Skip to content. | Skip to navigation

Personal tools
You are here: GES DISC Home Additional Features Technology Lab S4PM 5.7.0 RELEASE NOTES

S4PM 5.7.0 RELEASE NOTES

Release Date

July 25, 2005

Functional Changes

  1. Parameter @all_external_datatypes Dropped - In the Stringmaker string-specific configuration file, the array @all_external_datatypes no longer needs to be specified. Instead, Stringmaker will compute it and ignore the @all_external_datatypes parameter in the string configuration file.

  2. New Parameter for Controlling Privileged Users - Access control lists for privileged users is handled differently. Rather than hardwiring the names of these users in the Stringmaker configuration files, instead a @privileged_users array is set in the Stringmaker global configuration file (s4pm_stringmaker_global.cfg) or actually in any configuration file if desired.

  3. Additional Consistency Checking Added in Stringmaker - Consistency checking has been added to Stringmaker so that if some parameters are specified or set, Stringmaker will let you know if you have failed to also include dependent parameters (e.g failed to specify the staging directory if you specified some data types as going to the data pool).

  4. Modify Max Jobs Tool Enhanced - In previous releases, the Stringmaker jobs configuration file (s4pm_stringmaker_jobs.cfg) had to be left checked out of SCCS or RCS (i.e be writable) in order for the Modify Max Jobs tool to work. In this release, it can be left checked in. Arbitrary methods for checking in and checking out files can be specified in the s4pm_configurator.cfg file. The Modify Max Jobs tool will automatically apply these methods if they are set.

  5. New Parameter For Data Expiration Times Introduced - In previous releases, the expiration times for all data types had to be specified in the Stringmaker string-specific configuration file with a hash setting each data type individually. As a shortcut, this release allows $data_expiration_max_hours to be set once in the Stringmaker global configuration file and have it apply it to all data types. It is only useful if a single expiration time makes sense for all data types.

  6. New Parameters for Setting Ordering Tool Parameters - The time increments as shown in the Compose Data Request tool and the Fill Hole tool are now configurable via the %ordering_tool_parms hash. If not set, it defaults to 7200 seconds for the large increments in the Compose Data Request tool and 300 seconds for the small increments in the Fill Hole tool.

  7. SimplifiedFile Accumulation Production Rule - Added a new parameter to the parameters available in the Stringmaker algorithm configuration file: %file_accumulation_parms. This single hash specifies everything that is needed for implementing the file accumulation production rule and replaces the previous multi-step "recipe" that was needed before. In 5.7.0, the new %file_accumulation_parms and the old "recipe" are both supported, but the old recipe should be considered deprecated.

  8. Automated Update of Configurator Configuration File - Release 5.6.2 required you to update or create the s4pm_configurator.cfg file in the Stringmaker root directory manually. In this release, the file is created or updated automatically.

  9. Configurable S4PM File Names Introduced - This release introduces configurable S4PM file names via the $s4pm_filename_pattern parameter in the Stringmaker data types configuration file (s4pm_stringmaker_datatypes.cfg). If set, the environment variable S4PM_CONFIGDIR must also be set to the location of the Stringmaker directory (which is equal to the $stringmaker_root parameter in the Stringmaker global configuration file). The default is the standard S4PM file name pattern.

  10. S4PM-DME E-Mail Parameter Added - In S4PM-DME, an e-mail notification can optionally be sent to notify the recipient that a new Data Mining string has been created. This has been implemented via a -m email_address option on the s4pm_create_string.pl script.

  11. FIFO Sort Order in On-Demand Find Data - For On-Demand strings, the job sort order in the Find Data station was changed from its default (by algorithm) to FIFO (first in, first out). This has been demonstrated to help in keeping large orders moving through the system together leading to improved performance overall. (Trouble Ticket 15111)

Detailed File Changes

Algorithm.pm

  • Modified accumulation_start() so that it uses the processing start parameter when determining the start of an accumulation window.

  • Modified to read new optional parameter in the algorithm configuration files: %file_accumulation_parms.

Configurator.pm

  • Added the function get_machine_name() that returns the machine name for a particular string ID.

S4P::TimeTools.pm

  • Added a number of functions to support flexible file names in S4PM:

    S4P::TimeTools::CCSDStoy(), S4P::TimeTools::CCSDStoY(), S4P::TimeTools::CCSDStom(), S4P::TimeTools::CCSDStod(), S4P::TimeTools::CCSDStoH(), S4P::TimeTools::CCSDStoM(), S4P::TimeTools::CCSDStoS(), S4P::TimeTools::CCSDStoj(), S4P::TimeTools::CCSDStoB(), S4P::TimeTools::CCSDStob(), S4P::TimeTools::CCSDStou()

S4PM.pm

  • Added a number of functions to support flexible file naming within S4PM. The function S4PM::make_patterned_filename() will generate a file name when given a file name pattern, data type name and version, and a CCSDS date/time string. S4PM::parse_patterned_filename() will retrieve the data type name and version, and date/time string when given a pattern and a file name. S4PM::make_patterned_glob() will generate a glob pattern.

s4pm_allocate_disk.pl

  • Modified to use the new S4PM::parse_patterned_filename() routine.

  • Made error messages more clear when a data type could not be found in one of the Allocate Disk configuration files, particularly if $datatype evaluates to the empty string.

s4pm_create_string.pl

  • Added the -m option that allows an email notification to go to some recipient notifying them of the creation of a new DME string.

s4pm_derived.cfg

  • Changed sort order for Find Data station to FIFO in on-demand strings.

s4pm_find_data.pl

  • Modified to use the new S4PM::make_patterned_glob() routine.

s4pm_poll_data.pl

  • Commented out a S4P::perish() statement and a print statement that were causing the chain logs to grow to very large sizes.

s4pm_prepare_run.pl

  • Modified code to make use of the new S4PM::make_patterned_filename() routine. This allows the script to work with the new flexible file naming feature. Other code cleanup was done too.

  • Fix for Trouble Ticket 15093 causing failures when an algorithm in on-demand was configured to read its PSPEC file directly rather than indirectly via the PCF.

s4pm_purge_bad_data.pl

  • Modified code to make use of the new S4PM::make_patterned_filename() routine. This allows the script to work with the new flexible file naming feature.

s4pm_register_data.pl

  • Replaced called to S4PM::make_s4pm_filename() to S4PM::make_patterned_filename(), a function that allows flexible S4PM file names.

s4pm_regular_block.pl

  • Added support for new boundaries: START_OF_2HOUR, START_OF_4HOUR, START_OF_6HOUR, START_OF_12HOUR.

s4pm_request_data.pl

  • Bug fix for TT 15176. The fix makes sure that the data type name is part of the request stub file name so that later, when it comes time to delete the stub file, the Sweep Data station can find it more efficiently.

  • Added the -x option to the call to ssh when the SCLI is invoked on an external host. This disables X forwarding which was causing problems for DME.

s4pm_run_algorithm.pl

  • Cleaned up the call to S4PM::parse_s4pm_filename().

s4pm_run_easy.pl

  • Modified to call S4PM::make_patterned_filename() instead of S4PM::make_s4pm_filename().

  • Fixed a scoping bug that prevented the output tar option from working correctly.

s4pm_select_data.pl

  • Fixed a bug in the file accumulation production rule whereby the processing start time in the output work order was incorrect. This was fixed by having it make a call to Algorithm::accumulation_start() to determine the proper processing start time.

s4pm_ship_data.cfg

  • Removed the $ecs_mode parameter as the mode is now determined automatically.

s4pm_ship_data.pl

  • Removed need to specify mode in the s4pm_ship_data.cfg file and instead determine mode via S4PM::mode_from_path().

s4pm_static.cfg

  • Added code to place the $s4pm_filename_template parameter in the s4pm_register_data.cfg and s4pm_prepare_run.cfg configuration files.

s4pm_stringmaker.pl

  • Added extra checking of dependent parameters (e.g. if you specify data types in @datapool_insert_datatypes, then you need to set $datapool_staging_dir as well.

  • Added a subroutine to compute @all_external_datatypes so that there is no need to specify it in the string-specific configuration file.

  • Added code to read in the new optional %file_accumulation_parms hash.

  • Added get_punt_uses() function and a number of supporting functions to compute the uses that go into the s4pm_failed_pge_handler.cfg file. The uses in this file is the mechanism for balancing the books when a PGE is punted. The uses were computed via hardwired, MODIS-specific lines in the s4pm_stringmaker_derived.cfg file. This is now fully computed based upon mix of algorithms running in the string.

  • Added a subroutine to build the multi-string s4pm_configurator.cfg file. It backs up the old one first.

  • Added checking to be sure that the uses set in the algorithm configuration files were not zero or blank.

s4pm_stringmaker_derived.cfg

  • Minor correction to spelling of 'Uninstall Algorithm' interface in the Configurator station. It had been 'Uninstall Algorithm'.

  • Modified so that in the Insert Datapool station, the machine name on which to stage the data is passed along with the directory name in the $datapool_staging_dir parameter, rather than being hardwired.

  • Modified to use the new @privileged_users parameter rather than hardwiring the names of special privileged users.

  • Modified to key off of the setting for $domain to decide if the command for invoking the Modify Max Jobs window should have the -t (for submitting a Trouble Ticket via Remedy) or not. This is not very satisfying, but it addresses a current bug in 5.6.2 for users beyond the GES DAAC.

  • Modified the handling of increments in the Compose Data Request and Fill Hole tools. The larger increment was hardwired to 7200 and the smaller was set based on algorithm name (10 for an algorithm beginning with 'Ai' for example. This is still the default. However, the parameter hash %ordering_tool_parms can now override these defaults. It contains two attributes: 'increment' to set the larger increment and 'files_per_hour' to set the small.

  • Modified to use the new optional %files_accumulation_parms hash for implementing the file accumulation production rule.

  • Replaced hardwired, MODIS-specific (and not completely correct) code for computing uses that go into the s4pm_failed_pge_handler.cfg file with a line that sets it to the %punt_uses hash that is computed by s4pm_stringmaker.pl.

  • Removed setting of the $ecs_mode variable in the s4pm_ship_data.cfg file as it is no longer needed. Set the $max_item_count to 40 as default.

  • Fixed a bug in 5.6.2 whereby the stations list compiled by Stringmaker always included the sub_notify station regardless of whether it actually existed or not. If it didn't exist, this caused problems in bringing up the S4PM Monitor (Tkstat). The fix is to only include the sub_notify station in the stations list if it already exists.

  • Changed sort order for Find Data station to FIFO in on-demand strings.

s4pm_stringmaker_global.cfg

  • Added new parameter, @privileged_users, that contains the list of users that are allowed to execute some critical functions (e.g. release job now). This allows a better mechanism than we have now which is to hardwire the names of privileged users in the s4pm_stringmaker_derived.cfg file.

  • Added new parameter, $data_expiration_max_hours, that can be used to set the expiration time (in hours) of all data types at once. If used, this would alleviate the need to specify it in the Stringmaker string-specific configuration file for individual data types.

s4pm_stringmaker_static.cfg

  • Modified to use the new @privileged_users parameter rather than hardwiring the names of special privileged users.

  • Added code to place the $s4pm_filename_template parameter in the s4pm_register_data.cfg and s4pm_prepare_run.cfg configuration files.

s4pm_stringmaker_string.cfg

  • Updated the prolog on the use of the $datapool_staging_dir parameter.

  • Updated the prolog to remove description of @all_external_datatypes and example of its usage (since it is computed automatically now).

  • Reformatted the prolog to make it easier on outside users.

  • Added parameter and description of $s4pm_filename_template for setting the file name patterns in S4PM.

s4pm_tk_admin.pl

  • Added s4pm_configurator.cfg.backup to the skip patterns list.

  • Added code to remove NEWDATA*.log files from the Select Data station and TRIGGER*.log files from the Prepare Run station during Full Clean Out.

s4pm_tk_max_children.pl

  • Modified to remove the assumption that string IDs have the form: S4PMnn_XX_nn as we do at the DAAC. Now, any name will work. The script now makes use of the s4pm_configurator.cfg file to find out what machine the string is being run on.

  • Modified to make use of $cfg_checkout and $cfg_checkin parameters in the s4pm_configurator.cfg file. Thus, the script will handle the checking out and checking back in of the s4pm_stringmaker_jobs.cfg file. As a result, the s4pm_stringmaker_jobs.cfg doesn't have to be perpetually checked out.

  • Added the -t option which turns on the submission of a Trouble Ticket after update have been made. This means that users of S4PM outside of the GES DAAC aren't forced to contend with Remedy!

Document Actions
NASA Logo - nasa.gov
NASA Privacy Policy and Important Notices
Last updated: Dec 08, 2010 11:06 AM ET
Top