Skip to content. | Skip to navigation

Personal tools
You are here: GES DISC Home Additional Features Technology Lab S4PM 5.26.0 RELEASE NOTES

S4PM 5.26.0 RELEASE NOTES

Release Date

April 30, 2008

Functional Changes

  1. Bug 6538 Fixed: Stationmaster FAILED.WORK_ORDERS; Duplicate Work Orders - Now checks to see if candidate work orders are files, thus ignoring RUNNING and FAILED directories.

  2. Bug 6550 Fixed: Failed to Query Data Type With Coverage Great Than 24 Hours - Added "&overlap" to MRI query to invoke an overlap style query.

  3. Bug 6634 Fixed: Allocate Disk Job Should Fail If Database Not Readable - A modification was made so that an anomaly is raised (station turns yellow) if the Allocate Disk database file is unreadable.

  4. Bug 6664 Fixed: Make cfg_max_failures Job-Specific - $cfg_max_failures can now be set to an anonymous hash instead of a scalar, like: $cfg_max_failures = {'ABLE' => 1} in which case it will stop processing ABLE jobs when a failure for that job_type is in the station, but continue processing other job_types.

  5. Bug 6738 Fixed: Upgrade S4PM System to be Capable of Handling Multi-File Granule as Trigger - s4pm_run_algorithm.pl handles multiple multi-file granules within the same LUN

  6. Bug 6861 Fixed: Use of $use_legacy_dataversion Prevents Environment Variables From Being Set in a Station - This bug prevented station configuration files from having their own environment variables defined whenever the $use_legacy_dataversion option was set. This has been fixed.

  7. Bug 6863 Fixed: s4pa_mri_query Dataset Version Problem - Added 'version_string' attribute to configuration hash to override value of version for MRI requests.

  8. Bug 7138 Fixed: Stringmaker Log File in Multi-User Mode Not Writable - A bug prevented Stringmaker from being run by multiple users (in multi-user mode) due to permissions on the Stringmaker log file. This has been fixed.

  9. Bug 7174 Fixed: Stringmaker Configuration File Permissions Problems in Multi-User Mode - In multi-user mode, the s4pm_configurator.cfg file in the Configurator station gets created with incorrect permissions. This was fixed.

  10. Bug 7175 Fixed: Multiple UPDATE_POOLS Work Orders Cause Incorrect Disk Allocations - This bug occurred when Stringmaker was run more than once resulting in multiple UPDATE_POOLS work orders in the Allocate Disk station. The unintended consequence was that each such work order augmented the sizes of the disk pools. The fix was to make the UPDATE_POOLS work order file names non-unique so that each overwrote any preexisting work order.

  11. Bug 7176 Fixed: Annoying Resize in Tkstat GUI Status Bar for Long Work Order Names - Replaced homemade status bar with Tk::Balloon.

  12. Bug 7270 Fixed: DATA Directory Has Wrong Permissions in Multi-User Mode - A bug that caused the DATA directory to have incorrect permissions in multi-user mode has been fixed.

  13. Bug 7289 Fixed: Allocate Disk Database Permissions Problems in Multi-User Mode - A bug in the S4PM Admin tool caused the permissions on the Allocate Disk database file to be incorrect. This has been fixed.

  14. Enhanced Support for Multi-File Granules - Several enhancements have been made:

    1. The %files_per_granule hash has been added to the Stringmaker data types configuration file. See example within s4pm_stringmaker_datatypes.cfg.

    2. Because of the above enhancement, the PCF template no longer is used to tell S4PM that a data type is a multi-file granule. You do this in the Stringmaker algorithm configuration file. In previous releases, S4PM got this information from the template when an output entry was repeated more than once.

    3. Multiplicity of PCF entries due to multi-file granules versus time coverage is now disambiguated. That is, you can have both. The number of entries for a particular LUN is the number of outputs needed to cover the time period times the number of files per granule. It is up to the algorithm to figure out how to use the output PCF entries provided (e.g. it needs to know how to interleave the individual files per granule).

    4. Chaining is fully supported with multi-file granules. On input, each file of a multi-file granule will be available in the PRODUCT INPUT section of the PCF.

  15. Station Anomaly Handling - Stationmaster (or any other program) can now raise anomalies, by writing to files in the station directory of the form ANOMALY-something.log. These are error or warning conditions that are not serious enough to warrant shutting down the station, but should be looked at. The drill-down interface in tkstat.pl (TkJob) will now recognize such files (by the filename) and present the user with the option to Clear Anomaly, at which point it will delete the anomaly file. Current anomalies raised by stationmaster are:

    1. MAXFAIL: failed jobs >= $cfg_max_failures

    2. SLOW_FS: filesystem slow, detected by unexpected zero-length work orders or duplicate work orders

    3. BAD_CONFIG: a work order of unrecognized job_type was found and moved to FAILED.WORK_ORDERS

    4. BAD_PERM: bad permissions, recognized by failure to write a virtual job work order

    5. TOO_QUIET: station has gone too long without receiving a work order (configured via $cfg_max_interval)

  16. Fixes and Enhancements to the S4PM Monitor (Tkstat):

    1. The Success column header has been renamed OK to save space.

    2. Bug 6433 (tkstat enhancement request: display job count in a station): the number of pending jobs and max_children are now displayed.

    3. The counter reset button has been moved to the bottom of the window.

    4. The station button turns yellow if any ANOMALY files are detected in the station.

    5. The drill-down S4PM Job Monitor has a new button to Clear Anomaly.

    6. A station "status bar" shows the anomaly filename (if it exists) when mousing over the station button. If there are more than one, it shows one of them with an ellipsis (...).

    7. The "you are here" bar for jobs has been shortened to be more aligned with the job grid, and allow the station status bar.

    8. Bug 6665 Fixed: Enable Cut and Paste in Tkstat Dialogs - Dialog boxes have been changed from labels to scrolled read-only text areas.

  17. Tkargus Refresh - Changed refresh behaviour of tkargus.pl to do it in small chunks, not all at once, so that the interface will remain responsive. Also added the ability to refresh one particular machine on demand.


  18. Automatic Backup of Stringmaker Configuration Files - When Stringmaker is run, all Stringmaker configuration files are saved in a tar file automatically. This allows one to more easily roll back to a previous configuration if something goes horribly wrong.


  19. New Stringmaker Option to Not Generate Inactive Stations - A new option has been added to Stringmaker to only generate active stations used in a string and not generate the inactive ones. The default is for Stringmaker to generate all possible stations with a configuration parameter in the station.cfg file ($cfg_disable) to tell Stationmaster whether or not the station is active. With the -h option, inactive stations are not generated in the first place.

Detailed File Changes

S4P.pm

  • Anomalies: Added raise_anomaly() and clear_anomaly().

S4P::PDR.pm

  • If PDR parsing fails, reports which FILE_GROUP it failed on (starting with 1).

S4P::S4PTk.pm

  • Bug 6665 - Made use of a Scrolled Read-Only Text area instead of a label for both S4P::S4PTk::redirect_logger and S4P::S4PTk::confirm.

S4P::TkJob.pm

  • Add Clear Anomaly button.

 

S4PM::DataRequest

  • Bug 6550: Failed to query AUREPHMF - Added "&overlap" to Machine Request Interface (MRI) query to invoke an overlap style query.
  • Bug 6863: s4pa_mri_query dataset version problem - Added 'version_string' attribute to configuration hash to override value of version for MRI requests.

s4pm_allocate_disk.pl

  • For Bug 6634, added a check to make sure that the database file is readable. An anomaly is raised and a message is written to the log file if it is not.

s4pm_export.pl

  • Multi-file Granules: in LocalGranuleID/symlink mode, now symlinks to metadata file of multi-file granules (only) instead of dying as it used to.

s4pm_prepare_run.pl

  • For Bug 6738, added code to make use of the new files_per_granule attribute available for output data types. Changes were made where the OUTPUT PRODUCTS section of the PCF are written to read in the %cfg_files_per_granule hash value for the data type (if available) from the s4pm_prepare_run.cfg file and apply it when determining the number of output entries to have in the runtime PCF. Thus, a LUN can now support both multiple entries to cover the time period and the multiple files per granule at the same time. It is up to the algorithm, however, to make proper use of the PCF entries available.

s4pm_run_algorithm.pl

  • Changed EXPORT work order to resolve the file paths of the files making up multi-file granules.
  • Look for output metadata files first in the pathname specified by the PCF, then start guessing names if not found (used to ignore the PCF entry)

s4pm_run_test.pl

  • Modified subroutine guess_dir() which tries to determine the location of S4PM executables. It had been including s4pm_stringmaker_derived.cfg. But in some installation sites, configuration files may be installed into different directories from code files. So, it was left out.

s4pm_stringmaker.pl

  • Added subroutine mk_backout_tar() to save and tar up all Stringmaker configuration files with a time-stamped file name.
  • Added code to prune away inactive stations from the %stations hash and unneeded configuration files from the %config_files hash if the -h option is invoked.
  • Added subroutine make_kit() to eventually support S4PM dynamic load balancing. As step in this process is to enable the generation of an S4PM "kit", a self-consistent tar ball containing a particular S4PM string configuration, all its stations and the S4P/S4PM code included. This functionality is not complete and therefore, the -k option should not be used yet.
  • For Bug 7138, added a call to change_mode() after closing the log file so that permissions will be set appropriately.
  • For Bug 7174, added a call to change_mode() after closing the configuration file so that permissions will be set appropriately.
  • For Bug 7175, removed the process ID from the UPDATE_POOLS work order file name. This makes all such work orders have the same file name and each new one will overwrite any preexisting one. Also, made sure that this work order had the correct permissions in multi-user mode.
  • For Bug 7270, added a line to adjust the permissions of the DATA directory if in multi-user mode and the DATA directory is not a symbolic link to some pre-existing disk area.

s4pm_stringmaker_datatypes.cfg

  • In support of multi-file granules in algorithm chaining (Bug 6738), added description of the new %files_per_granule hash that can optionally be used to set the number of files per granule for a data type.

s4pm_stringmaker_derived.cfg

  • For Bug 6861, changed settings of environment variable S4P_LEGACY_DATAVERSION to use a ||= (or equal) rather than a simple = (equal), thus, avoiding overwriting any other environment variables that may have been set.
  • Modified station configurations so that $cfg_root is set to a relative path (e.g. ../..) rather than an absolute path unless the station is a symlink.
  • Modified settings in the Acquire Data station to only create symlinks between it and the Request Data station if both stations are enabled (e.g. $cfg_disable = 0).

s4pm_stringmaker_static.cfg

  • In support of Bug 6738, added code to read in the %files_per_granule hash from the Stringmaker data types configuration file and write it as %cfg_files_per_granule in the s4pm_prepare_run.cfg file in the Prepare Run station.

s4pm_tk_admin.pl

  • For Bug 7289, modified subroutine ResetDatabases(). First, changed system() calls to S4P::exec_system() calls. Second, changed invocation of cpooldb.pl to s4p_pooldb.pl instead as the latter one is more robust and it supports multi-user mode.

stationmaster.pl

  • Raise anomalies for several non-fatal conditions (see Functional Changes)
  • Move max_failure checking to its own routine, check_max_failures, and support job-type-specific max_failure settings
  • Bug 6538 Fixed: S4P stationmaster FAILED.WORK_ORDERS; duplicate workorders in S4PA - Check to see if candidate work orders are files, thus ignoring RUNNING and FAILED directories.
  • Added ActivityMonitor to raise TOO_QUIET anomaly if $cfg_max_interval seconds go by without a work order

tkargus.pl

  • Change to refresh behaviour (required extensive rewrite).
  • Recognizes 'localhost' as a valid machine, and will skip ssh in that case.

tkstat.pl

  • Significant layout changes (see Functional Changes).

 

 

Document Actions
NASA Logo - nasa.gov
NASA Privacy Policy and Important Notices
Last updated: Dec 08, 2010 12:06 PM ET
Top