Skip to content. | Skip to navigation

Personal tools
You are here: GES DISC Home Additional Features Technology Lab S4PM 5.16.0 RELEASE NOTES

S4PM 5.16.0 RELEASE NOTES

Release Date

October 31, 2006

Functional Changes

  1. Development of S4PM Moved to CVS - All development of S4PM has been transitioned from SCCS to CVS.

  2. PAN Interface with Providers - Added ability to return SUCCESSFUL PANs back to non-ECS data providers.

  3. Fix to Poll PDR Station - A fix to handle remote directories containing a large number of files was made to the Poll PDR station.

  4. Auto Acquire/Request Turbo Mode - Turbo mode (-t) causes Auto Request to skip over data gaps faster than the fixed X rate would ordinarily allow.

  5. Fill Hole Tool Update - A string can now be configured so that the Fill Hole ordering tool includes all the input data types, not just those not assigned to the INPUT disk pool. This is done via the new include_all_input attribute in the %ordering_tool_parms hash (normally set in the Stringmaker String configuration file).

  6. Remove Job Failsafe - The remove_job.pl script checks to see if a station directory is on the chopping block and errors out if so.

  7. Stage For Pickup Location Fixed - The Stage For Pickup station (used primarily for DME) was hardwired to be under $HOME. The proper directory location is the directory set by the $global_root parameter in the Stringmaker Global configuration file which does default to $HOME.

  8. Custom Find Data Support - Support was added to Find Data that allows custom commands or scripts to be run for particular input LUNs. Custom find is set up via the %custom_find hash in the s4pm_stringmaker_algorithm.cfg file. When invoked, Find Data relies on the custom script to return the pathname of a data file found. To support the custom script, a parameter file is created containing the values of parameters normally used by Find Data itself (e.g. data directory, the processing start and stop time, the input data's currency and boundary). It is up to the custom script to use this parameter file (or not) and whatever other means it has at its disposal to locate an input file. As a reference implementation, the script s4pm_find_by_coverage.pl is included. This script examines the metadata files in the data directory for the first one whose data coverage overlays the processing period. If such a metadata file is found, it returns the full pathname of the data file; otherwise it returns nothing. See Stringmaker Algorithm configuration file man page for more details on this new option.

  9. Simplified String Configurations - For simple S4PM strings, the directory structure has been made even more simple. By setting only the $s4pm_root parameter (and leaving out $ingest_root, $ecs_root, $global_root, and even $instance), the entire string is built under a single root directory. Other simplications were made too: The $ingest_root paramter now specifies where the PAN, PDR, and DATA directories go. To have the DATA directory somewhere else, use $data_root. $data_root can be used in conjuction with $ingest_root or not. The $instance parameter is now optional also.

Detailed File Changes

All Source Files

  • CVS tags were inserted into the tops of all source files.

S4P.pm

  • Modified remove_job() to check for presence of a station.cfg file before removing, to eliminate inadvertant station cleanups.

S4P::TimeTools.pm

  • Modified S4P::TimeTools::CCSDSa_DateCompare so that it truncates off any fractional seconds before doing the comparison. Without this, for example, time strings 2006-12-14T04:02:00Z and 2006-12-14T04:02:00.000000Z were not seen as equal.

S4PMDM.pm

  • This module supports the S4PM Data Mining Edition. Added $data_root parameter so that it appears in the work order that initiates the creation of a new user string.

s4pm_acquire_data.pl

  • Clean up of the prolog documentation.
  • Added -P option to return SUCCESSFUL PANs to the URL given in the ORIGINATING_SYSTEM attribute of the PDR work order.

s4pm_auto_request.pl

  • Added turbo mode (-t) to skip over data gaps faster than the fixed X rate would ordinarily allow.
  • Fixed a logic error encountered when there is a data gap.
  • Modified to truncate fractional seconds in begin time to make the bins come out evenly.

s4pm_find_data.cfg

  • Added documentation on the new %cfg_custom_find hash. See s4pm_find_data.pl for details.

s4pm_find_data.pl

  • Added a new optional parameter to the Find Data configuration file: %cfg_custom_find, that allows custom commands or scripts to be associated with particular LUNs. The algorithm in the custom script replaces the default methods used in s4pm_find_data.pl. To implement this, the new run_custom_find() subroutine was added and modifications were put into get_data_pathname() as well as minor changes in other areas.

s4pm_find_by_coverage.pl

  • Reference implementation of the new custom find capability in Find Data. This custom find looks at the coverage time in the metadata to determine if the data fully covers the processing period. It returns the path for the first one that passes the coverage test.

s4pm_remote_polling_pdr.pl

  • Fixed bug that caused a null return when attempting to get back a list of files in the remote directory via ftp. Modified code so that through ftp, a cd into the relevant directory is performed first before querying for the list of files it contains. Also, rather than querying via 'ls', 'dir' is used instead. The results are then parsed.
  • Added -P <URL> option to embed a PAN destination in the ORIGINATING_SYSTEM attribute of the output PDR work orders.

s4pm_run_algorithm.pl

  • Added a new option, -a node_name, to explicitly set the value of the node name in PDR work orders going from Run Algorithm to the Export station. The default is to use S4P::PDR::gethost().

s4pm_run_test.pl

  • This script, part of the S4PM_TEST package, was updated to accommodate the simpler string configuration available via Stringmaker in this release.

s4pm_stringmaker.pl

  • Removed references to $ENV{'HOME'} and replaced with $global_dir, a parameter set in the Stringmaker Derived configuration file. This change only affects DME.
  • Modified subroutine update_configurator() so that it uses $stringmaker_root rather than $stringmakerdir as the location of the s4pm_configurator.cfg file.
  • Added support for reading in the optional %custom_find parameter from the Stringmaker Algorithm configuration files and using it to build up a %all_custom_find hash to be used by s4pm_stringmaker_derived.cfg.

s4pm_stringmaker_algorithm.cfg

  • Added documentation and examples for the new %custom_find option that allows custom scripts or commands to be run for inputs of a particular LUN.

s4pm_stringmaker_derived.cfg

  • Modified how the @fillhole_datatypes array is populated depending upon whether or not the 'include_all_input' was set in the %ordering_tool_parms hash.
  • Modified the Stage For Pickup station so that it is placed under a $globaldir and not hardwired to the user's home directory.
  • Refactored for simplifying how various options play together. In the simplest configuration (where only $s4pm_root is set and not $ingest_root, $ecs_root, or $global_root), the entire string is built under a single directory. Adding $ingest_root causes the DATA, PAN, and PDR directories to be under $ingest_root. The $instance paramter is now optional. To have PAN and PDR under $ingest_root and DATA somewhere else (as we need for ECS), the $data_root parameter needs to be set. I also added some checking so that if $ecs_root is not set, links to things like the ECS acquire script and SCLI.log are not created.
  • Changed warning message content for when $external_archive_system is set to either 's4pa' or 'both', yet the %s4pa_data_map hasn't been defined. The warning had been misworded slightly.
  • Added support for the new %all_custom_find hash that s4pm_stringmaker.pl provides to add an optional %cfg_custom_command hash to the s4pm_find_data.cfg file in the Find Data station.
  • Modified the Run Algorithm section so that if $host_alias is defined, the new -a option is included when s4pm_run_algorithm.pl is invoked.
  • In Acquire Data station, configured the %cfg_commands hash for two work order types only: 'ACQUIRE_DATA' and 'DN.*'. The previous 'D*.{PDR,wo}' was removed in favor of the 'DN.*'.
  • In Poll PDR, modified %cfg_commands so that if 'pan_url' is set in the %pdr_polling_parms, the call to s4pm_remote_polling_pdr.pl includes the -P option for making PANs. Also, in this case, the work order generated is POLL_ACQUIRE_DATA rather than ACQUIRE_DATA which goes to the Acquire Data station. Also, modified the Acquire Data station confguration so that the POLL_ACQUIRE_DATA work order is set in the %cfg_commands hash and includes the -P option on the s4pm_acquire_data.pl script.

s4pm_stringmaker_host.cfg

  • Modified documentation on the $ingest_root parameter and how it works (see 3rd item under s4pm_stringmaker_derived.cfg).
  • Modified comments on the $host_alias parameter to include the fact that the alias is applied to the PDRs generated by the Run Algorithm station in the NODE_NAME attribute as well as to the PDRs that Acquire Data sees.

s4pm_stringmaker_host.template

  • Updated to stay in sync with changes in the s4pm_stringmaker_host.cfg file. This template file is part of the S4PM_TEST package.

s4pm_stringmaker_string.cfg

  • Modified the description of the %ordering_tool_parms hash to include the new 'include_all_input' attribute.
  • Modified documentation on the $instance paramter since it is now optional.
  • Added a new attribute to the %pdr_polling_parms hash, pan_url, which if set enables PAN generation by the Poll PDR station. It is actually the Acquire Data station that sends the PAN to the data provider if data are successfully acquired.

s4pm_stringmaker_string.template

  • Updated to stay in sync with changes in the s4pm_stringmaker_string.cfg file. This template file is part of the S4PM_TEST package.

stringWO.cfg

  • This file is part of the S4PM Data Mining Edition. Added $data_root_base path information used to build string work order for Stringmaker.
Document Actions
NASA Logo - nasa.gov
NASA Privacy Policy and Important Notices
Last updated: Dec 08, 2010 11:06 AM ET
Top