S4PM 5.15.0 RELEASE NOTES

Release Date

August 29, 2006

Functional Changes

1. Sweep Data Fix - A fix was made so that Sweep Data work orders containing URLs (http or ftp) don't fail. This only affected strings getting data from S4PA systems.
   
   
2. Better Support For Simple Installations - There is now better support for installations of S4PM on machines that are not automounting home directories. In Stringmaker, the setting of $global_root is now optional and defaults it to $s4pm_root. Further, if $ecs_root and $ingest_root parameters are omitted as well, the entire installation will be contained under the single $s4pm_root directory.
   
   
3. Updated TEST Package - The S4PM_TEST package was updated to take advantage of the simpler, single root directory installation capability.
   
   
4. Support For Both Input From and Export To S4PA - In the previous release, the %s4pa_data_map was used to define where S4PA input data were located (needed by the Compose Request Tool). This hash has now been extended to also define where products destined to be exported to an S4PA go.
   
   
5. Full Incorporation of S4PM-DME - This is the first release that fully incorporates and includes the Data Mining Edition of S4PM or S4PM-DME. Documentation on setting up and configuring DME, however, is still lacking. In addition to the four S4PM packages already included (S4PM, S4P, S4PM_CFG, S4PM_TEST), there are four new packages for DME: S4PMDM, S4PMDM_CFG, S4PMDM_CGI, and S4PMDM_HTML).
   
   
6. Full Support for Getting Data via PDR - This release has been tested successfully with the MODIS LAADS system, an external provider of data via PDR. The station that polls for the PDRs is the Poll PDR station. Testing has been successful in both the forward and reprocessing modes (that is, ordering via the Compose Data Request tool).

Detailed File Changes

S4P::FileGroup.pm

• Replaced the Net::SSH2 with system sftp call in sub ssh_get. This change will greatly improve the performance of sftp download and simplify the ssh key exchange issue. User should either setup sftp alias or ~/.ssh/config for special ssh ProxyCommand to get through the firewall if necessary.

S4P::MetFile.pm

• Change to xml2odl() so that it uses the embedded ODL object if available for making the ODL file (bug 68).

s4p_remote_polling_pdr.pl

• *Switched to using command line sftp from Net::SSH2.
• Replaced the command line switch, "-h", with "-r"; "-h" is now used to print out comand syntax.
• Added an additional switch, "-f", so that command parameters can be specified in a configuration file. The configuration file can also include an additional item, %cfg_data_version, for use in validating support for datasets/versions in a PDR.

S4PM::AlgorithmIO.pm

• In subroutine data_version(), dropped the formatting of the data version to exactly three characters.

S4PM::QC.pm

• Fixed call to S4P::logger. It had been S4P::looger.

s4pm_acquire_data.pl

• Added the -w option for "Winnowing" mode. This causes Acquire Data to discard FILE_GROUPs from PDRs for datatypes that are not found in the allocation database. (This will produce an INFO message, though.)
• Added logic to detect when remote machine is on local network and bypass firewall settings. Also removed s4pm user's settings of environment variable, and added settings to the startup for s4pm_tk_compose_request.pl. However, for next release, we should move the same logic from s4pm_acquire_data.pl into the PDR module so that it can be used by s4pm_tk_compose_request.pl as well (Bug 106).
• Added fix so that an appropriate UR is assigned for files coming in via PDR via the Poll PDR station.

s4pm_run_test.pl

• This is part of the S4PM_TEST package. Updated to take advantage of the simpler installation support that allows an entire string to be under one root directory.

s4pm_select_data.pl

• In subroutine validate_work_order(), removed the contraint on data type versions that they be three characters in length.

s4pm_select_data_handles.pl

• In subroutine validate_work_order(), removed the contraint on data type versions that they be three characters in length.

s4pm_stringmaker.pl

• Fix to sub update_configurator() to use the parameter $stringmakerdir rather than $stringmaker_root for finding the Stringmaker String configuration file. Without this fix, the parameter $stringmaker_root must be set. With, it can be inferred.

s4pm_stringmaker_datatypes.cfg

• Documented the extended use of the %s4pa_data_map parameter to define destination non-ECS archives as well as input archives.

s4pm_stringmaker_datatypes.template

• This is part of the S4PM_TEST package. It was synced up with the latest s4pm_stringmaker_datatypes.cfg file.

s4pm_stringmaker_derived.cfg

• Fix in the construction of the s4pm_export.cfg file to support output to multiple archives.
  *Added appropriate handling for sites not running ECS. The $ecs_root and $ingest_root are now handled with reasonable defaults.
• Modified so that if the $global_root parameter is not set, it defaults to $s4pm_root and this results in the complete S4PM string being under one root directory, namely $s4pm_root, rather than split with some stations being under the user's home directory. This is apprpropriate for sites only deploying a single string on a box.
• Enabled subscription request interface for DME in Acquire Data station. Note that the interface itself for S4PA has not yet been written.
• Added fix so that production summary data types would be omitted from being included in QC checks and from the generated QC.cfg file.
• Include the -w option for running s4pm_acquire_data.pl when PDR polling is used.
• Fixes to the configuration of the Poll PDR station.
• Fixed a problem with algorithms generating production summaries not getting the needed -f option on the s4pm_run_algorithm.pl command line.
• To handle the extended use of the %s4pa_data_map hash, the building of the s4pm_export.cfg file in the Export station was heavily modified as well as cleaned up. Modifications were also made in the Receive PAN station so that there was an entry for each destination configured in the Export station.
• Modified command line call to s4p_remote_polling_pdr.pl in the Poll PDR station. The -h option was changed to -r reflecting a change in the script's interface.

s4pm_stringmaker_global.cfg

• Cleaned up documentation on the $stringmaker_root parameter.

s4pm_stringmaker_global.template

• This is part of the S4PM_TEST package. It was synced up with the latest s4pm_stringmaker_global.cfg file.

s4pm_stringmaker_host.cfg

• Cleaned up documentation on $toolkit_root and $ecs_root parameters. Documented that the $ecs_root and $ingest_root parameters are now optional and appropriate defaults for sites not running ECS.

s4pm_stringmaker_host.template

• This is part of the S4PM_TEST package. It was synced up with the latest s4pm_stringmaker_host.cfg file.

s4pm_stringmaker_string.template

• This is part of the S4PM_TEST package. It was synced up with the latest s4pm_stringmaker_string.cfg file.

s4pm_sweep_data.pl

• Modified code that attempts to determine the data type from the .ur file. The code now tries first to get data type from .ur file, but if that fails, it gets it from the file name itself.
• Also, modified sub compare_urs() so that it works for URs that are FTP or HTTP URLs as well as ECS-style URs. Also, if the UR in a .ur file fails parsing, rather than the script failing, it simply notes that the accompanying request or acquire stub file won't get cleaned out.

s4pm_sweep_data_handles.pl

• Modified sub compare_urs() so that it works for URs that are FTP or HTTP URLs as well as ECS-style URs. Also, if the UR in a .ur file fails parsing, rather than the script failing, it simply notes that the accompanying request or acquire stub file won't get cleaned out.

s4pm_tk_admin.pl

• Added oldlist.ext to the skip list in subroutine skip() since this file is produced by the Poll PDR station.

stationmaster.pl

• Stationmaster outputs to the log file the command being run as set in the %cfg_commands hash in the station.cfg file for the station. But when the work order type for a command included wildcard characters (e.g. '*') for the hash key, the echoing of the command to the log file did not work. This was fixed so that the command is written to the log file even for work order types with wildcard characters.