August 29, 2006
Sweep Data Fix - A fix was made so that Sweep Data work orders containing URLs (http or ftp) don't fail. This only affected strings getting data from S4PA systems.
Better Support For Simple Installations - There is now better support for installations of S4PM on machines that are not automounting home directories. In Stringmaker, the setting of $global_root is now optional and defaults it to $s4pm_root. Further, if $ecs_root and $ingest_root parameters are omitted as well, the entire installation will be contained under the single $s4pm_root directory.
Updated TEST Package - The S4PM_TEST package was updated to take advantage of the simpler, single root directory installation capability.
Support For Both Input From and Export To S4PA - In the previous release, the %s4pa_data_map was used to define where S4PA input data were located (needed by the Compose Request Tool). This hash has now been extended to also define where products destined to be exported to an S4PA go.
Full Incorporation of S4PM-DME - This is the first release that fully incorporates and includes the Data Mining Edition of S4PM or S4PM-DME. Documentation on setting up and configuring DME, however, is still lacking. In addition to the four S4PM packages already included (S4PM, S4P, S4PM_CFG, S4PM_TEST), there are four new packages for DME: S4PMDM, S4PMDM_CFG, S4PMDM_CGI, and S4PMDM_HTML).
Full Support for Getting Data via PDR - This release has been tested successfully with the MODIS LAADS system, an external provider of data via PDR. The station that polls for the PDRs is the Poll PDR station. Testing has been successful in both the forward and reprocessing modes (that is, ordering via the Compose Data Request tool).
Detailed File Changes
- Change to xml2odl() so that it uses the embedded ODL object if available for making the ODL file (bug 68).
*Switched to using command line sftp from Net::SSH2.
Replaced the command line switch, "-h", with "-r"; "-h" is now used to print out comand syntax.
Added an additional switch, "-f", so that command parameters can be specified in a configuration file. The configuration file can also include an additional item, %cfg_data_version, for use in validating support for datasets/versions in a PDR.
- Fixed call to S4P::logger. It had been S4P::looger.
Added the -w option for "Winnowing" mode. This causes Acquire Data to discard FILE_GROUPs from PDRs for datatypes that are not found in the allocation database. (This will produce an INFO message, though.)
Added logic to detect when remote machine is on local network and bypass firewall settings. Also removed s4pm user's settings of environment variable, and added settings to the startup for s4pm_tk_compose_request.pl. However, for next release, we should move the same logic from s4pm_acquire_data.pl into the PDR module so that it can be used by s4pm_tk_compose_request.pl as well (Bug 106).
Added fix so that an appropriate UR is assigned for files coming in via PDR via the Poll PDR station.
Fix to sub update_configurator() to use the parameter $stringmakerdir rather than $stringmaker_root for finding the Stringmaker String configuration file. Without this fix, the parameter $stringmaker_root must be set. With, it can be inferred.
- Fix in the construction of the s4pm_export.cfg file to support output to multiple archives.
*Added appropriate handling for sites not running ECS. The $ecs_root and $ingest_root are now handled with reasonable defaults.
- Modified so that if the $global_root parameter is not set, it defaults to $s4pm_root and this results in the complete S4PM string being under one root directory, namely $s4pm_root, rather than split with some stations being under the user's home directory. This is apprpropriate for sites only deploying a single string on a box.
- Enabled subscription request interface for DME in Acquire Data station. Note that the interface itself for S4PA has not yet been written.
- Added fix so that production summary data types would be omitted from being included in QC checks and from the generated QC.cfg file.
- Include the -w option for running s4pm_acquire_data.pl when PDR polling is used.
- Fixes to the configuration of the Poll PDR station.
- Fixed a problem with algorithms generating production summaries not getting the needed -f option on the s4pm_run_algorithm.pl command line.
- To handle the extended use of the %s4pa_data_map hash, the building of the s4pm_export.cfg file in the Export station was heavily modified as well as cleaned up. Modifications were also made in the Receive PAN station so that there was an entry for each destination configured in the Export station.
- Modified command line call to s4p_remote_polling_pdr.pl in the Poll PDR station. The -h option was changed to -r reflecting a change in the script's interface.
Modified code that attempts to determine the data type from the .ur file. The code now tries first to get data type from .ur file, but if that fails, it gets it from the file name itself.
Also, modified sub compare_urs() so that it works for URs that are FTP or HTTP URLs as well as ECS-style URs. Also, if the UR in a .ur file fails parsing, rather than the script failing, it simply notes that the accompanying request or acquire stub file won't get cleaned out.
Modified sub compare_urs() so that it works for URs that are FTP or HTTP URLs as well as ECS-style URs. Also, if the UR in a .ur file fails parsing, rather than the script failing, it simply notes that the accompanying request or acquire stub file won't get cleaned out.
Stationmaster outputs to the log file the command being run as set in the %cfg_commands hash in the station.cfg file for the station. But when the work order type for a command included wildcard characters (e.g. '*') for the hash key, the echoing of the command to the log file did not work. This was fixed so that the command is written to the log file even for work order types with wildcard characters.