S4PM 5.17.0 RELEASE NOTES

Release Date

December 19, 2006

Functional Changes

1. Bug 298 Fixed: Metadata Not Found With Level 0 Input - A bug causing failures to find metadata files associated with Level 0 files has been fixed.
   
   
2. Bug 356 Fixed: Multi-File Granule Output Not Handled Correctly - This fix addresses a shortcoming in S4PM, namely that it didn't handle multi-file granule output from an algorithm. Multi-file granule output should now be handled including chaining, that is, where multi-file output is used as input to an upstream algorithm. The way to configure multi-file granule output is via the PCF template. Simply include multiple version entries in the PCF. Note that this cannot be combined with the multiplicity that S4PM already supports to make sure that there are enough output entries in the PCF (for a particular LUN) to cover the processing period. Only one type of multiplicity can be handled at a time for a LUN. A failure results in Prepare Run if you try to do both.
   
   
3. Bug 454 Fixed: Incorrect Logic in AIRS Pentad Production Rule - Bugs affecting the AIRS pentad production rule were fixed in this release.
   
   
4. Bug 495 Fixed: Added Support for Customized Production Summary Versions - In previous releases, the Production Summary was fixed to 001. In this release, it can be set via the data_version attribute in the %production_summary_parms hash in the Stringmaker Algorithm configuration file.
   
   
5. Bug 496 Fixed: S4PM Admin Tool Doesn't Clean Out All PDRs - A fix was made to the S4PM Admin Tool so that all PDRs are deleted when a full cleanout is run.
   
   
6. Further Progress In Supporting Native File Names - More work was done to lay the ground work for supporting native file names in S4PM via file handles, a concept in which S4PM refers to all data files via handle files. Handle files contain a list of files making up a granule (metadata file, XML file, and the data file or files themselves). The names of the individual data files within are native, that is, S4PM doesn't coerce them into any convention. Rather, only the handle files themselves adhere to a file naming convention for S4PM internal use. In prior releases, several scripts that specifically supported the file handle concept were added to the baseline with distinct file names (e.g. there was s4pm_find_data.pl and s4pm_find_data_handles.pl). In this release, these separate scripts have been merged back into the main scripts (e.g. s4pm_find_data.pl now handles both the file handle and the non file handle case). More work needs to be done and native file name support is not yet ready to be used.
   
   
7. Improved Error Handling in ODL Parser - Implemented additional error handling to detect unbalanced ODL trees.
   
   
8. Case Insensitive Search of ODL Tree - Implemented case insensitive search of ODL trees by setting a flag.
   
   
9. EDOS PDR and PAN Handling - Implemented PDRs and PANs with binary headers to support EDOS ingest to S4PA.
   
   
10. DME Installation Instruction - Modified the email instruction sent to dm_staff@listserv.gsfc.nasa.gov to allow easier creation of new data miner string.

Detailed File Changes

newUsersMail.pl

• This file is part of the S4PM_DM package. Modified instruction to reflect currrent g0dup05 data mining system.

S4P::FileGroup

• Implemented data_files() to return non-metadata files.

S4P::OdlBlock

• Implemented case insensitive search (CASE_INSENSITVE argument should be set to 1 in S4P::OdlTree:search()).

S4P::OdlGroup

• Implemented getNodeType() to return the node type (GROUP).

S4P::OdlObject

• Implemented getNodeType() to return the node type (GROUP).

S4P::OdlTree

• Implemented case insensitive search (CASE_INSENSITVE argument should be set to 1 in S4P::OdlTree:search()).

S4P::PAN

• Functions are provided to extract fields from an EDOS-style PDR to build an EDOS-style PAN. If the PAN description is "SUCCESSFUL", the file transfer disposition will be marked successful (0); if it contains CHECKSUM, the file disposition will be set to Checksum Mismatch (3); if METADATA, it will be set to Invalid PDS/EDS Construction Record Data (9); for all other values, it will be set to File Unreadable (8).

S4P::PCF.pm

• Added a method to obtain granules from a PCF lun, hashed on the corresponding metadata files. This supports the multi-file granule changes to s4pm_run_algorithm.pl (Bug 356).

S4P::PDR

• If an input file contains an EDOS-style PDR (begins with \x09\x00), the binary Ground Message Header is stripped from the front and stored as though it were a hex string for a new field called GROUNDMSGHDR. If a PDR is written that has a GROUNDMSGHDR, it is written as a SIPS-style PDR with this extra field which is recognized in subsequent reads. Functions are provided to fetch and store the value, and test if it is null.

S4PM::Algorithm.pm

• Fixes related to the START_OF_AIRS_PENTAD production rule. Small change to update_info() and test condition added to get_accumulation_start().

S4PM::DataRequest.pm

• Fix for Bug 298 causing s4pm_tk_compose_request.pl to fail periodically to find metadata files associated with Level 0 PDS files. The S4PM::DataRequest module was modified to look for the metadata file with the pattern P*1.PDS. The ECS convention has been to associate the metadata file with the first packet file and this convention will be adhered to with S4PA as well.

s4pm_allocate_disk.pl

• Fix for Bug 356. The code was modified so that if INSERT_MULTIDIRECTORY_HERE is found rather than INSERT_DIRECTORY_HERE, the directory assigned is a subdirectory under the disk pool directory, one named for the file name.

s4pmdm_SetUser.pl

• This file is part of the S4PM_DM package. Changed new call to s4pmdm_SetUser.pl -u $user -d $output_dir where $user = user login name and $output_dir = user data output directory (absolute path)

s4pm_find_data.pl

• The script s4pm_find_data_handles.pl was merged into s4pm_find_data.pl, a single script that handles *.ur files as file handles or as simple files containing only URs.
• Changed import of s4pm_allocate_disk.pl to use Safe instead of a straight eval.

s4pm_find_data_handles.pl

• Dropped from the baseline as a separate file; it was subsumed into s4pm_find_data.pl.

s4pm_prepare_run.pl

• Fix for Bug 356, incorrect PDR generated when output is multi-file granules. The problem was that s4pm_prepare_run.pl simply did not handle multi-file granule output. This functionality has been added in. If the template PCF lists multiple versions for a particular LUN, the script assumes that we are dealing with a multi-file granule. The normal INSERT_DIRECTORY_HERE placeholder placed in the output PCF is replaced with INSERT_MULTIDIRECTORY_HERE instead so that s4pm_allocate_disk.pl will know what to do. In addition, the output file names are tagged with the version numbers so that each file making up the multi-file granule gets a unique file name.
• Made configuration file optional.

s4pm_preselect_data.pl

• The script s4pm_preselect_data_handles.pl was merged into s4pm_preselect_data.pl, a single script that handles *.ur files as file handles or as simple files containing only URs.

s4pm_preselect_data_handles.pl

• Dropped from the baseline as a separate file; it was subsumed into s4pm_preselect_data.pl.

s4pm_run_algorithm.pl

• Modified to handle multi-file output granules, using the modification noted elsewhere to PCF.pm.

s4pm_run_test.pl

• Modified to retrieve the host name via the Sys::Hostname::hostname() function rather than via hostname. This file is part of the S4PM_TEST package.

s4pm_select_data.pl

• The script s4pm_select_data_handles.pl was merged into s4pm_select_data.pl, a single script that handles *.ur files as file handles or as simple files containing only URs.
• Fix in get_process_end_time() for START_OF_AIRS_PENTAD production rule.

s4pm_select_data_handles.pl

• Dropped from the baseline as a separate file; it was subsumed into s4pm_select_data.pl.

s4pm_stringmaker.pl

• Added fix for Bug 495. If the data_version is set in the %production_summary_parms hash, it will be used for the version ID of the production summary data type. Otherwise, it defaults to 001.

s4pm_stringmaker_algorithm.cfg

• Modified description of the %production_summary_parms parameter to describe the new optional data_version attribute.

s4pm_sweep_data.pl

• The script s4pm_sweep_data_handles.pl was merged into s4pm_sweep_data.pl, a single script that handles *.ur files as file handles or as simple files containing only URs.
• Made fix to subroutine wipe_handle() to handle the case where the so-called "handle" is still a plain old data file. The .ur file wasn't getting cleaned out; now it is.

s4pm_sweep_data_handles.pl

• Dropped from the baseline as a separate file; it was subsumed into s4pm_sweep_data.pl.

s4pm_tk_admin.pl

• Fix for Bug 496 to clean out all PDRs. The fix was implemented by having the tool read the %datatype_destination hash from the s4pm_export.cfg file. This hash lists all PDR directories in the system. The fix was implemented in FullCleanOut().
• Modified sub get_instance() so that it doesn't fail if the tool is run in a non ECS standard directory.

s4pm_tk_select_string.pl

• Removed a single line that had been commented out anyway.

SimAlg1.pl, SimAlg2.pl

• Updated both of these simulated algorithms (part of the S4PM_TEST suite) to use S4PM::parse_patterned_filename() rather than the antiquated S4PM::parse_modis_filename().