December 19, 2006
Bug 298 Fixed: Metadata Not Found With Level 0 Input - A bug causing failures to find metadata files associated with Level 0 files has been fixed.
Bug 356 Fixed: Multi-File Granule Output Not Handled Correctly - This fix addresses a shortcoming in S4PM, namely that it didn't handle multi-file granule output from an algorithm. Multi-file granule output should now be handled including chaining, that is, where multi-file output is used as input to an upstream algorithm. The way to configure multi-file granule output is via the PCF template. Simply include multiple version entries in the PCF. Note that this cannot be combined with the multiplicity that S4PM already supports to make sure that there are enough output entries in the PCF (for a particular LUN) to cover the processing period. Only one type of multiplicity can be handled at a time for a LUN. A failure results in Prepare Run if you try to do both.
Bug 454 Fixed: Incorrect Logic in AIRS Pentad Production Rule - Bugs affecting the AIRS pentad production rule were fixed in this release.
Bug 495 Fixed: Added Support for Customized Production Summary Versions - In previous releases, the Production Summary was fixed to 001. In this release, it can be set via the data_version attribute in the %production_summary_parms hash in the Stringmaker Algorithm configuration file.
Bug 496 Fixed: S4PM Admin Tool Doesn't Clean Out All PDRs - A fix was made to the S4PM Admin Tool so that all PDRs are deleted when a full cleanout is run.
Further Progress In Supporting Native File Names - More work was done to lay the ground work for supporting native file names in S4PM via file handles, a concept in which S4PM refers to all data files via handle files. Handle files contain a list of files making up a granule (metadata file, XML file, and the data file or files themselves). The names of the individual data files within are native, that is, S4PM doesn't coerce them into any convention. Rather, only the handle files themselves adhere to a file naming convention for S4PM internal use. In prior releases, several scripts that specifically supported the file handle concept were added to the baseline with distinct file names (e.g. there was s4pm_find_data.pl and s4pm_find_data_handles.pl). In this release, these separate scripts have been merged back into the main scripts (e.g. s4pm_find_data.pl now handles both the file handle and the non file handle case). More work needs to be done and native file name support is not yet ready to be used.
Improved Error Handling in ODL Parser - Implemented additional error handling to detect unbalanced ODL trees.
Case Insensitive Search of ODL Tree - Implemented case insensitive search of ODL trees by setting a flag.
EDOS PDR and PAN Handling - Implemented PDRs and PANs with binary headers to support EDOS ingest to S4PA.
DME Installation Instruction - Modified the email instruction sent to firstname.lastname@example.org to allow easier creation of new data miner string.
Detailed File Changes
Functions are provided to extract fields from an EDOS-style PDR to build an EDOS-style PAN. If the PAN description is "SUCCESSFUL", the file transfer disposition will be marked successful (0); if it contains CHECKSUM, the file disposition will be set to Checksum Mismatch (3); if METADATA, it will be set to Invalid PDS/EDS Construction Record Data (9); for all other values, it will be set to File Unreadable (8).
If an input file contains an EDOS-style PDR (begins with \x09\x00), the binary Ground Message Header is stripped from the front and stored as though it were a hex string for a new field called GROUNDMSGHDR. If a PDR is written that has a GROUNDMSGHDR, it is written as a SIPS-style PDR with this extra field which is recognized in subsequent reads. Functions are provided to fetch and store the value, and test if it is null.
Fix for Bug 356. The code was modified so that if INSERT_MULTIDIRECTORY_HERE is found rather than INSERT_DIRECTORY_HERE, the directory assigned is a subdirectory under the disk pool directory, one named for the file name.
The script s4pm_find_data_handles.pl was merged into s4pm_find_data.pl, a single script that handles *.ur files as file handles or as simple files containing only URs.
Changed import of s4pm_allocate_disk.pl to use Safe instead of a straight eval.
Fix for Bug 356, incorrect PDR generated when output is multi-file granules. The problem was that s4pm_prepare_run.pl simply did not handle multi-file granule output. This functionality has been added in. If the template PCF lists multiple versions for a particular LUN, the script assumes that we are dealing with a multi-file granule. The normal INSERT_DIRECTORY_HERE placeholder placed in the output PCF is replaced with INSERT_MULTIDIRECTORY_HERE instead so that s4pm_allocate_disk.pl will know what to do. In addition, the output file names are tagged with the version numbers so that each file making up the multi-file granule gets a unique file name.
Made configuration file optional.
The script s4pm_select_data_handles.pl was merged into s4pm_select_data.pl, a single script that handles *.ur files as file handles or as simple files containing only URs.
Fix in get_process_end_time() for START_OF_AIRS_PENTAD production rule.
The script s4pm_sweep_data_handles.pl was merged into s4pm_sweep_data.pl, a single script that handles *.ur files as file handles or as simple files containing only URs.
Made fix to subroutine wipe_handle() to handle the case where the so-called "handle" is still a plain old data file. The .ur file wasn't getting cleaned out; now it is.
Fix for Bug 496 to clean out all PDRs. The fix was implemented by having the tool read the %datatype_destination hash from the s4pm_export.cfg file. This hash lists all PDR directories in the system. The fix was implemented in FullCleanOut().
Modified sub get_instance() so that it doesn't fail if the tool is run in a non ECS standard directory.