Skip to content. | Skip to navigation

Personal tools
You are here: GES DISC Home Additional Features Technology Lab S4PM 5.18.1 RELEASE NOTES

S4PM 5.18.1 RELEASE NOTES

Release Date

February 1, 2007

Functional Changes

  1. Bug 422 Fixed: Enhance PDR validation - Traps malformed FILE_GROUPs with no FILE_SPECs or END_OBJECT.

  2. Bug 495 Fixed: Production Summary Version ID Hardwired - A bug prevented the production summary data type from having a version other than 001.

  3. Bug 555 Fixed: Bug in s4pm_find_by_coverage.pl - A modification was made to this script so that by default it will consider a data file as found if it partially overlaps the processing period in coverage whereas the -full option is needed to demand that there be complete overlap.

  4. Bug 558 Fixed: Allow Appending Command Line Options - A new hash, %commands_addenda, was added to support the adding of command line options to commands running in S4PM stations. The primary hash key is the station name and the secondary hash key is the work order type. The value is something to be added to the command associated with that work order type in that station.

  5. Bug 559 Fixed: Allow FIFO to Override Default Sort Order - A modification to Stringmaker now allows you to override the default $cfg_sort_jobs parameter with an alternative, such as FIFO.

  6. Bug 560 Fixed: Change s4pm_export.pl to Support AIRS Granule Browse - A fix was made to the Export station code to address the output EXPORT work order for AIRS Browse granules.

  7. Bug 561 Fixed: Invalid PDRs from s4pm_run_algorithm.pl for AIRS browse - A fix was made to produce valid PDRs for AIRS Browse granules.

  8. Bug 563 Fixed: Redefinition of copy subroutine in S4P - One line was changed in S4P::FileGroup.pm.

  9. Bug 564 Fixed: Errors from MetFile.pm called in s4pm_export.pl - Added "no warnings 'uninitialized'".

  10. Bug 566 Fixed: Add Capability for Disk Pool Map Overrides - A minor modification was made to Stringmaker so that the default disk pool map (disk pool to data type) could be overridden in the Stringmaker String configuration file.

  11. Bug 579 Fixed: Stage For Pickup Station Not Created in Data Mining - A bug that resulted in a malformed Stage For Pickup station in Data Mining was fixed.

  12. Bug 582 Fixed: Bug in S4PM::parse_patterned_filename - A fix was made, actually in s4pm_sweep_data.pl, to address this issue.

  13. Bug 605: Can't Start Compose Request GUI in Acquire Data Station - A change to Stringmaker was made to ensure that that last line of every configuration file generated contains a 1. This didn't fully address this bug, but it did highlight the need for this particular fix.

  14. Bug 635 Fixed: XML and Request Data Stub Files Not Getting Deleted - This was traced to several bugs in the wipe_handle() subroutine of the s4pm_sweep_data.pl script.

  15. Bug 644 Fixed: Granule Container Directories Not Getting Deleted - This was traced to another bug in the wipe_handle() subroutine of s4pm_sweep_data.pl and fixed.

  16. Bug 654 Fixed: XML Files Not Moved Along With Data - A bug resulting in XML files being left behind after data were moved from the INPUT disk pool to the data type disk pool was fixed.

  17. Bug 660 Fixed: DataRequest::find_urls_by_datatime problem with .jpg files - Modified to be more general.

  18. Extensions in S4P::FileGroup and S4P::FileSpec to Support Browse Images - Several modifications were made to FileGroup.pm and FileSpec.pm to handle Browse image files needed for AIRS.

  19. Run Tokens - The ability for multiple stations to "share" a set of processors has been added. This is done through a tokenmaster station, running the new script s4p_token.pl, and a %cfg_token configuration variable (keyed on job_type) in the various stations sharing the tokens. Modifications have been made to S4P.pm and stationmaster.pl to spawn jobs but not begin actual execution until granted a token from the tokenmaster station. Once the job succeeds or fails, the token is released.

  20. Data Handles Support - Implemented data handle support (formally called file handles) in Register Data via the -H option.

  21. S4P Enhancements - A number of changes were made in S4P modules to support PDRs and PANs.

Detailed File Changes

S4P.pm

  • Added request_token(), grant_token() and release_token() to support sharing of processors by multiple stations.

S4P::FileGroup.pm

  • Added new method browse_file() to return the pathname of a Browse file in the FILE_GROUP.
  • For Bug 563, changed "use File::Copy;" to "use File::Copy();" to avoid conflict with FileGroup::copy().

S4P::FileSpec.pm

  • Modified guess_file_type() method to add in the .jpg and .jpeg file name extensions for a Browse file.

S4P::MetFile.pm

  • Fixed Bug 564 by adding 'no warnings "uninitialized"' to the top of the file. Parsing should still work as designed.

S4P::PDR.pm

  • For Bug 422, added traps for malformed FILE_GROUPs with no FILE_SPECs or END_OBJECT.

S4PM.pm

  • Modified to remove the new data handle routines; these have been placed in a separate module.
  • Removed redundant 'use File::Basename;' statement near beginning.
  • Modified subroutine free_disk() to use the S4PM::Handles::get_granule_size() function to get granule size rather than doing it manually.

S4PM::DataRequest.pm

  • Fix for Bug 660. Modified find_urls_by_datatime() so that Browse files did not have to be named with "Browse" in the name.

S4PM::Handles.pm

  • New module containing handle routines. These were originally placed in the S4PM.pm module. Now, they've been separated out into their own module.
  • Revised the get_filenames_from_handle() and get_metadata_from_handle() routines to support data handles that prepend DATA=, MET=, XML=, and BROWSE= to the respective lines in the handle file. The UR remains restricted to the first line of the file and without anything prepended (to support backward compatibility).
  • Added new routines get_xml_from_handle() and get_browse_from_handle(). The latter assumes only a single browse per data file granule and one that is packaged with the data file it is associated with. If no XML file, the get_xml_from_handle() returns undef. If no browse (as described here), the get_browse_from_handle() also returns undef.

s4p_token.pl

  • New script to manage tokens in a tokenmaster station. It responds to requests for tokens by renaming them. It then polls for the existence of the token file, and when gone, exits 0. If it times out with the file still in existence, it exits 2.

s4pm_export.pl

  • For Bug 560, added subroutine is_jpeg() to check file for jpeg-ness, and if so, "rename" using the symlink function, in subroutine lgid_symlinks(), to replace the trailing HDF with jpg. N.B.: this will not work if file ends in something other than HDF or hdf.

s4pm_find_data.pl

  • Change to get_handle_pathname() to support AIRS near real-time where there is the possibility of duplicate input files.
  • Modified to use file handle routines from the new Handles.pm module.

s4pm_find_by_coverage.pl

  • For Bug 555, changed the default behavior so that a data file whose coverage at least partially overlaps the processing period is considered found. To get the previous behavior of demanding complete overlap of the processing period, the -full option is now needed.

s4pm_register_data.pl

  • Added support for data handles via the -H option:
    • The UR file in turn lists the UR with the data, metadata, and browse files listed after that in the form TYPE=PATH (e.g. MET=/some/path/pool/data_files/file.met). These files are stored with their native file names in a data_files subdirectory to make globbing more efficient.
    • Handles export to the EXPORT station:
      • If the FILE_GROUP STATUS is set to EXPORT or EXPORT_ONLY, an EXPORT PDR is generated for the Export station. N.B.: If you are using browse, this works only for non-linkage browse. And it does not work for EXPORT_PH.
      • If the FILE_GROUP STATUS is set to EXPORT_ONLY, the FILE_GROUP is not added to the INSERT work order.

s4pm_run_algorithm.pl

  • For Bug 561, added a line to do another file PDR recount after the browse handling.
  • For Data Handles, modified code to route data exporting through the Register Data station. Added merge_pdrs(), which merges the EXPORT PDR into the REGISTER PDR. Set the FILE_GROUP STATUS attribute in the PDRs to EXPORT for files to be exported, EXPORT_ONLY for files to be exported but not registered, and left unset for files that are not to be exported.

s4pm_stringmaker.pl

  • Related to Bug 605, modified to force an extra line at the bottom of every configuration file containing simply a 1. This insures that every eval or Safe:rdo returns true.

s4pm_stringmaker_derived.cfg

  • For Bug 566, replaced a straight = with a ||= instead for the setting of the %all_pool_map hash. This will allow overrides to be specified in one of the Stringmaker configuration files.
  • For Bug 558, added some code at the bottom to appended any settings in the new %commands_addenda hash to impacted station commands.
  • For Bug 559, the setting of the $cfg_sort_jobs parameter was recast with a ||= rather than just = thus allowing it to be overridden in the Stringmaker string configuration file. This wasn't done, however, in the Track Requests station since it is somewhat special.
  • Got rid of some leftover references to s4pm_sweep_data_handles.pl, s4pm_find_data_handles.pl, s4pm_select_data_handles.pl, and s4pm_preselect_data_handles.pl. These were superfluous given that the originally separated handles code has been merged into the main code.
  • Added -H command line options to s4pm_register_data.pl and s4pm_run_algorithm.pl if the $use_datahandles parameter is set.
  • Added the sending of EXPORT output work orders to Export from Register Data if data handles are employed.

s4pm_stringmaker_string.cfg

  • For Bug 558, added documentation on the new %commands_addenda hash.
  • Changed parameter name $use_filehandles to $use_datahandles (the term "file handles" already has a meaning not related to S4PM).

s4pm_sweep_data.pl

  • Fix for Bug 582. The problem was that when local file names are used (the -l flag with s4pm_register_data.pl), the Sweep Data can't always get the data type from the file name or the UR file. Thus, it fails. A subroutine was added, get_metadata_shortname(), to get the data type from the associated metadata file is all else fails.
  • Modified to use data file handle routines from the new Handles.pm module.
  • Fix for Bug 635. The wipe_handle() subroutine simply wasn't deleting XML files if they existed. Also, a logic bug resulted in the .ur file getting deleted before it could be used to find any associated request data stub file. These were fixed.
  • For Bug 644, the code in wipe_handle() was attempting to remove a directory via unlink, which doesn't work. It was changed to rmdir instead.

s4pm_tk_admin.pl

  • Fix for Bug 579 in subroutine wipe_file(). The deletion of files had been too ambitious (it deleted all files in directories listed in the %datatype_destination hash of the s4pm_export.cfg file). In Data Mining, this meant that the station.cfg and the script itself were deleted from the Stage For Pickup station. The fix was to make s4pm_tk_admin.pl avoid deleting script or configuration files.

stationmaster.pl

  • Added code to request a token if $cfg_token{$job_type} is set in the station.cfg. Code also release token when job succeeds or fails.

 

Document Actions
NASA Logo - nasa.gov
NASA Privacy Policy and Important Notices
Last updated: Dec 08, 2010 11:06 AM ET
Top