S4PM 5.14.0 RELEASE NOTES

Release Date

July 25, 2006

Functional Changes

1. Stationmaster Enhancements:
   
• A station-specific custom sort function can now be specified. See stationmaster.pl man page for details.

• The virtual job feature now includes $cfg_virtual_feedback, which will take into account the number of failed jobs of a given type before spinning off a new one. See stationmaster.pl man page for details on this parameter and virtual jobs in general.
  
  

2. Acquire Data - This new station that handles the acquiring of data from S4PA or S4PA-like sources is now fully functional.
   
• Now Accepts DNs Instead of DANs - In tandem with S4PA changes, input work orders to the Acquire Data station have been changed from DAN.* to D*.{PDR,wo}. That is, it accepts work orders with either the PDR or wo file name extensions. Work orders from S4PA have the PDR extension; work orders from LAADS have the wo extension.
• Can acquire data from LAADS FTP site.
  
  
3. New Poll PDR Station - To support bringing in data from LAADS (the MODIS distribution system), a new Poll PDR station has been created to poll a remote site for PDRs.
   
   
4. Nascent File Handle Functionality - Infrastructure code has been added to eventually (in a later release) support file handles for data. When fully functional, this will allow data files to retain their original file names since S4PM will be dealing only with file handles instead. File handles will be the *.ur files overloaded with the full pathnames of the data and metadata files with which they are associated. This functionality is not yet complete and should not be used. New scripts: s4pm_find_data_handles.pl, s4pm_sweep_data_handles.pl, s4pm_select_data_handles.pl, and s4pm_preselect_data_handles.pl have been added to the baseline temporarily to specifically work with file handles (this guarantees backward compatibility until the new code is fully tested). In a later release, the functionality in these new scripts will be merged into the original scripts; at that point, the scripts will fully support strings employing file handles and those that don't.
   
   
5. New Auto Acquire Stations - To handle automated acquires of data from S4PA systems, there are now Auto Acquire stations that feed requests to the Acquire Data station; these are analogous to the Auto Request station that feeds requests to the Request Data station. One is created for each S4PA system feeding data to the S4PM string (in addition to any Auto Request station to handle data from ECS).

Detailed File Changes

S4P::FileSpec.pm

• Added ability to parse attribute 'pathname' in new method

S4P::MetFile.pm

• Added hdfeos2odl to extract metadata from HDF-EOS using ncdump

S4PM.pm

• Added a number of functions to encapsulate polling of files and accesses to metadata and URs . These functions support both the old-style URs as well as the new-style URs functioning as file handles. These functions were added:
  
  
  • @files = get_filenames_from_handle($handle);
    
    This function takes a data file or a file handle as input and returns the corresponding full pathnames of the data files. If the handle is the old-style UR file, the function simply returns the name passed to it with the .ur truncated off. If the handle is a true handle file, the function returns a list comprised of lines 2 through the next to the last line (the last line is the metadata file). If $handle is a file, it simply returns back the file; if it's a directory correpsonding to a multi-file granule, it returns the list of data files within.
    
    
  • $metadata = S4PM::get_metadata_from_handle($handle);
    
    This function takes a data file or a file handle and returns the full pathname of the metadata file corresponding to the data file or handle file passed in.
    
    
  • $ur = S4PM::get_ur_from_handle($handle);
    
    This function returns the UR from the file handle, which is assumed to be the first line in the handle file.
    
    
  • @handles = S4PM::get_matching_handles($dir, $pattern);
    
    Given a directory and file pattern, this function returns an array of matching handle files, new or old-style.
    
    
  • $count = S4PM::matching_handle_count($dir, $pattern);
    
    Given a directory and file pattern, this function returns the number of handle files matching the pattern.
    
    
  • $res = S4PM::is_multi_file_granule($handle);
    
    Returns 1 if the granule associated with the data file or handle passed in contains more than 1 data file; 0 otherwise.
    
    
  • $size = S4PM::get_granule_size($handle);
    
    This function takes a data file or file handle and returns the total size in bytes of all associated data files.
    
    
  • S4PM::touch_granule($handle);
    
    Given a handle, this function simply executes a 'touch' on all components of the granule including the file handle itself. It works with both old- and new-style file handles. There is no return value.

S4PM::Algorithm.pm

• Added a companion function to trigger_data() that returns the trigger data type's version: trigger_version().

S4PM::DataRequest.pm

• Fixed symlink_metfiles() so that it doesn't complain about failing to symlink if the source directory isn't there in the first place. For AIRS L0, several App ID data types are produced only rarely and therefore, S4PA won't have made directories for them.

s4pm_acquire_data.pl

• Extracts metadata from HDF-EOS file if no metadata file is available
• Can work with firewalls now.
• Fixed S4P::perish() call to hardcode exit code instead of return code from write_file, which returns 0 on error.

s4pm_find_data_handles.pl

• New file that took the original s4pm_find_data.pl script and modified it fairly significantly to deal with both old-style and new-style URs. Changed get_data_pathname() to get_handle_pathname() and modified it to deal with URs as handles rather than as just files. The get_ur() function was dropped in favor of the new S4PM::get_ur_from_handle() function. The fill_in_optional_file_groups() and fill_in_required_file_groups() functions were also modified to deal with handles appropriately, whether old- or new-style. Finally, mk_array_of_file_specs() and mk_array_of_multifile_specs() were changed somewhat to deal with handles. This flavor of Find Data is only used if $use_filehandles is set.

s4pm_preselect_data_handles.pl

• New file that took the original s4pm_preselect_data.pl script and modified to make use of the new S4PM polling and file handle functions. Changes were mostly in poll_for_data() and in make_patterns_hash(). This script is only used if $use_filehandles is set.

s4pm_select_data_handles.pl

• New file that took the original s4pm_select_data.pl script and modified slightly to account for the new interface for S4PM::Algorithm::trigger_datatype() function. This script is only used if $use_filehandles is set.

s4pm_stringmaker_derived.cfg

• In the Acquire Data station, changed configuration for input work orders such that they are named DN.* rather than DAN.* (S4PA is renaming this in tandem).
• Modified Acquire Data station so that there is a subdirectory named ACQUIRES symbolically linked to ../request_data/REQUESTS.
• Also in Acquire Data, changed interface to s4pm_tk_compose_request.pl to pass the -o ACQUIRE_DATA option and replaced REQUEST_DATA in %cfg_commands with ACQUIRE_DATA to handle the new work order names generated by Compose Request.
• Modified to make use of the new @input_external_archive_systems and @output_external_archive_systems arrays throughout.
• Added the Auto Acquire stations so that there is a separate one for each S4PA archive system. The name of the FTP host is used in the name of the station. Thus, for example: Auto Request G0spp12, Auto Request G3dpp01, or Auto Request Local, etc.

s4pm_stringmaker_global.cfg

• Moved the $external_archive_system parameter out of here and into the s4pm_stringmaker_string.cfg file where it makes far more sense; it can be different for different strings on the same box.

s4pm_stringmaker_host.cfg

• Moved the $host parameter out of the s4pm_stringmaker_string.cfg into this file as it makes far more sense here. It doesn't, however, really matter where it is set. But this is a more fitting home.
• Added new parameter $host_alias to allow for an alternate name for the local machine (for machines that are dual-homed). Without this, the Acquire Data station may receive PDRs for local data, but because the PDR contains the alternate domain name, it isn't recognized as local. This leads to Acquire Data trying to FTP the data over rather than setting up symbolic links instead.

s4pm_stringmaker_static.cfg

• Minor modification in the Sweep Data station so that a link to the new s4pm_sweep_data_handles.pl file is created.

s4pm_stringmaker_string.cfg

• Added new %pdr_polling_parms hash to support polling of remote PDRs for ingest of data into S4PM.
• Moved the $host parameter out of this file and into the s4pm_stringmaker_host.cfg file as it makes far more sense there. It doesn't, however, really matter where it is set. But the other file is a more fitting home.
• Added the $use_filehandles parameter to turn on or off the use of file handles in S4PM.
• Moved the $external_archive_system parameter out of the s4pm_stringmaker_global.cfg file into this one. It makes far more sense to place it here.

s4pm_sweep_data_handles.pl

• New file that took the original s4pm_sweep_data.pl script and modified it rather substantially to make use of the new file handle function in the S4PM module. Among other changes, the old wipe_directory() and wipe_file() functions were replaced with a single generalized wipe_handle() function. This script is only used if $use_filehandles is set.

s4pm_tk_admin.pl

• Added cleaning out of request stub files in acquire_data/ACQUIRES directory.

stationmaster.pl

• Added customizable sort function capability.