S4PM 4.5.0 RELEASE NOTES

Release Date

April 30, 2003

Functional Changes

1. New buttons have been added to the TkJob GUI for the Gran Find station: Ignore Required forces Gran Find to stop looking for required inputs and release the job. Before it does so, one final look for all required inputs is performed. 'Ignore Optional' replaces Release Job Now and is functionally equivalent. The name change was to make it more consistent with the Ignore Required button and more clear.

The Ignore Required button should be used only VERY RARELY with some AIRS PGEs. Since you are forcing the job to be released without at least some required input, the PGE may fail outright unless it has been coded to handle this situation.

2. Quality Control (QC) functionality has been added to S4PM. This gives the ability to block data from being exported to the ECS archive or from being used in downstream processing based on product files and/or
product metadata files. A S4PM instance can be configured with QC on or off; it is off by default.

Current QC involves three checks:

1. For HDF files, validate that the HDF file is valid and not corrupted.
2. Check the file size to see that it is within reasonable bounds.
3. Compute the checksum for the file (not really QC, but can be used later to check for file corruption).

The above are all separate scripts associated with particular data types. Each data type can individually be configured to invoke one or more of the above checks and each data type can be individually configured as to what action is taken when a QC check fails.

Current actions are:

1. Log the QC check failure and fail the PGE.
2. Log the QC check failure, but the PGE still succeeds.
3. Block export of the product.
4. Block the product from being used in downstream processing.

If it is configured so that the PGE succeeds on QC failure (action 2), then actions 3 and 4 can be set independently. That is, you can have one or both happen.

Other QC scripts can easily be added and some definitely will be.

In the current release, QC failures are set to only be logged but not otherwise fail or block.

A later release will have failure handlers for QC failures.

Detailed File Changes

airs_L0_check.pl

• Replaced getFilenamePattern() function (which is deprecated) with the new get_filename_pattern() function.

attitude_check.pl

• Got rid of superfluous 'use Date::Calc;' statement. It wasn't necessary in any case.

data_catcher.pl

• Got rid of superfluous 'use Date::Calc;' statement. It wasn't necessary in any case.

granfind.pl

• Minor change: Got rid of the 'use Date::Calc;' statement since it is no longer needed (it had been superfluous for a long time now).
• Combined the functionalities of release_job_nowait() and expire_current_timer() into one generalized function: signal_file().
• Added new signal file handler: IGNORE_REQUIRED, intended for required inputs it release the job whether or not all required input was found. It is assumed that the PGE can handle such situations.
• To maintain consistency with the new IGNORE_REQUIRED file signal, changed RELEASE_JOB_NOW to IGNORE_OPTIONAL.

pge_stats.pl

• Replaced Date::Calc::Day_of_Year() function with TimeTools::day_of_year() equivalent.
• Replaced Date::Calc::Add_Delta_Days() function with TimeTools::add_delta_days() equivalent.

prespecify_data.pl

• Replaced Date::Calc::Day_of_Year() function with TimeTools::day_of_year() equivalent.

run_pge.pl

• Quit if configuration file is specified but not found/read.
• Simplified routine that figures out the UPDATE work order for multi-file vs. single-file granules, using hashes. This also fixes a bug present for Direct Broadcast PGEs.
• Simplified recognition of "true" product input files: i.e., anything with a leading '/'.
• Do not quit if no temporary scratch files are there (e.g. LogStatus).
• Added QC (quality control) functionality. This has the ability to block catches or exports of files that fail ESDT-specific QC tests.

S4PM.pm

• Minor fix to the get_box_mode_instance_gear() subroutine which was causing the SHOW_PGE job in Repeat Daily to fail.

s4pm_derived.cfg

• Changed the setting of $data_dir to doing so only if it has not already been set. This is in support of g0spp12 where we set this directory in the string-specific configuration file (on g0spp12, there is no link between /vol1/<mode>/s4pins and /usr/ecs/<mode>/<machine>/s4ins).
• Changes to accommodate QC functionality.

s4pm_pge_esdt.cfg

• Updated to reflect the most recent PGE and ESDT configurations used in OPS.
• Changes to accommodate QC functionality.

s4pm_static.cfg

• Modified the %cfg_manual_overrides associated with the Gran Find station so that the signal files used now are IGNORE_OPTIONAL and IGNORE_REQUIRED. See under granfind.pl.

specify_data.pl

• Replaced Date::Calc::Day_of_Year() function with TimeTools::day_of_year() equivalent.

stringmaster.pl

• Moved creation of MPGE02.cfg from s4pm_static.cfg to s4pm_derived.cfg, using info new %browse_map in s4pm_pge_esdt.cfg. This creates the appropriate configuration only in the strings where that PGE runs, rather than all strings.
• Replaced reference in run_pge command to "-f MoPGE02.cfg" with "-f ../MoPGE02.cfg". Both work, but the former is counter-intuitive.
• Replaced hard-coded paths to binaries in s4pm_derived.cfg.

TimeTools.pm

• Got rid of need for Date::Calc module by coding up equivalent subroutines:
  • is_leapyear()
  • day_of_year()
  • add_delta_days()
  • doy_to_ymd()
  • days_in_year()
  • day_of_week()
  • add_delta_dhms()
  • seconds_to_dhms()
  • delta_dhms()
  • ... and more.
• These are meant to replace the Date::Calc functions used within TimeTools.pm and elsewhere.

tkS4PMadmin.pl

• Went back to old way of defining the s4ins directory so that it works on both the SGI and Linux boxes.
• Added capability for running non-interactively from the command line via:
• tkS4PMadmin.pl -task [task] -force
• where [task] is one of:
• FullCleanOut, Archive, PH, Blocks, Data, IntegrityCheck, ExtraFiles,
• FailedJobs, ResetDatabases, and StationLogFiles
• Only one task can be run at a time.
• The -force means that you won't be asked to confirm the changes that you are about to make. This will allow other scripts to do, for instance, full clean out of S4PM, without having to run a GUI.

trigger_pge.pl

• Replaced Date::Calc::Day_of_Year() function with TimeTools::day_of_year() equivalent.
• Made fix so that if metadata is to be retrieved from the metadata file rather than the HDF file (a Specify Data configuration file setting), what goes into the PCF, 6th field is the metadata file name only and not the entire path. It already works fine if HDF file is chosen.