S4PM 3.3.0 RELEASE NOTES

Release Date

March 27, 2002

Functional Changes

1. Spacecraft Maneuvers - If a L0 (or PM Attitude) granule fails because it does not start on the even hour, this will be apparent in the presence of a:

DO.FIX_TIME.<job_id>.wo

file in the failed directory. Operators should first determine if the L0 granule should be processed, usually by checking with Ingest as to whether that is the best L0 granule for that period. If it is NOT to be processed, operator should click the Purge L0 button in the tkjob interface. This will delete the data from disk and deposit a work order in Clean Data to deallocate space from the INPUT pool. If it IS to be processed, operator should click the Fix L0 Time button. This will round the data time in the output work order down to the nearest even hour. If an AMATTN0 granule fails because it apparently includes a maneuver, this will appear as a:

DO.BAD_QA .<job_id>.wo

file in the failed job directory. The log will also indicate that a threshold has been exceeded. The operator should check the time period to verify that it is a maneuver. If so, the input data should be purged, by pressing the Purge Bad Data button. This will delete the data from disk and deposit a work order in Clean Data to deallocate space from the INPUT pool. The operator should also keep an eye out for spinning PGE01 jobs in granfind. Check the sleep.message to see if it is waiting for the purged data. If so, the Expire Current Timer can be used to get past the AM1ATTN0 dependency and use AM1ATTNF instead.

If there is no maneuver, the threshold trip may be spurious. The data can be forced into the system by clicking the Bypass QA button, which will recycle the DO.BAD_DATA file in the Data Catcher station, causing it to be executed without the -q option, thus skipping the QA step.

Because a work order may contain a mix of data failing QA and having offset times, the failed directory will not be cleaned automatically by any of the Fix/Bypass or Purge failure handlers. Instead, after the operator has dispensed with all of the anomalies in a given failed directory, the directory must be removed using the Remove Job handler. However, in some cases, it may be helpful to keep the failed directory in place as a reminder to check the Granule Find station for jobs that are waiting for AM1ATTN0 granules that have in fact been purged and will never arrive.

IMPORTANT: Generally, operators should not use the Restart Job button in Data Catcher, unless they have been hand editing failed work orders. This is something that they should do only if they /REALLY/ know what they are doing.

2. S4PM System Cleanup - A new GUI, tkS4PMadmin.pl, replaces the scripts pretest_cleanup.pl and init_allocdisk.pl. These latter scripts no longer exist.

To run the tkS4PMadmin.pl GUI:

cd /vol1/OPS/s4pm/<instance>/stations/<gear>

where <instance> is one of 'terra', 'aqua_modis', or 'aqua_airs' and <gear> is one of 'forward' or 'reprocessing'

3. tkS4PMadmin.pl - The tkS4PMadmin.pl GUI offers more features than simply cleaning up
everything. To clean up everything, select Full clean out of S4PM, the bottom check box. Then click Submit. This has the same functionality as running both pretest_cleanup.pl and init_allocdisk.pl.

See descriptions of other features of this GUI by clicking on the question-head button next to each item.

Detailed File Changes

tkS4PMadmin.pl

• New Tk GUI script to eventually replace pretest_cleanup.pl and init_allocdisk.pl test scripts. tkS4PMadmin.pl is more robust and can be extended to other S4PM administrative duties.

run_pge.pl

• Small fix to allow inputs in the PCF to start with 'AM1' or 'PM1'. This is in support of the Terra and Aqua DPREP PGEs.

attitude_check.pl

• New script to check attitude files for signs of a spacecraft maneuver. See detailed description under data_catcher.pl.

purge_input_data.pl

• New script to handle the purging of data that doesn't pass QA. See detailed description under data_catcher.pl.

data_catcher.pl

• Data Catcher now has the ability to utilize data type-specific QA scripts. In particular, AM1ATTN0 can be checked for threshold violations (default is 0.0002 radians). This feature is invoked by: (1) Adding the -q flag to the data_catcher call in station.cfg, and (2) Specifying the script to be executed in data_catcher.cfg, e.g.:

  %quality_assessment = (
       'AM1ATTN0' => 'attitude_check.pl -t .0002'
  );

  The handling of bad input data is also changed. If short L0 granules are detected, a PDR-style file named DO.FIX_TIME.<job_id>.$$.wo is written with the short granules. If a granule fails QA, it is extracted into a PDR-style file named DO.BAD_QA.<job_id>.$$.wo If any input granules are good, they are extracted into a PDR file with the same name as the input work order, but with the process_id attached AND the new work order is AUTOMATICALLY recycled into the parent directory for immediate processing.Owing to this change, failure handling has the following changes to station.cfg:

  (1) Modify %cfg_failure_handlers to:

  %cfg_failure_handlers = (
  'Fix L0 Time' => '/bin/mv DO.FIX_TIME..wo ..',
  'Bypass QA' => '/bin/mv DO.BAD_QA..wo ..',
  'Purge Ragged L0' => "purge_bad_data.pl -p 'DO.FIX_TIME..wo' -c ./../clean_data",
  'Purge Bad-QA Data' => "purge_bad_data.pl -p 'DO.BAD_QA..wo' -c ../../clean_data",
  'Remove Job' => 'remove_job.pl'
  );

  Thus, the only action to move the extracted work orders to the parent directory for processing (see #2 below).

(2) Add to %cfg_commands:

'FIX_TIME' => "perl ../data_catcher.pl -F -t -f ../data_catcher.cfg",
'BAD_QA' => "perl ../data_catcher.pl -t -f ../data_catcher.cfg",

Note that FIX_TIME calls data_catcher with the -F option, and BAD_QA calls data_catcher WITHOUT the -q option.

The behaviour of the failure handlers is as follows:

Purge Ragged L0: Delete files from disk, send work order to Clean Data station to deallocate space.

Purge Bad-QA Data: Delete files from disk, send work order to Clean Data station to deallocate space.

Fix L0 Time: Rerun data_catcher with -F option to round L0 start time down to nearest two-hour boundary.

Bypass QA: Rerun data_catcher without -q option to bypass QA step.