S4PM 5.5.0 RELEASE NOTES

The New Configurator Station - There is a new station in this release that is a part of all strings: Configurator. The purpose of this station is to allow the string to reconfigure itself. Currently, it only works for modifying the max jobs (see next), but will be extended to other functionalities in the future.

Update To Modify Max Jobs Tool - The Modify Max Jobs has been changed. First, the button to invoke it has been moved from the S4PM Monitor window to the Configurator station. To run it, right-click on the Configurator station button.

More Enhancements for Modify Max Jobs Tool - In previous releases, the Modify Max Jobs allowed one to modify the s4pm_max_children.cfg file and then run Stringmaster. If jobs were modified for strings other than the current string, it did not run Stringmaster on those other strings. Secondly, one had to bounce the affected stations in order for the changes to be reflected in the stations' behavior.

In this release, the Modify Max Jobs provides the same functionality but with enhancements:

(1) Stringmaster is automatically run on all affected strings, not just the one you happen to be currently working with.

(2) The affected stations in the affected strings do not have to be bounced if they are already up and running. This will be done automatically. For stations that happen to be down, there is no bouncing; when these stations are brought up in the normal way, the max jobs changes will be handled.

Hence, the Modify Max Jobs of one string can be used to modify any or all the strings on that box.

New Argus Interface for Cross String Monitoring - The first release of the Argus interface is available in this release. The executable is: tkargus.pl and the command to run it is:

tkargus.pl -c ~/s4pm_cfg/s4pm_tkargus.cfg

To simply running this program, an alias has been set up: argus

Note that this GUI takes a few seconds to start up.

New Directory To Support the Configurator Station - The directory ~/s4pm_cfg had been used to store Stringmaster configuration files and was kind of special because of this. In this release, the Configurator station actually resides in this very same directory. Therefore, it is even more special. It contains a station.cfg file as well as other station files. Please try not to clutter up this station nor remove any of these important files.

New Auto Request Station for Automating Data Orders - This release contains the first appearance of the Auto Request station. This station is only a part of non forward processing strings. It processes work orders generated by the Auto Request
window (see next).

Auto Request Details - The Auto Request tool allows requests for data to be automated and balanced among two or more defined strings. This tool obviates the need for manually ordering data via the the Compose Data Request tool. At the same time, it provides load balancing of data requests across the defined strings.

The executable is: s4pm_tk_auto_request.pl

and it is run:

s4pm_tk_auto_request.pl [request_data_station_list]

where [request_data_station_list] is a list of the full paths to Request Data station directories of the strings that are to be managed by the Auto Request tool. For example:

s4pm_tk_auto_request.pl \
~/g0spg10/terra/reprocessing/request_data ~/g0spg01/terra/reprocessing/request_data

will bring up the Auto Request tool that would provide automated data requests and load balancing across two strings: MODIS Terra Reprocessing on g0spg10 and MODIS Terra Reprocessing on g0spg01.

STOP and RECONFIG Work Order Fixes - STOP and RECONFIG work orders are now fully functional. This feature wasn't reliable in earlier releases. In this release, a STOP work order dropped into any station directory (e.g. DO.STOP.20.wo) will cause the station to shut down. A RECONFIG work order dropped into any station (e.g. DO.RECONFIG.0.wo) will cause a station to reread its station.cfg file and any changes therein to take effect. Note that even if a station is configured with $cfg_input_work_order_suffix set to something other than .wo, the station will still respond to STOP and RECONFIG work orders with the .wo file name extension.

Fixed several problems in the xml2met conversion:
+ handle ZoneIdentifier
+ add QAPercentCloudCover
+ suppress unset QA flags and explanations
+ add LocalVersionID
+ fix handling of StartOrbitNumber/StopOrbitNumber
+ handle multiple sensors, instruments, etc.

The fork_command() routine has been modified to work with Windows process creation.

In stop_station(), changed exec_system("kill...") to native Perl kill(). Also read PID from station.pid file, if it exists.

In write_job_status(), get user_id from directory stat() call before trying getpwuid in Win32.

Added error checking to close statement in write_file().

In restart_job(), changed system("copy...") to File::Copy::copy() call.

In check_station(), the code attempts an exclusive file lock if the shared file lock succeeds and the OS is Linux, to get around an NFS problem.

New script for the new station that automates requests of data via the *Auto Request* tool. This station is only used only in reprocessing.

New script for the new Configurator station. This station is used by the string to self-configure itself. In this release, changes made via the Modify Max Jobs tool are handled via this station.

Modified some of the default setting for the s4pm_poll_data.cfg file.

For on-demand processing, replaced the s4pm_failed_pge_handler.pl failure handler with s4pm_failed_service_handler.pl.

Added handling of the $scli_host variable in the string-specific configuration file in the Request Data station.

Modified so that the datapool polling directory is not assumed, but rather it is based on the parameter $data_source_polling_dir which should be set in the string-specific config file (or the s4pm_pge_esdt.cfg file).

Modified to link the new Configurator station to the station root directory.

Fixed logic error in PAN generation for ersatz ingest.

Changed where UPDATE work order is moved to from 'grancentral' to 'track_data'.

Changed references to clean_data to sweep_data.

Modified so that it no longer handles services in On-Demand processing as well as PGEs. Services are now handled via s4pm_failed_service_handler.pl.

New failure handler for services running in the Run Algorithm station in On-Demand processing. It is based on s4pm_failed_pge_handler.pl, but without the handling of a PH file and accompanying PDR.

Modification in how the PGE name is parsed from the work order name. A bug in this was reported by a user. The parse has been made successful even if the PGE name has embedded dots or hyphens, e.g. PGE FLASH4.1-4.1P3 is successfully parsed out of a work order named: DO.MOREDATA_FLASH4.1-4.1P3.2004338074000

Modified this template configuration file with more realistic values based upon experience.

New script that allows ODL requests to the Split Services station to be preprocessed prior to being handled by the main station script. This was added to handle virtual data product processing.

Added optional command line parameter $SCLI_host. If $SCLI_host is specified then SCLI acquire is executed on remote host via ssh. If $SCLI_host is not specified then the SCLI acquire is executed on the local host.

Modified to output the new PROCESS_ORDER work order type to send it to the s4pm_split_services.pl script.

Added data type D4LAEMIS to s4pm_register_data.cfg file that gets built by Stringmaster and the offsets hash.

Added the parameter $scli_host that, if set, contains the host with SCLI that is to be used for routing order requests. o Added the parameter $data_source_polling_dir to set the root of the datapool (or similar) directory.

Modified to handle the new station.pid files.

New *Auto Request* window that feeds work orders to the new Auto Request station.

Modified so that rather than running Stringmaster itself, it instead sends STRING_MAX_JOBS work orders to the modified strings' Configurator station.

Added 'disable' attribute for disabled stations (see stationmaster changes).

Refactored refresh() method into reconfig() and refresh(), allowing tkstat to pick up station reconfigurations without restarting.

The station.lock file is now used only for preventing duplicate stationmasters. A new file, station.pid, is used for recording the process_id for killing stations. N.B.: When upgrading to 5.5.0, all stations should be stopped and restarted.

Stationmaster now recognizes the DO.STOP.* work order, which stops the station, and DO.RECONFIG.*, which causes it to reread its configuration file.

In Windows stationmaster can now spin off more than one child, as a "fork" (or rather a Win32 process).

Several die() statements were changed to S4P::perish to show up in the station.log file.

The absolute value of the process id is now used in constructing directory names (Windows has negative process ids).

Several error traps and log messages were added.

In Windows, get username from ENV{'USERNAME'} instead of getpwuid($<).

Implements "disabled" stations, which cannot be started up by any means so long as $cfg_disable is set to be nonzero.

A new application that provides a summary look at many strings and many machines at once. Clicking on station/string buttons brings up a S4PM Monitor window (aka tkstat.pl) for that station in the selected string.

Recognizes disabled stations (see stationmaster.pl changes), and desensitizes station button and suppresses popup menu.

The Refresh button now picks up changes in the station.cfg files of the stations.

A "read-only" mode was added for managers (tkstat.pl -R ...).

A number of changes have been made to support S4P and S4PM in Windows. These changes are mostly backward compatible. However, the first stationmaster change below requires restarting stations when upgrading to 5.5.0.

NASA Official: Steve Kempler
Website Curator: Stephen W Berrick

Last updated: February 15, 2006 01:29:54 GMT

S4PM