Ocean Color Time-Series Project:   2005 Annual Report

Major Accomplishments To Date

1)  Giovanni.  This user-friendly data analysis tool has generated wide acclaim for ease of use and versatility.  It has been used in 18 presentations and publications that we are aware of, and others have been seen at meetings.  It has recently been expanded to include aerosols.

2)  SeaWiFS/Aqua Merged Data Products Completed.  Covers the period July, 2002-December, 2005, including The output products coming out of the merging process are daily global 9-km resolution maps of: Chlorophyll concentration (CHL), CHL confidence intervals, Colored dissolved and detrital absorption coefficient at 443 nm (CDM), CDM confidence intervals, Particulate backscattering coefficient at 443 nm (BBP), BBP confidence intervals, Coverage

3)  Refined Nimbus-7 Orbit.  Required for improved products from CZCS data, this has been the result of intensive investigation and can potentially contribute to improvements in all Nimbus-7 sensor data records.

4)  Data Merger Workshop.  An international group of ocean color data merging experts convened to discuss the state and future of ocean color data merging.  A report has been produced and is in the final stages before publication.

5)  Merging Procedure Documentation.  A paper in Remote Sensing of the Environment detailed the methodology for producing consistent merged SeaWiFS/MODIS datasets with improved spatial coverage for daily observations.

6)  Analysis of Recent Ocean Chlorophyll Trends.  A Geophysical Research Letters paper documented recent changes in global ocean chlorophyll, showing that previous declines in the open ocean occurring since 1980 have leveled off.  However, increases in coastal regions have occurred.  Furthermore, open ocean gyres are declining in chlorophyll, which appears to be associated with increasing temperatures.  There were over 40 hits in various sites on the Internet and 2 telephone interviews from reporters were conducted. 

7)  Consolidation of Ocean Color Data.  A major objective of the Ocean Color Time Series effort has been to consolidate data under a single location and provide access to all data through a single web site.  This has been accomplished through the Ocean Color Web site.

8)  Continued ESDS Participation.  Consistent and active participation be team members in general ESDS activities and in the Reuse sub-group has been maintained.

9)  Participation in IOCCG Workshop on Ocean Color Applications.  Project personnel have actively participated in this working group, led by Christopher Brown.

10)  Initial Full Processing OCTS.  The first full processing of OCTS using modern algorithm methodologies was accomplished.  Results indicated problem areas, especially in calibration as well as time trends, that are now being investigated.

11)  Analysis of Recent Changes in the Chesapeake Bay.  Abstract of the paper published in Geophysical Research Letters:  High freshwater flow delivers excess nutrients to Chesapeake Bay, leading to increased phytoplankton biomass, turbidity, and eutrophication. Low flow in 2002 was associated with a persistent drought that terminated abruptly in autumn 2002, followed by extremely high flow in 2003. This large difference in flow caused improved water quality in 2002 as nutrient loading subsided, and degraded water quality in 2003 with increased loading associated with high flow. Chl a in the study area was significantly higher in 2003 than in 2002, consistent with strong forcing by freshwater flow and nutrient loading in the nutrient-limited region of the Bay.  

12)  Participation in Ocean Color Algorithm Workshops.  The GSM01 model was tested, along other algorithms or models, against different in situ or synthetic data sets. First, IOCCG convened a working group to look at ocean color algorithms that produce inherent optical properties (IOPs). One of the objectives of this working group was to perform cross comparisons and evaluations on existing ocean-color IOP inversion algorithms. Eight algorithms were tested. The results and recommendations of the IOCCG working group on ocean color algorithms are compiled in a report currently under review and will be published by IOCCG in 2006.  The first part of the second ocean color algorithms exercise, the Ocean Color Bio-optical Algorithm Mini-Workshop (OCBAM), took place in September at UNH. The workshop aimed at evaluating ocean color algorithms that produce chlorophyll retrievals and other constituents and/or related inherent optical properties.

13)   UCSB Computing Cluster Designed and Built.  We have designed and built a computing cluster to be shared by the two UCSB REASoN projects. The cluster initially comprised 32 identical nodes, each configured with: AMD Athlon XP 2800 processor, 1 GB RAM, 36 GB 10K RPM SATA 150 Disk, gigabit Ethernet.

Specific Activities and Accomplishments for 2005, GSFC

A.  Data Processing

1.  CZCS

1.1.  Removed redundant scenes, producing a Level-1A archive of unique data

1.2.  The CZCS-capable MSl1info program was made ready for official distribution.

1.3.  Work began on adding CZCS processing to the Level-2 processing program: MSl12.  The program was linked with the CZCS read and navigation routines and Rayleigh tables were made by borrowing closely matching tables from other sensors.  Information on the CZCS spectral response functions (RSRs) was located and those functions, as well as Rayleigh tables should be provided soon.  The program currently runs to completion using the white aerosol and single scattering option.  The navigation is working correctly.  Most of the view geometry data produced for CZCS in the MSl12 implementation was verified.  The sensor azimuth angle was found to have artifacts and to be significantly different from values computed using a more complete model.  The better computation will be incorporated into the software.  The calibration, navigation and view geometry of CZCS data in the MSl12 (Level-2 processing) program was verified.  This required some changes to the l1czcs (create Level-1 CZCS file) and czl1merge (merge CZCS L1 data for an orbit) programs as well.  Some problems in the automated processing of the CZCS were investigated.  One problem found was that 2 files had the incorrect number of data records in the trailing header, which caused memory problems.  This was fixed to use the number of records reported by the VAX header.  Also, several CZCS files contained no navigation.  The Level-1 processing code was updated to signal this problem so the Level-2 processing could be avoided.  Work began on getting the MSl12 program to produce water-leaving radiances and chlorophyll with a single aerosol model.

1.4.  Artifacts were seen in the CZCS nLw and chlor_a.  The horizontal lines at about every 16 lines were found to originate in the 750 nm band and were present in the raw counts.  In addition, vertical bands at every 10 pixels were seen in the raw counts for the 443, 520 and 550 nm bands.  As the artifacts are in the counts, little correction can be easily done.  A method was found for identifying the lines in the CZCS 750 nm band that are low by about 1 digital count, on average.  This may be useful if the line artifacts have impact in the processing.

1.5.  Rayleigh tables were made for CZCS using the CZCS spectral response functions, courtesy Menghua Wang.

1.6.  SMMR files containing Nimbus-7 orbit data were placed on disk and programs were made to unpack the files and sample orbit data, creating 1-day files of orbit data for the entire 8 year set of files.  Coordinate transformations needed for the use of the orbit data were examined.  Examination of the Nimbus-7 orbit information revealed that there were several periods where orbit data was not available at the CZCS times.  This required the use of an orbit propagation model to supply orbit data for these times.  Software was written to examine the entire CZCS mission and the Nimbus-7 orbit data and locate time periods where orbit data would have to be interpolated to the CZCS times.  In general, the orbit data comes every other day and about 1300 gaps exist in the orbit record that will need to be filled by the orbit model.  An orbit propagation model was adapted for use with the Nimbus-7 orbit information.  The model was configured to produce the optimum orbit prediction during tests with several orbit files.  Further modifications were made to account for the variable drag that occurred over the mission as well as the apparent orbit raising that occurred in the first year of the mission.  Software is currently being written to re-write the orbit data so that: orbit data for only the day of the file is contained within the file; all CZCS times will have orbit data either from the original files or filled using the orbit propagation model; and the coordinate system for the orbit data will be converted from 1950 Inertial to Earth Centered, Earth Fixed coordinates.  The resulting orbit data will be ready for interpolation into the Level-1 CZCS files.  Software has been written for filling the Nimbus-7 orbit data with orbit model interpolations and extrapolations.  Testing has been done and several changes were made to improve the filling and resolve problems.  The orbit model was updated to do better drag fitting and to perform the fit over the entire data range instead of stressing the initial conditions.  Both of these changes helped to bring the estimated error in the fit down from 10s of km in some cases to less than 1 km and frequently in the 15 m range.  Gap ranges were adjusted to take advantages of the orbit model.  Also, the ranges were adjusted to account for the 5 leap second events which occurred through the CZCS mission.  Left unaccounted for, the leap seconds would induce orbit errors of up to 420 km in the position vectors.  The software was modified to perform extrapolations of the orbit for cases where the end period has no orbit information (this was necessary for cases where a leap second made it impossible to interpolate a period).  The filling software was also changed to avoid running the orbit model on previously modeled data.  The Nimbus 7 gap filling software was completed with the addition of a routine to convert the 1950 inertial coordinates to earth fixed.  The conversion was checked out and appears to work correctly.  Software for filling the Nimbus 7 orbit data with orbit model interpolations and extrapolations was run on all of the CZCS time period.  In the process of running the software, a bad day of orbit data was found and had to be removed to allow processing to continue.  The CZCS level-1 processing programs: l1czcs and czl1merge were enhanced to use the SMMR orbit information and create interpolated orbit data for each scan line.  A new SDS was placed in the dataset to indicate the amount of position error in the modeled portions of the orbit information.  These programs were tested and delivered.  Plots of the orbit position error were created for all of the orbit files for the entire CZCS mission.  This indicated that there were several cases where error was greater than 2 km, which is the initial limit on good quality orbit data.  All cases where the error was greater than 1 km were examined and the time range used in the modeling was adjusted until the error was reduced, frequently to less than 100 meters.  The resulting position files for the CZCS mission were delivered to the operational data area.

Efforts continued to confirm the first-year orbit raising for Nimbus-7 as indicated by the SpaceTrak two-line elements (TLEs).  The various team representatives contacted were not able to provide confirmation.  However, one individual, Dr. A. Fleig, suggested that an early mission orbit below nominal (955 km) would also be accompanied by a deviation in the nodal drift rate from one revolution per year (i.e., Sun synchronous), assuming a constant inclination angle.  The node crossing times from the TLEs were examined and indeed showed the expected deviation from Sun synchronicity for the first year of the mission.  This appears to provide the desired confirmation of

the orbit raising.

1.7.  Work started on getting a set of ancillary data for the CZCS period. TOMS version 8 ozone data was acquired on a DVD and the raw ascii files were placed on disk.  The ancillary meteorological data used previously were examined for use.  The previous ancillary data contains only wind speed and pressure, so a NetCDF archive from the Climate Data Center was determined to be a better source.  The data for 1978 to 1986 was acquired and the 'ancnrt' program is being converted to ingest this data source.  All the ancillary meteorological data for the CZCS period (1978 - 1986) was processed into standard HDF files.  The TOMS ozone data was also processed into standard HDF files for the CZCS period.  TOMS data for the CZCS period was reviewed so that partially filled files and inconsistent files could be removed.

1.8.  Near-IR correction methods were developed for CZCS processing.

1.9.  Mueller's CZCS ringing correction was examined and was implemented in the Level-2 processing and verified to be working properly.  Test results for one case were generated.  While the Mueller ringing mask works generally well for clouds over the ocean, it appears to mask an unreasonable amount of chlorophyll and aerosol optical thickness signals to the east of coastal areas.  This happens because visible radiances over land are frequently much less bright than they would be for clouds, while the 750 nm signal remains high.  As the ringing only affects the visible bands, a better estimate of those radiances would result in a better ringing mask.  Unfortunately, the visible bands saturate easily.  A compromise was implemented to only use the excess radiance at 750 nm in the Mueller mask algorithm if the total radiances in any of bands 1 - 3 (443, 520 and 550 nm) were saturated.  This method does give the desired effect of less ringing masking at the coast while still doing masking when clouds or bright desert regions are involved.  Also, the method reverts to the standard mask for cloud cases.  The ringing mask algorithm of Evans and Gordon 94 was also implemented outside of the Level-2 program and was found to be more conservative than the Mueller algorithm in many cases and it also did not mask far enough in cases with very bright clouds.  In the test case used, the assumption of oceanic nLw_520 < 0.7 was not true over large portions of the region, resulting in the maximum masking.  Several problems in previous ringing mask code were found while the current work was done.  The CZCS ringing masking code, containing the modification for the coastline, was added to the Level-2 processing code.

1.10.  Nimbus Experiment Team (NET) water-leaving radiances and coincident CZCS retrievals were found and plotted to do a rough comparison.  Only 8 of the 42 NET points had clear, matching satellite observations.  Some of the points were found to be in high (>0.3) optical thickness regions.

1.11.  The validity of using an epsilon value for an entire image was tested.  A typical CZCS file was run through Level-2 processing using 9 different constant aerosol models and the distribution of epsilon was analyzed.  The epsilon distributions of several models overlapped and the standard deviation of epsilon was about 0.05 for any one model, showing how much error there could be by assigning the same epsilon across the dataset.  This amount of error corresponds to a 6% chlorophyll error and an 8, 10 and 13% error in water-leaving radiance in the 443, 520 and 550 bands.

2.  OCTS

2 .1. Established a strategy for reprocessing OCTS in conformance with MODIS Aqua.  V0 processing of OCTS was completed and is undergoing analysis.  Tracked down operations and event history of OCTS and compared with unusual behavior in OCTS trends.  A change in tilt strategy appeared to correspond with degraded radiance data especially in 410 nm.

B.  Data Access

1.  Completed the Ocean Color Time-Series Project home page on the "reason" machine, which hosts Giovanni, and added a link to the January status report on that machine.   Created a new home page and status report on the developmental site for our new Web server -- the new DISC Web site debuted in April.    Most of the Ocean Color Time-Series Project pages are on the new site, and links have been added from the "reason" page to the new site.   The new site allows rapid implementation of new Web pages, and will foster the growth of the Laboratory for Ocean Color Users as the year progresses.

2.  Created the capability to output graphics from Giovanni in either PNG or Postscript, in addition to GIF.  Reprocessed (by OBPG) MODIS-Aqua and SeaWiFS Level 3 monthly data were ingested into Giovanni.  MODIS-Aqua SST data was added to the operational system.  Addition of 8-day data (SeaWiFS and MODIS-Aqua) to Giovanni was given a high priority.  The addition of SST data to Giovanni for synergy with ocean color data was completed. Giovanni was prepared for ingest of the reprocessed SeaWiFS data files.  Reprocessed SeaWiFS data was ingested for Giovanni, allowing reinstatement of the anomaly detection capability based on comparison with SeaWiFS climatological data.  This addition required slight changes to Giovanni to account for the new grid size.  A short Help section on the anomaly capability was also added to Giovanni.  Reprocessed MODIS-Aqua data was being ingested for Giovanni at the end of the month. 

3.  The "first stage" of the Giovanni intercomparison capability was promoted to the operational system.   Staff has already provided suggestions for enhancements to the intercomparison capability, including comparisons to climatological means for anomaly analysis.   SeaWiFS climatological data files were acquired and placed in the developmental system as the initial step for adding this capability.  The intercomparison (SeaWiFS with MODIS-Aqua) capability and the

anomaly analysis capability (SeaWiFS vs. SeaWiFS climatology) capability was promoted to the operational Giovanni-Oceans system.

4.  Announcements were sent to the ocean color mailing list (at the GES DISC) noting the "opening" of LOCUS, the return of the anomaly analysis capability, and confirming the transfer of archive and distribution responsibility for SeaWiFS data to the OBPG.

5.  G. Leptoukh performed additional statistical analyses of the potential differences occurring in averaging of the mapped (SMI) data products and averaging of the binned data products. 

6.  GES DISC summer student Erica Pinkney, from Elizabeth City State University (ECSU), created five of six planned educational modules that will be the first major component of the Laboratory for Ocean Color Users (LOCUS).  Pinkney also worked on a Giovanni + SeaDAS analysis of the potential effects of a strong upwelling event in July 2003 on chlorophyll concentrations in the Gulf Stream off the coasts of North and South Carolina, which is planned to be developed into a LOCUS research project and potential publication via ongoing collaboration with the Center of Excellence in Remote Sensing Education and Research (CERSER) at ECSU.  The initial Educational Module release (6 modules) was finished and linked to the LOCUS main page.  A message was sent to the mailing list for the "Cutting Edge" oceanographic education workshop requesting review and critical commentary on the modules.  Continuing collaboration with CERSER on a LOCUS demonstration research project (an extension of the work performed by Pinkney at the GES DISC during the summer and is planned to result in a publication) was discussed with CERSER staff.  A new LOCUS tutorial featuring the ocean color anomaly capability was completed.  This tutorial will be linked to the LOCUS page.  It is entitled: "Different Kinds of  Chlorophyll Concentration Anomalies in the North Atlantic Bloom”.  The tutorial also includes Giovanni images derived from MODIS-Aqua SST data.  Discussions with Dr. Malcolm LeCompte of ECSU/CERSER solidified the research collaboration on the examination of the chlorophyll effects of North Carolina coastal upwelling which was initiated during Pinkney's summer student tenure.  Work on this is expected to accelerate as the ECSU CERSER students address their research projects.  CERSER was undertaking the acquisition and installation of SeaDAS.  Review input on the Educational Modules was received from participants in the July "Cutting Edge" workshop.

7.  The Ocean Color Time-Series Web site and the LOCUS Web site were updated with new links.

8.   A rough draft and all graphics for a new tutorial on the use of temperature-chlorophyll scatter plots was completed, and will be promoted to the LOCUS Web site during the week of January 9-13, 2006.

9.   We initiated integration with the OBPG for providing direct (one-click) ordering of L2 and L3 daily granules from Giovanni corresponding to the parameter, temporal and spatial selections made by Giovanni users. 4. 

C.  Publications / Presentations

 1.  Revision of the Journal of Marine Systems special volume paper, "Remotely-sensed chlorophyll a observations of the northern Red Sea indicate seasonal variability and influence of coral reefs", was completed, and the paper and supporting material was returned to the editors.   The paper has been officially accepted.  The special issue is based on the AGU Fall meeting session in which our poster about the Time-Series Project appeared.  This paper included the first observational support for the results of a numerical circulation model (published in 1997) of the northern Red Sea, an important demonstration of how Giovanni analyses can augment oceanographic research.

2.  The paper "Recent Trends in Global Ocean Chlorophyll" by W. Gregg, N. Casey and C. McClain was published in Geophysical Research Letters.  There were over 40 hits in various sites on the Internet and 2 telephone interviews from reporters were conducted.  Abstract: A 6-year time series of remotely-sensed global ocean chlorophyll was evaluated using linear regression analysis to assess recent trends. Global ocean chlorophyll has increased 4.1% (P < 0.05). Most of the increase has occurred in coastal regions, defined as bottom depth < 200 m, where an increase of 10.4% was observed. The main contributors to the increase were the Patagonian Shelf, Bering Sea, and the eastern Pacific, southwest African, and Somalian coasts.  Although the global open ocean exhibited no significant change, 4 of the 5 mid-ocean gyres (Atlantic and Pacific) showed declines in chlorophyll over the 6 years. In all but the North Atlantic gyre, these were associated with significant increases in sea surface temperature in at least one season. These results suggest that changes are occurring in the biology of the global oceans. 

3.  The paper "Remotely-sensed chl a at the Chesapeake Bay mouth is correlated with annual freshwater flow to Chesapeake Bay" was published March 1 in Geophysical Research Letters.   (The abstract is provided in the 'Major Accomplishments' section.

4.  J. Acker wrote an Earth Observatory article on the Chesapeake Bay paper which appeared in GRL.  The text of the Earth Observatory article on Giovanni and the Chesapeake Bay drought/flood response was finalized.   Article:  Drought and Deluge Change Cheseapeake Bay Biology

5.  Staff attended the 8th International Conference on Remote Sensing for Marine and Coastal Environments in Halifax, Nova Scotia.  The presentation concerned the status of and potential for development of suspended sediment data products.  Demonstration of Giovanni capabilities combined with nLw(555) data products, which is a proxy for suspended sediments in coastal regions, elicited favorable comments and expressed interest in Giovanni (including the addition of anomaly analysis capability).

6.  J. Acker attended the Cutting Edge Workshop, "Teaching About the Ocean System Using New Research Techniques: Data, Models and Visualization", at the University of Washington.  A poster on Giovanni was presented during the poster session, and an invited talk on new science opportunities (focusing on remote sensing) was given on the second day.  Giovanni was also demonstrated during the workshop's "Share Fair" of oceanographic teaching resources.  The presentations resulted in strong interest in Giovanni and potential participation in LOCUS.

7.  A proceedings paper, "The Laboratory for Ocean Color Users (LOCUS) -- Merging Research with Oceanographic Education", was successfully submitted for the Marine Technology Society/IEEE Oceans '05 meeting, which took place in Washington, DC, in September.  J. Acker presented an oral presentation as well.  Potential collaboration with NOAA oceanographic education and outreach was discussed with several participants.

8.  J. Acker participated in the IOCCG workshop at the Graduate School of Oceanography, University of Rhode Island, dedicated to the completion of a report on the reasons for, and applications of, ocean color remote sensing.  Other participants at the workshop included Christopher Brown (leader), Trevor Platt, John Marra, Heidi Dierssen, and James Yoder.  Collaboration with C. Brown is expected late in the year in final preparation of the report following Brown's trip to Europe to obtain report input from European ocean color researchers.

9.  J. Acker presented the poster "Giovanni -- A Vital Tool Enabling Rapid and Accurate Climate Data Analysis for Evaluation, Assessment, and Research" at the Climate Change Science Program Workshop "Climate Science in Support of Decision Making" in November.

10.  A poster was presented on statistical considerations in Giovanni at the 2005 Fall AGU meeting.

Greg Leptoukh presented the poster, with the Ocean Color Time-Series Giovanni instance being the main target.  The presentation was entitled "Statistical Considerations of Data Processing in Giovanni Online Tool".

11.  An abstract featuring Giovanni was submitted to the U.S. Climate Change Science Program workshop "Climate Science in Support of Decision Making" which took place in December.   The presentation covered several different aspects of Giovanni, including ocean color data.

12.  An abstract was submitted to the AGU Oceans 2006 meeting featuring the Giovanni chlorophyll concentration anomaly capability.  This work builds on the initial results of the tutorial, and is entitled "Spatial and Temporal Coherence of SeaWiFS Chlorophyll Concentration Anomalies in the North Atlantic Bloom (1998-2005) Examined with Giovanni". 

13.   The Science Focus! article "The Blue, the Bluer, and the Bluest Ocean",

http://disc.sci.gsfc.nasa.gov/oceancolor/scifocus/oceanColor/oceanblue.shtml, which features use of Giovanni, was linked to the Science Focus! home page.

D.  Data Merging

E.  ESDSWG Participation

Participated in the NASA Earth Science Data Systems Working Group (ESDSWG) Reuse Working Group monthly conference calls, reviewed and provided comments for the Reuse Portal web site, and participated in a Reuse Working Group survey.  Reviewed and made extensive written comments as a contributing author for the ESDSWG Reuse Working Group's Reuse Enablement System (RES) Trade Study Report to be submitted to NASA HQ.  Co-authored poster presentation entitled 'Earth Science Software Reuse Enablement Systems' (J.J. Marshall, et al) presented at the Fall 2005 AGU Meeting.  Reviewed and provided comments for NASA Headquarters Draft Policy for System Software Acquisition for NASA Data System Solicitation.  Reviewed/commented on proposed language for NASA procurement and grant notices regarding software reuse.  Submitted information for the web portal on three meetings of interest to the Earth science software reuse community.  Attended the 3-day meeting of ESDSWG in Baltimore and participated in the Software Reuse Working Group break-out sessions. 

UCSB REASoN Ocean Color Data Merging - Annual Report (Jan. 2005-Dec 2005)

SCIENCE COMPONENT

Ocean Color Data Merging Model, Codes and Data Processing

Various codes related to the ocean color data merging model (GSM01; Maritorena et al., 2005) or data I/Os have been developed or modified in order to adapt the data merging processing scheme to run on the computer cluster we have developed for our merging activities. The main change is related to how normalized water-leaving radiances are extracted from the SeaWiFS and MODIS-Aqua HDF files. In the past, this was performed using the NASA SeaDAS software but to improve efficiency and speed we are now using a dedicated code that extracts the radiance data from the original level-3 binned HDF (Hierarchical Data Format) files and format them for input in the GSM01 model. Other updates to the model include a modified weighting function of the input LwNs by their uncertainty level and a minor change in the calculation of the products confidence intervals. We have defined a naming convention so the date, sensor(s) used, product, version number can be straightforwardly identified by the users.

After downloading the whole SeaWiFS (9-km) and Aqua(4.5-km) global daily Level-3 binned data archives from the NASA Ocean Color server the normalized water-leaving radiances were extracted, formatted and version 1.0 of the GSM01 SeaWiFS/MODIS-Aqua merged data set was initiated at the end of 2004. This version covered the July 4, 2002 - December 24, 2004 time period. The output products coming out of the merging process are daily global 9-km resolution maps of:

• Chlorophyll concentration (CHL)
• CHL confidence intervals
• Colored dissolved and detrital absorption coefficient at 443 nm (CDM)
• CDM confidence intervals
• Particulate backscattering coefficient at 443 nm (BBP)
• BBP confidence intervals
• Coverage

The same suite of products was also generated for each individual sensor (SeaWiFS and Aqua). The first version of the merged and individual sensor data sets was completed in February 2005. After MODIS-Aqua reprocessing 1.1 and SeAWiFS reprocessing 5.1, the whole daily L3 binned data were downloaded again for both sensors. In December 2005, the whole merging and individual sensor processings were performed again. This version (2.0) of the merged data set covers the July 4, 2002 - December 31, 2005 period. The merged and individual sensor products can be seen at

http://www.icess.ucsb.edu/~manuela/oceancolor.html .

Data QC and future developments

Our initial analyses indicate the quality of the merged products is very good overall. In version 1.0 of the merged and unmerged data sets, an error in the way the SeaWIFS data were projected resulted in some localized stripping. This problem was fixed in version 2.0. In few instances some discontinuities appear in the merged imagery, mostly in the particulate backscattering coefficient product (bbp[443]). Disagreements between the Lwns from both sensors is part of the explanation for these issues (Maritorena & Siegel, 2005). The recent reprocessing of SeaWiFS and Aqua has improved the overall consistency between SeaWiFS and Aqua Lwns and discontinuities appear even less frequent in the version 2.0 of the merged data set than in the initial version. Also, some high chlorophyll concentration values appear sometimes in oligotrophic waters. This problem is related to failures in the CDM retrievals. In the current version of the merging code, a valid retrieval for one variable is kept even if another variable fails for the same bin which results in the "spike" chlorophyll values seen in very clear waters. In the next version of the merging code, a bin will be flagged as failed if any of the 3 retrievals fails. We are still investigating these issues and we are developing analyses to better understand when, where and why issues or discrepancies occur in merged and unmerged products. We are also developing QC procedure to track potential problems more efficiently. We also track lineage of all codes and data sets involved in the merging procedure (see the Technology component section below). We are also in the process of testing multi-days composite images from the daily merged data. We are testing 2 to 8 days combinations and we will evaluate which is the best compromise in terms of coverage. We will add MODIS-Terra data to the merging process if they become available. There is also a good chance that we will be able test MERIS data with our merging procedure.

Data distribution

We have devised a scheme to distribute the merged data sets. At first, users will be able to get access to the data through a simple Ftp server. The system will then evolved into a more sophisticated user-friendly server that will ultimately have data sub-sampling and slicing capabilities.

Publications

Beside the paper documenting our merging procedure (Maritorena & Siegel, 2005), we have recently published four papers using the GSM algorithm (Siegel et al. 2005a; Siegel et al. 2005b; Behrenfeld et al. 2005; Westberry et al., 2005). The first deals with the independence and interdependencies among the optical properties retrieved by the GSM model. The second paper documents differences in chlorophyll retrievals between the SeaWiFS/MODIS operational algorithm and the GSM01 model. These differences are important at high latitudes and seem to be related to the amount of CDOM in the water. The third one describes a new approach, based on carbon rather than Chl, to estimate primary productivity from space. The fourth one uses a modified version of the GSM model designed to allow the detection of Trichodesmium spp. Blooms using ocean color remote sensing. We have also developed new algorithms for determining spectral light penetration for ultraviolet wavebands on global scales based upon the GSM output (CDM, etc.).

Maritorena, S. and D.A. Siegel.2005: Consistent merging of satellite ocean color data using a semi-analytical model, Remote Sensing of Environment, 94, 429-440

Behrenfeld, M.J., E. Boss, D.A. Siegel and D.M. Shea, 2005, Carbon-based ocean productivity and phytoplankton physiology from space. Global Biogeochemical Cycles, 19, GB1006, doi:10.1029/2004GB002299.

Siegel, D. A. S. Maritorena, N. B. Nelson, M. J. Behrenfeld, and C. R. McClain. 2005. Colored dissolved organic matter and its influence on the satellite-based characterization of the ocean biosphere. Geophys. Res. Letters, 32, L20605, doi:10.1029/2005GL024310.

Siegel, D.A., S. Maritorena, N.B. Nelson and M.J. Behrenfeld. 2005. Independence and Interdependencies Among Global Ocean Color Properties: Reassessing the Bio-Optical Assumption. Journal of Geophysical Research , VOL. 110, C07011, doi:10.1029/2004JC002527.

Westberry T.K., D.A. Siegel and A. Submaranium, 2005: An Improved Bio-Optical Model for the Remote Sensing of Trichodesmium spp. Blooms. Journal of Geophysical Research, 110 (C6): Art. No. C06012.

Meetings, Workshops

We did participate in an IOCCG (International Ocean Colour Coordination Group ) Working group on ocean color data merging. This working group is led by Dr. Watson Gregg (NASA/GSFC). The working group met in Bethesda, MD in May 2005 and looks into the strengths and weaknesses of different approaches for data merging and is tasked to make recommendations on various issues such as data access, required merged products, sensors and data characterization, merging methods and evaluation criteria. A first draft of the report was finalized in December 2005 and is currently under review at IOCCG.

We have also participated in two ocean color algorithms workshops in which the GSM01 model was tested, along other algorithms or models, against different in situ or synthetic data sets. First, IOCCG convened a working group to look at ocean color algorithms that produce inherent optical properties (IOPs). One of the objectives of this working group was to perform cross comparisons and evaluations on existing ocean-color IOP inversion algorithms. Eight algorithms were tested. The results and recommendations of the IOCCG working group on ocean color algorithms are compiled in a report currently under review and will be published by IOCCG in 2006. The first part of the second ocean color algorithms exercise, the Ocean Color Bio-optical Algorithm Mini-Workshop (OCBAM), took place in September at UNH. The workshop aimed at evaluating ocean color algorithms that produce chlorophyll retrievals and other constituents and/or related inherent optical properties. The algorithms evaluated needed to be useable with current ocean color satellite data, and their suitability for producing climate data records (CDRs) was also examined. The performance of the algorithms were be tested using a data set derived from the NASA bio-Optical Marine Algorithm Data set (NOMAD). The main conclusion of the 1st part of OCBAM is that no further improvements will be achieved in the chlorophyll algorithms without accounting for the effects of other constituents. In other words, algorithms that yield information about dissolved and particulate materials, backscattering or IOPs in general are the future of ocean color. More algorithms will be evaluated during the 2nd phase of OCBAM (1st semester of 2006). Algorithms and models will be tested against NASA's NOMAD data and the NASA OBPG matchup data set. Algorithms will also be used and compared using satellite data to assess their robustness and consistency when applied globally.

TECHNOLOGY COMPONENT

Computing Cluster

We have designed and built a computing cluster to be shared by the two UCSB REASoN projects. The cluster initially comprised 32 identical nodes, each configured with:

• processor: AMD Athlon XP 2800
• RAM: 1 GB
• disk: 36 GB 10K RPM SATA 150
• network: gigabit Ethernet

An additional identical node serves as the cluster controller. Ten new cluster nodes (w/ 1GB memory, high performance SATA disk drives, 2.8GhZ AMD processors and 100/1000 network interfaces) were added in November 2005. Local connectivity is provided by 2 gigabit Ethernet switches. All cluster systems run the Linux operating system (currently Fedora Core 3) and support the MATLAB and IDL/ENVI (including SeaDAS) analysis environments. Initially, We have implemented our own job submission and scheduling environment, since the standard packages we had been using (e.g. NPACI ROCKS) were found to be dependent on specific operating system releases. With the new nodes, we were able to update the cluster to ROCKS software w/ 40 compute nodes. The cluster is fully operational, and has been used in the development of the versions 1.0 and 2.0 of the GSM merged and unmerged data sets..

Online Storage

Our online storage strategy is simple: all of our data is kept online, in network-accessible filesystems. We implement this strategy by purchasing storage in units of fully-redundant NFS server "bricks" (e.g. if we require 8 additional TB then we purchase 2 8-TB servers). The replicate server (a "back-up brick", or BUB) is housed in a separate building and synchronized (via rsync) nightly with its primary twin. A separate backup server keeps rolling 30-day snapshots and monthly backups of all user files (as opposed to data granules), making recovery from most user "cockpit errors" completely automatic and nearly instantaneous. At the end of 2005, 8.1 TB of RAID6 disk space was added to the original 16 TB online disk space (i.e. 16 primary + 16 backup). Most of this storage is dedicated to product generation, including local copies of the input MODIS and SeaWiFS granules. The storage resides on a dual AMD 2.2GhZ processor system w/ 2GB of memory, RAID1 OS drives, and dual 100/1000 network connections, one to the public network and one to the private/cluster network.

Data Management

Our primary data management effort is a redesign of the ESSW "Lab Notebook" metadata management system. ESSW version 3 will track the lineage of all data products generated by our REASoN projects. To date, we have assembled a database of all metadata tags used to describe our REASoN processing, and are conducting evaluations of several commercial and open source workflow management systems, including Thetus Publisher, VisiQuest, and the Kepler framework. We are also exploring partnering with the JPL "GENESIS" ESIP, specifically to connect their SciFlo framework to ES3.

We are also developing software that will transparently record system events (e.g. file I/O, program invocation) that can be used to automatically capture product lineage. To date, we have developed prototypes for IDL and shell scripts. The IDL system requires a "pre-compilation" step but no modification to IDL code. The shell script system requires using a shell with system call tracing enabled, which is completely transparent and requires no modification to any existing programs. Both prototypes write activity records as XML messages intended for ingest into the new Lab Notebook. Our next step will be to test these prototypes in production runs of IDL/SeaDAS scripts and locally-written standalone programs.

The MODster namespace redirection server, prototyped during our previous ESIP award, is being reimplemented as Java servlets, with the intention of handing it off the ESIP Federation's Products and Services Committee to operate as a Federation service. The new version of MODster also features a "site crawl" capability, which allows it to redirect granule requests to large sites with stable URL hierarchies, without any effort on the host site's part. This capabilityis currently being tested with the GSFC DAAC data pool, and with JPL's online SST products.

We have implemented procedures for automatically retrieving MODIS and SeaWiFS data granules from the GSFC, EDC, and NSIDC DAACs. All datasets we retrieve (or create) are kept online, and are accessible by at least 3 mechanisms: through our local distributed filesystem; by granule name through MODster; and by attribute through our local implementation of the Alexandria Digital Library.

Publications

Bose, R. and Frew, J., 2005. Lineage Retrieval for Scientific Data Processing: A Survey. ACM Computing Surveys, vol. 37, no. 1, pp. 1-28.

Janée, G. and Frew, J., 2005. A Hybrid Declarative/Procedural Metadata Mapping Language Based on Python. Lecture Notes in Computer Science, Volume 3652, Sep 2005, Pages 302 - 313.

Meetings, workshops

Co-PI Frew chaired the 17th International Scientific and Statistical Database Management (SSDBM) Conference, held at UCSB on 27-29 June 2005. Several of the papers dealt with Earth science data management issues. The proceedings are available at http://2005.ssdbm.org/, and will be published on the ACM Digital Symposium DVDs and in the ACM Digital Library.

Frew, J. "Earth System Science Server." Presentation to the Geospatial Data Providers Workshop, National Geospatial Digital Archive, Santa Barbara, CA, 07 March 2005.

Metzger, D. and Frew, J. "MODster Namespace Management." Presentation to the ESIP Federation Summer Conference, San Diego, CA, 13 June 2005.

Frew, J. "MODster: Namespaces and Redirection for Earth Science Data." Presentation to the Workshop on Grid Middleware and Geospatial Standards for Earth System Science Data, National e-Science Centre, Edinburgh, Scotland, 06 September 2005.