Reprocessing JWST Data
After observations are executed on the telescope, JWST science data are processed to produce fully calibrated data products. See the JWST Science Calibration Pipeline Overview for details. The data calibration is being continuously improved over time as better, in-flight characterizations of instrument performance are derived. JWST science data are periodically reprocessed to leverage new improvements and provide the best quality products to the community. Reprocessing may happen after:
- new versions of calibration software are installed in Operations,
- new calibration reference data are delivered to the Calibration Reference Data System (CRDS),
- new data are obtained for an observing program, e.g., where individual observations need to be repeated.
The schedule for reprocessing data, and related details, are discussed in the article Reprocessing JWST Data.
The Product Pedigree
Pedigree of Your Data
It is useful to identify the pedigree of your data to help you figure out if you have the best available products. There are a few keyword records in the headers of science FITS files that record this information:
| Keyword | Description |
|---|---|
CAL_VER | The JWST Science Calibration Pipeline version used to process data (with a value like "1.4.6"). |
CRDS_CTX | The context of calibration reference data that was used to process data (with a value like "jwst_1061.pmap"). The context is basically a self-consistent mapping of best reference files to be used for processing to a specific version of the calibration pipeline. |
DATE | The date/time that a product was created. |
The operational version of the pipeline is typically updated every few months, but the pipeline is also released independently to the user community for custom processing. See JWST Operational Pipeline Build Information for details. Reference file updates may happen as often as a few times per month.
Pedigree of Data in MAST
You can view pedigree of the data products in MAST by performing a search of the JWST Instrument Keywords collection for the appropriate instrument, and selecting the Program ID for your data. After the search, look for the above-named pedigree keywords in the results table. If the values of the pedigree keywords are higher than those of the data on your machine, or the creation date is later than that of your data, the products in the Archive are newer
There are dozens of fields displayed in the results table. To find your fields of interest more quickly, click the "Edit Columns" button, click the "Select None" button, and then check the boxes next to the pedigree keyword names, as well as that for the filename column.
Notifications of Data Updates
...from MAST
You may elect to receive notifications from MAST when data from particular JWST Programs or Observations are reprocessed. See Program Subscriptions and Notifications in the Portal Guide to learn how to subscribe.
Some JWST programs, or even observations within programs, can take many hours (or even days) to process. If you subscribe to JWST data it is strongly recommended that you choose to receive notifications no more frequently than "Daily" to avoid being bombarded with messages. To manage the frequency of your subscriptions see the section on Updating Subscriptions.
Stale Data
You will see a warning dialog in the MAST Portal if you attempt to download Observations that are queued to be reprocessed. You have the option to proceed with the download, cancel, or subscribe to receive a notification when the data have been updated in MAST.
The wait time specified in the dialog for calibration to complete is not currently meaningful. It may take many hours or even a few days for updated data to be available in MAST.
...from CRDS
You may elect to be notified when new reference files are delivered to CRDS by subscribing to a list server. There are separate distribution lists for each instrument. See the sub-section "Reference Files and my data" in the article JWST Data Calibration Reference Files. The notification email will include a list of updated reference file names and a summary of why the reference files were updated. It may be take some time between the delivery of a new reference file and its application the operational pipeline.