This page offers links to recommended documentation and addresses frequently asked questions related to JWST data.  Check the material here, your question might already be answered!

On this page...

Frequently Asked Questions about JWST Data

My API queries for data taken with particular instruments no longer return the expected results. Why?

The names of the JWST science instruments as viewed in MAST have been augmented to include configuration information. This affects the way that instrument names should be specified in the Advanced Search in the Portal, and in Observations.query_criteria() searches with the astroquery.mast API. Other kinds of queries are not affected. See the article JWST Instrument Names for details.

I am a PI of a JWST program. How do I authorize my Co-Is to access the data?

Direct your browser to the MyST site, and follow the instructions in the MAST Accounts page.

How do I learn when JWST observations have been obtained?

There are multiple ways to discover JWST Observations. For instance:

  • To be notified when JWST data appear in the archive, become available to all users, or are reprocessed, use the subscription service in the MAST Portal.  This is the recommended way to find out when data from program IDs get ingested or receive updates.
  • In the MAST Portal, click Advanced Search, enter "JWST" in the Mission filter and enter a range of observation start dates, e.g., "2022-07-02"  and "2022-07-09in the Start Time filter boxes, then click the Search button.  This will search for all JWST observations that were started between 2 July 2022 through 9 July 2022.
  • Using the Python library astroquery.mast, form a query_criteria search with the parameter obs_collection="JWST" and some other parameters such as instrument_name="MIRI*" . See the astroquery.mast documentation for details.

How do I know whether data from my program have been reprocessed?

It is best to "subscribe" to data from a program so that you are notified after reprocessed data are available in MAST. See the article Program Subscriptions and Notifications for details. Select the Reprocessed check-box under subscription type; it is best to select a Daily notification frequency to avoid being bombarded with notifications (see next FAQ).

Absent a subscription, it is more challenging to determine whether extant data in MAST have been reprocessed recently. First read the JDox article  JWST Operational Pipeline Build Information for context. In particular it calls out two FITS header keywords to look at: CAL_VER and CRDS_CTX. If files you have on your machine show an earlier version than what is shown in MAST, you may want to re-retrieve the data. To see which versions are in the archive,

  • Do an Instrument Keyword search for data from the relevant instrument and the Program ID of interest
  • In the results table, view the following fields (you might want to “Edit Columns” in the Portal to show only these):
    • Filename
    • Date (this is the date the file was created, which reflects when processing was last performed)
    • cal_ver to show the version of the processing software
    • crds_ctx to show the context of the CRDS reference files

It is possible that files in MAST are awaiting a new reprocessing because of very recent updates to reference files. There is no easy way to tell that, except by subscribing to the relevant data.

I subscribed to data notifications for a program, but now I get dozens of identical reprocessing notifications per day. What can I do?

You probably requested a "Fast" notification frequency, and possibly for product levels of "2" and higher. Notifications are generated less than an hour after new data products are ingested into the Archive (whether new or reprocessed). For many programs it takes hours, or sometimes days, for all data to be processed. During this time, many notifications are generated as the L-2 products are archived; potentially dozens or even hundreds of notifications. Unfortunately, the notifications are not specific enough to link to only the data that have been reprocessed. Instead, the link will take you to a Portal results page with all data from the program.

Editing two settings in the subscriptions menu should help:

  1. Select a notification frequency of "Daily" rather than "Fast." Then individual triggers will be combined into a single notification for that day.
  2. Request notifications for product Level "3" rather than "2." The L-3 products are (usually) combined from L-2b (calibrated) products, so except for extracted spectra, there are many fewer products from which notifications will be generated.

If you have already established a subscription, you can click the "Subscriptions" link in the Portal and edit the frequency and product level parameters for each subscription. Read Program Subscriptions and Notifications to see how to do this.

What is the usual delay between the end of an observation and when it first appears in MAST?

From hours to days. That is a difficult question to answer with precision because it depends on several factors. A lot of the data gets processed within a few hours, but there are a few edge cases where it takes more than a day, or even several days. But that’s after the data processing system gets all of the pieces from Flight Operations. Many hours can elapse between the end of an exposure and transmission of that data to the ground. This is largely because JWST does not have continuous contact with the ground in the high-bandwidth channel that is required to transmit high-volume science data. That said, it is likely that most data will appear in MAST within a day or two of the end of the last exposure in an observation.

Why do I sometimes get a "404 File Not Found" error when I try to retrieve files?

Data from most programs have been reprocessed fairly often. When this happens, updated files eventually replace the prior version. However, there is a latency between the time when updated files are created and when the files are migrated to the area where they are discoverable in MAST. Sometimes the delay can be more than a day. Unfortunately, there is no easy workaround. Patience is the key. 

JWST Generates so many files. Which are most important?

The Level-2b (calibrated) and Level-3 (calibrated and combined) products are usually most appropriate for science analysis. If you wish to re-calibrate data, or combine calibrated products in a different way, you may need to retrieve lower-level products. Consult the JDox article Getting Started with JWST Data for details.

The Portal won't load Observations into my Basket. What do I do?

Some Observations (i.e., rows in the Portal Search Results table) are associated with a very large number of files. Loading too many Observation into the Basket at once can result in a pop-up error message like this one:

This problem of loading too many files in the basket is particularly likely for JWST Observations with many extracted spectra (e.g., NIRCam Wide-Field Slitless Spectra), or Observations containing lots of spatial dithers or other spacecraft maneuvers (as for an extended mosaic or moving targets). To get around this problem you can do the following:

  1. Select fewer Observations, then load those into the Basket, select files for download, then empty the Download Basket when finished. Repeat as necessary until you have retrieved files from all Observations of interest.
  2. Use the astroquery.mast API. See the Astroquery Search and Retrieval section of the Using MAST APIs article. Follow the link to the Jupyter Notebook for API Large Downloads.

Using the API may be the only choice for the very largest programs, where even one Observation over-fills the Portal Download Basket. There is no way to tell whether there are too many files for the Basket other than to try to load an Observation.

How do I create a DOI to reference the data I analyzed in my publication?

Navigate your browser to the special DOI Portal. Follow the instructions in the Special Searches article in the Portal Guide, specifically the section on Create a DOI.

DOI for Uncalibrated Products

If you have re-reduced uncalibrated or intermediate products yourself with custom options in the JWST pipeline, rather than using the high-level products in MAST, you can still create a DOI for your data. Simply load the level-2b (calibrated) or Level-3 (calibrated & combined) products into the DOI basket; users who follow the DOI link will be able to access the lower-level products. We recommend that you include a note in the "About this data" field when creating your DOI that says which specific products you used for your analysis. 



Primary Reference Sources

For Further Help...

JWST Help Desk

MAST Help Desk

Data Use | Acknowledgements | DOI | Privacy

Send comments & corrections on this MAST document to: archive@stsci.edu.