Frequently Asked Questions about JWST Data

I am a PI of a (partially) executed JWST program.

When can I get my data?

Cycle-1 data started to appear in MAST beginning on 14 July 2022.

How do I authorize my Co-Is to have access to data from that program?

Direct your browser to the MyST site, and follow the instructions in the MAST Accounts page.

How do I learn when JWST observations have been obtained?

There are multiple ways to discover JWST Observations. For instance:

• To be notified when JWST data appear in the archive, become available to all users, or are reprocessed, use the subscription service in the MAST Portal.  This is the recommended way to find out when data from program IDs get ingested or receive updates.
• In the MAST Portal, click Advanced Search, enter "JWST" in the Mission filter and enter a range of observation start dates, e.g., "2022-07-02"  and "2022-07-09" in the Start Time filter boxes, then click the Search button.  This will search for all JWST observations that were started between 2 July 2022 through 9 July 2022.
• Using the Python library astroquery.mast, form a query_criteriasearch with the parameters obs_collection="JWST" and some other parameter such as instrument_name="MIRI" . See the astroquery.mast documentation for details.

How do I know whether data from my program have been reprocessed?

It is best to "subscribe" to data from a program so that you are notified after reprocessed data are available in MAST. See the article Program Subscriptions and Notifications for details. Select the Reprocessed check-box under subscription type; it is best to select a Daily notification frequency to avoid being bombarded with notifications (see next FAQ).

Absent a subscription, it is more challenging to determine whether extant data in MAST have been reprocessed recently. First read the JDox article JWST Operational Pipeline Build Information for context. In particular it calls out two FITS header keywords to look at: CAL_VER and CRDS_CTX. If files you have on your machine show an earlier version than what is shown in MAST, you may want to re-retrieve the data. To see which versions are in the archive,

• Do an Instrument Keyword search for data from the relevant instrument and the Program ID of interest
• In the results table, view the following fields (you might want to “Edit Columns” in the Portal to show only these):
  • Filename
  • Date (this is the date the file was created, which reflects when processing was last performed)
  • cal_ver to show the version of the processing software
  • crds_ctx to show the context of the CRDS reference files

It is possible that files in MAST are awaiting a new reprocessing because of very recent updates to reference files. There is no easy way to tell that, except by subscribing to the relevant data.

I subscribed to data notifications for a program, but now I get dozens of identical notifications per day when those data are reprocessed.

Why are there so many notifications?

You probably requested a "RapidFast" notification frequency, and possibly for product levels of "2" and higher. Notifications are generated less than an hour after new data products are ingested into the Archive (whether new or reprocessed). For many programs it takes hours, or sometimes days, for all data from a given program to be processed. During this time, many notifications are generated as many (perhaps dozens) of the L-2 products are archived; potentially dozens or even hundreds of notifications. Unfortunately, the notifications are not specific enough to link to only the data that have been reprocessed. Instead, the link will take you to a Portal results page with all data from the program.

How can I reduce the number of notifications?

Editing two settings in the subscriptions menu should help:

1. Select a notification frequency of "Daily" rather than "RapidFast." Then individual triggers will be combined into a single notification for that day.
2. Request notifications for product Level "3" rather than "2." The L-3 products are (usually) combined from L-2b (calibrated) products, so except for extracted spectra, there are many fewer products from which notifications will be generated.

If you have already established a subscription, you can click the "Subscriptions" link in the Portal and edit the frequency and product level parameters for each subscription. Read Program Subscriptions and Notifications to see how to do this.

What is the usual delay between the end of an observation and when it first appears in MAST?

That is a difficult question to answer with precision. A lot of the data gets processed within an hour or two, but there are a few edge cases where it takes more than a day. But that’s after the data processing system gets all of the pieces from Flight Operations. Many hours can elapse between the end of an exposure and transmission of that data to the ground. This is largely because JWST does not have continuous contact with the ground in the high-bandwidth channel that is required to transmit high-volume science data. That said, it is likely that most data will appear in MAST within a day or two of the end of the last exposure in an observation. 

JWST Generates so many files. Which are most important?

Generally the Level-2b (calibrated) and Level-3 (calibrated and combined) products are most appropriate for science analysis. If you wish to re-calibrate data, or combine calibrated products in a different way, you may need to retrieve lower-level products. Consult the JDox article Getting Started with JWST Data for details.

The Portal won't load Observations into my Basket. What do I do?

Some Observations (i.e., rows in the Portal Search Results table) are associated with a very large number of files. Loading too many Observation into the Basket at once can result in a pop-up error message like the following:

This problem is particularly likely for JWST Observations with many extracted spectra (e.g., NIRCam Wide-Field Slitless Spectra), or Observations containing lots of spatial dithers or other spacecraft maneuvers (as for an extended mosaic or moving targets). To get around this problem you can do the following:

1. Select fewer Observations, then load those into the Basket, select files for download, then empty the Download Basket when finished. Repeat as necessary until you have retrieved file from all Observations of interest.

2. Use the MAST API. See the Astroquery Search and Retrieval section of the Using MAST APIs article. Follow the link to the Jupyter Notebook for API Large Downloads.

   Warning
   Using the API may be the only choice for the very largest programs, where even one Observation over-fills the Portal Download Basket. There is no way to tell whether there are too many files for the Basket other than to try to load an Observation.

I'm writing a paper that analyzes data from JWST. How do I create a DOI to reference the data I used?

Navigate your browser to the special DOI Portal. Follow the instructions in the Special Searches article in the Portal Guide, specifically the section on Create a DOI.

Why do some JWST Observations appear in searches where they should not?

Unfortunately, a few JWST NIRSpec observations have bad footprints: the area they subtend on the sky is very much larger than it should be. Thus some spatial searches will match all observations with footprints that intersect with your search area, plus these observations with errant footprints.

This issue will be fixed following the February 2023 upgrade to the pipeline processing software. Within a few weeks of the upgrade, the offending data will be reprocessed and the footprints will become sensible again.

Primary Reference Sources

• JWST User Documentation Home (JDox)
• MAST JWST Archive Manual
• MAST Portal Guide

For Further Help...

JWST Help Desk

• https://stsci.service-now.com/jwst (requires login)

MAST Help Desk

• https://stsci.service-now.com/mast (requires login)
• email archive@stsci.edu