Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DraftMASTDOCS and version 22.07
Excerpt
hiddentrue

Panel
borderColor#00617E
borderWidth2px
borderStylesolid

Key elements of MAST interfaces are summarized here, and include Portal tutorials for users new to MAST.

On this page...

Table of Contents
maxLevel3
indent24px

Getting Started with MAST

Login

Certain operations require that you log into MAST. This includes access to most data from active JWST observing programs. See MAST Accounts for details.

Tip

If you have a MyST account and login to MAST you may subscribe to notifications about JWST program data availability, even for observations that have not yet executed.

Exclusive Access Data

All data from JWST will eventually become public, and anyone may conduct searches for JWST data. However, when data fall under an exclusive access period (EAP) the files may only be retrieved by authorized persons. To retrieve EAP data from MAST:

  • You must have a MyST account. See this URL to create one: https://proper.stsci.edu/proper/authentication/auth
  • Login to MAST at any point prior to retrieving data. See MAST Accounts for details.
  • Those with MyST accounts must be authorized by the PI of the program to retrieve EAP data.
    • PIs of observing programs may grant access to Co-Is (or anyone with a MyST account) by visiting MyST and selecting Registered UsersUpdateManage Access to Exclusive Access Science Data.

MAST Interfaces

Web Applications

MAST offers a multiple web-based tools to search for and retrieve science and engineering data, or for other information. Links to these tools appear in the table below.

Search TypeSelect a collection...URL
Observations of a target or fieldMAST Observations by Object Name or RA/Dechttps://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html
Observations Matching a Program IDObservations by Program ID
Files matching Instrument Keyword ValuesJWST Instrument Keywords
Wavefront Sensing productsJWST WSS
DOI Portal

https://mast.stsci.edu/portal/Mashup/Clients/DOI/DOIPortal.html
Engineering Database
https://mast.stsci.edu/portal/Mashup/Clients/jwstedb/jwstedb.html
Keyword Dictionary
https://mast.stsci.edu/portal/Mashup/Clients/jwkeywords/index.html
Note
titleBrowser Issues

Note that the MAST Portal has incompatibilities with certain browsers and settings. In particular, pop-up blockers must grant an exception for the mast.stsci.edu domain. For details see Introduction to the MAST Portal in the Portal Guide.

API

Many users find scripted access to MAST searches and data retrievals to suit their needs better. The most popular interface offered is the Python package astroquery.mast. A brief tutorial is appears near the end of this article, and more extensive documentation is offered in the JWST Archive Manual in the Using MAST APIs chapter.


Anchor
Portal Tutorials
Portal Tutorials
Portal Tutorials

The MAST Portal search panel is shown below. It is a starting point for your interactions with MAST.

Panel
borderColor#00617E
bgColor#E3EEF1
borderWidth2
borderStylesolid

All of the searches described below start with selecting the appropriate collection for a search, from the drop-down menu shown at right.

Search results are displayed below the search panel, as shown in the example below.

Panel
borderColor#00617E
bgColor#E3EEF1
borderWidth2
borderStylesolid


Results consist of Filter dialogs (left); a table of matched results (center) showing rows for public (white background), Exclusive Access (yellow background, with the padlock icon), and planned but not yet executed observations (orange background); and the AstroView tool (right) which shows the spatial extent (colored footprints) of the exposures superimposed on an image of the sky. For details see Field Guide to the Portal.

Searches with the Portal

Some types of Portal search return Observations, which are really collections of all available data products, including uncalibrated, intermediate, fully calibrated, and combined science data. See Linkages in the Portal to learn more about how various kinds of data products are associated in the Portal. See Download Basket to learn how to select and download one or more of these types of products.

Target Name or Coordinates

A basic search will return Observations from all hosted MAST missions. This search will provide access to all levels of JWST data products.

Expand
titleStandard Search for Target/Coordinates...

InstructionNotes
1Select 'MAST Observations by Object Name or RA/Dec' from the 'Select a collection…’ pull-down menu at the left of the search panel.
2

Enter a target name and (optionally) a search radius, which will be resolved to coordinates. See Basic Search for syntax rules.

m57 r=1m

3Click the Search button to the right of the target dialog.

4
Optional: Filter the results in the left panel by selecting Mission: JWST and Instrument. For example: MIRI and MIRI/IMAGE

5

Evaluate the search results grid. It will look something like the example.


An advanced search allows you to customize the query by additional attributes, and can return Observations for all hosted MAST missions. This search will provide access for access to all levels of JWST data products (unless you choose to restrict the calibration level). 

Expand
titleAdvanced Search for Targets...

InstructionNotes
1Select 'MAST Observations by Object Name or RA/Dec' from the 'Select a collection…’ pull-down menu on the upper-left.
2To search using more criteria than target name or position, click the Advanced Search link below the target dialog.

3

Search by multiple criteria with filter dialogs. In this case:

  • Mission: JWST will greatly accelerate the query
  • Instrument: MIRI. Note typing the name of the instrument in the dialog box will accelerate the selection.
  • Target Name: NGC6720*. Including the wildcard at the end will select the target name and variants, which may include offset positions.

4

Click the search button at upper-left.

5

The search results grid will look similar to the example. Select one or more Observations by ticking the box(es) at the left of the results table, then choose a download method


Program ID

This type of search will return all Observations for a given observing program (a.k.a. proposal number). This is a quick way for Investigator Teams to search for all data related to their science program. 

Note

This search will match program IDs for both JWST and HST, even though the results for one mission are not scientifically related to the other. It is easy to filter the results table for the mission of interest.

Expand
titleSearch by Proposal ID...

InstructionNotes
1Select MAST Observations by Proposal ID from the 'Select a collection…’ pull-down menu on the upper-left.
2Enter one or more proposal numbers, then click the Search button.

3Filter the results to select the common target VV191. (Note the variations in the spelling.)

4The search results grid will look something like the example. Select one or more Observations by ticking the box(es) at the left of the results table, then choose a download method

JWST Instrument Keywords

This type of search for JWST data provides a much larger set of criteria, including important metadata that are not available in standard searches such as:

  • the flag for time-series observations (tsovisit)
  • exposure type (exp_type, which could indicate, e.g., coronographic observations)
  • observation sequence identifier (observtn)
  • program category (category, e.g., COM and ERS where all data are instantly public)

Beware that this type of search matches data for only one selected JWST instrument, and only the highest-level calibrated science FITS files

Note
titleHighest-level products only!

This kind of search will not provide access to ancillary files, nor to uncalibrated (e.g., Level-1b) files if higher-level products exist. It will also not match calibrated files that are not in FITS format, such as source catalogs or light curves.

Expand
titleInstrument Keywords...

InstructionNotes
1Select JWST Instrument Keywords from the 'Select a collection…’ pull-down menu on the upper-left, and select the instrument of interest from the pull-down menu below that.

2Click the Advanced Search button.

3Open filter dialogs of interest and enter search parameters. In this example we search COM and ERS programs for Single-Object Slitless Spectra taken in Time-series mode.

4Click the Search button.

5Optional: Filter the results table for Level-3 (fully calibrated and combined) data products.

6The search results grid will look something like the example. Select one or more Observations by ticking the box(es) at the left of the results table, then choose a download method
Tip

There is a way, using scripted queries, to search by values of instrument keywords and return Observations, that is, complete collections of all data products that match the query criteria. See API Tutorials for details.


Anchor
Retrieval
Retrieval
Data Retrieval Options

There are multiple options for downloading data products from MAST. Each option has its strengths, as noted below. 

Tip

Searches that return Observations to the results table contain links to all available JWST products associated with each Observation. Place one or more results in the Download Basket to view and select exactly which products (e.g., uncalibrated data) to download.

Direct Download

This is the simplest form of download, and works for individual entries in the results table. The download bundle is streamed to your machine through your browser.

Note
titleMRP files only
This option does not download all available products, but rather the most highly calibrated and combined science products. It is similar to downloading from the basket when the Minimum Recommended Products option is selected.
Expand
titleDirect Streaming Download...

InstructionNotes
1Click the file icon in one of the rows in the results table.

2

After a moment a pop-up dialog may appear, asking if the file should be saved. Note that the files are zipped for transfer efficiency. 

Tip

Whether a dialog appears, and the specific wording in the dialog, depend upon browser settings. The location where the files will be written also depends upon your browser settings.

Anchor
Basket
Basket
Download Basket

This form of download is intended for retrieving multiple observations at once, and for retrieving all (or only selected) types of products. There are three options: immediate streaming download (which imposes a size limit), and retrieval by curl script or ftp staging (which do not).

Streaming Download

This example applies to results that contain Observations, rather than individual files (as you would have after an Instrument Keyword search).

Expand
titleStreaming Download from Basket...

InstructionNotes
1Select one or more rows in the results grid.
2Click the basket button just above the grid to add the selected files to your download basket.

3

The Download Manager window may look something like that below. Note that the number of files associated with JWST Observations can  be very large (nearly 7500 in this case).

4

The next task is to select which files to include in the download bundle, which consists of two steps:

4a

Recommended: use one or more filter dialogs to select types of products to include or exclude.

  • Note that selecting Minimum Recommended Products will hide lower-level (uncalibrated) products from the file selector tree.
  • Selecting SCIENCE and INFO categories will include all files necessary for re-running the calibration pipeline on your own machine.
  • Not including the AUXILIARY category will have the effect of excluding guide-star files. This may greatly decrease the volume of data (7282 files in this example)
  • Selecting the UNCAL group will include uncalibrated (Level 1-b) files.

    Tip

    Your selection for the MRP checkbox will be remembered for subsequent downloads during your Portal session.

4b

Required: Select specific files or sub-directories of files to include.

  • check the boxes of observations and/or files you wish to download.
  • Note: checking the box of a parent observation automatically checks the boxes of all child files in the dataset.
  • optional: click the triangle on the left to review the detailed contents of the sub-directory.

6
Optional: After selecting exposures of interest, click the Retrieve References button to bring up a dialog where you can choose one or more of all the calibration reference files that were used by the CAL pipeline to produce the selected data products.

7Click the Download button.

8

Select the format for the download package from the pull-down, then click the Download but

Optional: Check the "Remove completed files from basket" box. This will purge download basket, which may help avoid confusion the next time you prepare to download files using the basket.

Download via cURL Script

This type of retrieval is asynchronous and has no limits to the volume of data to download. It is more robust against internet interruptions, and can be resumed if interrupted.

Expand
titleDownload cURL retrieval script...

InstructionNotes
1Follow instructions 1–7 for Streaming Download (above).
2Select Curl from the file Format pull-down menu.

3

Save the cURL shell script in a directory where you wish the data files to be stored.

Tip

Whether a dialog appears, and the specific wording in the dialog, depend upon browser settings. The location where the files will be written also depends upon your browser settings.

4

Bring up a shell and execute the downloaded script.

Note: If you are retrieving Exclusive Access Protected data, you will need to either provide your MyST login credentials, or have set an environment varialbe with your Auth.MAST token, in order for the shell to complete the retrieval successfully.

Tip

The cURL script is targeted to a bash shell. Specifying bash on the command line allows it to be invoked from any unix shell.

Code Block
languagebash
bash MAST_2018-10-31T2028.sh
Note
titleMAST Auth Tokens

Authentication for access to EAP data via a cURL or other scripts is managed in MAST via tokens. See MAST API Tokens to learn how to create or update a MAST.auth token.

Batch Download

Note

Staging files for ftps retrieval is currently supported for a few MAST missions (including JWST and HST), but not all. It is highly recommended to instead retrieve large numbers of files via cURL scripts.

Expand
titleStaging for FTPS Retrieval...

InstructionNotes
1Follow instructions 1–6 for Immediate Download (above).
2Click the Batch Retrieval button.

3

Select staging from the file Delivery Method pull-down menu and (if you are not logged in) provide the email address where you wish the notification to be sent.

Note: If you are logged in to MAST, the e-mail address on record will be pre-filled.

4
Acknowledge the email pop-up notification, and check your email at the delivery address you specified.

5
The email notification will give the location of the files that have been staged.
6
Use a third-party client such as Cyberduck or wget to retrieve the files.

API Tutorial

The following tutorial introduces the basics of MAST queries with the astroquery.mast package. More extensive documentation is offered in the JWST Archive Manual in the Using MAST APIs chapter.

Expand
titleAPI Example...

The set of all available parameters for this type of query is summarized here.  Consider a search for JWST/NIRCam observations in observing program 1073, using the F277W filter. Use the .query_criteria() method.

Code Block
languagepy
matched_obs = Observations.query_criteria(
        obs_collection = 'JWST'
        , proposal_id = '1073'
        , instrument_name = 'Nircam'
        , filters = 'F277W'
        )

The result is an astropy Table object, with one row per matched observation.

Tip

If for a particular query you are not interested in observations from any mission except JWST, specify it with the parameter obs_collection='JWST'. This will narrow the list of possible matches considerably, and speed up your query.

Having found Observations that match your criteria, the next step is to fetch a table of data products associated with each Observation.

Note

Some observations contain huge numbers of associated or linked files, sometimes in excess of 10,000 products. This is particularly true for NIRSpec MSA, or MIRI and NIRCam slitless spectroscopy, but may also be true for large mosaics of images. Each observation is likely to contain many files in common, such as guide-star files and ancillary products.

It is strongly recommended to retrieve product lists one or a few at a time from each observation to avoid server timeouts, and then to construct a set of unique products from the combined observations to avoid large numbers of duplicated products.

The following retrieves a list of tables of data products for each observation, and returns combined table containing unique data products. 

Code Block
languagepy
if (matched_obs > 0):
    t = [Observations.get_product_list(obs) for obs in matched_obs]
    files = unique(vstack(t), keys='productFilename')

If at least one observation matched the search criteria the above call returns a table of unique products, one per row, which could number in the hundreds or thousands. You may wish to filter the results by masking all but a limited number of file suffixes and excluding certain sub-strings. 

The selected products may now be downloaded to your local machine. Note that you will need to login with a valid Auth.MAST token to download exclusive access (EAP) data. If the token is needed but not supplied, the you will be prompted to enter one. 

You may download all the products in the files table, or select a subset if you prefer. The following will select L-1b products (i.e. the raw, uncalibrated, *_uncal.fits) from the data product list for retrieval. Other types of products may instead be selected, either by matching product name sub-strings or selecting named products in the  productSubGroupDescription.

Code Block
languagepy
manifest = Observations.download_products(
           files, 
           productSubGroupDescription='UNCAL',
           curl_flag=True
           )

Setting the optional curl_flag parameter to True will instead download a bash script that contains cURL commands to fetch the files at a later time. This approach is highly recommended for large numbers of files. The name of the download script will be something like: mastDownload_YYYYMMDDhhmmss.sh, where the latter part of the name is a numeric timestamp. What remains is to invoke the downloaded script on your machine to retrieve the files. 


What's in the Box

All data products for all selected observations will be bundled together for delivery. When the Zip or tar file is unpacked, data for each observation/visit/exposure will appear in a separate sub-directory. 

For each sub-directory, the data bundle includes by default the highest-level data products, plus all parent data. For example, if an observation/visit/exposure combination resulted in Level-2 data products, all Level-1 products would automatically be included unless the user explicitly chooses otherwise.

Tip
titleThe Download Manifest

The zip (or tar) file will include a file called MANIFEST.HTML which lists each file name, a short description, and whether access is restricted. It will also note any files that could not be downloaded and the reason why (e.g., if you do not have permission to retrieve them).

File Names and Content

The semantic content of science files may be inferred from the file suffix, and the filename signature. For example, files ending in _uncal.fits are Level-1b uncalibrated science products. See the JWST documentation for details: https://jwst-docs.stsci.edu/understanding-data-files/file-naming-conventions-and-data-products.


For Further Reading...

Reference Documents

User Support