Versions Compared

Old Version 15

changes.mady.by.user Thomas Dutkiewicz

Saved on

compared with

New Version 16

changes.mady.by.user Thomas Dutkiewicz

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DraftMASTDOCS and version 22.07
Excerpt
hiddentrue

Learn how to make effective use of MAST application programming interfaces (APIs).

Panel
borderColor#00617E
borderWidth2px
borderStylesolid

MAST application programming interfaces (APIs) provide a direct and efficient means for experienced users to search for and retrieve data products. The APIs that are most relevant for JWST are illustrated via a collection of tutorials which you can customize to your own scientific needs.

On this page...

Table of Contents
maxLevel2
indent24px

MAST Web Services

Much of the functionality of MAST is provided through a variety of web services. The MAST Portal itself is a Web application that calls MAST web services to provide data search, selection, and retrieval functionality. Most of these same services are available to users via multiple MAST application programming interfaces (APIs). These APIs offer a very efficient means to create custom, scripted access to JWST data products. The figure below is a simplified view of the interaction between users and MAST back-end services.

Panel
borderColor#00617E
bgColor#E3EEF1
borderWidth2
borderStylesolid


Cartoon of MAST user interaction with MAST web servicesImage Modified

Figure 1 — Users (right) interact with a MAST Web UI or an API, either of which call back-end services (left) to access databases and data files. The information is retrieved and is forwarded back to the interface.

Anchor
auth
auth
Auth.MAST

Most data in MAST can be retrieved anonymously. But retrieving protected data with an API requires both authentication and authorization. Authorization is granted by MAST to program Principal Investigators, and may be granted by a PI to their collaborators. Authentication in the Portal is achieved by logging in with your MyST credentials; authentication for scripted retrieval is achieved with a MAST API token. These tokens, which are long alpha-numeric strings, may be provided to a script:

  • with a command-line argument
  • by a user responding to a prompt
  • by storing the token string in the shell environment variable $MAST_API_TOKEN

    • Users of bash can do this in the shell they use to access an API with the following command (or, optionally, store in their .bashrc file):

      Code Block
      languagebash
      export MAST_API_TOKEN=<token string>

Generate the token with the tokens page.

Note
titleToken Expiration Policy

For security reasons MAST tokens expire after 10 days of inactivity or 60 days after creation, whichever comes first. If you have a script which used to work but now fails with an authorization  authorization error, check to see if your token is still active. It is easy to generate a new MAST token.

Anchor
API Tutorials
API Tutorials
API Tutorials

The following subsections provide a variety of examples of how to use the MAST APIs with python or shell scripts. For a more extensive discussion of MAST APIs, with many more worked examples, see MAST Web Services. Some of the tutorials below take the form of Jupyter notebooks, which reside in an STScI notebook repository.

warning

The MAST server and the service endpoints are different for the pre-launch JWST archive vs. the multi-mission MAST archive that will be used for in-flight data.

Astroquery Search and Retrieval

The Python package astroquery provides a general means of creating custom scripts to access astronomical data repositories; astroquery.mast provides specific functionality for the MAST archive, which is analogous to Portal searches.

Note
titleLarge JWST Queries
The most highly calibrated and combined JWST observations may contain a very large number (thousands) of data products. These observations may cause timeouts or other performance problems for the Portal application. The tutorials mentioned here provide a robust method for dealing with large numbers of observations and data products.
AnchorAstroqueryAstroquery

Curl Download

pOne One option for retrieving data via the Portal Download Basket is to retrieve a "Curl" script. This is in reality a bash script which includes cURL commands to actually retrieve the data. See a description of an example download script to learn how it works; from this you can create your own download script. 

Tip

You must know the URI(s) for the data file(s) you wish to download. These may be obtained with a different API, e.g., with an astroquery search.

Warning
This notebook searches the Operational MAST archive, which will be valid for JWST post-launch, but will not work for pre-launch JWST data

.

AnchorMashupMashup

MAST API

The MAST web services may be called directly, through a mashup interface. This is a collection of interfaces that allows more functionality and flexibility for tailoring request parameters. It does, however, require significant knowledge of the supported web services, the formatting of service requests and response results, and careful programming to wrangle the arcane syntax of web service calls. See:

WarningThis notebook searches the Operational MAST archive, which will be valid for JWST post-launch, but will not work for pre-launch JWST data.

Anchor
Duplication Check
Duplication Check
Duplication Check

It is important for investigators who are preparing JWST observing proposals to know in advance if their intended targets might duplicate existing or planned JWST observations. This Jupyter notebook will help: 

Note

While helpful, JWST planned observations in MAST do not provide sufficient information to determine if a genuine duplication exists. For instance, parallel observations are omitted, and the prime target orientation may not be known until the visits are scheduled. The results in the notebook will include a link to the program APT file, which should be consulted to evaluate any potential duplication identified in MAST.

Anchor
EDB Retrieval
EDB Retrieval
JWST EDB Retrieval

The JWST Engineering Database contents are described in the chapter on Engineering Data. The EDB Retrieval tutorial shows how to retrieve multiple mnemonics over independent time ranges using python. There is also an example python script which you may customize for your own use. Both of these methods require a valid MAST.auth token.

For Further Reading...