Example Bash Script

When retrieving a selection of files from the MAST Portal, one option is to download an auto-generated bash (shell) script. These scripts use the cURL utility to retrieve the selected files. cURL is robust, reliable, and supports many features including authentication. It is also the preferred tool for downloading large volumes of data because of the data volume and rate limits imposed by the Portal Web interface.

If you would like to customize the example script provided here to access data files in MAST, you will first need to determine the URIs for the files of interest using another API, e.g.,  astroquery.mast.

Invoke the script in a unix terminal to download the files:

bash MAST_2022-04-30T2140.sh

A look at the example script MAST_2022-04-30T2153.sh shows that, apart from some housekeeping, it does a few main things:

  1. For EAP protected data, fetch the MAST API Token
  2. Create a folder for the payload
  3. Create a MANIFEST.HTML file to report status of the requested vs. retrieved data files
  4. Retrieve each data file with a cURL command

Items 1 and 4 are described below in more detail.

The API Token

If the requested files include at least one with EAP protection, a MAST API token is required. As the following code snippet shows, the MAST token may be supplied in multiple ways.

# Check for command-line argument
if [ -z "$1" ]
then
  # Check for environment variable
  if [ -z "$MAST_API_TOKEN" ]
  then
    # Prompt the user
    echo "Please enter your token here: "
    read MAST_API_TOKEN
  fi
else
  MAST_API_TOKEN=$1
fi

Retrieve Files

The following code snippet shows how each file is retrieved with cURL. For readability, the cURL command is re-formatted here as multi-line, with linux/MacOS escape characters.

# Announce the file
cat <<EOT
<<< Downloading 
    File: mast:JWST/product/jw01409-o031_t014_nircam_clear-f212n_i2d.fits
      To: ${DOWNLOAD_FOLDER}/JWST/jw01409-o031_t014_nircam_clear-f212n/jw01409-o031_t014_nircam_clear-f212n_i2d.fits
EOT
# Retrieve the file
curl -H "Authorization: token $MAST_API_TOKEN" \
     --globoff \
     --location-trusted \
     -f \
     --progress-bar \
     --create-dirs $CONT \
     --output ./${DOWNLOAD_FOLDER}'/JWST/jw01409-o031_t014_nircam_clear-f212n/jw00839-o003_t001_miri_f560w_i2d.fits' \
'https://mast.stsci.edu/jwst/api/v0.1/Download/file?bundle_name=MAST_2022-04-30T2153&uri=mast:JWST/product/jw01409-o031_t014_nircam_clear-f212n_i2d.fits'

# Announce the second file
cat <<EOT
<<< Downloading 
    File: mast:JWST/product/jw01409-o031_t014_nircam_clear-f212n_cat.ecsv
    To: ${DOWNLOAD_FOLDER}/JWST/jw01409-o031_t014_nircam_clear-f212n/jw01409-o031_t014_nircam_clear-f212n_cat.ecsv
EOT
# Retrieve the second file
curl -H "Authorization: token $MAST_API_TOKEN" \
     --globoff \
     --location-trusted \
     -f \
     --progress-bar \
     --create-dirs $CONT \
     --output ./${DOWNLOAD_FOLDER}'/JWST/jw01409-o031_t014_nircam_clear-f212n/jw01409-o031_t014_nircam_clear-f212n_cat.ecsv' \
'https://mast.stsci.edu/jwst/api/v0.1/Download/file?bundle_name=MAST_2022-04-30T2153&uri=mast:JWST/product/jw01409-o031_t014_nircam_clear-f212n_cat.ecsv'

It is worth calling out two important cURL command-line options:

  • -H: pass custom header to the server. In this case, the MAST auth token
  • --location-trusted: Follow re-directs, and send auth to other hosts

For Further Reading...

Send comments & corrections on this MAST document to: archive@stsci.edu