Use the download overlay to refine your selection of specific files to retrieve, and to select a method for download.
On this page...
Download Overlay Overview
The download overlay, with selectors or action buttons in numbered boxes. The overlay selectors vary slightly among missions, as described below.
After choosing Datasets in the search window, use the download overlay panel to refine your selection of the underlying files. Use the selectors to refine the selection criteria, then use the action buttons to review the products and choose a method for downloading files.
The Elements in Detail
The selector and action elements are described below.
1: Selection Summary
The selection overview displays the number and volume of files you've selected. It also displays the total number of files that are available for download. In this case, we've selected "All" but excluded guidestar files.
Note: If some selected files are not available to you due to an exclusive access restriction, you will get a warning with the number of files that have been excluded. You must log in using your MyST account and be authorized to retrieve them. See Downloading Exclusive Access Products for more details.
2: Quick Select Options
You can use the "quick selections" to do just that: quickly select a subset of files.
All will select all file categories across processing levels, but for JWST will not include guidestar files. These must be added manually using the dedicated selector described below.
None clears all selections.
Recommended is the minimum set of files that are recommended for scientific analysis. This is a good shortcut for users are satisfied with the data calibration, but may not be applicable to your particular scientific needs.
3: Data Processing Levels
This selector shows or hides types of products based on their classification as defined by the archive or mission pipelines. The example at left applies to the HST mission. You can select entire calibration levels (e.g "Uncalibrated"), as well as individual file types within those levels ("e.g. CSPEC_THUMB").
The suffix indicates the semantic type of the file contents (e.g., CSPEC), and is not the file extension (e.g., .fits
). The specific product suffix depends on the instruments of each Mission. View the mission-specific conventions in MAST Data Product Types.
For HST
HST results now include data from HASP, the Hubble Advanced Spectral Products. You will see this option available for most COS and STIS observations, in addition to the standard product levels.
For JWST
An option to in the top right corner. This option provides the files necessary to run the calibration pipeline yourself.
will appear4: Guide Star Files
Note that this selector is only applicable to the JWST mission.
Guidestar files are not included by default, but may be selected. Only the highest calibration level of products are available.
5: Regex Matching
Regular expressions are a powerful selection mechanism, and the Download Overlay supports Javascript-style regular expressions. Using expressions is not for the faint of heart, however, and it takes some practice and experience to use them effectively. See Regex in the Download Overlay Page for detailed examples.
6: View and Export Selection
View All Selected Files will open a preview of the files you've selected so far. This is particularly useful if you have entered a regular expression (see above).
Export Selected File List saves this list of files as a CSV file, with columns corresponding to those displayed in the preview. See the corresponding sections below for more details on both.
6.1 The Preview and Pagination
When previewing file metadata, the filenames are separated into groups based on the parent Datasets.
Large numbers of products will span multiple pages. The current product range and total number of products are displayed at the bottom of the product-selection table. Paging controls can be modified to navigate more efficiently.
Note: not all metadata are visible in the graphic shown at left. It may be necessary for you to scroll horizontally to see all metadata.
6.2 Exporting to CSV
Dataset,Filename,Product Level,Suffix,Instrument,Filter / Grating jw02734001001_02101_00001,jw02734001001_02101_00001-seg001_nis_cal.fits,2b,_cal,NIRISS,NIS_SOSSTA jw02734001001_02101_00001,jw02734001001_02101_00001-seg001_nis_uncal.fits,1b,_uncal,NIRISS,NIS_SOSSTA jw02734001001_02101_00001,jw02734001001_02101_00001-seg001_nis_rate.fits,2a,_rate,NIRISS,NIS_SOSSTA jw02734001001_02101_00001,jw02734001001_02101_00001-seg001_nis_rateints.fits,2a,_rateints,NIRISS,NIS_SOSSTA jw02734-o002_t002_niriss,jw02734-o002_t002_niriss_clear-gr700xd-substrip256_x1dints.fits,3,_x1dints,NIRISS,NIS_SUBSTRIP256
The file list can also be exported to CSV format. However, the CSV differs slightly: it does not visually separate files based on the parent Dataset ID. Instead, each row begins with the Dataset ID corresponding to the product filename.
Some column names contain spaces; this table should only be parsed using commas.
7: Download Folder Organization
You may choose a NESTED structure, where the main folder contains subdirectories for each parent dataset where the related files are located (middle-left graphic).
Or choose a FLAT directory structure, where files appear at the top of the download folder tree (lower-left graphic).
8: Initiating the Download
START DOWNLOAD will initiate a streaming file retrieval through the browser and produce a local ZIP file with the results. Note that the download location is set by the preference settings of the web browser. Some browsers allow a choice of download location; otherwise, the ZIP file will be downloaded to your default location.
API QUERY will generate a bash script of curl commands that you can use to retrieve files at a later time. This may be the only option for a large file payload (>1.0 TB uncompressed), and is more robust on slow connections.
To download non-public products via API Query, an Auth.MAST token must be created and provided to the generated script when prompted. You will only be prompted for this Auth.MAST token if there are non-public products requested for download. For more information on Auth.MAST tokens, please visit the Auth.MAST documentation.
8.1 Downloading Exclusive Access Products
Some new data from Hubble and JWST are temporarily available only to the Investigator teams, which is indicated with a yellow lock symbol and a warning. These products will appear in the total file count, but will not be available to download except to signed-in, authorized users.
You will need to be logged in (see pull-down menu at upper-right) and be authorized to retrieve EA files. See MAST Accounts for details.
Post-download: the Download Manifest
Within the downloaded folder, you will find a manifest.html file. This manifest includes information about key aspects of file metadata and download status.
The table below highlights these columns. Of note is that "Access" will display either "PUBLIC" or "EXCLUSIVE_ACCESS"; public data is available to all users, while exclusive access data requires authentication. "Logged In User" will give your user name if you have logged in, and "anonymous" otherwise. "Status" will generally be "OK"; see the examples below for cases where it is something different.
URI | File | Access | Status | Logged In User |
---|---|---|---|---|
mast:MISSION/product/FILENAME | MISSION/path/FILENAME | PUBLIC EXCLUSIVE_ACCESS | OK Duplicate of [file] [File] associated with [obs] ... SUCCESS | anonymous YourName |
Example 1: De-duplication
If you selected products that are shared across multiple observations (e.g., guide star files), these files will only be downloaded once. This minimizes download time and storage utilization on your machine.
URI | File | Access | Status | Logged In User |
---|---|---|---|---|
mast:JWST/product/file1.fits | JWST/product/file1.fits | PUBLIC | OK | anonymous |
mast:JWST/product/pool.csv | JWST/product/pool.csv | PUBLIC | OK | anonymous |
mast:JWST/product/file2.fits | JWST/product/file2.fits | PUBLIC | OK | anonymous |
mast:JWST/product/pool.csv | JWST/product/pool.csv | PUBLIC | Duplicate of file pool.csv in folder jw0133_01_01 | anonymous |
Example 2: Exclusive Access Data
When downloading data that requires authentication, the status for all files will display the authentication message (regardless of whether they are publicly available). In the case of duplicated files, the "SUCCESS" will be replaced with a duplication message like the one in example 1.
URI | File | Access | Status | Logged In User |
---|---|---|---|---|
mast:JWST/product/file3.fits | JWST/product/file3.fits | EXCLUSIVE_ACESS | JWST/product/file3.fits associated with jw123_0_1 ... SUCCESS | YourName |
mast:JWST/product/pool.csv | JWST/product/pool.csv | PUBLIC | JWST/product/pool.csv associated with jw123_0_1 ... SUCCESS | YourName |
mast:JWST/product/pool.csv | JWST/product/pool.csv | PUBLIC | JWST/product/pool.csv associated with jw123_0_1 ... Duplicate of file pool.csv associated with jw123_0_1 | YourName |