Page History
| Excerpt | ||
|---|---|---|
| ||
A tour of the UI component used to select specific files for download |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Use the download overlay to refine your selection of specific files to retrieve, and to select a method for download. On this page...
|
Component Overview
This page describes the Download Overlay, used for selecting files to download.
Component Features and Functionality
A description of the Download Overlay: 'Select By Product Type', 'Select By Product Extension', 'Select By String Match', 'Download Button' (Download By Streaming To A Zip File), and the 'Show API Query Button' (Download Via A Script).
|
Download Overlay Overview
| Panel | ||||||
|---|---|---|---|---|---|---|
|
| ||
The download overlay, with selectors or action buttons in numbered boxes. The overlay selectors vary slightly among missions, as described below. |
After choosing Datasets in the search window, use the download overlay panel to refine your selection of the underlying files. Use the selectors to refine the selection criteria, then use the action buttons to review the products and choose a method for downloading files.
The Elements in Detail
The selector and action elements are described below.
1: Selection Summary
The selection overview displays the number and volume of files you've selected. It also displays the total number of files that are available for download. In this case, we've selected "All" but excluded guidestar files.
Note: If some selected files are not available to you due to an exclusive access restriction, you will get a warning with the number of files that have been excluded. You must log in using your MyST account and be authorized to retrieve them. See Downloading Exclusive Access Products for more details.
2: Quick Select Options
You can use the "quick selections" to do just that: quickly select a subset of files.
All will select all file categories across processing levels, but for JWST will not include guidestar files. These must be added manually using the dedicated selector described below.
None clears all selections.
Recommended is the minimum set of files that are recommended for scientific analysis. This is a good shortcut for users are satisfied with the data calibration, but may not be applicable to your particular scientific needs.
3: Data Processing Levels
This selector shows or hides types of products based on their classification as defined by the archive or mission pipelines. The example at left applies to the HST mission. You can select entire calibration levels (e.g "Uncalibrated"), as well as individual file types within those levels ("e.g. CSPEC_THUMB").
| Note |
|---|
The suffixindicates the semantic type of the file contents (e.g., CSPEC), and is not the file extension (e.g., |
For HST
HST results now include data from HASP, the Hubble Advanced Spectral Products. You will see this option available for most COS and STIS observations, in addition to the standard product levels.
For JWST
An option to 
will appear in the top right corner. This option provides the files necessary to run the calibration pipeline yourself.
4: Guide Star Files
Note that this selector is only applicable to the JWST mission.
Guidestar files are not included by default, but may be selected. Only the highest calibration level of products are available.
5: Regex Matching
Regular expressions are a powerful selection mechanism, and the Download Overlay supports Javascript-style regular expressions. Using expressions is not for the faint of heart, however, and it takes some practice and experience to use them effectively. See Regex in the Download Overlay Page for detailed examples.
6: View and Export Selection
View All Selected Files will open a preview of the files you've selected so far. This is particularly useful if you have entered a regular expression (see above).
Export Selected File List saves this list of files as a CSV file, with columns corresponding to those displayed in the preview. See the corresponding sections below for more details on both.
6.1 The Preview and Pagination
When previewing file metadata, the filenames are separated into groups based on the parent Datasets.
Large numbers of products will span multiple pages. The current product range and total number of products are displayed at the bottom of the product-selection table. Paging controls can be modified to navigate more efficiently.
Note: not all metadata are visible in the graphic shown at left. It may be necessary for you to scroll horizontally to see all metadata.
6.2 Exporting to CSV
| Code Block | ||||
|---|---|---|---|---|
| ||||
Dataset,Filename,Product Level,Suffix,Instrument,Filter / Grating
jw02734001001_02101_00001,jw02734001001_02101_00001-seg001_nis_cal.fits,2b,_cal,NIRISS,NIS_SOSSTA
jw02734001001_02101_00001,jw02734001001_02101_00001-seg001_nis_uncal.fits,1b,_uncal,NIRISS,NIS_SOSSTA
jw02734001001_02101_00001,jw02734001001_02101_00001-seg001_nis_rate.fits,2a,_rate,NIRISS,NIS_SOSSTA
jw02734001001_02101_00001,jw02734001001_02101_00001-seg001_nis_rateints.fits,2a,_rateints,NIRISS,NIS_SOSSTA
jw02734-o002_t002_niriss,jw02734-o002_t002_niriss_clear-gr700xd-substrip256_x1dints.fits,3,_x1dints,NIRISS,NIS_SUBSTRIP256 |
The file list can also be exported to CSV format. However, the CSV differs slightly: it does not visually separate files based on the parent Dataset ID. Instead, each row begins with the Dataset ID corresponding to the product filename.
| Note |
|---|
Some column names contain spaces; this table should only be parsed using commas. |
7: Download Folder Organization
You may choose a NESTED structure, where the main folder contains subdirectories for each parent dataset where the related files are located (middle-left graphic).
Or choose a FLAT directory structure, where files appear at the top of the download folder tree (lower-left graphic).
8: Initiating the Download
START DOWNLOAD will initiate a streaming file retrieval through the browser and produce a local ZIP file with the results
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
This filter shows or hides types of products based on their classification as defined by the archive or mission pipelines. Examples include 'CALIBRATED', 'UNCALIBRATED', and 'REFERENCE'. Each option can be toggled on or off, displaying or hiding all products under that type from the file-selection table. When a type is selected from the dropdown list, it shows as checked. Alternatively, use 'SHOW ALL' or select one or more types after clicking 'SHOW NONE' inside the drop-down menu.
| Info |
|---|
By default, datasets for all product types are shown in the file-selection table. |
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
| Widget Connector | ||||
|---|---|---|---|---|
|
This video clip demonstrates how to choose products to display in the overlay based on the product suffix.
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
Another way to refine data products displayed in the table is to use the 'Suffixes' drop-down. It is used to display products based on the product suffix. After clicking the menu, scroll through the available list of suffixes and select those of interest to display in the file-selection table. Alternatively, use 'SHOW ALL' or select individual suffix after clicking 'SHOW NONE' inside the drop-down menu.
| Note |
|---|
'Suffix' indicates the type of data the file contains, not to a file extension. For example, if a file ends in '_x1d.fits', the product suffix is 'x1d' (based on data type containing calibrated science spectra) while the file extension is 'fits'. |
| Info |
|---|
There are various product suffixes depending the instruments of each Mission. For HST data, see the HST User documentation page for its filename format convention. |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
|
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
The string search box in the Download Overlay can be used to match product names, instruments, and filters/gratings. This can be used to further refine the displayed list of products for easy selection to display in the file-selection table. This search box offers type-ahead functionality. For example, the search box can be used to display only "txt" file extension type by entering ".txt" into the search box, which will only display products that include the string. It is not limited to file types, any string in a file name can be searched. These can then be selected for download using the selection checkboxes to the left of the product filename in the file-selection table. Also, the 'ALL', 'NONE', and 'RECOMMENDED' table selection buttons may be used to select or deselect from the currently displayed products.
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
| Widget Connector | ||||
|---|---|---|---|---|
|
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
Any products displayed in the overlay must be selected using the checkboxes to download them. The 'ALL', 'NONE', and 'RECOMMENDED' table selection buttons will select or deselect from currently displayed products. The selection checkbox next to a given Observation ID
will select or deselect all products of that Observation matching the filters applied. The selection checkbox next to a displayed product
will select or deselect only that product.
| Info |
|---|
NOTE: Products that are selected will remain selected if they are later hidden using the filters in the Download Overlay. When in doubt, look at the number of Observations and Files at the top of the Overlay, or get a complete list of products and their selection status by displaying all Product Types and Product Extensions and remove any string matching in the search box. |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
If there are a large number of products available for the selected observations and filters, they may span multiple pages. Paging allows a large number of products to be made available without impacting the speed of the browser. At the bottom of the product-selection table, the current product range (e.g., 1 - 100) and total number of products (e.g., 584) matching the current 'File Type' and 'File Suffix' selections are displayed . These paging controls can be used to change how many products are displayed on a page and to navigate through the pages of available products.
| Info |
|---|
NOTE: Products that are selected will remain selected while paging through all products. Additionally, the 'ALL', 'NONE', and 'RECOMMENDED' table selection buttons will select or deselect from all currently displayed products across all pages (not one page at a time). |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
When selecting multiple products that may be spread between multiple filters by category or suffix, or across multiple pages, it can be helpful to see a summary of only the selected products. A product summary can be displayed by clicking the 'Files=xx' link at the top of the overlay, where xx is the number of products currently selected. This opens a file summary overlay displaying ONLY the selected product files, grouped by their suffix.
| Info |
|---|
NOTE: Products can be deselected from the file summary overlay by unchecking the checkbox next to a product name and then pressing the 'UPDATE SELECTIONS' button. If products were deselected and the Close ('X') button in the top-right corner is pressed, the updates will not be saved and the unchecked products will not be removed from the selected files. |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
. Note that the download location is set by the preference settings of the web browser. Some browsers allow a choice of download location; otherwise,
otherwisethe
ZipZIP file will be downloaded to
theyour default location
the browser downloads all files (Likely the download folder).
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
API QUERY
' button will display the overlay below.| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
Another way to download products is to request a download script using the Mission Search API. Click the 'DOWNLOAD WITH API QUERY' button to show the Download By API Query Overlay. This overlay displays API commands based on the products selected to download the files one at a time. These commands can be copied from the overlay into a text editor and then run as an executable script.
Such API download scripts can be useful in a number of scenarios where a search was conducted at a different time than you want to download the products. Examples include slow internet connections, not being at the machine where the files will be stored, or very large product requests where the products can be requested in smaller batches at different times.will generate a bash script of curl commands that you can use to retrieve files at a later time. This may be the only option for a large file payload (>1.0 TB uncompressed), and is more robust on slow connections.
| Info |
|---|
To download non-public products via API Query, an Auth.MAST token must be created and provided to the generated script when prompted. You will only be prompted for this Auth.MAST token if there are non-public products requested for download. For more information on Auth.MAST tokens, please visit the Auth.MAST documentation. |
| Anchor |
|---|
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
Example of non-public products in the Download Overlay if not authorized and/or not signed in. Instead of a selection checkbox, a locked icon with a yellow background will appear, indicating permission is not granted to download those products.
| borderColor | #00617E |
|---|---|
| bgColor | #E3EEF1 |
| borderWidth | 2 |
Some missions, such as Hubble, have products that are not publicly available to everyone. These non-public (or "exclusive access") products will appear in the download overlay but will not be available for download except to those who sign in to the STScI account and authorized to retrieve those products.
|
Some new data from Hubble and JWST are temporarily available only to the Investigator teams, which is indicated with a yellow lock symbol and a warning. These products will appear in the total file count, but will not be available to download except to signed-in, authorized users.
You will need to be logged in (see pull-down menu at upper-right) and be authorized to retrieve EA files. See MAST Accounts for details.
Post-download: the Download Manifest
Within the downloaded folder, you will find a manifest.html file. This manifest includes information about key aspects of file metadata and download status.
The table below highlights these columns. Of note is that "Access" will display either "PUBLIC" or "EXCLUSIVE_ACCESS"; public data is available to all users, while exclusive access data requires authentication. "Logged In User" will give your user name if you have logged in, and "anonymous" otherwise. "Status" will generally be "OK"; see the examples below for cases where it is something different.
| URI | File | Access | Status | Logged In User |
|---|---|---|---|---|
| mast:MISSION/product/FILENAME | MISSION/path/FILENAME | PUBLIC EXCLUSIVE_ACCESS | OK Duplicate of [file] [File] associated with [obs] ... SUCCESS | anonymous YourName |
Example 1: De-duplication
If you selected products that are shared across multiple observations (e.g., guide star files), these files will only be downloaded once. This minimizes download time and storage utilization on your machine.
| URI | File | Access | Status | Logged In User |
|---|---|---|---|---|
| mast:JWST/product/file1.fits | JWST/product/file1.fits | PUBLIC | OK | anonymous |
| mast:JWST/product/pool.csv | JWST/product/pool.csv | PUBLIC | OK | anonymous |
| mast:JWST/product/file2.fits | JWST/product/file2.fits | PUBLIC | OK | anonymous |
| mast:JWST/product/pool.csv | JWST/product/pool.csv | PUBLIC | Duplicate of file pool.csv in folder jw0133_01_01 | anonymous |
Example 2: Exclusive Access Data
When downloading data that requires authentication, the status for all files will display the authentication message (regardless of whether they are publicly available). In the case of duplicated files, the "SUCCESS" will be replaced with a duplication message like the one in example 1.
| URI | File | Access | Status | Logged In User |
|---|---|---|---|---|
| mast:JWST/product/file3.fits | JWST/product/file3.fits | EXCLUSIVE_ACESS | JWST/product/file3.fits associated with jw123_0_1 ... SUCCESS | YourName |
| mast:JWST/product/pool.csv | JWST/product/pool.csv | PUBLIC | JWST/product/pool.csv associated with jw123_0_1 ... SUCCESS | YourName |
| mast:JWST/product/pool.csv | JWST/product/pool.csv | PUBLIC | JWST/product/pool.csv associated with jw123_0_1 ... Duplicate of file pool.csv associated with jw123_0_1 | YourName |
| Anchor | ||||
|---|---|---|---|---|
|