Page History

Versions Compared

Key

This line was added.
This line was removed.
Formatting was changed.

Comment: Published by Scroll Versions from space DraftMASTDOCS and version 1.4

Excerpt

hidden	true

Choose an option for retrieving files from MAST

Panel

borderColor	#00617E
borderWidth	2px
borderStyle	solid

On this page...

Table of Contents

maxLevel	3
indent	24px

Direct from Search Results

Retrieving data from MAST in the Portal context requires first executing a successful search (see the chapter on Searching). Each of the rows in the search results includes a File icon which enables a direct download of a single file or observation (see One-Click Download for details).

Using the Download Manager

In the Portal Download Manager you must check one of more directories or file boxes in the File panel before download methods become available to you (see Download Basket).

Note

title	Portal Log-on

You must be logged in to the Portal to download data that is marked as Exclusive Access. Such observations appear with a yellow background in the Search Results table.

Warning

title	Browser Pop-ups

The Portal requires that pop-up blockers be disabled for the domain: https://mast.stsci.edu/ in order to download data. Check the documentation for your browser to determine how this is done, and be alert for notifications from the browser that popups are being blocked. In many cases it simply requires giving approval for pop-ups when you are asked.

Options

Click one of the buttons described below to choose the details of your download.

Browser Download

Follow the steps below to initiate a download of the selected files. The files will be bundled into a .zip or a .tar (optionally compressed) file.

	Instruction	Notes
1	Click the Download button.	Image Modified
2	Select your prefered form for the download bundle and click Download.	Image Modified
3	Save the file bundle to your local storage. The appearance of the file-save dialog depends upon your browser settings.	Image Modified

Curl Script

You may elect to download a custom Bash script, which when run on your host system will retrieve your selected data files using cURL commands.

Instruction

Notes

Click the Download button.

Image Modified

Select Curl from the file Format pull-down menu.

Image Modified

Save the cURL shell script in a directory where you wish the data files to be stored.

Download to local disk dialog (browser specific) Image Modified

Bring up a shell and execute the script.

Note: If you are retrieving Exclusive Access Protected data, you will need to either provide your MyST login credentials, or have established a MAST Auth token, in order for the shell to complete the retrieval successfully.

Note: The cURL script is targeted to a bash shell. Specifying bash on the command line allows it to be invoked from any unix shell.

Code Block

language	bash

bash MAST_2018-10-31T2028.sh

Note

title	Choosing Shippable Media

Requesting data via shippable media will involve considerable delay. Note that you that you will need to certify that you are unable to use another means of data retrieval.

Note

title	MAST Auth Tokens

Authentication for access to Exclusive Access data via a cURL or other scripts is managed in MAST via tokens. See MAST API Tokens to learn how to create or update a MAST.auth token.

Batch Retrieval

The Batch Retrieval option allows the selected files to be written to external media for subsequent retrieval. Follow the steps below to stage data on the archive.stsci.edu ftp server. Note that requesting data via shippable media will incur a substantial delay, and requires that you certify that you cannot retrieve data via staging.

Note
Staging files for ftp retrieval is currently supported for a few MAST missions (including JWST and HST), but not all. It is highly recommended to instead retrieve large numbers of files via cURL.

	Instruction	Notes
1	Click the Batch Retrieval button.	Image Modified
2	Select

staging

Staging from the

file

Delivery Method pull-down menu and (if you are not logged in) provide the email address where you wish the notification to be sent.

Note: If you are logged in to MAST, the e-mail address on record will be pre-filled.

Image Removed

Image Added

options dialog


3	Acknowledge the pop-up notification, and check your

messages at the delivery email address you specified.

Image Removed

Image Added
4	The email notification will give the location of the files that have been staged.

Image Removed

5Bring up a shell**

Delivery status request email message Image Added

Access the staging disk using an encrypted FTPS connection. See below for suggestions for modern, third-party clients.

Warning

The MAST FTP server (archive.stsci.edu) no longer supports unencrypted FTP connections. Command-line ftp clients, as in Linux or MacOS, may not work with our server.

Please see https://archive.stsci.edu/ftp.html for more information.

Options for retrieving staged data via third-party clients:

Please see the Archive Support Knowledge Base for more details on each of these clients, and to explore other options.

Expand

title	Cyberduck (Click here to expand...)

To download the app or for more help, please see the Cyberduck site.

Launch the app and select Open Connection. Image Added

Under the FTP dropdown select FTP-SSL (Explicit AUTH TLS)
Server: archive.stsci.edu
Select Anonymous Login OR enter your MyST login credentials
Select Connect

Image Added

You will see two folders: "pub" and "stage":

Pub is all of the public data available on MAST
Stage is the data that has been added to the staging area

Under "stage", navigate to the directory specified in the email notification and you should see the folder(s) containing your download.

To download the entire directory highlight the folder, then under Action on the menu bar select Download.
If you only want a few files from the directory, expand the desired directory, and then select the files you want to download.

Expand

title	FileZilla (Click here to expand...)

To download the app or for more help, please see the FileZilla site.

Launch the app.

In the Host field enter ftps://archive.stsci.edu, then your MyST email address and password in the Username and Password fields, respectively. Select Quickconnect.

If TLS connection is not set as the default, Quickconnect may be unsuccessful. If so, navigate to the Site Manager. Image Added
Select New Site
Protocol: FTP - File Transfer Protocol
Host: ftps://archive.stsci.edu
Encryption: Require explicit FTP over TLS
Logon Type: Select Anonymous OR enter your MyST credentials
Select Connect

Filezilla FTP Panel Image Added

Under Remote site, you will see two folders: "pub" and "stage"

Pub is all of the public data available on MAST
Stage is the data that has been added to the staging area

Under "stage", navigate to the directory specified in the email notification and you should see the folder(s) containing your download.

Once you locate the folder(s) or file(s) you want to download, right click and select Download.

Expand

title	wget (Click here to expand...)

To install wget or for more help, please see the wget site.

Bring up a shell.

If you requested a retrieval anonymously, run:

Code Block
wget -r --ftp-user='anonymous' --ask-password ftps://archive.stsci.edu/stage/anonymous/<anonymous requestID>/

If you were logged into MAST when you requested that your data be staged, run:

Code Block
wget -r --ftp-user='<your full MyST email>' --ask-password ftps://archive.stsci.edu/stage/<userID>/<requestID>/

Note, for internal users, there may still be a certificate problem; if you encounter an error add '--no-check-certificate' to the command.

Expand

title	curl (Click here to expand...)

To install curl or for more help, please see the curl site. Curl is likely already installed on many machines, and if not, is freely available as open source software.

Bring up a shell.

If you requested a retrieval anonymously, run:

Code Block
curl --ssl-reqd -O ftp://archive.stsci.edu/stage/anonymous/<anonymous requestID>/<filename>

If you were logged into MAST when you requested that your data be staged, run:

Code Block
curl --ssl-reqd -O ftp://archive.stsci.edu/stage/<userID>/<requestID>/<filename> -u '<your full MyST email>'

Note that you must include the filename in these commands. If you do not know the name of the file you'd like to retrieve, or want to download an entire directory, we recommend wget or one of the other methods detailed here.

Expand

title	ftp (Click here to expand...)

Warning
The MAST FTP server (`archive.stsci.edu``)` no longer supports unencrypted FTP connections. Command-line ftp clients may not work with our server.

Bring up a shell (see note below) and initiate an ftp session on archive.stsci.edu.

Code Block
% ftp archive.stsci.edu Connected to archive.stsci.edu

The appropriate login credentials will be one of the following:

If you were logged into MAST when you requested that your data be staged, use your MyST login credentials.
If you requested a retrieval anonymously (for non-exclusive access data), use:

User

- Name: anonymous
- Password:

<any syntactically valid

- <your full MyST email address>

Code Block

language	bash

% ftp archive.stsci.edu Connected to archopsvm1.stsci.edu. # [Welcome message...]

Name: anonymous
331 Guest login ok, type your email address as


the password
Password:

Navigate to the directory specified in the email notification and initiate the transfer of data files.

Code Block

languagebash


ftp> cd /stage/anonymous/

anonymous_20190107Z_2339f1c0

<anonymous requestID>
# turn off interactive prompt:
ftp> prompt
ftp> mget *

Note

title	MAC OS X 10.14+ Users

Apple has removed the native ftp application for Mac OS 10.14 (Mojave) and higher. A solution is to install a third-party file-transfer application, such as Cyberduck or wget

. If using CyberDuck, be sure to open your connection as FTP-SSL, since MAST now uses secure ftp (ftps)

See this article for help.

Other Download Methods

It is possible to search for and retrieve data using MAST web services, or the python Python astroquery.mast library. These methods are described in Beyond the Portal.

For Further Reading...

The Download Bundle

Content

Space Tools

Versions Compared

Old Version 60

New Version 61

Key

Direct from Search Results

Using the Download Manager

Options

Browser Download

Curl Script

Batch Retrieval

Options for retrieving staged data via third-party clients:

Other Download Methods

For Further Reading...