For information on absolute astrometry, Hubble Advanced Products (HAP), and the DrizzlePac software, please see the following resources.

________________

TIMELINE:

December 2019:  MAST data products now include updated absolute astrometry for all WFC3 and ACS images. The World Coordinate System (WCS) in the image header were updated and may include  one or more corrections. The first makes use of a updated version of the Hubble Guide Star Catalog (GSC version 2.4.0) which updates the coordinates of the guide stars with the positions from Gaia DR1. This reduces the typical uncertainties in the positions of the guide stars to ~200 mas over the entire sky. Combining this with knowledge of the instrument distortions, an a priori correction is made for any data acquired prior to October 2017 (GSC240).  When possible, an second correction is applied by aligning sources in each HST image directly to an external reference catalog catalog, referred to as an a posteriori correction. While some observing modes (e.g. grism and moving target observations) cannot be aligned to any catalog, or the alignment may fail due to a lack of sources in either the HST image or the reference catalog, approximately 80% of ACS/WFC and 50% of WFC3/IR frames have been directly aligned. For these data products, the typical pointing uncertainty is reduced to ~10 mas, although the uncertainties increase for observations further in time from the Gaia reference epoch (2015.0 for DR1, 2015.5 for DR2). The software used to produce these drizzled products is described on the Pipeline Astrometric Calibration page.

December 2020: MAST began production of Hubble Advanced Products in the HST data calibration pipeline for WFC3 and ACS. These Hubble Legacy Archive (HLA)-style mosaics comprise the data from a single HST visit which are aligned to a common astrometric reference frame. These new 'Hubble Advanced Products' (HAP) are referred to as 'Single Visit Mosaics' (SVMs) and are described in a MAST Newsletter article from December 2020. The data products are all drizzled onto the same north-up pixel grid and may include improved relative alignment across filters for datasets acquired within the same visit, enabling easy comparison of the images through multiple filters. When possible, sources in the images have been aligned directly to the Gaia catalog to improve the WCS. SVM data products with both relative alignment (by filter) and absolute alignment to Gaia will contain the string 'FIT_SVM_GAIA' in the 'WCSNAME' keyword in the science extension of the image header. The software used to compute these new data products is described in the DrizzlePac documentation for Single Visit Mosaic Processing.

November 2021: MAST began production of HAP source catalogs as part of the SVM data products. BecauseSVMproductsincludeanadditionalrelativealignmentacrossfiltersina visit,the drizzled imagesmayusedto generatepoint sourceand segmentcatalogsduring pipeline processing. Thesecatalogssupersedethoseproducedby theHubbleLegacy Archiveand will be the basis ofthenext version of the Hubble Source Catalog.

April 2022:  A new Hubble Advanced Product (HAP) is now  distributed through MAST. These are cross-visit, cross-proposal mosaics called Multi-Visit Mosaics (MVM), which combine public observations of fields observed multiple times by ACS and WFC3 into a set of products drizzled onto a common, pre-defined pixel grid. These new products were described in a MAST Newsletter article from May 2022 and complement the existing HAP Single Visit Mosaics (SVM) released in December 2020.

August 2022:  A new instrument science report was published: 'Improved Absolute Astrometry for ACS and WFC3 Data Products' (ACS ISR 2022-03; WFC3 ISR 2022-06). 

Abstract:
As of late-2019, MAST data products for ACS and WFC3 include improved absolute astrometry in the image header World Coordinate System (WCS). The updated WCS solutions are computed during pipeline processing by aligning sources in the HST images to a select set of reference catalogs (e.g. Gaia eDR3). We compute statistics on the alignment fraction for each detector and estimate the uncertainties in the WCS solutions when aligning to different reference catalogs. We describe two new types of Hubble Advanced Products (HAP), referred to as Single Visit Mosaics (SVMs) and Multi Visit Mosaics (MVM), which began production in MAST in late-2020 and mid-2022, respectively. The SVM products include an additional relative alignment across filters in a visit, and the drizzled images are used to generate point source and segment catalogs during pipeline processing. These catalogs supersede those produced by the Hubble Legacy Archive and will be the basis of the next version of the Hubble Source Catalog. The MVM data products combine all ACS/WFC, WFC3/UVIS, or WFC3/IR images falling within a pre-defined 0.2° x 0.2° 'sky cell' for each detector+filter, which are drizzled to a common all-sky pixel grid. When combining observations over a large date range, MVMs may have photometric errors of several percent or systematic alignment errors when combining visits with different catalog solutions. We therefore recommend these to be used as ‘discovery images’ for comparing observations in different detectors and passbands and not for precise photometry.

Usage 

Images downloaded from the archive after reprocessing with the new Enhanced Pipeline Products code will have headerlets added as extra extensions to the FITS file. A new python notebook, 'Using updated astrometry solutions', will familiarize users with the structure of the new FITS images and demonstrate how the primary WCS may be changed to any other preferred solution. These instructions will also show how to back out the new WCS updates entirely if desired (see the section below on 'Caveats').

Alternatively, any of the new WCS solutions may be downloaded from MAST/STScI as separate headerlet files and applied to existing data. For users who wish to manually reprocess existing data, the 'updatewcs' task in the STWCS package as used by the Enhanced Pipeline Products code will be able to automatically connect to the astrometry database to retrieve and apply the headerlets. Python functions for creating, updating, and applying headerlets to FITS images are described via the Headerlet User Interface.

Text modified from HST Astrometry Project Overview

Guide Star Catalogs

Historically, the accuracy of HST absolute astrometry has been limited primarily by uncertainties in the celestial coordinates of the guide stars . GSC1 as specified in the Guide Star Catalog. GSC 1.1 had nominal rms errors per coordinate of ~0.5 arcsec per coordinate, with errors as large as ~1‐3 arcsec reported near the plate edges. This accuracy improved substantially in October 2005 (during Cycle 15) with the introduction of GSC 2.3.2, which had where rms errors per coordinate of were reduced to ~0.3 arcsec over the whole sky.As of November 2019, an   An updated version of the catalog (GSC 2.4.0) has now been releasedwas released in October 2017, improving the celestial coordinates with the positions from GAIA Gaia DR1 and reducing errors to < 30mas over the entire sky. Thus, after After including uncertainties in the positions of the science Instruments instruments (SIs) in the alignment of the focal plane to the Fine Guidance Sensors (FGS), the total error in HST absolute astrometry is ~1 arcsec for observations made with GSC 1.1, ~0.3 arcsec for those made with GSC 2.3.2, and ~0.1 arcsec when using the new 2 arcsec for those with GSC 2.4.  .0. These errors are reduced to ~10 mas for observations with a posteriori alignment to Gaia. A summary of the GSC properties and associated errors is given pointing errors over the HST lifetime and the expected accuracy of the updated WCS solutions is provided in Table 1. 

Table 1: GSC Properties Key Guide Star Catalog releases and associated errors 

...

HST Astrometry Project

The coordinates populated in the FITS headers of HST observations retrieved from DADS (the HST Data Archiving and Distribution Service) were derived based on the guide star coordinates in use at the time of the observation. As the accuracy in these catalogs were refined over time, the pointing accuracy of HST has also improved. Table 1 lists the catalog in use at the time of installation of the three main imaging cameras (WFPC2, ACS, and WFC3) and the typical errors at each epoch.

The goal of the HST Astrometry Project is to correct these inconsistencies in the archival data products as much as possible.  As observations are processed or reprocessed in the HST pipeline, their World Coordinate System (WCS) will be updated to use the most accurate solution available. There are 2 two types of corrections that can be performed.:

• a priori priori         : correct the coordinates of the guide stars in use at the time of observation to the coordinates of those guide stars as determined by GAIA and apply Gaia, applying a global offset to the WCS
• a posteriori posteriori  : identify sources in the HST image and cross-match with positions from an external reference catalog (such as GAIAGaia) to derive a new WCS based on improve the WCS (fitting x/y to RA/Dec)

Note that the a priori corrections will still include any errors in the HST focal plane alignment and are only relevant for observations which executed prior October 2017 (eg. prior to the release of GSC 2.4.0), and these will still include small errors in the alignment of the science instruments to the HST focal plane. The a posteriori corrections corrections are limited to imaging instruments for which there are enough an adequate number sources to define a good reference catalog for matching. These solutions remove uncertainties in the focal plane and are expected to have the smallest absolute astrometric error.

Implementation

The key to implementing improvements to the astrometry is the use of headerlets, self-contained FITS extensions containing a WCS transformation which can be attached to a FITS file and applied to the primary WCS. An observation can have multiple headerlets, each of which may have astrometry derived by differing methods. As HST data is processed/reprocessed, all available headerlets will be present as FITS extensions in the archived image with the best solution applied to the primary WCS.  A full description of headerlets and how the WCS information is stored may be found in the Headerlet Primer for the DrizzlePac software.

A database has been created to contain all the headerlets that is used by the xxxtask  to save and retrieve headerlets during pipeline processing. (Do we need this part?) This database is accessed using a restful web service as well as a simple web-form. Prior to enabling this database in the HST pipeline, it was prepopulated with headerlets derived from:

• the original pipeline astrometry based on guide star positions at the time of observation
• a priori corrected astrometry based on updating the guide star positions to the GAIA DR1 reference frame
• a posteriori corrected astrometry based on sources in the Hubble Legacy Archive (HLA) images fitted to the Hubble Source Catalog (HSC). These positions are primarily based on the Pan-STARRS catalog, which is on the GAIA reference frame but with larger errors. These are only available for a subset of the ACS and WFC3 images that were public prior to October 2017.

Pipeline processing now includes a source finding step and cross-identification to a limited set of external reference catalogs; namely, GAIA DR1 and DR2.  Additional astrometric catalogs (like Pan-STARRS, SDSS, 2MASS, GSC 2.4 etc) may be added to the pipeline in order to obtain the best a posteriori solution, only after they have been determined to provide high quality alignment. In addition to loading this headerlet to the database, it will append all other available headerlets to the image before it is archived and available to users.

WCS Naming Conventions

Successfully aligning an observation to GAIA to Gaia using the a posteriori processing will result in an update of the 'active' WCS of the image with the new solution and the new headerlet extension. This headerlet not only includes the WCS keywords which define the transformation from pixels to GAIAGaia-aligned positions on the sky, but it also contains information about how this solution was derived along with the errors to be expected based on the fit. 

The various WCS solutions are identified by the WCSNAME keyword found in each FITS headerlet and use the following naming convention: 

wcsName = OriginalSolution - CorrectionType

where  where OriginalSolution may be either

•    OPUS : initial ground system wcs, no distortion correction
•    IDC_xxxxxxxxx : initial distortion corrected wcs  (where xxxxxxxxx = geometric distortion model used, eg. the rootname of the IDCTAB reference file)

and  and CorrectionType may have several forms

•    GSC240 : 'a priori' WCS where guide star coordinates are corrected from the original reference frame (e.g. GSC1.1 or GSC2.3) to the GAIA Gaia DR1-based GSC2.4.0
•    HSC30 :   'a priori' WCS corrected from the original reference frame to the Hubble Source Catalog (HSC v3.0) frame, which is based on GAIA on Gaia DR1
•    FIT-IMG-ReferenceCatalog RefCat  : 'a posteriori' WCS derived from matching matched to a reference catalog, where 'IMG' implies individual image catalog fits each FLT is separately aligned to the reference catalog
•    FIT-REL-ReferenceCatalog RefCat   : 'a posteriori' WCS derived from matching matched to a reference catalog, where 'REL' implies images were aligned to each other before a catalog alignmentthat FLTs within the same filter within the same visit are aligned before a global catalog alignment
•    FIT-SVM-RefCat : 'a posteriori' WCS matched to a reference catalog, where 'SVM' implies that FLTs in multiple filters within the same visit are aligned before a global catalog alignment

         and REFcat may be one of the following when an adequate number of matches are found in the HST frame to compute the linear transformations (shift, rotation, scale) to sky coordinates:

•    Gaia eDR3
•    GSC 2.4.2
•    2MASS

More details on interpreting the WCS names may be found on the Astrometry in Drizzled Products page. Several examples are listed below A list of possible 'active' WCSNAME values populated in the image headers is provided in Table 2.

Table 2: Sample active WCSNAME keyword values and the corresponding WCSTYPE description. The best WCS keywords in the image header is the FIT-SVM solution which has the best relative and absolute astrometry. 

Usage 

(Still needs revision)

Images downloaded from the HST archive after reprocessing with the new Enhanced Pipeline Products code will already have headerlets added as extra extensions to the FITS file. The primary WCS may be changed to any other preferred solution using the xxx 'apply_headerlet_as_primary' ? task. Alternately, any of the available headerlets may be downloaded from MAST/STScI and applied to existing data.

If you run your own version of the pipeline it (?) should automatically connect to the database and retrieve/apply the headerlets.

→ LINK to JUPYTER NOTEBOOK (Varun)

Some suggested basic examples for looking at the data:

____

from astropy.io import fits
fits.info('/internal/hladata/ENVS_OUTPUT/ALIGNDEV_12Oct19/popen-gw1/test_alignpipe_randomlist_J8C020/j8c041sdq_flc.fits')

No. Name Ver Type Cards Dimensions Format
0 PRIMARY 1 PrimaryHDU 279 ()
1 SCI 1 ImageHDU 253 (4096, 2048) float32
2 ERR 1 ImageHDU 57 (4096, 2048) float32
3 DQ 1 ImageHDU 49 (4096, 2048) int16
4 SCI 2 ImageHDU 249 (4096, 2048) float32
5 ERR 2 ImageHDU 57 (4096, 2048) float32
6 DQ 2 ImageHDU 49 (4096, 2048) int16
7 WCSCORR 1 BinTableHDU 59 14R x 24C [40A, I, A, 24A, 24A, 24A, 24A, D, D, D, D, D, D, D, D, 24A, 24A, D, D, D, D, J, 40A, 128A]
8 HDRLET 1 NonstandardExtHDU 22 (60480,)
9 HDRLET 2 NonstandardExtHDU 26 (112320,)
10 HDRLET 3 NonstandardExtHDU 26 (112320,)
11 HDRLET 4 NonstandardExtHDU 26 (112320,)
12 HDRLET 5 NonstandardExtHDU 26 (112320,)
13 HDRLET 6 NonstandardExtHDU 26 (112320,)
14 WCSDVARR 1 ImageHDU 15 (64, 32) float32
15 WCSDVARR 2 ImageHDU 15 (64, 32) float32
16 D2IMARR 1 ImageHDU 15 (64, 32) float32
17 D2IMARR 2 ImageHDU 15 (64, 32) float32
18 WCSDVARR 3 ImageHDU 15 (64, 32) float32
19 WCSDVARR 4 ImageHDU 15 (64, 32) float32
20 D2IMARR 3 ImageHDU 15 (64, 32) float32
21 D2IMARR 4 ImageHDU 15 (64, 32) float32
22 HDRLET 7 NonstandardExtHDU 26 (112320,)
23 HDRLET 8 NonstandardExtHDU 26 (112320,)

___

from stwcs.wcsutil.headerlet import headerlet_summary
headerlet_summary('/internal/hladata/ENVS_OUTPUT/ALIGNDEV_12Oct19/popen-gw1/test_alignpipe_randomlist_J8C020/j8c041sdq_flc.fits',columns='HDRNAME')

EXTN              HDRNAME                                                   WCSNAME                            DESCRIP
8         j8c041sdq_flt_OPUS-hlet.fits                                 OPUS
9        OPUS2019-06-04                                                    IDC_0461802ej                     Default WCS from Pipeline Calibration
10       j8c041sdq_flt_OPUS-GSC240-hlet.fits                 OPUS-GSC240                     Guide Stars updated to GAIA coordinates
11        j8c041sdq_flt_IDC_0461802ej-GSC240-hlet.fits IDC_0461802ej-GSC240     Guide Stars updated to GAIA coordinates
12       j8c041sdq_flt_OPUS-HSC30-hlet.fits                    OPUS-HSC30                      Guide Stars updated to GAIA coordinates
13       j8c041sdq_flt_IDC_0461802ej-HSC30-hlet.fits    IDC_0461802ej-HSC30       Guide Stars updated to GAIA coordinates
22      IDC_0461802ej                                                          IDC_0461802ej
23      IDC_0461802ej-FIT_REL_GAIADR2                          IDC_0461802ej-FIT_REL_GAIADR2

____

from stwcs.wcsutil import headerlet
headerlet.get_headerlet_kw_names('/internal/hladata/ENVS_OUTPUT/ALIGNDEV_12Oct19/popen-gw1/test_alignpipe_randomlist_J8C020/j8c041sdq_flc.fits',kw='WCSNAME')

Implementation

The key to implementing improvements to the astrometry is the use of headerlets, self-contained FITS extensions containing a WCS transformation which can be attached to a FITS file and applied to the primary WCS. An observation can have multiple headerlets, each of which may have astrometry derived by differing methods. As HST data is processed/reprocessed, all available headerlets will be present as FITS extensions in the archived image with the best solution applied to the primary WCS.  More details on how the WCS information is stored in headerlets may be found on the page Astrometry in Drizzled Products.

Caveats

While the majority of calibrated HST data products are now aligned to a common absolute reference frame, further improvements may be possible via manual realignment using the drizzlepac tools.  This is particularly true for exposures acquired in the same visit where the WCSNAMEs does not contain the string 'FIT_SVM_***'.  For standard drizzled data products:

• Short and long exposures obtained in the same visit may no longer be aligned due to potentially different number of Gaia matches.
• Exposures in different filters (eg. narrowband vs broadband) which were obtained in the same visit may no longer be aligned to one another, for example, if each filter had a different number of matches to Gaia.

Furthermore, grism images will now be offset from their direct image counterparts, where only the later of which may be aligned to an external reference catalog. In order to preserve relative alignment between grism and direct images, users may wish to back out the updated WCS solutions entirely, as described in Section 5 of the python notebook, '['OPUS', 'IDC_0461802ej', 'OPUS-GSC240', 'IDC_0461802ej-GSC240', 'OPUS-HSC30', 'IDC_0461802ej-HSC30', 'IDC_0461802ej', 'IDC_0461802ej-FIT_REL_GAIADR2']Using updated astrometry solutions'.