One of the main tasks of the TSO WG on FY2020 is to perform testing of the steps made by the JWST Calibration Pipeline for TSO-like data. This tracking sheet contains all the information regarding pipeline testing by the members of the TSO WG. Nestor Espinoza and Sarah Kendrew will be populating this page heavily in the weeks of July 20 and 27, 2020, for NIRISS and MIRI TSO testing, respectively. Nikolay Nikolov and Tony Keyes will be populating at their own pace for NIRCam and NIRSpec, respectively.
General guidelines for Pipeline Testing
- Please, test every step identified below. If for some reason you don't have time to test a step, let Nestor Espinoza and Sarah Kendrew know.
- Use the comments in the tables to post brief comments about the status of that testing step (think of it as a quick overview for others to know what is going on). For extended commentary, or details on how you performed the testing, use the text below the tables.
- If a bug is found, please report it in Jira as a "Bug" Jira ticket. Please copy the links to those tickets in the table below under the comments.
- For algorithm improvements/enhancements, please file an "Improvement" Jira ticket. Please copy the links to those tickets in the table below under the comments.
In general, if bugs are found, the best would be to discuss them first in one of our meetings before going ahead and submitting them (unless this is high-priority, in which case we could schedule a special meeting for this). This is left at the discretion of each TSO WG member.
General comments
Before jumping to read the below table, some important comments to keep in mind while reading them:
- NIRISS testing is being done using awesimsoss NIRISS/SOSS simulations of K2-141 (which has a short transit duration, hence the selection). Care must be taken as this is the first end-to-end simulation of awesimsoss (which is under active development). In-progress notebooks of each stage testing can be found here.
- MIRI testing is being done using MIRISim simulations of MIRI/LRS.
- NIRCam testing is being done using Mirage simulations.
- NIRSpec testing is being done on CV3 data.
Stage 1: Detector Processing TSO testing
| Pipeline step | NIRISS status | NIRISS comments | NIRCam status | NIRCam comments | MIRI status | MIRI comments | NIRSpec status | NIRSpec comments |
|---|---|---|---|---|---|---|---|---|
| group_scale | DONE | The step is supossed to do nothing for NIRISS/SOSS (here, a group = 1 frame). Indeed, nothing changes when running this step, except for the length of the ASDF element in the HDU list object, which is normal. | TODO | N/A | TODO | |||
| dq_init | DONE | This step correctly initializes the data-quality flags, as it is supposed to do, according to the reference mask in CRDS. | TODO | DONE | (high-level test) | TODO | ||
| saturation | DONE | This step does what is expected to do: it captures saturated pixels and flags them in the GROUPDQ masks. The simulations used for this validation had saturated pixels and they were correctly identified (flagged) by the pipeline. | TODO | DONE | (high-level) | TODO | ||
| superbias | DONE | Tested my own bias substraction using reference files and got exact same results. | TODO | N/A | TODO | |||
| reset | N/A | N/A | DONE | This was tested on a TSO imaging file separately since this step does not currently run by default. The test passed with no issues. | N/A | |||
| linearity | DONE | Tested linearity correction using reference files and got exact same results. | TODO | TODO | TODO | |||
| rscd | N/A | N/A | TODO | N/A | ||||
| dark_current | DONE | Dark frames perform expected corrections on the data. | TODO | TODO | TODO | |||
| refpix | DONE | Currently, does not do anything to NIRISS/SOSS simulations, as reference pixels there are set to zero. When injecting a fake strain of reference pixels got some deviations that most likely have to do with DMS-detector frame rotations. | TODO | DONE | This step is sufficiently tested in unit tests. It is skipped for MIRI subarray data so only applies to FULL array imaging TSOs, and note that MIRISim's implementation of Refpix is different from the pipeline step, so may give unexpected results. | TODO | ||
| jump | DONE | Step produced no errors. Detected jumps are consistent with pipeline specifications. | TODO | DONE | high level testing is done. Inserted cosmic rays of varying strengths were detected (based on threshold) and flagged neighboring pixels (based on level above background) | TODO | ||
| ramp_fitting | ONGOING | *rateints files show small spikes that do not show on my own ramps, even when being careful with GROUPSDQ (see notebook). | ONGOING | produced rateints with 7.5 and independent linear fits; substantial difference in a subset of data. | TODO | TODO | ||
| gain_scale | TODO | TODO | DONE | Documentation states 'This currently only applies to NIRSpec exposures that are read out using a subarray pattern, in which case a gain setting of 2 is used instead of the standard setting of 1.' This step is properly skipped for MIRI data. | TODO | |||
Extended comments by instrument branch:
- NIRISS
- NIRCam
- MIRI:
- NIRSpec:
Stage 2: Spectroscopic processing TSO testing
| Pipeline step | NIRISS status | NIRISS comments | NIRCam status | NIRCam comments | MIRI status | MIRI comments | NIRSpec status | NIRSpec comments |
|---|---|---|---|---|---|---|---|---|
| assign_wcs | TODO | TODO | ONGOING | TODO | ||||
| extract_2d | TODO | TODO | TODO | TODO | ||||
| flat_field | TODO | TODO | ONGOING | TODO Build 7.5: Failed current validation test | passes for NRS2, but fails for NRS1:
| |||
| srctype | TODO | TODO | TODO | TODO Build 7.5: Logic is not correct, always defaults to POINT, even when provided EXTENDED |
| |||
| photom | TODO | TODO | TODO | TODO | ||||
| extract_1d | DONE | This step currently extracts the whole subarray as a spectrum, which produced order overlap, which makes this step produce an unusable product for science. An algorithm is in development for this step. For details, see JP-201 - Getting issue details... STATUS . | TODO | TODO | TODO Build 7.5: methodology needs to be vetted by TSOWG | additional comments pending |
Extended comments by instrument branch:
- NIRISS:
- NIRCam
- MIRI:
- NIRSpec:
Stage 2: Image Processing TSO testing
| Pipeline step | NIRISS status | NIRISS comments | NIRCam status | NIRCam comments | MIRI status | MIRI comments | NIRSpec status | NIRSpec comments |
|---|---|---|---|---|---|---|---|---|
| assign_wcs | N/A | TODO | TODO | N/A | ||||
| flat_field | N/A | TODO | TODO | N/A | ||||
| photom | N/A | TODO | TODO | N/A |
Extended comments by instrument branch:
- NIRISS:
- NIRCam
- MIRI:
- NIRSpec:
Stage 3: Time-Series Observations (TSO) Processing testing
| Pipeline step | NIRISS status | NIRISS comments | NIRCam status | NIRCam comments | MIRI status | MIRI comments | NIRSpec status | NIRSpec comments |
|---|---|---|---|---|---|---|---|---|
| outlier_detection | TODO | TODO | TODO | TODO | ||||
| tso_photometry | TODO | TODO | TODO | TODO | ||||
| extract_1d | TODO | TODO | TODO | TODO | ||||
| white_light | TODO | TODO | TODO | TODO |
Extended comments by instrument branch:
- NIRISS:
- NIRCam
- MIRI:
- NIRSpec:
