This page archives Slack comments from Day 2, Session 2 of the Improving JWST Data Products Workshop (IJDPW).
Jeff Valenti - This channel is now open for business.
David Law - @Oliver King, Can you comment on what your final approach was for working with the data in which you saturated after just 1 group?
Oliver King - Regions saturated in the first group ultimately aren’t recoverable (with our routine at least), so we have to leave those as missing data.
David Law - Ah, sorry, I meant where 1 group was unsaturated but group 2 saturated.
Howard Bushouse - What about pixels that saturated after group 1, i.e. in group 2 or later? Are you able to recover a useable slope from group 1?
Oliver King - If I remember right, we disabled the step that drops the first frame when there’s only 1 group - that increases the uncertainty a bit, but it’s better than having no data.
Jeff Valenti - @Oliver King, Did I hear correctly that the current pipeline doesn't flag partially saturated groups reliably?
Taylor Bell - I believe the NIRSpec/PRISM JTEC-ERS observations of WASP-39b had issues with unflagged partial saturation and/or non-linearity issues and/or leaking flux. I myself didn't analyse the data for that paper, but my understanding is that the team had to expand the saturation flag to the entire column (to avoid leaking issues) and/or expand the saturation flag to the previous group (to avoid partial saturation and/or non-linearity).
Alicia Canipe - If it’s useful, I was just trying to remind myself what was implemented in the pipeline (can’t find specific readthedocs info on partial saturation). Here’s what the CalWG outlined in the 4 October 2022 notes on saturated pixel propagation:
- Only propagate flags for hard saturated pixels, combining “saturated” with “do_not_use” (bits 0 and 1, ie bit values 1 and 2, in the DQ arrays for the rateint and rate.fits files.
- Pixels that are partially saturated are not flagged in the DQ arrays for the rateint and rate.fits files.
- Pixels are still flagged (either partial or full saturation) for the affected groups in the ramp.fits files.
Oliver King - There were certainly some pixels where the e.g. 5 group data was darker than the e.g. 1-4 group data - it wasn’t massively common and usually happened on the edges of saturated regions. I haven’t had a detailed look at this in a while, so it may have improved in more recent pipeline versions.
Everett Schlawin - Similar to Jeff @Oliver King, in theory, I thought that default stage 1 saturation flagging would already reduce the number of groups where there is partial saturation. Does it misidentify saturated pixels? (I have seen cases where it has with e.g. anomalous jump detection).
Oliver King - Do you mean misidentifying unsaturated pixels as saturated, or saturated as unsaturated?
Everett Schlawin - Based on the presentation it looks like there may have been over-flagging of groups. (in my experience there were issues with anomalous flagging of saturated pixels as jumps and throwing away good early groups) . But after some discussion it looks like maybe the automatic MIRI first/last group flagging might have been a reason here.
David Law - @Oliver King, Can you remind me what the percentage correction was that you ended up making for the 'effective' flatfield on the Saturn data? I.e., what were the amplitudes of the swirls? Edit: Looks like your later figure answered this, about 0.9 to 1.1.
Oliver King - Yep - it varied a bit with wavelength, but was usually within +-10%
Michael Regan - We could do a slope fit with one sample (we do it in the near-IR). We don’t due to the uncertainty in the zero point. By default MIRI drops the first and last frame. The last frame is corrupted by the reset but maybe when it super bright that doesn’t matter. We know that group 1 can be used for bright pixels.
Jeff Valenti - @Oliver King, Bummer that the synthetic flats for Jupiter and Saturn are different.
Michael Regan - I think there’s a nomenclature problem with saturation.
James Davies - @Oliver King, Using the planet surface as a “flatfield lamp” is really clever. Can this be used for more complicated sources (i.e. points) with complicated backgrouds? Or does undersampling of the PSF complicate this?
Oliver King - I think the undersampling of the PSF may make this difficult. The atmospheres we’re looking at are relatively spatially homogenous, so neighbouring pixels are nearly always going to have a similar brightness which certainly makes bootstrapping the flat field much easier. It may be possible with other sources, but generally the more sharp the spatial variation, the harder the flat field generation is going to be.
Kevin Volk - Is this charge migration rather than "saturation"? The behaviour is similar to what we see in NIRISS that leads to the correction in the pipeline. Or is it something else?
Ben Lew - Is this because of imperfect non-linearity correction rather than saturation?
Michael Regan - I thought about charge migration/BFE but there is almost zero contrast between pixels. It could be a non-linearity problem.
Timothy Brandt - In a sense, saturation is just an extreme version of nonlinearity, right? You want to only use the groups where you trust the nonlinearity correction; it sounds like that threshold needs to be adjusted.
Michael Regan - Just to be clear NL is completely different in the Near-IR. These errors are primarily due to Brighter Fatter.
Jane Morrison - @David Grant, You picked a very difficult region to judge the non-linearity. You are in a region where non-linearity and brighter-fatter are at work. See The Brighter-Fatter Effect in the JWST MIRI Si:As IBC detectors I. Observations, impact on science, and modelling.
Jeff Valenti - @David Grant mentioned JWST-TST DREAMS: Quartz Clouds in the Atmosphere of WASP-17b.
Jeff Valenti - @David Grant, Is the code you mentioned exotic-miri?
Michael Regan - @David Grant, Don’t Alpha and Beta depend on the current difference between the pixels?
Mike Engesser - What values for Alpha and Beta have you found are characteristic? This correction is likely contaminated by the IPC kernel, which is calculated in a similar way and has values of ~3% to adjacent columns and 2% to adjacent rows.
David Grant - Across 3 TSO exoplanet datasets (MIRI LRS SLITLESS) I was finding alpha ~ -0.15, beta ~-0.03. The minus sign comes from the definition of the difference as DN_00 - DN_ij.
Everett Schlawin - @David Grant, Excellent work on the dimmer slimmer algorithm! Have you assessed how the spectrum changes (such as on the commissioning target that should be pretty flat)?
David Grant - Thanks @Everett Schlawin, I have only been scoring based on how linear the ramps are, rather than changes to end science products like transmission spectra. Happy to run it all the way through and get back to you.
Sarah Kendrew - agree that it would be nice to see impact of your corrections on the final science products. I’d also be interested in seeing what parameter space of data you’ve tested this on - ngroups, max fluence.
Taylor Bell - Yeah, very interested to see the impact on final science products! Happy to chat more about this later today or later this week.
David Grant - @Sarah Kendrew, 3 datasets so far, ngroups = {9, 100, 175}. All produce alpha and beta of remarkable similarity.
Sarah Kendrew - How similar were the max counts reached in the integrations for these different groups?
Eddie Bergeron - I've measured the fraction of charge lost to each neighbor of every pixel on an H2RG in the lab by programming artificial hot pixel grids of various brightnesses onto the array and shining a (fairly faint) uniform flatfield flux onto the grid. You can then directly measure the fraction of the incident flux that has migrated to the neighbors. Attached .ppt shows some figures. At max contrast (hot pix near saturation) the fractions going to each neighbor is ~16% and 4% for adjacent and diagonal. These are averages of all the pix on the array (as I recall). There is virtually no spatial variation across the array - I'll try to make a map to post. So if you add those numbers up you get 80% - i.e. a loss of 20% of the total flux. Still need to tweak all this up, but it does appear that there is some loss in the migration process ( @perhaps recombination). See brighter-fatter_per_pixel_NIRSpec_H2RG_ODL.pptx.
Eddie Bergeron - I assume the alpha and betas would be some kind of integral of these curves for David's algorithm (and its MIRI rather than H2RG).
Taylor Bell - @Eddie Bergeron, very interesting! David said that his measured alphas were ~15% and betas were ~3% which is remarkably close to the values you mention above.
David Grant - @Sarah Kendrew, I have checked and all the max counts for these tests were all very similar ~58,000. Would be worth testing on non-LRS MIRI data for sure.
Sarah Kendrew - Thanks! Yes, would be interested in seeing what numbers come out depending on how deep into the well we go.
Timothy Brandt - @David Law, @Leo Drake Deming, Possibly relevant to the conservation of charge upon saturation/charge leakage is Figure 3 of Data reduction pipeline for the CHARIS integral-field spectrograph I: detector readout calibration and data cube extraction.
Jeff Valenti - @Anna de Graaff mentioned jwst-msafit
Jeff Valenti - @Anna de Graaff, Sorry if you said this, but what stage of pipeline output do you model? Looking at the repository, it seems that the fitting doesn't use STScI pipeline software at all, right?
Anna de Graaff - Sorry, I'm not so well-versed in ST pipeline language, as I have been mainly working with the ESA pipeline. When I run my mock detector through the ST pipeline I run only the Spec2 and Spec3 pipelines (skipping some steps, e.g. flat-fielding). So I think that is the rate.fits files?
Jeff Valenti - That sounds right. The input to spec2 is a rate (or rateints) file. Thanks!
Ben Lew - Thanks for the fantastic work @Anna de Graaff!!! Could we use the code to model the dispersed PSF profile of a point source for optimal extraction? Can we use it for modeling the potential slit loss of fixed slit?
Anna de Graaff - Yes, absolutely! something I didn't highlight is that my code can output a full mock detector, which you can then slot into some NS data and run through a pipeline of your choosing. I.e. then you can evaluate for a known input what your algorithm is doing. I haven't explicitly modelled the fixed slit, but we might be able to construct something from the libraries that are there for the MOS shutters
Ben Lew - That will be great, thank you!
Jane Rigby - Q for @Anna de Graaff, could you please provide more details about what kinds of dedicated calibrations you’d like to see for MSA mode (rather than scaling from fixed slit)?
Jane Morrison - @Anna de Graaff, Excellent talk.
Anna de Graaff - thanks!
Jane Rigby - @Aarran Shaw’s poster presentation — very interesting! One follow-up question: there should be a database of the clock corrections that are done during every JWST ground contact. Is that dataset useful to you? (or is the cadence too low?) If so, do you have access to it? It seems to me it should be possible to point users to that data if it’s already public, and if it’s not, look into releasing it. Inviting experts from the Ops side to chime in.
Aarran Shaw - Hi Jane, thanks for the question. That is something that would be incredibly useful, and might even contain info about the clock drift!
Taylor Bell - @Néstor Espinoza, super glad and excited to have this tool! Thanks for making this
Néstor Espinoza - Thank you! The FGS data is beautiful!
Everett Schlawin - @Aarran Shaw, excellent to see the accuracy of JWST's clock measured and better than 1 second! Were the short and long wavelength times consistent? Were detailed pixel clocking times within the frame important?
Aarran Shaw - Great questions, thank you. This is all quite preliminary, so unfortunately I haven’t investigated your second question, but I will make a note of that and investigate. However, we did find (again, from this essentially first pass of the data) that the offset was consistent in SW and LW channels.
Jeff Valenti - @Aarran Shaw, Flight operations applies a correction to the onboard clock every DSN contact (nominally twice a day). Looking at the size of those corrections as a function of time since the last contact should provide a lot of information about clock drift.
Jane Rigby - Xpost with my comment above. @Jeff Valenti, I just reached out to mission systems engineering to ask whether uses have access to that dataset, and CCed you.
Jeff Valenti - @Aarran Shaw, I guess you got the time of your exposure from the FITS header. What keyword(s)?
Jeff Valenti - @Aarran Shaw responded: I will get back to you on the keyword (I don’t have my computer at this very moment) but I know the cadence was 5.29s including all overheads.
Everett Schlawin - Not the INT_TIMES extension? I put in a help desk ticket about offsets with the INT_TIMES and headers and think that was (or already) being addressed so that the FITS headers and INT_TIMES agree.
Aarran Shaw - It was indeed INT_TIMES, specifically the int_mid_BJD_TDB values from that extension.
