This page archives Slack comments from the focused discussion on persistence during the Improving JWST Data Products Workshop (IJDPW).
Harry Ferguson
Regarding persistence -- bright snowballs leave persistence that can show up in subsequent images and isn't caught by jump detection. The persistence signal looks a lot like a faint galaxy.
Yanu
@James Davies snowblind has an algorithm to flag those if you supply the list of subsequent images
Mario Gennaro
The Jupiter "offense" is something that the NIRCam team is trying to work on. We originally didn't conceive any specific mechanism for identifying "bad actors" and/or "sensistive observations" (like HST/WFC3 instead does). The rationale was that the e-folding time for NIRCam persistence was considered to be very short. Now, from preliminary analysis, our longer tau is of the order of ~1000 seconds (give it or take). That is not bad, but when you have HUGELY saturated observations (like the Jupiter ones in question, which, BTW used F150W2 and F322W2 to observe rings/moons, hence didn't care that the planet itself was going to overwhelm the pixel full-well) even 1000s are not enough.
We are working on scheduling mitigation strategies that would avoid that in the future. While the solution has not been fully identified yet, options include creating "buffers" after BAD ACTORS and/or prior to to SENSITIVE observations. This deals with persistence from without a given program. We are also working on a tool to predict self-persistence (ie from previous dithers/mosaic tiles from the same observation/program). This latter effort could lead to eg optimized dithers/PAs/expsoure parameters, but obviously if one has to observe a given patch of the sky becuase, you know, it's their target, there may not be a perfect solution to completely avoiding self-persistence.
Mario Gennaro
@Benjamin Johnson when you mention "improved persistence flagging" I am guessing you mean:
- have a model of the decay of persistence with time, possibly per pixel (calibrated eg on bright fields folllowed by draks - which is what we are doing in our NRC CAL program for persistence)
- measure fluxes in your exposures i=0,...m
- from each of the above predict the "persistence flux" for pixel (x,y)_j in exposure m+1
- set an acceptable threshold (eg 10% of the background or whatever is suitable to the specific science)
- FLAG those pixels (ie do not use them for that specific m+1 exposure)
I think that approach is viable. This was done in eg WFC3/IR with the persistence products (and it is something that our tool of my comment above may be able to achieve).
What I am less optimistic about is to use the above model to actually "recover" the pixel.
Benjamin Johnson
yes, thank you.Ya I agree that recovery is a big lift, treating decay of the persistence during the ramp fitting would likely be quite difficult, plus other difficulties I am probably missing.
Gabe Brammer
Another sneaky persistence effect I noticed was in the first exposures from the GLASS ERS program, e.g., the F090W visit jw01324001001_02201 with EXPSTART = 2022-06-28 22:04:43. Prior to that exposure was a long stare at the LMC with NIRSpec https://www.stsci.edu/cgi-bin/get-visit-status?id=1133&markupFormat=html&observatory=JWST., where the last exposure had EXPEND = 2022-06-28 21:17:16 . Would NIRCam have been continuously flushing during that time?
Mario Gennaro
unless there was a parallel, yes
10.7s full frame idle reset
but no DARK in place
Gabe Brammer
For what it's worth, that type of persistence will never be recorded in, e.g., trapsfilled files because there were never any reads downlinked.
Mario Gennaro
indeed. One could perhaps model the exact pointing, use a 2MASS image+catalog and predict persistence from observations prior to yours (like you did for WFC3...which what we are trying to do for NIRCam as well!)
but you did that on a generic rectified grid, so again, one can use that approach to have a order-of-magintude prediction in a simil-NIRCam detector, but doing that EXACTLY and pixel-by-pixel would be quite the challenge
Gabe Brammer
yes, that's what I was thinking. I guess you don't want to move any wheels that you don't have to, but is there a filter or something you could put in NIRCam while other instruments might be observing bad actor fields?
Mario Gennaro
still would need to move wheels
Gabe Brammer
I guess you'd rather deal with some persistence than have extra wheel moves
Mario Gennaro
thanks to the perfect launch we have to think >> 5years
Thimothy Brandt
Possibly ignorant question on persistence: is any of this done at the group stage? If not, is it straightforward to do so?
Mario Gennaro
when I have studied persistence in WFC3 I used the IMA. The plan for NIRCam is indeed to use a similar approach, ie use the group-by-group prime differences, which allows us the best possible time-sampling (given the data). The way I have always treated persistence when modeling it, is to use the integral of the "instantaneous flux" - which could eg be an exponential model- over the time under consideration (eg the sample time in the IMA or the group time in the JWST case)
basically the "up the ramp fit" does not work for non-linear stuff, so you can't use the rate images (or if you do, you have to properly integrate/average your model)
Thimothy Brandt
I was thinking about modeling and subtracting persistence (and 1/f noise) from the individual groups, then doing up-the-ramp on these cleaned ramps. You could remove snowballs in the same way: mask pixels out to a certain radius from the snowball and then model the snowball beyond this. I am largely ignorant of how snowballs are actually treated. Modeling the snowball really needs to be done at the group level though.
Mario Gennaro
yeah, your approach of subtracting persistence first and then fitting would be the way to go in order to recover the data (because indeed the non-linear effects of persistence are skewing/screwing the up the ramp fit, so one can't simply "subtract" persistence from the rate files) but I don't know how well we can model persistence at the pixel level that you'd actually be recovering the flux correctly. For now my one goal for NIRCam would be to at least be able to flag pixels that are affected by a given amount of persistence, above some user selectable threshold, by knowing something about their recent illumination history. Admittedly "throwing away" those pixels sounds like a waste so hopefully one can work the model to a level of trust that we can actually use it the way you outline. My holy grail for persistence would be to use some machine learning approach and look at the full history of each pixel and be able to model the total number of traps, their capture/release timescales, the level of charge at which they "activate" etc., and at that point one would know everything about the pixels. Even then though, one needs to know the full history of illumination of a given pixel in order to correctly model its persistence. And that full history may not simply be know (think of eg what happens when NIRCam is not being used. It does receive light, although it does idle reset, but then we don't have a way to know exactly how it was illuminated because those "data" are flushed down the space-toilet)
Yanu
From my experience with F090W data, persistence seems to stay longer on some detectors (a3, b3, b4)
Mario Gennaro
That is our assesment from commissioning as well, in particualr a3 might have a tau of 2000-3000 seconds (preliminary). We haven't fully analysed the persistence CAL programs yet, but we surely hope to do so in the coming months and have a better answer
Zhen-Kai Gao
I found some rings (in the upper center of the first image) in some NIRCam F277W images (e.g., jw01837004020_02201_00001_nrcblong and jw01837004020_02201_00002_nrcblong) and also scratches (on the lower right of both images) in many F277W images (jw01837004001-jw01837004024, module B only). I am not sure if these are persistence. Does anyone know if there are any known algorithms that can be used to identify them and mask them?
(the first image should be "jw01837004020_02201_00001_nrcblong" (module B not A). sorry for the typo.)
Yanu
I'm not sure if it's persistence but I've had some artifacts on F090W images which stay for a few exposures and gradually fade out, the easiest solution is to stack images with the feature and then flag affected pixels, which turned out to be a bit difficult because in my experience, it's hard to flag them automatically: if you try to apply some sort of threshold, then you end up also flagging some random pixels on the stacked image as well, so I made a tool to manually select which features on the stack to flag, then I just put "do not use" flag in dq array before stage 3. Depending on how many exposures are affected by the feature and how many dithers you have it may be easier or more difficult to do this. Bright stars are also difficult because they appear on the stacked images too (I kinda mitigated this with sigma clipping but it's still not always super optimal).
Mario Gennaro
do those artifacts fade with time? The "ring" thing may be a "gingko leaf" artifact, which should not fade: https://jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam-instrument-features-and-caveats/nircam-ginkgo-leaf . Those scratch-like things instead I cannot easily think of what they are. If they are persistence, you'd see them fade with time (eg if you sort the exposures or even INT-by-INT or group-by-group. @Martha Boyer @Bryan Hilbert, any ideas?
Martha Boyer
Yeah, I think the ring is a ginko leaf, or other scattered light. Not surprising with so many bright stars around. The scratches are weird though — they’re showing up in about the same spot on BOTH modules?? Unless maybe the labels on the images are wrong?
Bryan Hilbert
That's very weird. Being in both modules doesn't fit with persistence. Unless it's persistence from earlier scattered light? I'll try pulling these up in JWQL and see if there are any hints.
Zhen-Kai Gao
@Martha Boyer @Bryan Hilbert Sorry I made a mistake. The first (left) image should be "jw01837004020_02201_00001_nrcblong" (module B not A). (edited)
@Yanu The tool you made for flagging artifacts sounds super useful. Do you plan to put this tool on github?
@Mario Gennaro @Martha Boyer These artifacts do not fade with time. Thank you for introducing the "gingko leaf" artifact. I will try to see if "gingko leaf" tends to appear in regions with many bright stars around. If so, a possible solution to identify "gingko leaf" might be to first select images within regions with higher density of stars, and then we can flag them manually without having to visually inspect a few thousand of images.
Bryan Hilbert
For the image on the left, I don't see any NIRCam observations in the ~3 hours prior. I do see the same streak, in jw01827004020_02201_00002_nrcblong, which was taken immediately after the image on the left. It's hard to say using JWQL if the feature is fading at all with time, especially if it is persistence and we're in the tail of the exponential decay and the images were taken back to back. That would be my bet though. I'll check the other image as well. With the artifact in the same location and orientation in two images with different pointings.....maybe this is persistence from scattered light?
The situation is similar for the image on the right. I don't see any NIRCam observations in the preceding few hours. The scratch is present in all of the 1837 Obs 4 images taken at that time.
Eddie Bergeron
Bryan, can you look backwards in time until you find the actual previous obs?
Zhen-Kai Gao
The same streaks can also be seen in "jw0172704*nrcblong", but some of them are fainter (or spread out?).
Bryan Hilbert
jw01483275001_06201_00001 is a NIRCam dark (A mod only unfortunately) that was ~7 hours before the image on the right. To get to the previous observation that used B5, you have to go back 19.5 hours to jw02073003008_03101_00003_nrcblong, where I don't see any hint of the scratch.
Which observations in 1727? What's the next number after jw0172704?
Zhen-Kai Gao
043, 044, 045, 046, 047, 048
Marshall Perrin
The ring-like artifact I have seen before, and appears to be scattered light from the bright saturated star in the FOV. F277W appears particularly problematic for this so I think we can infer confidently that it’s an optical ghost arising from the filter. Not persistence.In at least some datasets I have seen this move with dithers, in directions opposite to the dithers motions of the bright saturated star. Dithering and masking seems an effective strategy to remove it.
Eddie Bergeron
I downloaded the two uncal files jw01837_004 and _020 nrcblong. Attached animated gif of the two CDSs. The "scratch" is moving slightly between the two obs, so it's not fixed to the detector (not persistence). Looks like its an internal reflection of some kind.
Yanu
I will put the tool on github at some point, but it seems that it won't be that useful for you if the artifact is moving
Zhen-Kai Gao
Are these artifacts also "gingko leaf"?
