This page archives Slack comments from the splinter session 1/f noise correction of the Improving JWST Data Products Workshop (IJDPW).
Nov 15th
Marco Sirianni
Dear all - thanks for providing your input in preparation to the 1/f exercise. I will summarize all your input in a slide or two and post them here and also use it to present the exercise on Wed. The following are the few datasets selected for this exercise. I have put some basic comments. Most of the people indicated that their tool works on the rate file, others on the uncal. Please retrieve ahead of Wed (to save some time) the appropriate dataset level for your tool from MAST.
NIRCAM
Sparse field : jw01345001001_10201_00001_nrca3
Crowded field : jw01074001001_02101_00001_nrca1
Extended object : jw02107025001_02101_00001_nrcb2
NIRSpec
dark frame : jw01484177001_04201_00001_nrs1 and jw01484177001_04201_00001_nrs2
IFU: jw01335004001_03101_00002_nrs1. and jw01335004001_03101_00002_nrs2 (this is a IRS2 exposure)
MOS: jw01345061001_07101_00003_nrs1 and jw01345061001_07101_00003_nrs2 (this is a IRS2 exposure) -
NIRISS
crowded field : jw01501001001_26101_0001
Everett Schlawin
is there a script for this already set up? Sorry if I'm missing something obvious
Dan Coe
something like this should work:
import os
import astroquery
from astroquery.mast import Observations
mast_dir = 'mast:jwst/product' # Download from MAST
data_dir = 'data' # save downloaded data
os.makedirs(data_dir, exist_ok=True)
# JWST images to be analyzed
image_files = [
"jw01345001001_10201_00001_nrca3_rate.fits", # sparse
"jw01074001001_02101_00001_nrca1_rate.fits", # crowded
"jw02107025001_02101_00001_nrcb2_rate.fits", # extended
]
for image_file in image_files:
# Download file (if not already downloaded)
mast_path = os.path.join(mast_dir, image_file)
local_path = os.path.join(data_dir, image_file)
Observations.download_file(mast_path, local_path=local_path)
Everett Schlawin
Here's an update to Dan Coe's script (but it's having issues with two of the nrs files:
import os
import astroquery
from astroquery.mast import Observations
mast_dir = 'mast:jwst/product' # Download from MAST
data_dir = 'data' # save downloaded data
os.makedirs(data_dir, exist_ok=True)
# JWST images to be analyzed
image_files = [
"jw01345001001_10201_00001_nrca3_rate.fits", # sparse
"jw01074001001_02101_00001_nrca1_rate.fits", # crowded
"jw02107025001_02101_00001_nrcb2_rate.fits", # extended
"jw01484177001_04201_00001_nrs1_rate.fits", ## problem?
"jw01484177001_04201_00001_nrs2_rate.fits",## problem?
"jw01335004001_03101_00002_nrs1_rate.fits",
"jw01335004001_03101_00002_nrs2_rate.fits",
"jw01345061001_07101_00003_nrs1_rate.fits",
"jw01345061001_07101_00003_nrs2_rate.fits",
"jw01501001001_26101_00001_nis_rate.fits"
]
for image_file in image_files:
# Download file (if not already downloaded)
#for product in =['_rate.fits','_uncal.fits']:
for product in ['_rate.fits']:
get_file = image_file.replace('_rate.fits',product)
mast_path = os.path.join(mast_dir, get_file)
local_path = os.path.join(data_dir, get_file)
if os.path.exists(image_file) == False:
Observations.download_file(mast_path, local_path=local_path)
Dan Coe
Melanie Clarke pointed out the darks don’t have rate.fits, so we need the uncal’s:
“jw01484177001_04201_00001_nrs1_uncal.fits”, # dark
“jw01484177001_04201_00001_nrs2_uncal.fits”, # dark
Timothy Brandt
A topic to think about here: all of the 1/f noise that we can remove at the group stage will result in a better suppression (because it is a more accurate model of the noise itself--the groups are combined with different weights). So, would the approaches that people are using easily translate to the group stage? For example, if you are using the median of pixels flagged as good for 1/f suppression, then this median could be done at the group stage pretty straightforwardly. Additional benefits would be slightly better ramps and potentially significantly better jump detection.
Michael Regan
I agree. The method that Eddie and I use works on the groups.
Timothy Brandt
Great. After this step, is the noise white from group-to-group as far as you can measure? In other words, is it white with group value at a given pixel?
Michael Regan
Here’s a before and after correction (It’s IRS^2 which is cleaner than regular) (edited) 
Timothy Brandt
Looks good, thanks. I'm also interested in how well the noise matches the assumptions of the covariance matrix, i.e., that remaining read noise is uncorrelated from group-to-group. To the extent that this is satisfied we are on firmer statistical ground and can expect things like ramp fitting and outlier rejection to work better.
Michael Regan
Right, that’s what I want to do too.
Everett Schlawin
In my experience, the 1/f subtraction at the group stage has only marginally improved 1/f noise over the rate files, but can dramatically help with other effects such as improving the jump detection and amplifier offsets. Has anyone found a case where the group level 1/f subtraction was significantly better than at the rate or rateints level in terms of the striping along the fast read direction?
Timothy Brandt
The weights for the different groups depend on S/N, so the 1/f subtraction will introduce S/N-dependent artifacts if done post-ramp. I would expect a pre-ramp subtraction to be significantly better in this respect. As you say it should also help with jump detection, and it will improve the ramp fitting (which will lead to a small gain in S/N).
Everett Schlawin
I haven't done a comparison in a while (I pretty much always do 1/f at the group level and not usually after). But a comparison I did a while ago on ground-based darks, I found that the difference between slope and group was a standard deviation was 1 ppm different on top of a read noise in a time series of 200 ppm,. This did not take into account S/N-dependent artifacts because it was just darks and simulated images
Timothy Brandt
If noise is Gaussian and you're doing some sort of least-squares fit for 1/f then I believe that suppression of 1/f noise should be commutative with ramp fitting if the ramp weights are fixed.
If you are doing some sort of median fit for 1/f noise I think it is not quite commutative, but so close that it probably doesn't matter.
Everett Schlawin
Yes, I have usually been subtracting the median of a row and have found it to be close to commuting with the ramp fitting steps
Nestor Espinoza
What I’ve found is that doing group-level 1/f corrections is hampered typically from offsets introduced by previous steps (e.g., imperfect superbias, zodiacal background, etc.). Interestingly, we and others have managed to figure out ways around this, including the CDS technique I’ll share in the afternoon.
Michael Regan
The effect we see in the rate image is the average of all of the 1/f in the individual groups. So, the variation has more variation in the rate image while in the groups the values change much slower. This makes the correction more accurate.
Everett Schlawin
Yeah, you and other detector experts convinced me long ago to to apply 1/f subtractions at the group level because of this "muddling/mixing" effect in the rate/rateints files, but I have yet to see a case where the improvement was significant (other than ancillary benefits like jump detections, amplifier offsets). I haven't spent the time to do much testing on doing the 1/f corrections later because what I have works OK. Perhaps someone else has tested both, who can provide a good example!
Although I just remembered an experiment I did on GTO data using NIRSPec with my fanciest gaussian-process "GROEBAK" 1/f correct and did get better noise so at least for that observation the effect was noticeable (but not strictly an apples-to-apples comparison)
Michael Regan
To really get to the best solution we need to do stuff like measuring the amount of correlated noise. To versions can look similar but if one has significantly lower correlated noise both aperture photometry of points sources and extended sources will be better.
Michael Regan
I am sure it is NOT clear
Nov 16
Thomas Williams
I’m at another session so probably won’t be able to join, but for reference here’s a typical output from our code (this is NGC628, which I think might be one of your test cases). It’s a good example of how this works in the presence of extended emission:
Varun Bajaj
Your algorithm estimates the "true" background by comparing multiple dithers, correct? Did you happen to have implementation details of that somewhere? I'm definitely curious!
Thomas Williams
This operates just on a single tile, but we also do that too! The way we do it is make a stacked image-> compare each individual image to this average, median along rows per-amplifier and subtract.There’s also a second step with a median filter perpendicular to the stripe direction for any 1/f noise that might overlap in every tile and so doesn’t get averaged out, but this is generally a pretty minor effect (it’s worse at shorter wavelengths though)
Varun Bajaj
Hmmm, interesting- so for the stacked image, is that drizzled and then reprojected into the exposure (distorted) frame?
Thomas Williams
we just use astropy reproject but yes exactly. There’s some stuff we do in the full stacked image plane such as creating a source mask and doing the filter to avoid edge effects, and those are reprojected in the same way
Nathan Adams
Hi All
I will be late joining remotely due to teaching.I have run the NIRCam example files through my pipeline from uncal to end of stage 2 and background subtraction. The product immediately after 1/f on the cal.fits and the result after a 2D background is subtracted off of that is available here if people want to look at it: https://livemanchesterac-my.sharepoint.com/:f:/g/personal/nathan_adams_manchester_ac_uk/ErW4AViuNX9Bl1qn0qq3xFABymJk7e36gZZX-tGQOOM-UA?e=JjSoBc files with 'nobg' have background removal applied after the 1overf step (note this is agresive and designed for sparse fields). Files with 'amp' in the name are those with amplifier by amplifier subtraction applied, without it's just one row.We are presently using this code written by Chris Willott which was released pretty early on so i suspect others will have a better solution and I'm keen to hear how they work. https://github.com/chriswillott/jwst/blob/master/image1overf.py , it is very unintensive and takes a handful of seconds per cal frame.
Mic Bagley
I'm in a meeting at the moment but will come join soon!
Melany Clarke
NSClean notebooks:
MOS:
https://github.com/spacetelescope/jwst-caveat-examples/blob/main/NIRSPEC_MOS/nrs_mos_clean_1f_noise_workaround.ipynb
For the NIRSpec dark, I made a notebook to test NSClean for this workshop. It’s not available online, but I am attaching here.
nrs_dark_clean_1f_noise_workaround.ipynb
Marco Sirianni
Box link to retrieve the entire dataset:
Ryan Endsley
I'm dropping the 1/f subtracted NIRCam rate files using my script in this box folder:
For the jw01345 exposure, I also tested the 1/f subtraction at the group level. The performance with that approach is poorer than my current approach of applying it on the rate file. Happy to hear thoughts from others.
Everett Schlawin
I'm still doing 1/f subtractions and dealing with some bug with NIRSpec but will re-join later after going to the TSO session
The PFET noise can vary between amplifiers whereas the asic noise should be the same
Dan Coe
Here’s our draft JDox page you probably can’t access:
https://outerspace.stsci.edu/pages/viewpage.action?pageId=210306095
I’ll also export a PDF…and here’s the Github notebook:
https://github.com/dancoe/NIRCam/blob/main/NIRCam%201overf%20noise%20removal%20methods.ipynb (edited)
D_Coe_210306095_d3224165d90a415abc6fff3034451335-161123-1130-1294.pdf
Savannah Gramze
https://github.com/keflavich/brick-jwst-2221/blob/main/reduction/destreak.py
this is our version of Massimo's destreaker, with compute_zero_spacing_approximation to add back large scale structure - important for ISM. Works better in crowded fields. Images are a comparison to cal file and @Nathan Adams’s image1overf
Nathan Adams
to be clear it isn't my code but by Chris Willott (PI of CANUCS)
Eddie Bergeron
Here's the powerpoint presentation on the upcoming superstripe/substripe ASIC readout modes.
E_Bergeron_superstripe_substripe.pptx
Tim Brandt
Thank you !
Marco Sirianni
Many thanks for the very useful discussion ! Please use the momentum to provide any additional input and comments. One thing in particular we would like you to share with us is the suggested metric(s) we should adopt to evaluate the different solutions. I will briefly post in this chat a few bullets to capture the core of the discussion of this morning for the 4:30 briefing. Feel free to add/modify in the thread.
Everett Schlawin
One metric I'm wondering about is (when there are multiple integrations or dithers, which would require more data), the standard deviation of photometry on point or extended sources over time
Marco Sirianni
Here are a few points we could report from today session: please feel free to comment/add in the thread:
- A lot of good technical discussion and exchange of ideas on how to optimize existing tools using knowledge on the detector.
- There is the consensus that having in the pipeline one optimal solution that fits all cases won't be possible. However implementing in the pipeline a solution that is good (not optimal) for most cases and does not do harm should be possible, and it is a recommended way forward
- The Pipeline should also provide (one o more) optimized solutions in parallel to the default one.- Many of the current tools work on the rate image and they can be optimized to produce good results in specific astronomical scenes. However, "removing the scene" and working on the underlining 1/f noise at group level would be the optimal approach and would be beneficial for outlier detection and snowball correction.
- removing the scene has its own challenges - (relatively easy with NINT >> 1, possible with multiple dithers or mosaic tiles) implementing a solution in the pipeline that uses information of the scene from another exposure is potentially possible but not straightforward.
- There is still not a consensus on the metric to use to evaluate the effectiveness of a solution (but many are on the table, power spectrum, noise statistics, standard deviation of photometry on multiple integrations/dithers)
- An upcoming SUBSTRIPE Mode will allow to have top & bottom reference pixels for any subarray and interleaved reference rows between science rows and facilitate the removal of the 1/f noise. (edited)
Marshall Perrin
This is the approach taken in the spaceKLIP package for coronagraphy, with an implementation of 1/f noise removal at the group level that works in the above manner, developed by Jarron Leisenring at U of Arizona.
The code for that step is available here if it’s useful for others. (Apologies I was not able to join today, so am not sure if this was discussed in person or not!)
MIchele Perna
hello, here you can find the results of my test. I applied my 1/f single-column corrections with and without the masks required for NSClean
https://drive.google.com/drive/folders/1-2so2-IyRrvJocEnO3LSNd-sJ6A55t6W?usp=drive_link
Eddie Bergeron
Thanks Michele, I'm having a look now. What's your opinion on the three?
Nov 17th
Everett Schlawin
Inspired by other folks' excellent work and following Chris Willott's photutils background estimation, I got ROEBA to work while preserving the background and subtraction 1/f group by group. This resulted in many fewer artifacts than my first run on NGC 1566!
Michael Regan
Looks Good
Everett Schlawin
Here is as far as I got - just NIRCam and NIRISS
Michele Perna
hello again, I attach here a few slides presenting the results of the tests I did together with @eddie bergeron. Briefly: I compared my 1/f single-column corrections with the NSClean ones, for NIRSpec IFU data sets listed above (PID 1335, o002) and DARK observations (PID 1121, o002). NSClean seems to work better than the single-column correction. But I am not sure if the NSClean modelled 1/f looks reasonable or not; it may introduce a spatial bias in the count rate images (??). fits files here https://drive.google.com/drive/folders/1-2so2-IyRrvJocEnO3LSNd-sJ6A55t6W?usp=drive_link .
NIRSpecIFU_1of_tests_perna.pdf
Eddie Bergeron
Note the applied NSClean correction has a similar spatial bias (those broad dark/bright horizontal bands) in both the dark dataset (top) and a science dataset (bottom). In the cleaned dark image it looks like there is some kind of horizontal band artifact roughly where the dark bands are in the correction. Is that something that should be masked and isn't? Can't really see that in the science image, but the correction residual is the same in both. The correction bands almost look like "ringing" around that artifact. That feature looks a little bit like the irs2 correction artifacts across bad refcolumn samples (rows here are columns on the actual detectors). Those will be at different locations in nrs1, so Michele, if you get a chance can you repeat this same analysis on the nrs1?
