This page archives Slack comments from Day 1, Session 2 of the Improving JWST Data Products Workshop (IJDPW).
Marco Sirianni - We are starting section 2.
Jeff Valenti - @Thomas Williams - Can you put some numbers on the alignment offsets you see in standard pipeline products?
Thomas Williams - For NIRCam typically quite small (maybe 10ths of an arcsec?), for MIRI it can be ~arcsec or more.
Tony Sohn - FYI, 10th of an arcsecond level offsets are consistent with what the instrument teams find caused by uncertainties in SIAF alignments, especially when using FGS2 as the guider. We'll be making improvements to these in the coming months.>1 arcsec level offsets are typically caused by guiding on different guide star than commanded. This can happen if e.g., there is a similarly bright star near the guide star.
Howard Bushouse - @Thomas Williams The skymatch step should not need any true background signal. It compares the total signal (background+sources) level in the overlap regions of images and tries to minimize all the offsets. Residuals might indicate slight errors in overall flux calibration between detectors (or something like that). So it's not finding or subtracting true background, but only offsets between images (the background will likely still be in there, unless you explicitly ask the step to subtract it).
Thomas Williams - I agree! I’m not sure why it doesn’t work, but we tried flipping every switch and never converged to something that just worked, sometimes we could get improved results on one filter/galaxy combo but made others worse. Our implementation seems to work generally for everything (but is slower)
Varun Bajaj - I believe its because of how it calculates the average surface brightness in the regions- with lost of extended emission sigma clipped statistics on the image itself (Rather than a difference of two regions) gives wonky numbers. Or at least that's my working theory
Macarena Garcia Marin - @Thomas Williams could you give us a sense of the computing power you need to reduce these datasets?
Thomas Williams - It’s a few100GB of RAM, and about 4 days for the full Cycle 1 dataset
Macarena Garcia Marin - Thanks!
Zhiyuan Ma - @Thomas Williams On the sky matching, I believe the default Spitzer data pipeline (MOPEX) does the overlapping-based sky match, and my personal experience is that if the single image have sky gradient, that gradient tends to "propagate" and cause the final mosaic to have gradients. For example, if there are three frames in a row and overlapping in part between 1 & 2 and 2 & 3, but the sky is not uniform in 2, then the resultant image will have mismatched overall sky. The solution that I ended up using was. to just remove the sky for all frames and do the mosaics with sky-free images. Do you see the same thing with your data?
Thomas Williams - We haven’t but we just fit a single offset, could well be but is being wiped out by our data being bright and hiding it. It’s low level if it is.
James Davies - @Thomas Williams can you tell us about doing 1/f noise removal on these images where you have no areas where there’s no source emission?
Thomas Williams - Trade secret *wink*. We do a Butterworth filter to remove large-scale structure beforehand. Does well to remove the small-scale stuff, but leaves big ripples in, which we try and correct for by comparing overlapping dithers after correcting local astrometry. It’s not perfect but seems to do well, and importantly doesn’t oversubtract by bright sources.
Macarena Garcia Marin - @Thomas Williams do you use any multi-process option?
Thomas Williams - Everywhere we can typically stuff runs on each tile in multiprocessing, so everything except lv3 runs in MP.
Jeff Valenti - @Thomas Williams - Given what we heard this morning, could some or all of your custom steps be incorporated into the STScI pipeline framework?
Thomas Williams - Probably, but I haven’t looked into it. There are quite a lot of options that I’m not sure are well handled with the pre/post stuff @James Davies talked about, since we hook in different parameters for different filters sometimes.
Gabe Brammer - Comment that in general it seems like the MIRI flat is the cause a lot of the issues with the MIRI backgrounds and gradients. The image attached below compares the rate and cal files for jw01180007001_02201_00009_mirimage_rate.fits, where the latter was processed in MAST with R_FLAT = 'crds://jwst_miri_flat_0619.fits' / Flat reference file name. My general assumption is that the background should be flat after flat-fielding? I think we need to be careful what we call a multiplicative contribution from the flat and an additive background (sky + dark).
David Law - Anecdotally, the MIRI MRS flat seems to be causing a number of issues that I've seen in the IFU data too, despite the flat doing a good job of correcting the sources used to derive the flat. I'm starting to consider that the effective flatfield may be dependent on the exposure setup, and if this is the case for the MRS it may also be relevant for the MIRI imager.
Zhiyuan Ma - Not sure if this is of any connection. Just as a matter of fact, we've seen in the NIRCam data that in the LW bands (F356W and F444W), we found some gradient (much less in the latest pmap though) that are multiplicative (i.e., in the flatfielding).
Angelo Adamo - @Thomas Williams did you have a solution for the 1/f noise? The target fills up the entire FoV.
Thomas Williams - From my slides (I didn’t have time for this):
Angelo Adamo - Thanks!
Thomas Williams - First step is butterworth filter to remove large-scale structure and get stripes there (after the filter it’s essentially the CEERs remstriping, which @Mic Bagley talked about). Second step is combine dithers and then subtract off the big stuff. Works better with more dithers!
Angelo Adamo - Thomas can you tell exactly what filtering functions you use to perform large-scale filtering? I see you show ngc628 in your example here. We have emission throughout the frame.
Thomas Williams - A classic galaxy. It’s a Butterworth filter, but the brightest stuff is replaced before to avoid Fourier “ringing” and there’s some masking going on afterwards as well. it works well even in the F200W where some of the frames are just pure emission.
Angelo Adamo - Would you be willing to share your approach to remove the 1/f? Our main problem are the narrow bands. F187N in particular.
Thomas Williams - Yep, we’re seeing that for the Cycle 2 stuff where we have narrowband. It’s rough. We can definitely add you to the repo so you can have a look at the code, if you let me know your github username?
Angelo Adamo - angelaadamo. Can you please add also @Varun Bajaj
Varun Bajaj - My username is Vb2341 on github!
Thomas Williams - I’ve added you, you want to look in the single_tile_destripe and multi_tile_destripe steps we have there
Angelo Adamo - Got it. Thanks a million!
Angelo Adamo - @Varun Bajaj has a poster about MIRI mosaic assembly and he’s super expert on the astrometry. He has done a wonderful job with the FEAST data..please let us know if we can help in some way
Thomas Williams - It’s on my list of things to have a look at for sure, be interesting to see if we’re converging in the same directions.
James Davies - @Benjamin Johnson When you mask snowballs and end up areas that have offsets within the masked area, is it possible that that could be showing problems with the linearity correction?
Jeff Valenti - @Benjamin Johnson - Please provide an example exposure that exhibits the issue with erroneous snowball flagging in jwst 1.11
Michael Regan - @Benjamin Johnson Can you point me to the 1/f -> snowball example? James is correct. Any jump can expose curvature in the ramp.
Howard Bushouse - Speaking of persistence, I'm curious whether anyone has tried to make use of the "trapsfilled" product from the Detector1 persistence step to make corrections to subsequent exposures (you have to use the trapsfilled product from exposure 1 to correct exposure 2, etc.)?
Bryan Hilbert - This won't work for NIRCam at the moment because our other persistence reffiles (I forget the name at the moment), are all zeros.
Michael Regan - I think that’s true for all instruments.
James Davies - Yes, it’s all zeros, so not useful.
Kevin Volk - For NIRISS the persistence reference files are not very good because we had limited CV3 data, and we see differences on-sky compared to ground testing in the persistence pattern (I do not know why).
Gabe Brammer - Is there a plan for the trapsfilled images to eventually know about data from other programs that might precede a particular exposure, like the HST WFC3 persist products?
James Davies - What would be very useful would be to have the saturated pixels put back into the rate file DQ array. Currently they are not propagated. Saturated pixels are the most likely to persist between exposures, or at least are the low-hanging fruit.
Howard Bushouse - That would best be implemented via a new DQ flag other than "SATURATED", because we don't want folks to think that a given pixel is still saturated in a rate or cal image, if it's been corrected properly in Detector1. So perhaps another flag, something like "persistence_likely"?
Howard Bushouse - @Gabe Brammer No plans for that at the current time. Perhaps eventually?
James Davies - But we do propagate “JUMP_DET” already to the rate files, and there is not confusion. Train people to use DO_NOT_USE only perhaps?
Bryan Hilbert - You can get persistence from pixels that do not saturate, although the persistence in that case is less. But a name like "persistence_likely' for pixels that reached saturation might be confusing in light of that. I could see some use in distinct bits for SATURATED and e.g. REACHED_SATURATION. Or a redefinition of SATURATED pre/post-detector1, along with some prominent documentation about it.
James Davies - @Benjamin Johnson I’ve noticed the jump step can flag what is essentially 1/f noise when using the default 4 sigma detection threshold. I generally run it at 5-6 sigma, and it solves the problem. Would be interested in your experience.
Martha Boyer - Same. I usually run with 6-8 sigma for the absolute flux calibration data. (those data can be particularly troublesome b/c some of the subarrays don’t have reference pixels
Benjamin Johnson - Thanks, glad that it's a likely a simple fix.
Michael Regan - It should not do that. 1/f is around 2 sigma peak-to-peak. There are some problems with the read noise reference files that underestimate the read noise which might be the source of the problem.
Timothy Brandt - 1/f noise is not removed at all prior to ramp fitting, right? I have found that the read noise in the data files is much closer to the actual read noise if the reference pixels are used for 1/f noise removal at the groups stage.
Michael Regan - The read noise should include the 1/f. WIthout it both the jump step and ramp fit will not work correctly.
Timothy Brandt - Hmm, interesting. I found the read noise values in the reference files to be significantly high if I don't take out some of the 1/f noise.
Bryan Hilbert - Some of the 1/f noise is removed as part of the reference pixel correction step, early in the detector1 pipeline, prior to ramp fitting. But the limited number of reference pixels means that only some fraction of the 1/f is removed.
Timothy Brandt - Ok, that is what I meant. So that is done prior to ramp fitting. Thanks.
Loic Albert - @Benjamin Johnson Have you tried using galaxies as your astrometric anchor?
Benjamin Johnson - Yes, our reference catalog is composed mostly of galaxies. At the moment we use Gaia only for final checks.
Greg Sloan - nJy is a unit that makes me very happy
James Davies - @Benjamin Johnson for missing flight distortion maps, are there any available for the narrow band filters? We’ve seen multi-pixel offsets between, say F444W and F470N.
Bryan Hilbert - Coming soon. Armin Rest can speak to the schedule for those.
Benjamin Johnson - I haven't looked at the narrow bands. There are a couple of medium bands that are missing for the A module.
Bryan Hilbert - There should also be a Global Alignment update coming soonish. That will correct offsets between the A and B modules, which can lead to very visible alignment failures in some cases when running with default parameter values/workflow.
Benjamin Johnson - Excellent, I should have been mroe precise in my talk, currently we compute two tweaks per visit, one for module A and one for module B
Harry Ferguson - @Benjamin Johnson: @Mic Bagley mentioned un-masked outliers as a remaining problem. Do you any remaining worries about outliers?
Benjamin Johnson - We see a few outliers, which we suspect will be improved by cross-exposure snowball persistence flagging. But the rate is low and we are pretty happy there, though we have 6 or 9 dithers in most cases.
Jane Rigby - Taylor Hutchison has an almost-ready-to-submit paper describing a method to remove the outliers that the pipeline doesn’t catch. Taylor.a.Hutchison@nasa.gov
Benjamin Johnson - To give a little more info, we have definitely seen outliers make it through the stage3 median flagging because they have very large errors attached, so they do not reach the uncertainty normalized threshold, yet they still have enough weight in the stack to generate a 'hotspot'. These are usually single band. We have not worked out yet the relation of these to the persistence of snowball cores across exposures.
Marcio Melendez - @Benjamin Johnson have you try to use WebbPSF to subtract the persistence from the sensing program (weak lens image, F212N +8WL)? This could be a good alternative instead of masking that region (if that’s your current procedure).
Benjamin Johnson - No, we haven't tried that. Is the persistence decay expected to be stable across an area of the detector that size?
Marcio Melendez - If from the sensing program it should always be in A3 (FP1 position), as for the magnitude it will be changing as time goes but it is a single parameter to adjust. Not sure if this answer your question.
Benjamin Johnson - Yes, it's from wavefront sensing and always in A3. My question is whether every pixel in that area might decay at a slightly different rate, making it a more than 1 parameter model. Still should be easy to fit out in source free exposures, thanks for the suggestion of the model to use.
Marcio Melendez - In addition, images from the sensing program are publicly available so you could use those as well to do your subtraction.
Jeff Valenti - Wild idea posed as a general question - would astrometric alignment of individual exposures be substantially improved by using Gaia and perhaps other catalogs to improve astrometry of the FGS ID image and then relying on precise knowledge of the JWST focal plane?
Tony Sohn - The FGS ID image is not directly used to solve the astrometry of SI images. It's just the location of where in the FGS detectors guide stars were measured that contributes to the WCS. Everything after that is affected by how well we know the SI's location and orientation *relative to* the FGS1 on the focal plane.
Tony Sohn - The current alignment issues (or absolute astrometry) users are seeing are mostly due to some uncertainties in SI's focal plane locations+orientations. These are being worked on.
David Law - Agree to Tony's first point. On the second point, the two limiting factors that I've seen are the significant failure cases in guide star coordinates (sometimes 1'' or more off) and in the SC roll uncertainty (especially for MIRI far away from the FGS).
Jeff Valenti - @Ryan Endsley - Have you tried your full processing pipeline on a crowded star field? Even one example would be interesting? Can somebody suggest a good test observation?
Martha Boyer - ERS data for M92: PID 1334. Or LMC data from cal program 1476 (more filters).
Ryan Endsley - @Jeff Valenti, I'll give that a try later today and get back to you on this thread.
Ryan Endsley - Hi @Jeff Valenti @Martha Boyer, I tested my processing pipeline on the F115W LMC data up through the 2D background subtraction. I assume that was the main reason why you requested a reduction of a crowded field? I've uploaded validation plots for each of the data files which you can find here: Box link. Overall, I think the code works well in the majority of cases, though there are some instances where the script seems to be over-correcting the background in the corners of the detector. The most extreme example is attached here, but I emphasize this seems to be an exceptional case.
Ryan Endsley - If you'd like me to test the reduction all the way to the final mosaic, please let me know.
Kevin Volk - A general comment, I think all the "1/f" noise corrections that have been discussed also do a correction for the other direction on the array. This is not actually 1/f noise, do we have any understanding of where it comes from? (There was previous mention of something coming in from the reference pixel correction step, but I was not clear on what it was.)
Timothy Brandt - You mean the different reference voltage for each readout channel?
Michael Regan - I think he’s talking about a per column correction. I don’t understand how this could happen.
Timothy Brandt - It does happen that some columns are strange. It's uncommon but I do see it.
Kevin Volk - There is a residual channel to channel offset as you say, but what is discussed looks like a variable change across the channel.
Timothy Brandt - I don't know how this could happen in hardware. Pixels are read out sequentially, left to right or right to left, then stepping up or down the detector as each row is read out. I'm not sure how to get a weird column though you're right, I have seen it.
Nathan Adams - @Ryan Endsley the 2d background on an amplifier by amplifier basis is a neat idea. Something that should be straight forward to implement on any other 2d background script so i may give it a go. thanks for the idea. I do think I've seen similar dark spots around bright stars in other surveys, not just that one CEERS image.
Ryan Endsley - Thanks @Nathan Adams! Glad the idea sounded useful. Happy to chat about it moving forward, including if you find it fails in some instances.
Mic Bagley - Agreed! That's a very cool idea and it's cool to see how well it works. Also I like your two-step wisp removal @Ryan Endsley. I've really struggled with that extra light on the left edge of B4 (I think, or maybe A4, I can't remember) and your 2nd subtraction catches it nicely!
Michael Regan - Can you explain what happens with the bad pixel masking
Jeff Valenti - @Ryan Endsley - Please document here what WCS information you would like output to facilitate analysis.
Bryan Hilbert - @Ryan Endsley there will be a global alignment update for NIRCam in the nearish future that should improve the alignment results when working with module A and B data together. I've seen cases where the current relative positions of the detectors leads to alignment offsets of ~1-3 pixels. The workaround for this is to align one module at a time.
Mic Bagley - @Ryan Endsley we've also found much better results fitting the modules separately. We've also fully separated the TweakReg step from stage 3 and run it iteratively and sometimes even on individual SW detectors, too. So @Bryan Hilbert it's good to know about the global update, we'll be sure to keep an eye out!
Bryan Hilbert - Unlike tweakreg, JHAT does not presume to know the relative locations of the detectors. So in many cases, you should be able to use a single call to JHAT using all of your data and get a good alignment.
Ryan Endsley - Thanks for the feedback, @Bryan Hilbert and @Mic Bagley! I'm realizing that I forgot to ask in my talk if there's a way to run alignment on the cal files before stage 3. The way stage 3 alignment seems to work right now is by building a source catalog as it relatively aligns each exposure to the previous one. But that creates challenges for running a full mosaic of non-overlapping exposures through stage 3 in my experience.
Bryan Hilbert - There is also a recent update to the piipeline (I'm not sure if it's live yet), where you can add a field to your asn file that will cause tweakreg to split up your data and not rely on it's knowledge of relative detector locations. I haven't tried this out myself though.
Ryan Endsley - Ah perfect, I'll keep an eye out for that update!
Bryan Hilbert - I think you can run alignment anytime after the assign_wcs step, which is early in the image2 pipeline. But in that case your data wouldn't be flat fielded. I imagine that could create other issues.
Ryan Endsley - The issue is that I don't know how to run tweakreg such that it modifies the gwcs data in the ASDF extension. I can easily change the WCS info in the SCI extension, but the stage 3 step doesn't seem to use that information at all.
Bryan Hilbert - That's right. The pipeline steps only pay attention to the WCS data in the ASDF extension. Tweakreg should be updating the ASDF extension as well. It's possible to manually update the gwcs in the ASDF extension. I'd need to look up the best way to do that though.
Ryan Endsley - Hmmm ok, let me post something about this in another channel that's more targeted towards community questions. I can share my code to see if someone can identify where it fails.
Bryan Hilbert - Looks lik pipeline version 1.12.5 contains the update for the grouping of files in the association file that I mentioned earlier.
Ryan Endsley - Thank you for checking. I will test it out in the next couple days.
Mic Bagley - We pulled the portions of tweakreg that update the wcs out and sometimes use that to apply wcs to images without needing to run the step, if something like that would be helpful (not sure if that's exactly your question)
Bryan Hilbert - It sounds like the adjust_wcs() tweakreg method that just mentioned in Savannah's poster may be able to do that? I've never tried it myself.
Mic Bagley - I couldn't get that to work, but that was probably user error
Howard Bushouse - @Ryan Endsley (he/him) Totally agree with the desire for some way to save/store tweakreg failures for individual images when you're processing a large number of inputs (based on my own experience). Scanning through thousands of lines of processing logs is not fun or efficient.
Tyler Pauly - FWIW, the default log config options allow you to divert certain log levels to a file, i.e. store all WARNINGs in a given pipeline_log_warnings.txt : logging_configuration
Howard Bushouse - @Ryan Endsley (he/him) Regarding the saving and/or use of a reference WCS in tweakreg, several options have been added in the latest version (jwst 1.12.x) to allow for that, e.g. using a predefined WCS object as an input reference, against which all images are aligned. Check the latest docs for info: tweakreg/utils.
Jeff Valenti - @Armin Rest - for reference: jhat.readthedocs.io
James Davies - @Armin Rest, these alignment diagnostic plots are super useful. It would be great if the default official pipeline could output diagnostic plots like this, and not just for alignment.
Howard Bushouse - Feel free to submit a Pull Request ...
David Law - I remember discussing the utility of diagnostic plots and quality metrics in general before; one possibility being not just plots but QA bitmasks in the header alerting of potential issues detected during processing.
Jo Taylor - @Armin Rest JHAT seems to have a lot of overlap with drizzlepac functionality. As someone who is not an expert in either, can you explain different/new features in JHAT?
Howard Bushouse - On a related note, the tweakreg step in the jwst pipeline is really just a wrapper around functionality in the "tweakwcs" package, also used in "drizzlepac". The original vision was that twakreg would be used to perform "good enough" alignment in the default operational pipeline, and then folks could use the far more extensive capabilities in "tweakwcs" to do more hardcore off-line WCS manipulation for specific observations.
Anil Seth - @Armin Rest My understanding is the absolute point accuracy of JWST is ~0.1" (1-sigma). Two questions related to this:
- What is the source of the large WCS offsets (10 pixels!) that you are finding? Is it unknown positions of the instruments on the focal plane?
- In a single visit, between dithers, how accurate are you finding the dithers to be (i.e. stddev of |request_offset-observed_offset|)? I’m wondering if for the IFUs we need to worry about correcting astrometry between dithers.
Thanks!!!
David Law - Specifically on the IFU case: relative offsets between dithers are very good.
Tony Sohn - Hi Anil. As for (1), the >1 arcsec offsets are usually due to guiding issues. For example, there's a good chance that in crowded fields, a nearby star is used for guiding instead of the commanded guide star. Telescope think it's using the commanded guide star, so the WCS is populated using that info.
Benjamin Johnson - For NIRCAM data the dithers also appear very good, a few mas rms.
David Law - Empirically for MIRI IFU I've found a single offset is sufficient for all exposures associated with a given guide star acquisition.
Tony Sohn - The uncertainties in SI positions on the focal plane is known to be on the order of ~<0.1 arcsec. This means, if the correct guide star is used, then the offset is ~<0.1 arcsec (up to 0.3 arcsec for MIRI as David pointed out below). That's where the pointing accuracy comes from.
Paul Goudfrooij - Did @Armin Rest say that he used a HST catalog as reference for the MIRI case? (HST absolute pointing can be off by an arcsec).
Sarah Kendrew - @Paul Goudfrooij He did.
Tony Sohn - I suppose the HST catalog has been first aligned with absolute astrometry (e.g., Gaia) in this case?
Sarah Kendrew - HST v3 is aligned to Gaia iirc.
David Law - Also worth noting that the absolute pointing accuracy is lower for MIRI because it's further away from the FGS, generally I've seen about 0.3''.
Paul Goudfrooij - @Tony Sohn I wasn’t clear about that.
Paul Goudfrooij - @Anil Seth dither accuracy is typically/nominally of order 5 mas.
Anil Seth - Thanks everyone!
James Davies - @Armin Rest does the chunky PSF of JWST cause matching problems?
Armin Rest - Not really, it even works well with coronography masks, which definitely makes the psf funky.
Mike Engesser - I dont think Armin mentioned it, but for everyones reference, we have found that after running JHAT, alignment for NIRCam can usually be as good as <0.1 pixel.
Sarah Kendrew - @Armin Rest does JHAT accept final mosaics (i2d files) as input? given that the relative calibration is very good generally, it would be great to be able to pass the larger field for final alignment of fields where tweakreg failed to do the absolute alignment on individual exposures.
Mike Engesser - Yes, it can align level 3 data! I’ve had it work for mosaics as large as 80Gb.
Howard Bushouse - I believe it's the case that the absolute alignment phase of tweakreg takes place after all the individual images have been aligned relative to one another and then treats them as a whole for the absolute alignment, so in theory it should take advantage of the whole FoV provided by a mosaic (in theory!).
Mike Engesser - We often align level 3 nircam data to HST WFC3 data, for example.
Sarah Kendrew - ah - I didn’t know that Howard, thanks.
Howard Bushouse - It seems to be the case that absolute alignment is just difficult overall for MIRI imaging, due to the dramatic drop in detected point sources as a function of wavelength.
Mario Gennaro - I created a JHAT channel. Maybe I'll rename it "alignment".
Sarah Kendrew - There is already a topic-alignment channel btw.
Jeff Valenti - @Savannah Gramze - Could you provide an example of an observation that was particularly challenging to align? It's useful to have such cases to test automated procedures.
Savannah Gramze - Sure, project id 2221, observations of cloudc were a struggle to align. When investigating the problem, I found that the offsets between visits were as much as 6". My advisor, working on brick data from project id 1182, found a 20" offset.
Varun Bajaj - This is definitely a limit of a tweakreg style algorithm, as the further you have to search, the chances of a false positive alignment goes up ~quadratically.
Armin Rest - @Savannah Gramze I might be able to help you, maybe during hack day?
Michael Regan - @Varun Bajaj Why wouldn’t this just replace the current pipeline steps?
Varun Bajaj - Were you referring to the MIRI background processing? If so, it does replace the current pipeline step as I've implemented it.
Jeff Valenti - @Varun Bajaj - Do you advocate incorporating OnePassPhot into the standard pipeline as an option? As the default? Mode dependent.
Steph LaMassa - NIRISS has a request to update DAOStarFind with IRAFStarFind for tweakreg due to the undersampling issue. This ticket should be part of the discussion for prioritization for the next pipeline build.
Varun Bajaj - I would strongly recommend using PSF fitting to get the source positions instead of the default algorithm. Plus, once you've aligned/combined everything, you have a pretty decent stellar photometry (and astrometry) catalog!
Paul Goudfrooij - @Varun Bajaj how fast or slow is doing WebbPSF fitting to source positions relative to IRAFStarFinder for example? Is it fast enough to run in a standard pipeline?
Varun Bajaj - I answered Paul in person, but for posterity: for fitting ~1000 sources, its on the order of ~10 seconds on my laptop, so in terms of running in the pipeline its is far from the slowest step. Its not as fast as the default finder algorithms, but its going an order of magnitude more precise so...
Massimo Stiavelli - Somebody should give @Meredith Durbin extra telescope time asap!
Paul Goudfrooij - By the way, there is already a pipeline ticket out there to (significantly) improve the source finding algorithm used in tweakreg to avoid the “bad” centroiding by DAOStarFinder for undersampled images.
Everett Schlawin - @Meredith Durbin - very glad to see these lightcurves! Were the Stage 3 catalogs of sufficient quality? Looks like you may have done your own photometric extraction.
Meredith Durbin - Thank you! I did do my own photometry on the stage 2 products with Dolphot but I’d definitely be interested in testing out the stage 3 extraction!
James Davies - @Varun Bajaj could you post links to the 2 repos (your QR codes from your poster) here when available?
Varun Bajaj - @James Davies yep! For the MIRI/NIRCam background matching : jwst-mosaic-skymatch (the docs are still a WIP, but the notebook is a demo of the usageFor the One Pass PSF Fitting there are some docs and a notebook now too! One-Pass-Fitting
Thomas Williams - I should have said this earlier, but the background matching stuff is super cool to see! And I’m glad we seem to be doing ~the same thing, makes me think we’re not far off optimum. We do a two-pass thing by matching dithers first, then stacking individual dithers to get an offset between mosaic tiles, do you do something similar? We found we had to to maximise S/N and overlap between the tiles, since by default they only overlap 10%. Especially an issue for the NIRCam shorts
Varun Bajaj - I have both methods implemented if im understanding you correctly
Thomas Williams - Very neat. And do you have an idea for how much better your PSF photometry is over the standard pipeline source extraction or something like IRAFStarFinder in the longer MIRI wavelengths? They basically don’t detect anything for us, so it’d be cool if this was a way to actually find some sources. Inheriting solutions from the shorter MIRI bands gets us to a fraction of a pixel but it doesn’t seem like an ideal long-term solution (edited)
Varun Bajaj - The default behavior groups exposures by their visit, and computes the intravisit offsets and drizzles them together, then it takes those drizzled image "tiles" and computes offsets between the tiles, and adds those offsets etc between the tiles back to the intravisit values
Thomas Williams - yep, that’s exactly what we do.
Varun Bajaj - Awesome yeah! I could never get montage to work, so I just ended up hacking it together from pipeline steps. It was pretty simple once I got that figured out
Thomas Williams - (well, we just use reproject rather than drizzle, but I suspect the difference there is minimal for much faster computation)
Varun Bajaj - I haven't tested the PSF photometry for wavelengths as red as yours, but in principle, the algorithm is so simple (for detection) that I think it should work. just fine, as long as things are local maxima, it will at least try to fit them
Thomas Williams - It’s tricky I think because the sources are embedded in the ISM structure so you can see them by eye but they’re not much above a local maxima. And in some cases there’s basically nothing (especially for lower metallicity galaxies)
Varun Bajaj - My finding algorithm just looks for a peak (regardless of background). It just makes sure its at least X bright, and X separated from any other local max
Thomas Williams - If I ever get some free time I’ll definitely give it a go, it’d be nice to have something that is more repeatable than “look for things by eye”
Varun Bajaj - I'd be happy to at least try to see what happens!
Thomas Williams - It runs on the CRF files, right? Can definitely send them over if you want to have a go on ngc628?
Varun Bajaj - I believe it should yes! The CRFs are just outlier rejected cal files, more or less, right?
Larry Bradley - @Thomas Williams There's also a peak finder in photutils: local-peak-detection. Essentially it's a 2D maximum filter within a defined footprint shape/size.
James Davies - Has anyone tried skimage.registration.phase_cross_correlation to align miri images with few or no point sources?
Thomas Williams - yep! Didn’t work very well when we tried it out (we use Adam Ginsburg’s implementation, keflavich/image_registration). This was to try and align between different filters though, didn’t try on a per-tile level
Varun Bajaj - I'm not too surprised it didn't work so well between filters, way out in the red the different bands don't even look like each other anymore. That said, for the same band, if you undistorted the images first I could see that algorithm working decently well to get alignment within a pixel.
