| Data Segmentation |
| - Files need to be segmented into chunks if they are too large
- Baseline version of segmentation code works, but doesn't optimize for different values of NGROUPS
- In some cases, the choice of 2GB per segment results in large level-2a products (e.g. 6 GB in size). The 2GB segment size is fine for a single level-1b FITS file. In the case that NGROUPS=3 though, there's very little "compression" due to ramp fitting in the level-2a products, and because the level-2a products inflate the original raw data from int16 to float32, as well as add several more float32 data arrays of the same size (ERR, DQ, ...), resulting in the level-2a products that were ~6 GB in size.
- Decision: Use size_level1b = size_level2a * ngroups / 10. Pick what the size_level2a we want, and then L1b and segment size will follow from there via the equation
- Work hasn't started on new version
- Will need to briefly discuss file segments in JDox TSO strategy pages and link to more detailed discussion
|