Related issues

[alexander-held]

This collects issues / pull and merge requests that relate to work on this project.

related to coffea

• postprocess step in processor Making postprocess() optional? scikit-hep/coffea#1383
• robust processing: Robust processing: partial aggregation, recovery, resubmission scikit-hep/coffea#1393, feat: add checkpointing functionality to coffea.processor.Runner scikit-hep/coffea#1420; prototype for a more robust dask scheduling logic feat: improved DaskExecutor scheduling logic scikit-hep/coffea#1468
• four-vector math on individual objects with coffea behaviors: Four-vector calculations on individual candidates vs arrays of candidates scikit-hep/coffea#1396
• nanoevents behavior in jitted code: Using nanoevents behaviors in numba-jitted code scikit-hep/coffea#1397
• custom code execution during coffea preprocessing: Injecting user code into pre-processing for custom metadata extraction scikit-hep/coffea#1401
• catching any Python exceptions during processing feat: allow user to pass in tuple of exceptions to skipbadfiles scikit-hep/coffea#1441
• RNTuple with coffea RNTuple input handling (plus PHYSLITE schema) scikit-hep/coffea#1436
• tracking failed files in coffea Return information about failed chunks scikit-hep/coffea#1446
• distribution with coffea Runner interface and fixed worker allocation Dask distribution breaks with hard-coded scatter and fixed worker allocation scikit-hep/coffea#1490

related to uproot and awkward

• fancy indexing: Unexpected fancy indexing behavior scikit-hep/awkward#3619
• behaviors dropping custom properties in jitted code: Array behaviors in jit-compiled code scikit-hep/awkward#3623
• added safetensors conversion feat: add to_/from_safetensors scikit-hep/awkward#3685
• large RSS memory use with uproot.iterate (Potential) memory leak with uproot.iterate scikit-hep/uproot5#1528, fix: make step sizes of iterate more accurate scikit-hep/uproot5#1606
• reading single basket over xrootd (basket size performance debugging) Reading single basket over xrootd scikit-hep/uproot5#1582

related to ServiceX

• Adding a new ScienceImage to the TopCP ServiceX backend Update AnalysisBase and TOPCPToolkit versions in Dockerfile ssl-hep/science-images#50
• Rucio DS identifier for ServiceX Rucio dataset not resolved if no scope ssl-hep/ServiceX_analysis_utils#34
• UI for selecting TopCP image List available Science images for TopCP ssl-hep/ServiceX_frontend#682

related to schema

• interface for event-level selection flags Event-level selection flags scipp-atlas/atlas-schema#77
• repr length of arrays with ntuple schema and systematics Limit repr length for parameters scikit-hep/awkward#3705
• mixin for trigger match Trigger match mixin scipp-atlas/atlas-schema#79

uncategorized

• atlasopenmagic warnings in notebooks: Configuration of warnings module atlas-outreach-data-tools/atlasopenmagic#31
• identifying weight systematics more easily: https://gitlab.cern.ch/atlas/athena/-/merge_requests/81794
• ATLAS basket size studies
  • https://gitlab.cern.ch/atlas/athena/-/merge_requests/85105
  • https://gitlab.cern.ch/atlas/athena/-/merge_requests/85645
• panda removed file tracking https://its.cern.ch/jira/browse/ATLASPANDA-1647

histogramming as a service

• developed: https://github.com/pfackeldey/histserv
• filling remote hist with growing axes is a major performance bottleneck (growing is very costly), see: boost-histogram axes growth is a performance bottleneck... pfackeldey/histserv#1

[ariostas]

[ ] large RSS memory use with uproot.iterate (Potential) memory leak with uproot.iterate scikit-hep/uproot5#1528

This PR should make things better scikit-hep/uproot5#1606

[alexander-held]

Thanks for pointing that out! That's great to see, I think with this we can check off the item for now.