Skip to content

Add GOES Dataset for Gen2#340

Open
kevinyang-cky wants to merge 6 commits intomainfrom
goes4gen2
Open

Add GOES Dataset for Gen2#340
kevinyang-cky wants to merge 6 commits intomainfrom
goes4gen2

Conversation

@kevinyang-cky
Copy link
Copy Markdown
Collaborator

Description

The GOES dataset complies with the Gen2 data scheme and provides access to the Level 2 Cloud and Moisture Imagery product (MCMIP), which packages all GOES channels into a single file as either reflectance or brightness temperature.

 
Features:

  • Supports reading locally or streaming directly from AWS in NetCDF4 format using the S3Fs library. Streaming all 16 channels, either over the full disk or CONUS, takes under 10 seconds (conservative estimate).
  • Supports single and multi-step training.
  • Supports subsetting over the GOES curvilinear grid.

There might be some merge conflict with #332 on config/multi_source_data.yaml and credit/datasets/multi_source.py, so once that is merged, this PR can make additional adjustments.

Special thanks to @charlie-becker for extensive discussion throughout the development process.

Related Issues and PRs

Closes #237

Checklist

  • I am familiar with the contributing guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date.

Copy link
Copy Markdown
Collaborator

@charlie-becker charlie-becker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for implementing this! The code looks good and I appreciate the testing scheme. Please resolve the merge conflict (just syncing multi_source.py to include the HRRR datasets) and then we can merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend Data Loader for Remotely Sensed Data

3 participants