Summary of lake-temperature-model-prep outputs for Machine Learning Strategy Writing tasks

[padilla410]

Overview

On 2022-07-07 @amsnyder asked me about the provenance of the following pipeline output files:

• 1_crosswalk_fetch/out/ned_centroid_elevations.feather
• 7_config_merge/out/canonical_lakes_area.rds
• 7_config_merge/out/nml_Kw_values.rds
• 7_config_merge/out/nml_meteo_fl_values.rds
• 7b_temp_merge/out/merged_temp_data_daily.feather
• 8_viz/inout/lakes_summary.feather

She needs this information to evaluate the data needs associated with ML projects that use outputs from this repo.

Data Summary

File	Desc	Other notes
1_crosswalk_fetch/out/ned_centroid_elevations.feather	Crosswalk between lakes of interest and USGS NED elevation. Values are accessed using the elevatr package.
7_config_merge/out/canonical_lakes_area.rds	Table of surface areas for all lakes of interest. Derived from 1_crosswalk_fetch/out/canonical_lakes_sf.rds and NHDHR geodatabases.
7_config_merge/out/nml_Kw_values.rds	Kw is the light extinction coefficient (see GLM &light doc). This data set matches each lake of interest to a Kw value. Values are derived from one of three data sources: LAGOS secchi data, WQP secchi data, landsat secchi data from Max Glines	The Glines data was first uploaded by Jordan Read 2022-01-07. This might be the relevant paper.
7_config_merge/out/nml_meteo_fl_values.rds	Crosswalk that matches each lake of interest to an NLDAS meteo file.	The NLDAS files are generated by Jordan Read on his local machine. Here is an example snippet
7b_temp_merge/out/merged_temp_data_daily.feather	Derived from 7b_temp_merge/out/merged_temp_data_alltimes.feather using reduce_temp_data function. The ~alltimes.feather data set combines WQP depth/temp data with cooperator temp/depth data. The reduce_temp_data function IDs unique combos of lake-date-depth values or munges lakes that have non-unique values.
8_viz/inout/lakes_summary.feather	This is really just a summary file of all of the upstream info that comes out of the pipeline for use in downstream modeling efforts.	table column names (15):  "site_id", "GNIS_Name", "n_obs", "n_profiles", "longitude", "latitude", "zmax", "hypsography", "hypsography_digitizing", "kw", "kw_file", "meteo", "obs_category", "main_source", "Lake_Source"

[padilla410]

@jread-usgs and @hcorson-dosch please feel free to weigh in on the above summary

[amsnyder]

Thanks so much for compiling this @padilla410!

Do you (or anyone else) have any more information about who your cooperators are that provide temperature + depth data? If it's a big lift, don't worry about it, but if you have a list somewhere handy, I'd like to get it!

[jordansread]

@amsnyder I'm wondering how extensive you'd like that list to be. For WQP data, we track the organization source (there are hundreds in the final dataset) and then for the more custom parser datasets that are added (usually .xls .csv, etc), we have some metadata for each file that captures the origin. These files are often subgroups within orgs, for example, the MN DNR may have a research group working on lakes X, Y, and Z that collected data historically, all in different formats.

We can get you that WQP source ID information and also access to the metadata files that explain each of the custom files, but it would be a lot of work for anyone to translate the latter into a more condensed format (but it would provide value, so I'm not claiming it wouldn't potentially be worth it).

[amsnyder]

I guess I'm just trying to get a more general sense of what "cooperator data" means. Some things I'm thinking about:

• Is the cooperator data all publicly accessible online? Or is some of it privately shared with your group?
• Are there specific types of organizations you collect data from (government, research institutions, NGOs)? Are there types of organizations you would not accept data from?

[jordansread]

Here is the complete list of "cooperator data" in the pipeline right now (with file hash): https://github.com/USGS-R/lake-temperature-model-prep/blob/main/6_temp_coop_fetch/log/6_temp_coop_fetch_tasks.ind

My understanding is that some of these data may be accessible online, but we weren't aware of it (e.g., some of these data were also entered into STORET by the state in some cases). They are mostly "grey" though. Local files, or in some cases, paper files that our collaborators digitized. Most of these came in when our collaborators/co-PIs sent inquiries to the state agencies for data, but there are several exceptions to that (e.g., a retired water scientist who collects data on Ten Mile Lake contributed to this as well). I don't think there are specific organizations that we'd not accept data from, but we do end up filtering out data that doesn't include adequate metadata, which usually means we can't resolve which lake it was (i.e., the lake ID is ambiguous for some reason) and/or can't resolve the depth of the measurement(s). Because temperature measurements are quite a bit less error-prone compared to other water quality measurements, we have a lower bar for communicating the sampling methods.

For this question:

government, research institutions, NGOs

gov: yes; research institutions: yes, especially UW-Madison, U-MN, Iowa, and MSU; NGOs: perhaps some but I can't think of them (there are certainly some in the WQP/STORET dataset but I'm not sure about coop data w/o taking a more detailed look).

[amsnyder]

Thanks @jread-usgs - this is very helpful! I think I've got what I need for now. :)