review code used to pull NWIS data

[jordansread]

https://github.com/USGS-VIZLAB/gages-through-ages/blob/master/scripts/fetch/getData_siteRecords.R

One thing I am noticing is that we have some duplicated rows:

library(dplyr) # using data.in from process.disch_sites
filter(data.in$`disch-data`) %>% distinct() %>% nrow
[1] 582882
filter(data.in$`disch-data`) %>% nrow
[1] 583902

here is one site in particular:

filter(data.in$`disch-data`, site_no == '02053200') %>% .$year %>% length()
[1] 116
filter(data.in$`disch-data`, site_no == '02053200') %>% .$year %>% unique() %>% length()
[1] 58

[jordansread]

Looks like nDays is creating additional non-unique rows:

filter(data.in$`disch-data`, site_no == '02053200') %>% select(-nDays)  %>% distinct() %>% nrow
[1] 58
filter(data.in$`disch-data`, site_no == '02053200') %>% distinct() %>% nrow
[1] 116

[jordansread]

On top of the other distinct I had above, if we only keep site_no and year, we have 17,737 duplicated rows

select(site_no, year) %>% distinct %>% nrow

[jordansread]

These sites seem to be missing too:
https://waterdata.usgs.gov/pr/nwis/inventory/?site_no=50063440
https://waterdata.usgs.gov/hi/nwis/uv?site_no=16229000
https://waterdata.usgs.gov/nwis/inventory/?site_no=02430615

this one is in HCN (hydro-clim), but limited data:
https://waterdata.usgs.gov/nwis/inventory/?site_no=04127918

And these are only the sites that I am cross-ref'ing w/ hydroclim, so there are likely more if we can figure out why these didn't end up (maybe with the exception of the last one).

[ldecicco-USGS]

https://waterservices.usgs.gov/nwis/site/?sites=50063440,16229000&hasDataTypeCd=dv&seriesCatalogOutput=TRUE&parameterCd=00060&format=rdb

== 404

[ldecicco-USGS]

So, not sure why they wouldn't show up in the site file, but that's why their not showing up in our data. I'll keep poking around

[ldecicco-USGS]

Pulling internal data instead of external, gets some of these:

> "50063440" %in% x$site_no
[1] TRUE
> "16229000" %in% x$site_no
[1] TRUE
> "02430615" %in% x$site_no
[1] FALSE
> "04127918" %in% x$site_no
[1] TRUE

[ldecicco-USGS]

And when you go to the NWIS sites that doesn't show up with the internal data set to TRUE, you see this:

So...I'm not surprised to not get that data