dplyr::left_join error in convert_glatos_to_att()

[chrisholbrook]

Error encountered with convert_glatos_to_att() when input detectionObj contains 6 million rows:

Error in dplyr::left_join() at glatos/R/util-convert_glatos_to_att.r:115:3:
! This join would result in more rows than dplyr can handle.

# dtc_file is a GLATOS detection export csv file (from GLATOS data portal)
# rcv_file is a GLATOS receiver locations csv file (from GLATOS data portal)

detectionObj <- read_glatos_detections(dtc_file)
receiverObj <- read_glatos_receivers(rcv_file)
  

dtc_att <- convert_glatos_to_att(detectionObj, 
                                 receiverObj, 
                                 crs = sf::st_crs(4326))

[mhpob]

I don't believe this is a length thing at 6 million rows (though that error says it probably will be at 2.1 billion!): I created a toy dataset with 6m rows and it ran fine.

It should be that detectionObj dictates the maximum amount of rows, assuming there are fewer station records than detections or number of fish detected. We're seeing 460x the rows than the input dataset, so some sort of multimatch is going on, inflating what should be the maximum number of rows by 460x.

The joins in convert_glatos_to_att happen when the algorithm:

1. creates tag metadata by taking the unique tags in detectionObj, getting common names from WORMS, and joins them together: https://github.com/ocean-tracking-network/glatos/blob/main/R/util-convert_glatos_to_att.r#L83-L94
2. builds on the tag metadata by joining with release records: https://github.com/ocean-tracking-network/glatos/blob/main/R/util-convert_glatos_to_att.r#L97-L115
3. joins detectionObj with receiverObj to match detections with corresponding receiver: https://github.com/ocean-tracking-network/glatos/blob/main/R/util-convert_glatos_to_att.r#L119-L132

Could the common names in detectionObj match with multiple scientific names, returned by WORMS a la #237? That could inflate things at step 1. Depending on how many receivers you have it could inflate things further in step 2. The error says that line 115 is the culprit -- I don't know if that includes the roxygen comments or not. If it doesn't, step 2 is the issue, making the WORMs thing possible.

@chrisholbrook do you recall what the species were in the 6m row dataset?

[chrisholbrook]

do you recall what the species were in the 6m row dataset?

Lake Whitefish

Looks to only match 1 scientific name in worms:

> glatos:::query_worms_common("Lake Whitefish")
[1] "https://www.marinespecies.org/rest/AphiaRecordsByVernacular/Lake%20Whitefish"
[[1]]
[1] "Coregonus clupeaformis"

[mhpob]

Darn. No idea what the multi-match is here without looking at the data. Have you tried going line-by-line through the function's code? I'm really going to be surprised if this is being tripped up by "only" 6m rows.

[mhpob]

It looks like this is doing a VERY GREEDY join, matching every detection to every deployment record, then filtering to only detections at the right station, while the receiver was deployed.

I moved to dplyr::left_join(detections, receivers, by = dplyr::join_by(between(detection_time, deploy_time, recover_time))), but I'm not getting an identical object.

A non-equi join may be the path forward on this, but I need to test that output and get it to match.

[chrisholbrook]

On hold; currently planning to develop crosswalk to ATO and deprecate convert_glatos_to_att().