Shannon review of IMOS to OTN outputs

[slandovs]

Addressing four questions from Bruce:

1. what is there that shouldn't be
2. what is not there that should be?
3. what is there but incorrect?
4. what is not there but could be, based on the source IMOS data?

GENERAL NOTES:

• the columns are not in the same order between the outputs and the OTN templates, but I am assuming this does not matter
• NA should not be autofilled into the blank cells (this is happening in both the rcvr and tag outputs)
• any edits I made on column headers below, I wrote in lower case despite column headers being in all caps

RECEIVER METADATA:
Goal: to have IMOS receiver metadata in OTN deployment metadata format

Q. 1

• I am not sure what the column with header "-" is supposed to be, but I am assuming it shouldn't be there?

Q. 2

• transmitter column
• transmitter_model column
• deployed_by column
• filename column
• do you have any examples from IMOS that have VR2AR rcvrs instead of VR2W? I would be curious if the IMOS template includes the transmitter columns then!

Q. 3

• deploy_date_time: formatting needs to be yyyy-mm-ddThh:mm:ss
• deploy_date_time: column header should be "deploy_date_time (yyyy-mm-ddThh:mm:ss)"
• ins_model_number: should be ins_model_no
• ins_serial_number: should be ins_serial_no
• ar_model_number: should be ar_model_no
• ar_serial_number: should be ar_serial_no
• recovered: values do not align with the OTN requirements (only accept y, n, l, f, moved, failed)
• recover_date_time: formatting needs to be yyyy-mm-ddThh:mm:ss
• download_date_time: formatting needs to be yyyy-mm-ddThh:mm:ss

Q. 4

• recover_lat: from receiver_recovery_latitude in IMOS sheet
• recover_long: from receiver_recovery_longitude in IMOS sheet

TAGGING METADATA:
Goal: to to have IMOS animal measurements and transmitter deployment metadata in OTN tagging metadata format

Q. 1
nothing

Q. 2

• animal_id column
• tag_activation_date column
• capture_location column
• capture_latitude column
• capture_longitude column
• length2 (m) column
• length2_type column
• dna_sample_taken column
• treatment_type column
• release_group column
• colums AM to BE (didn't want to type them all out)
• lots of these columns are not essential, but we include them in our metadata template so I wanted to note them

Q. 3

• sex: formatting should be M, F, or U
• utc_release_date_time: formatting needs to be yyyy-mm-ddThh:mm:ss
• comments: even when there are comments in the IMOS transmitter deployment sheet, an NA is still being added (for example "NA; xxxx"

Q. 4

• tag_implant_method: placement column in IMOS transmitter deployment sheet
• transmitter_sensor_type: this is in IMOS data and could be added to the output. We don't include it in our tagging metadata sheets, but often people add this column in. Just wanted to note that!

DETECTION EXTRACTS:
Goal: to have IMOS detections in OTN matched to animals detection extract format

Q. 1

• receiver_id column
• transmitter_name column
• transmitter_serial column

Q. 2

• collectioncode column
• catalognumber column
• datelastmodified column
• detectedby column
• receiver_group column
• bottom_depth column
• receiver_depth column
• codespace column
• sensorraw column
• the_geom column
• yearcollected column
• monthcollected column
• daycollected column
• julianday column
• timeofday column
• local_area column
• notes column
• citation column
• unqdetecid column
• contact_poc column
• contact_pi column
• I do think some of these columns (like receiver_group, etc) might be found in the IMOS detections file, I just don't feel familiar enough with this to tell
• lots of these columns also show up in the metadata above, they just also need to be included here now!

Q. 3

• date_ane_time: should be datecollected with formatting yyyy-mm-dd hh:mm:ss
• date_ane_time: has "Z" for timezone included, but that should be in its own column, "timezone" and formatted as "UTC"
• receiver: should just have the serial number, not model-serial
• transmitter: should be called tagname
• sensor_value: should be called sensorvalue
• sensor_unit: should be called sensorunit
• station_name: should be called station

Q. 4

• scientificname: from IMOS column species_scientific_name
• commonname: from IMOS column species_common_name
• sensorname: from IMOS column transmitter_id
• sensortype: from IMOS column transmitter_sensor_type
• do we know if any processing has been done to the transmitter_sensor_value in the IMOS sheet? I am trying to determine if this is equivalent to OTN sensorraw or sensorvalue

[jackVanish]

Columns being populated with NA is a write.csv option, not a surimi option, so we're golden on that front.

[jackVanish]

Regarding the comments in tag files still containing NAs, this is an artefact of how we're generating that column. The tag output is actually a synthesis of two files, the IMOS tag file and the IMOS animal measurements file, both of which contain a comments field. We get them both and merge them; however, the column in the animal measurements file (in this case) actually does contain 'NA' as its data. So, not a mistake per se, but something I'll nonetheless write in a catch for.

[jackVanish]

'placement' is being assigned to TAG_IMPLANT_TYPE, I confirmed with Shannon that this is as it should be. However, we were missing TAG_IMPLANT_METHOD so that's now being populated as a blank column.

[jackVanish]

'collectioncode' on the detections file I'm going to do the same as we do on the OTN -> IMOS side and have it exposed as a parameter to the user.,