Guidance: How to add objects from visitor folder to dataset

[dvirlar2]

The code below is what I used to add data from visitor/Fischer into their appropriate dataset. Make sure this documentation get's added to the reference manual's Edit EML section.

# Script Goal: Add files from visitor directory to data package

## -- load libraries -- ##
library(dataone)
library(datapack)
library(uuid)
library(arcticdatautils)
library(EML)


## -- get dataset -- ##
d1c <- dataone::D1Client() # removed for privacy concerns 

packageId <- "resource_map_pid_here"
dp <- getDataPackage(d1c, identifier = packageId, lazyLoad=TRUE, quiet=FALSE)

xml <- selectMember(dp, "sysmeta@fileName", ".xml")


## -- retrieve all files -- ##
# Get file paths to the data
files <- list.files("/home/visitor/Fischer", full.names = TRUE)


## -- Add files to package -- ##
for(i in 1:length(files)){
   # create new data object for each file
  data_object <- new("DataObject", format="netCDF-4", filename=files[[i]])

  # add that object to the data package
  dp <- addMember(dp, data_object, xml)
}


## -- update package -- ##
myAccessRules <- data.frame() # removed for privacy concerns

newPackageId <- uploadDataPackage(d1c, dp, public = FALSE, 
                                  accessRules = myAccessRules, quiet = FALSE)

[dvirlar2]

This might work out really well for large data workflows

[dvirlar2]

When uploading files from visitor, the data objects are stored as pids in the system metadata, but do not automatically generate entities from these objects. As such, after uploading files from visitor to the intended dataset, we then need to create the entities themselves from the data pids.

## -- get dataset -- ##
d1c <- dataone::D1Client("PROD", "urn:node:ARCTIC")

packageId <- ""
dp <- getDataPackage(d1c, identifier = packageId, lazyLoad=TRUE, quiet=FALSE)

xml <- selectMember(dp, "sysmeta@fileName", ".xml")

doc <- read_eml(getObject(d1c@mn, xml))


## -- create entities from pids -- ##
ids <- get_package(d1c@mn, packageId)

# save data pids from working version to a variable
dat_pids <- ids$data

# loop through all pids to create entities
for(i in 1:length(dat_pids)){
  doc$dataset$otherEntity[[i]] <- pid_to_eml_entity(d1c@mn,
                    dat_pids[i],
                    entity_type = "otherEntity"
                    )
}

eml_validate(doc)
   # TRUE

In this example, the files uploaded were different from the Fischer ones. Because they are .nc files, I thought I should create dataTables instead of otherEntities. This resulted in a failed eml validation. I tried to again create dataTables, but also adding a unique entityDescription (which just included the file name as an easy band-aid solution), and this did not validate the eml either. Based on the error message I received, I'm guessing the entities as dataTables also needed to have attributeLists. I'm fairly certain that would have gotten around the validation issue.

Because I was uploading 500-1000 files, to 8 separate datasets, I prioritized getting the entities created and eml validated in one step, as opposed to doing that and adding attributes at the same time. For large or complex datasets, I would recommend following this step-by-step method as opposed to everything at once.