integration with TCIA data retriever

[kirbyju]

Hi, TCIA recently started a partnership with the NCI CRDC folks who run the Clinical Translational Data Commons and General Commons to host TCIA's controlled access datasets. We also have an older dataset in the Integrated Canine Data Commons.

I'm very interested in having our team integrate the gen3 download capabilities directly into our "TCIA Data Retriever" application so that our users don't need to install multiple different applications when they find a dataset they like on our site but we could use some high level advice on this.

The workflow I'm envisioning is that we would generate bulk download "manifests" containing drs_uri's in them for each controlled access dataset to post on our dataset homepages, or users could go to the web GUI for these Commons and export subsets using the spreadsheet export on the cart pages after building a cohort. Then, upon seeing that a user has selected a JSON or spreadsheet manifest containing drs_uri's, our Retriever would automatically know to grab the data from CRDC by parsing those URI's.

In most cases (everything except the canines data) the data are controlled access, so I'm assuming we would also need to instruct users on how to apply for access through dbgap, create their json API key at https://nci-crdc.datacommons.io/ and specify that file location to the Retriever.

Also, the tool we're working on is based in Go. Do you happen to have an sdk for that or would we have to re-write what you're doing here?

Happy to setup a meeting if that would be easier.

Thanks!
Justin

[kirbyju]

Scratch that. I got some help from the gen3 slack channel and we got this working.