Skip to content

Commit e082690

Browse files
rtapellartapella
authored
Merge pull request #25 from unity-sds/update-notebooks-with-dockstore-info
Update notebooks with dockstore info
2 parents cbfb64e + f1abe31 commit e082690

21 files changed

+261
-16
lines changed

‎.gitattributes

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.png filter=lfs diff=lfs merge=lfs -text

‎img/jl_01.png

-89 KB
Loading

‎img/jl_02.png

-12.9 KB
Loading

‎img/jl_03.png

-16.4 KB
Loading

‎jupyter-notebooks/.DS_Store

-6 KB
Binary file not shown.

‎jupyter-notebooks/tutorials/1_working_with_applications.ipynb

Lines changed: 125 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,130 @@
66
"metadata": {},
77
"source": [
88
"# Working with Applications\n",
9-
"The intent of this tutorial is to help familiarize yourself with submitting applications to the Unity Application Catalog. Adding applications to the application catalog allows you and others to generate data by submitting jobs that use an application selected from the catalog."
9+
"The intent of this tutorial is to help familiarize yourself with submitting applications to the Unity Application Catalog. Adding applications to the application catalog allows you and others to generate data by submitting jobs that use an application selected from the catalog.\n",
10+
"\n",
11+
"***Note, accounts and repositories hosted on GitHub.com are currently supported. GitHub Enterprise is not supported.***"
12+
]
13+
},
14+
{
15+
"cell_type": "markdown",
16+
"id": "7fe60b57-62c8-44a8-aa20-cf5dd7551af1",
17+
"metadata": {
18+
"jp-MarkdownHeadingCollapsed": true,
19+
"tags": []
20+
},
21+
"source": [
22+
"## I. Create a Dockstore account"
23+
]
24+
},
25+
{
26+
"cell_type": "markdown",
27+
"id": "ac226cbe-bcd9-4365-b4a0-a3ad55fb3a90",
28+
"metadata": {
29+
"tags": []
30+
},
31+
"source": [
32+
"#### 1. Visit [Unity's Dockstore Test Instance](http://uads-test-dockstore-deploy-lb-1762603872.us-west-2.elb.amazonaws.com:9998/).\n",
33+
"_For the purposes of this tutorial, we will be using Unity's test environment._\n",
34+
"\n",
35+
"![Dockstore Home](images/01-Dockstore-Home.png \"Dockstore Homepage\")\n",
36+
"<br><br>\n",
37+
"\n",
38+
"#### 2. Click Register in the top right corner and you will be redirected to the Dockstore Registration page.\n",
39+
"\n",
40+
"![Dockstore Registration Page](images/02-Register.png \"Dockstore Registeration Page\")\n",
41+
"<br><br>\n",
42+
"\n",
43+
"#### 3. Agree to Dockstore's Terms of Service and Privacy by enabling the checkbox and then clicking the \"Register with GitHub\" button.\n",
44+
"\n",
45+
"![Dockstore TOS and Privacy Policy Acknowledgement](images/03-Enable-Checkbox.png \"Acknowledge TOS and Privacy Policy\")\n",
46+
"<br><br>\n",
47+
"\n",
48+
"#### 4. Authorize the JPL Dockstore Application access to your GitHub account.\n",
49+
"\n",
50+
"![Unity Dockstore Application Authorization](images/04-Dockstore-App-Authorization.png \"Authorize Unity Dockstore Application\")\n",
51+
"<br><br>\n",
52+
"\n",
53+
"#### 5. Verify your Github Username is correct and click `Next`.\n",
54+
"\n",
55+
"![Verify GitHub Username](images/05-Verify-Username.png \"Verify Github Username\")\n",
56+
"<br><br>\n",
57+
"\n",
58+
"#### 6. On the Link External Accounts screen, scroll to the bottom and click `Next`.\n",
59+
"\n",
60+
"![Link External Accounts Screen](images/06-02-Link-External-Accounts-Click-Next.png \"Click next at the bottom of the Link External Accounts Screen\")\n",
61+
"<br><br>\n",
62+
"\n",
63+
"#### 7. On the Setup Dockstore CLI screen, scroll to the bottom and click `Next`.\n",
64+
"\n",
65+
"![Setup Dockstore CLI screen](images/07-02-Setup-CLI-Click-Next.png \"Click next at the bottom of the Setup Dockstore CLI screen.\")\n",
66+
"<br><br>\n",
67+
"\n",
68+
"#### 8. When you reach the \"Next Steps\" screen, you're account is set up and ready to be configured with your GitHub Account and Application Repositories.\n",
69+
"\n",
70+
"![Next Steps Dockstore Registration Screen](images/08-Next-Steps.png \"Final Dockstore Setup Screen\")\n",
71+
"<br><br>\n",
72+
"\n",
73+
"---"
74+
]
75+
},
76+
{
77+
"cell_type": "markdown",
78+
"id": "1ae3c385-7190-4a5e-963f-617190d1a64c",
79+
"metadata": {
80+
"jp-MarkdownHeadingCollapsed": true,
81+
"tags": []
82+
},
83+
"source": [
84+
"---\n",
85+
"## II. Configure GitHub Account and Repository Access for Dockstore\n"
86+
]
87+
},
88+
{
89+
"cell_type": "markdown",
90+
"id": "b32832c9-f7e5-4578-95b3-9bb6347e4a38",
91+
"metadata": {
92+
"tags": []
93+
},
94+
"source": [
95+
"\n",
96+
"#### 1. Visit [GitHub](https://www.github.com) and login to your account.\n",
97+
"#### 2. Visit [https://github.com/apps/jpl-uads-dockstore](https://github.com/apps/jpl-uads-dockstore) and click the \"Configure\" button on the right of the page.\n",
98+
"\n",
99+
"![JPL Unity Dockstore GitHub Configuration](images/10-Configure-Dockstore-App.png \"Configure JPL Unity Dockstore GitHub App\")\n",
100+
"<br><br>\n",
101+
"\n",
102+
"#### 3. Configure the JPL Unity Dockstore application by providing it access to either:\n",
103+
"\n",
104+
"##### All the repositories in your account or\n",
105+
"##### Specific repositories by selecting those that Dockstore should have access to. When you have finished setting your repository information, click `Install`.\n",
106+
"\n",
107+
"![Repository Selection Screen for Dockstore](images/11-Install-Dockstore-App.png \"Install JPL Unity Dockstore Application\")\n",
108+
"<br><br>\n",
109+
"\n",
110+
"#### 4. Add `.dockstore.yml` to your repository.\n",
111+
"\n",
112+
"Add a `.dockstore.yml` configuration file to your repository. Documentation to configure this file can be found [here](https://docs.dockstore.org/en/stable/getting-started/github-apps/github-apps.html). Once you have added this file to your repository, it will take approximiately 5 minutes for Dockstore to process it. once completed, your application will show in your Dockstore account under \"My Workflows\"."
113+
]
114+
},
115+
{
116+
"cell_type": "markdown",
117+
"id": "721fbed7-ac51-4b31-98c1-8f3c7b5b8073",
118+
"metadata": {
119+
"jp-MarkdownHeadingCollapsed": true,
120+
"tags": []
121+
},
122+
"source": [
123+
"---\n",
124+
"## III. Continue development and maintenance of application"
125+
]
126+
},
127+
{
128+
"cell_type": "markdown",
129+
"id": "2595b6d6-fe13-4ac0-aa26-6ab6b29d7644",
130+
"metadata": {},
131+
"source": [
132+
"Now that your application is integrated with Dockstore, as you make changes, they will automatically be reflected in Dockstore."
10133
]
11134
}
12135
],
@@ -26,7 +149,7 @@
26149
"name": "python",
27150
"nbconvert_exporter": "python",
28151
"pygments_lexer": "ipython3",
29-
"version": "3.9.12"
152+
"version": "3.9.6"
30153
}
31154
},
32155
"nbformat": 4,

‎jupyter-notebooks/tutorials/3_working_with_jobs.ipynb

Lines changed: 93 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -138,12 +138,99 @@
138138
" print(\"Token received.\")"
139139
]
140140
},
141+
{
142+
"cell_type": "markdown",
143+
"id": "98017416-3d3c-4225-843f-8f2fa35be36b",
144+
"metadata": {},
145+
"source": [
146+
"## 1. Query Dockstore"
147+
]
148+
},
149+
{
150+
"cell_type": "code",
151+
"execution_count": null,
152+
"id": "ce34b502-6b55-42d1-bce8-f0a382ea392f",
153+
"metadata": {},
154+
"outputs": [],
155+
"source": [
156+
"r = requests.get(\"http://uads-test-dockstore-deploy-lb-1762603872.us-west-2.elb.amazonaws.com:9998/api/workflows/published\")\n",
157+
"r.raise_for_status()\n",
158+
"json = r.json()\n",
159+
"\n",
160+
"print(\"\\n\\nList of Application Packages available in Dockstore:\")\n",
161+
"JSON(json)"
162+
]
163+
},
164+
{
165+
"cell_type": "markdown",
166+
"id": "59a80d9c-6a9a-4612-bda5-39eaf64bf8e9",
167+
"metadata": {},
168+
"source": [
169+
"## 2. Select an application from Dockstore\n",
170+
"\n",
171+
"In the previous step we queried Dockstore and retrieved a list of applications. From this list of applications, select one and set the `workflow_id` and `workflow_version_id` in the cell below.\n",
172+
"\n",
173+
"`workflow_id` is a field labeled `id` from Dockstore.\n",
174+
"`workflow_version_id` is a field labeled `id` in the `workflowVersions` object.\n",
175+
"\n",
176+
"The ID values set below reference an example Sounder SIPS L1B application registered in Unity's Dockstore Test environment."
177+
]
178+
},
179+
{
180+
"cell_type": "code",
181+
"execution_count": null,
182+
"id": "b0c6b491-8130-4f8c-a8a3-5bd19c90ba24",
183+
"metadata": {},
184+
"outputs": [],
185+
"source": [
186+
"workflow_id = 16\n",
187+
"workflow_version_id = 31"
188+
]
189+
},
190+
{
191+
"cell_type": "markdown",
192+
"id": "9e36a5fb-25ab-40ad-a9cb-ed2c06597cea",
193+
"metadata": {},
194+
"source": [
195+
"## 3. Fetch the application's metadata\n",
196+
"\n",
197+
"The code below will download a ZIP file containing the CWL files associated with the selected package. The name of the zip will the name of the application (aka workflow). The information in the ZIP file will be used to create a JSON payload needed to deploy the application to Unity's Science Processing Service.\n",
198+
"\n",
199+
"***Note, this ZIP file contains a file named `.dockstore.yml` which when unpackad will not be visible in JupyterLab's Folder/File View, it will be viewable via a Terminal window.***"
200+
]
201+
},
202+
{
203+
"cell_type": "code",
204+
"execution_count": null,
205+
"id": "5f1c287f-939c-424d-bb12-b5145643ba38",
206+
"metadata": {},
207+
"outputs": [],
208+
"source": [
209+
"r_zip = requests.get(\"http://uads-test-dockstore-deploy-lb-1762603872.us-west-2.elb.amazonaws.com:9998/api/workflows/{workflowId}/zip/{workflowVersionId}\".format(workflowId=16,workflowVersionId=31))\n",
210+
"r_zip.raise_for_status()\n",
211+
"\n",
212+
"workflow_name = json[0]['workflowName']\n",
213+
"open(\"{}.zip\".format(workflow_name), 'wb').write(r_zip.content)"
214+
]
215+
},
216+
{
217+
"cell_type": "markdown",
218+
"id": "69d43073-3e99-4ea2-8021-9104b4003e84",
219+
"metadata": {},
220+
"source": [
221+
"## 4. Deploy Application\n",
222+
"\n",
223+
"Now that we have collected the information about the application from Dockstore, we can package it up and prep it for deployment to Unity's Science Processing Service.\n",
224+
"\n",
225+
"Coming soon..."
226+
]
227+
},
141228
{
142229
"cell_type": "markdown",
143230
"id": "9f1cdbbc-6c38-4c2a-955a-08ac8d13e366",
144231
"metadata": {},
145232
"source": [
146-
"## 1. List Processes\n",
233+
"## 5. List Processes\n",
147234
"\n",
148235
"The Unity API defines a way of viewing all deployed application packages (a.k.a., Processes) on the system using the 'get_apps' function. After a successful deployment of an application package to the Processing endpoint, you should see a new entry for the deployed application.\n",
149236
"\n",
@@ -173,7 +260,7 @@
173260
"id": "00fc3f8f-8d05-4765-92cb-99bc25778a02",
174261
"metadata": {},
175262
"source": [
176-
"## 2. Execute a job\n",
263+
"## 6. Execute a job\n",
177264
"Before deploying Applications and working with jobs, it is assumed that a system administrator has deployed an ADES. These are often called \"venues\" or \"environments\", and some examples may be dev, test, prod, etc. To run a Job, you need a Process available as well (a deployed Application). In this case we have a demo application deployed and referenced in the setup step 0 above.\n",
178265
"\n",
179266
"With an ADES and a Process ready to accept our Job, we can submit a Job along with any input data or parameters that are needed (as defined by a template job definition for a particular Application). In this example case, none are needed so `inputs` is blank. The response will provide a Job ID that we will store in a variable called `job_id` for use later.\n",
@@ -222,7 +309,7 @@
222309
"id": "7aed28d0-ab28-445c-bbb7-9ead5f5aa160",
223310
"metadata": {},
224311
"source": [
225-
"## 3. Get the Job status\n",
312+
"## 7. Get the Job status\n",
226313
"The code below will demonstrate how one can check the status of the Job. The potential status responses are documented [here] _need a reference to process status_.\n",
227314
"\n",
228315
"In this example, we will look up the status of the predfined application name from the setup step 0, and the job ID that was returned previously. This function will loop/block until the process ends. You will see a printout every 5 seconds."
@@ -260,7 +347,7 @@
260347
"id": "7198115b-2142-4ac7-8581-b1664b2278e7",
261348
"metadata": {},
262349
"source": [
263-
"## 4. Get Job results\n",
350+
"## 8. Get Job results\n",
264351
"Now that we've monitored the status of a Job, and verified that is has completed, we can retreive information about where the generated data is located. \n",
265352
"\n",
266353
"In this example, we will use the predefined Process name, and the Job ID that was returned previously."
@@ -307,7 +394,7 @@
307394
"id": "1d9e1a16-cc62-4d0a-87a8-f1c0256c9ac3",
308395
"metadata": {},
309396
"source": [
310-
"## 5. List the Jobs processed by a given Process\n",
397+
"## 9. List the Jobs processed by a given Process\n",
311398
"\n",
312399
"What if I restarted my notebook and I have no Job ID? The API can ask the process endpoint to list the Jobs for a given Process (deployed Application).\n"
313400
]
@@ -330,14 +417,6 @@
330417
"print(\"\\n\\nFull JSON output data of get_jobs_for_app\")\n",
331418
"JSON(jobs)"
332419
]
333-
},
334-
{
335-
"cell_type": "code",
336-
"execution_count": null,
337-
"id": "2f68959b-883b-4aee-a0ce-4105a4cf5aca",
338-
"metadata": {},
339-
"outputs": [],
340-
"source": []
341420
}
342421
],
343422
"metadata": {
@@ -356,7 +435,7 @@
356435
"name": "python",
357436
"nbconvert_exporter": "python",
358437
"pygments_lexer": "ipython3",
359-
"version": "3.10.5"
438+
"version": "3.9.6"
360439
}
361440
},
362441
"nbformat": 4,

‎jupyter-notebooks/tutorials/images/01-Dockstore-Home.png

Lines changed: 3 additions & 0 deletions
Loading

‎jupyter-notebooks/tutorials/images/02-Register.png

Lines changed: 3 additions & 0 deletions
Loading

‎jupyter-notebooks/tutorials/images/03-Enable-Checkbox.png

Lines changed: 3 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)