Skip to content

Commit f143daa

Browse files
aBurmeseDevrlhagerm
aBurmeseDev
and
rlhagerm
authored
Python: Add S3 Batch scenario with CloudFormation integration (#7572)
* Python: Add S3 Batch scenario * chore(README): Add README for S3 Batch Python * test: Add unit tests for S3 Batch Scenario Python * Fix job activation and update tests * chore(metadata): Add new metadata to s3-control-metadata * Add job status, priority updates, tagging * feat: add type hints and fix status message * chore(metadata): add snippet tags and metadata for every operations * refactor: standardize user input handling with helper and update metadata.yaml * refactor: restructure classes to follow Python examples standards * refactor: remove hard-coded region and use default config * refactor: use client injection pattern to follow Python examples standards * test: add service method stubbing pattern to follow Python Examples Standards * chore: fix snippet tags * chore: address metadata and README items * add clean up resources for when scenario fails * Update with status checking for cancellation, and simplified integration test. --------- Co-authored-by: Rachel Hagerman <[email protected]>
1 parent 2df2372 commit f143daa

File tree

9 files changed

+1091
-11
lines changed

9 files changed

+1091
-11
lines changed

‎.doc_gen/metadata/s3-control_metadata.yaml‎

Lines changed: 102 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,15 @@ s3-control_Hello:
1313
- description:
1414
snippet_tags:
1515
- s3control.java2.list_jobs.main
16+
Python:
17+
versions:
18+
- sdk_version: 3
19+
github: python/example_code/s3/scenarios/batch
20+
sdkguide:
21+
excerpts:
22+
- description:
23+
snippet_tags:
24+
- python.example_code.s3control.list_jobs
1625
services:
1726
s3-control: {ListJobs}
1827

@@ -29,15 +38,24 @@ s3-control_CreateJob:
2938
- s3control.java2.create_job.async.main
3039
- description: Create a compliance retention job.
3140
snippet_tags:
32-
- s3control.java2.create_job.compliance.main
41+
- s3control.java2.create_job.compliance.main
3342
- description: Create a legal hold off job.
3443
snippet_tags:
35-
- s3control.java2.create_job.compliance.main
44+
- s3control.java2.create_job.compliance.main
3645
- description: Create a new governance retention job.
3746
snippet_tags:
38-
- s3.java2.create_governance_retemtion.main
47+
- s3.java2.create_governance_retemtion.main
48+
Python:
49+
versions:
50+
- sdk_version: 3
51+
github: python/example_code/s3/scenarios/batch
52+
sdkguide:
53+
excerpts:
54+
- description:
55+
snippet_tags:
56+
- python.example_code.s3control.create_job
3957
services:
40-
s3-control: {CreateJob}
58+
s3-control: {CreateJob}
4159
s3-control_PutJobTagging:
4260
languages:
4361
Java:
@@ -49,8 +67,17 @@ s3-control_PutJobTagging:
4967
- description:
5068
snippet_tags:
5169
- s3control.java2.job.put.tags.main
70+
Python:
71+
versions:
72+
- sdk_version: 3
73+
github: python/example_code/s3/scenarios/batch
74+
sdkguide:
75+
excerpts:
76+
- description:
77+
snippet_tags:
78+
- python.example_code.s3control.put_job_tagging
5279
services:
53-
s3-control: {PutJobTagging}
80+
s3-control: {PutJobTagging}
5481
s3-control_DescribeJob:
5582
languages:
5683
Java:
@@ -62,8 +89,17 @@ s3-control_DescribeJob:
6289
- description:
6390
snippet_tags:
6491
- s3control.java2.describe_job.main
92+
Python:
93+
versions:
94+
- sdk_version: 3
95+
github: python/example_code/s3/scenarios/batch
96+
sdkguide:
97+
excerpts:
98+
- description:
99+
snippet_tags:
100+
- python.example_code.s3control.describe_job
65101
services:
66-
s3-control: {DescribeJob}
102+
s3-control: {DescribeJob}
67103
s3-control_DeleteJobTagging:
68104
languages:
69105
Java:
@@ -75,8 +111,17 @@ s3-control_DeleteJobTagging:
75111
- description:
76112
snippet_tags:
77113
- s3control.java2.del_job_tagging.main
114+
Python:
115+
versions:
116+
- sdk_version: 3
117+
github: python/example_code/s3/scenarios/batch
118+
sdkguide:
119+
excerpts:
120+
- description:
121+
snippet_tags:
122+
- python.example_code.s3control.delete_job_tagging
78123
services:
79-
s3-control: {DeleteJobTagging}
124+
s3-control: {DeleteJobTagging}
80125
s3-control_GetJobTagging:
81126
languages:
82127
Java:
@@ -88,8 +133,17 @@ s3-control_GetJobTagging:
88133
- description:
89134
snippet_tags:
90135
- s3control.java2.get_job_tagging.main
136+
Python:
137+
versions:
138+
- sdk_version: 3
139+
github: python/example_code/s3/scenarios/batch
140+
sdkguide:
141+
excerpts:
142+
- description:
143+
snippet_tags:
144+
- python.example_code.s3control.get_job_tagging
91145
services:
92-
s3-control: {GetJobTagging}
146+
s3-control: {GetJobTagging}
93147
s3-control_UpdateJobStatus:
94148
languages:
95149
Java:
@@ -101,8 +155,17 @@ s3-control_UpdateJobStatus:
101155
- description:
102156
snippet_tags:
103157
- s3control.java2.cancel_job.main
158+
Python:
159+
versions:
160+
- sdk_version: 3
161+
github: python/example_code/s3/scenarios/batch
162+
sdkguide:
163+
excerpts:
164+
- description:
165+
snippet_tags:
166+
- python.example_code.s3control.update_job_status
104167
services:
105-
s3-control: {UpdateJobStatus}
168+
s3-control: {UpdateJobStatus}
106169
s3-control_UpdateJobPriority:
107170
languages:
108171
Java:
@@ -114,8 +177,17 @@ s3-control_UpdateJobPriority:
114177
- description:
115178
snippet_tags:
116179
- s3control.java2.update_job.main
180+
Python:
181+
versions:
182+
- sdk_version: 3
183+
github: python/example_code/s3/scenarios/batch
184+
sdkguide:
185+
excerpts:
186+
- description:
187+
snippet_tags:
188+
- python.example_code.s3control.update_job_priority
117189
services:
118-
s3-control: {UpdateJobPriority}
190+
s3-control: {UpdateJobPriority}
119191
s3-control_Basics:
120192
synopsis: learn core operations for &S3Control;.
121193
category: Basics
@@ -132,5 +204,24 @@ s3-control_Basics:
132204
- description: An action class that wraps operations.
133205
snippet_tags:
134206
- s3control.java2.job.actions.main
207+
Python:
208+
versions:
209+
- sdk_version: 3
210+
github: python/example_code/s3/scenarios/batch
211+
sdkguide:
212+
excerpts:
213+
- description: Learn S3 Batch Basics Scenario.
214+
snippet_tags:
215+
- python.example_code.s3control.helper.S3BatchScenario
135216
services:
136-
s3-control: {CreateJob, DeleteJobTagging, DescribeJob, GetJobTagging, ListJobs, PutJobTagging, UpdateJobPriority, UpdateJobStatus}
217+
s3-control:
218+
{
219+
CreateJob,
220+
DeleteJobTagging,
221+
DescribeJob,
222+
GetJobTagging,
223+
ListJobs,
224+
PutJobTagging,
225+
UpdateJobPriority,
226+
UpdateJobStatus,
227+
}

‎python/example_code/s3/scenarios/batch/README.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Amazon S3 Batch for the SDK for Python (boto3)
2+
3+
## Overview
4+
5+
This example demonstrates how to use the AWS SDK for Python (boto3) to work with Amazon Simple Storage Service (Amazon S3) Batch Scenario. The scenario covers various operations such as creating an AWS Batch compute environment, creating a job queue, creating a job defination, and submitting a job, and so on.
6+
7+
Here are the top six service operations this scenario covers.
8+
9+
1. **Create an AWS Batch computer environment**: Creates an AWS Batch computer environment.
10+
11+
2. **Sets up a job queue**: Creates a job queue that will manage the submission of jobs.
12+
13+
3. **Creates a job definition**: Creates a job definition that specifies how the jobs should be executed.
14+
15+
4. **Registers a Job Definition**: Registers a job definition making it available for job submissions.
16+
17+
5. **Submits a Batch Job**: Submits a job.
18+
19+
6. **Checks the status of the job**: Checks the status of the job.
20+
21+
## ⚠ Important
22+
23+
- Running this code might result in charges to your AWS account. For more details, see [AWS Pricing](https://aws.amazon.com/pricing/) and [Free Tier](https://aws.amazon.com/free/).
24+
- Running the tests might result in charges to your AWS account.
25+
- We recommend that you grant your code least privilege. At most, grant only the minimum permissions required to perform the task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
26+
- This code is not tested in every AWS Region. For more information, see [AWS Regional Services](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services).
27+
28+
## Code examples
29+
30+
### Prerequisites
31+
32+
To run these examples, you need:
33+
34+
- Python 3.x installed.
35+
- Run `python pip install -r requirements.txt`
36+
- AWS credentials configured. For more information, see [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).
37+
38+
#### Running the workflow
39+
40+
To run this workflow, pull AWS tokens and run the command below:
41+
42+
```bash
43+
python s3_batch_scenario.py
44+
```
45+
46+
## Additional resources
47+
48+
- [Amazon S3 Developer Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html)
49+
- [Amazon S3 API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)
50+
- [boto3 Amazon S3 reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html)
51+
52+
---
53+
54+
© Amazon.com, Inc. or its affiliates. All Rights Reserved.
55+
56+
SPDX-License-Identifier: Apache-2.0

0 commit comments

Comments
 (0)