Skip to content

Commit 4f8e156

Browse files
jimrightjimright
authored
Add module for CML on AWS permissions (#94)
Signed-off-by: Jim Enright <[email protected]>
1 parent 2629166 commit 4f8e156

File tree

19 files changed

+482
-11
lines changed

19 files changed

+482
-11
lines changed

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ This repository contains a number of Terraform modules for creation of the pre-r
1616
| [terraform-aws-fw-vpc](modules/terraform-aws-fw-vpc/README.md) | Module for creation of the VPC networking resources on AWS suitable for running a Firewall in a distributed architecture on AWS. Can be used to create a networking VPC which runs the AWS Network Firewall and connects to a Cloudera on cloud full-private deployment. |
1717
| [terraform-aws-tgw](modules/terraform-aws-tgw/README.md) | Module for creation of AWS Transity Gateway (TGW) and attaching a specified list of VPCs via the TGW. This module can be used to assist in deploying Cloudera Data Platform (CDP) Public Cloud in a fully private networking configuration where a CDP VPC and Networking VPC are connected using the Transit Gateway. |
1818
| [terraform-aws-bastion](modules/terraform-aws-bastion/README.md) | Module to create a Bastion EC2 instance on AWS. This module can be used to assist in deploying Cloudera Data Platform (CDP) Public Cloud in a secure environment, where the CDP Environment requires a Bastion host. |
19-
| [terraform-aws-proxy](modules/terraform-aws-proxy/README.md) | Module to create and configure and EC2 Auto-Scaling Group for a highly available Squid Proxy service with Network Load Balancer (NLB) to forward traffic to the proxy instances. This module can be used to assist in deploying Cloudera Data Platform (CDP) Public Cloud in a fully private networking configuration where a the CDP Environments uses a proxy config via the NLB. |
19+
| [terraform-aws-cml-permissions](modules/terraform-aws-cml-permissions/README.md) | Module to create AWS IAM policies required for the Cloudera Machine Learning (CML) backup and restore. |
20+
| [terraform-aws-permissions](modules/terraform-aws-permissions/README.md) | Module for creation of the AWS IAM permissions required by the (CDP) Public Cloud environment and datalake deployment. Note that this module is called from the terraform-cdp-aws-prereqs module. |
2021
| [terraform-aws-firewall](modules/terraform-aws-firewall/README.md) | Module to create and configure to create and configure an AWS Network Firewall. This module can be used to assist in deploying Cloudera Data Platform (CDP) Public Cloud in a fully private networking configuration where the CDP Environment is connected to a Networking VPC running the Firewall. |
2122
| [terraform-azure-nfs](modules/terraform-azure-nfs/README.md) | Module for creation of Azure NFS File Share required for Cloudera Machine Learning (CML) Public Cloud. Also optionally creates a Virtual Machine which can be used to mount and set the required ownership for CML workspace's projects folder.|
2223
| [terraform-azure-cdw-permissions](modules/terraform-azure-cdw-permissions/README.md) | Module for creation of the Azure Kubernetes Service (AKS) managed identity required for the Cloudera Data Warehouse (CDW) service.|

‎modules/terraform-aws-cml-permissions/README.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
<!-- BEGIN_TF_DOCS -->
2+
# Terraform Module for AWS IAM Permissions for CML
3+
4+
This module contains resource files and example variable definition files for creation of the AWS IAM policies required for the Cloudera Machine Learning (CML) backup and restore functionality. This requirement is described [in this section](https://docs.cloudera.com/machine-learning/cloud/workspaces/topics/ml-backup-restore-prerequisites.html) of the CML documentation.
5+
6+
## Usage
7+
8+
The [examples](./examples) directory has an example AWS IAM policy creation on AWS:
9+
10+
* `ex01-minimal-inputs` uses the minimum set of inputs for the module.
11+
12+
An example `terraform.tfvars.sample` values file is included to show input variable values.
13+
14+
## Requirements
15+
16+
| Name | Version |
17+
|------|---------|
18+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
19+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | ~>5.30 |
20+
21+
## Providers
22+
23+
| Name | Version |
24+
|------|---------|
25+
| <a name="provider_aws"></a> [aws](#provider\_aws) | ~>5.30 |
26+
| <a name="provider_http"></a> [http](#provider\_http) | n/a |
27+
28+
## Modules
29+
30+
No modules.
31+
32+
## Resources
33+
34+
| Name | Type |
35+
|------|------|
36+
| [aws_iam_policy.cml_backup_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
37+
| [aws_iam_policy.cml_restore_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
38+
| [aws_iam_role_policy_attachment.cdp_xaccount_role_cml_backup_attach](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
39+
| [aws_iam_role_policy_attachment.cdp_xaccount_role_cml_restore_attach](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
40+
| [aws_iam_role.xaccount_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_role) | data source |
41+
| [http_http.cml_backup_policy_doc](https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http) | data source |
42+
| [http_http.cml_restore_policy_doc](https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http) | data source |
43+
44+
## Inputs
45+
46+
| Name | Description | Type | Default | Required |
47+
|------|-------------|------|---------|:--------:|
48+
| <a name="input_cml_backup_policy_name"></a> [cml\_backup\_policy\_name](#input\_cml\_backup\_policy\_name) | CDP CML Backup Policy name | `string` | n/a | yes |
49+
| <a name="input_cml_restore_policy_name"></a> [cml\_restore\_policy\_name](#input\_cml\_restore\_policy\_name) | CDP CML Restore Policy name | `string` | n/a | yes |
50+
| <a name="input_xaccount_role_name"></a> [xaccount\_role\_name](#input\_xaccount\_role\_name) | Name of existing cross account Assume role Name. | `string` | n/a | yes |
51+
| <a name="input_cml_backup_policy_doc"></a> [cml\_backup\_policy\_doc](#input\_cml\_backup\_policy\_doc) | Contents of CDP CML Backup Policy Document. If not specified document is downloaded from Cloudera Document repository | `string` | `null` | no |
52+
| <a name="input_cml_restore_policy_doc"></a> [cml\_restore\_policy\_doc](#input\_cml\_restore\_policy\_doc) | Contents of CDP CML Restore Policy Document. If not specified document is downloaded from Cloudera Document repository | `string` | `null` | no |
53+
| <a name="input_tags"></a> [tags](#input\_tags) | Tags applied to provised resources | `map(any)` | `null` | no |
54+
55+
## Outputs
56+
57+
| Name | Description |
58+
|------|-------------|
59+
| <a name="output_aws_cml_backup_policy_arn"></a> [aws\_cml\_backup\_policy\_arn](#output\_aws\_cml\_backup\_policy\_arn) | CML Backup IAM Policy ARN |
60+
| <a name="output_aws_cml_restore_policy_arn"></a> [aws\_cml\_restore\_policy\_arn](#output\_aws\_cml\_restore\_policy\_arn) | CML Restore IAM Policy ARN |
61+
<!-- END_TF_DOCS -->

‎modules/terraform-aws-cml-permissions/data.tf‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Copyright 2025 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
data "aws_iam_role" "xaccount_role" {
16+
17+
name = var.xaccount_role_name
18+
}
19+
20+
# HTTP get request to download policy documents
21+
# ..CML Backup Policy
22+
data "http" "cml_backup_policy_doc" {
23+
url = "https://docs.cloudera.com/machine-learning/cloud/cml-backup-policy.json"
24+
}
25+
26+
# ..CML Restore Policy
27+
data "http" "cml_restore_policy_doc" {
28+
url = "https://docs.cloudera.com/machine-learning/cloud/cml-restore-policy.json"
29+
}

‎modules/terraform-aws-cml-permissions/defaults.tf‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Copyright 2025 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
locals {
16+
17+
# CML Backup Policy document
18+
cml_backup_policy_doc = coalesce(var.cml_backup_policy_doc, data.http.cml_backup_policy_doc.response_body)
19+
20+
# CML Backup Policy document
21+
cml_restore_policy_doc = coalesce(var.cml_restore_policy_doc, data.http.cml_restore_policy_doc.response_body)
22+
23+
}

‎modules/terraform-aws-cml-permissions/doc_fragments/header.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Terraform Module for AWS IAM Permissions for CML
2+
3+
This module contains resource files and example variable definition files for creation of the AWS IAM policies required for the Cloudera Machine Learning (CML) backup and restore functionality. This requirement is described [in this section](https://docs.cloudera.com/machine-learning/cloud/workspaces/topics/ml-backup-restore-prerequisites.html) of the CML documentation.
4+
5+
## Usage
6+
7+
The [examples](./examples) directory has an example AWS IAM policy creation on AWS:
8+
9+
* `ex01-minimal-inputs` uses the minimum set of inputs for the module.
10+
11+
An example `terraform.tfvars.sample` values file is included to show input variable values.

‎modules/terraform-aws-cml-permissions/examples/ex01-minimal-inputs/main.tf‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Copyright 2025 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
terraform {
16+
required_version = ">= 1.5.7"
17+
required_providers {
18+
cdp = {
19+
source = "cloudera/cdp"
20+
version = ">= 0.6.1"
21+
}
22+
aws = {
23+
source = "hashicorp/aws"
24+
version = "~>5.30"
25+
}
26+
}
27+
}
28+
29+
provider "aws" {
30+
region = var.aws_region
31+
}
32+
33+
module "xaccount_iam_role" {
34+
source = "../../../terraform-aws-cred-permissions"
35+
36+
tags = var.tags
37+
38+
# Using CDP TF Provider cred pre-reqs data source for values of xaccount account_id and external_id
39+
xaccount_account_id = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.account_id
40+
xaccount_external_id = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.external_id
41+
42+
xaccount_policy_name = "${var.env_prefix}-xaccount-policy"
43+
xaccount_account_policy_doc = base64decode(data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.policy)
44+
45+
xaccount_role_name = "${var.env_prefix}-xaccount-role"
46+
47+
# Create assume role trust relationship required for CML backup and restore
48+
create_cml_assume_role_policy = true
49+
}
50+
51+
module "ex01_minimal_inputs" {
52+
source = "../.."
53+
54+
tags = var.tags
55+
56+
cml_backup_policy_name = "${var.env_prefix}-cml-backup-policy"
57+
cml_restore_policy_name = "${var.env_prefix}-cml-restore-policy"
58+
59+
xaccount_role_name = module.xaccount_iam_role.aws_xaccount_role_name
60+
61+
depends_on = [module.xaccount_iam_role]
62+
}
63+
64+
# Use the CDP Terraform Provider to find the xaccount account and external ids
65+
data "cdp_environments_aws_credential_prerequisites" "cdp_prereqs" {}
66+
67+
# ------- Outputs -------
68+
output "xaccount_role_arn" {
69+
value = module.xaccount_iam_role.aws_xaccount_role_arn
70+
71+
description = "The ARN of the created Cross Account Role"
72+
}

‎modules/terraform-aws-cml-permissions/examples/ex01-minimal-inputs/terraform.tfvars.sample‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Copyright 2025 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# ------- Global settings -------
16+
env_prefix = "<ENTER_VALUE>" # Required name prefix for cloud and CDP resources, e.g. cldr1
17+
18+
# ------- Cloud Settings -------
19+
aws_region = "<ENTER_VALUE>" # Change this to specify Cloud Provider region, e.g. eu-west-1
20+
21+
# ------- Resource Tagging -------
22+
# **NOTE: An example of how to specify tags is below; uncomment & edit if required
23+
tags = {
24+
owner = "<ENTER_VALUE>"
25+
project = "<ENTER_VALUE>"
26+
enddate = "<ENTER_VALUE>"
27+
}

‎modules/terraform-aws-cml-permissions/examples/ex01-minimal-inputs/variables.tf‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Copyright 2025 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# ------- Global settings -------
16+
variable "aws_region" {
17+
type = string
18+
description = "Region which Cloud resources will be created"
19+
}
20+
21+
variable "env_prefix" {
22+
type = string
23+
description = "Shorthand name for the environment. Used in resource descriptions"
24+
}
25+
26+
variable "tags" {
27+
type = map(any)
28+
description = "Tags applied to provised resources"
29+
30+
default = null
31+
}

‎modules/terraform-aws-cml-permissions/main.tf‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Copyright 2025 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# ------- Cross Account Policy -------
16+
resource "aws_iam_policy" "cml_backup_policy" {
17+
name = var.cml_backup_policy_name
18+
description = "CDP CML Workspace Backup policy"
19+
20+
tags = merge(var.tags, { Name = var.cml_backup_policy_name })
21+
22+
policy = local.cml_backup_policy_doc
23+
}
24+
25+
resource "aws_iam_policy" "cml_restore_policy" {
26+
name = var.cml_restore_policy_name
27+
description = "CDP CML Workspace Restore policy"
28+
29+
tags = merge(var.tags, { Name = var.cml_restore_policy_name })
30+
31+
policy = local.cml_restore_policy_doc
32+
}
33+
34+
# Attach CML backup policy to the xaccount role
35+
resource "aws_iam_role_policy_attachment" "cdp_xaccount_role_cml_backup_attach" {
36+
role = data.aws_iam_role.xaccount_role.name
37+
policy_arn = aws_iam_policy.cml_backup_policy.arn
38+
}
39+
40+
# Attach CML restore policy to the xaccount role
41+
resource "aws_iam_role_policy_attachment" "cdp_xaccount_role_cml_restore_attach" {
42+
role = data.aws_iam_role.xaccount_role.name
43+
policy_arn = aws_iam_policy.cml_restore_policy.arn
44+
}

‎modules/terraform-aws-cml-permissions/outputs.tf‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright 2025 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
output "aws_cml_backup_policy_arn" {
16+
value = aws_iam_policy.cml_backup_policy.arn
17+
18+
description = "CML Backup IAM Policy ARN"
19+
}
20+
21+
output "aws_cml_restore_policy_arn" {
22+
value = aws_iam_policy.cml_restore_policy.arn
23+
24+
description = "CML Restore IAM Policy ARN"
25+
}

0 commit comments

Comments
 (0)