-
Notifications
You must be signed in to change notification settings - Fork 274
RC RDI: RDI Cloud quick start #2326
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| --- | ||
| Title: RDI on Redis Cloud quick start | ||
| linkTitle: Quick start | ||
| alwaysopen: false | ||
| categories: | ||
| - docs | ||
| - operate | ||
| - rc | ||
| description: Learn how to create a data pipeline between a PostgreSQL source database created with Terraform and a Redis Cloud target database. | ||
| hideListLinks: true | ||
| weight: 1 | ||
| --- | ||
|
|
||
| The [`rdi-cloud-automation` GitHub repository](https://github.com/redis-field-engineering/rdi-cloud-automation) contains a Terraform script that quickly sets up a PostgreSQL source database on an EC2 instance and all required permissions and network setup to connect it to a Redis Cloud target database. | ||
|
|
||
| {{< note >}} | ||
| This guide is for demonstration purposes only. It is not recommended for production use. | ||
| {{< /note >}} | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| To follow this guide, you need to: | ||
|
|
||
| 1. Create a [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS). | ||
|
|
||
| Turn on Multi-AZ replication and [manually select the availability zones]({{< relref "/operate/rc/databases/configuration/high-availability#availability-zones" >}}) when creating the database. | ||
|
|
||
| 1. Install the [AWS CLI](https://aws.amazon.com/cli/) and set up [credentials for the CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html). | ||
|
|
||
| 1. Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli). | ||
|
|
||
| ## Create a data pipeline | ||
|
|
||
| 1. On the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. | ||
| 1. Select **Create pipeline**. | ||
| {{<image filename="images/rc/rdi/rdi-create-data-pipeline.png" alt="The create pipeline button." width=200px >}} | ||
| 1. Select **PostgreSQL** as the source database type. | ||
| {{<image filename="images/rc/rdi/rdi-select-source-db.png" alt="The select source database type list." width=500px >}} | ||
| 1. Under **Setup connectivity**, save the provided ARN and extract the AWS account ID for the account associated with your Redis Cloud cluster from it. | ||
|
|
||
| {{<image filename="images/rc/rdi/rdi-setup-connectivity-arn.png" alt="The select source database type list." width=80% >}} | ||
|
|
||
| The AWS account ID is the string of numbers after `arn:aws:iam::` in the ARN. For example, if the ARN is `arn:aws:iam::123456789012:role/redis-data-pipeline`, the AWS account ID is `123456789012`. | ||
|
|
||
| ## Create the source database and network resources | ||
|
|
||
| 1. Clone or download the [`rdi-cloud-automation` GitHub repository](https://github.com/redis-field-engineering/rdi-cloud-automation). | ||
cmilesb marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| 1. In a terminal window, go to the `examples` directory. | ||
|
|
||
| 1. Run `terraform init` to initialize the Terraform working directory. | ||
|
|
||
| 1. Open the `example.tfvars` file and edit the following variables: | ||
|
|
||
| - `region`: The AWS region where your Redis Cloud database is deployed. | ||
| - `azs`: The availability zone IDs where your Redis Cloud database is deployed. | ||
| - `port`: The port number for the new PostgreSQL source database. | ||
| - `name`: A prefix for all of the created AWS resources | ||
cmilesb marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - `redis-account`: The AWS account ID for your Redis Cloud cluster you saved earlier. | ||
|
|
||
| 1. To view the configuration, run: | ||
|
|
||
| ```sh | ||
| terraform plan -var-file=example.tfvars | ||
| ``` | ||
|
|
||
| 1. To create the AWS resources, run: | ||
|
|
||
| ```sh | ||
| terraform apply -var-file=example.tfvars | ||
| ``` | ||
|
|
||
| This example creates the following resources on your AWS account: | ||
|
|
||
| - An AWS KMS key with the required permissions for RDI | ||
| - A VPC with a public and private subnet and all necessary route tables | ||
| - An EC2 instance running a PostgreSQL database with a security group that allows access from Redis Cloud | ||
| - An AWS Secrets Manager secret for the PostgreSQL database credentials | ||
| - A Network Load Balancer (NLB), a listener, and target group to route traffic to the EC2 instance with AWS PrivateLink | ||
| - An AWS PrivateLink endpoint service for the PostgreSQL database | ||
|
|
||
| Creating the AWS resources will take some time. After the resources are created, you'll be able to view them in the AWS management console. | ||
|
|
||
| Save the following outputs: | ||
|
|
||
| - `database`: The name of the PostgreSQL database. | ||
| - `port`: The port number for the PostgreSQL database. | ||
| - `secret_arn`: The ARN of the AWS Secrets Manager secret for the PostgreSQL database credentials. | ||
| - `vpc_endpoint_service_name`: The name of the AWS PrivateLink endpoint service for the PostgreSQL database. | ||
|
|
||
| If you lose any outputs, run `terraform output` to view them again. | ||
|
|
||
| ## Define source connection | ||
|
|
||
| 1. Return to the [Redis Cloud console](https://cloud.redis.io/). Go to your target database and select the **Data Pipeline** tab. | ||
| 1. Select **Define source database**. | ||
| {{<image filename="images/rc/rdi/rdi-define-source-database.png" alt="The define source database button." width=200px >}} | ||
| 1. Enter a **Pipeline name**. | ||
| {{<image filename="images/rc/rdi/rdi-define-pipeline-cidr.png" alt="The pipeline name and deployment CIDR fields." >}} | ||
| 1. A **Deployment CIDR** is automatically generated for you. If, for any reason, a CIDR is not generated, enter a valid CIDR that does not conflict with your applications or other databases. | ||
| 1. Enter the terraform outputs in the following fields: | ||
| - **PrivateLink service name**: `vpc_endpoint_service_name` | ||
| - **Database**: `database` | ||
| - **Port**: `port` | ||
| - **Source database secrets ARN**: `secret_arn` | ||
| 1. Select **Start pipeline setup**. | ||
| {{<image filename="images/rc/rdi/rdi-start-pipeline-setup.png" alt="The start pipeline setup button." width=200px >}} | ||
|
|
||
| At this point, Redis Cloud will provision the pipeline infrastructure that will allow you to define your data pipeline. | ||
|
|
||
| {{<image filename="images/rc/rdi/rdi-pipeline-setup-in-progress.png" alt="The Pipeline setup in progress screen." width=75% >}} | ||
|
|
||
| Pipelines are provisioned in the background. You aren't allowed to make changes to your data pipeline or to your database during provisioning. This process will take about an hour, so you can close the window and come back later. | ||
|
|
||
| When your pipeline is provisioned, select **Complete setup**. | ||
|
|
||
| {{<image filename="images/rc/rdi/rdi-complete-setup.png" alt="The complete setup button." width=200px >}} | ||
|
|
||
| ## Define data pipeline | ||
|
|
||
| After your pipeline is provisioned, you will be able to define your pipeline. You will select the database schemas, tables, and columns that you want to import and synchronize with your primary database. | ||
|
|
||
| See [Define data pipeline]({{<relref "/operate/rc/databases/rdi/define#define-data-pipeline">}}) for detailed steps on defining your data pipeline. | ||
|
|
||
| After you define your data pipeline, it will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. | ||
|
|
||
| After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. You can see this by connecting to your source database and making changes to the data, and then connecting to your target Redis database and verifying that the changes are reflected there. | ||
|
|
||
| You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{<relref "/operate/rc/databases/rdi/view-edit">}}) to learn more. | ||
|
|
||
| ## Delete sample resources | ||
|
|
||
| {{< warning >}} | ||
| Make sure to [delete your data pipeline]({{<relref "/operate/rc/databases/rdi/view-edit#delete-pipeline">}}) before deleting the sample resources. | ||
| {{< /warning >}} | ||
|
|
||
| To delete the sample resources created by Terraform, run: | ||
|
|
||
| ```sh | ||
| terraform destroy -var-file=example.tfvars | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.