Infrastructure Cloud Blueprint for CI/CD

Description of intent

The infraestructure created will use Tofu version 1.9.0 to create a deployment system utilizing the following AWS services:

• S3 and ECR to store artifacts
• CodePipeline to orchestrate CodeBuild and CodeDeploy execution.
• An ECS cluster with a Service running under Fargate

It will have an orchestrator called "Agent" written in Python that will:

• Upload to S3 a Componenet Manifest in YAML format that will describe the artifact to deploy to ECS
  • This will trigger a CodePiline run with the file as input
  • CodePipeline will trigger the followin steps:
    • A Build stage where the Componenet manifest is read and the configuration to deploy to ECS is created.
  • A "User acceptance" step where the user has to accept the ployment specifics
  • A Deployment stage where the script is run and the applycation is deployed to ECS

This project provides a complete blueprint for implementing a CI/CD pipeline to deploy containerized applications to AWS ECS using a blue/green deployment strategy.

Architecture Overview

The following diagram illustrates how all components of the CI/CD pipeline work together:

graph TB
    subgraph "CO/RO Workflow"
        CDRO[CDRO] -->|Creates| Manifest[Component Manifest]
        CDRO -->|Runs| Agent[Deployment Agent]
        Agent -->|Uploads| S3Upload[Upload to S3]
    end

    subgraph "AWS CI/CD Pipeline"
        S3Upload -->|Triggers| CP[AWS CodePipeline]
        
        subgraph "Source Stage"
            CP -->|Source Stage| SrcStage[S3 Source]
            SrcStage -->|Download| SrcArtifact[Source Artifact]
        end
        
        subgraph "Build Stage"
            SrcArtifact -->|Input to| BuildStage[CodeBuild]
            BuildStage -->|Process Manifest| GenFiles[Generate Deployment Files]
            GenFiles -->|Creates| AppSpec[appspec.yaml]
            GenFiles -->|Creates| TaskDef[taskdef.json]
            BuildStage -->|Output| BuildArtifact[Build Artifact]
        end
        
        subgraph "Deploy Stage"
            BuildArtifact -->|Input to| DeployStage[CodeDeploy]
            DeployStage -->|Blue/Green| ECSService[ECS Service]
        end
    end
    
    subgraph "Runtime Environment"
        ECSService -->|Updates| TaskDef[ECS Task Definition]
        TaskDef -->|Container Config| Container[ECS Container]
        ALB[Application Load Balancer] -->|Routes Traffic| TargetGroups[Blue/Green Target Groups]
        TargetGroups -->|Points to| ECSService
    end

Loading

Project Components

1. Agent

The deployment agent (agent/agent.py) is a Python utility that:

• Takes a component manifest as input
• Uploads it to an S3 bucket
• Triggers CodePipeline execution
• Monitors and reports on the deployment status

Usage:

python agent/agent.py [s3_bucket] [codepipeline_name] [manifest_file_path]

2. Manifest Templates

Located in agent/templates/, these define the structure of:

• component_manifest_template.yaml: The main deployment configuration file
• appspec.template.yaml: Template for CodeDeploy AppSpec
• taskdef.template.json: Template for ECS Task Definition

3. Scripts

The scripts/ directory contains utilities to support the CI/CD process:

• generate_deployment_files.py: Transforms component manifests into AWS deployment artifacts

4. Terraform Infrastructure

The terraform/ directory contains Infrastructure as Code (IaC) definitions for all AWS resources required:

• AWS CodePipeline, CodeBuild, and CodeDeploy
• Amazon ECS Cluster, Service, and Task Definition
• Application Load Balancer and Target Groups
• All required IAM Roles and Security Groups

Workflow Explanation

1. CDRO Preparation

   • Generate a component manifest (YAML file) defining the application to deploy
   • Run the deployment agent to initiate the deployment process

2. CI/CD Pipeline Execution

   • Source Stage: CodePipeline retrieves source code and manifest from S3
   • Build Stage: CodeBuild processes the manifest and generates deployment files:
     • appspec.yaml: Instructions for CodeDeploy
     • taskdef.json: ECS task definition template
   • Deploy Stage: CodeDeploy performs blue/green deployment to ECS

3. Blue/Green Deployment Process

   • New task definition is registered in ECS
   • New tasks (green environment) are started
   • Green environment is tested
   • Traffic is shifted from blue to green environment
   • Old tasks (blue environment) are terminated

Getting Started

Prerequisites

• AWS Account
• Terraform installed locally
• Python 3.6+
• AWS CLI configured

Setup Instructions

1. Deploy AWS Infrastructure

cd terraform
terraform init
terraform apply -var-file=environments/dev/terraform.tfvars

2. Create a Component Manifest

Create a component_manifest.yaml based on the template:

version: '1.0'
description: 'My Application'
artifact:
  name: 'my-app'
  version: '1.0.0'
  type: 'docker'
  repository: '123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app'
  environment:
    variables:
      NODE_ENV: 'production'
  deployment:
    service: 'my-app-service'
    cluster: 'my-cluster'
    task_definition: 'my-app-task-def'
    desired_count: 2

3. Deploy Using the Agent

# Install requirements
pip install -r agent/requirements.txt

# Run deployment agent
python agent/agent.py my-artifact-bucket my-pipeline ./component_manifest.yaml

Key Features

• Component-Based Deployments: Flexible component manifest for application configuration
• Blue/Green Deployments: Zero-downtime deployments with automatic rollback capability
• Infrastructure as Code: Complete AWS infrastructure defined with Terraform
• Monitoring and Logs: Integrated CloudWatch logging and monitoring

Directory Structure

infra-cloud-bp/
├── agent/
│   ├── agent.py                     # Deployment agent
│   └── templates/
│       ├── component_manifest_template.yaml
│       ├── appspec.template.yaml
│       └── taskdef.template.json
├── buildspecs/
│   └── build_stage.yml             # CodeBuild specification
├── scripts/
│   ├── generate_deployment_files.py # Deployment files generator
│   ├── appspec.template.yaml
│   └── README.md
└── terraform/
    ├── main.tf                     # Main Terraform configuration
    ├── variables.tf                # Variable definitions
    ├── outputs.tf                  # Output values
    ├── README.md                   # Terraform documentation
    └── environments/
        ├── dev/                    # Development configuration
        │   └── terraform.tfvars
        └── prod/                   # Production configuration
            └── terraform.tfvars

Please read the individual README files in each directory for more detailed information about specific components.