Airflow 3.1.0 Docker Compose Setup

A complete Docker Compose setup for Apache Airflow 3.1.0 optimized for Windows 11 development.

Project Structure

./
├── dags/                   # Place your DAG files here
├── logs/                   # Airflow logs (auto-generated)
├── plugins/                # Custom Airflow plugins
├── config/                 # Airflow configuration files
├── docker-compose.yml      # Main Docker Compose configuration
├── .env                    # Environment variables (customize this)
├── .env.example           # Environment template
└── README.md              # This file

Quick Start

1. Prerequisites

   • Docker Desktop for Windows 11
   • At least 4GB RAM allocated to Docker
   • WSL2 backend enabled (recommended)

2. Setup

   # Copy environment template
   cp .env.example .env
   
   # Edit .env file with your settings
   # Generate a new Fernet key:
   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

3. Start Airflow

   # Initialize database (first time only)
   docker-compose up airflow-init
   
   # Start all services
   docker-compose up -d

4. Access Airflow

   • Web UI: http://localhost:8080
   • Get admin password: scripts/get-admin-password.bat (Windows) or scripts/get-admin-password.sh (Linux/Mac)
   • Username: admin
   • Password: Auto-generated by Simple Auth Manager (changes on restart)

Services

Core Services (Always Running)

• airflow-api-server: Web UI and REST API (port 8080)
• airflow-scheduler: Task scheduling and monitoring
• airflow-dag-processor: DAG file parsing
• airflow-triggerer: Deferred and async task handling
• postgres: Database backend (PostgreSQL 17)

CeleryExecutor Services (Profile: celery)

• redis: Message broker for task distribution
• airflow-worker: Celery worker processes (scalable via NUMBER_OF_WORKERS)
• airflow-scheduler-celery: Scheduler with Celery configuration
• airflow-api-server-celery: API server with Celery configuration

Monitoring Services (Profile: flower)

• flower: Celery worker monitoring dashboard (port 5555)

Configuration

Environment Variables (.env)

Key configuration options in your .env file:

# Executor Configuration
AIRFLOW_EXECUTOR=LocalExecutor          # or CeleryExecutor
COMPOSE_PROFILES=celery,flower          # Enable Celery + Flower

# Worker Scaling (CeleryExecutor only)
NUMBER_OF_WORKERS=2                     # Number of worker replicas

# Security Keys (Generate new ones for production)
AIRFLOW_FERNET_KEY=your-fernet-key
AIRFLOW_SECRET_KEY=your-secret-key

# Service Ports
AIRFLOW_WEBSERVER_PORT=8080
FLOWER_PORT=5555

# Database & Redis
POSTGRES_PASSWORD=airflow
REDIS_PASSWORD=airflow

Docker Compose Profiles

Control which services start using profiles:

• Default: Core Airflow services with LocalExecutor
• celery: Adds Redis, Celery workers, and Celery-configured services
• flower: Adds Flower monitoring dashboard

Examples:

# LocalExecutor only
docker-compose up -d

# CeleryExecutor with monitoring
COMPOSE_PROFILES=celery,flower docker-compose up -d

# Or set in .env file
echo "COMPOSE_PROFILES=celery,flower" >> .env
docker-compose up -d

Executors

LocalExecutor (Default)

• Single-node execution
• Good for development and small workloads
• No additional services required

CeleryExecutor (Distributed)

• Multi-worker distributed execution
• Scalable for production workloads
• Requires Redis message broker

To enable CeleryExecutor:

1. Set AIRFLOW_EXECUTOR=CeleryExecutor in .env
2. Set COMPOSE_PROFILES=celery,flower in .env
3. Restart services: docker-compose up -d

Worker Scaling:

• Configure NUMBER_OF_WORKERS=2 in .env to set worker count
• Workers will be identical with same configuration
• Monitor workers via Flower at http://localhost:5555

Authentication

Simple Auth Manager (Default)

• Auto-generated passwords for enhanced security
• JWT token support for API access
• Lightweight and fast
• Passwords stored in /opt/airflow/simple_auth_manager_passwords.json.generated

# Start with Simple Auth Manager (default)
docker-compose up -d

# Get current admin password
scripts/get-admin-password.bat  # Windows
scripts/get-admin-password.sh   # Linux/Mac

Configuration Migration from Airflow 2.x to 3.x

Key Changes

This setup uses the correct Airflow 3.1.0 configuration parameters. If you're migrating from Airflow 2.x, note these important changes:

1. Secret Key Configuration

Airflow 2.x (Deprecated):

AIRFLOW__WEBSERVER__SECRET_KEY=your-secret-key

Airflow 3.x (Current):

AIRFLOW__API__SECRET_KEY=your-secret-key
AIRFLOW__SIMPLE_AUTH_MANAGER__JWT_SECRET_KEY=your-secret-key

2. Service Architecture

Airflow 2.x:

• Single webserver component

Airflow 3.x:

• Separate API server (replaces webserver)
• Dedicated DAG processor service
• Enhanced triggerer for async operations

3. Database Commands

Airflow 2.x (Deprecated):

airflow db upgrade

Airflow 3.x (Current):

airflow db migrate

4. Authentication Manager

Airflow 2.x:

• Flask-AppBuilder based authentication

Airflow 3.x:

• SimpleAuthManager (default)
• FastAPI-based authentication
• Improved JWT token handling

Troubleshooting Configuration Issues

JWT Token Errors

If you see "JWT token is not valid: Signature verification failed" errors:

1. Check secret key consistency:

   # Verify all services use the same secret key
   docker-compose exec airflow-api-server-celery printenv | grep SECRET
   docker-compose exec airflow-worker printenv | grep SECRET

2. Regenerate secret keys:

   # Generate new secure secret key
   python -c "import secrets; print(secrets.token_urlsafe(32))"
   # Update AIRFLOW_SECRET_KEY in .env file

3. Restart services:

   docker-compose down
   docker-compose up -d

Deprecation Warnings

If you see deprecation warnings about configuration parameters:

• Update your configuration to use the Airflow 3.x parameter names shown above
• Remove any old Airflow 2.x configuration parameters
• Restart services after configuration changes

Scaling and Performance

Worker Scaling (CeleryExecutor)

Method 1: Environment Variable (Recommended)

# In .env file
NUMBER_OF_WORKERS=4

Method 2: Docker Compose Scale

docker-compose up --scale airflow-worker=3

Worker Configuration

Each worker is configured with:

• Concurrency: 16 parallel tasks per worker
• Autoscaling: 16 max, 4 min processes
• Memory Limit: 8GB per worker
• Task Limit: 1000 tasks before worker restart

Monitoring

• Flower Dashboard: http://localhost:5555 (CeleryExecutor only)
  • Real-time worker status
  • Task execution monitoring
  • Worker resource usage
• Airflow Web UI: http://localhost:8080
  • DAG execution status
  • Task logs and details
  • System health metrics

Troubleshooting

CeleryExecutor Issues

Workers not starting:

1. Check COMPOSE_PROFILES=celery,flower in .env
2. Verify AIRFLOW_EXECUTOR=CeleryExecutor in .env
3. Ensure Redis is healthy: docker-compose ps redis

Tasks stuck in queued state:

1. Check worker status in Flower: http://localhost:5555
2. Verify worker logs: docker-compose logs airflow-worker
3. Restart workers: docker-compose restart airflow-worker

Flower not accessible:

1. Ensure flower profile is enabled in COMPOSE_PROFILES
2. Check if port 5555 is available
3. Verify Flower service is running: docker-compose ps flower

Performance Tuning

Increase worker capacity:

# In .env
NUMBER_OF_WORKERS=4  # More workers

Adjust worker concurrency: Edit AIRFLOW__CELERY__WORKER_CONCURRENCY in docker-compose.yml

Windows 11 Notes

• Uses forward slashes for cross-platform volume compatibility
• Optimized for Docker Desktop with WSL2 backend
• Compatible with existing conda environments
• CeleryExecutor works seamlessly with WSL2 Docker backend

Documentation

Comprehensive Guides

• Complete Usage Guide - Comprehensive guide covering all aspects of using Airflow 3.1.0 with Docker Compose
• DAG Development Best Practices - Best practices for developing DAGs in Airflow 3.1.0
• Windows 11 Troubleshooting Guide - Solutions for common Windows 11 specific issues

Authentication Guides

• Simple Auth Manager Usage - Using the default Simple Auth Manager with JWT tokens

Next Steps

This setup is ready for development. Add your DAG files to the dags/ directory and they will be automatically detected by Airflow.

For detailed usage instructions, DAG development best practices, and troubleshooting help, see the comprehensive documentation guides above.