Service Deployment Manager

A comprehensive container deployment and management platform that streamlines Docker Compose application deployments with real-time monitoring, automated scaling, and multi-environment support.

🚀 Features

Core Deployment Capabilities

• Docker Compose Automation: Seamless deployment of containerized applications
• Multi-Environment Support: Deploy to local, remote VMs, or cloud infrastructure
• Azure Spot VM Integration: Cost-effective deployment using Azure Spot VMs
• Workspace Management: Isolated environments for different projects and users

Monitoring & Observability

• Real-time Log Streaming: Live log monitoring with service-specific filtering
• Performance Analytics: Container statistics and resource utilization tracking
• Error Detection: Automated error identification and reporting
• Health Monitoring: Continuous health checks for deployed services

Management Features

• Job Queue System: Asynchronous build and deployment operations
• Version Control: Track and manage deployed application versions
• Resource Cleanup: Automated cleanup of unused containers and images
• Load Balancing: Traefik integration for automatic service discovery

Security & Access Control

• Authentication Middleware: Secure API access with token-based authentication
• User Isolation: Per-user workspace and resource isolation
• Secure VM Access: SSH key management for remote deployments

🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Frontend      │    │   API Gateway   │    │   Core Engine   │
│   Dashboard     │◄──►│   FastAPI       │◄──►│   Docker Utils  │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                │                        │
                                ▼                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Database      │    │   Log Parser    │    │   VM Manager    │
│   MongoDB       │    │   Fluentd       │    │   Azure VMs     │
└─────────────────┘    └─────────────────┘    └─────────────────┘

📋 Prerequisites

• Docker & Docker Compose: Version 20.10+
• Python: 3.9+
• MongoDB: 4.4+
• Azure CLI: (for Azure VM deployments)
• SSH Access: For remote VM deployments

🚀 Quick Start

1. Clone and Setup

git clone <repository-url>
cd ServiceDeploymentManager
cp .env.example .env
# Edit .env with your configuration

2. Configure Environment

Copy the example environment file and configure your settings:

cp .env.example .env

Edit the .env file with your configuration. See the Environment Configuration section below for detailed descriptions of all variables.

3. Deploy with Docker Compose

# Development
docker-compose up -d

# Production
docker-compose -f docker-compose-prod.yml up -d

# With Traefik (recommended for production)
./deploy.sh traefik=true

4. Access the Platform

• Dashboard: http://localhost:8005
• API Documentation: http://localhost:8005/docs
• Health Check: http://localhost:8005/health

📚 API Documentation

Workspace Management

# Create workspace
POST /api/workspaces
{
  "username": "user@example.com",
  "workspace_name": "my-app",
  "workspace_path": "/workspaces/user/my-app"
}

# List workspaces
GET /api/workspaces/{username}

# Upload project
POST /api/workspaces/upload/{username}/{workspace_name}

Deployment Operations

# Build and deploy
POST /api/docker/build_deploy/{username}/{workspace_name}

# Monitor deployment
GET /api/jobs/{job_id}

# View logs
GET /api/logs/{username}/{workspace_name}

# Bring down deployment
POST /api/docker/down/{username}/{workspace_name}

VM Management

# Get VM stats
GET /api/vm/{username}/{workspace_name}/stats

# Allocate VM
POST /api/vm/{username}/{workspace_name}/allocate

# VM health check
GET /api/vm/{username}/{workspace_name}/health

🏗️ Project Structure

├── app/                          # Main application code
│   ├── routes/                   # API endpoints
│   │   ├── docker.py            # Deployment operations
│   │   ├── workspaces.py        # Workspace management
│   │   ├── logs.py              # Log streaming
│   │   └── vm.py                # VM operations
│   ├── controllers/             # Business logic
│   ├── repositories/            # Data access layer
│   ├── models/                  # Data models
│   ├── docker/                  # Docker utilities
│   │   ├── docker_compose_utils.py
│   │   ├── docker_compose_remote_vm_utils.py
│   │   ├── docker_context_manager.py
│   │   ├── services_ports_identifier.py
│   │   ├── traefik_labeler.py
│   │   └── helper_functions.py
│   ├── vm_manager/              # Azure VM management
│   ├── workspace_monitoring/    # Log monitoring
│   ├── log_parser/              # Log parsing utilities
│   └── templates/               # Web interface
├── tests/                       # Test suites
├── logs/                        # Application logs
├── fluentd/                     # Log aggregation config
├── denv/                        # Python virtual environment
└── docker-compose*.yml         # Deployment configurations

🔧 Environment Configuration

The Service Deployment Manager uses environment variables for configuration. Copy .env.example to .env and configure the following variables:

Required Variables

Database Configuration

# MongoDB connection string - Required for data persistence
MONGODB_URL=mongodb+srv://username:password@cluster.mongodb.net/database
# Alternative for local MongoDB:
# MONGODB_URL=mongodb://localhost:27017

# Database name (optional, defaults to "deployment_manager")
DATABASE=deployment_manager

Authentication

# API authentication token - Required for secure access
AUTH_TOKEN=your-secure-token-here

# App builder token for workspace management
APP_BUILDER_TOKEN=your-app-builder-token

Docker & Deployment Configuration

Volume Mapping

# Maps local directory to container path for workspace storage
# Format: /local/path:/container/path
BASE_VOLUME_DIR_MAP=/Users/username/deployments:/app/docker/deployments

Environment Detection

# Flask environment (development/production)
FLASK_ENV=development

# Local deployment flag
LOCAL_DEPLOY=true

API Configuration

# Base URL for the API service
API_BASE_URL=https://your-domain.com
LOCAL_API_BASE_URL=http://localhost:8005

# Subdomain for service routing
SUBDOMAIN=your-subdomain.com

Logging Configuration

# Path where log watcher stores and monitors logs
LOG_WATCHER_PATH=/app/watcher

# Traefik configuration location
TRAFFIC_TOML_LOCATION=/app/traefik

Redis Configuration

# Redis host for caching and job queuing
REDIS_HOST=deploy-redis
# Redis port (optional, defaults to 6379)
REDIS_PORT=6379

Azure VM Management (Optional)

Configure these variables to enable Azure Spot VM deployments:

# Azure subscription and resource configuration
AZURE_SUBSCRIPTION_ID=your-subscription-id
AZURE_RESOURCE_GROUP=your-resource-group
AZURE_VNET_RESOURCE_GROUP=your-vnet-resource-group  # Optional, defaults to AZURE_RESOURCE_GROUP

# Network configuration
AZURE_VNET_NAME=your-vnet-name
AZURE_SUBNET_NAME=your-subnet-name
AZURE_LOCATION=East US

# VM configuration
AZURE_VM_ADMIN_USERNAME=azureuser
AZURE_SSH_PUBLIC_KEY=ssh-rsa AAAAB3NzaC1yc2E...
VM_SIZE=Standard_B2als_v2

Mock Mode Configuration

For testing and development without actual deployments:

# Enable mock mode to simulate deployments
ENABLE_MOCK_MODE=true

# Mock job response type: completed, failed, running
MOCK_JOB_RESPONSE_TYPE=completed

# Delay in seconds for mock job completion
MOCK_JOB_DELAY_SECONDS=10

WhatsApp Messaging (Optional)

For deployment notifications via WhatsApp:

# Twilio credentials for WhatsApp integration
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your-auth-token
TWILIO_WHATSAPP_FROM=whatsapp:+14155238886

AI Integration (Optional)

For intelligent error analysis and suggestions:

# OpenAI configuration
OPENAI_API_KEY=sk-...
OPENAI_API_ORG=org-...

# Azure OpenAI configuration
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZUREAI_ENDPOINT_KEY=your-key
AZURE_OPENAI_DALLE_ENDPOINT=https://your-dalle-resource.openai.azure.com/
AZURE_OPENAI_DALLE_KEY=your-dalle-key

# Azure AI configuration
AZURE_ENDPOINT=https://your-ai-resource.cognitiveservices.azure.com/

# Anthropic configuration
ANTHROPIC_API_KEY=sk-ant-...

Test Configuration

For running tests:

# Test user credentials
TEST_USERNAME=test_user
TEST_PASSWORD=test_password

How to Set Environment Variables

Method 1: Using .env file (Recommended)

# Copy the example file
cp .env.example .env

# Edit the file with your values
nano .env

Method 2: Docker Compose override

Create a docker-compose.override.yml file:

version: '3.8'
services:
  app:
    environment:
      - MONGODB_URL=your-mongodb-url
      - AUTH_TOKEN=your-token

Method 3: System environment variables

export MONGODB_URL="your-mongodb-url"
export AUTH_TOKEN="your-auth-token"

Method 4: Docker run command

docker run -e MONGODB_URL="your-url" -e AUTH_TOKEN="your-token" your-image

Environment Variable Priority

The application reads environment variables in this order:

1. System environment variables
2. .env file in the project root
3. Default values in the code

Security Best Practices

• Never commit .env files to version control
• Use strong, unique tokens for AUTH_TOKEN
• Rotate API keys regularly
• Use read-only database credentials when possible
• Enable Azure Key Vault for production secrets

Validation

To verify your environment configuration:

# Check if required variables are set
python -c "
import os
required = ['MONGODB_URL', 'AUTH_TOKEN']
missing = [var for var in required if not os.getenv(var)]
if missing:
    print(f'Missing required variables: {missing}')
else:
    print('All required variables are set')
"

Configuration Examples

Local Development

MONGODB_URL=mongodb://localhost:27017
AUTH_TOKEN=dev-token-123
LOCAL_DEPLOY=true
FLASK_ENV=development
BASE_VOLUME_DIR_MAP=/tmp/deployments:/app/docker/deployments
ENABLE_MOCK_MODE=true

Production with Azure

MONGODB_URL=mongodb+srv://user:pass@cluster.mongodb.net/prod
AUTH_TOKEN=secure-production-token
LOCAL_DEPLOY=false
FLASK_ENV=production
AZURE_SUBSCRIPTION_ID=your-sub-id
AZURE_RESOURCE_GROUP=prod-rg
BASE_VOLUME_DIR_MAP=/opt/deployments:/app/docker/deployments
ENABLE_MOCK_MODE=false

Environment Variables Reference

Variable	Required	Default	Description
MONGODB_URL	✅	-	MongoDB connection string
AUTH_TOKEN	✅	-	API authentication token
DATABASE	❌	deployment_manager	Database name
BASE_VOLUME_DIR_MAP	❌	-	Local to container volume mapping
FLASK_ENV	❌	development	Flask environment
LOCAL_DEPLOY	❌	true	Enable local deployment mode
LOG_WATCHER_PATH	❌	/app/watcher	Log monitoring path
REDIS_HOST	❌	redis	Redis server hostname
REDIS_PORT	❌	6379	Redis server port
ENABLE_MOCK_MODE	❌	false	Enable mock deployment mode
MOCK_JOB_RESPONSE_TYPE	❌	completed	Mock job response type
AZURE_SUBSCRIPTION_ID	❌	-	Azure subscription ID
AZURE_RESOURCE_GROUP	❌	-	Azure resource group
VM_SIZE	❌	Standard_B2ats_v2	Azure VM size
TWILIO_ACCOUNT_SID	❌	-	Twilio account SID
OPENAI_API_KEY	❌	-	OpenAI API key

Docker Context Setup

For remote VM deployments:

# Create Docker context for remote VM
docker context create myvm --docker "host=ssh://user@vm-ip"
docker context use myvm

📊 Monitoring & Logs

Real-time Log Streaming

# Stream logs for a workspace
GET /api/logs/{username}/{workspace_name}/tail?lines=100

# Filter by service
GET /api/logs/{username}/{workspace_name}?service=web&lines=50

# Time-based filtering
GET /api/logs/{username}/{workspace_name}?minutes=30

Performance Metrics

# Get deployment statistics
GET /api/stats/{username}/{workspace_name}

# VM health monitoring
GET /api/vm/{username}/{workspace_name}/health

# Docker stats
GET /api/docker/stats/{username}/{workspace_name}

🧪 Testing

Run Test Suite

# Install test dependencies
pip install -r requirements.txt

# Run comprehensive API tests
python tests/test_workspace_api_comprehensive.py
python tests/test_docker_compose_utils.py
python tests/test_logs_apis.py

# Run VM deployment tests
python tests/test_vm_build_and_deploy_remotely.py

# Test error identification
python tests/test_server_error_identifier.py

Test Configuration

Tests assume the service is running on localhost:8005. Update API_BASE_URL in test files if different.

🚀 Deployment Scenarios

Local Development

docker-compose up -d

Remote VM Deployment

# Configure VM in .env
AZURE_SUBSCRIPTION_ID=your-id
AZURE_RESOURCE_GROUP=your-rg

# Deploy with VM support
./deploy.sh traefik=true

Production with Load Balancing

# Uses Traefik for automatic service discovery
docker-compose -f docker-compose-traefik.yml up -d
docker-compose -f docker-compose-prod.yml up -d

🔧 Key Components

Docker Compose Remote VM Utils

• Build Operations: Automated Docker Compose builds on remote VMs
• Deployment Management: Deploy, monitor, and cleanup containerized applications
• Context Management: Seamless switching between local and remote Docker contexts
• Service Discovery: Automatic port identification and Traefik configuration

VM Manager

• Azure Integration: Spot VM allocation and management
• SSH Key Management: Secure access to remote infrastructure
• Resource Optimization: Cost-effective cloud resource utilization

Log Management

• Fluentd Integration: Centralized log aggregation
• Real-time Streaming: Live log monitoring and filtering
• Error Detection: Automated error identification and alerting

Workspace Controller

• Multi-tenancy: Isolated environments per user/project
• File Management: Project upload and organization
• Access Control: Secure workspace isolation

🛠️ Development

Adding New Features

1. Create new route in app/routes/
2. Implement business logic in app/controllers/
3. Add data models in app/models/
4. Write tests in tests/

Docker Compose Project Structure

Projects should include:

• docker-compose.yml: Service definitions
• Dockerfile: Container build instructions
• .env: Environment variables (optional)

Custom Exceptions

The application includes comprehensive exception handling:

• DockerComposeBuildFailedException
• DockerComposeDeployFailedException
• DockerComposeDownFailedException
• DockerContextSetException

🔐 Security

• Authentication: Token-based API authentication via middleware
• Isolation: User workspace isolation with unique naming
• SSH Security: Secure VM access with key management
• Network Security: Traefik integration with SSL support

📈 Scaling

• Horizontal Scaling: Multiple worker processes with uvicorn
• Resource Management: Automatic cleanup of unused resources
• Cost Optimization: Azure Spot VM integration
• Load Balancing: Traefik integration for service discovery

🔄 Cleanup Operations

The platform includes comprehensive cleanup capabilities:

Project-Specific Cleanup

• Remove containers for specific projects
• Clean up project-related images and volumes
• Maintain isolation between different deployments

Complete VM Cleanup

• Stop all containers on a VM
• Remove all unused images, volumes, and networks
• System-wide Docker cleanup for VM reuse

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Submit a pull request

Code Style

• Follow Python PEP 8 guidelines
• Use type hints where applicable
• Include comprehensive docstrings
• Write unit tests for new features

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

• Documentation: /docs endpoint for interactive API docs
• Health Checks: /health endpoint for service monitoring
• Logs: Comprehensive logging with structured output
• Error Tracking: Automated error detection and reporting

🎯 Use Cases

Development Teams

• Rapid prototyping and testing
• Isolated development environments
• Automated CI/CD integration

DevOps Engineers

• Multi-environment deployments
• Resource optimization
• Monitoring and observability

Cloud Cost Optimization

• Spot VM utilization
• Automated resource cleanup
• Pay-per-use infrastructure

📊 Monitoring Dashboard Features

• Real-time container statistics
• Log aggregation and search
• Service health monitoring
• Resource utilization tracking
• Error rate monitoring
• Deployment history

---

For more detailed information, visit the interactive API documentation at /docs when the service is running.