Review pr 261 #266
Open
Review pr 261 #266
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Overview Implements complete Kubernetes/OpenShift deployment strategy with Helm charts, auto-scaling, high availability, and comprehensive documentation. ## What's New ### Kubernetes Manifests - ✅ Complete K8s manifests in `deployment/k8s/base/` - ✅ Namespace, ConfigMaps, and Secrets templates - ✅ StatefulSets for PostgreSQL, Milvus, MinIO, etcd - ✅ Deployments for Backend, Frontend, MLFlow - ✅ Services for all components - ✅ Ingress with TLS and OpenShift Routes - ✅ HorizontalPodAutoscaler for auto-scaling ### Helm Chart - ✅ Production-ready Helm chart in `deployment/helm/rag-modulo/` - ✅ Environment-specific values (dev, staging, prod) - ✅ Configurable resources and scaling policies - ✅ Support for multiple cloud providers ### Deployment Scripts - ✅ `deployment/scripts/deploy-k8s.sh` - Raw K8s deployment - ✅ `deployment/scripts/deploy-helm.sh` - Helm deployment - ✅ Environment validation and health checks - ✅ Automated deployment workflow ### Makefile Targets (40+ new commands) **Kubernetes:** - `make k8s-deploy-dev/staging/prod` - Deploy to K8s - `make k8s-status` - Show deployment status - `make k8s-logs-backend/frontend` - Stream logs - `make k8s-port-forward-*` - Port forwarding - `make k8s-shell-backend` - Open pod shell - `make k8s-cleanup` - Clean up resources **Helm:** - `make helm-install-dev/staging/prod` - Install chart - `make helm-upgrade-dev/staging/prod` - Upgrade release - `make helm-rollback` - Rollback release - `make helm-status` - Show release status - `make helm-uninstall` - Remove release **Cloud Providers:** - `make ibmcloud-deploy CLUSTER_NAME=<name>` - IBM Cloud - `make openshift-deploy` - OpenShift - Support for AWS EKS, Azure AKS, Google GKE **Documentation:** - `make docs-install` - Install MkDocs - `make docs-serve` - Serve docs locally - `make docs-build` - Build static site - `make docs-deploy` - Deploy to GitHub Pages ### CI/CD Workflows - ✅ `.github/workflows/k8s-deploy-production.yml` - Production deployment - ✅ `.github/workflows/k8s-deploy-staging.yml` - Staging/PR deployment - ✅ Automated build, push, and deploy pipeline - ✅ Health checks and verification ### Documentation (MkDocs) - ✅ Updated `mkdocs.yml` with complete navigation - ✅ `docs/deployment/QUICKSTART.md` - 5-minute quick start - ✅ `docs/deployment/kubernetes.md` - Complete K8s guide - ✅ `docs/deployment/index.md` - Deployment overview - ✅ `docs/README.md` - MkDocs writing guide - ✅ `docs/MKDOCS_SETUP.md` - Setup summary - ✅ Custom styling in `docs/stylesheets/extra.css` ## Key Features ### High Availability - Backend: 3 replicas with auto-scaling (2-10 pods) - Frontend: 2 replicas with auto-scaling (2-5 pods) - Rolling updates with zero downtime - Health probes (liveness, readiness, startup) ### Auto-Scaling - HPA based on CPU (70%) and Memory (80%) - Intelligent scale-up/down policies - Resource limits enforced ### Persistent Storage - PostgreSQL: 50Gi (prod), 10Gi (dev) - Milvus: 100Gi (prod), 20Gi (dev) - MinIO: 100Gi (prod), 20Gi (dev) - etcd: 10Gi (prod), 5Gi (dev) ### Security - Secrets management templates - TLS/SSL with cert-manager integration - OpenShift SCC support - Network policies ready ### Monitoring - Prometheus metrics endpoints - HPA metrics collection - Comprehensive logging ## Cloud Provider Support ### IBM Cloud Kubernetes Service ```bash make ibmcloud-deploy CLUSTER_NAME=<cluster-name> ``` ### OpenShift ```bash make openshift-deploy ``` ### AWS EKS / Azure AKS / Google GKE See docs/deployment/kubernetes.md for details ## Files Changed - Modified: Makefile, mkdocs.yml, docs/deployment/index.md - Added: 45+ new files for K8s, Helm, docs, scripts ## Testing - ✅ All pre-commit checks passed - ✅ Helm chart lints successfully - ✅ K8s manifests validate - ✅ Deployment scripts tested Closes #260
) Implements IBM Docling integration with AI-powered table extraction (TableFormer) and layout analysis (DocLayNet) to significantly improve document processing quality. Key Features: - DoclingProcessor with comprehensive text, table, and image extraction - Feature flag control (ENABLE_DOCLING) for transparent deployment - Automatic fallback to legacy processors on error - Support for PDF, DOCX, PPTX, HTML, and image formats - 313% improvement in chunk extraction vs legacy processors - Table detection: 3 tables vs 0 (legacy) - Image detection: 13 images vs 0 (legacy) Implementation: - New DoclingProcessor class with DocumentConverter integration - Enhanced metadata extraction with table/image counts - Page number tracking with new Docling API compatibility - Chunking strategy integration for optimal text segmentation - Type-safe implementation with mypy validation Testing: - 14 comprehensive unit tests (100% passing) - Real PDF comparison validation - Debug utilities for development - All critical code quality checks passing Technical Details: - Updated transformers to 4.56.2 for compatibility - Handled Docling API changes (tuple unpacking, page_no attribute) - Multiple text item types support (TextItem, SectionHeaderItem, ListItem, CodeItem) - Separate counters for tables, images, and chunks - Code quality: 9.64/10 (docling_processor.py), 9.84/10 (document_processor.py) Closes #255
Resolved conflicts: - backend/core/config.py: Combined Docling and hierarchical chunking settings - backend/pyproject.toml: Added both docling+transformers and pydub dependencies - backend/poetry.lock: Regenerated after dependency resolution - .linting-progress.json: Removed (deleted in main) All conflicts resolved and dependencies updated.
- Free up ~10GB by removing unnecessary pre-installed tools (.NET, Android SDK, GHC, CodeQL) - Clean Docker system before builds to free up space - Enable Docker BuildKit for better caching and smaller layers - Add disk space reporting before/after cleanup Fixes: GitHub Actions 'No space left on device' error
Prevents 'No space left on device' errors in GitHub Actions by freeing ~10-14GB: Modified workflows: - ci.yml (main CI/CD pipeline) - k8s-deploy-staging.yml (staging deployment) - k8s-deploy-production.yml (production deployment) - dev-environment-ci.yml (already fixed in previous commit) Added test workflow: - test-disk-cleanup.yml (manual validation workflow) Each cleanup step: 1. Removes pre-installed tools: .NET (~3-4GB), Android SDK (~3-4GB), GHC (~1-2GB), CodeQL (~1-2GB) 2. Cleans Docker system cache (~1-2GB) 3. Reports disk space before/after Validation performed: ✓ All YAML syntax valid ✓ Cleanup runs BEFORE Docker builds in all workflows ✓ Industry-standard pattern (used by Kubernetes, TensorFlow, etc.) Fixes: https://github.com/manavgup/rag_modulo/actions/runs/18222106174/job/51884221970
Make the staging deployment workflow conditional so it doesn't fail when no Kubernetes cluster is configured. Changes: - Added conditional to deploy-helm job (only runs if KUBECONFIG exists) - Added deployment-skipped job for when cluster is not available - Both jobs provide informative PR comments This allows the workflow to succeed even without a K8s cluster while still building and pushing Docker images to GHCR.
- Fix staging workflow to skip deployment for PRs (no KUBECONFIG available) - Add informational job for PRs that builds images but skips deployment - Add disk space cleanup step to staging workflow - Create complete Helm chart templates: - Namespace, ConfigMap, Secrets - Backend and Frontend Deployments and Services - Ingress with TLS support - HorizontalPodAutoscaler for auto-scaling - Fix Helm template helpers and validation - All workflows now pass linting and are ready for deployment Resolves: PR #261 deployment failures
Contributor
🚀 Development Environment OptionsThis repository supports Dev Containers for a consistent development environment. Option 1: GitHub Codespaces (Recommended)Create a cloud-based development environment:
Option 2: VS Code Dev Containers (Local)Use Dev Containers on your local machine:
Option 3: Traditional Local SetupSet up the development environment manually: # Clone the repository
git clone https://github.com/manavgup/rag_modulo.git
cd rag_modulo
git checkout review-pr-261
# Initialize development environment
make dev-init
make dev-build
make dev-up
make dev-validateAvailable CommandsOnce in your development environment: make help # Show all available commands
make dev-validate # Validate environment setup
make test-atomic # Run atomic tests
make test-unit # Run unit tests
make lint # Run lintingServices AvailableWhen running
This automated message helps reviewers quickly set up the development environment. |
- Add IBM Code Engine staging deployment workflow - Uses local disk cache (like IBM mcp-context-forge project) - Buildx v3 with explicit cache restore - Cache mode=min to prevent exhaustion - Comprehensive timeouts to prevent hangs - Clean IBM-style structure with emojis and sections - Remove failing K8s staging workflow - Was failing 100% of runs (11/11) - Caused by GitHub Actions cache exhaustion - Not needed without K8s cluster - Add local workflow testing with act - New script: scripts/test-workflows.sh - Test workflows before pushing to GitHub - Documentation: docs/development/testing-workflows-locally.md - Add deployment documentation - Quick start: docs/deployment/ibm-code-engine-quickstart.md - Setup time: 1-2 hours - Cost: $0-20/month (vs $300+ for K8s) - Update CLAUDE.md - Add workflow testing section - Add deployment documentation references - Document new CI/CD workflows Resolves CI/CD timeout issues and prepares for IBM Cloud deployment.
- Add scripts/ibm-create-secrets.sh to automatically create secrets from .env - Update quickstart guide to use the automated script - Prevents API keys from being hardcoded in documentation - Add api-key.json to gitignore for security This allows secure secret management without exposing credentials in git.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.