🚀 [v0.2.0-beta.0] Arquitetura Modular, CI/CD e Segurança#12
Merged
Conversation
…sional dashboard - Implement RESTful API versioning (/api/v1/public) - Add Redis-based analytical cache for BOM and ABC Curve - Introduce Sandbox mode for non-destructive ETL testing - Secure public routes with Kong Gateway limits and CORS - Optimize database queries using indexed date ranges - Build professional Tool-Landing Page with multi-format exports
…ecture, accessibility API: - Add get_composicao_man_hours (hora-homem endpoint) - Add get_candidatos_otimizacao (optimization endpoint) - Add nivel field to ComposicaoBOMItem schema - Fix Redis cache decorator dict-vs-object access Demo Frontend: - CSS: @layer cascade system, color-mix(), container queries, clamp() - JS: Dependency Injection wiring, getChartTheme() single source of truth - Accessibility: WCAG 2.1 AA, forced-colors, reduced-motion, keyboard nav - Remove 150+ lines of duplicated code (shimmer, downloadAsFile, closeMobileMenu, onclick→data-format) - Responsive: 320px → 3840px, dvh, mobile-first breakpoints Docs: - Update main README with new BI endpoints, demo section - Rewrite demo/README with full architecture docs - Add logs/ to .gitignore
…ments
API:
- CORS configurável via ALLOWED_ORIGINS (env var)
- Structured Logging em JSON com timestamp, endpoint, duration
- Endpoint /health com status de DB e Redis
- Cache decorator nos CRUDs básicos (get_insumo, search_insumo, etc.)
- invalidate_cache() com scan+cursor para limpeza por padrão
- Limite de recursão (nivel < 10) em todas as CTEs (ABC, Man-Hours, ABC por classificacao)
- Trends com relativedelta em vez de timedelta(days=30*N)
- Filtros classificacao/grupo movidos do Python para o SQL
- Alembic migrations setup + schema inicial (7 tabelas + view vw_composicao_itens_unificados)
- 2 novos endpoints BI: GET /bi/composicao/{codigo}/produtividade e GET /bi/insumo/{codigo}/onde-usado
- Testes de cache refatorados com unittest.mock (sem Redis real), 5/5 passando
Demo:
- Export BOM to PDF com jsPDF + autoTable
- Focus trap no modal + aria-live nos resultados dinâmicos
- HTML corrigido (tags mal fechadas)
- Responsividade: comparison grid, trends form, comparison form em mobile/tablet
- Heatmap: mapa Leaflet coroplético com GeoJSON oficial IBGE + fallback bar chart
- Comparison chart: categorias por Material/Mão de Obra/Equipamento em vez de INSUMO/COMPOSICAO
- Chart.getChart() em todos os 6 módulos (abc, compare, comparison, trends, heatmap, modal)
- Leaflet CDN, legendas dark mode, scroll-zoom desativado
- Quality checklist 8/8 itens verificados
Documentação:
- docs/WorkPlan.md reescrito: tabela completa de 18+ endpoints, 12 schemas, arquitetura
- docs/SPRINT_ENRIQUECIMENTO.md (documentação das sprints 1a, 1b, 1c)
- docs/workplans/SPRINT_202605_API_DEMO_ENHANCEMENT.md (plano desta sessão)
- docs/plans/SPRINT_HEATMAP_LEAFLET.md (plano do mapa Leaflet)
Infra:
- Alembic adicionado ao requirements.txt
- GeoJSON oficial do IBGE (27 UFs) servido via FastAPI StaticFiles
…hancement - PRD_DATA_TRACEABILITY.md: Full audit, requirements, architecture - SPRINT_3.1: Schema migration (traceability columns + audit log) - SPRINT_3.2: ETL enhancement (UPSERT, DELETE by period, version tracking) - SPRINT_3.3: API enhancement (traceability exposure, audit endpoint)
- Add Alembic migration 002: traceability columns + audit log table
- Update database.py: UPSERT on append, audit logging, version/run_id propagation
- Update etl_pipeline.py: DELETE by period (not TRUNCATE), extract SINAPI version
- Update api/schemas.py: TraceabilityMixin, AuditEvent schema
- Update api/crud.py: return traceability fields, add get_audit_events
- Update api/main.py: add /audit/{tipo}/{codigo} endpoint
- Add comprehensive PRD and sprint documentation
- Remove AutoSINAPI/ from .gitignore (was incorrectly ignoring ETL source) - Include database.py and etl_pipeline.py traceability enhancements
- database.py: UPSERT on append, audit logging, version/run_id propagation - etl_pipeline.py: DELETE by period (not TRUNCATE), extract SINAPI version - Remove embedded .git from AutoSINAPI/ (was preventing tracking) - Update .gitignore to not ignore AutoSINAPI/ toolkit
New test files:
- test_migration.py: Validates Alembic 002 migration (traceability columns)
- test_traceability_db.py: UPSERT, audit log, version propagation
- test_traceability_etl.py: ETL traceability (DELETE by period, version extraction)
- test_traceability_api.py: API traceability (audit endpoint, schemas)
- test_sandbox_integration.py: E2E integration with mock SINAPI data
Updated test files:
- test_database.py: Added UPSERT behavior, traceability propagation tests
- test_pipeline.py: Added sinapi_versao extraction, DELETE by period tests
- test_file_input.py: (already existed, staged)
Features tested:
- Migration 002 creates traceability columns + audit log table
- _append_data() now does UPSERT (not just INSERT IGNORE)
- sinapi_versao and etl_run_id propagated through ETL
- DELETE by period replaces TRUNCATE for structure tables
- New /audit/{tipo}/{codigo} API endpoint
- TraceabilityMixin in Pydantic schemas
- Create api/sandbox_utils.py for sandbox mode (AUTOSINAPI_SANDBOX) - Update config.py to support mode='sandbox' with sandbox_ table prefix - Update database.py: always propagate sinapi_versao/etl_run_id - Fix test_migration.py: correct assertions for Alembic 002 migration - Fix test_database.py: use call.args[0] for TextClause content checks - Fix test_pipeline.py: use _execute_phase_1_acquisition mock, accent namin - Fix test_traceability_etl.py: same pattern fixes - Fix test_traceability_api.py: use app.dependency_overrides for DB mock - Skip integration tests requiring real PostgreSQL - 50 total: 37 ETL + 13 API tests passing, 2 skipped
- etl_pipeline.py: Phase 0 checks config.DB_TABLE_INSUMOS (not hardcoded) - config.py: add DB_TABLE_AUDIT_LOG for sandbox prefix support - database.py: use config table names for audit log DDL and queries - database.py: drop/create audit log with config table name
- Fix etl_pipeline.py: use config table names for Phase 0 check - Fix database.py: use config DB_TABLE_AUDIT_LOG, add uuid import - Fix database.py: change etl_run_id to VARCHAR(36) type - Fix database.py: add DISTINCT ON dedup for UPSERT queries - Fix etl_pipeline.py: add column existence checks in placeholder gen - Fix etl_pipeline.py: revert structure tables to TRUNCATE (no data_referencia) - Add run_sandbox.py for sandbox ETL execution - Migration 002 applied to real DB - Sandbox ETL populated 2025-07: 1,160,750 records - Traceability fields verified: sinapi_versao, etl_run_id
Changes requirements.txt to install ./AutoSINAPI/ locally so the Docker build uses the local ETL toolkit with all traceability fixes (UPSERT, version propagation, audit log)
…bilize UI tests - Update CustomJSONEncoder to support SQLAlchemy RowMapping objects. - Fix API tests returning 500 ResponseValidationError. - Implement robust UF property detection in heatmap.js. - Ensure Heatmap and History charts occupy 100% width with 16/9 aspect ratio in modals. - Fix UI test race conditions by awaiting module logic and data population. - Add Grid vs Table toggle in the Comparison UI. - Handle missing insumos classifications by grouping them as 'GERAL' in Trends analysis. - Adjust Trends Y-axis to start at zero. - Add UF filter bindings missing in dom.js. - Document fixes in docs/history/2026-05-22_Stabilization_and_UI_Refinements.md
…integrity - Refactor Trends endpoint to support agrupar_por (classificacao, grupo, item) and codigos filter. - Update UI to allow switching trend dimensions and individual item analysis. - Fix Processor to correctly extract Grupo column from 'Analítico' Excel sheet. - Harden ETL pipeline to protect classifications during placeholder merging. - Ensure etl_run_id and sinapi_versao are propagated to all 10 DB tables. - Standardize metadata with UPPER(TRIM()) in trend analysis. - Update documentation in READMEs and history records.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 [v0.2.0-beta.0] Arquitetura Modular, CI/CD e Segurança
Este Pull Request marca a maturidade estrutural da API do autoSINAPI. O projeto deixou de ser um "MonoRepo" amontoado para se tornar um Ecossistema Profissional, separando a inteligência de extração (Toolkit) da camada de entrega e segurança (API + Gateway).
✨ O que há de novo?
1. Arquitetura Orientada a Submódulos (
feat/submodule)AutoSINAPI) agora é consumido como um Git Submodule. Isso permite versionamento independente e reutilização do Toolkit em outros projetos sem acoplar a infraestrutura da API.Dockerfilee odocker-compose.ymlforam reescritos para baixar e instalar o submódulo de forma autônoma e segura.2. Automação e Qualidade (
feat/ci)ci.ymlque sobe temporariamente o Banco de Dados, o Redis e o Kong para testar se a API consegue importar e se comunicar com o Toolkit antes de permitir qualquer merge.3. Auditoria e Segurança (
chore/security)docker-compose.ymlem versões anteriores foi revogado e substituído pela variável de ambiente${CLOUDFLARE_TUNNEL_TOKEN}..env.exampleAtualizado: Documentação de variáveis de ambiente foi melhorada para guiar novos usuários de forma segura.🛠️ Melhorias na UI e Endpoints Internos (
feat/ui-api)etl_run_id) do banco de dados.--recursive(git clone --recursive ...) para garantir que o código do submódulo venha junto.make setupfoi criado para ajudar a inicializar o ambiente e puxar o submódulo caso o usuário tenha esquecido a flag na clonagem.✅ Checklist de Validação
pytest).env,.secretsou token no histórico recente.README.mdeCONTRIBUTING.md) refatorada para explicar a arquitetura.