Feature/pipeline #538

pontusab · 2025-07-17T12:13:54Z

No description provided.

.github/workflows/beta.yml

+    name: 🔧 Setup & Cache
+    runs-on: ubuntu-latest
+    outputs:
+      cache-hit: ${{ steps.cache-deps.outputs.cache-hit }}
+      turbo-cache-hit: ${{ steps.cache-turbo.outputs.cache-hit }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - uses: oven-sh/setup-bun@v1
+        with:
+          bun-version: latest
+
+      - name: 📦 Cache dependencies (with compression)
+        uses: actions/cache@v4
+        id: cache-deps
+        with:
+          path: |
+            node_modules
+            ~/.bun/install/cache
+          key: deps-${{ runner.os }}-${{ hashFiles('**/bun.lock', '**/package.json') }}
+          restore-keys: |
+            deps-${{ runner.os }}-
+          enableCrossOsArchive: true
+
+      - name: 🚀 Cache Turbo (with compression)
+        uses: actions/cache@v4
+        id: cache-turbo
+        with:
+          path: |
+            .turbo
+            **/dist
+            **/.next/cache
+          key: turbo-beta-${{ runner.os }}-${{ github.sha }}
+          restore-keys: |
+            turbo-beta-${{ runner.os }}-
+          enableCrossOsArchive: true
+
+      - name: 📦 Install dependencies (with parallel installs)
+        if: steps.cache-deps.outputs.cache-hit != 'true'
+        run: |
+          bun install --frozen-lockfile --concurrent=10
+
+  # Change detection (fast, separate job)
+  detect-changes:


To fix the issue, add a permissions block to the root of the workflow file. This block will define the least privileges required for the workflow to function correctly. Based on the provided snippet, the workflow primarily involves reading repository contents for caching and change detection. Therefore, the contents: read permission is sufficient.

The permissions block should be added near the top of the file, after the name field, to apply to all jobs in the workflow that do not have their own permissions block.

.github/workflows/beta.yml

+    name: 🔍 Detect Changes
+    runs-on: ubuntu-latest
+    outputs:
+      has-api-changes: ${{ steps.changes.outputs.api }}
+      has-dashboard-changes: ${{ steps.changes.outputs.dashboard }}
+      has-website-changes: ${{ steps.changes.outputs.website }}
+      has-engine-changes: ${{ steps.changes.outputs.engine }}
+      has-email-changes: ${{ steps.changes.outputs.email }}
+      has-packages-changes: ${{ steps.changes.outputs.packages }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: 🔍 Advanced change detection
+        uses: dorny/paths-filter@v3
+        id: changes
+        with:
+          filters: |
+            api:
+              - 'apps/api/**'
+              - 'packages/!(email|ui)/**'
+            dashboard:
+              - 'apps/dashboard/**'
+              - 'packages/**'
+            website:
+              - 'apps/website/**'
+              - 'packages/!(email)/**'
+            engine:
+              - 'apps/engine/**'
+              - 'packages/engine-client/**'
+            email:
+              - 'packages/email/**'
+            packages:
+              - 'packages/**'
+              - 'turbo.json'
+              - 'package.json'
+
+  # Parallel quality checks (matrix build for speed)
+  quality-checks:


To fix the issue, we need to add a permissions block to the workflow. This block should be added at the root level of the workflow to apply to all jobs, or it can be added to individual jobs if different permissions are required for each. Based on the workflow's functionality, the minimal permissions required are contents: read, as the jobs primarily read repository contents.

Steps to implement the fix:

Add a permissions block at the root level of the workflow file (.github/workflows/beta.yml) to limit the GITHUB_TOKEN permissions to contents: read.

Ensure that the permissions block is correctly indented and formatted according to YAML syntax.

.github/workflows/beta.yml

+    name: ${{ matrix.check-name }}
+    runs-on: ubuntu-latest
+    needs: [setup, detect-changes]
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - check-name: "🔦 Lint"
+            check-type: "lint"
+          - check-name: "🪐 TypeScript"
+            check-type: "typecheck"
+          - check-name: "🧪 Tests"
+            check-type: "test"
+          - check-name: "🔒 Security"
+            check-type: "security"
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - uses: oven-sh/setup-bun@v1
+        with:
+          bun-version: latest
+
+      - name: 📦 Restore dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            node_modules
+            ~/.bun/install/cache
+          key: deps-${{ runner.os }}-${{ hashFiles('**/bun.lock', '**/package.json') }}
+
+      - name: 🚀 Restore Turbo cache
+        uses: actions/cache@v4
+        with:
+          path: |
+            .turbo
+            **/dist
+            **/.next/cache
+          key: turbo-beta-${{ runner.os }}-${{ github.sha }}
+          restore-keys: |
+            turbo-beta-${{ runner.os }}-
+
+      - name: 📦 Install dependencies (if cache miss)
+        if: needs.setup.outputs.cache-hit != 'true'
+        run: bun install --frozen-lockfile
+
+      - name: 🏗️ Build dependencies (only for typecheck)
+        if: matrix.check-type == 'typecheck'
+        run: |
+          # Build core dependencies that other packages rely on for TypeScript
+          bunx turbo build --filter='@midday/engine-client' --filter='@midday/engine' --filter='@midday/utils' --filter='@midday/ui' --filter='@midday/tsconfig'
+
+      - name: 🔦 Run Lint
+        if: matrix.check-type == 'lint'
+        run: |
+          if [ "${{ github.event.inputs.force_deploy }}" = "true" ]; then
+            bunx turbo lint
+          else
+            bunx turbo lint --filter='[HEAD^1]'
+          fi
+
+      - name: 🪐 Run TypeScript Check
+        if: matrix.check-type == 'typecheck'
+        run: |
+          if [ "${{ github.event.inputs.force_deploy }}" = "true" ]; then
+            bunx turbo typecheck
+          else
+            # Include dependencies in TypeScript check to ensure types are available
+            bunx turbo typecheck --filter='[HEAD^1]...'
+          fi
+
+      - name: 🧪 Run Tests
+        if: matrix.check-type == 'test'
+        run: |
+          if [ "${{ github.event.inputs.force_deploy }}" = "true" ]; then
+            bunx turbo test
+          else
+            bunx turbo test --filter='[HEAD^1]'
+          fi
+
+      - name: 🔒 Security Checks
+        if: matrix.check-type == 'security'
+        run: |
+          # Security audit
+          bun audit --audit-level moderate || true
+          echo "Security audit completed"
+
+      - name: 🔍 Secret Scanning
+        if: matrix.check-type == 'security'
+        uses: trufflesecurity/trufflehog@main
+        with:
+          path: ./
+          base: main
+          head: HEAD
+          extra_args: --debug --only-verified
+        continue-on-error: true
+
+  # Build shared artifacts (parallelized)
+  build-shared:


To fix the issue, we need to add a permissions block to the quality-checks job to explicitly limit the permissions of the GITHUB_TOKEN. Since the job does not require write access, we can set the permissions to contents: read. This ensures that the job has only the access it needs to perform its tasks.

The changes should be made in the .github/workflows/beta.yml file, specifically within the quality-checks job definition. No additional imports, methods, or definitions are required.

.github/workflows/beta.yml

+    name: 🏗️ Build Shared
+    runs-on: ubuntu-latest
+    needs: [setup, detect-changes, quality-checks]
+    if: needs.quality-checks.result == 'success' && (needs.detect-changes.outputs.has-engine-changes == 'true' || needs.detect-changes.outputs.has-packages-changes == 'true' || github.event.inputs.force_deploy == 'true')
+    steps:
+      - uses: actions/checkout@v4
+      - uses: oven-sh/setup-bun@v1
+        with:
+          bun-version: latest
+
+      - name: 📦 Restore dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            node_modules
+            ~/.bun/install/cache
+          key: deps-${{ runner.os }}-${{ hashFiles('**/bun.lock', '**/package.json') }}
+
+      - name: 🚀 Restore Turbo cache
+        uses: actions/cache@v4
+        with:
+          path: |
+            .turbo
+            **/dist
+            **/.next/cache
+          key: turbo-beta-${{ runner.os }}-${{ github.sha }}
+          restore-keys: |
+            turbo-beta-${{ runner.os }}-
+
+      - name: 🏗️ Build Engine and Dependencies (parallel)
+        run: |
+          # Build all core packages in parallel with optimal concurrency
+          bunx turbo build --filter=@midday/engine --filter=@midday/engine-client --filter=@midday/utils --filter=@midday/ui --concurrency=50%
+
+      - name: 📦 Cache Engine build
+        uses: actions/cache@v4
+        with:
+          path: apps/engine/dist
+          key: engine-build-beta-${{ github.sha }}
+
+      - name: 📦 Cache Core Packages
+        uses: actions/cache@v4
+        with:
+          path: |
+            packages/engine-client/dist
+            packages/utils/dist
+            packages/ui/dist
+          key: core-packages-beta-${{ github.sha }}
+
+  # Deploy Engine first (dependency for other apps)
+  deploy-engine:


To fix the issue, add a permissions block to the Build Shared job to explicitly limit the permissions of the GITHUB_TOKEN. Since the job primarily interacts with repository contents (e.g., checking out code and caching dependencies), the contents: read permission is sufficient. This ensures the job has only the minimal permissions required to complete its tasks.

The permissions block should be added under the build-shared job definition, ensuring it applies only to this job.

.github/workflows/beta.yml

+    name: 🚀 Deploy Engine (Beta)
+    runs-on: ubuntu-latest
+    needs: [detect-changes, quality-checks, build-shared]
+    if: needs.quality-checks.result == 'success' && (needs.detect-changes.outputs.has-engine-changes == 'true' || github.event.inputs.force_deploy == 'true') && (needs.build-shared.result == 'success' || needs.build-shared.result == 'skipped')
+    environment: beta
+    outputs:
+      deployment-url: ${{ steps.deploy.outputs.deployment-url }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: oven-sh/setup-bun@v1
+        with:
+          bun-version: latest
+
+      - name: 📦 Restore Engine build
+        uses: actions/cache@v4
+        with:
+          path: apps/engine/dist
+          key: engine-build-beta-${{ github.sha }}
+
+      - name: 📦 Restore Core Packages
+        uses: actions/cache@v4
+        with:
+          path: |
+            packages/engine-client/dist
+            packages/utils/dist
+            packages/ui/dist
+          key: core-packages-beta-${{ github.sha }}
+
+      - name: 📦 Restore dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            node_modules
+            ~/.bun/install/cache
+          key: deps-${{ runner.os }}-${{ hashFiles('**/bun.lock', '**/package.json') }}
+
+      - name: 🚀 Deploy to Cloudflare (Staging)
+        id: deploy
+        uses: cloudflare/wrangler-action@v3
+        with:
+          packageManager: bun
+          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
+          workingDirectory: "apps/engine"
+          wranglerVersion: "3.93.0"
+          command: deploy --minify src/index.ts --env staging
+
+      - name: 🏥 Health check
+        run: |
+          sleep 30
+          curl -f https://staging-engine.midday.ai/health || echo "Health check failed, but continuing..."
+
+      - name: 📢 Deployment notification
+        if: always()
+        uses: 8398a7/action-slack@v3
+        with:
+          status: ${{ job.status }}
+          channel: '#deployments'
+          webhook_url: ${{ secrets.SLACK_WEBHOOK_URL }}
+        continue-on-error: true
+
+  # Deploy API second (dependency for dashboard)
+  deploy-api:


To fix the issue, add an explicit permissions block to the deploy-engine job. This block should specify the minimal permissions required for the job to function correctly. Based on the operations performed in the job, the contents: read permission is sufficient for most steps, while additional permissions (e.g., deployments: write) may be required for deployment-related tasks.

The permissions block should be added directly under the deploy-engine job definition, ensuring it applies only to this job. This approach avoids affecting other jobs in the workflow.