Skip to content

Add BigQuery backend connection testing - #3874

Draft
ruslandoga wants to merge 2 commits into
Logflare:mainfrom
ruslandoga:rd/bigquery-connection-test
Draft

Add BigQuery backend connection testing#3874
ruslandoga wants to merge 2 commits into
Logflare:mainfrom
ruslandoga:rd/bigquery-connection-test

Conversation

@ruslandoga

@ruslandoga ruslandoga commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Implements O11Y-1947 with a REST-based BigQuery connection test that deterministically selects an attached source—preferring a direct attachment and falling back to a rule source—and inserts one labeled probe event into its existing table.

The probe requires a same-owner source association, performs no dataset or table DDL, and returns source_required before making a BigQuery request when none exists.

REST responses are normalized to atom-only sanitized errors, and the LiveView explains how to satisfy the source prerequisite.

It intentionally does not wait for Storage Write gRPC support.

Tests: the BigQuery adaptor file passes with 35 tests and one property, the backend component file passes with 5 tests, and Dialyzer, formatting, diff checks, and targeted strict Credo pass.

@spec create_connection_test_table(Backend.t(), String.t(), String.t()) ::
{:ok, connection_test_table_status()} | {:error, connection_test_error()}
defp create_connection_test_table(backend, project_id, dataset_id) do
case Google.BigQuery.create_table(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Severity: MEDIUM

project_id and dataset_id are user-controlled backend fields, but this call uses Logflare’s shared ingest credential to create _logflare_connection_test in that GCP dataset, even without an attached source. A tenant can target another customer’s project where the shared account is authorized, causing cross-tenant table creation and probe writes.
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: Add a project/dataset ownership check at the beginning of test_connection/1 (around line 189) to verify that the user-controlled project_id and dataset_id actually belong to the requesting tenant before invoking any GCP API calls with Logflare's shared ingest credentials.

Specifically:

  1. In test_connection/1, extract user_id from the %Backend{} struct pattern match.
  2. Fetch the user via Users.Cache.get(user_id) and add a with clause guard that compares project_id (from backend.config) against the user's bigquery_project_id (from the User struct) and also checks dataset_id against user.bigquery_dataset_id. If the values do not match, return {:error, :invalid_config} (or a dedicated :unauthorized error atom) before any GCP call is made.
  3. For tenants using Logflare's own shared project (where user.bigquery_project_id is nil), also guard that the provided project_id equals GCPConfig.default_project_id() and the dataset_id matches the expected user-scoped dataset.

This ensures that the shared ingest credential is never used to probe a GCP project/dataset that the requesting tenant does not own.

@ruslandoga
ruslandoga marked this pull request as draft August 25, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant