Skip to content

hub/analytics: make analytics cookie optional #8452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 0 additions & 22 deletions src/.claude/settings.local.json

This file was deleted.

59 changes: 42 additions & 17 deletions src/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,16 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

# CoCalc Source Repository

* This is the source code of CoCalc in a Git repository
* It is a complex JavaScript/TypeScript SaaS application
* CoCalc is organized as a monorepository (multi-packages) in the subdirectory "./packages"
* The packages are managed as a pnpm workspace in "./packages/pnpm-workspace.yaml"
- This is the source code of CoCalc in a Git repository
- It is a complex JavaScript/TypeScript SaaS application
- CoCalc is organized as a monorepository (multi-packages) in the subdirectory "./packages"
- The packages are managed as a pnpm workspace in "./packages/pnpm-workspace.yaml"

## Code Style

- Everything is written in TypeScript code
- Indentation: 2-spaces
- Run `pretter -w [filename]` after modifying a file (ts, tsx, md, json, ...) to format it correctly.
- All .js and .ts files are formatted by the tool prettier
- Add suitable types when you write code
- Variable name styles are "camelCase" for local and "FOO_BAR" for global variables. If you edit older code not following these guidlines, adjust this rule to fit the files style.
Expand All @@ -23,28 +24,32 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Development Commands

### Essential Commands

- `pnpm build-dev` - Build all packages for development
- `pnpm clean` - Clean all node_modules and dist directories
- `pnpm database` - Start PostgreSQL database server
- `pnpm hub` - Start the main hub server
- `pnpm psql` - Connect to the PostgreSQL database
- `pnpm test` - Run full test suite
- `pnpm test-parallel` - Run tests in parallel across packages
- `pnpm depcheck` - Check for dependency issues
- `prettier -w [filename]` to format the style of a file after editing it
- after creating a file, run `git add [filename]` to start tracking it

### Package-Specific Commands
- `cd packages/[package] && pnpm tsc` - Watch TypeScript compilation for a specific package

- `cd packages/[package] && pnpm build` - Build and compile a specific package
- for packages/next and packages/static, run `cd packages/[package] && pnpm build-dev`
- `cd packages/[package] && pnpm tsc:watch` - TypeScript compilation in watch mode for a specific package
- `cd packages/[package] && pnpm test` - Run tests for a specific package
- `cd packages/[package] && pnpm build` - Build a specific package
- **IMPORTANT**: When modifying packages like `util` that other packages depend on, you must run `pnpm build` in the modified package before typechecking dependent packages

### Development Setup
1. Start database: `pnpm database`
2. Start hub: `pnpm hub`
3. For TypeScript changes, run `pnpm tsc` in the relevant package directory
### Development

- After code changes, run `pretter -w [filename]` to ensure consistent styling
- After TypeScript or `*.tsx` changes, run `pnpm build` in the relevant package directory

## Architecture Overview

### Package Structure

CoCalc is organized as a monorepo with key packages:

- **frontend** - React/TypeScript frontend application using Redux-style stores and actions
Expand All @@ -62,25 +67,29 @@ CoCalc is organized as a monorepo with key packages:
### Key Architectural Patterns

#### Frontend Architecture

- **Redux-style State Management**: Uses custom stores and actions pattern (see `packages/frontend/app-framework/actions-and-stores.ts`)
- **TypeScript React Components**: All frontend code is TypeScript with proper typing
- **Modular Store System**: Each feature has its own store/actions (AccountStore, BillingStore, etc.)
- **WebSocket Communication**: Real-time communication with backend via WebSocket messages

#### Backend Architecture

- **PostgreSQL Database**: Primary data store with sophisticated querying system
- **WebSocket Messaging**: Real-time communication between frontend and backend
- **Conat System**: Container orchestration for compute servers
- **Event-Driven Architecture**: Extensive use of EventEmitter patterns
- **Microservice-like Packages**: Each package handles specific functionality

#### Communication Patterns

- **WebSocket Messages**: Primary communication method (see `packages/comm/websocket/types.ts`)
- **Database Queries**: Structured query system with typed interfaces
- **Event Emitters**: Inter-service communication within backend
- **REST-like APIs**: Some HTTP endpoints for specific operations

### Key Technologies

- **TypeScript**: Primary language for all new code
- **React**: Frontend framework
- **PostgreSQL**: Database
Expand All @@ -91,40 +100,56 @@ CoCalc is organized as a monorepo with key packages:
- **SASS**: CSS preprocessing

### Database Schema

- Comprehensive schema in `packages/util/db-schema`
- Query abstractions in `packages/database/postgres/`
- Type-safe database operations with TypeScript interfaces

### Testing

- **Jest**: Primary testing framework
- **ts-jest**: TypeScript support for Jest
- **jsdom**: Browser environment simulation for frontend tests
- Test files use `.test.ts` or `.spec.ts` extensions
- Each package has its own jest.config.js

### Import Patterns

- Use absolute imports with `@cocalc/` prefix for cross-package imports
- Example: `import { cmp } from "@cocalc/util/misc"`
- Type imports: `import type { Foo } from "./bar"`
- Destructure imports when possible

### Development Workflow
1. Changes to TypeScript require compilation (`pnpm tsc` in relevant package)

1. Changes to TypeScript require compilation (`pnpm build` in relevant package)
2. Database must be running before starting hub
3. Hub coordinates all services and should be restarted after changes
4. Use `pnpm clean && pnpm build-dev` when switching branches or after major changes

# Workflow
- Be sure to typecheck when you're done making a series of code changes

- Be sure to build when you're done making a series of code changes
- Prefer running single tests, and not the whole test suite, for performance

## Git Workflow

- Never modify a file when in the `master` or `main` branch
- All changes happen through feature branches, which are pushed as pull requests to GitHub
- When creating a new file, run `git add [filename]` to track the file.
- Prefix git commits with the package and general area. e.g. 'frontend/latex: ...' if it concerns latex editor changes in the packages/frontend/... code.
- When pushing a new branch to Github, track it upstream. e.g. `git push --set-upstream origin feature-foo` for branch "feature-foo".

# important-instruction-reminders
# Important Instruction Reminders

- Do what has been asked; nothing more, nothing less.
- NEVER create files unless they're absolutely necessary for achieving your goal.
- ALWAYS prefer editing an existing file to creating a new one.
- NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User.
- REFUSE to modify files when the git repository is on the `master` or `main` branch.
- NEVER proactively create documentation files (`*.md`) or README files. Only create documentation files if explicitly requested by the User.

# Ignore

- Ignore files covered by `.gitignore`
- Ignore everything in `node_modules` or `dist` directories
- Ignore all files not tracked by Git, unless they are newly created files
77 changes: 30 additions & 47 deletions src/packages/conat/core/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,53 +28,56 @@ cd packages/server
*/

import type { ConnectionStats, ServerInfo } from "./types";

import { delay } from "awaiting";
import { EventEmitter } from "events";
import { throttle } from "lodash";
import { Server } from "socket.io";

import { getClientIpAddress } from "@cocalc/util/get-client-ip-address";
import { getLogger } from "@cocalc/conat/client";
import { UsageMonitor } from "@cocalc/conat/monitor/usage";
import { type ConatSocketServer } from "@cocalc/conat/socket";
import {
isValidSubject,
isValidSubjectWithoutWildcards,
} from "@cocalc/conat/util";
import { Server } from "socket.io";
import { delay } from "awaiting";
import { once, until } from "@cocalc/util/async-utils";
import { is_array } from "@cocalc/util/misc";
import { reuseInFlight } from "@cocalc/util/reuse-in-flight";
import { Metrics } from "../types";
import {
ConatError,
connect,
Client,
type ClientOptions,
ConatError,
connect,
MAX_INTEREST_TIMEOUT,
STICKY_QUEUE_GROUP,
} from "./client";
import {
RESOURCE,
MAX_CONNECTIONS_PER_USER,
MAX_CONNECTIONS,
MAX_PAYLOAD,
MAX_SUBSCRIPTIONS_PER_CLIENT,
MAX_SUBSCRIPTIONS_PER_HUB,
} from "./constants";
import { Patterns } from "./patterns";
import { is_array } from "@cocalc/util/misc";
import { UsageMonitor } from "@cocalc/conat/monitor/usage";
import { once, until } from "@cocalc/util/async-utils";
import {
clusterLink,
type ClusterLink,
clusterStreams,
type ClusterStreams,
trimClusterStreams,
createClusterPersistServer,
Sticky,
Interest,
hashInterest,
hashSticky,
Interest,
Sticky,
trimClusterStreams,
} from "./cluster";
import { type ConatSocketServer } from "@cocalc/conat/socket";
import { throttle } from "lodash";
import { getLogger } from "@cocalc/conat/client";
import { reuseInFlight } from "@cocalc/util/reuse-in-flight";
import { type SysConatServer, sysApiSubject, sysApi } from "./sys";
import {
MAX_CONNECTIONS,
MAX_CONNECTIONS_PER_USER,
MAX_PAYLOAD,
MAX_SUBSCRIPTIONS_PER_CLIENT,
MAX_SUBSCRIPTIONS_PER_HUB,
RESOURCE,
} from "./constants";
import { Patterns } from "./patterns";
import { forkedConatServer } from "./start-server";
import { stickyChoice } from "./sticky";
import { EventEmitter } from "events";
import { Metrics } from "../types";
import { sysApi, sysApiSubject, type SysConatServer } from "./sys";

const logger = getLogger("conat:core:server");

Expand Down Expand Up @@ -1755,27 +1758,7 @@ export function randomChoice(v: Set<string>): string {

// See https://socket.io/how-to/get-the-ip-address-of-the-client
function getAddress(socket) {
const header = socket.handshake.headers["forwarded"];
if (header) {
for (const directive of header.split(",")[0].split(";")) {
if (directive.startsWith("for=")) {
return directive.substring(4);
}
}
}

let addr = socket.handshake.headers["x-forwarded-for"]?.split(",")?.[0];
if (addr) {
return addr;
}
for (const other of ["cf-connecting-ip", "fastly-client-ip"]) {
addr = socket.handshake.headers[other];
if (addr) {
return addr;
}
}

return socket.handshake.address;
return getClientIpAddress(socket.handshake) ?? socket.handshake.address;
}

export function updateInterest(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ export const EmailAddressSetting = ({
return;
}
try {
// anonymouse users will get the "welcome" email
// anonymous users will get the "welcome" email
await webapp_client.account_client.send_verification_email(!is_anonymous);
} catch (error) {
const err_msg = `Problem sending welcome email: ${error}`;
Expand Down
24 changes: 13 additions & 11 deletions src/packages/frontend/user-tracking.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,31 +7,33 @@
// client code doesn't have to import webapp_client everywhere, and we can
// completely change this if we want.

import { query, server_time } from "./frame-editors/generic/client";
import { analytics_cookie_name as analytics, uuid } from "@cocalc/util/misc";
import { redux } from "./app-framework";
import { redux } from "@cocalc/frontend/app-framework";
import {
query,
server_time,
} from "@cocalc/frontend/frame-editors/generic/client";
import { get_cookie } from "@cocalc/frontend/misc";
import { webapp_client } from "@cocalc/frontend/webapp-client";
import { uuid } from "@cocalc/util/misc";
import { version } from "@cocalc/util/smc-version";
import { get_cookie } from "./misc";
import { webapp_client } from "./webapp-client";

import { ANALYTICS_COOKIE_NAME } from "@cocalc/util/consts";

export async function log(eventName: string, payload: any): Promise<void> {
const central_log = {
id: uuid(),
event: `webapp-${eventName}`,
value: {
account_id: redux.getStore("account")?.get("account_id"),
analytics_cookie: get_cookie(analytics),
analytics_cookie: get_cookie(ANALYTICS_COOKIE_NAME),
cocalc_version: version,
...payload,
},
time: server_time(),
};

try {
await query({
query: {
central_log,
},
});
await query({ query: { central_log } });
} catch (err) {
console.warn("WARNING: Failed to write log event -- ", central_log);
}
Expand Down
15 changes: 9 additions & 6 deletions src/packages/hub/analytics-script.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,16 @@
* e.g. this filters the SSO auth pages, which are uninteresting referrals
*/

// variable PREFIX, NAME, DOMAIN and ID are injected in the hub's http server
declare var NAME, ID, DOMAIN, PREFIX, window, document;
// variable PREFIX, NAME, DOMAIN, ID, and ANALYTICS_ENABLED are injected in the hub's http server
declare var NAME, ID, DOMAIN, PREFIX, ANALYTICS_ENABLED, window, document;

// write cookie. it would be cool to set this via the http request itself,
// but for reasons I don't know it doesn't work across subdomains.
const maxage = 7 * 24 * 60 * 60; // 7 days
document.cookie = `${NAME}=${ID}; path=/; domain=${DOMAIN}; max-age=${maxage}`;
// write cookie only if analytics is enabled (for privacy in cookieless mode)
if (ANALYTICS_ENABLED) {
// it would be cool to set this via the http request itself,
// but for reasons I don't know it doesn't work across subdomains.
const maxage = 7 * 24 * 60 * 60; // 7 days
document.cookie = `${NAME}=${ID}; path=/; domain=${DOMAIN}; max-age=${maxage}`;
}

const { href, protocol, host, pathname } = window.location;

Expand Down
Loading