Skip to content

Commit 4899c9f

Browse files
brendan-kellambrendan-kellam
authored
feat(ee): GitLab permission syncing (#585)
1 parent 384aa9e commit 4899c9f

File tree

10 files changed

+211
-49
lines changed

10 files changed

+211
-49
lines changed

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
## Added
11+
- [Experimental][Sourcebot EE] Added GitLab permission syncing. [#585](https://github.com/sourcebot-dev/sourcebot/pull/585)
12+
1013
### Fixed
1114
- [ask sb] Fixed issue where reasoning tokens would appear in `text` content for openai compatible models. [#582](https://github.com/sourcebot-dev/sourcebot/pull/582)
1215
- Fixed issue with GitHub app token tracking and refreshing. [#583](https://github.com/sourcebot-dev/sourcebot/pull/583)

‎docs/docs/configuration/auth/providers.mdx‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,14 @@ Optional environment variables:
5252

5353
[Auth.js GitLab Provider Docs](https://authjs.dev/getting-started/providers/gitlab)
5454

55+
Authentication using GitLab is supported via a [OAuth2.0 app](https://docs.gitlab.com/integration/oauth_provider/#create-an-instance-wide-application) installed on the GitLab instance. Follow the instructions in the [GitLab docs](https://docs.gitlab.com/integration/oauth_provider/) to create an app. The callback URL should be configurd to `<sourcebot_deployment_url>/api/auth/callback/gitlab`, and the following scopes need to be set:
56+
57+
| Scope | Required | Notes |
58+
|------------|----------|----------------------------------------------------------------------------------------------------|
59+
| read_user | Yes | Allows Sourcebot to read basic user information required for authentication. |
60+
| read_api | Conditional | Required **only** when [permission syncing](/docs/features/permission-syncing) is enabled. Enables Sourcebot to list all repositories and projects for the authenticated user. |
61+
62+
5563
**Required environment variables:**
5664
- `AUTH_EE_GITLAB_CLIENT_ID`
5765
- `AUTH_EE_GITLAB_CLIENT_SECRET`

‎docs/docs/features/permission-syncing.mdx‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ We are actively working on supporting more code hosts. If you'd like to see a sp
3535
| Platform | Permission syncing |
3636
|:----------|------------------------------|
3737
| [GitHub (GHEC & GHEC Server)](/docs/features/permission-syncing#github) | ✅ |
38-
| GitLab | 🛑 |
38+
| [GitLab (Self-managed & Cloud)](/docs/features/permission-syncing#gitlab) | ✅ |
3939
| Bitbucket Cloud | 🛑 |
4040
| Bitbucket Data Center | 🛑 |
4141
| Gitea | 🛑 |
@@ -59,6 +59,18 @@ Permission syncing works with **GitHub.com**, **GitHub Enterprise Cloud**, and *
5959
- A GitHub OAuth provider must be configured to (1) correlate a Sourcebot user with a GitHub user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
6060
- OAuth tokens must assume the `repo` scope in order to use the [List repositories for the authenticated user API](https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#list-repositories-for-the-authenticated-user) during [User driven syncing](/docs/features/permission-syncing#how-it-works). Sourcebot **will only** use this token for **reads**.
6161

62+
## GitLab
63+
64+
Prerequisite: [Add GitLab as an OAuth provider](/docs/configuration/auth/providers#gitlab).
65+
66+
Permission syncing works with **GitLab Self-managed** and **GitLab Cloud**. Users with **Guest** role or above with membership to a group or project will have their access synced to Sourcebot. Both direct and indirect membership to a group or project will be synced with Sourcebot. For more details, see the [GitLab docs](https://docs.gitlab.com/user/project/members/#membership-types).
67+
68+
69+
**Notes:**
70+
- A GitLab OAuth provider must be configured to (1) correlate a Sourcebot user with a GitLab user, and (2) to list repositories that the user has access to for [User driven syncing](/docs/features/permission-syncing#how-it-works).
71+
- OAuth tokens require the `read_api` scope in order to use the [List projects for the authenticated user API](https://docs.gitlab.com/ee/api/projects.html#list-all-projects) during [User driven syncing](/docs/features/permission-syncing#how-it-works).
72+
73+
6274
# How it works
6375

6476
Permission syncing works by periodically syncing ACLs from the code host(s) to Sourcebot to build an internal mapping between Users and Repositories. This mapping is hydrated in two directions:

‎packages/backend/src/constants.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ export const SINGLE_TENANT_ORG_ID = 1;
55

66
export const PERMISSION_SYNC_SUPPORTED_CODE_HOST_TYPES = [
77
'github',
8+
'gitlab',
89
];
910

1011
export const REPOS_CACHE_DIR = path.join(env.DATA_CACHE_DIR, 'repos');

‎packages/backend/src/ee/repoPermissionSyncer.ts‎

Lines changed: 55 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import { Redis } from 'ioredis';
77
import { PERMISSION_SYNC_SUPPORTED_CODE_HOST_TYPES } from "../constants.js";
88
import { env } from "../env.js";
99
import { createOctokitFromToken, getRepoCollaborators, GITHUB_CLOUD_HOSTNAME } from "../github.js";
10+
import { createGitLabFromPersonalAccessToken, getProjectMembers } from "../gitlab.js";
1011
import { Settings } from "../types.js";
1112
import { getAuthCredentialsForRepo } from "../utils.js";
1213

@@ -16,7 +17,9 @@ type RepoPermissionSyncJob = {
1617

1718
const QUEUE_NAME = 'repoPermissionSyncQueue';
1819

19-
const logger = createLogger('repo-permission-syncer');
20+
const LOG_TAG = 'repo-permission-syncer';
21+
const logger = createLogger(LOG_TAG);
22+
const createJobLogger = (jobId: string) => createLogger(`${LOG_TAG}:job:${jobId}`);
2023

2124
export class RepoPermissionSyncer {
2225
private queue: Queue<RepoPermissionSyncJob>;
@@ -109,28 +112,31 @@ export class RepoPermissionSyncer {
109112
}
110113

111114
private async schedulePermissionSync(repos: Repo[]) {
112-
await this.db.$transaction(async (tx) => {
113-
const jobs = await tx.repoPermissionSyncJob.createManyAndReturn({
114-
data: repos.map(repo => ({
115-
repoId: repo.id,
116-
})),
117-
});
118-
119-
await this.queue.addBulk(jobs.map((job) => ({
120-
name: 'repoPermissionSyncJob',
121-
data: {
122-
jobId: job.id,
123-
},
124-
opts: {
125-
removeOnComplete: env.REDIS_REMOVE_ON_COMPLETE,
126-
removeOnFail: env.REDIS_REMOVE_ON_FAIL,
127-
}
128-
})))
115+
// @note: we don't perform this in a transaction because
116+
// we want to avoid the situation where a job is created and run
117+
// prior to the transaction being committed.
118+
const jobs = await this.db.repoPermissionSyncJob.createManyAndReturn({
119+
data: repos.map(repo => ({
120+
repoId: repo.id,
121+
})),
129122
});
123+
124+
await this.queue.addBulk(jobs.map((job) => ({
125+
name: 'repoPermissionSyncJob',
126+
data: {
127+
jobId: job.id,
128+
},
129+
opts: {
130+
removeOnComplete: env.REDIS_REMOVE_ON_COMPLETE,
131+
removeOnFail: env.REDIS_REMOVE_ON_FAIL,
132+
}
133+
})))
130134
}
131135

132136
private async runJob(job: Job<RepoPermissionSyncJob>) {
133137
const id = job.data.jobId;
138+
const logger = createJobLogger(id);
139+
134140
const { repo } = await this.db.repoPermissionSyncJob.update({
135141
where: {
136142
id,
@@ -194,6 +200,33 @@ export class RepoPermissionSyncer {
194200
},
195201
});
196202

203+
return accounts.map(account => account.userId);
204+
} else if (repo.external_codeHostType === 'gitlab') {
205+
const api = await createGitLabFromPersonalAccessToken({
206+
token: credentials.token,
207+
url: credentials.hostUrl,
208+
});
209+
210+
const projectId = repo.external_id;
211+
if (!projectId) {
212+
throw new Error(`Repo ${id} does not have an external_id`);
213+
}
214+
215+
const members = await getProjectMembers(projectId, api);
216+
const gitlabUserIds = members.map(member => member.id.toString());
217+
218+
const accounts = await this.db.account.findMany({
219+
where: {
220+
provider: 'gitlab',
221+
providerAccountId: {
222+
in: gitlabUserIds,
223+
}
224+
},
225+
select: {
226+
userId: true,
227+
},
228+
});
229+
197230
return accounts.map(account => account.userId);
198231
}
199232

@@ -221,6 +254,8 @@ export class RepoPermissionSyncer {
221254
}
222255

223256
private async onJobCompleted(job: Job<RepoPermissionSyncJob>) {
257+
const logger = createJobLogger(job.data.jobId);
258+
224259
const { repo } = await this.db.repoPermissionSyncJob.update({
225260
where: {
226261
id: job.data.jobId,
@@ -243,6 +278,8 @@ export class RepoPermissionSyncer {
243278
}
244279

245280
private async onJobFailed(job: Job<RepoPermissionSyncJob> | undefined, err: Error) {
281+
const logger = createJobLogger(job?.data.jobId ?? 'unknown');
282+
246283
Sentry.captureException(err, {
247284
tags: {
248285
jobId: job?.data.jobId,

‎packages/backend/src/ee/userPermissionSyncer.ts‎

Lines changed: 60 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,13 @@ import { Redis } from "ioredis";
66
import { PERMISSION_SYNC_SUPPORTED_CODE_HOST_TYPES } from "../constants.js";
77
import { env } from "../env.js";
88
import { createOctokitFromToken, getReposForAuthenticatedUser } from "../github.js";
9+
import { createGitLabFromOAuthToken, getProjectsForAuthenticatedUser } from "../gitlab.js";
910
import { hasEntitlement } from "@sourcebot/shared";
1011
import { Settings } from "../types.js";
1112

12-
const logger = createLogger('user-permission-syncer');
13+
const LOG_TAG = 'user-permission-syncer';
14+
const logger = createLogger(LOG_TAG);
15+
const createJobLogger = (jobId: string) => createLogger(`${LOG_TAG}:job:${jobId}`);
1316

1417
const QUEUE_NAME = 'userPermissionSyncQueue';
1518

@@ -110,28 +113,31 @@ export class UserPermissionSyncer {
110113
}
111114

112115
private async schedulePermissionSync(users: User[]) {
113-
await this.db.$transaction(async (tx) => {
114-
const jobs = await tx.userPermissionSyncJob.createManyAndReturn({
115-
data: users.map(user => ({
116-
userId: user.id,
117-
})),
118-
});
119-
120-
await this.queue.addBulk(jobs.map((job) => ({
121-
name: 'userPermissionSyncJob',
122-
data: {
123-
jobId: job.id,
124-
},
125-
opts: {
126-
removeOnComplete: env.REDIS_REMOVE_ON_COMPLETE,
127-
removeOnFail: env.REDIS_REMOVE_ON_FAIL,
128-
}
129-
})))
116+
// @note: we don't perform this in a transaction because
117+
// we want to avoid the situation where a job is created and run
118+
// prior to the transaction being committed.
119+
const jobs = await this.db.userPermissionSyncJob.createManyAndReturn({
120+
data: users.map(user => ({
121+
userId: user.id,
122+
})),
130123
});
124+
125+
await this.queue.addBulk(jobs.map((job) => ({
126+
name: 'userPermissionSyncJob',
127+
data: {
128+
jobId: job.id,
129+
},
130+
opts: {
131+
removeOnComplete: env.REDIS_REMOVE_ON_COMPLETE,
132+
removeOnFail: env.REDIS_REMOVE_ON_FAIL,
133+
}
134+
})))
131135
}
132136

133137
private async runJob(job: Job<UserPermissionSyncJob>) {
134138
const id = job.data.jobId;
139+
const logger = createJobLogger(id);
140+
135141
const { user } = await this.db.userPermissionSyncJob.update({
136142
where: {
137143
id,
@@ -183,6 +189,37 @@ export class UserPermissionSyncer {
183189
}
184190
});
185191

192+
repos.forEach(repo => aggregatedRepoIds.add(repo.id));
193+
} else if (account.provider === 'gitlab') {
194+
if (!account.access_token) {
195+
throw new Error(`User '${user.email}' does not have a GitLab OAuth access token associated with their GitLab account.`);
196+
}
197+
198+
const api = await createGitLabFromOAuthToken({
199+
oauthToken: account.access_token,
200+
url: env.AUTH_EE_GITLAB_BASE_URL,
201+
});
202+
203+
// @note: we only care about the private and internal repos since we don't need to build a mapping
204+
// for public repos.
205+
// @see: packages/web/src/prisma.ts
206+
const privateGitLabProjects = await getProjectsForAuthenticatedUser('private', api);
207+
const internalGitLabProjects = await getProjectsForAuthenticatedUser('internal', api);
208+
209+
const gitLabProjectIds = [
210+
...privateGitLabProjects,
211+
...internalGitLabProjects,
212+
].map(project => project.id.toString());
213+
214+
const repos = await this.db.repo.findMany({
215+
where: {
216+
external_codeHostType: 'gitlab',
217+
external_id: {
218+
in: gitLabProjectIds,
219+
}
220+
}
221+
});
222+
186223
repos.forEach(repo => aggregatedRepoIds.add(repo.id));
187224
}
188225
}
@@ -212,6 +249,8 @@ export class UserPermissionSyncer {
212249
}
213250

214251
private async onJobCompleted(job: Job<UserPermissionSyncJob>) {
252+
const logger = createJobLogger(job.data.jobId);
253+
215254
const { user } = await this.db.userPermissionSyncJob.update({
216255
where: {
217256
id: job.data.jobId,
@@ -234,6 +273,8 @@ export class UserPermissionSyncer {
234273
}
235274

236275
private async onJobFailed(job: Job<UserPermissionSyncJob> | undefined, err: Error) {
276+
const logger = createJobLogger(job?.data.jobId ?? 'unknown');
277+
237278
Sentry.captureException(err, {
238279
tags: {
239280
jobId: job?.data.jobId,
@@ -260,7 +301,7 @@ export class UserPermissionSyncer {
260301

261302
logger.error(errorMessage(user.email ?? user.id));
262303
} else {
263-
logger.error(errorMessage('unknown user (id not found)'));
304+
logger.error(errorMessage('unknown job (id not found)'));
264305
}
265306
}
266307
}

‎packages/backend/src/env.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ export const env = createEnv({
5656

5757
EXPERIMENT_EE_PERMISSION_SYNC_ENABLED: booleanSchema.default('false'),
5858
AUTH_EE_GITHUB_BASE_URL: z.string().optional(),
59+
AUTH_EE_GITLAB_BASE_URL: z.string().default("https://gitlab.com"),
5960
},
6061
runtimeEnv: process.env,
6162
emptyStringAsUndefined: true,

0 commit comments

Comments
 (0)