Disclosure: this bug was found and this report was written by Claude Code (Anthropic's CLI coding agent), while diagnosing a real production failure on my self-hosted instance. I am submitting it after reviewing it. I have deliberately left the "not generated by AI" checkbox unchecked rather than tick it inaccurately. Every claim below is backed by rows from my own database and by the 0.27.4 source; please judge it on that evidence.
What happened?
The doc gc / document-cleanup feature deletes AFFiNE's own internal workspace docs, because they are never listed as pages in the workspace root doc and are therefore treated as orphans.
On 2026-08-24 at 04:00:00–04:00:02 UTC, the first-ever document_cleanup_execute sweep on my instance deleted 19 docs across all 5 workspaces. 17 of the 19 were internal docs:
db$<ws>$docPropertiesdb$<ws>$foldersdb$<ws>$explorerIcondb$<ws>$docCustomPropertyInfouserdata$<user>$<ws>$settingsuserdata$<user>$<ws>$favorite
(The other 2 were genuinely orphaned page docs — the feature's legitimate target.)
User-visible consequences:
- Sidebar folder trees were lost in 4 workspaces (
db$<ws>$folders deleted).
- Journal template setting was lost (
userdata$…$settings holds templateDoc.value.journalTemplateId).
- Doc properties were lost in 4 workspaces.
snapshot_histories had 0 rows for the deleted docs, so there is no version history to restore from.
Expected: internal/reserved docs (db$…, userdata$…) should never be cleanup candidates. They are storage AFFiNE itself creates and depends on, and by design they never appear in the root doc's page list.
Root cause (0.27.4 source)
In packages/backend/native/src/runtime/storage_runtime/document_cleanup.rs:
load_stored_doc_activity() scans every stored guid, sparing only the workspace root:
SELECT guid AS doc_id, updated_at FROM snapshots WHERE workspace_id = $1
UNION ALL SELECT guid, created_at FROM updates WHERE workspace_id = $1
UNION ALL SELECT guid, timestamp FROM snapshot_histories WHERE workspace_id = $1
WHERE doc_id <> $1
Liveness is then decided purely by page membership in the root doc:
fn root_contains(root: CurrentDoc, doc_id: &str) -> RuntimeResult<bool> {
let projection = affine_doc_loader::project_workspace_root(root.blob, true)...;
Ok(projection.doc_ids.iter().any(|id| id == doc_id))
}reconcile_workspace() marks everything stored-but-not-a-live-page, and execute_document_cleanup_candidates deletes it after gracePeriodDays (default 30).
Internal docs are never in projection.doc_ids, so they are marked on every nightly run. Grepping that file for userdata, db$, prefix, starts_with, is_internal, exclude, reserved returns no matches — there is no allowlist or prefix filter at all.
Why it fired when it did
document_cleanup (mark_only) began running on my instance 2026-07-25 — consistent with #15282 (feat(server): impl doc gc, merged 2026-07-19). 07-25 + 30 days grace = 2026-08-24, the date of the first execute sweep. So instances that adopted doc gc around then are reaching the deletion boundary now.
It recurs
missing_since only resets when the doc sees activity, and internal docs are re-marked nightly. So any internal doc that goes 30 days without a write is deleted again. After I recreated a folder, its clock simply restarted.
The client does not heal it
After db$<ws>$folders was deleted, my browser tab was still connected, so it pushed only incremental ops against a parent that no longer existed server-side. Decoding the resulting snapshot shows the folder's name as a live but orphaned item with an unresolvable parent:
4522778557396753:0-6 parent=iKaPhGpcOnekUzNW4gyfo ... $$DELETED=true
2985492552941628:0 parent=- del=false ["Network Redesign_"] <- orphan, never materialises
A page reload did not re-create the missing parent node. Net effect: the folder still renders from browser-local state, but the server copy is permanently unusable — so this is silent, and users will only notice on a new browser or device.
Suggested fix
Exclude reserved doc-id prefixes (db$, userdata$) from the candidate scan in load_stored_doc_activity() / reconcile_workspace(), or make the live set include workspace-scoped internal docs rather than pages only.
Not fixed on canary
I diffed document_cleanup.rs between v0.27.4 and canary (2026-08-25): the changes are refactoring (function signatures, merge_current_doc arity, doc_blob_ref_projections added to the delete list). The same grep for internal-doc filtering still returns nothing, so v0.27.5-beta.0 does not address this.
No way to disable or tune it
In packages/backend/server/src/core/storage/blob-job.ts, gracePeriodDays = 30 is a hardcoded handler default and the cron enqueues with an empty payload:
@Cron(CronExpression.EVERY_DAY_AT_4AM)
… queue.add('backendRuntime.executeDocumentCleanupCandidates', {})
async executeDocumentCleanupCandidates({ workspaceId, gracePeriodDays = 30, limit = 100 })There is no config binding, and nothing in app_configs for it — so self-hosters cannot opt out while waiting for a fix.
Environment
- Distribution version: Linux (self-hosted, Docker,
ghcr.io/toeverything/affine:0.27.4)
- Self-hosting: Yes
- Self-hosting version: 0.27.4
- Database: external PostgreSQL
Relevant log output
storage_reconciliation_runs — the first document_cleanup_execute rows ever recorded on this instance:
started_at kind workspace_id doc_id
2026-08-24T04:00:02.514Z document_cleanup_execute fe07bcd8… userdata$ae5e65c2…$fe07bcd8…$settings
2026-08-24T04:00:02.475Z document_cleanup_execute fe07bcd8… db$fe07bcd8…$folders
2026-08-24T04:00:02.348Z document_cleanup_execute 2cf4d0c0… userdata$ae5e65c2…$2cf4d0c0…$favorite
2026-08-24T04:00:02.268Z document_cleanup_execute 2cf4d0c0… db$2cf4d0c0…$folders
2026-08-24T04:00:02.169Z document_cleanup_execute 2cf4d0c0… db$2cf4d0c0…$explorerIcon
2026-08-24T04:00:02.082Z document_cleanup_execute 2cf4d0c0… db$2cf4d0c0…$docProperties
2026-08-24T04:00:01.715Z document_cleanup_execute edecd35c… userdata$ae5e65c2…$edecd35c…$settings
2026-08-24T04:00:01.391Z document_cleanup_execute edecd35c… db$edecd35c…$docProperties
2026-08-24T04:00:00.702Z document_cleanup_execute 6bc3499a… db$6bc3499a…$docCustomPropertyInfo
Row counts for the deleted settings doc — note snapshot_histories: 0, i.e. unrecoverable:
{"snapshots":1,"doc_grants":1,"updates":0,"snapshot_histories":0,"workspace_pages":0,
"doc_access_policies":0,"doc_blob_refs":0,"comments":0,"replies":0}Aggregate confirming this was the first execute sweep, after a month of mark-only runs:
kind mode n first last
document_cleanup mark_only 155 2026-07-25T02:00:00.247Z 2026-08-24T02:00:06.499Z
document_cleanup_execute execute 19 2026-08-24T04:00:00.201Z 2026-08-24T04:00:02.514Z
What happened?
The
doc gc/ document-cleanup feature deletes AFFiNE's own internal workspace docs, because they are never listed as pages in the workspace root doc and are therefore treated as orphans.On 2026-08-24 at 04:00:00–04:00:02 UTC, the first-ever
document_cleanup_executesweep on my instance deleted 19 docs across all 5 workspaces. 17 of the 19 were internal docs:db$<ws>$docPropertiesdb$<ws>$foldersdb$<ws>$explorerIcondb$<ws>$docCustomPropertyInfouserdata$<user>$<ws>$settingsuserdata$<user>$<ws>$favorite(The other 2 were genuinely orphaned page docs — the feature's legitimate target.)
User-visible consequences:
db$<ws>$foldersdeleted).userdata$…$settingsholdstemplateDoc.value.journalTemplateId).snapshot_historieshad 0 rows for the deleted docs, so there is no version history to restore from.Expected: internal/reserved docs (
db$…,userdata$…) should never be cleanup candidates. They are storage AFFiNE itself creates and depends on, and by design they never appear in the root doc's page list.Root cause (0.27.4 source)
In
packages/backend/native/src/runtime/storage_runtime/document_cleanup.rs:load_stored_doc_activity()scans every stored guid, sparing only the workspace root:Liveness is then decided purely by page membership in the root doc:
reconcile_workspace()marks everything stored-but-not-a-live-page, andexecute_document_cleanup_candidatesdeletes it aftergracePeriodDays(default 30).Internal docs are never in
projection.doc_ids, so they are marked on every nightly run. Grepping that file foruserdata,db$,prefix,starts_with,is_internal,exclude,reservedreturns no matches — there is no allowlist or prefix filter at all.Why it fired when it did
document_cleanup(mark_only) began running on my instance 2026-07-25 — consistent with #15282 (feat(server): impl doc gc, merged 2026-07-19). 07-25 + 30 days grace = 2026-08-24, the date of the first execute sweep. So instances that adopted doc gc around then are reaching the deletion boundary now.It recurs
missing_sinceonly resets when the doc sees activity, and internal docs are re-marked nightly. So any internal doc that goes 30 days without a write is deleted again. After I recreated a folder, its clock simply restarted.The client does not heal it
After
db$<ws>$folderswas deleted, my browser tab was still connected, so it pushed only incremental ops against a parent that no longer existed server-side. Decoding the resulting snapshot shows the folder's name as a live but orphaned item with an unresolvable parent:A page reload did not re-create the missing parent node. Net effect: the folder still renders from browser-local state, but the server copy is permanently unusable — so this is silent, and users will only notice on a new browser or device.
Suggested fix
Exclude reserved doc-id prefixes (
db$,userdata$) from the candidate scan inload_stored_doc_activity()/reconcile_workspace(), or make the live set include workspace-scoped internal docs rather than pages only.Not fixed on canary
I diffed
document_cleanup.rsbetweenv0.27.4andcanary(2026-08-25): the changes are refactoring (function signatures,merge_current_docarity,doc_blob_ref_projectionsadded to the delete list). The same grep for internal-doc filtering still returns nothing, sov0.27.5-beta.0does not address this.No way to disable or tune it
In
packages/backend/server/src/core/storage/blob-job.ts,gracePeriodDays = 30is a hardcoded handler default and the cron enqueues with an empty payload:There is no config binding, and nothing in
app_configsfor it — so self-hosters cannot opt out while waiting for a fix.Environment
ghcr.io/toeverything/affine:0.27.4)Relevant log output
storage_reconciliation_runs— the firstdocument_cleanup_executerows ever recorded on this instance:Row counts for the deleted settings doc — note
snapshot_histories: 0, i.e. unrecoverable:{"snapshots":1,"doc_grants":1,"updates":0,"snapshot_histories":0,"workspace_pages":0, "doc_access_policies":0,"doc_blob_refs":0,"comments":0,"replies":0}Aggregate confirming this was the first execute sweep, after a month of mark-only runs: