diff --git a/docs/adr/gateway-capabilities-and-delivery-semantics.md b/docs/adr/gateway-capabilities-and-delivery-semantics.md new file mode 100644 index 000000000..51760e5b3 --- /dev/null +++ b/docs/adr/gateway-capabilities-and-delivery-semantics.md @@ -0,0 +1,241 @@ +# ADR: Gateway Capabilities and Delivery Semantics + +- **Status:** Proposed +- **Date:** 2026-08-06 +- **Author:** @NeoHsu +- **Related:** + - [Custom Gateway](custom-gateway.md) + - [Unified Binary](unified-binary.md) + - [Multi-Platform Adapters](multi-platform-adapters.md) + - [Teams bot-owned message mutations](teams-owned-message-mutations.md) + - [Teams message reactions preview](teams-message-reactions-preview.md) + +--- + +## Context + +OpenAB has two paths for webhook-based chat platforms: + +1. **Unified:** platform adapters and Core run in one process. +2. **Standalone:** Core connects to `openab-gateway` through `/ws`. + +Before this decision, Core inferred behavior from adapter-wide methods and platform-name allowlists. That caused three classes of error: + +- a shared Unified adapter could apply Telegram streaming settings to Teams; +- Standalone Core could not know whether a Gateway acknowledged send, edit, or delete operations; +- a transport timeout could not be distinguished from an explicit platform rejection. + +The Standalone event channel is a bounded in-process broadcast channel. It has no durable inbox, replay, shared deduplication store, or consumer-group semantics. Multiple Core consumers therefore receive duplicate events rather than distributed work. + +## Decision + +### 1. Proposed baseline product decisions + +The following decisions define this proposed single-process baseline: + +| ID | Decision | +| --- | --- | +| D1 | The baseline supports one process replica and one active Standalone Core consumer per platform. Ingress after local enqueue is best-effort; there is no crash replay or exactly-once claim. A second consumer is warned at high severity and reported as unsupported, but is not rejected in this backward-compatible release. | +| D2 | Standalone uses an optional, client-initiated capability handshake. A peer that does not complete a supported handshake remains in legacy mode; missing ACKs are not delivery failures in legacy mode. | +| D3 | `GatewayEvent.event_id` is correlation metadata, never a platform activity ID. New↔new create/send returns the real platform message ID. Teams client presentation is outside transport acknowledgement semantics. | +| D4 | Teams supports Microsoft commercial public cloud only. Sovereign-cloud and custom-proxy endpoints require an explicit future cloud profile. | +| D5 | User-visible status and content streaming are independent capabilities. Teams processing status must not reuse a streaming placeholder implicitly. | + +### 2. Platform-aware capability contract + +Each adapter exposes capabilities for the actual `ChannelRef.platform`: + +```rust +struct AdapterCapabilities { + send_ack: bool, + edit_ack: bool, + delete_ack: bool, + supports_target_message_id: bool, + supports_reactions: bool, + can_edit: bool, + can_delete: bool, + streaming_mode: StreamingMode, + show_streaming_placeholder: bool, + message_limit: MessageLimit, + status_backend: StatusBackend, +} +``` + +Capability defaults fail closed: + +- no required ACK; +- no additive command-target field or native reaction support; +- no edit or delete support; +- streaming disabled; +- status side effects disabled; +- a conservative 4,096-character message limit. + +Direct adapters derive a backward-compatible capability view from their existing methods. Unified and Standalone shared adapters override it by platform. + +A valid negotiated hello is authoritative. If a platform is omitted from a valid hello, Core uses fail-closed defaults rather than optimistic legacy behavior. Legacy behavior is used only before a supported hello is accepted. + +### 3. Optional Standalone hello exchange + +Core sends this additive control frame immediately after connecting: + +```json +{ + "schema": "openab.gateway.client_hello.v1", + "protocol_version": 1, + "client_name": "openab-core/", + "requested_platforms": ["teams"] +} +``` + +A new Gateway responds: + +```json +{ + "schema": "openab.gateway.hello.v1", + "protocol_version": 1, + "capabilities": { + "teams": { + "send_ack": true, + "edit_ack": true, + "delete_ack": true, + "supports_target_message_id": true, + "supports_reactions": false, + "can_edit": true, + "can_delete": true, + "streaming_mode": "disabled", + "show_streaming_placeholder": true, + "message_limit": { "unit": "characters", "max": 4096 }, + "status_backend": "none" + } + }, + "topology": { + "active_consumers": 1, + "supported": true, + "delivery_mode": "best_effort_broadcast" + } +} +``` + +Rules: + +- unknown JSON fields are additive and may be ignored; +- an empty `requested_platforms` list requests all configured adapters; the stock Core uses this because one Standalone socket can carry events from several platforms; +- protocol version mismatch, malformed hello, or no hello keeps Core in legacy mode; +- Gateway continues to accept `openab.gateway.reply.v1` as the first frame, so an old Core works with a new Gateway; +- a new Core may send `client_hello` to an old Gateway; the old Gateway may log it as an invalid reply but must keep the connection usable; +- operations emitted before a valid hello is processed use legacy semantics; +- control frames are prioritized over broadcast events once received. + +Recommended Standalone rollout order remains Gateway first, then Core, but either side may be upgraded first. + +### 4. Structured write outcome + +Gateway keeps the existing `openab.gateway.response.v1` fields and adds optional fields: + +- `outcome`: `delivered`, `rejected`, or `unknown`; +- `error_code`; +- `retry_after_ms`. + +The internal result is: + +```rust +enum WriteOutcome { + Delivered { message_id: Option }, + Rejected { + code: String, + message: String, + retry_after_ms: Option, + }, + Unknown { code: String, message: String }, +} +``` + +Semantics: + +- create/send delivery requires a non-empty real message ID when that operation advertises required ACK support; +- edit/delete delivery does not require a message ID in its ACK; +- explicit platform refusal is `Rejected`; +- an ambiguous POST timeout or disconnect is `Unknown` and must not be retried blindly; +- legacy responses without `outcome` map from the existing `success`, `message_id`, and `error` fields; +- Core waits only for an operation whose capability advertises the corresponding ACK; +- Teams advertises `send_ack = true` only after its event-route send path emits a terminal structured response with a non-empty Bot Framework activity ID on delivery; +- Teams advertises edit/delete ACK and `supports_target_message_id` only after bot-owned mutation enforcement emits a terminal response on every command path; +- Teams advertises `supports_reactions = true` and `status_backend = reactions` only under the explicit public-preview `reactions_enabled` opt-in; the default remains false/`none`; +- `supports_reactions` is independent from the selected progress backend so permanent batch receipts can coexist with a processing message; new Core normalizes an old peer's `status_backend = reactions` to reaction support; +- configured Teams processing messages are selected Core-side only after a valid hello advertises required send/edit/delete ACKs, additive command targets, and bot-owned edit/delete; no valid hello means no message status; +- negotiated required ACK timeout defaults to 12 seconds and is configurable as `[gateway].gateway_ack_timeout_secs`; +- configuration rejects zero, a budget at or above `pool.prompt_hard_timeout_secs`, and a Teams budget at or below the 10-second Connector timeout; +- legacy response waits preserve their previous best-effort behavior. + +The 12-second Gateway budget must remain greater than the Teams Bot Connector request timeout (10 seconds) and less than the ACP turn hard timeout. + +### 5. Topology guardrail + +Each Gateway process counts active `/ws` Core consumers: + +- one consumer: `topology.supported = true`; +- more than one: emit an error-level log and return `topology.supported = false` with `delivery_mode = "best_effort_broadcast"`; +- disconnect decrements the count through a drop guard, including task cancellation paths. + +This detects unsupported fan-out inside one Gateway process. It cannot detect multiple independent Gateway replicas because the baseline deliberately has no shared state. The Helm deployments therefore remain fixed at `replicas: 1` with `Recreate` strategy. External deployments must follow the same constraint. + +Rejecting the second consumer would be a breaking change and is deferred. + +## Compatibility Matrix + +| Core | Gateway | Behavior | +| --- | --- | --- | +| old | old | Existing protocol and fire-and-forget behavior. | +| old | new | Gateway accepts a reply without hello; additive response fields are ignored. | +| new | old | Core sends optional hello, receives none, and stays in legacy mode; missing ACK is not failure. | +| new | new | Valid hello enables platform-aware capabilities, operation-specific required ACKs, structured outcomes, and topology reporting. | + +## Security and Reliability Boundaries + +- Capability negotiation is not authentication or authorization. Existing WebSocket token, platform webhook authentication, tenant checks, L2 scope, and L3 identity gates remain authoritative. +- Hello frames contain no platform credentials, service URLs, route records, or user identifiers. +- Advertising a capability does not make an operation safe by itself; the adapter must emit the corresponding ACK on every terminal path before the flag is enabled. +- `Unknown` preserves ambiguity instead of creating duplicates through automatic retry. +- This baseline does not claim durable enqueue, replay, duplicate-safe multi-consumer operation, or exactly-once delivery. + +## Consequences + +### Positive + +- Teams no longer inherits generic Gateway or Telegram streaming/status behavior; reaction availability, processing-message selection, and progressive content each require their own explicit opt-in and capability gate. +- Core no longer needs write-path platform allowlists such as `EDIT_RESPONSE_PLATFORMS`. +- New platform features can be introduced additively without forcing a lockstep Core/Gateway deployment. +- Operators and Core can identify unsupported multi-consumer topology. +- Delivery uncertainty is represented explicitly and can be handled without unsafe retry. + +### Negative + +- Capability DTOs are mirrored in Core and Gateway and require wire-compatibility tests. +- The first operation may use legacy behavior if it races ahead of hello processing. +- Existing adapters that cannot return a stable message ID must advertise conservative send-once behavior until their delivery path is upgraded. +- Cross-replica topology remains undetectable without a shared coordination system. + +## Alternatives Rejected + +1. **Platform-name allowlists in Core.** Rejected because they drift whenever an adapter changes behavior and cannot represent deployment-specific support. +2. **Mandatory hello before accepting replies.** Rejected because it breaks old Core deployments during rolling upgrade. +3. **Treat every timeout as rejection.** Rejected because the platform may have committed an ambiguous POST. +4. **Retry ambiguous POST automatically.** Rejected because it can duplicate user-visible activities. +5. **Reject the second consumer immediately.** Deferred because this is a breaking operational change. +6. **Claim HA from multiple broadcast consumers.** Rejected because broadcast fan-out is not work distribution and has no shared idempotency state. + +## Verification + +Automated verification must cover: + +- capability DTO defaults and wire round trips; +- all three structured outcomes plus legacy response decoding; +- old Core→new Gateway reply without hello; +- new Core→old Gateway legacy fallback; +- new↔new capability selection; +- requested-platform filtering; +- second-consumer unsupported topology and disconnect decrement; +- Unified Teams isolation from Telegram streaming settings; +- additive reaction-support decoding and processing-message fail-closed capability selection; +- Teams progressive-response selection only under explicit opt-in plus every required write primitive, in Standalone and Unified modes; +- configurable 12-second ACK default. diff --git a/docs/adr/teams-ephemeral-ingress-state.md b/docs/adr/teams-ephemeral-ingress-state.md new file mode 100644 index 000000000..28dd9f7b1 --- /dev/null +++ b/docs/adr/teams-ephemeral-ingress-state.md @@ -0,0 +1,151 @@ +# ADR: Teams Ephemeral Ingress Route and Duplicate Suppression + +- **Status:** Proposed +- **Date:** 2026-08-07 +- **Author:** @NeoHsu +- **Related:** + - [Gateway capability and delivery semantics](gateway-capabilities-and-delivery-semantics.md) + - [Custom Gateway](custom-gateway.md) + - [Teams real send acknowledgement](teams-real-send-acknowledgement.md) + - [Teams bot-owned message mutations](teams-owned-message-mutations.md) + +--- + +## Context + +Bot Framework may retry the same webhook activity. Before this decision, the +Teams adapter published every authenticated retry, keyed reply routing only by +conversation ID, accepted messages with missing routing identifiers, and +ignored the result of the local Gateway broadcast. An HTTP 200 could therefore +mean that no Core consumer received the event. + +The proposed delivery boundary remains deliberately narrow: + +- one Gateway process replica; +- one supported Standalone Core consumer; +- process-local state only; +- no crash replay, durable inbox, shared idempotency store, or exactly-once + claim. + +## Decision + +### Required message fields + +A message activity proceeds only when it contains non-empty values for: + +- Bot Framework channel ID; +- tenant ID; +- conversation ID; +- activity ID; +- sender ID; +- service URL. + +Structural presence is checked before JWT key lookup; route and dedupe state are +created only after JWT and tenant authorization. The service URL must also pass +the Microsoft commercial public-cloud endpoint policy. Invalid or missing +fields return HTTP 400 and do not create route or dedupe state. Non-message and +structurally valid empty-text activities retain their existing HTTP 200 ignore +behavior. + +The adapter also parses optional Bot Framework `replyToId`, Team ID, and channel +ID into gateway-local route state. These values are not sent to the agent. + +### Composite identity + +Both route correlation and duplicate suppression use the composite identity: + +```text +(app_id, tenant_id, conversation_id, activity_id) +``` + +This prevents an activity ID collision from crossing applications, tenants, or +conversations. A generated `GatewayEvent.event_id` is a separate correlation +index for the future outbound route lookup. It is never a Bot Framework +activity ID. + +### Publication state machine + +Each composite key follows: + +```text +Vacant -> Publishing -> Accepted + | + +-> local publish failure -> Vacant +``` + +The first request owns publication. A concurrent duplicate that observes +`Publishing` waits on the same in-process completion signal. It returns HTTP +200 only if the owner reaches `Accepted`; otherwise it returns HTTP 503. + +An `Accepted` duplicate returns HTTP 200 without publishing another +`GatewayEvent`. A failed local broadcast removes the publishing entry before +returning HTTP 503, so a later Bot Framework retry may publish again. + +The Gateway checks the result of `broadcast::Sender::send` rather than a +separate receiver-count preflight, avoiding a check-then-send race. A successful +local broadcast is the process-local acknowledgement boundary; it does not prove ACP or +outbound completion. + +### Bounded process-local state + +Three positive settings control state: + +| Setting | Default | Purpose | +| --- | ---: | --- | +| `teams.dedupe_ttl_secs` | 600 | Accepted duplicate suppression window | +| `teams.route_ttl_secs` | 3600 | Authenticated ephemeral route lifetime | +| `teams.max_route_entries` | 10000 | Independent capacity bound for route and dedupe maps; bot-owned mutation state uses the same independent bound | + +Equivalent Standalone environment variables are +`TEAMS_DEDUPE_TTL_SECS`, `TEAMS_ROUTE_TTL_SECS`, and +`TEAMS_MAX_ROUTE_ENTRIES`. + +Expired entries are removed during reservation and by a shared background +sweeper used in both Standalone and Unified mode. At capacity, the oldest +accepted dedupe entry or route may be evicted with a warning. Active +`Publishing` entries are never evicted to admit another key; saturation returns +HTTP 503. A stale publishing owner is failed after a bounded internal timeout so +waiters cannot remain blocked indefinitely. + +The initial route implementation retained the legacy conversation-to-service-URL +cache for the existing outbound path. The real-send implementation removed that +compatibility cache: outbound sends +now resolve the authenticated route directly by `event_id` and verify the +reply's conversation before using its gateway-local service URL. + +## Security and privacy + +- Service URLs remain gateway-local and are excluded from Gateway wire events, + agent prompts, response payloads, and logs. +- Endpoint validation happens before route persistence. +- Logs may include tenant, conversation, sender, and validated service host, + but never the full service URL or app secret. +- Route state is not promoted to a proactive conversation reference. +- Duplicate suppression occurs only after JWT and tenant checks; unauthenticated + input cannot poison the cache. + +## Compatibility + +This is a correctness change for malformed or undeliverable Teams webhooks: + +| Condition | Previous behavior | New behavior | +| --- | --- | --- | +| Missing required route field | Often HTTP 200 | HTTP 400 | +| Accepted duplicate | Published again | HTTP 200 without republish | +| No local event consumer | HTTP 200 | HTTP 503, no tombstone | +| Local publish succeeds | HTTP 200 | HTTP 200 and route accepted | + +No Gateway wire field is removed or made mandatory. Unified and Standalone use +the same adapter state machine. Existing configuration remains valid through +the documented defaults. + + +## Consequences + +- Duplicate suppression does not survive restart and does not span replicas. +- Capacity eviction may shorten the effective dedupe window under sustained + overload; warnings make this visible. +- A successful broadcast can still be lost after process failure or consumer + lag. Durable delivery requires a later inbox/outbox design. +- [Real send acknowledgement](teams-real-send-acknowledgement.md) uses this + route for real activity IDs and outbound correlation. diff --git a/docs/adr/teams-message-reactions-preview.md b/docs/adr/teams-message-reactions-preview.md new file mode 100644 index 000000000..2012ca64d --- /dev/null +++ b/docs/adr/teams-message-reactions-preview.md @@ -0,0 +1,133 @@ +# ADR: Teams Public-Preview Message Reactions + +- **Status:** Proposed +- **Date:** 2026-08-09 +- **Author:** @NeoHsu +- **Related:** + - [Gateway capability and delivery semantics](gateway-capabilities-and-delivery-semantics.md) + - [Teams ephemeral ingress state](teams-ephemeral-ingress-state.md) + - [Teams bot-owned message mutations](teams-owned-message-mutations.md) + +--- + +## Context + +Microsoft's Teams SDK exposes public-preview add/remove reaction operations on +the Bot Connector conversation API. They use the authenticated Bot Framework +`serviceUrl`, conversation ID, activity ID, and bot token already required for +normal sends. They do not require Microsoft Graph, RSC, a delegated user token, +or a new manifest permission. + +OpenAB previously treated `add_reaction` and `remove_reaction` as successful +no-ops for Teams. Enabling the preview by default would change existing +behavior, create new status side effects, and rely on a tenant feature that is +not yet generally available. + +## Decision + +### Explicit opt-in + +Add this first-class setting: + +```toml +[teams] +reactions_enabled = false +``` + +The environment fallback is `TEAMS_REACTIONS_ENABLED`. Only `true` or `1` +enables the preview. Missing, false, zero, empty, or invalid values remain +fail-closed at `false`. + +When disabled: + +- reaction commands preserve the legacy successful no-op; +- no route lookup, token request, or Connector write occurs; +- Teams advertises `status_backend = none` to a negotiated Core. + +When enabled, Standalone and Unified advertise `supports_reactions = true`. +They select `status_backend = reactions` unless Core explicitly selects a +separate processing-message backend. In that combined mode, reactions provide +only permanent queued receipts while a turn-local message provides transient +progress. Streaming remains disabled and reaction support does not enable any +other Teams capability. + +### Bot Connector operations + +OpenAB uses the existing Bot Framework token and validated public-cloud +`serviceUrl`: + +```text +PUT {serviceUrl}/v3/conversations/{conversationId}/activities/{activityId}/reactions/{reactionType} +DELETE {serviceUrl}/v3/conversations/{conversationId}/activities/{activityId}/reactions/{reactionType} +``` + +All path values are appended as URL path segments. Empty reaction writes send +`Content-Length: 0`; the Connector otherwise rejects PUT requests that have +neither content length nor chunked framing. The +existing same-origin redirect policy, timeout, bounded error body, token +redaction, and commercial public-cloud endpoint policy apply unchanged. + +Reaction writes share the idempotent PUT/DELETE outcome classifier. HTTP +success is `Delivered`; explicit 3xx/4xx is `Rejected`; timeout, disconnect, or +5xx is `Unknown`. An explicit `429` with `Retry-After` no greater than one +second receives at most one internal retry. A bounded retry emits a warning +containing only the static operation name and `retry_after_ms`; it does not log +the Connector URL, conversation, activity ID, or token. There is no +POST/fresh-send fallback. + +### Scope and target trust + +A reaction target may be: + +- an authenticated inbound activity retained in the process-local route index; +- the authenticated reply-chain root of the command's origin route; or +- a confirmed bot-owned activity in the process-local ownership index. + +New-field commands require a live origin event route and cannot cross app, +tenant, or conversation scope. Legacy commands without a separate origin are +accepted only when app, conversation, and activity resolve to one unique route; +cross-tenant ambiguity fails closed. + +Reaction writes use the same fixed tenant/conversation write shards as sends, +edits, and deletes, with route state revalidated after lock acquisition. + +### Reaction IDs + +Core emits Unicode status emoji, while the Connector preview expects Teams +reaction IDs. OpenAB maps all default status and completion emoji to IDs from +the Microsoft Teams reactions reference. The generic controller's hard-coded +soft- and hard-stall states are included: `🥱` maps to +`1f971_yawningface`, while `😨` maps to the distinct `fearful` ID so a later +`😱` / `screamingfear` error swap cannot add and remove the same reaction. A +configured value may also be an ASCII reaction ID containing only letters, +digits, `_`, or `-`, up to 128 bytes. Unknown Unicode and unsafe identifiers +are rejected before HTTP. + +### Deliberate exclusions + +This preview slice does not: + +- process inbound `messageReaction` activities; +- add Graph or RSC permissions; +- add reaction-specific required ACK negotiation; +- promise availability outside Microsoft commercial public cloud; +- make native reactions part of the default processing-indicator contract; +- persist route evidence across restart or replicas. + + +## Consequences + +### Positive + +- Operators can test visible Teams reactions without widening Graph authority. +- Existing deployments remain unchanged until explicit opt-in. +- Reactions cannot be aimed at arbitrary activity IDs or leak `serviceUrl`. +- Default OpenAB status emoji work without operator-specific mapping. + +### Negative + +- Preview availability and rendering remain tenant-dependent. +- Rapid generic status transitions may encounter the platform's reaction rate + limit; bounded retries do not guarantee every cosmetic transition is shown. +- Inbound user reactions remain ignored until a separate behavior and trust + contract is approved. diff --git a/docs/adr/teams-owned-message-mutations.md b/docs/adr/teams-owned-message-mutations.md new file mode 100644 index 000000000..94e0357a3 --- /dev/null +++ b/docs/adr/teams-owned-message-mutations.md @@ -0,0 +1,229 @@ +# ADR: Teams Bot-Owned Message Mutations + +- **Status:** Proposed +- **Date:** 2026-08-09 +- **Author:** @NeoHsu +- **Related:** + - [Teams real send acknowledgement](teams-real-send-acknowledgement.md) + - [Teams ephemeral ingress state](teams-ephemeral-ingress-state.md) + - [Gateway capability and delivery semantics](gateway-capabilities-and-delivery-semantics.md) + - [Teams public-preview message reactions](teams-message-reactions-preview.md) + +--- + +## Context + +The [real-send decision](teams-real-send-acknowledgement.md) returns the Bot +Framework activity ID for every confirmed Teams send. +That ID is required for Bot Connector update and delete APIs, but accepting an +arbitrary activity ID from Core would allow OpenAB to attempt mutation of an +inbound user message or a bot message from another tenant or conversation. + +The existing `GatewayReply.reply_to` field is overloaded by older command +paths: normal sends use it as the origin OpenAB event ID, while edit and delete +commands historically place the platform message target in it. Keeping that +overload for new peers would again risk passing an OpenAB event ID to a Bot +Connector activity URL. + +The reliability boundary remains single-process and process-local. It does not +provide durable ownership, cross-replica coordination, or mutation of messages +created before restart. + +## Decision + +### Additive command target + +`openab.gateway.reply.v1` gains an optional `target_message_id` field. The +capability contract gains a fail-closed `supports_target_message_id` flag. + +For a negotiated peer that advertises support, Core sends command replies as: + +```json +{ + "reply_to": "evt_origin", + "target_message_id": "platform_activity_id", + "command": "edit_message" +} +``` + +`reply_to` remains origin event correlation. `target_message_id` is the +platform activity targeted by edit, delete, or an opt-in reaction command. +Normal sends never interpret `reply_to` as a command target. + +Compatibility behavior is: + +- new Core + new Gateway: preserve the origin event and send the explicit + target field; +- new Core + old Gateway, or an operation before hello completes: omit the new + field and copy the command target into legacy `reply_to`; +- old Core + new Gateway: when the field is absent, treat `reply_to` as the + legacy command target; +- Unified: use the explicit field for Teams and legacy form for adapters that + do not advertise support. + +Missing capability fields default to false. The protocol version remains v1 +because both capability and reply fields are additive and covered by +old-peer decoding tests. + +### Process-local ownership index + +After a Teams create/send returns `Delivered` with a non-empty activity ID, the +Gateway records: + +```text +(app_id, tenant_id, conversation_id, activity_id) + -> authenticated route + ownership_created_at +``` + +The ownership index: + +- is process-local; +- uses `teams.route_ttl_secs` as its lifetime; +- has an independent `teams.max_route_entries` capacity bound; +- evicts its oldest entry at capacity with an operator warning; +- is swept by the existing Teams ingress cleanup task; +- stores the already validated gateway-local service URL and never exposes it + to Core, the agent, ACK payloads, or logs. + +Inbound activity IDs are not inserted. Only a confirmed outbound activity can +be edited or deleted. + +A new-field command must still have a live origin event route. The route fixes +the app, tenant, and conversation scope before ownership lookup. A missing or +expired origin returns `target_origin_not_found`; a channel mismatch returns +`target_scope_mismatch`; a target outside that exact scope returns +`message_not_owned`. + +A legacy command has no separate origin event. Gateway may use a unique owned +entry matching the configured app, command conversation, and target activity. +If the same legacy tuple exists in more than one tenant, it fails closed with +`target_scope_ambiguous`. + +### Bot Connector operations + +Owned edits call: + +```text +PUT {serviceUrl}/v3/conversations/{conversationId}/activities/{activityId} +``` + +Owned deletes call: + +```text +DELETE {serviceUrl}/v3/conversations/{conversationId}/activities/{activityId} +``` + +Conversation and activity IDs continue to use URL path-segment encoding and the +commercial-public-cloud endpoint policy. OAuth acquisition, same-origin +redirect policy, 5-second connect timeout, 10-second request timeout, 4 KiB +error body cap, and redaction rules are unchanged. + +Mutation outcomes are classified as follows: + +- HTTP success: `Delivered` without a required message ID; +- explicit `3xx` or `4xx`: `Rejected`; +- `429`: `Rejected` with parsed `retry_after_ms` unless the bounded internal + retry succeeds; +- `5xx`, request timeout, disconnect, or transport failure: `Unknown`; +- route, ownership, target, or command validation failure before HTTP: + `Rejected`. + +POST sends are never retried. PUT and DELETE perform at most one internal retry +only after an explicit `429` response proves that attempt was rejected, and +only when `Retry-After` is present and no greater than one second. A second +`429`, a longer delay, a timeout, disconnect, or `5xx` is returned immediately. +Core does not retry a terminal `Rejected` or `Unknown` outcome. + +A delivered delete removes the ownership entry. A rejected delete retains it +for a later corrected attempt; an unknown delete retains it because the Gateway +cannot safely infer whether Teams applied the operation. Delivered edits retain +the original ownership timestamp rather than turning active mutation into +unbounded retention. + +### Operation-specific acknowledgement + +A configured Teams adapter advertises: + +```text +send_ack = true +edit_ack = true +delete_ack = true +supports_target_message_id = true +can_edit = true +can_delete = true +streaming_mode = disabled +``` + +New Standalone peers wait for the existing configured Gateway ACK timeout on +edit and delete. Delivered edit/delete ACKs do not require a message ID. +Rejected and unknown outcomes propagate as errors and are not converted to +fire-and-forget success. + +Legacy peers retain their previous missing-ACK semantics. Unified returns the +same outcome directly and overrides native delete instead of falling back to an +edit-to-zero-width operation. + +Enabling edit/delete capability does not enable progressive response. Teams +`streaming_mode` remains disabled; progressive response owns its policy and +lifecycle separately. + +### Same-conversation ordering + +Every Teams send, edit, and delete acquires a fixed process-local write shard +computed from tenant and conversation ID. There are 64 shards: + +- writes in the same tenant/conversation are serialized; +- different conversations normally proceed independently; +- a hash collision may conservatively serialize unrelated conversations; +- the fixed array prevents an attacker or busy tenant from growing a lock map + without bound. + +Ownership and route state are revalidated after lock acquisition. This prevents +a queued edit from running after a preceding delete removed ownership. + +## Compatibility matrix + +| Core | Gateway | Command targeting and ACK behavior | +| --- | --- | --- | +| old | old | Legacy `reply_to` wire shape; Teams edit/delete remain unsupported and fail closed in a legacy Gateway without bot-owned mutation support. | +| old | new | Legacy target fallback is accepted only if it resolves to a unique bot-owned activity; missing ACK remains non-fatal to old Core. | +| new | old | No target-field capability is advertised, so Core copies the target into legacy `reply_to`; missing required ACK is not enabled. | +| new | new | Origin and target remain separate; ownership is enforced; edit/delete receive operation-specific structured ACKs. | +| Unified | embedded | Teams uses the explicit target and returns the structured mutation outcome directly. | + +## Security and reliability boundaries + +- Inbound user activities cannot enter the ownership index and therefore cannot + be edited or deleted. +- New-field mutation cannot cross app, tenant, or conversation scope. +- Ambiguous legacy scope fails closed. +- Service URLs and credentials remain Gateway-local. +- Ownership disappears on restart and is not shared across replicas. +- A successful write ACK is not a durable event log or exactly-once guarantee. +- External deletion, app uninstall, or platform-side retention can make a + locally owned ID invalid; the Connector response remains authoritative. +- This decision does not add proactive references, persistent ownership, or + multi-consumer coordination. + + +## Consequences + +### Positive + +- OpenAB can update and delete only activities it confirmed creating. +- Event correlation and platform command targets are no longer overloaded for + new peers. +- Edit/delete uncertainty is explicit and cannot trigger blind fresh sends. +- Same-conversation writes cannot overtake one another within a process. +- Rolling upgrades remain non-lockstep. + +### Negative + +- Restart, TTL expiry, or capacity eviction makes older bot messages immutable + through OpenAB. +- New-field commands require their origin route to remain live even if ownership + was recorded slightly later. +- Fixed lock-shard collisions can reduce concurrency. +- The one-second `429` retry bound may return a retryable rejection to Core + instead of waiting for a longer platform delay. +- Connector mutation behavior and availability remain controlled by Microsoft. diff --git a/docs/adr/teams-real-send-acknowledgement.md b/docs/adr/teams-real-send-acknowledgement.md new file mode 100644 index 000000000..044acb2dd --- /dev/null +++ b/docs/adr/teams-real-send-acknowledgement.md @@ -0,0 +1,205 @@ +# ADR: Teams Real Send Acknowledgement and Reply Correlation + +- **Status:** Proposed +- **Date:** 2026-08-08 +- **Author:** @NeoHsu +- **Related:** + - [Teams ephemeral ingress state](teams-ephemeral-ingress-state.md) + - [Gateway capability and delivery semantics](gateway-capabilities-and-delivery-semantics.md) + - [Custom Gateway](custom-gateway.md) + - [Teams bot-owned message mutations](teams-owned-message-mutations.md) + +--- + +## Context + +The [ephemeral-ingress decision](teams-ephemeral-ingress-state.md) introduced an +authenticated, bounded, gateway-local route for each Teams message activity. Before this decision, outbound Teams delivery still used a +conversation-only service URL cache, copied the OpenAB `event_id` into Bot +Connector `replyToId`, and discarded the activity ID returned by Teams in +Unified mode. Standalone Core could not require a send acknowledgement because +the Teams capability advertised `send_ack = false`. + +Those behaviors violated the identifier contract: + +- `GatewayEvent.event_id` is OpenAB correlation metadata, not a Bot Framework + activity ID; +- a successful create/send must return the real platform activity ID; +- a timed-out or disconnected POST may already have completed and must not be + represented as a safe rejection or retried blindly. + +Teams controls channel-root, channel-reply, Personal, and group-chat +presentation; HTTP transport tests cannot define or guarantee that UX. + +## Decision + +### Route resolution + +A commandless outbound reply resolves `GatewayReply.reply_to` exclusively as an +OpenAB `event_id` in the bounded ingress registry. The resolved route supplies: + +- bot app and tenant scope; +- conversation ID and type; +- inbound activity and reply-chain identifiers; +- the validated gateway-local service URL; +- optional Team and channel identifiers. + +The outbound `GatewayReply.channel.id` must equal the route's conversation ID. +A missing, expired, or capacity-evicted event route returns +`route_not_found`; a conversation mismatch returns `route_mismatch`. Both are +`Rejected` outcomes and occur before OAuth or Bot Connector I/O. + +The conversation-only compatibility service URL map is removed. Service URLs +remain inside authenticated route state and are never sent to Core or the +agent. + +### Normal send and explicit quote + +Normal responses do not infer a Bot Connector `replyToId` from `event_id` or +from the triggering inbound activity. They use the Bot Connector +`SendToConversation` endpoint: + +```text +POST {serviceUrl}/v3/conversations/{conversationId}/activities +``` + +Only `GatewayReply.quote_message_id` expresses an explicit quote request. The +adapter uses the Bot Connector `ReplyToActivity` endpoint and also carries the +target as `Activity.replyToId`, but only when the target is known in the same +app, tenant, and conversation scope: + +```text +POST {serviceUrl}/v3/conversations/{conversationId}/activities/{activityId} +``` + +Known targets include: + +- the current authenticated inbound activity; +- its authenticated `replyToId` reply-chain root; +- another activity still present in the same bounded ingress route index. + +An empty or unknown target falls back to a plain send. It is never looked up in +another tenant or conversation and never causes `event_id` to reach a Bot +Connector activity URL or body field. This endpoint distinction follows +Microsoft's [Bot Connector API reference](https://learn.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-api-reference?view=azure-bot-service-4.0#reply-to-activity), +which directs replies to a specific activity through `ReplyToActivity` rather +than `SendToConversation`. + +### Structured send outcomes + +Teams sends produce one of the existing protocol outcomes: + +| Condition | Outcome | Code / data | +| --- | --- | --- | +| HTTP success with non-empty activity ID | `Delivered` | real `message_id` | +| Invalid or missing route | `Rejected` | `invalid_route`, `route_not_found`, or `route_mismatch` | +| OAuth acquisition fails before Connector POST | `Rejected` | `connector_auth_failed` | +| Connector `3xx` or `4xx` response | `Rejected` | stable rejection code | +| Connector `429` | `Rejected` | `rate_limited` plus bounded `retry_after_ms` | +| Connector `5xx` | `Unknown` | `connector_server_error` | +| POST timeout, disconnect, or transport error | `Unknown` | `request_timeout` or `transport_error` | +| Success response is malformed or omits activity ID | `Unknown` | `invalid_success_response` or `missing_activity_id` | + +A success response without a usable activity ID is `Unknown`, not `Rejected`, +because Teams may already have created the message. OpenAB does not +fresh-send or automatically retry `Rejected` or `Unknown` results on this path. +Error body size, redaction, endpoint validation, redirect policy, and request +timeouts remain governed by the Bot Connector transport-hardening decision. + +`Retry-After` accepts either delta seconds or an HTTP date and is converted to a +bounded millisecond value. Recording it does not introduce an automatic retry. + +### Standalone acknowledgement + +A configured Teams adapter now advertises: + +```text +send_ack = true +edit_ack = false +delete_ack = false +``` + +After a valid new↔new hello negotiation, Core includes a request ID and waits for +the operation-specific send ACK. Gateway emits +`openab.gateway.response.v1` with additive structured outcome fields on every +terminal commandless-send path. `Delivered` must contain a non-empty real +Bot Framework activity ID; Core rejects an otherwise successful ACK without +one. + +A legacy Core may omit the request ID or include one for its existing +best-effort streaming correlation. New Gateway emits no unsolicited frame when +the ID is absent. When it is present, the response keeps the legacy fields and +adds outcome metadata that old peers can ignore; a missing response remains +non-fatal under legacy Core semantics. New Core connected to an old Gateway +receives no supported hello and likewise retains legacy missing-ACK behavior. + +### Unified acknowledgement + +Unified mode calls the same Teams route and Connector implementation in +process. `ChatAdapter::send_message` and `send_message_with_reply` return a +`MessageRef` containing the real Bot Framework activity ID. `Rejected` and +`Unknown` outcomes become errors; Unified no longer fabricates a synthetic ID +for Teams delivery. + +Other Unified platforms retain their existing behavior. At this decision's +boundary, Teams capabilities report required send acknowledgement while edit, +delete, streaming, and status remain disabled. The later +[bot-owned mutation decision](teams-owned-message-mutations.md) enables guarded +edit/delete without enabling streaming. + +### Activity DTO + +The inbound DTO parses the route and presentation fields, including +`activity.id`, `activity.replyToId`, `conversation.id`, `conversation.conversationType`, +`channelData.team.id`, `channelData.channel.id`, and `recipient.id`. +Parsing these fields does not define or guarantee Teams presentation behavior. + +## Compatibility + +| Core | Gateway | Send behavior | +| --- | --- | --- | +| old | old | Existing legacy fire-and-forget behavior. | +| old | new | Event route is used; no request ID means no response frame, while a legacy request ID receives a backward-compatible response. Missing ACK remains non-fatal to old Core. | +| new | old | No valid hello; Core keeps legacy missing-ACK semantics. | +| new | new | Teams advertises required send ACK and returns a structured terminal outcome with the real activity ID on delivery. | +| Unified | embedded | The same outcome is returned directly without a WebSocket ACK frame. | + +Operations emitted before a valid hello is processed continue to use legacy +Core semantics. The Gateway still classifies the internal result, but does not +send an unsolicited response when the reply has no request ID. + +## Security and reliability boundaries + +- Route lookup is process-local, bounded, and TTL-limited. +- Service URLs never cross the Gateway boundary or appear in outcome messages. +- Quote targets cannot cross app, tenant, or conversation scope. +- Unsupported commands are rejected before route lookup and network I/O; + reactions remain an intentional no-op until a Teams status backend exists. +- A Gateway broadcast ACK is not a durable record and does not provide crash + replay. +- This decision does not claim exactly-once delivery, multi-consumer work + distribution, proactive-send support, or restart-persistent message + ownership. + + +## Consequences + +### Positive + +- OpenAB correlation IDs can no longer leak into Bot Connector activity fields. +- New Standalone and Unified sends expose the real Teams activity ID. +- Rejection and ambiguous delivery are distinct, preventing blind duplicate + sends after POST uncertainty. +- The temporary conversation-only service URL cache is eliminated. +- Explicit quote targets fail safely to plain send when route evidence is + missing. + +### Negative + +- Replies fail after route expiry, capacity eviction, or process restart. +- A successful Teams write with a malformed response is reported as unknown + even though a message may be visible to the user. +- Teams clients control scope-specific thread and quote presentation; successful + transport correlation does not guarantee visible quote chrome. +- Required ACKs make Connector latency visible to new Core peers and consume + the configured 12-second Gateway ACK budget. diff --git a/docs/adr/teams-typed-scope-and-mention-routing.md b/docs/adr/teams-typed-scope-and-mention-routing.md new file mode 100644 index 000000000..c1345b5c1 --- /dev/null +++ b/docs/adr/teams-typed-scope-and-mention-routing.md @@ -0,0 +1,191 @@ +# ADR: Teams Typed Scope and Mention Routing + +- **Status:** Proposed +- **Date:** 2026-08-07 +- **Author:** @NeoHsu +- **Related:** + - [Multi-platform adapter architecture](multi-platform-adapters.md) + - [Identity trust-none](identity-trust-none.md) + - [Teams ephemeral ingress state](teams-ephemeral-ingress-state.md) + +--- + +## Context + +Before typed scope was added, Teams published only a conversation route, +sender, raw text, and an empty mention list. Core therefore evaluated every Gateway event with +`is_dm = false`, could not distinguish Personal, group chat, and Team channel +scope, and could not prove that a structured Teams mention targeted the +receiving bot. Pure text could also resemble an `@mention` without carrying an +authenticated mention entity. + +This proposal must add scope and mention evidence without changing outbound +routing, widening Graph authority, or requiring a lockstep Core/Gateway rollout. + +## Decision + +### Additive wire fields + +`openab.gateway.event.v1` gains three optional fields. The schema and protocol +version remain unchanged because old decoders ignore unknown fields and new +decoders default absent fields: + +```rust +struct GatewayScope { + tenant_id: Option, + team_id: Option, + channel_id: Option, + conversation_type: String, + trust_scope_id: String, + is_dm: bool, +} + +struct RecipientInfo { + id: String, + name: String, +} + +struct MentionInfo { + id: String, + text: String, +} + +struct GatewayEvent { + // existing fields remain unchanged + scope: Option, + recipient: Option, + mention_entities: Vec, +} +``` + +The existing `mentions` array remains a list of mentioned entity IDs for +cross-platform structural gating. `mention_entities` carries the exact Teams +entity text needed for safe recipient-mention removal. Neither field carries a +service URL, token, or proactive conversation reference. + +`ChannelInfo.id` remains the Bot Connector conversation ID used for session and +outbound routing. It must not be replaced by `trust_scope_id`. + +### Teams scope derivation + +After JWT, tenant, required-route, and public-cloud service URL validation, the +Gateway canonicalizes known Bot Framework conversation types: + +| `conversationType` | `is_dm` | Required typed fields | Opaque `trust_scope_id` shape | +| --- | --- | --- | --- | +| `personal` | `true` | tenant + conversation | `teams:{tenant}:personal:{conversation}` | +| `groupChat` | `false` | tenant + conversation | `teams:{tenant}:group-chat:{conversation}` | +| `channel` | `false` | tenant + Team + channel | `teams:{tenant}:team:{team}:channel:{channel}` | + +The key is opaque and is never parsed for authorization. Raw Team and channel +IDs remain separate fields for allowlist matching. Unknown conversation types, +`is_dm` contradictions, empty `trust_scope_id`, or channel scope missing Team or +channel IDs fail closed in the new Core. + +### Scope policy and compatibility + +First-class Teams scope settings are: + +```toml +[teams] +allowed_teams = [] +allowed_channels = [] +allow_personal = true +allow_group_chats = true +``` + +Rules: + +- Personal is admitted only when `allow_personal = true`. +- Group chat is admitted only when `allow_group_chats = true`. +- For Team channels, both lists empty means L2 open. If either list is non-empty, + a Team ID or channel ID match admits the scope. +- L3 `allowed_users` remains an independent security gate and is never bypassed + by an L2 scope match. + +Presence of any new Teams scope field or corresponding environment variable +opts into typed policy. If none is present, Core preserves the existing generic +Gateway L2 behavior: `GATEWAY_ALLOWED_CHANNELS` and `[gateway].allowed_channels` +continue matching `ChannelInfo.id` (the conversation ID). This legacy fallback +is explicit and observable; it prevents a rolling upgrade from silently +opening or closing an existing deployment. New Gateway events without a new +Core are ignored additively. New Core receiving an old event without `scope` +uses the legacy gate. + +### Mention trust and trigger matrix + +The Gateway parses only `Activity.entities[]` entries whose type is `mention` +and whose `mentioned.id` is non-empty. It publishes their IDs in `mentions` and +their ID/text pairs in `mention_entities`. `Activity.recipient.id` is published +separately. + +New↔new trigger behavior is: + +| Scope | Trigger | +| --- | --- | +| Personal | Every otherwise trusted user message; mention not required | +| Group chat | A structured mention entity has `mentioned.id == recipient.id` | +| Team channel root/reply | A structured mention entity has `mentioned.id == recipient.id` | +| Unknown/malformed scope | Drop before commands, sessions, reactions, or ACP | + +Pure text such as `@OpenAB` or `OpenAB` without a matching entity never +satisfies group/channel mention gating. Thread presence does not bypass Teams +mention gating; ambient/RSC reading remains a separate feature. + +After structural mention and L2/L3 trust gates allow the event, Core removes +only entity text associated with the recipient bot ID. It maps entity order to +text occurrences, removes matched recipient ranges in reverse order, and trims +only the resulting edges. Other user/bot mentions and arbitrary whitespace are +preserved. A malformed recipient entity may trigger by ID but is not removed +unless its exact non-empty text occurs. Mention-only text is ignored when no +attachment blocks remain. + +Core sets `SenderContext.receiver_id` from `recipient.id`; sender identity, +conversation routing, event correlation, and message ID semantics remain +unchanged. + +## Security and reliability boundaries + +- Scope and mention evidence is created only after existing Bot Framework JWT, + tenant, and service URL validation. +- Mention markup is not authority; structured entity IDs are. +- Scope allowlists do not replace L3 user trust. +- This decision adds no Graph, RSC, delegated token, manifest permission, ambient + reading, attachment download, or persistent conversation state. +- Service URLs and credentials remain Gateway-local. +- Standalone and Unified paths use the same Core scope, mention, and prompt + normalization helpers. + +## Acceptance criteria + +Automated tests must cover: + +- additive new/old wire decoding; +- Personal, groupChat, and channel scope derivation; +- missing Team/channel fields and unknown conversation types; +- typed Team-or-channel allowlist semantics and legacy conversation fallback; +- correct `is_dm` and L3 identity ordering; +- genuine recipient mention, pure-text spoof, reply mention, multi-mention, + duplicate mention, and malformed entity cases; +- removal of only the recipient mention while preserving other text; +- slash-command recognition after recipient mention removal; +- Standalone and Unified use of the same helpers; +- config, environment fallback, Helm, and platform-schema conformance. + +## Consequences + +### Positive + +- Core receives an explicit, authenticated Teams scope instead of guessing from + a route ID. +- Structured mention gating prevents textual mention spoofing. +- Personal behavior remains mention-free and existing generic L2 restrictions + retain a non-lockstep fallback. +- Agent context can identify the receiving bot in multi-agent deployments. + +### Negative + +- Typed Teams L2 policy requires additional Core configuration and tests. +- Old Core cannot enforce new group/channel mention semantics until upgraded. +- Mention text lacks explicit character offsets, so cleanup relies on entity + order plus exact text matching and deliberately leaves unmatched markup.