Skip to content
This repository was archived by the owner on Nov 29, 2023. It is now read-only.

Commit bda3a2a

Browse files
marcelohossomimarcelohossomi
authored
Merge pull request #992 from Bandwidth/vapi-1390-real-time-transcription
VAPI-1390: Real-Time Transcription
2 parents e6b9b1f + c4b2110 commit bda3a2a

9 files changed

+1035
-158
lines changed

‎site/docs/voice/bxml/startTranscription.mdx

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
---
2+
id: startTranscription
3+
title: Start Transcription
4+
slug: /voice/bxml/startTranscription
5+
description: A general overview of Bandwidth's startTranscription BXML Verb
6+
keywords:
7+
- bandwidth
8+
- voice
9+
- bxml
10+
- start
11+
- transcribing
12+
hide_title: false
13+
image: ../../static/img/bandwidth-logo.png
14+
---
15+
16+
import Tabs from '@theme/Tabs';
17+
import TabItem from '@theme/TabItem';
18+
19+
The `StartTranscription` verb allows a segment of a call to be transcribed, and optionally for the live transcription to be sent off to another destination for additional processing.
20+
The transcription will continue until the call ends or the [`<StopTranscription>`][1] verb is used.
21+
When a `destination` is specified, live transcription updates for one or both sides (tracks) of the call will be sent to the specified destination.
22+
A total of 4 concurrent track transcriptions are allowed on a call. A `<StartTranscription>` request that uses `both` tracks will count as 2 of the permitted 4 concurrent track transcriptions.
23+
24+
A call has only two tracks, which are named after the direction of the media from the perspective of the Programmable Voice platform:
25+
- `inbound`: media received by Programmable Voice from the call executing the BXML;
26+
- `outbound`: media sent by Programmable Voice to the call executing the BXML.
27+
28+
Note that this has no correlation to the direction of the call itself. For example, if either an inbound or outbound call is being transcribed and executes a `<SpeakSentence>`, the `inbound` track will be the callee's audio and the `outbound` track will be the text-to-speech audio.
29+
30+
## Text Content
31+
32+
There is no text content available to be set for the `<StartTranscription>` verb.
33+
34+
## Attributes
35+
36+
| Attribute | Description |
37+
|:-------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
38+
| name | (optional) A name to refer to this transcription by. Used when sending [`<StopTranscription>`][1]. If not provided, it will default to the generated transcription id as sent in the [`Real-Time Transcription Started`][2] webhook. |
39+
| tracks | (optional) The part of the call to send a transcription from. `inbound`, `outbound` or `both`. Default is `inbound`. |
40+
| transcriptionEventUrl | (optional) URL to send the associated Webhook events to during this real-time transcription's lifetime. Does not accept BXML. May be a relative URL. |
41+
| transcriptionEventMethod | (optional) The HTTP method to use for the request to `transcriptionEventUrl`. GET or POST. Default value is POST. |
42+
| username | (optional) The username to send in the HTTP request to `transcriptionEventUrl`. If specified, the `transcriptionEventUrl` must be TLS-encrypted (i.e., `https`). |
43+
| password | (optional) The password to send in the HTTP request to `transcriptionEventUrl`. If specified, the `transcriptionEventUrl` must be TLS-encrypted (i.e., `https`). |
44+
| destination | (optional) A websocket URI to send the transcription to. A transcription of the specified tracks will be sent via websocket to this URL as a series of JSON messages. See below for more details on the websocket packet format. |
45+
| stabilized | (optional) Whether to send transcription update events to the specified `destination` only after they have become stable. Requires `destination`. Defaults to `true`. |
46+
47+
If the `destination` and `transcriptionEventUrl` attributes are specified, then the [Real-Time Transcription Started][2], [Real-Time Transcription Rejected][3] and [Real-Time Transcription Stopped][4] events will be sent to the URL when the transcription starts, if there is an error starting the transcription and when the transcription ends respectively. BXML returned in response to this callback will be ignored.
48+
If the `transcriptionEventUrl` attribute is specified, then the [Real-Time Transcription Available][5] event will be sent once the transcription has ended providing a URL from where the transcription can be downloaded. BXML returned in response to this callback will be ignored.
49+
50+
:::note
51+
While multiple real-time transcriptions for the same call are allowed, each real-time transcription MUST have a unique name. Attempting to start a real-time transcription on the same call with the name of an already existing real-time transcription will result in a [Real-Time Transcription Rejected][3] event.
52+
:::
53+
54+
## Webhooks Received
55+
56+
| Webhooks | Can reply with more BXML |
57+
|:---------------------------|:-------------------------|
58+
| [Real-Time Transcription Started][2] | No |
59+
| [Real-Time Transcription Rejected][3] | No |
60+
| [Real-Time Transcription Stopped][4] | No |
61+
| [Real-Time Transcription Available][5] | No |
62+
63+
## Nested Tags
64+
65+
You may specify up to 12 `<CustomParam/>` elements nested within a `<StartTranscription>` tag. These elements define optional user specified parameters that will be sent to the destination URL when the real-time transcription is first started.
66+
67+
### CustomParam Attributes
68+
69+
| Attribute | Description |
70+
|:----------|:---------------------------------------------------------------|
71+
| name | (required) The name of this parameter, up to 256 characters. |
72+
| value | (required) The value of this parameter, up to 2048 characters. |
73+
74+
## Websocket Packet Format
75+
76+
If a `destination` is specified, it will be sent JSON messages for the duration of the real-time transcription. There will be an initial `start` message when the connection is first established. This will be followed by zero or more `transcription` messages containing transcription updates for the tracks being transcribed. Finally, when a real-time transcription is stopped, a `stop` message will be sent.
77+
78+
### Start and Stop Message Parameters
79+
80+
| Parameter | Description |
81+
|:-------------|:------------|
82+
| eventType | What type of message this is, one of `start`, or `stop` |
83+
| metadata | Details about the real-time transcription this message is for. See further details below. |
84+
| customParams | (optional) (`start` message only) If any `<CustomParam/>` elements were specified in the `<StartTranscription>` request, they will be copied here as a map of `name : value` pairs |
85+
86+
#### Metadata Parameters
87+
88+
| Parameter | Description |
89+
|:------------------------------|:------------|
90+
| accountId | The user account associated with the call |
91+
| callId | The call id associated with the real-time transcription |
92+
| realTimeTranscriptionId | The unique id of the real-time transcription |
93+
| transcriptionName | The user supplied name of the real-time transcription |
94+
| tracks | A list of one or more tracks being transcribed in real-time |
95+
| tracks.name | The name of the track being transcribed, will be used to identify which transcription updates belong to which track |
96+
| stabilized | Whether transcription updates will be sent only after they have become stable or not |
97+
98+
### Transcription Message Parameters
99+
100+
| Parameter | Description |
101+
|:----------|:-----------------------|
102+
| eventType | Will always be `transcription` |
103+
| track | The name of the track this transcription update is for, will be one of the names specified in the `start` message |
104+
| startTime | The time at which this segment started |
105+
| endTime | The time at which this segment ended |
106+
| isPartial | Indicates if the segment is complete |
107+
| language | The detected language of the segment |
108+
| transcript | The transcription of this segment as a flattened string |
109+
| items | The list of items making up this segment |
110+
| items.content | A word or punctuation |
111+
| items.startTime | The time at which this item started |
112+
| items.endTime | The time at which this item ended |
113+
| items.confidence | The confidence score associated with a word or phrase in your transcript. |
114+
| items.stable | Indicates whether the specified item is stable (true) or if it may change when the segment is complete (false). |
115+
| items.type | Either `PRONUNCIATION` or `PUNCTUATION` |
116+
117+
## Examples
118+
119+
### A `start` Websocket Message
120+
121+
```json
122+
{
123+
"eventType": "start",
124+
"metadata": {
125+
"accountId": "5555555",
126+
"callId": "c-2a913f94-7fa91773-a426-4118-8b8b-b691ab0a0ae1",
127+
"realTimeTranscriptionId": "s-2a913f94-93e372e2-60da-4c89-beb0-0d3a219b287c",
128+
"transcriptionName": "live_audience",
129+
"tracks": [
130+
{
131+
"name": "inbound",
132+
},
133+
{
134+
"name": "outbound",
135+
}
136+
]
137+
},
138+
"customParams": {
139+
"foo": "bar",
140+
"foos": "bars"
141+
}
142+
}
143+
```
144+
145+
### A `transcription` Websocket Message
146+
```json
147+
{
148+
"eventType": "transcription",
149+
"track": "inbound",
150+
"startTime": "2023-03-31T20:05.101Z",
151+
"endTime": "2023-03-31T20:07.493Z",
152+
"isPartial": false,
153+
"language": "en-US",
154+
"transcript": "hello world!",
155+
"items": [
156+
{
157+
"content": "hello",
158+
"startTime": "2023-03-31T20:05.101Z",
159+
"endTime": "2023-03-31T20:06.285Z",
160+
"confidence": 0.9,
161+
"stable": true,
162+
"type": "PRONUNCIATION"
163+
},
164+
{
165+
"content": "world",
166+
"startTime": "2023-03-31T20:06.984Z",
167+
"endTime": "2023-03-31T20:07.493Z",
168+
"confidence": 0.6,
169+
"stable": true,
170+
"type": "PRONUNCIATION"
171+
},
172+
{
173+
"content": "!",
174+
"startTime": "2023-03-31T20:07.493Z",
175+
"endTime": "2023-03-31T20:07.493Z",
176+
"confidence": 0.9,
177+
"stable": false,
178+
"type": "PUNCTUATION"
179+
}
180+
]
181+
}
182+
```
183+
184+
### A `stop` Websocket Message
185+
186+
```json
187+
{
188+
"eventType": "stop",
189+
"metadata": {
190+
"accountId": "5555555",
191+
"callId": "c-2a913f94-7fa91773-a426-4118-8b8b-b691ab0a0ae1",
192+
"realTimeTranscriptionId": "s-2a913f94-93e372e2-60da-4c89-beb0-0d3a219b287c",
193+
"transcriptionName": "live_audience",
194+
"tracks": [
195+
{
196+
"name": "inbound",
197+
},
198+
{
199+
"name": "outbound",
200+
}
201+
]
202+
}
203+
}
204+
```
205+
206+
[1]: /docs/voice/bxml/stopTranscription
207+
[2]: /docs/voice/webhooks/realTimeTranscriptionStarted
208+
[3]: /docs/voice/webhooks/realTimeTranscriptionRejected
209+
[4]: /docs/voice/webhooks/realTimeTranscriptionStopped
210+
[5]: /docs/voice/webhooks/realTimeTranscriptionAvailable

‎site/docs/voice/bxml/stopTranscription.mdx

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
id: stopTranscription
3+
title: Stop Transcription
4+
slug: /voice/bxml/stopTranscription
5+
description: A general overview of Bandwidth's StopTranscription BXML Verb
6+
keywords:
7+
- bandwidth
8+
- voice
9+
- bxml
10+
- stop
11+
- transcribing
12+
hide_title: false
13+
image: ../../static/img/bandwidth-logo.png
14+
---
15+
16+
import Tabs from '@theme/Tabs';
17+
import TabItem from '@theme/TabItem';
18+
19+
The `StopTranscription` verb is used to stop a real-time transcription that was started with a previous [`<StartTranscription>`][1] verb.
20+
21+
If there is no real-time transcription with the given name active on the call, this verb has no effect.
22+
If no `name` is specified, all active call transcriptions (does not include transcribed recordings) are stopped.
23+
24+
## Text Content
25+
26+
There is no text content available to be set for the `<StopTranscription>` verb.
27+
28+
## Attributes
29+
30+
| Attribute | Description |
31+
|:-------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
32+
| name | (optional) The name of the real-time transcription to stop. This is either the user selected name when sending the [`<StartTranscription>`][1] verb, or the system generated name returned in the [Real-Time Transcription Started][2] webhook if `<StartTranscription>` was sent with no `name` attribute. If no `name` is specified, then all active call transcriptions will be stopped. |
33+
34+
## Webhooks Received
35+
36+
| Webhooks | Can reply with more BXML |
37+
|:---------------------------|:-------------------------|
38+
| [Real-Time Transcription Stopped][3] | No |
39+
| [Real-Time Transcription Available][4] | No |
40+
41+
## Examples
42+
43+
[1]: /docs/voice/bxml/startTranscription
44+
[2]: /docs/voice/webhooks/realTimeTranscriptionStarted
45+
[3]: /docs/voice/webhooks/realTimeTranscriptionStopped
46+
[4]: /docs/voice/webhooks/realTimeTranscriptionAvailable

‎site/docs/voice/webhooks/mediaStreamStopped.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ This event may be sent to the url specified when sending a [`<StopStream>`][1] v
2929
| enqueuedTime | (optional) If [call queueing](/apis/voice#operation/createCall/) is enabled and this is an outbound call, time the call was queued, in ISO 8601 format. |
3030
| errorMessage | (optional) If the `cause` value was anything other than `closed`, this will contain details about what went wrong. |
3131
| eventTime | The approximate UTC date and time when the event was generated by the Bandwidth server, in ISO 8601 format. This may not be exactly the time of event execution. |
32-
| eventType | The event type, value is `mediaStreamStarted` |
32+
| eventType | The event type, value is `mediaStreamStopped` |
3333
| from | The provided identifier of the caller: can be a phone number in E.164 format (e.g. +15555555555) or one of `Private`, `Restricted`, `Unavailable`, or `Anonymous`. |
3434
| mediaStream | Details about the stream that was stopped |
3535
| mediaStream.destination | The destination URL to which the stream sent media |

‎site/docs/voice/webhooks/realTimeTranscriptionAvailable.mdx

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
---
2+
id: realTimeTranscriptionAvailable
3+
title: Real-Time Transcription Available
4+
slug: /voice/webhooks/realTimeTranscriptionAvailable
5+
description: A general overview of Bandwidth's Real-Time Transcription Available Webhook
6+
keywords:
7+
- bandwidth
8+
- voice
9+
- webhook
10+
- transcribing
11+
- available
12+
hide_title: false
13+
image: ../../static/img/bandwidth-logo.png
14+
---
15+
16+
This event may be sent to the url specified when sending a [`<StartTranscription>`][1] verb.
17+
18+
## Request Parameters
19+
20+
| Property | Description |
21+
|:------------------------|:-------------|
22+
| accountId | The user account associated with the call. |
23+
| answerTime | Time the call was answered, in ISO 8601 format. |
24+
| applicationId | The id of the application associated with the call. |
25+
| callId | The call id associated with the event. |
26+
| callUrl | The URL of the call associated with the event. |
27+
| direction | The direction of the call. Either `inbound` or `outbound`. The direction of a call never changes. |
28+
| enqueuedTime | (optional) If [call queueing](/apis/voice#operation/createCall/) is enabled and this is an outbound call, this is the time the call was queued, in ISO 8601 format. Otherwise, this is omitted. |
29+
| eventTime | The approximate UTC date and time when the event was generated by the Bandwidth server, in ISO 8601 format. This may not be exactly the time of event execution. |
30+
| eventType | The event type, value is `realTimeTranscriptionAvailable` |
31+
| from | The provided identifier of the caller: can be a phone number in E.164 format (e.g. +15555555555) or one of `Private`, `Restricted`, `Unavailable`, or `Anonymous`. |
32+
| realTimeTranscription | Details about the transcription. |
33+
| realTimeTranscription.id | The unique id of the transcription. |
34+
| realTimeTranscription.name | The name of this transcription. If the `name` attribute was specified in the [`StartTranscription`][1] verb, then this will be the value of that attribute, otherwise it will default to the transcription id.
35+
| realTimeTranscription.startTime | The approximate UTC date and time the transcription was started |
36+
| realTimeTranscription.tracks | The segments of the call that are being sent in the transcription, values will be one or both of `inbound` and `outbound` |
37+
| realTimeTranscription.status | The status of the transcription. Can be either `available`, meaning that the transcription is ready for downloading, or `failed` otherwise. |
38+
| realTimeTranscription.url | The URL of the transcription. |
39+
| realTimeTranscription.completedTime | The time at which the transcription was completed and ready for download. |
40+
| realTimeTranscription.destination | (optional) The destination URL to which the transcription is sending media |
41+
| realTimeTranscription.stabilized | (optional) Whether to send transcription update events to the specified `destination` only after they have become stable. Requires `destination`. |
42+
| startTime | Time the call was started, in ISO 8601 format. |
43+
| to | The phone number that received the call, in E.164 format (e.g. +15555555555). |
44+
| tag | (optional) The `tag` specified on call creation. If no `tag` was specified or it was previously cleared, this field will not be present. |
45+
46+
## Expected Response
47+
48+
```http
49+
HTTP/1.1 204
50+
```
51+
52+
## Examples
53+
54+
### Real-Time Transcription Available event with destination
55+
56+
```json
57+
POST http://myapp.example/realTimeTranscriptionEvents
58+
Content-Type: application/json
59+
60+
{
61+
"accountId" : "55555555",
62+
"answerTime" : "2022-06-30T18:55:02.080Z",
63+
"applicationId" : "7fc9698a-b04a-468b-9e8f-91238c0d0086",
64+
"callId" : "c-95ac912f-68aacdd7-4a8e-4223-a7fd-020e02fa6bf2",
65+
"callUrl" : "https://voice.bandwidth.com/api/v2/accounts/55555555/calls/c-95ac912f-68aacdd7-4a8e-4223-a7fd-020e02fa6bf2",
66+
"direction" : "outbound",
67+
"enqueuedTime" : "2022-06-30T18:54:59.172Z",
68+
"eventTime" : "2022-06-30T18:55:02.489Z",
69+
"eventType" : "realTimeTranscriptionAvailable",
70+
"from" : "+15551112222",
71+
"realTimeTranscription" : {
72+
"id" : "t-95ac90b3-bfc81595-35fc-4b64-8265-fab6855b74a2",
73+
"name" : "example_transcription",
74+
"startTime" : "2022-06-30T18:55:02.489Z",
75+
"tracks" : ["inbound", "outbound"],
76+
"destination" : "wss://websocket.myapp.example",
77+
"stabilized" : "true",
78+
"status" : "available",
79+
"url" : "https://voice.bandwidth.com/api/v2/accounts/55555555/calls/c-95ac912f-68aacdd7-4a8e-4223-a7fd-020e02fa6bf2/transcriptions/t-95ac90b3-bfc81595-35fc-4b64-8265-fab6855b74a2",
80+
"completedTime" : "2022-06-30T18:55:02.489Z",
81+
},
82+
"startTime" : "2022-06-30T18:54:59.175Z",
83+
"to" : "+15553334444"
84+
}
85+
```
86+
87+
[1]: /docs/voice/bxml/startTranscription
88+
[2]: /docs/voice/bxml/startTranscription

0 commit comments

Comments
 (0)