TwilioRealtimeTransportLayer

An adapter to connect a websocket that is receiving messages from Twilio’s Media Streams API to the OpenAI Realtime API via WebSocket.

It automatically handles setting the right audio format for the input and output audio, passing the data along and handling the timing for interruptions using Twilio’s mark events.

It does require you to run your own WebSocket server that is receiving connection requests from Twilio.

It will emit all Twilio received messages as twilio_message type messages on the * handler. If you are using a RealtimeSession you can listen to the transport_event.

Example

const transport = new TwilioRealtimeTransportLayer({
  twilioWebSocket: twilioWebSocket,
});

transport.on('*', (event) => {
  if (event.type === 'twilio_message') {
    console.log('Twilio message:', event.data);
  }
});

Extends

OpenAIRealtimeWebSocket

Constructors

Constructor

new TwilioRealtimeTransportLayer(options): TwilioRealtimeTransportLayer

Parameters

Parameter	Type
`options`	`TwilioRealtimeTransportLayerOptions`

Returns

TwilioRealtimeTransportLayer

Overrides

OpenAIRealtimeWebSocket.constructor

Accessors

_tracingConfig

Set Signature

set _tracingConfig(tracingConfig): void

Sets the internal tracing config. This is used to track the tracing config that has been set during the session.create event.

Parameters

Parameter	Type
`tracingConfig`	`null` \| `RealtimeTracingConfig`

Returns

void

Inherited from

OpenAIRealtimeWebSocket._tracingConfig

connectionState

Get Signature

get connectionState(): WebSocketState

The current connection state of the WebSocket connection.

Returns

WebSocketState

Inherited from

OpenAIRealtimeWebSocket.connectionState

currentModel

Get Signature

get currentModel(): OpenAIRealtimeModels

The current model that is being used by the transport layer.

Returns

OpenAIRealtimeModels

Set Signature

set currentModel(model): void

The current model that is being used by the transport layer. Note: The model cannot be changed mid conversation.

Parameters

Parameter	Type
`model`	`OpenAIRealtimeModels`

Returns

void

Inherited from

OpenAIRealtimeWebSocket.currentModel

muted

Get Signature

get muted(): null

Always returns null as the WebSocket transport layer does not handle muting. Instead, this should be handled by the client by not triggering the sendAudio method.

Returns

null

Inherited from

OpenAIRealtimeWebSocket.muted

status

Get Signature

get status(): "connected" | "disconnected" | "connecting"

The current status of the WebSocket connection.

Returns

"connected" | "disconnected" | "connecting"

Inherited from

OpenAIRealtimeWebSocket.status

Methods

_cancelResponse()

_cancelResponse(): void

Send a cancel response event to the Realtime API. This is used to cancel an ongoing response that the model is currently generating.

Returns

void

Inherited from

OpenAIRealtimeWebSocket._cancelResponse

_interrupt()

_interrupt(_elapsedTime): void

Do NOT call this method directly. Call interrupt() instead for proper interruption handling.

This method is used to send the right events to the API to inform the model that the user has interrupted the response. It might be overridden/extended by an extended transport layer. See the TwilioRealtimeTransportLayer for an example.

Parameters

Parameter	Type
`_elapsedTime`	`number`

Returns

void

Overrides

OpenAIRealtimeWebSocket._interrupt

_setInputAndOutputAudioFormat()

_setInputAndOutputAudioFormat(partialConfig?): Partial<RealtimeSessionConfig>

Parameters

Parameter	Type
`partialConfig`?	`Partial`<`RealtimeSessionConfig`>

Returns

Partial<RealtimeSessionConfig>

close()

close(): void

Close the WebSocket connection.

This will also reset any internal connection tracking used for interruption handling.

Returns

void

Inherited from

OpenAIRealtimeWebSocket.close

connect()

connect(options): Promise<void>

Parameters

Parameter	Type
`options`	`RealtimeTransportLayerConnectOptions`

Returns

Promise<void>

Overrides

OpenAIRealtimeWebSocket.connect

emit()

emit<K>(type, ...args): boolean

Type Parameters

Type Parameter
`K` extends keyof `RealtimeTranportEventTypes`

Parameters

Parameter	Type
`type`	`K`
…`args`	`OpenAIRealtimeEventTypes`[`K`]

Returns

boolean

Inherited from

OpenAIRealtimeWebSocket.emit

interrupt()

interrupt(): void

Interrupt the ongoing response. This method is triggered automatically by the client when voice activity detection (VAD) is enabled (default) as well as when an output guardrail got triggered.

You can also call this method directly if you want to interrupt the conversation for example based on an event in the client.

Returns

void

Inherited from

OpenAIRealtimeWebSocket.interrupt

mute()

mute(_muted): never

Will throw an error as the WebSocket transport layer does not support muting.

Parameters

Parameter	Type
`_muted`	`boolean`

Returns

never

Inherited from

OpenAIRealtimeWebSocket.mute

off()

off<K>(type, listener): EventEmitter<EventTypes>

Type Parameters

Type Parameter
`K` extends keyof `RealtimeTranportEventTypes`

Parameters

Parameter	Type
`type`	`K`
`listener`	(…`args`) => `void`

Returns

EventEmitter<EventTypes>

Inherited from

OpenAIRealtimeWebSocket.off

on()

on<K>(type, listener): EventEmitter<EventTypes>

Type Parameters

Type Parameter
`K` extends keyof `RealtimeTranportEventTypes`

Parameters

Parameter	Type
`type`	`K`
`listener`	(…`args`) => `void`

Returns

EventEmitter<EventTypes>

Inherited from

OpenAIRealtimeWebSocket.on

once()

once<K>(type, listener): EventEmitter<EventTypes>

Type Parameters

Type Parameter
`K` extends keyof `RealtimeTranportEventTypes`

Parameters

Parameter	Type
`type`	`K`
`listener`	(…`args`) => `void`

Returns

EventEmitter<EventTypes>

Inherited from

OpenAIRealtimeWebSocket.once

resetHistory()

resetHistory(oldHistory, newHistory): void

Reset the history of the conversation. This will create a diff between the old and new history and send the necessary events to the Realtime API to update the history.

Parameters

Parameter	Type	Description
`oldHistory`	`RealtimeItem`[]	The old history of the conversation.
`newHistory`	`RealtimeItem`[]	The new history of the conversation.

Returns

void

Inherited from

OpenAIRealtimeWebSocket.resetHistory

sendAudio()

sendAudio(audio, options?): void

Send an audio buffer to the Realtime API. This is used for your client to send audio to the model to respond.

Parameters

Parameter	Type	Description
`audio`	`ArrayBuffer`	The audio buffer to send.
`options`?	{ `commit`: `boolean`; }	The options for the audio buffer.
`options.commit`?	`boolean`	‐

Returns

void

Inherited from

OpenAIRealtimeWebSocket.sendAudio

sendEvent()

sendEvent(event): void

Send an event to the Realtime API. This will stringify the event and send it directly to the API. This can be used if you want to take control over the connection and send events manually.

Parameters

Parameter	Type	Description
`event`	`RealtimeClientMessage`	The event to send.

Returns

void

Inherited from

OpenAIRealtimeWebSocket.sendEvent

sendFunctionCallOutput()

sendFunctionCallOutput(
   toolCall,
   output,
   startResponse?): void

Send the output of a function call to the Realtime API.

Parameters

Parameter	Type	Description
`toolCall`	`TransportToolCallEvent`	The tool call to send the output for.
`output`	`string`	The output of the function call.
`startResponse`?	`boolean`	Whether to start a new response after sending the output.

Returns

void

Inherited from

OpenAIRealtimeWebSocket.sendFunctionCallOutput

sendMessage()

sendMessage(message, otherEventData): void

Send a message to the Realtime API. This will create a new item in the conversation and trigger a response.

Parameters

Parameter	Type	Description
`message`	`RealtimeUserInput`	The message to send.
`otherEventData`	`Record`<`string`, `any`>	Additional event data to send.

Returns

void

Inherited from

OpenAIRealtimeWebSocket.sendMessage

updateSessionConfig()

updateSessionConfig(config): void

Updates the session config. This will merge it with the current session config with the default values and send it to the Realtime API.

Parameters

Parameter	Type	Description
`config`	`Partial`<`RealtimeSessionConfig`>	The session config to update.

Returns

void

Inherited from

OpenAIRealtimeWebSocket.updateSessionConfig