OpenAIRealtimeWebRTC

Transport layer that’s handling the connection between the client and OpenAI’s Realtime API via WebRTC. While this transport layer is designed to be used within a RealtimeSession, it can also be used standalone if you want to have a direct connection to the Realtime API.

Unless you specify a mediaStream or audioElement option, the transport layer will automatically configure the microphone and audio output to be used by the session.

Extends

OpenAIRealtimeBase

Implements

RealtimeTransportLayer

Constructors

Constructor

new OpenAIRealtimeWebRTC(options): OpenAIRealtimeWebRTC;

Parameters

Parameter	Type
`options`	`OpenAIRealtimeWebRTCOptions`

Returns

OpenAIRealtimeWebRTC

Overrides

OpenAIRealtimeBase.constructor

Accessors

_tracingConfig

Set Signature

set _tracingConfig(tracingConfig): void;

Sets the internal tracing config. This is used to track the tracing config that has been set during the session.create event.

Parameters

Parameter	Type
`tracingConfig`	`null` \| `RealtimeTracingConfig`

Returns

void

Inherited from

OpenAIRealtimeBase._tracingConfig

connectionState

Get Signature

get connectionState(): WebRTCState;

The current connection state of the WebRTC connection including the peer connection and data channel.

Returns

WebRTCState

currentModel

Get Signature

get currentModel(): OpenAIRealtimeModels;

The current model that is being used by the transport layer.

Returns

OpenAIRealtimeModels

Set Signature

set currentModel(model): void;

The current model that is being used by the transport layer. Note: The model cannot be changed mid conversation.

Parameters

Parameter	Type
`model`	`OpenAIRealtimeModels`

Returns

void

Inherited from

OpenAIRealtimeBase.currentModel

muted

Get Signature

get muted(): boolean;

Whether the session is muted.

Returns

boolean

Whether the input audio track is currently muted null if the muting is not handled by the transport layer

Implementation of

RealtimeTransportLayer.muted

Overrides

OpenAIRealtimeBase.muted

status

Get Signature

get status(): "connecting" | "connected" | "disconnected";

The current status of the WebRTC connection.

Returns

"connecting" | "connected" | "disconnected"

Implementation of

RealtimeTransportLayer.status

Overrides

OpenAIRealtimeBase.status

Methods

close()

close(): void;

Close the connection to the Realtime API and disconnects the underlying WebRTC connection.

Returns

void

Implementation of

RealtimeTransportLayer.close

Overrides

OpenAIRealtimeBase.close

connect()

connect(options): Promise<void>;

Connect to the Realtime API. This will establish the connection to the OpenAI Realtime API via WebRTC.

If you are using a browser, the transport layer will also automatically configure the microphone and audio output to be used by the session.

Parameters

Parameter	Type	Description
`options`	`RealtimeTransportLayerConnectOptions`	The options for the connection.

Returns

Promise<void>

Implementation of

RealtimeTransportLayer.connect

Overrides

OpenAIRealtimeBase.connect

emit()

emit<K>(type, ...args): boolean;

Type Parameters

Type Parameter
`K` extends keyof `RealtimeTranportEventTypes`

Parameters

Parameter	Type
`type`	`K`
…`args`	`OpenAIRealtimeEventTypes`[`K`]

Returns

boolean

Implementation of

RealtimeTransportLayer.emit

Inherited from

OpenAIRealtimeBase.emit

interrupt()

interrupt(): void;

Interrupt the current response if one is ongoing and clear the audio buffer so that the agent stops talking.

Returns

void

Implementation of

RealtimeTransportLayer.interrupt

Overrides

OpenAIRealtimeBase.interrupt

mute()

mute(muted): void;

Mute or unmute the session.

Parameters

Parameter	Type	Description
`muted`	`boolean`	Whether to mute the session.

Returns

void

Implementation of

RealtimeTransportLayer.mute

Overrides

OpenAIRealtimeBase.mute

off()

off<K>(type, listener): EventEmitter<EventTypes>;

Type Parameters

Type Parameter
`K` extends keyof `RealtimeTranportEventTypes`

Parameters

Parameter	Type
`type`	`K`
`listener`	(…`args`) => `void`

Returns

EventEmitter<EventTypes>

Implementation of

RealtimeTransportLayer.off

Inherited from

OpenAIRealtimeBase.off

on()

on<K>(type, listener): EventEmitter<EventTypes>;

Type Parameters

Type Parameter
`K` extends keyof `RealtimeTranportEventTypes`

Parameters

Parameter	Type
`type`	`K`
`listener`	(…`args`) => `void`

Returns

EventEmitter<EventTypes>

Implementation of

RealtimeTransportLayer.on

Inherited from

OpenAIRealtimeBase.on

once()

once<K>(type, listener): EventEmitter<EventTypes>;

Type Parameters

Type Parameter
`K` extends keyof `RealtimeTranportEventTypes`

Parameters

Parameter	Type
`type`	`K`
`listener`	(…`args`) => `void`

Returns

EventEmitter<EventTypes>

Implementation of

RealtimeTransportLayer.once

Inherited from

OpenAIRealtimeBase.once

resetHistory()

resetHistory(oldHistory, newHistory): void;

Reset the history of the conversation. This will create a diff between the old and new history and send the necessary events to the Realtime API to update the history.

Parameters

Parameter	Type	Description
`oldHistory`	`RealtimeItem`[]	The old history of the conversation.
`newHistory`	`RealtimeItem`[]	The new history of the conversation.

Returns

void

Implementation of

RealtimeTransportLayer.resetHistory

Inherited from

OpenAIRealtimeBase.resetHistory

sendAudio()

sendAudio(audio, options): void;

Send an audio buffer to the Realtime API. If { commit: true } is passed, the audio buffer will be committed and the model will start processing it. This is necessary if you have disabled turn detection / voice activity detection (VAD).

Parameters

Parameter	Type	Description
`audio`	`ArrayBuffer`	The audio buffer to send.
`options`	{ `commit?`: `boolean`; }	The options for the audio buffer.
`options.commit?`	`boolean`	‐

Returns

void

Implementation of

RealtimeTransportLayer.sendAudio

Inherited from

OpenAIRealtimeBase.sendAudio

sendEvent()

sendEvent(event): void;

Send an event to the Realtime API. This will stringify the event and send it directly to the API. This can be used if you want to take control over the connection and send events manually.

Parameters

Parameter	Type	Description
`event`	`RealtimeClientMessage`	The event to send.

Returns

void

Implementation of

RealtimeTransportLayer.sendEvent

Overrides

OpenAIRealtimeBase.sendEvent

sendFunctionCallOutput()

sendFunctionCallOutput(
   toolCall,
   output,
   startResponse): void;

Send the output of a function call to the Realtime API.

Parameters

Parameter	Type	Default value	Description
`toolCall`	`TransportToolCallEvent`	`undefined`	The tool call to send the output for.
`output`	`string`	`undefined`	The output of the function call.
`startResponse`	`boolean`	`true`	Whether to start a new response after sending the output.

Returns

void

Implementation of

RealtimeTransportLayer.sendFunctionCallOutput

Inherited from

OpenAIRealtimeBase.sendFunctionCallOutput

sendMessage()

sendMessage(message, otherEventData): void;

Send a message to the Realtime API. This will create a new item in the conversation and trigger a response.

Parameters

Parameter	Type	Description
`message`	`RealtimeUserInput`	The message to send.
`otherEventData`	`Record`<`string`, `any`>	Additional event data to send.

Returns

void

Implementation of

RealtimeTransportLayer.sendMessage

Inherited from

OpenAIRealtimeBase.sendMessage

updateSessionConfig()

updateSessionConfig(config): void;

Updates the session config. This will merge it with the current session config with the default values and send it to the Realtime API.

Parameters

Parameter	Type	Description
`config`	`Partial`<`RealtimeSessionConfig`>	The session config to update.

Returns

void

Implementation of

RealtimeTransportLayer.updateSessionConfig

Inherited from

OpenAIRealtimeBase.updateSessionConfig