RealtimeAgent
A specialized agent instance that is meant to be used within a RealtimeSession
to build
voice agents. Due to the nature of this agent, some configuration options are not supported
that are supported by regular Agent
instances. For example:
model
choice is not supported as all RealtimeAgents will be handled by the same model within aRealtimeSession
modelSettings
is not supported as all RealtimeAgents will be handled by the same model within aRealtimeSession
outputType
is not supported as RealtimeAgents do not support structured outputstoolUseBehavior
is not supported as all RealtimeAgents will be handled by the same model within aRealtimeSession
voice
can be configured on anAgent
level however it cannot be changed after the first agent within aRealtimeSession
spoke
Example
Section titled “Example”const agent = new RealtimeAgent({ name: 'my-agent', instructions: 'You are a helpful assistant that can answer questions and help with tasks.',})
const session = new RealtimeSession(agent);
Extends
Section titled “Extends”Agent
<RealtimeContextData
<TContext
>,TextOutput
>
Type Parameters
Section titled “Type Parameters”Type Parameter | Default type |
---|---|
|
|
Constructors
Section titled “Constructors”Constructor
Section titled “Constructor”new RealtimeAgent<TContext>(config): RealtimeAgent<TContext>
Parameters
Section titled “Parameters”Parameter | Type |
---|---|
|
|
Returns
Section titled “Returns”RealtimeAgent
<TContext
>
Overrides
Section titled “Overrides”Agent< RealtimeContextData<TContext>, TextOutput>.constructor
Properties
Section titled “Properties”handoffDescription
Section titled “handoffDescription”handoffDescription: string;
A description of the agent. This is used when the agent is used as a handoff, so that an LLM knows what it does and when to invoke it.
Inherited from
Section titled “Inherited from”Agent.handoffDescription
handoffs
Section titled “handoffs”handoffs: (Agent<any, "text"> | Handoff<any, "text">)[];
Handoffs are sub-agents that the agent can delegate to. You can provide a list of handoffs, and the agent can choose to delegate to them if relevant. Allows for separation of concerns and modularity.
Inherited from
Section titled “Inherited from”Agent.handoffs
inputGuardrails
Section titled “inputGuardrails”inputGuardrails: InputGuardrail[];
A list of checks that run in parallel to the agent’s execution, before generating a response. Runs only if the agent is the first agent in the chain.
Inherited from
Section titled “Inherited from”Agent.inputGuardrails
instructions
Section titled “instructions”instructions: string | (runContext, agent) => string | Promise<string>;
The instructions for the agent. Will be used as the “system prompt” when this agent is invoked. Describes what the agent should do, and how it responds.
Can either be a string, or a function that dynamically generates instructions for the agent. If you provide a function, it will be called with the context and the agent instance. It must return a string.
Inherited from
Section titled “Inherited from”Agent.instructions
mcpServers
Section titled “mcpServers”mcpServers: MCPServer[];
A list of Model Context Protocol servers the agent can use. Every time the agent runs, it will include tools from these servers in the list of available tools.
NOTE: You are expected to manage the lifecycle of these servers. Specifically, you must call
server.connect()
before passing it to the agent, and server.cleanup()
when the server is
no longer needed.
Inherited from
Section titled “Inherited from”Agent.mcpServers
model: string | Model;
The model implementation to use when invoking the LLM. By default, if not set, the agent will use the default model configured in modelSettings.defaultModel
Inherited from
Section titled “Inherited from”Agent.model
modelSettings
Section titled “modelSettings”modelSettings: ModelSettings;
Configures model-specific tuning parameters (e.g. temperature, top_p, etc.)
Inherited from
Section titled “Inherited from”Agent.modelSettings
name: string;
The name of the agent.
Inherited from
Section titled “Inherited from”Agent.name
outputGuardrails
Section titled “outputGuardrails”outputGuardrails: OutputGuardrail<AgentOutputType<unknown>>[];
A list of checks that run on the final output of the agent, after generating a response. Runs only if the agent produces a final output.
Inherited from
Section titled “Inherited from”Agent.outputGuardrails
outputType
Section titled “outputType”outputType: "text";
The type of the output object. If not provided, the output will be a string.
Inherited from
Section titled “Inherited from”Agent.outputType
resetToolChoice
Section titled “resetToolChoice”resetToolChoice: boolean;
Wether to reset the tool choice to the default value after a tool has been called. Defaults
to true
. This ensures that the agent doesn’t enter an infinite loop of tool usage.
Inherited from
Section titled “Inherited from”Agent.resetToolChoice
tools: Tool<RealtimeContextData<TContext>>[];
A list of tools the agent can use.
Inherited from
Section titled “Inherited from”Agent.tools
toolUseBehavior
Section titled “toolUseBehavior”toolUseBehavior: ToolUseBehavior;
This lets you configure how tool use is handled.
- run_llm_again: The default behavior. Tools are run, and then the LLM receives the results and gets to respond.
- stop_on_first_tool: The output of the frist tool call is used as the final output. This means that the LLM does not process the result of the tool call.
- A list of tool names: The agent will stop running if any of the tools in the list are called. The final output will be the output of the first matching tool call. The LLM does not process the result of the tool call.
- A function: if you pass a function, it will be called with the run context and the list of
tool results. It must return a
ToolsToFinalOutputResult
, which determines whether the tool call resulted in a final output.
NOTE: This configuration is specific to FunctionTools
. Hosted tools, such as file search, web
search, etc. are always processed by the LLM
Inherited from
Section titled “Inherited from”Agent.toolUseBehavior
readonly voice: string;
The voice intended to be used by the agent. If another agent already spoke during the RealtimeSession, changing the voice during a handoff will fail.
Accessors
Section titled “Accessors”outputSchemaName
Section titled “outputSchemaName”Get Signature
Section titled “Get Signature”get outputSchemaName(): string
Ouput schema name
Returns
Section titled “Returns”string
Inherited from
Section titled “Inherited from”Agent.outputSchemaName
Methods
Section titled “Methods”asTool()
Section titled “asTool()”asTool(options): FunctionTool
Transform this agent into a tool, callable by other agents.
This is different from handoffs in two ways:
- In handoffs, the new agent receives the conversation history. In this tool, the new agent receives generated input.
- In handoffs, the new agent takes over the conversation. In this tool, the new agent is called as a tool, and the conversation is continued by the original agent.
Parameters
Section titled “Parameters”Parameter | Type | Description |
---|---|---|
|
{ |
Options for the tool. |
|
( |
A function that extracts the output text from the agent. If not provided, the last message from the agent will be used. |
|
|
The description of the tool, which should indicate what the tool does and when to use it. |
|
|
The name of the tool. If not provided, the name of the agent will be used. |
Returns
Section titled “Returns”A tool that runs the agent and returns the output text.
Inherited from
Section titled “Inherited from”Agent.asTool
clone()
Section titled “clone()”clone(config): Agent<RealtimeContextData<TContext>, "text">
Makes a copy of the agent, with the given arguments changed. For example, you could do:
const newAgent = agent.clone({ instructions: 'New instructions' })
Parameters
Section titled “Parameters”Parameter | Type | Description |
---|---|---|
|
|
A partial configuration to change. |
Returns
Section titled “Returns”Agent
<RealtimeContextData
<TContext
>, "text"
>
A new agent with the given changes.
Inherited from
Section titled “Inherited from”Agent.clone
emit()
Section titled “emit()”emit<K>(type, ...args): boolean
Type Parameters
Section titled “Type Parameters”Type Parameter |
---|
|
Parameters
Section titled “Parameters”Parameter | Type |
---|---|
|
|
… |
|
Returns
Section titled “Returns”boolean
Inherited from
Section titled “Inherited from”Agent.emit
getAllTools()
Section titled “getAllTools()”getAllTools(): Promise<Tool<RealtimeContextData<TContext>>[]>
ALl agent tools, including the MCPl and function tools.
Returns
Section titled “Returns”Promise
<Tool
<RealtimeContextData
<TContext
>>[]>
all configured tools
Inherited from
Section titled “Inherited from”Agent.getAllTools
getMcpTools()
Section titled “getMcpTools()”getMcpTools(): Promise<Tool<RealtimeContextData<TContext>>[]>
Fetches the available tools from the MCP servers.
Returns
Section titled “Returns”Promise
<Tool
<RealtimeContextData
<TContext
>>[]>
the MCP powered tools
Inherited from
Section titled “Inherited from”Agent.getMcpTools
getSystemPrompt()
Section titled “getSystemPrompt()”getSystemPrompt(runContext): Promise<undefined | string>
Returns the system prompt for the agent.
If the agent has a function as its instructions, this function will be called with the runContext and the agent instance.
Parameters
Section titled “Parameters”Parameter | Type |
---|---|
|
|
Returns
Section titled “Returns”Promise
<undefined
| string
>
Inherited from
Section titled “Inherited from”Agent.getSystemPrompt
off<K>(type, listener): EventEmitter<EventTypes>
Type Parameters
Section titled “Type Parameters”Type Parameter |
---|
|
Parameters
Section titled “Parameters”Parameter | Type |
---|---|
|
|
|
(… |
Returns
Section titled “Returns”EventEmitter
<EventTypes
>
Inherited from
Section titled “Inherited from”Agent.off
on<K>(type, listener): EventEmitter<EventTypes>
Type Parameters
Section titled “Type Parameters”Type Parameter |
---|
|
Parameters
Section titled “Parameters”Parameter | Type |
---|---|
|
|
|
(… |
Returns
Section titled “Returns”EventEmitter
<EventTypes
>
Inherited from
Section titled “Inherited from”Agent.on
once()
Section titled “once()”once<K>(type, listener): EventEmitter<EventTypes>
Type Parameters
Section titled “Type Parameters”Type Parameter |
---|
|
Parameters
Section titled “Parameters”Parameter | Type |
---|---|
|
|
|
(… |
Returns
Section titled “Returns”EventEmitter
<EventTypes
>
Inherited from
Section titled “Inherited from”Agent.once
processFinalOutput()
Section titled “processFinalOutput()”processFinalOutput(output): string
Processes the final output of the agent.
Parameters
Section titled “Parameters”Parameter | Type | Description |
---|---|---|
|
|
The output of the agent. |
Returns
Section titled “Returns”string
The parsed out.
Inherited from
Section titled “Inherited from”Agent.processFinalOutput
toJSON()
Section titled “toJSON()”toJSON(): object
Returns a JSON representation of the agent, which is serializable.
Returns
Section titled “Returns”object
A JSON object containing the agent’s name.
name: string;
Inherited from
Section titled “Inherited from”Agent.toJSON
create()
Section titled “create()”static create<TOutput, Handoffs>(config): Agent<unknown, TOutput | HandoffsOutputUnion<Handoffs>>
Create an Agent with handoffs and automatically infer the union type for TOutput from the handoff agents’ output types.
Type Parameters
Section titled “Type Parameters”Type Parameter | Default type |
---|---|
|
|
|
[] |
Parameters
Section titled “Parameters”Parameter | Type |
---|---|
|
|
Returns
Section titled “Returns”Agent
<unknown
, TOutput
| HandoffsOutputUnion
<Handoffs
>>
Inherited from
Section titled “Inherited from”Agent.create