Voice Agents on Twilio

Twilio offers a Media Streams API that sends the raw audio from a phone call to a WebSocket server. This setup can be used to connect your voice agents to Twilio. You can use the default Realtime Session transport in websocket mode to connect the events coming from Twilio to your Realtime Session. However, this requires you to set the right audio format and adjust your own interruption timing as phone calls will naturally introduce more latency than a web-based conversation.

To improve the setup experience, we’ve created a dedicated transport layer that handles the connection to Twilio, including interruption handling and audio forwarding.

Caution

This adapter is still in beta. You may run into edge case issues or bugs. Please report any issues via GitHub issues and we’ll fix quickly.

Setup

1. Make sure you have a Twilio account and a Twilio phone number.

2. Set up a server that authenticates requests from Twilio.

   If you are developing locally, you will need to configure a local tunnel like ngrok or Cloudflare Tunnel to make your local server accessible to Twilio. The server must validate X-Twilio-Signature on both the incoming call webhook and the WebSocket upgrade before creating a transport or connecting to OpenAI. Use the official Twilio request validator, the account’s primary Auth Token, and a configured public URL. Do not derive that URL from request Host or forwarded headers. TwilioRealtimeTransportLayer receives an already authenticated WebSocket; the adapter does not authenticate HTTP requests.

3. Install the Twilio adapter by installing the extensions package:

   npm install @openai/agents-extensions

4. Import the adapter and model to connect to your RealtimeSession:

   import { TwilioRealtimeTransportLayer } from '@openai/agents-extensions';
   import { RealtimeAgent, RealtimeSession } from '@openai/agents/realtime';
   
   const agent = new RealtimeAgent({
     name: 'My Agent',
   });
   
   // Authenticate the HTTP webhook and WebSocket upgrade in your server first.
   // Create a new transport mechanism that will bridge the connection between Twilio and
   // the OpenAI Realtime API.
   const twilioTransport = new TwilioRealtimeTransportLayer({
     twilioWebSocket: websocketConnection,
   });
   
   const session = new RealtimeSession(agent, {
     // set your own transport
     transport: twilioTransport,
   });

5. Connect your RealtimeSession to Twilio:

   import type { RealtimeSession } from '@openai/agents/realtime';
   
   // Connect only after the server has authenticated the Twilio WebSocket upgrade.
   async function connectAuthenticatedSession(
     session: RealtimeSession,
     apiKey: string,
   ) {
     await session.connect({ apiKey });
   }

Any event and behavior that you would expect from a RealtimeSession will work as expected including tool calls, guardrails, and more. Read the Voice agents guide for more information on how to use the RealtimeSession with voice agents.

Tips and considerations

1. Speed is the name of the game.

   In order to receive all the necessary events and audio from Twilio, you should create your TwilioRealtimeTransportLayer instance as soon as the authenticated WebSocket connection is available and immediately call session.connect() afterwards.

2. Access the raw Twilio events.

   If you want to access the raw events that are being sent by Twilio, you can listen to the transport_event event on your RealtimeSession instance. Every event from Twilio will have a type of twilio_message and a message property that contains the raw event data.

3. Watch debug logs.

   Sometimes you may run into issues where you want more information on what’s going on. Using a DEBUG=openai-agents* environment variable will show all the debug logs from the Agents SDK. Alternatively, you can enable just debug logs for the Twilio adapter using DEBUG=openai-agents:extensions:twilio*.

Full example server

The runnable Twilio example includes authenticated Fastify routes and Realtime session setup. Its README describes the required server-side OPENAI_API_KEY, TWILIO_AUTH_TOKEN, and TWILIO_PUBLIC_BASE_URL settings. The public base URL is an HTTPS origin without a path prefix, query string, or fragment. Update that setting when a tunnel’s public origin changes.

The server rejects invalid call and upgrade signatures before opening a Realtime connection. Twilio Media Stream URLs use wss:// and do not support query parameters. See the Twilio Stream documentation for custom parameters.