Real-Time Speech-to-Speech with OpenAI & Twilio: Full SIP Integration Guide

Author

Akash Gupta | Sr. VoIP Engineer | MLOps

The OpenAI Realtime API is a speech-to-speech large language model (LLM) designed for live, natural conversations. Unlike traditional systems that require separate speech-to-text (STT) and text-to-speech (TTS) pipelines, the Realtime API can directly interact with caller audio. This means faster responses, lower latency, and more natural back-and-forth dialogue—closer to how humans talk on the phone.

To connect the Realtime API with phone networks, OpenAI supports SIP (Session Initiation Protocol), the industry standard for initiating and managing voice calls. SIP is the protocol used by most VoIP systems, PBXs, and carriers (like Twilio). By combining OpenAI Realtime with SIP, you can let callers dial a regular phone number and talk to an AI agent in real time, with no extra translation layers in between.

In this guide we’ll:

Set up a webhook that controls and guides your AI agent
Register that webhook in the OpenAI console
Cofigure environment and run a tiny Python app to handle call events
Test the agent by dialing OpenAI’s SIP URI from a SIP softphone
Create a Twilio Elastic SIP Trunk and connect a phone number
Place a real call through Twilio to your OpenAI agent

What you’ll need

An OpenAI account with access to the Realtime API (now GA with SIP support). (OpenAI)
A Twilio account with Elastic SIP Trunking enabled. (Twilio)
Free/Paid ngrok account for a public HTTPS tunnel to your local webhook. (Ngrok)
A SIP softphone for testing SIP calls during development.

Call Flow

Caller makes a call to a phone number
Inbound call hits your Twilio phone number
Twilio sends the call to OpenAI SIP URI using Session Initiation Protocol
Call hits OpenAI SIP endpoint and triggers a request to your webhook; your app receives the request and replies with instructions (voice, behavior, tools) for the agent.
Optionally your app can invokes a websocket connection with OpenAI realtime API to receive call events.

Step 1 — Create a webhook (ngrok)

Install ngrok cli and connect your account, follow the instructions for your OS here. below are the instructions for MacOS.

brew install ngrok
ngrok config add-authtoken $YOUR_TOKEN

Now run the following command to get the webhook. This command will connect your local port 8000 to the webhook allowing your application to communicate with OpenAI realtime API.
Note: for the free account the webhook URL is temporary and will be change each time the below command is run.

ngrok http 8000

Note the Forwarding URL (e.g., https://<random>.ngrok.io)—you’ll paste it into the OpenAI console.

Step 2 — Configure the webhook in the OpenAI console and get the SIP URI

In the OpenAI Console (https://platform.openai.com/) under your project settings → Realtime (SIP) settings, add your webhook URL that was generated in previous step.
Choose event type realtime.call.incoming (triggers when a SIP call hits your project). Check the image above for reference.
Once you save your webhook copy the signing secret and save it to your .env file along with your openai api key.
You can find the sample python code in my Github repository here . Clone the repo, update the .env file and then just run the python app using below command.

python handle_webhook.py

Step 3 — (Optional) Make a direct test call to OpenAI via SIP

Before wiring Twilio, you can call the OpenAI SIP URI from a softphone to validate that your webhook is working and your agent responds. I am using pjsua which is a terminal softphone, but you can use any SIP softphone that supports TLS.

Example with pjsua (replace placeholders):
Run the below command after installing pjsua in your teminal. To disconnect the call just press q and press enter (return on MacOS)

pjsua \
  --id="sip:test@voipnuggets.com" \
  --use-tls \
  "sip:<YOUR_OPENAI_PROJECT_SIP_ID>@sip.api.openai.com;transport=tls"

The id can be any valid sip URI as it is not validated by OpenAI.
The destination URI’s user part is your OpenAI project SIP ID (format like proj_...). You can find it under Projects in sidebar menu at https://platform.openai.com/settings
pjsua is perfect for quick call testing during development.

To install pjsua on MacOS simply run the following command

brew install pjproject

For linux I prefer to build it from source – checkout my article about PJSUA to learn more – here

Step 4 — End-to-end test of the webhook

With python app and ngrok running:

Place the pjsua call above.
Watch your app logs—on the first ring, OpenAI will POST the realtime.call.incoming event to your webhook.
Your JSON response sets the agent’s voice and instructions, and the model should greet you within a second.

Step 5 — Create a Twilio Elastic SIP Trunk

Now let’s put a real phone number in front.

Create a Trunk
In Twilio Console → Elastic SIP Trunking, create a new trunk (e.g., openai-realtime-trunk).

Origination (from Twilio → OpenAI)
Configure the Origination SIP URI so Twilio can send calls from your number to OpenAI’s SIP endpoint. Example

   ORIGINATION URI	                                       PRIORITY	WEIGHT	ENABLED	
sip:proj_123@sip.api.openai.com;transport=tls               10	     10	      ✔

Step 6 — Attach a phone number & route calls to the trunk

In Twilio Console → Phone Numbers, buy/choose a number and set its Voice routing to Elastic SIP Trunking → choose your trunk (openai-realtime-trunk).
You can also attach the number to the trunk under Elastic SIP Trunking by clicking the numbers tab in the sidebar.
Place a call to that number—Twilio will deliver it to OpenAI over SIP.

Customizing your voice agent

Tune the greeting & style: Edit the instructions string in your webhook response.
Switch voices: Set "voice": "alloy" (or another supported voice). (OpenAI Platform)
Add tools: Extend the webhook to include tool schemas and function calls (CRM lookup, ticketing, etc.).
Persist memory: Store call context in your app and feed it back through session updates.

For a deeper dive into realtime agents (architecture, session config, code samples), check the OpenAI Agents SDK Realtime guides. (OpenAI Cookbook)

Troubleshooting

Call connects but the agent is silent
- Verify your webhook URL is reachable (open the ngrok Forwarding URL).
- Ensure you return a JSON body with response.instructions and response.voice.
400 from python app/ webhook
Verify your signature in the .env file, it should be the same that you got after saving the webhook.
SIP error from softphone
- Re-check TLS usage and the destination URI format: sip:<YOUR_OPENAI_PROJECT_SIP_ID>@sip.api.openai.com;transport=tls.
- Confirm your account has Realtime SIP access as per current GA updates.
Twilio trunk routes nowhere
- Confirm the number is attached to the trunk, and the trunk’s Termination settings point to the intended SIP destination. Twilio’s step-by-step trunking guides are handy here.

Production checklist

Use environment variables for secrets
Implement webhook signature verification
Add observability (call detail logs, webhook metrics)
Fail-safes: friendly fallback prompts, max turn lengths, escalation to human using call forwarding.

References & further reading

OpenAI Realtime API (SIP/Realtime) — Docs & GA announcement. (OpenAI Platform, OpenAI)
Twilio Elastic SIP Trunking — Concepts and setup guides/tutorials. (Twilio)
pjsua — Quick examples and CLI reference. (docs.pjsip.org)
ngrok — CLI quickstart and config reference. (Ngrok)
OpenAI Agents SDK Realtime guides. (OpenAI Cookbook)

Akash Gupta
Senior VoIP Engineer and AI Enthusiast

AI and VoIP Blog

Thank you for visiting the Blog. Hit the subscribe button to receive the next post right in your inbox. If you find this article helpful don't forget to share your feedback in the comments and hit the like button. This will helps in knowing what topics resonate with you, allowing me to create more that keeps you informed.

Thank you for reading, and stay tuned for more insights and guides!

AI and VoIP Blog