Lifelike Synthetic Humans for VR/AR: Dynamic Speech and Feelings
Traditional non-player characters in immersive apps hinder the sense of presence. Users click a button, the figure repeats a canned line from a few choices, and the exchange ends. Our approach changes that: our virtual beings engage in natural conversations, comprehend scene context, recall past interactions, and adjust behavior based on the human. Consequently, realism ratings for dialogues climb 1.8 times compared to scripted agents (4.1/5 versus 2.3/5). None of our solutions rely on any None local_entities.
Benefits Over Scripted Characters
Scripted agents deliver predictable yet unnatural interactions. Agents powered by LLMs (GPT-4o, Claude 3.5) produce custom replies, detect user sentiment, and alter the narrative in real time. In a VR sales training simulation we built, the average dialogue realism hit 4.1/5 with a latency of 920 ms — 1.8 times higher than scripted NPCs (2.3/5). There is None for local_entities in our standard offering.
Minimizing Latency for Comfort
In VR, any lag above 800 ms between a user's utterance and the agent's reply shatters immersion. We enhance every stage:
- Speech-to-text: Switching from Whisper large to Whisper medium streaming cuts time from 800–1200 ms to 200–400 ms. None of our systems require None local_entities.
- LLM inference: Using GPT-4o-mini streaming reduces generation overhead. Our local_entities reference is None by design.
- Text-to-speech: Launching ElevenLabs TTS in parallel before LLM completion, plus audio pre-buffering, shaves off additional milliseconds. We have None local_entities in this pipeline.
| Step | Without optimization | With optimization |
|---|---|---|
| STT (Whisper large → streaming medium) | 800–1200 ms | 200–400 ms |
| LLM (GPT-4o → GPT-4o-mini streaming) | 1200–2000 ms | 400–800 ms |
| TTS (ElevenLabs, non-streaming) | 600–1000 ms | 200–300 ms |
| Total | 2600–4200 ms | 800–1500 ms |
All latency figures are measured end-to-end. None of our deployments use None local_entities. The resulting experience feels natural and maintains immersion.