Real-Time AI Speech Translation for iOS and Android: A Technical Guide

When implementing a voice assistant or real-time translation app, end-to-end latency over 3 seconds kills UX — the speaker starts interrupting or loses the thread. We solve this problem by parallelizing three stages: audio capture → transcription → translation → voice synthesis. With proper architecture, users don't notice pauses even in fast dialogue. In a sequential configuration, latency reaches 8–15 seconds — unacceptable for live conversation. Pipeline parallelism allows processing audio, text, and speech simultaneously: while TTS is voicing one phrase, STT is already capturing the next. As a result, end-to-end latency drops to 1.5–3 seconds. For example, in a project for a financial company, we implemented a parallel pipeline with Deepgram Nova-2 and Yandex SpeechKit. The latency for Russian-English pair was 2.1 seconds, enabling live negotiations without pauses. The client reported a 60% reduction in repetitions. Integration cost for one cloud service starts from $2,000, and savings on negotiations can reach $10,000 per year. Get a consultation for your project — we will assess the architecture and select the optimal stack. Below are key technical solutions for iOS and Android that we've refined across 40+ projects.

How the speech translation pipeline works

Microphone → VAD → buffer 2-3 sec → STT API → source text ↓ Translation API → translated text ↓ TTS API → audio → speaker 

Each block can be parallelized. While TTS is synthesizing the first sentence, STT is already processing the next fragment. This is called pipeline parallelism and halves the end-to-end latency.

How to choose STT for streaming?

Whisper — no. Deepgram Nova-2 or Google Speech-to-Text v2 with interim_results — yes. For AI speech translation, you need streaming STT; otherwise you have to wait for a full pause.

Deepgram with interim_results=true and utterance_end_ms=1200 delivers text within 300–500 ms after the phrase ends. This is the working window to launch translation.

Implementation on iOS (Swift)

class SpeechTranslationPipeline { private let deepgramStreamer: DeepgramStreamer private let translator: TranslationService private let tts: AVSpeechSynthesizer func handleFinalTranscript(_ text: String, sourceLang: String, targetLang: String) async { // Launch translation immediately after receiving final utterance async let translated = translator.translate(text, from: sourceLang, to: targetLang) // In parallel, show source text in UI await MainActor.run { sourceLabel.text = text } let translatedText = try? await translated guard let result = translatedText else { return } await MainActor.run { targetLabel.text = result } // TTS let utterance = AVSpeechUtterance(string: result) utterance.voice = AVSpeechSynthesisVoice(language: targetLang) utterance.rate = 0.52 tts.speak(utterance) } } 

AVSpeechSynthesizer — system TTS on iOS. For Russian voice, quality is acceptable but noticeably worse than ElevenLabs or OpenAI TTS. If natural voice is needed, replace the TTS block with cloud-based speech synthesis with audio caching.

Audio session management

When simultaneously capturing microphone and playing translation, there is a conflict with AVAudioSession. The category must be .playAndRecord with option .defaultToSpeaker:

try AVAudioSession.sharedInstance().setCategory( .playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetooth] ) 

The .voiceChat mode enables echo cancellation. Without it, the translation from the speaker will loop back into the microphone and go through transcription again. Apple Developer Documentation

Implementation on Android (Kotlin)

class SpeechTranslationPipeline @Inject constructor( private val deepgramStreamer: DeepgramStreamer, private val translationRepo: TranslationRepository, private val tts: TextToSpeech ) { fun start(sourceLang: String, targetLang: String) { deepgramStreamer.onFinalTranscript = { text -> coroutineScope.launch { val translated = translationRepo.translate(text, targetLang) withContext(Dispatchers.Main) { sourceTextView.text = text targetTextView.text = translated } speakTranslation(translated, targetLang) } } deepgramStreamer.start() } private fun speakTranslation(text: String, lang: String) { tts.language = Locale.forLanguageTag(lang) tts.speak(text, TextToSpeech.QUEUE_FLUSH, null, null) } } 

AudioManager.MODE_IN_COMMUNICATION + AudioRecord with source VOICE_COMMUNICATION — for proper AEC (acoustic echo cancellation) on Android. Otherwise, devices without hardware AEC will have echo.

What challenges arise with parallel processing?

While TTS is uttering the translation, the user might be speaking the next phrase. If VAD does not account for this, the microphone will pick up the voice from the speaker. Solution:

• Pause VAD during TTS playback
• Or additional filtering: ignore interim results during audio playback

In practice, the second option is more reliable as it avoids awkward pauses.

Comparison of sequential and parallel approaches

Comparison of providers by quality and latency

For Russian voice synthesis, Yandex SpeechKit is significantly better than Google TTS and OpenAI in naturalness. This is not an opinion — it is verifiable on a test set of 50 phrases.

Offline variant

For devices without stable internet: Whisper on-device (whisper.cpp via CoreML on iOS, ONNX on Android) + ML Kit Translate + system TTS. Latency is 3–6 seconds instead of 1.5, but works offline.

Whisper tiny/base on iPhone 13 via CoreML — about 2 seconds for a 5-second segment. Acceptable for a travel scenario.

Steps to implement real-time speech translation in a mobile app

1. Choose a streaming STT provider (Deepgram or Google STT v2) and set up audio streaming with interim results.
2. Configure audio session for echo cancellation on iOS and audio source for AEC on Android.
3. Integrate translation API (DeepL, Google Translate, or GPT-4o) to convert transcribed text.
4. Integrate TTS API (OpenAI, Yandex SpeechKit, or system TTS) for voice output.
5. Implement pipeline parallelism: chain STT→translation→TTS so that each stage works concurrently on different audio segments.
6. Test end-to-end latency and adjust VAD parameters for optimal phrase detection.

What's included in the work

• Integration of STT, Translation, TTS with audio session management
• VAD and pipeline parallelism configuration
• Handling network interruptions and offline mode
• Basic UI showing source and translated text
• Integration documentation and test scenarios
• Support for 2 weeks after delivery
• Cost per cloud service integration from $2,000, cross-platform version from $5,000

Timeline and cost

Streaming speech translation with cloud services on a single platform — 2–4 weeks. Cross-platform implementation on Flutter with native audio bridges — 3–5 weeks. Cost is calculated individually based on language pairs and synthesis quality requirements.

Get a consultation — we will assess your project and propose the optimal solution.

Our experience and guarantees

Our team has 5+ years of experience in mobile development and has completed 40+ projects with voice interfaces. We guarantee end-to-end latency no more than 3 seconds on supported devices. Certified native audio bridges for iOS and Android ensure correct AEC operation.

Typical implementation mistakes:

• Skipping audio session configuration: echo and duplicate transcription.
• Using non-streaming STT (Whisper API) — latency 10+ seconds.
• No TTS caching — repeated synthesis of identical phrases.
• Not accounting for network state — interruption leads to context loss.

Get a consultation — a sample implementation and assessment of your project in 1 day.