Mobile App Development for Robot Control

An engineer faces the challenge of remotely controlling a robot via a mobile device with latency under 100 ms. The choice of data transfer protocol determines reaction speed, safety, and development cost. Incorrect architecture can lead to connection loss and accidents. We specialize in developing robot control applications — from AGVs to collaborative manipulators. Over 5+ years, we have delivered 15+ projects, achieving latencies below 100 ms even over LTE.

Which protocol should you choose for minimal latency?

The transport selection defines latency, reliability, and development complexity. In our practice, we use three main protocols:

UDP provides 10x lower latency than MQTT but does not guarantee delivery — this is an advantage for real-time scenarios where a fresh command is more important than reliability. MQTT with QoS 1 is more reliable but adds overhead. For ROS robots, WebSocket via rosbridge is optimal, although JSON serialization can become a bottleneck at high frequencies (according to the official rosbridge_protocol documentation).

Why is a watchdog mandatory?

A critical scenario is connection loss. Without protection, the robot may continue moving, which is dangerous. We implement a watchdog in steps:

1. On the robot, launch a timer of 1–2 seconds.
2. If no command arrives within that time, the robot transitions to safe_stop (stop all motors) or holds position.
3. The phone displays connection status and automatically reconnects with a Quality of Service indicator.

Additionally, we use PIN/QR authentication to prevent control of another robot.

How to reduce video stream latency?

For H.264 streams from IP cameras on the robot, we use ExoPlayer (Android) or AVPlayer with HLS (iOS). HLS latency is 5–30 seconds, which is unacceptable for control. The correct approach is WebRTC (latency < 200 ms) or RTSP via LibVLC/ffmpeg in a MediaCodec pipeline (latency 100–300 ms).

WebRTC is more complex to set up but offers the lowest latency and adaptive bitrate. For local control (Wi-Fi), RTSP provides an optimal balance.

Transport and protocols (in detail)

ROS 2 + rosbridge (WebSocket)

The most common stack for ROS robots: on the robot, rosbridge_server is launched; the phone connects via WebSocket and publishes/subscribes to topics using the JSON protocol rosbridge_protocol. For mobile clients, there are roslibjs-compatible wrappers, but for native Android/iOS we write the client ourselves — this is 300–400 lines of code with reconnection and message queue.

Problem: rosbridge is JSON over WebSocket, which is slow for high-frequency topics. The /cmd_vel topic with Twist commands at 20 Hz generates ~40 kbps of traffic. If the robot is on Wi-Fi with real throughput of 1 Mbps, it's fine. If on LTE with jitter, packets arrive in bursts, causing delayed command execution. Solution: reduce command frequency to 10 Hz and add a watchdog — if no command from the phone for 500 ms, the robot transitions to safe_stop.

MQTT for lightweight control

For IoT robots without ROS (AGVs, sorters, custom platforms) — MQTT broker (Mosquitto or EMQ X) plus a lightweight command protocol. The phone publishes to robot/{id}/cmd, the robot subscribes to that topic. Telemetry goes the other way: robot/{id}/state, robot/{id}/battery. We use MQTT with QoS 1 (at least once) for control commands — QoS 0 loses packets over unstable Wi-Fi, QoS 2 creates extra round trips. Retained message for robot/{id}/state allows a newly connected client to immediately get the current state without waiting for the next update.

UDP for real-time (< 50 ms)

If minimal latency is required, use a direct UDP socket on the control port. On Android: DatagramSocket in CoroutineScope(Dispatchers.IO), sending every 50 ms. No delivery guarantees — this is an advantage: old commands don't block new ones in the queue. Used for manipulator control where command latency is more important than delivery guarantee.

robot/{id}/cmd — control commands (velocity, gripper) robot/{id}/state — telemetry (position, battery) robot/{id}/video — stream metadata (URL, codec) 

Mobile client architecture

class RobotControlViewModel( private val robotRepository: RobotRepository ) : ViewModel() { private val _robotState = MutableStateFlow<RobotState>(RobotState.Disconnected) val robotState: StateFlow<RobotState> = _robotState fun sendVelocityCommand(linear: Float, angular: Float) { viewModelScope.launch { robotRepository.publishVelocity( TwistCommand(linear = linear, angular = angular) ) } } fun connect(robotIp: String) { viewModelScope.launch { robotRepository.connect(robotIp) .onEach { state -> _robotState.value = state } .launchIn(this) } } } 

RobotRepository encapsulates the specific transport — WebSocket, MQTT, or UDP. Changing the transport does not affect the ViewModel or UI. This is critical: in real projects, the hardware or protocol often changes between prototype and production.

Virtual joystick and input handling

MotionEvent.ACTION_MOVE fires up to 60 times per second during fast finger movement. Sending a command on every event overloads the channel. We use throttleLatest(50) from Kotlin Coroutines Flow — it takes the latest value within a 50 ms window. Old intermediate values are discarded, and response latency stays below 50 ms.

Dead zone in the joystick center: we filter 10–15% of the radius to zero. Without this, micro-tremors cause constant low-speed commands, making the robot "twitch" at rest.

What is included in development?

We provide a full service package:

• Protocol architecture design (transport selection, command scheme)
• Native mobile app development for Android and/or iOS
• Video streaming integration via WebRTC or RTSP
• Watchdog and emergency recovery scenario development
• Testing on a real robot under conditions close to operation
• Technical documentation and operator training

Timelines

A basic client with joystick, telemetry, and video stream on one platform — 3–5 weeks. A cross-platform solution supporting multiple protocols, a facility map, and autonomous missions — 2–4 months. Exact timelines are determined after analyzing the robot platform and latency requirements.

We will evaluate your project and offer an optimal solution. Get a consultation on protocol architecture. Contact us for a detailed discussion.