Setting up real device testing via Firebase Test Lab

Setting Up Testing on Real Devices via Firebase Test Lab

Emulator lies. Not maliciously, but systematically: GPU rendering different, Bluetooth unavailable, camera simulated, memory behavior differs from real device. Firebase Test Lab provides access to physical devices — Pixel 8, Samsung Galaxy S24, tablets, old Xiaomi with custom ROMs. A test passing on emulator may fail on Samsung One UI due to custom WebView or changed permission system.

What Firebase Test Lab Does

Two testing modes:

Instrumentation Tests — runs your Espresso / XCUITest tests on selected devices. Full control what's checked.

Robo Test — automatic crawler. Run APK without single written test, Robo itself crawls interface through Accessibility tree, taps buttons, fills fields with random data, searches for crashes. Useful for quick validation of new builds before release.

Platform support: Android (Espresso, UI Automator, Appium) and iOS (XCUITest). Firebase Test Lab for iOS — separate story with smaller device fleet, but real iPhone and iPad still more valuable than simulator.

Setup: From Build Upload to Results

Android

APK and test APK build:

./gradlew assembleDebug assembleAndroidTest

Upload and execution via gcloud:

gcloud firebase test android run \
  --type instrumentation \
  --app app/build/outputs/apk/debug/app-debug.apk \
  --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
  --device model=Pixel8,version=34,locale=en,orientation=portrait \
  --device model=a54xnsxx,version=13,locale=en,orientation=portrait \
  --results-bucket gs://my-project-test-results \
  --results-dir "run_$(date +%Y%m%d_%H%M%S)"

model=a54xnsxx is Samsung Galaxy A54. Available models list: gcloud firebase test android models list.

Device Matrix

Choosing matrix — separate analytical task. No point running tests on 50 devices. Principle:

Realistic matrix for most projects: 3–5 devices. More — costlier and longer, but not necessarily more informative.

iOS

For iOS need .ipa with development signature (not distribution). Upload:

gcloud firebase test ios run \
  --test MyAppTests.zip \
  --device model=iphone15pro,version=17.4,locale=en_US,orientation=portrait

MyAppTests.zip is archive with .xctestrun file and test products. Built through xcodebuild build-for-testing.

Robo Test with Custom Script

Pure Robo crawler sometimes gets stuck on login screen — doesn't know credentials. Robo Script allows setting initial actions:

[
  {
    "eventType": "VIEW_TEXT_CHANGED",
    "replacementText": "[email protected]",
    "elementDescriptors": [{"resourceName": "com.example.app:id/email_input"}]
  },
  {
    "eventType": "VIEW_CLICKED",
    "elementDescriptors": [{"resourceName": "com.example.app:id/login_button"}]
  }
]

After authorization Robo continues crawl of already authorized app. Fast way to check regressions without writing tests.

CI Integration

GitHub Actions:

- name: Set up gcloud
  uses: google-github-actions/setup-gcloud@v2
  with:
    service_account_key: ${{ secrets.GCLOUD_SERVICE_ACCOUNT_KEY }}
    project_id: my-firebase-project

- name: Run tests on Firebase Test Lab
  run: |
    gcloud firebase test android run \
      --app app-debug.apk \
      --test app-debug-androidTest.apk \
      --device model=Pixel8,version=34 \
      --timeout 10m

Service account needs role Firebase Test Lab Admin. Results auto-saved to Cloud Storage — screenshots, video, logcats, XML report.

Results Analysis

Firebase Test Lab console shows for each run:

• Video recording of test execution
• Logcat with tag filters
• Screenshots at key points
• Coverage report (if jacoco enabled)
• XML results in JUnit format (for Allure / Jenkins)

Timeline

2–3 days — Firebase Test Lab setup, device matrix configuration, CI integration, Robo Script setup for authorization. Plus time for first results analysis and fixing device-specific issues. Cost is calculated individually.