Setting up real device testing via AWS Device Farm

Setting Up Testing on Real Devices via AWS Device Farm

AWS Device Farm is cloud farm of real mobile devices from Amazon. If project infrastructure already on AWS (CodePipeline, CodeBuild, S3, IAM), Device Farm integrates organically: same IAM roles, same AWS CLI, same access policy. No separate login to third-party system needed.

Two Working Models

Automated Testing — runs Appium, Espresso, XCUITest, Calabash tests on device fleet. Upload APK/IPA and test package, select devices, run.

Remote Access — interactive session with real device in browser. Useful for manually reproducing bug, checking UI on specific model, or debugging automated tests.

Setting Up Appium Tests for Device Farm

Device Farm requires Appium tests packaged in ZIP with specific structure. For WebdriverIO (Node.js):

tests.zip/
├── package.json
├── package-lock.json
├── node_modules/        # all dependencies included
└── test/
    └── specs/
        └── login.test.js

All node_modules must be inside archive — Device Farm doesn't run npm install. WebdriverIO config for Device Farm:

// wdio.conf.devicefarm.js
exports.config = {
  runner: 'local',
  specs: ['./test/specs/**/*.js'],
  // capabilities taken from Device Farm environment via env vars
  capabilities: [{
    platformName: process.env.DEVICEFARM_DEVICE_PLATFORM_NAME,
    'appium:deviceName': process.env.DEVICEFARM_DEVICE_NAME,
    'appium:platformVersion': process.env.DEVICEFARM_DEVICE_OS_VERSION,
    'appium:app': process.env.DEVICEFARM_APP_PATH,
    'appium:automationName': process.env.DEVICEFARM_DEVICE_PLATFORM_NAME === 'iOS' ? 'XCUITest' : 'UiAutomator2',
  }],
  hostname: 'localhost',
  port: 4723,
};

Device Farm itself launches Appium server on device, environment variables with device parameters passed automatically.

Native Tests: Espresso

Native Espresso without Appium executes faster. Upload via AWS CLI:

# Create project (once)
PROJECT_ARN=$(aws devicefarm create-project --name "MyApp" --query 'project.arn' --output text)

# Upload APK
APP_UPLOAD=$(aws devicefarm create-upload \
  --project-arn $PROJECT_ARN \
  --name "app-debug.apk" \
  --type ANDROID_APP \
  --query 'upload.{arn:arn,url:url}' --output json)

APP_URL=$(echo $APP_UPLOAD | jq -r '.url')
APP_ARN=$(echo $APP_UPLOAD | jq -r '.arn')

curl -T app/build/outputs/apk/debug/app-debug.apk "$APP_URL"

# Upload test APK
TEST_UPLOAD=$(aws devicefarm create-upload \
  --project-arn $PROJECT_ARN \
  --name "app-debug-androidTest.apk" \
  --type INSTRUMENTATION_TEST_PACKAGE \
  --query 'upload.{arn:arn,url:url}' --output json)

TEST_ARN=$(echo $TEST_UPLOAD | jq -r '.arn')
curl -T app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk "$(echo $TEST_UPLOAD | jq -r '.url')"

Run execution:

aws devicefarm schedule-run \
  --project-arn $PROJECT_ARN \
  --app-arn $APP_ARN \
  --device-pool-arn $POOL_ARN \
  --name "Espresso Run $(date)" \
  --test type=INSTRUMENTATION,testPackageArn=$TEST_ARN,filter="com.example.LoginTest"

Device Pool Configuration

Device pool — set of filter rules:

aws devicefarm create-device-pool \
  --project-arn $PROJECT_ARN \
  --name "Android Top Devices" \
  --rules '[
    {"attribute": "PLATFORM", "operator": "EQUALS", "value": "ANDROID"},
    {"attribute": "OS_VERSION", "operator": "GREATER_THAN_OR_EQUALS", "value": "13"},
    {"attribute": "MANUFACTURER", "operator": "IN", "value": "[\"Samsung\",\"Google\"]"},
    {"attribute": "AVAILABILITY", "operator": "EQUALS", "value": "HIGHLY_AVAILABLE"}
  ]' \
  --max-devices 5

AVAILABILITY: HIGHLY_AVAILABLE — Device Farm selects only devices available right now. Without this run may queue.

CodePipeline Integration

In AWS CodeBuild specification:

phases:
  build:
    commands:
      - ./gradlew assembleDebug assembleAndroidTest
      - |
        APP_ARN=$(aws devicefarm create-upload \
          --project-arn $DEVICE_FARM_PROJECT_ARN \
          --name "app.apk" --type ANDROID_APP \
          --query 'upload.arn' --output text)
        # ... upload and run
      - aws devicefarm get-run --arn $RUN_ARN --query 'run.result'

CodeBuild IAM role must have permissions devicefarm:* on specific project or *. Minimum set: CreateUpload, ScheduleRun, GetRun, ListArtifacts.

Results Analysis

After run completion artifacts available via API:

aws devicefarm list-artifacts \
  --arn $JOB_ARN \
  --type FILE \
  --query 'artifacts[*].{name:name,url:url}' \
  --output table

Typical artifacts: Logcat, Screenshots, Video, Test spec output. Download to S3 or view in Device Farm console.

Timeline

2–3 days — AWS Device Farm setup, device pool creation, build upload configuration, CodePipeline integration or GitHub Actions, first run and results review. Cost is calculated individually.