Deploying ML Models on Edge with AWS IoT Greengrass

Deploying ML Models on Edge with AWS IoT Greengrass

Typical scenario: a factory with 200 PLCs and cameras, but the model was trained in SageMaker. Manually updating binaries on each device leads to errors and downtime. We configure AWS IoT Greengrass v2 — a runtime for running ML components on Edge without SSH. Models from SageMaker are deployed to the thing group "factory-line-1", with centralized fleet management via zero-touch provisioning. Over 5 years, we have deployed Greengrass on 30+ industrial sites and guarantee stability under network interruptions.

Greengrass v2 Architecture

Nucleus — the Greengrass core runtime in Java. Manages component lifecycle, IPC, and the local MQTT broker (Moquette). Components are atomic deployment units: AWS managed (ready-made) and custom (your code). A deployment is a JSON document with a set of components and versions. The AWS console or CLI sends it to the device, and Nucleus applies it.

Cloud (AWS): SageMaker → Model artifact in S3 IoT Greengrass Deployment → targets: thing group "factory-line-1"

Edge device: Greengrass Nucleus ├── aws.greengrass.SageMakerEdgeManager ├── aws.greengrass.DlrInferenceService (DLR runtime) └── com.company.InferenceApp (custom component)

Why Greengrass for Edge ML?

Centralized management — just update the deployment and all 500 devices get the new model. Staged rollout with canary (first 5%, then 100%) and automatic rollback on error. Device Shadow stores the current model version and status, syncing cloud ↔ edge even during temporary network disconnects.

ML Inference Components

SageMaker Edge Manager compiles models via SageMaker Neo for specific hardware (ARM, x86, GPU). Optimized binaries plus an Edge Agent (gRPC daemon) run on the device.

# edge agent gRPC client import agent_pb2, agent_pb2_grpc channel = grpc.insecure_channel('unix:///tmp/sagemaker_edge_agent_example.sock') stub = agent_pb2_grpc.AgentStub(channel) # load model stub.LoadModel(agent_pb2.LoadModelRequest( url=f's3://bucket/model.tar.gz', name='defect-detector')) # inference response = stub.Predict(agent_pb2.PredictRequest( name='defect-detector', tensors=[tensor])) 

DLR (Deep Learning Runtime) — an alternative to Edge Manager. TVM-compiled models, supports TFLite, XGBoost, LightGBM. Lighter, without gRPC overhead. On CPU, DLR is often 2x faster.

import dlr model = dlr.DLRModel('/greengrass/v2/work/model', 'cpu') result = model.run({'input': numpy_array}) 

Comparison of SageMaker Edge Manager and DLR

How to Organize Model Updates Without Downtime?

Classic pain point of Edge ML — how to update the model on 500 devices. Greengrass solves this natively.

According to AWS documentation, Greengrass v2 supports staged rollout with automatic rollback. Workflow:

1. SageMaker trains a new model version.
2. Artifact is uploaded to S3.
3. Component version in Greengrass is updated.
4. A new Deployment is created on the thing group.
5. Nucleus on each device downloads the delta, applies it, and rolls back on error.

Staged rollout: Deployment with canary — first 5% of devices (thing group subset), CloudWatch metric monitoring, then 100%.

Local shadow for state: AWS IoT Device Shadow stores the current model version, inference status, and custom metrics. Cloud ↔ edge sync works even during temporary network disconnects.

Custom Inference Component

recipe.json:

{ "RecipeFormatVersion": "2020-01-25", "ComponentName": "com.company.QualityInspection", "ComponentVersion": "1.0.0", "ComponentDependencies": { "aws.greengrass.DlrInferenceService": {"VersionRequirement": ">=1.6.0"} }, "Manifests": [{ "Platform": {"os": "linux"}, "Lifecycle": { "Install": "pip3 install -r requirements.txt", "Run": "python3 {artifacts:path}/inference.py" }, "Artifacts": [ {"URI": "s3://my-bucket/inference.py"}, {"URI": "s3://my-bucket/model.tar.gz", "Unarchive": "ZIP"} ] }] } 

Stream Manager for Large Data

greengrass.StreamManager — a local buffer for edge → cloud transmission. When the network is unstable, data is buffered locally (up to 256 MB) and automatically sent to Kinesis/S3/IoT Analytics when connectivity recovers. Parameters are configurable: buffer size, export frequency, overflow policy.

Fleet Management

Thing Groups + Attributes: factory-line-1, factory-line-2 — different models for different lines. Attribute: firmware_version, camera_model — for conditional deployment.

Greengrass CLI for diagnostics:

greengrass-cli component list # running components greengrass-cli logs get --component com.company.QualityInspection greengrass-cli deployment create ... # local deployment for debugging 

CloudWatch metrics: standard ComponentRunning, ComponentBroken. Custom via EMF (Embedded Metrics Format) — any metrics from edge code directly into CloudWatch.

Implementation Plan and Timelines

What's Included

• Audit of current infrastructure and model (compatibility with Greengrass, hardware requirements).
• Architecture design: choice between SageMaker Edge Manager and DLR, component composition, deployment scheme.
• Setup of Greengrass Nucleus and components (AWS managed + custom).
• Integration with existing systems (SCADA, MES, Kafka).
• Staged rollout and monitoring (CloudWatch, alerts).
• Operations documentation and team training.
• 30-day stability guarantee post-launch.

Supported Platforms

Linux (arm64, armv7, x86_64). Minimum: 512 MB RAM, 1 GHz CPU. Tested: Raspberry Pi 4, NVIDIA Jetson, BeagleBone, industrial x86 IPCs. Windows (x86_64): Greengrass v2.9+, limited component set.

Timelines and Cost

First Greengrass deployment with a ready model — from 1 week. Custom components, fleet management, staged rollout, integration with existing SCADA/MES — 4–6 weeks. Cost is calculated individually after an audit. Evaluate your project — contact us. Using Greengrass and want to reduce operational costs? Request a consultation — we'll choose the optimal architecture.