AI Computer Vision Systems for Industrial Robots

Without vision, a robot is a blind automaton strictly following a program. CV adds adaptability: grasping arbitrarily oriented parts, in-process inspection, navigation in dynamic environments, and safe human collaboration per ISO 15066. We integrate computer vision systems into industrial manipulators and mobile robots — this combination delivers: 6DoF pose estimation for bin picking, depth-guided grasping, and semantic mapping for AMRs. Savings on a single bin picking station are substantial due to reduced manual labor and defects.

How CV Solves the Bin Picking Problem

Bin picking is one of the most demanded and challenging tasks. Parts in a container overlap each other, are chaotically oriented, and often have reflective surfaces. The primary method is 6DoF pose estimation: determining position (x,y,z) and rotation (roll,pitch,yaw) of each part. We use RGB-D cameras (RealSense, Azure Kinect) and one of the following models:

• FoundationPose — state-of-the-art for parts with a known CAD model. Achieves ADD-0.1d 78–89%.
• GDR-Net — geometrically discretized rendering, works without CAD but with lower accuracy.
• PVPN (Point Voting) — robust to heavy noise and partial occlusions.

Example implementation in PyTorch (abbreviated):

import numpy as np import cv2 import torch from dataclasses import dataclass from typing import Optional @dataclass class ObjectPose: object_class: str position_xyz: tuple[float, float, float] # mm in camera coordinate system rotation_matrix: np.ndarray # 3×3 euler_angles: tuple[float, float, float] # roll, pitch, yaw in degrees confidence: float grasp_point: tuple[float, float, float] # recommended grasp point grasp_approach: np.ndarray # approach vector class BinPickingSystem: """ Bin picking system: detection and pose estimation of parts in a container. Methods: 1. FoundationPose / DenseFusion: based on RGB-D 2. GDR-Net: geometrically discretized rendering 3. PVPN: point-wise voting Camera: Intel RealSense D435i or Azure Kinect. CAD model of the part is required for FoundationPose. """ def __init__(self, pose_model_path: str, cad_model_path: str, object_classes: list[str], camera_intrinsics: dict, device: str = 'cuda'): self.device = device self.object_classes = object_classes self.camera_intrinsics = camera_intrinsics # fx, fy, cx, cy # Load pose estimation model self.pose_model = torch.load(pose_model_path, map_location=device).eval() # CAD model for rendering (used by FoundationPose) self.cad_model = self._load_cad_model(cad_model_path) # YOLO for initial object detection from ultralytics import YOLO self.detector = YOLO(pose_model_path.replace('pose', 'det')) def _load_cad_model(self, cad_path: str): """Load .ply or .obj CAD model""" try: import open3d as o3d return o3d.io.read_triangle_mesh(cad_path) except ImportError: return None def estimate_poses(self, rgb: np.ndarray, depth: np.ndarray) -> list[ObjectPose]: """ Estimate object poses. rgb: (H, W, 3) uint8 depth: (H, W) float32 in millimeters """ # 1. Detect objects for ROI detections = self.detector(rgb, conf=0.4, verbose=False) poses = [] for box in detections[0].boxes: cls_id = int(box.cls.item()) if cls_id >= len(self.object_classes): continue x1, y1, x2, y2 = map(int, box.xyxy[0]) # Crop RGB and depth patches rgb_crop = rgb[y1:y2, x1:x2] depth_crop = depth[y1:y2, x1:x2] if rgb_crop.size == 0: continue # 2. Pose estimation on the patch pose = self._estimate_single_pose( rgb_crop, depth_crop, cls_id, (x1, y1) ) if pose: poses.append(pose) # Sort by Z height (top parts first) poses.sort(key=lambda p: p.position_xyz[2]) return poses @torch.no_grad() def _estimate_single_pose(self, rgb_crop: np.ndarray, depth_crop: np.ndarray, cls_id: int, offset: tuple) -> Optional[ObjectPose]: """Pose estimation for a single object""" from torchvision import transforms transform = transforms.Compose([ transforms.ToTensor(), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) ]) from PIL import Image pil = Image.fromarray(cv2.cvtColor(rgb_crop, cv2.COLOR_BGR2RGB)) rgb_tensor = transform(pil).unsqueeze(0).to(self.device) depth_tensor = torch.from_numpy(depth_crop).unsqueeze(0).unsqueeze(0).float().to(self.device) # Concatenate RGB + depth depth_norm = depth_tensor / 1000.0 # mm → meters # Simplified model takes 4-channel input input_tensor = torch.cat([ rgb_tensor, torch.nn.functional.interpolate( depth_norm, size=rgb_tensor.shape[-2:], mode='bilinear' ) ], dim=1) output = self.pose_model(input_tensor) # output: (1, 6) — translation(3) + rotation_euler(3) if output is None or output.shape[-1] < 6: return None out_np = output.squeeze().cpu().numpy() tx, ty, tz = out_np[:3] * 1000 # meters → mm rx, ry, rz = np.degrees(out_np[3:6]) # Rotation matrix from Euler angles R, _ = cv2.Rodrigues(np.array([np.radians(rx), np.radians(ry), np.radians(rz)])) # Grasp point: object center + offset upward along normal grasp_z = tz - 30 # 30mm above center grasp_point = (tx, ty, grasp_z) approach_vec = R @ np.array([0, 0, -1]) # approach direction conf = float(torch.sigmoid( self.pose_model.confidence_head(output) if hasattr( self.pose_model, 'confidence_head') else torch.tensor(0.0) ).item()) if hasattr(self.pose_model, 'confidence_head') else 0.8 return ObjectPose( object_class=self.object_classes[cls_id], position_xyz=(round(tx, 1), round(ty, 1), round(tz, 1)), rotation_matrix=R, euler_angles=(round(rx, 1), round(ry, 1), round(rz, 1)), confidence=round(conf, 3), grasp_point=grasp_point, grasp_approach=approach_vec ) 

Why 6DoF Pose Estimation Is Critical for Collaborative Robots

Collaborative robots work in the same space as humans. An error in determining the part pose leads to collision or damage. For cobot applications per ISO 15066, grasp repeatability of ±1 mm and latency under 50 ms are required. Only 6DoF pose estimation provides the accuracy needed for safe approach and grasp.

Method comparison metrics:

FoundationPose outperforms GDR-Net by 1.2× in ADD when CAD is available, but without CAD GDR-Net wins due to not requiring a model. PVPN is more robust to occlusions but slower (15 FPS vs 30 FPS for FoundationPose).

How We Design a Computer Vision System for Robots

The process starts with an audit of your production: what operations are performed, what parts, what current issues. Then we select hardware (cameras, lighting, controllers) and develop the CV algorithm. Steps:

1. Data collection: capturing scenes at your facility, labeling poses (6DoF) using our tools.
2. Model selection: FoundationPose, GDR-Net, or custom Transformer depending on CAD availability and acceptable latency.
3. Training and validation: on synthetic and real data. We aim for ADD-0.1d > 85%.
4. Integration: into a ROS2 node for the manipulator or OPC-UA for PLC. We ensure real-time performance.
5. Testing: on the production line for two weeks. We record KPIs (grasp cycle time, success rate).

Vision for AMR Navigation

Mobile robots (AMR/AGV) use CV for obstacle detection, people detection, and map building. Typical architecture: YOLOv8 on RGB-D, depth segmentation, sector division for trajectory planning. Example code snippet:

class AMRNavigationVision: """ Computer vision for autonomous mobile robots (AMR). Tasks: obstacle detection, semantic mapping, human recognition for cobot safety (ISO/TS 15066 protected/restricted speed zones). """ def __init__(self, obstacle_model_path: str, device: str = 'cuda'): from ultralytics import YOLO self.obstacle_model = YOLO(obstacle_model_path) self.device = device # Semantic map: {cell_id: label} self.semantic_map: dict = {} def process_navigation_frame(self, rgb: np.ndarray, depth: np.ndarray) -> dict: """ Analyze a frame for AMR navigation. Returns: obstacles, nearest_human_dist_m, clear_path_sectors. """ results = self.obstacle_model(rgb, conf=0.4, verbose=False) obstacles = [] nearest_human_dist = float('inf') h, w = depth.shape[:2] sector_width = w // 5 # 5 sectors: LL/L/C/R/RR for box in results[0].boxes: x1, y1, x2, y2 = map(int, box.xyxy[0]) cls_name = results[0].names[int(box.cls.item())] cx = (x1 + x2) // 2 cy = (y1 + y2) // 2 # Median depth in bbox depth_crop = depth[y1:y2, x1:x2] valid_depths = depth_crop[depth_crop > 0] dist_m = float(np.median(valid_depths)) / 1000.0 if len(valid_depths) > 0 else 0 obstacles.append({ 'class': cls_name, 'bbox': [x1, y1, x2, y2], 'distance_m': round(dist_m, 2), 'sector': min(cx // sector_width, 4) }) if cls_name == 'person' and dist_m < nearest_human_dist: nearest_human_dist = dist_m # Determine clear sectors blocked_sectors = {o['sector'] for o in obstacles if o['distance_m'] < 1.5} clear_sectors = [s for s in range(5) if s not in blocked_sectors] # ISO/TS 15066: if person < 0.5m → STOP; 0.5–1.5m → reduced speed safety_mode = ('STOP' if nearest_human_dist < 0.5 else 'REDUCED_SPEED' if nearest_human_dist < 1.5 else 'NORMAL') return { 'obstacles': obstacles, 'nearest_human_m': round(nearest_human_dist, 2), 'clear_sectors': clear_sectors, 'safety_mode': safety_mode } 

What Is Included in the CV Project Work

We provide the full cycle: requirements analysis, equipment selection, camera-robot calibration, model training on your data, integration with the controller (ROS2/OPC-UA), testing under production conditions. Deliverables:

• Pose estimation model (ONNX/TensorRT)
• Integration module for PLC
• Safety operation documentation
• Operator training
• 6-month warranty support

Timeline and Cost

Timeline — from 8 to 20 weeks depending on complexity. Cost is calculated individually after an audit of your production. We guarantee stage transparency and fix KPIs in the contract.

Our Experience and Guarantees

Years of experience in industrial CV, dozens of projects from bin picking to inspection. Certified engineers in PyTorch, ROS2, OpenCV. We use official libraries: OpenCV and ISO 15066. We guarantee model accuracy (ADD and recall are specified in the contract).

Get a consultation for your project — contact us to discuss the task and create a prototype. Request a preliminary audit of your production — it is free.