AI systems for autonomous vessels: from perception to certification

Imagine your vessel navigating in dense fog, the radar showing dozens of targets, but you don't know which are small buoys and which are large tankers. Every second of delay in threat recognition is a collision risk. Standard systems (radar + AIS) don't solve the problem: they don't classify objects, and a man overboard goes unnoticed until the last moment. We built an AI system that merges cameras, LiDAR, radar, and AIS into a unified perception pipeline, automatically applies COLREG rules, and detects a man overboard in seconds. Over 10 projects completed—from fishing trawlers to container ships. Thanks to automation, clients reduce crew costs up to 30%, and the system pays back in 2–3 years.

Why marine autonomy is harder than automotive?

At sea, there are no clear lanes, markings, or signs. Instead—waves, spray on lenses, fog, blinding sun on water, and night lights. A camera can go blind, LiDAR can drown in rain, radar can give false echoes from waves. Multisensor fusion with temporal and spatial correction is required. Additionally, you must interpret vessel lights (COLREGs 72) and predict maneuvers. A situation assessment error can cost millions of dollars and human lives.

How we build perception: sensor fusion

We use YOLOv8l, fine-tuned on marine data, in conjunction with ARPA radar and AIS. Ouster OS1-64 LiDAR covers the near zone (up to 120 m). Full perception cycle latency—150–250 ms. In fog, the system automatically increases radar and LiDAR weight, using the camera for classification. Below is a pipeline snippet in Python.

import numpy as np
import cv2
from ultralytics import YOLO

class MarinePerceptionSystem:
    def __init__(self, config: dict):
        # Vessel and obstacle detector: YOLOv8l fine-tuned on marine data
        self.vessel_detector = YOLO(config['vessel_model'])
        # Radar data (ARPA/AIS)
        self.radar_parser = RadarARPAParser(config['radar_port'])
        self.ais_receiver = AISReceiver(config['ais_port'])
        # LiDAR (Ouster OS1) for near zone < 100m
        self.lidar_processor = MarineLiDAR(config['lidar_config'])
        self.camera_matrix = np.array(config['cam_intrinsics'])

    def fuse_detections(self, frame: np.ndarray, radar_tracks: list, ais_contacts: list, lidar_points: np.ndarray) -> list[dict]:
        # Camera: detect vessels, buoys, flotsam, man overboard
        cam_dets = self.vessel_detector(frame, conf=0.4)
        fused_contacts = []
        for det in cam_dets[0].boxes:
            cls = self.vessel_detector.model.names[int(det.cls)]
            bbox = list(map(int, det.xyxy[0]))
            bearing = self._bearing_from_bbox(bbox, frame.shape)
            # Distance from LiDAR (if points in sector)
            distance = self._lidar_distance_in_sector(lidar_points, bearing)
            # Match with AIS (if MMSI in same direction)
            ais_match = self._match_ais(bearing, distance, ais_contacts)
            contact = {
                'class': cls,
                'bearing_deg': bearing,
                'distance_m': distance,
                'confidence': float(det.conf),
                'bbox': bbox,
                'ais_data': ais_match,
                'source': 'camera+lidar'
            }
            # Supplement with radar track
            radar_match = self._match_radar(bearing, distance, radar_tracks)
            if radar_match:
                contact['cog'] = radar_match.get('cog')  # course over ground
                contact['sog'] = radar_match.get('sog')  # speed over ground
                contact['tcpa'] = radar_match.get('tcpa')  # time to closest point of approach
                contact['cpa'] = radar_match.get('cpa')  # closest point of approach
            fused_contacts.append(contact)
        return fused_contacts

    def _bearing_from_bbox(self, bbox: list, frame_shape: tuple) -> float:
        cx = (bbox[0] + bbox[2]) / 2
        fov_h = 60  # horizontal field of view in degrees
        return (cx / frame_shape[1] - 0.5) * fov_h  # relative to heading

How do we ensure COLREG compliance?

The built-in COLREGPlanner analyzes the situation according to the International Regulations for Preventing Collisions at Sea (COLREGs). The system determines overtaking, head-on, or crossing and issues a maneuver command. tcpa and cpa are considered for urgency. In a real project for a ferry, the system reduced false alarms by 70% compared to manual planning.

class COLREGPlanner:
    """ COLREG rules: determine situation type and required maneuver. """

    def assess_situation(self, own_vessel: dict, target: dict) -> dict:
        bearing_to_target = target['bearing_deg']
        tcpa = target.get('tcpa', float('inf'))
        cpa = target.get('cpa', float('inf'))

        situation = 'safe'
        action = 'none'

        # COLREG Rule 13: overtaking
        if -22.5 <= bearing_to_target <= 22.5 and tcpa < 12 * 60:
            situation = 'overtaking'  # We are overtaking: give way
            action = 'alter_course_starboard'

        # COLREG Rule 14: head-on
        elif abs(bearing_to_target) < 5:
            situation = 'head_on'
            action = 'alter_course_starboard'

        # COLREG Rule 15: crossing
        elif 0 < bearing_to_target < 112.5:
            situation = 'crossing_give_way'
            action = 'alter_course_starboard_or_reduce_speed'
        elif -112.5 < bearing_to_target < 0:
            situation = 'crossing_stand_on'
            action = 'maintain_course_and_speed'

        return {
            'situation': situation,
            'action': action,
            'tcpa_minutes': tcpa / 60,
            'cpa_meters': cpa,
            'urgency': 'HIGH' if cpa < 500 and tcpa < 5 * 60 else 'MEDIUM' if cpa < 1000 else 'LOW'
        }

What about man overboard detection?

MOBDetector—critical function. A person in water is a small object (30×40 px at 50 m), can be hidden by waves. Our MOBDetector uses a fine-tuned YOLO for RGB and thermal camera for night, filtering false positives by horizon. In one trial, the system detected a MOB at 200 m with sea state 3, while the watchstander noticed it only after 40 seconds.

class MOBDetector:
    def __init__(self):
        self.detector = YOLO('yolov8m_mob.pt')  # fine-tuned on marine humans
        self.thermal_model = ThermalPersonDetector()  # for night

    def detect(self, frame: np.ndarray, thermal_frame: np.ndarray = None) -> list:
        # RGB detection
        rgb_dets = self.detector(frame, conf=0.35, classes=[0])  # person
        dets = list(rgb_dets[0].boxes)

        # Night or low visibility—thermal
        if thermal_frame is not None:
            thermal_dets = self.thermal_model.detect(thermal_frame)
            dets.extend(thermal_dets)

        # Filter: person in water has bbox near horizon
        horizon_y = frame.shape[0] * 0.4  # approximate
        mob_candidates = []
        for det in dets:
            bbox = list(map(int, det.xyxy[0]))
            if bbox[1] > horizon_y:  # below horizon = in water
                mob_candidates.append({
                    'bbox': bbox,
                    'confidence': float(det.conf)
                })
        return mob_candidates

Comparison: our system vs traditional approach

Typical mistakes in marine autonomy implementation

• Using automotive models without fine-tuning on marine data—they give 70% false positives on waves.
• Neglecting sensor calibration: 1-degree misalignment gives 30 m error at 2 km.
• Lack of fail-safe mode when GPS or AIS is lost—vessel should transition to safe drift.
• Ignoring classification society requirements (DNV, Lloyd's) during design phase.

Process of work

1. Analytics—vessel audit, requirements gathering, KPI agreement (e.g., MOB detection probability >95% at sea state 4).
2. Design—system architecture, stack selection (PyTorch, ONNX, Triton Inference Server).
3. Implementation—model training, pipeline coding, integration with NMEA 2000 and CAN bus.
4. Testing—unit, integration, pool and sea trials.
5. Deployment—installation, configuration, documentation handover, crew training.

What is included in the project work?

• Architectural documentation—detailed description of sensor configuration, computing modules, and algorithms.
• Trained models—for vessel, buoy, MOB detection, horizon segmentation, and light classification.
• Integration pipeline—linking with onboard systems (NMEA 2000, CAN bus, ARPA, AIS).
• Test bench—ability to simulate scenarios.
• Crew training—3–5 days onboard, including interface practice and emergency scenarios.
• Certification documentation—package for classification society.
• Support—12 months technical support with model update warranty.

System specifications

Timeline estimates

Contact us for a vessel audit—we will select the optimal sensor and computing configuration. Request a consultation to assess the applicability of AI autonomy for your vessel. Receive a detailed analysis and commercial proposal.