Jetson-Based Autonomous Driving System

Embedded autonomous driving system with perception, decision-making, and control on NVIDIA Jetson

Overview

This project implements an embedded autonomous driving system on the NVIDIA Jetson platform, covering the complete pipeline of:

Perception → Decision-Making → Control

The system integrates deep learning-based perception with rule-based planning and real-time control under embedded resource constraints.

Key capabilities include:

  • real-time object detection and lane perception
  • rule-based driving decision-making
  • closed-loop vehicle control
  • deployment on embedded GPU (Jetson)

System Architecture

The system follows a modular pipeline:

Perception → Environment Understanding → Behavior Planning → Vehicle Control

At each control cycle:

  1. Capture camera input
  2. Detect objects (vehicles, pedestrians, traffic lights)
  3. Extract lane geometry and estimate vehicle offset
  4. Generate behavior decisions (e.g., stop, turn, slow down)
  5. Output steering and speed commands

All modules run in real time on the Jetson onboard system.

Perception Module

Object Detection (YOLOv5)

  • Used YOLOv5 for detecting vehicles, pedestrians, and traffic lights
  • Trained on a custom dataset with augmentation (brightness, scaling)
  • Optimized with TensorRT for real-time inference on embedded GPU

Outputs:

  • bounding boxes + class labels
  • traffic light states (red / yellow / green)

Lane Detection (Classical CV)

  • Perspective transformation (bird’s-eye view)
  • Canny edge detection
  • Hough transform for lane extraction
  • Polynomial fitting for lane center estimation

The system computes lateral deviation for steering control.

Sensor Fusion (Camera + Radar)

To improve robustness, a camera–radar fusion module was implemented:

  • Camera → semantic understanding
  • Radar → reliable distance measurement

Fusion steps:

  • coordinate alignment between sensors
  • association between radar returns and detected objects
  • distance estimation refinement

This improves obstacle localization under challenging lighting conditions.

Behavior Planning (Core Contribution)

A rule-based decision system with FSM (Finite-State Machine) was designed to control vehicle behavior.

Decision Rules

  • Traffic light
    • red → stop
    • green → go
  • Obstacle
    • object in safety region → slow down / stop
  • Lane following
    • maintain alignment with lane center
  • Crosswalk
    • detected → reduce speed

FSM Design

The system defines states such as:

  • driving
  • stopping
  • obstacle avoidance

Transitions are triggered by perception outputs, ensuring:

  • stable behavior
  • interpretable logic
  • real-time responsiveness

Control Module

Steering Control

Proportional controller:

[ \delta = k_p \cdot e ]

  • (e): lateral deviation
  • (\delta): steering command

Speed Control

  • normal speed → lane following
  • reduced speed → crosswalk / obstacles
  • stop → red light

Embedded Optimization

To achieve real-time performance on Jetson:

  • TensorRT acceleration for YOLOv5
  • reduced input resolution
  • asynchronous perception and control pipeline
  • GPU utilization tuning

Technical Stack

Hardware

  • NVIDIA Jetson (embedded GPU)
  • monocular RGB camera
  • millimeter-wave radar

Software

  • Python, PyTorch, TensorRT

Algorithms

  • YOLOv5 object detection
  • classical lane detection (Canny + Hough)
  • rule-based planning + FSM
  • proportional control

Outcome

The system demonstrates a complete embedded autonomous driving pipeline capable of:

  • real-time perception on embedded hardware
  • stable lane following and obstacle avoidance
  • interpretable decision-making via FSM
  • closed-loop control under resource constraints

Key Takeaways

  • Built a full perception–decision–control system from scratch
  • Gained hands-on experience in embedded AI deployment
  • Developed understanding of decision-making systems in autonomous driving

This project marked my transition from general computer vision to autonomous driving systems, and motivated my later research in multi-agent perception and communication-efficient occupancy prediction.