|
| 1 | +<p align="center"> |
| 2 | + <img src="media/bannernobg.png" alt="VisionPilot Banner" height="200" /> |
| 3 | +</p> |
| 4 | + |
| 5 | +# VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception (BeamNG.tech) |
| 6 | + |
| 7 | +<p align="center" style="margin-bottom:0;"> |
| 8 | + <a href="https://star-history.com/#visionpilot-project/VisionPilot&Date"> |
| 9 | + <img src="https://api.star-history.com/svg?repos=visionpilot-project/VisionPilot&type=Date" alt="Star History Chart" height="300" /> |
| 10 | + </a> |
| 11 | +</p> |
| 12 | + |
| 13 | +## Table of Contents |
| 14 | + |
| 15 | +- [VisionPilot: Autonomous Driving Simulation, Computer Vision \& Real-Time Perception (BeamNG.tech)](#visionpilot-autonomous-driving-simulation-computer-vision--real-time-perception-beamngtech) |
| 16 | + - [Table of Contents](#table-of-contents) |
| 17 | + - [Overview](#overview) |
| 18 | + - [Demos](#demos) |
| 19 | + - [Emergency Braking (AEB) Demo](#emergency-braking-aeb-demo) |
| 20 | + - [Sign Detection \& Detection and classification](#sign-detection--detection-and-classification) |
| 21 | + - [Traffic Light Detection \& Classification Demo](#traffic-light-detection--classification-demo) |
| 22 | + - [Latest Lane Detection Demo (v2)](#latest-lane-detection-demo-v2) |
| 23 | + - [Previous Lane Detection Demo (v1)](#previous-lane-detection-demo-v1) |
| 24 | + - [Foxglove Visualization Demo](#foxglove-visualization-demo) |
| 25 | + - [Segmentation Demo](#segmentation-demo) |
| 26 | + - [Sensor Suite](#sensor-suite) |
| 27 | + - [Roadmap](#roadmap) |
| 28 | + - [Perception](#perception) |
| 29 | + - [Sensor Fusion \& Calibration](#sensor-fusion--calibration) |
| 30 | + - [Control \& Planning](#control--planning) |
| 31 | + - [Simulation \& Scenarios](#simulation--scenarios) |
| 32 | + - [Visualization \& Logging](#visualization--logging) |
| 33 | + - [Deployment \& Infrastructure](#deployment--infrastructure) |
| 34 | + - [README To-Dos](#readme-to-dos) |
| 35 | + - [Other](#other) |
| 36 | + - [Note on Installation](#note-on-installation) |
| 37 | + - [Known Limitations](#known-limitations) |
| 38 | + - [Simulator-Specific Limitations](#simulator-specific-limitations) |
| 39 | + - [Credits](#credits) |
| 40 | + - [Citation](#citation) |
| 41 | + - [BeamNG.tech Citation](#beamngtech-citation) |
| 42 | + - [License](#license) |
| 43 | + |
| 44 | +## Overview |
| 45 | + |
| 46 | +A modular Python project for autonomous driving research and prototyping, fully integrated with the BeamNG.tech simulator and Foxglove visualization. This system combines traditional computer vision and state-of-the-art deep learning (CNN, U-Net, YOLO, SCNN) with real-time sensor fusion and autonomous vehicle control to tackle: |
| 47 | + |
| 48 | +- Lane detection (Traditional CV & SCNN) |
| 49 | +- Traffic sign classification & detection (CNN, YOLO) |
| 50 | +- Traffic light detection & classification (YOLO) |
| 51 | +- Vehicle & pedestrian detection and recognition (YOLO) |
| 52 | +- Multi-sensor fusion (Camera, LiDAR, Radar, GPS, IMU) |
| 53 | +- Multi-model inference, real-time simulation, autonomous driving with PID control (BeamNG.tech) |
| 54 | +- Containerized multi-model architecture (Docker-based), orchestrated via a central inference aggregator service |
| 55 | +- Cruise control |
| 56 | +- Automatic Emergency Breaking AEB |
| 57 | +- Real-time visualization and monitoring (Foxglove WebSocket) |
| 58 | +- Modular configuration system (YAML-based) |
| 59 | +- Drive logging and telemetry |
| 60 | + |
| 61 | +## Demos |
| 62 | + |
| 63 | +### Emergency Braking (AEB) Demo |
| 64 | + |
| 65 | +Watch the Emergency Braking System (AEB) in action with real-time radar filtering and collision avoidance: |
| 66 | + |
| 67 | +<img src="media/demo_gifs/aeb_gif.gif" alt="AEB Demo" width="600" height="337" /> |
| 68 | + |
| 69 | +**Extended Demo:** [Watch the full video here](https://www.youtube.com/watch?v=Z8Y2-MpmrRg) |
| 70 | + |
| 71 | +--- |
| 72 | + |
| 73 | +### Sign Detection & Detection and classification |
| 74 | + |
| 75 | +This demo shows real-time traffic sign detection and classification: |
| 76 | + |
| 77 | +<img src="media/demo_gifs/sign_demo.gif" alt="Sign Detection Demo & Vehicle Pedestrian" width="600" height="337" /> |
| 78 | + |
| 79 | +**Extended Demo:** [Watch the full video here](https://youtu.be/ujGkQJ2BqV0) |
| 80 | + |
| 81 | +> VisionPilot does not yet support multi-camera. This is for demonstration purposes only. |
| 82 | +
|
| 83 | +--- |
| 84 | + |
| 85 | +### Traffic Light Detection & Classification Demo |
| 86 | + |
| 87 | +This demo shows real-time traffic light detection and classification: |
| 88 | + |
| 89 | +<img src="media/demo_gifs/traffic_light_demo.gif" alt="Traffic Light Detection & Classification Demo" width="600" height="337" /> |
| 90 | + |
| 91 | +> No extended Demo avaliable yet. |
| 92 | +
|
| 93 | +--- |
| 94 | + |
| 95 | +### Latest Lane Detection Demo (v2) |
| 96 | + |
| 97 | +Watch the improved autonomous lane keeping demo (v2) in BeamNG.tech, featuring smoother fused CV+SCNN lane detection, stable PID steering, and robust cruise control: |
| 98 | + |
| 99 | +<img src="media/demo_gifs/lane.gif" alt="Lane Detection Demo" width="600" height="337" /> |
| 100 | + |
| 101 | +**Extended Demo:** [Watch the full video here](https://www.youtube.com/watch?v=7eA_XfIkLWQ) |
| 102 | + |
| 103 | +> Note: Very low-light (tunnel) scenarios are not yet supported. |
| 104 | +
|
| 105 | +### Previous Lane Detection Demo (v1) |
| 106 | + |
| 107 | +The original demo is still available for reference: |
| 108 | + |
| 109 | +[Lane Keeping & Multi-Model Detection Demo (v1)](https://youtu.be/f9mHigMKME8) |
| 110 | + |
| 111 | +--- |
| 112 | + |
| 113 | +### Foxglove Visualization Demo |
| 114 | + |
| 115 | +See real-time LiDAR point cloud streaming and autonomous vehicle telemetry in Foxglove Studio: |
| 116 | + |
| 117 | +<img src="media/demo_gifs/foxglove.gif" alt="Foxglove Visualization Demo" width="600" height="337" /> |
| 118 | + |
| 119 | +**Extended Demo:** [Watch the full video here](https://www.youtube.com/watch?v=4HJDvL2Q6AY) |
| 120 | + |
| 121 | +--- |
| 122 | + |
| 123 | +### Segmentation Demo |
| 124 | + |
| 125 | +See real-time image segmentation using front and rear cameras: |
| 126 | + |
| 127 | +<img src="media/demo_gifs/segmentation.gif" alt="Segmentation Demo" width="600" height="337" /> |
| 128 | + |
| 129 | +**Extended Demo:** [Watch the full video here](https://youtu.be/4PAqcUKqn6c?si=UHw-mw7iLZKGXvav) |
| 130 | + |
| 131 | +> More demo videos and visualizations will be added as features are completed. |
| 132 | +
|
| 133 | +## Sensor Suite |
| 134 | + |
| 135 | +The vehicle is equipped with a comprehensive multi-sensor suite for autonomous perception and control: |
| 136 | + |
| 137 | +| Sensor | Specification | Purpose | |
| 138 | +| --------------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | |
| 139 | +| **Front Camera** | 1920x1080 @ 50Hz, 70° FOV, Depth enabled | Lane detection, traffic signs, traffic lights, object detection | |
| 140 | +| **LiDAR (Top)** | 80 vertical lines, 360° horizontal, 120m range, 20Hz | Obstacle detection, 3D scene understanding | |
| 141 | +| **Front Radar** | 200m range, 128×64 bins, 50Hz | Collision avoidance, adaptive cruise control | |
| 142 | +| **Rear Left & Right Radar** | 30m range, 64×32 bins, 50Hz | Blindspot monitoring, rear object detection | |
| 143 | +| **Dual GPS** | Front & rear positioning @ 50Hz | Localization | |
| 144 | +| **IMU** | 100Hz update rate | Vehicle dynamics, pose estimation | |
| 145 | + |
| 146 | +<table> |
| 147 | + <tr> |
| 148 | + <td align="center"><img src="media/beamng_images/sensors.png" alt="Sensor Array 1" width="280"/></td> |
| 149 | + <td align="center"><img src="media/beamng_images/radar_front.png" alt="Sensor Array 2" width="280"/></td> |
| 150 | + <td align="center"><img src="media/beamng_images/lidar.png" alt="Sensor Array 3" width="280"/></td> |
| 151 | + </tr> |
| 152 | + <tr> |
| 153 | + <td align="center"><em>Sensor Array</em></td> |
| 154 | + <td align="center"><em>Front Radar</em></td> |
| 155 | + <td align="center"><em>Lidar Visualization</em></td> |
| 156 | + </tr> |
| 157 | +</table> |
| 158 | + |
| 159 | +> Configuration files are located in the `/config` directory: |
| 160 | +
|
| 161 | +## Roadmap |
| 162 | + |
| 163 | +### Perception |
| 164 | + |
| 165 | +- [x] Sign classification & Detection (CNN / YOLOv11m) |
| 166 | +- [x] Traffic light classification & Detection (CNN / YOLOv11m) |
| 167 | +- [x] Lane detection Fusion (SCNN / CV) |
| 168 | +- [x] Advanced lane detection using OpenCV (robust highway, lighting, outlier handling) |
| 169 | +- [x] Integrate Majority Voting system for CV |
| 170 | +- [x] ⭐ Semantic Segmentatation (Already built not implemented here yet) |
| 171 | +- [x] ⭐ Real-Time Object Detection (Cars, Trucks, Buses, Pedestrians, Cyclists) (Trained) |
| 172 | +- [ ] 🔥 Speed Estimation using detection from camera and lidar |
| 173 | + - [ ] Potentially use Multiple Object Tracking (MOT) for better speed estimation |
| 174 | +- [ ] Pedestrian intent prediction (crossing, standing, walking along road) |
| 175 | +- **Note:** Would Have to be tested in Carla as BeamNG.tech does not have pedestrians implemented |
| 176 | +- [ ] Vehicle State Classification (Break Lights, Turn Signals, Reverse Lights) |
| 177 | +- [ ] 🔥 Handle dashed lines better in lane detection |
| 178 | +- [ ] 🔥 Lidar Object Detection 3D |
| 179 | +- [ ] Detect multiple lanes |
| 180 | +- [ ] 💤 Multi Camera Setup (Will implement after all other camera-based features are finished) |
| 181 | +- [ ] 💤 Overtaking, Merging (Will be part of Path Planning) |
| 182 | + |
| 183 | +### Sensor Fusion & Calibration |
| 184 | + |
| 185 | +- [ ] 🔥 Kalman Filtering |
| 186 | +- [x] Integrate Radar |
| 187 | +- [x] Integrate Lidar |
| 188 | +- [ ] Integrate GPS |
| 189 | +- [ ] Integrate IMU |
| 190 | +- [ ] Ultrasonic Sensor Integration |
| 191 | +- **Note:** Can easily be implemented with prebuilt Beamng ADAS module |
| 192 | +- [ ] Map Matching algorithm |
| 193 | +- [ ] 💤 SLAM (simultaneous localization and mapping) |
| 194 | + - [ ] Build HD Map from Scratch |
| 195 | + - [ ] Localize Vehicle on HD Map |
| 196 | +- [ ] Sensor Health Monitoring & Redundancy |
| 197 | + - [ ] Redundant Front Radar for AEB |
| 198 | + - [ ] Sensor status diagnostics and failover |
| 199 | + |
| 200 | +### Control & Planning |
| 201 | + |
| 202 | +- [x] Integrate vehicle control (Throttle, Steering, Braking Implemented) (PID needs further tuning) |
| 203 | +- [x] Integrate PIDF controller |
| 204 | +- [x] ⭐ Adaptive Cruise Control (Currently only basic Cruise Control implemented) |
| 205 | +- [x] ⭐ Automatic Emergency Braking AEB |
| 206 | + - [ ] Support using Camera and Lidar detections |
| 207 | +- [ ] Trajectory Predcition for surrounding vehicles |
| 208 | +- [ ] Blindspot Monitoring (Using left/right rear short range radars) |
| 209 | +- [ ] Traffic Rule Enforcement (Stop at red lights, stop signs, yield signs) |
| 210 | +- [ ] Dynamic Target Speed based on Speed Limit Signs |
| 211 | +- [ ] Global Path planning |
| 212 | +- [ ] Local Path planning |
| 213 | +- [ ] Lane Change Logic |
| 214 | +- [ ] Parking Logic (Path finding / Parallel or Perpendicular) |
| 215 | +- [ ] 💤 End-to-end driving policy learning (RL, imitation learning) |
| 216 | +- [ ] 💤💤 Advanced traffic participant prediction (trajectory, intent) |
| 217 | + |
| 218 | +### Simulation & Scenarios |
| 219 | + |
| 220 | +- [x] Integrate and test in BeamNG.tech simulation (replacing CARLA) |
| 221 | +- [x] Modularize and clean up BeamNG.tech pipeline |
| 222 | +- [x] Tweak lane detection parameters and thresholds |
| 223 | +- [ ] Fog Weather conditions (Rain or snow not supported in BeamNG.tech) |
| 224 | +- [ ] Traffic scenarios: driving in heavy, moderate, and light traffic |
| 225 | +- [ ] Test all Systems in different lighting conditions (Day, Night, Dawn/Dusk, Tunnel) |
| 226 | +- [ ] 💤💤 Test using actual RC car |
| 227 | + |
| 228 | +### Visualization & Logging |
| 229 | + |
| 230 | +- [x] ⭐ Full Foxglove visualization integration (Overhaul needed) |
| 231 | +- [x] Modular YAML configuration system |
| 232 | +- [x] Real-time drive logging and telemetry |
| 233 | +- [ ] Real time Annotations Overlay in Foxglove |
| 234 | +- [ ] Show predicted trajectories in Foxglove |
| 235 | +- [ ] Show Global and local path plans in Foxglove |
| 236 | +- [ ] Live Map Visualization |
| 237 | + |
| 238 | +### Deployment & Infrastructure |
| 239 | + |
| 240 | +- [ ] Containerize Models for easy deployment and scalability (Also eliminates dependency issues) |
| 241 | + - [ ] Message Broker (redis/rabbitmq) |
| 242 | + - [ ] Create docker compose |
| 243 | + - [ ] Aggregator service |
| 244 | + - [ ] Refactor beamng.py |
| 245 | + |
| 246 | +### README To-Dos |
| 247 | + |
| 248 | +- [x] Add demo images and videos to README |
| 249 | +- [ ] Add performance benchmarks section |
| 250 | +- [x] Add Table of Contents for easier navigation |
| 251 | + |
| 252 | +### Other |
| 253 | + |
| 254 | +- [x] Vibe-Code a website for the project |
| 255 | +- [x] Redo project structure for better modularity |
| 256 | + |
| 257 | +> Driver Monitoring System would've been pretty cool but human drivers are not implemented in BeamNG.tech |
| 258 | +
|
| 259 | +## Legend |
| 260 | + |
| 261 | +> 🔥 = High Priority |
| 262 | +
|
| 263 | +> ⭐ = Complete but still being improved/tuned/changed (not final version) |
| 264 | +
|
| 265 | +> 💤 = Minimal Priority, can be addressed later |
| 266 | +
|
| 267 | +> 💤💤 = Very Low Priority, may not be implement |
| 268 | +
|
| 269 | +## Note on Installation |
| 270 | + |
| 271 | +> **Status:** This project is currently in **active development**. A stable, production-ready release with pre-trained models and complete documentation will be available eventually. |
| 272 | +
|
| 273 | +## Known Limitations |
| 274 | + |
| 275 | +- **Tunnel/Low-Light Scenarios:** Camera depth perception fails below certain lighting thresholds |
| 276 | +- **Multi-Camera Support:** Single front-facing camera only (future roadmap) |
| 277 | +- **Dashed Lane Detection:** Requires improvement for better accuracy |
| 278 | +- **PID Controller Tuning:** May oscillate on aggressive maneuvers |
| 279 | +- **Real-World Testing:** Only validated in simulation (BeamNG.tech), for now... |
| 280 | + |
| 281 | +### Simulator-Specific Limitations |
| 282 | + |
| 283 | +- Rain/snow physics not supported in BeamNG.tech |
| 284 | +- No native ROS2 support (custom bridge required) |
| 285 | +- Pedestrians |
| 286 | +- Human Drivers |
| 287 | + |
| 288 | +## Credits |
| 289 | + |
| 290 | +**Datasets:** |
| 291 | + |
| 292 | +- CU Lane, LISA, GTSRB, Mapillary, BDD100K |
| 293 | + |
| 294 | +**Simulation & Tools:** |
| 295 | + |
| 296 | +- BeamNG.tech by [BeamNG GmbH](https://www.beamng.tech/) |
| 297 | +- Foxglove Studio for visualization |
| 298 | + |
| 299 | +**Special Thanks:** |
| 300 | + |
| 301 | +- Kaggle for free GPU resources (model training) |
| 302 | +- Mr. Pratt (teacher/supervisor) for guidance |
| 303 | + |
| 304 | +## Citation |
| 305 | + |
| 306 | +If you use VisionPilot in your research, please cite: |
| 307 | + |
| 308 | +```bibtex |
| 309 | +@software{visionpilot2025, |
| 310 | + title={VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception}, |
| 311 | + author={Julian Stamm}, |
| 312 | + year={2025}, |
| 313 | + url={https://github.com/visionpilot-project/VisionPilot} |
| 314 | +} |
| 315 | +``` |
| 316 | + |
| 317 | +### BeamNG.tech Citation |
| 318 | + |
| 319 | +> **Title:** BeamNG.tech |
| 320 | +> **Author:** BeamNG GmbH |
| 321 | +> **Address:** Bremen, Germany |
| 322 | +> **Year:** 2025 |
| 323 | +> **Version:** 0.35.0.0 |
| 324 | +> **URL:** https://www.beamng.tech/ |
| 325 | +
|
| 326 | +## License |
| 327 | + |
| 328 | +This project is licensed under the MIT License - see [LICENSE](LICENSE) file for details. |
0 commit comments