Skip to content

Commit d37fcca

Browse files
committed
updated website
1 parent cf7dfec commit d37fcca

16 files changed

Lines changed: 444 additions & 74 deletions

README copy.md

Lines changed: 328 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,328 @@
1+
<p align="center">
2+
<img src="media/bannernobg.png" alt="VisionPilot Banner" height="200" />
3+
</p>
4+
5+
# VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception (BeamNG.tech)
6+
7+
<p align="center" style="margin-bottom:0;">
8+
<a href="https://star-history.com/#visionpilot-project/VisionPilot&Date">
9+
<img src="https://api.star-history.com/svg?repos=visionpilot-project/VisionPilot&type=Date" alt="Star History Chart" height="300" />
10+
</a>
11+
</p>
12+
13+
## Table of Contents
14+
15+
- [VisionPilot: Autonomous Driving Simulation, Computer Vision \& Real-Time Perception (BeamNG.tech)](#visionpilot-autonomous-driving-simulation-computer-vision--real-time-perception-beamngtech)
16+
- [Table of Contents](#table-of-contents)
17+
- [Overview](#overview)
18+
- [Demos](#demos)
19+
- [Emergency Braking (AEB) Demo](#emergency-braking-aeb-demo)
20+
- [Sign Detection \& Detection and classification](#sign-detection--detection-and-classification)
21+
- [Traffic Light Detection \& Classification Demo](#traffic-light-detection--classification-demo)
22+
- [Latest Lane Detection Demo (v2)](#latest-lane-detection-demo-v2)
23+
- [Previous Lane Detection Demo (v1)](#previous-lane-detection-demo-v1)
24+
- [Foxglove Visualization Demo](#foxglove-visualization-demo)
25+
- [Segmentation Demo](#segmentation-demo)
26+
- [Sensor Suite](#sensor-suite)
27+
- [Roadmap](#roadmap)
28+
- [Perception](#perception)
29+
- [Sensor Fusion \& Calibration](#sensor-fusion--calibration)
30+
- [Control \& Planning](#control--planning)
31+
- [Simulation \& Scenarios](#simulation--scenarios)
32+
- [Visualization \& Logging](#visualization--logging)
33+
- [Deployment \& Infrastructure](#deployment--infrastructure)
34+
- [README To-Dos](#readme-to-dos)
35+
- [Other](#other)
36+
- [Note on Installation](#note-on-installation)
37+
- [Known Limitations](#known-limitations)
38+
- [Simulator-Specific Limitations](#simulator-specific-limitations)
39+
- [Credits](#credits)
40+
- [Citation](#citation)
41+
- [BeamNG.tech Citation](#beamngtech-citation)
42+
- [License](#license)
43+
44+
## Overview
45+
46+
A modular Python project for autonomous driving research and prototyping, fully integrated with the BeamNG.tech simulator and Foxglove visualization. This system combines traditional computer vision and state-of-the-art deep learning (CNN, U-Net, YOLO, SCNN) with real-time sensor fusion and autonomous vehicle control to tackle:
47+
48+
- Lane detection (Traditional CV & SCNN)
49+
- Traffic sign classification & detection (CNN, YOLO)
50+
- Traffic light detection & classification (YOLO)
51+
- Vehicle & pedestrian detection and recognition (YOLO)
52+
- Multi-sensor fusion (Camera, LiDAR, Radar, GPS, IMU)
53+
- Multi-model inference, real-time simulation, autonomous driving with PID control (BeamNG.tech)
54+
- Containerized multi-model architecture (Docker-based), orchestrated via a central inference aggregator service
55+
- Cruise control
56+
- Automatic Emergency Breaking AEB
57+
- Real-time visualization and monitoring (Foxglove WebSocket)
58+
- Modular configuration system (YAML-based)
59+
- Drive logging and telemetry
60+
61+
## Demos
62+
63+
### Emergency Braking (AEB) Demo
64+
65+
Watch the Emergency Braking System (AEB) in action with real-time radar filtering and collision avoidance:
66+
67+
<img src="media/demo_gifs/aeb_gif.gif" alt="AEB Demo" width="600" height="337" />
68+
69+
**Extended Demo:** [Watch the full video here](https://www.youtube.com/watch?v=Z8Y2-MpmrRg)
70+
71+
---
72+
73+
### Sign Detection & Detection and classification
74+
75+
This demo shows real-time traffic sign detection and classification:
76+
77+
<img src="media/demo_gifs/sign_demo.gif" alt="Sign Detection Demo & Vehicle Pedestrian" width="600" height="337" />
78+
79+
**Extended Demo:** [Watch the full video here](https://youtu.be/ujGkQJ2BqV0)
80+
81+
> VisionPilot does not yet support multi-camera. This is for demonstration purposes only.
82+
83+
---
84+
85+
### Traffic Light Detection & Classification Demo
86+
87+
This demo shows real-time traffic light detection and classification:
88+
89+
<img src="media/demo_gifs/traffic_light_demo.gif" alt="Traffic Light Detection & Classification Demo" width="600" height="337" />
90+
91+
> No extended Demo avaliable yet.
92+
93+
---
94+
95+
### Latest Lane Detection Demo (v2)
96+
97+
Watch the improved autonomous lane keeping demo (v2) in BeamNG.tech, featuring smoother fused CV+SCNN lane detection, stable PID steering, and robust cruise control:
98+
99+
<img src="media/demo_gifs/lane.gif" alt="Lane Detection Demo" width="600" height="337" />
100+
101+
**Extended Demo:** [Watch the full video here](https://www.youtube.com/watch?v=7eA_XfIkLWQ)
102+
103+
> Note: Very low-light (tunnel) scenarios are not yet supported.
104+
105+
### Previous Lane Detection Demo (v1)
106+
107+
The original demo is still available for reference:
108+
109+
[Lane Keeping & Multi-Model Detection Demo (v1)](https://youtu.be/f9mHigMKME8)
110+
111+
---
112+
113+
### Foxglove Visualization Demo
114+
115+
See real-time LiDAR point cloud streaming and autonomous vehicle telemetry in Foxglove Studio:
116+
117+
<img src="media/demo_gifs/foxglove.gif" alt="Foxglove Visualization Demo" width="600" height="337" />
118+
119+
**Extended Demo:** [Watch the full video here](https://www.youtube.com/watch?v=4HJDvL2Q6AY)
120+
121+
---
122+
123+
### Segmentation Demo
124+
125+
See real-time image segmentation using front and rear cameras:
126+
127+
<img src="media/demo_gifs/segmentation.gif" alt="Segmentation Demo" width="600" height="337" />
128+
129+
**Extended Demo:** [Watch the full video here](https://youtu.be/4PAqcUKqn6c?si=UHw-mw7iLZKGXvav)
130+
131+
> More demo videos and visualizations will be added as features are completed.
132+
133+
## Sensor Suite
134+
135+
The vehicle is equipped with a comprehensive multi-sensor suite for autonomous perception and control:
136+
137+
| Sensor | Specification | Purpose |
138+
| --------------------------- | ---------------------------------------------------- | --------------------------------------------------------------- |
139+
| **Front Camera** | 1920x1080 @ 50Hz, 70° FOV, Depth enabled | Lane detection, traffic signs, traffic lights, object detection |
140+
| **LiDAR (Top)** | 80 vertical lines, 360° horizontal, 120m range, 20Hz | Obstacle detection, 3D scene understanding |
141+
| **Front Radar** | 200m range, 128×64 bins, 50Hz | Collision avoidance, adaptive cruise control |
142+
| **Rear Left & Right Radar** | 30m range, 64×32 bins, 50Hz | Blindspot monitoring, rear object detection |
143+
| **Dual GPS** | Front & rear positioning @ 50Hz | Localization |
144+
| **IMU** | 100Hz update rate | Vehicle dynamics, pose estimation |
145+
146+
<table>
147+
<tr>
148+
<td align="center"><img src="media/beamng_images/sensors.png" alt="Sensor Array 1" width="280"/></td>
149+
<td align="center"><img src="media/beamng_images/radar_front.png" alt="Sensor Array 2" width="280"/></td>
150+
<td align="center"><img src="media/beamng_images/lidar.png" alt="Sensor Array 3" width="280"/></td>
151+
</tr>
152+
<tr>
153+
<td align="center"><em>Sensor Array</em></td>
154+
<td align="center"><em>Front Radar</em></td>
155+
<td align="center"><em>Lidar Visualization</em></td>
156+
</tr>
157+
</table>
158+
159+
> Configuration files are located in the `/config` directory:
160+
161+
## Roadmap
162+
163+
### Perception
164+
165+
- [x] Sign classification & Detection (CNN / YOLOv11m)
166+
- [x] Traffic light classification & Detection (CNN / YOLOv11m)
167+
- [x] Lane detection Fusion (SCNN / CV)
168+
- [x] Advanced lane detection using OpenCV (robust highway, lighting, outlier handling)
169+
- [x] Integrate Majority Voting system for CV
170+
- [x] ⭐ Semantic Segmentatation (Already built not implemented here yet)
171+
- [x] ⭐ Real-Time Object Detection (Cars, Trucks, Buses, Pedestrians, Cyclists) (Trained)
172+
- [ ] 🔥 Speed Estimation using detection from camera and lidar
173+
- [ ] Potentially use Multiple Object Tracking (MOT) for better speed estimation
174+
- [ ] Pedestrian intent prediction (crossing, standing, walking along road)
175+
- **Note:** Would Have to be tested in Carla as BeamNG.tech does not have pedestrians implemented
176+
- [ ] Vehicle State Classification (Break Lights, Turn Signals, Reverse Lights)
177+
- [ ] 🔥 Handle dashed lines better in lane detection
178+
- [ ] 🔥 Lidar Object Detection 3D
179+
- [ ] Detect multiple lanes
180+
- [ ] 💤 Multi Camera Setup (Will implement after all other camera-based features are finished)
181+
- [ ] 💤 Overtaking, Merging (Will be part of Path Planning)
182+
183+
### Sensor Fusion & Calibration
184+
185+
- [ ] 🔥 Kalman Filtering
186+
- [x] Integrate Radar
187+
- [x] Integrate Lidar
188+
- [ ] Integrate GPS
189+
- [ ] Integrate IMU
190+
- [ ] Ultrasonic Sensor Integration
191+
- **Note:** Can easily be implemented with prebuilt Beamng ADAS module
192+
- [ ] Map Matching algorithm
193+
- [ ] 💤 SLAM (simultaneous localization and mapping)
194+
- [ ] Build HD Map from Scratch
195+
- [ ] Localize Vehicle on HD Map
196+
- [ ] Sensor Health Monitoring & Redundancy
197+
- [ ] Redundant Front Radar for AEB
198+
- [ ] Sensor status diagnostics and failover
199+
200+
### Control & Planning
201+
202+
- [x] Integrate vehicle control (Throttle, Steering, Braking Implemented) (PID needs further tuning)
203+
- [x] Integrate PIDF controller
204+
- [x] ⭐ Adaptive Cruise Control (Currently only basic Cruise Control implemented)
205+
- [x] ⭐ Automatic Emergency Braking AEB
206+
- [ ] Support using Camera and Lidar detections
207+
- [ ] Trajectory Predcition for surrounding vehicles
208+
- [ ] Blindspot Monitoring (Using left/right rear short range radars)
209+
- [ ] Traffic Rule Enforcement (Stop at red lights, stop signs, yield signs)
210+
- [ ] Dynamic Target Speed based on Speed Limit Signs
211+
- [ ] Global Path planning
212+
- [ ] Local Path planning
213+
- [ ] Lane Change Logic
214+
- [ ] Parking Logic (Path finding / Parallel or Perpendicular)
215+
- [ ] 💤 End-to-end driving policy learning (RL, imitation learning)
216+
- [ ] 💤💤 Advanced traffic participant prediction (trajectory, intent)
217+
218+
### Simulation & Scenarios
219+
220+
- [x] Integrate and test in BeamNG.tech simulation (replacing CARLA)
221+
- [x] Modularize and clean up BeamNG.tech pipeline
222+
- [x] Tweak lane detection parameters and thresholds
223+
- [ ] Fog Weather conditions (Rain or snow not supported in BeamNG.tech)
224+
- [ ] Traffic scenarios: driving in heavy, moderate, and light traffic
225+
- [ ] Test all Systems in different lighting conditions (Day, Night, Dawn/Dusk, Tunnel)
226+
- [ ] 💤💤 Test using actual RC car
227+
228+
### Visualization & Logging
229+
230+
- [x] ⭐ Full Foxglove visualization integration (Overhaul needed)
231+
- [x] Modular YAML configuration system
232+
- [x] Real-time drive logging and telemetry
233+
- [ ] Real time Annotations Overlay in Foxglove
234+
- [ ] Show predicted trajectories in Foxglove
235+
- [ ] Show Global and local path plans in Foxglove
236+
- [ ] Live Map Visualization
237+
238+
### Deployment & Infrastructure
239+
240+
- [ ] Containerize Models for easy deployment and scalability (Also eliminates dependency issues)
241+
- [ ] Message Broker (redis/rabbitmq)
242+
- [ ] Create docker compose
243+
- [ ] Aggregator service
244+
- [ ] Refactor beamng.py
245+
246+
### README To-Dos
247+
248+
- [x] Add demo images and videos to README
249+
- [ ] Add performance benchmarks section
250+
- [x] Add Table of Contents for easier navigation
251+
252+
### Other
253+
254+
- [x] Vibe-Code a website for the project
255+
- [x] Redo project structure for better modularity
256+
257+
> Driver Monitoring System would've been pretty cool but human drivers are not implemented in BeamNG.tech
258+
259+
## Legend
260+
261+
> 🔥 = High Priority
262+
263+
> ⭐ = Complete but still being improved/tuned/changed (not final version)
264+
265+
> 💤 = Minimal Priority, can be addressed later
266+
267+
> 💤💤 = Very Low Priority, may not be implement
268+
269+
## Note on Installation
270+
271+
> **Status:** This project is currently in **active development**. A stable, production-ready release with pre-trained models and complete documentation will be available eventually.
272+
273+
## Known Limitations
274+
275+
- **Tunnel/Low-Light Scenarios:** Camera depth perception fails below certain lighting thresholds
276+
- **Multi-Camera Support:** Single front-facing camera only (future roadmap)
277+
- **Dashed Lane Detection:** Requires improvement for better accuracy
278+
- **PID Controller Tuning:** May oscillate on aggressive maneuvers
279+
- **Real-World Testing:** Only validated in simulation (BeamNG.tech), for now...
280+
281+
### Simulator-Specific Limitations
282+
283+
- Rain/snow physics not supported in BeamNG.tech
284+
- No native ROS2 support (custom bridge required)
285+
- Pedestrians
286+
- Human Drivers
287+
288+
## Credits
289+
290+
**Datasets:**
291+
292+
- CU Lane, LISA, GTSRB, Mapillary, BDD100K
293+
294+
**Simulation & Tools:**
295+
296+
- BeamNG.tech by [BeamNG GmbH](https://www.beamng.tech/)
297+
- Foxglove Studio for visualization
298+
299+
**Special Thanks:**
300+
301+
- Kaggle for free GPU resources (model training)
302+
- Mr. Pratt (teacher/supervisor) for guidance
303+
304+
## Citation
305+
306+
If you use VisionPilot in your research, please cite:
307+
308+
```bibtex
309+
@software{visionpilot2025,
310+
title={VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception},
311+
author={Julian Stamm},
312+
year={2025},
313+
url={https://github.com/visionpilot-project/VisionPilot}
314+
}
315+
```
316+
317+
### BeamNG.tech Citation
318+
319+
> **Title:** BeamNG.tech
320+
> **Author:** BeamNG GmbH
321+
> **Address:** Bremen, Germany
322+
> **Year:** 2025
323+
> **Version:** 0.35.0.0
324+
> **URL:** https://www.beamng.tech/
325+
326+
## License
327+
328+
This project is licensed under the MIT License - see [LICENSE](LICENSE) file for details.

components/Navbar.tsx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ const Navbar: React.FC = () => {
1818
}, []);
1919

2020
const navLinks = [
21+
{ name: 'Home', path: '/' },
2122
{ name: 'Technical', path: '/technical' },
2223
{ name: 'Demos', path: '/demos' },
2324
{ name: 'Models', path: '/models' },

index.html

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,13 @@
44
<meta charset="UTF-8" />
55
<link rel="icon" type="image/svg+xml" href="/vite.svg" />
66
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
7-
<title>VisionPilot | Autonomous Driving Simulation</title>
7+
<meta name="description" content="VisionPilot: A modular Python project for autonomous driving research, integrating BeamNG.tech soft-body physics with state-of-the-art computer vision (YOLO, SCNN), sensor fusion, and real-time control." />
8+
<meta name="keywords" content="Autonomous Driving, BeamNG.tech, Computer Vision, YOLO, SCNN, Sensor Fusion, Robotics, Simulation, Python, Deep Learning, VisionPilot" />
9+
<meta name="author" content="Julian Stamm" />
10+
<meta property="og:title" content="VisionPilot | Autonomous Driving Simulation & Perception" />
11+
<meta property="og:description" content="VisionPilot: A modular Python project for autonomous driving research, integrating BeamNG.tech soft-body physics with state-of-the-art computer vision (YOLO, SCNN), sensor fusion, and real-time control." />
12+
<meta property="og:type" content="website" />
13+
<title>VisionPilot | Autonomous Driving Simulation & Perception</title>
814
<!-- Google tag (gtag.js) -->
915
<script async src="https://www.googletagmanager.com/gtag/js?id=G-6H1HVLTXWB"></script>
1016
<script>

0 commit comments

Comments
 (0)