Navisioner is a robot
guides visually impaired individuals from the starting point to their destination,
avoids and responds to surrounding hazards, and safely assists them along the way.

---

Table of Contents

• Project Overview
• Structure
• Reference
• Technologies
      (A) Voice Commands and Route Voice Guidance
      (B) Robot Localization and Pose Estimation
      (C) Sidewalk Detection
      (D) Traffic Light Detection
      (E) Obstacle Detection

Project Overview

Structure

Technologies

(A) Voice Commands and Route Voice Guidance

The user sets the departure and destination points by voice using the Whisper STT model on a Flask server and receives voice guidance through the Web Speech API. The configured route is converted into latitude and longitude values and searched using the Tmap server. The four pieces of information obtained are converted into ROS message format using roslibjs and sent to the robot via the Rosbridge server. The robot then begins navigation based on the provided route.

(B) Robot Localization and Pose Estimation

The robot estimates its current position based on the latitude and longitude values received via GPS during navigation and estimates its orientation using the IMU.

It compares the waypoints of the pedestrian route received from the web server with the robot's current GPS position to determine whether a waypoint has been reached. Once a waypoint is reached, the required heading to the next waypoint is calculated, and the robot rotates accordingly. During the rotation, the IMU's z-axis data is used to check whether the target heading has been achieved. After reaching the target heading, the robot starts moving toward the next waypoint.

During this process, functions such as pedestrian path detection, obstacle detection, and voice guidance for navigation are performed to ensure safe travel to the next waypoint.

When the robot reaches the final waypoint, it is considered the destination, and the robot stops.

To consider the perspective of visually impaired individuals, I simulated the service with my eyes closed and found that the directional changes were not as noticeable as expected. To provide directional guidance while ensuring safety, the robot is programmed to stop and then rotate after providing a voice announcement for maneuvers related to "turn type" rotations. For tracking other waypoints, the Pure Pursuit algorithm is used.

(C) Sidewalk Detection

We performed transfer learning on Ultralytics’s YOLOv8 Segmentation model using 45,000 images from AI Hub, an open dataset platform. The dataset includes annotations in box and polygon formats for 29 types of objects obstructing Indian pedestrian pathways, as well as polygon annotations for sidewalk surface conditions.

Due to the performance limitations of edge devices, there was an issue with slow real-time processing speed.
To address this, we converted the PyTorch model to TensorRT and applied INT8 quantization to accelerate the inference speed. As a result, we were able to improve latency by 65% while reducing mAP loss.

The detection process is as follows:
The RGB video is captured from the camera, and the pedestrian road area is segmented.
An algorithm compares the detected sidewalk's two edges with experimentally obtained threshold points to localize the robot, ensuring that it moves only within the sidewalk.

This diagram simplifies the algorithm.
In the first image, when comparing the threshold with the sidewalk edges, the robot is at the center of the sidewalk, so it moves straight ahead.
Depending on the condition, if the robot is shifted to the left, it moves right; if shifted to the right, it moves left.
In this way, the robot's angular velocity and speed are adjusted and controlled via ROS according to each situation.

(D) Traffic Light Detection

Using the crosswalk location information received from the Tmap API, the robot verifies that it has reached the crosswalk. If it is determined that the crosswalk has been reached, the YOLOv8 model will crop only the traffic light in the image, and use this cropped image to determine if it is a green light and to cross.

The cropped image is determined whether it is a green light through three steps.

1. Check the color with the HSV filter.
2. A moving average filter is applied to prevent noise-induced errors, and it is considered green only when the average value is above 0.5.
3. Using the ‘red flag’, it does not cross when the signal's remaining time is short, but moves only when it changes from red to green. This allows the robot to safely cross the crosswalk.

(E) Obstacle Detection

Reference

Reference
yolo_ros
scout_ros2
ugv_sdk
ROS2 Foxy
zed-ros2-wrapper