Affordable Robotic Assistant with Real-Time Object Recognition Using CNN for Visually Impaired Users

Illustrative Image: Affordable Robotic Assistant with Real-Time Object Recognition Using CNN for Visually Impaired Users
Image Source & Credit: Frontiers
Ownership and Usage Policy

A study by Oluyele et al. (2024) titled “Robotic assistant for object recognition using convolutional neural network” published in ABUAD Journal of Engineering Research and Development reveals that the convolutional neural network (CNN) achieved over 90% accuracy on the test set for the selected object classes, demonstrating strong performance in real-world conditions.

“

Low-cost robotic assistant using CNN achieved over 90% accuracy in real-time object recognition tasks.– Oluyele et al. 2024

This paper presents the design, development, and implementation of a real-time object recognition system for a robotic assistant, aimed at enabling autonomous visual perception and decision-making. Leveraging the pattern-recognition capabilities of convolutional neural networks (CNNs), the study equips a mobile robotic platform with the ability to identify and distinguish objects in complex, dynamic environments. A custom CNN architecture was developed and optimized for both speed and accuracy on embedded hardware, achieving over 90% classification accuracy across selected object categories. Integrated directly into the robot’s control loop, the trained model enabled real-time inference at 10–15 frames per second, allowing the system to smoothly navigate and interact with its surroundings. The robot demonstrated robust performance in cluttered scenes, showing resilience to occlusion and varying lighting conditions. This work lays foundational infrastructure for vision-based service robots in domestic settings, warehouses, and assistive technologies for visually impaired users. Future directions include expanding the object vocabulary, incorporating multi-object tracking, and applying active learning techniques for continuous model refinement in real-world deployments. The study also highlights potential avenues for comparative analysis with transfer-learning methods, such as fine-tuning pre-trained models, to explore trade-offs between accuracy and computational efficiency. Additionally, ensuring model security in edge-computing contexts remains a crucial consideration, particularly when operating in public or sensitive environments.

The study explores the following methodology:

Robotic Object Recognition System Overview: This system features a mobile robotic chassis equipped with a camera module for continuous scene capture and real-time object recognition. An onboard processing unit—such as a Raspberry Pi or NVIDIA Jetson—executes deep learning inference to identify target objects in dynamic environments.

Dataset Preparation and Augmentation: To train the recognition model, a dataset of labeled images is compiled, encompassing target object classes captured under diverse lighting conditions and angles. Data augmentation techniques such as rotation, scaling, and noise injection are applied to enhance the model’s generalization capability and robustness.

CNN Architecture and Training: The core of the system is a custom convolutional neural network (CNN) composed of alternating convolutional and pooling layers, followed by fully connected layers. ReLU activations are used to introduce non-linearity, while dropout layers help prevent overfitting. The dataset is partitioned into training, validation, and test sets. The model is trained using stochastic gradient descent (SGD) with learning-rate decay, and its performance is continuously evaluated using metrics such as accuracy, precision, and recall on the validation set.

System Integration and Deployment: Upon achieving satisfactory performance, the trained CNN model is exported to a lightweight format, such as TensorFlow Lite, for deployment on the embedded hardware. The robot’s control software integrates an inference loop that processes live camera input and triggers corresponding actions based on detected objects, enabling autonomous interaction with the environment.

What the Authors Found

The authors found that the convolutional neural network (CNN) achieved over 90% accuracy on the test set for the selected object classes, demonstrating strong performance in real-world conditions. Running at 10–15 frames per second on an embedded platform, the system enabled smooth real-time navigation and interaction. It also proved robust in cluttered environments, reliably distinguishing objects despite occlusion and varying lighting conditions.

Why is this important

Solving Real-World Accessibility Challenges: The robotic assistant addresses a critical need for visually impaired individuals who often struggle to locate everyday objects like phones, chairs, or mice—something traditional aids like canes or guide dogs can’t do. By providing active object recognition with audio feedback, it fills a crucial gap in assistive technology.

Autonomous Mobility and Intelligence: Unlike static or wearable solutions, this mobile robot can autonomously navigate, recognize objects, and inform users in real time. This empowers visually impaired users to explore indoor spaces more confidently and independently.

Affordable Assistive Technology: With a cost of just $172.59, this system delivers smart vision assistance at a fraction of the price of high-end tools like OrCam ($4,250), making it a viable option for low-income communities and individuals.

Flexible and Customizable Design: Built on a Raspberry Pi and using open-source frameworks like YOLOv3 and TensorFlow, the system is highly adaptable. It can be customized, expanded, or repurposed for various applications—including schools, rehabilitation centers, and homes.

Context-Aware, Locally Trained Model: The model was trained on images collected in Nigeria, ensuring better performance and cultural relevance compared to generic datasets. This localized approach enhances object recognition in real-world environments.

What the Authors Recommended

• Currently recognizing only mobile phones, computer mice, and chairs, the system would benefit from supporting more everyday items like keys, wallets, books, and utensils. This expansion would enhance its usefulness across varied indoor environments and daily routines.
• With just 2,895 locally sourced images, the dataset limits the model’s generalizability. Incorporating a larger, more diverse dataset—including images from different locations, lighting conditions, and object orientations—would significantly improve recognition accuracy in unpredictable settings.
• Equipping the robot with the ability to learn from new data and user feedback over time would make it more intelligent and responsive. This adaptive feature would allow for improved performance in recognizing unfamiliar or misidentified objects.
• While the robot currently operates in confined indoor spaces, extending its mobility to outdoor environments using GPS and obstacle-aware routing would increase its accessibility and real-world usability.
• Leveraging lightweight pre-trained models like MobileNet or EfficientDet via transfer learning could improve detection accuracy while maintaining speed on low-power devices. Additionally, enhancing human-robot interaction through voice control or a refined user interface would offer a more seamless and intuitive user experience.

In conclusion, the study by Oluyele et al. (2024) marks a significant advancement in affordable, AI-powered assistive technology by successfully integrating a high-accuracy CNN-based object recognition system into a mobile robotic assistant. With its strong real-time performance, adaptability, and low-cost design, this innovation holds immense potential to improve the independence and quality of life for visually impaired individuals. By expanding object categories, enhancing learning capabilities, and refining system integration, future developments can further position this solution as a transformative tool for accessible and inclusive smart environments.