Drishya is a smart glass developed to provide real-time audio feedback to visually impaired people. It aims to predict objects, find their location and provide feedback to the user through speakers present inside the arm of the smart glasses. The unique feature of the smart glass is that it would be connected to a protocol-like matter which will be able to integrate with all the smart home devices, and will work as an all-in-one solution to assist visually impaired people in their day-to-day life. This innovative solution addresses a real-world problem and aims to improve the accessibility for visually impaired individuals.
Accessibility is an afterthought, be it an improvement in society, culture, or technology. Though we have billions of dollars invested in the field of Metaverse, for people who can see; what about the people who cannot see? About 90% of the world's visually impaired people are living in developing countries. They cannot easily recognize obstacles, people, travel routes independently, or use stairs properly. They struggle with their day-to-day life and everyday activities. No safety features are present on the normal blind stick which is mostly used by visually impaired people as for their aid and support.
Our solution is to develop a smart glass which will not only be able to predict the objects but also find their location and provide feedback to the user. The visually impaired person will be provided with the feedback through speakers that would be present inside the arm of the smart glasses. In our prototype, the bounding boxes in the FOV of the camera will be recited to the ears of the user with a simple location-based quadrant mapping with a central field. If an object is in any of these locations, its position along with the object type will be played through the speakers. This enables the user to be conscious of their surroundings.
Drishya started out as a thought during one of the hackathon that I participated with my friends. Later we started to put effort into it and develop a prototype.
Initial concepts were simply a camera attached to a glass.
Later we iterated to design our own product that incorporated the camera, different sensors and a small computer for on device computations.
Drishyas' architecture was hybrid. It could do computation on device but for more complex tasks it could request the could to do the heavy lifting for it.
We ran a on device image segmentation model NanoDet Plus, the on device model was very lightweight and only could identify 70 different types of objects. We made an algorithm to determine the most important objects the user should be aware about, this was based on the score of how far the objects are, type of object and location of the object. We iterated the device to have a depth camera incorporated to identify the environment around them. If additional information was need then the user can request the cloud with the push of a button on the remote.
The remote was wireless and had six inputs. It was based of NRF device and could last months with a full charge. The main feature of the remote was the asymmetricity, usually for anyone who are not able to look at the remote will not be able identify the orientation and layout of the buttons if the remote was symmetric. So I designed the remote to have a contrasting layout when held at different position to be able to easily identify the buttons on the remote.
We ran some tests with the blind association of Nepal. It brought a lot of positive feedbacks. We even got some ideas and could understand their demands that could be incorporated in the device.
Drishya is an attempt to address the group of people who are denied from knowledge, it is an attempt to make the dauting life of visually impaired people easier and it is an attempt to give them the freedom they deserve.
We took Drishya to multiple competitions and won different awards. Some of them are Imagine Cup, Hult Prize, ICT Award Nepal.