1 month ago
SoundVision – Mobile App Innovation for Visually Impaired
Table of Contents:
- What is SoundVision?
- How did it all start?
- The SoundVision mobile app
- Indoor navigation
- Brightness recognition
- Distance detection
- Other functionalities and capabilities
- Conclusion
What is SoundVision?
SoundVision boasts multiple functionalities that vastly improve blind and visually impaired individuals’ lifestyles by enabling them to socialize, do daily chores, and spend more time in the open. The easy-to-use and affordable solution consists of a compact, lightweight handheld device and an Android/iOS native app that this article will focus on.
How Did It All Start?
From a hackathon and а PowerPoint presentation with whales to a Grand Stevie and numerous other formidable accolades. That is the short story of SoundVision – the innovative solution that has the potential to make the life of blind and visually impaired people more independent and mobile.
Little did we know when the idea won the hackathon and took inspiration from the ultrasound that whales utilize to navigate and echolocate food… but these really were the humble and unassuming beginnings of a signature project – a project where Scalefocus’ mobile app development expertise truly shone through.
The SoundVision Mobile App
SoundVision’s mobile application is an upgrade over the hardware device as it provides a larger number of computational-intensive functions that require more computing power or server connection. Object and banknote recognition are among those, and so is indoor navigation, which is validated with BLE (Bluetooth Low Energy) beacons. As a matter of fact, all the functionality provided by the ML engine to recognize objects, currencies, and anything else you feed it to train itself is available through the app.
Indoor Navigation
Indoor navigation is among SoundVision’s most impactful features and is extremely valuable in terms of accumulated knowledge about location approximation. It enables the user to download maps of public facilities and navigate inside premises where GPS is virtually useless. The functionality combines data for buildings, suites, rooms, obstacles, Points of Interest (PoI), beacons, routes, and floor exits for navigation through Bluetooth beacons.
As GPS is quite imprecise inside buildings, Scalefocus experimented and added an orientation functionality based on Bluetooth beacons. To maximize the accuracy of the positioning system, we tried an approach based on sensor fusion, where the position is calculated by two systems – an RSSI-based triangulation of Beacons’ distances and a dead reckoning system, before merging these two positions using a 2D Kalman Filter. This approach benefits from the increased accuracy that multiple sensors provide, as the advantages of each system can largely cancel out the disadvantages of the other one.
Last but not least, during the development process, indoor navigation was perfected to such an extent that it can be implemented outside the SoundVision context as a standalone application – in supermarkets, administrative buildings, shopping malls, airports, etc.
Brightness Recognition
For implementing the brightness recognition functionality in standalone mode, in terms of hardware, we only utilize the phone’s back camera, and, in terms of software, we use the CameraX API, more specifically, the Preview and the Image Analysis use cases.
To extract valuable information from the camera frames, we turned off all the automatic camera settings (auto-focus, auto-exposure, and auto-white balance) and manually set camera exposure settings (mainly the sensor sensitivity and the sensor exposure time values). Our goal was to maximize the difference in the brightness of the images captured in a room when the lights are turned on/off or outside in the daylight. As we got the images in a YUV format, we extracted brightness information from them by averaging the values from the Y plane. Real-time brightness recognition on a separate worker thread guarantees that the UI will not get blocked.
The brightness value is then extracted into sound, primarily by playing with the background sound sample rate and frequency. We also use a custom view for showing the information in a simple visual format. To make the transitions of this view smoother, we animate those changes and calculate in real-time how many frames we analyze per second. Finally, the animation duration of the custom view is changed dynamically to smoothen transitions.
Distance Detection
SoundVision utilizes sounds and vibrations to represent the environment and help blind and visually impaired people understand how close they are to objects. Initially, the idea was to use ultrasound for sonar navigation and detection of obstacles, but it turned out the method would need to be more precise, especially with smaller objects.
Instead, we utilized LiDAR infrared laser detector as it is very accurate about the exact distance to the object. When the users point the device in the direction of an object, they get feedback via the vibration and the pitch of the sound made by the device. Curiously enough, LiDAR is commonly used for vacuum cleaner robots, which is quite a departure from the whales we took the idea from, but its precision is hard to ignore. Unsurprisingly, it is already a mainstay in the automotive and other high-profile industries.
Other Functionalities and Capabilities
Server connection with a trained algorithm for object recognition is one of the project’s major integrations. This means the device or telephone camera collects data from the environment and sends it to the app and then to the server, which recognizes the object, banknote, or currency (a car, a door, 5 EUR note, 20 EUR note, etc.) and provides feedback to the user.
Conceptually, the mobile app connects via Bluetooth Low Energy to the hardware device with its processor and OS. Simultaneously, the app utilizes all resources of the phone – internet connection, cameras, and computing power.
Other technologies Scalefocus has leveraged to implement the object recognition feature are the Open CV and Tesseract libraries. These have also been useful for the Optical character recognition (OCR) or text recognition functionality, where users can scan a text, and the device reads it back to them.
SoundVision supports over-the-air audit and update of the device and carries out the firmware download before uploading it to the device. The app also has a component that allows collaboration with the device’s omnidirectional joystick that can be used as a Human Interface Device (HID). That is really convenient for the users, as the configuration of phones that blind and visually impaired people use is quite different from the standard ones, and they can even use the joystick to navigate through menus and functionalities without taking the phone out of their pocket.
Finally, Android and iOS SDKs allow integration with 3rd party apps, which means developers of other solutions can utilize the SoundVision app functionalities (e.g. text recognition or indoor navigation). The same applies to the white cane device features, for example, the built-in digital compass or the LiDAR laser detector can be paired to various phone applications.
Conclusion
SoundVision is among the projects that our mobile experts are rightfully proud of – after all, there is no way to overstate its social impact. And that is not all – the Scalefocus mobile team also has a proven track record in many other areas, including Fintech, Telehealth, Gaming, etc.
Native mobile app development, hybrid mobile app development, app modernization, consultancy, and UX design – we provide end-to-end mobile app development, support, and maintenance services that has helped our customers ensure the seamless mobile experiences their users expect.
