The challenge
Accurately representing a user's environment using vision processing algorithms
The challenge is to develop novel vision processing methods that aim to extract and process salient information from a scene and represent that information using only the limited amount of phosophenes (points of light) from the retinal implant (i.e. bionic eye).
The aim is to represent the environment using vision processing algorithms that enable individuals implanted with the bionic eye to experience improved functional vision.
Our response
Developing a retinal implant and software library
We developed a retinal implant and supporting technologies, including vision processing and a software library, to restore sight to a degree in people with profound vision loss due to Retinitis Pigmentosa or Macular Degeneration.
Three profoundly blind individuals received the implant in 2012-2014 and into a further four in 2018.
Our technology selects and displays information to solve tasks required for independent living. We pioneered vision processing technologies that take image streams from a head worn camera and perform processing to turn them into stimulation patterns on the retinal implant, resulting in the perception of spots of light by the patients.
Advanced methods select and display key information such as orientation, mobility, and finds items on a table top.
This led to significant papers including a 'first-in-humans' study where we showed that the supra-choroidal approach was an effective and safe retinal implant therapy.
Critically, the paper showed end-to-end results where CSIRO vision processing enabled profoundly blind subjects to pass ultra-low vision tests. In the Journal of Neural Engineering, we reported two important breakthroughs:
- For the first time, our advanced vision processing methods led to significantly improved performance on visual tasks.
- Participants using our advanced methods for scene understanding from colour and depth images performed significantly better in mobility tasks.
The results
Vision processing in the bionic eye
This research is embodied as software that converts an image stream captured by a head-worn camera into a useful visual representation for implantees via electrical stimulation for a retinal implant. This might include highlighting the ground, obstacles, or faces.
The Bionic Eye consists of the implantable hardware, an external processor, and image processing software. The implant sits in the eye, behind the retina, and stimulates nerve cells providing the perception of vision.
The electrical impulses it produces are 'seen' as spots of light by the recipient. The implant is connected to an external smartphone sized device and a wearable camera. The device captures video from the camera and encodes it for the implant.
1. The camera in the glasses the user wears are captured in real time
2. The image is sent to the processing device (a small black box located externally on the user)
3. That processed data is then sent to the implant and then to the brain for user navigation
In 2017, spin out company Bionic Vision Technologies, which CSIRO is a shareholder, raised US$18M in capital.
Following a successful two-year feasibility study conducted on patients with retinitis pigmentosa, Bionic Vision Technologies is on track to complete clinical trials in early 2024, with plans for the device to become available commercially in the US and Europe in late 2024 subject to additional capital funding.
"The team at CSIRO's Data61 are excited about the success of the trials Bionic Vision Technologies is conducting, with the possibility that individuals with retinitis pigmentosa could have their vision restored an amazing impact," says CSIRO's Data61 project lead, Dr Nariman Habili.
"CSIRO's Data61 is proud to have contributed a software library that implements the algorithms that enable users to see while using this technology, and look forward to continuing to help drive this great application of AI-powered tech in the future."