Foveated rendering—Eye tracking technology in focus
Facebook’s patent “US20180046248A1—Systems and methods of eye tracking control” was published in February 2018. It got me wondering what they were up to. In December 2016, Facebook purchased Eye Tribe. Eye Tribe, a Danish start-up, developed eye tracking developer kits and software for gaze-based interaction with tablets and virtual reality (VR) headsets. And Eye Tribe had also been developing foveated rendering technology too.
I hadn't heard of foveated rendering, so I thought it might be interesting to see what foveation rendering is, who else is focussing on it and for what purposes—with the aid of intellectual property (IP) data found within PatSnap.
Word wheel showing key topics and subtopics for “foveated” patents (Source: Patsnap platform)
Foveated rendering (also known as foveated imaging, space variant imaging or gaze contingent imaging), is a graphics rendering technique which uses eye tracking to reduce the rendering workload on user interfaces by reducing image quality in peripheral vision—outside the gaze-zone of the fovea (the centre of the retina).
Foveated rendering has the benefit of saving power and memory on a display by only generating perfect graphics where you are actually looking. This allows for high resolution imaging at gaze-point even with limitations such as slow bandwidths and wireless devices.
Patent application trends
By looking at the 622 simple patent families referring to “foveation” we can see a big upturn in patent applications since 2012.
The key players
It is possibly unsurprising to see Microsoft leading the patenting charge, in terms of volume, within this technology area.
One of Microsoft’s most recent patents “US20180047203A1—Variable rate shading” describes how to render graphics in a graphical processing unit (GPU) with a flexible and dynamic, application-directed mechanism for varying the rate at which fragment shading is performed for rendering an image to a display.
A second, “US9891716—Gesture recognition in vehicles” describes how to perform gesture recognition using a time of flight (TOF) sensor and a computing system in a vehicle. The use of clustering algorithms along with foveation can allow faster and more robust recognition of gestures.
But if we look at the timeline of these patents we can see that Microsoft’s activity is on the decline. Hot on its tails and leading the latest charge is Magic Leap.
Florida based, Magic Leap, is a start-up developing head-mounted virtual retinal displays (Magic Leap One), which uses Augmented Reality (AR) to superimpose three dimensional (3D) computer-generated imagery over real world objects.
Magic Leap’s latest patent publications includes “WO2018039270A1—Virtual, augmented, and mixed reality systems and methods”. This patent discusses a display system configured to show virtual, augmented or mixed reality image data that may introduce optical distortions or aberrations to the image data.
Looking at the key patents that have been cited in other IP documents we can see that “US5262871—Multiple resolution image sensor” is the most influential patent covering foveation. The patent, dating back to 1989, has now expired, meaning anyone is free to use and build upon this technology.
AR and VR are some of the earliest adopters of foveation rendering. In the future, eye-tracking and user facing sensors are likely to be integrated into every digital device you might use, from smart phones and tablets, to workstations and even the cinema. Which suggests that foveation is a technology that we could well see making its way to a screen near you, soon.
Read more like this: