In an intriguing new study, researchers at the University of Maryland have managed to transform eye reflections into somewhat identifiable 3D scenes. This remarkable experiment utilizes Neural Radiance Fields (NeRF), a form of AI technology capable of recreating environments from two-dimensional photographs. While the technology is still in its early stages, with practical applications yet to be seen, this initial study paints a captivating picture of future possibilities, including the reconstruction of environments from a mere set of portrait photographs.
The research team embarked on this fascinating journey by examining the subtle play of light reflected in human eyes, captured through a series of consecutive images from a single sensor. Their objective was to gain insight into the person’s immediate surroundings based on these reflections. The method involved taking several high-resolution images from a stationary camera position as an individual, moving in front of the camera, glanced towards it. The reflections in the eyes were then magnified, isolated, and analyzed to determine the direction of the gaze in each photograph.
The results of this process offered an intriguing, albeit rudimentary, reconstruction of the environment as reflected in the human eye under controlled conditions. The experiment even led to the creation of a dreamlike scene captured from a synthetic eye. However, the same process applied to eye reflections from popular music videos of Miley Cyrus and Lady Gaga only generated indistinct forms, thought to be an LED grid and a camera on a tripod. This outcome highlights the distance yet to be covered before this technology becomes truly useful in real-world scenarios.
The team faced a host of challenges to achieve even these basic reconstructions. The cornea’s natural “noise,” for instance, complicates the separation of reflected light from the intricate patterns of the human iris. To tackle this, they employed strategies such as cornea pose optimization and iris texture decomposition during the training phase. These methods estimated the cornea’s position and orientation and extracted unique features from the individual’s iris, respectively. Additionally, radial texture regularization loss, a machine learning technique, was used to simulate smoother textures and further isolate and enhance the reflected images.
However, despite their remarkable progress and innovative problem-solving, the team still acknowledges that there are substantial hurdles to overcome. The real-world results so far have been derived from lab setups that involved closely captured facial shots, controlled lighting, and deliberate subject movements. They anticipate challenges in more unconstrained scenarios, such as video conferencing with natural head movements, due to issues like lower sensor resolution, dynamic range, and motion blur. The team also recognized that their universal assumptions about iris texture might be too simplistic for broader application, given the broader range of eye movements in real-life scenarios.
Nonetheless, the team views their work as a stepping stone towards future advancements. They expressed hope that their efforts would stimulate further exploration into utilizing unexpected visual cues to reveal information about our surroundings, thereby pushing the boundaries of 3D scene reconstruction. While this evolving technology could potentially raise privacy concerns in its mature form, rest assured, the current version is still struggling to clearly recognize a Kirby doll, even in ideal conditions.
