Investigating the secrets of neural networks

Researcher Miika Aittala’s goal is to create visually realistic worlds in virtual environments. He employs neural networks and 3D modelling as his tools, with the outcomes in high demand in, for example, movie productions. This area of research is trending strongly around the world, and competition for the cleverest methods and researchers is intense.   

Aittala took his first step towards becoming a researcher as a teenager at the end of the 1990s. He found a copy of the 3D Studio MAX professional modelling software at Tallinn’s Mustamäe market. Inspired by the special effectsin films like Jurassic Park and Toy Story, Aittala used the software to create 3D animations and images. He might have ended up producing effects for cinema – but research was to win him over. 

From VTT and Aalto to the world

Aittala began to study technology and, in the aftermath of a summer job, also ended up completing his Master’s thesis on graphics and augmented reality at VTT Technical Research Centre of Finland. His ambitious work led to doctoral studies and a place in Professor Jaakko Lehtinen’s research group at Aalto University.

“I retained my interest in graphics after completing my Master’s, and it felt natural to start researching it when the University offered me the opportunity. I was quickly working with distinguished and well-known researchers.” 

Aittala’s doctoral thesis studied the 3D modelling and reproduction of materials to generate realistic computer graphics. Even then, his aim was to create graphics tools that were as simple, elegant and computationally light as possible.  

Aittala visited a graphics research retreat organised by MIT Professor Frédo Durand in the United States to talk about his research just a little bit before finishing his award-winning doctoral thesis. They agreed on close research cooperation during their meeting. 

This is how he started his journey to the world’s premiere technology university and its famed graphics laboratory. 

Enormous data volumes enable neural networks

The limits of traditional methods have been met in many graphics research problems, but neural networks have enabled fresh approaches to them. Although the basic principles of neural networks have been know for a long time, a proper understanding of their potential has only developed in recent years.  

“Unlike just a decade or two ago, there is an enormous amount of data, for example photos, available on the Internet, and this can be used to teach neural networks. Computing power is likewise on an entirely different order of magnitude than before.”

The computations we now perform in a few days would have taken years in the 1990s, and photos snapped with a phone can, at best, be processed in less than a second. Earlier, there was no inkling that good results would be achieved only with the arrival of great computing power.

“A large part of current research resembles charting unknown territory by trying various tools for different tasks,” Aittala says.

“I assume that, in twenty years, the overall picture will be much clearer, and the ideas now prevailing will look like stumbling, yet important first steps. The field is moving forward so rapidly
that what was considered wisdom just a couple of years ago can have been upturned several times already, however.”  

Visually believable worlds

Materials modelling and reproduction is utilised especially in various graphics applications like special effects production or augmented and virtual reality.

Neural networks are strong in approximate intuitive inference: if, for example, a picture in itself does not contain enough data, it can be complemented with the aid of computation. This creates visually credible virtual surfaces, which are more significant in gaming or cinematic applications than perfectly realistic reflections.

Neural network inference follows along the lines of human intuition, and will also generate an estimate of the material’s nature based on a “fingerprint” formed by visual clues. Each clue, which describes the material’s surface, forms its own map: a relief map shows surface shapes, specular maps detail the shininess of each pixel on a material’s surface, while reflection and base colour maps describe the other properties of the surface.

The different components of an image can be directly utilised in rendering, i.e. the creation of new virtual images frame by frame.

“A single movie scene can contain a million things. Each detail on the screen must be in order so that nothing out of place sticks out. Viewers are really good at noticing if the graphics are done wrong. This is why every movie production employs such huge numbers of artists. If I can simplify and ease their work, they can concentrate on more essential artistic issues,” Aittala says.

Flexibility of neural networks behind the AI boom

Data is accumulated for neural networks to use for learning how to make independent deductions, and the results of this are applied in, for example, the creation of virtual realities. The inference process performed by the network remains a mystery, however. Neural networks are free to form flexible descriptions or results from the input data, but they provide no explanation for the outcome.

Miika Aittala considers this development stage necessary and interesting.

“From a researcher’s perspective, things are happening rapidly and in entirely new ways. Neural networks allow us to tackle problems that were difficult to grasp in any way earlier.
The operating model as a whole will start to become clearer once neural networks have been utilised for a sufficiently long time.”

There is a long way to go before an independently operating and general artificial intelligence comes to be. Existing neural networks are only taught to solve a single, precisely defined problem at a time. A new research problem always requires a new neural network architecture.

Aittala’s new project explores how to remove blur from photographs suffering from camera shake. He also participated in a joint study arranged by NVIDIA, Aalto University and MIT that cleaned up grainy images with the aid of neuralnetworks. This study made the news around the world in July 2018.

American graphics technology company NVIDIA has licensed and started to commercialise the findings of Aittala’s doctoral research. This could result in programs game makers could use to
generate virtual realities. Applications might also end up in the hands of consumers, for example as devices that can create 3D models of any object or surface.

“Neural networks help us achieve results that couldn’t even be dreamed of just a few years ago. It remains to be seen whether this field of research will grow more established, or if the current development, which is characterised by ever-increasing enthusiasm and new breakthroughs, will continue.”•

Text: Tiina Aulanko-Jokirinne. Photo: Mikko Raskinen.

This article is published in the Aalto University Magazine issue 23 (issuu.com), October 2018.