In the late 1960s, the English mathematician John Conway began experimenting with a strange form of computer known as a cellular automaton. This device consists of a grid of squares that turn black or white depending on the color of the squares around them.
The computation proceeds in rounds. In each round, the automaton updates the color of each square based on the color of its neighbors according to a set of specified rules. The resulting pattern then becomes the starting point for the next round of computation and so on.
Conway found to his surprise that simple rules can produce remarkably complex behaviors. Some patterns even moved across the grid, rather like living things. Conway called his approach the Game of Life and others have since discovered patterns that perform calculations or are even capable of reproducing themselves completely.
That has raised all kinds of questions about how cellular automata can capture the behavior of living things and the way they grow. But while there has been much success in creating two-dimensional patterns, there has been less success in creating 3D shapes.
Minecraft Morphogenesis
Now Shyam Sudhakaran at the University of Copenhagen in Denmark and colleagues have found a way to program cellular automata to build complex 3D shapes in Minecraft, the online gaming environment which allows players to assemble structures from a range of fundamental building blocks. The new technique grows, in a matter of seconds, complex structures such as cathedrals, temples and even functional moving machines. It suggests an entirely new approach to engineering and developing novel structures and machines.
While the new approach is based on Conway’s rule-based updates, it differs in some important ways. The most important difference is a fundamental change in the way the update rules operate that determine how a block changes according to the pattern of blocks around it.
Until recently, these rules were specified by simple logic. For example, the rules in the Game of Life are:
1. Any black square with two or three black neighbors survives. 2. Any white square with three black neighbors becomes a black square. 3. All other black squares become white in the next generation and all other white squares remain white.
Depending on the original seed pattern, this simple logic produces remarkably complex outcomes.
Of course, computer scientists have played with all kinds of more complex rules for updating the grid. In recent years, they have started to use neural networks to do the job.
In so-called neural cellular automata, the neural network takes as its input, the pattern of surrounding squares. These needn’t be immediate neighbors but could extend to all those within a given area. It then decides whether the square will be updated to become black or white, or indeed any other color if those are allowed options.
Neural networks also have the ability to learn. This makes it possible to train them to produce a particular shape given a certain seed pattern. The neural network knows the pattern it must produce as a goal and at each step compares its own creation to the goal. It then updates the pattern by trial and error in an attempt to reduce the difference between its creation and the goal.
Over a large number of steps it learns how to reduce the difference to zero. At this point, the neural network has produced a copy of the original pattern.
Neural Network Engineering
Sudhakaran and co’s approach is essentially the same in Minecraft, albeit with more possible outcomes because of the extra dimension and because the researchers allow the neural network to choose between 50 different building blocks in each update.
Nevertheless, the team show how its neural network can learn to create a wide variety of structures, many with complex internal details and some with moving parts such as a piston and a walking caterpillar.
The team show off their results in an entertaining YouTube video.
That’s interesting work, which hints at a new way to engineer complex devices — simply train a neural network to copy them.
One interesting comparison is the way the human body develops from a single cell, through a process of cell division, to a fully formed baby made of many different cell types — heart cells, bones cells, blood cells, brain cells and so on — all distributed in a complex 3D shape. The code behind this process is our genome and there is a sense in which this is “learned” over many generations of trial and error during the process of evolution.
All this learning is compressed into our genetic code, just as the learning to create a Minecraft structure is compressed into the neural network.
There are important differences, however. The neural network is a kind of black box that makes decisions in a way that is not visible or capable of being easily understood.
It’s not possible to identify the part of the network responsible for green leaves on trees, for example, or to cut and paste it into the network that creates caterpillars and thereby give them leaves too.
At least not yet. It may be that the structure of these neural networks will yield secrets in the future. Perhaps it will become possible to copy and paste “artistic styles” in Minecraft structures, which is exactly what neural networks have made possible for images.
Of course, there is no reason why objects in Minecraft could not have a kind of genome that encodes their structure and how to build them. Indeed, it would be straightforward to make new structures by combining the codes of old ones, adding mutations, selecting the best outcomes and then iterating. In other words, by evolving.
Indeed, it might be interesting to combine evolution and the neural network approach. So there is much more to come from this approach.
If nothing else, it reveals the huge potential of Minecraft as a laboratory for exploring engineering in an entirely new way. Conway, who recently died from a coronavirus-related illness, would surely have been impressed.
Ref: https://arxiv.org/abs/2103.08737 : Growing 3D Artefacts and Functional Machines with Neural Cellular Automata
