The modern approaches in AI for utilizing Deep Learning to solve mathematical modeling challenges have transformed many different fields, starting with Image Detection and Natural Language Processing, just recently moving into protein folding and weather prediction.

The latest developments here have been generative models, the most prominent being language generation like chatGPT and image generation using text inputs like DALL-E and Stable Diffusion. These approaches make use of large sums of data and the rise of GPU computational power to create artificial intelligence systems capable of finding patterns where prior approaches have failed.

The approach of this project is to use what has been developed so far in AI, develop it further and use it to solve one of the biggest scientific mysteries, the emergence of complex life from a single cell. The field of biology has been transformed into a big data field of its own.

New automated data gathering techniques like single cell sequencing are giving insights into the complexity of life at an unprecedented scale and resolution. However, ever since the beginning of genetic sequencing, the amount of data has outpaced our ability to make sense of it.

One major inspiration is the field of artificial life, and it is our conviction that – especially in biology – it has a lot to offer to advance the application of AI as well as improve its foundations in terms of efficiency and reliance on data.

State of the Art and what to expect
The current state of the art in Artificial Intelligence is being pushed forward by so-called foundation models based of a subset of novel architectures best suited to make use of the now available large resource, both in terms of computation and data. These models are currently built either based on a transformer architecture or a latent diffusion architecture with a massively oversized parameter count a the order of magnitude of 175 Billion parameters in the case of GPT3, to raise a recent example. These models are then trained (pre-trained) on large datasets usually sampled from internet sources.

The models show emergent properties which grow with parameter size. They manage to generalize beyond their training data in fields like text and image generation. It is still common practice to take these pretrained foundation models and train them on a smaller dataset about a specific problem to further improve results. However modern foundation models show the ability to learn patterns from a few examples provided in the input (the so-called prompt). With the recent practice of prompt engineering this is a field to optimize the results produced by good prompts.

This leads – in the limit – to the replacement of fine-tuning by prompt engineering. With this upside one can consider the use of such models on biological data and this has already shown to lead to impressive results in the form of language models capable of solving the protein folding problem to a degree which was considered unthinkable when it was released in the form of Alphafold 2 and RosettaFold.

This happens at a time when the data available in biology is seeing its own revolution with new assays capable of collecting data at new resolutions and in new magnitudes. The first of these databases are the massive amounts of gene sequencing data, an area that has a cost per sequencing development which even leaves behind Moore’s Law and correspondingly the sequences of proteins expressed by these genes.

In terms of proteome analysis the human proteome project released its dataset in 2020 after 10 years of work, in large parts based on the capabilities in mass spectrography available, providing a large annotated human protein database. Genome-Wide-Association-Studies can map between phenotypical measurements and the underlying genes involved in their development.

Single-Cell data refers to the collection of gene expression, chromatin accessibility and proteome data at single-cell resolution at a large scale, allowing the monitoring of cell development from stem cell to fully differentiated cells at all stages of development. As is the case in other fields like image processing and natural language processing, the computer-aided analysis of biological data is currently being pushed forward by advances in artificial intelligence, protein folding being the most prominent example.

On its own, these developments show great promise in unraveling the patterns hidden in the massive datasets collected. However, there are limits to the current approaches in artificial intelligence. Especially the high resource requirements in terms of data and computation are beginning to reach their limits when based on the current foundation models, which is why large research groups at Google, Meta and OpenAI have begun incorporating other modes of model training not relying on model size and large data. Indeed the large requirements limit the amount of researchers capable of building such massive models to a few research labs with the required financial backing to train such resource intensive models.

So while it is great that such models exist and are often available to the public and the rest of the research community for use, its sheer resource demands limit the amount of researchers capable of actually building such models themselves.

The goal of this project is to find a way around this dilemma. We propose that the solution can be found in the field of artificial life in order to model complex biological phenomena, which would also yield the ability to model other complex phenomena ranging from particle physics to population dynamics, ecology, climate and other fields studied by the interdiciplinary complexity science.

The approach taken here is a first principles approach rooted in theoretical biology. If one takes the analysis of complexity in biology by Robert Rosen seriously, one comes to the conclusion that current reductionistic models (necessarily) fall short of explaining complex models because they are ‘too empoverished in entailment’.

What is necessary is a model of the developing phenotype taking into account the interaction of genome, transcriptome, proteome, metabolome and the environment as well as the communication/interpretation/modeling across different layers of abstraction both upwards (emergent behavior through interaction of low-level agent) but also downwards (flow of higher-level-information from higher levels of hierarchy back to lower levels of organisation).

As Mary Jane West-Eberhard writes in her book ‘Developmental Plasticity and Evolution’, such a phenotype-development-based model is essential to both developmental biology as well as to evolutionary biology in order to understand how and where evolutionary pressure drives the evolution of changing phenotypes and eventually genetic changes.

What would this entail, though, in terms of artificial intelligence design? If one considers intelligence as part of a developing phenotype the question arises how its development takes place in space and time. Genes obviously are too limited to describe at detail the entire structure of the organism (genetic bottleneck), but they also seem to contain enough information to drive an initial cell to process its environment in a way to drive the development of the necessary complexity for intelligence over time and also influencing the ongoing processing of environmental information for intelligent behavior like learning as well as other forms of adaptation (as well as a selection for adaptability itself!).

So not encoding the intelligence and adaptation directly but encoding robust strategies for its development in an environment seems to be the path to intelligence in nature. This is in direct contrast to the static supervised learning approach common in current deep learning approaches where a fixed set of rules is learned to solve a singular task given by a fixed dataset. There are ideas for incorporating these features in artificial intelligence models and they are beginning to be explored in the field of artificial life which is dealing with modeling biology using computational means.

Starting with initial approaches like Conway’s game of life, Lindemann systems, boids and genetic algorithms, the central idea in modeling life in the field of artificial life is the notion of emergence, where small units, be they cellular automata, agents or digital ‘genotypes’ interact to produce phenomena which cannot be explained at the level of its low-level components, but are a result of their interaction at massive scale.

What is missing here, the authors of this proposal claim, is the downward path of information flowing from higher levels of hierarchy downwards to adapt the low-level interaction to further the demands of higher levels of organisation. In the case of Conway’s game of life one can see that the three simple rules which drive the cellular automata can give rise to gliders, glider cannons and even a turing-complete computer, but it cannot create complex evolution.

The reason for this is that once higher levels of hierarchy are reached, they cannot adapt the lower levels for further development to grow towards the next hierarchical level in complexity. It is this feedback loop that we intend to incorporate into models to allow them to learn to adapt across grown hierarchical levels to incoming information, to reorganize at all levels to meet the demands of a changing environment as natural systems do.

Even the most flexible of current deep learning models, the transformer, lacks the flexibility to do such things. A model more susceptible to being adapted to these needs is the model of latent diffusion models, where the model does not learn a single output, but learns to process input to its next iteration and only by repeated application of the model does the final output emerge. But again, this processing is fixed and hence not flexible enough.

Another problem in current models is that they need to be trained at the level of complexity desired in the end. The processed data must fit the level of complexity of the training data. Even if out-of-distribution generalisation is possible, solving hard problems while having been only trained on easy ones is not possible for current AI systems.

To solve this dilemma there has recently been developed an approach of training a recursive model on simple and small datasets and allow the model to be run for longer and have it solve more complex problems, extrapolating to more difficult problems by – essentially – processing the problem for longer, or evolving for longer to create a solution. To model complexity one needs a model capable of similar complexity itself, the model must have the necessary capabilities to model the flexibility of the system it is supposed to model.

We contend that current AI systems are lacking in that department. What we intend to build is an AI system truly capable of adaptively reacting to its environment. Not only will we incorporate the notion of downward feedback and multilevel flexibility, we will also build the model in a way which progressively adapts to its environment step by step (and adapts its ability to adapt), we will also make use of learning it recursively so that our model can move beyond its training data to increasing complexity if allowed to run for longer timeframes. This model, if trained on current datasets should be able to extend beyond current models in capability.

The goal is to model biological data like the development of organisms, but it may have become apparent in this proposal that this approach would also be able to model and solve other problems in complexity while also holding the promise of improving the results on current application of AI.