General Theory of Neural Networks

What’s wild to me is that Donald Hoffman is also proposing a similar foundation for his metaphysical theory of consciousness, ie that it is a fundamental property and that it exists outside of spacetime and leads via a markov chain of conscious agents (in a Network as described above)

Ie everything that exists may be the result of some kind of Uber Network existing outside of space and time

It’s a wild theory but the fact that these networks keep popping up and recurring at level upon level when agency and intelligence is needed is crazy

https://youtu.be/yqOVu263OSk?si=SH_LvAZSMwhWqp5Q

The existence of a universal function approximator or function representation is not particularly unique to neural networks. Fourier transforms can represent any function as a (potentially) infinite vector on an orthonormal basis.

What would be particularly interesting is if there were a proof that some universal approximators were more parameter efficient than others. The simplicity of the neural representation would suggest that it may be a particularly useful - if inscrutable approximator.

There are a whole lot more activation functions used nowadays in NNs

https://dublog.net/blog/all-the-activations/

The author is extrapolating way too much. The simplest model of X is similar to the simplest model of Y, therefore the common element is deep and insightful, rather than mathematical modelers simply being rationally parsimonious.

This is another example of Markov Chains in the wild - so that’s what he’s seeing

The general nn is a discrete implementation of that

https://en.m.wikipedia.org/wiki/Markov_chain

Despite vast implementation constraints spanning diverse biological systems, a clear pattern emerges the repeated and recursive evolution of Universal Activation Networks (UANs). These networks consist of nodes (Universal Activators) that integrate weighted inputs from other units or environmental interactions and activate at a threshold, resulting in an action or an intentional broadcast. Minimally, Universal Activator Networks include gene regulatory networks, cell networks, neural networks, cooperative social networks, and sufficiently advanced artificial neural networks.

Evolvability and generative open-endedness define Universal Activation Networks, setting them apart from other dynamic networks, complex systems or replicators. Evolvability implies robustness and plasticity in both structure and function, differentiable performance, inheritable replication, and selective mechanisms. They evolve, they learn, they adapt, they get better and their open-enedness lies in their capacity to form higher-order networks subject to a new level of selection.

"Topology is all that matters" --> bold statement, especially when you read the paper. The original authors were much more reserved in terms of their conclusions.

God this grandiose prose style is insufferable. Calm down.

Anyway, this doesn't even try to make the case that that equation is universal, only that "learning" is a general phenomena of living systems, which can be modeled probably in many different ways.

How does the attention operator in transformers, in which input data is multiplied by input data (as opposed other neural network operations in which input data is multiplied by model weights) fit into the notion of a universal activator?

People seem to be obsessed with finding fundamental properties in neural networks, but why not simply marvel at the more basic incredible operations of addition and multiplication, and stop there?

There are some interesting parallels to ideas in this article and IIT. The focus on parsimony in networks, and pruning connections that are redundant to reveal the minimum topology (and the underlying computation)is reminiscent of parts of IIT: I’m thinking of the computation of the maximally irreducible concept structure via searching for a network partition which minimizes the integrated cause-effect information in the system. Such redundant connections are necessarily severed by the partition.

We must always remember that all models are wrong, though some are useful.

In the biology there are families of neurons, each one with different morphologies.

can't rule out it was generated by ChatGPT

I switched off at paragraph two:

"Prokaryotes emerged 3.5 billion years ago, their gene networks acting like rudimentary brains. These networks controlled chemical reactions and cellular processes, laying the foundation for complexity."

... for which there is no evidence at all. Psuedo-science, aka Fantasy.

If there is a 'god equation' it will almost certainly include a+b=c because we use it all the time to describe "diverse biological systems with vast implementation constraints".

This article is lacking originality and insight to such degree that I susupect it is patentable.