Update

The project to construct an automated pattern generator and recognizer is coming in fits and starts. I finally got my sweet Octave code to work (Octave is like MATLAB, but free), only to find out that I have serious memory leak problems. This sets me way back, and makes me think that I should go back to my original idea of developing C++ code, and just use Octave as an algorithmic development tool.

Some other major issues include:
Numerical integrators. I only have a very basic working knowledge of numerical integrators, and at the moment the 4th order Runge-Kutta solver I am using in Octave appears to give good enough answers (at least when the solutions don't go shooting off to infinity). The question now becomes if it is worth it to try and develop my own numerical solver (probably using Numerical Recipes in C as a major resource) as well as an automated equation generator in C++? The nice thing about Octave/MATLAB is that data is much easier to manipulate than in C++.

But the biggest problem is the following: how does one quantify and qualify patterns in an automated fashion? Generating random non-linear differential equations is all well and good, but the whole point of this exercise is to basically find emergent phenomenon and characterize them. Then use what is learned by discovering emergent phenomenon to predict the existence of other emergent phenomenon in higher level structures.

The following is a very short summary of what I currently know about non-linear differential equations:
-Whenever nullclines (a nullcline is a line where the value of one of the time derivative is always zero) cross, there is a fixed point.
-Fixed points come in various flavors, from stable to unstable, and spirals, nodes and saddles. There are also other fixed points that are harder to examine (degenerate nodes, stars, centers, and some others...)
-To get chaos you have to have at least three dimensions
-Linear stability analysis at certain types of nodes (saddle, spiral, and stable) can use the Jacobian to determine the stability of a node in a nonlinear system.
-There are different types of bifurcations that can occur (where a fixed point undergoes some sort of transformation). These include saddle-node bifurcations, Hopf bifurcations, pitchfork bifurcations, and others.

Say you want to generate arbitrary emergent phenomenon for N "agents." What kind of system of equations would you use? Say the agents are homogenous (they all have the same set of interactions, but distributed amongst themselves in different ways). How does one characterize said emergent phenomenon? I guess the simplest way would be to start out with very simple equations (e.g. equations with only one or two terms), and see what the behavior is. Then bump up the complexity by adding new terms. So, for example, each individual would be represented by a single variable (this variable could be extended to any number of variables later on...). This variable would dictate something (such as position, hunger, velocity, size, concentration, or any other metric one wished to develop an interaction (and hopefully emergence) framework for). Then, the variables between individuals would be coupled, such that the concentration value (for example) of one agent has an effect on the rate of change of the concentration value for another agent. Then one could add environmental factors as well into some or all of the equations. This could be a natural decay rate of concentration, or influx rate. Environmental variables in themselves that change over time (e.g. spatial food/energy concentration function) could be incorporated.

Some other points to cover on these ideas in the next post:
Importance of connectivity between agents (random, uniform, higher order structure (which probably means higher order information)
Importance of correlations between agents
Importance of nonlinearity between agents
Defining emergent behavior with:
1) Correlation Functions (e.g. given I know that agent 1 is doing action x, what can I say about what agent 2)
2) Spatio-Temporal Distributions (how random/non-random is the distribution of agents in space-time)
3a) Information (e.g. given a set of inputs, what does the set of outputs look like (how discriminating/robust is the system between certain inputs)
3b) Does the system destroy, create, or merely transform information?
3c) Can we find symbols in the information patterns created by the system?
Other random thoughts with respect to making this project easier and more importantly, fruitful...

Hello!? Is anyone there?

The strange thing about emergent phenomenon is that it can often seem like magic to even the most scientifically inclined person. I think it is largely because our minds evolved to be able to recognize and characterize certain kinds of patterns. Enormously complicated interconnected nonlinear systems were not one of the things our minds are naturally built to deal with. We like to understand systems as a series of rules where every action has a cause and effect. The problem with emergent systems is that they are often composed of many heterogeneous parts that act in nonlinear ways. This means that dissecting the tangent of causality in such systems is practically impossible, because it gets lost in a sea of other interactions that are affecting the system in often unpredictable ways.

I read somewhere that the butterfly effect really is not as significant as it would seem. In essence a butterfly could flap its wings and cause a storm halfway across the world, assuming the weather patterns are chaotic or highly sensitive to initial conditions. Yet to trace the line of causality to that specific butterfly would be impossible. The problem is that there are so many other variables in the system (other butterflies to start with) that are doing things to change the system, just like the hypothetical storm-causing butterfly. These lines of causality become completely tangled within this system and we lose our ability to say that a single butterfly at a specific point in space-time caused a horrific storm. Instead, all we are left with is the knowledge that, yes it could have been the butterfly, but it could have more likely been any number of other factors.

Complications in causality factors into many other complex dynamical systems. That's one of the reasons why fighting cancer is so hard. Cancer does not have a simple on-off switch. It is a complicated set of malfunctions and conditions that can change a cell from its normal healthy cell-type (or personality, say) into an unhealthy, voracious cell-type (or a psychopathic maniac). It is hard to trace the causes to a single gene or environmental factor. We do know that there are things that can tend to cause cancer, and there are many genes that have been identified as increasing the chance of getting cancer, but there is still no comprehensive understanding of the full set of underlying mechanisms and how different contributing factors have different causes and effects that lead up to cancer.

Other systems where lines of causality become tangled range from ecosystems, to economics, to the weather, and to biology in general. I wonder if it would be possible to develop a general framework to deal with these kinds of systems. It may be necessary to throw out a lot of our preconceptions about how to look at these kinds of systems. Science, for example, is very good at breaking up a system into pieces, examining the pieces, and seeing how they fit back together. The problem with this reductionist method is that it can miss out on higher level phenomenon that is only present when the system is functioning as a whole. Another example of this is in complicated signaling transduction pathways or gene networks, scientists often can not analyze the system as a whole, so they look at pieces of the system, and then try to put it together and reproduce the dynamics of the whole system.

I wonder if it is possible to come up with an evolving classification system for emergent phenomenon that would be able to recognize certain types of phenomenon and be able to predict the set of interactions underlying the phenomenon. If it could find an appropriate set of interactions, model, or even a suite of possible models, then it could be possible to be able to know the complicated lines of causality, or at least what the probable lines of causality were. The reason why the classification scheme of emergent phenomenon would have to be evolving in itself is because if it couldn't evolve there is no way it could deal with novel and unpredicted dynamics.

There is at least one huge potential problem with this classification scheme idea (and there are probably more). I've mentioned in my previous post The Adjacent Possible that there are an essentially infinite number of possible interactions, and therefore it is impossible to predict all possible configurations between objects in phase space. Hence, trying to catalog all emergent phenomenon may seem like a silly and useless idea. Yet, I have a hunch that there are certain types of interactions that are more likely to occur between objects than others, and that though there may be a large number of possible interactions and also a large number of potential objects, the distribution of objects and interaction is not uniform. There may be a few incredibly wacky interactions and even wackier objects (a conscious entity is a pretty wacky object in my opinion), but the majority of objects and interactions of interest fall into a relatively small number of classification schemes (at least in comparison to infinity). My justification for this hunch is largely based on the feeling that if the distribution of objects and interactions in the universe was entirely uniform, it would be impossible for complicated systems to develop.

The reasoning goes as follows: Stuart Kauffman discusses a hypothetical situation where all the biochemicals in all of biology were combined into a single vat. In the first few instants there would be an untold number of interactions between these biochemical objects. This would be similar to the case where the distribution of interactions and objects was made uniform from its natural non-uniform shape. Yet after a few instants this bubbling froth of biology would eventually create molecules that would hijack the entire system, poisoning most other molecules, until it finally reached a state where there were only a relatively small set of interactions occurring. Hence, biology has generated ways to keep the distribution of objects and interactions between objects very non-uniform...because if it ever approached uniformity everything would fall apart.

This seems to suggest to me that most emergent systems will not have a uniform distribution of underlying objects and interactions. In fact, the distribution of interactions and objects must be skewed in such a way as to give the system some sort of useful higher level property. Any sort of non-uniformity in the distribution of objects and interactions on any and all levels of an emergent system seems to suggest that though the number of possible interactions and objects is essentially infinite there is a much smaller number of interactions and objects that are actually implemented in reality.

More on this later...

Nonlinear Differential Equations...

I stumbled upon a very useful program today called Octave. It is a high-level numerical oriented programming language that is somewhat similar to MATLAB. After many hours of tweaking the settings, downloading files, and writing some functions that write other functions, I've stumbled upon some interesting preliminary findings. Basically, I wrote a program that generates systems of nonlinear ordinary differential equations randomly. Then I used the built-in numerical integrator to see what the behavior of these functions looks like. In this first graph, lines 1-3 are the time solutions of the variables x,y, and z with the initial conditions (0,0.3,0.1). I checked for longer times of integration and it appears that the system eventually converges on a fixed point. The initial behavior of the system is very interesting, because it acts very similarly to a damped oscillator. Yet another interesting aspect is we can see that the variables x and z are out of sync up until around 5000 time units, and then they suddenly come right into harmony. If this is not order for free I do not know what is!















The next image is the phase space of the x and y dimensions. This is a very interesting pattern, it looks sort of like a wing or a leaf. The magnitude of the time solution of the y variable is very small compared to that of the x and z variables, but it still has interesting behavior.














This image demonstrates the phase space of the solution along the x-z plane. It looks like there are two basins of attraction, and eventually the basin on the right determines the long term behavior of the system. The point at which the x and z variables come into sync is probably when the function reaches an "event horizon" where it is forced to enter the basin of attraction on the right.














Finally, this last image is the phase space in the y- z plane. This is a totally different pattern, that brings to mind a cell under going mitosis. The dynamics of this system of non-linear differential equations is obviously very complicated, but there do seem to be some underlying patterns that govern the entire system. Hopefully I will be able to develop a more systematic system for identifying these underlying patterns as well as finding that elusive beast of emergence.

Symbols and topology

In Douglas Hofstadter's book, Gödel Escher Bach an Eternal Golden Braid he goes into great detail on how inanimate matter can become conscious without the coexistence of some higher metaphysical entity such as the soul. His argument boils down to a relatively simple, if highly non-intuitive concept. It goes as follows: given a set of objects (say, for example, ants, or birds, or slime mold cells, or neuron cells, you get the picture...) there exists a set of interactions between the objects that allows them to organize themselves spontaneously. This is similar to some of the arguments presented by Steven Strogatz in his popular novel Sync, and I sort of eluded to it earlier in the post Order for Free. Once these objects organize spontaneously symbols can be encoded in patterns within these objects. These symbols are usually higher order phenomenon that are represented in the way in which the objects orient themselves in space and time. Hofstadter writes a clever interlude about a bunch of ants, where each ant is mindlessly going about its business, but the collective of ants is able to perform complicated interactions at a different level than the individual ants. A symbol is a tough object to define because it is highly dependent upon the context in which it resides. Random noise could be described as a pattern of sorts, but random noise does not have a few essential features that makes it symbol-like. A symbol has to be robust, insomuch as that it can have a variable response to similar inputs or parameters in its environment, but the variable response can not exceed a critical threshold such that the behavior of the system can not be distinguished from random behavior. This is very likely a very subtle point which may take some serious further investigation.

What do symbols do? Well that is a very good question. In my last post (The Adjacent Possible) I talked about the extra flaps of skin on the squirrel from Kauffman's book. This is a symbol of sorts, because it is a pattern among the spatio-temporal distribution of squirrel cells. The devil is in the details of course, because there are any number of other "patterns" in the spatio-temporal distribution of squirrel cells, but the key question is what patterns are important and which ones exist but do not serve any purpose. This begins to border on philosphical (and therefore there could be less opportunity to dissect analytically) for a few reasons. First of all, the idea of purpose is a little strange here, because the flaps of skin do not know anything about their greater purpose. Very likely they are the result of some random mutation or series of random mutation in a growth plan of the skin along the sides of the squirrel. There is probably a complicated network or series of networks of cellular signaling that determines the orientation, density, and type of epidermal cells that span this extra flap of skin. It seems unlikely but not necessarily impossible that a single mutation could produce such a pattern, but the point is that because of some change in the overall system of the squirrel, a new robust phenomenon has arisen. This new pattern, or symbol as it were, leads to a whole new functionality, which opens up an entire new region in the phase space of possible "spatio-temporal orientations" for not only the skin cells in the flaps of the flying squirrel, but also for future generations of squirrels. Yet this new functionality could quite possibly never be discovered without the occurence of a totally different event (the hawk!). And this is where the layers of stochasticity become coupled to the symbol (pattern). Mutation initially creates this pattern by feeding variation into the bottom of the system. This variation percolates up through the layers of organization and through some very likely complicated series of causes and effects (that probably are impossible to predict ab initio) which produces the new pattern or symbol. Yet at the top layer, the hawk has to come swooping in so that the squirrel can jump out of the tree and fly. If the squirrel never deviates from its normal behavior than it will never discover the greater functionality, or new region of phase-space that has opened up, and future generations of squirrels will be less likely to benefit from this beneficial pattern.

Symbols are usually used in a more abstract way than flaps of skin on a small furry mammal. Yet, this circles back to the original idea. The squirrel example adds a lot of added complexity, what with selection, additional stochastic processes, the the phenomenon of skin flaps being at such a high level in organization that the transition between the very basic level (say the genes) and the phenotype (the flaps) is anything but transparent. Yet going back to Hofstadter's ants, or an even more general example, with some even more boring agents, it seems possible that there exists any number of interesting behavior or symbols given the right rule set between the ants. The question becomes, can such rule sets be found such that the behavior of the ants (or agents) is something both interesting and useful in some manner? And by interesting I mean that it is robust, it responds to various inputs or initial conditions in dynamic and sometimes unpredictable ways, and that there is also something novel about the behavior.

This is a very hard idea to explain, but here it is worded differently: how does one distinguish mutliple levels in the system of ants. Obviously there is the base level, where the ants (or agents or whatever) go about mindlessly, doing whatever they do, not really thinking on a higher level. Then above the ants, because of their interactions is a much more subtle phenomenon. In Hofstadter's example, the patterns that the ants produce enable it to hold a conversation with other fictious characters. The patterns are having the conversation though, not the ants. Its as if by magic because of the interactions between the ants an entirely new level of organization and "symbols" is created. Within this higher level of organization there are new rules, new capabilities, and new constraints. The new capabilities come from the opening up of phase space that the interactions produce. This means that the ants, who as a disconnected uncoordinated mob, would not be able to accomplish much of anything (such as gather resources, create a suitable living environment, protect themselves) with any great success, are able to accomplish both new and greater things. For example, a single ant could never build a complicated colony in which to live, work, and propogate. But a colony of ants can. Also, a single ant could not protect itself from certain predators, but a colony of ants just by virtue of it being composed of many ants has a natural buffer against predation, because a predator would have to wipe out the entire colony...and the analogies go on. Colonies of ants can cover more distance, forage more efficiently, etc...But there is also more constraints for this higher level of organization. The colony can only grow so big, based upon the available resources and the environment in general. The colony is more adaptable than a single ant...it can respond almost like a super organism, if say a predator destroyed part of the ant mound, the ants could work in concert to repair the broken area, but there are limits to this adaptability. Also, ants who live by the colony, die by the colony.

Anyway, the whole point of this long rambling, is that interactions between agents in a system can produce very interesting behavior...behavior that can be described as being a symbol even. Now, the natural extension of this that Hofstadter goes into in great detail, is that the mind is essentially many many levels of these kinds of symbols, chugging along doing their thing, whether programmed in from birth, or whether acquired through interactions with the environment. Here is the key idea: the patterns created by the interactions between the base units (or the symbols on the first level above the bottom) can also interact with one another to produce higher level patterns above them. In terms of programming language this is sort of like the jump from machine language to assembly language, and then from assembly language to a higher level language like C++.

There is something very important to notice about the mind though. This is the fact that these levels of symbols can talk to one another. It is not like they are completely seperated from each other, not affecting the behavior of each other running like they are completely independent of one another. No, it is in fact very important to recognize the recursive relationsip between the levels of the system. Hofstadter's most controversial and contentious point is the idea that consciousness is the highest level symbol the mind has. He argues this because when someone become conscious, they become more aware of themself. In terms of the symbol analogy, all the different symbols that make up you, from every purely physical experience you have had, to every emotional experience you have had, to every thought and internal model you have built to try and reconcile the world in your head to the world outside of you starts to realize something. What all these symbols start to realize is that they exist. Whereas most other things that we know of, from the simplest to the most complex organisms, go about doing things for whatever reason, we are one of the only entities that we know of that can look aback upon ourselves and say AHA! I am a thing that thinks about other things, that feels things, and that has symbols for other things. It is as if all the symbols that flow through your mind do a doubletake, and realize that they are there in the first place. Hofstadter describes this as the top level of the mind looking down at the layers beneath it. This is one of the most amazing occurences in the history of life on earth, because by gaining the power to look down upon the symbols that make up ourselves we essentially bootstrap ourselves into consciousness and all the wonders of being conscious entities...(e.g. complex emotions, from love to hate, a sense of humor, the appreciation of beauty, and perhaps even a soul).

Now what I want to know is why isn't there a way to be able to both create such systems that build layer after layer of symbols on top of each other, each new layer adding some sort of new phenomenon or ability to the system. I have a hunch that there are a variety of factors that are very important in generating such systems. One such factor is the topology both within and between levels. The topology, or connectivity between both the base units or agents, as well as the symbols in each level above the base level, is probably highly correlated with the corresponding functionality. Also, I am pretty sure that the cross-talk between levels is also very important in gaining certain higher level functionality that is beyond the scope of isolated levels of symbols working independently of one another. Finally, the biggest mystery is the types of interactions between elements that are necessary to construct such elaborate systems of symbols and patterns.

These ideas drive to the very core of what I believe are some of the most interesting questions out there at the moment. I truly hope to develop a general framework for creating such systems of multiple levels of symbols, and maybe even developing an entirely new language or something! to deal with such systems.

I sincerely hope to write that C++ program ASAP so I can start posting actual results including graphical representations of some interesting systems.

The Adjacent Possible

One of the most interesting aspects of the creation of new patterns and "stories" is that there are always aspects of a system that are beyond the capability to know ab initio. Stuart Kauffman discusses this issue in great detail in his book Investigations. He claims that the phase space for biology can never be completely defined, because the order of possible interactions is so large that it is essentially infinite and that in a way, there are always aspects of a system that are beyond our knowledge until they actually occur.

This makes me wonder if there is a way to shortcut the "innovation" process. Say, for example, one has two widgets. These widgets would have a fantastical, magical behavior, if they were able to enter into a specific configuration with respect to one another (e.g. a rule set) at the right place in space and time. In Stuart's book, he references the flying squirrel, whose abnormality in skin proves to be incredibly useful in the right context, when a hawk attempts to swoop down and eat the poor little animal (e.g. some aspect that seems apparently random or even a hindrance, ends up being very useful). Now in this case, it seems that there is no way of knowing ahead of time the usefulness of extra skin flaps when presented with an unbelivable number of possible circumstances that any object can enter in the course of its existence.

But, on the other hand, what if there was a way to create some sort of function, or program that searches specifically for new and innovative tools. Say, for example, this function was recursive, insomuch as it would use new tools (or patterns) to improve itself and its searching ability. This function would grow and morph into something more complicated as time went on. It would need a substrate on which to grow, and a plethora of information and constraints to establish incentives for it to innovate. Patterns in a vaccuum are relatively useless, but patterns that serve a purpose are infinitely more interesting and meaningful.

And here is where the tough questions begin to arise. Can such a function define its own purpose? Or does said purpose have to be constructed artificially ahead of time? These are difficult questions that may take some serious investigation to answer. It seems possible that such a program could start out with a very simple purpose, which as a by product of its activities would change in some way, and in turn would cause the function to change as well. A simple purpose would be an optimization problem...to see if the function could find a solution to the problem. But as the function analyzed and searched for a solution, it would change the landscape of the problem space, causing it to have to develop further more complicated techniques to search for a solution. The question becomes, how does one couple the search for a solution to the solution itself, and how can one incoporate the necessary freedom to allow for unpredictable but interesting behavior.

There is much to think about and elaborate on in the coming weeks and months.

Quick thought...

...a simple c++ program that given a system of ordinary differential equations does the following:
a) Finds all fixed points and steady-states
b) Classifies stability and type of steady-state
c) Gives a plot of the phase-space given different initial conditions with..
d) ...the possibility of a uncertainity (or a distribution) of initial conditions, and the corresponding effect on the trajectories through phase space

This information will then be used to further develop a theory as to the genesis of patterns based upon interactions between elements, as well as, perhaps, information storage, transformation, and encoding within dynamical systems.

Order for Free?

Assuming any complicated system of interconnected parts can be broken down into both deterministic and stochastic interactions, is there a way to quantify and qualify all salient aspects of the system? Salient is a tricky word, because it implies some sort of functionality or significance, which are only meaningful in the context of a human defined ontology. It seems that there should a way to define such systems, and search for patterns in the genesis of other patterns.

For example, consider the following system of ODE's:

x' = x^2 - y^2
y' = x^2-x+xy

where x' and y' are dx/dt and dy/dt respectively. The Jacobian matrix of this system looks as follows:

A =[ (2x) (-2y)
(2x-1 +y) (x) ]


The fixed points in this system occur where the nullclines intersect, or in other words where

0 = x^2-y^2
0 = x^2-x+ xy

intersect. This occurs at the points (1/2,1/2) and (-1/2,-1/2) in the x,y plane. The Jacobian evaluated at these points is A(1/2,1/2) = [1,-1;1/2,1/2] and A(-1/2,-1/2) = [-1,1;-5/2,-1/2]. For (1/2,1/2) the trace is 3/2 and the determinant is 1. For (-1/2,-1/2) the trace is -3/2 and the determinant is 3. This suggests that the first fixed point is unstable and the second is stable, with the dynamics of the two points being spirals (since the discriminant is negative in both cases). Hence the long-term behavior of the system will eventually approach (-1/2,-1/2) as long as it does not start exactly at (1/2,1/2) with no noise.

This tends to suggest that such a system would destroy information over time. In other words, if the initial conditions contain important information (say, the position in two dimensional space of some particle whose time evolution of its x and y coordinates is dictated by the previous system of equations) that information is eventually transformed into (1/2,1/2) after a long enough period of time. The system might transition through some interesting intermediate locations in phase space, but it will eventually end up at the stable fixed point.

Could this be an example of order for free? It is hard to say. It all comes down to what order means. If in this case order means a reduction in information, then yes, this system definitely reduces the information encoded in the initial conditions into the fixed point. But it also transforms that information into a trajectory through space-time that asymptotically reaches the fixed point, and different initial conditions could have wildly different trajectories. It would be interesting to see if the distribution of trajectories given some uncertainty in initial conditions adds any more information to the system...e.g. nearby trajectories tend to be like one another, or, on the other hand, are radically different.

Plus, adding uncertainty to the equations themselves could present some interesting results. In that case the system would never reach a total equilbrium, but would probably oscillate around a stochastic equilibrium.

Hopefully in my next posts I will have some more interesting thoughts with respect to more complicated structures...here are some rough ideas:
-multiple fixed points (or steady-states)
-how to perturb a system to transfer between fixed points (or steady-states)
-in the case of polynomial ODE's, the importance of the order of the equations
-connectivity (for systems with greater number of nodes)
-systems where each element has multiple dimensions (e.g. interactions between particles in space, each particle has three dimensions which describe its position)
-the importance of stochasticity and types of stochasticity (PDE, Monte Carlo, etc...)
-how does information (and perhaps entropy) play a role...and most importantly...
-how can a computer recognize a novel and useful phenomenon and classify it as something new and significant?
-and then, how can that computer take that information and apply it to both information it already has available as well as new information presented to it?