On Nonlinearity VI

"Up until now most economists have concerned themselves with linear systems, not because of any belief that the facts were so simple, but rather because of the mathematical difficulties involved in nonlinear systems [... Linear systems are] mathematically simple, and exact solutions are known. But a high price is paid for this simplicity in terms of special assumptions which must be made." (Paul A Samuelson, "Foundations of Economic Analysis", 1966)

"Linear relationships are easy to think about: the more the merrier. Linear equations are solvable, which makes them suitable for textbooks. Linear systems have an important modular virtue: you can take them apart and put them together again - the pieces add up. Nonlinear systems generally cannot be solved and cannot be added together. [...] Nonlinearity means that the act of playing the game has a way of changing the rules. [...] That twisted changeability makes nonlinearity hard to calculate, but it also creates rich kinds of behavior that never occur in linear systems." (James Gleick, "Chaos: Making a New Science", 1987)

"Never in the annals of science and engineering has there been a phenomenon so ubiquitous‚ a paradigm so universal‚ or a discipline so multidisciplinary as that of chaos. Yet chaos represents only the tip of an awesome iceberg‚ for beneath it lies a much finer structure of immense complexity‚ a geometric labyrinth of endless convolutions‚ and a surreal landscape of enchanting beauty. The bedrock which anchors these local and global bifurcation terrains is the omnipresent nonlinearity that was once wantonly linearized by the engineers and applied scientists of yore‚ thereby forfeiting their only chance to grapple with reality." (Leon O Chua, "Editorial", International Journal of Bifurcation and Chaos, Vol. l (1), 1991) 

"It remains an unhappy fact that there is no best method for finding the solution to general nonlinear optimization problems. About the best general procedure yet devised is one that relies upon imbedding the original problem within a family of problems, and then developing relations linking one member of the family to another. If this can be done adroitly so that one family member is easily solvable, then these relations can be used to step forward from the solution of the easy problem to that of the original problem. This is the key idea underlying dynamic programming, the most flexible and powerful of all optimization methods." (John L Casti, "Five Golden Rules", 1995)

"When it comes to modeling processes that are manifestly governed by nonlinear relationships among the system components, we can appeal to the same general idea. Calculus tells us that we should expect most systems to be 'locally' flat; that is, locally linear. So a conservative modeler would try to extend the word 'local' to hold for the region of interest and would take this extension seriously until it was shown to be no longer valid." (John L Casti, "Five Golden Rules", 1995)

"A system at a bifurcation point, when pushed slightly, may begin to oscillate. Or the system may flutter around for a time and then revert to its normal, stable behavior. Or, alternatively it may move into chaos. Knowing a system within one range of circumstances may offer no clue as to how it will react in others. Nonlinear systems always hold surprises." (F David Peat, "From Certainty to Uncertainty", 2002)

"In a linear system a tiny push produces a small effect, so that cause and effect are always proportional to each other. If one plotted on a graph the cause against the effect, the result would be a straight line. In nonlinear systems, however, a small push may produce a small effect, a slightly larger push produces a proportionately larger effect, but increase that push by a hair’s breadth and suddenly the system does something radically different." (F David Peat, "From Certainty to Uncertainty", 2002)

"Complex systems are full of interdependencies - hard to detect - and nonlinear responses." (Nassim N Taleb, "Antifragile: Things That Gain from Disorder", 2012)

Catastrophe Theory III

"On the plane of philosophy properly speaking, of metaphysics, catastrophe theory cannot, to be sure, supply any answer to the great problems which torment mankind. But it favors a dialectical, Heraclitean view of the universe, of a world which is the continual theatre of the battle between 'logoi', between archetypes." (René F Thom, "Catastrophe Theory: Its Present State and Future Perspectives", 1975)

"At the large scale where many processes and structures appear continuous and stable much of the time, important changes may occur discontinuously. When only a few variables are involved, as well as an optimizing process, the event may be analyzed using catastrophe theory. As the number of variables in- creases the bifurcations can become more complex to the point where chaos theory becomes the relevant approach. That chaos theory as well as the fundamentally discontinuous quantum processes may be viewed through fractal eyeglasses can also be admitted. We can even argue that a cascade of bifurcations to chaos contains two essentially structural catastrophe points, namely the initial bifurcation point at which the cascade commences and the accumulation point at which the transition to chaos is finally achieved." (J Barkley Rosser Jr., "From Catastrophe to Chaos: A General Theory of Economic Discontinuities", 1991)

"Catastrophe theory is a local theory, telling us what a function looks like  in a small neighborhood of a critical point; it says nothing about what the function may be doing far away from the singularity. Yet most of the applications of the theory [...]  involve extrapolating these rock-solid, local results to regions that may  well be distant in time and space from the singularity." (John L Casti, "Five Golden Rules", 1995)

"Chaos and catastrophe theories are among the most interesting recent developments in nonlinear modeling, and both have captured the interests of scientists in many disciplines. It is only natural that social scientists should be concerned with these theories. Linear statistical models have proven very useful in a great deal of social scientific empirical analyses, as is evidenced by how widely these models have been used for a number of decades. However, there is no apparent reason, intuitive or otherwise, as to why human behavior should be more linear than the behavior of other things, living and nonliving. Thus an intellectual movement toward nonlinear models is an appropriate evolutionary movement in social scientific thinking, if for no other reason than to expand our paradigmatic boundaries by encouraging greater flexibility in our algebraic specifications of all aspects of human life." (Courtney Brown, "Chaos and Catastrophe Theories", 1995)

"[...] chaos and catastrophe theories per se address behavioral phenomena that are consequences of two general types of nonlinear dynamic behavior. In the most elementary of behavioral terms, chaotic phenomena are a class of deterministic processes that seem to mimic random or stochastic dynamics. Catastrophe phenomena, on the other hand, are a class of dynamic processes that exhibit a sudden and large scale change in at least one variable in correspondence with relatively small changes in other variables or, in some cases, parameters." (Courtney Brown, "Chaos and Catastrophe Theories", 1995)

"Chaos and catastrophe theories directly address the social scientists' need to understand classes of nonlinear complexities that are certain to appear in social phenomena. The probabilistic properties of many chaos and catastrophe models are simply not known, and there is little likelihood that general procedures will be developed soon to alleviate the difficulties inherent with probabilistic approaches in such complicated settings." (Courtney Brown, "Chaos and Catastrophe Theories", 1995)

"Fundamental to catastrophe theory is the idea of a bifurcation. A bifurcation is an event that occurs in the evolution of a dynamic system in which the characteristic behavior of the system is transformed. This occurs when an attractor in the system changes in response to change in the value of a parameter. A catastrophe is one type of bifurcation. The broader framework within which catastrophes are located is called dynamical bifurcation theory." (Courtney Brown, "Chaos and Catastrophe Theories", 1995)

"Probably the most important reason that catastrophe theory received as much popular press as it did in the mid-1970s is not because of its unchallenged mathematical elegance, but because it appears to offer a coherent mathematical framework within which to talk about how discontinuous behaviors - stock market booms and busts or cellular differentiation, for instance - might emerge as the result of smooth changes in the inputs to a system, things like interest rates in a speculative market or the diffusion rate of chemicals in a developing embryo. These kinds of changes are often termed bifurcations, and playa central role in applied mathematical modeling. Catastrophe theory enables us to understand more clearly how - and why - they occur." (John L Casti, "Five Golden Rules", 1995)

"The goal of catastrophe theory is to classify smooth functions with degenerate critical points, just as Morse's Theorem gives us a complete classification for Morse functions. The difficulty, of course, is that there are a lot more ways for critical points to 'go bad' than there are for them to stay 'nice'. Thus, the classification problem is much harder for functions having degenerate critical points, and has not yet been fully carried out for all possible types of degeneracies. Fortunately, though, we can obtain a partial classification for those functions having critical points that are not too bad. And this classification turns out to be sufficient to apply the results to a wide range of phenomena like the predator-prey situation sketched above, in which 'jumps' in the system's biomass can occur when parameters describing the process change only slightly." (John L Casti, "Five Golden Rules", 1995)

"The reason catastrophe theory can tell us about such abrupt changes in a system's behavior is that we usually observe a dynamical system when it's at or near its steady-state, or equilibrium, position. And under various assumptions about the nature of the system's dynamical law of motion, the set of all possible equilibrium states is simply the set of critical points of a smooth function closely related to the system dynamics. When these critical points are nondegenerate, Morse's Theorem applies. But it is exactly when they become degenerate that the system can move sharply from one equilibrium position to another. The Thorn Classification Theorem tells when such shifts will occur and what direction they will take." (John L Casti, "Five Golden Rules", 1995)

On Chaos IV

"One of the central problems studied by mankind is the problem of the succession of form. Whatever is the ultimate nature of reality (assuming that this expression has meaning), it is indisputable that our universe is not chaos. We perceive beings, objects, things to which we give names. These beings or things are forms or structures endowed with a degree of stability; they take up some part of space and last for some period of time." (René Thom, "Structural Stability and Morphogenesis", 1972)

"'Disorder' is not mere chaos; it implies defective order." (John M Ziman, "Models of Disorder", 1979)

"Chaos and catastrophe theories are among the most interesting recent developments in nonlinear modeling, and both have captured the interests of scientists in many disciplines. It is only natural that social scientists should be concerned with these theories. Linear statistical models have proven very useful in a great deal of social scientific empirical analyses, as is evidenced by how widely these models have been used for a number of decades. However, there is no apparent reason, intuitive or otherwise, as to why human behavior should be more linear than the behavior of other things, living and nonliving. Thus an intellectual movement toward nonlinear models is an appropriate evolutionary movement in social scientific thinking, if for no other reason than to expand our paradigmatic boundaries by encouraging greater flexibility in our algebraic specifications of all aspects of human life." (Courtney Brown, "Chaos and Catastrophe Theories", 1995)

"[...] chaos and catastrophe theories per se address behavioral phenomena that are consequences of two general types of nonlinear dynamic behavior. In the most elementary of behavioral terms, chaotic phenomena are a class of deterministic processes that seem to mimic random or stochastic dynamics. Catastrophe phenomena, on the other hand, are a class of dynamic processes that exhibit a sudden and large scale change in at least one variable in correspondence with relatively small changes in other variables or, in some cases, parameters." (Courtney Brown, "Chaos and Catastrophe Theories", 1995)

"Nature normally hates power laws. In ordinary systems all quantities follow bell curves, and correlations decay rapidly, obeying exponential laws. But all that changes if the system is forced to undergo a phase transition. Then power laws emerge-nature's unmistakable sign that chaos is departing in favor of order. The theory of phase transitions told us loud and clear that the road from disorder to order is maintained by the powerful forces of self-organization and is paved by power laws. It told us that power laws are not just another way of characterizing a system's behavior. They are the patent signatures of self-organization in complex systems." (Albert-László Barabási, "Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday Life", 2002)

"Chaos is not pure disorder, it carries within itself the indistinctness between the potentialities of order, of disorder, and of organization from which a cosmos will be born, which is an ordered universe." (Edgar Morin, "Restricted Complexity, General Complexity" [in (Carlos Gershenson et al [Eds.], "Worldviews, Science and Us: Philosophy and Complexity", 2007)])

"Chaos can be understood as a dynamical process in which microscopic information hidden in the details of a system’s state is dug out and expanded to a macroscopically visible scale (stretching), while the macroscopic information visible in the current system’s state is continuously discarded (folding)." (Hiroki Sayama, "Introduction to the Modeling and Analysis of Complex Systems", 2015)

"God has put a secret art into the forces of Nature so as to enable it to fashion itself out of chaos into a perfect world system." (Immanuel Kant)

"Science, like art, music and poetry, tries to reduce chaos to the clarity and order of pure beauty." (Detlev W Bronk)

On Fractals II

"A fractal is a mathematical set or concrete object that is irregular or fragmented at all scales [...]" (Benoît Mandelbrot, "The Fractal Geometry of Nature", 1982)

"A fractal is by definition a set for which the Hausdorff-Besicovitch dimension strictly exceeds the topological dimension." (Benoît Mandelbrot, "The Fractal Geometry of Nature", 1982)

"In the mind's eye, a fractal is a way of seeing infinity." (James Gleick, "Chaos: Making a New Science, A Geometry of Nature", 1987)

"One reason nature pleases us is its endless use of a few simple principles: the cube-square law; fractals; spirals; the way that waves, wheels, trig functions, and harmonic oscillators are alike; the importance of ratios between small primes; bilateral symmetry; Fibonacci series, golden sections, quantization, strange attractors, path-dependency, all the things that show up in places where you don’t expect them [...] these rules work with and against each other ceaselessly at all levels, so that out of their intrinsic simplicity comes the rich complexity of the world around us. That tension - between the simple rules that describe the world and the complex world we see - is itself both simple in execution and immensely complex in effect. Thus exactly the levels, mixtures, and relations of complexity that seem to be hardwired into the pleasure centers of the human brain - or are they, perhaps, intrinsic to intelligence and perception, pleasant to anything that can see, think, create? - are the ones found in the world around us." (John Barnes, "Mother of Storms", 1994)

"We are approaching a more fluid state. I have talked about cultural boiling. The idea of the phase-transition period which, in fractal mathematics, is the chaotic flux between one state and another. [...] Culturally, and as a species, we are approaching a phase-transition. I don’t know quite what that means, on a human level." (Alan Moore, [interview], 1998)

"If financial markets aren't efficient, then what are they? According to the 'fractal market hypothesis', they are highly unstable dynamic systems that generate stock prices which appear random, but behind which lie deterministic patterns." (Steve Keen, "Debunking Economics: The Naked Emperor Of The Social Sciences", 2001)

"Do I claim that everything that is not smooth is fractal? That fractals suffice to solve every problem of science? Not in the least. What I'm asserting very strongly is that, when some real thing is found to be un-smooth, the next mathematical model to try is fractal or multi-fractal. A complicated phenomenon need not be fractal, but finding that a phenomenon is 'not even fractal' is bad news, because so far nobody has invested anywhere near my effort in identifying and creating new techniques valid beyond fractals. Since roughness is everywhere, fractals - although they do not apply to everything - are present everywhere. And very often the same techniques apply in areas that, by every other account except geometric structure, are separate." (Benoît Mandelbrot, "A Theory of Roughness", 2004) 

"Only at the edge of chaos can complex systems flourish. This threshold line, that edge between anarchy and frozen rigidity, is not a like a fence line, it is a fractal line; it possesses nonlinearity." (Stephen H Buhner, "Plant Intelligence and the Imaginal Realm: Beyond the Doors of Perception into the Dreaming of Earth", 2014)

[fractal:] "A fragmented geometric shape that can be split up into secondary pieces, each of which is approximately a smaller replica of the whole, the phenomenon commonly known as self similarity." (Khondekar et al, "Soft Computing Based Statistical Time Series Analysis, Characterization of Chaos Theory, and Theory of Fractals", 2013)

"If you have a hammer, use it everywhere you can, but I do not claim that everything is fractal." (Benoît Mandelbrot)

On Fractals I

"Fractal geometry is not just a chapter of mathematics, but one that helps Everyman to see the same world differently." (Benoît Mandelbrot, "The Fractal Geometry of Nature", 1982)

"I coined fractal from the Latin adjective fractus. The corresponding Latin verb frangere means 'to break': to create irregular fragments [...] how appropriate for our needs!" (Benoît Mandelbrot, "The Fractal Geometry of Nature", 1982)

"The chaos theory will require scientists in all fields to, develop sophisticated mathematical skills, so that they will be able to better recognize the meanings of results. Mathematics has expanded the field of fractals to help describe and explain the shapeless, asymmetrical find randomness of the natural environment." (Theoni Pappas, "More Joy of Mathematics: Exploring mathematical insights & concepts", 1991)

"The term chaos is used in a specific sense where it is an inherently random pattern of behaviour generated by fixed inputs into deterministic (that is fixed) rules (relationships). The rules take the form of non-linear feedback loops. Although the specific path followed by the behaviour so generated is random and hence unpredictable in the long-term, it always has an underlying pattern to it, a 'hidden' pattern, a global pattern or rhythm. That pattern is self-similarity, that is a constant degree of variation, consistent variability, regular irregularity, or more precisely, a constant fractal dimension. Chaos is therefore order (a pattern) within disorder (random behaviour)." (Ralph D Stacey, "The Chaos Frontier: Creative Strategic Control for Business", 1991)

"I believe that scientific knowledge has fractal properties; that no matter how much we learn, whatever is left, however small it may seem, is just as infinitely complex as the whole was to start with. That, I think, is the secret of the Universe." (Isaac Asimov, "A Way of Thinking", The Magazine of Fantasy and Science Fiction, 1994)

"It is time to employ fractal geometry and its associated subjects of chaos and nonlinear dynamics to study systems engineering methodology (SEM). [...] Fractal geometry and chaos theory can convey a new level of understanding to systems engineering and make it more effective." (Arthur D Hall, "The fractal architecture of the systems engineering method", "Systems, Man and Cybernetics", Vol. 28 (4), 1998)

"The self-similarity of fractal structures implies that there is some redundancy because of the repetition of details at all scales. Even though some of these structures may appear to teeter on the edge of randomness, they actually represent complex systems at the interface of order and disorder."  (Edward Beltrami, "What is Random?: Chaos and Order in Mathematics and Life", 1999)

"In plain English, fractal geometry is the geometry of the irregular, the geometry of nature, and, in general, fractals are characterized by infinite detail, infinite length, and the absence of smoothness or derivative." (Philip Tetlow, "The Web’s Awake: An Introduction to the Field of Web Science and the Concept of Web Life", 2007)

"The economy is a nonlinear fractal system, where the smallest scales are linked to the largest, and the decisions of the central bank are affected by the gut instincts of the people on the street." (David Orrell, "The Other Side Of The Coin", 2008)

"Geometric pattern repeated at progressively smaller scales, where each iteration is about a reproduction of the image to produce completely irregular shapes and surfaces that can not be represented by classical geometry. Fractals are generally self-similar (each section looks at all) and are not subordinated to a specific scale. They are used especially in the digital modeling of irregular patterns and structures in nature." (Mauro Chiarella, "Folds and Refolds: Space Generation, Shapes, and Complex Components", 2016)

On Differential Equations III

"The integrals which we have obtained are not only general expressions which satisfy the differential equation, they represent in the most distinct manner the natural effect which is the object of the phenomenon [...] when this condition is fulfilled, the integral is, properly speaking, the equation of the phenomenon; it expresses clearly the character and progress of it, in the same manner as the finite equation of a line or curved surface makes known all the properties of those forms." (Jean-Baptiste-Joseph Fourier, "Théorie Analytique de la Chaleur", 1822)

"Most surprising and far-reaching analogies revealed themselves between apparently quite disparate natural processes. It seemed that nature had built the most various things on exactly the same pattern; or, in the dry words of the analyst, the same differential equations hold for the most various phenomena. (Ludwig Boltzmann, "On the methods of theoretical physics", 1892)

"Part of the charm in solving a differential equation is in the feeling that we are getting something for nothing. So little information appears to go into the solution that there is a sense of surprise over the extensive results that are derived." (George R Stibitz & Jules A Larrivee, "Mathematics and Computers", 1957)

"Finite systems of deterministic ordinary nonlinear differential equations may be designed to represent forced dissipative hydrodynamic flow. Solutions of these equations can be identified with trajectories in phase space. For those systems with bounded solutions, it is found that nonperiodic solutions are ordinarily unstable with respect to small modifications, so that slightly differing initial states can evolve into considerably different states. Systems with bounded solutions are shown to possess bounded numerical solutions. (Edward N Lorenz, "Deterministic Nonperiodic Flow", Journal of the Atmospheric Science 20, 1963)

"A system may be specified in either of two ways. In the first, which we shall call a state description, sets of abstract inputs, outputs and states are given, together with the action of the inputs on the states and the assignments of outputs to states. In the second, which we shall call a coordinate description, certain input, output and state variables are given, together with a system of dynamical equations describing the relations among the variables as functions of time. Modern mathematical system theory is formulated in terms of state descriptions, whereas the classical formulation is typically a coordinate description, for example a system of differential equations." (E S Bainbridge, "The Fundamental Duality of System Theory", 1975)

"General systems theory deals with the most fundamental concepts and aspects of systems. Many theories dealing with more specific types of systems (e. g., dynamical systems, automata, control systems, game-theoretic systems, among many others) have been under development for quite some time. General systems theory is concerned with the basic issues common to all these specialized treatments. Also, for truly complex phenomena, such as those found predominantly in the social and biological sciences, the specialized descriptions used in classical theories (which are based on special mathematical structures such as differential or difference equations, numerical or abstract algebras, etc.) do not adequately and properly represent the actual events. Either because of this inadequate match between the events and types of descriptions available or because of the pure lack of knowledge, for many truly complex problems one can give only the most general statements, which are qualitative and too often even only verbal. General systems theory is aimed at providing a description and explanation for such complex phenomena." (Mihajlo D. Mesarovic & Yasuhiko Takahare, "General Systems Theory: Mathematical foundations", 1975)

"The successes of the differential equation paradigm were impressive and extensive. Many problems, including basic and important ones, led to equations that could be solved. A process of self-selection set in, whereby equations that could not be solved were automatically of less interest than those that could." (Ian Stewart, "Does God Play Dice? The Mathematics of Chaos", 1989)

"The results of mathematics are seldom directly applied; it is the definitions that are really useful. Once you learn the concept of a differential equation, you see differential equations all over, no matter what you do. This you cannot see unless you take a course in abstract differential equations. What applies is the cultural background you get from a course in differential equations, not the specific theorems. If you want to learn French, you have to live the life of France, not just memorize thousands of words. If you want to apply mathematics, you have to live the life of differential equations. When you live this life, you can then go back to molecular biology with a new set of eyes that will see things you could not otherwise see." (Gian-Carlo Rota, "Indiscrete Thoughts", 1997)

"Complex systems defy intuitive solutions. Even a third-order, linear differential equation is unsolvable by inspection. Yet, important situations in management, economics, medicine, and social behavior usually lose reality if simplified to less than fifth-order nonlinear dynamic systems. Attempts to deal with nonlinear dynamic systems using ordinary processes of description and debate lead to internal inconsistencies. Underlying assumptions may have been left unclear and contradictory, and mental models are often logically incomplete. Resulting behavior is likely to be contrary to that implied by the assumptions being made about' underlying system structure and governing policies." (Jay W Forrester, "Modeling for What Purpose?", The Systems Thinker Vol. 24 (2), 2013)

 "Among all of the mathematical disciplines the theory of differential  equations is the most important […]. It furnishes the explanation of all those elementary manifestations of nature which involve time." (Sophus Lie)

Previous Post <<||>> Next Post

Thermodynamics III

"My analysis of living systems uses concepts of thermodynamics, information theory, cybernetics, and systems engineering, as well as the classical concepts appropriate to each level. The purpose is to produce a description of living structure and process in terms of input and output, flows through systems, steady states, and feedbacks, which will clarify and unify the facts of life." (James G Miller, "Living Systems: Basic Concepts", 1969)

"In an isolated system, which cannot exchange energy and matter with the surroundings, this tendency is expressed in terms of a function of the macroscopic state of the system: the entropy." (Ilya Prigogine, "Thermodynamics of Evolution", 1972)

"[The] system may evolve through a whole succession of transitions leading to a hierarchy of more and more complex and organized states. Such transitions can arise in nonlinear systems that are maintained far from equilibrium: that is, beyond a certain critical threshold the steady-state regime become unstable and the system evolves into a new configuration." (Ilya Prigogine, Gregoire Micolis & Agnes Babloyantz, "Thermodynamics of Evolution", Physics Today 25 (11), 1972)

"The evolution of a physicochemical system leads to an equilibrium state of maximum disorder." (Ilya Prigogine, "Thermodynamics of Evolution", 1972)

"The functional order maintained within living systems seems to defy the Second Law; nonequilibrium thermodynamics describes how such systems come to terms with entropy." (Ilya Prigogine, "Thermodynamics of Evolution", 1972)

"When matter is becoming disturbed by non-equilibrium conditions it organizes itself, it wakes up. It happens that our world is a non-equilibrium system." (Ilya Prigogine, "Thermodynamics of Evolution", 1972)

"There is nothing supernatural about the process of self-organization to states of higher entropy; it is a general property of systems, regardless of their materials and origin. It does not violate the Second Law of thermodynamics since the decrease in entropy within an open system is always offset by the increase of entropy in its surroundings." (Ervin László, "Introduction to Systems Philosophy", 1972)

"Concepts form the basis for any science. These are ideas, usually somewhat vague (especially when first encountered), which often defy really adequate definition. The meaning of a new concept can seldom be grasped from reading a one-paragraph discussion. There must be time to become accustomed to the concept, to investigate it with prior knowledge, and to associate it with personal experience. Inability to work with details of a new subject can often be traced to inadequate understanding of its basic concepts." (William C Reynolds & Harry C Perkins, "Engineering Thermodynamics", 1977)

"Just like a computer, we must remember things in the order in which entropy increases. This makes the second law of thermodynamics almost trivial. Disorder increases with time because we measure time in the direction in which disorder increases."  (Stephen Hawking, "A Brief History of Time", 1988)

"Life is nature's solution to the problem of preserving information despite the second law of thermodynamics." (Howard L Resnikoff, "The Illusion of Reality", 1989)

Complex Systems VI

"In short, complex adaptive systems are characterized by perpetual novelty." (M Mitchell Waldrop, "Complexity: The Emerging Science at the Edge of Order and Chaos", 1992)

"[...] it's essentially meaningless to talk about a complex adaptive system being in equilibrium: the system can never get there. It is always unfolding, always in transition. In fact, if the system ever does reach equilibrium, it isn't just stable. It's dead." (M Mitchell Waldrop, "Complexity: The Emerging Science at the Edge of Order and Chaos", 1992)

"If universality is one of the observed characteristics of complex dynamical systems in many fields of study, a second characteristic that flows from the study of these systems is that of emergence. As self-organizing systems go about their daily business, they are constantly exchanging matter and energy with their environment, and this allows them to remain in a state that is far from equilibrium. That allows spontaneous behavior to give rise to new patterns." (Terry Cooke-Davies et al, "Exploring the Complexity of Projects", 2009)

"The difference between complex adaptive systems and self-organizing systems is that the former have the capacity to learn from their experience, and thus to embody successful patterns into their repertoire, although there is actually quite a deep relationship between self-organizing systems and complex adaptive systems. Adaptive entities can emerge at high levels of description in simple self-organizing systems, i.e., adaptive systems are not necessarily self-organizing systems with something extra thrown in." (Terry Cooke-Davies et al, "Exploring the Complexity of Projects", 2009)

"Most systems in nature are inherently nonlinear and can only be described by nonlinear equations, which are difficult to solve in a closed form. Non-linear systems give rise to interesting phenomena such as chaos, complexity, emergence and self-organization. One of the characteristics of non-linear systems is that a small change in the initial conditions can give rise to complex and significant changes throughout the system. This property of a non-linear system such as the weather is known as the butterfly effect where it is purported that a butterfly flapping its wings in Japan can give rise to a tornado in Kansas. This unpredictable behaviour of nonlinear dynamical systems, i.e. its extreme sensitivity to initial conditions, seems to be random and is therefore referred to as chaos. This chaotic and seemingly random behaviour occurs for non-linear deterministic system in which effects can be linked to causes but cannot be predicted ahead of time." (Robert K Logan, "The Poetry of Physics and The Physics of Poetry", 2010)

"Complex systems seem to have this property, with large periods of apparent stasis marked by sudden and catastrophic failures. These processes may not literally be random, but they are so irreducibly complex (right down to the last grain of sand) that it just won’t be possible to predict them beyond a certain level. […] And yet complex processes produce order and beauty when you zoom out and look at them from enough distance." (Nate Silver, "The Signal and the Noise: Why So Many Predictions Fail-but Some Don't", 2012)

"We forget - or we willfully ignore - that our models are simplifications of the world. We figure that if we make a mistake, it will be at the margin. In complex systems, however, mistakes are not measured in degrees but in whole orders of magnitude." (Nate Silver, "The Signal and the Noise: Why So Many Predictions Fail-but Some Don't", 2012)

"If an emerging system is born complex, there is neither leeway to abandon it when it fails, nor the means to join another, successful one. Such a system would be caught in an immovable grip, congested at the top, and prevented, by a set of confusing but locked–in precepts, from changing." (Lawrence K Samuels, "Defense of Chaos: The Chaology of Politics, Economics and Human Action", 2013) 

"Simplicity in a system tends to increase that system’s efficiency. Because less can go wrong with fewer parts, less will. Complexity in a system tends to increase that system’s inefficiency; the greater the number of variables, the greater the probability of those variables clashing, and in turn, the greater the potential for conflict and disarray. Because more can go wrong, more will. That is why centralized systems are inclined to break down quickly and become enmeshed in greater unintended consequences." (Lawrence K Samuels,"Defense of Chaos: The Chaology of Politics, Economics and Human Action", 2013)

"One of the remarkable features of these complex systems created by replicator dynamics is that infinitesimal differences in starting positions create vastly different patterns. This sensitive dependence on initial conditions is often called the butterfly-effect aspect of complex systems - small changes in the replicator dynamics or in the starting point can lead to enormous differences in outcome, and they change one’s view of how robust the current reality is. If it is complex, one small change could have led to a reality that is quite different." (David Colander & Roland Kupers, "Complexity and the art of public policy : solving society’s problems from the bottom up", 2014)

Complex Systems IV

"With the growing interest in complex adaptive systems, artificial life, swarms and simulated societies, the concept of 'collective intelligence' is coming more and more to the fore. The basic idea is that a group of individuals (e. g. people, insects, robots, or software agents) can be smart in a way that none of its members is. Complex, apparently intelligent behavior may emerge from the synergy created by simple interactions between individuals that follow simple rules." (Francis Heylighen, "Collective Intelligence and its Implementation on the Web", 1999)

"A system may be called complex here if its dimension (order) is too high and its model (if available) is nonlinear, interconnected, and information on the system is uncertain such that classical techniques can not easily handle the problem." (M Jamshidi, "Autonomous Control on Complex Systems: Robotic Applications", Current Advances in Mechanical Design and Production VII, 2000)

"Bounded rationality simultaneously constrains the complexity of our cognitive maps and our ability to use them to anticipate the system dynamics. Mental models in which the world is seen as a sequence of events and in which feedback, nonlinearity, time delays, and multiple consequences are lacking lead to poor performance when these elements of dynamic complexity are present. Dysfunction in complex systems can arise from the misperception of the feedback structure of the environment. But rich mental models that capture these sources of complexity cannot be used reliably to understand the dynamics. Dysfunction in complex systems can arise from faulty mental simulation-the misperception of feedback dynamics. These two different bounds on rationality must both be overcome for effective learning to occur. Perfect mental models without a simulation capability yield little insight; a calculus for reliable inferences about dynamics yields systematically erroneous results when applied to simplistic models." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"Much of the art of system dynamics modeling is discovering and representing the feedback processes, which, along with stock and flow structures, time delays, and nonlinearities, determine the dynamics of a system. […] the most complex behaviors usually arise from the interactions (feedbacks) among the components of the system, not from the complexity of the components themselves." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"To avoid policy resistance and find high leverage policies requires us to expand the boundaries of our mental models so that we become aware of and understand the implications of the feedbacks created by the decisions we make. That is, we must learn about the structure and dynamics of the increasingly complex systems in which we are embedded." (John D Sterman, "Business dynamics: Systems thinking and modeling for a complex world", 2000) 

"Falling between order and chaos, the moment of complexity is the point at which self-organizing systems emerge to create new patterns of coherence and structures of behaviour." (Mark C Taylor, "The Moment of Complexity: Emerging Network Culture", 2001)

"[…] most earlier attempts to construct a theory of complexity have overlooked the deep link between it and networks. In most systems, complexity starts where networks turn nontrivial." (Albert-László Barabási, "Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday Life", 2002)

"[…] networks are the prerequisite for describing any complex system, indicating that complexity theory must inevitably stand on the shoulders of network theory. It is tempting to step in the footsteps of some of my predecessors and predict whether and when we will tame complexity. If nothing else, such a prediction could serve as a benchmark to be disproven. Looking back at the speed with which we disentangled the networks around us after the discovery of scale-free networks, one thing is sure: Once we stumble across the right vision of complexity, it will take little to bring it to fruition. When that will happen is one of the mysteries that keeps many of us going." (Albert-László Barabási, "Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday Life", 2002)

"A sudden change in the evolutive dynamics of a system (a ‘surprise’) can emerge, apparently violating a symmetrical law that was formulated by making a reduction on some (or many) finite sequences of numerical data. This is the crucial point. As we have said on a number of occasions, complexity emerges as a breakdown of symmetry (a system that, by evolving with continuity, suddenly passes from one attractor to another) in laws which, expressed in mathematical form, are symmetrical. Nonetheless, this breakdown happens. It is the surprise, the paradox, a sort of butterfly effect that can highlight small differences between numbers that are very close to one another in the continuum of real numbers; differences that may evade the experimental interpretation of data, but that may increasingly amplify in the system’s dynamics." (Cristoforo S Bertuglia & Franco Vaio, "Nonlinearity, Chaos, and Complexity: The Dynamics of Natural and Social Systems", 2003) 

Stephen Wolfram - Collected Quotes

"Cellular automata are discrete dynamical systems with simple construction but complex self-organizing behaviour. Evidence is presented that all one-dimensional cellular automata fall into four distinct universality classes. Characterizations of the structures generated in these classes are discussed. Three classes exhibit behaviour analogous to limit points, limit cycles and chaotic attractors. The fourth class is probably capable of universal computation, so that properties of its infinite time behaviour are undecidable." (Stephen Wolfram, "Nonlinear Phenomena, Universality and complexity in cellular automata", Physica 10D, 1984)

"Cellular automata are mathematical models for complex natural systems containing large numbers of simple identical components with local interactions. They consist of a lattice of sites, each with a finite set of possible values. The value of the sites evolve synchronously in discrete time steps according to identical rules. The value of a particular site is determined by the previous values of a neighbourhood of sites around it." (Stephen Wolfram, "Nonlinear Phenomena, Universality and complexity in cellular automata", Physica 10D, 1984)

"Cellular automata may be considered as discrete dynamical systems. In almost all cases, cellular automaton evolution is irreversible. Trajectories in the configuration space for cellular automata therefore merge with time, and after many time steps, trajectories starting from almost all initial states become concentrated onto 'attractors'. These attractors typically contain only a very small fraction of possible states. Evolution to attractors from arbitrary initial states allows for 'self-organizing' behaviour, in which structure may evolve at large times from structureless initial states. The nature of the attractors determines the form and extent of such structures." (Stephen Wolfram, "Nonlinear Phenomena, Universality and complexity in cellular automata", Physica D (10), 1984)

"Computation offers a new means of describing and investigating scientific and mathematical systems. Simulation by computer may be the only way to predict how certain complicated systems evolve." (Stephen Wolfram, "Computer Software in Science and Mathematics", 1984)

"Scientific laws give algorithms, or procedures, for determining how systems behave. The computer program is a medium in which the algorithms can be expressed and applied. Physical objects and mathematical structures can be represented as numbers and symbols in a computer, and a program can be written to manipulate them according to the algorithms. When the computer program is executed, it causes the numbers and symbols to be modified in the way specified by the scientific laws. It thereby allows the consequences of the laws to be deduced." (Stephen Wolfram, "Computer Software in Science and Mathematics", 1984)

"Scientific laws have conventionally been constructed in terms of a particular set of mathematical functions and constructs, and they have often been developed as much for their mathematical simplicity as for their capacity to model the salient features of a phenomenon." (Stephen Wolfram, "Computer Software in Science and Mathematics", 1984)

"Computational reducibility may well be the exception rather than the rule: Most physical questions may be answerable only through irreducible amounts of computation. Those that concern idealized limits of infinite time, volume, or numerical precision can require arbitrarily long computations, and so be formally undecidable." (Stephen Wolfram, Undecidability and intractability in theoretical physics", Physical Review Letters 54 (8), 1985)

"In the existing sciences much of the emphasis over the past century or so has been on breaking systems down to find their underlying parts, then trying to analyze these parts in as much detail as possible. [...] But just how these components act together to produce even some of the most obvious features of the overall behavior we see has in the past remained an almost complete mystery." (Stephen Wolfram, "A New Kind of Science", 2002)

"Over and over again we will see the same kind of thing: that even though the underlying rules for a system are simple, and even though the system is started from simple initial conditions, the behavior that the system shows can nevertheless be highly complex." (Stephen Wolfram, "A New Kind of Science", 2002)

"The idea of describing behavior in terms of mathematical equations works well where the behavior is fairly simple. It almost inevitably fails whenever the behavior is more complex. Indeed, there are many common phenomena about which theoretical science has had remarkably very little to say. Degree of difficulty encountered in mathematical representation of a phenomenon increases exponentially by the degree of its complexity." (Stephen Wolfram, "New Kind of Science", 2002)

"Our best way of summarizing and communicating knowledge tends to be through language. And when mathematics became formalized, it did so essentially by emulating the symbolic structure of traditional human natural language." (Stephen Wolfram, Some Modern Perspectives on the Quest for Ultimate Knowledge, 2007)

"Could it be that some place out there in the computational universe, we might find our physical universe?" (Stephen Wolfram, "Computing a Theory of Everything", 2010)

On Nonlinearity IV (Organizations)

"In strategic thinking, one first seeks a clear understanding of the particular character of each element of a situation and then makes the fullest possible use of human brainpower to restructure the elements in the most advantageous way. Phenomena and events in the real word do not always fit a linear model. Hence the most reliable means of dissecting a situation into its constituent parts and reassembling then in the desired pattern is not a step-by-step methodology such as systems analysis. Rather, it is that ultimate nonlinear thinking tool, the human brain. True strategic thinking thus contrasts sharply with the conventional mechanical systems approach based on linear thinking. But it also contrasts with the approach that stakes everything on intuition, reaching conclusions without any real breakdown or analysis. [...] No matter how difficult or unprecedented the problem, a breakthrough to the best possible solution can come only from a combination of rational analysis, based on the real nature of things, and imaginative reintegration of all the different items into a new pattern, using nonlinear brainpower. This is always the most effective approach to devising strategies for dealing successfully with challenges and opportunities, in the market arena as on the battlefield." (Kenichi Ohmae, "The Mind Of The Strategist", 1982)

"So we pour in data from the past to fuel the decision-making mechanisms created by our models, be they linear or nonlinear. But therein lies the logician's trap: past data from real life constitute a sequence of events rather than a set of independent observations, which is what the laws of probability demand.[...] It is in those outliers and imperfections that the wildness lurks." (Peter L Bernstein, "Against the Gods: The Remarkable Story of Risk", 1996)

"In a linear world of equilibrium and predictability, the sparse research into an evidence base for management prescriptions and the confused findings it produces would be a sign of incompetence; it would not make much sense. Nevertheless, if organizations are actually patterns of nonlinear interaction between people; if small changes could produce widespread major consequences; if local interaction produces emergent global pattern; then it will not be possible to provide a reliable evidence base. In such a world, it makes no sense to conduct studies looking for simple causal relationships between an action and an outcome. I suggest that the story of the last few years strongly indicates that human action is nonlinear, that time and place matter a great deal, and that since this precludes simple evidence bases we do need to rethink the nature of organizations and the roles of managers and leaders in them." (Ralph D Stacey, "Complexity and Organizational Reality", 2000)

"The world is nonlinear. Trying to make it linear for our mathematical or administrative convenience is not usually a good idea even when feasible, and it is rarely feasible." (Donella H Meadow, "Thinking in Systems: A Primer", 2008)

"Complexity theory shows that great changes can emerge from small actions. Change involves a belief in the possible, even the 'impossible'. Moreover, social innovators don’t follow a linear pathway of change; there are ups and downs, roller-coaster rides along cascades of dynamic interactions, unexpected and unanticipated divergences, tipping points and critical mass momentum shifts. Indeed, things often get worse before they get better as systems change creates resistance to and pushback against the new. Traditional evaluation approaches are not well suited for such turbulence. Traditional evaluation aims to control and predict, to bring order to chaos. Developmental evaluation accepts such turbulence as the way the world of social innovation unfolds in the face of complexity. Developmental evaluation adapts to the realities of complex nonlinear dynamics rather than trying to impose order and certainty on a disorderly and uncertain world." (Michael Q Patton, "Developmental Evaluation", 2010)

"Internal friction is exacerbated by the fact that in business as in war, we are operating in a nonlinear, semi-chaotic environment in which our endeavors will collide and possibly clash with the actions of other independent wills (customers, suppliers, competitors, regulators, lobbyists, and so on). The internal and external worlds are in constant contact and the effects of our actions are the result of their reciprocal interaction. Friction gives rise to three gaps: the knowledge gap, the alignment gap, and the effects gap. To execute effectively, we must address all three. Our instinctive reaction to the three gaps is to demand more detail. We gather more data in order to craft more detailed plans, issue more detailed instructions, and exercise more detailed control. This not only fails to solve the problem, it usually makes it worse. We need to think about the problem differently and adopt a systemic approach to solving it." (Stephen Bungay, "The Art of Action: How Leaders Close the Gaps between Plans, Actions, and Results", 2010)

"Motivation is a fine example of social complexity. It is nonlinear and sometimes unpredictable. It cannot be defined or modeled with a single diagram." (Jurgen Appelo, "Management 3.0: Leading Agile Developers, Developing Agile Leaders", 2010)

"We have minds that are equipped for certainty, linearity and short-term decisions, that must instead make long-term decisions in a non-linear, probabilistic world. (Paul Gibbons, "The Science of Successful Organizational Change", 2015)

On Nonlinearity V (Chaos I)

"When one combines the new insights gained from studying far-from-equilibrium states and nonlinear processes, along with these complicated feedback systems, a whole new approach is opened that makes it possible to relate the so-called hard sciences to the softer sciences of life - and perhaps even to social processes as well. […] It is these panoramic vistas that are opened to us by Order Out of Chaos." (Ilya Prigogine, "Order Out of Chaos: Man's New Dialogue with Nature", 1984)

"Algorithmic complexity theory and nonlinear dynamics together establish the fact that determinism reigns only over a quite finite domain; outside this small haven of order lies a largely uncharted, vast wasteland of chaos." (Joseph Ford, "Progress in Chaotic Dynamics: Essays in Honor of Joseph Ford's 60th Birthday", 1988)

"The term chaos is used in a specific sense where it is an inherently random pattern of behaviour generated by fixed inputs into deterministic (that is fixed) rules (relationships). The rules take the form of non-linear feedback loops. Although the specific path followed by the behaviour so generated is random and hence unpredictable in the long-term, it always has an underlying pattern to it, a 'hidden' pattern, a global pattern or rhythm. That pattern is self-similarity, that is a constant degree of variation, consistent variability, regular irregularity, or more precisely, a constant fractal dimension. Chaos is therefore order (a pattern) within disorder (random behaviour)." (Ralph D Stacey, "The Chaos Frontier: Creative Strategic Control for Business", 1991)

"In the everyday world of human affairs, no one is surprised to learn that a tiny event over here can have an enormous effect over there. For want of a nail, the shoe was lost, et cetera. But when the physicists started paying serious attention to nonlinear systems in their own domain, they began to realize just how profound a principle this really was. […] Tiny perturbations won't always remain tiny. Under the right circumstances, the slightest uncertainty can grow until the system's future becomes utterly unpredictable - or, in a word, chaotic." (M Mitchell Waldrop, "Complexity: The Emerging Science at the Edge of Order and Chaos", 1992)

"There is a new science of complexity which says that the link between cause and effect is increasingly difficult to trace; that change (planned or otherwise) unfolds in non-linear ways; that paradoxes and contradictions abound; and that creative solutions arise out of diversity, uncertainty and chaos." (Andy P Hargreaves & Michael Fullan, "What’s Worth Fighting for Out There?", 1998)

"Let's face it, the universe is messy. It is nonlinear, turbulent, and chaotic. It is dynamic. It spends its time in transient behavior on its way to somewhere else, not in mathematically neat equilibria. It self-organizes and evolves. It creates diversity, not uniformity. That's what makes the world interesting, that's what makes it beautiful, and that's what makes it work." (Donella H Meadow, "Thinking in Systems: A Primer", 2008)

"Complexity theory can be defined broadly as the study of how order, structure, pattern, and novelty arise from extremely complicated, apparently chaotic systems and conversely, how complex behavior and structure emerges from simple underlying rules. As such, it includes those other areas of study that are collectively known as chaos theory, and nonlinear dynamical theory." (Terry Cooke-Davies et al, "Exploring the Complexity of Projects", 2009)

"Most systems in nature are inherently nonlinear and can only be described by nonlinear equations, which are difficult to solve in a closed form. Non-linear systems give rise to interesting phenomena such as chaos, complexity, emergence and self-organization. One of the characteristics of non-linear systems is that a small change in the initial conditions can give rise to complex and significant changes throughout the system. This property of a non-linear system such as the weather is known as the butterfly effect where it is purported that a butterfly flapping its wings in Japan can give rise to a tornado in Kansas. This unpredictable behaviour of nonlinear dynamical systems, i.e. its extreme sensitivity to initial conditions, seems to be random and is therefore referred to as chaos. This chaotic and seemingly random behaviour occurs for non-linear deterministic system in which effects can be linked to causes but cannot be predicted ahead of time." (Robert K Logan, "The Poetry of Physics and The Physics of Poetry", 2010)

"Even more important is the way complex systems seem to strike a balance between the need for order and the imperative for change. Complex systems tend to locate themselves at a place we call 'the edge of chaos'. We imagine the edge of chaos as a place where there is enough innovation to keep a living system vibrant, and enough stability to keep it from collapsing into anarchy. It is a zone of conflict and upheaval, where the old and new are constantly at war. Finding the balance point must be a delicate matter - if a living system drifts too close, it risks falling over into incoherence and dissolution; but if the system moves too far away from the edge, it becomes rigid, frozen, totalitarian. Both conditions lead to extinction. […] Only at the edge of chaos can complex systems flourish. This threshold line, that edge between anarchy and frozen rigidity, is not a like a fence line, it is a fractal line; it possesses nonlinearity." (Stephen H Buhner, "Plant Intelligence and the Imaginal Realm: Beyond the Doors of Perception into the Dreaming of Earth", 2014)

"To remedy chaotic situations requires a chaotic approach, one that is non-linear, constantly morphing, and continually sharpening its competitive edge with recurring feedback loops that build upon past experiences and lessons learned. Improvement cannot be sustained without reflection. Chaos arises from myriad sources that stem from two origins: internal chaos rising within you, and external chaos being imposed upon you by the environment. The result of this push/pull effect is the disequilibrium [...]." (Jeff Boss, "Navigating Chaos: How to Find Certainty in Uncertain Situations", 2015)

On Nonlinearity III

"Finite systems of deterministic ordinary nonlinear differential equations may be designed to represent forced dissipative hydrodynamic flow. Solutions of these equations can be identified with trajectories in phase space. For those systems with bounded solutions, it is found that nonperiodic solutions are ordinarily unstable with respect to small modifications, so that slightly differing initial states can evolve into considerably different states. Systems with bounded solutions are shown to possess bounded numerical solutions. (Edward N Lorenz, "Deterministic Nonperiodic Flow", Journal of the Atmospheric Science 20, 1963)

"We've seen that even in the simplest situations nonlinearities can interfere with a linear approach to aggregates. That point holds in general: nonlinear interactions almost always make the behavior of the aggregate more complicated than would be predicted by summing or averaging." (Lewis Mumford, "The Myth of the Machine" Vol 1, 1967)

"The structure of a complex system is not a simple feedback loop where one system state dominates the behavior. The complex system has a multiplicity of interacting feedback loops. Its internal rates of flow are controlled by non‐linear relationships. The complex system is of high order, meaning that there are many system states (or levels). It usually contains positive‐feedback loops describing growth processes as well as negative, goal‐seeking loops." (Jay F Forrester, "Urban Dynamics", 1969)

"I would therefore urge that people be introduced to [the logistic equation] early in their mathematical education. This equation can be studied phenomenologically by iterating it on a calculator, or even by hand. Its study does not involve as much conceptual sophistication as does elementary calculus. Such study would greatly enrich the student’s intuition about nonlinear systems. Not only in research but also in the everyday world of politics and economics, we would all be better off if more people realized that simple nonlinear systems do not necessarily possess simple dynamical properties." (Robert M May, "Simple Mathematical Models with Very Complicated Dynamics", Nature Vol. 261 (5560), 1976)

"Most physical systems, particularly those complex ones, are extremely difficult to model by an accurate and precise mathematical formula or equation due to the complexity of the system structure, nonlinearity, uncertainty, randomness, etc. Therefore, approximate modeling is often necessary and practical in real-world applications. Intuitively, approximate modeling is always possible. However, the key questions are what kind of approximation is good, where the sense of 'goodness' has to be first defined, of course, and how to formulate such a good approximation in modeling a system such that it is mathematically rigorous and can produce satisfactory results in both theory and applications." (Guanrong Chen & Trung Tat Pham, "Introduction to Fuzzy Sets, Fuzzy Logic, and Fuzzy Control Systems", 2001) 

"Thus, nonlinearity can be understood as the effect of a causal loop, where effects or outputs are fed back into the causes or inputs of the process. Complex systems are characterized by networks of such causal loops. In a complex, the interdependencies are such that a component A will affect a component B, but B will in general also affect A, directly or indirectly.  A single feedback loop can be positive or negative. A positive feedback will amplify any variation in A, making it grow exponentially. The result is that the tiniest, microscopic difference between initial states can grow into macroscopically observable distinctions." (Carlos Gershenson, "Design and Control of Self-organizing Systems", 2007)

"Where simplifications fail, causing the most damage, is when something nonlinear is simplified with the linear as a substitute. That is the most common Procrustean bed." (Nassim N Taleb, "Antifragile: Things that Gain from Disorder", 2012)

"Complex systems defy intuitive solutions. Even a third-order, linear differential equation is unsolvable by inspection. Yet, important situations in management, economics, medicine, and social behavior usually lose reality if simplified to less than fifth-order nonlinear dynamic systems. Attempts to deal with nonlinear dynamic systems using ordinary processes of description and debate lead to internal inconsistencies. Underlying assumptions may have been left unclear and contradictory, and mental models are often logically incomplete. Resulting behavior is likely to be contrary to that implied by the assumptions being made about' underlying system structure and governing policies." (Jay W Forrester, "Modeling for What Purpose?", The Systems Thinker Vol. 24 (2), 2013)

"There is no linear additive process that, if all the parts are taken together, can be understood to create the total system that occurs at the moment of self-organization; it is not a quantity that comes into being. It is not predictable in its shape or subsequent behavior or its subsequent qualities. There is a nonlinear quality that comes into being at the moment of synchronicity." (Stephen H Buhner, "Plant Intelligence and the Imaginal Realm: Beyond the Doors of Perception into the Dreaming of Earth", 2014)

"Exponentially growing systems are prevalent in nature, spanning all scales from biochemical reaction networks in single cells to food webs of ecosystems. How exponential growth emerges in nonlinear systems is mathematically unclear. […] The emergence of exponential growth from a multivariable nonlinear network is not mathematically intuitive. This indicates that the network structure and the flux functions of the modeled system must be subjected to constraints to result in long-term exponential dynamics." (Wei-Hsiang Lin et al, "Origin of exponential growth in nonlinear reaction networks", PNAS 117 (45), 2020)

On Nonlinearity II

"Indeed, except for the very simplest physical systems, virtually everything and everybody in the world is caught up in a vast, nonlinear web of incentives and constraints and connections. The slightest change in one place causes tremors everywhere else. We can't help but disturb the universe, as T.S. Eliot almost said. The whole is almost always equal to a good deal more than the sum of its parts. And the mathematical expression of that property - to the extent that such systems can be described by mathematics at all - is a nonlinear equation: one whose graph is curvy." (M Mitchell Waldrop, "Complexity: The Emerging Science at the Edge of Order and Chaos", 1992)

"Today the network of relationships linking the human race to itself and to the rest of the biosphere is so complex that all aspects affect all others to an extraordinary degree. Someone should be studying the whole system, however crudely that has to be done, because no gluing together of partial studies of a complex nonlinear system can give a good idea of the behaviour of the whole." (Murray Gell-Mann, 1997)

"Much of the art of system dynamics modeling is discovering and representing the feedback processes, which, along with stock and flow structures, time delays, and nonlinearities, determine the dynamics of a system. […] the most complex behaviors usually arise from the interactions (feedbacks) among the components of the system, not from the complexity of the components themselves." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"The mental models people use to guide their decisions are dynamically deficient. […] people generally adopt an event-based, open-loop view of causality, ignore feedback processes, fail to appreciate time delays between action and response and in the reporting of information, do not understand stocks and flows and are insensitive to nonlinearities that may alter the strengths of different feedback loops as a system evolves." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"Most physical processes in the real world are nonlinear. It is our abstraction of the real world that leads us to the use of linear systems in modeling these processes. These linear systems are simple, understandable, and, in many situations, provide acceptable simulations of the actual processes. Unfortunately, only the simplest of linear processes and only a very small fraction of the nonlinear having verifiable solutions can be modeled with linear systems theory. The bulk of the physical processes that we must address are, unfortunately, too complex to reduce to algorithmic form - linear or nonlinear. Most observable processes have only a small amount of information available with which to develop an algorithmic understanding. The vast majority of information that we have on most processes tends to be nonnumeric and nonalgorithmic. Most of the information is fuzzy and linguistic in form." (Timothy J Ross & W Jerry Parkinson, "Fuzzy Set Theory, Fuzzy Logic, and Fuzzy Systems", 2002)

"Swarm intelligence can be effective when applied to highly complicated problems with many nonlinear factors, although it is often less effective than the genetic algorithm approach [...]. Swarm intelligence is related to swarm optimization […]. As with swarm intelligence, there is some evidence that at least some of the time swarm optimization can produce solutions that are more robust than genetic algorithms. Robustness here is defined as a solution’s resistance to performance degradation when the underlying variables are changed. (Michael J North & Charles M Macal, Managing Business Complexity: Discovering Strategic Solutions with Agent-Based Modeling and Simulation, 2007)

"[…] our mental models fail to take into account the complications of the real world - at least those ways that one can see from a systems perspective. It is a warning list. Here is where hidden snags lie. You can’t navigate well in an interconnected, feedback-dominated world unless you take your eyes off short-term events and look for long-term behavior and structure; unless you are aware of false boundaries and bounded rationality; unless you take into account limiting factors, nonlinearities and delays. You are likely to mistreat, misdesign, or misread systems if you don’t respect their properties of resilience, self-organization, and hierarchy." (Donella H Meadows, "Thinking in Systems: A Primer", 2008)

"You can’t navigate well in an interconnected, feedback-dominated world unless you take your eyes off short-term events and look for long term behavior and structure; unless you are aware of false boundaries and bounded rationality; unless you take into account limiting factors, nonlinearities and delays." (Donella H Meadow, "Thinking in Systems: A Primer", 2008)

"A network of many simple processors ('units' or 'neurons') that imitates a biological neural network. The units are connected by unidirectional communication channels, which carry numeric data. Neural networks can be trained to find nonlinear relationships in data, and are used in various applications such as robotics, speech recognition, signal processing, medical diagnosis, or power systems." (Adnan Khashman et al, "Voltage Instability Detection Using Neural Networks", 2009)

"Linearity is a reductionist’s dream, and nonlinearity can sometimes be a reductionist’s nightmare. Understanding the distinction between linearity and nonlinearity is very important and worthwhile." (Melanie Mitchell, "Complexity: A Guided Tour", 2009)

On Nonlinearity I

"In complex systems cause and effect are often not closely related in either time or space. The structure of a complex system is not a simple feedback loop where one system state dominates the behavior. The complex system has a multiplicity of interacting feedback loops. Its internal rates of flow are controlled by nonlinear relationships. The complex system is of high order, meaning that there are many system states (or levels). It usually contains positive-feedback loops describing growth processes as well as negative, goal-seeking loops. In the complex system the cause of a difficulty may lie far back in time from the symptoms, or in a completely different and remote part of the system. In fact, causes are usually found, not in prior events, but in the structure and policies of the system." (Jay Wright Forrester, "Urban dynamics", 1969)

"Self-organization can be defined as the spontaneous creation of a globally coherent pattern out of local interactions. Because of its distributed character, this organization tends to be robust, resisting perturbations. The dynamics of a self-organizing system is typically non-linear, because of circular or feedback relations between the components. Positive feedback leads to an explosive growth, which ends when all components have been absorbed into the new configuration, leaving the system in a stable, negative feedback state. Non-linear systems have in general several stable states, and this number tends to increase (bifurcate) as an increasing input of energy pushes the system farther from its thermodynamic equilibrium. " (Francis Heylighen, "The Science Of Self-Organization And Adaptivity", 1970)

"[The] system may evolve through a whole succession of transitions leading to a hierarchy of more and more complex and organized states. Such transitions can arise in nonlinear systems that are maintained far from equilibrium: that is, beyond a certain critical threshold the steady-state regime become unstable and the system evolves into a new configuration." (Ilya Prigogine, Gregoire Micolis & Agnes Babloyantz, "Thermodynamics of Evolution", Physics Today 25 (11), 1972)

"An artificial neural network is an information-processing system that has certain performance characteristics in common with biological neural networks. Artificial neural networks have been developed as generalizations of mathematical models of human cognition or neural biology, based on the assumptions that: 1. Information processing occurs at many simple elements called neurons. 2. Signals are passed between neurons over connection links. 3. Each connection link has an associated weight, which, in a typical neural net, multiplies the signal transmitted. 4. Each neuron applies an activation function (usually nonlinear) to its net input (sum of weighted input signals) to determine its output signal." (Laurene Fausett, "Fundamentals of Neural Networks", 1994)

"Symmetry breaking in psychology is governed by the nonlinear causality of complex systems (the 'butterfly effect'), which roughly means that a small cause can have a big effect. Tiny details of initial individual perspectives, but also cognitive prejudices, may 'enslave' the other modes and lead to one dominant view." (Klaus Mainzer, "Thinking in Complexity", 1994)

"[…] nonlinear interactions almost always make the behavior of the aggregate more complicated than would be predicted by summing or averaging."  (John H Holland," Hidden Order: How Adaptation Builds Complexity", 1995)

“[…] self-organization is the spontaneous emergence of new structures and new forms of behavior in open systems far from equilibrium, characterized by internal feedback loops and described mathematically by nonlinear equations.” (Fritjof  Capra, “The web of life: a new scientific understanding of living  systems”, 1996)

"Bounded rationality simultaneously constrains the complexity of our cognitive maps and our ability to use them to anticipate the system dynamics. Mental models in which the world is seen as a sequence of events and in which feedback, nonlinearity, time delays, and multiple consequences are lacking lead to poor performance when these elements of dynamic complexity are present." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"Even if our cognitive maps of causal structure were perfect, learning, especially double-loop learning, would still be difficult. To use a mental model to design a new strategy or organization we must make inferences about the consequences of decision rules that have never been tried and for which we have no data. To do so requires intuitive solution of high-order nonlinear differential equations, a task far exceeding human cognitive capabilities in all but the simplest systems."  (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"All forms of complex causation, and especially nonlinear transformations, admittedly stack the deck against prediction. Linear describes an outcome produced by one or more variables where the effect is additive. Any other interaction is nonlinear. This would include outcomes that involve step functions or phase transitions. The hard sciences routinely describe nonlinear phenomena. Making predictions about them becomes increasingly problematic when multiple variables are involved that have complex interactions. Some simple nonlinear systems can quickly become unpredictable when small variations in their inputs are introduced." (Richard N Lebow, "Forbidden Fruit: Counterfactuals and International Relations", 2010)

On Randomness XII (Chaos I)

"Chaos is but unperceived order; it is a word indicating the limitations of the human mind and the paucity of observational facts. The words ‘chaos’, ‘accidental’, ‘chance’, ‘unpredictable’ are conveniences behind which we hide our ignorance." (Harlow Shapley, "Of Stars and Men: Human Response to an Expanding Universe", 1958)

"The term ‘chaos’ currently has a variety of accepted meanings, but here we shall use it to mean deterministically, or nearly deterministically, governed behavior that nevertheless looks rather random. Upon closer inspection, chaotic behavior will generally appear more systematic, but not so much so that it will repeat itself at regular intervals, as do, for example, the oceanic tides." (Edward N Lorenz, "Chaos, spontaneous climatic variations and detection of the greenhouse effect", 1991)

"The term chaos is used in a specific sense where it is an inherently random pattern of behaviour generated by fixed inputs into deterministic (that is fixed) rules (relationships). The rules take the form of non-linear feedback loops. Although the specific path followed by the behaviour so generated is random and hence unpredictable in the long-term, it always has an underlying pattern to it, a 'hidden' pattern, a global pattern or rhythm. That pattern is self-similarity, that is a constant degree of variation, consistent variability, regular irregularity, or more precisely, a constant fractal dimension. Chaos is therefore order (a pattern) within disorder (random behaviour)." (Ralph D Stacey, "The Chaos Frontier: Creative Strategic Control for Business", 1991)

"In nonlinear systems - and the economy is most certainly nonlinear - chaos theory tells you that the slightest uncertainty in your knowledge of the initial conditions will often grow inexorably. After a while, your predictions are nonsense." (M Mitchell Waldrop, "Complexity: The Emerging Science at the Edge of Order and Chaos", 1992)

"Intriguingly, the mathematics of randomness, chaos, and order also furnishes what may be a vital escape from absolute certainty - an opportunity to exercise free will in a deterministic universe. Indeed, in the interplay of order and disorder that makes life interesting, we appear perpetually poised in a state of enticingly precarious perplexity. The universe is neither so crazy that we can’t understand it at all nor so predictable that there’s nothing left for us to discover." (Ivars Peterson, "The Jungles of Randomness: A Mathematical Safari", 1997)

"When we look at the world around us, we find that we are not thrown into chaos and randomness but are part of a great order, a grand symphony of life. Every molecule in our body was once a part of previous bodies-living or nonliving-and will be a part of future bodies. In this sense, our body will not die but will live on, again and again, because life lives on. We share not only life's molecules but also its basic principles of organization with the rest of the living world. Arid since our mind, too, is embodied, our concepts and metaphors are embedded in the web of life together with our bodies and brains. We belong to the universe, we are at home in it, and this experience of belonging can make our lives profoundly meaningful." (Fritjof Capra, "The Hidden Connections", 2002)

"[…] we would like to observe that the butterfly effect lies at the root of many events which we call random. The final result of throwing a dice depends on the position of the hand throwing it, on the air resistance, on the base that the die falls on, and on many other factors. The result appears random because we are not able to take into account all of these factors with sufficient accuracy. Even the tiniest bump on the table and the most imperceptible move of the wrist affect the position in which the die finally lands. It would be reasonable to assume that chaos lies at the root of all random phenomena." (Iwo Białynicki-Birula & Iwona Białynicka-Birula, "Modeling Reality: How Computers Mirror Life", 2004) 

"Although the potential for chaos resides in every system, chaos, when it emerges, frequently stays within the bounds of its attractor(s): No point or pattern of points is ever repeated, but some form of patterning emerges, rather than randomness. Life scientists in different areas have noticed that life seems able to balance order and chaos at a place of balance known as the edge of chaos. Observations from both nature and artificial life suggest that the edge of chaos favors evolutionary adaptation." (Terry Cooke-Davies et al, "Exploring the Complexity of Projects", 2009)

"Most systems in nature are inherently nonlinear and can only be described by nonlinear equations, which are difficult to solve in a closed form. Non-linear systems give rise to interesting phenomena such as chaos, complexity, emergence and self-organization. One of the characteristics of non-linear systems is that a small change in the initial conditions can give rise to complex and significant changes throughout the system." (Robert K Logan, "The Poetry of Physics and The Physics of Poetry", 2010)

"A system in which a few things interacting produce tremendously divergent behavior; deterministic chaos; it looks random but its not." (Christopher Langton) 

On Randomness V (Systems I)

"Is a random outcome completely determined, and random only by virtue of our ignorance of the most minute contributing factors? Or are the contributing factors unknowable, and therefore render as random an outcome that can never be determined? Are seemingly random events merely the result of fluctuations superimposed on a determinate system, masking its predictability, or is there some disorderliness built into the system itself?” (Deborah J Bennett, "Randomness", 1998)

"The self-similarity of fractal structures implies that there is some redundancy because of the repetition of details at all scales. Even though some of these structures may appear to teeter on the edge of randomness, they actually represent complex systems at the interface of order and disorder."  (Edward Beltrami, "What is Random?: Chaos and Order in Mathematics and Life", 1999)

"Emergent self-organization in multi-agent systems appears to contradict the second law of thermodynamics. This paradox has been explained in terms of a coupling between the macro level that hosts self-organization (and an apparent reduction in entropy), and the micro level (where random processes greatly increase entropy). Metaphorically, the micro level serves as an entropy 'sink', permitting overall system entropy to increase while sequestering this increase from the interactions where self-organization is desired." (H Van Dyke Parunak & Sven Brueckner, "Entropy and Self-Organization in Multi-Agent Systems", Proceedings of the International Conference on Autonomous Agents, 2001)

"Entropy [...] is the amount of disorder or randomness present in any system. All non-living systems tend toward disorder; left alone they will eventually lose all motion and degenerate into an inert mass. When this permanent stage is reached and no events occur, maximum entropy is attained. A living system can, for a finite time, avert this unalterable process by importing energy from its environment. It is then said to create negentropy, something which is characteristic of all kinds of life." (Lars Skyttner, "General Systems Theory: Ideas and Applications", 2001)

"If a network is solely composed of neighborhood connections, information must traverse a large number of connections to get from place to place. In a small-world network, however, information can be transmitted between any two nodes using, typically, only a small number of connections. In fact, just a small percentage of random, long-distance connections is required to induce such connectivity. This type of network behavior allows the generation of 'six degrees of separation' type results, whereby any agent can connect to any other agent in the system via a path consisting of only a few intermediate nodes." (John H Miller & Scott E Page, "Complex Adaptive Systems", 2007)

"Although the potential for chaos resides in every system, chaos, when it emerges, frequently stays within the bounds of its attractor(s): No point or pattern of points is ever repeated, but some form of patterning emerges, rather than randomness. Life scientists in different areas have noticed that life seems able to balance order and chaos at a place of balance known as the edge of chaos. Observations from both nature and artificial life suggest that the edge of chaos favors evolutionary adaptation." (Terry Cooke-Davies et al, "Exploring the Complexity of Projects", 2009)

"Most systems in nature are inherently nonlinear and can only be described by nonlinear equations, which are difficult to solve in a closed form. Non-linear systems give rise to interesting phenomena such as chaos, complexity, emergence and self-organization. One of the characteristics of non-linear systems is that a small change in the initial conditions can give rise to complex and significant changes throughout the system. This property of a non-linear system such as the weather is known as the butterfly effect where it is purported that a butterfly flapping its wings in Japan can give rise to a tornado in Kansas. This unpredictable behaviour of nonlinear dynamical systems, i.e. its extreme sensitivity to initial conditions, seems to be random and is therefore referred to as chaos. This chaotic and seemingly random behaviour occurs for non-linear deterministic system in which effects can be linked to causes but cannot be predicted ahead of time." (Robert K Logan, "The Poetry of Physics and The Physics of Poetry", 2010)

"Second Law of thermodynamics is not an equality, but an inequality, asserting merely that a certain quantity referred to as the entropy of an isolated system - which is a measure of the system’s disorder, or ‘randomness’ - is greater (or at least not smaller) at later times than it was at earlier times." (Roger Penrose, "Cycles of Time: An Extraordinary New View of the Universe", 2010)

"[...] a high degree of unpredictability is associated with erratic trajectories. This not only because they look random but mostly because infinitesimally small uncertainties on the initial state of the system grow very quickly - actually exponentially fast. In real world, this error amplification translates into our inability to predict the system behavior from the unavoidable imperfect knowledge of its initial state." (Massimo Cencini, "Chaos: From Simple Models to Complex Systems", 2010)

"Although cascading failures may appear random and unpredictable, they follow reproducible laws that can be quantified and even predicted using the tools of network science. First, to avoid damaging cascades, we must understand the structure of the network on which the cascade propagates. Second, we must be able to model the dynamical processes taking place on these networks, like the flow of electricity. Finally, we need to uncover how the interplay between the network structure and dynamics affects the robustness of the whole system." (Albert-László Barabási, "Network Science", 2016)

On Networks X (Neural Networks I)

"A neural network is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: 1. Knowledge is acquired by the network through a learning process. 2. Interneuron connection strengths known as synaptic weights are used to store the knowledge." (Igor Aleksander, "An introduction to neural computing", 1990) 

"Neural Computing is the study of networks of adaptable nodes which through a process of learning from task examples, store experiential knowledge and make it available for use." (Igor Aleksander, "An introduction to neural computing", 1990)

"A neural network is characterized by A) its pattern of connections between the neurons (called its architecture), B) its method of determining the weights on the connections (called its training, or learning, algorithm), and C) its activation function." (Laurene Fausett, "Fundamentals of Neural Networks", 1994)

"An artificial neural network is an information-processing system that has certain performance characteristics in common with biological neural networks. Artificial neural networks have been developed as generalizations of mathematical models of human cognition or neural biology, based on the assumptions that: 1. Information processing occurs at many simple elements called neurons. 2. Signals are passed between neurons over connection links. 3. Each connection link has an associated weight, which, in a typical neural net, multiplies the signal transmitted. 4. Each neuron applies an activation function (usually nonlinear) to its net input (sum of weighted input signals) to determine its output signal." (Laurene Fausett, "Fundamentals of Neural Networks", 1994)

"At the other far extreme, we find many systems ordered as a patchwork of parallel operations, very much as in the neural network of a brain or in a colony of ants. Action in these systems proceeds in a messy cascade of interdependent events. Instead of the discrete ticks of cause and effect that run a clock, a thousand clock springs try to simultaneously run a parallel system. Since there is no chain of command, the particular action of any single spring diffuses into the whole, making it easier for the sum of the whole to overwhelm the parts of the whole. What emerges from the collective is not a series of critical individual actions but a multitude of simultaneous actions whose collective pattern is far more important. This is the swarm model." (Kevin Kelly, "Out of Control: The New Biology of Machines, Social Systems and the Economic World", 1995)

"The most familiar example of swarm intelligence is the human brain. Memory, perception and thought all arise out of the nett actions of billions of individual neurons. As we saw earlier, artificial neural networks (ANNs) try to mimic this idea. Signals from the outside world enter via an input layer of neurons. These pass the signal through a series of hidden layers, until the result emerges from an output layer. Each neuron modifies the signal in some simple way. It might, for instance, convert the inputs by plugging them into a polynomial, or some other simple function. Also, the network can learn by modifying the strength of the connections between neurons in different layers." (David G Green, "The Serendipity Machine: A voyage of discovery through the unexpected world of computers", 2004)

"It is not only a metaphor to transform the Internet to a superbrain with self-organizing features of learning and adapting. Information retrieval is already realized by neural networks adapting to the information preferences of a human user with synaptic plasticity. In sociobiology, we can 1 earn from populations of ants and termites how to organize traffic and information processing by swarm intelligence. From a technical point of view, we need intelligent programs distributed in the nets. There are already more or less intelligent virtual organisms {'agents'), learning, self-organizing and adapting to our individual preferences of information, to select our e-mails, to prepare economic transactions or to defend the attacks of hostile computer viruses, like the immune system of our body." (Klaus Mainzer, "Complexity Management in the Age of Globalization", 2006)

"A neural network is a particular kind of computer program, originally developed to try to mimic the way the human brain works. It is essentially a computer simulation of a complex circuit through which electric current flows." (Keith J Devlin & Gary Lorden, "The Numbers behind NUMB3RS: Solving crime with mathematics", 2007)

"A network of many simple processors ('units' or 'neurons') that imitates a biological neural network. The units are connected by unidirectional communication channels, which carry numeric data. Neural networks can be trained to find nonlinear relationships in data, and are used in various applications such as robotics, speech recognition, signal processing, medical diagnosis, or power systems." (Adnan Khashman et al, "Voltage Instability Detection Using Neural Networks", 2009)

"An artificial neural network, often just called a 'neural network' (NN), is an interconnected group of artificial neurons that uses a mathematical model or computational model for information processing based on a connectionist approach to computation. Knowledge is acquired by the network from its environment through a learning process, and interneuron connection strengths (synaptic weighs) are used to store the acquired knowledge." (Larbi Esmahi et al, "Adaptive Neuro-Fuzzy Systems", 2009)