Out of Context: On Learning (Definitions)

"Learning is, in too many cases, but a foil to common sense; a substitute for true knowledge." (William Hazlitt, "Table Talk; or, Original Essays", 1821-1822)

"Learning is its own exceeding great reward; and at the period of which we speak, it bore other fruits, not unworthy of it." (William Hazlitt, "The Plain Speaker", 1826)

"Learning is motivated by intent and understanding by visualization." (A B Garrett, "Visualization: A Step to Understanding", Journal of Chemical Education Vol. 25 (10), 1948)

"Learning is a property of all living organisms." (Winfred B. Hirschmann, "Profit from the Learning Curve", Harvard Business Review, 1964)

"Learning is any change in a system that produces a more or less permanent change in its capacity for adapting to its environment." (Herbert A Simon, "The Sciences of the Artificial", 1968)

"Learning is finding out what you already know." (Richard Bach, "Illusions: The Adventures of a Reluctant Messiah", 1977)

"Teaching is only demonstrating that it is possible. Learning is making it possible for yourself." (Paulo Coelho, "The Pilgrimage", 1987)

"Learning is the process of obtaining new knowledge. It results in a better reaction to the same inputs at the next session of operation. It means improvement. It is a step toward adaptation. Learning is a major characteristic of intelligent systems." (Nikola K Kasabov, "Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"Learning is a multi-faceted, integrated process where changes with any one element alters the larger network. Knowledge is subject to the nuances of complex, adaptive systems." (George Siemens, "Knowing Knowledge", 2006)

"Learning is the process of creating networks." (George Siemens, "Knowing Knowledge", 2006)

"Learning is a process of modifying or completely changing our mental models based on new experiences or evidence." (Edward D Hess, "Learn or Die: Using Science to Build a Leading-Edge Learning Organization", 2014)

"Learning is a dynamic event and so the belief that learning is primarily about the acquisition of facts is fundamentally flawed - the acquisition and manipulation of data is at best a prerequisite to learning. Real learning involves acquiring knowledge and understanding." (William Byers, "Deep Thinking: What Mathematics Can Teach Us About the Mind", 2015)

"A little learning is a dangerous thing." (Alexander Pope)

"Learning is a treasure which accompanies its owner everywhere." (proverb)

"Learning is the only thing the mind never exhausts, never fears, and never regrets." (Leonardo da Vinci)

"Learning is not attained by chance. It must be sought for with ardor and attended to with diligence." (Abigail Adams)

John D Kelleher - Collected Quotes

"A predictive model overfits the training set when at least some of the predictions it returns are based on spurious patterns present in the training data used to induce the model. Overfitting happens for a number of reasons, including sampling variance and noise in the training set. The problem of overfitting can affect any machine learning algorithm; however, the fact that decision tree induction algorithms work by recursively splitting the training data means that they have a natural tendency to segregate noisy instances and to create leaf nodes around these instances. Consequently, decision trees overfit by splitting the data on irrelevant features that only appear relevant due to noise or sampling variance in the training data. The likelihood of overfitting occurring increases as a tree gets deeper because the resulting predictions are based on smaller and smaller subsets as the dataset is partitioned after each feature test in the path." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies", 2015)

"Decision trees are also discriminative models. Decision trees are induced by recursively partitioning the feature space into regions belonging to the different classes, and consequently they define a decision boundary by aggregating the neighboring regions belonging to the same class. Decision tree model ensembles based on bagging and boosting are also discriminative models." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies", 2015)

"Decision trees are also considered nonparametric models. The reason for this is that when we train a decision tree from data, we do not assume a fixed set of parameters prior to training that define the tree. Instead, the tree branching and the depth of the tree are related to the complexity of the dataset it is trained on. If new instances were added to the dataset and we rebuilt the tree, it is likely that we would end up with a (potentially very) different tree." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies", 2015)

"It is important to remember that predictive data analytics models built using machine learning techniques are tools that we can use to help make better decisions within an organization and are not an end in themselves. It is paramount that, when tasked with creating a predictive model, we fully understand the business problem that this model is being constructed to address and ensure that it does address it." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, worked examples, and case studies", 2015)

"The main advantage of decision tree models is that they are interpretable. It is relatively easy to understand the sequences of tests a decision tree carried out in order to make a prediction. This interpretability is very important in some domains. [...] Decision tree models can be used for datasets that contain both categorical and continuous descriptive features. A real advantage of the decision tree approach is that it has the ability to model the interactions between descriptive features. This arises from the fact that the tests carried out at each node in the tree are performed in the context of the results of the tests on the other descriptive features that were tested at the preceding nodes on the path from the root. Consequently, if there is an interaction effect between two or more descriptive features, a decision tree can model this."  (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies", 2015)

"There are two kinds of mistakes that an inappropriate inductive bias can lead to: underfitting and overfitting. Underfitting occurs when the prediction model selected by the algorithm is too simplistic to represent the underlying relationship in the dataset between the descriptive features and the target feature. Overfitting, by contrast, occurs when the prediction model selected by the algorithm is so complex that the model fits to the dataset too closely and becomes sensitive to noise in the data."(John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies", 2015)

"Tree pruning identifies and removes subtrees within a decision tree that are likely to be due to noise and sample variance in the training set used to induce it. In cases where a subtree is deemed to be overfitting, pruning the subtree means replacing the subtree with a leaf node that makes a prediction based on the majority target feature level (or average target feature value) of the dataset created by merging the instances from all the leaf nodes in the subtree. Obviously, pruning will result in decision trees being created that are not consistent with the training set used to build them. In general, however, we are more interested in creating prediction models that generalize well to new data rather than that are strictly consistent with training data, so it is common to sacrifice consistency for generalization capacity." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies", 2015)

"When datasets are small, a parametric model may perform well because the strong assumptions made by the model - if correct - can help the model to avoid overfitting. However, as the size of the dataset grows, particularly if the decision boundary between the classes is very complex, it may make more sense to allow the data to inform the predictions more directly. Obviously the computational costs associated with nonparametric models and large datasets cannot be ignored. However, support vector machines are an example of a nonparametric model that, to a large extent, avoids this problem. As such, support vector machines are often a good choice in complex domains with lots of data." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies", 2015)

"When we find data quality issues due to valid data during data exploration, we should note these issues in a data quality plan for potential handling later in the project. The most common issues in this regard are missing values and outliers, which are both examples of noise in the data." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, worked examples, and case studies", 2015)

"A neural network consists of a set of neurons that are connected together. A neuron takes a set of numeric values as input and maps them to a single output value. At its core, a neuron is simply a multi-input linear-regression function. The only significant difference between the two is that in a neuron the output of the multi-input linear-regression function is passed through another function that is called the activation function." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"Data scientists should have some domain expertise. Most data science projects begin with a real-world, domain-specific problem and the need to design a data-driven solution to this problem. As a result, it is important for a data scientist to have enough domain expertise that they understand the problem, why it is important, an dhow a data science solution to the problem might fit into an organization’s processes. This domain expertise guides the data scientist as she works toward identifying an optimized solution." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"One of the biggest myths is the belief that data science is an autonomous process that we can let loose on our data to find the answers to our problems. In reality, data science requires skilled human oversight throughout the different stages of the process. [...] The second big myth of data science is that every data science project needs big data and needs to use deep learning. In general, having more data helps, but having the right data is the more important requirement. [...] A third data science myth is that modern data science software is easy to use, and so data science is easy to do. [...] The last myth about data science [...] is the belief that data science pays for itself quickly. The truth of this belief depends on the context of the organization. Adopting data science can require significant investment in terms of developing data infrastructure and hiring staff with data science expertise. Furthermore, data science will not give positive results on every project." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"One of the most important skills for a data scientist is the ability to frame a real-world problem as a standard data science task." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"Presenting data in a graphical format makes it much easier to see and understand what is happening with the data. Data visualization applies to all phases of the data science process."  (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"The goal of data science is to improve decision making by basing decisions on insights extracted from large data sets. As a field of activity, data science encompasses a set of principles, problem definitions, algorithms, and processes for extracting nonobvious and useful patterns from large data sets. It is closely related to the fields of data mining and machine learning, but it is broader in scope." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"The patterns that we extract using data science are useful only if they give us insight into the problem that enables us to do something to help solve the problem." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"The promise of data science is that it provides a way to understand the world through data." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"Using data science, we can uncover the important patterns in a data set, and these patterns can reveal the important attributes in the domain. The reason why data science is used in so many domains is that it doesn’t matter what the problem domain is: if the right data are available and the problem can be clearly defined, then data science can help."  (John D Kelleher & Brendan Tierney, "Data Science", 2018)

"We humans are reasonably good at defining rules that check one, two, or even three attributes (also commonly referred to as features or variables), but when we go higher than three attributes, we can start to struggle to handle the interactions between them. By contrast, data science is often applied in contexts where we want to look for patterns among tens, hundreds, thousands, and, in extreme cases, millions of attributes." (John D Kelleher & Brendan Tierney, "Data Science", 2018)

On Knowledge (1800-1824)

"Knowledge is only real and can only be set forth fully in the form of science, in the form of system." (G W Friedrich Hegel, "The Phenomenology of Mind", 1807)

"It is not knowledge, but the act of learning, not possession but the act of getting there, which grants the greatest enjoyment. When I have clarified and exhausted a subject, then I turn away from it, in order to go into darkness again; the never satisfied man is so strange if he has completed a structure, then it is not in order to dwell in it peacefully, but in order to begin another. I imagine the world conqueror must feel thus, who, after one kingdom is scarcely conquered, stretched out his arms for others." (Carl F Gauss, [Letter to Farkas Bolyai] 1808)

"Thus then does the Doctrine of Knowledge, which in its substance is the realisation of the absolute Power of intelligising which has now been defined, end with the recognition of itself as a mere Schema in a Doctrine of Wisdom, although indeed a necessary and indispensable means to such a Doctrine: - a Schema, the sole aim of which is, with the knowledge thus acquired, - by which knowledge alone a Will, clear and intelligible to itself and reposing upon itself without wavering or perplexity, is possible, - to return wholly into Actual Life; - not into the Life of blind and irrational Instinct which we have laid bare in all its nothingness, but into the Divine Life which shall become visible to us." (Johann G Fichte, "Outline of the Doctrine of Knowledge", 1810)

"The most important questions of life are, for the most part, really only problems of probability. Strictly speaking one may even say that nearly all our knowledge is problematical; and in the small number of things which we are able to know with certainty, even in the mathematical sciences themselves, induction and analogy, the principal means for discovering truth, are based on probabilities, so that the entire system of human knowledge is connected with this theory." (Pierre-Simon Laplace, "Theorie Analytique des Probabilités", 1812)

"One may even say, strictly speaking, that almost all our knowledge is only probable; and in the small number of things that we are able to know with certainty, in the mathematical sciences themselves, the principal means of arriving at the truth - induction and analogy - are based on probabilities, so that the whole system of human knowledge is tied up with the theory set out in this essay." (Pierre-Simon Laplace, "Philosophical Essay on Probabilities", 1814) 

"[...] all knowledge, and especially the weightiest knowledge of the truth, to which only a brief triumph is allotted between the two long periods in which it is condemned as paradoxical or disparaged as trivial." (Arthur Schopenhauer, "The World as Will and Representation", 1819)

"The highest knowledge can be nothing more than the shortest and clearest road to truth; all the rest is pretension, not performance, mere verbiage and grandiloquence, from which we can learn nothing." (Charles C Colton, "Lacon", 1820)

"We [...] are profiting not only by the knowledge, but also by the ignorance, not only by the discoveries, but also by the errors of our forefathers; for the march of science, like that of time, has been progressing in the darkness, no less than in the light." (Charles C Colton, "Lacon", 1820)

"The aim of every science is foresight. For the laws of established observation of phenomena are generally employed to foresee their succession. All men, however little advanced make true predictions, which are always based on the same principle, the knowledge of the future from the past." (Auguste Compte, "Plan des travaux scientifiques nécessaires pour réorganiser la société", 1822)

On Knowledge (1960-1969)

"Any pattern of activity in a network, regarded as consistent by some observer, is a system, Certain groups of observers, who share a common body of knowledge, and subscribe to a particular discipline, like 'physics' or 'biology' (in terms of which they pose hypotheses about the network), will pick out substantially the same systems. On the other hand, observers belonging to different groups will not agree about the activity which is a system." (Gordon Pask, "The Natural History of Networks", 1960)

"The most important maxim for data analysis to heed, and one which many statisticians seem to have shunned is this: ‘Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.’ Data analysis must progress by approximate answers, at best, since its knowledge of what the problem really is will at best be approximate." (John W Tukey, "The Future of Data Analysis", Annals of Mathematical Statistics, Vol. 33, No. 1, 1962)

"Incomplete knowledge must be considered as perfectly normal in probability theory; we might even say that, if we knew all the circumstances of a phenomenon, there would be no place for probability, and we would know the outcome with certainty." (Félix E Borel, Probability and Certainty", 1963)

"When a science approaches the frontiers of its knowledge, it seeks refuge in allegory or in analogy." (Erwin Chargaff, "Essays on Nucleic Acids", 1963)

"In its efforts to learn as much as possible about nature, modem physics has found that certain things can never be ‘known’ with certainty. Much of our knowledge must always remain uncertain. The most we can know is in terms of probabilities." (Richard P Feynman, "The Feynman Lectures on Physics", 1964)

"A model is a useful (and often indispensable) framework on which to organize our knowledge about a phenomenon. […] It must not be overlooked that the quantitative consequences of any model can be no more reliable than the a priori agreement between the assumptions of the model and the known facts about the real phenomenon. When the model is known to diverge significantly from the facts, it is self-deceiving to claim quantitative usefulness for it by appeal to agreement between a prediction of the model and observation." (John R Philip, 1966)

"It is a commonplace of modern technology that there is a high measure of certainty that problems have solutions before there is knowledge of how they are to be solved." (John K Galbraith, "The New Industrial State", 1967)

"The aim of science is not so much to search for truth, or even truths, as to classify our knowledge and to establish relations between observable phenomena in order to be able to predict the future in a certain measure and to explain the sequence of phenomena in relation to ourselves." (Pierre L du Noüy, "Between Knowing and Believing", 1967)

"It [knowledge] is clearly related to information, which we can now measure; and an economist especially is tempted to regard knowledge as a kind of capital structure, corresponding to information as an income flow. Knowledge, that is to say, is some kind of improbable structure or stock made up essentially of patterns - that is, improbable arrangements, and the more improbable the arrangements, we might suppose, the more knowledge there is." (Kenneth E Boulding, "Beyond Economics: Essays on Society", 1968)

"Knowing reality means constructing systems of transformations that correspond, more or less adequately, to reality. They are more or less isomorphic to transformations of reality. The transformational structures of which knowledge consists are not copies of the transformations in reality; they are simply possible isomorphic models among which experience can enable us to choose. Knowledge, then, is a system of transformations that become progressively adequate." (Jean Piaget, "Genetic Epistemology", 1968)

"Scientific knowledge is not created solely by the piecemeal mining of discrete facts by uniformly accurate and reliable individual scientific investigations. The process of criticism and evaluation, of analysis and synthesis, are essential to the whole system. It is impossible for each one of us to be continually aware of all that is going on around us, so that we can immediately decide the significance of every new paper that is published. The job of making such judgments must therefore be delegated to the best and wisest among us, who speak, not with their own personal voices, but on behalf of the whole community of Science. […] It is impossible for the consensus - public knowledge - to be voiced at all, unless it is channeled through the minds of selected persons, and restated in their words for all to hear." (John M Ziman, "Public Knowledge: An Essay Concerning the Social Dimension of Science", 1968)

"The idea of knowledge as an improbable structure is still a good place to start. Knowledge, however, has a dimension which goes beyond that of mere information or improbability. This is a dimension of significance which is very hard to reduce to quantitative form. Two knowledge structures might be equally improbable but one might be much more significant than the other." (Kenneth E Boulding, "Beyond Economics: Essays on Society", 1968)

"Discovery always carries an honorific connotation. It is the stamp of approval on a finding of lasting value. Many laws and theories have come and gone in the history of science, but they are not spoken of as discoveries. […] Theories are especially precarious, as this century profoundly testifies. World views can and do often change. Despite these difficulties, it is still true that to count as a discovery a finding must be of at least relatively permanent value, as shown by its inclusion in the generally accepted body of scientific knowledge." (Richard J. Blackwell, "Discovery in the Physical Sciences", 1969)

"It is not enough to observe, experiment, theorize, calculate and communicate; we must also argue, criticize, debate, expound, summarize, and otherwise transform the information that we have obtained individually into reliable, well established, public knowledge." (John M Ziman, "Information, Communication, Knowledge", Nature Vol. 224 (5217), 1969)

"Models constitute a framework or a skeleton and the flesh and blood will have to be added by a lot of common sense and knowledge of details."(Jan Tinbergen, "The Use of Models: Experience," 1969)

"The 'flow of information' through human communication channels is enormous. So far no theory exists, to our knowledge, which attributes any sort of unambiguous measure to this 'flow'." (Anatol Rapoport, "Modern Systems Research for the Behavioral Scientist", 1969)

On Patterns (2000-2009)

"In a linear world of equilibrium and predictability, the sparse research into an evidence base for management prescriptions and the confused findings it produces would be a sign of incompetence; it would not make much sense. Nevertheless, if organizations are actually patterns of nonlinear interaction between people; if small changes could produce widespread major consequences; if local interaction produces emergent global pattern; then it will not be possible to provide a reliable evidence base. In such a world, it makes no sense to conduct studies looking for simple causal relationships between an action and an outcome. I suggest that the story of the last few years strongly indicates that human action is nonlinear, that time and place matter a great deal, and that since this precludes simple evidence bases we do need to rethink the nature of organizations and the roles of managers and leaders in them." (Ralph D Stacey, "Complexity and Organizational Reality", 2000)

"The central proposition in [realistic thinking] is that human actions and interactions are processes, not systems, and the coherent patterning of those processes becomes what it becomes because of their intrinsic capacity, the intrinsic capacity of interaction and relationship, to form coherence. That emergent form is radically unpredictable, but it emerges in a controlled or patterned way because of the characteristic of relationship itself, creation and destruction in conditions at the edge of chaos." (Ralph D Stacey et al, "Complexity and Management: Fad or Radical Challenge to Systems Thinking?", 2000)

"Although the detailed moment-to-moment behavior of a chaotic system cannot be predicted, the overall pattern of its 'random' fluctuations may be similar from scale to scale. Likewise, while the fine details of a chaotic system cannot be predicted one can know a little bit about the range of its 'random' fluctuation." (F David Peat, "From Certainty to Uncertainty", 2002)

"There are endless examples of elaborate structures and apparently complex processes being generated through simple repetitive rules, all of which can be easily simulated on a computer. It is therefore tempting to believe that, because many complex patterns can be generated out of a simple algorithmic rule, all complexity is created in this way." (F David Peat, "From Certainty to Uncertainty", 2002)

"Randomness is a difficult notion for people to accept. When events come in clusters and streaks, people look for explanations and patterns. They refuse to believe that such patterns - which frequently occur in random data - could equally well be derived from tossing a coin. So it is in the stock market as well." (Didier Sornette, "Why Stock Markets Crash: Critical events in complex financial systems", 2003)

"Learning is the process of creating networks. Nodes are external entities which we can use to form a network. Or nodes may be people, organizations, libraries, web sites, books, journals, database, or any other source of information. The act of learning (things become a bit tricky here) is one of creating an external network of nodes - where we connect and form information and knowledge sources. The learning that happens in our heads is an internal network (neural). Learning networks can then be perceived as structures that we create in order to stay current and continually acquire, experience, create, and connect new knowledge (external). And learning networks can be perceived as structures that exist within our minds (internal) in connecting and creating patterns of understanding." (George Siemens, "Knowing Knowledge", 2006)

"Some number patterns, like even and odd numbers, lie on the surface. But the more you learn about numbers, both experimentally and theoretically, the more you discover patterns that are not so obvious. […] After a hidden pattern is exposed, it can be used to find more hidden patterns. At the end of a long chain of patterned reasoning, you can get to very difficult theorems, exploring facts about numbers that you otherwise would not know were true." (Avner Ash & Robert Gross, "Fearless Symmetry: Exposing the hidden patterns of numbers", 2006)

"Still, in the end, we find ourselves drawn to the beauty of the patterns themselves, and the amazing fact that we humans are smart enough to prove even a feeble fraction of all possible theorems about them. Often, greater than the contemplation of this beauty for the active mathematician is the excitement of the chase. Trying to discover first what patterns actually do or do not occur, then finding the correct statement of a conjecture, and finally proving it - these things are exhilarating when accomplished successfully. Like all risk-takers, mathematicians labor months or years for these moments of success." (Avner Ash & Robert Gross, "Fearless Symmetry: Exposing the hidden patterns of numbers", 2006)

"There is a big debate as to whether logic is part of mathematics or mathematics is part of logic. We use logic to think. We notice that our thinking, when it is valid, goes in certain patterns. These patterns can be studied mathematically. Thus, logic is a part of mathematics, called 'mathematical logic'." (Avner Ash & Robert Gross, "Fearless Symmetry: Exposing the hidden patterns of numbers", 2006) 

"The system is highly sensitive to some small changes and blows them up into major alterations in weather patterns. This is popularly known as the butterfly effect in that it is possible for a butterfly to flap its wings in São Paolo, so making a tiny change to air pressure there, and for this tiny change to escalate up into a hurricane over Miami. You would have to measure the flapping of every butterfly’s wings around the earth with infinite precision in order to be able to make long-term forecasts. The tiniest error made in these measurements could produce spurious forecasts. However, short-term forecasts are possible because it takes time for tiny differences to escalate."  (Ralph D Stacey, "Strategic Management and Organisational Dynamics: The Challenge of Complexity" 5th Ed. , 2007)

"Perception requires imagination because the data people encounter in their lives are never complete and always equivocal. [...] We also use our imagination and take shortcuts to fill gaps in patterns of nonvisual data. As with visual input, we draw conclusions and make judgments based on uncertain and incomplete information, and we conclude, when we are done analyzing the patterns, that out picture is clear and accurate. But is it?" (Leonard Mlodinow, "The Drunkard’s Walk: How Randomness Rules Our Lives", 2008)

"Why is the human need to be in control relevant to a discussion of random patterns? Because if events are random, we are not in control, and if we are in control of events, they are not random. There is therefore a fundamental clash between our need to feel we are in control and our ability to recognize randomness. That clash is one of the principal reasons we misinterpret random events."  (Leonard Mlodinow, "The Drunkard’s Walk: How Randomness Rules Our Lives", 2008)

"In emergent processes, the whole is greater than the sum of the parts. A mathematical phenomenon that appears in certain dynamic systems also occurs within biological systems, from molecular interactions within the cells to the cognitive processes that we use to move within society. [...] Emergent patterns of ideas, beauty, desires, or tragicomedy wait, ready to trap the next traveler in their complex domain of neatly patterned squares - the never-ending world of chess metaphors." (Diego Rasskin-Gutman, "Chess Metaphors: Artificial Intelligence and the Human Mind", 2009)

"Obviously, the final goal of scientists and mathematicians is not simply the accumulation of facts and lists of formulas, but rather they seek to understand the patterns, organizing principles, and relationships between these facts to form theorems and entirely new branches of human thought." (Clifford A Pickover, "The Math Book", 2009)

"The master of chess is deeply familiar with these patterns and knows very well the position that would be beneficial to reach. The rest is thinking in a logical way (calculating) about how each piece should be moved to reach the new pattern that has already taken shape in the chess player’s mind. This way of facing chess is closely related to the solving of theorems in mathematics. For example, a mathematician who wishes to prove an equation needs to imagine how the terms on each side of the equal sign can be manipulated so that one is reduced to the other. The enterprise is far from easy, to judge by the more than two hundred years that have been needed to solve theorems such as that of Fermat (z^n = x^n + y^n), using diverse tricks to prove the equation." (Diego Rasskin-Gutman, "Chess Metaphors: Artificial Intelligence and the Human Mind", 2009)

On Stability II

"Stability is commonly thought of as desirable, for its presence enables the system to combine of flexibility and activity in performance with something of permanence. Behaviour that is goal-seeking is an example of behaviour that is stable around a state of equilibrium. Nevertheless, stability is not always good, for a system may persist in returning to some state that, for other reasons, is considered undesirable." (W Ross Ashby, "An Introduction to Cybernetics", 1956)

"Effect spreads its 'tentacles' not only forwards (as a new cause giving rise to a new effect) but also backwards, to the cause which gave rise to it, thus modifying, exhausting or intensifying its force. This interaction of cause and effect is known as the principle of feedback. It operates everywhere, particularly in all self-organising systems where perception, storing, processing and use of information take place, as for example, in the organism, in a cybernetic device, and in society. The stability, control and progress of a system are inconceivable without feedback." (Alexander Spirkin, "Dialectical Materialism", 1983)

"Structure is the type of connection between the elements of a whole. […] . Structure is a composite whole, or an internally organised content. […] Structure implies not only the position of its elements in space but also their movement in time, their sequence and rhythm, the law of mutation of a process. So structure is actually the law or set of laws that determine a system's composition and functioning, its properties and stability." (Alexander Spirkin, "Dialectical Materialism", 1983)

"Stability theory is the study of systems under various perturbing influences. Since there are many systems, many types of influences, and many equations describing systems, this is an open-ended problem. A system is designed so that it will be stable under external influences. However, one cannot predict all external influences, nor predict the magnitude of those that occur. Consequently, we need control theory. If one is interested in stability theory, a natural result is a theory of control." (Richard E Bellman, "Eye of the Hurricane: An Autobiography", 1984)

"All systems evolve, although the rates of evolution may vary over time both between and within systems. The rate of evolution is a function of both the inherent stability of the system and changing environmental circumstances. But no system can be stabilized forever. For the universe as a whole, an isolated system, time’s arrow points toward greater and greater breakdown, leading to complete molecular chaos, maximum entropy, and heat death. For open systems, including the living systems that are of major interest to us and that interchange matter and energy with their external environments, time’s arrow points to evolution toward greater and greater complexity. Thus, the universe consists of islands of increasing order in a sea of decreasing order. Open systems evolve and maintain structure by exporting entropy to their external environments." (L Douglas Kiel, "Chaos Theory in the Social Sciences: Foundations and Applications", 1996)

"The phenomenon of emergence takes place at critical points of instability that arise from fluctuations in the environment, amplified by feedback loops." (Fritjof Capra, "The Hidden Connections", 2002)

"This spontaneous emergence of order at critical points of instability is one of the most important concepts of the new understanding of life. It is technically known as self-organization and is often referred to simply as ‘emergence’. It has been recognized as the dynamic origin of development, learning and evolution. In other words, creativity-the generation of new forms-is a key property of all living systems. And since emergence is an integral part of the dynamics of open systems, we reach the important conclusion that open systems develop and evolve. Life constantly reaches out into novelty." (Fritjof  Capra, "The Hidden Connections", 2002)

"Classification is only one of the mathematical aspects of catastrophe theory. Another is stability. The stable states of natural systems are the ones that we can observe over a longer period of time. But the stable states of a system, which can be described by potential functions and their singularities, can become unstable if the potentials are changed by perturbations. So stability problems in nature lead to mathematical questions concerning the stability of the potential functions." (Werner Sanns, "Catastrophe Theory" [Mathematics of Complexity and Dynamical Systems, 2012])

"An important aspect of the global theory of dynamical systems is the stability of the orbit structure as a whole. The motivation for the corresponding theory comes from applied mathematics. Mathematical models always contain simplifying assumptions. Dominant features are modeled; supposed small disturbing forces are ignored. Thus, it is natural to ask if the qualitative structure of the set of solutions - the phase portrait - of a model would remain the same if small perturbations were included in the model. The corresponding mathematical theory is called structural stability." (Carmen Chicone, "Stability Theory of Ordinary Differential Equations" [Mathematics of Complexity and Dynamical Systems, 2012])

"This spontaneous emergence of order at critical points of instability, which is often referred to simply as 'emergence', is one of the hallmarks of life. It has been recognized as the dynamic origin of development, learning, and evolution. In other words, creativity-the generation of new forms-is a key property of all living systems." (Fritjof Capra, "The Systems View of Life: A Unifying Vision", 2014)

Frederick Mosteller - Collected Quotes

 "As usual we may make the errors of I) rejecting the null hypothesis when it is true, II) accepting the null hypothesis when it is false. But there is a third kind of error which is of interest because the present test of significance is tied up closely with the idea of making a correct decision about which distribution function has slipped furthest to the right. We may make the error of III) correctly rejecting the null hypothesis for the wrong reason." (Frederick Mosteller, "A k-Sample Slippage Test for an Extreme Population", The Annals of Mathematical Statistics 19, 1948)

"Errors of the third kind happen in conventional tests of differences of means, but they are usually not considered, although their existence is probably recognized. It seems to the author that there may be several reasons for this among which are 1) a preoccupation on the part of mathematical statisticians with the formal questions of acceptance and rejection of null hypotheses without adequate consideration of the implications of the error of the third kind for the practical experimenter, 2) the rarity with which an error of the third kind arises in the usual tests of significance." (Frederick Mosteller, "A k-Sample Slippage Test for an Extreme Population", The Annals of Mathematical Statistics 19, 1948)

"For many purposes graphical accuracy is sufficient. The speed of graphical processes, and more especially the advantages of visual presentation in pointing out facts or clues which might otherwise be overlooked, make graphical analysis very valuable." (Frederick Mosteller & John W Tukey, "The Uses and Usefulness of Binomial Probability Paper?", Journal of the American Statistical Association 44, 1949)

"If significance tests are required for still larger samples, graphical accuracy is insufficient, and arithmetical methods are advised. A word to the wise is in order here, however. Almost never does it make sense to use exact binomial significance tests on such data - for the inevitable small deviations from the mathematical model of independence and constant split have piled up to such an extent that the binomial variability is deeply buried and unnoticeable. Graphical treatment of such large samples may still be worthwhile because it brings the results more vividly to the eye." (Frederick Mosteller & John W Tukey, "The Uses and Usefulness of Binomial Probability Paper?", Journal of the American Statistical Association 44, 1949)

"Scientific and technological advances have made the world we live in complex and hard to understand. […] Science itself shows the same growing complexity. We often hear that 'one man can no longer cover a broad enough field' and that 'there is too much narrow specialization'. And yet these complexities must be met - and resolved. At all levels, decisions must be made which involve consideration of more than a single field." (Frederick Mosteller et al, "The Education of a Scientific Generalist", Science 109,1949)

"Mathematical models for empirical phenomena aid the development of a science when a sufficient body of quantitative information has been accumulated. This accumulation can be used to point the direction in which models should be constructed and to test the adequacy of such models in their interim states. Models, in turn, frequently are useful in organizing and interpreting experimental data and in suggesting new directions for experimental research." (Robert R. Bush & Frederick Mosteller, "A Mathematical Model for Simple Learning", Psychological Review 58, 1951)

"Almost any sort of inquiry that is general and not particular involves both sampling and measurement […]. Further, both the measurement and the sampling will be imperfect in almost every case. We can define away either imperfection in certain cases. But the resulting appearance of perfection is usually only an illusion." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"Because representativeness is inherent in the sampling plan and not in the particular sample at hand, we can never make adequate use of sample results without some measure of how well the results of this particular sample are likely to agree with the results of other samples which the same sampling plan might have provided. The ability to assess stability fairly is as important as the ability to represent the population fairly. Modern sampling plans concentrate on both." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"By sampling we can learn only about collective properties of populations, not about properties of individuals. We can study the average height, the percentage who wear hats, or the variability in weight of college juniors [...]. The population we study may be small or large, but there must be a population - and what we are studying must be a population characteristic. By sampling, we cannot study individuals as particular entities with unique idiosyncrasies; we can study regularities (including typical variabilities as well as typical levels) in a population as exemplified by the individuals in the sample." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"In many cases general probability samples can be thought of in terms of (1) a subdivision of the population into strata, (2) a self-weighting probability sample in each stratum, and (3) combination of the stratum sample means weighted by the size of the stratum." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"That which can be and should be representative is the sampling plan, which includes the manner in which the sample was drawn (essentially a specification of what other samples might have been drawn and what the relative chances of selection were for any two possible samples) and how it is to be analyzed. [...] It is clear that many [...] groups fail to be represented in any particular sample, yet this is not a criticism of that sample. Representation is not, and should not be, by groups. It is, and should be, by individuals as members of the sampled population. Representation is not, and should not be, in any particular sample. It is, and should be, in the sampling plan." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"The main purpose of a significance test is to inhibit the natural enthusiasm of the investigator." (Frederick Mosteller, "Selected Quantitative Techniques", 1954)

"We must emphasize that such terms as 'select at random', 'choose at random', and the like, always mean that some mechanical device, such as coins, cards, dice, or tables of random numbers, is used." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"We have made the sampling plan representative, not by giving each individual an equal chance to enter the sample and then weighting them equally, but by a more noticeable process of compensation, where those individuals very likely to enter the sample are weighted less, while those unlikely to enter are weighted more when they do appear. The net result is to give each individual an equal chance of affecting the (weighted) sample mean." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"We realize that if someone just 'grabs a handful', the individuals in the handful almost always resemble one another (on the average) more than do the members of a simple random sample. Even if the 'grabs' [sampling] are randomly spread around so that every individual has an equal chance of entering the sample, there are difficulties. Since the individuals of grab samples resemble one another more than do individuals of random samples, it follows (by a simple mathematical argument) that the means of grab samples resemble one another less than the means of random samples of the same size. From a grab sample, therefore, we tend to underestimate the variability in the population, although we should have to overestimate it in order to obtain valid estimates of variability of grab sample means by substituting such an estimate into the formula for the variability of means of simple random samples. Thus using simple random sample formulas for grab sample means introduces a double bias, both parts of which lead to an unwarranted appearance of higher stability." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"Weighing a sample appropriately is no more fudging the data than is correcting a gas volume for barometric pressure." (Frederick Mosteller et al, "Principles of Sampling", Journal of the American Statistical Association Vol. 49 (265), 1954)

"A primary goal of any learning model is to predict correctly the learning curve - proportions of correct responses versus trials. Almost any sensible model with two or three free parameters, however, can closely fit the curve, and so other criteria must be invoked when one is comparing several models." (Robert R Bush & Frederick Mosteller, "A Comparison of Eight Models?", Studies in Mathematical Learning Theory, 1959)

"A satisfactory prediction of the sequential properties of learning data from a single experiment is by no means a final test of a model. Numerous other criteria - and some more demanding - can be specified. For example, a model with specific numerical parameter values should be invariant to changes in independent variables that explicitly enter in the model." (Robert R Bush & Frederick Mosteller,"A Comparison of Eight Models?", Studies in Mathematical Learning Theory, 1959)

"In the testing of a scientific model or theory, one rarely has a general measure of goodness-of-fit, a universal yardstick by which one accepts or rejects the model. Indeed, science does not and should not work this way; a theory is kept until a better one is found. One way that science does work is by comparing two or more theories to determine their relative merits in handling relevant data."(Robert R Bush & Frederick Mosteller, "A Comparison of Eight Models?", Studies in Mathematical Learning Theory, 1959)

"In a problem, the great thing is the challenge. A problem can be challenging for many reasons: because the subject matter is intriguing, because the answer defies unsophisticated intuition, because it illustrates an important principle, because of its vast generality, because of its difficulty, because of a clever solution, or even because of the simplicity or beauty of the answer." (Frederick Mosteller, "Fifty Challenging Problems in Probability with Solutions", 1965)

"Using data from the population as it stands is a dangerous substitute for testing." (Frederick Mosteller & Gale Mosteller, "New Statistical Methods in Public Policy. Part I: Experimentation", Journal of Contemporary Business 8, 1979)

"Although we often hear that data speak for themselves, their voices can be soft and sly." (Frederick Mosteller, "Beginning Statistics with Data Analysis", 1983)

"The law of truly large numbers states: With a large enough sample, any outrageous thing is likely to happen." (Frederick Mosteller, "Methods for Studying Coincidences", Journal of the American Statistical Association Vol. 84, 1989)

"It is easy to lie with statistics, but easier to lie without them [...]" (Frederick Mosteller)

On Technology III

"Technology means the systematic application of scientific or other organized knowledge to practical tasks." (John K Galbraith, "The New Industrial State", 1967)

"Networks constitute the new social morphology of our societies, and the diffusion of networking logic substantially modifies the operation and outcomes in processes of production, experience, power, and culture. While the networking form of social organization has existed in other times and spaces, the new information technology paradigm provides the material basis for its pervasive expansion throughout the entire social structure." (Manuel Castells, "The Rise of the Network Society", 1996)

"Beauty is more important in computing than anywhere else in technology because software is so complicated. Beauty is the ultimate defense against complexity." (David Gelernter, "Machine Beauty: Elegance And The Heart Of Technolog", 1998)

"Modelling techniques on powerful computers allow us to simulate the behaviour of complex systems without having to understand them.  We can do with technology what we cannot do with science.  […] The rise of powerful technology is not an unconditional blessing.  We have  to deal with what we do not understand, and that demands new  ways of thinking." (Paul Cilliers,"Complexity and Postmodernism: Understanding Complex Systems", 1998)

"A primary reason that evolution - of life-forms or technology - speeds up is that it builds on its own increasing order." (Ray Kurzweil, "The Age of Spiritual Machines: When Computers Exceed Human Intelligence", 1999) 

"As systems became more varied and more complex, we find that no single methodology suffices to deal with them. This is particularly true of what may be called information intelligent systems - systems which form the core of modern technology. To conceive, design, analyze and use such systems we frequently have to employ the totality of tools that are available. Among such tools are the techniques centered on fuzzy logic, neurocomputing, evolutionary computing, probabilistic computing and related methodologies. It is this conclusion that formed the genesis of the concept of soft computing." (Lotfi A Zadeh, "The Birth and Evolution of Fuzzy Logic: A personal perspective", 1999)

"We do not learn much from looking at a model - we learn more from building the model and manipulating it. Just as one needs to use or observe the use of a hammer in order to really understand its function, similarly, models have to be used before they will give up their secrets. In this sense, they have the quality of a technology - the power of the model only becomes apparent in the context of its use." (Margaret Morrison & Mary S Morgan, "Models as mediating instruments", 1999)

"Periods of rapid change and high exponential growth do not, typically, last long. A new equilibrium with a new dominant technology and/or competitor is likely to be established before long. Periods of punctuation are therefore exciting and exhibit unusual uncertainty. The payoff from establishing a dominant position in this short time is therefore extraordinarily high. Dominance is more likely to come from skill in marketing and positioning than from superior technology itself." (Richar Koch, "The Power Laws", 2000)

"Mythology and science both extend the scope of human beings. Like science and technology, mythology, as we shall see, is not about opting out of this world, but about enabling us to live more intensely within it." (Karen Armstrong, "A Short History Of Myth", 2004)

"In an age when technology is integrating us more tightly together and delivering tremendous flows of innovation, knowledge, connectivity and commerce, the future belongs to those who build webs not walls, who can integrate not separate, to get the most out of these flows." (Thomas L Friedman, The New York Times, 2016)

Tipping Point I

"For any given population of susceptibles, there is some critical combination of contact frequency, infectivity, and disease duration just great enough for the positive loop to dominate the negative loops. That threshold is known as the tipping point. Below the tipping point, the system is stable: if the disease is introduced into the community, there may be a few new cases, but on average, people will recover faster than new cases are generated. Negative feedback dominates and the population is resistant to an epidemic. Past the tipping point, the positive loop dominates .The system is unstable and once a disease arrives, it can spread like wildfire that is, by positive feedback-limited only by the depletion of the susceptible population." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"If the contact rate, infectivity, and duration of infection are small enough, the system is below the tipping point and stable. Such a situation is known as herd immunity because the arrival of an infected individual does not produce an epidemic (though a few unlucky individuals may come in contact with any infectious arrivals and contract the disease, the group as a community is protected). However, changes in the contact rate, infectivity, or duration of illness can push a system past the tipping point." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"The existence of the tipping point means it is theoretically possible to completely eradicate a disease. Eradication does not require a perfect vaccine and universal immunization but only the weaker condition that the reproduction rate of the disease fall and remain below one so that new cases arise at a lower rate than old cases are resolved." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world. ", 2000)

"The sharp boundary between an epidemic and stability defined by the tipping point in the deterministic models becomes a probability distribution characterizing the chance an epidemic will occur for any given average rates of interaction, infectivity, and recovery. Likewise, the SI and SIR models assume a homogeneous and well-mixed population, while in reality it is often important to represent subpopulations and the spatial diffusion of an epidemic." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"In the real world, advertising is expensive and does not persist indefinitely. The marketing plan for most new products includes a certain amount for a kickoff ad campaign and other initial marketing efforts. If the product is successful, further advertising can be supported out of the revenues the product generates. If, however, the product does not take off, the marketing budget is soon exhausted and external sources of adoption fall. Advertising is not exogenous, as in the Bass model, but is part of the feedback structure of the system. There is a tipping point for ideas and new products no less than for diseases." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"The tipping point is that magic moment when an idea, trend, or social behavior crosses a threshold, tips, and spreads like wildfire." (Malcolm T Gladwell, "The Tipping Point: How Little Things Can Make a Big Difference", 2000)

"This possibility of sudden change is at the center of the idea of the Tipping Point and might well be the hardest of all to accept. [...] The Tipping Point is the moment of critical mass, the threshold, the boiling point." (Malcolm T Gladwell, "The Tipping Point: How Little Things Can Make a Big Difference", 2000)

"But in mathematics there is a kind of threshold effect, an intellectual tipping point. If a student can just get over the first few humps, negotiate the notational peculiarities of the subject, and grasp that the best way to make progress is to understand the ideas, not just learn them by rote, he or she can sail off merrily down the highway, heading for ever more abstruse and challenging ideas, while an only slightly duller student gets stuck at the geometry of isosceles triangles." (Ian Stewart, "Why Beauty is Truth: A history of symmetry", 2007)

"The product that first gets over its own tipping point attracts many consumers and this may make the competing product less attractive. Being the first to reach this tipping point is very important - more important than being the 'best' in an abstract sense." (David Easley & Jon Kleinberg, "Networks, Crowds, and Markets: Reasoning about a Highly Connected World", 2010)

"Stochastic variability and tipping points in the catch are two different dynamical phenomena. Yet they are both compatible with real-world data [...]" (John D W Morecroft, "Strategic Modelling and Business Dynamics: A Feedback Systems Approach", 2015)

On Imagination (1900-1924)

"This is the greatest degree of impoverishment; the [mental] image, deprived little by little of its own characteristics, is nothing more than a shadow. […] Being dependent on the state of the brain, the image undergoes change like all living substance, - it is subject to gains and losses, especially losses. But each of the foregoing three classes has its use for the inventor. They serve as material for different kinds of imagination - in their concrete form, for the mechanic and the artist; in their schematic form, for the scientist and for others." (Théodule-Armand Ribot, "Essay on the Creative Imagination", 1900)

"This means that it is not a dead thing; it is not at all like a photographic plate with which one may reproduce copies indefinitely. Being dependent on the state of the brain, the image undergoes change like all living substance, - it is subject to gains and losses, especially losses. But each of the foregoing three classes has its use for the inventor. They serve as material for different kinds of imagination - in their concrete form, for the mechanic and the artist; in their schematic form, for the scientist and for others." (Théodule-Armand Ribot, "Essay on the Creative Imagination" , 1900)

"We form in the imagination some sort of diagrammatic, that is, iconic, representation of the facts, as skeletonized as possible. The impression of the present writer is that with ordinary persons this is always a visual image, or mixed visual and muscular; but this is an opinion not founded on any systematic examination." (Charles S Peirce, "Notes on Ampliative Reasoning", 1901)

"Imagination is as vital to any advance in science as learning and precision are essential for starting points." (Percival Lowell, "The Solar System", 1903)

"Nature talks in symbols; he who lacks imagination cannot understand her." (Abraham Miller, "Unmoral Maxims", 1906)

"Mathematics makes constant demands upon the imagination, calls for picturing in space (of one, two, three dimensions), and no considerable success can be attained without a growing ability to imagine all the various possibilities of a given case, and to make them defile before the mind's eye." (Jacob W A Young, "The Teaching of Mathematics", 1907)

"The motive for the study of mathematics is insight into the nature of the universe. Stars and strata, heat and electricity, the laws and processes of becoming and being, incorporate mathematical truths. If language imitates the voice of the Creator, revealing His heart, mathematics discloses His intellect, repeating the story of how things came into being. And the value of mathematics, appealing as it does to our energy and to our honor, to our desire to know the truth and thereby to live as of right in the household of God, is that it establishes us in larger and larger certainties. As literature develops emotion, understanding, and sympathy, so mathematics develops observation, imagination, and reason." (William E Chancellor, "A Theory of Motives, Ideals and Values in Education" 1907)

"The beautiful has its place in mathematics as elsewhere. The prose of ordinary intercourse and of business correspondence might be held to be the most practical use to which language is put, but we should be poor indeed without the literature of imagination. Mathematics too has its triumphs of the Creative imagination, its beautiful theorems, its proofs and processes whose perfection of form has made them classic. He must be a 'practical' man who can see no poetry in mathematics." (Wiliam F White, "A Scrap-book of Elementary Mathematics: Notes, Recreations, Essays", 1908)

"No system would have ever been framed if people had been simply interested in knowing what is true, whatever it may be. What produces systems is the interest in maintaining against all comers that some favourite or inherited idea of ours is sufficient and right. A system may contain an account of many things which, in detail, are true enough; but as a system, covering infinite possibilities that neither our experience nor our logic can prejudge, it must be a work of imagination and a piece of human soliloquy: It may be expressive of human experience, it may be poetical; but how should anyone who really coveted truth suppose that it was true?" (George Santayana, "The Genteel Tradition in American Philosophy", 1911)

"Only in men’s imagination does every truth find an effective and undeniable existence." (Joseph Conrad, "Some Reminiscences", 1912)

"What is the imagination? Only an arm or weapon of the interior energy; only the precursor of the reason." (Ralph W Emerson, "Miscellanies, Natural history of intellect", 1912)

"The concept of an independent system is a pure creation of the imagination. For no material system is or can ever be perfectly isolated from the rest of the world. Nevertheless it completes the mathematician’s ‘blank form of a universe’ without which his investigations are impossible. It enables him to introduce into his geometrical space, not only masses and configurations, but also physical structure and chemical composition." (Lawrence J Henderson, "The Order of Nature: An Essay", 1917)

"[…] because mathematics contains truth, it extends its validity to the whole domain of art and the creatures of the constructive imagination." (James B Shaw, "Lectures on the Philosophy of Mathematics", 1918)

"Nature uses human imagination to lift her work of creation to even higher levels." (Luigi Pirandello, "Six Characters in Search of an Author", 1921)

"The story of scientific discovery has its own epic unity - a unity of purpose and endeavour - the single torch passing from hand to hand through the centuries; and the great moments of science when, after long labour, the pioneers saw their accumulated facts falling into a significant order - sometimes in the form of a law that revolutionised the whole world of thought - have an intense human interest, and belong essentially to the creative imagination of poetry." (Alfred Noyes, "Watchers of the Sky", 1922)

Complex Systems IV

"With the growing interest in complex adaptive systems, artificial life, swarms and simulated societies, the concept of 'collective intelligence' is coming more and more to the fore. The basic idea is that a group of individuals (e. g. people, insects, robots, or software agents) can be smart in a way that none of its members is. Complex, apparently intelligent behavior may emerge from the synergy created by simple interactions between individuals that follow simple rules." (Francis Heylighen, "Collective Intelligence and its Implementation on the Web", 1999)

"A system may be called complex here if its dimension (order) is too high and its model (if available) is nonlinear, interconnected, and information on the system is uncertain such that classical techniques can not easily handle the problem." (M Jamshidi, "Autonomous Control on Complex Systems: Robotic Applications", Current Advances in Mechanical Design and Production VII, 2000)

"Bounded rationality simultaneously constrains the complexity of our cognitive maps and our ability to use them to anticipate the system dynamics. Mental models in which the world is seen as a sequence of events and in which feedback, nonlinearity, time delays, and multiple consequences are lacking lead to poor performance when these elements of dynamic complexity are present. Dysfunction in complex systems can arise from the misperception of the feedback structure of the environment. But rich mental models that capture these sources of complexity cannot be used reliably to understand the dynamics. Dysfunction in complex systems can arise from faulty mental simulation-the misperception of feedback dynamics. These two different bounds on rationality must both be overcome for effective learning to occur. Perfect mental models without a simulation capability yield little insight; a calculus for reliable inferences about dynamics yields systematically erroneous results when applied to simplistic models." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"Much of the art of system dynamics modeling is discovering and representing the feedback processes, which, along with stock and flow structures, time delays, and nonlinearities, determine the dynamics of a system. […] the most complex behaviors usually arise from the interactions (feedbacks) among the components of the system, not from the complexity of the components themselves." (John D Sterman, "Business Dynamics: Systems thinking and modeling for a complex world", 2000)

"To avoid policy resistance and find high leverage policies requires us to expand the boundaries of our mental models so that we become aware of and understand the implications of the feedbacks created by the decisions we make. That is, we must learn about the structure and dynamics of the increasingly complex systems in which we are embedded." (John D Sterman, "Business dynamics: Systems thinking and modeling for a complex world", 2000) 

"Falling between order and chaos, the moment of complexity is the point at which self-organizing systems emerge to create new patterns of coherence and structures of behaviour." (Mark C Taylor, "The Moment of Complexity: Emerging Network Culture", 2001)

"[…] most earlier attempts to construct a theory of complexity have overlooked the deep link between it and networks. In most systems, complexity starts where networks turn nontrivial." (Albert-László Barabási, "Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday Life", 2002)

"[…] networks are the prerequisite for describing any complex system, indicating that complexity theory must inevitably stand on the shoulders of network theory. It is tempting to step in the footsteps of some of my predecessors and predict whether and when we will tame complexity. If nothing else, such a prediction could serve as a benchmark to be disproven. Looking back at the speed with which we disentangled the networks around us after the discovery of scale-free networks, one thing is sure: Once we stumble across the right vision of complexity, it will take little to bring it to fruition. When that will happen is one of the mysteries that keeps many of us going." (Albert-László Barabási, "Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday Life", 2002)

"A sudden change in the evolutive dynamics of a system (a ‘surprise’) can emerge, apparently violating a symmetrical law that was formulated by making a reduction on some (or many) finite sequences of numerical data. This is the crucial point. As we have said on a number of occasions, complexity emerges as a breakdown of symmetry (a system that, by evolving with continuity, suddenly passes from one attractor to another) in laws which, expressed in mathematical form, are symmetrical. Nonetheless, this breakdown happens. It is the surprise, the paradox, a sort of butterfly effect that can highlight small differences between numbers that are very close to one another in the continuum of real numbers; differences that may evade the experimental interpretation of data, but that may increasingly amplify in the system’s dynamics." (Cristoforo S Bertuglia & Franco Vaio, "Nonlinearity, Chaos, and Complexity: The Dynamics of Natural and Social Systems", 2003) 

Peter M Senge - Collected Quotes

"’Mental models’ are deeply ingrained assumptions, generalizations, or even pictures or images that influence how we understand the world and how we take action. Very often, we are not consciously aware of our mental models or the effects they have on our behavior. […] Mental models focus on the openness needed to unearth shortcomings in our present ways of seeing the world. [...] Mental models are deeply held internal images of how the world works, images that limit us to familiar ways of thinking and acting. Very often, we are not consciously aware of our mental models or the effects they have on our behavior." (Peter M Senge, "The Fifth Discipline", 1990)

"Mental models are the images, assumptions, and stories which we carry in our minds of ourselves, other people, institutions, and every aspect of the world. Like a pane of glass framing and subtly distorting our vision, mental models determine what we see. Human beings cannot navigate through the complex environments of our world without cognitive ‘mental maps’; and all of these mental maps, by definition, are flawed in some way." (Peter M Senge, "The Fifth Discipline Fieldbook: Strategies and Tools for Building a Learning Organization", 1994)

"[…] new insights fail to get put into practice because they conflict with deeply held internal images of how the world works [...] images that limit us to familiar ways of thinking and acting. That is why the discipline of managing mental models - surfacing, testing, and improving our internal pictures of how the world works - promises to be a major breakthrough for learning organizations." (Peter M Senge, "The Fifth Discipline: The Art and Practice of the Learning Organization", 1990)

"Systems thinking is a discipline for seeing the 'structures' that underlie complex situations, and for discerning high from low leverage change. That is, by seeing wholes we learn how to foster health. To do so, systems thinking offers a language that begins by restructuring how we think." (Peter M Senge, "The Fifth Discipline: The Art and Practice of the Learning Organization", 1990)

"Systems thinking is a discipline for seeing wholes. It is a framework for seeing interrelationships rather than things, for seeing patterns of change rather than static 'snapshots'. It is a set of general principles- distilled over the course of the twentieth century, spanning fields as diverse as the physical and social sciences, engineering, and management. [...] During the last thirty years, these tools have been applied to understand a wide range of corporate, urban, regional, economic, political, ecological, and even psychological systems. And systems thinking is a sensibility for the subtle interconnectedness that gives living systems their unique character." (Peter M Senge, "The Fifth Discipline: The Art and Practice of the Learning Organization", 1990)

"Systems thinking is a framework for seeing interrelationships rather than things, for seeing patterns rather than static snapshots. It is a set of general principles spanning fields as diverse as physical and social sciences, engineering and management." (Peter M Senge, "The Fifth Discipline: The Art and Practice of the Learning Organization", 1990)

"The problem with mental models lie not in whether they are right or wrong - by definition, all models are simplifications. The problem with mental models arise when they become implicit - when they exist below the level of our awareness. […] models, if unexamined, limit an organization's range of actions to what is familiar and comfortable. [...] Each person's mental model focuses on different parts of the system. Each emphasizes different cause-effect chains. This makes it virtually impossible for a shared picture of the system as a whole to emerge in normal conversation." (Peter M Senge, "The Fifth Discipline: The Art and Practice of the Learning Organization", 1990)

On Networks XI (Neural Networks II)

"The first attempts to consider the behavior of so-called 'random neural nets' in a systematic way have led to a series of problems concerned with relations between the 'structure' and the 'function' of such nets. The 'structure' of a random net is not a clearly defined topological manifold such as could be used to describe a circuit with explicitly given connections. In a random neural net, one does not speak of 'this' neuron synapsing on 'that' one, but rather in terms of tendencies and probabilities associated with points or regions in the net." (Anatol Rapoport, "Cycle distributions in random nets", The Bulletin of Mathematical Biophysics 10(3), 1948)

"The terms 'black box' and 'white box' are convenient and figurative expressions of not very well determined usage. I shall understand by a black box a piece of apparatus, such as four-terminal networks with two input and two output terminals, which performs a definite operation on the present and past of the input potential, but for which we do not necessarily have any information of the structure by which this operation is performed. On the other hand, a white box will be similar network in which we have built in the relation between input and output potentials in accordance with a definite structural plan for securing a previously determined input-output relation." (Norbert Wiener, "Cybernetics: Or Control and Communication in the Animal and the Machine", 1948)

"Neural nets have no central control in the classical sense. Processing is distributed over the network and the roles of the various components (or groups of components) change dynamically.  This does not preclude any part of the network from developing a regulating function, but that will be determined by the evolutionary needs of the system." (Paul Cilliers, "Complexity and Postmodernism: Understanding Complex Systems", 1998)

"Neural networks conserve the complexity of the systems they model because they have complex structures themselves. Neural networks encode information about their environment in a distributed form. […] Neural networks have the capacity to self-organise their internal structure." (Paul Cilliers, "Complexity and Postmodernism: Understanding Complex Systems", 1998)

"The internal structure of a connectionist network develops through a process of self-organisation, whereas rule-based systems have to search through pre-programmed options that define the structure largely in an a priori fashion. In this sense, learning is an implicit characteristic of neural networks. In rule-based systems, learning can only take place through explicitly formulated procedures." (Paul Cilliers, "Complexity and Postmodernism: Understanding Complex Systems", 1998)

"An artificial neural network is a massive parallel distributed processor made up of simple processing units. It has the ability to learn from experiential knowledge expressed through interunit connections strengths, and can make such knowledge available for use." (Yorgos Goletsis et al, "Bankruptcy Prediction through Artificial Intelligence", 2009)

"ANN is a pattern matching technique that uses training data to build a model and uses the model to predict unknown samples. It consists of input, output, and hidden nodes and connections between nodes. The weights of the connections are iteratively adjusted in order to get an accurate model." (Indranil Bose, "Data Mining in Tourism", 2009)

"Just as they did thirty years ago, machine learning programs (including those with deep neural networks) operate almost entirely in an associational mode. They are driven by a stream of observations to which they attempt to fit a function, in much the same way that a statistician tries to fit a line to a collection of points. Deep neural networks have added many more layers to the complexity of the fitted function, but raw data still drives the fitting process. They continue to improve in accuracy as more data are fitted, but they do not benefit from the 'super-evolutionary speedup'."  (Judea Pearl & Dana Mackenzie, "The Book of Why: The new science of cause and effect", 2018)

"[a neural network is] a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs." (Robert Hecht-Nielsen)

Stephen G Haines - Collected Quotes

"Although it is important to understand each individual characteristic, keep in mind that it is the relationship between these parts and characteristics, and their fit into one whole system, that is key. Systems dynamics are all about relationships." (Stephen G Haines, "The Managers Pocket Guide to Systems Thinking & Learning", 1998)

"If life on earth is governed by the natural laws of living systems, then a successful participant should learn the concepts and principles." (Stephen G Haines, 1998)

"In a closed system, the change in entropy must always be 'positive', meaning toward death. However, in open biological or social systems, entropy can be arrested and may even be transformed into negative entropy. - a process of more complete organization and enhanced ability to transform resources. Why? Because the system imports energy and resources from its environment, leading to renewal. This is why education and learning are so important, as they provide new and stimulating input (termed neg-entropy) that can transform each of us." (Stephen G Haines, "The Managers Pocket Guide to Systems Thinking & Learning", 1998)

"Information concerning the system’s outputs or process is fed back into the system as an input, perhaps leading to changes in the transformation process to achieve more effective future outputs. Often this informational input helps us get to the root of problems.
Feedback can be either positive or negative. Positive feedback indicates that the steady state of a system is presently effective. Negative feedback indicates that the system is deviating from a prescribed course and should readjust to a new steady state. Some systems-related field, such as cybernetics, are based on negative feedback." (Stephen G Haines, "The Managers Pocket Guide to Systems Thinking & Learning", 1998)

"[Systems thinking is] A new way to view and mentally frame what we see in the world; a worldview and way of thinking whereby we see the entity or unit first as a whole, with its fit and relationship to its environment as primary concerns." (Stephen G Haines, "The Managers Pocket Guide to Systems Thinking & Learning", 1998)

"The beauty of this [systems thinking] mindset is that its mental models are based on natural laws, principles of interrelationship, and interdependence found in all living systems. They give us a new view of ourselves and our many systems, from the tiniest cell to the entire earth; and as our organizations are included in that great range, they help us define organizational problems as systems problems, so we can respond in more productive ways. The systems thinking mindset is a new orientation to life. In many ways it also operates as a worldview - an overall perspective on, and understanding of, the world." (Stephen G Haines, "The Managers Pocket Guide to Systems Thinking & Learning", 1998)

"The Systems Thinking Approach is an absolute necessity to make sense of and succeed in today’s complex world." (Stephen G Haines, 1998)

"The whole idea of a system is to optimize - not maximize - the fit of its elements in order to maximize the whole. If we merely maximize the elements of systems, we end up suboptimizing the whole [...]" (Stephen G Haines, "The Managers Pocket Guide to Systems Thinking & Learning", 1998)

"Delay time, the time between causes and their impacts, can highly influence systems. Yet the concept of delayed effect is often missed in our impatient society, and when it is recognized, it’s almost always underestimated. Such oversight and devaluation can lead to poor decision making as well as poor problem solving, for decisions often have consequences that don’t show up until years later. Fortunately, mind mapping, fishbone diagrams, and creativity/brainstorming tools can be quite useful here." (Stephen G Haines, "The Manager's Pocket Guide to Strategic and Business Planning", 1998)

"Simple analytic thinking [as opposed to systems thinking] focuses on cause-and-effect: one cause for every one effect. It asks the all too common either/or question. Its weakest link, and the reason it’s not working in today’s world, is that it doesn’t take into consideration the environment, other systems, and the multiple and/or delayed causality that surrounds each cause and effect. Nor does it consider a part’s interrelationships and interdependencies with other parts." (Stephen G Haines, "The Manager's Pocket Guide to Strategic and Business Planning", 1998)

"To begin with, we must understand that any mindset consists of mental models, or concepts, that influence our interpretation of situations and predispose us to certain responses. These models, which are replete with beliefs and assumptions, thus strongly determine the way we understand the world and act in it. The irony is, they become so ingrained in us, as tendencies and predispositions, that we seldom pay attention to them." (Stephen G Haines,  "The Manager's Pocket Guide to Strategic and Business Planning", 1998)

"All systems have a tendency toward maximum entropy, disorder, and death. Importing resources from the environment is key to long-term viability; closed systems move toward this disorganization faster than open systems." (Stephen G Haines, "The Systems Thinking Approach to Strategic Planning and Management", 2000)

"Systems thinking practices the exact opposite of this analytic approach. Systems thinking studies the organization as a whole in its interaction with its environment. Then, it works backwards to understand how each part of that whole works in relation to, and support of, the entire system’s objectives. Only then can the core strategies be formulated." (Stephen G Haines, "The Systems Thinking Approach to Strategic Planning and Management", 2000)

"Systems, and organizations as systems, can only be understood holistically. Try to understand the system and its environment first. Organizations are open systems and, as such, are viable only in interaction with and adaptation to the changing environment." (Stephen G Haines, "The Systems Thinking Approach to Strategic Planning and Management", 2000)

On Networks X (Neural Networks I)

"A neural network is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: 1. Knowledge is acquired by the network through a learning process. 2. Interneuron connection strengths known as synaptic weights are used to store the knowledge." (Igor Aleksander, "An introduction to neural computing", 1990) 

"Neural Computing is the study of networks of adaptable nodes which through a process of learning from task examples, store experiential knowledge and make it available for use." (Igor Aleksander, "An introduction to neural computing", 1990)

"A neural network is characterized by A) its pattern of connections between the neurons (called its architecture), B) its method of determining the weights on the connections (called its training, or learning, algorithm), and C) its activation function." (Laurene Fausett, "Fundamentals of Neural Networks", 1994)

"An artificial neural network is an information-processing system that has certain performance characteristics in common with biological neural networks. Artificial neural networks have been developed as generalizations of mathematical models of human cognition or neural biology, based on the assumptions that: 1. Information processing occurs at many simple elements called neurons. 2. Signals are passed between neurons over connection links. 3. Each connection link has an associated weight, which, in a typical neural net, multiplies the signal transmitted. 4. Each neuron applies an activation function (usually nonlinear) to its net input (sum of weighted input signals) to determine its output signal." (Laurene Fausett, "Fundamentals of Neural Networks", 1994)

"At the other far extreme, we find many systems ordered as a patchwork of parallel operations, very much as in the neural network of a brain or in a colony of ants. Action in these systems proceeds in a messy cascade of interdependent events. Instead of the discrete ticks of cause and effect that run a clock, a thousand clock springs try to simultaneously run a parallel system. Since there is no chain of command, the particular action of any single spring diffuses into the whole, making it easier for the sum of the whole to overwhelm the parts of the whole. What emerges from the collective is not a series of critical individual actions but a multitude of simultaneous actions whose collective pattern is far more important. This is the swarm model." (Kevin Kelly, "Out of Control: The New Biology of Machines, Social Systems and the Economic World", 1995)

"The most familiar example of swarm intelligence is the human brain. Memory, perception and thought all arise out of the nett actions of billions of individual neurons. As we saw earlier, artificial neural networks (ANNs) try to mimic this idea. Signals from the outside world enter via an input layer of neurons. These pass the signal through a series of hidden layers, until the result emerges from an output layer. Each neuron modifies the signal in some simple way. It might, for instance, convert the inputs by plugging them into a polynomial, or some other simple function. Also, the network can learn by modifying the strength of the connections between neurons in different layers." (David G Green, "The Serendipity Machine: A voyage of discovery through the unexpected world of computers", 2004)

"It is not only a metaphor to transform the Internet to a superbrain with self-organizing features of learning and adapting. Information retrieval is already realized by neural networks adapting to the information preferences of a human user with synaptic plasticity. In sociobiology, we can 1 earn from populations of ants and termites how to organize traffic and information processing by swarm intelligence. From a technical point of view, we need intelligent programs distributed in the nets. There are already more or less intelligent virtual organisms {'agents'), learning, self-organizing and adapting to our individual preferences of information, to select our e-mails, to prepare economic transactions or to defend the attacks of hostile computer viruses, like the immune system of our body." (Klaus Mainzer, "Complexity Management in the Age of Globalization", 2006)

"A neural network is a particular kind of computer program, originally developed to try to mimic the way the human brain works. It is essentially a computer simulation of a complex circuit through which electric current flows." (Keith J Devlin & Gary Lorden, "The Numbers behind NUMB3RS: Solving crime with mathematics", 2007)

"A network of many simple processors ('units' or 'neurons') that imitates a biological neural network. The units are connected by unidirectional communication channels, which carry numeric data. Neural networks can be trained to find nonlinear relationships in data, and are used in various applications such as robotics, speech recognition, signal processing, medical diagnosis, or power systems." (Adnan Khashman et al, "Voltage Instability Detection Using Neural Networks", 2009)

"An artificial neural network, often just called a 'neural network' (NN), is an interconnected group of artificial neurons that uses a mathematical model or computational model for information processing based on a connectionist approach to computation. Knowledge is acquired by the network from its environment through a learning process, and interneuron connection strengths (synaptic weighs) are used to store the acquired knowledge." (Larbi Esmahi et al, "Adaptive Neuro-Fuzzy Systems", 2009)

On Self-Organization V

"Clearly, if the state of the system is coupled to parameters of an environment and the state of the environment is made to modify parameters of the system, a learning process will occur. Such an arrangement will be called a Finite Learning Machine, since it has a definite capacity. It is, of course, an active learning mechanism which trades with its surroundings. Indeed it is the limit case of a self-organizing system which will appear in the network if the currency supply is generalized." (Gordon Pask, "The Natural History of Networks", 1960)

"It is inherent in the logical character of the abstract self-organizing system that all available methods of organization are used, and that it cannot be realized in a single reference frame. Thus, any of the tricks which the physical model can perform, such as learning and remembering, may be performed by one or all of a variety of mechanisms, chemical or electrical or mechanical." (Gordon Pask, "The Natural History of Networks", 1960)

"An autopoietic system is organized (defined as a unity) as a network of processes of production (transformation and destruction) of components that produces the components that: (a) through their interactions and transformations continuously regenerate and realize the network of processes (relations) that produce them and, (b) constitute it (the machine) as a concrete unity in the space in which they exist by specifying the topological domain of its realization as such a network." (Francisco Varela, "Principles of Biological Autonomy", 1979)

"Self-organization is seen as the process by which systems of many components tend to reach a particular state, a set of cycling states, or a small volume of their state space (attractor basins), with no external interference." (Luis M Rocha, "Syntactic Autonomy", Proceedings of the Joint Conference on the Science and Technology of Intelligent Systems, 1998) 

"Autopoietic systems, then, are not only self-organizing systems, they not only produce and eventually change their own structures; their self-reference applies to the production of other components as well. This is the decisive conceptual innovation. […] Thus, everything that is used as a unit by the system is produced as a unit by the system itself. This applies to elements, processes, boundaries, and other structures and, last but not least, to the unity of the system itself." (Niklas Luhmann, "The Autopoiesis of Social Systems", 1990)

"The second law of thermodynamics, which requires average entropy (or disorder) to increase, does not in any way forbid local order from arising through various mechanisms of self-organization, which can turn accidents into frozen ones producing extensive regularities. Again, such mechanisms are not restricted to complex adaptive systems." (Murray Gell-Mann, "What is Complexity?", Complexity Vol 1 (1), 1995)

"I propose a new concept based on an interpretation of ecosystems: sympoietic systems. These are complex, self-organizing but collectively producing, boundaryless systems. A subsequent distinction between sympoietic and autopoietic systems is discussed. This distinction arises from defining a difference between three key system characteristics: 1) autopoietic systems have self-defined boundaries, sympoietic systems do not; 2) autopoietic systems are self-produced, sympoietic systems are collectively produced; and, 3) autopoietic systems are organizationally closed, sympoietic systems are organizationally ajar." (Beth Dempster, "Sympoietic and Autopoietic Systems: A New Distinction for Self-Organizing Systems". 2000)

"Emergent self-organization in multi-agent systems appears to contradict the second law of thermodynamics. This paradox has been explained in terms of a coupling between the macro level that hosts self-organization (and an apparent reduction in entropy), and the micro level (where random processes greatly increase entropy). Metaphorically, the micro level serves as an entropy 'sink', permitting overall system entropy to increase while sequestering this increase from the interactions where self-organization is desired." (H Van Dyke Parunak & Sven Brueckner, "Entropy and Self-Organization in Multi-Agent Systems", Proceedings of the International Conference on Autonomous Agents, 2001)

"Nature normally hates power laws. In ordinary systems all quantities follow bell curves, and correlations decay rapidly, obeying exponential laws. But all that changes if the system is forced to undergo a phase transition. Then power laws emerge-nature's unmistakable sign that chaos is departing in favor of order. The theory of phase transitions told us loud and clear that the road from disorder to order is maintained by the powerful forces of self-organization and is paved by power laws. It told us that power laws are not just another way of characterizing a system's behavior. They are the patent signatures of self-organization in complex systems." (Albert-László Barabási, "Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday Life", 2002)

"A self–organizing system acts autonomously, as if the interconnecting components had a single mind. And as these components spontaneously march to the beat of their own drummer, they organize, adapt, and evolve toward a greater complexity than one would ever expect by just looking at the parts by themselves." (Lawrence K Samuels, "Defense of Chaos: The Chaology of Politics, Economics and Human Action", 2013)