On Algorithms: Definitions

"Algorithms are a set of procedures to generate the answer to a problem." (Stuart Kauffman, "At Home in the Universe: The Search for Laws of Complexity", 1995)

"An algorithm is a simple rule, or elementary task, that is repeated over and over again. In this way algorithms can produce structures of astounding complexity." (F David Peat, "From Certainty to Uncertainty", 2002)

"An algorithm refers to a successive and finite procedure by which it is possible to solve a certain problem. Algorithms are the operational base for most computer programs. They consist of a series of instructions that, thanks to programmers’ prior knowledge about the essential characteristics of a problem that must be solved, allow a step-by-step path to the solution." (Diego Rasskin-Gutman, "Chess Metaphors: Artificial Intelligence and the Human Mind", 2009) 

"[...] algorithms, which are abstract or idealized process descriptions that ignore details and practicalities. An algorithm is a precise and unambiguous recipe. It’s expressed in terms of a fixed set of basic operations whose meanings are completely known and specified. It spells out a sequence of steps using those operations, with all possible situations covered, and it’s guaranteed to stop eventually." (Brian W Kernighan, "Understanding the Digital World", 2017)

"An algorithm is the computer science version of a careful, precise, unambiguous recipe or tax form, a sequence of steps that is guaranteed to compute a result correctly." (Brian W Kernighan, "Understanding the Digital World", 2017)

"Algorithms describe the solution to a problem in terms of the data needed to represent the  problem instance and a set of steps necessary to produce the intended result." (Bradley N Miller et al, "Python Programming in Context", 2019)

"An algorithm, meanwhile, is a step-by-step recipe for performing a series of actions, and in most cases 'algorithm' means simply 'computer program'." (Tim Harford, "The Data Detective: Ten easy rules to make sense of statistics", 2020)

Brian Christian - Collected Quotes

"As with all issues involving overfitting, how early to stop depends on the gap between what you can measure and what really matters. If you have all the facts, they’re free of all error and uncertainty, and you can directly assess whatever is important to you, then don’t stop early. Think long and hard: the complexity and effort are appropriate." (Brian Christian & Thomas L Griffiths, "Algorithms to Live By: The Computer Science of Human Decisions", 2016)

"Bayes’s Rule tells us that when it comes to making predictions based on limited evidence, few things are as important as having good priors - that is, a sense of the distribution from which we expect that evidence to have come. Good predictions thus begin with having good instincts about when we’re dealing with a normal distribution and when with a power-law distribution. As it turns out, Bayes’s Rule offers us a simple but dramatically different predictive rule of thumb for each." (Brian Christian & Thomas L Griffiths, "Algorithms to Live By: The Computer Science of Human Decisions", 2016)

"Game theory covers an incredibly broad spectrum of scenarios of cooperation and competition, but the field began with those resembling heads-up poker: two-person contests where one player’s gain is another player’s loss. Mathematicians analyzing these games seek to identify a so-called equilibrium: that is, a set of strategies that both players can follow such that neither player would want to change their own play, given the play of their opponent. It’s called an equilibrium because it’s stable—no amount of further reflection by either player will bring them to different choices. I’m content with my strategy, given yours, and you’re content with your strategy, given mine." (Brian Christian & Thomas L Griffiths, "Algorithms to Live By: The Computer Science of Human Decisions", 2016)

"In other words, overfitting poses a danger any time we’re dealing with noise or mismeasurement—and we almost always are. There can be errors in how the data were collected, or in how they were reported. Sometimes the phenomena being investigated, such as human happiness, are hard to even define, let alone measure. Thanks to their flexibility, the most complex models available to us can fit any patterns that appear in the data, but this means that they will also do so even when those patterns are mere phantoms and mirages in the noise." (Brian Christian & Thomas L Griffiths, "Algorithms to Live By: The Computer Science of Human Decisions", 2016)

"Many prediction algorithms, for instance, start out by searching for the single most important factor rather than jumping to a multi-factor model. Only after finding that first factor do they look for the next most important factor to add to the model, then the next, and so on. Their models can therefore be kept from becoming overly complex simply by stopping the process short, before overfitting has had a chance to creep in. A related approach to calculating predictions considers one data point at a time, with the model tweaked to account for each new point before more points are added; there, too, the complexity of the model increases gradually, so stopping the process short can help keep it from overfitting." (Brian Christian & Thomas L Griffiths, "Algorithms to Live By: The Computer Science of Human Decisions", 2016)

"Randomness seems like the opposite of reason - a form of giving up on a problem, a last resort. Far from it. The surprising and increasingly important role of randomness in computer science shows us that making use of chance can be a deliberate and effective part of approaching the hardest sets of problems. In fact, there are times when nothing else will do."  (Brian Christian & Thomas L Griffiths, "Algorithms to Live By: The Computer Science of Human Decisions", 2016)

"The effectiveness of regularization in all kinds of machine-learning tasks suggests that we can make better decisions by deliberately thinking and doing less. If the factors we come up with first are likely to be the most important ones, then beyond a certain point thinking more about a problem is not only going to be a waste of time and effort - it will lead us to worse solutions. Early Stopping provides the foundation for a reasoned argument against reasoning, the thinking person’s case against thought. But turning this into practical advice requires answering one more question: when should we stop thinking?" (Brian Christian & Thomas L Griffiths, "Algorithms to Live By: The Computer Science of Human Decisions", 2016)

"The greater the uncertainty, the bigger the gap between what you can measure and what matters, the more you should watch out for overfitting - that is, the more you should prefer simplicity." (Brian Christian & Thomas L Griffiths, "Algorithms to Live By: The Computer Science of Human Decisions", 2016)

On Algorithms I

"Mathematics is an aspect of culture as well as a collection of algorithms." (Carl B Boyer, "The History of the Calculus and Its Conceptual Development", 1959)

"An algorithm must be seen to be believed, and the best way to learn what an algorithm is all about is to try it." (Donald E Knuth, The Art of Computer Programming Vol. I, 1968)

"Scientific laws give algorithms, or procedures, for determining how systems behave. The computer program is a medium in which the algorithms can be expressed and applied. Physical objects and mathematical structures can be represented as numbers and symbols in a computer, and a program can be written to manipulate them according to the algorithms. When the computer program is executed, it causes the numbers and symbols to be modified in the way specified by the scientific laws. It thereby allows the consequences of the laws to be deduced." (Stephen Wolfram, "Computer Software in Science and Mathematics", 1984)

"Algorithmic complexity theory and nonlinear dynamics together establish the fact that determinism reigns only over a quite finite domain; outside this small haven of order lies a largely uncharted, vast wasteland of chaos." (Joseph Ford, "Progress in Chaotic Dynamics: Essays in Honor of Joseph Ford's 60th Birthday", 1988)

"On this view, we recognize science to be the search for algorithmic compressions. We list sequences of observed data. We try to formulate algorithms that compactly represent the information content of those sequences. Then we test the correctness of our hypothetical abbreviations by using them to predict the next terms in the string. These predictions can then be compared with the future direction of the data sequence. Without the development of algorithmic compressions of data all science would be replaced by mindless stamp collecting - the indiscriminate accumulation of every available fact. Science is predicated upon the belief that the Universe is algorithmically compressible and the modern search for a Theory of Everything is the ultimate expression of that belief, a belief that there is an abbreviated representation of the logic behind the Universe's properties that can be written down in finite form by human beings." (John D Barrow, New Theories of Everything", 1991)

"Algorithms are a set of procedures to generate the answer to a problem." (Stuart Kauffman, "At Home in the Universe: The Search for Laws of Complexity", 1995)

"Let us regard a proof of an assertion as a purely mechanical procedure using precise rules of inference starting with a few unassailable axioms. This means that an algorithm can be devised for testing the validity of an alleged proof simply by checking the successive steps of the argument; the rules of inference constitute an algorithm for generating all the statements that can be deduced in a finite number of steps from the axioms." (Edward Beltrami, "What is Random?: Chaos and Order in Mathematics and Life", 1999)

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

"An algorithm is a simple rule, or elementary task, that is repeated over and over again. In this way algorithms can produce structures of astounding complexity." (F David Peat, "From Certainty to Uncertainty", 2002)

"Many people have strong intuitions about whether they would rather have a vital decision about them made by algorithms or humans. Some people are touchingly impressed by the capabilities of the algorithms; others have far too much faith in human judgment. The truth is that sometimes the algorithms will do better than the humans, and sometimes they won’t. If we want to avoid the problems and unlock the promise of big data, we’re going to need to assess the performance of the algorithms on a case-by-case basis. All too often, this is much harder than it should be. […] So the problem is not the algorithms, or the big datasets. The problem is a lack of scrutiny, transparency, and debate." (Tim Harford, "The Data Detective: Ten easy rules to make sense of statistics", 2020)

On Algorithms II

"The vast majority of information that we have on most processes tends to be nonnumeric and nonalgorithmic. Most of the information is fuzzy and linguistic in form." (Timothy J Ross & W Jerry Parkinson, "Fuzzy Set Theory, Fuzzy Logic, and Fuzzy Systems", 2002)

"Knowledge is encoded in models. Models are synthetic sets of rules, and pictures, and algorithms providing us with useful representations of the world of our perceptions and of their patterns." (Didier Sornette, "Why Stock Markets Crash - Critical Events in Complex Systems", 2003)

"Swarm Intelligence can be defined more precisely as: Any attempt to design algorithms or distributed problem-solving methods inspired by the collective behavior of the social insect colonies or other animal societies. The main properties of such systems are flexibility, robustness, decentralization and self-organization." ("Swarm Intelligence in Data Mining", Ed. Ajith Abraham et al, 2006)

"The burgeoning field of computer science has shifted our view of the physical world from that of a collection of interacting material particles to one of a seething network of information. In this way of looking at nature, the laws of physics are a form of software, or algorithm, while the material world - the hardware - plays the role of a gigantic computer." (Paul C W Davies, "Laying Down the Laws", New Scientist, 2007)

"An algorithm refers to a successive and finite procedure by which it is possible to solve a certain problem. Algorithms are the operational base for most computer programs. They consist of a series of instructions that, thanks to programmers’ prior knowledge about the essential characteristics of a problem that must be solved, allow a step-by-step path to the solution." (Diego Rasskin-Gutman, "Chess Metaphors: Artificial Intelligence and the Human Mind", 2009)

"Programming is a science dressed up as art, because most of us don’t understand the physics of software and it’s rarely, if ever, taught. The physics of software is not algorithms, data structures, languages, and abstractions. These are just tools we make, use, and throw away. The real physics of software is the physics of people. Specifically, it’s about our limitations when it comes to complexity and our desire to work together to solve large problems in pieces. This is the science of programming: make building blocks that people can understand and use easily, and people will work together to solve the very largest problems." (Pieter Hintjens, "ZeroMQ: Messaging for Many Applications", 2012)

"These nature-inspired algorithms gradually became more and more attractive and popular among the evolutionary computation research community, and together they were named swarm intelligence, which became the little brother of the major four evolutionary computation algorithms." (Yuhui Shi, "Emerging Research on Swarm Intelligence and Algorithm Optimization", Information Science Reference, 2014)

"Again, classical statistics only summarizes data, so it does not provide even a language for asking [a counterfactual] question. Causal inference provides a notation and, more importantly, offers a solution. As with predicting the effect of interventions [...], in many cases we can emulate human retrospective thinking with an algorithm that takes what we know about the observed world and produces an answer about the counterfactual world." (Judea Pearl & Dana Mackenzie, "The Book of Why: The new science of cause and effect", 2018)

"An algorithm, meanwhile, is a step-by-step recipe for performing a series of actions, and in most cases 'algorithm' means simply 'computer program'." (Tim Harford, "The Data Detective: Ten easy rules to make sense of statistics", 2020)

"Big data is revolutionizing the world around us, and it is easy to feel alienated by tales of computers handing down decisions made in ways we don’t understand. I think we’re right to be concerned. Modern data analytics can produce some miraculous results, but big data is often less trustworthy than small data. Small data can typically be scrutinized; big data tends to be locked away in the vaults of Silicon Valley. The simple statistical tools used to analyze small datasets are usually easy to check; pattern-recognizing algorithms can all too easily be mysterious and commercially sensitive black boxes." (Tim Harford, "The Data Detective: Ten easy rules to make sense of statistics", 2020)

"Each of us is sweating data, and those data are being mopped up and wrung out into oceans of information. Algorithms and large datasets are being used for everything from finding us love to deciding whether, if we are accused of a crime, we go to prison before the trial or are instead allowed to post bail. We all need to understand what these data are and how they can be exploited." (Tim Harford, "The Data Detective: Ten easy rules to make sense of statistics", 2020)

David Berlinski - Collected Quotes

"The body of mathematics to which the calculus gives rise embodies a certain swashbuckling style of thinking, at once bold and dramatic, given over to large intellectual gestures and indifferent, in large measure, to any very detailed description of the world. It is a style that has shaped the physical but not the biological sciences, and its success in Newtonian mechanics, general relativity and quantum mechanics is among the miracles of mankind. But the era in thought that the calculus made possible is coming to an end. Everyone feels this is so and everyone is right." (David Berlinski, "A Tour of the Calculus", 1995)

"Yet everything has a beginning, everything comes to an end, and if the universe actually began in some dense explosion, thus creating time and space, so time and space are themselves destined to disappear, the measure vanishing with the measured, until with another ripple running through the primordial quantum field, something new arises from nothingness once again." (David Berlinski, "A Tour of the Calculus", 1995)

"The motion of the mind is conveyed along a cloud of meaning. There is this paradox that we get to meaning only when we strip the meaning from symbols." (David Berlinski, "The Advent of the Algorithm: The Idea that Rules the World", 2000)

"A definition in mathematics is an exercise in uncovering the essence of things, one reason that good definitions are so hard to pull off, since a definition brings the essence to light, and the light brings the definition to life." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"A five-dimensional space is not a strange deformation of ordinary space, one that only mathematicians can see, but a place where numbers are collected in ordered sets. When string theorists talk of the eleven dimensions required by their latest theory, they are not encouraging one another to search for eight otherwise familiar spatial dimensions that have somehow become lost. They are saying only that for their purposes, eleven numbers are needed to specify points. Where they are is no one’s business." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005) 

"A group is a collection of objects, one that is alive in the sense that some underlying principle of productivity is at work engendering new members from old. […] Like many other highly structured objects, groups have parts, and in particular they may well have subgroups as parts, one group nested within a large group, kangarette to kangaroo." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"A proof in mathematics is an argument and so falls under the controlling power of logic itself. […] Within mathematics, a proof is an intellectual structure in which premises are conveyed to their conclusions by specific inferential steps. Assumptions in mathematics are called axioms, and conclusions theorems. This definition may be sharpened a little bit. A proof is a finite series of statements such that every statement is either an axiom or follows directly from an axiom by means of tight, narrowly defined rules. The mathematician’s business is to derive theorems from his axioms; if his system has been carefully constructed, a gross cascade of theorems will flow from a collection of carefully chosen axioms." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"Beyond the theory of complex numbers, there is the much greater and grander theory of the functions of a complex variable, as when the complex plane is mapped to the complex plane, complex numbers linking themselves to other complex numbers. It is here that complex differentiation and integration are defined. Every mathematician in his education studies this theory and surrenders to it completely. The experience is like first love." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"But like every profound mathematical idea, the concept of a group reveals something about the nature of the world that lies beyond the mathematician’s symbols. […] There is […] a royal road between group theory and the most fundamental processes in nature. Some groups represent- they are reflections of - continuous rotations, things that whiz around and around smoothly." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"If the method of proof offers the mathematician the prospect of certainty, it is a form of certainty that is itself conditional. A proof, after all, conveys assumptions to conclusions, or axioms to theorems. If the hammer of certainty falls on the theorems, it cannot fall on the axioms with equal force." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"Mathematics is insight and invention and the flash of something grasped at once, but it is also something salt-cleaned and stout as a Gothic cathedral." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"Practical geometry is an empirical undertaking, living and breathing and sweating in the real world where measurements are always approximate and things are fudged or smeared or jumbled up. Within Euclidean geometry points are concentrated, lines straightened, angles narrowed; idealizations are made, and some parts of experience discarded and other parts embraced." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"Set theory is unusual in that it deals with remarkably simple but apparently ineffable objects. A set is a collection, a class, an ensemble, a batch, a bunch, a lot, a troop, a tribe. To anyone incapable of grasping the concept of a set, these verbal digressions are apt to be of little help. […] A set may contain finitely many or infinitely many members. For that matter, a set such as {} may contain no members whatsoever, its parentheses vibrating around a mathematical black hole. To the empty set is reserved the symbol Ø, the figure now in use in daily life to signify access denied or don’t go, symbolic spillovers, I suppose, from its original suggestion of a canceled eye." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"The calculus is a theory of continuous change - processes that move smoothly and that do not stop, jerk, interrupt themselves, or hurtle over gaps in space and time. The supreme example of a continuous process in nature is represented by the motion of the planets in the night sky as without pause they sweep around the sun in elliptical orbits; but human consciousness is also continuous, the division of experience into separate aspects always coordinated by some underlying form of unity, one that we can barely identify and that we can describe only by calling it continuous." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"There is weirdness in non-Euclidean geometry, but not because of anything that geometers might say about the ordinary fond familiar world in which space is flat, angles sharp, and only curves are curved. Non-Euclidean geometry is an instrument in the enlargement of the mathematician’s self-consciousness, and so comprises an episode in a long, difficult, and extended exercise in which the human mind attempts to catch sight of itself catching sight of itself, and so without end." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"What a wealth of insight Euler’s formula reveals and what delicacy and precision of reasoning it exhibits. It provides a definition of complex exponentiation: It is a definition of complex exponentiation, but the definition proceeds in the most natural way, like a trained singer’s breath. It closes the complex circle once again by guaranteeing that in taking complex numbers to complex powers the mathematician always returns with complex numbers. It justifies the method of infinite series and sums. And it exposes that profound and unsuspected connection between exponential and trigonometric functions; with Euler’s formula the very distinction between trigonometric and exponential functions acquires the shimmer of a desert illusion." (David Berlinski, "Infinite Ascent: A short history of mathematics", 2005)

"Mathematicians are like pilots who maneuver their great lumbering planes into the sky without ever asking how the damn things stay aloft. […] The computer has in turn changed the very nature of mathematical experience, suggesting for the first time that mathematics, like physics, may yet become an empirical discipline, a place where things are discovered because they are seen. [...] The existence and nature of mathematics is a more compelling and far deeper problem than any of the problems raised by mathematics itself." (David Berlinski)

On Heuristics II

"Models of bounded rationality describe how a judgement or decision is reached (that is, the heuristic processes or proximal mechanisms) rather than merely the outcome of the decision, and they describe the class of environments in which these heuristics will succeed or fail." (Gerd Gigerenzer & Reinhard Selten [Eds., "Bounded Rationality: The Adaptive Toolbox", 2001)

"A second class of metaphors - mathematical algorithms, heuristics, and models - brings us closer to the world of computer science programs, simulations, and approximations of the brain and its cognitive processes." (Diego Rasskin-Gutman, "Chess Metaphors: Artificial Intelligence and the Human Mind", 2009)

"In particular, the accurate intuitions of experts are better explained by the effects of prolonged practice than by heuristics. We can now draw a richer and more balanced picture, in which skill and heuristics are alternative sources of intuitive judgments and choices." (Daniel Kahneman, "Thinking, Fast and Slow", 2011)

"This is the essence of intuitive heuristics: when faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution." (Daniel Kahneman, "Thinking, Fast and Slow", 2011)

"Heuristics are an evolutionary solution to an ongoing problem: we have limited mental resources. As such, they have a very long and thoroughly time-tested history of helping us - on average - make better decisions." (Peter H Diamandis, "Abundance: The Future is Better Than You Think", 2012)

"Heuristics are simplified rules of thumb that make things simple and easy to implement. But their main advantage is that the user knows that they are not perfect, just expedient, and is therefore less fooled by their powers. They become dangerous when we forget that." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"The art of reasoned persuasion is an iterative, recursive heuristic, meaning that we must go back and forth between the facts and the rules until we have a good fit. We cannot see the facts properly until we know what framework to place them into, and we cannot determine what framework to place them into until we see the basic contours of the facts." (Joel P Trachtman, "The Tools of Argument", 2013)

"Heuristic decision making is fast and frugal and is often based on the evaluation of one or two salient bits of information." (Amitav Chakravarti, "Why People (Don’t) Buy: The Go and Stop Signals", 2015)

"A heuristic is a strategy we derive from previous experience with a similar problem." (Darius Foroux, "Think Straight", 2017)

"The social world that humans have made for themselves is so complex that the mind simplifies the world by using heuristics, customs, and habits, and by making models or assumptions about how things generally work (the ‘causal structure of the world’). And because people rely upon (and are invested in) these mental models, they usually prefer that they remain uncontested." (Dr James Brennan, "Psychological  Adjustment to Illness and Injury", West of England Medical Journal Vol. 117 (2), 2018)