On Distributions III

"Linear regression assumes that in the population a normal distribution of error values around the predicted Y is associated with each X value, and that the dispersion of the error values for each X value is the same. The assumptions imply normal and similarly dispersed error distributions." (Fred C Pampel, "Linear Regression: A primer", 2000)

"The principle of maximum entropy is employed for estimating unknown probabilities (which cannot be derived deductively) on the basis of the available information. According to this principle, the estimated probability distribution should be such that its entropy reaches maximum within the constraints of the situation, i.e., constraints that represent the available information. This principle thus guarantees that no more information is used in estimating the probabilities than available." (George J Klir & Doug Elias, "Architecture of Systems Problem Solving" 2nd Ed, 2003) 

"The principle of minimum entropy is employed in the formulation of resolution forms and related problems. According to this principle, the entropy of the estimated probability distribution, conditioned by a particular classification of the given events (e.g., states of the variable involved), is minimum subject to the constraints of the situation. This principle thus guarantees that all available information is used, as much as possible within the given constraints (e.g., required number of states), in the estimation of the unknown probabilities." (George J Klir & Doug Elias, "Architecture of Systems Problem Solving" 2nd Ed, 2003)

"First, if you already know that the population from which your sample has been taken is normally distributed (perhaps you have data for a variable that has been studied before), you can assume the distribution of sample means from this population will also be normally distributed. Second, the central limit theorem […] states that the distribution of the means of samples of about 25 or more taken from any population will be approximately normal, provided the population is not grossly non-normal (e.g. a population that is bimodal). Therefore, provided your sample size is sufficiently large you can usually do a parametric test. Finally, you can examine your sample. Although there are statistical tests for normality, many statisticians have cautioned that these tests often indicate the sample is significantly non normal even when a t-test will still give reliable results." (Steve McKillup, "Statistics Explained: An Introductory Guide for Life Scientists", 2005)

"In the laws of probability theory, likelihood distributions are fixed properties of a hypothesis. In the art of rationality, to explain is to anticipate. To anticipate is to explain." (Eliezer S. Yudkowsky, "A Technical Explanation of Technical Explanation", 2005)

"Linear correlation analysis assumes that the data are random representatives taken from the larger population of values for each variable, which are normally distributed and have been measured on a ratio, interval or ordinal scale. A scatter plot of these variables will have what is called a bivariate normal distribution. If the data are not normally distributed, or the relationship does not appear to be linear, they may be able to be analysed by nonparametric tests for correlation [...]" (Steve McKillup, "Statistics Explained: An Introductory Guide for Life Scientists", 2005)

"The central limit theorem says that, under conditions almost always satisfied in the real world of experimentation, the distribution of such a linear function of errors will tend to normality as the number of its components becomes large. The tendency to normality occurs almost regardless of the individual distributions of the component errors. An important proviso is that several sources of error must make important contributions to the overall error and that no particular source of error dominate the rest." (George E P Box et al, "Statistics for Experimenters: Design, discovery, and innovation" 2^nd Ed., 2005)

"Two things explain the importance of the normal distribution: (1) The central limit effect that produces a tendency for real error distributions to be 'normal like'. (2) The robustness to nonnormality of some common statistical procedures, where 'robustness' means insensitivity to deviations from theoretical normality." (George E P Box et al, "Statistics for Experimenters: Design, discovery, and innovation" 2^nd Ed., 2005)

"Traditional statistics is strong in devising ways of describing data and inferring distributional parameters from sample. Causal inference requires two additional ingredients: a science-friendly language for articulating causal knowledge, and a mathematical machinery for processing that knowledge, combining it with data and drawing new causal conclusions about a phenomenon." (Judea Pearl, "Causal inference in statistics: An overview", Statistics Surveys 3, 2009)

"The elements of this cloud of uncertainty (the set of all possible errors) can be described in terms of probability. The center of the cloud is the number zero, and elements of the cloud that are close to zero are more probable than elements that are far away from that center. We can be more precise in this definition by defining the cloud of uncertainty in terms of a mathematical function, called the probability distribution." (David S Salsburg, "Errors, Blunders, and Lies: How to Tell the Difference", 2017)

On Distributions I

"The state of a system at a given moment depends on two things - its initial state, and the law according to which that state varies. If we know both this law and this initial state, we have a simple mathematical problem to solve, and we fall back upon our first degree of ignorance. Then it often happens that we know the law and do not know the initial state. It may be asked, for instance, what is the present distribution of the minor planets? We know that from all time they have obeyed the laws of Kepler, but we do not know what was their initial distribution. In the kinetic theory of gases we assume that the gaseous molecules follow recti-linear paths and obey the laws of impact and elastic bodies; yet as we know nothing of their initial velocities, we know nothing of their present velocities. The calculus of probabilities alone enables us to predict the mean phenomena which will result from a combination of these velocities. This is the second degree of ignorance. Finally it is possible, that not only the initial conditions but the laws themselves are unknown. We then reach the third degree of ignorance, and in general we can no longer affirm anything at all as to the probability of a phenomenon. It often happens that instead of trying to discover an event by means of a more or less imperfect knowledge of the law, the events may be known, and we want to find the law; or that, instead of deducing effects from causes, we wish to deduce the causes."  (Henri Poincaré, "Science and Hypothesis", 1902)

"If the number of experiments be very large, we may have precise information as to the value of the mean, but if our sample be small, we have two sources of uncertainty: (I) owing to the 'error of random sampling' the mean of our series of experiments deviates more or less widely from the mean of the population, and (2) the sample is not sufficiently large to determine what is the law of distribution of individuals." (William S Gosset, "The Probable Error of a Mean", Biometrika, 1908)

"We know not to what are due the accidental errors, and precisely because we do not know, we are aware they obey the law of Gauss. Such is the paradox." (Henri Poincaré, "The Foundations of Science", 1913)

"The problems which arise in the reduction of data may thus conveniently be divided into three types: (i) Problems of Specification, which arise in the choice of the mathematical form of the population. (ii) When a specification has been obtained, problems of Estimation arise. These involve the choice among the methods of calculating, from our sample, statistics fit to estimate the unknow n parameters of the population. (iii) Problems of Distribution include the mathematical deduction of the exact nature of the distributions in random samples of our estimates of the parameters, and of other statistics designed to test the validity of our specification (tests of Goodness of Fit)." (Sir Ronald A Fisher, "Statistical Methods for Research Workers", 1925)

"An inference, if it is to have scientific value, must constitute a prediction concerning future data. If the inference is to be made purely with the help of the distribution theory of statistics, the experiments that constitute evidence for the inference must arise from a state of statistical control; until that state is reached, there is no universe, normal or otherwise, and the statistician’s calculations by themselves are an illusion if not a delusion. The fact is that when distribution theory is not applicable for lack of control, any inference, statistical or otherwise, is little better than a conjecture. The state of statistical control is therefore the goal of all experimentation. (William E Deming, "Statistical Method from the Viewpoint of Quality Control", 1939)

"Just as in applied statistics the crux of a problem is often the devising of some method of sampling that avoids bias, our problem is that of finding a probability assignment which avoids bias, while agreeing with whatever information is given. The great advance provided by information theory lies in the discovery that there is a unique, unambiguous criterion for the 'amount of uncertainty' represented by a discrete probability distribution, which agrees with our intuitive notions that a broad distribution represents more uncertainty than does a sharply peaked one, and satisfies all other conditions which make it reasonable." (Edwin T Jaynes, "Information Theory and Statistical Mechanics" I, 1956)

"Normality is a myth; there never has, and never will be, a normal distribution." (Roy C Geary, "Testing for Normality", Biometrika Vol. 34, 1947)

"[A] sequence is random if it has every property that is shared by all infinite sequences of independent samples of random variables from the uniform distribution." (Joel N Franklin, 1962)

"Mathematical statistics provides an exceptionally clear example of the relationship between mathematics and the external world. The external world provides the experimentally measured distribution curve; mathematics provides the equation (the mathematical model) that corresponds to the empirical curve. The statistician may be guided by a thought experiment in finding the corresponding equation." (Marshall J Walker, "The Nature of Scientific Thought", 1963)

On Distributions II

"Pencil and paper for construction of distributions, scatter diagrams, and run-charts to compare small groups and to detect trends are more efficient methods of estimation than statistical inference that depends on variances and standard errors, as the simple techniques preserve the information in the original data." (William E Deming, "On Probability as Basis for Action" American Statistician Vol. 29 (4), 1975)

"When the statistician looks at the outside world, he cannot, for example, rely on finding errors that are independently and identically distributed in approximately normal distributions. In particular, most economic and business data are collected serially and can be expected, therefore, to be heavily serially dependent. So is much of the data collected from the automatic instruments which are becoming so common in laboratories these days. Analysis of such data, using procedures such as standard regression analysis which assume independence, can lead to gross error. Furthermore, the possibility of contamination of the error distribution by outliers is always present and has recently received much attention. More generally, real data sets, especially if they are long, usually show inhomogeneity in the mean, the variance, or both, and it is not always possible to randomize." (George E P Box, "Some Problems of Statistics and Everyday Life", Journal of the American Statistical Association, Vol. 74 (365), 1979)

"Continuous distributions are basic to the theory of probability and statistics, and the calculus is necessary to handle them with any ease." (Richard Hamming, "Methods of Mathematics Applied to Calculus, Probability, and Statistics", 1985)

"We will use the convenient expression 'chosen at random' to mean that the probabilities of the events in the sample space are all the same unless some modifying words are near to the words 'at random'. Usually we will compute the probability of the outcome based on the uniform probability model since that is very common in modeling simple situations. However, a uniform distribution does not imply that it comes from a random source; […]" (Richard W Hamming, "The Art of Probability for Scientists and Engineers", 1991)

"Data that are skewed toward large values occur commonly. Any set of positive measurements is a candidate. Nature just works like that. In fact, if data consisting of positive numbers range over several powers of ten, it is almost a guarantee that they will be skewed. Skewness creates many problems. There are visualization problems. A large fraction of the data are squashed into small regions of graphs, and visual assessment of the data degrades. There are characterization problems. Skewed distributions tend to be more complicated than symmetric ones; for example, there is no unique notion of location and the median and mean measure different aspects of the distribution. There are problems in carrying out probabilistic methods. The distribution of skewed data is not well approximated by the normal, so the many probabilistic methods based on an assumption of a normal distribution cannot be applied." (William S Cleveland, "Visualizing Data", 1993)

"Fitting data means finding mathematical descriptions of structure in the data. An additive shift is a structural property of univariate data in which distributions differ only in location and not in spread or shape. […] The process of identifying a structure in data and then fitting the structure to produce residuals that have the same distribution lies at the heart of statistical analysis. Such homogeneous residuals can be pooled, which increases the power of the description of the variation in the data." (William S Cleveland, "Visualizing Data", 1993)

"Many good things happen when data distributions are well approximated by the normal. First, the question of whether the shifts among the distributions are additive becomes the question of whether the distributions have the same standard deviation; if so, the shifts are additive. […] A second good happening is that methods of fitting and methods of probabilistic inference, to be taken up shortly, are typically simple and on well understood ground. […] A third good thing is that the description of the data distribution is more parsimonious." (William S Cleveland, "Visualizing Data", 1993)

"Probabilistic inference is the classical paradigm for data analysis in science and technology. It rests on a foundation of randomness; variation in data is ascribed to a random process in which nature generates data according to a probability distribution. This leads to a codification of uncertainly by confidence intervals and hypothesis tests." (William S Cleveland, "Visualizing Data", 1993)

"When distributions are compared, the goal is to understand how the distributions shift in going from one data set to the next. […] The most effective way to investigate the shifts of distributions is to compare corresponding quantiles." (William S Cleveland, "Visualizing Data", 1993)

"When the distributions of two or more groups of univariate data are skewed, it is common to have the spread increase monotonically with location. This behavior is monotone spread. Strictly speaking, monotone spread includes the case where the spread decreases monotonically with location, but such a decrease is much less common for raw data. Monotone spread, as with skewness, adds to the difficulty of data analysis. For example, it means that we cannot fit just location estimates to produce homogeneous residuals; we must fit spread estimates as well. Furthermore, the distributions cannot be compared by a number of standard methods of probabilistic inference that are based on an assumption of equal spreads; the standard t-test is one example. Fortunately, remedies for skewness can cure monotone spread as well." (William S Cleveland, "Visualizing Data", 1993)

"A normal distribution is most unlikely, although not impossible, when the observations are dependent upon one another - that is, when the probability of one event is determined by a preceding event. The observations will fail to distribute themselves symmetrically around the mean." (Peter L Bernstein, "Against the Gods: The Remarkable Story of Risk", 1996)

On Sampling III

"The fact must be expressed as data, but there is a problem in that the correct data is difficult to catch. So that I always say 'When you see the data, doubt it!' 'When you see the measurement instrument, doubt it!' [...]For example, if the methods such as sampling, measurement, testing and chemical analysis methods were incorrect, data. […] to measure true characteristics and in an unavoidable case, using statistical sensory test and express them as data." (Kaoru Ishikawa, Annual Quality Congress Transactions, 1981)

"The law of truly large numbers states: With a large enough sample, any outrageous thing is likely to happen." (Frederick Mosteller, "Methods for Studying Coincidences", Journal of the American Statistical Association Vol. 84, 1989)

"A little thought reveals a fact widely understood among statisticians: The null hypothesis, taken literally (and that’s the only way you can take it in formal hypothesis testing), is always false in the real world. [...] If it is false, even to a tiny degree, it must be the case that a large enough sample will produce a significant result and lead to its rejection. So if the null hypothesis is always false, what’s the big deal about rejecting it?" (Jacob Cohen,"Things I Have Learned (So Far)", American Psychologist, 1990)

"When looking at the end result of any statistical analysis, one must be very cautious not to over interpret the data. Care must be taken to know the size of the sample, and to be certain the method forg athering information is consistent with other samples gathered. […] No one should ever base conclusions without knowing the size of the sample and how random a sample it was. But all too often such data is not mentioned when the statistics are given - perhaps it is overlooked or even intentionally omitted." (Theoni Pappas, "More Joy of Mathematics: Exploring mathematical insights & concepts", 1991)

"When the sample size is small or the study is of one organization, descriptive use of the thematic coding is desirable." (Richard Boyatzis, "Transforming qualitative information", 1998)

On Ecology VI

"Ecology is the scientific study of the interactions that determine the distribution and abundance of organisms." (Charles J Krebs, "Ecology", 1972)

"Ecology, on the other hand, is messy. We cannot find anything deserving of the term law, not because ecology is less developed than physics, but simply because the underlying phenomena are more chaotic and hence less amenable to description via generalization." (Lev Ginzburg & Mark Colyvan," Ecological Orbits: How Planets Move and Populations Grow", 2004)

"Limiting factors in population dynamics play the role in ecology that friction does in physics. They stop exponential growth, not unlike the way in which friction stops uniform motion. Whether or not ecology is more like physics in a viscous liquid, when the growth-rate-based traditional view is sufficient, is an open question. We argue that this limit is an oversimplification, that populations do exhibit inertial properties that are noticeable. Note that the inclusion of inertia is a generalization - it does not exclude the regular rate-based, first-order theories. They may still be widely applicable under a strong immediate density dependence, acting like friction in physics." (Lev Ginzburg & Mark Colyvan, "Ecological Orbits: How Planets Move and Populations Grow", 2004)

"An ecology provides the special formations needed by organizations. Ecologies are: loose, free, dynamic, adaptable, messy, and chaotic. Innovation does not arise through hierarchies. As a function of creativity, innovation requires trust, openness, and a spirit of experimentation - where random ideas and thoughts can collide for re-creation." (George Siemens, "Knowing Knowledge", 2006)

"Knowledge flow can be likened to a river that meanders through the ecology of an organization. In certain areas, the river pools and in other areas it ebbs. The health of the learning ecology of the organization depends on effective nurturing of flow." (George Siemens, "Knowing Knowledge", 2006)

"[ecology:] the scientific study of the distribution and abundance of organisms and the interactions that determine distribution and abundance." (Michael Begon et al, "Ecology: From individuals to ecosystems", 2006)

"The living world can be viewed as a biological hierarchy that starts with subcellular particles, and continues up through cells, tissues and organs. Ecology deals with the next three levels: the individual organism, the population (consisting of individuals of the same species) and the community (consisting of a greater or lesser number of species populations). At the level of the organism, ecology deals with how individuals are affected by (and how they affect) their environment. At the level of the population, ecology is concerned with the presence or absence of particular species, their abundance or rarity, and with the trends and fluctuations in their numbers. Community ecology then deals with the composition and organization of ecological communities." (Michael Begon et al, "Ecology: From individuals to ecosystems", 2006)

"In ecology, we are often interested in exploring the behavior of whole systems of species or ecosystem composed of individual components which interact through biological processes. We are interested not simply in the dynamics of each species or component in isolation, but the dynamics of each species or component in the context of all the others and how those coupled dynamics account for properties of the system as a whole, such as its persistence. This is what people seem to mean when they say that ecology is ‘holistic’, an otherwise rather vague term." (John Pastor, "Mathematical Ecology of Populations and Ecosystems", 2008)

"Much of what we deal with in ecology are rates of change of biological objects: growth of an organism, decay of a dead leaf, fluctuations in populations, accumulation or erosion of soil, increases or decreases in lake levels, etc. But rates of change are some of the hardest things to measure. What we measure are static properties such as the sizes of objects at different times and then infer that change has taken place between those two measurements." (John Pastor, "Mathematical Ecology of Populations and Ecosystems", 2008)

"Therefore, mathematical ecology does not deal directly with natural objects. Instead, it deals with the mathematical objects and operations we offer as analogs of nature and natural processes. These mathematical models do not contain all information about nature that we may know, but only what we think are the most pertinent for the problem at hand. In mathematical modeling, we have abstracted nature into simpler form so that we have some chance of understanding it. Mathematical ecology helps us understand the logic of our thinking about nature to help us avoid making plausible arguments that may not be true or only true under certain restrictions. It helps us avoid wishful thinking about how we would like nature to be in favor of rigorous thinking about how nature might actually work." (John Pastor, "Mathematical Ecology of Populations and Ecosystems", 2008)

On Numbers: Prime Numbers I

“A prime number is one (which is) measured by a unit alone.” (Euclid, “The Elements”, Book VII) 

“Numbers prime to one another are those which are measured by a unit alone as a common measure.” (Euclid, “The Elements”, Book VII)

"Till now the mathematicians tried in vain to discover some order in the sequence of the prime numbers and we have every reason to believe that there is some mystery which the human mind shall never penetrate. To convince oneself, one has only to glance at the tables of the primes, which some people took the trouble to compute beyond a hundred thousand, and one perceives that there is no order and no rule. This is so much more surprising as the arithmetic gives us definite rules with the help of which we can continue the sequence of the primes as far as we please, without noticing, however, the least trace of order." (Leonhard Euler, "Letters of Euler on different subjects in physics and philosophy. Addressed to a German princess, 1768)

 "Mathematicians have tried in vain to this day to discover some order in the sequence of prime numbers, and we have reason to believe that it is a mystery into which the mind will never penetrate." (Leonhard Euler)

"The problem of distinguishing prime numbers from composite numbers and of resolving the latter into their prime factors is known to be one of the most important and useful in arithmetic. […] The dignity of the science itself seems to require that every possible means be explored for the solution of a problem so elegant and so celebrated.” (Carl Friedrich Gauss, "Disquisitiones Arithmeticae”, 1801)

“The difference of two square numbers is always a product, and divisible both by the sum and by the difference of the roots of those two squares; consequently the difference of two squares can never be a prime number.” (Leonhard Euler, “Elements of Algebra”, 1810)

"We found a beautiful and most general proposition, namely, that every integer is either a square, or the sum of two, three or at most four squares. This theorem depends on some of the most recondite mysteries of numbers, and it is not possible to present its proof on the margin of this page." (Pierre de Fermat)

"A prime number, which exceeds a multiple of four by unity, is only once the hypotenuse of a right triangle." (Pierre de Fermat)

"The theory of Numbers has always been regarded as one of the most obviously useless branches of Pure Mathematics. The accusation is one against which there is no valid defence; and it is never more just than when directed against the parts of the theory which are more particularly concerned with primes. A science is said to be useful if its development tends to accentuate the existing inequalities in the distribution of wealth, or more directly promotes the destruction of human life. The theory of prime numbers satisfies no such criteria. Those who pursue it will, if they are wise, make no attempt to justify their interest in a subject so trivial and so remote, and will console themselves with the thought that the greatest mathematicians of all ages have found it in it a mysterious attraction impossible to resist." (Georg H Hardy, 1915)

“The mystery that clings to numbers, the magic of numbers, may spring from this very fact, that the intellect, in the form of the number series, creates an infinite manifold of well distinguishable individuals. Even we enlightened scientists can still feel it e.g. in the impenetrable law of the distribution of prime numbers." (Hermann Weyl, “Philosophy of Mathematics and Natural Science”, 1927)

“The mystery that clings to numbers, the magic of numbers, may spring from this very fact, that the intellect, in the form of the number series, creates an infinite manifold of well-distinguished individuals. Even we enlightened scientists can still feel it, e.g., in the impenetrable law of the distribution of prime numbers.” (Hermann Weyl, “Philosophy of Mathematics and Natural Science”, 1949)

On Numbers: Prime Numbers III

“One of the remarkable aspects of the distribution of prime numbers is their tendency to exhibit global regularity and local irregularity. The prime numbers behave like the ‘ideal gases”’which physicists are so fond of. Considered from an external point of view, the distribution is - in broad terms - deterministic, but as soon as we try to describe the situation at a given point, statistical fluctuations occur as in a game of chance where it is known that on average the heads will match the tail but where, at any one moment, the next throw cannot be predicted.” (Gerald Tenenbaum & Michael M France, “The Prime Numbers and Their Distribution”, 2000)

“"As archetypes of our representation of the world, numbers form, in the strongest sense, part of ourselves, to such an extent that it can legitimately be asked whether the subject of study of arithmetic is not the human mind itself. From this a strange fascination arises: how can it be that these numbers, which lie so deeply within ourselves, also give rise to such formidable enigmas? Among all these mysteries, that of the prime numbers is undoubtedly the most ancient and most resistant." (Gerald Tenenbaum & Michael M France, “The Prime Numbers and Their Distribution”, 2000)

“Prime numbers belong to an exclusive world of intellectual conceptions. We speak of those marvelous notions that enjoy simple, elegant description, yet lead to extreme - one might say unthinkable - complexity in the details. The basic notion of primality can be accessible to a child, yet no human mind harbors anything like a complete picture. In modern times, while theoreticians continue to grapple with the profundity of the prime numbers, vast toil and resources have been directed toward the computational aspect, the task of finding, characterizing, and applying the primes in other domains." (Richard Crandall and Carl Pomerance, “PrimeNumbers: A Computational Perspective”, 2001)

“The primes have tantalized mathematicians since the Greeks, because they appear to be somewhat randomly distributed but not completely so."(Timothy Gowers, “Mathematics: A Very Short Introduction”, 2002)

“The beauty of mathematics is that clever arguments give answers to problems for which brute force is hopeless, but there is no guarantee that a clever argument always exists! We just saw a clever argument to prove that there are infinitely many primes, but we don't know any argument to prove that there are infinitely many pairs of twin primes.”  (David Ruelle, “The Mathematician's Brain”, 2007)

“Although the prime numbers are rigidly determined, they somehow feel like experimental data." Timothy Gowers, “Mathematics: A Very Short Introduction”, 2002)

“[Primes] are full of surprises and very mysterious […] They are like things you can touch. […][ In mathematics most things are abstract, but I have some feeling that I can touch the primes, as if they are made of a really physical material. To me, the integers as a whole are like physical particles.” (Yoichi Motohashi, “The Riemann Hypothesis: The Greatest Unsolved Problem in Mathematics”, 2002)

“[…] despite their apparent simplicity and fundamental character, prime numbers remain the most mysterious objects studied by mathematicians. In a subject dedicated to finding patterns and order, the primes offer the ultimate challenge.” (Marcus du Sautoy, “The Music of the Primes”, 2003)

“The primes have been a constant companion in our exploration of the mathematical world yet they remain the most enigmatic of all numbers. Despite the best efforts of the greatest mathematical minds to explain the modulation and transformation of this mystical music, the primes remain an unanswered riddle.” (Marcus du Sautoy, “The Music of the Primes”, 2003)