On Optimization I

"The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming." (Donald E Knuth, "Computer Programming as an Art", 1968)

"In most engineering problems, particularly when solving optimization problems, one must have the opportunity of comparing different variants quantitatively. It is therefore important to be able to state a clear-cut quantitative criterion." (Yakov Khurgin, "Did You Say Mathematics?", 1974)

"Linear programming is viewed as a revolutionary development giving man the ability to state general objectives and to find, by means of the simplex method, optimal policy decisions for a broad class of practical decision problems of great complexity. In the real world, planning tends to be ad hoc because of the many special-interest groups with their multiple objectives." (George Dantzig, "Reminiscences about the origins of linear programming", Mathematical programming: the state of the art", 1983) 

"It remains an unhappy fact that there is no best method for finding the solution to general nonlinear optimization problems. About the best general procedure yet devised is one that relies upon imbedding the original problem within a family of problems, and then developing relations linking one member of the family to another. If this can be done adroitly so that one family member is easily solvable, then these relations can be used to step forward from the solution of the easy problem to that of the original problem. This is the key idea underlying dynamic programming, the most flexible and powerful of all optimization methods." (John L Casti, "Five Golden Rules", 1995)

"Heuristic methods may aim at local optimization rather than at global optimization, that is, the algorithm optimizes the solution stepwise, finding the best solution at each small step of the solution process and 'hoping' that the global solution, which comprises the local ones, would be satisfactory." (Nikola K Kasabov, "Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"Mathematical programming (or optimization theory) is that branch of mathematics dealing with techniques for maximizing or minimizing an objective function subject to linear, nonlinear, and integer constraints on the variables." (George B Dantzig & Mukund N Thapa, "Linear Programming" Vol I, 1997)

"A heuristic is ecologically rational to the degree that it is adapted to the structure of an environment. Thus, simple heuristics and environmental structure can both work hand in hand to provide a realistic alternative to the ideal of optimization, whether unbounded or constrained." (Gerd Gigerenzer & Peter M Todd, "Fast and Frugal Heuristics: The Adaptive Toolbox" [in "Simple Heuristics That Make Us Smart"], 1999)