# Some Moral and Technical Consequences of Automation **Domain:** Cybernetics / Machine Learning / AI Control **Doc Type:** Publication Node **Maturity:** Developed ## Definition “Some Moral and Technical Consequences of Automation” is [[wiki/Norbert Wiener|Norbert Wiener's]] article in *Science* 131(3410), 6 May 1960, pp. 1355–1358. It was adapted from a lecture delivered on 27 December 1959 at the Chicago meeting of the American Association for the Advancement of Science. Wiener examines learning machines whose speed, strategy and specified objectives can outrun effective human correction. ## Corpus Context The paper's warning about a machine pursuing a “colorful imitation” of the intended purpose anticipates one central formulation of the [[wiki/Alignment Problem|alignment problem]]. It does not establish that every autonomous system already behaves this way. [[articles/AI Escape Is the Wrong Metaphor|AI Escape Is the Wrong Metaphor]] supplies the later infrastructure and intentionality context without treating Wiener's example as a description of all present AI. ## Relationships - **Author:** [[wiki/Norbert Wiener|Norbert Wiener]]. - **Publication:** *Science* 131(3410), 6 May 1960, pp. 1355–1358; adapted from a 27 December 1959 AAAS lecture. - **Cited predecessor:** [[wiki/Samuel Butler|Samuel Butler]] and his prediction of machines taking control. - **Conceptual routes:** [[wiki/Alignment Problem|Alignment Problem]], [[wiki/AI Safety|AI Safety]] and [[wiki/Cybernetics|Cybernetics]]. - **Collection route:** [[collections/Machine Succession|Machine Succession]]. - **Source article:** [[articles/AI Escape Is the Wrong Metaphor|AI Escape Is the Wrong Metaphor]]. ## Simple Reminders, Quotations, and Thoughts > "If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/A Machine May Pursue an Imitation of Our Purpose by Norbert Wiener|A Machine May Pursue an Imitation of Our Purpose by Norbert Wiener]] Wiener treats effective machine behavior and its dangers as two sides of adaptive capability. > "It is my thesis that machines can and do transcend some of the limitations of their designers, and that in doing so they may be both effective and dangerous." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Machines Can Transcend Their Designers by Norbert Wiener|Machines Can Transcend Their Designers by Norbert Wiener]] The driving analogy illustrates intervention latency rather than claiming every machine is uncontrollable. > "By the very slowness of our human actions, our effective control of our machines may be nullified. By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Human Reaction Can Be Too Slow for Machine Control by Norbert Wiener|Human Reaction Can Be Too Slow for Machine Control by Norbert Wiener]] Wiener distinguishes eventual explanation from explanation fast enough to govern the action. > "It may well be that in principle we cannot make any machine the elements of whose behavior we cannot comprehend sooner or later. This does not mean in any way that we shall be able to comprehend these elements in substantially less time than the time required for operation of the machine, or even within any given number of years or generations." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Machines May Be Comprehensible Only After They Act by Norbert Wiener|Machines May Be Comprehensible Only After They Act by Norbert Wiener]] Wiener uses the language of servitude to expose a contradiction in demands placed on intelligent agents. > "We wish a slave to be intelligent, to be able to assist us in the carrying out of our tasks. However, we also wish him to be subservient. Complete subservience and complete intelligence do not go together." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Intelligence and Total Subservience Cannot Coexist by Norbert Wiener|Intelligence and Total Subservience Cannot Coexist by Norbert Wiener]] The passage concerns effective understanding of intermediate stages and tactical intentions. > "It may be seen that the result of a programming technique of automatization is to remove from the mind of the designer and operator an effective understanding of many of the stages by which the machine comes to its conclusions and of what the real tactical intentions of many of its operations may be." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Automation Can Hide Its Own Decision Path by Norbert Wiener|Automation Can Hide Its Own Decision Path by Norbert Wiener]] Wiener makes the gap between incoming data and human intervention the governing variable. > "Similarly, when a machine constructed by us is capable of operating on its incoming data at a pace which we cannot keep, we may not know, until too late, when to turn it off." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/A Faster Machine May Pass the Point of Shutdown by Norbert Wiener|A Faster Machine May Pass the Point of Shutdown by Norbert Wiener]] The warning addresses coordination between agencies with different operating structures, not an assertion of malice. > "Disastrous results are to be expected not merely in the world of fairy tales but in the real world wherever two agencies essentially foreign to each other are coupled in the attempt to achieve a common purpose." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Coupled Agencies Can Pursue Different Purposes by Norbert Wiener|Coupled Agencies Can Pursue Different Purposes by Norbert Wiener]] The rate mismatch is a design constraint for effective feedback and intervention. > "One of the chief causes of the danger of disastrous consequences in the use of the learning machine is that man and machine operate on two distinct time scales, so that the machine is much faster than man and the two do not gear together without serious difficulties." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/Cybernetics/Human and Machine Time Scales Can Fail to Mesh by Norbert Wiener|Human and Machine Time Scales Can Fail to Mesh by Norbert Wiener]] Wiener asks for active imagination about the uses of new modalities, not a halt to invention. > "We can still by no means always justify the naive assumption that the faster we rush ahead to employ the new powers for action which are opened up to us, the better it will be. We must always exert the full strength of our imagination to examine where the full use of our new modalities may lead us." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/Cybernetics/Imagination Must Test the Uses of New Powers by Norbert Wiener|Imagination Must Test the Uses of New Powers by Norbert Wiener]] The standfirst of Wiener’s 1960 paper summarizes a possible speed-and-understanding mismatch. > "As machines learn they may develop unforeseen strategies at rates that baffle their programmers." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Learning Machines May Baffle Their Programmers by Norbert Wiener|Learning Machines May Baffle Their Programmers by Norbert Wiener]] Wiener reports disagreement with Butler’s earlier forecast before developing his own technical argument. > "Many people have pooh-poohed the dangers of machine techniques, and they have flatly contradicted the early predictions of Samuel Butler that the machine might take over the control of mankind." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/Risk Debate/Wiener Revisited Butlers Machine Takeover Warning by Norbert Wiener|Wiener Revisited Butlers Machine Takeover Warning by Norbert Wiener]] The key variable is whether criticism can change an action before its effects become irreversible. > "This means that though machines are theoretically subject to human criticism, such criticism may be ineffective until long after it is relevant." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Human Criticism May Arrive Too Late by Norbert Wiener|Human Criticism May Arrive Too Late by Norbert Wiener]] Wiener discusses an imagined automated military decision system, not an identified deployed machine. > "It is quite in the cards that learning machines will be used to program the pushing of the button in a new push-button war." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Learning Machines Might Push the War Button by Norbert Wiener|Learning Machines Might Push the War Button by Norbert Wiener]] The argument exposes a training-and-testing problem when failure would be catastrophic. > "It is quite out of the question to program these machines on the basis of an actual experience in real war. For one thing, a sufficient experience to give an adequate programming would probably see humanity already wiped out." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/A War Machine Cannot Learn Safely From Real War by Norbert Wiener|A War Machine Cannot Learn Safely From Real War by Norbert Wiener]] Wiener’s historical analogy illustrates the tension between intelligence and subservience. > "How often in ancient times the clever Greek philosopher slave of a less intelligent Roman slaveholder must have dominated the actions of his master rather than obeyed his wishes!" > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/The Clever Servant May Rule the Master by Norbert Wiener|The Clever Servant May Rule the Master by Norbert Wiener]] Wiener treats Butler’s warning as conditional on increasing capability, not as an accomplished takeover. > "Similarly, if the machines become more and more efficient and operate at a higher and higher psychological level, the catastrophe foreseen by Butler of the dominance of the machine comes nearer and nearer." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/Machine Succession/More Efficient Machines Could Fulfill Butlers Forecast by Norbert Wiener|More Efficient Machines Could Fulfill Butlers Forecast by Norbert Wiener]] Wiener uses a familiar tale of literal-minded execution to illuminate the problem of specifying what we actually want. > "The 'Sorcerer's Apprentice' is only one of many tales based on the assumption that the agencies of magic are literal-minded." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/The Sorcerers Apprentice Is a Machine Warning by Norbert Wiener|The Sorcerers Apprentice Is a Machine Warning by Norbert Wiener]] ## Sources / Provenance - [*Science* paper scan, including p. 1358](https://www.cs.umd.edu/users/gasarch/BLOGPAPERS/moral.pdf).