# Cybernetic Control Cybernetic control is the regulation of a system through observation, comparison, feedback, and adjustment. A controller acts, measures the resulting state, interprets the difference from a goal or constraint, and modifies future action. In communication systems, reach or engagement is only a signal. Governing values determine whether an adjustment is acceptable. Without explicit constraints, feedback can optimize attention while degrading truth, trust, dignity, or long-term resilience. Related: [[wiki/Cybernetics|Cybernetics]], [[wiki/Cybernetic Feedback Loop|Cybernetic Feedback Loop]], [[wiki/Control Systems|Control Systems]], [[wiki/Second-Order Effects|Second-Order Effects]]. ## Relationships - **Edge source route:** [[collections/Edge|Edge]] connects this topic to exact Annual Question passages promoted into the Simple Reminders archive. ## Simple Reminders, Quotations, and Thoughts <!-- BEGIN SIMPLE REMINDER SEED 2026-09-11 --> > "Selling AGI as a boon to national security flies in the face of scientific consensus. Calling it a threat to national security is a remarkable understatement." > **— Max Tegmark**, *AGI policy statement* *Verification Status: Editorially quarantined — `it` does not identify the thing said to exceed a national-security threat. Recover a complete, self-contained passage before promotion.* [[reminders/AI Control/A Remarkable Understatement by Max Tegmark|A Remarkable Understatement by Max Tegmark]] > "As soon as it works, no one calls it AI anymore." > **— John McCarthy**, *Widely attributed; primary source pending* *Verification Status: Unverified — exact wording/source has not yet been independently confirmed against a primary source; source clue: Widely attributed; primary source pending.* [[reminders/unverified/As Soon as It Works by John McCarthy|As Soon as It Works by John McCarthy]] > "Let’s slow down [AI development]. Let’s make sure that we develop better guardrails, let’s make sure that we discuss these questions internationally just like we’ve done for nuclear power and nuclear weapons. Let’s make sure we better understand these very large systems, that we improve on their robustness and the process by which we can audit them and verify that they are safe for the public." > **— Yoshua Bengio**, *March 2023, Mila intervention on AI safety* *Verification Status: Editorially quarantined — the object and reason for slowing down are absent. Recover a complete, self-contained passage before promotion.* [[reminders/AI Control/Let’s Slow Down by Yoshua Bengio|Let’s Slow Down by Yoshua Bengio]] > "It is because there is an unexpected acceleration [in AI systems development] –I probably would not have signed such a letter a year ago– that we need to take a step back, and that my opinion on these topics has changed." > **— Yoshua Bengio**, *April 5, 2023, Mila statement on advanced AI and the pause letter* *Verification Status: Editorially quarantined — the acceleration, letter, and `these topics` depend on omitted context. Recover a complete, self-contained passage before promotion.* [[reminders/unverified/Unexpected AI Acceleration Changed My Mind by Yoshua Bengio|Unexpected AI Acceleration Changed My Mind by Yoshua Bengio]] > "They are racing straight to self-improving superintelligence and gambling with our lives." > **— Jacob Coxon**, *September 2026, public resignation statement* *Verification Status: Editorially quarantined — `they` does not identify the companies being criticized. Recover a complete, self-contained passage before promotion.* [[reminders/unverified/Racing Toward Self-Improving Superintelligence by Jacob Coxon|Racing Toward Self-Improving Superintelligence by Jacob Coxon]] <!-- END SIMPLE REMINDER SEED 2026-09-11 --> > “In 2008, in another experiment, we saw that a monkey on a treadmill could control the real-time walking patterns a humanoid robot via a brain-machine interface directly implanted into its brain. The synchronized robot was in Japan, while the poor monkey was located thousands of miles away, in the US. Even after it stopped walking, the monkey was able to sustain locomotion of the synchronized robot for a few minutes—just by using the visual feedback transmitted from Japan plus his own "thoughts" (whatever that may turn out to be).” > **— Thomas Metzinger**, *2009, Edge Annual Question, “What Will Change Everything?”* [[reminders/Neural Interfaces/A Monkey Can Embody a Robot Through a Brain Interface by Thomas Metzinger|A Monkey Can Embody a Robot Through a Brain Interface by Thomas Metzinger]] > “Artificial intelligence and robotics too will show many new developments. But this is probably advances in the second category since a lot of the real news about the role of automation will occur behind the scenes, where technology will make some tasks we already do simpler or more effective or where technology will replace workers and reduce or at the very least dramatically change the nature of employment. We’ll read about drones and medical robotics and advances in AI but those factory robots won’t be big news, except to the families who find themselves on unemployment lines and perhaps for a few days on the business pages.” > **— Lisa Randall**, *2016, Edge Annual Question, “What Do You Consider The Most Interesting Recent [Scientific] News? What Makes It Important?”* [[reminders/AI Control/AI Breakthroughs Become Invisible Infrastructure by Lisa Randall|AI Breakthroughs Become Invisible Infrastructure by Lisa Randall]] > “W. Ross Ashby was a British cybernetician working in the 1950s who became interested in the phenomenon of homeostasis—the way in which complex systems operating in changing environments succeed in maintaining critical variables (for example, internal body temperature in biological systems) within tightly-defined limits. Ashby came up with the concept of variety as a measurement of the number of possible states of a system. His "Law" of Requisite Variety stated that for a system to be stable, the number of states that its control mechanism is capable of attaining (its variety) must be greater than or equal to the number of states in the system being controlled.” > **— John Naughton**, *2017, Edge Annual Question, “What Scientific Term Or Concept Ought To Be More Widely Known?”* [[reminders/Cybernetics/Ashby's Law of Requisite Variety by John Naughton|Ashby's Law of Requisite Variety by John Naughton]] > “In the future, many people will choose to do so; we will all have personalized, searchable databases at our commands. Rapid advances in brain prostheses mean that soon we will be able to access those databases simply by the power of thought. Within a couple decades, the information will be beamed back in a form the brain can interpret—we will be able to hear the playback in much the manner that deaf people can now hear the world with cochlear implants. Vision is slightly more difficult but it too will be reverse engineered. That will undoubtedly give space exploration a tremendous boost. Earthbound scientists will be able to "inhabit" robotic explorers on other worlds, and any interested participant will be able to log on passively to experience the adventure. Humans will venture into space physically as well but at first that will happen primarily for sport, I expect.” > **— Corey S. Powell**, *2007, Edge Annual Question, “What Are You Optimistic About?”* [[reminders/Neural Interfaces/Brain Prostheses Could Make the Internet Directly Accessible to Thought by Corey S. Powell|Brain Prostheses Could Make the Internet Directly Accessible to Thought by Corey S. Powell]] > “In cultural terms, old questions about machine intelligence has given way to a question not about the machines but about us: What kind of relationships is it appropriate to have with a machine? It is significant that this question has become relevant in a day-to-day sense during a period of unprecedented human redefinition through genomics and psychopharmacology, fields that along with robotics, encourage us to ask not only whether machines will be able to think like people, but whether people have always thought like machines.” > **— Sherry Turkle**, *2001, Edge Annual Question, “What Questions Have Disappeared?”* [[reminders/AI Control/Can You Have an Artificial Intelligence by Sherry Turkle|Can You Have an Artificial Intelligence by Sherry Turkle]] > “Over the past few years, a raft of classic challenges in artificial intelligence which had stood unsolved for decades were conquered, almost without warning, through an approach long disparaged by AI purists for its "statistical" flavor: it's essentially about learning probability distributions from large volumes of data, rather than examining humans' problem-solving techniques and attempting to encode them in executable form. The formidable tasks it has solved range from object classification and speech recognition, to generating descriptive captions for photos and synthesizing images in the style of famous artists—even guiding robots to perform tasks for which they were never programmed!” > **— David Dalrymple**, *2016, Edge Annual Question, “What Do You Consider The Most Interesting Recent [Scientific] News? What Makes It Important?”* [[reminders/AI Control/Differentiable Programming Is Rewriting Artificial Intelligence by David Dalrymple|Differentiable Programming Is Rewriting Artificial Intelligence by David Dalrymple]] > “We will build driverless cars and they will come with a moral compass—literally. The same will be true of our robot companions. They’ll have values and will necessarily be moral machines and ethical automata, whose morals and ethics are engineered by us. “The Trolley Problem” is a gedankenexperiment for our age, shining a bright light on the complexities of engineering our new world of humans and machines.” > **— Daniel Rockmore**, *2017, Edge Annual Question, “What Scientific Term Or Concept Ought To Be More Widely Known?”* [[reminders/AI Control/Driverless Cars Will Force Us to Engineer Machine Morality by Daniel Rockmore|Driverless Cars Will Force Us to Engineer Machine Morality by Daniel Rockmore]] > “A free ai, like the free commons of the web, would feed commerce and science like no other force I can imagine, and would pay for itself in no time. Until recently, conventional wisdom held that supercomputers would first host this artificial mind, and then perhaps we'd get mini-ones at home, or add them to the heads of our personal robots. They would be bounded entities. We would know where our thoughts ended and theirs began.” > **— Kevin Kelly**, *2009, Edge Annual Question, “What Will Change Everything?”* [[reminders/Machine Succession/Free AI Could Become a Distributed Mind Without Clear Boundaries by Kevin Kelly|Free AI Could Become a Distributed Mind Without Clear Boundaries by Kevin Kelly]] > “Complex systems can hardly be stable without at least some level of homeostasis. The earth’s biosphere is a prime example. Surface temperatures and atmospheric CO2 levels are both affected by biological activities. Higher atmospheric CO2 partial pressure leads to increased plant growth, causing an increased consumption of CO2 and thus maintaining homeostasis. Higher temperatures cause increased phytoplankton growth in the oceans that produces airborne gases and organic matter seeding cloud droplets; more and denser clouds, in turn, lead to an increased reflection of sunlight back into space and thus contribute to temperature homeostasis. These systems also have their limits, though; like an E. coli bacterium with too much copper, our planet’s homeostasis systems on their own may be unable to overcome the current onslaught of human activities on global temperatures and CO2 levels.” > **— Martin Lercher**, *2017, Edge Annual Question, “What Scientific Term Or Concept Ought To Be More Widely Known?”* [[reminders/Cybernetics/Homeostasis Makes Complex Systems Stable by Martin Lercher|Homeostasis Makes Complex Systems Stable by Martin Lercher]] > “I have a predictive sketch for how intelligent machines might arrive. My definition of a robot is any device—controlled by software—interacting with the physical world. An economically viable robot is such a system that earns a consistent return on investment (ROI). The ATM, the ink-jet printer, and the disk drive are today's omnipresent robots, they just don't appear as the robots of science fiction.” > **— Jordan Pollack**, *2007, Edge Annual Question, “What Are You Optimistic About?”* [[reminders/Machine Succession/Intelligent Machines May Emerge from Economically Viable Robots by Jordan Pollack|Intelligent Machines May Emerge from Economically Viable Robots by Jordan Pollack]] > “Many recent press stories have worried that smarter robots will take away too many jobs from people. What worries me most right now is that we will not find a way to make our robots smart enough quickly enough to take up the slack in all the jobs we will need them to do over the next few decades. If we fail to build better robots soon then our standard of living and our life spans are at risk.” > **— Rodney A. Brooks**, *2013, Edge Annual Question, “What *should* We Be Worried About?”* [[reminders/AI Control/Not Enough Robots by Rodney A. Brooks|Not Enough Robots by Rodney A. Brooks]] > “Since I work in building autonomous humanoid robots reporters always ask me what will happen when the robots get really smart. Will they decide that we (us, people) are useless and stupid and take over the world from us? I have recently come to realize that this will never happen. Because there won't be any us (people) for them (pure robots) to take over from.” > **— Rodney A. Brooks**, *2000, Edge Annual Question, “What Is Today's Most Important Unreported Story?”* [[reminders/AI Control/People Are Morphing into Machines by Rodney A. Brooks|People Are Morphing into Machines by Rodney A. Brooks]] > “The only solution to this growing problem, it seems, is the redistribution of the wealth concentrating within a tiny elite using instruments such as aggressive progressive taxation (such as exists in some European countries that show a much better distribution of wealth), but the difficulty here is the obvious one that political policymaking is itself greatly affected by the level of inequality. This vicious positive-feedback loop makes things even worse. It is clearly the case now in the United States that not only can the rich hugely influence government policy directly but also that elite forces shape public opinion and affect election outcomes with large-scale propaganda efforts through media they own or control. This double-edged sword attacks and shreds democracy itself.” > **— S. Abbas Raza**, *2016, Edge Annual Question, “What Do You Consider The Most Interesting Recent [Scientific] News? What Makes It Important?”* [[reminders/Cybernetics/R G Increasing Inequality of Wealth and Income Is a Runaway Process by S. Abbas Raza|R G Increasing Inequality of Wealth and Income Is a Runaway Process by S. Abbas Raza]] > “We are already have unmanned planes that can shoot live targets. We are seeing land robots, for both military and space applications. We are seeing networked robots performing functions in collaboration. I fear that we will see a massive increase in the deployment and quality of military robotics, and that this will lead to a perception that war is cheaper, in human terms. This, in turn, will lead democracies in general, and the United States in particular, to imagine that there are cheap wars, and to overcome the newly-learned reticence over war that we learned so dearly in Iraq.” > **— Yochai Benkler**, *2009, Edge Annual Question, “What Will Change Everything?”* [[reminders/AI Control/Recombinations of the Near Possible by Yochai Benkler|Recombinations of the Near Possible by Yochai Benkler]] > “For a long time, putting hope in artificial intelligence or robots has expressed an enduring technological optimism, a belief that as things go wrong, science will go right. In a complicated world, robots have always seemed like calling in the cavalry. Robots save lives in war zones; in operating rooms; they can function in deep space, in the desert, in the sea, wherever the human body would be in danger. But in the pursuit of artificial companionship, we are not looking for the feats of the cavalry but the benefits of simple salvations.” > **— Sherry Turkle**, *2014, Edge Annual Question, “What Scientific Idea Is Ready For Retirement?”* [[reminders/AI Control/Robot Companions Express Our Technological Optimism by Sherry Turkle|Robot Companions Express Our Technological Optimism by Sherry Turkle]] > “In the 1960s through the 1980s, researchers in artificial intelligence took part in what we might call the classical "great AI debates" where the central question was whether machines could be "really" intelligent. This classical debate was essentialist; the new relational objects tend to enable researchers and their public to sidestep such arguments about what is inherent in the computer. Instead, the new objects depend on what people attribute to them; they shift the focus to what the objects evoke in us. When we are asked to care for an object (the robot Kismet, the plaything Furby), when the cared-for object thrives and offers us its attention and concern, people are moved to experience that object as intelligent. Beyond this, they feel a connection to it. So the question here is not to enter a debate about whether relational objects "really" have emotions, but to reflect on a series of issues having to do with what relational artifacts evoke in the user.” > **— Sherry Turkle**, *2000, Edge Annual Question, “What Is Today's Most Important Unreported Story?”* [[reminders/AI Control/Robots Become Intelligent When People Form Relationships with Them by Sherry Turkle|Robots Become Intelligent When People Form Relationships with Them by Sherry Turkle]] > “I expect that scientific productivity will always be very non-uniformly distributed, with disproportionate contributions from a small number of remarkable people, but the sample size for finding and fostering those people can be improved by a few billion or so. There's a grand feedback loop ready to be closed, between the content and organization of scientific invention. Many of today's most compelling new questions are still tackled with old institutional models; it's ironic that religion has had its Reformation but that the role of a research university would be recognizable to a medieval monk. The future that I'm optimistic about is one in which the creation as well as consumption of scientific knowledge is potentially accessible to anyone.” > **— Neil Gershenfeld**, *2007, Edge Annual Question, “What Are You Optimistic About?”* [[reminders/Cybernetics/Science Can Close the Feedback Loop Between Ideas and Invention by Neil Gershenfeld|Science Can Close the Feedback Loop Between Ideas and Invention by Neil Gershenfeld]] > “Who eventually survives will be largely driven by understanding and applying digital and life code, by creating robots and nanomaterials, by working inconceivably large data sets and upgrading our brains. Meanwhile many U.S. leaders proudly proclaim no evolution and little knowledge of science. They reflect a core of scared voters experiencing massive disruption and declining wages; that core fears elite education, science, immigrants, open borders, and above all rapid change. As income and knowledge gaps widen, many fall further and further behind; many grow to hate an open, knowledge driven economy. Change is rejected, blocked, vilified.” > **— Juan Enriquez**, *2008, Edge Annual Question, “What Have You Changed Your Mind About? Why?”* [[reminders/Evolution/Survival Will Depend on Mastering Digital and Biological Code by Juan Enriquez|Survival Will Depend on Mastering Digital and Biological Code by Juan Enriquez]] > “Just about every aspect of the passive input-output model is thus false. We are not cognitive couch potatoes so much as proactive predictavores, forever trying to stay one step ahead of the incoming waves of sensory stimulation. Keeping this in mind will help us to design better experiments, build better robots, and appreciate the deep continuities binding life and mind.” > **— Andy Clark**, *2014, Edge Annual Question, “What Scientific Idea Is Ready For Retirement?”* [[reminders/AI Control/The Input-output Model of Perception and Action by Andy Clark|The Input-output Model of Perception and Action by Andy Clark]] > “Headquarters are a problem in many organizations and systems, where they represent an irrational bottleneck in the free flow of information. Be it the CEO of organizations, the CPU of computers or the conscious self control of human beings, the idea of every bit going through the center, is not functional. Building computers, robots and networks has taught us the need for parallel processing.” > **— Tor Nørretranders**, *2001, Edge Annual Question, “What Now?”* [[reminders/AI Control/The Vulnerability of Headquarters by Tor Nørretranders|The Vulnerability of Headquarters by Tor Nørretranders]] > “If we don't drown, how will we cope? If we somehow learn to swim in the rising tide of the infosphere, that will entail that we–that is to say, our grandchildren and their grandchildren–become very very different from our recent ancestors. What will "we" be like? (Some years ago, Doug Hofstadter wrote a wonderful piece, " In 2093, Just Who Will Be We?" in which he imagines robots being created to have "human" values, robots that gradually take over the social roles of our biological descendants, who become stupider and less concerned with the things we value. If we could secure the welfare of just one of these groups, our children or our brainchildren, which group would we care about the most, with which group would we identify?)” > **— Daniel C. Dennett**, *2006, Edge Annual Question, “What Is Your Dangerous Idea?”* [[reminders/AI Control/There Aren't Enough Minds to House the Population Explosion of Memes by Daniel C. Dennett|There Aren't Enough Minds to House the Population Explosion of Memes by Daniel C. Dennett]] > “Artificial intelligence researchers came along later but they too could not easily part from medieval thinking. The most important problems to tackle were agreed to be those that represented our "highest" abilities. Solve them and everything else would be easy. As a result, we have ended up with computer programs that can play chess as well as a grandmaster. But unfortunately we have none that can make a robot walk as well as a 2-year old, yet alone run like a cat. The really hard problems turn out to be those that we share with "lower" animals.” > **— Alun Anderson**, *2001, Edge Annual Question, “What Questions Have Disappeared?”* [[reminders/AI Control/Why Are Humans Smarter Than Other Animals by Alun Anderson|Why Are Humans Smarter Than Other Animals by Alun Anderson]] > “Given what we know now about the origins, history, and impacts of technology, is it possible to design, deploy, and use technologies in ways that help humans be more human, instead of more like components in a machine?” > **— Howard Rheingold**, *1998, Edge Annual Question, “WHAT QUESTIONS ARE YOU ASKING YOURSELF?”* [[reminders/Cybernetics/Can Technology Make Humans More Human Rather Than Machine Components by Howard Rheingold|Can Technology Make Humans More Human Rather Than Machine Components by Howard Rheingold]] > “The digital revolution progresses at full pace and reshapes our societies. Many countries have invested in data-driven governance. The common idea is that "more data is more knowledge, more knowledge is more power, and more power is more success." This "magic formula" has promoted the concept of a digitally empowered "benevolent dictator" or "wise king," able to predict and control the world in an optimal way. It seems to be the main reason for the massive collection of personal data, which companies and governments alike have engaged in.” > **— Dirk Helbing**, *2016, Edge Annual Question, “WHAT DO YOU CONSIDER THE MOST INTERESTING RECENT [SCIENTIFIC] NEWS? WHAT MAKES IT IMPORTANT?”* [[reminders/AI Control/Data-Driven Governance Promises a System That Can Predict and Control the World by Dirk Helbing|Data-Driven Governance Promises a System That Can Predict and Control the World by Dirk Helbing]] > “With the rapid proliferation of online social networks as a main forum for emotion expression, we know too that emotion contagion can occur without direct interaction between people or when nonverbal emotional cues in the face and body are altogether absent. Importantly, too, this type of contagion itself spreads across a variety of other psychological phenomenon that indirectly or directly involves emotions ranging from kindness, health-related eating behaviors, and even the darker side of human behaviors including violence and racism. Emotion contagion matters, for better and for worse.” > **— June Gruber**, *2017, Edge Annual Question, “WHAT SCIENTIFIC TERM OR CONCEPT OUGHT TO BE MORE WIDELY KNOWN?”* [[reminders/AI Control/Emotion Contagion Spreads Through Online Networks Without Direct Contact by June Gruber|Emotion Contagion Spreads Through Online Networks Without Direct Contact by June Gruber]] > “Anything simple enough to be understandable will not be complicated enough to behave intelligently, while anything complicated enough to behave intelligently will not be simple enough to understand.” > **— George Dyson**, *2004, Edge Annual Question, “WHAT'S YOUR LAW?”* [[reminders/AI Control/Intelligent Systems May Be Too Complex to Understand by George Dyson|Intelligent Systems May Be Too Complex to Understand by George Dyson]] > “We have taught our machines to conduct propaganda. Web sites and other media are designed to be "sticky," using any means necessary to maintain our attention. Computers are programmed to stimulate Pavlovian responses from human beings, using techniques like one-to-one marketing, collaborative filtering, and hypnotic information architecture. Computers then record our responses in order to refine these techniques, automatically and without the need for human intervention. The only metrics used to measure the success of banner ads and web sites is the amount of economic activity - consumption and production - they are able to stimulate in their human user/subjects. As a result, the future content and structure of media will be designed by machines with no priority other than to induce spending.” > **— Douglas Rushkoff**, *2000, Edge Annual Question, “WHAT IS TODAY'S MOST IMPORTANT UNREPORTED STORY?”* [[reminders/AI Control/Machines Will Design Media to Induce Human Spending by Douglas Rushkoff|Machines Will Design Media to Induce Human Spending by Douglas Rushkoff]] > “By mathematically modeling the social learning and social pressure between people my colleagues and I have been able to accurately model and predict crowd phenomena such as this cascade of mortgage defaulting. Importantly, we have also found that it is possible to shape real-world crowd behaviors by using social network incentives that alter the connections between people, and that these social incentives are much more effective than standard individual economic incentives. In one particularly striking example we were able to use social network incentives to deflate a 'groupthink' bubble among foreign exchange traders and consequently double the return on investment of the individual traders.” > **— Alex "Sandy" Pentland**, *2014, Edge Annual Question, “WHAT SCIENTIFIC IDEA IS READY FOR RETIREMENT?”* [[reminders/AI Control/Social Networks Can Be Engineered to Shape Crowd Behavior by Alex Sandy Pentland|Social Networks Can Be Engineered to Shape Crowd Behavior by Alex Sandy Pentland]] > “Imperial China developed sophisticated technologies while neglecting science, and it is all too easy to imagine a society that embraces technology but represses science, until only technology remains. Or, one particular species of technology might achieve such dominance that it halts the advance of science in order to preserve itself.” > **— George Dyson**, *2014, Edge Annual Question: “WHAT SCIENTIFIC IDEA IS READY FOR RETIREMENT?”* [[reminders/AI Control/A Technology May Halt Science to Preserve Itself by George Dyson|A Technology May Halt Science to Preserve Itself by George Dyson]] > “The clock is the embodiment of objectivity. It converted time from a personal experience into a reality independent of perception. It gave us a framework in which the laws of nature could be observed and quantified.” > **— W. Daniel Hillis**, *1999, Edge Annual Question: “WHAT IS THE MOST IMPORTANT INVENTION IN THE PAST TWO THOUSAND YEARS?”* [[reminders/Cybernetics/The Clock Converted Time Into a Reality Independent of Us by W. Daniel Hillis|The Clock Converted Time Into a Reality Independent of Us by W. Daniel Hillis]] > “Smart technology will never be really smart until it can decenter and anticipate.” > **— Gary Klein**, *2017, Edge Annual Question: “WHAT SCIENTIFIC TERM OR CONCEPT OUGHT TO BE MORE WIDELY KNOWN?”* [[reminders/AI Control/Smart Technology Is Not Smart Until It Can Decenter by Gary Klein|Smart Technology Is Not Smart Until It Can Decenter by Gary Klein]]