On a black night in the early 1980s, a fierce scream congealed the darkness deep within MIT's Artificial Intelligence Laboratory. The late-working engineer who went to investigate discovered Richard Stallman -- one of the nation's most brilliant programmers -- sitting in tears and screaming at his computer terminal, "How can you do this? How can you do this? How can you do this?" /1/
The image is no doubt provocative, revealing as it does a striking urge to personify the computer. And yet, perhaps we make too much of such occurrences. After all, the computer is hardly unique in this respect. Don't I curse the door that jams, implore my car's engine to start on an icy morning, and kick a malfunctioning TV with more than just mechanical intent? The fact is that we manifest a strong tendency to personify all the contrivances of our own devising. The computer simply takes its place among the numerous other objects to which, with a kind of animistic impulse, we attribute life.
This may be the point worth holding on to, however. Once we acknowledge our anthropomorphizing compulsions, we must immediately grant that the computer is ideally designed to provoke them. Whatever our considered, philosophical understanding of the computer and its intelligence, we also need to reckon with this "animistic" tendency -- at least we do if we seek self-knowledge, and if we would prevent our own subconscious impulses from infecting our philosophical inquiries.
Lacking any better term, I will call this wholly derivative intelligence of the computer its book value. However, the notion extends to other sorts of stored material in the computer besides text -- for example, voice recordings and video images. Just as a book displays "someone else's" intelligence and not its own, so also do the tape recorder and television. In what follows, I'll use "book value" to include all such derived content.
No one would claim that its book value represents what the computer itself does. Those who talk about how computers will be able to think and converse are not merely referring to the way tape recorders and books "speak." They have in mind an autonomous activity of the computer -- an activity directly expressing intelligence.
So I will use these terms -- "book value" and "activity" -- as rough markers capturing the distinction I'm after. While I have had, and will have, a great deal to say about the logical or mathematical character of computational activity, here I propose to look at certain peculiarities of book value in our culture. It is a curious fact that, today, book value -- the detached word -- readily takes on a life of its own, whether or not it is associated with the computer. At the same time, we seem strongly inclined to adopt an anthropomorphizing or superstitious stance toward this detached word.
I am currently writing a program that allows the user to build a network consisting of thoughts and relations. The user starts by building a thought node. Each thought node contains a pointer to a list of relation nodes. Each relation node contains a pointer to another thought node. Every time a new thought is created, a relation node is added to the relation list of the current thought....
What are we to make of this language? A few innocent-sounding sentences and suddenly we have a computer dealing with "thoughts" and relations -- all in the context of discussion about artificial intelligence! One easily overlooks the fact that the speaker is apparently talking about nothing more than a tool for creating a cross-referenced outline or network diagram. The computer itself is no more dealing in thoughts than does a typewriter.
Despite his loose language, this contributor may just conceivably have had properly humble intentions when he submitted his ideas for consideration. The same cannot be said for the writer who informed his colleagues that he had hit upon just the ticket for giving computers free will. It requires that we write a program for a "decision system with three agents":
The first agent generates a candidate list of possible courses of action open for consideration. The second agent evaluates the likely outcome of pursuing each possible course of action, and estimates its utility according to its value system. The third agent provides a coin-toss to resolve ties. Feedback from the real world enables the system to improve its powers of prediction and to edit its value system.
So much for free will. So much for the problem of values -- "ought" versus "is." So much for the question of who these "agents" are that consider possible courses of action, understand likely outcomes, and apply values. It is all beautifully simple.
This same contributor rebuts the notion that computers cannot experience feelings. His appeal is to diagnostic messages -- the words of advice or warning that programmers instruct the computer to print out when, for example, a user types something incorrectly.
A diagnostic message is a form of emotional expression. The computer is saying, "Something's wrong. I'm stuck and I don't know what to do." And sure enough, the computer doesn't do what you had in mind.
One wonders: are we up against an exciting new understanding of the human mind here, or an animistic impulse of stunning force? Do we confront theory or superstition? The USENET discussions in which such observations as these are launched continue month after month in all seriousness. The messages I have cited here were selected from hundreds of similar ones. We might hope that this is no more than the all-too-frequent descent of electronic discussion groups to a lowest common denominator. But there is evidence that the problem goes far beyond that.
As AI progresses (at least in terms of money spent), this malady gets worse. We have lived so long with the conviction that robots are possible, even just around the corner, that we can't help hastening their arrival with magic incantations. Winograd ... explored some of the complexity of language in sophisticated detail; and now everyone takes `natural-language interfaces' for granted, though none has been written. Charniak ... pointed out some approaches to understanding stories, and now the OWL interpreter includes a `story-understanding module'. (And, God help us, a top- level `ego loop.') /2/
I once sat in a conference where the head of a university computer science department was asked how a computer could possess a selfhood and a knowledge of itself. He immmediately replied that it is easy enough to create a program variable giving the computer a name for itself, and to cause the computer to associate the record of its past activity with this name. Thus, "since [the computer program] has a term for itself, it can remember what it did, and can tell you what it has thought about." So much for the nature of memory, personal experience, and selfhood.
These stories show that it is not only casual users who are liable to mistake book value for what the computer itself does or understands. If a word flashing on the screen suggests to our cooperating minds a native intelligence within the machine, sophisticated engineers appear similarly susceptible to the name of a variable in a program listing. One is reminded of the word's powerful evocations during ancient times. As we are often told, our ancestors did not always distinguish clearly between word and thing. The word bore within itself some of the inner, spiritual significance of the thing to which it referred. But this only makes our own behavior all the more puzzling, for we are certainly inclined to reject what we take to be the superstitions of the ancients. And we do not believe -- or at least we say we do not believe -- our machines to be possessed by spirits.
What, then, can we make of a technically trained person who casually attributes thoughts and feelings to a computer on the basis of its diagnostic messages? Has he become pathologically superstitious about machines, or has he instead become pathologically separated from his own world of thoughts and feelings, which he reduces to a collection of vague abstractions of the sort he can easily attribute to a computer? Or are these two sides of the same phenomenon?
Men are all alike.IN WHAT WAY
They're always bugging us about something or other.
CAN YOU THINK OF A SPECIFIC EXAMPLE
Well, my boyfriend made me come here.
YOUR BOYFRIEND MADE YOU COME HERE
He says I'm depressed much of the time.
I AM SORRY TO HEAR YOU ARE DEPRESSED
It's true. I am unhappy.
DO YOU THINK COMING HERE WILL HELP YOU NOT TO
BE UNHAPPYI need some help, that much seems certain.
WHAT WOULD IT MEAN TO YOU IF YOU GOT SOME HELP
Perhaps I could learn to get along with my mother.
TELL ME MORE ABOUT YOUR FAMILY
My mother takes care of me
WHO ELSE IN YOUR FAMILY TAKES CARE OF YOU
My father.
YOUR FATHER
You are like my father in some ways.
WHAT RESEMBLANCE DO YOU SEE
You are not very aggressive but I think you don't want me to
notice that.WHAT MAKE YOU THINK I AM NOT VERY AGGRESSIVE
You don't argue with me
WHY DO YOU THINK I DON'T ARGUE WITH YOU
You are afraid of me.
DOES IT PLEASE YOU TO BELIEVE I AM AFRAID OF YOU
My father is afraid of everybody
WHAT ELSE COMES TO MIND WHEN YOU THINK OF
YOUR FATHERBullies.
DOES THAT HAVE ANYTHING TO DO WITH THE FACT THAT YOUR BOYFRIEND MADE YOU COME HERE
As Weizenbaum points out, the computer's language analysis was very primitive. Its programming reflected a single context -- that of the therapeutic dialog -- and its "understanding" consisted substantially of keying in on certain words and drawing on a set of stock responses. (Of course, it also had to do some basic grammatical parsing of English text.) Nevertheless, Weizenbaum reports on the "shock" he experienced upon learning how seriously people took the program:
Once my secretary, who had watched me work on the program for many months and therefore surely knew it to be merely a computer program, started conversing with it. After only a few interchanges with it, she asked me to leave the room. Another time, I suggested I might rig the system so that I could examine all conversations anyone had with it, say, overnight. I was promptly bombarded with accusations that what I proposed amounted to spying on people's intimate thoughts .... I knew of course that people form all sorts of emotional bonds to machines .... What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people. /3/
There is a sense in which we must agree with Weizenbaum -- a sense that is central to my own argument. Yet I suspect he would acknowledge another side to the issue. How delusional is it to assume an intelligence behind the use of language? Language, as we all learn in school, is one of the chief distinguishing features of man. We simply never come across language that has not issued, in one way or another, from a human mind. If I find a series of words neatly impressed upon the sand of a desert island, I will conclude -- no doubt correctly -- that I am not alone. There is another human speaker on the island.
Furthermore, we have become fully attuned to mechanically mediated human communication. While telephone, radio, or TV might hopelessly disorient a time-traveling Roman, we take it as a matter of course that these devices put us in touch with other people. Weizenbaum's secretary, quite undistracted by the mechanical contrivances she was dealing with, immersed herself from habit in the meaning of the text addressed to her, and she felt (with good justification) that this text originated in another mind, one that had considered how to respond to just the sorts of comments she was making. What she was most likely not doing was considering explicitly whether she was speaking with the computer itself, or a programmer, or some other person. She was simply conversing with words. Who was behind them didn't matter. The episode may say more about the pervasive and accustomed anonymity of our society than anything else.
If a phone connection puts me at one mechanical remove from my conversational partner, a recorded phone message more than doubles the indirection, for here there is no possibility of interaction. But, no, that's not quite true. As phone systems become ever more sophisticated, I am allowed to push buttons in an increasingly articulate manner. At the same time, the spliced-together snippets of recorded speech manage to respond in an increasingly intelligent manner. And, like Weizenbaum's secretary, I follow along with all the seriousness of someone conversing with a real person, even if I am more or less aware of the arrangement's limitations. This awareness will no doubt attenuate with time, even as the mechanical devices gain in deftness.
Many in our society have only recently experienced the shock that comes when one first realizes that the "person" who rang the phone is really a recording. But those among us who are already accustomed to recordings will readily acknowledge a certain process of acclimatization: as the collage of recorded and "real" voices becomes more and more intricate, and as the underlying programming responds more and more flexibly to our needs, we make less and less of a distinction between the various levels of genuineness. We are comfortable doing business with the words themselves.
We live out much of our lives under the guidance of these words-as- artifacts. How can we do otherwise? How can I pierce behind the intricate mechanism mediating the words I hear, so as to discern the true speaker? How can I discover more than a few fragments of the individuality of Peter Jennings or Tom Brokaw behind the formulas and technology of the evening news? (Think how different it would be to watch and listen as the cameras inadvertently picked up a half-hour's off-the-air conversation between Brokaw and one of his colleagues. And how different again to participate in a face-to-face exchange with them.)
We are so used to words that have become disconnected from their human source that we scarcely notice the peculiarities of our situation. But we do notice on at least some occasions, as the unflattering stereotype of the bureaucrat makes clear. When I quibble with a disenfranchised clerk over some irrelevant regulation, with whom am I speaking? Not the clerk himself, for he has no authority to speak on his own account. (That is probably the main cause of my frustration; I thought I was going to converse with a person.) But if my conversation is not with the clerk, who is it with? Nobody, really -- which is why I quickly begin to blame the System.
The fact is that, like Weizenbaum's secretary, I am merely conversing with words, and these words are produced by a vague mechanism neither I nor anyone else can fully unravel. Yes, the words somehow originate with human beings, but they have been subjected to a kind of organizational/mechanical processing that renders them simplistic, too-logical, slightly out of kilter. They are impossible to trace, and so I don't try. Why try? It is the disembodied words that determine my fate. They are the reality.
One way to think of the challenge for our future is this: how can I work toward a society in which every transaction is as deeply human a transaction as possible? To make an exchange human is to reduce the distance between words and their source, to overcome the entire mediating apparatus, so that I am myself fully present in my words. Even when stymied by a bureaucracy, I can at least choose to address (and respect) the human being in front of me. This will, in fact, encourage him to step out of the System in some small way rather than retreat further into it as a defense against my anger. On the other hand, I support the System just so far as I give further impetus to the automated word.
There may appear to be a paradox here. On the one hand, the increasing objectification of what is most intimately human -- our speech; on the other hand, an anthropomorphizing liaison with our mechanical contrivances. But, of course, this is not really a paradox. When speech detaches itself from the speaker, it is indeed objectified, cut off from its human source. But it still carries -- if only via our subconscious -- some of its ancient and living powers. And so its association with machinery readily evokes our personification of the machinery.
We may think ourselves freed from superstition. But if we take superstition to be a susceptibility to the magical effects of words, then it would be truer to say that ours is the age in which superstition has come into its own. Having reduced the word to a dead abstraction residing outside ourselves, we subject ourselves to its invisible influence. Our single largest industry, centered on Madison Avenue (but also operative in every corporate Marketing Department) is dedicated to refining the instruments of magical control. Seeming to be alien, a hollow physical token and nothing more, approaching us only as a powerless shape from without, the word nevertheless has its way with us. We detach the word from ourselves and it overpowers us from the world.
Nor is it an accident that the great social and political ideologies have arisen only during the last couple of centuries. These elaborate word-edifices, detached from their human sources, sway and mobilize the surging masses. The passionate believer in an -ism -- what is it he believes in? An idea from which all human meaning has been purged, leaving only an abstraction and the controlling passion of belief itself. That is what makes the ghastly, inhuman contradictions possible: a communist workers' paradise to be achieved by disfranchising or massacring the workers; a capitalist common good to be achieved through the universal cultivation of selfishness.
It is quite otherwise with ideology. For Communist revolutionaries around the world, not much depended on the person of Marx or Lenin, or on the authenticity of words attributed to them. Faith was vested in empty generalities -- the proletariat, the revolution, the classless society. Words that change the course of the world are no longer bound to their human source. Unlike the symbol and parable -- and much more like scientific descriptions -- they have become abstract and capable of standing by themselves, for they are nearly contentless. They are automatic -- fit to be spoken by the System -- and therefore the human subconscious from which the System has arisen is invited to supply the real meaning.
Some people believe we have seen the end of ideology. My own fear is that we are seeing its perfection. The disembodied word no longer requires even the relatively impersonal support of the activist's cell, the political movement, the faceless bureaucracy, the machinelike corporation, the television evangelistic campaign. The machine itself, uncannily mimicking the human being, now bears the word alone -- and we call it information. Our -ism is declared in our almost religious devotion to a life determined, not from within ourselves, but by divine technological whim.
It is easy enough to see a danger in the reaction of Weizenbaum's secretary to ELIZA. For despite it being an understandable reaction in view of our culture's detachment of the word, it is clearly not a healthy reaction. She evidently proved blind to the laughable limitation of her therapist and the essential artificiality of the exercise, because she could not distinguish properly among book value, mechanical activity, and human presence. And having lost her ability to trace word to speaker, she must have lost as well some of her ability to deal truly with human beings in general; for the human being as productive spirit, as the source of the word, had at least temporarily escaped her reckoning.
We may scorn her for that, but it would not be wise. Better to note that the test she failed becomes daily more subtle, and that the rest of us, too -- whether in passively absorbing television commercials, or beating our heads against petty bureaucracies, or allowing electronic mail to put us into "automatic mode" -- fail it every day.
1. Cobb, 1990.
2. McDermott, 1981: 145-46.
3. Weizenbaum, 1976: 3-4.
Steve Talbott :: The Future Does Not Compute, Chapter 18