One Cannot Not Interact

Computational Design, University of Wuppertal
 

Monsieur Jourdain: And this, the way I speak. What name would be applied to...? Philosophy Master: The way you speak? Monsieur Jourdain: Yes. Philosophy Master: Prose Monsieur Jourdain: It's prose. Well, what do you know about that!? these forty years now, I've been speaking in prose without knowing it.   (Molière, The Bourgeois Gentleman)

A position paper for a workshop on human-computer interaction, beginning with a quote from literature is unusual. But here you have the essence of my position: All those involved in human-computer interaction (HCI) "speak" semiotics, whether they are aware of it or not. In paraphrasing Paul Watzslawick's famous axiom-One cannot not communicate-I submit (again) to the HCI community that one cannot not interact; and because interaction is based on signs, one cannot not "semiotize." That is, one cannot avoid semiotics. Indeed, we express ourselves through various signs; we interpret them, and thus become part of the infinite sign process. Our practical experiences (work, learning, various transaction, etc.) are more and more semiotic in nature, i.e., more and more mediated by signs, visual or of other types. As a semiotic engine, the computer accelerated the semiotization of human life, including the semiotization of the interaction between humans and machines.

As is the case with every broad statement, it means close to nothing unless it is substantiated. Here are the arguments for my axiom:

Interactions can be direct (e.g., raising one's hand in greeting, pounding on a table, pulling a lever, cracking an egg) or mediated (e.g., showing a picture in reference to a physical phenomenon, touching a soft-button on a copy machine screen in order to execute a desired operation, clicking on an icon in order to connect to a Web address). Our interaction with computers is mediated. There is no way to avoid this condition.

The intersection area-where and how the mediation takes place, i.e. how interactions occur-is of interest from many perspectives: of psychology, cognitive science, technology, education, communication theory, design, economics (how effective is the interaction; what costs does it entail?), culture (to what extent does it fit the cultural context), among others. Each of these disciplines can affect our understanding of HCI and can, within the framework of methodological investigation, support the implementation and evaluation of user interfaces and other aspects of interaction. Some of these perspectives are more important than others, some are easier to consider when proceeding with the task of making HCI possible and effective. The only characteristic impossible to eliminate is the semiotic characteristic, because regardless of the nature of the interaction considered, it always takes place through the intermediary of signs. This intermediary, which is the object of semiotics, constitutes an underlying element, in the absence of which no interaction is possible.

Monsieur Jourdain had spoken prose all his life without having the slightest inkling of it. People around him understood what he had to say. The same things holds true of HCI in respect to semiotics. A minority of those involved in the field are aware of semiotics. But all HCI professionals want to gain access to the secret of designing effective means and methods of interaction. Monsieur Jourdain confessed, "I am in love with a lady of great rank and quality and wish to ask your help in writing her a note. . . ." Not unlike him, HCI professionals are dedicated to (and sometimes in love with) the computer and would like all the help they can get in order to make the interaction better and more intuitive. They want to support a variety of interactions corresponding to the variety of users, and eventually they to reach that evasive goal (but a very assuring slogan!) of user-centered interaction, also known (in Norman's words) as human-centered design.

Short of understanding how signs are constituted and how interpretations emerge in a pragmatic context (i.e., what is called sign process), moreover, short of understanding what the relation between the machine, the program, and the user is, the HCI community will at best continue to provide solutions that in fact reduce the effectiveness of the computation it is asked to support instead of enhancing it. In view of all this, I argue for the following practical goals:

1. The necessity of a semiotic foundation.
   As long as the complexity of computation was very low (due to the fact that computers are still in a very early stage of development) and the interaction limited, the HCI community has been able to get away with even rudimentary semiotic notions. All kinds of dilettantes and peddlers of ideas picked up during a conference coffee break have been successful in promoting metaphors, visual conventions, and I/O devices whose packaging was better than their real performance. (Let's face it, after the desktop metaphor, which was conceived on a semiotic foundation, almost nothing of similar impact has emerged.) As the complexity of computation increases, richer interactions are possible. Especially as networking poses new challenges irreducible to the artistry of the self-declared information architects and screen designers, interactions take new forms. Henceforth, the necessity of a semiotic foundation will make it impossible for HCI professionals to avoid studying the various aspects of interaction that networking brings about. (Let me point to the relation between the represented and the representation, the dynamics of representation, visual codes, interpretation, multimedia, to name only a few of such aspects.)

   My Computational Design Program (see Computational Design. Design in the Age of a Knowledge Society, in formdiskurs, Journal of Design and Design Theory, 2, I/1996. Frankfurt/Main: Verlag Form. pp. 40-60) and Joseph Goguen's program at UCSD are dedicated to the exploration of the semiotic aspects of interaction. The Group at the PCU University in Rio de Janeiro pursues semiotic engineering as a well defined field of applied semiotics and information science. In Denmark, the Centre for Human-Machine Interaction (CHMI) is making progress along the line I suggested.

2. Integration of semiotic considerations at the level of operating systems
   Since an operating system is the program through which we address the computer (hardware, software), it follows that HCI considerations can no longer be limited to the cosmetics (or illustration) of the operating system's functionality, but need to start with the design of the operating system. This is where interaction is pre-defined. At this high level, semiotic considerations should go hand in hand with technical considerations. Whether multi-user, multiprocessing, multitasking, or multithreading, operating systems are high-level semiotic entities that act as intermediaries between applications and users. Regardless of what kind of representations are used (line commands, icons, dial controls, etc.), these are signs mediating the expression of practical goals with the aim of using the knowledge embodied in the machine (at the hardware level and in software expression) for certain practical purposes. This is why the integration of semiotic considerations at the level of operating systems is fundamental for improved human-machine interaction.

3. Semiotic fundation of academic programs in HCI
   In order to optimize the use of computation, we need to address the complexity of HCI. This interaction is not reducible to technological innovation, but essential to its success. Thus, it follows that there is a need for establishing academic programs dedicated exclusively to the study of interactions in all possible computing environments. These must integrate, as their foundation, the scientific and technological aspects of computation, as well as the semiotics of computation. In particular, design, which also has a semiotic foundation, has to become a subject matter for all computer science programs. Interactions can be enhanced and made richer by design; they can be customized and adapted to individual characteristics.

4. Complement usability with semiotic adequacy evaluations
   Last but not least: The obsession with user performance, expressed in usability studies, has to give way to a balanced understanding of what we evaluate. It is time to complement usability with semiotic adequacy evaluations (concerning types of signs, context and meaning, cultural characteristics).