Home → TutorialsBasic Idea

Sketchify: Sketching as a Rapid Manipulation of Interactive Material

Zeljko Obrenovic, TU/e Eindoven
Last modified: 27/08/2009

Table of Content

Introduction

Sketching is at the heart of design. Many studies of the design practice, such as recent contributions from Buxton, Krippendorff, and Moggridge, have called attention to sketching as an omnipresent element of any disciplined activity of design. Disciplines such as graphical design and architecture can boast a rich tradition in sketching, and offer courses to students in order to improve their sketching skills. However, interaction designers cannot efficiently employ existing sketching techniques while designing new user interfaces. Buxton has argued that while it is relatively easy to sketch the physical shape of an interaction device or the graphical layout of a user interface, interaction designers lack tools that enable them to sketch the dynamics of the interaction, let alone the overall user experience [Buxton 2007]. For example, pencil and paper provide few means to sketch speech interaction, or to illustrate interaction scenarios in domains such as ambient intelligence, tangible interaction, multimodal interaction or pervasive computing.  

The identified issue of lack of tools for sketching is also confirmed by our own experience in the education of students of interaction design. Based on these practical experiences and on our understanding of existing theoretical contributions, we introduced a novel approach and tool for sketching, adopting two main premises in our work:

  • The primary objective of interaction designers in the early design stage is clarifying the user experience and the associated user-system interaction [Moggridge 2007]. Interaction designers (and students) need better techniques than those currently available for sketching such experiences and interactions; analog to traditional sketching, these techniques need to combine speed and freedom of expression, and need to assist in producing an output that invites discussion.
  • Sketching should be extended beyond the simple creation of a pencil trace on paper to deal with important attributes of the overall user experience, especially time, phrasing, and feel [Buxton 2007].

To support and explore our view, we have developed AMICO Sketchify, a sketching tool that implements our extension of paper and pencil sketching to the more generic concept of rapid manipulation of interactive materials. Interactive material can be any piece of software/hardware that represents or simulates part of the interactive user experience, such as input from sensors, output in the form of audio, video or drawn sketches, interaction with Web services, etc. Through the manipulation of interactive materials, designers create interactive sketches, which are rough illustrations of the interaction scenarios or interaction techniques that they have in mind. With our tool designers can, for instance, combine elements of traditional freehand sketching with end-user programming, such as spreadsheets or scripting, needed in order to create intelligent system behavior.

Sketching as Rapid Manipulation of Interactive Material with a Diversity of Tools

AMICO Sketchify is an extensible toolset for sketching interactive systems. In this section we present the basic idea and the principles behind our toolset (see Video 1). In order to deal with many different attributes of the overall user experience, in addition to the look of the user interface, we propose extending the concept of paper and pencil sketching to the more generic concept of rapid manipulation of interactive material. We define interactive material as any piece of software/hardware that represents or simulates part of the system’s behavior and interaction, such as input from sensors, audio or video output, interaction with Web services, but also simple drawings. Through the manipulation of interactive materials, designers can create “interactive sketches” that provide a rough illustration of the interaction scenario or interaction technique that they have in mind. Such sketches should enable the designer, as well as end users, to experience, rather than to see (or imagine), the interaction. Paper and pencil sketching can be viewed as a special case of such an approach, where the interactive material consists of drawings and images, and the main way to manipulate the material is through pen-based selection and gesturing. We propose to treat sketching in much broader terms, more specifically, as a means to rapidly develop ideas, regardless of the media being used.

Video 1. Basic idea of the predecessor of Sketchify (called AMICO Sketchpad, version 0.5).

Conceptual Model

AMICO Sketchify implements the concept of rapid manipulation of interactive materials by combining several existing and proven approaches, including freehand sketching, end-user programming, and software services. Figure 1 illustrates the conceptual framework of AMICO Sketchify. It distinguishes two groups of components in the system:

  • Tools that help a designer to create or bring into the design space interactive materials and services, for instance through connections to simplified software services or links to external design environments.
  • Tools that enable a designer to sketch interaction through the rapid manipulation of these interactive materials, using freehand sketching, various forms of end-user programming, or paradigms already supported in available tools.

Figure 1. AMICO Sketchify combines traditional free-hand sketching with support for state transitions and animation, end-user programming, and simplified software services, using variables to connect all elements.

a

 

Integration and Variables

To integrate all elements of our solution, and to enable their orchestration and synergetic use, we applied a loosely coupled coordination model, where all elements of AMICO Sketchify communicate indirectly by exchanging messages through a centralized repository of variables. Alternatively, the tools can also communicate through files and through the system clipboard. AMICO Sketchify runs a blackboard server with a simple repository of named slots, called variables, enabling external applications to update and read variables using one of the many available communication interfaces. We reused components from the Adaptable Multi-Interface COmmunicator (AMICO) project (see Appendix B for details) to implement this server. Variables provide a simple and uniform abstraction mechanism, enabling a designer to work with very diverse elements using the same set of operations. Properties of freehand sketches, such as their position or transparency, or user actions such as item selections, can be mapped to blackboard variables. Spreadsheets or scripts can subsequently read, process, and update these variables. Software services can receive arguments and send back results through such variables (Figure 2 and Video 2). Lastly, through extension mechanisms other platforms can update, read or register for the notification of variables.

Figure 2. Communication of diverse elements of AMICO Sketchify through variables. In this example, a speech recognizer updates the variable “speech-command” with the recognized word. Within a spreadsheet, this variable is read, and the variable “tts-input” is updated in response. Update of this latter variable is propagated to a text-to-speech engine that pronounces the given text.

a

 

Video 2. Wii Remote as a service in AMICO Sketchify (see also how to open this example in Sketchify).

In addition to exchanging messages through variables, tools can also communicate through files and through the system clipboard. For example, our freehand sketching environment saves graphical elements of sketches into image files that can subsequently be opened in most existing image manipulation programs. Through the system clipboard, images or text can be copied between our environment and many other ones.

Examples to Import from URL in Sketchify

You can open the examples from this tutorial in Sketchify by using "Import from URL" function (in the File menu), and copy-and-paste there the URL of an example.

To Probe Further