Sketchify Services

Zeljko Obrenovic, TU/e Eindoven
Last modified: 27/08/2009

Table of Content

Introduction
Wii Remote Service
Text-to-speech Service
Speech Recognizer Service
Face Detector Service
Motion Detector Service
Face Expression Service
Google News Service
Examples to Import in Sketchify

Introduction

With Sketchify services, designers can introduce in their sketches real but “trimmed down” functionality of input/output devices and software components from various domains. We have incorporated many different services within AMICO Sketchpad, including text-to-speech engines and speech recognizers, camera-based face and motion detectors, VRPN devices (such as 3D trackers and buttons), MP3 and MIDI players, more specialized devices such as the Wii Remote, Nabaztag, or Phidgets, Web services (such as Google spelling checker and search engine), semantic services (such as the Wordnet definition service), and many others. Appendix D provides a complete list of the software services that are currently supported by our platform.

Sketchify services can bring within the reach of the designer a huge number of available software components. In order to accomplish this, we build on our previous work for integrating heterogeneous software components, where we used a service oriented approach[1] to connect components written in different languages [Obrenovic and Gasevic 2007]. In essence, our Sketchify services are standalone applications that AMICO Sketchpad runs as background processes, and which connect to AMICO Sketchpad through one of the many network interfaces being supported, updating and reading variables. Our services are simplified as we usually do not map the full functionality of the component, but only it’s most representative parts.

Table 1. Two Sketchify services from the domain of speech interaction, and the variables they use. The designer sees and interacts with these services only through these variables, while our platform hides the complexity of the service execution.

Software Service		Variables	Type	Description
	FreeTTS text-to-speech engine	tts-input	[in]*	Text to be pronounced.
	FreeTTS text-to-speech engine	tts-status	[out]	Status of the engine: ‘loading’, ‘ready’, ‘talking’
	Sphinix-4 speech recognizer	speech-command	[out]	Recognized phrase
	Sphinix-4 speech recognizer	sphinix4-status	[out]	Status of the engine: ‘loading’, ‘ready’, ‘recognizing’

* [in] = input parametro for service, [out] = result of the service

From the designer’s point of view, our platform offers a simple interface to start and stop services, hiding the complexity and diversity of technologies that a software service may use, and providing a simple and uniform variable-based interface towards them (Table 1). In this way, we can bring components from various domains within the reach of the designer, allowing a designer to directly experience possibilities and limitations of technologies, without requiring programming skills on the side of the designer.

Wii Remote Service

Wii Remote (Hhttp://en.wikipedia.org/wiki/Wii_Remote), which connects to a PC using a Bluetooth link, is a complex sensing platform. It can track infra-red sources, and contains three acceleration sensors, various buttons, a vibrator, a simple speaker and some status LED diodes. It can also be used to connect more devices, such as Wii Nunchuk, which contains a joystick and more buttons. Other related devices, such as Wii Fit, can also be used. Our Wii software service is based on the C# demo programs that come with WiimoteLib (http://www.codeplex.com/WiimoteLib).

	Variables	Type	Description
Wii Remote & Nunchuk Wii Fit	wii-accel-x	[out]	X-axis acceleration (in Gs)
	wii-accel-y	[out]	Y-axis acceleration (in Gs)
	wii-accel-z	[out]	Z-axis acceleration (in Gs)
	wii-ir-<object-id>-x	[out]	X position of a tracked infrared object (0..1.0)
	wii-ir-<object-id>-y	[out]	Y position of a tracked infrared object (0..1.0)
	wii -<button-id>-state	[out]	State of each Wii Remote button (‘True’ or ‘False”)
	wii-vibration	[in]	Causes the Wii Remote to vibrate for a given time (in milliseconds)
	wii-fit-balance-<n>	[out]	Data from one of four balance board sensors

Video. Wii Remote Service.

Text-to-Speech Service

Text-to-Speech service pronounces the text given in a variable.

Variables	Type	Description
tts-input	[in]	Text to be pronounced.
tts-status	[out]	Status of the engine: ‘loading’, ‘ready’, ‘talking’

Video. Text-to-speech Service.

Speech Recognizer Service

Speech recognizer service updates a variable based on the user speech input.

Variables	Type	Description
speech-command	[out]	Text produced by the recognizer
sphinix4-status	[out]	Status of the engine: ‘loading’, ‘ready’, ‘talking’

Video. Speech Recognize Service.

Face Detector Service

Face detector is a Web camera based service that detects faces in fron of the screen, and update the variables with coordinates of detected faces, and number of detected faces.

Variables	Type	Description
number-of-faces	[out]	Number of faces detected: 0, 1, 2, …
face-<id>-x1	[out]	Left X coordiate
face-<id>-y1	[out]	Top Y coordinate
face-<id>-x2	[out]	Right X coordiate
face-<id>-y2	[out]	Bottom Y coordinate

Video. Face Detector Sample.

Motion Detector Service

Motion detector service is a Web camera based service, which visually detects the intensity of motion in fron of the camera, and updates the varaible with the number of pixels that are different between current and previous image.

Variables	Type	Description
motion-intensity	[out]	Intensity of motion derived from the difference between successive images.

Video. Motion Detector Service.

Face Expressions Service

We also support a simple face expression animation module, based on The Expression Toolkit (Hhttp://expression.sourceforge.net/H) – an open-source procedural facial animation system. In our adaptation, the face animation runs in a separate window, and through variables a designer can set basic and complex facial expressions, as well as define the “mood” of the character.

Variables	Type	Description
face-expression	[out]	ID of the face expression to be animated (1..41)
face-composite-expression	[out]	ID of one of 12 complex face expressions to be animated
face-mood	[out]	Face mood during animation (‘happy’, ‘sad’, ‘angry’, ‘scared’, ‘tired’, ‘skeptical’)

Video. Face Expressions Service.

Google News Service

Google News service returns titles, and links to news for a given keywords.

Variables	Type	Description
google-news-query	[in]	User query
google-news-title-<n>	[out]	Title of the news item <n> (n = 1..5)
google-news-date-<n>	[out]	Date of the news item <n> (n = 1..5)
google-news-link-<n>	[out]	Link to the news item <n> (n = 1..5)
google-news-status	[out]	Status of search (‘ready’, ‘working’, ‘finished’)

Video. Google News Service.

Examples to Import from URL in Sketchify

You can open the examples from this tutorial in Sketchify by using "Import from URL" function (in the File menu), and copy-and-paste there the URL of an example.

Wii Remote Example, URL:
Wii Fit Examples, URL:

[1] We define a software service as a self-contained functional unit in which service consumers interact with the service through a well defined interface. In this model, the consumer does not know (or care) "how" the service implements the requested action - only that the service performs "what" is defined by its published interface.