AMICO Sketchify End-User Programming

Zeljko Obrenovic, TU/e Eindoven
Last modified: 27/08/2009

Table of Content

• Introduction
• Spreadsheets commands
• Scripting Support
  • Javascript
  • BeanShell
  • Python
  • Groovy
  • Ruby
  • TCL
  • Sleep
  • Haskell
  • Prolog
• Examples to Import in Sketchify

Introduction

In order to define rich interactions, designers need complex ways of influencing the behavior of sketches, for instance, by means of testing conditions, doing simple calculations, or creating sequences of actions. In many interaction scenarios, such as in speech applications, the sketches may not have any visible elements, and the sketch behavior becomes the only “object” being designed. Having in mind that most designers are not experienced programmers, and that there is a huge diversity between designers, we decided to connect our environment to a range of end-user programming tools which are likely to be accessible and usable for designers.  

In AMICO Sketchpad, spreadsheets and scripting languages can be used to quickly outline the behavior of sketches. Spreadsheets and scripts are proven, highly productive and simple to learn and use end-user development paradigms. With such tools designers, who are usually not experienced programmers, can quickly define more complex interaction scenarios, without requiring intensive programming. We currently support OpenOffice.org CALC spreadsheets, and several higher-level scripting languages including Javascript, Python, BeanShell, Groovy, Ruby, TCL, Sleep, Haskell, and Prolog.

In all cases, existing end-user development solutions  were extended with mechanisms to update and read AMICO Sketchpad variables, which is the chosen way to receive, cause or process interaction events (Appendix C). Designers can use any of these individual end-user programming tools, or can combine them, describing, for example, part of the behavior in spreadsheets, and another part in a script. Figure 1, illustrates how a simple “echo” behavior (on update of one variable, another variable is updated with the same value) can be accomplished with both spreadsheet formulas and simple script code in any of six different scripting languages.

Figure 1. Examples of spreadsheets formulas and simple scripts written in different languages. All examples implement the same “echo” function: on update of a variable, another variable is updated with the same value

Figure 2 and Video 1 illustrates how spreadsheets and scripts can be used in combination with simplified software services and freehand sketching. The motion detector software service tracks the intensity of the user’s motion in front of the camera, and updates the variable “motion-intensity”. This variable is passed to a spreadsheet that contains additional formulas to process this variable, more specifically, to map the motion intensity value into the range from 0.0 to 1.0. This derived value is passed into the new variable “transparency”, and the transparency of the freehand sketch responds to this variable. As a result, if the user is not moving, the image is invisible (completely transparent); the more she or he moves, the less transparent and more visible the image becomes. Figure 2b also shows how the same logic could be defined using a script instead of a spreadsheet.

Figure 2. Implementation of the interaction sketch described in Section 1. The logic behind the sketch can be manipulated through spreadsheet formulas (a),or by means of a script (b).

function variableUpdated( name, value, old ) { 
    if (name == "motion-intensity") { 
        t = parseFloat( value ) / 5000; 
        amico.update( "transparency", t > 1 ? "1" : "" + t ); 
    } 
}

Video 1. Creating a complex examples using spreadsheest (in version 0.5).

Video 2. Using Scripting in Sketchify (see also how to open this example in Sketchify).

Our support for end-user programming also allows for creating interactive sketches without graphical elements. For example, to sketch speech interaction, we may use a spreadsheet containing only a speech recognizer and a text-to-speech engine.

Our environment provides access to several end-user programming tools that can help designers to define more complex interaction behaviors in their sketches,. We currently support OpenOffice.org CALC spreadsheets, as well as several higher-level scripting languages including Javascript, Python, BeanShell, Groovy, Ruby, TCL, Sleep, Haskell, and Prolog.

Spreadsheets commands

Spreadsheets were included in AMICO Sketchify by means of add-ins for OpenOffice.org CALC and Microsoft Excel. From the end-user point of view the add-ons constitute only a few additional functions that are accessible as spreadsheet formulas (Table 1 and Video 3). The introduced functions allow users to  update or read all variables of any service connected to AMICO Sketchify. For example, the expression AMICO_WRITE("spelling";B10) is evaluated every time when cell B10 is updated, and calls the Google spelling checker service (which is triggered by updating the AMICO variable “spelling”); while expression AMICO_READ("spelling-suggestion") obtains the current value of the variable “spelling-suggestion”, which is updated by the Google spelling checker adapter. Reading a variable once also registers an application for notifications about future changes of this variable.

Spreadsheets run as separate process and connect to our middleware through TCP and UDP interfaces.

Video 3. Simple example of working with spreadsheets.

Scripting Languages

Scripting languages are relatively easy to learn by developers as they use typeless approaches to achieve a higher level of programming. This tends to result in much more rapid application development than when using low-level programming languages. However, there is a huge diversity of scripting languages, each being supported by a significant community. Therefore, instead of choosing one specific language, we decided to support most popular scripting languages, including:

1. Javascript, a dynamic, weakly typed, prototype-based language,
2. BeanShell and Groovy, two Java-based scripting languages,
3. Python, a very high-level programming language supporting multiple programming paradigms (object oriented, imperative, and functional),
4. Ruby, a dynamic, reflective, general purpose object-oriented programming language,
5. TCL, a popular tool command language,
6. Sleep, a procedural scripting language inspired by Perl and Objective-C,
7. Prolog, a logic programming language.

Except for the support for Prolog, which is based on the JLog project[1], all scripting support is based on the Java Scripting Project[2]. Our middleware runs scripting engines for each of these languages, extending each language with functions for updating and reading variables (i.e., functions UPDATE and GET). To receive notifications about variable updates (i.e., the REGISTER function), a script needs to contain a hook function called variableUpdated (see Figure 3a), which can be called by the middleware when a given variable is updated. For Prolog, which is a non-procedural scripting language, we introduced special predicates for updating, reading and receiving notifications about variable updates (see Figure3b). We also support XSLT scripts, implemented using standard Java XML libraries. These scripting extensions allow the developers to use (a mixture) of different programming modes, such as declarative programming, object-oriented programming, or logic programming, when creating their services.

Javascript

We support standard Javascript constructions, adding two commands to work with AMICO Sketchify varaibles:

• amico.update("variable-name", "variable-value");
• variable = amico.get( "variable-name" );

Code should be placed in the hook function which will be called by AMICO when registered varaibles are updated.

function variableUpdated( name, value ) {
    amico.update( "new-variable", value + "" );
}

BeanShell

We support standard BeanShell constructions, adding two commands to work with AMICO Sketchify varaibles:

• amico.update("variable-name", "variable-value");
• String variable = amico.get( "variable-name" );

Code should be placed in the hook function which will be called by AMICO when registered varaibles are updated.

void variableUpdated( String name, String value ) {
	amico.update("bsh-msg","value is " + value);
}

Python

We support standard Python constructions, adding two commands to work with AMICO Sketchify varaibles:

• amico.update("variable-name", "variable-value")
• variable = amico.get( "variable-name" )

Code should be placed in the hook function which will be called by AMICO when registered varaibles are updated.

def variableUpdated( name, value, oldValue ):
	if name == "py-input":
		amico.update("py-msg","python forever")
	return

Groovy

We support standard Groovy constructions, adding two commands to work with AMICO Sketchify varaibles:

• amico.update("variable-name", "variable-value")
• variable = amico.get( "variable-name" )

Code should be placed in the hook function which will be called by AMICO when registered varaibles are updated.

void variableUpdated( name, value ) {
	amico.update("groovy-msg", name + " = " + value)
}

Ruby

We support standard Ruby constructions, adding two commands to work with AMICO Sketchify varaibles:

• Java::Amico::Variable.update('variable-name', 'value')
• var = 'puts ' + Java::Amico::Variable.get('variable-name')

Code should be placed in the hook function which will be called by AMICO when registered varaibles are updated.

include Java

def variableUpdated( name, value ) 
   Java::Amico::Variable.update('ruby-msg', name + ' = ' + value)
   var = 'puts ' + Java::Amico::Variable.get('test')
   puts var
end

TCL

We support standard TCL constructions, adding two commands to work with AMICO Sketchify varaibles:

• java::call amico.Variable update "tcl-msg" "$name = $value"
• set string [java::call amico.Variable get "tcl-msg"]

Code should be placed in the hook function which will be called by AMICO when registered varaibles are updated.

package require java

proc variableUpdated {name value} {
	java::call amico.Variable update "tcl-msg" "$name = $value"
	set string [java::call amico.Variable get "tcl-msg"]
	puts "result is $string";
	return ""
}

Sleep

We support standard Sleep constructions, adding two commands to work with AMICO Sketchify varaibles:

• amico_update("variable-name", "value");
• $value = amico_get("variable-name");

Code should be placed in the hook function which will be called by AMICO when registered varaibles are updated.

sub variableUpdated 
{
	$name = $1;
	$value = $2;
	
	amico_update("sleep-msg", "$name is now $value");
	
	println(amico_get("sleep-msg"));
} 

Prolog

We support standard Prolog predicates, adding several predicates to work with AMICO Sketchify varaibles:

• variable(Name,Value)
• amico_update(Name,Value)
• amico_update_delayed(Name,Value,DelayInSeconds)
• amico_get(Name)

Code should be placed in expressions which will be called by AMICO when registered varaibles are updated.

variable(Name,Value) :- amico_update("prolog-msg",V).  
variable("prolog","test") :- amico_get('prolog',V), amico_update("prolog-test",V).

Examples to Import from URL in Sketchify

You can open the examples from this tutorial in Sketchify by using "Import from URL" function (in the File menu), and copy-and-paste there the URL of an example.

• Scripting Examples, URL:

References:

• Obrenović Z. and Gašević D., "End-User Service Computing: Spreadsheets as a Service Composition Tool," IEEE Transactions on Services Computing, vol. 1, no. 4, pp. 229-242, 2008.
• JavaScript Tutorial, http://www.w3schools.com/JS/