HandLense: A sensor analysis toolkit

The HandLense tool kit provides a means for analyzing, studying, and augmenting data collected by the house_n research group. It is a ‘work-in-progress’ and should be used with this in mind. Any comments, suggestions, or general inquires should be made to the house_n group at housen-handlense [at symbol] media [dot] mit [dot] edu.

II. Datasets

The HandLense software will only work with the PLIA2 dataset and datasets collected thereafter. It cannot be used with legacy datasets, such as the PLIA1 dataset. See the PlaceLab Data Overview website for information on recent datasets.

III. Introduction

This document provides a walkthrough of the HandLense toolkit. This document will take you step-by-step through the application and introduce the different components along the way. This guide is meant to be gone through in sequential order, but you may choose to go through in your own order or explore the application on your own. Enjoy.

IV. Installation

The HandLense toolkit is a Microsoft Windows application written in the programming language C#. The Microsoft Installer file is currently available at:

http://www.mit.edu/~rockinso/Handlense/HandlenseExeInstaller.msi

Send mail to housen-handlense [at symbol] media [dot] mit [dot] edu if you would like to be added to the distribution list and be notified of software updates.

i. Uninstall Previous Versions

If a previous versions HandLense have been installed, first use Add/Remove Program utility to uninstall this previous version

ii. Point your Browser to the Installer File

Point your favorite Internet browser to: http://www.mit.edu/~rockinso/Handlense/HandlenseExeInstaller.msi

iii. Choose to run the installer when prompted

You may be prompted several times to install HandLense. Just click ‘Run’ at the prompts .

iv. Choose to Install HandLense and Follow Dialogs

Follow the installation prompts. Upon successful installation, a shortcut to the HandLense program will be created on your Desktop. Double click this icon to start the HandLense toolkit.

NOTE: You may be prompted to install the .NET 2.0 framework. This is required to run the HandLense toolkit.

V. HandLense Container

The HandLense toolkit is a Multi-Document Interface (MDI). This means that the different HandLense components described in this document reside within a containing window. This containing window is aptly named HandLense Container. It is the main application window and the one presented when HandLense is first loaded. All the different components may be accessed through the Main Menu of this window.

VI. Loading Sensor Data

The first step in using the HandLense toolkit is to load the sensor data. To do this select File -> Open Data Folder from the HandLense Container Main Menu.

This selection will popup a Folder Browser Dialog. Point this dialog to the folder above the Day Sensor Folder (labeled 2006-04-24 in the sample data set). For example, if you download the sample dataset into a folder named House_n Sample Data, point the browser to this folder. Note: It is important you do not change the folder structure of the sample dataset.

Once you select the dataset to load you will see a progress bar informing you that data is being loaded.

VII. The Views

The components of the HandLense toolkit are called views. The main menu of the HandLense Container provides access to all the available views.

i. Playback Remote

The Playback remote is just that – a Media Player type control that allows easy control of sensor data playback.

From here you can seek to a specific time of day, browse the through the hours of data, step frame by frame through the data, or Fast-Forward and Fast-Reverse through the data. Data is loaded by hour, so every time you Play a new hour (e.g. 11 to 14 ) that hour is loaded. A dialog pops up to inform you that this data is being loaded.

Buttons:

1. Play/Pause Button: The Play button plays the data back in real time. Once pressed this button changes to a Pause button.

2. Stop Button: The Stop Button stops playback.

3. ‘<’ ‘>’ - These buttons step data progression a specified amount of time. The ‘>’ button steps data forward and the ‘<’ steps data backward. To set the amount of time each button progresses the data, right click on either button.

4. ‘<<’ ‘>>’ - These buttons enable non real-time playback of the data. ‘>>’ plays the data forward and ‘<<’ plays the data backward. To set the rate of playback associated with these buttons, right click on either button.

5. Seek Slider Bar – use the slider bar to seek through the hours of data.

6. Set Time – Click on the time display in the bottom right hand corner to bring up the Set Time dialog (may or may not have date and time displayed) and seek to any time you specify. You can specify the Hour: Minute: Second: Millisecond.

ii. Image Player

The Image Player is a window that displays the video data recorded. Right clicking on the window will auto rescale the window.

The first time you open the Image Player, the JPEG video files are indexed. The first time you run the HandLense application this indexing may take one to several minutes. Subsequent runs will only take a few seconds to index these files.

iii. Sensor Map

The Sensor Map is a floorplan map augmented with a visual representation of the sensors used during the study. This map allows an easier way of seeing what is happening when a large number of sensors are being used.

The layout of the Sensor Map is defined by an .SMAP file. An example file is provided with the Installer under the SensorMap folder that is copied to the installation folder. You will need to point the Open File Dialog to this file.

The map will populate with text and the little blue squares will flash during playback. The blue squares are the Switch Sensors and Object Motion Mites described in the dataset document. The text is the output of the various other sensors.

Right clicking on any point on the map with sensor will show you what sensors are at that point via a context menu. Some points have more than one sensor located there due to the different sensor types and/or different experiments being run.

A mouse over will show each sensor type and the description associated with this type are specified in the .SMAP file.

In addition, the context menu provides the ability to skip forwards or backwards to the closest activation in time of that sensor.

Right clicking on any point on the Sensor Map that does not contain a sensor displays a context menu that enables filtering of which sensors are show on the map.

iv. Sensor Report

The Sensor Report View is a means to have a persistent record of the sensor data that has been recorded for a specific time. It is a textual output of the sensor data displayed via the Sensor Map. This view is useful for debugging and a better understanding of what is happening for a short period of time.

The context menu, presented by right-clicking on the main body of the Sensor Report View, provides a means to enable the view to start running, if the view should auto scroll, clear the output, which sensors to show, and the ability to set a time marker. The default is for the view to not be running and all the sensors to be hidden.

v. MITes Acceleration Plot

The MITes Acceleration Plot displays a plot of the 3-Axis accelerometer MITes sensors that the participant wears on their wrists, arms, legs, and ankles.

Right clicking on the plot displays the sensor IDs displayed.

Selecting ‘Set Channels to View’ pops up a dialog box. If you uncheck ‘Plot All’ you can right click on the ID list and add, remove, or clear the IDs to be shown. Also, you are able to drag the numbers in the list to rearrange the order in which they are displayed.

vi. Annotation Tool

The annotation tool provides an integrated method for adding meta data to the sensor activations. This metadata can be things such as activities (e.g. ‘Preparing a Meal’), Ambulation (e.g. ‘Walking’), and Social Context (e.g. ‘Not Alone’). This type of metadata is useful for such things as training and testing inference algorithms.

Creating a New Annotation File

1.

Start the HandLense application and open the annotation tool (located in the Views menu under ‘Annotation’).

2. The annotation tool window will appear, but will be disabled. To create new file, select ‘New…’ from the AnnoTool menu.

3. The tool will attempt to find a default template to use for the new file. If one is not found, it will prompt you to select one by opening a file browser window. Select an appropriate template file (with extension *.atmpl) A default .atmpl file has been provided in the Annotation folder of the install directory.

4. Once the template has been loaded, the tool will be enabled and ready to use.

Opening an Existing File

1.

Open the annotation tool (as above) and select ‘Open…’ from the AnnoTool menu.

2. The tool will open a file browser window. Select a valid annotation file (with extension *.xml).

3. Once the file has been loaded, the tool will be enabled and ready to use.

Setting Up the Annotation Tool for Basics Data Annotation

Once the sensor data has been loaded (Section V), you will need to open the annotation tool, as well as the image player (Section VI - ii ), and playback remote(VI - i). These tools are all found in the Views menu under.

Using the Annotation Tool to Annotate Data

File Info Tab:

The File Info tab displays metadata for the annotation file. Once you have created a new file or loaded in an existing one, you are free to add, delete, or edit any of the information in the text boxes.

Annotator Tab:

The Annotator tab contains the actual annotation data and displays it in tabular form. For a new file, it will only contain a single, blank entry as shown above. Whenever a field (excluding those in the Time column) is clicked, it becomes editable and you can change its value either by typing it in or by selecting it from the dropdown list.

If you manually enter a value, it will auto complete as you type assuming the value appears in the list. Values that do not appear in the list will be rejected and will not be saved and displayed in the field.

You can add another entry to the table through the annotation context menu. This context menu appears when you right-click outside of the table area, i.e. right-click anywhere but on one of the cells in the table. You will have the option to start a blank label, or you may select a particular label from the submenu. They will be divided into the same categories represented by the columns of the table.

The keyboard shortcut for quickly adding a blank label is CTRL-Space. A newly added label will contain the current time, as shown in on playback remote, in its Time column and “start” in its Status column. Also, the sensor data playback will be automatically paused when a new label is added. You may start and stop the playback using the playback remote as well as the option in the annotation context menu.

The annotation tool provides some helpful features once a label has been created via the right click context menu. One such feature is available by right clicking anywhere except on a label row. Doing this displays a list of all un-stopped labels. That is labels that have a start label, but don’t have a paired stop label. This quick stop option is helpful in understanding what activities are currently transpiring and enables a rapid manner in which to pair labels.

In the table view these start labels without paired stop labels (referred to as unbalanced bouts) are marked as rows with a red background.

Paired Start and Stop labels (referred to as balanced bouts or just bouts) are marked by rows with a light blue background, one row for the start label and one for the stop label.

Right clicking on a label row displays a context menu that allows you to move the time context of the data being displayed to the beginning or end of the selected bout. That is, if you want the Playback Remote to auto seek to the time displayed by the start tag, you would click ‘Skip to beginning of bout’. You can also skip to the previous or next bout with an identical label (e.g. using phone).

The same context menu also allows you to delete the selected label, the selected bout (both the start and stop label rows), or set the selected label row to the time currently displayed by the Playback Remote.

The final option is to ‘Play bout. This option will seek the time context of the application to the time displayed by the start label. It will then prompt you press the ‘Play’ button on the Playback remote. Data will playback until the time displayed by the stop label has been reached, thus playing only that bout. This is useful for manually testing the accuracy of you labels.

vii. Notification Sounds

The Notification Sound component enables audio representation of sensor activations. Sounds may be selected to play when an activation specific to a category of sensor takes place. For example, a ding sound could be played when any switch sensor is activated.

VIII. Credits

HandLense was created by Randy Rockinson and Leevar Williams of the house_n group at MIT. Some charts (e.g. Histograms) were created using the ZedGraph control

IX. Bug reports

Please report bugs to housen-handlense [at symbol] media [dot] mit [dot] edu.

X. Citation

The PlaceLab was first introduced in a a CHI 2005 paper. This dataset is freely available for academic researchers to use in their own work given that this website (http://architecture.mit.edu/house_n/data/PlaceLab/PlaceLab.htm) is cited whenever the dataset is used, as well this academic overview article:

S. S. Intille, K. Larson, E. Munguia Tapia, J.S. Beaudin, P. Kaushik, J. Nawyn, and R. Rockinson, "Using a live-in laboratory for ubiquitous computing research" in Proceedings of PERVASIVE 2006. Berlin Heidelberg: Springer-Verlag, 2006. [Abstract] [PDF]