Evaluation Materials

Attention: open in a new window. PDFPrintE-mail

When creating evaluation tasks, one should take care to include some contexts that are not optimal – because it will never be optimal in the real usage context. A list of possible conditions that might limit the use of the respective application is given below:

 

Optimal conditions:

  • Optimal lighting and the user can easily look at the screen
  • Quiet environment without sound restrictions
  • The user holds the device in the hands (no gloves or similar)
  • The user has nothing else in the hands and can use both two handed and one handed grips
  • The context does not require any attention

 

Non-optimal conditions:

  • Non-optimal lighting (e.g. bright sunlight), or the user has or wants to look elsewhere (while crossing a street, negotiating rough terrain, etc.)
  • Noisy environment (e.g. in a crowd, by a busy street, at a train station, at a festival or fair, etc.) or an environment where sounds are not suitable (e.g. in a meeting, concert, theatre)
  • Situation which limits the user’s ability to touch the device - cold hands, using gloves (cold weather) or keeping the device in a pocket or a bag
  • Situation which limits the user’s ability to manipulate the device like having to hold something else in one or both hands (e.g. white cane, umbrella, bag, take away coffee, ice cream, pram, child etc.)
  • Context that requires attention (e.g. other people, traffic, sights, nature etc.)


Useful questionnaires

A simple but very useful questionnaire for measuring the task load is the NASA-TLX.

 

In many location based/navigational tasks it can be important to get a grip on the sense of direction. Either one can ask participants to estimate this during the interview, or if more precision is needed one can use the Santa Barbara Sense of Direction Scale. More information can be found at the Hegarty Spatial Thinking Lab.

 

Background questionnaires, consent forms etc can be found in HaptiMap deliverable D1.2.

 

Heuristic evaluation

A Heuristic evaluation of the UI is part of the design process. You should do a heuristic evaluation,  using an appropriate to the end-user set of rules, before field-testing. To do a heuristic evaluation, the tester should know about the application. However, it would be better if the tester is not the developer of the application him or herself.

 

Tools focusing on how to take the context into account can be found in the section Dynamic User Experiences.

 

Universal Design Principles


Multiple perspectives and techniques can be used to conduct the heuristic evaluation. One of them is to evaluate your application following the Universal Design Principles.

 

The principle 7 concerning "size and space for approach and use" might need adaptation for a mobile context. Instead of ability to reach the device, more relevant might be to check if the grip is stable enough to for use while moving. Instead of providing adequate space for assistive devices, checking if it can be used with busy hands (i.e. using a cane or with a wheelchair) and the body is moving.

 

Diverse conditions of use

Go back to the list of conditions in the introduction. For each of those, please state if your application will or will not work. You might need to specify if this answer is conditional to specific events or contexts.


Other resources

The HaptiMap deliverable D1.2 contains an overview over a range of suitable user study methods.


A pretty comprehensive reference on cognitive/heuristic walkthrough :
http://www.univ-valenciennes.fr/LAMIH-intra/site/specifique/publications/204316.pdf


One resource on Activity walkthrough:
http://delivery.acm.org/10.1145/1030000/1028052/p251-bertelsen.pdf?key1=1028052&key2=9040004031&coll=DL&dl=ACM&ip=130.235.212.94&CFID=18336735&CFTOKEN=33894031