Seminar 2 notes - Joakim

Seminar 2

Chapter 13

Chapter 13 introduces the concept of evaluation and the authors stresses the importance of evaluation as a part of the design process. It is important to evaluate continuously, known as formative evaluation, to check that the product fulfills the requirements. Both low-tech prototypes and full systems can be considered. At the end of the process a summative evaluation is carried out. Evaluation regards both usability and user experience.

There are different methods that can be broadly categorized as

• Controlled settings involving users; e.g. experiments in a laboratory for usability testing
• Natural settings involving users. Field studies in a realistic context
• Any settings not involving users. Analytics and models.

The first has the advantage of tight control but in an artifical environment which could bias the evaluation data. A natural setting has the advantage of the users being in the correct context (in the wild studies) but it might be difficult to correctly anticipate what will happen. The idea of a living lab is to try and combine these two. A quick and cheap way to assess a prototype or product is to use heuristics, that is apply knowledge of typical users, rules-of-thumb etc to create models of user behavior.

Things to consider

• Participants rights and consent
• Reliability (Are the results repeatable)
• Validity (Are we measuring the intended thing?) and ecological validity (Does the evaluation environment affect the results?)
• Biases (Results distorted systematically, e.g. a specific group of expert evaluators might be sensitive to one design flaw compared to others)
• Scope (Can we generalize the results?)

Chapter 14

Here evaluation studies are described in a spectrum of settings, from controlled to natural.

Usability testing

Usability is tested in a controlled setting, such as a lab or temprorary makeshift lab. The idea is to collect quantitative data about users performance on predefined tasks, e.g.

• Time to complete task
• Number of errors
• Number of navigations to manual
• etc.

Usability can also be tested remotely (users test in their own setting, data is logged). Experiments aim to test an hypothesis where the experimenter controls the independent variable(s) to test the dependent variable(s). The other variables are held constant.

Different ways of experiment design:

• Different-participant design (large randomly allocated group. Different participant perform in different conditions, need a large group to avoid bias)
• Same-participant design (all participants in all conditions)
• matched-participant design design (particpants paired based on characteristics, e.g. expertise)

In the wild studies

• How do people interact with technology in a natural setting?
• Evaluators want to discover how nw products or prototypes will be used. Explore novel designs.

Chapter 15

This chapter describes methods for understanding users through heuristics, remotely collected data or models. That is, methods without having to involve users in the evaluation step.

Heuristic evaluation

Experts evaluate interface through usability guidelines

Walkthroughs

Predict user problems by walking through a task noting problematic usability features. E.g. cognitive walkthrough.

Analytics

Evaluating user traffic through a system, e.g. web analytics (user activity such total visitors, traffic sources etc.)

Predictive models

Predict user behavior based on a model. For example Fitt's Law which predicts the time to reach a target with a pointing device

Questions

• Which evaluation methods are feasible within our project?
• Are the results valid? reliable? What about biases etc.
• How should we balance settings involving users with evaluation methods without involved users?

Notes Seminar 1: Kim

Notes Seminar 1
 
Chapter 7
In this chapter data gathering techniques were presented. It’s good to know that there are different techniques that can be combined and used in various ways depending on the goals. It’s always possible to tweak and restructure the techniques so that they fit the context they are used for, since every technique has advantages and disadvantages. Thinking about details when using the techniques for data gathering is very important so that the collected data is used in the right context. I think that the five key issues setting goals, identifying participants, relationships with participants, triangulation, and pilot studies that the authors wrote about are important to think about when it comes to data gathering. It was interesting to read about how interviewing children should be done in a different way compared to adults, because they think and react to situations differently to adults.


QUESTION: Instead of only using semi-structured interviews for our design project, should we use any other technique to strengthen our data gathering?

Chapter 8
This chapter presented how data analysis could be done. Knowing if the analysis should be qualitative, quantitative or a combination of them is a good first step. Using different data gathering techniques will yield different data suitable for the varying options of analysis. Some of the techniques might even yield data suitable for all the methods for analysis of data. One thing that I think is interesting is how data can be interpreted in different ways. Just like when gathering data, analyzing data must be adapted to each situation.

Chapter 10
This chapter presented a lot of thing that were new to me. I felt that this chapter had a messy structure and a lot of new information to process. Overall I got the feeling that there are a lot of things that needs to be considered and specified when it comes to establishing the requirements for a project. It’s also very important to do this right since it’s a critical part of a project. Not doing this in a proper way would most definitely make a project end in a disaster. It would be good to talk about this chapter during the seminar to make it more clear.

 

Reading seminar 1 notes- Sajaval Choudrey

Reading seminar 1 notes.

There are several different ways to gather data for instance: questionnaires interview, natural observations and etc. Data gathering is mainly done to capture users’ reactions and performances with a system or prototype so that a set of stable requirements can be established for said system. When gathering data, triangulation, which means to gather several kinds of data from different perspectives, is usually a good way to go about it. This is something written in the text and which I also agree on. Only going about data gathering in one way can be a bad idea as you only are gathering the views from one perspectives. Triangulation is something I feel can be implemented in the project in this course, as a way to improve our results.

I also agree on combining qualitative analysis and quantitative analysis. As written in the text, “Karapanos et al (2009) go further and suggest that averaging treats diversity among participants as error and proposes the use of a multidimensional scaling approach instead”, I agree and feel that the “multidimensional approach” could be in the form of a combination between qualitative and quantitative analysis. In data gathering and data analysis, I believe that a holistic view of the system and/or prototype being examined is important, this equates to examining as many aspects as possible from as many different viewpoints as possible.

The next step is to use all the data that has been gathered and analysed in order to create stable requirements. The literature mentions “personas” (e.g. novice or professional) and how requirements need to be adjusted to different kinds of personas. This made me wonder how large systems, that try to create requirements, deal with having to adjust to a widespread distribution of various personas. Are there any instances, when having too many personas (/ users with different experiences), in a system can create problems? Are some of the different experience levels of users ignored when building system?

Notes Seminar 1: Joakim

Notes Seminar 1

The first seminar will cover the initial phase of the project. We will discuss topics such as establishing needs and requirements, Data gathering and data analysis, interpretation and presentation. We want to understand the user and find out what the user needs!

By gathering data the needs and requirements of the users can be established. Both qualitative and quantitative data are typically gathered and triangulation used see the data in a wider perspective. Due to the rather small scope of our project, mostly qualitative data will be used, since we won't be able to gather enough data (using for exampe questionnaires) for quantitative analysis. I imagine a first step would be to conduct semi-structured interviews. Maybe combine this with naturalistic observations to see how the travellers interact with the environment. When sampling the target group (tourists) stratified sampling is a good approach. However, due to the limited time we will probably have trouble getting mutually exclusive subgroups to sample from.

Chapter 8 discusses analysis, interpretation and presentation of gathered data. Qualitative and quantitative data is to be handled differently, quantitative data is closely related to statistics where, with math, conclusions can be drawn. Qualitative data is not as straight-forward, but the goal is to recognize patterns and identify outliers.

Since the project is an iterative process we will after first iteration define scenarios and personas which probably will lead to new insights in needs and requirements.

My questions for this seminar is:

• First of all we need to agree on a specific project idea, what implications will this have on data gathering methods?
• Since the time for data gathering will be quite limited, which data gathering methods will best complement each other?

Exercise 3

During this exercise our goal was to develop a conceptual design. Through three different brainstorming exercises we developed two rapid off-line low-fidelity prototypes. Since the aim for the application is simplicity, both proposals are quite similar. There are however differences in the way the information is presented. We also discussed various techniques that can be applied in the design phase

• Since our app is simple we concluded that, at least for the time being, we are not trying out a "wizard of oz"-approach. It's assumed easier to get a software prototype up and running instead of someone playing the role of our system(app).
• We also concluded that a participatory design method is infeasible since we do not have the time to include end-users in our project.
• After low-fi prototyping we will move on to generate one, or a few, mock-ups to get a better feel for the application.

Parallell Design

Through the parallell design two views were constructed. This was due to the different choices of scenario and pain points. The first was a result of our persona Ben needing information regarding surrounding places of interest, while the other was pur secondary persona Susan needing time-table information. This resulted in one map view and one augmented reality view.

Collaborative iteration

We then went on and did a collaborative iteration starting from one of the initial designs of the parallell design exercise. Through the iteration this then came to include the idea of the second design, namely the map.

Word Association

The word association exercise gave rise to the idea that the map should be stylish and simplistic and only present information related to the route. No simple embedded google maps with too much information.

Iteration

We then iterated the above exercises to yield another proposal. The result of this was that we all realized that we had focused only on landscape mode design in the previous iteration. The second proposal was considering portrait mode instead which introduced certain limitations, due to a smaller width. We then discussed different ways of presenting information to the user. One was a sliding view from the bottom showing the relevant information after a user clicks.

Pain points

List of pain points

ISSUE/OPPORTUNITIES	Ben	Susan
Need timetable	1	1
Walking distances	3	1
Geographical information	3	1
Toilet locations	3	1
All information available in one place	2	1
Places of interest	1	2
Activities for children	5	1
Social activities (pubs/restaurant/etc)	1	3