Seminar 2 summary

Seminar 2 summary

Evaluation is a method to use in order to make sure that a whole design/product or parts of it is working properly from a user perspective. What part that is being evaluated can differ, depending on what the aim of the design/product is. The different evaluation methods were discussed, settings involving users (both controlled and in the wild) and settings not involving users.

During the seminar the following questions were discussed:
- What setting and method is appropriate for evaluating our application?
- Are the results valid? reliable? What about biases etc.?

In the next excersice we will perform a heuristic evaluation. This evaluation doesn't involve any potential user which could be a seen as a drawback. Furthermore, all experts are from the course. The group is therefore contemplating performing a cognitive walkthrough with a person that isn't taking the course, to avoid bias. We also would like to perform a evaluation in a natural setting to involve actual users. This will be done in the later stages, when we have a hi-tech prototype.

Doing a usability test in a controlled setting such as a laboratory, will not be feasible as the scope of the project does not allow such a evaluation.

Due to the fact that Augemented Reality (AR) is a relatively new technology, performing evaluations without the users could be less reliable and less valid compared to evaluating a more established technology since there is less knowledge about such systems. Furthermore users might not have any previous experiences with AR to base their expressions and opinions on, which can cause problems in regards to the validity of evaluation involving users.

Since only a prototype will be developed in this project, a final summative evaluation will not be conducted. However, starting from next weeks exercises formative evaluations will be performed.

Usability testing through experiments was discussed during the seminar and the group concluded that such a method of evaluation will not be performed as it usually is done in a later stage of the development process. Analytics, where user traffic through a system will not be performed due to the same reasons. We won't have a fully functional system and enough users to test this.

Seminar 2 notes - Sajaval Choudrey

Seminar 2

The book introduced evaluation together with its key concepts. The reason why we need to evaluate, is because users expect a lot from the systems that are available to them, in form of experience, design, usability etc. The system needs to go beyond the basic requirements of the customer, it also needs to provide the user with a pleasing experience and intuitive design. If one can identify all of the important aspects for the customer and also evaluate the product, it can become appealing to a wider audience. Depending on the system and product being created one needs to evaluate different things. Everything stems from the demands of the customer, one system might need to focus on speed during the evaluation, while another might need to focus on design and ease of use. The means of evaluating in terms of location is also dependent upon the system that is being evaluated. In the book it is mentioned that the evaluation of a toy can benefit from being made in a child’s room with the child using the toy. Other evaluations can be made directly in a lab. The nature of the evaluation is taken into consideration when deciding “when” to carry out the evaluation. Does it occur in an earlier stage of the development process? Or Is it left to the end when the product is finished?

How do we interpret data when evaluating? There are several things one should think of when doing so, is the system reliable (not prone to error or inconsistency)? Are the evaluation results valid (do they measure what we wanted)? Are there any biases (are the results of the evaluations affected by any personal traits or agendas)?

Evaluation can as stated in the book be summarized in the following categories:

1.       Controlled settings involving users (examples are laboratories and living labs): users’ activities are controlled in order to test. Usability testing is often performed in such a setting. Usability testing focuses on evaluating how usable a product is, what do users do? And he social of influences on the users behaviour.

2.       Natural settings involving users (examples are online communities and products that are used in public places): there is little or no control of users’ activities in order to determine how the product would be used in the real world. Field studies are often made in natural settings because of the fact that products are increasingly being designed for environments other than the office or the lab.

3.       Any settings not involving users: consultants and researchers critique, predict, and model aspects of the interface in order to identify. Experimental designs can be performed in such environments as well as controlled settings involving users

When users are hard to come by in the evaluating process inspections can be made. This involves getting experts in the field of interaction design, requirements and user behaviour, to evaluate a product. One such inspection method is heuristic evaluation which inspects the usability. Walkthroughs are another form of inspection methods where the product is “walked through” while at the same time taking notes of any issues occurring.

Question: which setting is the most appropriate for evaluating our app?

Seminar 2 notes - Kim

Seminar 2

Chapter 13
This chapter presented the main ideas of evaluation and different methods for evaluating designs or other parts of a design project. Evaluation is an important thing to do because it makes sure that for example a design is accepted among a wider group of users. In the text the author mentions that it’s important to know why an evaluation should be performed, what one should evaluate, where and when the evaluation should be done. It’s often possible to evaluate different aspects in different ways. An evaluation could for example range from a whole system to a smaller part of an application.

In the text the author writes about three different broad categories used to classify evaluations. These categories are controlled settings involving users, natural settings involving users and any settings not involving users. Each category has its advantages and disadvantages. It’s also important to know that using combinations of the different categories is possible to reach a better evaluation that fits to a specific task. Finally, the author mentions things that could influence the way the data is translated. These things are reliability, validity, ecological validity, biases and scope.

Question: What kind of evaluations would be suitable for our design project?

Chapter 14
In this chapter evaluation studies were presented. This involved where the evaluation took place, which in the text focused on usability tests in controlled laboratory settings and field studies in natural settings. In the text different ways of measuring quantitative performances during a test were presented. For example it’s possible to measure the time it takes to complete a task, number and type of errors per task or numbers of users making a specific type of error.  When an evaluation is performed in a controlled setting such as a lab, there are different type of equipment that can be used to gather data to the evaluation.

Hypotheses testing is a way of examining the relationships between variables. A variable can be independent or dependent. An independent variable is possible to manipulate. When the hypothesis is tested it’s common to state a null hypothesis and an alternative hypothesis.

When it comes to experimental designs, it’s important how participants should be involved and which participants that should be used. The author writes about the three different designs different-participant design, same-participant design and matched-pairs design. 

Field studies is a way of gathering data from natural settings. The data is mainly collected using observations and performing interviews. It could also be done by for example filming or capturing audio. The time of the field studies being done can vary from a minute to a several years. In the wild studies is a way of capturing how people are using new technologies in a natural setting.

Chapter 15
This chapter presented methods used to understand users through heuristics, remotely collected data or models that predicts a user’s performance. All the mentioned methods don’t involve any participation of a user. Instead role playing is used to simulate the user and analyzing different relevant aspects and identifying usability problems is done.

Using heuristic evaluations is done checking how different parts of the tested product stands against a list of heuristics. In each iteration any problems are identified and handled.

Walkthroughs is a method used for predicting problems for users without performing any user tests. This method involves going through a list of tasks and marking possible problems with the user features. Some walkthrough methods don’t include the users while other walkthrough methods involves the users as well as usability specialists and developers. In the text the author writes more about cognitive and pluralistic walkthroughs.

Analytics is a method used to evaluate the user traffic through a system. This means collecting data about parts of the system to be able to see if there are parts that are underused.

Seminar 2 notes - Joakim

Seminar 2

Chapter 13

Chapter 13 introduces the concept of evaluation and the authors stresses the importance of evaluation as a part of the design process. It is important to evaluate continuously, known as formative evaluation, to check that the product fulfills the requirements. Both low-tech prototypes and full systems can be considered. At the end of the process a summative evaluation is carried out. Evaluation regards both usability and user experience.

There are different methods that can be broadly categorized as

• Controlled settings involving users; e.g. experiments in a laboratory for usability testing
• Natural settings involving users. Field studies in a realistic context
• Any settings not involving users. Analytics and models.

The first has the advantage of tight control but in an artifical environment which could bias the evaluation data. A natural setting has the advantage of the users being in the correct context (in the wild studies) but it might be difficult to correctly anticipate what will happen. The idea of a living lab is to try and combine these two. A quick and cheap way to assess a prototype or product is to use heuristics, that is apply knowledge of typical users, rules-of-thumb etc to create models of user behavior.

Things to consider

• Participants rights and consent
• Reliability (Are the results repeatable)
• Validity (Are we measuring the intended thing?) and ecological validity (Does the evaluation environment affect the results?)
• Biases (Results distorted systematically, e.g. a specific group of expert evaluators might be sensitive to one design flaw compared to others)
• Scope (Can we generalize the results?)

Chapter 14

Here evaluation studies are described in a spectrum of settings, from controlled to natural.

Usability testing

Usability is tested in a controlled setting, such as a lab or temprorary makeshift lab. The idea is to collect quantitative data about users performance on predefined tasks, e.g.

• Time to complete task
• Number of errors
• Number of navigations to manual
• etc.

Usability can also be tested remotely (users test in their own setting, data is logged). Experiments aim to test an hypothesis where the experimenter controls the independent variable(s) to test the dependent variable(s). The other variables are held constant.

Different ways of experiment design:

• Different-participant design (large randomly allocated group. Different participant perform in different conditions, need a large group to avoid bias)
• Same-participant design (all participants in all conditions)
• matched-participant design design (particpants paired based on characteristics, e.g. expertise)

In the wild studies

• How do people interact with technology in a natural setting?
• Evaluators want to discover how nw products or prototypes will be used. Explore novel designs.

Chapter 15

This chapter describes methods for understanding users through heuristics, remotely collected data or models. That is, methods without having to involve users in the evaluation step.

Heuristic evaluation

Experts evaluate interface through usability guidelines

Walkthroughs

Predict user problems by walking through a task noting problematic usability features. E.g. cognitive walkthrough.

Analytics

Evaluating user traffic through a system, e.g. web analytics (user activity such total visitors, traffic sources etc.)

Predictive models

Predict user behavior based on a model. For example Fitt's Law which predicts the time to reach a target with a pointing device

Questions

• Which evaluation methods are feasible within our project?
• Are the results valid? reliable? What about biases etc.
• How should we balance settings involving users with evaluation methods without involved users?

Notes Seminar 1: Kim

Notes Seminar 1
 
Chapter 7
In this chapter data gathering techniques were presented. It’s good to know that there are different techniques that can be combined and used in various ways depending on the goals. It’s always possible to tweak and restructure the techniques so that they fit the context they are used for, since every technique has advantages and disadvantages. Thinking about details when using the techniques for data gathering is very important so that the collected data is used in the right context. I think that the five key issues setting goals, identifying participants, relationships with participants, triangulation, and pilot studies that the authors wrote about are important to think about when it comes to data gathering. It was interesting to read about how interviewing children should be done in a different way compared to adults, because they think and react to situations differently to adults.


QUESTION: Instead of only using semi-structured interviews for our design project, should we use any other technique to strengthen our data gathering?

Chapter 8
This chapter presented how data analysis could be done. Knowing if the analysis should be qualitative, quantitative or a combination of them is a good first step. Using different data gathering techniques will yield different data suitable for the varying options of analysis. Some of the techniques might even yield data suitable for all the methods for analysis of data. One thing that I think is interesting is how data can be interpreted in different ways. Just like when gathering data, analyzing data must be adapted to each situation.

Chapter 10
This chapter presented a lot of thing that were new to me. I felt that this chapter had a messy structure and a lot of new information to process. Overall I got the feeling that there are a lot of things that needs to be considered and specified when it comes to establishing the requirements for a project. It’s also very important to do this right since it’s a critical part of a project. Not doing this in a proper way would most definitely make a project end in a disaster. It would be good to talk about this chapter during the seminar to make it more clear.