Reflective Journal, 2016 July

Reflective Journal, 2016 July

2 Main findings from field research

The most challenging thing of the research is connecting 'needs' of designers regarding data visualization and finding of reader(or users) cognitive activities when they encounter narrative visualization.

Since last week, I conducted two part of field research. First one is expert interview in order to find problem possibly occurred when design process of data visualization contents and second one is exploring cognitive activities required when people try to make sense of the contents.

The participants of expert interviews were 3years experienced project manager of data visualization content and head director of Seoul Information Research Institution.

The below of the summary of their mentions during interview

P1.

1.              Data literacy,which means ability of interpret data based graphic are vary depend on person, and sometime people look the content more difficult than designer expected

2.              Few principles and rules which followed by designer for long time in western cultured doesn't work in certain cultural background.

3.              Therefore, for these reasons, user test could be useful for checking data literacy of expected users

P2

1.              Previously, effectiveness of communicating in data based graphic/ or inforgraphic were mainly studied by main researchers.

2.              However, as storytelling is usually regarding time flow or sequence of separated  contents, different reaction of users by time change were not studied enough.

Through, the interview I could find more clear problem statement of the research.

1.              Designers are hardly able to predict users' reaction due to the different level of data literacy depend on person.

2.              Due to the time flow/sequence as feature of data storytelling, it is hard to predict how each element of contents interact with users.

As a consquences, I decide to explore user's cognitive activities and how each element affect on it in order to find how can designer testing or evaluate their data visualization contents effectively. As I mentioned previous journal, I conducted think-aloud obervation session to 8 participants. I selected 3 case of data storytelling, considered recent research of data storytelling trend and techniques. And all the participants are between 20 ~ 30 and they are reading web based journalism contents instead newspaper.

Participant 1 is testing test object 3

Participant 6 is testing test object 2

Participant 4 is testing test object 1

After observation, I analysized transcribed data by grounded theory.

1.Initial open coding

The above is list of all the behavior, attitude and cognitive recognitions occurred during observations. And then I recategorized to find main activities to understand each stage of their process of making sense of the content by using axial coding.

2.Axial coding

The above diagram shows main categories of each stage can explains users comprehensive activities of making sense of contents. Then I tried to find their relationship and how people transit between stage and which behaviour or cognitive change trigger affect on this process.

3. Selective coding

The above diagram describe not only how people can fail to make sense, but also how they overcome.

The most important findings were main cause of barrier for making sense of the content. For the next step, I have to think about how can suggest evaluate tool to track users' failure of making sense of the content.

Feedback from Professor Lee and few changes in the research

I shared the current progress with Professor Lee and he gave me some feedback. First of all, my research have to ultimately show a design outcome, however he couldn’t expect any design outcomes in my research, because I was focusing too much discovering cognitive activities occurred when people read ‘interactive data storytelling’.

Therefore, I started to think about how this research contribute to designers who make data visualization based content by suggesting appropriate way to evaluate their content before publishing. Then, I decided to interview designers related to this area, data design and data visualization. I contacted 2 designers who work with me previously in data journalism project last year for interview and they also introduce me another 2 people who work in design agency related to data visualization. Finally, I booked the total 4 interview session and the below is the list and the summary of findings.

P1: Project manager of data design agency 3 years experience in data design consulting Conducted data journalism projects with media companies

P2: Researcher of information design institution Head researcher of Seoul Information design Institution Managing data design education program

P3: Designer of  Data Journalism team Data designer of journalism team in Seoul Broadcasting System Making daily data visualization based content

P4: Data Scientist  In IT company Analysing data through visualization process Co-working with designer and developer

Findings

1. Letting users understand storytelling is highest priority. However, providing advanced information with visualization is also important.

2. People have different data literacy, which means ability to interpret visualized data.

3. Some participants tend to talk direct opinion which possibly lead content.

4. In case of huge scale of project, how user interface encourage people to make sense is important issue.

And Professor Lee also mentioned that if I want to suggest how evaluate data storytelling, it would be better to specify the method. Therefore, I started to search previous studies on how evaluate data storytelling or data visualization based contents. The below list is the researches that I found related to evaluating data visualization based content.

Title	Author	Keywords	Publish year	Summary
Evaluating Information Visualization via the Interplay of Heuristic Evaluation and Question-Based Scoring	Marti A.Hearst	Information Interface and presentation, Visualization, Education, Assesment, Heuristic Evaluation, Usability Testing, Question Answering	2016	This research illustrate that Heuristic evaluation and interview can both evaluate usability of interactive data visualization.
Evaluating Data Storytelling Strategies: A Case Study on Urban Changes	Flavia De Simone	Human Reasoning, Data Storytelling, Eye tracking,	2014	The research compare understanding level of two articles written in same topics but different approaches, narrative only and narrative visualization. And evaluate participants understanding level by interview.
Ranking Visualizations of Correlation Using Weber’s Law	Lane Harrison	Perception, Visualization, Evaluation	2014	The research study which type of visualization communicate with people most accurately when visualize correlation between data by using Weber’s Law.

According to these researches, current approaches to evaluate data visualization based content can be categorized main 3 categories, heuristic evaluation, interview and testing visual perception. However, those approaches have limitation evaluate data visualization based content by comprehensive way for identifying whether people make sense of it and or which factor is obstacle for understanding. Because, heuristic evaluation and interview do not require immediate user response during the test. Therefore, all of the verbal or recorded data have to be relied on user memory, which have both limitation and bias.

So I decided to bring ‘think-aloud’ as research subject, which is possibly replace previous evaluation tools and has more advantages. According to the ‘Neilson Group’, think-aloud is Simple usability tests where users think out loud are cheap, robust, flexible, and easy to learn. And Thinking aloud should be the first tool in your UX toolbox, even though it entails some risks and doesn't solve all problems.

Debate on how utilize think-aloud

I also found a research about gap between theory and practice of think-aloud. This research survey how and why researchers and evaluate conduct think-aloud differently with theoretical instructions. Especially, this research showed why so many researcher used intervention during the test for various reasons.

The above chart shows the reasons for using intervention.

As a results, I decided to modify my research topic to ‘How to utilize ‘intervention’ in ‘think-aloud’ method as  an evaluation tool in the area of “interactive data storytelling”. Due to the change, I had to conduct more observation using think-aloud with intervention. I already conducted observation with classic think-aloud for 8 participants, however, because I didn’t have much time I could only conduct think-aloud observation with intervention for 4 people. At this time, I tried to give extra question when participants do not give any verbal description about the reason when they reach ‘Questioning’ stage which was defined in previous analysis.

Comparing with two version of think-aloud

	Without Intervention	With intervention
Participants’ Activities	Same activities were founds in both case; Constructing a Frame, Exploring Information, Questioning, Fail to make sense.
	It was able to observe natural flow of user activities by their own reliance. Emotional reaction and change of attitude was observed rather than deeper understanding of participant behavior.	Intervention affect flow of user activities. For example, once they asked question during the session, they tend to spend more time in ‘Questioning’ stage. Participants mostly describe specific reasons why they fail to make sense rather than emotional reactions.
Number of issue mentioned	1.6 average per a session Some participants tend to do not mention reason of questioning stage or fail to make sense	2.5 for average per a session Participants to mentioned more issues but difference was small.
Types of description in Questioning stage	Fail to encoding visual element Fail to encoding text element	Fail to encoding visual element Fail to encoding text element Expectation to overcome User interaction issue
Data Analysis	It is required to extra time to analyze transcription	Transcription was organized already by intervention.
Time	13 min for average per a session	7min 30s for average per a session

Classic think-aloud let evaluator observe more natural flow of how user interact and understand  ‘interactive data storytelling’ with change of their emotion and attitude. However, once participant move to questioning stage, it has limitations in deeper understanding why they fail to interpret objects.

Although interventions of evaluator affects participants activities during think-aloud session, same kinds of activities were able to observed; Constructing a frame, Exploring Information, Questioning and fail to make sense. As mentioned previously, Think aloud with intervention has more advantages in discovering issues, detail of description and convenience in data analysis.

As a result, this research concluded that think-aloud with intervention is more appropriate approach to evaluate data storytelling in terms of understanding level.

I will suggest design proposal of evaluation tool based on this findings until final presentation.