Reflective Journal, 2016 July
2 Main findings
from field research
The most challenging thing of the
research is connecting 'needs' of designers regarding data visualization and finding
of reader(or users) cognitive activities when they encounter narrative
visualization.
Since last week, I conducted two part of field research. First one is expert interview in order to find problem possibly occurred when design process of data visualization contents and second one is exploring cognitive activities required when people try to make sense of the contents.
The participants of expert interviews were 3years experienced project manager of data visualization content and head director of Seoul Information Research Institution.
The below of the summary of their mentions during interview
P1.
Since last week, I conducted two part of field research. First one is expert interview in order to find problem possibly occurred when design process of data visualization contents and second one is exploring cognitive activities required when people try to make sense of the contents.
The participants of expert interviews were 3years experienced project manager of data visualization content and head director of Seoul Information Research Institution.
The below of the summary of their mentions during interview
P1.
1.
Data literacy,which means ability of interpret data
based graphic are vary depend on person, and sometime people look the content
more difficult than designer expected
2.
Few principles and rules which followed by designer
for long time in western cultured doesn't work in certain cultural background.
3.
Therefore, for these reasons, user test could be
useful for checking data literacy of expected users
P2
1.
Previously, effectiveness of communicating in data
based graphic/ or inforgraphic were mainly studied by main researchers.
2.
However, as storytelling is usually regarding time
flow or sequence of separated contents, different reaction of users by
time change were not studied enough.
Through, the interview I could find more clear
problem statement of the research.
1.
Designers are hardly able to predict users' reaction
due to the different level of data literacy depend on person.
2.
Due to the time flow/sequence as feature of data
storytelling, it is hard to predict how each element of contents interact with
users.
As a consquences, I decide to explore user's cognitive
activities and how each element affect on it in order to find how can designer
testing or evaluate their data visualization contents effectively. As I
mentioned previous journal, I conducted think-aloud obervation session to 8
participants. I selected 3 case of data storytelling, considered recent
research of data storytelling trend and techniques. And all the participants
are between 20 ~ 30 and they are reading web based journalism contents instead
newspaper.
Participant 1 is testing test object 3
Participant 6 is testing test object 2
Participant 4 is testing test object 1
After
observation, I analysized transcribed data by grounded theory.
1.Initial
open coding
The
above is list of all the behavior, attitude and cognitive recognitions occurred
during observations. And then I recategorized to find main activities to
understand each stage of their process of making sense of the content by using
axial coding.
2.Axial
coding
The
above diagram shows main categories of each stage can explains users
comprehensive activities of making sense of contents. Then I tried to find their
relationship and how people transit between stage and which behaviour or
cognitive change trigger affect on this process.
3.
Selective coding
The above diagram describe not
only how people can fail to make sense, but also how they overcome.
The most important findings were
main cause of barrier for making sense of the content. For the next step, I
have to think about how can suggest evaluate tool to track users' failure of
making sense of the content.
Feedback from
Professor Lee and few changes in the research
I shared the current
progress with Professor Lee and he gave me some feedback. First of all, my
research have to ultimately show a design outcome, however he couldn’t expect
any design outcomes in my research, because I was focusing too much discovering
cognitive activities occurred when people read ‘interactive data storytelling’.
Therefore, I started
to think about how this research contribute to designers who make data
visualization based content by suggesting appropriate way to evaluate their
content before publishing. Then, I decided to interview designers related to
this area, data design and data visualization. I contacted 2 designers who work
with me previously in data journalism project last year for interview and they
also introduce me another 2 people who work in design agency related to data
visualization. Finally, I booked the total 4 interview session and the below is
the list and the summary of findings.
|
P1: Project manager of data
design agency
|
|
P2: Researcher of information design
institution
|
|
P3: Designer of Data Journalism team
|
|
P4: Data Scientist In IT company
|
Findings
1. Letting users understand
storytelling is highest priority. However, providing advanced information with
visualization is also important.
2. People have different data
literacy, which means ability to interpret visualized data.
3. Some participants tend to talk
direct opinion which possibly lead content.
4. In case of huge scale of project, how user interface
encourage people to make sense is important issue.
And Professor Lee
also mentioned that if I want to suggest how evaluate data storytelling, it
would be better to specify the method. Therefore, I started to search previous
studies on how evaluate data storytelling or data visualization based contents.
The below list is the researches that I found related to evaluating data
visualization based content.
|
Title
|
Author
|
Keywords
|
Publish year
|
Summary
|
|
Evaluating
Information Visualization via the Interplay of
Heuristic Evaluation and Question-Based Scoring |
Marti A.Hearst
|
Information
Interface and presentation,
Visualization, Education, Assesment, Heuristic Evaluation, Usability Testing, Question Answering |
2016
|
This research
illustrate that Heuristic
evaluation and interview can both evaluate usability of interactive data
visualization.
|
|
Evaluating Data
Storytelling Strategies: A Case Study on Urban Changes
|
Flavia De Simone
|
Human Reasoning,
Data Storytelling, Eye tracking, |
2014
|
The research
compare understanding level of two articles written in same topics but
different approaches, narrative only and narrative visualization. And
evaluate participants understanding level by interview.
|
|
Ranking
Visualizations of Correlation Using Weber’s Law
|
Lane Harrison
|
Perception,
Visualization, Evaluation |
2014
|
The research study
which type of visualization communicate with people most accurately when
visualize correlation between data by using Weber’s Law.
|
According to these
researches, current approaches to evaluate data visualization based content can
be categorized main 3 categories, heuristic evaluation, interview and testing
visual perception. However, those approaches have limitation
evaluate data visualization based content by comprehensive way for identifying
whether people make sense of it and or which factor is obstacle for
understanding. Because, heuristic evaluation and interview do not require
immediate user response during the test. Therefore, all of the verbal or
recorded data have to be relied on user memory, which have both limitation and
bias.
So I decided to bring ‘think-aloud’ as
research subject, which is possibly replace previous evaluation tools and has
more advantages. According to the ‘Neilson Group’, think-aloud is Simple usability tests where users think out loud are
cheap, robust, flexible, and easy to learn. And Thinking aloud should be the
first tool in your UX toolbox, even though it entails some risks and doesn't
solve all problems.
Debate on how
utilize think-aloud
I also found a research
about gap between theory and practice of think-aloud. This research survey how
and why researchers and evaluate conduct think-aloud differently with
theoretical instructions. Especially, this research showed why so many
researcher used intervention during the test for various reasons.
The above chart shows the
reasons for using intervention.
As
a results, I decided to modify my research topic to ‘How to utilize ‘intervention’ in ‘think-aloud’ method as an
evaluation tool in
the area of “interactive data storytelling”. Due to the change, I had to
conduct more observation using think-aloud with intervention. I already
conducted observation with classic think-aloud for 8 participants, however,
because I didn’t have much time I could only conduct think-aloud observation
with intervention for 4 people. At this time, I tried to give extra question
when participants do not give any verbal description about the reason when they
reach ‘Questioning’ stage which was defined in previous analysis.
Comparing with two version of
think-aloud
|
|
Without Intervention
|
With intervention
|
|
Participants’ Activities
|
Same activities were founds in both
case; Constructing a Frame, Exploring Information, Questioning, Fail to make
sense.
|
|
|
It was able to observe natural flow of
user activities by their own reliance. Emotional reaction and change of
attitude was observed rather than deeper understanding of participant behavior.
|
Intervention affect flow of user
activities. For example, once they asked question during the session, they
tend to spend more time in ‘Questioning’ stage. Participants mostly describe
specific reasons why they fail to make sense rather than emotional reactions.
|
|
|
Number of issue mentioned
|
1.6 average per a session
Some participants tend to do not mention reason of questioning stage or
fail to make sense
|
2.5 for average per a session
Participants to mentioned more issues but difference was small.
|
|
Types of description in Questioning
stage
|
Fail to encoding visual element
Fail to encoding text element
|
Fail to encoding visual element
Fail to encoding text element
Expectation to overcome
User interaction issue
|
|
Data Analysis
|
It is required to extra time to analyze transcription
|
Transcription was organized already by intervention.
|
|
Time
|
13 min for average per a session
|
7min 30s for average per a session
|
Classic think-aloud
let evaluator observe more natural flow
of how user interact and understand ‘interactive data
storytelling’ with change of their emotion and attitude. However, once
participant move to questioning stage, it has limitations in deeper understanding why they fail to interpret objects.
Although
interventions of evaluator affects participants activities during think-aloud
session, same kinds of activities were able to observed; Constructing a frame,
Exploring Information, Questioning and fail to make sense. As mentioned
previously, Think aloud with intervention has more advantages in discovering
issues, detail of description
and convenience in data analysis.
As a result, this research concluded that think-aloud with intervention is more
appropriate approach to evaluate data storytelling in terms of understanding
level.
I will suggest design proposal of evaluation tool
based on this findings until final presentation.
