CauseWorks: A Framework for Transforming User Hypotheses into
a Computational Causal Model
Thomas Kapler
a
, Derek Gray
b
, Holland Vasquez
c
and William Wright
d
Uncharted Software, Toronto, Canada
Keywords: Causality Analysis, User-driven Modelling.
Abstract: Causal Model building for complex problems has typically been completed manually by domain experts and
is a time-consuming, cumbersome process. Operational Design defines a process of rapid, structured discourse
for teams to envision systems and relationships about complex, “wicked” problems, however, the resulting
models are simple diagrams produced on whiteboards or slides, and as such, do not support computational
analytics, thus limiting usefulness. We introduce CauseWorks, an application that helps operators sketch”
complex systems and transforms sketches into computational causal models using automatic and semi-
automatic causal model construction from knowledge extracted from unstructured and structured documents.
CauseWorks then provides computational analytics to assist users in understanding and influencing the system.
We walk through human-machine collaborative model-building with CauseWorks and describe its application
to regional conflict scenarios. We discuss feedback from subject matter experts as well as lessons learned.
1 INTRODUCTION
Causal reasoning forms the basis for most complex
forms of reasoning, facilitating hypotheses,
inferences, explanations, and problem-solving
(Jonassen & Ionas, 2006). This is true for virtually all
domains: causal reasoning permeates science,
engineering, public health, finance, medicine, and
military planning and decision making (Keim et al.,
2010; Sedig et al., 2012; Schmitt, 2017). Indeed,
understanding causality, the influence by which one
factor or cause contributes to the production of
another factor has become an important topic in
visualization within the last decade (Pearl, 2009.)
Causality provides a way of understanding systems,
subsystems, how they operate dynamically, their
underlying characteristics, and forces that drive
change. As such, there is an increasingly important
role for visual analytics, to support the user’s
understanding of causality through interactive visual
interfaces.
The application of causality visualization
methods to complex wicked problems, while
a
https://orcid.org/0000-0003-1957-9131
b
https://orcid.org/0000-0003-4009-2125
c
https://orcid.org/0000-0001-9048-3445
d
https://orcid.org/0000-0002-5685-3999
emerging, is still limited (Proulx et al., 2019). Wicked
problems can be difficult to define, are not discreet,
and potential solutions are intertwined and complex
(Rittel & Webber, 1973). Government instability,
gray zone conflicts (Wirtz, 2017), food security
(Zhang & Vesselinov 2017), and climate change (Gil
et al., 2018) are examples of ill-structured, dynamic
situations which are poorly understood and where
solutions are neither readily available nor have
consensus. Most causality visualization systems
require a pre-existing model for the user to explore
(Sedlmair et al., 2012). However, with wicked
problems, key knowledge may reside within the mind
of the domain expert. As such, in addressing wicked
problems modelling efforts are often completed by
hand, involving the knowledge of many domain
experts, who may be novices in causal modeling.
Inherently, this process can be time consuming,
requiring access to domain and modelling experts,
which are costly and hard to procure (McPherson et
al., 2007).
50
Kapler, T., Gray, D., Vasquez, H. and Wright, W.
CauseWorks: A Framework for Transforming User Hypotheses into a Computational Causal Model.
DOI: 10.5220/0010194300500063
In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 3: IVAPP, pages 50-63
ISBN: 978-989-758-488-6
Copyright
c
2021 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
Figure 1: CauseWorks visual analytics interface for rapid creation of causal models showing a simple causal system of
relationships between two countries. Side panel summarizes user objectives and the interventions made to achieve them.
Green and red nodes indicate projected changes to factors resulting from interventions (in blue).
In developing solutions for wicked problems
within the military, military planners will combine
systems thinking, causal thinking and the Operational
Design (OD) process (described in the subsequent
section) to frame complex problems and arrive at
potential solutions. The Defence Advanced Research
Projects Agency’s (DARPA, 2018) Causal
Exploration (CX) Program, is focused on developing
a causal modeling platform to aid expert military
planners in decision-making in conflicts complicated
by political, economic, social and other non-military
factors where there is significant uncertainty about
the problem and appropriate objective. In this paper,
we present CauseWorks, a visual analytics interface
developed by Uncharted Software for teams to apply
causal modelling to complex problems. The problem
domain of focus is regional conflict analysis;
however, we note that the application of CauseWorks
could extend to other domains. The key contribution
in this paper is a framework to support OD experts
who are novice modelers in building computational
causal models for complex, wicked problems. The
specific contributions include: 1) a method for
capturing user-driven ideas and hypotheses, and
connecting them with a knowledge base spanning
thousands of documents, 2) a method for rapidly
transforming hypotheses into a computational causal
model backed by data, and 3) a method for interacting
with and visualizing casual analytics within the
context of the model to reveal system behaviors and
assist in solution development. In the subsequent
section, we provide additional background on OD to
set the stage for the application of CauseWorks.
2 OPERATIONAL DESIGN
The United States military is increasingly conducting
operations in complex, ill-structured environments
that are characterized by a diverse, ambiguous set of
actors, enemies, and unknowns, (Joint Publication JP
5-0, 2017). In order to develop the best course of
action, military planners need to develop a
comprehensive understanding of interconnected,
complex systems such as the governments,
population, security forces, and non-state actors that
make up the environment (Schmitt, 2017). This
understanding is achieved through OD, a method
used by military planners involving team
brainstorming, system sketching, and other methods,
of characterizing the intertwined relationships
between entities and factors involved in current and
desired states.
During the OD process, a team of six to nine
operational designers and domain experts will engage
in extensive discussion with three goals: 1) framing
the operational environment, 2) framing the problems
that permeate the environment, and 3) creating
approaches to transform the problem (
ATP 5-0.1,
CauseWorks: A Framework for Transforming User Hypotheses into a Computational Causal Model
51
2015)
. This involves rapid brainstorming of potential
variables on a whiteboard and developing lists of key
actors and factors, and hypotheses about the
relationships between them. The design team will
supplement initial hypotheses with supporting
evidence obtained from researching government
documents, articles, and additional relevant source
material. This process is completed manually, using
physical whiteboards and post-it notes that document
the team’s conceptualization of the environment.
Written narratives are composed in Microsoft (MS)
Word documents. Pre-existing templated conceptual
diagrams may be filled out in MS PowerPoint. This
process is completed in a short period of time (i.e.
days). Figure 2 provides an example of a system
sketch created by SMEs using the OD process to
solve a fictional problem.
Figure 2: Example system sketch of a fictional scenario
created during a user exercise with CX system designers
and SMEs.
During planning exercises attended by the
authors, several limitations to the current OD process
were noted. First, domain experts across a range of
specializations are often unavailable. Secondly, the
design team identifies actors, systems, variables and
proposes hypotheses manually, based on prior
knowledge. As such, the quality of the resulting
system concept model largely depends on the
expertise, interpersonal “chemistry” and creativity of
the design team’s ability to identify relevant variables
and connections between variables. In doing so, the
team may rely on heuristics, which may be error
prone and subject to human biases (Das & Teng,
1999). Additionally, in supporting hypotheses with
evidence, the team manually searches through
documents, which inherently is a cumbersome, time-
consuming process. The lack of time reduces the
opportunity for an evidence-based weighing of
alternatives. Additionally, the scale and scope are
limited. OD exercises over three days typically result
in systems with ten to twenty factors. Current and
future states are envisioned, but system dynamics and
changes in factors over time are not strongly
considered. Finally, because this process is completed
by hand, the end product does not result in a
computational causal model with which to leverage
analytical tools. Validation of conceptual models and
solutions are performed manually by reviewing static,
slide-based products with senior staff.
3 RELATED WORK
In this section, we relate our work to relevant tools
that support causal reasoning and causality
visualization, including tools for mind mapping,
argument mapping, and causal modelling.
Mind mapping is a tool for visualizing one’s
mental model of a problem. While traditionally,
mind-mapping has been completed by hand on
whiteboards, several digital tools have been
developed to facilitate this activity (e.g., Subramanian
& Krishnamurth, 2020; Shih et al., 2009). However,
as Chen, and colleagues (2020) point out, few of these
provide computer-based support for idea and
hypotheses generation, or assist users with the
learning process. With this limitation in mind, they
developed QCue, which provides the user with
system-generated ideas to assist the user in exploring
topics as they develop a mind-map and user-elicited
queries that allow the user to explore a given topic in
depth. Wright et al., (2017) presented Argument
Mapper, for developing hypotheses in the mind-
mapping process through computer-based analytics,
with the goal of reducing human cognitive bias.
Analysts construct argument trees composed of
hypotheses, sub-hypotheses, assumptions and
evidence, and assess the credibility and evidence of
each item. Support for the upper-level hypotheses is
automatically calculated. CauseWorks provides tools
for mind mapping, allowing users to sketch their
hypotheses about causal relationships and factors
within a given system and then automatically finds
supporting or refuting evidence. Like QCue, users are
presented with suggestions to help facilitate learning
and encourage creativity, however in CauseWorks,
suggestions include true model additions, thereby
expanding the scope of the causal model.
In the context of causal modelling, automated
algorithms exist for extracting causal relationships
from factors within multivariate data sets (Wang &
IVAPP 2021 - 12th International Conference on Information Visualization Theory and Applications
52
Mueller, 2017). Factors are synonymous with
variables and are attributes (characteristics) of an
entity or their environment that influence the question
of interest (McPherson et al., 2007). Causal
relationships can be depicted in a directed acyclic
graph (DAG), which consists of a set of nodes and
links (Von Landesberger et al., 2011; Pearl, 2009).
Nodes typically represent relevant factors while links
represent the causal connections between factors.
Link properties include direction, strength, and
certainty (Bae et al., 2017), described through visual
encodings including color, line width, fuzziness, and
transparency. A central component of CauseWorks is
a computational causal model. The model is
displayed using nodes and links that visually encode
different stages of model construction, as well as
several static and dynamic model properties. In
CauseWorks the user is not constrained to a DAG
automatic graph layout, rather CauseWorks supports
a user-driven layout with the option to leverage
automatic graph layouts.
Causality cannot be computed from the data
alone. Wang (2018) noted the need for the domain
expert to interact with the causal graph. Zhang et al.,
(2015) developed an interactive correlation map for
filtering edges with weak correlations. Wang and
Meuller (2016) presented a platform in which users
modify a causal graph by connecting nodes, assigning
causal direction, deleting or marking edges as
unknown. In Causemos (Proulx et al., 2020), expert
modellers start with a knowledge base of extracted
causal statements displayed in a causal graph, and
then refine it. In CauseWorks, causal relationships
and factors are extracted from source documents,
resulting in a model database that users can search,
pull content from and edit to construct a
computational model.
In general, causality visualization methods are
limited in expression, scale, dimensionality, and do
not provide sufficient support for “what if” analysis,
injecting interventions, and development of solutions
to impact system behaviour (Kapler & Wright, 2018).
The CauseWorks system offers advanced causal
analytics to assist the team with answering
sensemaking questions (e.g., “what-if” and “how-to”)
in addition to considering multiple approaches to
solution development. In the next section we describe
the implementation of CauseWorks.
4 CAUSEWORKS SYSTEM
DESCRIPTION
To understand the model construction interface of
CauseWorks, it is necessary to have a basic
understanding of the underlying system, and how it
automatically generates causal model elements for
users to leverage in constructing their own models.
CauseWorks is composed of a web client and a
server (Node.js), both written in the Javascript
language (see Figure 3). The client application
components are built on the EmberJS framework,
with both D3.js and Cytoscape.js visualization
libraries for graph rendering, augmented with SVG-
elements for specialized visuals and interactions. The
CauseWorks server marshals communication
between the client and back-end analytics. These
back-end analytics consist of 3
rd
party program
performers (details available from DARPA). Firstly,
three Natural Language Processing reader
components extract events, causal assertions and
associated locations and actors from a shared corpus
of thousands of documents related to a given problem
domain (including expert-authored reports, news
media articles and open source material).
Figure 3: CauseWorks system architecture includes a
federation of three NLP readers and three causal modelling
framework performers. The ICM combines all models into
a single database, and the Arbitrator routes work among
readers and frameworks. Analytics, search, and other tools
are exposed to CauseWorks as services via a common API.
A common ontology is utilized to align readers,
and to associate events with “FactorTypes” curated
for a specific problem domain. Secondly, reader-
results are merged and then processed into a causal
model database by three causal-model frameworks
that provide model construction, simulation and
analytic functions. Factors are created where events
match FactorType definitions. Factor value and trend
is inferred from event trends or structured data
timeseries. Then, causal relationships between factors
are generated based on the FactorTypes of the cause
CauseWorks: A Framework for Transforming User Hypotheses into a Computational Causal Model
53
and effect events that comprise causal assertions.
Strength is inferred from assertion count, confidence
and, in some cases, correlations between factor
historical trends. The process for associating
extracted data with user-created Factors leverages
users input values for FactorType, actors, and
locations. In addition, CauseWorks includes model
frameworks for pre-built subsystems (e.g. economy
model for a nation-state). Figure 3 provides an
overview of the CauseWorks system architecture.
5 SYSTEM OBEJECTIVES AND
DESIGN REQUIRMENTS
Key human performance factors for OD are focused
on human learning, thinking and creativity. Important
performance objectives include: 1) increase learning
and comprehension about new, unfamiliar, unknown
situations, 2) engage human critical thinking in a
group debate that questions and tests alternative
perspectives, 3) encourage creativity by considering
different perspectives and merging selected aspects,
and 4) use group discourse as the catalyst to develop
new ways of thinking about problems and identifying
innovative solutions (Schmitt, 2006; ATP 5-0.1,
2015; Joint Publication JP 5-0; 2017)
In developing CauseWorks, we followed a user-
centric approach, engaging with expert OD
practitioners, trainers and students. Early design
began with structured interviews and system concept
sketches. Full-day, problem-focused exercises were
performed for system designers to observe traditional
OD teams working through regional conflict
scenarios. Results included the following high-level
objectives for CauseWorks functionality.
1. This system should support the rapid pace of OD
team brainstorming and discourse without
disruptive “care and feeding” of the software tools
2. The system should enable creation of an
unconstrained, notional, causal system, i.e. a
“sketch” system
3. The system should enable transformation of the
hypothesized system into a computational causal
model with minimal user effort or input.
4. The system should display the causal model in a
manner that allows sense-making of causal
structure, causal “flow”, predictions and impacts
of changes.
5. The system should provide causal analytics to
assist the team in understanding, validating, and
improving the model, uncovering patterns and
behaviours, and assist in developing a solution.
Note that one key assumption repeated by subject
matter experts (SME’s) was that models are never
objectively “correct”, but some are useful (Box,
1979). For OD of complex problems, supporting
high-level thinking and introducing new factors for
consideration is more important than the accuracy of
a specific model. An initial CauseWorks application
and analytics system was developed, followed by a
series of scenario-focused exercises conducted
with SMEs to assess system performance, usability,
and collect feedback to inform subsequent
development cycles.
The following sections focus on CauseWorks
HMI and human-machine workflow for creating and
using causal models.
6 CAUSEWORKS VISUAL
ANALYTICS WORKFLOW
The following sections describe how the affordances,
interactions and visual encodings enable our
envisioned machine assistance mantra of “capture
sketch hypotheses of a system, transform it into a
causal model, and provide insights with causal
analytics”. While this workflow is described as a
linear process, it is important to note that a team can
work through the process iteratively and move back
and forth through each step.
The main purpose of CauseWorks is working with
causal models. Accordingly, the design focus is on
causal model construction and analytics presentation.
We begin by presenting high-level interface
components and key visual encodings, and
subsequently discuss how to use the system.
6.1 General Workspace & Visual
Encodings
Current OD methodology relies heavily on the use of
physical whiteboards to allow teams to freely capture
discussion points and sketch diagrams. CauseWorks
is similarly centered about a digital whiteboard
workspace.
The HMI is comprised of a whiteboard
workspace, side panel, and main menu bar (see Figure
1). It is within the whiteboard workspace that the
team sketches and constructs their causal model.
Sketching tools in the toolbar include the creation of
user nodes, groups, and simple shapes. CauseWorks
also includes tools for grouping nodes, and then
IVAPP 2021 - 12th International Conference on Information Visualization Theory and Applications
54
collapsing or hiding group contents to declutter or
simplify the display. Groups can also combine factor
values into a single aggregate factor. A small right-
side panel provides tabs to access additional functions
and thus avoid floating windows that obscure the
whiteboard. Tabs within the side-panel allow the
team to navigate whiteboards; search the document
corpus and model database; edit factors and
relationships and connect them with evidence; access
analytic functions; and develop approaches to achieve
objectives. These functions are described in
subsequent sections.
In addition, CauseWorks supports synchronous
and asynchronous collaboration through shared
whiteboards, shared models and shared analytics. A
team can flexibly work with the system on a large
touch screen display and/or multiple workstations,
enabling co-located group-based OD processes as
well as distribution of tasks among separate teams or
individuals. Features and observations pertaining to
team collaboration using CauseWorks will be
described in a subsequent report (Kapler, Gray,
Vasquez, and Wright in preparation).
Figure 4: Visual encodings for user nodes, user links, causal
factors and causal relationships.
Key visual encodings in CauseWorks delineate
different stages and states in the construction of the
causal model (see Figure 4). Throughout the model-
building process, the team works with “user” nodes
and links, and causal factors and relationships. User
nodes and links have a distinct orange color with
rounded corners. They have no causal function and
are used to capture notes and ideas and for sketching
systems diagrams. User links can however represent
qualitative relationships between nodes, with width
representing notional strength, a color ramp
representing polarity (double-encoded with “+” or “-
icon) and color saturation representing confidence.
When a user node or link is promoted to a
computational causal factor or relationship, they
visually change to using black text, and a grey
background. A dashed line or outline indicates items
that are NOT backed by evidence. User-originated
causal factors retain their rounded corners while
system-generated causal factors have sharp, right-
angle corners.
These visual encodings were designed in
conjunction with SMEs to help see at a glance the
pedigree and state of the model as it evolves,
indicating where there is supporting data, or where
additional work is required; for example, backfilling
details after an intense team discussion.
6.2 Sketch Hypotheses of Complex
System
The OD process begins with team discussion about
actors, factors and influences involved in a problem
environment. This is typically captured as a series of
point-form notes. In CauseWorks, users enter this
information directly onto whiteboards using user
nodes and user links, in a process we describe here as
“sketching”. This gives CauseWorks analytics a
means to access team thinking and problem context:
a critical step in providing model-building assistance.
Supporting the rapid pace of team discourse in a
digital system (vs. physical markers and whiteboards)
requires simple, efficient affordances. For rapid note-
taking, user nodes can be created in quick succession
by hitting the Tab key. Links are created by dragging
a handle between nodes in one stroke, with direction,
strength, polarity and confidence set simultaneously
in single gesture (see Figure 5.5: Interactive Link
Editor).
As the team sketches a rough causal system, they
enter important concepts in the text of user nodes.
User training includes guidance for labelling nodes
intended to become factors in the model. For
example, a label should include a measurable
concept, along with locations and actors (e.g.
“Economic Growth in Canada”). This specificity
aligns with typical web-search syntax and enables
CauseWorks to more effectively assist in evolving the
model. An in-context search affordance that leverages
the label is provided that operates in two ways. First
it finds related factors in the model database that can
be either dragged onto user nodes to substitute them
in-place (see Figure 5.4), or just added to the
whiteboard if they are of interest to the user. This is
one way that computational causal factors are
surfaced for consideration into the user’s model. In
many cases users may not have previously known
about or considered these factors, thus potentially
CauseWorks: A Framework for Transforming User Hypotheses into a Computational Causal Model
55
Figure 5.1-5.6: Interactions for transforming a sketch system into a computational causal model.
injecting new thinking and scope into the model.
Second, the search tool performs a Lucene text search
(Goetz, 2000) into the document corpus, which
provides familiar web-search-like results for
researching the topic represented by a node label.
Results can be attached as evidence to a factor, thus
allowing users to manually connect their hypotheses
to data sources. This association provides additional
opportunities for the machine to understand user
thinking, problem context and apply analytics.
6.3 Transform Hypotheses Sketch into
a Computational Causal Model
As the OD process moves into system framing, the
user continues to transform their sketch into a
computational causal model. User nodes and links
can be instantly converted into computational causal
factor nodes and causal relationship links by clicking
“add to ICM” (Figure 5.2). As such, the team can
construct causal factors and relationships purely
based on their own knowledge or intuition without
pulling from the model database. This gives
CauseWorks flexibility to support unconstrained
causal thinking about any domain. All causal factors
have three key attributes:
1. Initial Value: value between 0-100 representing a
factors current value today relative to historical
norms, with 50 being “average”. A 0-100 range
was selected instead of discrete qualitative values
because it allows for higher granularity to reveal
small directional changes (e.g. a change of +5%
may be insignificant, but the direction of change
is perceivable)
2. Trend: The trend of the value at the current time,
i.e. increasing, decreasing or staying level
3. Confidence: a scale of 0-10
User-derived causal factors are given default values;
however, these can be modified at any time to reflect
user’s beliefs. It is important to note that in
CauseWorks, the user’s working model consists only
of factors and relationships that are placed on a
whiteboard. A dynamic model forward projection
runs automatically whenever the user modifies the
model. The projections can be run over a variable
number of months or years and are displayed in
sparklines on the nodes themselves. The grey and
black timeline with grey fill shows the “baseline”
projected value for a factor starting from “now”
through a number of time-steps in the future,
determined by executing a simulation of the model
(see Figure 6).
IVAPP 2021 - 12th International Conference on Information Visualization Theory and Applications
56
Figure 6: Baseline projections (top) and “What-if”
projection overlay resulting from intervention (bottom).
The next stage in model evolution is to populate
new factors and relationships with evidence from the
corpus through a process referred to as “grounding”.
This involves defining a factor in ontological terms,
and then running a process to connect it to the results
of machine reading. Each factor includes properties
to align it to a pre-determined domain ontology that
includes FactorTypes, active and affected actors and
locations (see Figure 5.3).
“Factor types” are high-level concepts detected by
the reader subsystems in the document corpus (see
System Description). The form can be completed
manually by the user, or through the “auto-fill”
feature, which uses machine reading of the user-
entered node label to extract terms and complete the
grounding form with suggested values.
Once grounded, a Factor’s “Data” tab will list
discovered events, assertions, and their associated
extracts from structured and unstructured data
sources, ranked by relevance. Trends in the structured
data and event history are used to determine Factors’
initial values and trend attributes. Explanation of
value calculations are presented to users in the “Data”
tab (development of explanations is ongoing and is
outside of the scope of this paper).
A separate processing stage finds new causal
relationships and factors related to the new factor
based on information in the knowledge base and
model database. These new relationships are stored in
the model database, but are not added to the user’s
whiteboard. Rather, it is the role of the Suggestion
System to provide in-context, discovery of new causal
relationships for a selected factor, and quickly
incorporate them into the model with minimal effort.
Suggested relationships are displayed in context
around a selected factor on the whiteboard (see Figure
4.6). They are displayed temporarily, with a blue
outline to distinguish them. Suggestions are ranked
based on multiple criteria including strength and
context. To accept a suggestion into the model, the
user clicks on it and it is added to the whiteboard.
Thus suggestions surface new factors and
relationships for consideration to improve and expand
the users model.
6.4 Apply Causal Analytics
CauseWorks provides causal analytics services for
improving the model, revealing behaviours, and
achieving planning objectives. A key user experience
goal is to incorporate analytics into the model
building process, thus results are always presented
within the whiteboard workspace to ensure continuity
of thought and context.
6.4.1 “What if Projection”
The “What-if” projection is the most frequently used
analytical tool in CauseWorks. “What-if” projections
are generated when users create “interventions” on
Factors to change the system. An intervention is
defined as an applied change in the value of a factor
at some point in the future. Applying interventions
and assessing their effects is a key mechanism for
understanding and validating model behaviour (see
Figure 7). Interventions also capture the means to
achieve the overall planning goals.
Figure 7: Intervention and Objectives Editor. Click on chart
to add and remove interventions and objectives.
Fast, intuitive entry of interventions allows rapid
testing and exploration of model behaviour. Users can
“sketch” interventions over the baseline projection
within a full-size version of the sparkline chart. When
an intervention is created, CauseWorks automatically
runs a model simulation and the resulting “What-If”
projections are overlaid in the node sparklines for
comparison, using green or red segments to
emphasize values above or below the baseline.
Interventions are highlighted in the whiteboard with
blue arrows. Aggregate change in a factors average
value over time compared to the baseline are pre-
attentively indicated with a saturated scale of green or
red applied to causal factor backgrounds, expressing
increase or decrease, respectively (see Figure 1)
making changes clearly visible within the generally
monochrome graph.
CauseWorks: A Framework for Transforming User Hypotheses into a Computational Causal Model
57
6.4.2 Sensitivity
The sensitivity tool takes a factor as input and
highlights other factors that it is causally sensitive to.
Degree of sensitivity is displayed using a 5-point
scale, displayed as magenta bars shown on the factor
nodes. (see Figure 8.1). This helps the team answer
questions such as “which other factors are likely
going to have a large impact on this factor?” When
applied to an objective factor, this tool assists the
team in identifying influencing factors as targets for
an intervention.
6.4.3 Most Impact
Most Impact indicates nodes with a high general
impact on the system based on a series of simulations.
These are also indicated with a 5-point scale,
displayed as magenta bars. This helps the team
answer questions such as, “which are the central
factors that all factors are most sensitive to?
6.4.4 “Why” Projections
“Why” Projection takes a single factor and time range
as inputs and returns factors that cause changes in the
value of that factor over the given time frame. Each
resulting contributing factor is marked with 1 to 5
small arrows to indicate strength and direction of
impact (see Figure 8.2). The “Why” function can
operate on either the baseline or the what-if
projection. This helps the team answer specific
questions about projection results, such as, which
factors lead to a jump at time ‘t’ in this Factor?”
6.4.5 Causal Loops
Causal Loops takes two factors as inputs (A and B)
and outputs paths between them that include
reinforcing or dampening loops occurring along each
path. Loops are an important consideration in
thinking about complex systems as they can
significantly enhance effects of an intervention.
Loops are highlighted in the whiteboard with a
repeating animated cycle that is effective at
emphasizing complex graph paths (Ito et al., 2016).
Figure 8.3 includes a static image of a causal loop.
6.4.6 Causal Pathways
Causal Pathways takes two factors as inputs (A and
B) and returns a new whiteboard that contains all
significant paths in one direction from A to B. This
helps the team answer questions such as “how does a
change in factor A propagate to factor B?”
6.4.7 Approach Helper
CauseWorks provides tools for developing
“approaches” to influence a system to meet planning
objectives. These include tracking objectives,
suggesting interventions to achieve them, finding
unexpected impacts and opportunities, measuring
solution performance, and managing multiple
approach options (see Figure 1).
The key information that helps drive these
analytics are users’ objectives. Objectives are defined
as a desired change in the value of a factor at some
point in the future. Explicitly setting user objectives
captures critical information about the team’s goals,
which in-turn helps inform machine assistance
functions for suggesting factors, recommendations
for interventions to achieve objectives, and generally
improves system awareness of problem context and
scope. Objectives are visually represented by an
orange star icon on factors and in the sparklines.
Objectives are entered using the same interface as
interventions; users can set both time and value for an
objective with a single click on the timeline (see
Figure 7). An example of an objective is “increase
sanctions on Country B starting in October”.
In CauseWorks, an approach is a collection of
objectives and the interventions applied to achieve
them. Teams may develop multiple named
approaches to a planning problem, such as
“Aggressive Solution”, or “Least Risk”. Within the
Approach Management Tab, users can name,
organize, open, save and copy different approaches
(see Figure 8.4). For a selected approach, the
objective and intervention factors are displayed as a
list of nodes. The Approach Helper is an analytic
function that can automatically propose interventions
to meet objectives. It uses objectives in the active
Approach as the inputs and allows users to set
constraints on the timing and size of interventions it
proposes. When executed, this function will create
interventions on factors in the user model and add
them to the current Approach. The “Refine” tool is
similar to the Helper, however it only adjusts existing
interventions that the team has already set, optimizing
them to meet objectives.
When considering which factors to apply
interventions to, it is important to recognize that not
all factors represent something that can be influenced
directly as part of a solution (e.g., GDP of the USA).
As such, the team can toggle a flag to exclude certain
factors from intervention consideration.
IVAPP 2021 - 12th International Conference on Information Visualization Theory and Applications
58
Figure 8.1: Sensitivity results for “Populace Mood In
“Country B” indicated by magenta bars in the whiteboard
view.
Figure 8.3: Example of Causal Loops result.
Figure 8.2: “Why” results show contributions by othe
r
factors on “Populace Mood In Country B” for a time perio
d
set by user (not shown). Up arrow-stack indicates relative
amount of increasing influence.
Figure 8.4: Approach Tab showing objectives in “Populace
Mood in Country B” being met with interventions generate
d
b
y the Approach Helper tool. Score of “100%” indicates
objective target values are fully met in the What-i
f
projection.
Each Approach can also include an optional
automatically generated text summary of
interventions, impacts on objectives and alternative
intervention suggestions through an Approach
Narrative. The narrative display includes embedded
graphical representations of factors nodes, with
linked highlighting and drag-and-drop to the
whiteboard. This allows the user to connect the
narrative to the whiteboard view and helps call out 2
nd
and 3
rd
order effects that users may not have noticed.
As such, the narrative can surface potentially hidden
or surprising information, a key benefit of combined
human-machine systems (Wickens, Hollands,
Banbury, & Parasuraman, 2015). The narrative
engine is a separately developed plug-in component,
and its design and assessment is the focus of a report
by (Choudhry et al., 2020)
7 USER EVALUATION
OD SMEs were involved in CauseWorks
development from the early stages of the system
design through multiple hands-on evaluation
exercises. Observations and interviews were
conducted to determine design requirements and
inform the initial system design. Multi-day exercise-
based evaluations have been conducted every 6
months. Each involved a fictional scenario matched
to a corpus of scenario-related documents processed
into a knowledge base. Participants included
experienced OD experts, government–provided
problem domain experts, and OD students from the
US Military. Teams were formed to address problems
and present solutions over several days using
CauseWorks. The goal of these exercises was to
determine whether the system meets users’ needs, and
identify issues and functional gaps, to inform
subsequent CauseWorks iterations.
Two key concerns were closely followed by the
authors: 1) that users are able to quickly learn the
basic interactions necessary to sketch systems,
compose models, and develop approaches, and 2)
users can produce planning products within normal
planning timelines. HMI training took a half-day,
including lower-level system background. Once users
had hands-on CauseWorks, they were quickly able to
use the system, possibly due to intentional similarities
CauseWorks: A Framework for Transforming User Hypotheses into a Computational Causal Model
59
Figure 9: Affinity diagram of user feedback collected from planning exercises.
with tools such as MS PowerPoint. Within a 3-day
standard planning exercise, teams using CauseWorks
were able to generate models and approaches for their
target problems, and then brief superiors using the
live system. Further, they were able to adjust
solutions on-the-fly to respond to Commanders
questions and feedback. Significantly, this
demonstrates that it is possible to incorporate causal
modelling into the planning process with minimal
negative effect on schedule or workload.
Post-exercise surveys were conducted by
government staff and CX program performers to
collect feedback about CauseWorks. We compiled
survey answers from one exercise and constructed an
affinity diagram (Harboe & Huang, 2015) comprising
five themes in the data (see Figure 9). Many
participants noted benefits and contributions of the
system, suggesting that the CauseWorks provides
advantages over traditional operational design
methods. As with most prototypes, opportunities to
improve the system were noted. Some users observed
that creating and grounding factors was a
cumbersome multi-step process. Indeed, the
grounding process is not part of the traditional OD
process, however it is necessary to connect factors to
extracted data from the corpus. Additionally, SMEs
observed that as models increased in scale, there were
challenges with untangling links and grasping
complex causal flows. SMEs also questioned the
accuracy of event associations to Factors, though this
can be attributed to limitations in automatic event and
assertion extraction technologies. Finally, SMEs
provided recommendations and suggestions to
improve the interface, several of which have since
been addressed.
8 LESSONS LEARNED &
LIMITATIONS
In this section we discuss limitations and potential
future research suggested by this work. This includes
the efficacy of user-driven layouts, fixed vs.
zoomable whiteboards, the distribution of models
across multiple whiteboards, the construction of
causal models by novice modellers, and the use of
color to convey causal attributes and effects.
Figure 10: Example system sketch from problem-solving
exercise with CauseWorks. This model consists of
approximately 1000 nodes and edges created during a
multi-day exercise. Image intentionally blurred to obfuscate
confidential details.
An early design decision was to emphasize user-
driven layouts, rather than applying automated graph
layouts to the evolving causal model. This allows
teams to arrange information according to their own
criteria and logic and provides an additional
contextual dimension from which machine analytics
could extract meaning (e.g. users often cluster related
items). User layouts also support long-term shared
team cognition, as users become familiar with
IVAPP 2021 - 12th International Conference on Information Visualization Theory and Applications
60
groupings and arrangements and remember where to
find things. We observed that this contributed to
maintaining comprehension of models containing
nearly 1000 nodes and edges, well beyond our
scalability expectations (see Figure 10).
At this scale, maximizing use of screen space and
label readability has a significant impact on a teams’
ability to view a model together, even on an 80-inch
4k display. Independent virtual collaboration spaces
are a technical alternative to wall-size displays,
however the impact of virtual meetings on OD
discourse is unknown. A large touchscreen display
was available at one exercise, and we observed
stronger team engagement, discussion and model
development compared to a single-laptop-per-team
configuration. Users did ask for layout tools to
automatically arrange factors based on topical
groupings, such as geography or affiliation.
Early implementations of CauseWorks provided a
fixed-scale whiteboard, as most virtual whiteboards
take this approach (e.g. Google Jamboard) and there
are documented challenges associated with zoomable
workspaces. Zooming is a weak method for subgraph
comparison and in some instances, users prefer an
overview context compared to zoom interaction due
to simpler navigation (Büring, Gerken, & Reitere,
2006). However, as models increased in size and
complexity, SMEs demanded zoom-able whiteboards
to fit growing models into a single view.
A related challenge is how to effectively work
with large models that are distributed across multiple
whiteboards. Initial thinking was to place major
subsystems on separate whiteboards, however,
observations from exercises revealed difficulties in
recalling relationships between nodes on different
whiteboards. Indeed, as the distance between sources
of information increases, whether across multiple
displays or whiteboards, there is a cost to the user in
maintaining information in working memory
(Wickens et al., 2015) and users tend to perform
fewer information seeking behaviours (Fu and Gray,
2006). Eventually users elected to place large models
on a single whiteboard to avoid this effect. Further
research is required to effectively allow
comprehension of connections in systems that span
multiple pages.
Functions are needed to support model building
by domain experts and planning experts who are not
model building experts. Determining the appropriate
scope and level of detail to model impacts the speed
of model development, validity, speed of
experimentation and confidence in model results.
Building more complex models, because it is possible
to do so, does not necessarily equate to effective
modelling (Robinson, 2004). It is important to avoid
modelling every aspect of a system. Simpler models
can be developed faster, are more flexible, require
less data and are easier to interpret, validate and
maintain because the structure of the model is clearer
(Chwif, 2000). The ease with which CauseWorks
users can construct models impacts the size of the
systems they create. This can enable inclusion and
consideration of many more factors, causes and
solutions than might otherwise occur using traditional
methods, however one should acknowledge the
possibility of overwhelming human comprehension.
A cycle of model simplification or filtering could help
reduce a large model to make it more comprehensible.
In general, however, we observed that users find
value in building larger systems to reflect real-world
complexities, and also that the system can help with
sense-making of larger models through use of
analytics, and by enabling distributed work through
collaboration.
CauseWorks use of green and red for indicating
both link polarity and factor value change, merits
explanation. Use of color is important when a user
must attend to changing patterns on an interface
(Brewer,1996). Our hypothesis is that the shared
color schemes reinforce each other (increase/support
vs. decrease/oppose; Wickens et al., 2015), and thus
makes recalling combinations of link-factor effects
easier than with two separate, two-color schemes (4
colors total; Brewer, 1996). A 4-color scheme also
limits unique color use for other information. User
feedback on color usage includes mention that red is
often used to represent “enemy” in military
convention, which may cause confusion in
interpretation, however despite providing alternative
color options in CauseWorks, users continued to use
the green-red color scheme without issue. We suggest
that further research into optimal color schemes for
military use of causal model representations may be
useful.
9 CONCLUSIONS
Causal Model building for complex problems has
typically been completed manually by domain
experts and is a time-consuming, cumbersome
process. OD is a process of rapid, structured discourse
for teams to envision systems and relationships about
complex, “wicked” problems, however, the resulting
models are simple diagrams produced on whiteboards
or slides, and as such, do not support computational
analytics, thus limiting usefulness. In this paper we
introduced the HMI and workflow for CauseWorks, a
CauseWorks: A Framework for Transforming User Hypotheses into a Computational Causal Model
61
tool for expert planners (but novice model builders)
to create computational causal models of complex
problems. We presented how users can sketch
hypotheses about a system on a digital whiteboard
and connect it to automatically extracted information
that suggests system behaviour, thereby transforming
the sketch into a computational causal model.
CauseWorks also helps expand OD team thinking and
model development by suggesting new factors to add
to the model, and by providing analytics to support
sense-making and solution development. In applied
planning exercises, military operational planners with
no prior modelling experience were able to use
CauseWorks to construct and use computational
causal models to develop approaches for realistic
complex planning scenarios, within typical planning
time constraints. Planners thought CauseWorks
supported the OD process and helped them consider
new ideas.
Future work should investigate the following:
ways to present connections between models
spanning multiple whiteboards; assessment of model
characteristics built by novice modellers; deeper
investigation into causal symbology and color-use for
military applications. Formal experiments should be
performed to assess impact of using CauseWorks
modelling tools in operational design vs traditional
methods.
ACKNOWLEDGEMENTS
This work was supported by the Defense Advanced
Research Projects Agency (DARPA) under Contract
Number FA8650-17-C-7720. The views, opinions,
and findings contained in this report are those of the
authors and should not be construed as an official
Department of Defense position, policy, or decision.
This work has been approved for Public Release,
Distribution Unlimited. The authors wish to thank all
Causal Exploration collaborators for support and
encouragement.
REFERENCES
ATP 5-0.1, Army Design Methodology, HQ Dept of the
Army, July 2015.
Bae, J., Helldin, T., & Riveiro, M. (2017). Understanding
indirect causal relationships in node-link graphs.
Computer Graphics Forum, 36(3), 411-421.
Box, G. E. P. (1979), Robustness in the strategy of scientific
model building, in Launer, R. L.; Wilkinson, G. N.
(eds.), Robustness in Statistics, Academic Press, pp.
201–236.
Brewer, C. A. (1996). Guidelines for selecting colors for
diverging schemes on maps. Cartographic Journal,
33(2), 79-86
Büring, T., Gerken, J., & Reiterer, H. (2006). User
interaction with scatterplots on small screens - A
comparative evaluation of geometric-semantic zoom
and fisheye distortion. IEEE Transactions on
Visualization and Computer Graphics, 12(5), 829-836.
Chen, T. J., Subramanian, S. G., & Krishnamurthy, V. R.
(2020). QCue: Queries and Cues for Computer-
Facilitated Mind-Mapping. Paper presented at the
Proceedings – Graphics Interface
Choudhry, A., Sharma, M., et al. (2020). Once upon a time
in visualization: Understanding of textual narratives for
causality, IEEE Transactions on Visualization and
Computer Graphics
Chwif L, Barretto M.R.P., & Paul R.J. (2000). On
simulation model complexity. In: Joines JA, Barton
RR, Kang K and Fishwick PA (eds). Proceedings of the
2000 Winter Simulation Conference. IEEE:
Piscataway: NJ, pp 449–455.
Dang, T. N., Murray, P., Aurisano, J., & Forbes, A. G.
(2015). ReactionFlow: An interactive visualization tool
for causality analysis in biological pathways. BMC
Proceedings, 9
DARPA, (2018). https://www.darpa.mil/program/causal-
exploration.
Das, T. K., & Teng, B. S. (1999). Cognitive biases and
strategic decision processes: An integrative
perspective. Journal of Management Studies, 36(6),
757-778.
Fu, W. T., & Gray, W. D. (2006). Suboptimal tradeoffs in
information seeking. Cognitive Psychology, 52(3), 195-
242.
Gil, Y., Pierce, S. A., Babaie, H., Banerjee, A., Borne, K.,
Bust, G., . . . Shekhar, S. (2019). Intelligent systems for
geosciences: An essential research agenda.
Communications of the ACM, 62(1), 76-84.
Goetz, B. (2000). The Lucene search engine: Powerful,
flexible, and free. JavaWorld. http://www. javaworld.
com/javaworld/jw-09-2000/jw-0915-lucene. html.
Gray, W. D., & Fu, W. T. (2004). Soft constraints in
interactive behavior: The case of ignoring perfect
knowledge in-the-world for imperfect knowledge in-
the-head. Cognitive Science, 28(3), 359-382.
Harboe, G., & Huang, E. M. (2015). Real-world affinity
diagramming practices: Bridging the paper-digital gap.
In Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems (pp. 95-104).
Ito, T., & Misue, K. (2016). A visualization technique using
loop animations. In International Conference on
Human Interface and the Management of Information
(pp. 136-147).
Joint Publication JP 5-0, Joint Planning, Joint Chiefs of
Staff, June, 2017.
Jonassen, D. H., & Ionas, I. G. (2008). Designing effective
supports for causal reasoning. Educational Technology
Research and Development, 56(3), 287-308.
IVAPP 2021 - 12th International Conference on Information Visualization Theory and Applications
62
Keim, D. A., Kohlhammer, J., Ellis, G., & Mansmann, F.
(2010). Mastering the information age; solving
problems with visual analytics. Eurographics.
McPherson, K., Marsh, T., & Brown, M. (2007). Foresight
report on obesity. The Lancet, 370(9601), 1755.
Pearl, J. (2009). Causal inference in statistics: An
overview. Statistics Surveys, 3, 96-146.
Proulx, P., Husain, F., Gyori, B. M., & Pyarelal, A. (2019).
MOIRE: MOdeling and Inference from REading.
Demonstration presented in Modeling the World's
Systems Conference.
Rittel, H. W. J., Webber, M. M. (1973). Dilemmas in a
general theory of planning. Policy Sciences, 4(2), 155–
169.
Robinson, S. (2004). Simulation: The Practice of Model
Development and Use, John Wiley and Sons Ltd:
England.
Schmitt, J., A Systemic Concept for Operational Design,
(2006) US MC Warfighting Laboratory. http://www.
mcwl. usmc. mil/file_download.
Sedig, K., Parsons, P., & Babanski, A. (2012). Towards a
characterization of interactivity in visual analytics.
Journal of Multimedia Processing Technologies, 3(1),
12-28.
Sedlmair, M., Meyer, M., & Munzner, T. (2012). Design
study methodology: Reflections from the trenches and
the stacks. IEEE Transactions on Visualization and
Computer Graphics, 18(12), 2431-2440.
Shih, P. C., Nguyen, D. H., Hirano, S. H., Redmiles, D. F.,
& Hayes, G. R. (2009). GroupMind: Supporting idea
generation through a collaborative mind-mapping tool.
Paper presented at the GROUP'09 - Proceedings of the
2009 ACM SIGCHI International Conference on
Supporting Group Work, (pp.139-148).
Von Landesberger, T., Kuijper, A., Schreck, T.,
Kohlhammer, J., van Wijk, J. J., Fekete, J. D., &
Fellner, D. W. (2011). Visual analysis of large graphs:
state‐of‐the‐art and future research challenges.
Computer Graphics Forum, 30(6), 1719-1749
Wang, X. (2018). A Comparison of Causal Inference
Methods and Their Application in Big Data Analytics
(Doctoral Dissertation), Louisiana State University,
Louisiana, USA.
Wang, J., & Mueller, K. (2018). Visual causality analysis
made practical. Paper presented at the 2017 IEEE
Conference on Visual Analytics Science and
Technology, VAST 2017 Proceedings, 151-161.
Wickens, C. D., Hollands, J. G., Banbury, S., &
Parasuraman, R. (2015). Engineering psychology and
human performance. Psychology Press.
Wright, W., & Kapler, T. (2018). Challenges in Visualizing
Complex Causality Characteristics. IEEE Pacific
Visualization. 2018. Proceedings
Wright, W., Sheffield, D., & Santosa, S. (2017). Argument
mapper: Countering cognitive biases in analysis with
critical (visual) thinking. In Information Visualisation
(IV), 2017 21
st
International Conferences, IEEE, (pp.
250-255).
Wirtz, J. J. (2017). Life in the “Gray Zone”: observations
for contemporary strategists. Defense & Security
Analysis, 33(2), 106-114.
Zhang, Z., McDonnell, K. T., Zadok, E., & Mueller, K.
(2015). Visual correlation analysis of numerical and
categorical data on the correlation map. IEEE
Transactions on Visualization and Computer
Graphics, 21(2), 289-303.
Zhang, X., & Vesselinov, V. V. (2017). Integrated
modeling approach for optimal management of water,
energy and food security nexus. Advances in Water
Resources, 101, 1-10.
CauseWorks: A Framework for Transforming User Hypotheses into a Computational Causal Model
63