Evaluating A Novel Agile Requirements Engineering Method: A Case

Study

Tanel Tenso, Alex Norta and Irina Vorontsova

Department of Informatics, Tallinn University of Technology, Tallinn, Estonia

Keywords:

Visualization, Agile, Requirements Engineering, Modelling, Goals.

Abstract:

The use of agile methods during software development is a standard practice and user stories are an established

way of breaking complex system requirements into smaller subsets. However, user stories do not sufﬁce for

understanding the bigger picture of system goals. While methods exists that try to solve this problem, they lack

visual tool support and are too heavy for smaller projects. This article ﬁlls the gap by evaluating a novel agile

agent-oriented modelling (AAOM) method for requirements engineering. The AAOM-method comprises a

visual approach to agile requirements engineering that links goal-model creation techniques taken from agent-

oriented modelling and connects goals intuitively to user stories. A case study based evaluation explores the

applicability of AAOM for requirements engineering in an agile software development process.

1 INTRODUCTION

Requirements engineering (RE) is an important

software-development activity and traditionally con-

sidered one of the ﬁrst phases in software develop-

ment. RE is a process of formulating, documenting

and managing the requirements for software and com-

prises requirementsidentiﬁcation, analysis, documen-

tation and validation (Hull et al., 2010). Since RE is

the ﬁrst software-development phase, late detected er-

rors are very costly (Carlson and Matuzic, 2010) and

produce incorrect software that does not satisfy cus-

tomer needs.

Agile development is the most widely used

method (Version One, 2016) for developing software

systems that adhere to theagile manifesto (Beck et al.,

2001). Agile development methods are time-boxed,

iterative and incremental. Characteristic is also a

frequent delivery of usable software and collabora-

tion with customers. Additionally, the agile method

supports self-organizing cross-functional teamwork.

These factors play a role in the ability to quickly re-

spond to changes (Cao and Ramesh, 2008). For agile

afﬁliated RE, several variations exist that affect the

use of RE, e.g., Scrum (Schwaber, 2004), XP (Beck,

2000), Lean (Poppendieck and Poppendieck, 2007),

Kanban (Kniberg and Skarin, 2010). Common for

these agile variations are of a lacking intuitive align-

ment (Cao and Ramesh, 2008) between engineered

requirements and intuitive visual system development

support.

To address this gap we deﬁne a novel require-

ments engineering method, namely the agile agent-

oriented modelling (AAOM) method (Tenso and

Taveter, 2013). AAOM emerges through multiple ex-

periments in various projects varying from small scale

projects, for example adding release management ca-

pability for a task management system, to large scale

distributed projects like crisis simulation software de-

velopment. AAOM is derived from agent-oriented

modelling (AOM) (Sterling and Taveter, 2009), a

holistic method for analysing and designing socio-

technical systems consisting of humans and technical

components. More concretely, AAOM focuses on a

speciﬁc model type out of a larger model set that are

part of AOM, namely goal models comprising func-

tional goals, quality goals that are also knows as non-

functional goals, and afﬁliated roles these goals af-

fect.

To evaluate the applicability and usefulness of

AAOM, we choose a case study based research

methodology (Runeson et al., 2012; Yin, 2013). We

use as a running case a project for developing a

lost&found (l&f) mobile app. The business idea of

the mobile app is to reunite lost objects of any type

with their rightful owners. Instead of having to rely

on lost&found ofﬁces at police stations, airports, cin-

emas and so on, the l&f-app is a simple and quick mo-

bile solution to report ﬁndings by using smartphone

capabilities: a phone camera allows to instantly take

156

Tenso, T., Norta, A. and Vorontsova, I.

Evaluating A Novel Agile Requirements Engineering Method: A Case Study.

In Proceedings of the 11th International Conference on Evaluation of Novel Software Approaches to Software Engineering (ENASE 2016), pages 156-163

ISBN: 978-989-758-189-2

a picture of a found item while simultaneously the

photo-shot location is traced and stored via smart-

phone GPS. The app is also beneﬁcial for people who

lose something: the app announces the loss and re-

ceives notiﬁcations when an item with a similar de-

scription and attributes is found and entered into the

database. The l&f-app either establishes the associ-

ation between object and owner because of provided

descriptions, or the object is tagged with a visual iden-

tiﬁcation code a mobile phone can read. The l&f-

app also offers a so-called giveaway section for found

items that nobody claims.

The remainder of the paper is structured as fol-

lows. Earlier studies and theory of AAOM are in Sec-

tion 2. Research questions, case setup, data collec-

tion, analysis and validity procedures are described in

Section 3. In Section 4, we present results and ﬁnd-

ings of the analysis of collected data. Finally, Section

5 concludes ﬁndings and gives open issues for future

work.

2 RELATED WORK

To provide a basis for the evaluation, in Section 2.1,

we give more details about the application of AAOM.

In Section 2.2, we discuss similar methods to AAOM

for comparison.

2.1 AAOM-Method Explanation

Face-to-face communication in agile over written

speciﬁcations facilitates embracing change and apply-

ing iterative cycles for RE. Figure 1 depicts (Tenso

and Taveter, 2013) how AAOM activities ﬁt into the

agile development lifecycle. After an initial discov-

ery phase, the result is a set of further elaborated goal

models and afﬁliated user stories for which a prelim-

inary backlog is established. Next, in iterations, the

goal models and their user stories evolve by chang-

ing, eliminating, updating the latter. Once the main

development phase of a project ends, the system-

maintenance phase commences where the goal mod-

els and user stories are the foundation for exploring

how to add, modify, or remove features.

The AAOM method links goal models from AOM

(Sterling and Taveter, 2009) to user stories in accor-

dance with Figure 2. In the depicted model, goals are

shaped as parallelograms and quality goals shaped as

clouds represent functional- and non-functional sys-

tem requirements respectively. Goal models also con-

tain roles as sticky men with relationships between

roles and goals/quality goals. User stories are at-

tached in Figure 2 at the leave-level of sub-goals for

Figure 1: AAOM activities in agile development lifecycle.

The Notation For Goal Model

Reunite Item

Trustworty,

using

protocols

Lost Item

User

Enter lost item info Search lost item

User Stories

As an app user I want to be able to see list of all lost

items from database

As an app user I want to be able to see list of found items

from database

As an app user when clicking on item in a list of found/

lost items I want to see all available item information on a

screen (picture + all the info)

Symbol Meaning

Goal

Quality Goal

Role

Relationship between

goals

Goal and quality goal

relationship

Figure 2: Example of a goal model with user stories at-

tached.

establishing and tracing a connection to a system’s top

goal.

User stories are simple artefacts for agile software de-

velopment and requirements documentation (Cohn,

2004). A user story is a written sentence or two

that describe functionality from a system’s user point

of view. There are several formats and concepts, in

which Cohn’s (Cohn, 2004) deﬁnition is as follows:

As a <role/type of user>, I want <goal/desire> so

that <beneﬁt/reason> .

Examples:

As a user, I want to reserve a hotel room.

As a frequent ﬂyer, I want to rebook a past

trip, so that I save time booking trips I take

often.

A user story must be small enough for implementa-

tion within one development iteration (Cohn, 2004).

This limits implementation work per user story, pro-

viding a fast feedback and veriﬁcation of require-

ments for system development.

Similar to AAOM, other methods exist, that pro-

vide structure to agile requirements by organizing

Evaluating A Novel Agile Requirements Engineering Method: A Case Study

157

user stories. These methods are brieﬂy discussed in

the next section.

2.2 Earlier Studies

Similar to AAOM, there are methods for organiz-

ing user stories into structures to mitigate visibility

problems, for example Cohn (Cohn, 2004) suggests

Epics, that are bigger user stories grouping smaller

ones. Epics covering different levels of abstrac-

tion are also used in Scaled Agile Framework(SAFe)

(Lefﬁngwell, 2016). Large-Scale Scrum (LeSS) (Lar-

man and Vodde, 2008) concentrates on splitting re-

quirements into small Product Backlog Items, usu-

ally user stories. Disciplined Agile Delivery (DAD)

(Ambler and Lines, 2012) sums up many agile prac-

tices and introduces the term Portfolio management

that is requirements management in a hierarchical list

of work items. Scrum of Scrums (Sutherland et al.,

2007) includes team level planning and requirements

tracking between teams. The lean approach to ag-

ile requirements (Lefﬁngwell, 2010; Lefﬁngwell and

Aalto, 2009) divides requirements according to de-

tails onto team-, program- and portfolio level. De-

spite having several positive impacts and inﬂuenc-

ing AAOM theory aspects, these approaches are not

truly visual and, according to our experience not gras-

pable without special training by non IT stakehold-

ers. Furthermore, these methods are meant for enter-

prise scale usage and are too heavyweight for smaller

projects where it is important to establish a conversa-

tion with clients and align everybody to the same set

of goals.

On the other hand, goal modelling techniques ex-

ist exactly for depicting system goals in a visual way.

Goal-based requirements engineering is well estab-

lished, for example Hull et al. (Hull et al., 2010) sug-

gest representing use scenarios as a sequence of goals.

Lamsweerde approaches goal modelling from a for-

mal point of view, providing a mathematical proof and

meta model for goal based requirements engineer-

ing. His effors culminate in (Van Lamsweerde et al.,

2009), covering high-level system analysis with goal

techniques. One prominent method that includes goal

modelling is i* (Yu, 2009) that provides both tooling

and principles for the problem domain. Finally, an

example of more social goal-modelling techniques is

described in agent-oriented modelling (AOM) (Ster-

ling and Taveter, 2009) theory that is the basis for

AAOM. However, the aforementioned approaches

are too heavyweight to be included into short ag-

ile development feedback cycles and are meant for

model driven development. While AOM provides

the most lightweight goal modelling technique for re-

lating roles, functional- and non-functional require-

ments, its other models focus on the agent paradigm

and are thus too speciﬁc for wider system-design use.

Connecting agile user stories with goal models is a

novel approach introduced by AAOM.

3 CASE STUDY DESIGN

We choose a case study research method to conduct

the evaluation of the AAOM-method. First we de-

ﬁne research questions in Section 3.1 for guiding the

AAOM evaluation, followed by a justiﬁcation of case-

and subject selection in Section 3.2. Data sources of

evidence are discussed in Section 3.3, followed by an

analysis procedure in Section 3.4. A validity discus-

sion ﬁnalizes the case study design in Section 3.5.

3.1 Research Questions

Based on previous experiments with the AAOM

method and feedback gathered during them, the fol-

lowing main research question is devised:

How does AAOM help to improve software engineer-

ing requirements engineering activities?

This question can be reﬁned in various sub-questions:

RQ1: What are the beneﬁts of using AAOM from a

user perspective?

RQ2: What effect has project setup and tooling on

AAOM usage?

RQ3: What aspects of the AAOM-method usage

need further reﬁnement?

Aforementioned questions establish the basis for se-

lecting appropriate case, data sources and analysis

methods.

3.2 Case Selection

The l&f-app development project follows an agile

software development cycle, speciﬁc scrum tech-

niques such as planning with user stories, back-

log management and iterative development are used.

Three development iterations take place in which

work is visualized on a task board and meetings with

clients take place at the end of iterations.

For the l&f-project research, we set up as a single

case study with holistic design (Runeson et al., 2012).

The unit of analysis and the case is an AAOM RE

method application for iterative requirements gather-

ing. Four people participate in this project carrying

the labels role and experience. Role helps us to evalu-

ate the AAOM usage by different team members, ex-

perience provides reliability for gathered information.

Three participants ﬁll the role of client who order the

ENASE 2016 - 11th International Conference on Evaluation of Novel Software Approaches to Software Engineering

158

l&f-app development, whereas one of them has a high

level of experience as an ICT expert while the other

two are from different domains. One person acts as

an analyst and developer, being a beginner in the ﬁrst

role and highly experienced in the second. A high

level of experience means a person acts at least for

two years in a speciﬁc role.

The research team consists of three researchers

working in co-operation, providing peer-review to

each other. The procedures to observe a case include

taking part in all meetings between case subjects that

are modelling, demo and retrospective sessions while

video recoding them all. We act as silent participants

taking notes of the AAOM usage throughout differ-

ent meetings. Based on the research questions and

meeting notes, we devise interview questions and ses-

sions for gathering qualitative data. The analysis of

the gathered data by researchers provides answers to

the research questions.

3.3 Data Sources

Interviews are the most valuable data source for

the running case study. Based on recommendations

from (Runeson et al., 2012), interview planning com-

mences with selecting interviewees. There are four

people and three roles in the running case as men-

tioned in Section 3.2 and all participants are inter-

viewed. Since there is one person in both roles of

analyst and developer, different questions are posed

to her addressing both roles. Next follows the plan-

ning of interview sets. Ideally, interviews take place

multiple times, for example after elaborating a ﬁrst

branch of requirements, after every development iter-

ation and at the end of the project. As a limitation,

for this l&f-app project, we conduct only one set of

interviews after completing the planning session and

three development iterations.

For conducting interviews, we choose a semi-

structured format (Runeson et al., 2012). The re-

search questions stated in Section 3.1 and the meet-

ing notes guide the preparation of interview questions.

Different sets of questions target each respective role

without offering predeﬁned answers. Thus, intervie-

wees can not answer, e.g., ”yes” or ”no”, instead they

must express their own opinions. Questionnaires for

client, analyst and developer roles are depicted in a

longer version of this article

The interviews adhere to a time-glass model

(Runeson et al., 2012) so that an interview begins

with broad questions ﬁrst and continues with more

speciﬁc questions. At the end of an interview, again

broad questions are presented. Thus, four interviews

http://digi.lib.ttu.ee/i/?4064

include three with clients and one combined for the

developer/analyst. The interview structure is similar

in all cases and the interviewees are informed about

the interview structure during the process.

Each interview session lasts roughly one and a

half hours and starts with an introduction, followed

by role speciﬁc questions. The interviews are audio

recorded into MP4 ﬁles for subsequent post-interview

activities and analysis. In case an interviewee re-

sponds to a question brieﬂy, we ask additional ques-

tions on the same topic to gather more insight. At

the end of an interview, a participant learns that ev-

ery interview is transcribed and sent for verifying the

captured ideas are correct.

We also gather work artefacts, such as goal mod-

els, user stories and source code. Since a dedicated

development toolkit for AAOM does not exist yet, we

employ provisionally a set of freely available tools to

host our case. The ﬁrst tool, Draw.io

, is an online

diagramming tool to draw goal models for the l&f-

app. Another used tool is Trello

, a collaboration tool

to organize project tasks on boards. Trello visualizes

tasks in the form of user stories similar to post-it notes

in status columns to observe the progress during soft-

ware development, e.g., to do, pending, in progress,

completed, and so on.

The generated goal modelsand user stories are rel-

evant for investigating the evolution during the project

while the history of changes is recorded. Project

source code for the l&f-app in a version control sys-

tem allows to observe lines of code (LOC), changes

in LOC, time between LOC changes and links to

user stories. After evaluating these quantitative data

sources, the main ﬁnding is that there is no existing

body of knowledge for analysing them as needed for

the AAOM method usage validation. Thus, we use

these anecdotal data sources only to subjectively eval-

uate some statements received from interviewees.

As our main data source is interviews, we focus

in the next section on interview analysis. The latter

yields answers to the research questions.

3.4 Analysis Procedure

Interview analysing requires ﬁrst transcribing and

then coding (Salda˜na, 2015) the results and for both

we use the tool NVivo

. We transcribe the interviews

and for corrections and clariﬁcations, the interviewees

review the transcripts.

To code the interviews, we determine ﬁrst a list

of so-called a priori codes (Salda˜na, 2015) that are

https://www.draw.io

https://trello.com

http://www.qsrinternational.com/

Evaluating A Novel Agile Requirements Engineering Method: A Case Study

159

deduced from the research questions. Upon labelling

interviews with predeﬁned labels, we detect sentences

and blocks of text that do not ﬁt into existing codes.

The contents of these exceptions lead to the creation

of new so-called called grounded codes (Salda˜na,

2015).

To summarize coding, ﬁrst themes are introduced

(Salda˜na, 2015) for grouping codes. This axial coding

(Runeson et al., 2012) identiﬁes connections between

the themes and codes. Themes are also divided into

two sets, either based on relations to research ques-

tions (Section 3.1), or for not having any relation.

Additionally, we perform theoretical coding for

devising a set of attributes to codes for evaluating the

latter during analysis. The chosen attributes attached

to codes are polarity and type. Polarities show the

opinion of the interviewee and has three possible val-

ues, namely positive, neutral or negative. Types show

if a code and thus, the interviewee references an ex-

isting situation, or a need for change/addition. Type

has two values, namely statement or suggestion.

Polarity- and type values provide combinations

that describe whether a code proves, contradicts, or

has no relation to research. The ﬁrst combination of

six, negative suggestion, indicates a need to change

an existing part of case under investigation. Second,

positive suggestion is an additional idea, or improve-

ment put forward by the interviewee. Next, neutral

suggestion is not related to the AAOM method evalua-

tion, negative statement denotes a ﬂaw in the AAOM-

method while the interviewee has no improvement

suggestion. The ﬁnal two combinations are positive

statement for indicating a participant’s satisfaction

with AAOM and neutral statement for representing

an opinion, or a statement of affairs not related to the

research question.

We use a simple formula to evaluate which codes

have more value for analysis. Three components play

a role in determining code validity, namely ﬁrst ref-

erences that show how many times a code is men-

tioned in interviews. The second element is sources

denoting how many different interviewees mention

one code. Finally, role experience expresses an in-

terviewee’s experience in a role. Each component has

numeric values and a higher value increasing code va-

lidity. The formula to sort codes based on mentioned

components is as follows:

codevalue = (references∗ sources) + experience

3.5 Validity Procedure

To asses the validity of our research, we use criteria

proposed by A. K. Shenton (Shenton, 2004), namely

credibility, transferability, dependability and con-

ﬁrmability. To increase credibility, we employ the fol-

lowing strategies:

• We use a well established body of knowledge of

conducting case studies, mainly Runeson et al

(Runeson et al., 2012). To set up and receive

useful data from interviews, we use guidelines

by Robson (Robson, 2002) and for best-practices

to analyse gathered data, we consider Saldana

(Salda˜na, 2015).

• We conduct interviews with all participants in the

project covering all different roles. This is consid-

ered as a form of triangulation since we capture

viewpoints of all informants (Van Maanen, 1979).

Unfortunately, we are not able to triangulate via

other types of data sources, since there is a lack of

methods for analysing goal models.

• Before starting the l&f-app development, re-

searchers study AAOM as the unit of analysis and

also the context where it is applied, the ﬁeld of

ﬁnding and losing assets.

• To help ensure honesty in subjects, we inform

them that the data use is anonymous and that

their voice is recorded. All interviewees agree

with recording. The interviewees review the tran-

scribed documents to assure a valid transferral of

ideas.

• Three researchers participate in the case study

and provide constant peer reviewing to each other.

With that setup and constant debrieﬁng among

each other, the researchers’ vision is wider than

working alone.

• The researchers participating in the case study

have relevant related backgrounds. Thus, credi-

bility is assured by the extensive research experi-

ence.

The second criteria, transferability, can not be

demonstrated since the case covers only a speciﬁc

project and a speciﬁc set of individuals (Shenton,

2004). Still, the reporting context of this case study is

helpful for other researchers to compare their results

with ours (Shenton, 2004). To test transferability, we

conduct another case study with a similar setup to ver-

ify whether the same results reoccur. One more factor

assuring transferability is the fact that agile teams are

limited by deﬁnition to a size of 3 to 9 (Cockburn,

2006).

To address dependability, Shenton et al. (Shenton,

2004) recommend a case study report that includes

sections devoted to:

• the research design and its implementation

• the operational detail of data gathering

ENASE 2016 - 11th International Conference on Evaluation of Novel Software Approaches to Software Engineering

160

• reﬂective appraisal of the project

In this article, the research design is described in Sec-

tion 3 and data gathering details are depicted in Sec-

tion 3.3. We do not evaluate the effectiveness of pro-

cesses undertaken in the current case study. This will

be a evaluated by a future case study.

According to (Jensen, 2008), conﬁrmability is an

accurate means through which to verify the two basic

goals of qualitative research:

• to understand a phenomenon from the perspective

of the research participants and

• to understand the meanings people give to their

experiences

This research contains a threat to researcher bias be-

cause one researcher is the inventor of AAOM, a

method under investigation. We are aware of this

threat and avoid it with the same means as for credi-

bility - by providing detailed descriptions. According

to (Shenton, 2004), in a qualitative study, researchers

biases are inevitable. We also acknowledge the lim-

itation of one person acting in two roles possibly re-

moving one friction point between analyst and devel-

opers. This could have been mitigated by assigning

another person to that role for which we lack a sufﬁ-

cient project budget.

4 RESULTS

The main goal of the analysis is to understand whether

theories about the AAOM method are valid by ﬁnding

answers to questions speciﬁed in Section 3.1. We also

take into account any non-expected data found during

the interviews. All themes and codes discussed fur-

ther on, are presented in longer version of this arti-

cle

4.1 Beneﬁts of using AAOM From a

User Perspective (RQ1)

In order to ﬁnd answers to RQ1, we use ﬁve themes

that cover direct beneﬁts the participants state about

how AAOM improves communication by collabora-

tive modelling by including the participants. Addi-

tionally, a method comparison along with visual rep-

resentation provides insights to AAOM.

To evaluate results, we refer to a formula devised

in Section 3.4. The ﬁrst theme Beneﬁts shows all

codes are positive statements for using AAOM. All

the codes in this theme are positive statements and

adhere to the case-study research question about what

http://digi.lib.ttu.ee/i/?4064

the beneﬁts are of using AAOM. The highest ranked

codes are a secure feeling for the project direction,

mutual communication and discovering new angles in

requirements for the project. Four codes have a lower

evaluation score and thus, are more unreliable to draw

conﬁdent conclusions from.

Next, theme Collaborative Modelling relates to

the AAOM-theory of improving communication be-

tween clients and the development team by working

together on requirement elicitation. The highest rated

codes are all positive suggestions and thus, conﬁrm-

ing the expectations set by AAOM. Having everyone

“on the same page”, improves understandability and

pinpointing problems represents a beneﬁt for collab-

orative modelling. There is one negative suggestion

about composing goal models should be more struc-

tured.

The Method Comparison theme gathers the par-

ticipants’ experience with similar methods compared

to AAOM. Unfortunately, the experience with simi-

lar methods is low among participants in the running

case study. Consequently, a comparison with other

methods is not sufﬁcient and does not provide enough

results.

The ﬁnal two themes have less codes than Beneﬁts

and Collaborative Modelling while the codes have a

high formula value that shows same opinions from all

participants. The Participation theme gathers objec-

tive opinions about how the method includes everyone

in the project and it shows the ICT-expertise is an ad-

vantage. The theme Visual Representation consists of

only one strongly referenced code denoting that the

goal-model representation is a deﬁnite beneﬁt.

4.2 Effects of the Projects Setup and

Tooling on AAOM (RQ2)

Three themes in Section 3.1 cover RQ2 starting with

explaining the setup of elaboration sessions and ﬁnd-

ing the effects for the AAOM-method application.

The next theme is related to temporal measures as

well, explaining time usage to manage AAOM mod-

els. The ﬁnal theme covers effects of software based

tooling usage on AAOM application.

Which practices are favoured and which need im-

provement, are ranked again by formula results (Sec-

tion 3.4). The Elaboration Sessions theme covers

the AAOM-method’s application-session content, -

duration and -suitability. The highest ranked code

suggests the selected session length that is 1.5 hours,

is selected correctly even though one low ranked con-

tracting code is gathered to have shorter sessions. The

remaining codes are ranked relatively low with incon-

clusive ﬁndings.

Evaluating A Novel Agile Requirements Engineering Method: A Case Study

161

The theme Modelling Time Usage answers to

questions related to spent time on modelling activities

carried out using the AAOM method. Positive state-

ments are gathered for system requirements fast cap-

ture, fast development based on models and overall

effective time usage. One neutral statement is added

that moderate time is spent until an idea is formed

as a user story. The unit of measurement is the par-

ticipants’ subjective feelings about the time spent on

method activities.

In order to answer aspects of tool importance in

using AAOM, we ﬁnd a theme Tools Usage. There are

several different codes revealing several viewpoints.

The two highest ranked codes show that using freely

available tools is provisionally satisfactory while hav-

ing an integrated suit would reduce the amount of

work needed for completing tasks. Available com-

mercial tools are considered better while they are pro-

hibitively expensive, especially in smaller projects.

We conclude interest exists to use new tools that are

better tailored than the chosen free tools in the run-

ning case.

4.3 Further AAOM-reﬁnement Meeds

(RQ3)

For further improvementswe collect codes under only

one theme called Method Clariﬁcation that addresses

what practices of AAOM are clear and which need ex-

planation, or redeﬁning. On the positive side, we ﬁnd

that the sequence of activities for goal models com-

posing is clear. In top ranked codes we also ﬁnd pos-

itive statements being clear about concepts of quality

goals, user stories and roles.

On the negative side, we gather contradicting in-

formation if creating user stories for the lowest level

goals is clear. Both codes are highly ranked while

statements about the process being unclear are ranked

slightly higher. Studying this contradiction is future

research work in addition to the exact usage of qual-

ity goals in relation to user stories that is currently un-

clear. As a counterpart for top ranked positive state-

ments, there are few contradicting negativestatements

about same aspects. Analysing these contradictions

deeper, we ﬁnd that a lack of experience causes these

results.

4.4 Emerged Results

The emerged codes result from grounded coding that

is explained in Section 3.4. The participants ex-

press their feelings in the theme Drawbacks about the

project setup that is not directly related to AAOM

modelling while it still affect its usage. The most

mentioned code states that experienced participants

are needed to fully beneﬁt from the AAOM-method.

Thus, AAOM is not intuitively fully understandable

to all participants in requirements engineering. To

aid understandability, a solution could be a better user

guide for the AAOM method, as suggested by inter-

viewees in the second top mentioned code. Further-

more, the analyst is identiﬁed as carrying a vital role

in AAOM usage by having the biggest responsibility

in modelling the activities. As a ﬁnal statement worth

noting a doubt exists pertaining to the method suit-

ability in smaller projects due to a possible method

overhead.

The Expectations theme shows the participants’

positive impression. Working results and extensible

implementations are expected, goal models and user

stories are expected to be updated. A negative point

is that more participants are needed to accomplish the

goals pertaining to the project setup and not the im-

plementation phase of AAOM.

The results captured under the theme Modelling

Suitability support the theory about the method help-

ing to focus on objectives elicitation and organizing

thoughts to express a client’s feelings. Statements

gathered under the theme New Ideas give novel sug-

gestions stemming from the study participants such

as assigning ﬁnancial values to the goal, to use goal

models as a system documentation, and so on.

5 CONCLUSION

In this article, we evaluate the AAOM-method as

a novel method for requirements engineering in a

small-scale project that employs agile software de-

velopment for developing a mobile app from scratch.

A case-study based research approach is instrumen-

tal for an evaluation. Interviews with project partici-

pants provide the most relevant input for the AAOM-

method’s evaluation. Interviews are coded and ana-

lyzed to answers research questions, also new knowl-

edge outside the scope of research questions we gen-

erate.

The beneﬁts of using the AAOM-method are a

secure feeling for project direction, mutual commu-

nication and discovering new angles in requirements

for the project. The visual representation provided by

goal models is intuitively comprehensible. Further-

more, the AAOM-method enforces communication

via collaborative modeling, improving understand-

ability, pinpointing problems and involving partici-

pants. Time spent on AAOM activities for gathering

requirementsis found to be adequate and the overhead

during that process is marginally low. For projects

ENASE 2016 - 11th International Conference on Evaluation of Novel Software Approaches to Software Engineering

162

on a small budget, free tools with manual integration

sufﬁce while there is a desire for AAOM-tailored in-

tegrated tool support. Unclear procedures, that need

better guidance or redeﬁning, pertain to the way of

ﬁnding the lowest level of goal models with the cor-

responding deducing of user stories, and quality-goal

use. One of the discovered factors outside the scope

of research questions is that before using AAOM ef-

fectively, longer experience in ICT is needed or addi-

tional guidance should be available. Finally, the ana-

lyst role has been identiﬁed as too heavy in applying

the AAOM method.

The ﬁrst limitation is that the collected informa-

tion is mostly based on the client-side opinions be-

cause three participants out of four ﬁll the client role.

Only two of the four participants have experience par-

ticipating in software developmentprocesses and con-

sequently, feedback also stems from inexperienced

participants. Another limitation is a lack of inter-

view feedback with respect to other methods for agile

and requirements/engineering methods. This can re-

veal how well the AAOM-method is combinable with

other agile methods. A ﬁnal limitation is related to re-

search bias towards supporting factors of AAOM us-

age since one researcher is an inventor of AAOM.

Further studies and investigation must apply the

AAOM-method in diverse domains for demonstrat-

ing its universal applicability. Based on the feedback

from participants of the runningcase, an improvement

of the AAOM-method must include better explana-

tion of the role of quality goals. Since the analyst

plays an important role during applying the AAOM-

method, we need a deeper understanding of this es-

sential role.

REFERENCES

Ambler, S. W. and Lines, M. (2012). Disciplined agile de-

livery: A practitioner’s guide to agile software deliv-

ery in the enterprise. IBM Press.

Beck, K. (2000). Extreme programming explained: em-

brace change. Addison-Wesley.

Beck, K. et al. (2001). The agile manifesto.

Cao, L. and Ramesh, B. (2008). Agile requirements engi-

neering practices: An empirical study. IEEE Software,

25(1):60–67.

Carlson, D. and Matuzic, P. (2010). Practical Agile Re-

quirements Engineering. Technical report.

Cockburn, A. (2006). Agile software development: the co-

operative game. Pearson Education.

Cohn, M. (2004). User Stories Applied: For Agile Software

Development. Addison-Wesley.

Hull, E. et al. (2010). Requirements engineering. Springer.

Jensen, D. (2008). Conﬁrmability, page 113. Sage.

Kniberg, H. and Skarin, M. (2010). Kanban and Scrum-

making the most of both. Lulu.com.

Larman, C. and Vodde, B. (2008). Scaling lean & agile

development: thinking and organizational tools for

large-scale Scrum. Pearson Education.

Lefﬁngwell, D. (2010). Agile software requirements: lean

requirements practices for teams, programs, and the

enterprise. Addison-Wesley.

Lefﬁngwell, D. (2016). Scaled agile framework.

http://scaledagileframework.com.

Lefﬁngwell, D. and Aalto, J. (2009). A lean and scalable re-

quirements information model for the agile enterprise.

https://scalingsoftwareagility.wordpress.com/.

Poppendieck, M. and Poppendieck, T. (2007). Implement-

ing lean software development: from concept to cash.

Pearson Education.

Robson, C. (2002). Real word research. Blackwell.

Runeson, P., Host, M., Rainer, A., and Regnell, B. (2012).

Case study research in software engineering: Guide-

lines and examples. Wiley.

Salda˜na, J. (2015). The coding manual for qualitative re-

searchers. Sage.

Schwaber, K. (2004). Agile project management with

Scrum. Microsoft Press.

Shenton, A. K. (2004). Strategies for ensuring trustwor-

thiness in qualitative research projects. Education for

Information, 22(2):63–75.

Sterling, L. and Taveter, K. (2009). The art of agent-

oriented modeling. MIT Press.

Sutherland, J. et al. (2007). Distributed scrum: Agile project

management with outsourced development teams. In

System Sciences. IEEE.

Tenso, T. and Taveter, K. (2013). Requirements engineering

with agent-oriented models. In Evaluation of Novel

Approaches to Software Engineering, pages 254–259.

Van Lamsweerde, A. et al. (2009). Requirements engineer-

ing: from system goals to UML models to software

speciﬁcations. Wiley.

Van Maanen, J. (1979). The fact of ﬁction in organizational

ethnography. Administrative Science Quarterly, pages

539–550.

Version One, I. (2016). 9th annual state of ag-

ile survey. http://info.versionone.com/state-of-agile-

development-survey-ninth.html.

Yin, R. K. (2013). Case study research: Design and meth-

ods. Sage.

Yu, E. S. (2009). Social modeling and i*. In Conceptual

Modeling: Foundations and Applications, pages 99–

121. Springer.

Evaluating A Novel Agile Requirements Engineering Method: A Case Study

163