Understanding Software Testers in the Automotive Industry
A Mixed-method Case Study
Tabata Pérez Rentería y Hernández
and Nicola Marsden
IT Department, Heilbronn University, Max-Planck-Straße 39, 74081 Heilbronn, Germany
Keywords: Software Testing, Mixed-Method Study, Human Aspects.
Abstract: This paper presents the results of a mixed-method study performed in the software department of a large
automotive supplier operating in a global software engineering setting. The aim was to understand the social
dimension and human aspects involved in software testing in an intercultural setting. Qualitative and quanti-
tative analyses of testers' perception regarding their day-to-day activities and collaboration with other teams
was conducted. The findings suggest the testing team is motivated but recurrently affected by external fac-
tors such as late input for testers, improperly or missing requirements, and unrealistic project planning.
Testers identified human factors, such as openness and attitude of people, as relevant for effective collabora-
tion. Combining the findings of the quantitative and the qualitative studies, our research suggests that the
approach that testers take to their work can be characterized by a silo focus, i.e. rather than focusing on the
overall goals their perception revolves around their subunit of the organization.
1 INTRODUCTION
With the growing complexity required for producing
large software systems, software testing has been
acknowledged by the industry to play a critical role
in the development of useful software systems (De-
ak, 2012). Therefore, a considerable amount of
effort in the development of any large and useful
system is invested in testing (Martin, Rooksby,
Rouncefield, and Summerville, 2008). Nevertheless,
the current testing practices rely on experience
gained in the field rather than on a rigorous and
systematic approach (Engström and Runeson, 2010).
The scope of testing has broadened: instead of
testing a single piece of software, nowadays, either
an entire system or a system of systems is often
tested. Hence, testing has undergone several
transformations. For example, it has increasingly
become a professional team activity and is being
made accountable throughout the entire
organization. Technology transfer from research to
the work performed in the industry has demanded
the acquisition of new skills and consequently, a
restructuring in the current practices (Martin et al.,
2008).
Additional to the technical challenges, testing
faces difficulties on the human and organizational
levels, e.g. regarding the social perception of
software testers within the organization or the
technical impact of their role in the software
development process (Whittaker, 2000; Deak 2012).
Nonetheless, software testing has been increasingly
recognized as a key activity in the production of
software systems, the social perception of software
testers has not been equally acknowledged.
2 RELATED RESEARCH
The study presented in this paper is based on re-
search and findings regarding human aspects of
software testing, tester-developer relationships,
testing as a profession, and the specifics of software
in the automotive setting. The following sections
give a brief overview of the related research in these
areas.
2.1 Human Aspects of Software
Testing
Albeit research has strongly focused on studying the
technical components of software testing, there have
been prior efforts on studying the human factors
related to testing as well as a mean to better under-
stand software-engineering processes. By using
305
Pérez Rentería y Hernández T. and Marsden N..
Understanding Software Testers in the Automotive Industry - A Mixed-method Case Study.
DOI: 10.5220/0004992503050314
In Proceedings of the 9th International Conference on Software Engineering and Applications (ICSOFT-EA-2014), pages 305-314
ISBN: 978-989-758-036-9
Copyright
c
2014 SCITEPRESS (Science and Technology Publications, Lda.)
qualitative interviews, observations, etc. social con-
structions and the inner view of social subjects on
their environment becomes accessible. Shah and
Harrold (2013) performed an ethnographic study in a
large software company in India to understand the
human dimensions that influence software testing.
Shah and Harrold concluded that junior testers,
when mentored well, can develop a positive attitude
towards software testing and do not view it as a
secondary activity. They found that participants
view testing as a monotonous activity – especially
manual testing. Nonetheless, a factor that enhanced
motivation, despite the monotonous nature of the
activity, was the feeling of responsibility and owner-
ship. Furthermore, a relationship between responsi-
bility and power suggested higher enthusiasm in
participants over those who had responsibility with
no power. Likewise, studies conducted to software
developers found that effective leadership, a reason-
able level of responsibility, a positive working envi-
ronment, and a sense of being involved were factors
influencing their motivation (McLeod and Mac-
Donell, 2011). Both the organizational context as
well as the composition and the culture of the soft-
ware project team were equally found as factors
influencing motivation (Oz and Sosik, 2000).
2.2 Tester-developer Relationship
Even though software testing is a technical sub-
discipline of software engineering, where tools and
automated methods are used, it is branch that de-
pends on the interpersonal interactions of the people
involved in the production of software. It has been
found that conflict between testers and developers is
inevitable and management has to ensure conflicts
are dealt with constructively (Cohen et al., 2004).
But where does this conflict stem from? According
to the perception of a tester, the tester mindset is
completely opposite to the developer’s since the
former has to break code while the latter creates it
(Shah, Nersessian, Harrold, and Newstetter, 2012).
According to Cohen et al, while developers create
software “the way it should work”, testers develop
tests not only considering a software engineering
perspective but also taking into account an under-
standing from the business and from real-life use
cases.
Moreover, frequent conflicts between testers and
developers arise when they do not share common
work goals (Cohen et al., 2004). To align the
different mindsets, individual goals should be
connected to process metrics. Greater emphasis on
shared team goals rather than individual ones and
subsequent reward of the former, motivates people
to work together (Patel, Pettitt, and Wilson, 2012).
Shared knowledge and awareness should be
promoted for effective collaboration as well as for a
better understanding of the roles, responsibilities,
and skills of others. Collaborative awareness is
influenced by, and has an influence on,
communication, coordination, and organizational
culture: the shared attitudes, beliefs, and values that
emerged from the organization’s vision and
objectives and which influence employee behavior
(Patel, Pettitt, and Wilson, 2012). Past experience:
how successfully developer and tester teams have
worked together in the past should also be taken into
account as a factor influencing successful
collaboration (Patel, Pettitt, and Wilson, 2012).
Also, structured work activities to promote tester-
developer acquaintance, as well as testers’ early and
consistent involvement in the requirements are
recommended.
The tester-developer relationship is important
not only because it is related to job satisfaction of
the involved parties but also because it influences
the quality of the software produced. Pairing one
tester to one developer leads to positive changes as
higher job satisfaction for the people involved
(Zhang et al., 2010).
2.3 Testing as a Profession
Additional to the conflicts identified on the tester-
developer interactions, there are other challenges
that testers often face within their own organizations
and with the perception of testing as a profession.
For example, one of the key problems testers often
experience are the shortened times for testing. Often
development phases overrun into the time allocated
for testing. (Martin et al., 2008)
From the research performed to 127 software pro-
fessionals to discover the factors with negative in-
fluence on software testing practices, reduced time
for testing was also identified (Fernández-Sanz,
Villalba, Hilera, and Lacuesta, 2009). The shortened
time for testing stems from delays of the previous
development phases, the common practice to per-
form testing at the end of the project and only once
the code is available, the impossibility to postpone
the delivery to the customer, or due to financial
problems that arise along the project (Fernández-
Sanz et al., 2009).
Lack of specific testing training both while
attending University and as an IT professional were
highlighted as negative factors for the software
practice. Moreover lack of education on software
ICSOFT-EA2014-9thInternationalConferenceonSoftwareEngineeringandApplications
306
testing of other stakeholders also poses a negative
impact. For example, many managers did not
obtained a good training on software testing; thus,
do not value its potential for efficiency and quality.
On the academic arena, few teachers have in-depth
knowledge of the correct philosophy and techniques
for testing. To understand the software practice in
Canada, Garousi and Zhi (2013) surveyed software
professionals –including Project Managers, Software
Testers, Business and Systems Analysts and QA
Specialists and Leads. They wanted to know if
people had received testing training programs
(including university or on-site training courses, but
excluding self-study hours based on online resources
or books). From the 206 respondents, more than half
received at least 20 hours of formal training of
testing in a year -which showed an increasing
awareness of the importance of software testing
training. Nonetheless, 39% of the respondents
mentioned that they received no training on testing
before the survey was performed. Cost and time as
well as lack of support from high-level management
were identified as obstacles for not providing
training (Garousi and Zhi, 2013). Earlier research
from Cohen et al. (2004) identified that the lack of
support and status that testers face also make their
job more difficult and time consuming.
Regarding the perception of testing as a
profession, Deak (2012) assessed both students and
professionals. Her findings showed that students had
a negative attitude towards testing as a career:
testing translated into correcting other people's work
and being a failed and denominated developer (De-
ak, 2012). Testing is also viewed as an activity
which needs less qualification than other sub-
disciplines in the production of software systems
(Fernández-Sanz et al., 2009).
To comprehend the complexity of software
testing on its own, Taipale and Smolander conducted
a qualitative research in 26 organizations of high
technical level producing real-time software with
above than average criticality. From their findings,
they suggested several improvements and stressed
the importance of the early involvement of testing
and and planning of testing so testers could give
their contributions earlier (Taipale and Smolander,
2006). Their recommendations depict the
importance of equally considering software testers
as part of the software development lifecycle,
promoting effective interaction and communication
between testers and developers.
Another challenge software testers face is the
need to find solutions to ill-defined problems and
shape their work to changing organizational
demands (Martin et al., 2008). One of these
organizational demands is the involvement of the
customer in the decision-making process regarding
testing (Shah and Harrold, 2013).
2.4 Software in the Automotive
Setting
Especially in the automotive industry, all these chal-
lenges have had vast impact on the development of
software systems (Broy, 2006). Software has pro-
vided more and more functionality to the car; yet,
software development had to fit into a predominant-
ly vertically-organized automotive industry where
historically, mechanical engineers strove to build
cars with a set of independent sub-systems. This
independence allowed car manufacturers to out-
source production to third-party suppliers. In his
study about software engineering challenges on the
automotive industry, Broy found that requirements
management is a major challenge between the car
manufacturers and the suppliers. In many cases the
requirements, which are developed by the car manu-
facturer, are often not complete or precise enough
for the supplier to implement. Additionally, the
complexity of the solutions needed and the hard-
ware-dependent implementations hinder the reuse of
code from one car generation to the other. Even
though models of software engineering are progres-
sively influencing the mechanical engineering in the
automotive field, processes are still not adapted to
the software and intensive systems development
demands.
From the findings from Fernández et al. (2009)
regarding shortened testing phases due to committed
delivery dates to the customer, this constraint is
particularly strict in the automotive industry. All the
suppliers of a car manufacturer, such as the company
in our research, are responsible for delivering their
component to the car manufacture on-time so that
the end goal of assembling and producing the car is
achieved as planned. Thus, since testing is the last
phase the supplier has before it signs-off to
production, delays in development cut time for
testing with no opportunity for extension.
3 THE RESEARCH PROBLEM
Current research suggests that software testing –
especially in the automotive sector– is a complex
task, not only regarding the technical skills required,
but also regarding human and social aspects and the
personal ability of dealing with complexity.
UnderstandingSoftwareTestersintheAutomotiveIndustry-AMixed-methodCaseStudy
307
The research presented in this paper aims to
understand the situation of testers in an automotive
setting. While there has been research addressing the
social and human factors that affect software testers
in other contexts, this study takes a closer look at the
challenges that a team of senior software and system
testers, working in a matrix-organization, deal with
in the extremely time-constrained and safety-critical
automotive industry. We focused on highly qualified
testers, all having a university degree – thus to pre-
empt the view of testing as a profession that is done
by failed and denominated developers or as an
activity which needs less qualification (Deak, 2012;
Fernández-Sanz et al, 2009).
The present study was triggered by a request of
the managers of a software testing team in the
automotive industry. The concerns and perceptions
of the testing managers pointed to social problems
between testers and other stakeholders. Considering
for this research as well, we decided to analyze the
problem from the testers’ point of view. We aimed
to understand how testers conceptualize their most
recurrent problems and their collaboration with other
departments. Thus, this research is guided by the
following questions:
RQ1: What are the most recurrent problems
that testers face in their day-to-day activities?
RQ2: How do they collaborate with other
teams?
4 STUDY METHOD
In this section we present the details of our study by
first outlining the organizational context in which
this study takes place, then describing the partici-
pants, and finally presenting the mixed-method ap-
proach taken in this research.
4.1 Organizational Context
The testing group being examined in this study is
part of the automotive passive safety business unit of
a large multinational engineering company. The goal
of this testing group is to perform black box tests on
both software and system levels. Specifically for the
tests done on the system level, implicit hardware
tests are carried out since software and hardware
integration is needed for performing them. This is
relevant from the organizational perspective because
it implies the testing team not only interacts with
software stakeholders but also with other depart-
ments such as technical project management – with-
in the hardware team – and crash detection algo-
rithm development teams.
The organizational structure in place is a func-
tional matrix: employees are full members of a func-
tional department and project managers are limited
to coordinate their efforts while participating in a
project. Thus, testers are part of a (functional) test-
ing team and report to their testing line manager but
are also part of a particular customer project where
they also report to the project manager. The project
manager is responsible for the execution of the pro-
ject from beginning to end and receives the re-
sources allocated from each functional or line man-
ager.
Regarding communication: telephone as well as
computer-mediated communication tools (e.g.,
email, web conferencing, and enterprise social net-
work software) are available as means to interact
with people throughout the entire organization. In-
teraction with the customer or car manufacturer
occurs via telephone, email, or face-to-face meet-
ings. This interaction is conducted by a single per-
son: usually the project manager. The requirements
management process is tracked with a requirements
management tool that connects both the customer
and the supplier.
The supplier develops and manufactures safety-
critical software for the electronic control unit of
automobiles. Specialists from hardware, software,
and crash detection algorithms work in the develop-
ment of the product throughout its different phases.
Although generic software and hardware platforms
are used to leverage the development of specific
customers’ requests, customized development is
further needed.
Due to the real-time and safety-critical nature of
the systems, they have to undergo rigorous tests and
re-validation including requirements validation re-
views, design and code reviews as well as compo-
nent and integration testing. Testers can take several
weeks to develop the test specification, based on the
requirements, execute the test cases, and finally find
and log defects for a given feature.
4.2 Participants
During the five-month study period, there were
numerous informal interactions with testers, test
leads, and managers in the department (e.g., during
lunch, meetings, talks, and workshops). The partici-
pants were members of a testing team co-located in
Germany and India. For the study presented here, 15
members located in Germany participated.
11 testers
ICSOFT-EA2014-9thInternationalConferenceonSoftwareEngineeringandApplications
308
4 managers (either team leaders, line manag-
ers, or department managers)
For the first part of the study, 6 testers were in-
terviewed using a semi-structured interview and the
4 managers were interviewed using diverse custom-
ized interviews. For the second part of study, we
surveyed 11 testers –6 of them had already been
interviewed on the first part of the research.
The testing group in Germany has 15 members,
one of them is the section manager (line manager),
and 2 are team leads. From the 12 testers in the
team, one person deliberately declined to participate
in the study for privacy reasons so we studied the
remaining 11 testers. These are developing and run-
ning test cases, remotely coordinating other testers
working in India, or doing test tool development.
Regarding their experience, 14 of them are seniors
ranging from 7–25 years of experience in the com-
pany. Just one of them has one year of experience –
all of which was completed in this company. All of
the group members speak German as their native
language and English as a second language. Their
age ranges from 30 to 50 years old; there are 1 fe-
male and 14 males. They all hold a university degree
in electrical engineering, physics or computer sci-
ence.
The German group collaborates with an Indian
group of testers remotely. The German members
play a leading role and the Indian team a supportive
one. The Indian testing group, which is not included
in this study, has 69 members, one of them is the
Section Manager, and 5 are team leads. There are 63
testers that are developing and running test cases.
4.3 Procedure
To gain a deeper understanding of the reality lived
and perceived by the testers, an ethnographically-
informed study was performed. This study intended
to discover the opinion of the studied group and did
not aim to generalize or provide a statistically valid
view of a certain population. Therefore the sample
size was determined, as frequently done in qualita-
tive studies, in a non-probabilistic fashion (Guest,
Bunce, and Johnson, 2006). Saturation, or the point
at which no new information or themes are observed
in the data, was used to determine the sample size.
The procedure adopted for conducting this study
was exploratory sequential design, a mixed methods
approach, using a combination of qualitative and
quantitative methods (Creswell and Clark, 2007).
After the collection and analysis of qualitative data,
a second quantitative step – built from the explorato-
ry results – to test or generalize the findings from the
first qualitative research was taken. The key purpose
of this two-phase sequential design is to triangulate
the data. By involving more than one method to
gather data, an alternative perspective allows to
validate, extend, or challenge the initial findings
(Turner and Turner, 2009).
The following sections describe the four phases
completed in this study as part of a mixed-method
exploratory sequential design: qualitative data col-
lection, qualitative data analysis, quantitative data
collection, and quantitative data analysis.
4.3.1 Qualitative Data Collection
In exploratory sequential design, the first step per-
formed is a qualitative study. The qualitative data
collection tools in this study consisted of a combina-
tion of informal conversations, interviews, and doc-
ument analysis. The first couple of months were the
phase of immersion: accompanying the day-to-day
activities of the testers, we learned as much as pos-
sible about their daily operations, their processes,
their interactions, and the organization itself. Based
on the analysis of the details of the observations,
interview questions were developed. The interviews
performed were semi-structured, this means that a
set of similar questions were asked to the partici-
pants, following an interview guide (Bernard and
Ryan, 2010). With this guide, a set of basic ques-
tions were asked to all participants but depending on
how the interview progressed, more detailed ques-
tions were asked to gain more insight in a topic of
interest. The guide (presented in Appendix A) co-
vers questions about testers’ most recurrent prob-
lems and collaboration, as well questions on aca-
demic background, qualifications, and their profes-
sion’s self-perception.
Participants were approached via email. At the
meeting, before the interview started, the partici-
pants were informed once again of the goal of the
talk and reminded that the information shared would
be kept anonymous. Throughout the interview, con-
tinual paraphrasing of the participants’ answers was
done to confirm that the researcher had correctly
understood what the interviewee aimed to communi-
cate. The participant was requested to give acknowl-
edgment and feedback. During the interviews, the
answers were typewritten in a simple text file. Due
to the company's privacy policy the interviews were
not recorded. The interviews lasted from 80 up to
120 minutes. Once an interview was over, the an-
swers were reviewed. If needed, to confirm our
understanding and findings, we conducted follow-up
UnderstandingSoftwareTestersintheAutomotiveIndustry-AMixed-methodCaseStudy
309
meetings or emailed participants and asked for feed-
back or clarification.
In general, the participants showed an open and
cooperative attitude in the interview. Even though
the interviews were scheduled for one hour, in all
cases the interviews extended, mainly due to the
participants’ interest in talking about their vast expe-
rience.
4.3.2 Qualitative Data Analysis
For analyzing and putting together all the collected
data, each interview was re-read several times –
through the analysis phase – in order to identify
patterns that could allow the data to be grouped in
meaningful ways (Denzin and Lincoln, 2005). Cate-
gories and sub-categories emerged around the first
four interviews. An important fact is that most of the
interviewees have been working for a significantly
long period of time in the company and in some
cases only in this team since they joined. In spite of
starting in the company at distinct moments and
having different backgrounds, the collective opinion
was relatively similar. The number of times the
categories and subcategories appeared on the inter-
views was counted. Similar categories and subcate-
gories were stacked together in order to create a
more meaningful, summarized, and concise system
of categorization. In order to visualize all this infor-
mation, a mental map with relationships interpreted
from the data was built from the findings. This men-
tal map was built in two steps. The first consisted on
summarizing the categories and subcategories in a
spreadsheet. The second one was building a graph,
using the DGML (Directed Graph Markup Lan-
guage) specification, taking as input the categories
and subcategories from the first step. The categories
that were mentioned by most of the participants
combined with the most meaningful topics from the
interview were the ones chosen as input for the sur-
vey.
4.3.3 Quantitative Data Collection
In line with the exploratory sequential design the
collection and analysis of the qualitative data were
followed with quantitative methods to build on the
qualitative findings (Creswell and Clark, 2007). For
the quantitative step, a questionnaire with five-point
Likert-scaled answering options was created, rang-
ing from strongly disagree to strongly agree. The
questions of the online questionnaire matched the
questions performed in the face-to-face interview
performed in phase 1 of this study. Whereas the
Likert items in the online survey corresponded to the
highest mentioned and most relevant topics catego-
ries/subcategories identified as answers from the
qualitative study. In contrast to the open-ended ques-
tions from phase 1, the answer domain on the survey
was constrained to the selected categories and sub-
categories.
The identified categories included motivation,
training provided (by the company), recurrent prob-
lems, collaboration on the same team and with other
departments, and perception of processes and their
adherence.
The online survey was posted through the intra-
net of the company. An email was sent to all partici-
pants and a period of five days was given for people
to answer it. The survey was anonymous and all
testers – except for one who explicitly declined to
participate – answered it. Thus, 11 of the 12 testers
participated in the survey.
4.3.4 Quantitative Data Analysis
The results from the survey were analyzed by fre-
quencies. Since the goal of the quantification was to
receive an overview regarding the agreement to the
items identified and proposed with a five-point Lik-
ert scale, the two selections of “agree” and “strong-
ly” agree were taken together as markers of agree-
ment regarding the item. To see the summary of the
identified categories and subcategories as well as the
percentage of agreement versus disagreement, please
refer to Appendix B.
5 FINDINGS
The findings presented in this section are from both
the qualitative and the quantitative studies. We de-
scribe the most important categories or themes we
obtained from the qualitative study as well as the
overall agreement regarding these topics gathered in
the survey from the quantitative study.
Motivation was a key theme to understand since
it would depict how testers felt about being testers.
Both in the qualitative and the quantitative studies,
testers agreed on feeling satisfied with their job.
They have a positive attitude towards software test-
ing and do not consider it a boring and monotonous
activity. 91 % of the participants described testing as
an interesting job that allowed them to work on a
broad variety of topics. One participant explained
that “Testing is a very interesting job because it
requires a very complete view of the software. If you
have a broad view then you can do a whole range of
tests: with a bigger overview a better test variety.”
ICSOFT-EA2014-9thInternationalConferenceonSoftwareEngineeringandApplications
310
Another participant added,“It's very interesting be-
cause of the variations. You cover many different
subjects and you keep learning.”
91% of the testers agreed on having responsibil-
ity and ownership to do their job – which previous
research acknowledged as factors for enhanced mo-
tivation and enthusiasm among testers. Additionally,
91 % of testers shared that they had enough skill to
perform their job and the same percentage said they
enjoyed working with their testing team; 73 % of the
testers felt satisfied as a tester. “I feel happy to be a
tester […] Support is really good and then it is easy
to feel comfortable working with this team.”
One tester shared how he felt connected to
her/his job: “I was once in a meeting with higher
management, and they said that the work of each of
us had helped to save in average thirty lives. I felt
really connected to my job and why it really mat-
tered.”
To sustain the fact that this testing team is tech-
nically qualified, we learned from their line manag-
er, that almost all testers hold a Tester Certification
from the ISTQB. Therefore, according to our find-
ings, the testing team is highly qualified and moti-
vated to test the electronic control unit software.
So in line with findings in motivational psychol-
ogy (Deci and Ryan, 2012), our findings regarding
motivation show the importance of intrinsic motiva-
tion – the participants are able to relate their testing
activities to individual experiences of autonomy,
competence and relatedness and thus experience
high quality forms of motivation and engagement.
Regarding the training the company had provid-
ed to the testers, 91 % agreed on receiving some
kind of training since they joined the company. Most
of the testers shared that the training was of tech-
nical nature: 82 % received either the foundations
for understanding concepts related to the product
(Electronic Control Unit, hardware and software or
sensors) and 73 % received even more specific tech-
nical training to perform their daily job. When asked
about the training, one of the participants who had
worked in another department before (within the
same business unit), said: “The trainings [I received]
were very specific and role-oriented. When I
changed from role 1 to role 2, my view on the organ-
ization changed and I started to understand which
were the pain points of each role and why were they
colliding. I think that roles are focused on a very
narrow and specific scope. So I think, it strongly
depends on how managers are leading by goals and
that these goals are not only looking at the role's
self-optimization.” According to this participant in
one meeting he once argued about task responsibility
and the answer from the colleague was: “Are you
sure this is part of my role description to do?” An-
other participant added: “Sometimes I had to con-
vince the group leader of another person to allow
him to work with me. Seems we have different goals
between groups and that's why the members of a
team seek mostly their own.”
Regarding cross-functional training within the
department that would bring awareness to the im-
portance of shared goals we found that only 36 %
had received training that allowed them to under-
stand how each of the departments contributed to the
development of the product; 45 % of testers had
receiving organizational training or information on
how the company worked. When asked about their
role scope, one tester said: “The role is clear: to find
and log defects.” Later when we asked this partici-
pant about the overall goal of quality to the product
he asserted: “Yes, the overall goal is to close as
many bugs as possible. In the past we [testers] intro-
duced scripts to help the developers but this didn’t
work because there was no time and not enough
resources.”
Testers mentioned that the start of production is a
date agreed between the suppliers and the customer
that is virtually immovable. This is a constraint in
the automotive industry that specifically affects
testers: Since they are the last step before the prod-
uct is signed-off to production, delays from previous
phases reduce their time to perform testing.
Among the most recurrent problems they face,
82 % of the testers mentioned late inputs needed for
testing (software, hardware, or requirements), 73 %
of testers pointed to improperly specified require-
ments, 73 % to unrealistic project planning and es-
timation and 55 % mentioned reduced time to per-
form testing.
Regarding properly specified requirements, one
participant shared: “I think writing good require-
ments is difficult. But in one project we got really
good requirements when the customer reviewed
them directly, the results were better.”
Another participant said: “The problem is that
sometimes the OEM (customer) had the require-
ments and they were making changes which were
not easy to track on our side.”
For these four recurrent problems, their comple-
mentary percentages were all acknowledged, so
there was an overall agreement with them. With less
salience than the previous four factors, collaboration
with other departments was identified by 45 % of the
testers, whereas 36 % did not see it as a recurrent
problem.
UnderstandingSoftwareTestersintheAutomotiveIndustry-AMixed-methodCaseStudy
311
On the interviews participants related the four
acknowledged recurrent problems to the lack of
understanding of shared goals and to the people‘s
openness and attitude. Nevertheless, when we asked
on the survey whether they found collaboration a
recurrent problem the answers were mixed. Our
interpretation is that the team knows how to do its
job; yet exhibited one of the major pitfalls faced on
matrix organizations: a silo-focused behavior (Sy
and D’Annunzio, 2005). The data can be seen to
suggest that the shortage of training provided to
learn about how other departments work and lack of
communication or training regarding shared goals
contributes to the focus on the specific subunit of the
organization rather than a common goal. Moreover,
we speculate that this behavior is present in other
teams and departments as well, thus aggravating the
collaboration problem.
Regarding collaboration, when we asked the par-
ticipants what factors effective collaboration with
other departments was based on, two main human
factors were identified as most relevant: openness
and attitude of the people and the understanding of
shared goals and needs with 91 % and 82 % of
agreement respectively. Next came priorities as well
as schedule and frequency of communication with
73 % each. During the interviews, one tester shared:
“Well yes, in extremely critical or urgent cases, I
will make sure that things get done, I will contact
whoever is necessary to achieve the goal.” Another
participant added: “Usually because tasks are split
[with specialized roles] it is difficult to bring people
together. If the situation is drastic and it is close to
time of delivery then the project manager ranks the
importance as higher since delivery is at risk. There
seems to be better collaboration when problems are
evident.” 36 % agreed that collaboration improves
when a task has been escalated; 45 % of the testers
agreed that effective collaboration depends on how it
is put into practice in the project.
So the data regarding recurrent problems and
collaboration seem to hint back at the centrality of
the silo focus to understand how the testers are per-
ceiving their work: Collaboration with other depart-
ments is not seen as a major issue due to the focus
on their own team of testers. The fact that escalation
is seen as a possible means to improve collaboration
can be interpreted as reinforcing the mindset that it
is better to simply focus on the team’s goals – and if
that does not work any more, escalation can fix it.
Regarding processes and process adherence, we
wanted to learn how testers perceive processes in
regards to their job. During the interviews, all partic-
ipants found them useful, while on the survey 73 %
agreed, 18 % were undecided and 9 % disagreed.
During the interviews, testers mentioned that pro-
cesses had evolved in a positive way and that served
as a reminder to the tasks that they have to –which
received mixed opinions on the survey: 45 % agreed,
18 % was undecided, and 36 % disagreed. In general
we found that processes were perceived in a positive
light. Moreover one participant shared an interesting
insight on how to leverage change with the new
processes: “The last SPICE assessment was really
good and identified the weaknesses. It also specified
how we should receive some inputs for verification
criteria from the developers and including them in
the process should help”.
6 CONCLUSIONS AND FURTHER
WORK
We performed a mixed-method study in the software
department of a large automotive supplier operating
in a global software engineering setting. We set out
to understand the most recurrent problems testers
face in their day-to-day activities and how they col-
laborate with other teams in this setting. According
to our findings testers most recurrent problems are
of external nature: they receive late inputs – soft-
ware, hardware or requirements, improperly speci-
fied or incomplete requirements, unrealistic project
planning and estimation, and reduced time to per-
form testing. Unlike findings in previous studies, the
studied group of testers is highly motivated. They
are satisfied being testers, feel they have the skill
and ownership needed to do their job, and enjoy
working with their own team. Their years of experi-
ence and the training they received, mostly of tech-
nical nature, support their technical competence to
perform testing. During the qualitative study, partic-
ipants indirectly attributed their problems to human
factors such as people’s openness, attitude, and un-
derstanding of the shared goals and needs of others.
Nonetheless, collaboration was not identified as a
major recurrent problem. We believe that this is due
to a silo focus of the testing team, i.e. the testers
relate mainly to the other testers in their team and
focus mainly on their subunit rather than the overall
organization. In the matrix organization they work
in, they require input from different stakeholders and
during their testing they interface with them for
clarifications. The lack of clear goals for those inter-
faces as well as the scarce specialized training to
prepare people to develop the skills needed to per-
ICSOFT-EA2014-9thInternationalConferenceonSoftwareEngineeringandApplications
312
form in a matrix organization reinforce this silo-
focused behavior.
Based on our findings, analysis, and reflections,
we suggest that testers should be involved even
earlier in the requirements gathering stage. Not only
to review the specification but to provide input,
awareness regarding testability, and their
knowledge. Further research should focus on a vali-
dation of our findings and possible effects of reor-
ganizing software testing in the overall process of
the production of software systems. The size of the
population is not representative of software testers in
the automotive industry and we suggest that further
research should be conducted in other suppliers.
Nevertheless, we are aware that the functional ma-
trix organization, the fixed start date of production,
the complex and missing requirements, and the late
inputs for the testers are common among other sup-
pliers. We believe this case study sets out a starting
point for how software testing practice is carried out
by a group of highly qualified professionals. Further
research should look into the relationship between
the lack of cooperation within the testing team and
the impact this has on the quality of safety critical
products like the ones analyzed in this case study.
REFERENCES
Bernard, H. R. and Ryan, G. W., 2010. Analyzing qualita-
tive data: Systematic approaches. Sage.
Broy, M., 2006. Challenges in automotive software engi-
neering. In ICSE’06 Proceedings of the 28th ACM In-
ternational Conference on Software Engineering, 33–
42.
Cohen, C. F., Birkin, S. J., Garfield, M. J., and Webb,
H.W., 2004. Managing conflict in software testing.
Communications of the ACM, 47(1), 76–81.
Creswell, J. W. and Clark, V. L. P., 2007. Designing and
conducting mixed methods research. Wiley Online.
Deak, A., 2012. Understanding socio-technical factors
influencing testers in software development organiza-
tions. In Computer Software and Applications Confer-
ence (COMPSAC), 2012 IEEE, 1–4.
Deci, E. L. and Ryan, R. M., 2012. Motivation, Personali-
ty, and Development Within Embedded Social Con-
texts: Overview of self-determination theory. In R. M.
Ryan (Ed.), The Oxford Handbook of Human Motiva-
tion. 85–109. New York: Oxford University Press.
Denzin, N. K. and Lincoln, Y. S., 2005. The Sage Hand-
book of Qualitative Research. Sage.
Engström, E. and Runeson, P., 2010. A qualitative survey
of regression testing practices. In M. Ali Babar, Matias
Vierimaa and Markku Oivo (Eds.), Product-Focused
Software Process Improvement (LNCS Vol. 6156), 3–
16. Berlin, Heidelberg: Springer.
Fernández-Sanz, L., Villalba, M., Hilera, J., and Lacuesta,
R., 2009. Factors with negative influence on software
testing practice in Spain: A survey. In R. O’Connor,
N. Baddoo, J. Cuadrago Gallego, R. Rejas Muslera, K.
Smolander, and R. Messnarz (Eds.), Software Process
Improvement (Vol. 42), 1–12. Berlin, Heidelberg:
Springer.
Garousi, V., and Zhi, J., 2013. A survey of software test-
ing practices in Canada. Journal of Systems and Soft-
ware, 86(5), 1354–1376.
Guest, G., Bunce, A., and Johnson, L., 2006. How many
interviews are enough? An experiment with data satu-
ration and variability. Field methods, 18(1), 59–82.
Martin, D., Rooksby, J., Rouncefield, M., and Sommer-
ville, I., 2008. Cooperative work in software testing. In
Proceedings of the 2008 ACM international workshop
on Cooperative and human aspects of software engi-
neering, 93–96.
McLeod, L. and MacDonell, S. G. 2011. Factors that
affect software systems development project out-
comes: A survey of research. ACM Computing Surveys
(CSUR), 43(4), 24.
Oz, E. and Sosik, J. J. 2000. Why information systems
projects are abandoned: a leadership and communica-
tion theory and exploratory study.
Journal of Comput-
er Information Systems, 41(1), 66–78.
Patel, H., Pettitt, M., and Wilson, J. R. (2012). Factors of
collaborative working: A framework for a collabora-
tion model. Applied Ergonomics, 43(1), 1–26.
Shah, H., Nersessian, N. J., Harrold, M. J., and Newstetter,
W., 2012. Studying the influence of culture in global
software engineering: thinking in terms of cultural
models. In Proceedings of the ACM 4
th
International
Conference on Intercultural Collaboration, 77–86.
Shah, H. and Harrold, M. J., 2013. Culture and testing:
What is the relationship? In ICGSE’13, Proceedings of
the IEEE 8th International Conference on Global
Software Engineering, 51–60.
Sy, T. and D’Annunzio, L., 2005. Challenges and strate-
gies of matrix organizations. Human Resource Plan-
ning, 28–39.
Taipale, O. and Smolander, K., 2006. Improving software
testing by observing practice. In Proceedings of the
2006 ACM/IEEE International Symposium on Empiri-
cal Software Engineering, 262–271.
Whittaker, J. A., (2000). What is software testing? and
why is it so hard? Software, IEEE, 17(1),70–79.
Zhang, X., Dhaliwal, J., and Gillenson, M. L., 2010. Or-
ganizing software testing for improved quality and sat-
isfaction. Journal of Information Technology Man-
agement, 21(4),1-12.
APPENDIX
A: Interview Guide
1. What are the main challenges of your current
project?
UnderstandingSoftwareTestersintheAutomotiveIndustry-AMixed-methodCaseStudy
313
2. What can you say about the training you re-
ceived? Was this training useful on your day to
day job?
3. What have you learned from other projects that
in a current or future you wish to avoid and
how?
4. What are the most recurrent problems that the
testing team faces (think of internal factors
within your own testing team but also consider
other teams)
5. How would you describe the relationship be-
tween testers and other teams?
6. How do you perceive the collaboration?
7. How often do you interact or communicate
with them?
8. How do you view and feel about your job as a
Tester (self-image, skills, motivation, interest)?
9. How do you see testing as a profession?
10. How much responsibility and power do you
have regarding the tests you run?
11. How do managers or seniors relate to your
team?
12. What can you say about adherence to software
processes models (V-Model, SPICE)?
B: Agreement from Online Survey
Category: Recurrent Problems
Subcategory Agree Undecided Disagree
Late inputs 82% 18% 0%
Wrong or missing
requirements
73% 27% 0%
Unrealistic project
planning/estimation
73% 27% 0%
Reduced time to
perform testing
55% 45% 0%
Collaboration with
other Departments
45% 36% 18%
Category: Training
Subcategory Agree Undecided Disagree
Received training
while working in the
company.
91% 0% 9%
Technical Training 73% 9% 9%
Product Training 82% 9% 0%
Organizational 45% 27% 18%
Department (Interac-
tion with other Dept.)
36% 18% 36%
Category: Collaboration (depends on:)
Subcategory Agree Undecided Disagree
Openness and attitude
of others
91% 9% 0%
Understanding of
shared goalsand needs
from others
82% 18% 0%
Priorities and
schedule
73% 27% 0%
Frequency of
communication
73% 18% 9%
How it is enforced in
the project
45% 45% 9%
On task escalation 36% 45% 18%
Category: Motivation
Subcategory Agree Undecided Disagree
Satisfied as tester 73% 18% 9%
Find testing
interesting
91% 9% 0%
Enjoy working with
their team
91% 9% 0%
Have responsibility
over their work
91% 9% 0%
Have enough skills to
do their job
91% 9% 0%
Have power to decide
upon their daily tasks
64% 27% 9%
Category: Process and Process Adherence
Subcategory Agree Undecided Disagree
Processes are helpful 73% 18% 9%
Serve as reminder of
their job
45% 18% 36%
ICSOFT-EA2014-9thInternationalConferenceonSoftwareEngineeringandApplications
314
