INTERACTIVE QUESTIONNAIRES

Yann Veilleroy

, Frédéric Hoogstoel

and Luigi Lancieri

Institut Catholique de Lille, Lille, France

Equipe NOCE, Laboratoire d’Informatique Fondamentale de Lille (LIFL),

Université des Sciences et Technologie de Lille, Lille, France

Keywords: Interactive questionnaire, e-Brainstorming, Collective intelligence, Emergence, Malleability.

Abstract: The use of online interactive questionnaires is an interesting example of human-computer interactions

mediatizing human interactions to support the emergence of collective intelligence. To better understand

these interactions and their various effects, we propose to investigate the operating mode of interactive

questionnaires. First, we recall what the questionnaires are made of in order to know their anatomy. Then

we give two examples of interactive questionnaires: e-Brainstorming from the Orange Labs and the Real-

Time Delphi, one computerization of the Delphi method.

1 INTRODUCTION

Questionnaires are mainly used in surveys that

address many issues in a wide range of areas. They

have spread with computerization and the

development of the Web, which makes data

collection easy. Unlike traditional questionnaires,

where questions and answers are frozen at the

beginning of the survey, interactive questionnaires

are dynamic or tailorable, malleable: new questions

may arise, with new possible answers that modify

the initial questionnaire. Through several iterations,

the interaction leads to an adaptation via the

feedback loop of the socio-technical system. The

result of this adaptation can be the achievement of

consensus or an aid to problem resolution or the

discovery of multiple tracks.

This paper is organized as follows. First, we

present a state of the art of traditional surveys just

before discussing the changes involved by online

interactive questionnaires. Then, we develop the

underlying concept and method and we present some

related products. Finally, we discuss the advantages

and drawbacks of such techniques.

2 STATIC SURVEYS

Questionnaires are frequently used for surveys. The

following figure represents an overview of survey

concerns: the main issues of sampling, the classical

“errors”, and the different means to contact the

participants depending on their management.

In her “Fundamentals of Survey Research

Methodology”, Glasow (2005) begins with a quote

about the objectives of a survey: [Survey research is

used] “to answer questions that have been raised, to

solve problems that have been posed or observed, to

assess needs and set goals, to determine whether or

not specific objectives have been met, to establish

baselines against which future comparisons can be

made, to analyze trends across time, and generally,

to describe what exists, in what amount, and in what

context.” (Isaac and Michael, 1997, p. 136).

Errors generally arise for four reasons, Fricker

(2008) quotes Groves (1989): coverage, sampling,

non-response and measurement. The coverage error

occurs if the sample doesn't include all needed units

of the population. The sampling error occurs

because only a limited sample of the population is

invited to participate in a survey instead of the total

population, and some people in the sample tend to

make a measurement deviation. The measurement

error occurs when the answers given are not accurate

and the given answers difficult or impossible to use.

This happens, for instance, when the influence

during an interview is too much significant (Fricker,

2008). An important point to determine when

designing a survey is the mode (the channel of

communication). Roughly speaking, surveys are

classified into two categories: interviewer-

administered surveys and self- administered surveys.

De Leeuw (2005) explains mixed modes are a way

to reduce costs, errors and to get more data.

520

Veilleroy Y., Hoogstoel F. and Lancieri L..

INTERACTIVE QUESTIONNAIRES.

DOI: 10.5220/0003845105200523

In Proceedings of the 4th International Conference on Agents and Artiﬁcial Intelligence (ICAART-2012), pages 520-523

ISBN: 978-989-8425-95-9

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

Figure 1: Overview of survey concerns.

An evolution came with computerization; the e-

mail mode and the web mode are new internet-based

modes. The web mode gives lot of capabilities and

advantages: shorter transmitting time, lower delivery

cost, more design options, and less data entry time.

However, it has several flaws: sampling remains

difficult, “Coverage is the most widely recognized

shortcoming of internet-based surveys” (Fricker,

2002), the response rate is estimated approximately

11% lower than the traditional modes (Manfreda et

al., 2008) quoted by (Fan and Yan, 2010). The web

mode allows designing a large amount of possible

surveys, the interactive questionnaires fit well with

the online modes such as web mode.

3 INTERACTIVE SURVEYS

One interesting point with internet based

technologies relates to open-ended questions, Yun

(2000) quotes Bachmann and Elfrink (1996): open-

ended questions by e-mail collect longer answers,

more information than a paper survey. Keyboarding

can be easier than handwriting. This was also

observed at Murdoch University (Australia), where

students are surveyed to evaluate teaching. Collings

and Ballantyne (2004) made a comparison between

online and paper regarding the 2003 survey. The

students write more comments online than on paper,

when they do comment, the number of words is

higher. Collings and Ballantyne conclude saying that

the response rate focuses first the attention, but the

value of qualitative data, richness in responses is in

comment length.

This points out limitations of traditional static

surveys. Such surveys are conducted to gather

information about a population, but they are very

regulated, framed, constrained: participants can only

make a choice by a yes/no system, a check boxes

system, a scales system or respond in textboxes. We

can see that a better quality of information comes

from open-ended questions and interactivity because

respondents have things to say and they respond

willingly when they are given the opportunity. The

web mode lends itself to this form of questioning. A

questionnaire that allows people express themselves,

as in a real dialogue, or a debate, could collect lots

of valuable information. This would be another kind

of questionnaires, dynamic and interactive, with

another design, other goals and objectives. Now let’s

review two initiatives in the field of interactive

questionnaires: the e-Brainstorming and the Real-

Time Delphi. Roughly speaking, the e-

Brainstorming relies on multiple-choice questions as

first interaction mode and the Real-Time Delphi (RT

Delphi) is a web-based variant of the Delphi method

that we will remind.

3.1 e-Brainstorming

The e-Brainstorming, an initiative from the Orange

Labs of Caen in France (Lancieri et al., 2005), is a

computerized system of closed questions, a multiple

choice questionnaire system (MCQ), intended to

simplify and synthesize the opinions of a group. The

system uses the good will of the participants to

respond to open-ended questions as we have just

seen. Inside a question, respondents can leave a

comment: each question provides a free comments

zone, and the system allows them to add questions

and possible answers that are forwarded to all

respondent for a new questionnaire round. This is a

way to get rid of some limitations (lack of

cooperation, not enough choice) introduced by the

MCQ. The success rate of certain questions and the

amount of left comments on certain questions are

fuelling debates. The idea relies on human

intelligence: to solve a problem, half of the solution

INTERACTIVE QUESTIONNAIRES

521

is in formulation. The e-Brainstorming provides a

structuring frame to express, formalize and

reformulate ideas. There is no moderator, the group

has to be self-moderated.

The questionnaire is scripted with an easy tagged

language, writable in a form, in a mobile phone

application for instance, then sent to a web server.

After generation on a server, a return mail is sent to

the author with the URL of the web questionnaire, to

be distributed to the chosen participants. e-

Brainstorming can be applied in three modes:

without free comments, with free comments, with

free comments and capacity for adding new

questions. Among the features: new questions are

highlighted, participants can check a box “Does not

interest me”, they can access statistics and graphics;

the system uses traces to evaluate the collective

intelligence phenomenon. Data can be exported: it is

possible to exploit it in information processing

systems (semantic web, etc.). Cheap, easy and quick

to implement, it can be used with distributed

participants. This is another example demonstrating

that technology can change spatiotemporal

interactions between individuals (Lancieri et al., 2005).

3.2 Delphi, Real-time Delphi

“The primary strength of Delphi is its ability to

explore, coolly and objectively, issues that requires

judgment” (Gordon, 2003). In the 1950s, on demand

of the US Army, RAND (Research ANd

Development - a thinktank) was in charge of

creating a method to make forecasts from the

opinion of experts about the possibility (etc.) of an

enemy attack during the Cold War. RAND achieved

the Delphi method. A sample of experts on the topic

of the study enters an iterative process of

questionnaires, administered by researchers, which

stops when a pre-defined criterion has been reached.

Iterative process: the questionnaire has to be

prepared, the questions refined. In a first

questionnaire, the experts are asked to provide their

judgment. Data collected is analyzed, synthesized.

The next round begins with the sending of a new

questionnaire. The experts have to be more accurate,

some extreme positions have to be reassessed,

justified (Gordon, 2003). There are multiple rounds

until a defined criterion is reached, then the

administrator stops the questionnaires. Gordon says

that at the end, more often than not, group of experts

move toward a consensus. If not, the reasons are

known. The reasons and arguments for the

consensus as well as for disagreements can be highly

valuable and useful: with this material, it becomes

possible for planners to make judgments. Synthesis

is done to reduce the force of oratory of some people,

by this way every participant is equal; the Delphi

method was designed to encourage a true debate.

Okoli and Pawlowski (2004) have compared a

traditional survey approach with the Delphi method

in order to investigate what would be the critical

success factors for e-commerce in Sub Saharan

Africa. In a traditional survey, a questionnaire is

designed, the participants respond, data collected is

analyzed. In the Delphi method, a questionnaire is

designed, submitted to a group of experts, responses

are analyzed, then the questionnaire is re-designed

and re-submitted and this process is repeated. The

sample of the Delphi method is made of some

experts on the topic (literature recommends 10 to 18

experts). In a traditional survey, a large sample is

preferred, for generalization. Individuals responses

are averaged in a traditional survey while in the

Delphi method, “Studies have consistently shown

that for questions requiring expert judgment, the

average of individual responses is inferior to the

averages produced by group decision processes;

research has explicitly shown that the Delphi method

bears this out”. For a traditional survey, reliability is

an important concern, assured by multiple tests,

while in a Delphi study, the experts have to revise

their responses, the importance degree is different.

In a traditional survey, the construct validity is

assured in the design phase and participants only

respond. In the Delphi method, the experts are asked

to validate the design (researchers' interpretation and

categorization of the variables). Participants are

always anonymous in a traditional survey. In the

Delphi method, respondents are anonymous to each

other but not to researchers, then, if a participant

drops-out, researchers are able to discover the reason

by asking directly. Non-response is an error to be

reduced in traditional survey while in the Delphi

method, experts have been selected and solicited to

give their opinion. In a traditional survey, the quality

depends on the question, design, the follow-up, the

respondents... while in the Delphi method, multiple

iterations provide rich data, furthermore there may

be follow-up interviews: interviewers can come with

open-ended questions to learn more.

In 2004, the Defense Advanced Research Project

Agency (DARPA) asked for the development of a

Delphi-based method for improving the speed of the

Delphi method. A company, Articulate Software,

made the “Real-Time Delphi”. This computerized

method is quick and there can be more participants.

They introduced Artificial Intelligence and Natural

Language during the analysis phase of non-

numerical answers. It works “roundless”: every

ICAART 2012 - International Conference on Agents and Artificial Intelligence

522

participant can come, at anytime to update his or her

inputs. Each presented question comes with some

information (the average/median response of the

group, the number of responses, the reasons).

Respondents have got spaces to respond and justify

it. Real-Time Delphi has got a large range of

applications (Gordon and Pease, 2006).

4 DISCUSSION

There are elements that lead us to say e-

Brainstorming and the Real Time Delphi seem to

have some convergence points as interactive

questionnaires systems: they are well suited for

small groups of carefully selected experts. A second

point of convergence is the exploitation of what we

saw with open-ended questions: people are willing

to express themselves, in writing. e-Brainstorming

allows participants to add questions and to leave

comments. The Real Time Delphi tries to get the

opinion, the judgment and a justification of the

participants. They both take advantage of the ease of

keyboarding. A third point of convergence is: e-

brainstorming and RT Delphi solved the problem of

availability of a moderator thanks to

computerization but each in a different way: the first

uses multiple-choice questionnaire, the second uses

of Natural Language and Artificial Intelligence to do

the sort and classification tasks and to send back a

synthesis. e-Brainstorming relies on a self-regulation

from the participants because it is assumed that

multiple-choice questionnaire simplifies the

opinions (implicit management), RT Delphi uses a

sophisticated and instrumented device. Where they

differ is in the goal: the Delphi method seeks a

consensus with a predefined set of questions

carefully prepared and refined during the iteration

process. e-Brainstorming encourages participants to

formalize the subjects of interest in a structured

frame but participants can create new questions and

comment it, this can give an open direction to the

discussion; we might call such a questionnaire a

malleable questionnaire. The RT Delphi uses an

impressive computerization, e-Brainstorming uses a

cheaper, more creative and human method.

In this paper, we have seen that surveys have

evolved. Technologies such as web have allowed the

rise of different forms of interactive questionnaires.

We should continue to study interactive

questionnaires because they are group-meeting

methods that develop creativity, allow people to

share and pool together ideas, uses the good will and

where everyone is equal. They could be useful tools

for research purposes.

REFERENCES

Bachmann, D., & Elfrink, J. (1996). “Tracking the

progress of e-mail versus snail-mail.” Marketing

Research, 8(2), 31-35.

Collings, D., & Ballantyne, C. S. (2004, November 24-

25). “Online student survey comments: A qualitative

improvement?” A paper presented at the 2004

Evaluation Forum, Melbourne, Victoria.

de Leeuw E. D. (2005). “To mix or not to mix: data

collection modes in surveys.” Journal of Official

Statistics, 21, 233-255. http://igitur-archive.library.uu.

nl/fss/2011-0314-200305/EdL-to%20mix%202005.pdf

Fan W., Yan Z. (2010). “Factors affecting response rates

of the web survey: A systematic review.” Computers in

Human Behavior 26:132-139. DOI: 10.1016/

j.chb.2009.10.015.

Fricker, R. D., & Schonlau, M. (2002). “Advantages and

Disadvantages of Internet Research Surveys: Evidence

from the Literature.” Field Methods, 14(4), 347-365

Fricker, R. D. (2008). “Sampling methods for web and e-

mail surveys.” In N. Fielding, R. M. Lee & G. Blank

(Eds.), The SAGE Handbook of online research

methods (pp. 195-216). Los Angeles, CA: Sage.

Glasow, P. (2005, April). “Fundamentals of survey

research methodology”. Retrieved January 16, 2009:

http://www.mitre.org/work/tech_papers/tech_papers_0

5/05_0638/05_0638.pdf

Gordon, T. J. (2003). “The Delphi method.” In J. C. Glenn

& T. J. Gordon (Eds.), Futures research methodology -

V2.0. Washington, DC: American Council for the

United Nations University.

Gordon, T. & Pease, A. (2006). “RT Delphi: An Efficient,

'Round-Less' Almost Real Time Delphi Method.”

Technological Forecasting and Social Change, 73,321-333.

Groves, Robert M. (1989). “Survey Errors and Survey

Costs.” New York: John Wiley & Sons.

Isaac, S., & Michael, W. B. (1997). “Handbook in

research and evaluation: A collection of principles,

methods, and strategies useful in the planning, design,

and evaluation of studies in education and the

behavioral sciences”. (3rd Ed.). San Diego:

Educational and Industrial Testing Services.

Lancieri L., Lavallard A., Manson P. (2005). “E-

Brainstorming: Optimization of collaborative learning

thanks to online questionnaires”, proceedings of

CELDA IADIS International conference, 2005.

Manfreda K. L., Bosnjak M., Berzelak J., Haas I., Vehovar

V. (2008). “Web surveys versus other survey modes.”

International Journal of Market Research, 50, 79-104.

Okoli, C. & Pawlowski, S. D. (2004). “The Delphi method

as a research tool: an example, design considerations

and applications.” Info. & Management, 42, 15-29.

Yun G. W. and Trumbo, C. (2000) “Comparative

Response to a Survey Executed by Post, E-mail and

Web Form”, Journal of Computer-Mediated

Communication.http://onlinelibrary.wiley.com/doi/10.

1111/j.1083-6101.2000.tb00112.x/full

INTERACTIVE QUESTIONNAIRES

523