Are We Ready for Problem-based Learning? A Proposal of
Institutional Diagnosis in Computing Higher Education
Osmário de Carvalho Santos Filho and Simone C. dos Santos
a
Centro de Informática, Federal University of Pernambuco, Recife, Brazil
Keywords: Computing Higher Education, PBL, Institutional Diagnosis, Opinion Survey.
Abstract: In Computing Higher Education (CHE), the desired transformation of traditional teaching and learning
methods, almost always based on the transmission of information and content-based curricula, has been the
objective of several educational institutions that wish to combat students' demotivation and dropout. Among
successful approaches, Problem-Based Learning stands out as one of the most effective and radical methods
regarding pedagogical innovations. While the implementation of the PBL means a great opportunity to
achieve better educational performance, it also represents many challenges that can only be managed if they
are first known and understood. In this context, the motivation for this study comes from the following
research question: "How to know if an institution at CHE is ready to implement the PBL?". As a response, an
institutional diagnostic model regarding the adoption of PBL is proposed. From an opinion survey with 38
technical educational institutions in computing, involving 302 participants, the results showed that the model
reached its objective, allowing the identification of favorable, warning, and critical points regarding the
adoption of PBL in these institutions.
1 INTRODUCTION
In Computing Higher Education (CHE), the desired
transformation of traditional teaching and learning
methods, almost always based on the transmission
of information and content-based curricula, has been
the objective of several educational institutions that
aim to prevent students' demotivation and dropout
(Yang and Choi, 2017). There are diverse examples
of successful approaches, in particular those using
authentic problems, attractive technologies, and a
learning environment that reflects the labor market
to promote the engagement and motivation of the
students (Babori et. al., 2016; Kemavuthanon, 2017;
Martins et. al., 2018; Santos and Silva, 2018). Also,
several solutions seek to systematize teaching and
learning methods into methodologies, with concrete
proposals to help transform the traditional classroom
into a practical and stimulating environment. Thus,
teaching methodologies such as Problem-Based
Learning (PBL) (Yu, 2005; Jaryani et. al., Zhao and
Liu, 2011; Panwong and Kemavuthanon, 2014;
Ibrahim and Halim, 2014; Mäenpää et. al., 2017),
and its variants like Project-Based Learning (Bell,
a
https://orcid.org/0000-0002-7903-9981
2010), Case-Based Learning (Srinivasan et. al.,
2007) among others, have become popular in
computing education, bringing important benefits
such as increased engagement, motivation, and
development of technical and non-technical skills so
important to the professional life of the student.
Born in the medical education and defined as "a
learner-centered constructivist method that uses
real problems as a learning object" (Savery and
Duffy, 1994). PBL is considered one of the most
effective and radical methods regarding pedagogical
innovations. It advocates profound changes that
involve the institution as a whole, transforming its
learning environment, teachers' and students'
attitudes, curricula, operational and managerial
resources, infrastructure, assessment processes,
relationships with the labor market, and,
consequently, budget (Fink, Enemark and Moesby,
2002). While the implementation of the PBL means
a great opportunity to achieve better educational
performance, it also represents many challenges that
can only be managed if they are first known and
understood. In this context, it is necessary to
understand the challenges faced by all stakeholders
Filho, O. and dos Santos, S.
Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education.
DOI: 10.5220/0010433604130424
In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU 2021) - Volume 1, pages 413-424
ISBN: 978-989-758-502-9
Copyright
c
2021 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
413
in the educational process based on PBL and raise
the requirements to be adopted this approach
effectively. With this objective, the NEXT
(iNnovative Educational eXperience in Technology)
research group has been investigated the use of the
PBL approach in computer education for more a
decade (Oliveira, Santos and Garcia, 2013; Santos et
al., 2020), developing methodologies that can
support the pedagogical team in adopting this
approach. However, in recent consultancies in CHE
institutions that wish to transform their curricula and
traditional pedagogical proposals through the PBL,
it was encountered an imminent difficulty: the
institutions were not able to adopt the PBL and, even
more critical, they did not know how to start. With
this motivation, a master's research (Santos Filho,
2020) sought to investigate solutions that could help
higher education institutions identify their ability to
implement the PBL, making evident its favorable,
critical aspects and warning signs. From a
systematic literature review concerning PBL applied
in computing education from 1999 to 2019 (Santos
et al., 2020), several frameworks for applying PBL
were identified, proposing the key elements for
implementing the method and conducting the
classroom approach as described in (Yu, 2005;
Jaryani et. al., Zhao and Liu, 2011; Panwong and
Kemavuthanon, 2014; Ibrahim and Halim, 2014;
Babori et. al., 2016; Kemavuthanon, 2017; Mäenpää
et. al., 2017; Martins et. al., 2018; Santos and Silva,
2018). But no solution was found focusing on
investigating the situation of an educational
institution regarding the requirements necessary
for/adopting the PBL. So, the motivation for this
study comes from the following research question:
RQ - "How to know if an institution at CHE is ready
to adopt the PBL?".
In this context, this study proposes a model of
institutional diagnosis intending to investigate this
situation, considering two target-public:
teachers/tutors, giving their opinion on the essential
elements for the adoption of the method under the
pedagogical aspect, also reporting their perceptions
related to students in general; course
coordinators/managers, who have a more systemic
view of the educational institution and, therefore,
can contribute with their vision under structural and
organizational aspects, complementary to the
teachers' view. From these visions, the educational
institution will be able to identify its strengths and
improvement points regarding the implementation
of PBL, supporting its decisions of changes within
its pedagogical planning and educational strategy.
To report the early results of this research, this
paper is divided into 6 sections. After this brief
introduction, Section 2 presents the main PBL
references used as the foundation of the model,
highlighting concepts, challenges, and requirements
for adopting this approach. Section 3 describes the
research methodology and its steps. The institutional
diagnosis model is presented in Section 4. To assess
the practical applicability of this model, Section 5
describes the results and discussions of an opinion
survey with 38 educational institutions in
computing, which answered the proposed diagnosis
involving their teachers and course coordinators'
team. Finally, Section 6 discusses the conclusions
and future work.
2 THEORETICAL REFERENCES
According to (Ribeiro, 2008), the PBL is a teaching
and learning method that makes use of real problems
to motivate students to learn concepts, procedures,
and attitudes that will be important for their future
performance as citizens and professionals. As
emphasized in (Melo, 2013), the potential of the PBL
in the teaching and learning process of students is
capable of developing important skills such as self-
confidence, problem-solving, and autonomy. Thus,
the focus of the teaching process is the student, who
is stimulated to learn more autonomously and
cooperatively with his colleagues. The PBL method
still requires a more active posture from students,
choosing the best way to learn, conducting research,
and using educational resources that go beyond the
classroom. This also encourages students to reflect
critically on what is proposed to solve the problem,
important skills for future computer professionals in
constant learning due to technological advances. In
this context, the role of the teacher is to monitor and
provide feedback on learning, in addition to
identifying the difficulties encountered by students,
facilitating and guiding the progress of their learning
(Enemark, Kolmos and Moesby, 2006).
PBL follows some principles such as an authentic
learning environment and simulation of the situation
found in the professional environment, the use of real
problems as a learning object, the monitoring of
evaluation by continuous feedback (Ribeiro, 2008).
In (Santos et al., 2013), ten principles were defined
for the teaching of Computing that founded a
methodology called xPBL (Santos et al., 2014), as
shown in Figure 1.
CSEDU 2021 - 13th International Conference on Computer Supported Education
414
Figure 1: Ten PBL Principles and the xPBL elements.
The xPBL methodology defines five manageable
elements for PBL planning: 1) Problem; 2) Learning
Environment; 3) Human Capital, that includes
students, pedagogical team and market partners; 4)
Content, as a guide and support to solve problems;
And 5) Processes, concerning educational objectives
and assessment processes. These elements are aligned
with the ten PBL principles (as shown in Figure 1)
that are fundamental for the implementation of an
authentic PBL in computer education.
2.1 Challenges in PBL
The adoption of the PBL approach, as it is not trivial,
requires a series of changes, both in the attitude of
teachers and students (Moesby, 2004; Santos and
Silva, 2018; Aldabbus, 2018). According to (Moesby,
2004), it is common to evidence in the application of
the PBL method the resistance of teachers, who often
lack knowledge and experience in the teaching
methodology, and for this, need to be trained to face
the obstacles of the method.
Moreover, teachers should always be aligned with
the PBL teaching process, considering that this
approach requires a learning environment with a
flexible curriculum geared to the demands of the
professional market, in which the relevant issues are
raised by the students and no longer by the teacher
(Enemark, Kolmos and Moesby, 2006). As for the
students, according to (Santos and Silva, 2018), some
difficulties were observed in the application of the
PBL method: a certain initial discomfort with the
changes; lack of bibliographical research during the
activities; lack of ability with technological
resources; little involvement of the students in
groups, and student priorities. However, there are also
factors external to the learning environment that make
it difficult to implement the PBL, as mentioned by the
students, such as adequate time management, which
causes a drop-in student performance and project
quality (Hsu and Lin, 2016). In the application of the
PBL, the dedication of students inside and outside the
classroom is fundamental, and their active
participation in the learning process.
It is also necessary to face many obstacles in the
planning for the implementation of the PBL
methodology, such as the development of problems
or projects, the planning of classes with a new
approach, the lack of technological tools and trained
professionals to practice this method in the institution
(Herold, 2019). The initial lack of security in the
process of change to the PBL method needs to be
overcome, as well as the difficulties of aligning the
time between theory and practice, the adequacy of the
curriculum, the availability of financial resources, the
evaluation, and development of skills of the tutor
teacher (Souza and Dourado, 2015). In (Aldabbus,
2018), the author reinforces these challenges
emphasizing the lack of technology, flexible
schedules, and absence of curriculum policy as
difficulties in adopting the method. The study
(Krusche et al., 2016) reports that there are
institutional factors that hinder the adoption of the
PBL method such as infrastructure, class planning,
and didactic resources, in addition to an adequate
environment for the development of team activities,
technical-pedagogical support, and hiring of
monitors/tutors.
2.2 Requirements to Apply PBL
To overcome the challenges of adopting the PBL, it
is necessary to plan all the processes and resources
before implementation, assisting in the correct use of
the method, in the alignment of theory and practice,
and respecting the principles of this approach (Santos
et al., 2013). When planning to change a teaching and
learning process, it is important to carry out a
diagnosis in order to characterize the respective
educational institution that will face the changes.
Thus, it is possible to identify the main aspects
required for the effective implementation of the
approach.
Among the aspects to be analyzed, it is important
to point out: collaboration between courses from a
curriculum that allows the integration of knowledge
acquired by students; a central committee for
curriculum planning, to manage the content and
topics to be addressed in the course; training of
teachers to improve their didactics and learn new
teaching strategies; planning of teaching in small
groups and; availability of a period for study
(Alshaye, Tasir, and Jumaat, 2019). The
implementation of PBL also requires the
understanding and participation of various actors
(teachers, students, researchers, managers, real
clients) who will be active in the processes, therefore
the need to focus on team development and
curriculum development of PBL (Fink, Enemark and
Moesby, 2002; Koray and Koray, 2013). The PBL
Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education
415
curriculum should have a flexible format, be student-
centered, be interdisciplinary, have real problems,
and focus on research and investigation, promoting
critical thinking and development of technical and
non-technical skills (Hmelo-Silver, 2004; Melo,
2008). Content planning should focus on practical
classes, with the learning process focused on solving
problems that stimulate discussion, challenge
students, and stimulate their creativity (Krusche,
2016; Barron and Wells, 2013).
Other prerequisites in the process of adopting the
PBL are the provision of free space for self-learning,
a review of the role of teachers and departmental
autonomy, availability of financial resources to invest
in infrastructure, and aligning student selection
criteria to the profile expected by the PBL approach.
Santos et al. (2020) reinforce that the implementation
of the PBL method in an environment with traditional
teaching triggers the need for a series of changes,
such as the adaptation of the curriculum, the
formatting and organization of the learning
environment and the use of technologies.
PBL promotes many benefits for the student
learning from the alignment of academic training with
the requirements of the professional market (Herold,
2019). At this point, the PBL requires that the
institution has approximation with companies in the
labor market to provide teachers with new ideas and
relevant problems to be addressed in the classroom.
There are organizations that perform PBL method
consulting and analyze the institution at the
organizational, pedagogical and educational level,
thus supporting the design of a PBL-based
curriculum, evaluation processes and,
implementation of organizational and pedagogical
aspects (Moesby, 2004). Moesby reports that many
educational institutions analyze the needs of students
today, and this brings about various changes at
various levels: personal, organizational, and cultural.
These changes require the development of an action
plan, involving not only principals, but all those
involved in the educational process, such as teachers,
tutors, coordinators, and managers.
3 RESEARCH METHOD
This study has used a mix of methods, with emphasis
on qualitative research and descriptive approach.
According to Patton in (Patton, 2002), research is
said to be qualitative when it aims to investigate
what people do, know, think, and feel through data
collection techniques such as observation,
interviews, questionnaires, document analysis,
among others. Conducted by the NEXT research
group focusing on studies on the PBL approach
applied to computing teaching, this research was
motivated by direct observations in consulting
activities to support educational institutions in the
transformation of traditional methods to PBL, in
general, carried out during the process of
educational innovation and curriculum changes. At
this time, an analysis of the institution is essential to
identify the requirements for the implementation of
PBL. Figure 2 shows the research steps and the main
methods.
Figure 2: Methodological scheme.
After an ad hoc literature review seeking solutions
to help this investigation, the central research
question was defined: RQ - "How to know if an
institution at CHE is ready to adopt the PBL?
To find objective answers, this research question was
divided into three secondary questions:
Q1 - What are the favorable points for adopting
the PBL?
Q2 - What are the critical points for PBL
adoption that can negatively impact its
implementation?
Q3 - What are the warning points that the
institution needs to understand better to make
new decisions?
From these research questions, two objectives
were defined: (1) the conception of a model to assess
the ability to apply PBL and; (2) the application of
this model in real educational institutions to verify
its effectiveness. To achieve these objectives, two
research cycles were designed, as shown in Figure
2.
CSEDU 2021 - 13th International Conference on Computer Supported Education
416
The first cycle had focus on the "conception of
the institutional diagnosis", which included
searching the relevant literature on the adoption of
PBL in the teaching computing (Oliveira, Santos
and Garcia, 2013; Santos et al., 2020),
understanding of the main challenges encountered
by the institutions and the essential requirements for
the successful implementation of PBL, as discussed
in sections 2.1 and 2.2, respectively. From these
references, three dimensions of the model and its
aspects were identified:
Pedagogical, with the five aspects Problem,
Environment, Human Capital, Content and
Process, based on the methodology cited in
Section 2;
Structural, which includes Infrastructure and
Curriculum aspects, and;
Organizational, with the Political and
Evaluation aspects.
Besides, two target-public were defined: the
teacher/tutor and the course coordinator/manager,
both key actors in the process of transforming the
pedagogical approach. At this point, the student
inquiry was also considered, but it was decided to
capture the students' perspectives from their
teachers/tutors. As a result, a set of 85 assertions were
identified, with respective rationales and literature
references. After that, the assertions were analyzed by
two PBL specialist, discarding those outside of the
context and grouping others and, finally, defining 60
assertions distributed in two questionnaires of 30
questions each one, addressed to the two target-
public. Each assertion has its rationale and literature
references (URL in Appendix).
The second cycle was guided by the objective of
applying the model. To do so, this research has used
the opinion survey method proposed by Kitchenham
and Pfleeger (2008), defined in six stages: Setting
the objectives; Survey design; Developing the
survey instrument (i.e. the questionnaire);
Evaluating the survey instrument; Obtaining valid
data and; Analysing the data. The research objective
was to evaluate and identify the aspects required for
the adoption of PBL in higher education institutions
in computing, in both undergraduate and graduate
levels, checking the current situation with teachers
and course coordinators.
To create the research instrument, the
questionnaires and assertions defined in Cycle 1
were analyzed by seven specialists (6 Ph.D. and 1
Ph.D. student, all with more than 5 years of
experience in PBL) under the aspects of syntax,
semantics, level of the relevance of the questions
and completeness of the questionnaire. From this
evaluation, some updates, groupings, and
suggestions of new assertions were recommended:
an assertion to verify if the self-regulation and
metacognition of the students can be evidenced in
the respective institution; an assertion on feedback
in the evaluations of the teachers and other actors.
After the experts' evaluation, two forms
(Google) were created, each directed to a type of
participant (teacher or coordinator). The answers
were based on Likert's ordinal scale: 1) Totally
Disagree, 2) Partially Disagree, 3) Neither Agree
nor Disagree; 4) Partially Agree, and; 5) Totally
Agree.
For data analysis, consolidated charts were
generated for each question, with the appropriate
proportions of each answer chosen. For a qualitative
discussion of the results, three status were defined:
"favorable" (agrees), "warning" (neutral), and
"critical" (disagrees), facilitating the analysis of
responses towards the secondaries research
questions and future decision making. The results of
this assessment will be discussed in more detail in
Section 5.
3.1 Limitations and Threats to Validity
It is important to highlight some limitations of this
research. According to “Promoting Institutional &
Organisational Development” guideline in (DFID,
2003), a diagnosis must be a thorough task, based on
a careful selection of interested parties. However,
preliminary or partial analysis in the first contact
with the investigated institution can serve as a
reference base to have a more comprehensive and
in-depth diagnosis later.
Another important point is that an institutional
diagnosis is focused on a particular organization,
requiring time and effort to apply, discuss, and carry
out future interventions. This research focused
solely on the diagnosis application stage, using the
opinion survey method for large-scale application,
to understand its usefulness regarding the
implementation of PBL, making evident the
favorable, critical points and warning signs
institutions.
Finally, to facilitate the processing of the data
collected in the diagnosis, a Likert scale with five
values from "totally disagree" to "totally agree" was
used. However, the evaluation of the results adopted
a qualitative interpretation of the data, identifying the
favorable, critical points and warning signs, in
response to the secondary questions Q1 to Q3. Thus,
Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education
417
no statistical method was applied in this case, which
does not prevent it from being adopted in future
works.
4 PBL INSTITUTIONAL
DIAGNOSIS
The PBL institutional diagnosis is structured
according to Figure 3, while Table 1 and 2 present the
assertions for the Teacher/Tutor and
Coordinator/Manager questionaries, respectively.
Figure 3: Structure of the Institutional Diagnosis.
The scope of the diagnosis is focused on
computing courses, considering the PBL references
used for its definition and the CHE context. Two
questionnaires have created addressing collaborators
of the educational institution in the role of
teacher/tutor and coordinator/manager. Each
questionnaire has 30 assertions, in the format of
Google forms.
Regarding identification of the respondents, the
following fields were also included: name, institution,
position/function, e-mail, course modalities, course,
duration (in hours), teaching methodology, PBL
knowledge, and experience in PBL. The “course” and
“duration” fields are only included in the teacher’s
form, and the “teaching methodology” field asks what
type of methodology the teacher uses within three
alternatives (traditional, active, and hybrid), while in
the coordinator’s form asks if the institution
recommends any type of teaching methodology.
It is important to emphasize that the results of this
diagnosis provide transparency about the situation of
the institution from the perception of its employees.
With this information, discussions, and referrals can
be made, for example, regarding the training of
teachers, approximation with market companies, or
even acquisition of specific technologies.
Table 1: Teacher/Tutor Questionary.
PROBLEM
A1 The activities developed in the computing course use
real problems as a motivating element for student
learning.
A2 Students in the computing course seek to take
ownership of the problem to be solved, becoming
responsible for the learning itself.
A3 The problems, problem situations, or hypotheses
presented in the course are based on real contexts.
A4 Problems attributed to learners are stimulating as a
challenge to reasoning.
A5 In the course, problems or problem situations have
similar complexity to those found in real contexts.
A6 Students interact with real customers and users who
have problems to solve.
ENVIRONMENT
A7 The environment stimulates social skills and problem-
solving required by the labor market.
A8 The students’ learning environment seeks solutions to
real client demands in order to reflect conditions similar
to the professional market.
A9 Students demonstrate a professional attitude in the
learning environment in order to maintain the
authenticity of the labor market in the educational
environment.
A10 The students’ learning process is implemented in an
environment that provides conditions for students to
assume responsibilities assigned to certain functions
and positions in the Computing.
A11 The physical and technological infrastructure of the
learning environment stimulates and favors the
execution of learning dynamics, group work, and
collaborative activities.
A12 The learning environment fosters the understanding of
the concepts that will be aimed at solving problems in
the labor market context.
CONTENT
A13 In the course of computing, there is alignment between
theory and practice.
A14 There is integration between the courses in favor of the
educational proposal of the curriculum.
A15 The content planned to be addressed is based on
projects and practical activities.
A16 The content of the course is designed to be used as a
guide and support for problem-solving, therefore
flexible.
A17 The course structure is planned with content that will
generate dynamics in student learning inside and
outside the classroom.
A18 The course plan can be adjusted as the teaching
methodology changes.
CSEDU 2021 - 13th International Conference on Computer Supported Education
418
Table 1: Teacher/Tutor Questionary (cont.).
HUMAN CAPITAL
A19 Teachers monitor the resolution of problems by
students in order to stimulate resolution using
interdisciplinary knowledge.
A20 In solving real problems, there is the participation of the
real client to carry out the collaboration in the
requirements of the problem.
A21 The students demonstrate to be active and autonomous
in the construction of their knowledge from the solution
of the proposed problems.
A22 In problem-solving tasks, teams or small groups are
formed with 4 to 7 students to promote a greater
contribution among members.
A23 The students' learning process has a multidirectional
characteristic, that is, there is an effective relationship
between students, teacher, tutor, and real client.
A24 The pedagogical coordination seeks, together with the
teachers, to develop more collaborative and
multidirectional projects for their students.
PROCESS
A25 Teachers plan the content to be learned and evaluations
should reflect on the content assimilated.
A26 Teachers carry out continuous evaluation and
monitoring of the teaching and learning process, aiming
to help students in their own reflection on learning from
their difficulties and feedback.
A27 The institution seeks appropriate strategies for
monitoring and evaluating student learning.
A28 In the teaching-learning process, characteristics of
metacognition and self-regulation can be evidenced in
students.
A29 The teachers evaluate the learning outcomes with the
objective of following up the student in solving process,
before putting it into practice.
A30 The educational planning meets the students'
expectations regarding the objectives, goals, or
expected results of the course.
5 APPLYING PBL DIAGNOSIS:
RESULTS AND DISCUSSION
The proposed diagnosis was applied in Federal
Institutes of Education, Science, and Technology
(FITs), institutions created by law 11.892/08 to
promote basic, professional, and higher education,
but also to offer education in different teaching
modalities (Santos Filho, 2020). Each institute is led
by a hierarchy of directors made up of rector, pro-
rector, directors, coordinators, teachers, and
Table 2: Coordinator/Manager Questionary.
INFRASTRUCTURE
A1 The course environment (classroom, meeting room,
etc.) and communication tools (email, instant message,
groupware, etc.) facilitate interaction and collaboration
of students.
A2 The course environment provides free space and
sufficient time for self-learning by the student.
A3 There are sufficient materials, technological resources,
and systems in the course to assist teachers during
learning.
A4 Teachers demonstrate sufficient skills and capacity to
perform activities as tutors.
A5 Teachers play a role as facilitators of learning and
assist in the process of learner autonomy and
independence.
A6 Tutors are available in the course to provide group
mentoring.
A7 There is the role of the real client (in general, external
collaborators), participating in the teaching and
learning process, providing problems to be solved, and
accompanying their resolutions.
A8 In the course, there are trained collaborators available
to help in the tutoring process.
POLITICS
A9 The institution usually conducts training for teachers
and teaching staff for possible evolutions in the
teaching process.
A10 The course involves the participation of companies in
academic projects, to make the teaching and learning
process closer to reality.
A11 In the student selection process, criteria such as
interpersonal, autonomy, and creativity are considered,
such as teamwork skills and student proactivity.
A12 Teachers easily plan, implement, and evaluate their
courses.
A13 There is resistance from teachers when there are
changes in curriculum or pedagogical methodology.
A14 There is resistance from students when there are
changes in curriculum or pedagogical methodology.
A15 There is a culture of "learning by doing" in the
institution, stimulating professional practice.
A16 Class planning is carried out collaboratively, involving
the pedagogical team (e.g., teachers, tutors, real
client).
A17 The institution's budget includes resources for
practical approaches and continuous monitoring.
Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education
419
Table 2: Coordinator/Manager Questionary (cont.).
CURRICULUM
A18 The adaptation of the course curriculum to a new
pedagogical methodology may occur if it is necessary.
A19 The course brings interdisciplinarity and/or
multidisciplinarity to the classroom.
A20 The curriculum is flexible, providing a consistent body
of basic knowledge and autonomy to the student in the
choice of their specialties.
A21 The class has a flexible schedule, being able to adjust
according to the teacher's needs.
A22 There is a lack of alignment between the class load
and the content to be taught by the teacher.
A23 The course curriculum encourages problem-solving
and self-directed learning.
A24 The course is based on a curriculum that stimulates
interdisciplinary and/or multidisciplinary practice.
EVALUATION
A25 During the course, there is continuous monitoring and
feedback from teachers to students.
A26 There is effectiveness in the course evaluation process
through various evaluation criteria.
A27 The institution performs evaluations on the
pedagogical methodology and the level of tasks
required by the teachers.
A28 The evaluation system provides feedback on the
teachers' work and the students' learning.
A29 The course's subjects allow for evaluations of the
learning process and self-assessments of students.
A30 The institution performs evaluations regarding
content, process, performance, satisfaction, and results
generated from the course.
administrators. With a multi-campus structure of
more than 661 units, there are 38 FITs distributed in
various regions of Brazil based on professional and
technical education. Although not all campuses offer
courses in the area of computing, there are several
courses in this area such as Computer Technician,
Information Technology Management, Information
System Analysis and Development, Bachelor of
Computer Science, Databases, among others (Santos
Filho, 2020).
All 38 FITs responded to the survey, totalling 302
respondents (222 teachers and 80 coordinators).
Considering that teachers can teach in more than one
academic level, most of them work in technical (92%)
and higher (75%) courses, while a small portion of
them work in graduate (14%) and extension (22%)
courses. This scenario was similar to the coordinators
with technical (60%), higher (49%), graduate (3%),
and extension (8%). About 88% of the teachers stated
that they apply active methodologies in their classes,
while only 12% use traditional methodology. On the
other hand, 62.5% of the coordinators answered that
the institution where they work does not recommend
a teaching methodology, while 22.5% answered that
they recommend the active methodology and 15%
indicate the traditional methodology. As for
knowledge and experience in PBL, 57% of the
teachers and 50% of the coordinators stated that they
have median knowledge, with the level of experience
also median even lower (42% of the teachers and only
30% of the coordinators).
5.1 Results of the Teacher Survey
Figure 4 shows an overall result from the 222
teachers.
On theProblemaspect, most stated using real,
stimulating, and sufficiently complex problems as an
element of study (favorable), but were uncertain
about the appropriation of the problem-solving
process by the student (warning).
Figure 4: Teachers’ results.
It was also evident the lack of interaction of
students with real clients and users, compromising the
authenticity of the learning environment (critical
points). In PBL, real clients make it possible to build
effective solutions through interactions that help
students in the resolution process, providing feedback,
and evaluating partial results (Santos et al., 2020).
31,5%
12,2%
28,4%
32,9%
18,9%
7,7%
14,9%
11,3%
2,7%
5%
16,8%
13,6%
33,8%
23,1%
23,9%
16,3%
16,7%
22,2%
24,8%
6,3%
5,4%
26,2%
10%
6,8%
36%
24,3%
21,2%
5,5%
12,7%
10%
40%
29,7%
37,8%
40,1%
34,2%
10,8%
34,7%
23,4%
16,7%
25,5%
28,6%
41,6%
47,7%
37,1%
40,1%
34,4%
32,6%
34,8%
41%
12,7%
17,6%
27,2%
16,7%
17,6%
39,2%
35,6%
41%
24,5%
38,2%
36,1%
25,2%
38,7%
24,8%
21,6%
34,2%
25,7%
32,9%
40,5%
40,8%
37,3%
29,6%
32,5%
14,8%
23,5%
27,4%
32,6%
32,6%
27,2%
24,3%
19,4%
41,7%
26,3%
31,7%
21,3%
18,9%
27,5%
27,9%
42,7%
33,7%
43,4%
8,1%
14,4%
8,6%
5%
9,5%
28,8%
14,4%
16,7%
27,1%
23,6%
18,6%
10%
2,3%
14,5%
7,7%
14%
15,4%
12,2%
9,5%
32,6
27,6%
14%
29,4%
26,2%
4,5%
9,9%
8,1%
19,1%
13,6%
7,8%
0,5%
5%
0,4%
0,4%
3,2%
27%
3,1%
8,1%
12,7%
8,6%
6,4%
2,3%
1,4%
1,8%
0,9%
2,7%
2,7%
0,4%
29
7,7%
6,3%
12,2%
28,1%
1,4%
2,7%
1,8%
8,2%
1,8%
2,7%
0 102030405060708090100
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
A19
A20
A21
A22
A23
A24
A25
A26
A27
A28
A29
A30
Teachers' Evaluation
Totally Agree Partially Agree Neutral
Partially Desagree Totally Desagree
CSEDU 2021 - 13th International Conference on Computer Supported Education
420
Regarding the Environment aspect, teachers
partially agreed that the environment stimulates
social skills, solving problems required by the
professional market and fostering important concepts
(favorable), however, the authenticity of the
environment was again questioned, considering that
problems do not always reflect real market demands,
compromising the student's professional posture
(critical). According to (Bell, 2010), the PBL
promotes the construction of knowledge and skills
considered relevant to their professional practice,
therefore, an inappropriate environment can impact
this construction.
Concerning the Content aspect, as main
favorable points are the alignment between theory
and practice, integration between related courses,
content approaches using practical projects, and
appropriate subjects. Being institutions focused on
professional education, it is understood that the
course curricula have already been built for this
purpose. Two points indicated a warning sign:
flexibility of content in supporting the problem-
solving process and content that promotes dynamic
learning inside and outside the classroom. In PBL, the
content needs to be flexible and focused on the
problem-solving process (Krusche, 2016), with
teaching that is much more "learning to learn" than
"knowing a concept" that, far from practice, can
easily be forgotten.
Human Capital” was the most critical aspect.
Although teachers follow the resolution of problems
and stimulate group work with the formation of small
teams of students (favorable), many showed
uncertainty about the active posture and autonomy of
students. Critical points are the lack of participation
of the real client, a multidirectional relationship
where everyone learns from everyone and stimulation
of collaborative projects by the course coordinator.
Finally, the Process” aspect showed as favorable
points the content planning, evaluation, and
continuous monitoring by teachers and institutions.
However, warning signs for learning with
characteristics of metacognition and self-regulation
and a course planning that meets the expectations of
the students. The characteristics of metacognition and
self-regulation are present in the PBL approach,
enabling the student to perform self-reflection and
perform monitoring and evaluation of their learning
(Santos and Silva, 2018).
5.2 Results of the Coordinators Survey
Figure 5 presents the overall result of the
coordinators' responses.
In the "Infrastructure" aspect, the following are
highlighted as favorable points: the environment and
technological tools that facilitate interaction and
collaboration; a free space and sufficient time for self-
learning of the student; the existence of resources;
teachers with skills for tutoring activities and teachers
in the role of learning facilitators. The high bias in the
agreement of these points and the rate of uncertainty
in the first five assertions indicates a warning sign for
many institutions. Critical points were the absence of
tutors to support the teacher, collaboration with real
clients, and training in tutoring activities.
Concerning the "Politics" aspect, favorable points
are the realization of the training, ease in planning,
implementing and evaluating, and the culture of
"learning by doing" present in the institution.
However, once again the lack of interaction with the
labor market became evident, besides the absence of
a student selection process more appropriate to the
pedagogical approach and the collaborative planning
of courses.
Figure 5: Coordinators’ results.
It is also worth highlighting a warning regarding
the resistance of teachers and students to changes in
the curriculum or pedagogical methodology, and the
need for a planned budget for educational practices.
Regarding the Curriculumaspect, the following
points stand out as favorable: the adequacy of the
curriculum, an interdisciplinary and multidisciplinary
curriculum, and a flexible curriculum. However, the
33,8%
17,5%
12,5%
15%
16,3%
8,8%
3,8%
5%
12,5%
3,8%
13,8%
12,7%
20%
6,3%
11,3%
6,3%
5%
43,8%
17,5%
10%
6,3%
6,1%
3,8%
21,3%
13,8%
16,3%
3,8%
5%
10%
15%
40%
32,5%
40%
33,8%
40%
8,8%
8,8%
13,8%
23,8%
11,4%
17,5%
44,3%
17,5%
26,3%
43,8%
12,5%
16,3%
30%
41,3%
33,8%
11,3%
11,3%
28,8%
30%
37,5%
32,5%
28,8%
21,3%
38,8%
22,5%
21,2%
30%
27,4%
27,4%
32,4%
6,1%
10%
17,4%
30%
25,3%
26,2%
30,4%
38,7%
36,1%
30%
21,2%
41,1%
23,7%
30%
28,8%
22,3%
35%
34,9%
29,9%
29,9%
32,4%
21,1%
34,9%
31,2%
27,5%
3,7%
13,8%
18,8%
18,8%
10%
22,5%
23,6%
17,5%
23,7%
27,9%
7,5%
10,1%
13,8%
22,5%
12,5%
35%
21,3%
2,5%
8,8%
13,7
21,3%
31,3%
22,5%
16,3%
15%
13,8%
18,8%
13,8%
12,5%
20%
1,3%
6,2%
1,3%
5%
1,3%
53,8%
53,8%
46,3%
10%
31,6%
35%
2,5%
10%
8,8%
2,4%
25%
16,3%
2,4%
13,7
38,8%
16,3%
10%
2,5%
3,8%
5%
27,5%
25%
7,5%
15%
0 102030405060708090100
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
A19
A20
A21
A22
A23
A24
A25
A26
A27
A28
A29
A30
Coordinator's Evaluation
Totally Agree Partially Agree Neutral
Partially Desagree Totally Desagree
Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education
421
result also shows a warning about the curriculum's
ability to stimulate problem-solving and promote
self-directed learning. Also, the lack of flexible hours
and alignment of time to content was highlighted as
negative points.
Finally, as for the Evaluation aspect, the
following points stand out as favorable: monitoring
and feedback, effectiveness and evaluations of the
disciplines and the institution. Critical points were the
absence of feedback from the evaluation system on
the work of teachers and student learning, in addition
to the lack of evaluation of teaching methodology and
teacher performance.
5.3 General Discussion
Figure 6 presents an overview of the results from the
teachers' questionnaire.
Figure 6: Teachers’ results.
This figure shows a predominance of favorable
points on the Problem, Content, and Process axis,
with respect to the panorama of all the institutions
involved in the survey. Considering the education
purpose focused on professional performance, in
general, these institutions have worked with real and
relevant problems, with appropriate content for
problem-solving, and have a student evaluation
process. On the other hand, there was a greater
predominance of critical points in the Human Capital
axis, highlighting how impactful the PBL culture can
be in its adoption. Based on this understanding, PBL
training recommendations can be made regarding
each actor's responsibilities and roles in the learning
environment and the inclusion of new actors who can
make a difference, such as real clients and tutors. It is
worth noting the number of warning signs on the
Environment axis, indicating that there are still many
uncertainties regarding preparing the teaching and
learning environment for PBL in these institutions.
Figure 7 presents an overview of the questionnaire
results for the coordinators.
Comparing Figures 6 and 7, it can be clearly seen
how much the structural and organizational aspects
can negatively impact the PBL implementation,
having as main axes of concern the infrastructure and
the policy. Again, it became evident that it is
necessary to work on the institution's PBL culture,
involving educational managers and investing in
teacher training to prepare institutions for the radical
transformations that PBL brings. Other critical points
also highlight the need for flexibility of the content,
maintaining the alignment of theory with practice,
and a special concern with the continuous evaluation
process involving not only students, teachers, and
coordinators.
Figure 7: Coordinators’ results.
Finally, from this kind of overview, a discussion
with those involved in the diagnosis is recommended
with each institution to raise the points of
improvements and uncertainties, identifying and
managing the risks for implementing the PBL.
6 CONCLUSIONS
Based on the motivation of how to assess whether an
educational institution in the context of CHE is
prepared to adopt the PBL, this study proposed an
institutional diagnosis based on the pedagogical,
structural and organizational dimensions, evaluating
nine aspects with two groups of stakeholders:
teachers/tutors and coordinators/ managers. To
evaluate the model, this diagnosis was applied in 38
institutions of technical education in computing
through an opinion survey, obtaining an overview of
these institutions' situation with 302 respondents (222
teachers and 80-course coordinators in computing).
Due to these institutions' professional
characteristics, several favorable points were
identified, such as the use of real problems,
curriculum, and student evaluation. On the other
hand, critical points such as the academy's lack of
interaction with the labor market, teacher training,
and a more effective and ample assessment process
were also highlighted. In particular, some warning
CSEDU 2021 - 13th International Conference on Computer Supported Education
422
signs have also been identified regarding the
suitability of the learning environment for PBL and
the institutionalization of PBL culture.
The next steps are intended to make
improvements in questionnaires and apply them to
individual institutions, based on a careful analysis
with their stakeholders. It is also intended to apply
statistical methods for a more rigorous analysis.
ACKNOWLEDGEMENTS
The NEXT group thanks all the institutions that
participate in the survey and their respective
respondents, in addition to the group of experts who
evaluated and contributed to the production of the
survey questionnaires. Many thanks to all involved in
this study.
REFERENCES
Aldabbus, S, 2018. Project-Based Learning: Implementation
& Challenges. Int. Journal of Education, Learning and
Development, Vol.6, No.3, p. 71-79.
Alshaye, Tasir, Z. and Jumaat, N. F, 2019. The Conceptual
Framework of Online Problem-Based Learning
Towards Problem-Solving Ability and Programming
Skills. In IEEE Conference on e-Learning, e-
Management & e-Services (IC3e). IEEE, pp. 1-4.
Babori, A., Fassi, H.F, Hariri, A., Bideq, M., and Zaid, A.,
2016. Using problem based learning environment to
enhance algorithmic problem solving skill. In Global
Summit on Computer & Information Technology
(GSCIT). IEEE, pp. 60–65.
Barron, Lis; Wells, Lauren, 2013. Transitioning to the Real
World Through Problem- Based Learning: A
Collaborative Approach to Teacher Preparation. In
Journal of Learning in Higher Education.
Bell, S., 2010. Project-Based Learning for the 21st
Century: Skills for the Future. In The Clearing House,
83: 39–43, Taylor & Francis Group, LLC. ISSN: 0009-
8655print. DOI: 10.1080/00098650903505415.
DFID, 2003. Promoting Institutional & Organisational
Development. Department for International
Development, UK. ISBN 1 86192 535 2. Available in
http://www.kalidadea.org/castellano/materiales/evalua
cion/DFID%20promoting%20institutional%20develp
ment%20guide.pdf.
Enemark, S.; Kolmos, A.; Moesby, E., 2006. Promoting
and supporting PBL interests world wide: the profile of
the UICEE Centre for Problem Based Learning. In
Kolmos A. (Ed.), PBL at Aalborg University:
contributions to the International PBL Conference in
Lima July 17-24 (pp. 7-16).
Fink, F. K.; Enemark, S.; Moesby, E., 2002. UICEE Centre
for Problem Based Learning (UCPBL) at Aalborg
University. In: Proceedings of the 6th Baltic Region
Seminar on Engineering Education, Wismar, p. 6.
Herold, Robert, 2019. Problem based learning use in higher
education: a state university faculty study of pbl. Doctor-
ate in Education. California State University, Fresno.
Hmelo-Silver, Cindy E., 2004. Problem-Based Learning:
What and How Do Students Learn?. Educational
Psychology Review. 16 (3): 235–266.
Hsu, W.-C. and Lin, H.-C. K., 2016. Impact of applying
webgl technology to develop a web digital game-based
learning system for computer programming course in
flipped classroom. In EITT International Conference.
IEEE, pp. 64–69.
Ibrahim, N. and Halim, S. A., 2014. Generic framework
design of project-oriented problem-based learning
(popbl) for software engineering courses. In 8th.
Malaysian Software Engineering Conference
(MySEC). IEEE, 2014, pp. 359–364.
Jaryani, F., Zandi, B., Sahibudin, S., Salehy, S.,. Masrom,
M., and Zamani, M., 2010. Framework of a reflective
e-portfolio supported by outcome based education and
problem based learning. In Second International
Conference on Computer Research and Development.
IEEE, pp. 270–273.
Kemavuthanon, K., 2017. Integrated e-project
collaborative management system: Empirical study for
problem-based learning project. In 9th ICITEE
International Conference. IEEE, pp. 1–5.
Kitchenham, B. Pfleeger, S., 2008. Personal Opinion
Surveys. In: Guide to Advanced Empirical Software
Engineering. Springer, Cap. 3, p. 63-92.
Koray, O.; Koray, A., 2013. The effectiveness of problem-
based learning supported with computer simulations on
reasoning ability. In Procedia-Social and Behavioral
Sciences, 106, 2746 – 2755.
Krusche, S., Reichart, B., Tolstoi, P. and Bruegge, B., 2016.
Experiences from an experiential learning course on
games development. In Proceedings of the 47th ACM
Technical Symposium on Computing Science
Education. ACM, pp. 582–587.
Mäenpää, H., Varjonen, S., Hellas, A., Tarkoma, S, and
Männistö, T., 2017. Assessing iot projects in university
education: a framework for problem-based learning. In
Proceedings of the 39th International Conference on
Software Engineering: Software Engineering and
Education Track. IEEE Press, pp. 37–46.
Martins, V. F., De Almeida, I., Concilio, S., and De Paiva
Guimarães, M, 2018. Problem based learning associated
to the development of games for programming teaching.
In Computer Applications in Engineering Education, v.
26, n. 5. Scopus, 2018. pp. 1577-1589.
Melo, Roberto Correia, 2013. Estratégias de ensino e
aprendizagem baseadas em problemas (PBL) no ensino
tecnológico [Problem-based teaching and learning
(PBL) strategies in technological education]. In VIII
Workshop de Pós-Graduação e Pesquisa do Centro
Paula Souza, São Paulo.
Moesby, E., 2004. Reflections on making a change towards
Project Oriented and Problem-Based Learning
(POPBL).
Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education
423
Oliveira, A. M., Santos, S. C., Garcia, V. C.PBL in
Teaching Computing: An overview of the Last 15
Years,” presented at the IEEE Frontiers in Education
Conference (FIE), Oklahoma, OK, Oct. 2013, pp. 23-
26.
Panwong, P. and Kemavuthanon, K., 2014. Problem-based
learning framework for junior software developer:
empirical study for computer programming students. In
Wireless personal communications, vol. 76, no. 3, pp.
603–613.
Patton, M. Q., 2008. Qualitative Research & Evaluation
Methods. 3. ed. California: Sage Publications.
Ribeiro, L. R. D. C., 2008. Aprendizagem Baseada em
Problemas (PBL) na educação em Engenharia
[Problem-Based Learning (PBL) in Engineering
Education]. Revista de ensino de Engenharia, 27(2),
23-32.
Richardson, I., Reid, L., Seidman, S. B., Pattinson, B., and
Delaney, Y., 2011. Educating software engineers of the
future: Software quality research through problem-
based learning”. In 24th IEEE-CS Conference on
Software Engineering Education and Training
(CSEE&T). IEEE, 2011, pp. 91–100.
Santos, D. M. B., and Silva, C. A. S, 2018. Problem-Based
Learning in a Computer Engineering Program:
Quantitative Evaluation of the Students' Perspective. In
IEEE Latin America Transactions, v. 16, n. 7, Scopus,
2018, pp. 2061-2068.
Santos, S. C., Figueredo, C. O., and Wanderley, F., 2013.
PBL-test: A model to evaluate the maturity of teaching
processes in a pbl approach, in Frontiers in Education
Conference, IEEE, 2013, pp. 595–601.
Santos, S. C., Furtado, F., and Lins, W., 2014. xPBL: A
methodology for managing pbl when teaching
computing. IEEE Frontiers in Education Conference
(FIE) Proceedings. IEEE, pp. 1–8.
Santos, S. C., Reis P. B. S., Reis J. F. S, Tavares F., 2020.
Two Decades of PBL in Teaching Computing: A
Systematic Mapping Study, IEEE Transaction on
Education, pp. 1-15.
Santos Filho, O. C., 2020. Diagnóstico Institucional para
Adoção de PBL em Cursos de Computação: Uma
Pesquisa de Opinião em Institutos Federais de
Educação, Ciência e Tecnologia do Brasil (Institutional
Diagnosis for PBL Adoption in Computer Courses: An
Opinion Survey in Federal Institutes of Education,
Science and Technology in Brazil), Master thesis,
Informatics Center, UFPE.
Savery, J. R. and Duffy, T. M., 1994. Problem based
learning: An instructional model and its constructivist
framework.
Souza, S. C.; Dourado, L, 2015. Aprendizagem baseada em
problemas (ABP): Um método de aprendizagem
inovador para o ensino educativo [Problem-based
learning (PBL): An innovative learning method for
educational teaching]. Holos, v. 5, p. 182-200.
Srinivasan, Malathi; Wilkes, Michael; Stevenson, Frazier;
Nguyen, Thuan; Slavin, Stuart, 2007. Comparing
Problem-Based Learning with Case-Based Learning:
Effects of a Major Curricular Shift at Two Institutions.
In Academic Medicine, Volume 82 – Issue 1 - p 74-82.
doi: 10.1097/01.ACM.0000249963.93776.aa.
Yang, J.-Y. and Choi, C., 2017. Enhancing interactiveness
of problem-based programming course using
anonymous chatting service. In 16th International
Conference on Information Technology Based Higher
Education and Training (ITHET). IEEE, pp. 1–6.
Yu, W., 2005. Work in progress-a mobile computing
collaborative framework for problem-based learning
environment. In Proceedings Frontiers in Education
35th Annual Conference. IEEE, pp. F4E–12.
Zhao, L. and Liu, F., 2011. A new theoretical pbl model for
mis course design and research. In International
Conference on Management and Service Science.
IEEE, pp. 1–4.
APPENDIX
A rationale and literature references for Teachers’ and
Coordinators’ questionaries are available in:
https://docs.google.com/spreadsheets/d/1E0fKh-Nim
U-OAhCed-geJ8bKpH8QzgmtqKD1VQzUupo/edit?
usp=sharing
CSEDU 2021 - 13th International Conference on Computer Supported Education
424