Investigating the Relationship between Learners' Cognitive

Participation and Learning Outcome in Asynchronous Online

Discussion Forums

Zhi Liu

, Shiqi Liu

, Cuishuang Zhang

, Zhu Su

, Tianhui Hu

and Sannyuya Liu

1,2

National Engineering Laboratory for Educational Big Data, Central China Normal University,

Luoyu Road 152, 430079 Wuhan, China

National Engineering Research Center for E-Learning, Central China Normal University,

Luoyu Road 152, 430079 Wuhan, China

Keywords: Online Discussion, Learning Analytics, LDA Topic Modeling, Regression Analysis.

Abstract: In asynchronous forums of Blended Learning and E-learning, learners’ cognitive participation, such as

knowledge construction and critical-thinking dialogues, is a crucial factor for their learning outcome, which

has not yet been further exploited. This study investigated learners’ cognitive behaviors and implicit content

derived from posts by using a mixed-method of text mining and statistical analysis. We adopted a content

analysis approach to manual coding learners’ cognitive behaviors in a Blended Learning discussion forum.

Then we proposed an improved topic model called Cognitive Behavior Topic Model (CBTM) to detect

learner’s semantic content between three achievement groups (High/Medium/Low). Moreover, we performed

a statistical analysis to investigate the relationship among cognitive behaviors, cognitive content, and learning

outcome. The results showed that the high achievement group’s cognitive behavior frequency in all categories

is higher than the other, and effective order of behaviors with the learning outcome is “constructive > active

> interactive”. The “application practice” related topic is more effective for learning outcome than “theoretical

discussions”. Specifically, when the cognitive content changes from "theoretical discussion" to "application

practice", or the number of posts on the same cognitive content-related topic is large, the high-level cognitive

behaviors bound to the topic content will increase significantly. Therefore, this study could provide new

insights into theoretical and practical implications.

1 INTRODUCTION

Online Asynchronous forums in Blended Learning

and E-learning provide learners and educators with an

interactive learning environment for cognitive

participation such as Q&A and argumentation

(Almatrafi & Johri, 2019; Ezen-Can, Boyer, Kellogg,

& Booth, 2015). In those discussions, cognitive

participation refers to the information processing of

cognitive content by cognitive behaviors (Gerrig,

2012). Cognitive high-level participation is a crucial

factor that contributes to learning outcome (Chi &

Wylie, 2014). In recent years, researchers have

studied the effects of conversational cognitive

behaviors (Galikyan & Admiraal, 2019; Wang, Wen,

& Rosé, 2016; Wang, Yang, Wen, Koedinger, &

Rosé, 2015) on E-learning asynchronous forums.

However, cognitive behaviors and cognitive content,

as two closely related components, have not been

used by association modeling to analyze the impact

of cognitive participation on learning outcome,

especially in blended learning environments where

learners are more connected. Moreover, the present

cognitive content analysis requires a large amount of

manual manipulation (Atapattu, Thilakaratne,

Vivian, & Falkner, 2019), lacking automated

algorithms to analyze the cognitive participation

process, which has a negative impact on large-scale,

real-time, automatic cognitive detection and

intervention. the large amount of discussion data

generated by learners in blended learning

asynchronous forums provides the basis for research.

The purpose of this research is to design a

Cognitive Behavior Topic Model(CBTM) based text

mining algorithm to automatically analyze cognitive

behaviors and cognitive content in the asynchronous

forum, and further study the relationship between the

Liu, Z., Liu, S., Zhang, C., Su, Z., Hu, T. and Liu, S.

Investigating the Relationship between Learners’ Cognitive Participation and Learning Outcome in Asynchronous Online Discussion Forums.

DOI: 10.5220/0009338900260033

In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 2, pages 26-33

ISBN: 978-989-758-417-6

two factors and the learning outcome, to capture

higher-order cognition participation in providing

insights. It will provide effective algorithms and

suggestions for online learning discussions. The

organizational structure of this article is as follows:

we review related works on discourse cognitive

behaviors, content, and behavior-topic models in

Section 2. The study design and methodology are

described in Section 3. The analysis results and

findings of the study are presented in Section 4.

Section 5 summarizes and discusses the findings of

this study.

2 RELATIONS WORKS

Interactive-Constructive-Active-Passive (ICAP) (Chi

& Wylie, 2014) framework presents a guide for

evaluating the cognitive engagement of contributions

within online communities. Although initially

developed to understand classroom conversational

data, this framework has also been proved to be

effective in online environments to understand

learners' cognitive participation, which has been

utilized within MOOC studies to measure the

association between course materials and discussion

contributions (Wang et al., 2015).

As an explicit form of learners' thinking and

knowledge processing, interactive discourse is an

essential basis for discriminating learners' cognitive

patterns, knowledge construction levels, and

independent inquiry ability (Wang et al., 2015).

SPOC, as a kind of restricted learning community,

has produced a large number of cognitive discourse

samples related to learning behaviors in its forum. It

is a crucial carrier reflecting the knowledge

construction, cognitive strategies, and interactive

quality of learners. A large number of studies have

laid the foundation for the analysis of cognitive

discourses in MOOC. For example, Wang et al.

(2015) adopted a content analysis approach to analyze

learners' cognitively appropriate behaviors in a

MOOC discussion forum and further explored the

relationship between the quantity and quality of that

participation with their learning gains. To interpret

what kind of discussion behaviors could help learners

from the semantic level. Wang et al. (2016) proposed

to trigger the cognitive behaviors of higher-order

thinking through ICAP cognitive coding scheme, and

found that learners who displayed more higher-order

thinking behaviors learned more through more in-

depth engagement with course materials posted by

their discussion behaviors. Moreover, many

researchers investigated the relationship between

cognitive behaviors and learning outcomes in terms

of quantitative and probabilistic models. Galikyan &

Admiraal. (2019) verified the effectiveness of

individual learner’s cognitive behaviors in predicting

learner’s learning effect through multiple regression

analysis in an asynchronous discussion community

and discussed the complex dynamics of knowledge

construction in pre-service teacher education.

Atapattu et al. (2019) proposed a fusion neural word

embedding (Doc2Vec) model to automatically

identify teachers' cognitive participation in MOOC

communities, such as active participation,

constructive participation, etc. They explored the

content of constructive cognitive involvement in 67

cases. Recently, some studies did excellent works in

modeling cognitive behavior from the perspective of

the topic model. For example, Qiu et al. proposed an

LDA-based (Blei, Ng, & Jordan, 2003) behavior-

topic model, which combined the users’ subject

interest and behavior patterns. They proved that the

model could obtain more dominant behavioral

indicators (Qiu, Zhu, & Jiang, 2013). Peng et al.

proposed Behavior Emotion Topic Model (BETM) to

detect reviews' semantic content between two

achievement groups (completers and non-

completers), the results showed that posting

comments is an essential behavior of the completers,

while replying and giving peer-review"praise"

arenecessary behaviors of the non-completers (Peng

& Xu, 2020). The above researches on behaviors and

discussion topics only focus on the semantic

modeling of shallow behaviors (such as approval,

replying, posting, and other discourse operations) and

seldom on the semantic modeling of the content of

discourse behaviors from the cognitive level. In

addition, the existing research has not explored the

relationship between cognitive behavior and learning

outcomes in the forum of hybrid courses, nor has it

used automated methods to analyze the impact of

discourse content themes and cognitive behavior on

learning outcomes. Thus, in order to fix the shortage

of automatic semantic analysis of discourse cognitive

behaviors in the field of learning analytics, we

propose the CBTM-based text mining method, which

not only models the semantic content with the topic

factors from the cognitive level of discourse

(compared with the shallow behavior of BETM,

CBTM is a discourse behaviors encoded by ICAP

framework), but also conducts a specific group

dialogue-oriented cognitive behavior analysis method

through topic probability modeling.

Investigating the Relationship between Learners’ Cognitive Participation and Learning Outcome in Asynchronous Online Discussion

Forums

3 RESEARCH METHODS

3.1 Research Questions

Asynchronous forum discussions can support

learners’ cognitive participation activities such as

knowledge construction and critical-thinking

dialogues. To promote learners' high-order cognitive

participation in the forum and improve learning

effect, it is necessary to understand the topic content

corresponding to cognitive behaviors and their

relations to learning achievement. This study will

focus on the following issues:

(1) What is the relationship between different

categories of cognitive behaviors and the learning

outcome?

(2) What is the relationship between different

types of the cognitive content-related topic and the

learning outcome?

(3) What is the relationship between the cognitive

content-related topic and cognitive behaviors?

3.2 Dataset

The forum data set is retrieved from the second-year

undergraduate course "Introduction to Psychology" in

the fall of 2014 offered by a university from China.

This course serves as an introductory course in

psychology. The first goal is to enable college

students to understand the knowledge system of

psychology, to master the basic concepts and

principles of psychology, and to use the scientific

knowledge of psychology to understand and analyze

people's psychological phenomena. The second goal

is to enrich the knowledge structure of college

students, help learners to better understand

themselves and self-improvement, and enhance the

psychological quality of college students. The third

goal is to improve students' teaching ability and

quality of daily life. Learners can participate in

discussions initiated by instructors in online forums

after class. They can also initiate and participate in

discussions freely in Chinese. The issues discussed

include highly specialized “theoretical discussions”

discussions and “application practice” discussions.

This course offers ten teaching classes, with a total of

490 learners participating.

Learners in this course are in non-psychological

majors (e.g., literature, philosophy, and sports). We

assume that their initial level of psychological

knowledge is the same, so they are not pre-tested. The

total score of this course is 100 points. First, online

learning accounts for 15% of the time. Second, the

participation in asynchronous forum discussions and

electronic assignments accounted for 30%. Third, the

counts and quality of face-to-face discussions in

offline classrooms accounted for 25%. Fourth, the

final exam account for 30%, and the content of the

exams is about the understanding and application of

basic psychological knowledge. The performance

evaluation aims to assess learner participation and

knowledge from online and offline, respectively, and

can effectively assess learners’ learning outcomes.

The average learner score is 84.4 points, the

highest score is 97 points, the lowest score is 32

points, and the standard deviation is 6.9. For the

difference analysis (Kelley, 1939), we marked the

first 29% (n = 144) of the score as the high

achievement group, the medium 54% (n = 214) as the

medium achievement group, and the last 27% (n =

132) as the low achievement group.

Table 1: Coding rule of Cognitive Behaviors.

Cognitive Behavior Coding rule

Example

Active

Learner repeats or explain or cite

information that is existed in the

textbook.

心理学是研究心理和行为的科学。

/Psychology is the science of psychology

and behavior.

Constructive

Learners ask new questions or express

new ideas or compare examples.

我觉得创造力是可以被培养的，但可能也

有一定程度是遗传的。/I think creativity

can be cultivated, but it may also be inherited

to a certain degree.

Interactive

Learner acknowledgment or debate

peer’ contribution, or build a new idea

from it.

同意你的看法，我觉得一个人的性格应该

是先天和后天两者结合影响的。/I agree

with you, I think a person's personality should

be influenced by both innate and acquired.

CSEDU 2020 - 12th International Conference on Computer Supported Education

Learners’ postings generated a total of 7,574

discussions about learning content. Their maximum

number of posts is 158, the minimum number of posts

is 1, the average number of posts per person is 15.0,

the standard deviation is 22.3, and the average length

of each post is 141.5 words.

3.3 Research Data Analysis

3.3.1 Cognitive Behaviors Coding

The forum discussion reflects the cognitive behaviors

and content of the learners. As for the first research

question, we studied the cognitive behaviors in the

forum and analyzed their frequencies and relationship

with learning achievement. we used coding rules that

are adjusted from the IACP framework by Wang et

al. (2015), which were verified useful for research

about the MOOC forum. The simplification coding

rules are shown in Table 1. Then, we invited three

graduate learners majoring in education technology to

learn the coding rules and randomly select 50 posts to

conduct coding of cognitive behaviors. The

consistency Kappa coefficient reached 0.69. Next,

through discussion and clarification on some

ambiguous samples, their differences were resolved

through the discussion. After the three coders reached

a consensus, they simultaneously coded 7574

discussion texts related to the learning content.

3.3.2 Cognitive Behavior Topic Model

Before text modeling, we used HanNLP (hankcs,

2019) for Chinese word segmentation. At this time,

named entity words in the field of psychology were

added to the user dictionary. Finally, removing stop

words, the text is used as the input data for the model.

In the Cognitive Behavior Topic Model (CBTM),

the cognitive behavior is considered as a factor that

drives a topic. For a large-scale data set, researchers

or instructors tend to understand the topics discussed

by learners and corresponding cognitive behavior

patterns.

As is shown in Figure 1, the modeling process of

CBTM is as follows: we assume that when writing a

discussion post, each learner will choose a topic of

cognitive content (such as human motivation), then

choose a cognitive behavior processing pattern (such

as construction) for the content. This post is

composed of a series of words and semantic content

that fits the topic and behavior in a post (e.g., how to

enhance children's intrinsic motivation in teaching?).

Among them,





represents the Dirichlet prior

parameters of the author-topic distribution



, topic-

words distribution



, and topic-cognitive behaviors

distribution



. According to the conditional

dependencies of Bayesian networks, the joint

probabilities of topics, words, and cognitive

behaviors can be described formally in equation (1).

Figure 1: Generative process of CBTM.









,, | , , |,, , , (|,)(| )pwzc pwzc pcz pz



   



(1)

We used the Gibbs sampling method to estimate

the hidden variables in the CBTM. The parameters

are estimated as follows in equation (2).

lt tw

nT n V





































(2)

In order to solve the research question 1, we

conducted a multiple regression analysis of cognitive

behaviors and the learning outcome (The final score),

analyzed the cognitive behaviors differences between

different achievement groups, and obtained the

influence of each of three cognitive behaviors on the

learning outcome. To solve the research questions 2

and 3, we developed a cognitive behavior text mining

model to calculate the cognitive content-related topic

probability of each learner and then performed the

multiple regression analysis and difference analysis

to uncover the relationship between these topic

probabilities and learning outcome. Futhremore, the

cognitive content-related topic differences were

investigated in terms of different achievement groups.

Finally, we examined the differences between these

cognitive-behavior patterns corresponding to content-

related topic, as shown in Figure 2.

When the number of posts related to the cognitive

content topic increases, the proportion of higher-order

cognitive behaviors (interactive and constructive)

also increases. T11 is related to “motivation and

emotion.” Its proportion of higher-order cognitive

behavior is 74.8%, and its number of posts is 229. T2

is related to “the personality of children.” Its

proportion of high-order cognitive behaviors is

94.8%, and its number of posts is 859. For other

topics, as the number of posts on the same cognitive

content-related topic increases, the proportion of

higher-order cognitive behaviors also increases.

As the semantic of topic is trandferred from

"theoretical discussion" to "application practice," the

proportion of higher-order cognitive behaviors would

gradually increase. T2, T13, and T28 are about the

application of psychology in teaching. The average

proportion of higher-order cognitive behaviors is

93.2%. T20, T17, and T11 are about the terms of

psychological

theory, and the average proportion of

Table 4: Differences of significant topics between different achievement groups.

Topic

label

Cognitive content topic probability

Post-hoc test

HAG MAG LAG

Mean SD Mean SD Mean SD

T13

0.053

0.035 0.047 0.023 0.036 0.014

HAG > LAG

***

HAG> MAG

***

T11

0.034

0.012 0.032 0.010 0.031 0.006

HAG > LAG

***

HAG> MAG

***

T28

0.035

0.012 0.032 0.008 0.032 0.008

HAG > LAG

***

HAG> MAG

***

T17

0.035

0.011 0.033 0.014 0.031 0.008

HAG > LAG

***

HAG> MAG

***

(p<0.001

***

, p<0.01

, p<0.05

)

Investigating the Relationship between Learners’ Cognitive Participation and Learning Outcome in Asynchronous Online Discussion

Forums

Figure 2: Behavior Model of Cognitive Topic.

higher-order cognitive behaviors reaches 72.1%. The

proportion of higher-order cognitive behaviors

corresponding to the topic of " application practice "

is significantly larger than those corresponding to "

theoretical discussion."

5 SUMMARY AND DISCUSSION

In this study, we designed a cognitive behavior topic

model to analyze the relationship between cognitive

behaviors, cognitive content, and learning outcome in

discussion texts of asynchronous learning forums,

and reached the following conclusions:

To answer question 1, in a blended learning

environment, Constructive behavior have a greater

impact on the learning outcome than active and

interactive. Learners in the high achievement group

took all categories of cognitive behaviors. This

conclusion in the Blended Learning environment is

consistent with Wang’s results in the E-learning

environment(Wang et al., 2015) . But it is different

from ICAP theory(Chi & Wylie, 2014), i.e., learning

outcomes related to interactive behaviors are greater

than that related to other cognitive behaviors.

Through the results of instructional design and text

mining, we can infer that the first reason may be that

Wang et al. defined knowledge test scores as learning

outcomes, and our learning outcomes were mainly

indicated by learning activity participation and

knowledge understanding. None of these studies have

used critical thinking and interactive skills as teaching

goals (they are related to interactive ones). The

second reason may be that the cognitive content

related to high interactive behaviors involves the

basic knowledge points, not the content of the

examination. Therefore, educational discourse

analysis needs to be combined with educational goals

and instructional design, otherwise the causality of

the teaching process cannot be efficiently understood.

The reason might be that the interactive has fewer

times of occurrence, or the related discussed content

is only a regular knowledge point.

To answer question 2, The “application practice”

topic related has a greater impact on the learning

outcome than the “theoretical discussions” topic

content. The high achievement group seems to pay

more attention to the topic content related to the

“application practice” of psychology, and the general

topic has the highest regression coefficient to learning

outcome, indicating that the appropriate application

of knowledge in discussions can effectively promote

the learning outcome. One of the goals of this course

is to apply psychological knowledge to work and life,

and the results of the automatic text mining algorithm

are consistent with the teaching goals, so the

algorithm can be used as an effective component of a

learning management system. On the other hand, by

only counting cognitive behavior, it is difficult to

reflect the level of a learner's cognitive participation

fully.

To answer question 3, When the cognitive content

changes from "theoretical discussion" to "application

practice," or the number of posts on the same

cognitive content-related topic is increased, the high-

level cognitive behavior bound to the topic content

will increase significantly. Existing studies cannot

calculate the cognitive behavior patterns of cognitive

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

(n=859)

the personality

of children

T13

(n=755)

Study and

work with

psychology

T28

(n=276)

intelligence

and creativity

T20

(n=275)

Mood and

behavior

T17

(n=265)

Negative

reinforcement

of behaviorism

T11

(n=229)

motivation and

emotions

Active Constructive Interactive

CSEDU 2020 - 12th International Conference on Computer Supported Education

content using the original Lda(Ezen-Can et al., 2015;

Wang et al., 2016), and the semantic probability of

cognitive content cannot be obtained based on the

LIWC dictionary method (Moore, Oliver, & Wang,

2019). In this study, cognitive behavior and content

are jointly modeled, which can effectively provide

teachers with timely and profound dialogue analysis

results. Therefore, the content of the instructor-

directed discussions needs to be adjusted to adapt to

the learners' future work to promote higher-order

cognitive participation and learning achievements.

However, there are still some limitations to this

study. First, there may be some imbalances between

different types of cognitive behaviors within the

discussion posts due to specific instructional and

interactive design and forum activity settings.

Moreover, learning is an information-processing

process, and future work also needs to consider the

evolution of cognitive behaviors and discussed

content in terms of the time dimension.

ACKNOWLEDGMENTS

This work was supported by the Research Funds from

National Natural Science Foundation of China (grant

number: 61702207, 61937001, 61977030,

31600918), National Key Research & Development

Program of China (grant number:

2017YFB1401303),Hubei Provincial Natural Science

Foundation of China (grant number: 2018CFB518).

REFERENCES

Almatrafi, O., & Johri, A. (2019). Systematic Review of

Discussion Forums in Massive Open Online Courses

(MOOCs). IEEE Transactions on Learning

Technologies, 12, 413–428.

Atapattu, T., Thilakaratne, M., Vivian, R., & Falkner, K.

(2019). Detecting cognitive engagement using word

embeddings within an online teacher professional

development community. Computers & Education,

140, 103594.

Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent

Dirichlet Allocation. The Journal of Machine Learning

Research, 993–1022.

Chi, M. T. H., & Wylie, R. (2014). The ICAP Framework:

Linking Cognitive Engagement to Active Learning

Outcomes. Educational Psychologist, 49, 219–243.

Ezen-Can, A., Boyer, K. E., Kellogg, S., & Booth, S.

(2015). Unsupervised modeling for understanding

MOOC discussion forums: A learning analytics

approach. Proceedings of the Fifth International

Conference on Learning Analytics And Knowledge -

LAK ’15, 146–150. Poughkeepsie, New York: ACM

Press.

Galikyan, I., & Admiraal, W. (2019). Students’ engagement

in asynchronous online discussion: The relationship

between cognitive presence, learner prominence, and

academic performance. The Internet and Higher

Education, 43, 100692.

Gerrig, R. J. (2012). Psychology and Life. Pearson.

hankcs. (2019). Hankcs/HanLP [Python]. Retrieved from

https://github.com/hankcs/HanLP

Kelley, T. L. (1939). The selection of upper and lower

groups for the validation of test items. Journal of

Educational Psychology, 30, 17–24.

Moore, R. L., Oliver, K. M., & Wang, C. (2019). Setting

the pace: Examining cognitive processing in MOOC

discussion forums with automatic text analysis.

Interactive Learning Environments, 1–15.

Peng, X., & Xu, Q. (2020). Investigating learners’

behaviors and discourse content in MOOC course

reviews. Computers & Education, 143, 103673.

Qiu, M., Zhu, F., & Jiang, J. (2013). It Is Not Just What We

Say, But How We Say Them: LDA-based Behavior-

Topic Model. Proceedings of the 2013 SIAM

International Conference on Data Mining, 794–802.

Society for Industrial and Applied Mathematics.

Wang, X., Wen, M., & Rosé, C. P. (2016). Towards

triggering higher-order thinking behaviors in MOOCs.

Proceedings of the Sixth International Conference on

Learning Analytics & Knowledge, 398–407. ACM

Press.

Wang, X., Yang, D., Wen, M., Koedinger, K., & Rosé, C.

P. (2015). Investigating How Student’s Cognitive

Behavior in MOOC Discussion Forums Affect

Learning Gains. International Educational Data

Mining Society.

Investigating the Relationship between Learners’ Cognitive Participation and Learning Outcome in Asynchronous Online Discussion

Forums