Assessing Engineering Students’ Stress: Implementing the

Psychometric Synonym Technique

Abdul Saman

, Muhammad Jufri

and Hillman Wirawan

Department of Counseling, Faculty of Education, Universitas Negeri Makassar, Jl. Tamalate, Makassar, Indonesia

Department of Psychology Universitas Negeri Makassar, Jl. Mapala, Makassar, Indonesia

Department of Psychology Universitas Hasanuddin, Makassar, Indonesia

Keywords: stress, careless response, psychometric synonym, reliability, and data screening

Abstract: This study aims at investigating the engineering student’s stress at a vocational-technical school and

incorporating a psychometric synonym technique to screen student’s careless responses. This study

introduced the implementation of data screening technique to identify the students with careless or effortless

responses. Participants were 31 students (74.19% male) enrolling in the first and second year at the

Vocational-Technical high school. The 33-item Likert-type stress scale was administered to the participants.

The scale was well-constructed, and it satisfied the validity and reliability of an acceptable measure. The

results suggested that the stress level varied with a mean score of 69.13 (SD= 13.20). However, 15

participants showed low personal reliability index (r< .22) and some of them had personal reliability with a

negative value. These findings suggested that half of the participants potentially completed the

questionnaire with careless responses.

1 INTRODUCTION

Stress has been studied extensively by many

researchers across scientific fields such as

psychology, education, counselling, and

management. Many of the scientists found a great

effect of stress on human performance including

student’s academic achievement. For example, in

one study, 87% of students experienced academic

stress, and their stress levels were negatively

correlated with their academic achievement (Liu and

Lu, 2011). Meaning, the higher their stress level, the

more likely the students experienced poor academic

achievements.

As the parts of an educational process, learning

and teaching processes trigger some degree of stress.

Stress becomes one of the important variables in the

area of education as it contributes significant effect

on student psychological state. For instance, stress

influences smoking behavior among adolescences

(Unalan et al., 2008), young athlete performance

(Nicholls et al., 2009), and academic achievement

(Liu and Lu, 2011). Stress also influences the way

students work toward their goal. However, on the

other side, the student’s resilience within the process

also determines how they perceive stress (Gerber et

al., 2013). In brief, the learning process within

classroom potentially trigger students’ academic

stress, and this condition may contribute to other

important factors in education.

Technical-vocational schools run different

education system where the students are required to

complete technical-based skill modules. The

education curriculum is designed to foster the

student’s knowledge and skills. Regarding

workforce, the students are expected to fulfill the

needs of industries and organization. The education

focuses on shaping employability skills which

include personal qualifications and technical skills

(Bakar and Hanafi, 2007). However, the students in

the technical-vocational training and education also

experience academic stress which might hinder their

academic performance and lead to counterproductive

behavior (Unalan et al., 2008). Stress occurs in

everyday human life; it serves as motivation for

growth but damage if ineffectively managed

(Zitzow, 1992).

Experts, scientists, and practitioners in this

regard have developed various approaches to

assessing student’s stress as well as designing

appropriate interventions. One of the widely used

interventions is mindfulness (Brown and Ryan,

Saman, A., Jufri, M. and Wirawan, H.

Assessing Engineering Students’ Stress: Implementing the Psychometric Synonym Technique.

DOI: 10.5220/0008591805430547

In Proceedings of the 3rd International Conference on Psychology in Health, Educational, Social, and Organizational Settings (ICP-HESOS 2018) - Improving Mental Health and Harmony in

Global Community, pages 543-547

ISBN: 978-989-758-435-0

543

2003) and stress management (Unalan et al., 2008).

These two interventions were relevant to treat

students with moderate-to-high academic stress. On

the other hand, although many alternative

interventions are currently offered, identifying stress

level is another important task for teachers and other

practitioners. They are expected to bring valid and

reliable test results before the intervention.

Unfortunately, assessing the stress level requires

a systematic and professional approach. Although

psychometric practitioners and scale developer can

provide a robust measure for stress, they still have to

ensure that the participant provides only genuine

responses. In this case, a student may allegedly

complete the questionnaire (or survey) with

insufficient efforts or provide the assessor with

careless responses.

Using a self-report questionnaire in assessing

student’s stress provides many advantages for school

counsellor if used with careful supervision (Zitzow,

1992). Unfortunately, the students may disregard the

assessment or commit to careless responses. This is

an issue for both individual and classical assessment.

School counsellor or Psychologist could misinterpret

the data and consequently deliver incorrect

treatments.

To tackle this issue, therefore, many scientists

propose data screening techniques to identify

careless or effortless responses (Please see Curran,

2016; Desimone et al., 2015; Huang et al., 2015;

Meade and Craig, 2012). Researchers will be able to

detect careless responses or insufficient effort by

using the data screening technique like psychometric

synonym or bogus item technique (Desimone,

Harms and Desimone, 2015).

The Psychometric Synonym Technique assumes

that respondents do not appreciably change during

the course of assessment administration (Desimone,

Harms and Desimone, 2015). Thus, the respondent

who shows inconsistent responses over the

conceptually similar items should be treated as the

unreliable respondent. Desimone and others (2015)

suggested to identify the items that are conceptually

and statistically similar using a correlation

technique. Item pairs with correlation coefficient 0.6

or higher are defined as psychometrically synonym

(Meade and Craig, 2012).

A set of the synonym item pairs is used to

discriminate between effortful and effortless

respondents. Each respondent is assessed to detect

the correlation between the first and second set of

the items. Positive and higher correlations

coefficient indicate a reliable respondent. Meade and

Craig (2012) suggested a correlation coefficient

higher than 0.22 as the cut-off score. The personal

reliability index lower than 0.22 indicates careless

responses or an unreliable respondent.

Many previous studies focused on identifying

student’s stress either in individual- or group-unit

analysis. The self-report assessment technique was

ubiquitous among researchers (Alkhateeb, 2014;

Schwarz, 1999; Zitzow, 1992). The self-report stress

scale potentially reduces the validity of the

assessment procedure. As mentioned earlier,

students might contribute insufficient efforts or

respond carelessly to the items. This issue also

occurs when assessing engineering student’s stress.

Therefore, employing robust data screening

technique(s) will assist practitioners as well as

scientists in implementing a better self-report

assessment.

This study consists of two main parts. The first

study focuses on assessing engineering student’s

stress in the technical-vocational school. The second

study is designed to detect the participant’s

psychometric synonym index. In the end, this

provides practical implication of the psychometric

synonym technique for a self-report assessment.

2 METHOD

2.1 Participants

Participants were engineering students who enrolled

in technical-vocational education and training in one

of the vocational schools in Makassar, Indonesia.

The participants were 31 engineering students

randomly recruited from 305 students in the school.

Most of the participants were male (23, 74.19%)

with age ranged from 16 to 18 years old. The

participants were either in electrical engineering or

mechanical engineering program. Participants who

were under 18 years submitted permission from their

parents before participating in the study. This study

complied standard ethical codes for researching

participants under 18 years old. The participants had

rights to choose whether to participate or stop

completing the study at any time without any further

questions.

2.2 Measure

This study employed a 33-item stress scale with

Likert-type option. This scale was constructed by the

authors by collecting stress-related items from

various sources. The initial item pool consisted of

112 items and finally reached 48 items at the later

ICP-HESOS 2018 - International Conference on Psychology in Health, Educational, Social, and Organizational Settings

544

stage of the validation study. The scale was

constructed based on the guideline for measuring

non-cognitive variables (Hinkin, Tracey and Enz,

1997). The scale was constructed and administered

in Bahasa Indonesia by trained school counselor.

The response options ranged from 1 (strongly

disagree) to 5 (strongly agree). In the first

administration, the scale was a 48-item Likert type

scale, but 15 items were dropped due to lower inter-

item correlation (r< 0.30). The sample of the items

are “Saya tidak tertarik mengerjakan tugas sekolah

(I am not interested in doing school work)” and

“Saya mengkhawatirkan masa depanku (I am

worried about my future).” The initial 48-item had

.88 coefficient alpha and after dropping 15 items, the

alpha increased to .90. This was highly reliable

stress measure for research purpose. The final 33-

item scale was reliable and acceptable for research

purpose.

2.3 Procedure

The participants (n= 31) were asked to complete the

stress questionnaire. The scale was administered by

the school counselor in classroom using a paper-and-

pencil administration. The participants completed

the questionnaire in less than 30 minutes with mean

completion time was 20 minutes. The data were

collected and analyzed using reliability test (inter-

item correlation) and descriptive statistic technique.

In the next part of the study, the authors conducted

data screening to detect participant’s insufficient

efforts or careless responses. The psychometric

synonym technique (Desimone et al., 2015) was

implemented to the data to present a robust data

screening technique. In the end, the both results (i.e.,

before and after the data screening) were compared.

3 RESULTS AND DISCUSSION

3.1 Results

The participants completed the stress questionnaire

and the data were analyzed using descriptive statistic

technique. The following table 1 described the

descriptive statistics of the data.

Table 1: Descriptive statistics.

Variable

Academic stress 48 100 69.13 13.20

Note: N= 31, M= mean, SD= Standard Deviation,

in= Minimum,

ax= Maximum

The results showed that the participants had

stress score ranging from 48 to 100 with mean score

of 69.13 (SD= 13.20). Hypothetically, the score

might range from 33 to 165. The scores showed that

the participants exhibit a various degree of stress.

Students in the technical-vocational school, as well

as other students in regular schools, also have

dynamic academic stress levels.

In the next step of the analysis, the data were

screened using Psychometric Synonym technique.

The authors followed the guideline on how to run

Psychometric Synonym (please refer to Desimone,

Harms and Desimone, 2015 for more details). This

part of the analysis assisted the authors in

identifying participant’s insufficient effort in

completing the survey.

There were three main stages in conducting this

technique. First, the inter-correlations among the

items were computed. This yielded correlation

coefficients and the pairs of items that had r> .60

were included in the next step. Next, the authors

computed the correlation between the first set of the

items (item no. 13, 16, 18, 20, 23, and 33) and the

second set of the items (item no. 1, 4, 8, 9, 12, and

16). This step yielded correlation coefficients for

each participant in the survey. The next step, the

authors used the coefficients as the Psychometric

Synonym index. Participants who had a coefficient

index lower than .22 were considered as not having

enough effort to complete the questionnaire

(Desimone et al., 2015). The following table 2 listed

the first five participants in the list with their

Psychometric Synonym index:

Table 2: Psychometric synonym index

Partici

ants r index Decision

1 -0.5 Insufficient effort

2 0.59* Sufficient effort

3 -0.32 Insufficient effort

4 0.42* Sufficient effort

5 -0.71 Insufficient effort

= 31, *

>.22

The results suggested that 15 participants

completed the questionnaire with careless responses,

while 16 participants showed enough effort. Those

15 participants showed personal reliability index

lower than the acceptable cut-off score (r> .22) and

many of them had personal reliability with negative

value. The findings indicate that nearly half of the

participants responded to the questionnaire

carelessly or they completed the questionnaire with

Assessing Engineering Students’ Stress: Implementing the Psychometric Synonym Technique

545

insufficient effort. Therefore, the conclusion of the

students’ stress level could be misleading due to

insufficient effort in responding to each item.

3.2 Discussions

This study aimed at investigating the academic stress

among engineering students at the Technical-

Vocational School in Makassar, Indonesia. In

addition to the assessment, the authors also included

the implementation of Psychometric Synonym

technique as one of effective data screening

techniques. This study serves as an example on how

to assess student’s academic stress as well as

identifying their effort in completing the

questionnaire.

The results showed both common and surprising

findings. In the first stage of the analysis, the

descriptive statistic depicted normal description of

student’s academic stress. It is considered normal

that in one school the students or a group of students

have various level of academic stress. Like other

studies, students in many schools also experience

stress (Gerber et al., 2013; Liu and Lu, 2011;

Zitzow, 1992). In the second stage, the results

indicated that nearly half (15 out of 31) students

possibly had insufficient or carless effort in

completing the questionnaire. This technique is one

among many suggested data screening technique

(Desimone, Harms and Desimone, 2015; Meade and

Craig, 2012). Albeit this technique serves as a robust

and reliable guide for data screening, comparing two

or more data screening might provide better

information.

Education process – including learning and

teaching process – demands hard work and it

requires students to exert both their physical and

psychological energy. This, then, leads to stress

where the students should deal with all pressures in

order to performing well in completing academic

context. During this process, the students focus their

attention to both academic task and their

psychological constraint. Although most students

can cope and manage their stress, still many of them

fail in this process. This creates tremendous effect to

the academic performance and leads to poor

academic performance.

As mentioned earlier, a number of interventions

have been developed by experts. They intended to

assist students in dealing with their life stress as well

as academic stress. However, the major shortcoming

in implementing the interventions is assessing the

stress per se. The students may not realize the

importance of filling a stress questionnaire. As the

result, many of the students only submit careless

responses. In many cases, this will lead to poor

assessment results and potentially influence validity

and reliability of the measurement.

In this study, the engineering students were

highly influenced by their thoughts towards the

stress assessment. The results of this study suggested

that nearly half (15 out of 31) engineering students

failed at providing sufficient effort. Hypothetically,

their low psychometric synonym index (r< .22)

indicated that they might respond carelessly to each

item. This was in-line with many previous studies

where the measurement must be accompanied by

data screening technique (Desimone, Harms and

Desimone, 2015; Huang et al., 2012; Meade and

Craig, 2012). During the assessment process, the

students may not realize that the assessment was the

main information to initiate intervention for each

student. Failing in providing valid and reliable

measure leads to irrelevant interventions for

students.

This study brought a new concern on assessing

student’s stress, especially for classical

administration. Researchers and practitioners should

realize student’s insufficient effort to each item.

Using data screening technique provides extra

evidence for the assessment validity which later can

be used for interpreting the results. However, the

psychometric synonym technique is not the only

data screening technique. The data screening

techniques vary from the simplest one (e.g.,

including bogus item) to the more advanced

technique (e.g., personal reliability technique). In

addition, this study was also intended to show

teachers and practitioners the importance of valid

and reliable responses.

This study was able to implement the

psychometric synonym technique. Nevertheless,

there were some limitations related to the sample

size and the stress scale. First, the sample size was

considered small and may not represent the whole

population. The authors invited all students in the

school to participate. Unfortunately, only 31

participants who returned the questionnaire with

complete response. Although nearly half participants

were found to be careless, this does not conclude

that half of the student population at the school were

also careless.

Second, this study only employed one measure

(i.e., stress scale) to assess the student’s stress and to

detect any careless responses. This study does not

claim that the unreliable respondents will

consistently show careless or insufficient effort

ICP-HESOS 2018 - International Conference on Psychology in Health, Educational, Social, and Organizational Settings

546

across different measures. It requires different

measure or assessment to reach such conclusion.

Third, the stress scale used in this study still

needs further improvement. The authors had

computed validity and reliability test for the scale.

However, small sample size hindered the authors

from showing more evidence regarding validity. To

illustrate, it needs around 200 participants to

perform Confirmatory Factor Analysis (Myers, Ahn

and Jin, 2011).

4 CONCLUSIONS

This study was designed to assess the engineering

students’ stress at a vocational-technical school. In

addition, the data screening technique was also

included in identifying their true responses. The

results suggested that the students’ stress level was

normally distributed and it depicted that students

may have different level of stress across their

academic lives. Nevertheless, using the

Psychometric Synonym technique, this study also

found that students potentially responded to the

items carelessly or responded with insufficient

effort.

ACKNOWLEDGMENTS

Sincere thanks to the Government of Indonesia

through the Minister of Research, Technology and

Higher Education for providing research grant.

REFERENCES

Alkhateeb, H., 2014. Reading anxiety, classroom anxiety,

language motivation, reader self-perception, and

arabic achievement of arab-american students learning

arabic as a second language 1. Psychological Reports:

Sociocultural Issues in Psychology, 115(3), pp.918–

931.

Bakar, A.R. and Hanafi, I., 2007. Assessing Employability

Skills of Technical-Vocational Students in Malaysia.

Journal of Social Sciences, 3(4), pp.202–207.

Brown, K.W. and Ryan, R.M., 2003. The benefits of being

present: mindfulness and its role in psychological

well-being. Journal of personality and social

psychology, 84(4), pp.822–848.

Curran, P.G., 2016. Methods for the detection of

carelessly invalid responses in survey data. Journal of

Experimental Social Psychology, [online] 66, pp.4–19.

Desimone, J.A., Harms, P.D. and Desimone, A.J., 2015.

Best practice recommendation for data screening.

Journal of Organizational Behavior, 36(1), pp.171–

181.

Gerber, M., Brand, S., Feldmeth, A.K., Lang, C., Elliot,

C., Holsboer-Trachsler, E. and Pühse, U., 2013.

Adolescents with high mental toughness adapt better

to perceived stress: A longitudinal study with Swiss

vocational students. Personality and Individual

Differences, [online] 54(7), pp.808–814.

Hinkin, T.. R., Tracey, J.. B. and Enz, C.A., 1997. Scale

construction: Developing reliable and valid

measurement instruments. Journal of Hospitality and

Tourism Research, 21(1), pp.100–120.

Huang, J.L., Bowling, N.A., Liu, M. and Li, Y., 2015.

Detecting Insufficient Effort Responding with an

Infrequency Scale: Evaluating Validity and Participant

Reactions. Journal of Business and Psychology, 30(2),

pp.299–311.

Huang, J.L., Curran, P.G., Keeney, J., Poposki, E.M. and

DeShon, R.P., 2012. Detecting and Deterring

Insufficient Effort Responding to Surveys. Journal of

Business and Psychology, 27(1), pp.99–114.

Liu, Y. and Lu, Z., 2011. Longitudinal analysis of Chinese

high school student’s stress in school and academic

achievement. Educational Psychology, 31(1), pp.723–

729.

Meade, A.W. and Craig, S.B., 2012. Identifying careless

responses in survey data. Psychological Methods,

17(3), pp.437–455.

Myers, N.D., Ahn, S. and Jin, Y., 2011. Sample size and

power estimates for a confirmatory factor analytic

model in exercise and sport: A Monte Carlo approach.

Research quarterly for exercise and sport, [online]

82(3), pp.412–423.

Nicholls, A.R., Backhouse, S.H., Polman, R.C.J. and

McKenna, J., 2009. Stressors and affective states

among professional rugby union players.

Scandinavian Journal of Medicine and Science in

Sports, 19(1), pp.121–128.

Schwarz, N., 1999. Self-reports: How the questions shape

the answers. American Psychologist, 54(2), pp.93–

105.

Unalan, D., Celikten, M., Ozturk, A., Senol, V., Health,

H.B. and Mazicioglu, M., 2008. the Relationship

Between Vocational College Students ’ Smoking

Behavior and Self- Actualization , Interpersonal

Support and Stress Management.

Social Behavior and

Personality, 36(6), pp.721–726.

Zitzow, D., 1992. Assessing student stress: School

adjustment rating by self-report. School Counselor,

40(1), pp.20–23.

Assessing Engineering Students’ Stress: Implementing the Psychometric Synonym Technique

547