Validity of Processing Speed Ability Test based on Internal Structure

Fitri Andriani

, Cholichul Hadi

, Urip Purwono

, Siti Sulasmi



Faculty of Psychology, Universitas Airlangga

Faculty of Psychology, Padjajaran University

Keywords: Intelligence, Processing Speed Ability, CHC Theory

Abstract: This study constructed a new intelligence tests based on Cattel Horn Carrol theory (CHC theory). The aim is

to test some broad ability of the newly developed tests, namely processing speed ability, consisting of three

narrow abilities. In accordance with Cattel Horn Carrol theory, processing speed ability is one of the broad

abilities that contribute in shaping general intelligence. There are three narrow abilities of processing speed

ability, namely perceptual speed, number facility and rate of test taking. These abilities were studied

through 299 items and were piloted on 135 subjects. Through confirmatory factor analysis using the JASP

0.9 program, the model was found as a good fit (χ² = 0.341, p=0.559 (greater than 0.05), df =1, RMSEA =

0.000 (90% CI [0.000 - 0.0191], and those three factors (perceptual speed, number facility and rate of test

taking) were consistent with factors measured in processing speed ability tests. Based on the model

accuracy index, it shows that the proposed model is capable of describing the narrow ability of processing

speed ability.

1 INTRODUCTION

Cognitive ability has become an important subject in

psychology because it shows one of the individual

differences of concern by psychologists. Cognitive

ability can take the form of intellectual potential,

logical thinking ability, comprehension, analytical

ability, numerical ability, verbal ability and

intelligence etc. Scientists consider that intelligence

plays an important role in determining human

behavior. Intelligence is considered capable of

predicting individual success in careers

(Gottfredson, 2009; Kaufman et al., 2012), study

(Lohman and Gambrell, 2012; Lohman et al., 2008;

Naglieri et al., 2003; Vanderwood et al., 2001) and

survival (Vanderwood et al., 2001).

Based on its very important role, assessing (i.e.

knowing or measuring) the capacity of intelligence

in a person is important. Assessing someone's

intelligence is related to assessment activities that

ultimately relate to intelligence tests. A

psychological assessment activity is an attempt to

evaluate the individual who is facing the problem;

this activity explains that the information obtained

from the assessment that can be used to solve the

problem (Marnat, 2003). The assessment will be

related to the measurement activity which is a

systematic procedure for observing someone's

behavior and describing it with the help of a

numerical scale or categorization system.

Meanwhile, the test is one of the means of collecting

data on measurement activities. The test activity is

an objective and standard measurement of a

behavioral sample (Anastasi and Urbina, 1997).

Tests play an important role in psychological

decision-making regarding the fate of individuals,

groups and the wider community. However,

decision-making becomes less precise when the

available data is less accurate. Finally, the accuracy

of the data indicates the quality of assessment

activities that are influenced by the quality of a test

kit. While the quality of a test tool can be seen from

the evaluation that should be done before using the

test tool. One of the evaluations that can be done is

on the validity aspects of the test. This article

discusses the validity of the Speed Processing

Ability test as part of a common intelligence

developed based on the Cattel Horn Carrol theory

(CHC theory).

The Cattel Horn Carrol theory is the theory of

intelligence which is the integration of Gf-Gc theory

156

Andriani, F., Hadi, C., Purwono, U. and Sulasmi, S.

Validity of Processing Speed Ability Test based on Internal Structure.

DOI: 10.5220/0008586601560161

In Proceedings of the 3rd International Conference on Psychology in Health, Educational, Social, and Organizational Settings (ICP-HESOS 2018) - Improving Mental Health and Harmony in

Global Community, pages 156-161

ISBN: 978-989-758-435-0

g

General

Intelligence

Fluid

Intelligence

Crystallized

Intelligence

General

Memory&

Learning

Broad

Visual

Perception

Broad

Auditory

Perception

Broad

Retrieval

Ability

Broad

Cognitive

Speediness

Processing

Speed(RT

Decision

Speed)

General

(StratumIII)

Broad

(StratumII)

Narrow

(StratumI)

69narrowabilitiesfoundindatasetsanalyzedbyCarroll

Gf Gc Gy Gv Gu Gr Gs Gt

Figure 1: Carroll’s Three-Stratum Theory of Cognitive Abilities (1993) (Mcgrew and Flanagan, 1998).

from Cattel-Horn and three stratum theory from

Carroll (Beaujean, 2015; Furnham and Mansi,

2014). Raymond Cattell (1941, 1971). It has argued

that intelligence consists of two factors (g and s)

further developed by John Horn, who is also a

student of Cattell (1968, 1994). Then Carroll, in

1993, conducted a review and analysis of a number

of thousands of intelligence test databases and

produced three stratum models of human

intelligence and cognitive abilities (Carroll, 1993).

These models are considered capable of describing

the human cognitive structure as it uses a strong

empirical base. These opinions are integrated into

Cattell-Horn and Carroll theories (eventually

referred to as CHC). According to CHC theory,

intelligence consists of pervasive, broad and narrow

capabilities that are arranged hierarchically. Carrol

suggests there are at least 69 narrow abilities, as

shown in Figure 1. While McGrew conducted a

study again and found 59 narrow abilities (Floyd et

al., 2009; Mcgrew, 2009).

The CHC theory has broad implications for the

measurement of intelligence (McGrew, 1997, in

Gregory, 2011; McGrew in (Flanagan and Harrison,

2005b; Mcgrew, 2009). At present, CHC theory has

a major influence on the development of intelligence

tests, especially revised versions of previously

developed tests. Alfonso, Flanagan and Radwan

(Flanagan and Harrison, 2005b) have summarized

several tests whose development is influenced by the

CHC theory. For example, KABC, Kaufman

Assessment Battery for Children (1983) was revised

into KABC-II in 2004 through a measurement of

five broad abilities of CHC; SB-IV, Stanford–Binet

Intelligence Scales, Fourth Edition (1986) was

revised to SB5 in 2003 to which Working Memory

was added. In addition, WAIS-R, Wechsler Adult

Intelligence Scale - Revised (1981) developed into

WAIS-III in 1997 by comprehending Fluid

Reasoning and Working Memory. Whereas WPPSI-

R, Wechsler Preschool and Primary Scale of

Intelligence-Revised (1989) was revised to WPPSI-

III in 2002 by comprehending Processing Speed and

Fluid Reasoning. Another example is the revision of

WJ-R, Woodcock–Johnson Psycho-Educational

Battery-Revised (1989) to WJ III in 2001 through

the blueprint preparation. Quite major revision was

done to WISC-III, Wechsler Intelligence Scale for

Children - Third Edition (1991) which was revised

to WISC-IV in 2003 and that affected several

subtests, including: (1) elimination of Verbal and

Performance, (2) replacement of Freedom from

Distractibility Index to Working Memory Index, (3)

replacement of Perceptual Organization Index to

Perceptual Reasoning Index, and (4) comprehending

Fluid Reasoning and Processing Speed. Moreover,

development of new tests, namely RIAS, Reynolds

Intellectual Assessment Scales (2003), WRIT, Wide

Range Intelligence Test (2002), and KAIT, Kaufman

Adolescent and Adult Intelligence Test (1993) has

also been affected by the CHC theory.

On one hand, unfortunately, the tests mentioned

above have not been adopted in Indonesia.

Meanwhile, the adaptation process requires a long

Validity of Processing Speed Ability Test based on Internal Structure

157

and complicated procedure, psychometric criteria

that must be fulfilled, high cost due to the long

procedure, and surely the permission of the

publisher and test developer. In addition, most of

these tests are administered individually. Therefore,

from a practical point of view, test administration

that demands speed or which can be administered in

a classical manner is still less favorable.

On the other hand, practically, in accordance

with the experience of researchers as psychology

practitioners for several years, there are many

problems to be faced, especially in connection with

the intelligence assessment. Such issues may affect

the quality of the assessment itself, so that assessors

need to pay attention to this. These problems include

1) The availability of varied assessment tools is still

limited; 2) The most widely available assessment

tools are an external adaptation conducted in the

1960-1970s. At that time the method of adapting the

measuring instrument was not as advanced as it is

now, so it is likely that the adaptation is simply

translating; 3) The number of limited intelligence

tests means that the frequency of use becomes very

high. This makes the content of the test known to the

participants. In these conditions the assessment

result does not reflect the individual, so there is the

possibility that a psychological practitioner will gain

unreliable data. Finally, 4) Most of the existing test

kits are for individual administration, while group

administration tests are still indispensable.

This article is part of a larger research work that

includes developing intelligence tests based on the

CHC theory that is currently at the point of

preparing the items for each narrow ability (there are

4 broad abilities consisting of 12 narrow abilities).

This article specifically discusses the validity test of

cognitive speediness (some authors call it cognitive

speed processing or speed processing abilities)

which is one of the broad abilities of CHC theory.

According to McGrew, processing speed abilities

consist of three narrow abilities, Perceptual Speed,

Number Facility and Rate of Test Taking (Flanagan

& Harrison, 2005), which will be the basis for the

preparation of blueprint tests in this study. The test

to measure speed processing ability is interesting to

be developed because in addition to being an

important part of intelligence, this test also has a role

in other aspects of ability such as memory and fluid

intelligence. McGrew (Flanagan & Harrison, 2005)

summarizes some studies that show the role of speed

processing abilities against working memory and

fluid intelligence, i.e, research conducted by Fry and

Hale (2000); Conway, Cowan, Bunting, Therriault

and Minkof (2002), and Sub, Oberaurer, Wittmann,

Wilhelm and Schulze (2002).

As mentioned earlier, Cognitive Processing

Speed (Gs) refers to the speed of continuous

learning or automatic cognitive processes, especially

when high-level attention and concentration are

required. For example, the ability to perform simple

counts quickly demonstrates the ability of high

speed processing abilities. The ability to distinguish

two words also demonstrates the ability of high

processing speed ability. According to CHC theory,

as the broad ability, speed processing ability has

three narrow abilities, namely Perceptual Speed,

Number Facility and Rate of Test Taking. The

researcher hypothesizes that these three narrow

abilities consisting of Perceptual Speed, Number

Facility and Rate of Test Taking are substantially

appropriate to measure processing speed ability. In

addition, the researcher tries to involve gender

variables to see if there is a gender effect on the

processing speed ability.

2 METHOD

Research Design. The design of this study applies a

quantitative approach. In quantitative research, the

positivism paradigm becomes the basis for

determining the variables and hypotheses in

explaining the research and testing the hypotheses

associated with causal explanations in general

(Neuman, 2014). The technique of data collection in

this research is by using a survey. According to

Neuman (2014), data collecting methods using

survey techniques are conducted by asking people

who become the subject of research to fill in a

written questionnaire (paper based).

Participant. Participants in this study were those

who were involved in the selection of prospective

mentors of children prone to drop out of school

organized by the Social Office of Surabaya City

Government. The program is part of a campus social

responsibility involving students from several

universities in the city of Surabaya. There were 135

participants in this study consisting of 105 (78%)

women and 30 (22%) men from several universities

in Surabaya. Average and standard deviations of P

(Perceptual Speed), N (Number Facility) and R9

(Rate of Test Taking) are listed in the following

table.

ICP-HESOS 2018 - International Conference on Psychology in Health, Educational, Social, and Organizational Settings

158

Table 1: Descriptive Statistic Men and Women Participant.

Men Mean 31.46 19.86 28.52

SD 6.86 6.09 11.48

Women Mean 30.70 20.92 32.66

SD 5.46 5.51 11.31

Instruments. The instrument that became the

object of this research is a new instrument to

measure processing speed ability or cognitive

processing speed (Gs) consisting of three sub-tests.

The tests are compiled based on CHC theory.

According to McGrew (Flanagan and Harrison,

2005a; Mcgrew, 2009), the definition of processing

speed ability is the ability to automatically and

fluently perform relatively easy or overlearned

cognitive tasks, especially when high mental

efficiency (i.e. attention and focused concentration)

is required. This is the speed of executing relatively

overlearned or automatized elementary cognitive

processes. Gs consists of three narrow abilities,

which are Perceptual speed (P), Number facility (N)

and Rate of test taking (R9).

Perceptual speed (P): Ability to rapidly and

accurately search, compare (for visual similarities or

differences) and identify visual elements presented

side by side or separated in a visual field. Recent

research (Ackerman, Beier & Boyle, 2002;

Ackerman & Cianciolo, 2000; Ackerman & Kanfer,

1993), suggests that P may be an intermediate-

stratum ability defined by four narrow sub-abilities:

Pattern recognition, Scanning, Memory, and

Complex. Item form: there are a pair of combination

of letters and numbers. Participants are asked to

determine whether the pair is the same or not. There

are 100 items. Alpha reliability of 0.769. Work time

is 5 minutes.

Number facility (N): Ability to rapidly perform

basic arithmetic (i.e. add, subtract, multiply, divide)

and accurately manipulate numbers quickly. N does

not involve understanding or organizing

mathematical problems and is not a major

component of mathematical/quantitative reasoning

or higher mathematical skills.

Rate of test taking (R9): Ability to rapidly

perform tests that are relatively easy or overlearned

(require very simple decisions). This ability is not

associated with any particular type of test content or

stimuli. Item form: there is a pair of words,

participants are asked to cross out the same letter.

There are 99 items. The coefficient of alpha

reliability is 0.923. Work time is 5 minutes.

Examples of items for each narrow ability are in the

Appendix.

Data analysis. A first-order confirmatory factor

analysis was performed to test the hypothesis. To

test validity based on internal structure, confirmatory

factor analysis was conducted using JASP 0.9 (JASP

Team, 2018).

3 RESULT

The parameters used to test the accuracy of the

model were Chi Square (χ2), significance level (p),

CFI and root mean square error of approximation

(RMSEA). The result of data analysis showed Chi-

square (χ²) = 0.341 (9, N = 135), p = 0.559; CFI = 1.

000; SRMR = 0.017; RMSEA = 0.000 (90% CI

[0.000 - 0.0191]. The RMSEA described the

residuals contained in the model, so the expected

value was very small, under 0.08. In the arranged

model, the obtained RMSEA value was 0.000 which

proved that the arranged model was fit, just like any

other parameter which indicated that the accurate

index was fit. Full results are listed in Table 2.

Table 2 : Model Fit Parameter.

Parameter Model

χ² 0.341

Degreesoffreedom 1

p0.559

SRMR 0.017

RMSEA 0.0000

90PercentConfidenceInterval 0.000

–

0.191

p‐valueRMSEA<=0.05 0.620

Table 3 : Parameter Estimate.

Label Est p

Gs=~P 1.000

‐

Gs=~N

0.602 0.195

Gs=~R

1.204 0.182

Gender~Gs

‐0.030 0.135

4 DISCUSSION

This study aims to test the validity of the test

processing speed ability consisting of three narrow

abilities, namely Perceptual Speed, Number Facility

and Rate of Test Taking. Based on the result of

confirmatory factor analysis, the study shows that

Validity of Processing Speed Ability Test based on Internal Structure

159

the three mentioned factors prove to form a

constraint processing speed ability. It is indicated by

the parameters showing that the model is fit.

The parameters for the model accuracy test also

show that the model can explain the roles of all three

factors. RMSEA value of 0.000 proves that the

model is very fit. Similarly, the value of Chi Square

(χ²) whose magnitude is 0.341 with a significance

level (p) of 0.559 (greater than 0.05) indicates that

the index of accuracy meets the criteria. This means

that the model can describe a well-measured factor.

Thus, based on a trial that aims to estimate the

validity of internal structure, the research shows that

the prepared processing speed ability test has good

internal validity. This is evidenced from the index of

model accuracy that all meet the expected range of

values.

The researcher wanted to know why the

temporary model fit of the parameter estimate was

not significant. To answer this question, the

researcher did post hoc to know the magnitude of

statistical power. With a sample size of 135,

RMSEA of 0.000, alpha of 0.05 and df equal to 1,

the statistical power obtained was 5%. This

statistical power was very small, and this result at

once answered the question why the temporary

model fit parameter of each narrow ability was not

significant. With a sample size of 135, RMSEA of

0.000, alpha of 0.05 and df equal to 1, the statistical

power obtained was 5%. This statistical power was

very small, and this result at once answered the

question why the model was suitable while the

parameters of each narrow ability was not

significant. Simulation was also done to prove that

to obtain statistical power of 80% the required

sample was at least 78,490 people. Meanwhile, due

to the gender effect on processing speed ability, the

result showed that there was no difference between

men and women. This could be seen from the level

of significance (p) gender for processing speed

ability of 0.135 (see Table 1.3). This fact showed

that there was no influence or different ability of a

person in terms of processing speed ability if viewed

from the aspect of gender.

Results of the validity test found that all three

factors have a fit model in measuring speed

processing ability. However, some weaknesses are

identified, e.g. the statistical power which is weak

(only about 5% due to insufficient sample size). As

mentioned earlier, to obtain a stronger statistical

power (about 80%) the required sample was at least

78,490 people. To test a theoretical model, it should

include all indicators/sub-tests/narrow abilities as the

larger project of the study. Research on the planned

intelligence test has 12 sub-tests, so the degree of

freedom is 78. If assuming RMSEA is 0.04, and

wanting to obtain 80% statistical power, then the

required sample is 293.

Moreover, for test developers interested in the

same field, it is recommended to conduct further

research to enrich the validity of this test through

different validity sources, such as validity based on

criteria with other variables. And for practitioners,

this test can be used to measure speed processing

abilities. However, it is necessary to be careful in

interpreting the test results, as these tests are new so

there is little evidence available regarding the

validity of this test. Another future challenge is to

develop norms from larger groups that could be used

to interpret test results more accurately. The related

norms to interpretation are required for diagnostic

purposes, so that the existence of norms becomes

absolute if the test is to be used in a practical field.

REFERENCES

Anastasi, A. and Urbina, S., 1997. Psychological Testing.

7th Editio ed. New York: Prentice Hall.

Beaujean, A., 2015. John Carroll’s Views on Intelligence:

Bi-Factor vs. Higher-Order Models. Journal of

Intelligence, 3(4), pp.121–136.

Carroll, J.B., 1993. Human Cognitive Abilities: A Survey

of Factor-Analytic Studies. Educational Researcher.

New York: Cambridge University Press.

Flanagan, D.P. and Harrison, P.L., 2005a. Contemporary

intellectual assessment. The Guilford Press.

Flanagan, D.P. and Harrison, P.L., 2005b. Contemporary

Intellectual Assessment. the Guilford Press, .

Floyd, R.G., Shands, E.I., Rafael, F.A., Bergeron, R. and

Mcgrew, K.S., 2009. Intelligence The dependability of

general-factor loadings : The effects of factor-

extraction methods , test battery composition , test

battery size , and their interactions. Intelligence, 37(5),

pp.453–465.

Furnham, A. and Mansi, A., 2014. The self-assessment of

the Cattell – Horn – Carroll broad stratum abilities.

Learning and Individual Differences, 32, pp.233–237.

Gottfredson, L., 2009. Intelligence : Foundations and

Issues in Assessment. 50(3), pp.183–195.

JASP Team, 2018. JASP.

Kaufman, S.B., Reynolds, M.R., Liu, X., Kaufman, A.S.

and McGrew, K.S., 2012. Are cognitive g and

academic achievement g one and the same g? An

exploration on the Woodcock-Johnson and Kaufman

tests. Intelligence, 40(2), pp.123–138.

Lohman, D.F. and Gambrell, J.L., 2012. Journal of

Psychoeducational Assessment.

Lohman, D.F., Korb, K.A., Lakin, J.M., Korb, K.A. and

Lakin, J.M., 2008. Gifted Child Quarterly Identifying

Academically Gifted English- A Comparison of the

ICP-HESOS 2018 - International Conference on Psychology in Health, Educational, Social, and Organizational Settings

160

Raven , NNAT , and CogAT.

Marnat, G.G., 2003. Handbook Of Psychological

Assessment. 4th ed. Canada: John Wlley & Sons Inc.

Mcgrew, K.S., 2009. Editorial CHC theory and the human

cognitive abilities project : Standing on the shoulders

of the giants of psychometric intelligence research.

Intelligence, 37(1), pp.1–10.

Mcgrew, K.S. and Flanagan, D.P., 1998. The Intelligence

Test Desk Reference (ITDR): Gf-Gc Cross-Battery

Assessment. Boston: Allyn & Bacon.

Naglieri, J.A., Ford, D.Y., Naglieri, J.A. and Ford, D.Y.,

2003. Gifted Child Quarterly Addressing

Underrepresentation of Gifted Minority Children

Using the Naglieri Nonverbal Ability Test ( NNAT ).

Neuman, W.L., 2014. Social Research Methods:

Qualitative and Quantitative Approaches. Relevance

of social research, .

Vanderwood, M.L., McGrew, K.S., Flanagan, D.P. and

Keith, T.Z., 2001. The contribution of general and

specific cognitive abilities to reading achievement.

Learning and Individual Differences, 13(2), pp.159–

188.

APPENDIX

Appendix 1: Sample Item

NARROW

ABILITY

SAMPLE

ITEM

OPTION

Perceptual

Speed (P)

5+7=12 R W

10- 9=19 R W

Number

Facility

(N)

Aab ........ Aab = ≠

cdOp ........

cd0p

= ≠

Rate of

Test

Taking

(R9)

pagi --- puli

cross the same

letters between

the left and right

word.

denah--- parik

Validity of Processing Speed Ability Test based on Internal Structure

161