Calculation
of the acceptable level of performance (ALP) for a MCQ
test
1. Definition
The
acceptable level of performance is a threshold value making it possible to
decide (according to absolute criteria) whether a student "who knows
barely enough" should be passed or failed. Calculation of the ALP for a
test is not valid unless the number of MCQ is more than 30. Use of the ALP
involves an advance judgment (before the test) on the relative difficulty of
each question and enables a judgment based on the test as a whole to be made.
Calculation of the ALP depends on the collective decision of several teachers
each of whom has first made an independent judgment.
2. Procedure*
To calculate the
acceptable level of performance (ALP) of a student for a MCQ test:
2.I The
evaluation board decides what is the correct answer to each MCQ; Let us take
two MCQs which are worded identically but where the choice of answers is
different.
Question,
Which
of the following values corresponds to the number of red cells per mm3
of blood in a healthy adult?
This procedure was
developed from an article by L. Nedelsky (Absolute grading standards for
objective tests,. Educ. Psycbolog. Meas., 14,3-19,1954).
2.2 The
board decides which answer or answers must definitely be eliminated by the
student, other than by chance;
2.3 The board calculates the acceptability
index for each MCQ;
2.4 The ALP for the test as a whole is the
sum of the acceptability indexes for
each MCQ.
The acceptibility
index for a MCQ is calculated as follows:
Carefully study all
the choices offered and decide which the student "who knows just enough to
pass" should be able to reject. For example, if a question offers five
choices (only one of which is the correct answer) and it is deemed that the
student 'who knows just enough to pass" should be able to reject one of
these choices straightaway, it follows that the marginal student could obtain
the correct answer by mere chance approximately one time out of four. In this
case the acceptability index of the question is 0.25.
Let us take two MCQ which are worded
identically but where the choice of answers is different.
Question: Which of the following values corresponds to the number of
red cells per mm3 of blood in a healthy adult?
1 2
A 500000 A 4000000
B 1000000 B 4500000
C 2000000
C 4750000
D 3000000 D 5000000
E 5000000 E 5250000
In case 1, an acceptability index of
1.00 could be considered while in case 2 it could be 0.25.
3. Comments
The
ALP has little value if it is not based on a
detailed analysis of each of the questions in a test, including consideration
of incorrect choices just as much as of correct answers.
The
validity of the estimate of the ALP also depends on obtaining independent judgments
from several teachers who have paid attention to the educational objectives and the level for
which the examination is intended. The quality of the estimate will be the
greater the larger the number of teachers involved. When the differences
between the judgments’ obtained are relatively small, the extremes can serve to
define a "grey zone" below which the results will be regarded as
distinctly inadequate (failure) and above which the results will clearly
indicate a success. For example, if the mean of the estimates of one
teacher for the ALP of a test is 43%whereas two other teachers obtain figures
of 45% and 47%, respectively, then it could be recommended that any score below
43% should be regarded as a failure, that any score above 47% be regarded as a
success, while a score between 43% and 47% should be regarded as being in a
grey zone. It would remain to be defined what should be done in the latter
case. If the differences between the judgments obtained by several teachers are
large then the criteria of the educational objectives should be revised.
Resource
Educational Handbook for Health
Personnel (Sixth Edition)
J.J. Guilbert, World Health
Organization, Geneva
No comments:
Post a Comment