Calculation of the acceptable level of performance (ALP) for a MCQ test
The acceptable level of performance is a threshold value making it possible to decide (according to absolute criteria) whether a student "who knows barely enough" should be passed or failed. Calculation of the ALP for a test is not valid unless the number of MCQ is more than 30. Use of the ALP involves an advance judgment (before the test) on the relative difficulty of each question and enables a judgment based on the test as a whole to be made. Calculation of the ALP depends on the collective decision of several teachers each of whom has first made an independent judgment.
To calculate the acceptable level of performance (ALP) of a student for a MCQ test:
2.I The evaluation board decides what is the correct answer to each MCQ; Let us take two MCQs which are worded identically but where the choice of answers is different.
Which of the following values corresponds to the number of red cells per mm3 of blood in a healthy adult?
This procedure was developed from an article by L. Nedelsky (Absolute grading standards for objective tests,. Educ. Psycbolog. Meas., 14,3-19,1954).
2.2 The board decides which answer or answers must definitely be eliminated by the student, other than by chance;
2.3 The board calculates the acceptability index for each MCQ;
2.4 The ALP for the test as a whole is the sum of the acceptability indexes for each MCQ.
The acceptibility index for a MCQ is calculated as follows:
Carefully study all the choices offered and decide which the student "who knows just enough to pass" should be able to reject. For example, if a question offers five choices (only one of which is the correct answer) and it is deemed that the student 'who knows just enough to pass" should be able to reject one of these choices straightaway, it follows that the marginal student could obtain the correct answer by mere chance approximately one time out of four. In this case the acceptability index of the question is 0.25.
Let us take two MCQ which are worded identically but where the choice of answers is different.
Question: Which of the following values corresponds to the number of red cells per mm3 of blood in a healthy adult?
A 500000 A 4000000
B 1000000 B 4500000
C 2000000 C 4750000
D 3000000 D 5000000
E 5000000 E 5250000
In case 1, an acceptability index of 1.00 could be considered while in case 2 it could be 0.25.
The ALP has little value if it is not based on a detailed analysis of each of the questions in a test, including consideration of incorrect choices just as much as of correct answers.
The validity of the estimate of the ALP also depends on obtaining independent judgments from several teachers who have paid attention to the educational objectives and the level for which the examination is intended. The quality of the estimate will be the greater the larger the number of teachers involved. When the differences between the judgments’ obtained are relatively small, the extremes can serve to define a "grey zone" below which the results will be regarded as distinctly inadequate (failure) and above which the results will clearly indicate a success. For example, if the mean of the estimates of one teacher for the ALP of a test is 43%whereas two other teachers obtain figures of 45% and 47%, respectively, then it could be recommended that any score below 43% should be regarded as a failure, that any score above 47% be regarded as a success, while a score between 43% and 47% should be regarded as being in a grey zone. It would remain to be defined what should be done in the latter case. If the differences between the judgments obtained by several teachers are large then the criteria of the educational objectives should be revised.
Educational Handbook for Health Personnel (Sixth Edition)
J.J. Guilbert, World Health Organization, Geneva