Appropriateness measurement with polychotomous item response models and standardized indices

Source abstract

The test scores of some examinees on a multiple choice test may not provide satisfactory measures of their abilities. The goal of appropriateness measurement is to identify such individuals. Earlier theoretical and experimental work considered examinees answering all, or almost all, test items. This article reports research that extends appropriateness measurement methods to examinees with moderately high non‐response rates. These methods treat non‐response as if it were a deliberate option choice and then attempt to measure the ‘appropriateness’ of the pattern of option choices. Earlier studies used only the dichotomous pattern of ‘right’ and ‘not right’ answers. A general polychotomous model is introduced along with a technique called ‘standardization’ designed to reduce the observed confounding between measured appropriateness and ability. A standardized appropriateness index based on a polychotomous model yielded higher rates of detection of simulated spuriously low examinees than the analogous index based on a dichotomous model. However, the converse was true for simulated spuriously high examinees. Standardization was found to reduce greatly the interaction between ability and measured appropriateness.