Invariant Measurement with Raters and Rating Scales

The purpose of this book is to present methods for developing, evaluating and maintaining rater-mediated assessment systems. Rater-mediated assessments involve ratings that are assigned by raters to persons responding to constructed-response items (e.g., written essays and teacher portfolios) and other types of performance assessments. This book addresses the following topics: (1) introduction to the principles of invariant measurement, (2) application of the principles of invariant measurement to rater-mediated assessments, (3) description of the lens model for rater judgments, (4) integration of principles of invariant measurement with the lens model of cognitive processes of raters, (5) illustration of substantive and psychometric issues related to rater-mediated assessments in terms of validity, reliability, and fairness, and (6) discussion of theoretical and practical issues related to rater-mediated assessment systems. Invariant measurement is fast becoming the dominant paradigm for assessment systems around the world, and this book provides an invaluable resource for graduate students, measurement practitioners, substantive theorists in the human sciences, and other individuals interested in invariant measurement when judgments are obtained with rating scales.

George Engelhard, Jr. is a professor at The University of Georgia, USA, and a professor emeritus at Emory University, USA. Stefanie A. Wind is an assistant professor at The University of Alabama, USA.

Preface. I. Introduction 1. Introduction and Overview 2. Progress in the Social Sciences: An Historical and Philosophical Perspective II. Theories of Measurement and Judgment for Rating Scales 3. Measurement Models for Rater-Mediated Assessments: A Tale of Two Research Traditions 4. Lens Models of Human Judgment for Rater-Mediated Assessments III. Foundational Areas for Rating Scales 5. Validity, Invariant Measurement, and Rater-Mediated Assessments 6. Reliability, Precision, and Errors of Measurement for Ratings 7. Fairness in Rater-Mediated Assessment: Appropriate Interpretation and Use of Ratings 8. Case Study: Evidence for the Validity, Reliability, and Fairness of Ratings on a Middle Grades Writing Assessment IV. Technical Issues and IRT Models for Ratings 9. Models for Ratings Based on Item Response Theory 10. Parameter Estimation for the Polytomous Rasch Model V. Practical Issues 11. Model-Data Fit for Polytomous Rating-Scale Models 12. Designing Rater-Mediated Assessment Systems 13. Examining Rating Scale Functioning VI. Final Word 14. Invariant Measurement with Raters and Rating Scales: Summary and Discussion