Understanding Measurement: Reliability

This is a title in our Understanding Statistics series, which is designed to provide researchers with authoritative guides to understanding, presenting and critiquing analyses and associated inferences. Each volume in the series demonstrates how the relevant topic should be reported -- including detail surrounding what can be said, and how it should be said, as well as drawing boundaries around what cannot appropriately be claimed or inferred.
This volume addresses reliability, which is a fundamental aspect of any social science study that involves educational or psychological measurement. It not only has implications for the quality of test scores themselves, but also any statistical analysis conducted using those scores. Topics addressed in this book include cover three different types of reliability methods and appropriate standard errors of measurement: classical test theory methods, decision consistency indices, and generalizability theory coeffcients. After a brief introduction to the topic, the author outlines how to report reliability in professional journal articles. Meyer is known for his clear, accessible writing; like all books in this series, this volume includes examples of both good and bad write-ups for methods sections of journal articles.

Patrick Meyer, Assistant Professor, Curry School of Education, University of Virginia.

Chapter I: Introduction The Importance of Test Score Reliability Organization of the Book General Concepts The Measurement Procedure Sampling in Measurement Sampling the Measurement Procedure. Sampling Examinees. Replicating the Measurement Procedure and Sampling Examinees Classical Test Theory Replicating the Measurement Procedure Additional Restrictions that Define Classical Test Theory Sources of Variance The Reliability Coefficient Estimating Unobserved Quantities Classification decisions Replicating the Measurement Procedure Strong True Score Theory Distributional Assumptions Domain Scores. Conditional Distribution of Observed Scores. Observed Scores. Estimating Domain Scores. Reliability and Conditional Standard Error of Measurement Generalizability Theory Generalizability Study Variance Components. Decision Study Replicating the Measurement Procedure. Sources of Variance. Types of Decisions. Reliability Coefficients. Estimating Unobserved Quantities. Chapter Summary
Chapter 2: Data Collection Designs Description of Real-World Measurement Procedure Data Collections Designs in Classical Test Theory Whole Test Replications Test-Retest Reliability Coefficient Alternate Forms Reliability Coefficient Test-Retest with Alternate Forms Two Part-Test Replications Split-halves Other Two-Part Divisions. Multiple Part-test Divisions. Data Collection Designs for Classification Decisions Data Collection Designs in Generalizability Theory Common Data Collection Designs Random Designs Mixed Designs Classical Test Theory Designs Restated as Data Collection Designs Data Collection Designs for the ELA Benchmark Assessment Chapter Summary
Chapter 3: Assumptions Dimensionality Violation of Unidimensionality Error Score Correlations Violation of Uncorrelated Errors The Nature of Measurement Procedure Replications Classical Test Theory Congeneric Measures. Classical Congeneric Measures. Essentially Tau-equivalent Measures. Tau-equivalent Measures. Parallel Measures. Violation of Essential tau-equivalence Classification Decisions and Generalizability Theory Randomly Parallel Measures Distributional Assumptions Violation of Randomly Parallel Measures Chapter Summary
Chapter 4: Methods Reliability Coefficients for Relative Decisions Whole Test Replications Two Part-test Replications Multiple Part-test Replications Other Methods for Estimating Reliability Estimating Reliability for Absolute Decisions Two Whole-test Replications Squared Error Loss Coefficients Threshold Loss Indices One Whole-test Replication Squared Error Loss Coefficients Threshold Loss Indices Other Methods Chapter Summary
Chapter 5: Results Documentation of Score Reliability Characterize the Examinee Population Describe the Measurement Procedure and All Major Sources of Error Present Evidence in Support of Assumptions Provide Estimates of Reliability and the Standard Error of Measurement Benchmark ELA Assessment Sample Results Section Palmetto Achievement Challenge Test of Mathematics Sample Results Section The Responsive Classroom Efficacy Study and MSCAN Sample Results Section Chapter Summary
Chapter 6: Discussion and Recommended Readings ELA Benchmark Assessment Some Interesting Points about the Analysis Palmetto Achievement Challenge Test of Mathematics The Responsive Classroom Efficacy Study and MSCAN Recommended Readings References