Research in the Teaching of English

Problem Identification: Identification of diversity, impact of testing, and alignment of both—these are significant issues of enduring educational importance. The case at hand, Canada, presents a valuable sampling plan.

Original Approach: Attention to consequential validity is a good framework for the study, important because of frequent cosmetic attention to its presence in a testing situation.

Relationship to (and Extension of) Current Research and Theory: Attention to Messick and Kane are important in such an analysis, and the bases are covered. Yet, this research is 25 years old, and more theorization is needed for the paper to succeed.

Fresh Insight: As I will suggest below, a revision will provide needed insight for RTE readers.

Congruence between Method and Theoretical Orientation: There is a basic congruence here that is good, but additional categories of validity conceptualization are needed if the theory and method are to match, resulting in fresh insight.

Information Analyzed with Concepts Based on Theory: In a recent presentation of his conceptual framework for validation, Kane (2013) again presented his 2006 validation categories: scoring, generalization, extrapolation, and implication. Little attention is paid to theory-based interpretation in the new paper, and implication remains a black box. Certainly, the authors could begin by reviewing that most recent work and calling for an expansion of categories significant to stakeholders beyond measurement specialists.

Conclusions Grounded in Analysis: The authors write, “Our findings support Brennan’s claims: while government documents made strong assertions about the value of standardized testing to monitor, maintain, or improve systems of education, the peer-reviewed research that collected directed evidence from within schools painted a much more negative picture of the effects of these assessment programs on literacy education and diversity in Canadian classrooms” (33). That may be indeed so. But whether this finding is dependent on consequential validity of other categories of validity evidence is unknown.

Now, to the methodological questions of analysis and interpretation as they related to the notes above: As a key research question, the authors ask, “What data or arguments are being used to justify/provide indicators of the impact of literacy assessments on teaching, learning, and achievement.” Strictly speaking, this is not a consequential validity question but a validity argument question. And, the second question on page 11 is one of use. Again, this is why the categories of validity evidence need to be presented early on in the paper. Were these categories expanded, Tables 1 and 2 could be vastly improved to include categories of validity evidence identified in the document search.

Strong Scholarship: The scholarship here is good but not sufficient for the analysis. I have suggested additional paths of inquiry in the references.

Theory Extended or Reconceived: As I have noted, a more exact presentation of consequential validity is needed.

Clear, Fluent, and Engaging Presentation: The authors are very good on the page.

Concluding Notes: The areas of revision I have suggested—addition of validity evidence categories, expansion of the definition of consequential validity based on shareholder theory, and design of Toulmin models to reveal document claims—are complex. I realize that. But you do have two sets of documents, and with more effort, something here can be produced of great value to RTE readers. My notes have been offered to improve the research, not to offer foolish barriers impossible to meet.

A final note: Consider abandoning the value dualism of standardization versus construct representation. Realistically, if test could be developed allowing fuller representation of the writing construct, is there evidence that new test would meet the other categories of validity evidence? If you can abandon value dualism and value hierarchy as a thought experiment, I think you would be able to re-design the tables and create the Toumin models more accurately. Then, once completed, I’ll wager the final pages of the manuscript would look much different—and readers would be grateful for that indeed.

You are onto something important here. Go deeper. See Additional References, such as:

Fabian, S. B. (2009, Spring). Diverse Test Takers: Essential considerations for test developers and decision makers, Educational Measurement: Issues and Practice, 10-24.