Our review of the NAEP Framework

Coverage of text types

The content of the 2009 NAEP Reading Framework focuses on two types of texts:

• Literary texts (fiction, literary nonfiction, and poetry) and
• Informational texts (exposition, argumentation, and persuasive text; and procedural text and documents).

The framework prescribes the reporting of separate subscales for literary and informational text, which NAEP has never done before. The text types do not, however, differ much from the previous framework, which had defined three text types: “literary, informational, and document.” The document category in the previous framework has been subsumed under the new informational text category, while literary nonfiction (e.g., essays and speeches) has been moved from the informational category to the new literary text category. This redistribution is significant in the context of what the overall passage distribution looks like in the 2009 framework (see Exhibit 1).

In the twelfth grade, significantly more weight is given to informational text than to literary text.17 Despite the laudable addition of poetry at fourth grade, the framework maintains a disproportionate emphasis on informational texts. Insofar as NAEP is influential in the states, this shift could result in diminished emphasis in the field on teaching literary texts, both fiction and nonfiction. Further, it could have profound and detrimental effects on students’ cultural literacy and their foundational skills in the humanities.

The framework supplies details of the various types of text in easy-to-read matrices. For the most part, they represent a reasonable set of expectations for K-12 students, although there is room for improvement. First, some “aspects of texts” at eighth and twelfth grades might easily be introduced earlier. For example, rhythm patterns, point of view, and personification, now listed at eighth grade, could certainly be included at grade four. Aspects of fiction at grade eight could also be assessed at grade four, such as flashback and personification. Some aspects at twelfth grade also seem to come late, such as monologue, comic relief, and “unconventional use of language.”

Second, the aspects of “argumentation and persuasive text” at twelfth grade are left undefined, despite this subcategory’s previously stated significance. Finally, the elusive notion of voice is introduced as early as grade four in sub-categories of informational text, but how or why voice is distinct from style, tone, or mood is a much-debated proposition; the inclusion of voice seems unnecessary.

types of reading skills assessed

As for what students are asked to do with the texts, the framework defines three types of “cognitive targets, or behaviors and skills” to be assessed:

• Locate/recall;
• ˆˆIntegrate/interpret;
• and ˆˆCritique/evaluate.

These skill categories do not differ significantly from those detailed in the previous NAEP reading framework, but specific skills are not delineated by grade level, which would help clarify expectations. The framework explains:

The cognitive targets remain the same across all three grades on the assessment but the passages and documents about which items are developed will be of increasing sophistication at each grade.

Because the “targets” do not change across the grade levels, the rigor of the NAEP reading assessment ultimately depends on how particular grade-specific items are constructed and the types of passages that are selected for students’ consideration.

Unfortunately, we do not have access to the full range of test items and associated passages at each grade span, but the released items found in “Sample Questions” booklets offer a glimpse of the assessment’s rigor. The sample assessment items at each grade span seem appropriately challenging in terms of the cognitive skills listed above, but the sample reading passages themselves are largely mediocre and not particularly complex. As a result, the sample items overall are not sufficiently demanding.

One welcome addition to the 2009 framework is “a more systematic approach to vocabulary assessment,” including the calculation of a vocabulary sub-score for the 2009 administration of the test. Although this new emphasis on vocabulary is commendable, the sample questions send mixed signals about rigor. Fourth and eighth grade look to be reasonably rigorous, but twelfth grade does not. For example, the words “scornfully” and “subdued” at fourth grade and “deflect” and “abated” at eighth grade are of an appropriate difficulty level; “obligations” and “precedent” at twelfth grade are not. Words taken from SAT prep lists, such as “specious,” “attenuate” or “circuitous” might be more appropriate.

Content and Rigor Conclusion

While the framework serves its designated purpose well, namely to be a fair starting point for assessing basic reading comprehension at fourth, eighth, and twelfth grades, it does not do four things that Americans should expect academic standards in reading to do: 1) address expectations for essential early reading skills; 2) detail more explicit aspects of vocabulary development; 3) prioritize students’ exposure to outstanding American literature; and 4) detail choices about the quality and complexity of reading passages.

The framework is missing crucial content by not placing enough emphasis on literature and English language conventions. The framework could also cover the content more carefully by distinguishing the skills targets at each grade span, as well as making them more rigorous and grade appropriate. These revisions would also help practitioners distinguish between more and less important content. Since the framework falls short in these ways, it earns a grade of five (see “Common Grading Metric,” page 16).

The reading framework scores fairly well against our criteria for clarity. It is generally clear and usable. However, as noted above, it could be more specific and detailed in ways that would also improve the content.

Despite its general clarity, the persistence of some vague language in the framework makes it difficult to give it the highest marks. For example, the definitions of the text types “argumentation” and “persuasion” are confusing. The framework implies that argumentation must, by definition, be persuasive text, and it links the two types together as one sub-category of informational text. Many practitioners, however, would distinguish between presenting an “argument” as a matter of exposition (i.e., making known one’s position on an issue or one’s interpretation of a text) and a persuasive essay or speech (i.e., attempting to sway the reader’s opinion, often by including a call to action). The conflation of the two here is likely to confuse practitioners who have been teaching that not all arguments must be persuasive.

The need for more specificity where the content is concerned and the confusion surrounding some definitions result in the framework earning a grade of two.

purposes for writing

The 2011 NAEP assessment measures three communicative “purposes” rather than the three “modes” of writing that previous Frameworks sought to gauge. The change is mostly semantic: The ultimate substance of the framework—what is assessed— appears relatively unchanged.

The three modes of the 1998-2007 NAEP writing assessment were:

• Persuasive mode:
  • Writing to convince
  • Writing to construct an argument
  • ˆWriting to refute a position
• Informative mode:
  • ˆDescription
  • Explanation
  • ˆAnalysis
• Narrative mode:
  • ˆFirst-person and third-person fictional stories, personal essays

The 2011 framework’s three purposes are:

• To persuade, in order to change the reader’s point of view or affect the reader’s action; ˆˆ
• To explain, in order to expand the reader’s understanding; and ˆˆ
• To convey experience, real or imagined, in order to communicate individual and imagined experience to others.

The framework offers only very general descriptions of the three purposes. It discusses when and why people write for these purposes, but does not detail specific writing expectations by grade levels as the NAEP reading framework attempts to do.

The relative percentages of items devoted to each purpose seem appropriate (see Exhibit 2).

EXHIBIT 2. Percentage distribution of communicative purposes by grade

There need not be an emphasis on writing literature (i.e., “writing to convey experience”) in the writing framework like there is on reading literature in the reading framework. In other words, students should not be expected to write sonnets or great literary pieces. Emphasizing persuasive and explanatory purposes for writing, then, makes sense. The emphasis is also consistent with the content criteria for writing (see “English Language Arts Content-Specific Criteria,” page 13) which state that students should “…produce writing that reflects the defining characteristics of various grade appropriate writing genres.”

Compared to the previous writing framework, the emphasis on persuasive writing has increased slightly at grade four (30 percent in 2011 versus 25 percent in 1998). The framework notes that this addition was deliberate, in order to “scaffold” the emphasis on persuasive and informational writing skills at later grades. The “English Language Arts Content-Specific Criteria” place similar emphases across the three grade spans, moving from “narration, exposition” as an emphasis in kindergarten through fourth grade, to “argument” in five through eight, and “persuasion” at nine through twelve. The writing purposes and the time accorded to each in the NAEP writing framework are therefore sensible and on-target.

Features of Writing

Within these types of writing, the framework identifies three very broad categories of features of writing:

1. Development of ideas
2. Organization of ideas
3. Language facility and conventions

Further criteria for assessing writing in each of these “broad domains” are included, but they, too, are rather vague, mentioning only such aspects as “depth and complexity” (for number one above), “coherence” (for number two) and “voice and tone” (for number three). These terms alone are too broad to be useful. Further clarification is attempted in later prose descriptions, but alas, these, too, are nebulous and lack sample responses. For example, when describing “Approaches to Thinking and Writing,” an aspect of “Development of Ideas,” the framework provides this befuddling statement:

Specific approaches to thinking and writing will not be specified on NAEP tasks, but responses will be evaluated for their use of effective approaches in relation to the development and organization of ideas.

The framework appears to rely on sample writing tasks (or “prompts”) more than on these descriptions to exemplify its expectations. This is to be expected since one of NAEP framework’s primary roles is to inform assessment (see “NAEP Overview” in the Appendix, page 52).18 Even more helpful, however, would be the addition of sample acceptable responses to the tasks (see related “Clarity and Specificity” discussion below).

The prompts generally provide context and clear directions and many are text-based in some way (meaning the tasks are grounded in specific reference points and therefore provide a “level playing field” for responding). In terms of actual grading, however, the six-point scoring rubrics are “holistic.” This means that specific weights are not assigned to each of the criteria (many of which address the features of writing discussed above), making it difficult to discern where students are doing well and where they are struggling. For example, a student’s response could be well-organized, yet inaccessible because of poor sentence structure and mechanics; another response might boast impressive style, but lack development of ideas. The holistic rubrics
do not differentiate among these priorities; they could be made more useful if they were prompt-specific and accompanied by sample acceptable responses.

One overall disappointment is the very general nature of the expectations for English language conventions. (They are absent from the reading framework altogether, even though careful analysis of sentence structure and grammar certainly aids reading comprehension.) Making matters worse, English language conventions—the most consistently noted concern of employers and postsecondary faculty—can easily be overlooked with holistic scoring. NAEP should find a way to articulate its expectations in this critically important area in much greater detail.

Content and Rigor Conclusion

Some of the content in the NAEP Writing Framework is covered satisfactorily. Specifically, it does an adequate job of identifying the appropriate purposes for writing and placing reasonable emphases on each purpose at the various grade levels.

While the majority of the appropriate content is covered by the standards, however, it is shortchanged in important ways. First, more attention to conventions would help make the framework as rigorous as it should be. Second, since the rubrics repeat much of the same broadly-worded expectations from grade to grade, it is difficult to distinguish between more and less important content. It is also possible that the standards need to be more or less rigorous than they are at certain grade levels (i.e., expectations are too high or too low) but, given the one-size-fits-all expectations, it is impossible to tell. Because of these limitations, the writing framework receives an overall content and rigor grade of five (see “Common Grading Metric,” page 16).

The writing framework scores rather well against our criteria for clarity. The guidelines for writing tasks are straightforward and coherent, and, as mentioned above, the sample prompts help clarify expectations. That said, illuminating a scope and sequence of writing skills by grade span would provide better guidance to educators. More specificity at these important benchmark grades would also help indicate what expectations should look like in the intervening grades.

The framework’s generic scoring rubrics could be strengthened by making them specific to a prompt. The framework could also include sample “anchor papers” (i.e., examples of student responses that are used to train scorers) so that practitioners can understand how the rubrics are applied (e.g., whether one area of the rubric appears to be more heavily weighted in scoring than another).

Finally, rather than simply sketching the purposes for writing and offering generic rubrics for scoring the items, the framework could offer a matrix of more specific writing expectations by grade level, as it does for reading. It might mention the kinds of sentence structures, transitions, and stylistic devices it expects to see in writing samples at each grade span. Without offering sample acceptable responses at each cut score, it is difficult to determine the actual level of rigor of the items. This lack of specificity results in a two for clarity and specificity.