NAEP FrameworkView Best in Class
Our review of the NAEP FrameworkThis report is part of our "Stars by Which to Navigate?" report.
Both the NAEP 2009 Reading Framework and the 2011 Writing Framework reflect an unfortunate trend among developers of K-12 academic standards, namely the perceived need to assign utilitarian â€œreal worldâ€ purposes to all academic expectations. The NAEP 2009 reading framework openly acknowledges the influence of PISA, a proudly non-curriculum-based exam, which describes itself as a test that â€œassesses how far [sic] students near the end of compulsory education have acquired some of the knowledge and skills essential for full participation in society.â€15 Both frameworks were also developed in the wake of the 2004 release of the American Diploma Project (ADP) Benchmarks, which prompted many policymakers to evaluate state standards and high-school graduation requirements strictly in the insufficient light of the knowledge and skills that are necessary for college and work rather than also recognizing their vital role as the foundation of a robust liberal arts curriculum, for responsible adulthood, and for competent citizenship.
Both NAEP frameworks seek to connect the content and skills they delineate to their ostensible application in the postsecondary worlds of college and work. While it is trueâ€”and importantâ€”that students should be prepared to succeed in college and the workplace, not all valuable educational experiences, such as the analysis of great literature, will have a direct or perpetual application in the workplace. (Some arenâ€™t necessarily applicable in college, either. A student may never take a â€œpoetry courseâ€ in college, yet still should have learned to read poetry in high school.) Many things learned in school but not immediately useful after high school nevertheless remain essential for developing a studentâ€™s cultural literacy, capacity for empathy, understanding of history, and potential for effective citizenship. Fortunately, these attempted connections to real world applications appear
to function largely as rationales and do not completely eviscerate the substance of the NAEP assessment frameworks, which remains largely intact compared with previous iterations.
â€œReadingâ€ and â€œWritingâ€ are the only strands of English Language Arts assessed by NAEP, and each has its own framework. Thus, the reviews of each framework are presented separately here, with individual grades assigned. The 2009 Reading framework is addressed first, with a discussion of â€œcontent and rigorâ€ that is organized into two subcategories: â€œCoverage of Text Typesâ€ and â€œTypes of Reading Skillsâ€ assessed. A discussion of â€œclarity and specificityâ€ follows; the review ends with a summary and final grade. The Writing Framework is organized in similar fashion except there the â€œcontent and rigorâ€ section is subdivided into two categories titled â€œPurposes for Writingâ€ and â€œFeatures of Writing.â€
Both Frameworks are evaluated against their stated purposes. That is, the NAEP Reading Framework is assessed as a â€œreadingâ€ framework only. It does not address writing, communication skills, research, media, or language, all worthy components of a robust English Language Arts curriculumâ€”but not the territory that NAEP here seeks to occupy. Similarly, the NAEP Writing Framework is evaluated only as a â€œwritingâ€ framework. Therefore, the reading and writing criteria of the â€œEnglish Language Arts Content-Specific Criteriaâ€ (see page 13) are the sole criteria applied to the respective reviews.
Readers should bear in mind that the purpose of NAEP frameworks is to delineate the parameters and content of large scale, matrix sample assessments. In matrix sampling, no one student takes the whole test. Each test-taker responds to different sets of questions, which limits testing time yet still provides a picture of general student performance on a broad range of content and skills.
NAEP Reading Framework (2009)
Content & Rigor
Coverage of text types
The content of the 2009 NAEP Reading Framework focuses on two types of texts:
- Literary texts (fiction, literary nonfiction, and poetry) and
- Informational texts (exposition, argumentation, and persuasive text; and procedural text and documents).
The framework prescribes the reporting of separate subscales for literary and informational text, which NAEP has never done before. The text types do not, however, differ much from the previous framework, which had defined three text types: â€œliterary, informational, and document.â€ The document category in the previous framework has been subsumed under the new informational text category, while literary nonfiction (e.g., essays and speeches) has been moved from the informational category to the new literary text category. This redistribution is significant in the context of what the overall passage distribution looks like in the 2009 framework (see Exhibit 1).
In the twelfth grade, significantly more weight is given to informational text than to literary text.17 Despite the laudable addition of poetry at fourth grade, the framework maintains a disproportionate emphasis on informational texts. Insofar as NAEP is influential in the states, this shift could result in diminished emphasis in the field on teaching literary texts, both fiction and nonfiction. Further, it could have profound and detrimental effects on studentsâ€™ cultural literacy and their foundational skills in the humanities.
The framework supplies details of the various types of text in easy-to-read matrices. For the most part, they represent a reasonable set of expectations for K-12 students, although there is room for improvement. First, some â€œaspects of textsâ€ at eighth and twelfth grades might easily be introduced earlier. For example, rhythm patterns, point of view, and personification, now listed at eighth grade, could certainly be included at grade four. Aspects of fiction at grade eight could also be assessed at grade four, such as flashback and personification. Some aspects at twelfth grade also seem to come late, such as monologue, comic relief, and â€œunconventional use of language.â€
Second, the aspects of â€œargumentation and persuasive textâ€ at twelfth grade are left undefined, despite this subcategoryâ€™s previously stated significance. Finally, the elusive notion of voice is introduced as early as grade four in sub-categories of informational text, but how or why voice is distinct from style, tone, or mood is a much-debated proposition; the inclusion of voice seems unnecessary.
types of reading skills assessed
As for what students are asked to do with the texts, the framework defines three types of â€œcognitive targets, or behaviors and skillsâ€ to be assessed:
- and ÂˆÂˆCritique/evaluate.
These skill categories do not differ significantly from those detailed in the previous NAEP reading framework, but specific skills are not delineated by grade level, which would help clarify expectations. The framework explains:
The cognitive targets remain the same across all three grades on the assessment but the passages and documents about which items are developed will be of increasing sophistication at each grade.
Because the â€œtargetsâ€ do not change across the grade levels, the rigor of the NAEP reading assessment ultimately depends on how particular grade-specific items are constructed and the types of passages that are selected for studentsâ€™ consideration.
Unfortunately, we do not have access to the full range of test items and associated passages at each grade span, but the released items found in â€œSample Questionsâ€ booklets offer a glimpse of the assessmentâ€™s rigor. The sample assessment items at each grade span seem appropriately challenging in terms of the cognitive skills listed above, but the sample reading passages themselves are largely mediocre and not particularly complex. As a result, the sample items overall are not sufficiently demanding.
One welcome addition to the 2009 framework is â€œa more systematic approach to vocabulary assessment,â€ including the calculation of a vocabulary sub-score for the 2009 administration of the test. Although this new emphasis on vocabulary is commendable, the sample questions send mixed signals about rigor. Fourth and eighth grade look to be reasonably rigorous, but twelfth grade does not. For example, the words â€œscornfullyâ€ and â€œsubduedâ€ at fourth grade and â€œdeflectâ€ and â€œabatedâ€ at eighth grade are of an appropriate difficulty level; â€œobligationsâ€ and â€œprecedentâ€ at twelfth grade are not. Words taken from SAT prep lists, such as â€œspecious,â€ â€œattenuateâ€ or â€œcircuitousâ€ might be more appropriate.
Content and Rigor Conclusion
While the framework serves its designated purpose well, namely to be a fair starting point for assessing basic reading comprehension at fourth, eighth, and twelfth grades, it does not do four things that Americans should expect academic standards in reading to do: 1) address expectations for essential early reading skills; 2) detail more explicit aspects of vocabulary development; 3) prioritize studentsâ€™ exposure to outstanding American literature; and 4) detail choices about the quality and complexity of reading passages.
The framework is missing crucial content by not placing enough emphasis on literature and English language conventions. The framework could also cover the content more carefully by distinguishing the skills targets at each grade span, as well as making them more rigorous and grade appropriate. These revisions would also help practitioners distinguish between more and less important content. Since the framework falls short in these ways, it earns a grade of five (see â€œCommon Grading Metric,â€ page 16).
Clarity & Specificity
The reading framework scores fairly well against our criteria for clarity. It is generally clear and usable. However, as noted above, it could be more specific and detailed in ways that would also improve the content.
Despite its general clarity, the persistence of some vague language in the framework makes it difficult to give it the highest marks. For example, the definitions of the text types â€œargumentationâ€ and â€œpersuasionâ€ are confusing. The framework implies that argumentation must, by definition, be persuasive text, and it links the two types together as one sub-category of informational text. Many practitioners, however, would distinguish between presenting an â€œargumentâ€ as a matter of exposition (i.e., making known oneâ€™s position on an issue or oneâ€™s interpretation of a text) and a persuasive essay or speech (i.e., attempting to sway the readerâ€™s opinion, often by including a call to action). The conflation of the two here is likely to confuse practitioners who have been teaching that not all arguments must be persuasive.
The need for more specificity where the content is concerned and the confusion surrounding some definitions result in the framework earning a grade of two.
NAEP Writing Framework (2011)
Like the 2009 Reading Framework, the 2011 NAEP Writing Framework proposes a workforce-oriented rationale for its contents. The framework explains that â€œthe impact of communications technologies has changed the way people writeâ€ and asserts that â€œwriting in the 21st century is defined by its frequency and efficiencyâ€ rather than by its style, elegance, or even persuasiveness.
More specifically, the framework has the following goals:
- To encourage student writers to move beyond prescriptive or formulaic approaches in their writing.
- To assess grade eight and twelve studentsâ€™ writing using word processing software with commonly available tools. ÂˆÂˆ
- To measure studentsâ€™ ability to respond to a writing task in an on-demand scenario.
In short, the framework emphasizes studentsâ€™ ability to make choices about how to respond to a writing taskâ€”sometimes using a computerâ€”and do it quickly.
Content & Rigor
purposes for writing
The 2011 NAEP assessment measures three communicative â€œpurposesâ€ rather than the three â€œmodesâ€ of writing that previous Frameworks sought to gauge. The change is mostly semantic: The ultimate substance of the frameworkâ€”what is assessedâ€” appears relatively unchanged.
The three modes of the 1998-2007 NAEP writing assessment were:
- Persuasive mode:
- Writing to convince
- Writing to construct an argument
- ÂˆWriting to refute a position
- Informative mode:
- Narrative mode:
- ÂˆFirst-person and third-person fictional stories, personal essays
The 2011 frameworkâ€™s three purposes are:
- To persuade, in order to change the readerâ€™s point of view or affect the readerâ€™s action; ÂˆÂˆ
- To explain, in order to expand the readerâ€™s understanding; and ÂˆÂˆ
- To convey experience, real or imagined, in order to communicate individual and imagined experience to others.
The framework offers only very general descriptions of the three purposes. It discusses when and why people write for these purposes, but does not detail specific writing expectations by grade levels as the NAEP reading framework attempts to do.
The relative percentages of items devoted to each purpose seem appropriate (see Exhibit 2).
EXHIBIT 2. Percentage distribution of communicative purposes by grade
|Grade||To Persuade||To Explain||To Convey Experience|
There need not be an emphasis on writing literature (i.e., â€œwriting to convey experienceâ€) in the writing framework like there is on reading literature in the reading framework. In other words, students should not be expected to write sonnets or great literary pieces. Emphasizing persuasive and explanatory purposes for writing, then, makes sense. The emphasis is also consistent with the content criteria for writing (see â€œEnglish Language Arts Content-Specific Criteria,â€ page 13) which state that students should â€œ…produce writing that reflects the defining characteristics of various grade appropriate writing genres.â€
Compared to the previous writing framework, the emphasis on persuasive writing has increased slightly at grade four (30 percent in 2011 versus 25 percent in 1998). The framework notes that this addition was deliberate, in order to â€œscaffoldâ€ the emphasis on persuasive and informational writing skills at later grades. The â€œEnglish Language Arts Content-Specific Criteriaâ€ place similar emphases across the three grade spans, moving from â€œnarration, expositionâ€ as an emphasis in kindergarten through fourth grade, to â€œargumentâ€ in five through eight, and â€œpersuasionâ€ at nine through twelve. The writing purposes and the time accorded to each in the NAEP writing framework are therefore sensible and on-target.
Features of Writing
Within these types of writing, the framework identifies three very broad categories of features of writing:
- Development of ideas
- Organization of ideas
- Language facility and conventions
Further criteria for assessing writing in each of these â€œbroad domainsâ€ are included, but they, too, are rather vague, mentioning only such aspects as â€œdepth and complexityâ€ (for number one above), â€œcoherenceâ€ (for number two) and â€œvoice and toneâ€ (for number three). These terms alone are too broad to be useful. Further clarification is attempted in later prose descriptions, but alas, these, too, are nebulous and lack sample responses. For example, when describing â€œApproaches to Thinking and Writing,â€ an aspect of â€œDevelopment of Ideas,â€ the framework provides this befuddling statement:
Specific approaches to thinking and writing will not be specified on NAEP tasks, but responses will be evaluated for their use of effective approaches in relation to the development and organization of ideas.
The framework appears to rely on sample writing tasks (or â€œpromptsâ€) more than on these descriptions to exemplify its expectations. This is to be expected since one of NAEP frameworkâ€™s primary roles is to inform assessment (see â€œNAEP Overviewâ€ in the Appendix, page 52).18 Even more helpful, however, would be the addition of sample acceptable responses to the tasks (see related â€œClarity and Specificityâ€ discussion below).
The prompts generally provide context and clear directions and many are text-based in some way (meaning the tasks are grounded in specific reference points and therefore provide a â€œlevel playing fieldâ€ for responding). In terms of actual grading, however, the six-point scoring rubrics are â€œholistic.â€ This means that specific weights are not assigned to each of the criteria (many of which address the features of writing discussed above), making it difficult to discern where students are doing well and where they are struggling. For example, a studentâ€™s response could be well-organized, yet inaccessible because of poor sentence structure and mechanics; another response might boast impressive style, but lack development of ideas. The holistic rubrics
do not differentiate among these priorities; they could be made more useful if they were prompt-specific and accompanied by sample acceptable responses.
One overall disappointment is the very general nature of the expectations for English language conventions. (They are absent from the reading framework altogether, even though careful analysis of sentence structure and grammar certainly aids reading comprehension.) Making matters worse, English language conventionsâ€”the most consistently noted concern of employers and postsecondary facultyâ€”can easily be overlooked with holistic scoring. NAEP should find a way to articulate its expectations in this critically important area in much greater detail.
Content and Rigor Conclusion
Some of the content in the NAEP Writing Framework is covered satisfactorily. Specifically, it does an adequate job of identifying the appropriate purposes for writing and placing reasonable emphases on each purpose at the various grade levels.
While the majority of the appropriate content is covered by the standards, however, it is shortchanged in important ways. First, more attention to conventions would help make the framework as rigorous as it should be. Second, since the rubrics repeat much of the same broadly-worded expectations from grade to grade, it is difficult to distinguish between more and less important content. It is also possible that the standards need to be more or less rigorous than they are at certain grade levels (i.e., expectations are too high or too low) but, given the one-size-fits-all expectations, it is impossible to tell. Because of these limitations, the writing framework receives an overall content and rigor grade of five (see â€œCommon Grading Metric,â€ page 16).
Clarity & Specificity
The writing framework scores rather well against our criteria for clarity. The guidelines for writing tasks are straightforward and coherent, and, as mentioned above, the sample prompts help clarify expectations. That said, illuminating a scope and sequence of writing skills by grade span would provide better guidance to educators. More specificity at these important benchmark grades would also help indicate what expectations should look like in the intervening grades.
The frameworkâ€™s generic scoring rubrics could be strengthened by making them specific to a prompt. The framework could also include sample â€œanchor papersâ€ (i.e., examples of student responses that are used to train scorers) so that practitioners can understand how the rubrics are applied (e.g., whether one area of the rubric appears to be more heavily weighted in scoring than another).
Finally, rather than simply sketching the purposes for writing and offering generic rubrics for scoring the items, the framework could offer a matrix of more specific writing expectations by grade level, as it does for reading. It might mention the kinds of sentence structures, transitions, and stylistic devices it expects to see in writing samples at each grade span. Without offering sample acceptable responses at each cut score, it is difficult to determine the actual level of rigor of the items. This lack of specificity results in a two for clarity and specificity.
Summary and Grades: NAEP Reading and Writing
Policymakers could easily draw upon the conceptual frameworks delineated by NAEP for both reading and writing when developing national or state standards for English language arts. However, any such effort must keep in mind that actionable standards (at whatever level they are developed) must ultimately address with specificity all the domains of English language arts, and at each grade level, in order to be useful. Such an effort must, in particular, focus more than these frameworks do on literature (especially outstanding U.S. literature) and English language conventions. Using these NAEP frameworks as models for state standards might maximize studentsâ€™ performance on NAEP. But it may also lead to an English curriculum that slights both literature and English language conventions, jeopardizing studentsâ€™ chances of developing a strong foundation for a broad liberal arts education.
Our Review OfNAEP Framework
Read moreABOUT THIS REPORT
- Reading framework for the 2009 National Assessment of Educational Progress
- Reading assessment item specifications for the National Assessment of Educational Progress
- National Assessment of Educational Progress 2009 sample questions booklet: Grade 4 mathematics, reading, science
- National Assessment of Educational Progress 2009 sample questions booklet: Grade 8 mathematics, reading, science
- National Assessment of Educational Progress 2009 sample questions booklet: Grade 12 mathematics, reading, science
- Writing framework for the 2011 National Assessment of Educational Progress, pre- publication edition
- Writing specifications for the 2011 National Assessment of Educational Progress, pre- publication edition
Our review ofState Standards