Teaching Connection #112
Qualitative Assessment in the Department of Languages, Literatures, & Writing
When assessment of student learning comes to mind, most people assume quantitative assessment. Yet the assessment team in the Department of Languages, Literatures, & Writing realized that the learning accomplished by their students can’t easily or effectively be measured by or translated into numbers alone, particularly given the diversity of students that the department serves.
To gain more insight into new approaches, the entire faculty in the department read Writing Assessment, Social Justice & the Advancement of Opportunity, edited by Mya Poe, Asao Inoue and Norbert Elliot, which revealed that qualitative analysis can be a more inclusive and equitable way of gaining deeper insight into students’ learning. Taking this more inclusive approach was particularly important for this Regional campus department that serves larger numbers of underprepared students and students from low-income communities and under-invested school systems.
So, how are they going about this? They decided to supplement their traditional approaches to assessment with a new qualitative method.
Their current assessment plan focuses on the learning outcome, “Analyze complex tests within their social, historical and political contexts using the technique of close reading” and features a close reading assignment in EGS 410 as the means for assessing this outcome.
As a new addition to their assessment processes, they incorporated into the course a new assignment: a written reflection of the student’s process of close reading that would be completed after the students produced the written textual analysis essay. The LLW assessment team gathered artifacts (written essays and reflections) from 21 students. Team members continued their standard quantitative assessment method– that is, scoring each essay with a rubric that included four quality levels and descriptions for each level. The rubric for this outcome is below:
| Criterion | Mastery (4) | Accomplished (3) | Developing (2) | Beginning (1) |
|---|---|---|---|---|
| Analytical purpose |
Imaginative analysis richly supported with well-chosen textual detail; more than one kind of literary evidence may be drawn upon Demonstrates a clear purpose and argument |
Analysis stressed over summary/paraphrase; the discussion stays within the scope of the text and primarily focuses on literary structures and issues; may rely on limited evidence or only one kind of evidence Demonstrates a clear purpose and argument |
Either summary/paraphrase predominates over analysis, or extra-literary premises drawn from beyond the scope of the text predominate over textual evidence Confused, shifting, or unclear purpose and argument |
Little to no significant analysis, or a combination of excessive summary/paraphrase and extra-literary, extra-textual premises No demonstration of purpose and argument; writing does not speak to assignment prompt |
| Integration of textual material | Quotations and paraphrases introduced with sufficient background information and signal-phrases when necessary; quotations smoothly integrated into the flow and syntax of the prose; no significant errors in documentation conventions | Various quotations and/or paraphrases present; textual material well-chosen but not always smoothly integrated or not always introduced with sufficient background information and signal-phrases; there may be some errors in documentation conventions but not so as to impede comprehension | Some quotations present, but either not clearly introduced or not smoothly integrated, or both; a significant number of errors in documentation conventions or a few errors severe enough to impede comprehension | Little to no directly-quoted textual evidence, and/or little to no attempt at integration; serious errors in documentation conventions |
| Development of argument | Full and rich discussion of connections between evidence and claims (e.g., the implications of imagery or connotations of language); different kinds of evidence woven together to support claims; no significant redundancies, digressions or irrelevancies; strong and clear logical relationships among subtopics If assignment requires students to draw on external resources, resources are effectively used to engage in a critical conversation; writer adds something new to the conversation |
Adequate commentary clarifying or explicating the relationship of evidence to claim (e.g. discussion of tone, imagery, etc.); there may be minor redundancies, digressions or irrelevancies but not so as to impede comprehension or persuasiveness; logical relation of subtopics may be reasonable but under-developed
If assignment requires students to draw on external resources, resources are effectively used to engage in a critical conversation |
Some explanatory commentary provided, but evidence not always clearly or fully linked to claims; or the link is ambiguous or the explication is under-developed; or there are significant redundancies, digressions or irrelevancies that can impede comprehension or persuasiveness; or some problems with the logical relationship of subtopics If assignment requires students to draw on external resources, resources are not applied consistently/effectively |
Little explanatory commentary provided to link evidence to claims; or commentary not related to the link between evidence and claim; or there are major redundancies, digressions, or irrelevancies; or subtopics seem logically unrelated to one another If assignment requires students to draw on external resources, no resources are used |
| Clarity and quality of writing | Uses graceful language that skillfully communicates meaning to readers with clarity andfluency, and is virtually error free. | Uses straightforward language that generally conveys meaning to readers. The language in the document has few errors. | Uses language that generally conveys meaning to readers with clarity, although writing may include some errors. | Uses language that sometimes impedes meaning because of errors in usage. |
Yet in addition to this method, they elected to assess the reflection statements using a qualitative approach: grounded methodology. Grounded methods are open-ended evaluations of student work in which faculty move away from grading or other quantifying assessments and instead focus on communicating to one other the quality of the writing, including writing issues and concerns as well as strengths. From this discussion, several themes emerged such as “open-mindedness” (being open to new meanings), “understanding textual elements” (symbolism, details, organization, context), “grasping the meaning of texts,” “strategic approaches” (use of language, rhetorical strategies, word choice), “historical/social understanding of texts,” or “connections to personal life.” These themes provided the framework for a coding schematic so that the themes were then counted across all 21 artifacts.
This coding and conversation yielded insights into strengths (understanding textual elements and strategic approaches) and possible areas for improvement (social, historical and political contexts) in students’ capacity to analyze and interpret texts. This qualitative approach paired with the quantitative analysis of the written essays provided faculty with a new recognition: “student learning should not be isolated to or assessed by one artifact; a more equitable and comprehensive approach to assessment requires an assessment of the students’ learning ecology,” including “students’ own insights into their learning.”
For more information on this program’s assessment activity, contact Dr. Kelli Johnson, Associate Professor, Department of Languages, Literatures, & Writing at johnso58@miamioh.edu.
September 2023