The first time that I read Keston Fulcher’s “Weigh Pig, Feed Pig, Weigh Pig” article, it transformed the way that I approached student assessment. In a world filled with complex models, psychometrics, and assessment theories, this message is refreshingly simple. Assessment (weighing the pig) does not produce learning improvement (a fattened pig), curricular/pedagogical changes (feeding the pig) in response to assessment data does. Moreover, if we do not re-assess (weigh the pig again) once we have intervened (fed the pig), we cannot know if our efforts were effective in improving learning (fattening the pig).However, this analogy quickly breaks down if the scale used to weigh the pig (measure outcomes) is unreliable.
Therefore, in order to foster a culture of improvement, institutions must first establish an effective scale. In higher-ed, assessment designs (ADs) serve as the blueprints for measuring outcomes. Poorly crafted ADs produce unreliable data that proves useless in driving meaningful improvement. However, ensuring quality across your campus is no small task. In order to address the many challenges associated with designing assessments, consider incorporating the use of research questions. This practice can be instrumental in promoting faculty engagement, ensuring alignment with learning outcomes, defining the scope of the assessment, and providing context to various stakeholders. Put simply, a research question (RQ) represents what faculty want to learn about student learning in the context of a particular learning outcome (LO) assessment. Here is an example:
LO: The student will be able to defend a philosophically informed worldview.
RQ: Is the student able to construct a valid argument for (or against) a view of the human soul that allows for the long-term (i.e. immortal) persistence of immediate conscious awareness even after one’s body has died?
Let’s take a look how research questions overcome the challenges described above and pave the way to a culture of improvement.
- Faculty engagement
Requiring faculty to begin the assessment design process by formulating a RQ fosters a culture of intellectual inquiry rather than compliance. Asking faculty what students need to learn in order to be successful captures their intrinsic passion for preparing students to impact their field. When faculty are assessing what they value, they are engaged and have a vested interest in the success of the assessment.
- Alignment with learning outcome
As faculty ask questions about their LOs, they begin to form stronger connections between the LOs, the curriculum, and student learning. During the process of creating a RQ, faculty must identify the level of learning being called by the LO (according to Bloom’s Taxonomy) and ensure that what they want to discover (represented by the RQ) is consistent with those expectations. This sensitivity to alignment extends itself to key items in the assessment including the instrument and rubric, resulting in improved alignment across the design.
- Defining the scope of the assessment
LOs represent a large domain of knowledge/skills that a student will possess upon completion of a program. Attempting to assess an entire LO in one assessment can be difficult to manage and, with the exception of a capstone, it is nearly impossible to do via a single course-embedded assessment. Instead, ask programs to identify a particular subskill or piece of knowledge related to the LO that they want to assess. RQs provide the space to articulate this distinction which narrows the scope of the assessment and allows faculty to focus intensely on a particular skill.
- External stakeholder reviews
The use of RQs at my institution played a pivotal role in the success of our recent regional accreditation review. In addition to improved alignment, RQs possess a contextual value that helps external reviewers comprehend the design. Reviewers were able to analyze samples from unfamiliar disciplines and readily grasp the nature of the assessment. For example, as someone trained in education, I may not know what it looks like to “defend a philosophically informed worldview.” However, the RQ translates the outcome into the common language of student learning, allowing someone from any educational background to follow along.
Stephen Stafford is an Assistant Director of Institutional Assessment for Institutional Effectiveness at Liberty University. This article is the second in his student assessment series for Higher Ed Live. Read the first article, “4 Ways that Assessment Improves Student Learning.”