In a recent posting on his blog, John Gardner raised some intriguing questions about measuring the impact of our work with first-year students (e.g. How do we measure whether or not students love their university experience? How do we measure transformation?). These are, no doubt, important questions for First-Year Experience professionals to ask. And, if we see our work as more than just boosting retention stats, they are questions that we have to grapple with.
The trouble with questions about “love,” “transformation,” or “the impact of a mentor” is that these things are tough to measure. Granted, researchers can operationalize concepts like love and develop instruments that measure proxy indicators that the more fuzzy concepts are present (e.g. we measure “engagement” by measuring a variety of indicators including time spent working outside of class, interactions with faculty members, study w/ peers, etc.), but this approach has its problems. Before readers’ red-flags go up, this is not a rant against quantitative research or operationalism—this type of academic work has its place and helps us to answer critical questions about student experience. However, while p-values, standard error estimates, and confidence intervals, are sometimes seen as the coin of the realm in academia, for some questions (i.e. those mentioned by John), particularly those with deep personal relevance for us, the traditional quantitative approach can leave us feeling unfulfilled, still questioning whether or not our work has really mattered in the lives of individual students.
The good news is that there are other research approaches coming to be recognized as just as viable as more traditional quantitative approaches. One that may hold particular promise for FYE researchers is narrative inquiry. A blog post is not the place to attempt an exhaustive explanation of this discipline; rather, my intent here is to provide a brief sketch outlining the general landscape of the field. For those wanting more depth (and much more sophisticated) than is found here, these are some good places to start:
Narrative inquiry is a sub-discipline within the broader field of qualitative research. Based on the premise that storytelling is an inherent human quality and that we live “storied lives,” narrative researchers use narrative data sources (e.g. story-telling, journals, field notes, interviews) to study how we experience the world and then make meaning of these experiences. As an example, John’s question about love for an institution could be addressed by interviewing graduating seniors, soliciting stories from sophomore students, or analyzing journal writing from a freshman seminar course, then identifing meaningful themes that cut across the entire set of “data.” Ultimately, the researcher would aim to use these narrative sources to construct a “meta-narrative” that tells an overarching story related to the initial research question.
As with any research methodology, narrative inquiry has its critics. Common criticisms of narrative inquiry include a lack of generalizability, reliability, and validity (similar to the arguments against most qualitative approaches) and these are all fair claims. However, as a general rule, narrative inquirers don’t view these criteria as goals of their research. Rather, their focus is on transferability, apparency, and verisimilitude. To narrative researchers, the narrative approach is a way of addressing personal and human dimensions of experience in ways that quantitative data cannot—they are working from a completely different set of assumptions about research. Jean Clandinin and Michael Connelly (arguably the most well-known and well-respected narrative researchers) put it best in their paper “Stories of experience and narrative inquiry” when they said that “stories stand between the general and the particular, mediating the generic demands of science with the personal, practical, concrete demands of living” (1990).
Lest this post be viewed as a one-sided push for a problem-free research approach, it is important to recognize narrative’s dangers and pitfalls. Because narrative asks researchers to construct stories from qualitative data, the possibility exists for individuals to fake the data and concoct pure fiction, based on nothing but unfounded assumptions and only anecdotal data. Additionally, narrative could be used to misrepresent experience and promote deception. Finally, some narrative researchers (particularly when stakes are high) may present only “Hollywood stories” where, for example, students love everything about their university experience, become deep learners, and are transformed by their college experience for the rest of their lives. But, it bears mentioning, that similar claims could be made of virtually any approach to research, even quantitative methods.
While narrative inquiry may not have the tables and statistical significance that sways some political and campus leaders, there is rhetorical power in human stories that we can relate to and that elicit similar stories from our own experiences. What’s more, narrative inquiry is gaining traction among educational researchers, as evidenced by the recently formed Narrative Research SIG within AERA, the convening of the first ever Narrative Inquiry conference this last May, and the well-respected work of Clandinin and Connelly. Most importantly, narrative offers an approach for answering those questions, like John has posed, that are deeply meaningful to us personally, and whose answers we use to measure the impact of our careers.