Friday, July 30, 2010

What is success? And, how important are outcomes?

In my work I spend about 25% of my time supervising undergraduate peer mentors.  Peer mentors support students in their transition to the university, connect them with university resources, and become sounding boards for new college students as they learn how to learn.  Not suprisingly, these peer mentors are largely high-achieving, highly engaged students who do the sorts of deep learning that we wish happened for every college student.  

I meet with these peer mentors every other week to hear about their work with students, discuss challenges they face, and support them in their development.  One of the most frequent things I hear in these 30 minute conversations is "I'm not really sure if I'm making a difference" or "my students don't respond to me."  Essentially, the question these peer mentors ask is "Am I doing a good job?" and "How will I know?"

Those are fair questions, questions we have all likely asked at one point or another.  As a very young and very inexperienced high school soccer coach, I asked this question after almost every game we played (especially after losses).  Any time we lost I found myself in a dilemma:  I believed I was a good coach and that I worked hard to prepare my players for matches, so what did it mean when we lost?  Was the fact that the opposing team had scored one more goal that we did evidence that I wasn't successful?  These sorts of questions weighed more heavily on my mind when we lost 4 out of our last 5 games of the season after going undefeated up until that point.  Teachers face the same sorts of questions when they work hard to help students learn, but are then faced with test scores that are deemed unacceptable or below par.  

There was a time when I viewed teacher performance in very black and white terms:  If they can't get test scores up, then they don't deserve to be in the classroom.  I've softened my stance over the last couple of years and I'm not really sure anymore.  I don't know that I have a very good definition of what success is for teachers, peer mentors, or coaches.  My biggest question is how much attention we should pay to outcomes (i.e. test scores, pass rates, win-loss percentages, etc.).  There is definitely a place for measurment and assessment.  We live in a world that values numbers and percentages.  We like to count, sort, and rank things.  We believe that doing so gives us a basis to make accurate objective decsions about quality.  What's more, in an economy where resources are sparse, numbers matter a great deal.  But, when our work (and hearts) are wrapped up in helping human beings grow, learn, or change, a purely outcomes-based focus can be incredibly discouraging at those times when the people we work with "underperform."  

This post has really been a long and inarticulate way of asking how we can know when we're doing a good job as educators.  I still think that there are some key outcomes that we need to pay attention to, but some other possible indicators of success might be

1.  Effort, preparation, and thougtfulness.  While we can't control what learners do or how they respond to our invitations to grow, there are some important things that are in our control.  Preparation and intentionality seem to fit here.  Teachers and coaches should work tirelessly to be well prepared and to design learning experiences that meet learners where they are and that help move them towards meaningful learning goals.  

2.  Using feedback to improve.  While outcomes and measurements may not be the sole indicator of success, an educator has a responsibility to use feedback to refine the way they teach.  "Failure" (in the form of unsatisfactory outcomes) isn't failure if you use it to get better and change the way you do things.  

3.  Grit and persistence.  How we respond to challenges seems to be an important part of success.  While persistence in and of itself may not constitute success, there is a fair amount of evidence that the gritty, never-give-up attitude when sustained over time, does lead to success.  What' more, initial "failure" or low-performance provides nice opportunities for learning and reflection, and can be a foundation for eventual success (see this post on the virtue of being terrible, including painful footage of Charles Barkeley's golf swing).  

4.  Relationships.  I have always believed that learning is an inherently social pursuit and that relationships matter a great deal.  So, one indicator of an educator's success is the personal relationships they are cultivating with learners.  If a peer mentor has good relationships with their students (or even most of their students), I always take that as a sign that something useful is happening.  Likewise, the fact that my soccer players seemed to respect me and valued our relationship, gave me comfort after losses when I was questioning my value as a coach.  

As I finish this list I realize that this is touch-feely stuff.  I'm also willing to admit that there is danger in defining success in this way because of the slippery slope it can present (e.g. "Who cares if none of my students passed the class, they like me and I worked hard, so I am a success.").  So, how should we define success as educators?  And, how much attention should we pay to outcomes?



Friday, July 16, 2010

Diversifying our approach to problem solving in higher education

"Our Projects, if we are wise, will be myriad and quiet, not a grand few visible to the whole world."

This is a passage from Bill McKibben's recent book, eaarth.  In this part of his book, McKibben is arguing that it's time to shift our global paradigm from growth to maintenance and, that with limited resources, it's too risky to bet on a few, large projects or initiatives.  If we are going to make life on our polluted, resource-depleted planet, we need to minimize risk by finding lots of small ways to improve conditions.  In essence, in the end we'll be better off making lots of small wagers rather than a small number of huge investments.  

There seems to be some wisdom for higher education here--try a lot of different things, minimize your risk, pay attention to what works, and then replicate it if you can.  In highly centralized organizations like universities, however, we don't do well with this sort of thing.  It's more likely that we invest huge amounts of resources in large-scale initiatives that are very public and that carry huge risk.  For example, I've mentioned on this blog before how my institution will be launching a freshman mentoring program this fall wherein every freshman student is provided with an upperclassmen mentor.  This has meant vast changes in the way students register for courses, the development of technological solutions involving hours and hours of development and testing, the creation of a new full-time administrative position, and a comprehensive course redesign for one of the largest freshman courses on campus, not to mention a huge allocation of funds to pay and train the peer mentors.  The objectives of the program are to balance resources across high-demand first year courses, connect students with an upperclassmen who can support them in their transition, and increase the likelihood that freshmen will use campus resources (e.g. advisement centers, teaching assistants, etc.)

I hope that the program is a success because I value mentoring and believe that it can make a tremendous difference for first-year students.  But, McKibben's ideas have me wondering whether a decentralized effort across a variety of departments may have been just as useful in improving the first-year experience for BYU students.  What could academic advisors do in their sphere to help?  Is there a low-cost social media campaign that the Office of First-Year Experience could have piloted?  What if Residence Life had experimented with American Heritage course review sessions taught in the halls?  If any one (or all) of those things fail, no one has lost much time or money and central administration doesn't have to worry about looking bad.  As it is, the message has been "Freshman Mentoring cannot fail" (that sounds eerily simialr to the "to big to fail" rhetoric we've heard surrounding the automotive and banking industries as of late) and the individuals responsible for making it work are faced with making the impossible possible.

I recognize the value of campus-wide, coordinated initiatives where a diverse group of stakeholders work together to address a campus issue.  I hope that is what our Freshman Mentoring initiative is and that it works.  If not, we will have spent a lot of time, money, and social capital on a disastrous failure.  

So, the question I'm really asking here is when large-scale, resource intense initiatives are the answer, and when we'd be better off encouraging small innovations across a number of departments and areas in hopes that they add up to some sort of aggregate success.  



Friday, July 9, 2010

What schools could learn from cab companies

I spent the first year of my professional career as a teacher.  At the risk of sounding arrogant, I thought I was pretty good--I had good rapport with students, I tried to align my teaching with learning objectives, and I believed that I applied good pedagogical practices in my classroom.  It's probably pretty easy to spot the problem in this assessment--they are all my own personal perceptions of my ability and performance, and very subject to bias and inaccuracy.  This has bothered me lately and I've wondered how good a teacher I really was during that year.  As I look back on those experiences, two really important things seem to have been missing:  feedback and focused effort to learn from mistakes.  

While I tried to regularly evaluate my own teaching, like most of us I probably overestimated my abilities and was likely unaware of many of the mistakes I was making.  On a few occasions (three, that I can remember) I was observed by another teacher and then given some basic feedback at the end of the class session.  It's fair to assume that both my own evaluations and those of others might have had some slight improvement on my teaching.  But, in retrospect, I don't think I was much better in June when the school year ended than I was on the first day of school in September.  In fact, I probably developed some bad habits, got too comfortable with the role, and stopped doing some of the little things that make a big difference in one's teaching ability.  In short, I may have even been worse.  

At the same time I was having these depressing thoughts, I was reading Traffic, by Tom Vanderbilt.  The book uses driving patterns and habits as a context for exploring human behavior.  He tells a fascinating story about how cab companies and limo services have helped improve their drivers' performance using a technology called DriveCam.  DriveCam installs cameras on the rearview mirrors of cars that continusously buffer images (like TiVo) of what is happening both inside and outside the car.  Sensors monitor various measurable forces and when a "trigger" is detected (sharp turn of the steering wheel, significant decrease/increase in speed, etc.), the camera records ten seconds of footage both before and after the trigger.  This footage is then sent to a database and may be reviewed with the driver in an attempt to correct mistakes and improve safety.  Although it probably ruffles some feathers of drivers who feel like their privacy is being invaded, the bottom line is that transportation companies have a lot to lose when their drivers don't drive well.  What's more, this is great learning.  Drivers can view actual footage of their driving, spot mistakes and fix them, and focus on small elements of the driving performance that seem to make a big difference in achieving good outcomes, namely safety (this isn't unlinke the way elite athletes use taped performances to improve).

This left me wondering why something similar couldn't happen in classrooms, particularly those classrooms led by novice teachers.  We spend millions of dollars to equip classrooms with new technologies that are touted to improve student learning and increase engagement.  And, while ipods and laptops can help, the core factor influencing student learning still seems to be the teacher.  It seems fair to ask why technology can't be used to imrove core teacher practices that would have far-reaching impact upon student learning.  There is some work being done in this area (see the work of Peter Rich and this pilot project at the University of Central Florida), but it  seems to be on the periphery.  

What would happen if there were cameras installed in classrooms that could capture real footage of teacher performance.  Are there "triggers" that we would want to focus on?  What would they be?  Something like this would seem to fill a gap in current teacher development practices.  Teachers would have the opportunity to really see themselves teaching (as opposed to their perception of their teaching or someone else's interpretation), use mistakes to improve, and focus on critical parts of their performance that have been shown to lead to significant improvements in student learning.  










Friday, July 2, 2010

Measuring what matters: How much should we really care about retention?

In his February 2010 TED talk, CEO and author Chip Conley tells a fascinating story about the nation of Bhutan and their transformation from an isolated, undeveloped nation to a modern, technologically rich nation that still manages to maintain the essence of their original culture and traditions.  Bhutan's story is one of striking the balance between progress and innovation, while stilll maintaining core elements of an identity (a rare feat for any country, organization, or school in today's rapid-paced world).  One of the most interesting parts of Conlee's telling of the story is his reason for Bhutan's ability to transorm in these ways.  Quite simply, he believes that it is because Bhutan has learned to "count" the right thing, gross national happiness.  Forty years ago, Bhutan's King coined the term rather off-handedly to describe his commitment to building an economy that would allow for growth, while staying true to Bhutan's Buddhist roots.  Bhutanese officials ran with the concept, developed sophisticated instruments to measure the concept, and used it as a model for the development plan that brought Bhutan into the 21st century.  This all stands in stark contrast to most nations' preoccupation with Gross Domestic Product and their belief that it stands as the supreme indicator of a nation's well-being.

There seems to be a lesson for higher education in all of this, particularly the first-year experience movement.  Like anyone else, we count what is easily countable.  So, in many ways "retention" has become our GDP.  We work hard to measure it, argue over how it should be measured, showcase (or hide) it in reports to our administrations, and tout it at conferences.  This isn't to say that we shouldn't care about retention--the reality is that enrolled students bring money to the institution and that money keeps us running.  But, there seems to be some danger in retention becoming what Conlee describes as a "misplaced metric," an easy to count measure that gives little indication as to the real health of an institution.

So, what should we be counting?  In many ways this question hinges on how we define success in the FYE movement and the factors that we believe contributes to a vibrant campus community.  What does a "successful" student look like at the end of their first year?  What skills, habits, and attitudes would they possess?  While the National Survey of Student Engagement (NSSE) has contributed greatly to institutions' ability to measure certain behaviors and attitudes of students,  we don't seem to do much on our individual campuses to measure the equivalent of the "gross national happiness" for our campuses.  

What would these "intangibles" on our campuses be?  While they will vary slightly across institutions, some possibilities might include

A personal reason for being at a particular institution.  Do students know why they decided to come to your school?  Their purpose and commitment to the educational ideals and objectives at your institution will make a huge difference in their engagement and persistence.  If they don't have a set of fairly good reasons for choosing a particular campus, there is likely to be trouble down the road.

Understanding of and investment in an institutional mission.  This seems strongly correlated with the idea above, but it seems important for institutions to not only orient students to their physical surroundings, but to help them understand the culture and ideals of the institution they have enrolled in so they might become a part of the community and fulfill their role in it.  So, if you are a faith-based institution that espouses character development, do students believe in that mission and pursue that growth?  For liberal arts institutions, do your students value a well-rounded education and recognize the importance of breadth in their learning?  

Passion for learning.   What students believe and feel about learning are important.  We want life-long learners that continue to grow and make contributions to society after they leave our institutions.  Can the first-year experience nurture this passion?  How would it be measured?

This is obviously not an all inclusive list.  But, these would seem to be key indicators of the success of a FYE program.  There are others including deep learning behaviors, formation of supportive mentoring relationships, and the development of grit and persistence.  We probably can't measure them all, but what for you are the key indicators on your campus?  What could we start measuring on our campuses that would be meaningful and give us real insight into the success of our FYE programs?  

Universities need leaders who know what to count.