Library Assessment Conference 2016, Day 2: Is it a frog? A horse? Or both?

Brian Nosek faced multiple accusations of practicing witchcraft and/or being a magician this morning. He invited keynote attendees to informally participate in a couple of well-known perception studies, including the McGurk Effect (in which we tried to discern the vowel sounds we heard or saw in a five-second video clip). He flashed the following image on the screen for less than a second and asked which animal we saw:


What do you see? A frog? Or a horse? Or both?

Nosek pointed out that we were all given the same data – the same image – and yet we came to different conclusions. Once we’d determined the animal in the image, it was hard to un-see it–to look at the same data and come to a different conclusion. It takes effort to see the same information in a new way.

This paradigm was a helpful starting place for the rest of his talk, which focused on the importance of open science. Nosek emphasized that we need not just open access to published research, but open workflows, open data, and open curation. As the Executive Director for the Center for Open Science, he wants to see more researchers adopt habits that support “open, transparent, and reproducible science”–including the use of the Open Science Framework to manage, share, and analyze data. He admitted that he doesn’t use the University of Virginia Institutional Repository because it isn’t part of his workflow. “If you want researchers to use your IR,” he said, “you have to make it part of their workflow.”

Nosek criticized the existing ecosystem of scientific research, including the universities, publishers, funders, and societies who reinforce the dominant paradigm that rewards research for research’s sake. In one of his slides, he wrote, “Incentives for individual success are focused on getting it published, not getting it right.” (The effect of this state of affairs was the focus of his Reproducibility Project, written about in The Atlantic last year.)

As librarians who work with, support, and partner with researchers, the keynote had obvious practical implications for our institutions, and Nosek had a laundry list of ways we could support open science. But what does it mean for us as assessment librarians and practitioners? Nosek said that he asked himself,  “Did I get into science because I wanted to publish things, or because I wanted to discover things?”I would ask the same question to my assessment librarian peers–did you get into assessment to publish things about your library, or because you want to discover new things about users, services, and learning?

Before this trip, I wrote about my interest in Lise Doucette’s paper, and I made sure that I was in the front row at her talk this morning (kudos to Kat Bell for saving my seat while I was in another session!). (I also had the opportunity to bump into Lise during the reception last night, and had a lovely chat with her and her colleagues over barbecue sliders and mini Hershey bars.)

Doucette is currently on research leave at the University of Western Ontario, and she’s using her time to assess assessment work to determine the motivations for the assessment. Her work answers, in part, my version of Nosek’s question–why are you doing assessment in the first place?

In explaining the inspiration for her research, Doucette said, “We often hear of assessment as being a given, or common sense. Sometimes it’s even discussed in moralistic or paternalistic tones, so there is less of a perceived need for giving a rationale for why we do it.”

Doucette said her central research question is this: How do librarians identify and express motivators of assessment work?

To answer this question, Doucette analyzed conference proceedings from the Library Assessment Conference, which included 361 papers from 2006 to 2014. In her sample, she included 10% of papers in each year, for 39 papers total, with a mix of keynotes, papers, short papers represented in her sample.

Overall, she found two categories of assessment motivators–assessment that tries to prove (quality assurance, external) or to improve (quality enhancement, internal).

In her sample, she found that only about half of the papers sought to prove, while over 90% of papers sought to improve the library in some way, and about a third of the papers had both motivators.

I was tickled to see Deb Gilchrist’s 2014 plenary session as one of Doucette’s “improve” motivator examples — Gilchrist asked, “Does information literacy make a contribution to overall learning and transition for pre-college students?” (I was a librarian at Pierce College from 2012 to 2016 and was hired by Dr. Gilchrist.)

Doucette noted there was a strong sentiment of hope about the future in the “improve” motivator samples, with sentiments like, “We did some work, we not sure it will have an immediate application, but we hope it will have some meaning in the future.” In closing her lightning talk, she recommended the following:

Before beginning assessment work, identify
  • What values and goals are motivating your work?
  • What will certain types of results tell you?
  • Will the results be used, and how?
Clearly describe motivators of your work
  • Near the start of the paper/presentation
  • Avoid using empty or buzz words (or define them)

I think these are very reasonable and helpful questions to consider and I’m excited about the work that Lise Doucette is doing; I hope to see her present a longer paper at Library Assessment Conference 2018.


I skipped the box lunch today.

After seeing Lise’s talk, I rushed back over to the Learning session. I was really impressed by the presentations from Rachel Gammons and Lindsay Inge at the University of Maryland and Ann Medaille at the University of Nevada, Reno. Gammons and Inge talked abut their large-scale information literacy instruction program, which includes graduate student-taught course-integrated instruction in nearly all (95%) sections of English 101, with common lesson plans and a shared assessment. Students are asked to synthesize their learning in a tweet (140 characters or less), which you can read by following the #mylibrarymoment hashtag. Gammons and Inge sound like they’re doing very similar work to what we’d like to accomplish at Auraria Library, and I’m selfishly hopeful that we could arrange a conference call soon to compare models.

Ann Medaille’s talk, “Using Images to Understand Students’ Approaches to the Research Process”, shared a study in which 222 students (across programs/disciplines) were asked to draw the steps they completed in a recent writing assignment that required research. The drawings were analyzed for themes and similarities, and nine students about were interviewed about their drawings. They found that many students included depictions of frustration and anxiety, and a third of students drew help seeking behaviors. In interviews, students indicated that they were most likely to seek help from family and friends (rather than campus resources). I am completely in love with this study for many reasons (authentic, student-centered, creative, non-linear, represents research as an iterative/flexible process, it is NOT library-as-place-centered, etc.), and I pumped my fist in the air when Ann said they concluded, “We need to focus more on metacognitive skills.”This forever and ever, yes, please, more. Students need to understand and reflect on their own process, and I believe this is a critical skill that will serve them greatly when they go on to pursue careers in any field.

Still so much more to do today: poster sessions happening now (including information literacy posters, yes!), dinner with Luminary Andrew Asher, and the #critlib chat. (Apparently there’s a sportsball game on TV? Good thing I don’t let things like that distract me.)

God, can you imagine the poor beasts who don’t love what they do? It is something special to be here, surrounded by 640 people who may not love their work, but hell, they give a damn, and I’ll take that any day.

– – –

Listening to: HGTV (’cause I was lucky enough to find an LAC16 conference roommate who loves “House Hunters” as much as I do)

Best Conference Coffee: Nitro Cold Brew from Commonwealth Joe.


Wearing: Neutrogena “Healthy Peach” lipgloss, purchased at the Rite Aid on the other side of the tunnel.