Center for Academic Innovation
Lessons learned:
Trust is key
Diversity is strength
Stakeholders are central
Stakeholders
I let our stakeholder know we were excited.
Building that relationship is important.
Getting started
We began by asking, “who’s on our team, what are our strengths, and what is our purpose? We had structured discussions to answer this. Here are three artifacts.
Team
Strengths
Purpose
Team dynamics
An early meeting with the team.
We came with different backgrounds and strengths with at least one thing in common. We were all really busy.
Initial client meeting
This was a chance for us to clarify the scope and context of the project by asking the client follow-up questions.
To prep for the client meeting, we made a list of questions and goals. These were pretty general. “Tell me about your role,” “what does the workflow at CAI look like?” I also suggested to the team we leave the client contact with a 1-page-handout so he has some context about us (what our strengths, backgrounds, and research plans are). Sometimes an old-fashioned printed paper is better than email.
After the initial contact, we had context to ask deeper questions. In a follow-up email, we asked for a few documents to help us further understand the problem.
Celebrating small wins
Shree and I gave each other a virtual high five in an informal debrief after the meeting. She had six years of work experience and did an excellent job keeping time, framing the agenda, and holding focused discussion—and I made sure to tell her that. In turn, she noted that I helped keep the meeting focused and brought creative ideas to the team.
Interviews
We had this question, “how can we improve workflow at CAI?” We had some ideas but still a lot to learn.
Selection criteria
Our selection criteria for interviews included stakeholders from three different areas:
Student Residents in the certificate program
Learning Experience Designers (LXDs)
Leadership
Though we didn’t initially have a Student Resident participant, we later got one. We noted the importance of including Student Residents in future research.
Interview protocol
For each of stakeholder type, we wrote an interview protocol with necessary preamble (“can we record?”), research goals, overarching questions, contextual questions, and probing questions.
Recruiting participants
We didn’t have to worry about recruiting. Our client gave us names and emails and we reached out for scheduling.
Having done this a few different ways, I think…
The benefit of the client doing the recruiting is they know their audience. They know who will be “good to talk to” (open, friendly, opinionated).
The drawback is sampling biases. It was our task as researchers is to be aware of and disclose these to the client in the written report.
Just how selective to get on sampling depends on the goals and constraints of the project.
Conducting interviews
I was first to interview an LXD named Jacob. Our notetaker was Shree.
Even though we rehearsed the interview protocol several times, when push came to shove, the questions we wrote bombed. LXDs knew nothing about “workflow” and “project management.” They did, however, have valuable things to say about challenges within projects and working with students in the certificate program.
Warming up with an LXD in a conference room
Sitting with an LXD at their desk looking at a document
Thinking off the cuff
I had to pivot and go off-script. I asked things LXDs knew about, like, “who was the last student you worked with, can you tell me about that?” We got wonderful insights.
This, in fact, led to one of our client’s favorite recommendations: two “student resident personas” that would allow LXDs to tailor meaningful experiences based on students’ unique needs and profile, even as CAI continues to scale.
Collaboration story 1: “I wrote those for a reason”
Shree didn’t like how things were handled, though. We worked out our disagreements, and I reflected on what I learned from that experience here. To move forward, I also proposed we update the LXD interview protocol.
Here is where that landed:
Interview notes
Here, you can see part of my annotated interview notes, as I was thinking through the content (purple) and mechanics (blue) of the notes I took.
Other interviews
I also interviewed the Associate Director (leadership) and was the notetaker for interviews with an Accessibility Specialist (LXD) and a Student Resident.
Collaboration story 2: “you should just go here”
In the interview protocol I wrote for the Associate Director, we wanted to know about the leadership’s strategic planning process. This was really valuable (and I explain why at 1:48 in this video).
While we were sitting with the Associate Director discussing how this might look for the organization, my note-taker Orville turned his laptop screen around and said “you should just go here,” pointing to their organization’s mission and vision page of the website.
Later, in that same interview, he wrote down a question in Sharpie and tried hinting to me with somewhat inconspicuous gestures to read it.
I explained after that I needed to know how the Associate Director responded to the question authentically without our input, and he sort of invalidated her response. Would she have thought of the website independently? We’d never know.
On top of that, it’s just really important to maintain trust and empathy in the interview so the interviewee is comfortable sharing information. I feel we might have compromised that.
But, what was done was done. What mattered was how we moved forward.
Area of growth
As a notetaker, I used to write down every little thing the interviewee said. I’m getting better at being selective and thinking as I’m notetaking.
When I actively think and make connections across the interview, I ask better follow-up questions. That means more rich data for analysis.
Data analysis
Collaboration story 2: “I wasn’t braining”
I suggested interviewers take note of the 3 high-level ideas right after each interview while it’s fresh on their mind to share back with the team. The counterargument to this is you don’t want to seed the data with your own conclusions. I made the suggestion thinking it made sense to recognize our bias early on. If we had doubts, we could pass the idea by our professors or other teams.
Orville strongly disagreed and this was a late-night meeting at the peak intensity of the semester where we were all pretty beat. Orville and I went back and forth on this point but we weren’t getting anything across. It was a case where, really, instead of solving the discussion, we should have let it go. It was just too late in the evening to carry on. I reached out for a quick chat the next day.
TEXT
Analyzing the data
I love thematic content analysis. Honestly, I can’t believe I get to do this.
Once the interviews were transcribed and we aligned on our method, we met several times as a full-team to breakdown the data.
We produced and read transcripts of interviews
We listened to the audio recordings
We paused occasionally to discuss
We entered notes into a spreadsheet: direct quotes, questions, comments, insights
Solving a problem we didn’t expect
It was taking us days to break down one interview recording into affinity notes.
For every three minutes of discussion, we weren’t even getting in 1 minute of the recording. It was taking us three times as long as was recommended by our senior researcher.
I solved this by using a stopwatch and tracking our worst and best times, keeping everyone on track in a friendly, neutral way.
Clustering data in Miro
Once we had good data in excel, we transferred it into Miro to build an Affinity Wall.
We randomized all of our notes
We independently explored
We started forming group
We discussed as a team off and on again
Slowly, we started to form clusters with “lone wolves” and “families”
This KJ Method process is described by Raymond Scupin as 1) label making, 2) label grouping, 3) chart making and 4) writing of explanations.
A Loom video update to the team on Slack
The Affinity Wall
We built an affinity wall, presented it to our colleagues, got feedback, and re-did the whole thing. We thought we could do better, so we made a Version 2. It was exhausting. Looking back, well worth it.
Flow Models, & Working in the Dark
Getting approved
Working in the space
A rough sketch
Higher fidelity flow model with feedback in yellow boxes
The Final Product
Client Feedback
I’m always looking to improve my chops. :-)
How I grew as a UX Researcher:
Building trust on a team early on is really important
It’s okay to own your strengths—everybody brings something different to the table
Working with stakeholders and looping them in on the process is invaluable
Using continuous cycles of improvement is critical to staying sharp (document, reflect, use feedback, iterate)
Show me the receipts
Here’s how I used this experience to build a better team doing even better work this semester.