Reframing the User Research Conversation

Who has struggled with implementing user research? Do you have hard sell clients/stakeholders on the value of user research? Are the necessary costs of these activities debated?  Although many speak of current UX practices as the era of user-centered products, we still find ourselves explaining the what and the why of user research and how it is the beating heart of creating a great user experience.

When projects start, the costs and time required to implement user research often takes our stakeholders by surprise. Most often, clients who haven't had experience with user centered thinking and hence, don’t understand what user research entails. They think they want to experience user centered design processes, but often get cold feet at the changes that these processes bring.  At worst, the client doesn’t agree to research, which leaves us with our assumptions unvalidated and our designs untested. At best, we find a way forward, but often our start times are delayed, and expectations still aren’t clear. The London Pivotal Labs Product and Design teams decided to take steps to fix this broken chain and smooth our client on-boarding process to user research.

At Pivotal London, we were a small office for many months. We didn’t have extensive networks to reach into for recruiting participants and our capacity was low.  On a project that was particularly strapped for time, we engaged Criteria, in order to get five interviews scheduled with only two days to prep. Using a recruitment agency meant: 

• We cut the time needed to secure interviews by half.

• We could specify a specific profile of user.

• Our interview subjects were not tainted by personal connection, incorrect knowledge or leading set-ups.

• The attrition rate for interviews went down to nearly zero.

• The client received the invoice for the cost directly.

The office grew and we took on more projects that required a research plan. Many teams came up against a user research roadblock with clients. While using recruitment agencies helped cut down on admin time, sometimes it didn't make sense to use one, which left us double over an barrel when it came to getting user research off the ground.  

We found that different clients had different reactions to the costs of user research. Small start-ups worried about overhead costs. Large corporations worried about deadlines and IP. In most scenarios, getting ongoing buy-in through the course of the engagement was tricky at best.

Fast forward a few months. London Product and Design had tripled in size and had gathered enough information about what our user research challenges were to tackle the problem head on. We now had the capacity to create a strategy for how to bake this process into our engagements, which included using Criteria for all interview recruitment.

We came up with a concept of “Packages,” and workshopped how our experiences fit into what became small, medium and large. Our goal with these packages was to outline our research expectations and process, given the scope of the product, and get client buy-in. The outcome; the costs for discovery + framing and ongoing research were now baked into our engagements.

Clients and businesses always need to understand the costs, constraints and time structures around any work done.  The lack of transparency around the nuts and bolts of how user research happens put it at risk of being accomplished.  This puts our clients at a disadvantage when it comes to creating great products.  We believe that these templates can provide a foundation that can just as easily be used by start-ups and product companies as they can by consultancies.

The London Product + Design Team identified a gap in the sales process, and a risk when it came to delivering great lean practices to our clients. Our hope is that we can continue to work with these concepts and refine them as we get new clients and new products through the door. We would love to know if other offices are using similar techniques, and how these have helped or hindered your product development.

Pivotal Labs London Launches Product Office Hours

Remember Office Hours in University?  If you don’t, it was a time that professors just left open in their calendars and students could book some time during those blocks to come and have a chat.   As a student, you usually went to the office hours for the classes you cared about most, or maybe the class where you were really struggling but wanted to do well. It was part brainstorm, part catch-up, part therapy.  It was a chance to clear your head and perhaps it kept you out of the weeds for a few weeks after.  Sometimes it was the only time you got to talk about a paper or a project that you’d worked on for weeks or months.

Building products, as an entrepreneur or a product manager, can sometimes feel a lot like being a perpetual student.  You are always faced with new information coming at you, you have to synthesize it, make decisions and make the idea your own.  A lot of the time you’re going at it alone or with a very small team, and you often feel pretty isolated.  What do you do when challenges come up, like when sign-ups drop, or your growth levels out?  Who do you talk to?

Pivots have met a ton of CEOs and Product Managers at meet-ups and conferences who are working on great ideas and have really promising products.  We shoot the breeze, take a look at their products, offer ideas, and help them focus on the possible solutions.  The appreciation we receive often seems disproportionate to what we think we offer until it came clear how valuable collaboration and community is when you feel as though you’re doing it alone.  So we formalized the process and rolled out Pivotal Labs Product Office Hours.  This program of lunches, where Pivotal Product Managers and Designers meet with people who are building things and chat about the challenges they face, went live in NYC and San Francisco last year.  We are really psyched that we are now bringing it to our London office in Autumn 2014.

So what’s the deal, really (I hear you ask)?  The deal is that you send us an email to office-hours@pivotal.io, tell us a bit about yourself and your product (Make sure you mention you’re in London.)  Someone will be in touch to schedule our lunch and maybe ask some clarification questions.  Then you come down to our Old Street office on a Friday from 1230-130pm, eat some free lunch and a few Pivotal Product Managers and Designers offer you no-holds-barred advice on how to tackle your most pressing product issues.  Can you come in again in a few weeks to let us know how things are going?  YES, please do.  The lunch is free, the advice is free, the bus fare over costs £1.45.  There’s really no reason not to sign-up and come in.

So what’s in it for Pivotal Labs?  Well, we get to meet awesome people who are doing cool things.  We get to exercise our critique muscles and we get to have cool conversations.  If you want some more specifics, you can check out these blogs about WeeSpring and Notable and how to frame your problem, to get an idea of how these sessions go.  There is no general template, each session evolves in response to the product discussion at hand.

We hope to see you soon, our doors are open.

Great Things Come in Small Sizes: Writing Stories that Work for your Team

When I tell people a big part of my job as a Product Manager is to write stories, I tend to be met with blank stares or quizzical looks. But if you’ve been anywhere near a project at Pivotal Labs, you’d know that stories are the life’s blood of the development process.

Simply put, stories are features that are broken up into small pieces of work. For example, “As a user, I want to log-in to the website, so that I can see my homepage.” These are written by PMs and logged in a Tracker tool. Dev’s then “pick up” these stories and use them to direct their work. A list of stories is called the Backlog and reading a Backlog should give someone a general idea of the product that is being built. There are a number of benefits to clients that stories provide.

• Stories provide transparency. Stakeholders can read the stories in laymen’s terms, and ask questions about what is there and what is not.

• Stories provide control. Clients can look at how stories are prioritized in the backlog and have meaningful conversations about why certain features are more important than others.

• Stories help lower risk. Clients often have limited resources at their disposal and delivering smaller pieces of work at a rapid pace helps clients get their products in front of their audience faster.

When we work with clients that are new to working in an Agile environment, we explain user stories by presenting INVEST.

• Independent

• Negotiable

• Valuable

• Estimable

• Small

• Testable

Out of these, the word “small” is a bit quixotic because the appropriate story size is a range with both an upper and lower bound. So what is a good story size and what are the tradeoffs? At Pivotal, we prefer to have stories that are estimated to take less than 2 pair days. This suits our goals, but because the size range can vary between methodologies and practitioners, we get a lot of questions around what is correct versus not correct. Here is a quick rundown of what we see at Pivotal when we use small stories.

• Flexibility in planning, prioritizing, and cutting scope. One of the important skills we push our clients to develop is to see the complexity and cost of their vision.

• Greater accuracy of the velocity. Velocity tends to be more accurate when the stories are finer grained, because there is less likely to be hidden complexity and issues.

• Forces earlier and better understanding of the overall system.

As mentioned this process of turning stories into as small bits as possible has its trade-offs. Obviously the more moving pieces that you introduce into a process the more complex things can become. Each story becomes part of a larger picture and it is easy to get lost in the leaves. It’s also quite easy to have stories get duplicated, as well as have certain stories be blocked by work that is not completed yet.  But PMs should rigorously audit their backlogs to make sure that all stories are relevant and usable.  Most stories should be written within a few weeks of when a Dev would pick them up.

So what does this look like in practice? After an inception, the project team has a list of Epic stories. These are the higher-level stories out of which smaller stories are created. Here’s an illustration of this breakout process.

“Shopper should be able to recommend a product to a friend.”

After discussing the “how” of this story it can be broken up into smaller stories that capture facets of how a user will accomplish this task.

“Shopper can provide an email address of their friend.”
“Shopper can provide a customized message when recommending something to a friend.”
“A link is sent to the friend of shopper that links back to the recommended product.”

…and so on and so forth.

1 and 2 points story are generally the most common. 3 points story also occur regularly, but much less frequently. Higher points like 5 and 8 are rare and generally are because there’s technical unknowns and risks. At the same time, Pivotal Tracker assumes a starting velocity of 10 points, which also tends to be median for a pair. This translates to stories generally being about 1 pair day or less. Only rarely are stories at 2 pair days or more. Of course this is the estimate, and not the actual amount of time spent and this can change, shift or vary, but the team will learn to adjust estimates as the project moves forward.

Ultimately breaking up stories into the smallest possible increments does is makes it possible to predict with some accuracy when certain features will be complete. It’s never an exact science, and unforeseen things arise that can set predictions off track BUT Product Managers and Devs are able to control the size and breadth of stories. As development is often an uncertain practice, teams should grab onto the practices that help them gather more insight, more transparency and more control wherever and whenever they can. So…keep your stories small.

*Note: This post was started by Brendan Kao and finished/edited by Rosemary King an indeterminate amount of time later. 

Getting to Zero: Strategizing a Validation Process with Mobilise

How do you get started? This was a question that was being asked on both sides of the table at Pivotal Labs London’s inaugural office hours. Our Associate Director Robbie Clutton, Associate Director of Design Martina Schell, and I sat down with an organization called Mobilise Public Ltd to help them hash out their product hypothesis and potential strategy for an MVP launch. Mobilise asked us how these sessions usually went, we replied that the session really shapes itself and asked them to start describing the problem they are trying to solve.

Mobilise is a public sector company that is in the process of transitioning from service delivery to product delivery. Mobilise has extensive experience working with public and community sector in underserved areas and a deep understanding of their various users’ needs and desires. This wealth of experience means that Mobilise has many ideas of what products could be useful. This shift in business model was instigated when Mobilise spotted an opportunity to capitalize on underutilized public spaces that could help raise engagement in local communities. So their question for us was how they should best get started.

Their hunch on how to best evolve their business is building a platform that breaks down the barriers to using community spaces, while simultaneously giving small entrepreneurs an opportunity to engage directly with their own communities. Our aim during the session was to understand what Mobilise’s product hypothesis is for this product. Once you have a clear hypothesis the path to validate that hypothesis emerges. All product hypotheses should ruthlessly eliminate as much complexity as possible and drive to the core idea.

Mobilise’s hypothesis boils down to a simple concept: If the community spaces in question were easy to access and easy to use, then people in the surrounding community would be eager to use them.

After discussing the hypothesis it was clear that Mobilise would be well served by leveraging their existing relationships with community organizations and groups in order to help define the MVP. We identified several opportunities for research and discovery that should help them understand how their core user groups were completing these activities now. These included shadowing and interviews with organizations that have successfully built active communities with minimal help from digital products. It is also crucial that Mobilise get an understanding of what other tools to do similar work, and where gaps exist and what folks would like to see automated vs. what they would want to continue to do themselves.   Lots of companies view this type of intensive discovery work as overkill but with complex problems, unclear user needs, and premium development costs, making sure you validate your hypothesis with specific answers carries its weight in gold.

Sometimes the first step is discovering the first step.

There is No Manual: Changing Your Product’s Rules to Appeal to New Customer Segments

Every company wants to see their customer base grow. This is one of the baseline metrics of success. But if a product is built for one core group of users, how do you know which way to move your product as you try to appeal to a new segment? In our most recent product office hours, we spoke to a company called Effektif about this issue and how they could plot their path forward.

Effektif has spent the last few months building a BPM tool that could be billed to Enterprise level clients as the “simpler” solution. After a brief perusal of the product, it was clear that compared to market competitors like Salesforce and SAP, Effektif was a more straight-forward tool. In addition to continuing to improve the product for their Enterprise customers, Effektif wants to expand their market to small and medium sized companies. As Effektif’s rep, Peter Hilton says, “we see simpler tools as both the future for big companies and a way to introduce BPM to smaller companies,” which is a great hypothesis for them to validate.

It was clear that Effektif understands the Enterprise market really well. Their tool didn’t require massive tutorials, but there was an expectation that their customers would receive training and would check out a manual. The usability of Effektif’s BPM reflected this. There were several features and flows that I didn’t even notice until Peter gave us a demo and navigation was not intuitive.

I talked to Peter about my experience as a Product Manager at a smaller company and how I would have used this tool. A lot of the product’s value wasn’t initially apparent, but after I saw the demo I thought of several instances where it would have been incredibly useful.

During our session, we encouraged that Effektif stop thinking about how to market the value of their Enterprise BPM to small businesses, and start thinking about how to evolve a version of the tool into a Small-Business BPM. Applying old rules to new customers doesn’t work, you’ve got to create a new foundation of understanding. Effektif needs throw out their assumptions about small businesses and carve out time to discover what’s valuable and what’s not so important. This means getting out of the building and having conversations with people who work at small companies, which will drive out a new rules like, “I don’t have time to do a training,” “I need to be able to easily delegate these tasks without extensive discussion,” and “I really need a BPM that has X template.”

Without a solid understanding of what a new customer segment needs to accomplish, the chances of creating an indispensable product are slim. Kathy Sierra, one of my favorite product people once said, “People don’t want to be great at your product, they want to be great at the context.” In Effektif’s case, people want to be great at their jobs, and they want a tool that doesn’t get in the way of that. Attempting to capture new customers means throwing out the old rule book and finding the stuff that makes your product’s value crystal clear.

Open Signal: A Case Study of Rapid Fire User Research Overhaul

As a tech consultant, I help companies optimize how they build software. I talk to a lot of people about implementing new practices and processes that will help them put the user value at the center of the development practice. But with many of the companies I work with full assimilation of these concepts is a distant goal. Until yesterday, I’d never seen an organization learn about a process and start implementing it full-force the next day. That, my friends, is catching sight of a unicorn in the wild.

This unicorn is called OpenSignal and they came to visit me in a Pivotal Labs Office Hours session to chat about beta testing their new app. Ellie, Teresa, and Jaleh, told me about WifiMapper, a brilliant (IMO) native app tool that would help users locate public and private sources of wifi and provide some info about the location.

They had done a lot of work towards their research plan. They had recruited 60 people for a group of super-user participants. They had two focus groups scheduled in their office. They were preparing to conduct some pretty huge global surveys. They wanted the feedback that they got to drive their next stage of development and they came to me to figure out some ways to help them do that. How could they best decide which usability issues to work on and where they should focus enhancements?

I took a deep breath and broke the news that if I would recommend anything, it would be to cancel the focus groups and redo their whole research strategy.

Instead of getting mad, OpenSignal got curious. Here are some of the problems we talked about.

Problem 1: Focus Groups are ineffective

Focus groups are highly risky way of gathering information. If people are shy they won’t speak up. If they feel like others know more, they won’t say that they don’t understand. One aggressive person can take over the session and run it off the rails.

Essentially, people influence people.

Additionally, a focus group format would not allow OpenSignal to watch their users play with the app, only hear about how people felt about it after the fact. Getting late feedback means that they were going to miss out on 99% of the valuable information that they would have caught if they had seen the app used for the first time in front of their eyeballs.

Solution: 1-on-1’s

Scrap the focus group and reschedule each participant as an individual user interview. This will allow OpenSignal to go in-depth with each person, digging into their personal experiences, challenges and hearing how they are reacting to the tool as they using it.

The transcripts of these interviews will contain nuggets of pure gold and uncut diamonds. The added bonus is that user interviews have a relatively small overhead. Generally, five 1-hour interviews are enough to help surface patterns in user behavior and usability issues strongly enough to help set priorities. This leads us to the second major problem that OpenSignal had…

Problem 2: No synthesis means you’re losing stuff

No plan for how to use the feedback and incorporate it into their development pipeline. Currently, OpenSignal fed bugs directly into their development team, but no one was really prioritizing them. Suggestions, usability complaints, and opportunities were tackled on an ad hoc, scattershot basis, and the work wasn’t really well connected to overall feature goals or metrics.

When you don’t take the time to synthesis and plan, all of the value of user research is lost.

Solution: Download and find the themes

OpenSignal needed a few baseline synthesis activities that would help them shake out the nuggets of wisdom from their interviews, as well as their general feedback pipeline. A really simple one is the whole team going through the raw feedback and write one take-away per sticky, talking them through and then bucketing them into similarities. Then you pull out themes, challenges and opportunities out of the buckets. These can then be mapped to specific business goals.

The main point of having the synthesis session is to clearly identify the patterns that emerged from your conversation and map them to your overall business goals. This helps the team understand where user value maps to goals and also sets priorities. BUT in order to achieve this you need your whole team to participate, which leads us to their final problem.

Problem 3: Not Enough Org Participation

The three OpenSignal ladies who joined me for Product Office Hours were collaborating on the user research stuff, but were finding it difficult to get other parts of the organization consistently involved. They hadn’t found a way to involve the developers, and so in turn devs didn’t see user feedback as part of their own workflow. This has made it doubly difficult for them to incorporate the feedback organically into the development backlog.

It can be difficult for small start-ups when there is so much going on, but I tend to say:

Not making the time to participate in user research is like ignoring a chainsaw as you try to cut down a tree with a butter-knife.

Solution: Make it really available

I encouraged these ladies to start having conversations with OpenSignal leadership about user interviews and ask them to try to listen to at least 1–2 of the user interview sessions. With OpenSignal, an invite is probably all that is needed, but if there is resistance, sometimes a knuckle cracking is in order to get higher-ups to tune in. At a small organization like OpenSignal, making sure that leadership is as close to direct user feedback as possible is critical to keeping priorities straight.

With developers, I take a really soft approach. I book space for them to listen to the interviews. I invite them all to both the interviews and the synthesis, and then announce to the whole team whenever the interviews are happening, day before, morning of and five minutes before the interview starts. This helps it get it in front of their faces. The same goes for synthesis. Leave the door wide open, and bang a drum that it’s going on. More often than not, curiosity will get the better of them.

The Full Monty

The next afternoon I went down to the OpenSignal office and was floored by how quickly they had implemented some of my suggestions.

• They had cancelled the focus groups and scheduled several user interviews.

• They had taken all of the existing customer feedback that was in their ZenDesk and put it through a synthesis process on a wall in the middle of the office.

• They spoke about the new user research process with their CEO and invited him and other leadership folks to participate in the upcoming user interviews.

• Finally, they sat me down and conducted their first user interview with me as their guinea pig, with an amazing script that was copied from a script template. They nailed the exercise and delivered it like true pros. It was this process that offered the ah-ha moment for them as to why this stuff is the absolute Poo.

“So much of what you said is what I’ve been saying, but can’t get anyone to take action on.”

“We had so many conversations around whether or not people knew what to do there, but couldn’t decide how to proceed.”

This is what this type of direct user research helps organizations do, understand what’s important vs. what can be put off.

There is very little ambiguity when you’re sitting next to a user and they do not notice, ever, that a button is clickable, or that there is another page of information that they can access. A few days later I got an update from Ellie saying that they had several interviews, videoed them, and shared out the live-feed with the whole organization and a ton of people listened. They also did a synthesis exercise and presented the synthesis to the whole organization. In the words of Ellie herself: “It was fascinating seeing the users go through our app, and at points frustrating because we were thinking ‘why can’t you see the button? it’s right there!” I have never been so impressed by a company’s commitment to putting their customer at the center of their process. OpenSignal, kudos for just doing it. You guys rock.

Getting Back to Play: How Play and Playfulness Can Lead to Better Products

When I was about 10 I found a picture of a Jackabasselope, also known as a Jackalope. For those that don’t know, a Jackabasselope is a rabbit with horns. I think I found the image in a Far Side comic. I had a pet rabbit at the time, and I came up with elaborate stories about Jackabasselopes. They had horns made of pure gold, and would put curses on anyone that hunted them. I played pretend with this idea for hours. At one point I even made paper maché horns for poor Mr. Bun. Eventually I lost interest in these games, and forgot about the Jackabasselope.

Fast forward 20 years. I was gifted some deer horns by a friend’s father. I had them for several months, before an afternoon of crafting with gold leaf brought my Jackabassalope back from the dead. I spent some time applying layers of gold leaf to my horns, while making up stories that smaller Jackabasselopes have greater gold density in their horns and are therefore more desirable. Blah blah blah.

I realized that afternoon that I’d been playing pretend with this idea for nearly 20+ years and it resulted in a beautiful new display for my walls. his got me thinking about play and the effect that the act of play has on us as adults, specifically on adults who are in creative industries. How have we developed into creative people? How can we think about making leaps in innovation and creativity more consistently?

My mum is an early childhood development specialist and I grew up hearing her talk about how kids develop. From birth to the age of 6, children move through three different buckets of play. These are fluid states and they overlap and come up again depending on what a child is doing. 

States of Play

 Autosphere

Play as it connects to ourselves and what is directly around us. It’s here that we recognize what is “self” vs. what is “other.” It’s here that we explore the initial human geography and develop our sense of taste, touch, sight, sound and smell.

 Microsphere

This connects us to our immediate environment. We’ve expanded to include the people and objects around us, and we include them in our games. It’s a creation of configurations, patterns and rhythms. It’s where children can start to explore make believe worlds and use representation and pretend to start mastering skills. Rules are flexible and often discarded to make way for extraordinary circumstances.

Macrosphere

Play as it connects us to a larger societal context. In the macrosphere, children can take tasks or activities that are “adult” and bring new life to them simply because they don’t know what’s allowed vs. not allowed. When kids play in these different spheres, they develop all kinds of cognitive and emotional skills. The way we played as children deeply affects how we think as adults. Moreover, continuing to force ourselves into these playful states helps us as adults to think about things in innovative ways. Albert Einstein, Pablo Picasso, and Nobel prize winning scientist Andre Geim all credit continuing to view their work as play as key to producing their best efforts. 

Products That Make Use of Play

Some of the best products out there make use of these concepts of play and how it causes us to connect with ourselves, our direct communities and the greater world. Here’s two examples of products that make use of play:

 Google Doodles

With their Doodles Google took at concept as simple at a Logo and turned it into an opportunity to connect their users to the macrosphere, by using cartoons and gifs to educate us about events, scientific concepts, and moments in history. It’s a simple but effective way to get our eyeballs on things that we normally wouldn’t pay attention to, or put us in touch with a larger community.

Single Page Narrative Apps

Single Page narrative apps like EveryLastDrop have seen a huge explosion in popularity because it keeps users bounded in a microsphere type story. One perspective, one journey, and it let’s them be a part of it by lfetting them control the pace, experience movement, and patterns.

Play as a Mindset

Sometimes play can simply be a mindset or a process, like Why’s philosophy of Irresponsible Coding, in which he posits that behaving as if there are no rules, consequences or any way that things are “supposed to be done” helps drive him towards bigger technical breakthroughs.

So ask yourself:

• How does your product connect your customer to their world?

• Does it invite them to find out more?

• Does it invite them to participate in the story?

• Can you examine your product for opportunities to make it more playful?

• To hear more about play and creativity, come check out my talk at UserConf London this May!