Conversation with Janna Bastow

Conversation on customer feedback with Janna Bastow, Co-Founder of ProdPad and Mind the Product

Published

Key Points

  • The importance of customer feedback for Product Managers
  • Using active customers as a special source of understanding for issues in the product
  • Things to mind when speaking and interviewing customers
  • Prioritization of problems you identify when talking to customers
  • Having a group of power users in Slack to validate new features and ideas
  • Quantitative methods to use for tracking over time
  • The need to have raw customer feedback as input to the product team and the role of the support team
  • The usefulness of a public roadmap as a way to improve research and communication with customers
  • Some ways to approach hard-to-reach and busy customers

Transcript

Daniel:

Can you tell me a little bit about yourself and your background in product management?

Janna:

Sure. So I’m Janna Bastow, and I’ve been a product person of some sort or another for about a decade now. In terms of my background, I’m an accidental product manager, I kinda fell into it. I was previously a customer support rep for a company that happened to have a digital product, and I was picked out by the dev team and by some others on the team who basically realized that I was good at reporting bugs. So they plucked me out of customer support and made me a junior product manager, and I remember at the time thinking, “Oh, this sounds like an improvement. What is it though?” So one of the first things I did when I became a junior product manager was Google, “what is product management?” Just to get an understanding as to where it’s come from.

I have since then worked for several different start-ups and have done a bunch of different consulting and training and mentoring for various PMs and product companies. And several years ago I started ProdPad, which is a product management software, and since then I’ve been working with thousands of different product managers to build a product tool that helps them do their jobs.

Daniel:

Awesome. Okay, so the issue of customer feedback can be very broad as it applies to almost anything that we can do on the product. So in your view, what is customer feedback and how relevant is it to your work as a PM?

Janna:

So customer feedback to me is anything that, any snippets that comes in from customers that outlines some sort of interest or intent or suggestion or whatever it is that they have to say about your product. More often than not, I see so many companies, even great companies with great product functions, missing out on the customer feedback part. It’s something that gets stuck with support, but doesn’t necessarily make its way through to product in a transparent way. So what we believe is that the best thing to do is make sure that you’re capturing the stuff, make it easy for people to give you feedback, and then feed that into what’s going on in your backlog. It doesn’t necessarily mean that everything that’s on your backlog or everything that’s in your roadmap should be directly based on what customers are asking for. They don’t always know everything, but it should definitely be inspired by what you’re learning from the customers.

Daniel:

Right. Okay, so let’s get started on the part of learning about customers. I mean when you’re trying to find out about new things or problem areas that you think that… I mean new things or problem areas that might be important to users and customers, what kinds of things are you looking for? What kind of questions do you ask on customer interviews or any other methods to reach them?

Janna:

So I like picking out customers who I know are relatively active and if asked on Twitter, they’d say that they love ProdPad. I love asking them questions like, “What frustrates you most about the product? Or what isn’t going right?” for example. Basically, prodding them to find the really tough stuff, because invariably you’ll find even your happiest customer will have some niggly piece that they think would actually make their life easier, and sometimes it’s like actually we have that functionality already, it’s just here’s how you find it. So the note to ourselves is that we could actually make that more obvious, for example. Or write a blog post on it or help other customers. In other cases, it might be something really small that we had overlooked, but that was really ticking this customer off. So if we’re able to find those things then… And it’s the type of things that they wouldn’t otherwise bother us with. They wouldn’t think that it was worth their time to or worth our time to hear it, but we find it to be a really valuable question.

Daniel:

Right. And when applying maybe a broader view into what customer feedback might be, interviews might be very exploratory. More than tactical issues. Maybe try to find out about big problem areas that your product doesn’t tackle, or possibly untapped opportunities already on your product. So they are more of discovery kind of question. What interview techniques, what observation techniques, what kind of things did you try to use to get to these kind of insight?

Janna:

So it depends on the stage of your product and the type of customers and the type of users that you have. Some things that I know have worked really well, is just making sure that you’ve got several people who are potential users in a room every week and you’re ready to ask them questions. And it could be anything from asking them about, “Here’s some napkin drawings I did this afternoon, what do you think?” Through to, sit them down in front of the computer and have them, or in front of a mobile or whatever it is that your product is on, and actually have them work through it and talk through their issues or talk through their questions or whatever else. Other times, with ProdPad itself, we’re actually quite lucky that our users are product managers and therefore are pretty accustomed to the types of questions and how to structure something that’s actually useful.

Janna:

So oftentimes, I’ll just be able to prod them and say, “Hey, I’d like to talk about my product.” And they know the kind of process to go through, and will happily give feedback in let’s say user story format, or will provide mock-ups of what they think would suit best. So it definitely does depend on your customers. I think with our user base it’s almost unique because our customers are product people who do this for a living.

Daniel:

Right. And using that kind of techniques, what would you say that are things to avoid? Things that you shouldn’t be doing to get the best data possible?

Janna:

There’s the obvious stuff like don’t lead the customer. You don’t wanna say, “Isn’t this new redesign great? Huh? Don’t you love it?” Because that will just lead them to saying, “Yeah, it’s great. Thanks.” And there is kind of a fine art in asking questions that are leading versus not, and of course it depends on again who your user is, what the perception of the product is, what are their experience is already, but your whole goal is to have them guide through and to not have you get involved if they get stuck. You wanna see where they get stuck, you don’t want to just point it out and kind of ruin the fun for them.

Daniel:

Right, okay. And say that as you start talking to your customers, you find out about a few things that you might try to work on. How do you measure their relative value? How do you compare them and decide which to pursue?

Janna:

Yep, so one popular way that people see for prioritizing is to add up the number of people who’ve asked for it, or add up a number of votes. I don’t agree with this process, I don’t think that features should be built based on how many people it’s going to affect, or how many people want it. Often because votes tend to go for the sexier features as opposed to stuff that might actually make the difference, like no one thinks that redoing how your app asks for payment for subscription is an interesting or a sexy feature, but it’s the type of thing that actually might increase your end conversion rates. Whereas, a customer might ask for something that is like custom reports, or something like that, that might be important to them, but doesn’t necessarily move the needle on the overall product. That said, it is very important to quantify this stuff and to look at how many customers are asking for things, and to use that as just one piece of the puzzle to figure out what to build.

Once you know that a bunch of people have asked for something, you should be having conversations with them and understanding what it is that they are trying to achieve, what the problem actually is without trying to get into solutioneering and telling them this is what it’s going to look like, because at this point in time you don’t know. But it also comes down to, what are the goals of the company? What’s the main KPI that you’re focusing on right now? Or what needs to be achieved by the product overall? If your main driver is getting conversion rates up or something like that, then it’s very possible that something that an advanced user that has been using a product for two years is going to have much say on.

Daniel:

Right. Alright. Okay. As you move forward and you are building the thing, do you actively seek for customer input into what you’re building?

Janna:

Yes.

Daniel:

And how do you approach that? What kind of process and/ or technique do you use to get their inputs on what you’re building?

Janna:

So we’ve actually created a Slack group which is basically just for internal ProdPad users only, so people who are actively using it and wanna get involved or wanna have that closer conversation with us. So it’s sometimes just as informal as throwing a link in there to some mock-ups and seeing what people weigh in on and seeing what people think. Or throwing in a question like, “Hey, could everybody tell me what tags they have set up or what stages they have set up?” Just so we understand how people are using page X. Other times, it’s more formal. But for the most part, we talk to customers, we prompt them, if we see that they are using it a lot, we prompt them if we are see they are not using it very much at all with specific questions to tease out what’s going wrong.

Daniel:

Right. During this stage, do you think there’s a room for using quantitative methods or is it more on the qualitative side?

Janna:

It should always be a mix of.

Daniel:

Right. So, what kind of quantitative methods do you use? I mean, you try to find out about things using the qualitative research and interviews and asking the right questions, but afterwards, how do you… Might approach a quantitative analysis of all of this?

Janna:

So, we’ve done a couple of questions in the past that are relatively popular, so I would be surprised if you haven’t come across them before, to get more of a quantitative look at stuff. Number one is the NPS, so how likely are you to recommend this product or this feature to a colleague or a friend.

Daniel:

Right.

Janna:

Which you can quantify by figuring out where they sit between -100 and +100.

Daniel:

Yeah.

Janna:

The other one is from the Customer Development Survey that Sean Ellis did and the question is, “How would you feel if we were to take this product away or if this product no longer existed?” And they rate it on a scale on from ‘not very disappointed at all’ to ‘extremely disappointed,’ so where they kind of sit within there. And you can see actually where are people going to be really upset if you disappear or just somewhat upset, or would they not even notice?

Daniel:

Right. And another question might be… Okay, so that kind of gives you a notion of satisfaction and how they are feeling about the product. When you are shipping new things, how does success look like? What kind of things do you define to say this feature or this release is succeeding or it’s not?

Janna:

So that goes back to looking at what we were actually trying to do in the first place, so before we build anything we always outline, what problem are we trying to solve? And what are we expecting when we actually do this? Is this going to increase our upsell rate or is it going to decrease churn or is it going to increase feature usage of this particular feature for example, and then once it’s out there, we can measure it and understand as to, did it actually increase feature usage of this feature? Or did it actually decrease churn? Or whatever else.

Daniel:

Right, so, okay, do you have a specific goal or is it more… I mean, if you don’t hit that goal, what do you do in that case?

Janna:

Usually, you figure out what went wrong and you would start iterating. Whenever you’re setting a goal and you’re outlining a problem, you’re not necessarily coming up and saying, “I know exactly how to solve this,” you’re saying, “I know of maybe three ways that we can try to solve this, let’s try this one first based on the fact that a lot of people have asked for it and that I think it’s the most impactful in this particular area. It’s the most obvious thing to do.” And you work on that one, and once it’s out there you can measure it and say, “Okay, great. Did this actually hit the overall goal or should we maybe move to step number two or go back and brainstorm other ways that we could hit this goal?”

Daniel:

Right. Okay, at the start off our conversation, you mentioned that one of the problems that you see is that feedback often gets talking, support doesn’t flow transparently towards the product team. So what kind of process do you recommend to make this a healthy process and a healthy feedback loop?

Janna:

Make sure that any feedback that comes from customers is seen and is visible to the product team. They shouldn’t have to have a summed up version of it or be… They don’t wanna just have this list of stuff that has been summarized by the support manager saying, “People are saying X,” because oftentimes there’s a lot of context and a lot of information in what people are actually saying, so it’s best to conserve that so you know that Joe at Acme Co actually said this particular thing, and this actually hints to what his problem is. It might not be the same solution or feature suggestion he’s just made. And same thing goes the other way, is making sure that whatever is happening in product is actually visible to the support team in a way that is understandable. They should be able to see as to whether there’s a particular feature in the backlog that is getting requested a lot or whether there’s a particular thing coming up in the roadmap that might solve some problems for the customers.

Daniel:

Right. Let me wind back a little bit on that. On one side, when the customer feedback comes in, it might need some back and forth until you actually understand what the customer is saying. So is that back and forth something that support should be doing or is it something that product should be doing or is it something that the product should be enabling customer support to do in a way that let’s them scale that kind of feedback?

Janna:

It’s usually the role of support to be doing that back and forth unless you’re talking about a very, very early stage MVP and you don’t have separate support and product functions. At that point in time, there’s no reason not to be talking to every customer.

Daniel:

Right.

Janna:

When you’ve got a more established product and you’re talking about possibly hundreds or thousands of customers asking for stuff, your support will be your front line for that. But as soon as they determine that this is a feature request and not a bug, then they should have enough information there to say, “Okay, so we’ve now got… A customer’s asking for this.” Feature requests and stuff like that should be preserved and put somewhere that the product managers can absolutely see it and peruse through it and actually figure out how it fits within the grander plan.

Daniel:

Alright, okay. So you mentioned also the issue of having a publicly visible roadmap, I know that it’s something that you guys at ProdPad believe a lot in. That also has some… People don’t often feel comfortable with showing their roadmap or what they’re working on to customers, what counter arguments do you have for that kind of position?

Janna:

So I think one of the things that scares people off the most from showing off their roadmap is that they don’t wanna show either specific features that they may or may not actually get into, but in particular, they don’t wanna make any promises of dates that they know they’re probably gonna miss. If you’re putting dates on your roadmap, then you’re pretty much kidding yourself unless you’ve actually got a culture of almost waterfall-like building out massive project plans and knowing when things are gonna be delivered. More realistically, any company that’s working in any sort of lean or agile way will have a rough idea of where it is that they wanna go and what’s on that roadmap, but doesn’t necessarily have the list-by-list feature of what’s gonna be built over the next two years. So the number one thing that the roadmap does is that it communicates the product vision more than anything else. It communicates the product vision in a way that basically says, “We wanna be the X of Y,” which is something that your customers should know is where it is that you’re trying to be and what you aspire to be. But at the same time, it also outlines, “These are the things that we wanna achieve along the way.” And ideally those aren’t necessarily features or solutions that you’re going to be coming up with, but problems that you’re gonna solve along the way.

Janna:

And that way, what you can basically do is start using that roadmap as a way to communicate with your customers and actually tease out more information from them. So what you’re basically saying is, “Yeah, you know what? Down the line we definitely wanna have a mobile app,” for example, “So we’re gonna put that on the roadmap, it’s definitely something that we intend to do.” However, every time somebody asks about it, saying, “Hey, what’s gonna be on the mobile app,” or, “Hey, are you planning on doing a mobile app yet?” You can actually turn around to them and say, “Yeah, it’s something that’s definitely on our roadmap, but we’re still working out exactly what it might look like.”

For example, what would you like to do on a mobile? What are the key things that you’d like to be able to talk about and to be able to do within… While you’re on mobile." And this can actually come back in, be fed back into your product backlog so that when you get to that stage of deciding as to how you’re gonna do a mobile app, you actually have people who have given lots of feedback. Same thing goes with being able to say, “Hey, you keep asking for a custom report, what do those look like?” And they might say, “It’s really important that I have these,” and then you can say, “Well actually, they’re in the future. It’s something that we wanna work on, but look at all this other cool stuff we think that we need to do first.” And right then and there, you can get buy-in from your customer.

Or they can say, “No no, custom reports is the most important thing. It should be done next.” But more often than not, they’ll say, “Oh yeah, you’re right, you’re planning on building a mobile app too. Yeah, I’d rather have that first,” validating what order you put your roadmap in.

😀

Daniel:

Yeah, I mean the thing that I would say would jump out the most is, okay, no dates, it’s very freeing but there might be an issue there with managing expectations in the sense that people know they can actually accept that, “Okay, maybe these things are not… This is the order of things that are going to be built. But I really would like to have timeframe because this really is the third thing that is most important for me… I wanna know when will I have it.” Does that come up with you? Do you find that to be an issue?

Janna:

You’ll occasionally have somebody who is absolutely sold on the idea that they need a particular feature before the product is valuable to them. This actually comes back to more of a sales culture than anything else. And making sure that your sales team or your sales literature sells what you have today, rather than tries to sell your product based on what it will be. Because the thing is is that, unless this customer is paying you a lot of money and you’re spending that money on a kick-ass team who can plan out and pretty much guarantee something delivered by a particular date, you’re not going to be able to deliver that thing for them. Chances are timelines are going to slip or something else might come in.

Additionally, if you make a promise to a customer saying, “This is gonna be done by Q3 this year,” you run into the problem that, what happens if something changes in the market and that thing is no longer important? Or if a competitor comes up with something and you have to react. Or if some new technology comes through that makes it easier or harder to do something. You don’t wanna be stuck committed to one particular customer’s request that you’ve put a date on and you’ve promised is gonna happen. And so when you actually release the roadmap, you need to give that level of expectation. And I’ll be honest, customers ask that question, “When can I have X?” whether or not you have a public roadmap or not. 😀 Having that public roadmap actually helps take that pain off because you can say, “This is the order we’re working on things, but I can’t tell you how long it’s gonna take to get even here, let alone these future things that you’re talking about.”

Daniel:

Right. You talked about the relationship with sales which is also an important part of feedback that comes in. They’re actively talking to customers, prospects and they’re valuable input into the product’s cycle, but it’s often a conflictive relationship with product and sales. So, what would you recommend as an approach to handle this relationship?

Janna:

So a few things there. Number one: Keeping communications open and friendly between salespeople and product managers. It’s really hard to want put your finger on how you’re supposed to do that, but remember that a product manager’s main role is not necessarily as a doer, not necessarily as a checklist of things to do, but actually managing the relationships between key people. And so having sales on board is hugely important and it depends on how your sales team has been performing in the past and how they treat the product and everything like that. So a few things that you want to be able to do; make sure that the sales team knows that you are not going to just cave in to the pressure of doing individual or one-off product features. And this is about managing expectations, saying, “You know what, hey, we’re gonna build the best thing we possibly can and every month I hope that what you’re selling is better than the thing you were selling last month.”

But don’t, make it clear you’re not gonna have these one-off feature requests just built because you think it’s gonna close one particular sale. And also, empower them to have good conversations with their customers, With potential customers. They should be just as ready to say to a customer, “This is what we have today, this is how it’s gonna change your life now.” But if a customer asks for something, then allowing them to have the conversation saying, “It’s really interesting you asked for that. We don’t have it but, hey, we’re always taking feedback from customers. We’d love to find out what it is that you’d like to achieve with X.” And just by doing so, helps them actually sell the product based on the openness of how the product works and how customer support and product is going to treat them when it comes to future requests like, “No, we won’t just build whatever, but we’ll absolutely listen to what you have to say.”

Daniel:

Right. Another issue that can be considered around this topic is, you have multiple user segments, customer segments that are giving you feedback through multiple channels. And it’s very important to have a clear picture of who’s talking to you and who are you talking to. Can you describe how would you segment your users in a way that is effective to handle their feedback?

Janna:

Yeah. So, it’s always important to make sure that you’ve got the context as to who is asking and when and how. So making sure that you are segmenting out fresh trial users versus your more advanced users. You will be able to identify which of your users are power users. They’re gonna have completely different requests to different feedback and a different take on your product than somebody who’s just joined and seeing it with fresh eyes for the first time. Also being able to figure out as to whether this was something that, did this come out of them being prodded for feedback, did you ask them something like, “What frustrates you about our product?” Or were they actually just so frustrated that they had to tell you what was frustrating them with your product, which is two different takes on it.

So it’s always worth making sure you’ve got that kind of data, metadata, around where the feedback comes from. But at the end of the day, it’s stuff that the customers want. It’s got to be kind of filed in one place, and all of it is potentially valid feedback to go into your backlog or to help you figure out what goes in your backlog.

Daniel:

So, do you have more of a functional segmentation of users or is it more kind of a demographic-based segmentation? Usually there… Both approaches are useful but people lean towards one or to the other.

Janna:

Yeah. Our product is B2B so it makes a lot more sense to understand their function. What’s their role in the product? Are they one of the main admins or editors or are they a reviewer who uses it once a week? Are they a power user who’s been using it for two years, or are they a brand new fresh user? Are they using it with a team of 100 people or is it just themselves because they’re using it for a side project? So we actually prefer to look at things this way. To be honest, it doesn’t really make a difference to us as to whether they’re American or European or something in between. We don’t really find any sort of segmentation differences in terms of how they give their feedback or what the approach is.

Daniel:

Okay, cool. Since you talk to a lot of PMs and an issue I’ve seen around is the fact that, especially in large organizations for some reason, they don’t have direct access to customers. Do you have any comments on why do you think that might be and how can they try to overcome that position?

Janna:

So companies and product managers in larger companies often do struggle with that, where they don’t have direct access to the customers, and the only thing I can say to that is that they’ve got to start trying. They’ve gotta find ways to talk to the team who’s talking to the customers, making sure that they don’t have silos, they don’t have walls there. And that could just be as simple as finding out who is in the support team and where their feedback actually goes. You can kinda presume that if you’ve got a big company like that, then they’re gonna have some sort of tracking tool. Hopefully, with some sort of analytics on there, they can understand where feedback is coming from. Hopefully, with some way piping it through to somewhere that the product managers can see it. But it will depend on a case by case basis just depending on what the structure of the company is and how they work. But again, this comes down to the product manager being the person who’s gotta be the communicator between them, and so they’ve gotta find ways to communicate with the customer side, the customer-facing side of the business.

Daniel:

Right. Another angle to that is when customers aren’t available, they don’t… They’re too busy, they’re not interested in talking to you, maybe because the relationship with the product is not as important to them as it is to you. Are there ways around that?

Janna:

If you literally cannot speak to the customers, then you’ve gotta work a lot harder on getting the right data and seeing what they do. If you can’t talk to the customers and ask them what’s bugging them, then you should be able to look at the data in detail and find out what’s causing people to drop off or what are they doing or what aren’t they doing. It’s one approach. There’s still no replacement for actually being able to talk to a customer and ask them what was up.

Daniel:

Yeah.

Janna:

Always be able to articulate it better than a CSV with a bunch of events on it and activities on it. But if you’ve got no qualitative input or any way of talking to customers, then you’ve gotta work off data.

Daniel:

Right. On the subject of qualitative feedback, you have any personal system to kind of organize all of this and make sense of it? It applies both to what’s coming in right now and probably what might have come in in the past. Is it some…

Janna:

Yep.

Daniel:

Is it something that you do directly on ProdPad? Is it something that you do or using some other tool?

Janna:

Yeah. We actually have a module directly built in to ProdPad which gathers customer feedback from various sources. So, for example, we use Zendesk for our support for whenever we’ve got feedback that comes in from there, we pipe it into ProdPad. We’ve got the Slack chat group of our most powerful users, most engaged users, so whenever they’ve got feedback it pipes through automatically. We’ve also got it hooked up with various forms and other places that people can drop in feedback. So all of that comes into ProdPad which we can then use to tags or filter search on and then link to various stuff that’s going on in our product backlog itself.

Daniel:

Cool. Okay, so I think we’ve run through the major questions that I had for you here. I wanted to maybe get some of your input on bad experiences that you’ve had, things that you have learned and mistakes that you’ve perhaps made when dealing with customer feedback and how can PMs avoid those mistakes?

😀

Janna:

So I’ve got one. When we first starting building ProdPad, it was myself, my co-founder and we were both product managers for different teams. And we just needed tools to help us do our jobs. Nothing out there existed. So we started building ProdPad and we used it internally only for the first couple of years of existence. It didn’t even have a name, it was just this tool that we used.

Daniel:

Right.

Janna:

But what went wrong was that as we started to expand it to the rest our teams, we got a lot of confidence saying, “Okay, looks like this is actually being used by our companies, within our own companies. Let’s try to expand this out to other people.” And what we learned was that we built ProdPad pretty much in a vacuum. We hadn’t spoken to anybody. We’d just been building it for our own use cases. And even though, we thought we were our ideal users, product managers at tech companies. Even though we were pretty much exactly who are end-users were, what we ended up building was so narrow-focused that it only worked for us. So we had to learn from that. We had to throw out a whole bunch of work and take it back to the drawing board and talk to people who were trying to use it, talk to our very early customers and based on that we actually redid some major components within the app before it actually started being picked up and being used by people. So even though you think you are your user, you are not your user.

Daniel:

😀 Great, okay. So on the flip side of that and trying to wrap it up, what would your top recommendations be for product managers that are trying to make sense of customer feedback? What would their most important things to keep in mind around customer feedback be?

Janna:

So number one, keep in mind that if somebody has a particular feature request and they’ve outlined how they want it to work, don’t necessarily take that verbatim. It doesn’t necessarily mean that they’ve got the best answer for it. Ask them further questions to understand what their actual problem is because you might find that the solution to that problem could be something completely other than what they’ve suggested. So make sure that you’re not just listening verbatim to what they’re saying, but actually parse it and figure out what their issue is underlying and work to solve it using the best solution as opposed to what they have proposed.

Daniel:

Great. Okay, so I guess this is it. I wanted to thank you again for your time and it was fun for me, I hope it was also fun for you. Thanks a lot!

Janna:

Wonderful. Yeah. Absolutely. Anytime.