Episode 199: The True Cost of AI: Beyond the Hype and Into Reality with Jessica Hall
I recently had the pleasure of interviewing Jessica Hall, the Chief Product Officer at Just Eat Takeaway, on a recent episode of the Product Thinking podcast. Jessica has an impressive background, having navigated her company through the complexities of mergers and acquisitions, which have brought both challenges and opportunities for innovation in product management.
During our conversation, Jessica shared insights into how her team is working to create a unified platform across multiple markets. This journey has involved consolidating various customer-facing applications to deliver a consistent experience worldwide. I found it fascinating to hear about the meticulous research and collaboration involved in understanding the unique needs of different regions, all while ensuring that product development remains efficient and responsive to customer demands.
We explored a range of topics, including the importance of building a capable, diverse team to address the challenges AI presents, such as data governance and bias. Jessica shared her practical experiences, highlighting how she has implemented robust frameworks to ensure transparency and ethical use of AI, while also fostering a culture of continuous learning and curiosity within her teams.
Read on, to learn how we unpacked the reality of AI in product development and discover how to harness its potential effectively!
You’ll hear us talk about:
07:45 - Creating Customer Closeness Through Direct Engagement
Jessica passionately advocates for the principle of "customer closeness," highlighting how integral it is for product managers to engage directly with customers. She has initiated a program called "Just Eat Takeaway Meets," which allows team members to immerse themselves in various roles within the company – whether that be delivering food, working in customer service, or experiencing the restaurant environment firsthand. This practice fosters a deeper understanding of the customer's journey and the challenges they face, which can be invaluable for product development.
15:07 - The Hidden Costs of AI Implementation
Jessica and Melissa get deep into the costs of AI, both the well-known and hidden costs that can appear unwanted during implementation. While many view AI and large language models (LLMs) as cost-cutting tools, the reality can be quite different. She discusses how organizations often overlook the full spectrum of costs associated with implementing AI technologies, such as self-hosting on platforms like AWS, which can be quite expensive. There's also the ongoing expense of maintaining skilled teams who can effectively manage and evolve these technologies to contend with. Jessica points out that businesses need to assess not just the upfront costs but also the long-term financial implications tied to customer acquisition and conversion metrics. This broader understanding helps organizations avoid the pitfalls of adopting AI for the sake of it without a clear strategy, ensuring they approach such implementations with a realistic mindset.
22:02 - The Power of Diverse Teams in AI Development
Jessica advocates for the importance of diversity in teams working on AI initiatives. She believes that having a variety of perspectives leads to more innovative solutions and a greater understanding of the diverse user base that these technologies serve. She stresses that it's not enough to have a team that simply identifies as diverse; team members must actively engage with different viewpoints and experiences. Jessica encourages her team to seek out customer insights from varied backgrounds, promoting empathy and a deeper understanding of user needs. This inclusivity not only sparks creativity but also drives better product outcomes, ensuring that the solutions developed resonate with a broader audience and address underrepresented issues.
Episode Resources:
Other Resources:
Follow/Subscribe Now:
Facebook | Instagram | Twitter | LinkedIn
Intro - 00:00:01: Creating great products isn't just about product managers and their day-to-day interactions with developers. It's about how an organization supports products as a whole. The systems, the processes, and cultures in place that help companies deliver value to their customers. With the help of some boundary-pushing guests and inspiration from your most pressing product questions, we'll dive into this system from every angle and help you think like a great product leader. This is the Product thinking Podcast. Here's your host, Melissa Perri.
Melissa - 00:00:37: Hello, and welcome to another episode of the Product Thinking Podcast. Joining me today is Jessica Hall, the chief product officer of Just Eat. Just Eat might not be familiar to you as a brand, but you probably have ordered food from one of their many delivery services. Jessica has been at Just Eat for three years, now working across portfolio, tech delivery, and operations, before becoming their CPO in 2022. You may have heard her talk at the Women of Silicon Roundtable event in London last November, where she discussed embracing transformation, unlocking potential, and welcoming change. I can't wait to hear how one manages and innovates the product of such a huge brand. But before we talk to Jessica, it's time for Dear Melissa. So this is a segment of the show where you can ask me any of your burning product management questions, and I answer them every single week. Go to dearmelissa.com and let me know what you're thinking. Here's this week's question.
Dear Melissa, I chanced upon your interview with Lenny about SafeAgile. Having come from a banking background, I personally share the struggles of having a title of a product manager, but was just seen as a product owner, interfacing with engineering. Having recently moved into a new role of product director for enabling platform that delves into building internal tools for business operations, fraud compliance, customer support, and the mobile platform, I'm having my struggles if this is really a PM role versus a technical product manager role. The role mostly serves the internal customers. The conversations aren't about problems to solve, but solutions or workflows to build. I wanted to get your guidance on, does building for an internal stakeholder really need a product manager versus a project manager? If it does need a technical PM, how should I reframe this in a way that I can power myself and my team?
I'm going to first think that when you say technical PM, you're meaning project manager. So let me make those distinctions right now. So there are technical product managers. There are technical project managers. There are project managers. There are product owners. There's a lot of stuff out there in these terms. So let's just talk about the difference between what you're doing, what good product management looks like, and how we should think about it. First things first, a technical product manager is still a product manager. And these people are usually working on backend platforms that have to do with code-based problems rather than user-facing problems. The developers are still your customers. You want to make sure that you are enabling them to do their job well. But a lot of the stuff that you're overseeing is a strategy around a platform. So things like API strategies, things like how we think about data classification, AI models, all those types of things could be relevant to your role if you are on the more technical side of a product manager. But at the end of the day, all product managers have a very important job to do, and that is to reduce the risk of failure on their product. And when we look at risks in product, there are three main risks. There's the feasibility issue. Can we actually build it? There's a desirability issue. Do our customers or users actually want this? Will they use it? And then there's the viability issue, which is, does it work for our business? Now, when we think of internal tools, like you're talking about with workflows, those still have desirability issues. And I don't want you to forget about that. So think of all the internal tools that you have ever dealt with. Some of them suck, right? Some of them are really, really bad.
And it's because sometimes we lose that mindset of a product manager when we build for the people that are inside our company instead of the focus that we have when we look externally because of our customers and our brand risk. There's still risk to your team if you're not making desirable products internally. Now, that risk might be lower. It might be easier to innovate on those things and throw stuff out there and not have it be perfect because it's internal to us. But at the same time, there's risk. And I'm going to tell you a little bit about it. So I'm going to give you a great example. Somebody in one of my workshops a while ago came in and said, hey, I actually had a failed product where I thought that people just had to use it. There was no thinking about this. It was just implement this workflow, implement the solution. And what happened? When they did it by the book and they implemented it, half the developers quit because they refused to use it. Great example. They had to take it all back, say, no, no, no, no, please don't quit. Please come back here. We don't want you quitting. We need to use it. When we talk about internal stuff too, there was a huge issue in healthcare when I was working at Athena Health. There were tons of nurses quitting their jobs across the US because they did not want to use the tools that the doctors and the hospital systems were forcing them to use. They said, you have to use this electronic health record system. And said, no. And they started leaving to go to other hospitals that use different electronic health record systems. And this was for all the health record systems. And that became a really big part of improving the UX across the board for any company out there. So that was a trend, right? And in that case too, they're thinking those are internal employees. I don't have to worry about them. I just have to worry about patients. Again, internal tools.
We always assume that people just have to use it. They don't. They could find another job. Wouldn't you be much happier if you work somewhere where they actually cared about the tools that you have to use, that you have to go through? So think about it that way. Another way too, is what is a risk? So when you're working on things like fraud and compliance, you don't have to just do exactly what people say from a workflow implementation. It's not like somebody's just handing you these things down and say, go do it. You should get creative about how you should do it. How can I protect the business better from fraud, from compliance, from risk on these backend systems? Can I make them more accurate? Can I make it easier for people to actually intervene in these situations? Can I make it more secure? All of those things have components to it where you could start to figure out what is the right solution. So at the end of the day, I don't want you to think that just because you're working on something that's technical, that there isn't risk associated in it and that there's not anything to think through when it comes to the solution. We should be paying way more attention to how we build internal tools and how we do our processes. In the advent of AI, everybody's talking about how a lot of these workflows will go away and our jobs will be replaced. And I don't think we're quite there yet, but that's a great example of a solution that is helping people do their jobs better and faster. It's being more efficient. But if we had just built a workflow to do those things instead and people click 80,000 times, it would be a different solution.
You still have control over these things. So I want you to really embrace that. And as you think about that with your role as a product manager, you are a technical product manager. How do you reduce the risk that it's desirable, feasible, and viable? Really look at those three things and figure out what you can do and what you can control. And I hope that helps. Again, if you have any questions for me, please go to dearmelissa.com and let me know what they are. Now, let's go talk to Jess. Are you eager to dive into the world of angel investing? I was too, but I wasn't sure how to get started. I knew I could evaluate the early stage companies from a product standpoint, but I didn't know much about the financial side. This is why I joined Hustle Fund's Angel Squad. They don't just bring you opportunities to invest in early stage companies, they provide an entire education on how professional investors think about which companies to fund. Product leaders make fantastic angel investors. And if you're interested in joining me at Angel Squad, you can learn more at hustlefund.vc/mp. Find the link in our show notes. Welcome to the podcast, Jess.
Jessica - 00:07:42: Hi, thank you so much for having me. It's great to be here with you.
Melissa - 00:07:45: Can you tell our listeners a little bit about your career journey and what led you to Just Eat?
Jessica - 00:07:51: My career journey has been quite varied. I had a really good start in my career when I joined a large global retailer based in the UK on an early careers program. And it was fantastic. It opened my eyes to what the world of technology could actually be and gave me the opportunity to learn things that I didn't know. So I learned to code and I worked as a developer for a period of time, although I always tell people I'm really a terrible software developer and that's not a skill that I claim. But it springboarded me into lots of other things like business analysis, working on technical program delivery and understanding software development life cycles. And not only that, I've been in the industry for a while. So I started in that waterfall way of working. And so I've also been on the agile journey in large companies, which has been more of a latter part of my career. So I moved around, I did lots of sideways moves and took the opportunity to work on different technologies, different customer propositions, things that I would call product today. But back then, we didn't necessarily have the terminology product around that work. But diving into things like designing user interfaces and user experiences in retail environments, for example, I loved it. But I also love the work of organization, what makes an organization tick? What is the culture? How do you organize for success? How do you create meaningful, interesting work? And so that's been a thread more latterly in my career, bringing those two together. So I've worked in transformation roles, bringing together teams, restructuring. I moved to another retailer where I did some merger work and coming to Just Eat Takeaway, that was what I initially came to do.
I was hired originally to help work on how we could bring the organization to a better place and bring two large organizations together following a merger, which I did for the first, I think, 10 months that I was at the business. And from there, I became chief product officer, which I've been doing now for two and a half, nearly three years. And I count myself so lucky to get to do this job because it's amazing. The impact and the type of work and the team that I have and all of that is fantastic. But if you are thinking about your career journey and how do I get to be a chief product officer or, you know, how do I grow my career? I think one of the most impactful things for me has been doing sideways moves and taking opportunities and saying yes. That meant that there's been some moments of huge, steep learning curves for me. But by doing sideways moves and working in different places and challenging myself to learn, I've been able to have this broad view and connect all the dots. And now in the job that I do today, the remit is very broad. And if I hadn't done that, I think I would struggle to be effective in this role now. So whenever I'm mentoring somebody, whenever anyone asks, I always say, like, say yes, figure out how to learn and be curious and do the sideways moves that give you the opportunities to grow.
Melissa - 00:11:03: I think that's really good advice for people, especially with the sideways moves. That's what I've seen help everybody get into the C-suite, like you said. What were some of the sideways moves that you did? That you felt gave you an opportunity to learn something that you would not have traditionally learned in a regular product management path?
Jessica - 00:11:22: So I think, first of all, right at the beginning of my career, I did a number of different things that I wasn't necessarily well suited to, but gave me a real insight. So it sort of ages me somewhat to tell some of these stories. But early in my career, for example, I was working in a business that was like pre DevOps. So we had a different service management team. And I worked in the service management team for a while. It was all around how do you get the prices to the shelf edge for customers and how do you get them onto the till? And it was sort of an everyday thing. The prices were changing all the time. It's a competitive grocery environment. So you need to have the best prices. So there's a lot of change, a lot of shops, like thousands of shops. And it's got to be right legally, because otherwise you're going to be in trouble for incorrect prices on the shelf versus the checkout or whatever. So my job wasn't to develop any of that software. My job was to make sure that it was working properly. And I can tell you, there were lots of times that went wrong. And it usually went wrong in the middle of the night, because that's when the prices were changing. And I found it a really difficult job. It was hard to be motivated. And it was hard to solve those problems sometimes because you'd be thinking to yourself, well, we could solve this upstream if we made these changes instead of having to fix it here every night. So why can't we do that? And that made me really curious about why does this not work? And why is there not a motivation to fix that problem? Now, obviously, DevOps is the answer to that problem. Own your code, run your code. But we didn't have that.
And I think had I not done that job, I wouldn't have realized how important that is, what it feels like to sort of be on the other side of the fence and someone's just literally chucking bombs at you. And you're like, you're just trying to figure out how to make it work. And as the person doing that, maybe not being considered as important as the person writing the software. So now my perspective is quite different. I wouldn't ever run a team that way. That's not how we work. But understanding the importance of both sides. And I guess that was a very early window on customer closeness. And that's something that I hold true to today. And anyone who works with me, anyone who knows me will know at some point they will hear customer closeness from me. That customer closeness is a bit different from maybe what it is today, where I might talk to a courier or a restaurant or a consumer. Again, it's just all about knowing who's the user. Or who's getting the experience? What does it mean? That's one of the things. But I think otherwise, I would say I did a few different project management type roles where I worked on different types of technology, customer in-store technology, like checkout, self-service checkouts, marketing technology, loyalty technology, and then more organizational roles that allowed me to see how those pieces came together. All of that's made me understand the entire ecosystem rather than just one specific area.
Melissa - 00:14:25: That ecosystem is very important for CTOs, and especially at a company that's as big as Just Eat Takeaway. I think a lot of people are familiar with the company just from the name, but they're probably familiar with a lot of your brands. Can you explain a little bit about the company? Tell me a little bit about this merger you were talking about, and what markets and what brands do you support?
Jessica - 00:14:46: Yes. So we are a global delivery company, food delivery mainly. We operate in 19 markets. We're known as Grubhub in the US, Skip the Dishes in Canada. Across Europe, we have a number of brands, but in the UK, we're Just Eat, we're Takeaway, and a number of other brands across Europe. We have actually 84 million customers. We have about 750,000 partners on our platform. And a partner could be a restaurant, a grocer, a pharmacy. We have electricals, we have all pet foods, gifting, all sorts of things. We have thousands, hundreds of thousands of couriers in our network who are delivering. And then obviously we have things like customer service and other elements to the platform. So that's what we do. And we are all about basically creating what I would call a digital high street and bringing convenience to customers. So we will get you what you want really quickly. Hopefully in about half an hour. And that digital high street is really important because much like other businesses who are in our industry, we started with prepared meals, takeaways, and we've expanded our offering over and over again so that we're now creating a much broader experience. That's grocery, like I mentioned. You can also, you can order an iPad, you can order gifts, flowers, pet food, you name it. We're probably selling it, or we're going to be. And we're bringing that convenience to everyday life. And what I would say is there's been a massive acceleration of that since COVID as people have got much more online, much more comfortable with that. But also consumer expectations have massively changed on how long they are willing to wait for something to arrive. And we're in that sweet spot of being able to bring the big brands, local independents across a variety of different categories right to your door.
Melissa - 00:16:39: So from a product management standpoint, you've got all these different brands that you manage. How are you structured? From a product? Do you have like a central platform that empowers them? Do you have teams that are in different countries kind of running their own things? What brings us all together?
Jessica - 00:16:52: We've been on a journey is the answer to that. And I'll tell you where we got to, but we're a business that has actually grown from mergers and acquisitions. So that has brought with it some challenges. We've inherited different company cultures and teams, and we've inherited different tech stacks. And we've been on a journey of consolidation for some time, but we've really amplified that this year. And we are very close to finishing a consolidation of all of our customer facing apps and web so that everyone globally will have the same experience. And we've been working on that more or less start to finish this year. So it's pretty impressive what the team have managed to achieve. And we've gone in that direction because we see that consumer desires, expectations, trends are moving very quickly, especially in terms of e-commerce and online. And what we recognize is that actually, if we can have as much of a single platform as possible, we can develop once and we can roll it out many times. So that means every market can very quickly benefit from all of the features that we have, rather than having to develop them in different tech stacks or in different places. It's a challenge for us because we do need to make sure that the features we develop are usable in every location. Now, of course, we can turn them on and off and we do all of that. But things like translations and flows and small changes like where do you show the monetary symbol? The euro symbol is sometimes before the number and sometimes after the number in different European countries. So all of that stuff has to come into play and we have to really know our markets well. But it's really worth having the one platform we can deploy multiple times because we see that our speed to market is much, much faster. And in an ultra competitive environment where customers have really high expectations, that really matters.
Melissa - 00:18:47: How do you organize or help the product managers think through what's needed in each one of those countries? Like, do you have somebody who's dedicated to the market? Or are you going out and doing research when it comes to specific features and bringing it back in?
Jessica - 00:19:00: All of that. So our team, my team is very distributed. I have people in a whole load of different countries who are all part of one global team. So we already have a wealth of information from people who are on the ground using our platform in different places or understanding the cultural and consumer side of life in those countries. But more than that, yes, we work with the kind of leadership and the people in each of the markets. And we have a team, our product success managers, we call them, and they are dedicated to working with the markets all the time to understand what regulatory change is coming, what do they really need in their market and why. And those members of my team also help to bring together the asks so that it might seem that we need different things for different countries, but actually the technical implementation of that might be the same. We might be able to reuse it. So they play a really important role in kind of synthesizing all of those different asks into what we actually really need to do. But I also have a brilliant research department that works really closely with the product managers to implement and run research.
We've massively improved that over the last few years with significant investment into different languages so that we can really get close to our customers, our restauranteurs, et cetera, getting lots more research done, having it very focused on the things that we actually want to address, as well as exploratory kind of research and horizon scanning on what's coming. So lots of that going on. And then, like I said earlier, because my team, if they're watching this, will laugh, I also say to them, you have to get close to the customer, for me, even just a couple of days ago, I was in a grocer picking an order from one of our customers to see what it's actually like in the shop. And I was talking to my team today about a certain restaurant and going and seeing how it runs because it's really eye-opening once you get there and you see what the reality is on the ground. So all of that together helps us to figure out what we need to do and how we're going to go about doing it. And then we go through planning process where we look at the priorities, the commercial value and what we need to go after to make those very difficult trade-off decisions. Because I think the trade-off part is the hardest part of it, being a product manager.
Melissa - 00:21:22: When you're trying to help your team get closer to the customer, do you have any prescribed cadences to do that or any tools that they can leverage to help them go out there? Or is it more that they're taking the initiative on their own?
Jessica - 00:21:34: So I encourage everybody to take their own initiative because I think you can have micro-experiences very easily. Any time I go into, say, a coffee shop or a restaurant that is on our platform or on a competitive platform, I've got half an eye on what are the couriers doing when they arrive? What's the restaurant doing? And that's very easy. However, this is so important that I actually have started a program for our department and that's been in place for about two years now. And we call it Just Eat Takeaway and we set up experiences. So you can go and do a shift in a restaurant. You can go and do a career shift and drive a car or ride a bike and deliver people's orders. You can go and sit in customer service and you can listen or do chats or see the social media, whatever elements we're doing. And then we also have a very well-established customer closeness program. And I speak to at least one customer a month for an hour face-to-face, asking them questions and placing orders with them to see what they think. And anyone in our organization can do that as well. So there are a whole load of ways to do it. But in my opinion, curiosity is one of the most important skills that a product manager can kind of nurture and encourage. And it's those micro pieces that are actually the real value. Just seeing the small pieces and saying to yourself, I wonder why that happened or I wonder what they're going to do next. And having that broad view. And I think related to that also, having opinions about stuff but not being afraid to change your mind, and asking yourself, why do you think that and what might a different customer think or feel?
Melissa - 00:23:20: I love the fact that you can go out there and deliver people's foods and try it like first hand. That's really neat.
Jessica - 00:23:25: If you're in Amsterdam, if you're in London, if you're in Toronto or Winnipeg, I can often be seen out delivering food.
Melissa - 00:23:33: I love that. Putting yourself in the shoes of the customer.
Jessica - 00:23:36: You don't know our brand, by the way. We are Bright Orange, so I'm easy to spot.
Melissa - 00:23:41: So one of the big strategies for your company is around AI and integrating AI into your roadmap. Can you tell me a little bit about how that came about and what you're thinking when it comes to that?
Jessica - 00:23:52: Yeah, so I think AI is going to be very transformational for the industry, for society, for businesses. There's loads of excitement about it now, and rightly so, because I think it's really interesting. However, yes, we are looking into it. But I think the real key is what is the problem that you're trying to solve? Which is the fundamental product question always. So I'm trying to kind of step away from some of the hype and say, OK, well, which business problems really matter to US? Which customer problems really matter to us? And is AI a tool that can help us solve that problem better? So the result is at the heart of what we're doing rather than the technology itself. And I think that's really important. Having said that, I think whenever new technology starts to gain traction, you know, it's our job to also understand. To research it and to know how to apply it. So we've been experimenting with a few things and in a few different areas. One of the things that's quite interesting is personalization and improving the customer journey. So I'm sure many of you that are watching or listening are always thinking about, like, how do I improve conversion? How do I reduce friction in my journey? We all know that's important. So we introduced an AI assistant and we've had a bit of a learning journey with that. We have built something where you can build a basket in the chat. So we are taking the ordering journey from minutes to seconds, which is great. I think that's a real improvement. However, we've had to do some learning about what is it that customers want. And what they really want is hyper personalization. They want to say, hi, I'm feeling tired. I'd like some comfort food or I'm after something healthy. What do you recommend? And they want to get some instant recommendations. Build the basket and go. And that's where we've had to kind of experiment.
And there's been some learnings there on speed, the accuracy and the types of personalization. Thank you. And we've also been analyzing what people write about so that we can start to build it out. So we also see not only do they want to order, they want to know where their order is and they want to ask for help through this interface. So we started to build in customer service flows and we started to look at how do you automate when things go wrong? Because unfortunately, sometimes they go wrong and you need a refund or there's an item missing from your order. So what can we do to make that experience also really frictionless? How do we automate it so we just say, yep, your refund's on the way, really sorry. And like they can go on with their day. They don't need to send an email or phone somebody or any of that. So these are areas we've been experimenting with. With 750,000 partners on our platform, you can imagine that the amount of data we have around menus and things like that is huge. And setting up a new partner requires us to set up a new set of data every time. So other things we've been looking at which are really impactful are we've trialed an AI menu upload tool.
So you can take the printed menu from a restaurant, scan it, create all the data entities that you need, upload and get that partner online. That was something that could sometimes take up to four hours. And we've reduced that massively. We've reduced it by over 50%. And every day as it's learning, as we're improving it, it's getting faster and faster. So we can onboard our partners more quickly. We can give our colleagues more meaningful work. So they're not tired. They're not typing in menus. They're working on customer problems and other things. So there's lots that we're experimenting with, I think, to make the customer journey more efficient. To remove some of the repetitive work in our business so that people can work on more interesting problems. And I think there's loads more to come with that as well.
Melissa - 00:27:35: Those are really cool examples of how you're leveraging it. When you are debating whether or not AI is a tool that can help solve this customer problem, what are the types of things that you're thinking through to say, yes, this is the right way to go or the right solution for this? And versus we should try something else?
Jessica - 00:27:50: So I think there's always a whole load of inputs. And it really depends on the situation. So there's a cost piece in this. We are excited about AI. And there's lots of conversation about it in the industry. But implementing LLMs, training LLMs, and running them can be really expensive. And if we're talking about something that has a very small impact, it might be great for your customers. They might really love it. But it might not commercially really move the needle. Then you have to ask the question, like, is this the right investment to make? Or are there cheaper alternatives that are as good or 90% as good? And that's always a judgment call that you have to make. I think the other piece is just keeping it simple. And that's one of the things that I hold really, really true. And I challenge my team on as well. Sometimes with the AI solutions, we can be guilty of kind of overcomplicating things or overengineering it. And actually, simplicity is the answer. So asking yourself, is this the simple answer? Is this the right thing to do? That's really important. And then I think once you get to that point, it's then a question of like, what's your tech stack? What data do you have? Do you have enough data that you can make this meaningful and that you can run it effectively? You know, you have to ask yourself all of those kind of things. How accurate is it going to be based on the information that you've got? You know, and is that a risk you're willing to take? Because with generative AI, there is also a risk element of this because we are still... We're still learning and the LLMs are learning and it's changing all the time. We've seen some mishaps in the industry and there will be more to come, I'm sure. But it's all of these factors together. And I think I would go back to what I said also about hype, not getting kind of dazzled by the exciting potential, but actually asking like, what is the problem we're solving? And does this really solve that problem?
Melissa - 00:29:46: After working with hundreds of companies to transform product management, I've discovered one thing that consistently holds both organizations and product managers back from reaching their full potential, the ability to craft great product strategy. That's why I created Product Institute's latest course, Mastering Product Strategy. I've taken my hands-on experience and turned it into interactive lessons that will teach you exactly how to create strategies that drive real business results. Lock in your spot now during the presale with code holiday and save $200 before December 31st. Visit product institute.com today. I've been playing with LLMs on our side too for like our Product Institute training to see if we can make things more interactive. And what surprised me about it was some of the stuff that you just said too that I don't think a lot of people are looking at. And one was the cost side of it, right? Like there's a scale. And I think a lot of people are like, oh, LLMs, it's gonna reduce the cost. You're like for some, if you want to self-host, for example, an LLM on AWS, like it's expensive. Sure, it could replace a couple thousand people if you have that. But if you don't and you're doing it for the wrong reasons, like that's expensive. And I don't think people are actually looking at it.
Jessica - 00:30:49: Yeah, the cost side is quite interesting. And depending on the size of your business and what you're going after, it's definitely worth looking at. That is one of the things we assessed when we were doing our AI assistant for customers. Like what is the cost per chat? And when you're thinking about conversion, customer acquisition, which I'm sure many people who are listening will be very familiar with that, then you have to think about what is the cost of a chat? You know, what's it gonna do for your first, or the second order and your conversion metrics? Are you seeing it as a way of doing acquisition? And like, have you built that cost in? And it can be surprisingly expensive. And I think that probably the other thing is it isn't just the cost of the technology, it's the cost of maintaining the team that needs to continue to work on that, the skill sets that you need to build. And that's something I also take very seriously, not just with AI, machine learning and other technology in that kind of space, but are you building a department that is fit for the future? Are you training your colleagues, your team on the things that they need to know to be effective into the future? And so, you know, as you adopt these technologies, it isn't just about adopting the technology, it's about creating the entire ecosystem of your department that's able to take it forward.
Melissa - 00:32:06: Yeah, it's building in a capability to your business and that's not free.
Jessica - 00:32:10: Yeah, it is. And it's not a reason not to do it, but that expense piece, there's lots of other things around the data governance and how you govern it. And bias is a great topic to talk about on this as well. And how do you tackle all of those parts of the solution as well? I'm not for a moment saying don't do it. You've got to go into it eyes open and not be dazzled, but really make the right decision for you.
Melissa - 00:32:35: Yeah. I think you're being realistic about it. The hype cycle is like, oh, this can solve everything, right? And everybody wants to go grab AI and LLMs. And when you dive into it, you're like, oh, this can solve for certain things in certain situations, not everything. And I think that reality is hitting a lot of people right now. And that's for the best. You just mentioned something too about bias. And we have talked a lot about bias in AI and it's a computer. It's generating stuff based off the data you give it. How do you help run the governance around the AI that you have? How do you mitigate for that risk? You also talked about it. Giving the wrong answer or doing something that's not desired because they do hallucinate. What do you do to manage risk around there?
Jessica - 00:33:14: So all of these things, I would say, we do have a robust data governance policy and procedures already in place. And AI falls into that category already. So we already have much of that in place, but we have developed that out further to make sure we really get this right, or at least that we are giving it the best chance we can to be on top of some of that stuff. Because you know, it's a lot of work. It's a lot of work. It's a lot of work. It's a lot of work. It's a lot of work. It's a lot of work. It's a lot of work. You know, it is new and is developing all the time. So we do have a cross-functional team of people who look at the solutions from all perspectives. So we have data protection office, legal, tech, product, various different people who come together regularly to talk about, you know, how are we using it? What are we doing? What are we learning? What's the industry saying? And making sure we're really on top of all of that. And we try to be transparent with customers as well. So that's a lot of work. And I think it's a lot of work. And I think it's a lot of work. AI assistant that I'm talking about, we give the customers the choice to opt in to us analyzing their chats. We want to, because we want to know what they want, but we're giving them the choice and we're being transparent about that in the first chat that they have. We ask them that question. So I think that's also really important. Not only that, we've set up an AI guild. So anyone with an interest across the entire organization, not just product, not just tech people, but anyone can get involved in the AI guild. And through that guild, we are educating the customers. And we're giving them the choice to opt in to us analyzing their chats. We're educating people on the foundations, the policies, safe use or correct use within our guidelines so that we can make sure that we don't allow this to go unchecked and that we are really eyes open to the risks and the way we use it.
When it comes to bias and inaccuracies, it's difficult. So we've created a cross-functional team that work on the AI system that's made up of a number of different people, different backgrounds, and it's quite a diverse team. And I think it is very important to have diverse teams building solutions, not just for AI, but especially for AI. Because if we don't think about how we train the models, we risk exacerbating some of the bias that we see in society today. I am not for one moment saying that we have solved that problem or that we are the perfect business and we know exactly how we're doing it, but we're giving it a very good go and we are really aware that that's something we have to be on top of. In terms of inaccuracies, it's difficult. The longer the conversation goes on, the more likely the AI is to hallucinate because of short-term memory and all of the things that how LLMs operate. So from my perspective, it's not just for the inaccuracies, it's also better for the customer if we answer their question quickly and they don't have to ask five times for the meal that they want, but they can get a pretty good answer straight away. And speed is important too in that. So actually, initially, we found that our assistant was too slow. Customers were waiting too long, so they were asking again, and that's much more likely to lead to inaccuracies. So those are some of the things we've done. But I think just like everybody else, we're on that journey and we're learning and developing alongside the technology, and that's what makes it really exciting as well.
Melissa - 00:36:26: When you're thinking about building diverse teams, especially around AI, what types of factors are you putting into play to make sure that the team feels capable to think through a lot of the risks?
Jessica - 00:36:36: So I think I'm all about empowering teams. And what empowerment means to me is that they understand the goal that we're after. That's not up for debate. But they have voice to be able to speak up and say and tell us the truth about how we're going to get there or what might be going on. So that's very important. I will come back to customer closeness like a broken record, but that's also key. We always hire talent and build teams of like the best possible teams that we can build. And we have a number of policies and I'm a strong believer in diversity and inclusion and creating a culture that brings genuine inclusion so that, you know, people can speak up with different opinions, different backgrounds, and it is listened to. That's important. But what I also say is that even if you consider yourself to be diverse, so for example, I'm a woman in the tech industry. There's fewer of us than there are men. But I'm not the most diverse person. So even those of us that consider ourselves to be in some way diverse, we still need to put our feet in the shoes of other customers and users. And we need to ask ourselves those questions. So have we looked at it from the perspective of a different group? And that goes for everybody in my team. And so I come back to the customer closeness because when I talked earlier about, you know, I talk to a customer face to face every month. I often ask the team that helps me find these customers, hey, you know, I want to talk to somebody with a disability because I want to know how they use our accessibility features and if they're good enough.
Or I want to talk to someone from a certain economic background because I want to know about this type of thing or that type of thing. And that's what I'm encouraging my teams to do as well, to ask questions in groups that they aren't part of. Not only that, because this is a topic that I could talk about forever. I also talk about what media do you consume? What articles do you read? Who do you follow? And can you diversify that? So, you know, in my, we don't call it Twitter anymore, do we? X, I'm still kind of used to it. In my feed, I follow people from lots of different communities that have different perspectives. And it's really opened my eyes to certain challenges that they have and how we can solve those problems. So it's a whole holistic piece and we're never finished with it because we need to be curious and asking those questions. I literally could talk about this. Because I think this sparks creativity for everybody. I think it helps us to like achieve purpose and meaning in lots of the work we do. It's such an important topic for me.
Melissa - 00:39:12: Tell me a little bit. You've been building these teams. What are some of the results that you've seen from putting the diverse teams together?
Jessica - 00:39:17: Well, that's quite hard to quantify sometimes.
Melissa - 00:39:21: Qualitative is fine too.
Jessica - 00:39:23: Because there's loads of studies out there that will tell you that a diverse team will bring you more revenue or whatever. But what I would say is where you get a high performing team with diverse perspectives. And when I'm talking about a high performing team, I'm talking about a team that is able to challenge each other and not fall out. Who's able to step in and out of people's remits and it's all part of achieving something together. The sense of accomplishment that I hear from the people in those teams is second to none. They enjoy the work and the challenge and the new perspective is something that sort of powers them forwards. But I've also seen in a number of scenarios better outcomes because we've thought of maybe less obvious use cases or things that we can do that just change things for a certain group. And actually we get great feedback from them that they really like that. And that all comes from bringing those different perspectives to the group. And those different perspectives come from lots of different ways. I'm not only talking about gender. I'm talking about, you know, cultural diversity because in different countries we do things differently. I'm talking about social background. I'm talking about race. Like all of these factors bring a different perspective. And I think that is really at the heart of innovation and creativity, to be honest.
Melissa - 00:40:41: I completely agree. And I've seen the same thing. We put teams together where people are not having the same experience and all of a sudden they're challenging each other, thinking outside the box. And I think that makes better products at the end of the day.
Jessica - 00:40:53: Yeah, I couldn't agree more. And I think the sense of achievement from those teams when they recognize that they're serving a greater group of customers or they've achieved something for a certain group that hasn't been recognized before is also something that stays with people and is something that they come back to throughout their career as a thread.
Melissa - 00:41:14: One of the things I wanted to talk about when it comes to AI too is we talked a little bit about the technology implications, the implications to the org about having to hire or stand up these teams. We talked about the cost side of it. What about the user acceptance side? Some people are very afraid of AI these days or they're like, oh, I don't want to give those people my data or I don't want to do this or they're wary when it comes to that stuff. Have you seen any challenges on that side?
Jessica - 00:41:37: We've not seen any overt challenge from any specific customer. But yes, I'm really aware of this. And I think generally in society, we've all become a lot wiser to the value of our own personal data, our digital footprint. And, you know, the fact that stuff lives forever on the Internet pretty much. And there's definitely a shift in consumer attitudes. But it is for this reason that we, you know, we put that right at the beginning of the chat with our AI assistant for customers. We tell them we would like to analyze your tasks. Is that OK? Because we want to give them control over that. But I think this is going to be one of those key elements of successful AI implementations because there's lots of fear around it and there's lots of misunderstandings. And. Yeah. When you're a product manager and you're providing someone with an app, you don't have much space to explain to them exactly what you're going to do or exactly how it works or any of that stuff. You can't really allay those fears. So I think you just have to be open and transparent to as much as you can. And I think we'll see in society that this will be one of the key elements of successful AI implementations as we move forward.
Melissa - 00:42:44: When it comes to AI helping you with decision making on the product management side, how are you leveraging it to help make better strategic decisions?
Jessica - 00:42:52: That's a great question. And I think I'm not sure we're there yet, if I'm honest. We're still planning and looking at things from a commercial value and customer experience perspective. And I don't think we're yet at a place that we're using AI to make those decisions. However, we are experimenting with these tools internally, as well as with our customers and our partners and things. For example, we've got an HR chat. So if you're not sure what the policy is, or you want to know something, you can chat. And we've had that live for a few weeks now. So we're seeing how employees, how our staff are responding to that and what they're finding. But we're not yet leveraging it in our planning. And I think that's something that's still to come. I think there will definitely be opportunities. And I expect that lots of the tools we use are going to be bringing out more AI tooling to help us make those decisions as we move forward.
Melissa - 00:43:44: You know, that's kind of refreshing to hear, because I feel like a lot of people start from the other perspective, which was, how do we just use it to help me do my job better internally, which isn't a bad place to start. But a lot of them are not doing what you're doing, which is starting from the commercial aspect and coming back.
Jessica - 00:43:57: I think you have to if you want to be successful. I mean, no doubt that AI is going to revolutionize things and, you know, will be a significant change. You know, for example, I think probably operating systems will massively change and actually be based on LLMs and things like that. Rather than, you know, file systems and all the things we have today. And being a software developer will also massively change. We're already seeing those tools. But if we don't think commercially, especially as product managers at the intersection we sit at, you know, if we don't bring it back to that every single time, then we're not doing our job properly. That's my perspective on it.
Melissa - 00:44:34: When you're looking at the landscape of AI and with things moving so quickly, what are you observing in the market? Or what are you looking for to keep ahead of these technological changes?
Jessica - 00:44:43: I would first of all say my ethos is very simple because you'll keep hearing it. My first perspective is broad thinking is really important. The more you consume from different sources and broad perspective, the more dots that you can connect. And suddenly you come up with the idea. So in terms of like, how do you keep ahead? I'm just constantly reading, talking to people, asking questions, listening to podcasts and kind of pulling in all these different perspectives and trying to see, is there a threat? Is there not a threat? Is there something that I hear that really sparks something in me? And then I want to go and have a look at that. And I do that for tech. But in the world of product with consumer facing products is not only about the tech, it's about the customers. And we can all probably think of examples of, you know, startups and tech that was maybe just a bit ahead of its time. And now it, you know, it would be adopted. And that's because it all comes down to consumer confidence, consumer attitudes. Way of life.
For example, we do in-car ordering with Mercedes because we've seen the adoption of electric vehicles and in Europe as in the US for sure, customers are doing really long journeys sometimes and where they would have done a five minute fuel up, they're now charging for an hour. So we've seen an opportunity to respond to that by allowing them to use that hour effectively to have a delivered meal or to pick up some groceries, or maybe if they're on their way somewhere, they can get a gift on the way. Delivered to them while they charge their car. That's not something that would have worked probably even three years ago or four years ago, because the adoption of electric vehicles just wasn't where it is today. So it's about technology, but it's about the environment and whether the two are overlapping enough that now is the moment to do that, or these all play together. So I am always looking at the tech and educating myself and encouraging my teams as well. But the consumer behavior trends, that is equally as important.
Melissa - 00:46:48: I think that's great advice to leave everybody out there. Look at the consumer behaviors, then bring it back to tech. See if we can solve it that way. Thanks so much, Jess, for being on the podcast. If people want to learn more about you, where can they go?
Jessica - 00:47:00: The best place to find me is on LinkedIn. And that's where I share what we're working on and some of the interesting stuff that I am seeing.
Melissa - 00:47:07: Great. And we will put your LinkedIn link into our show notes at https://dearmelissa.com. Thank you, everybody. Thanks for listening to this episode of the Product thinking Podcast. We'll be back next week with another amazing guest. So make sure you like and subscribe so you never miss another episode. We'll see you next time.
Jessica - 00:47:22: Thanks very much for having me.