Episode 188: How AI is Redefining Enterprise Productivity with Tamar Yehoshua of Glean
I recently had the pleasure of speaking with Tamar Yehoshua, President of Product and Technology at Glean, about the dynamic intersection of AI, product management, and enterprise technology. Tamar, with her deep experience leading product teams at Google and Slack, provided a fascinating look into the rapidly evolving world of AI and its impact on how we work.
In our conversation on the Product Thinking podcast, Tamar delved into the intricate process of developing AI-driven products, the unique challenges of managing product teams in a startup environment, and the strategic considerations of scaling in the ever-changing tech landscape. We explored the critical role of retrieval-augmented generation (RAG) technology, the importance of maintaining a strong engineering focus, and the future of productivity in the workplace. Tamar's insights were not only insightful but also incredibly relevant for anyone navigating the complexities of product management in today's fast-paced, AI-powered world.
You’ll hear us talk about:
06:36 - The Critical Role of Security in Enterprise Search
When discussing Glean's approach to enterprise search, Tamar emphasizes the paramount importance of security and privacy. Glean’s platform deals with vast amounts of proprietary information within enterprises, making it essential to ensure robust security measures. Tamar explains that the platform is designed to adhere strictly to the privacy controls and policies of various SaaS applications, requiring meticulous attention to detail to maintain compliance. She highlights that Glean offers both a hosted service and the option for customers to host the service within their own cloud instances, ensuring a high level of data isolation and security.
This focus on security is critical for building trust with enterprise clients, particularly when dealing with large organizations that have stringent security requirements. Tamar’s insight into the complexities of securing enterprise data showcases the deep technical challenges involved in developing a product like Glean. It also reflects the broader industry trend where security and data privacy have become fundamental selling points for technology solutions, especially in the enterprise space.
27:28 - The Importance of Personalized Retrieval in Enterprise AI
Tamar Yehoshua emphasizes the challenges of integrating large language models (LLMs) with enterprise data. Unlike general LLMs, enterprise solutions require sophisticated retrieval mechanisms to ensure that only authorized and relevant information is accessed. Tamar explains that simply dumping all enterprise data into an LLM is not feasible due to issues with permissions and the specificity required in enterprise contexts. Glean’s approach is to index all enterprise data, enforce metadata-based permissions, and personalize results based on the user’s role and relationships within the company.
38:17 - The Evolving Role of AI in Enhancing Productivity
In discussing the future of AI, Tamar touches on the emerging trend of agentic workflows, where AI systems can perform multi-step tasks autonomously. Although these systems are still in their infancy and require user intervention, Tamar is optimistic that they will soon become more reliable. She believes that as AI models improve, these workflows will enable users to offload routine tasks, allowing them to focus on more strategic and creative work.
Tamar envisions a future where AI significantly enhances workplace productivity by handling time-consuming tasks such as summarizing meeting notes or updating databases. She also addresses concerns about AI potentially replacing jobs, advising professionals to embrace AI as a tool to improve their own efficiency and capabilities.
Episode Resources:
Other Resources:
Follow/Subscribe Now:
Facebook | Instagram | Twitter | LinkedIn
Intro - 00:00:01: Creating great products isn't just about product managers and their day-to-day interactions with developers. It's about how an organization supports products as a whole. The systems, the processes, and cultures in place that help companies deliver value to their customers. With the help of some boundary-pushing guests and inspiration from your most pressing product questions, we'll dive into this system from every angle and help you think like a great product leader. This is the Product Thinking Podcast. Here's your host, Melissa Perri.
Melissa - 00:00:37: Hello, and welcome to another episode of the Product Thinking Podcast. Joining us today is Tamar Yehoshua, President of Product and Technology at Glean, the AI search company helping enterprises effectively harness their data to make better decisions. She's an experienced executive with a profound background in leading and delivering innovative products on a massive scale. Tamar has deep experience in search, large secure systems, and workplace products, which has been accumulated from her experience as a chief product officer of Slack, VP of product for identity and privacy at Google, and also the VP of advertising technology at Amazon. But before we talk to Tamar, it's time for Dear Melissa. So this is a segment of the show where you can ask me any of your burning product management questions, and I answer them every single week. Go to dearmelissa.com and let me know what your questions are. If you leave me a voicemail, I do answer them faster. So let's go to the phones today.
Caller - 00:01:31: Dear Melissa, you recently talked about the use of AI in product management. I'm wondering in the cybersecurity industry or banking industries, which are industries which are super secure, where the use of AI is quite restrictive. How do you see that we can still use AI? And how can we leverage all the capabilities, given that it's quite restrictive? Thanks a lot.
Melissa - 00:01:53: So when we're talking about AI and cybersecurity and banking, these companies have been using AI for a very long time. But there's two different facets of this. There is, can AI be used as part of your product strategy that helps us deliver value to customers? And then there's also, can we use AI as product managers in this company to do our jobs more effectively? So when we think of that strategy piece, can we use AI to be part of our product strategy when it comes to banking and cybersecurity? Of course. You don't have to just use GPT-4. Many of these companies, when they have privacy concerns, are worried that you're passing customer information back into a central hub. So if we think about OpenAI, the way that that AI works is that it lives on external servers. So we would be passing information back out to external servers. But when we do product strategy with AI, a lot of companies are building their own AI models internally, or they're using open source to build these models so that they can host them locally on servers and you're not passing them to a third party. Is that the models can run on your own servers. So if we think about it, banks are using AI all the time right now, and they have been for a long time. They've been using them in chatbots to help with help. They've been doing it to better user experiences so that you can come in and ask for what you want and find it really easily or open a bank account faster, anything like that. They also use it to track customer data and then use that in useful ways to come back and prevent fraud, for instance. So if somebody is using your credit card and they saw that you were just in New Jersey and all of a sudden somebody is using it in Mexico. They'll be like, oh, my God, that's fraud. So they will stop that. And that's all built on AI models.
So a lot of this is baked into the way that we do product strategy. And AI has been doing this for a long time. What I think you're asking about, though, is how do I leverage AI as a product manager in this space when I can't use things like ChatGPT? That part's a little difficult because we have to look at what's available to you as a product manager when it comes to useful tools and what will help you with those privacy concerns for your company. So a lot of companies I know will not let people use ChatGPT because, again, the interface could be passing potentially confidential information back into the back end of OpenAI. Everybody's worried that's going to get lost in the sauce and mixed up with everything else and then come back to other people. So your confidential information will be put into the algorithm and will pop up to somebody you don't want it to pop up to. That's the fear. So that's why a lot of companies don't want you using ChatGPT. But that's not the only model out there and that's not the only AI thing out there. So what I'm seeing as a trend, though, too, is that because this is a concern, many companies when it comes to AI are building open source models. And the companies that are harnessing AI to help you be a better product manager, are also thinking about how do we host this on a single server and not integrate with things like GPT. So, for instance, just today, you won't hear this today, but just today, Llama 3 did come out from Facebook. I can't tell you how great it is because I have not looked at it yet, but they did an open-source model. And this is really big progress because what that allows companies to do is host those models internally. So they're not passing information back to Facebook about your confidential information. So you can host it on your own servers. You can run these things.
You can still answer your questions and build your own interface to be able to do the things that you would with ChatGPT. But Llama 3 is not external, right? It's going to be something that you host internal. And that's what we're seeing. A lot of companies are signing agreements, too, with OpenAI so that they can use things like ChatGPT. It's just that you wouldn't go through that interface. You would usually build an interface yourself that hooks into your own models or own separate server where it's running data and not passing it to other people. So we're seeing that in a lot of large companies as well. What I think is going to happen is there's going to be a trend, of more and more companies understanding that AI is really important. And with these large LLMs, you can't just be passing data out there to everybody else and using it to train algorithms. You're going to see a lot more companies go to using dedicated servers per company that they sign contracts with. And you're also going to see a lot more open source models out there where you might be able to download it, create your own way, your own interfaces to actually chat with it and do your job better. So that's what I would look out for. I think give it a couple more months and you're gonna see many, many tools out there where you can start to grab it and get going. In the meantime, if you want stuff that will help you process PRDs faster or use it in that way, talk to your developers and see if you can harness some of these open source models and create simple interfaces on your end so that you can leverage AI in those types of ways to do your job a little more efficiently. That's what I would look at in the meantime. You're also gonna hear about how Glean really protects the privacy because they do use AI models in the next segment with Tamar. So let's go talk to her.
Advertisement - 00:06:29: Are you eager to dive into the world of angel investing? I was too, but I wasn't sure how to get started. I knew I could evaluate the early stage companies from a product standpoint, but I didn't know much about the financial side. This is why I joined Hustle Fund's Angel Squad. They don't just bring you opportunities to invest in early stage companies, they provide an entire education on how professional investors think about which companies to fund. Product leaders make fantastic angel investors. And if you're interested in joining me at Angel Squad, you can learn more at hustlefund.vc/mp. Find the link in our show notes.
Melissa - 00:07:04: Hi, Tamara. Welcome to the show.
Tamar - 00:07:06: Hey, Melissa. Thank you so much for having me.
Melissa - 00:07:09: It's awesome to speak to you because you have so much profound experience in product and technology at some of the world's largest companies, ones that we all know and use like Slack, and now you're at Glean. Can you tell us a little bit about your career journey so far and what led you into product and technology?
Tamar - 00:07:25: I started out as an engineer and then gradually moved into engineering management and then into product management and then more like GM roles. And I think one of the things that defines my career is when I made changes, I was always looking at the technology changes and going from software into the internet and then from the internet into more into like Google search and then into Slack. I'm like looking at what are the things that are the most interesting? Where are like the interesting new companies? Where are the interesting people going? Where are you going to learn the most? And then follow that. So that's kind of. The way that my career has progressed and sometimes ways that people looked at me and like, why are you doing that? Like, why are you leaving this great job where you're managing a large team and a great company to go to like a company nobody's ever heard of? So there is a couple of transition points in my career like that.
Melissa - 00:08:22: Well, with the latest one too, you were at Slack. So you were the chief product officer of Slack. You went to IVP, Venture Partners. You were a venture capitalist for a little bit. And now you're at Glean, where you're the president of product and technology. Can you tell me a little bit about what motivated the shift from an operational company like Slack into venture capitalist? And what made you go, oh, I got to join Glean?
Tamar - 00:08:45: It's a really good question. So when I left Slack, Slack was amazing. I had such an incredible opportunity to join pre-IPO, watching incredible scale at the company and then through the acquisition. And I wanted to do something really different after Slack, try something different. I'd been leading products and tech teams for decades. And how do I learn a new skill, challenge myself in a new way? And executives are always looking at venture capital as, okay, when I'm done with the executive job, let me try venture capital. And I had a bunch of friends in venture capital and I knew IVP really well, and they had invested in Slack and I knew one of the general partners really well. So I'm like, let me just go and see what it's like. Loved it. I learned so much. And it was really good timing because it was right when the AI wave was starting. And so ChatGPT had come out. There were all these new AI startups. Nobody knew quite what to make of it. So instead of jumping to one company, I got to go and meet with a ton of different AI companies and get kind of the view of the landscape. And, that's what I really enjoyed. But I also realized that the job of a venture capitalist is very different than the job of a product or technology leader. It's very hard. It's a lot of hustle. And it's much more like a sales job where you're meeting a lot of people, you're trying to close a deal.
And it's much less about building a team, working with the same set of people, building a product, working with customers. And I really missed that aspect. So after about eight or nine months, I decided, you know what, I should go back to operating. And at the same time, I was on the deal team. For people who don't know what a deal team is at a venture capital, when you're looking at a specific company to invest in, there's a deal team that goes really deep on that. You look at all the metrics, you do customer calls. And so I was on the deal team for Glean and went deep. And I also knew the CEO Arvind from Google from years ago. And I was like, wow, this is really cool. And for people who don't know what Glean is, think about it started as enterprise search. Think of Google for your enterprise, it can get you any results based on documentation in your enterprise, and then add an assistant. So think ChatGPT for your enterprise. So I'm going to ask Glean any questions and it will know everything about proprietary data in my enterprise. So the customer calls were just incredible. People are like, this is an amazing product. And so that's how I ended up at Glean.
Melissa - 00:11:07: With the enterprise search and trying to go across with an enterprise search like Glean, trying to break into the enterprises, and get them to open up access to all of their information and their data to do this. It sounds like a big challenge, but a lot of people are trying to get in there. How does Glean position itself to really win? And how do you get the trust of the enterprises there? Why do they turn to you to say, hey, this is a solution versus all the other ones that we want?
Tamar - 00:11:34: First, you have to have a problem that you're solving, that they understand and that's acute. When Arvind, Arvind's the founder and CEO, when he started Glean, he was looking for an enterprise, search solution in the company he was at, which was Rubrik, which he was also the co-founder and tested a lot of other products and none of them were very good. And so he decided to build one. And because when you go to almost anyone in a large corporation, not a corporation with 50 or a hundred people, but corporation with thousands of people, and you say, can you find the information that you need? They'll be like, oh my God, no, it's such a pain. All right. Can you find the people that you need? Oh, it's so hard to find the right expert to talk to, about such and such. And so it's a very relatable problem. So when you go in and you tell somebody, we want to enable you to find any information that you need in the company. And now with the assistant, we want me, to be able to answer any question that you have related to any content in your organization. People get it. So that's the first thing you have to be solving a problem that people get. And then the first thing that people care about is, wait, what? You're going to index all of my content? So you have to have very strong answer on security and the policies about making sure that you can pass through a CISO, which is not easy these days and looking at all of your, your content. And you want to make sure that you adhere to the permissions of all of the SaaS tools where you're ingesting the data from.
Melissa - 00:13:03: So when you look at Glean, how are you approaching security and making people feel comfortable there? What are your key features that make these enterprises go, oh, wow, this is like this is a solution for us?
Tamar - 00:13:16: Security and getting privacy right is everything like under the iceberg. It's the really hard part. You have to get all the connectors. You have to understand the metadata of every SaaS app to make sure you are adhering to whatever policies and privacy controls that that application has. So it's a lot of very detailed work, and you just got to get it right. And we explain our approach, our algorithms, how we do it. And then people can try it. You do a proof of concept, and you show them that it works. So it's very important, to show them, go through the architecture of the system. And then we also, from a security perspective, we have a SaaS-hosted offering, but we also will host your documents in a customer's cloud instance in their VPC. So it's single-tenant in their VPC, and that's really important for large customers, that their content is not merged with anyone else's. So we have a very strong security stance, and we've been developing this over the last five years. So, it takes a while to get to that adherence to meet the requirements of all of the CISOs at large Fortune 500 companies.
Melissa - 00:14:31: What other challenges do you see with Glean's marketplace, right, which is enterprise search, or with this transformative vision of what AI can bring to the workplace?
Tamar - 00:14:40: We want to make sure that people understand the power of what we have, because we've got the ability to think about how executives and organizations have assistants. So we're building an assistant. So our vision is that everyone will have an assistant. You may have many assistants. And part of the problem is behavior change. If you get this product and you're not quite sure what you can do with it, and how do you educate people in what they can do with it? How do you get them to use it? We find with our assistant that it sometimes takes a little bit for people to try it. But then once they try it, retention is really high. So once you feel like, I know what this can do. And I've talked to a lot of other CPOs working in the AI space, and that behavior change is a big delta. Like, you try ChatGPT for your personal use. It was really cool. And then you forget it's there. And then you do things that it could help you with. But how do you get that into your normal workflow? How do you remember that, oh, wow, this can really help me? So we're trying to get a lot more directive and be very clear on what kind of workflows we can help with. So we launched this concept called Glean Apps so that people can build apps on top of the generic Glean Assistant for specific use cases and share those with their organization.
Melissa - 00:15:54: How do you educate people to, about those distinctive areas where you can use the assistant or where AI is going to work? Do you build it into the product? Are there like, besides the apps pieces, are there like a guided flow? How do you educate people about what's possible so they're not just using it for, let's say, like lightweight tasks or things that don't matter? Like, how do you get them to that aha moment?
Tamar - 00:16:15: So everything that you said, you have to do everything. You have to do onboarding. You have to have documentation with it. We have a prompting guide to help people, which like, these are the type of things you do. In product, we suggest things. So when you open up the assistant, there's suggestions of what you can do. And we have other things coming down the pike that will help guide even more. So it's an evolution of learning from our customers what works for them and looking at the data to say, okay, this does work. But it has to be both in product, the marketing materials and the enablement materials of like our customer success managers. So all across the board.
Melissa - 00:16:52: So when you're thinking about prioritizing the different features too for Glean that make these enterprises very excited to use it, you mentioned like security is definitely a barrier there. What types of things are you thinking about from a user perspective? We've got, how do we get in and get them to use an AI assistant, but what other things are on your mind, right? Leading product and technology to say we have to prioritize this. This is how we're thinking about them to really harness this wave of AI here.
Tamar - 00:17:19: We built basically a knowledge engine of your organization. So we understand your organization. We understand the content, the people, the experts, and we can answer any question that you have. Now, that foundation of that content can be used in so many different ways. So a customer success manager can come to help them prepare for a meeting with their customer. I may have, be joining a sales call. I can go to Glean and say, brief me on this customer. I might be engineer and saying, well, help me with this incident management. So once you have this content, there's so much more you can do. So this is why I'm so excited about the Glean platform. So we built a platform that ranges from no code to code. So you can build a no code workflow with the Glean apps, or you can use our APIs. A lot of our large customers actually access Glean through our APIs, not through our UI. And they build their own custom applications on top of the content. So when you're building a platform, what people do on top of the platform sometimes surprises you. Like, oh, wow, I didn't realize you could do that. That's really cool. Like, we have some people building apps for competitive analysis or for helping with deal approval. That's where it gets really exciting to me is give people the tools that they need so that they can build things on top of this content and that saves them time. We go to a lot of large organizations and the CIOs will be like, yeah, we have a team in IT who's trying to build this for our company. And then we realize it's really hard and we're a year later and it doesn't really work. We would love to just swap out and not like, let's do the stuff that we're good at. You do the stuff that you're good at. And then they can build on top of it.
Melissa - 00:18:57: With this too, it sounds like there's a really good opportunity for the user research there, right? Seeing what people are doing with it, seeing what trends are. How are you encouraging your team to keep in touch with what are users doing? How do we follow these trends? How to identify, you know, potential new opportunities?
Tamar - 00:19:13: We really need to look at what's happening in the market and use the tools that are in the market. So at Glean itself, we've asked people to use AI tools, use tools from other companies, see what people are doing. Because you don't want to get stuck in a place where everybody else is using the new stuff and you're just doing things the same way. And so by using these technologies in-house, then you know what people are doing. Of course, talking to customers. Like all of our product managers and leaders talk to customers all the time. So you have to talk to them. You have to sit down with them. You have to understand what the workflows are. What are the problems they're trying to solve? And they tell you. So when you've got a product that... Is sticky, that's when you get tons of feedback. As a product manager, the worst is when you put a product out there and you get no feedback. That means nobody cares. But we get a ton of feedback. We're like, oh, we really want to use it for this, but it doesn't quite work. Can you fix it? And those are things that we're working very actively on of trying to go to where our customers want to go and where they're pulling us.
Melissa - 00:20:18: With that prioritization, there's so many different directions that you can go in, right? You've got all this feedback coming in. You're at this flux where you're growing very quickly. All these large enterprises, they see the power. They want the help with it. How do you think about prioritizing and what we do now versus later, especially as you're scaling and as a five-year-old company?
Tamar - 00:20:39: This is the canonical product manager roadblock question. How do you balance it all? And so, we did a lot of work in our planning for this year. And one, we're scaling really fast, scaling in customers. And so you just, you got to keep the wheels on the bus. You have to make sure that things work and you have to keep investing in automation and in processes so that you don't scale linearly with your customers. So a lot of effort just goes into scaling. I remember when I got to Slack, we were just coming off of a year of PRQ, which was performance, reliability, and quality. I think every fast-growing company goes through a phase where it's like, whoa, we need to fix and redo a bunch of infrastructure in order to be able to scale. So you got to do that. It's kind of table stakes that you have to do. Then you want to do what the customers are asking for and make sure that you do enough that you're not letting them down. But then you want to have enough resources to be leapfrogging and to be going where the industry is going or ahead of where the industry is going.
So you have to be really careful to make sure that you can invest in all three of those because I think if you miss any one pillar there, it's a problem. The leapfrogging and the moving forward doesn't have to be a large team. It can be small groups of people that are the key people that are thinking ahead. And you need to isolate some resources in order to be able to do that. But that's the hard part. The hard part is how do you get people who are going to be working on this stuff that's going to be really important in a year or two. And in the time I've been in this industry, I don't think I've ever seen it move as fast as it is with AI. Every week there's a new company doing something or a new launch from a big company. And so it's pretty incredible to see the pace. But they're also the tools are better. You've got LLMs that can do so much more than what you could build before. So you have to be very aware of what it can do and keeping pace.
Melissa - 00:22:37: When you're thinking about that balance too, in your organization for how do we work on stability, how do we work on growth, in a lot of growth stage organizations, especially ones that take venture capital, some of them will sacrifice like growth at all costs. And I hear them kind of go back to, oh, like, should we really prioritize stability or the board won't understand stability? Having seen the venture capital aside too, how do you recommend product leaders out there justify the need for making sure that you can invest in those areas so that you can scale but while also managing the expectations of growth and adding new customer features and bringing more people on board?
Tamar - 00:23:13: You know, if your product's working and the business is working, the board and the VCs don't get involved at this level. That's what I've seen. Like, I don't get these kind of questions. They get involved if something's breaking. Like, if there's customers who are super unhappy and are churning because the product isn't working. Or if you're not innovating and pulling new things and launching new things. Or if revenue grows slows. So, if the company is working, then you have the ability to prioritize and put resources where you need. If, as I say, churn is a big issue that people look at. So, if customers are churning because of performance or reliability or it's not meeting their needs, then you really need to worry. But I don't find people get involved in that level to that level. They get involved when there's clearly a problem. So, you just need to make sure that you're allocating the resources to not create those problems. And to be ahead of it so that you can make the right decisions. But I've never had an issue. Where somebody's like, you should take resources off of stability. Like, I just have never in my career gotten to that level of detail with somebody on the board.
Melissa - 00:24:19: That's good to hear. I think it also is about probably controlling the narrative with the board too. Making sure that you're telling the right vision and the story. How do you make sure that they understand where you're going, where this vision is? And how do you recommend other people communicate that clearly to them? Like, what are some mistakes maybe that you've seen people make in board meetings or with the board in that sense?
Tamar - 00:24:41: Well, one, you have to have a vision. That I think one mistake is people come to board meetings and they have a list of features. And they show a roadmap and say, this is what I'm going to launch in this quarter, in the next quarter, in the next quarter. And like everybody's eyes glaze over because the level of detail of the features are not telling a story. So, you have to tell a story of why what you're building is important. Why customers care. And you actually have to have a vision of this is what we're trying to accomplish. And this is what we're working towards. And then show a demo. People love demos. People love seeing. And it can be like mock-ups. It doesn't have to be something live. But bring them along on the journey of what you're building. And don't get into too much detail. Have that as backup. So, you have to know what you're doing. So, if somebody says, well, how are you going to achieve that? And what are you launching next quarter? You've got to know. And you've got to be able to say, oh, well, we're launching ABC. And it's for these customers. CDE. And this is why. So, you have to show that you know your roadmap and you know what's there. But don't start with that. Start with the problem you're solving and why you're solving it. And why it's going to make a difference for the business. Why is this connected to what's happening? And how is this going to enable your business to continue to grow?
Melissa - 00:25:58: Did you know I have a course for product managers that you could take? It's called Product Institute. Over the past seven years, I've been working with individuals, teams, and companies to upscale their product chops through my fully online school. We have an ever-growing list of courses to help you work through your current product dilemma. Visit productinstitute.com and learn to think like a great product manager. Use code THINKING to save $200 at checkout on our premier course, Product Management Foundations. I think that's really nice advice for some product leaders out there listening. And I do like the fact that they're probably not going to get involved unless you're doing something wrong. So maybe it's something for people to step back and actually reflect on. Back to Glean a little bit and where you're at right now and your growth story. When you're also balancing that side with innovation and looking at what's next, how do you stay ahead of the market, right? Like, what are you looking at? What are you observing to say, oh, this is something that we might need to harness, or here's how things are going to change?
Tamar - 00:26:57: There's a Bezos quote. I think it's from him. I heard it from him when I was at Amazon, which is, be competitor aware and customer obsessed. So know what your competitors are doing, but don't be too focused on them. Be much more focused on your customers and what they need and the technology and where it's going. Make sure you're reading the latest LLN papers. Make sure you understand where you think that that's going to go. But I was talking to a startup founder in a different domain, not competing with Glean at all small company they just started. And he told me that his co-founder deleted the Slack channel that talked about competitors and said, you're spending way too much time obsessing about these competitors. Forget it. Focus on the customer. I was like, oh, my God, that is brilliant. Because, like, sales and marketing needs to worry about competitors. And I need to be aware of what's happening. But it's so much more productive for product people to really think about the problem that they're solving. Because your competitors might not be solving the right problem. They might not be doing it in the right way. Just because they take your marketing copy and put it on their website doesn't mean that they're doing the same thing also. So just don't obsess about that. But I think in this age of AI, what people should obsess about is where are the foundation models going? And what are you developing? And is it going to get better as the foundation models get better? Or is it going to become obsolete as the foundation models get better? Because they will get better. And you want that to be a positive. And you want that benefit to go to your customers directly so that what you're building on top of it will always be better with better foundation models.
Melissa - 00:28:34: I think that's a nice way of looking at technological trends. I find that a lot of product managers, too, who are getting started, they're trying to look at technology out there and say, how do I harness this? Or what should I be looking for in trends? I see a lot of mistakes, too, with companies who are just like, oh, AI, let me slap some on there. And trying to figure out how to harness it. When you are working with your product managers on your teams. How are you getting them to embrace technology or look at it? What are some techniques that you would advise for product managers to do to become familiar with what these trends are going to be? And how to stay ahead of that so they can understand how to digest it back into their product?
Tamar - 00:29:13: Try it. You know what I said before. You got to try it. You got to be hands-on. If there's a thing, my AI came out and I asked our designers to use it. I got access to the early beta to know the people there and then try it. Let us know what do you think? So this is in a different domain. It's not competing. So any company that comes out with an AI offering, use it. Assess it. Is it good? Do you like it? How is this doing for you? Using Glean. Like, we use Glean a lot at Glean. And we build our own workflows. So a product manager built a workflow to go through a set of gong calls and then put them in a spreadsheet in a specific format. And then be able to query that and say, what are the most requested features? And it was hard. Like, it wasn't like, oh, this is an easy thing to do. He was telling me the other day that things like it got wrong when you assess the gong calls first of what the Glean person on the call said and what the customer said. So then he had to adjust the prompt to make sure that it was clear that only listen to what the customer is saying that they want as a feature, not what the sales rep is saying. So things like that. There's a lot of nuance. And you learn by building this and then using it. So you should build the stuff that's going to help you in your day-to-day work, and then use it. That's really the only way of feeling it and being hands-on.
Melissa - 00:30:32: With AI too, what you were just getting into as well is you have to come up with the right answer, right? Like this going out there, trying to sift through all this enterprise data at Glean and come back with the right answer. What are ways that you are looking at how accurate your answers are, right? Like how do you measure that and get feedback, especially when your customers are in the enterprise or it might be sensitive data that you can't get, you know, can't just read everybody's answers? What types of ways can you make sure that this is working well?
Tamar - 00:31:00: So the system product at Glean is built on RAG technology. RAG is just retrieval, retrieval is just search. So it's built on our enterprise search, our core enterprise search. And the techniques that we use first, we have to validate enterprise search. And the same techniques we use are techniques that Google uses for their eval, except that we can't ask third-party raters to rate it exactly because of what you said, it's sensitive information. But we can run our own golden sets and queries. And so we have our own like, eval infrastructure for the search eval, and that's a very important component for the assistant because the assistant basically takes the query, asks search for the relevant snippets, and then generates a natural language. And we have signals that we get from our users. They do thumbs up, thumbs down on the response. We look at all the thumbs down. We don't get a lot of thumbs up. We usually get thumbs down, the traditional and all of these kind of Other techniques. And then you look at them and you try and assess them and bucket them and see how they're working. We also recently. I put out a paper a couple of weeks ago called the AI evaluator, which is using LLM as a judge. So other companies are doing that as well. But how do you do that in the enterprise? So I encourage people to read that paper, which it talks about how we use LLMs to validate the results of the LLMs that we're using in our product.
Melissa - 00:32:19: When you're doing this eval side too, is there like a team associated with this where you're using people to help go validate it internally? Or is it something that the LLM side sounds like you automated these pieces?
Tamar - 00:32:31: Right now we have an eval team who are building the tools. And our engineers, our assistant quality and our search quality engineers look at the data. We don't have third parties. I mean, maybe as we get larger again, because as I said, you can't have consultants because it's enterprise data. So it's a little bit trickier.
Melissa - 00:32:48: And for the RAG technology that you did develop here, there's probably a bunch of people who are listening to this podcast who are not familiar with that type of technology on AI. Can you explain a little bit more about why that's core to your product and what that actually does?
Tamar - 00:33:03: What you want to do with RAG is you want to retrieve the most relevant information and then use that to generate the answer because a generic LLM does not understand enterprise data. And you can't just dump. Let's say you crawled all the enterprise data. And put it into a model to fine tune it. You can't do that either because it won't adhere to permissions. You can't put all of your documents from your HR team into an LLM and then somebody can query when's the next layoff. That's not going to work. So you have to only bring back, only retrieve documents that you, the person who's asking the question, has the authority to read. We retrieve them. So first we index all the information. We put it into our own search index. We adhere to have metadata around the permissions. And then we use ranking signals to know what's the most relevant for you. We use personalization. So we know who you report to. We know who reports to you. If you're asking for an onboarding document, we will retrieve the onboarding document that's relevant for you, not somebody in marketing. So that's the personalization. We use topicality. What's the topic of the document you're asking for? Semantic search. What is, do we understand the semantics? And then our retrieval engine is a hybrid of keyword search and semantic vector-based embeddings. So we do vector embeddings with snippets to retrieve them in a semantic way. And then we combine those.
Melissa - 00:34:26: It sounds like it probably took a lot of rigorous research to get into this RAG technology and make it as great as it is. Can you talk a little bit about what went into developing that? And what does that team look like who are investing in it and are working on those technological advances? I imagine it's a little bit different than just a traditional product team that might be working on a user-facing feature like an app or something like that.
Tamar - 00:34:49: It's very engineering heavy. Our co-founder, Arvind, was an early search infrastructure engineer at Google. And our three leads in the AI team are all from Google, from search ranking and from Google Brain. And so they took a lot of what they knew from building consumer search. And Google's been using AI and ML in Google Search for years. That's why transformers were invented at Google to help Google Search. So from 2019, when Glean started, they were using things like BERT, which is large language models in search. And so this was from the very beginning using the latest AI before ChatGPT and GPT-3 and all of that. And so understanding that, as I said before, under the iceberg of all of the things that are really hard, I think one of the things that they did really well is they started with what's hard. They didn't start with, let me just take a language model, add it to like something simple, and get, they started with the real guts and the core. And because there was such a deep understanding of search in the team, they understood what was hard. So you ask the organization, there's a search quality team, a system quality team, an eval team, and embeddings and ML foundations team. That's the AI side. And then there's the product team, and then there's the infrastructure team. So that's kind of how engineering is set up. And then we're very much a very low ratio. We have a very few PMs, but we have very senior PMs who have a very broad scope, which is something I believe in for startups. If you want to move fast, you want to have all the communication go to one person. It breaks after a certain point, and we are hiring. But we have a head of product, and then we have all senior ICPMs in that org.
Melissa - 00:36:40: Wow, that's really interesting. For the senior ICPMs, are they dedicated to a certain team? Are they over, you said they have a large scope. Are they dedicated to specific teams? Are they working like really deeply with individual engineers? Or do they kind of like stay up more with engineering leaders?
Tamar - 00:36:56: It's the traditional. You assign a PM to a team, you make sure that they have an engineering team to work with. It's just the scope of the engineering team may be larger. And it may span, like the assistant spans the assistant product team and the assistant quality team. Because at the end of the day, what the user is going to see is a combination between the quality of the answer and the features that are there. So that's kind of one example.
Melissa - 00:37:18: I love hearing you say this, because I do think we have to raise up some product managers sometimes to be a little bit higher than just one to one on like small feature level so that they can work on strategy. There are a lot of people probably listening to this who adhere or have been told there is a certain ratio of like five to 9 engineers to like 1 PM. Do you subscribe by anything like that? Or are you just looking more at like boundaries of what they're working on?
Tamar - 00:37:42: It really depends on the phase of the company. When I was at Slack, anytime we did headcount planning. We always looked at the ratios. And because you had like a thousand engineers, so you have to operate at that level. And it depended like infrastructure at a different ratio than the growth team. The growth team had way more PM, for engineers than the infrastructure. So you have to understand which need what kind of ratio. But when you get that big, you kind of have no choice but to operate in that way at the executive level. When you're a startup and you're small, it's different. When you're a startup, you don't have the bandwidth to train, junior PMs, because it takes time. So the senior PMs have to spend time with the people, the junior ones, which you do have that. You need that redundancy and you need to do that when you're bigger. So it really you have to understand the phase of the company that you're in and the phase of the lifecycle and adjust to it. So our ratio is very different than I would have operated at Slack because it's different. But I do think we need more PMs. It's not that we don't. It's just we have to be very careful in how we do that. And I think as we grow, as Glean gets bigger, a lot of the senior PMs we have, they'll be leaders and they'll hire teams and they'll kind of grow with them, which is the same thing we did at Slack. In Slack in the early days, far fewer PMs. And then you kind of grow with the organization. You have no choice but to specialize.
Melissa - 00:39:03: With the side two, you've got this heavy engineering component. And I think out there, there's some companies, not all of them, right, but like some companies are like, what do I do with product management when it comes to this heavily technical stuff, right? Do we have technical product managers? Do we have platform product managers? Like, what are you thinking when it comes to the strategy side? How do you like run that when it comes to those very technical components of like search and AI and your models?
Tamar - 00:39:29: Depends on the engineering leads you have, and it depends on the culture of your company. So some engineering leads double as the product leader and they write the specs for what you need to do for that technical area. Like what type of deployments are we going to support? It's all driven by the engineering leads. We don't have any product managers on it. And they're really good at it, like they understand the customers. They understand what's needed. Would it help to have a PM? It would offload them. But right now, again, it's churning too fast because we're small. And as we onboard new customers, each new customer might have something we haven't seen before. So we have to understand what's doable and what isn't. So I think you just go faster in that case. But on the search ranking side, we just hired a PM who's going to work on search and work very closely with search ranking because you want to make sure that the changes you're making are impacting the end user and not just looking at the quality of those changes. When I was in Google search, it also was very similar. There were no PMs early on that worked with the ranking team. And as we got larger, we did have some PMs who worked with the ranking team. And so you want to leverage people's time. So the engineering leaders, you want to make sure that they're highly leveraged and not spending time on things that they shouldn't be spending time on. And if there is a PM who can help balance that, then absolutely hire them. But not too early when you want the engineers to be deeply empathetic with the customers.
Melissa - 00:41:00: What's a good sign for companies who are thinking, like, maybe I need to bring in a PM over these areas? Like, an engineering leader has been running it. We've been doing well, but this might be the time when I need a PM. What's some signs you would look at?
Tamar - 00:41:13: When it's not working anymore. When it's breaking. When the engineering leader is constantly, like, behind and doesn't have the time. And honestly complaining. Like, I got to glean and some of the engineering leaders are like, please hire more PMs. So you want there to be some kind of pull and you want them to recognize the value and to need it. And obviously you don't want to go too far that breaks for the customer. But you can start to feel within the organization when it's not working and when you really need to get more. And then, you know, sometimes there's alignment and there's agreement. It's just finding the right person may take longer than you want.
Melissa - 00:41:53: Well, it's really nice that your engineers were actually asking for the help, too. Because probably some people who are listening to this are like, oh, my engineers don't really need me. But I think that showcases how valuable product management is at your company, too. To show, like, people this is what we do.
Tamar - 00:42:06: I've heard that, but I haven't run into that personally. I think if your PMs are good, they're going to show how they're adding value pretty quickly.
Melissa - 00:42:14: So with Glean, too, you're in this really rapidly evolving AI space. Like you said, things are changing literally every day. I remember I was looking at some models like six months ago, and now they're completely different. So much better. So much closer to, like, you know, where OpenAI's models are. What are you looking at going forward in the AI space? Like, what are you paying attention to? What are you looking at? And what are you like closely monitoring to say like, hey, this might help lean or this might change the landscape of what we're working on?
Tamar - 00:42:44: I think what's really interesting is the agentic workflows. People have started launching things like Devin is super cool, but they don't work yet. They still need a lot of massaging and intervention, but they will work. Like as the models get better, they're going to work. And that's really interesting to see how you can have multi-stacks that the LLM will come up with. So that's one area that we're watching very closely.
Melissa - 00:43:11: When you're thinking, too, about the future of Glean, how are you thinking about the realm of enterprise workflow and productivity? What kind of trends do you see happening out there in how we do our work?
Tamar - 00:43:23: I think that we are going to be able to get a lot more leverage for our time. So take the example of the Gong calls that I gave to the PM who wrote a clean app. Save the time of listening to the Gong calls, updating the spreadsheet, summarizing them. You just saved hours. And then you can actually go to the creative work, to the strategy work of, well, what do I do with these answers of what people are asking for? So it's not quite there. There's still quite a bit of manual work you need to do to say, is this right? Are these answers right? But I think that that's what we're going to see. We're going to see a lot of these tasks get done by AI, by LLMs. And in the horizontal platforms like Glean, you're not going to need a lot of specialized tools. You'll need some, like lawyers are going to need a specialized tool because that's so distinct. So there's going to be some, but most of them, people will be able to create their own, like the custom GPTs and like the Glean apps, their own automations in a very easy way. So that they can spend more time on higher leverage items. I'm really excited about what that's going to feel like in the workplace.
Melissa - 00:44:27: There's a lot of talk out there too, when it comes to AI and everybody keeps saying this about like a lot of these things, people might be replaced, right? When it comes to some of these tasks and what we're doing. And obviously we're talking about some efficiency plays here. What would you say to people out there who are listening, product managers, other people? Who are worried about that, right? What would you say to concentrate on to make sure, to ensure that they have a place in this new way of working in these new workplaces?
Tamar - 00:44:54: Don't be scared of AI. Use it. Use it to make yourself better. If you don't, somebody else will. And then your job may become obsolete, just like with programming languages. As there are new languages out there, if you're an engineer and you're not learning the new languages and you're still like programming in COBOL or FORTRAN, you're gonna have fewer opportunities. So understand the tools, use the tools, use them in a way that makes you more productive, because I believe some jobs are going to go away, but new ones will be created. So be one of the people who's learning and can have the opportunity to do some of those new jobs.
Melissa - 00:45:31: I think that's definitely wise words for people out there, especially if we're working in tech because we're moving very quickly. Thank you so much, Tamar, for being on the show. If people wanna learn more about you, where can they go?
Tamar - 00:45:41: LinkedIn. I post on LinkedIn just Tamar Yehoshua.
Melissa - 00:45:46: Okay. We will definitely link to your profile. And for Glean, what's the website we can go to to learn more?
Tamar - 00:45:50: Glean.com.
Melissa - 00:45:51: We'll definitely send people there. Thank you so much for listening to the Product Thinking Podcast. We'll be back next Wednesday with another episode. We will have all those links to Glean and Tamar's profile at our website at productthinkingpodcast.com. And of course, if you have any product management questions for me, go to dearmelissa.com and let me know what they are. I answer them every single episode. Thanks, and we'll see you next time.