Episode 110: Breaking Down Today's Machine Learning Technology with Christina Pawlikowski
Melissa Perri is joined by Christina Pawlikowski, a teaching fellow at Harvard and co-founder of Causal, to help demystify machine learning and AI on this episode of Product Thinking. Christina discusses language models, the different types of machine learning, how they can be used to solve problems, and the importance of good data and ethical considerations when using machine learning algorithms. Christina Pawlikowski is a teaching fellow at Harvard University and co-founder of Causal, a company that helps businesses make better decisions with causal inference.
You’ll hear Melissa and Christina talk about:
How machine learning is essentially creating an algorithm or a model that can make good predictions based on data. There are three types of machine learning: supervised learning, unsupervised learning, and reinforcement learning. Good training data is crucial for machine learning algorithms to be effective.
When considering using machine learning, it's important to ask questions about things like how complex the decision that needs to be made is, whether the model has to produce a definitive answer, how high the stakes are, and how quickly the answer needs to come back.
Ethical considerations are important when feeding data into a machine learning model, especially when making decisions with high stakes.
GPT-3 and Chat GPT are examples of language models that use neural nets to generate predictions about what word or sentence comes next based on probabilities.The accuracy of a machine learning model is only as good as the quality of the data that is fed into it.
When incorporating ML into a product, it's important to plan for scenarios where the model is wrong and to consider ethical considerations such as false positives and false negatives.
Data scientists play a crucial role in assembling and cleaning training data, building and testing the model, and deploying it in production. The process may involve collaboration with machine learning engineers or other teams.
The cadence of working on machine learning is different from working on traditional UX-focused teams, with more downtime and exploratory time upfront. Slack time is important for data scientists and machine learning engineers to keep up with new techniques, write papers, and attend conferences.
Artificial general intelligence is probably further off than we think, and AI alignment is an important field to prevent any negative outcomes.
Resources:
Transcript:
Christina:
If you're looking at a problem where it's like, look, we have tons of training data. The data's really high quality. Either the stakes are fairly low or there can be a human in the loop. We feel like we've got the right infrastructure to sort of feed a model, then that's great, that's a really good fit. Once those things aren't true, it's still possible. Fraud. The stakes are really high and the latency requirements are really low and the quality of the data needs to be really high. But obviously that all still happens in an automated way. It just means your model is gonna take a lot more work and fine tuning.
Melissa:
Hello and welcome to another episode of the Product Thinking Podcast. Today we're talking all about machine learning, AI, GPT3, all of those fun terms that are flying around the internet right now causing such a stir. And we're gonna talk about what product managers actually need to know about all these things and help demystify them a little bit. So today I'm joined by Christina Pawlikowski, who is my teaching fellow Harvard, and also the co-founder of Causal. Welcome Christina.
Christina:
Hi, thanks for having me.
Melissa:
So Christina, you have now been working in machine learning. You know, you are the co-founder of this, but you've been a product manager at many places, TripAdvisor, you led product management at a financial institution before and now you're co-founder of your own company. How did you get into the machine learning world from this
Christina:
A little bit by accident, honestly. I got an MBA at Harvard and then I joined TripAdvisor right out of my MBA in their product rotation program. And I got really lucky. I started on the personalization team. That's where I met Jeff who ran the machine learning team at TripAdvisor for years. He's now my co-founder and I just fell in love with that kind of product work and I ended up doing it for effectively all of my time at TripAdvisor.
Melissa:
Christina and I, we were talking about this during one of our classes at Harvard too about what do product managers need to know about machine learning and AI. Now we've got GPT3 hitting the storm of the internet with their chap GPT models and what we found by talking to a bunch of students is that everybody wanted to do something with machine learning, but a lot of people didn't know what machine learning entailed. So Christina, can you walk us through a little bit about what is machine learning and what problems can you actually solve with it? Like what are good problems to actually solve with all of these models that we've been hearing about?
Christina:
Yeah, so there's a lot of different ways to sort of summarize machine learning. So the way I think about it is like essentially it's can you create an algorithm or a model where you feed this model some data and it comes up with a good prediction chat. GPT3 obviously feels like magic. I mean under the hood what's happening is it's just generating predictions for the next word in the sentence. I mean that's really like an oversimplification, but I think like that's the sort of core way I would think about machine learning. And if you talk to data scientists and practitioners, the way they categorize the different sort of types of ML is based on how does the model learn. So there's sort of three bins, there's probably more than that. I think of it as three. There's supervised learning and unsupervised learning and then reinforcement learning.
And so in supervised learning you're giving your your model a bunch of labeled training data, comment moderation is a really good example of this. You can give the model basically like here are 10,000 comments on a sort of social post that a human has looked at and like these are the ones that are good and these are the ones that should be filtered out. And so they get this like labeled training data. So supervised learning is usually classification like is this X or is it Y regressions? So like logistic and linear regressions are also supervised learning. And then the sort of second category is unsupervised. So that's more like you just give the model a bunch of data and you ask it to come up with like natural clusters. So the way we used this at TripAdvisor is you can just give it all of the photographs at TripAdvisor and it will put photographs that it thinks are similar together.
And then as a human you're looking at it and you're like, oh obviously this is all pictures of like pools or pictures of bathrooms cuz people take a lot of pictures of hotel bathrooms for some reason and you can sort of label the clusters yourself Another way unsupervised learning is used as like anomaly detection. So does this particular piece of traffic or does this particular transaction look like very different from most of the ones we see? So you might use that in like fraud detection. And then the last is reinforcement learning, which is interesting. So reinforcement learning is more, I think of it as more like continuous, that's maybe not quite the right word. But essentially in reinforcement learning you are giving the model room to like try new things and then you're giving it feedback on like whether or not the thing has worked.
It's easier to talk about reinforcement learning in the the context of a specific example but a technique that's very commonly used called bandits as reinforcement learning. And so you can imagine that you would set up like a test on your website where you feed in 10 copy options and you use a bandit and then you tell the model that you want it to optimize for click through rate and it will just slowly feed more and more traffic through the option that gets the most clicks. And then what's interesting about reinforcement learning is you get this interesting balance between optimizing for the options that are getting the most clicks but sort of continuing to explore the new options to make sure there isn't something better. So it's a really interesting sort of like balance.
Melissa:
Which one is and chat GPTs in?
Christina:
Yeah, so GPT3 is actually like an interesting blend of a bunch of different things. Let's back up. Let's talk about what is a language model maybe more generally. Yeah,
Christina:
At its simplest level, like a language model is just a way of given one word or predict the next word. You could do it at the letter level too, right? So like Q is usually followed by you or like jumbo is usually followed by shrimp language models have been around for a while, but what's new about them in the last few years is using this technique called neural nets to make them better. So in a standard language model where you're just sort of training on the amount of text that's out there, I mean essentially you're limited by how many potential pairs of words there are, right? So if you are saying I wanna look at everything that's ever been written in English and calculate probabilities between word pairs in the English language there's 29 billion word pairs, right? Given that there's about 170,000 words in English.
And so that means for like word pairs you can do it fine with what's just out there written in the universe, but as soon as you're trying to generate like sentences or paragraphs you get to like combinations. The number of combinations is like effectively more than the atoms in the universe. So you need some other way of doing this core task of a language model which is calculating the probabilities of what the next word in the sentence is. And so what is happening at the base is they're using this technique called neural nets to essentially generate or predict the probabilities.
Melissa:
Exactly. So we've got basically let's say in the case of GPT3 or chat GBT three, they're feeding them all the stuff on the internet from what I understand a bunch of yes websites and stuff. And it's making a language model about what typically follows different words, whether it's sentence structures, how all these words interact with each other so that I can start to predict when you ask it a question that this might be the most likely answer based on what it's already read. Yeah,
Christina:
Yeah. So it's predicting the probability of what comes next. And so we can go into neural nets in more depth if you want. They're sort of roughly speaking, modeled on the human brain. They're actually a surprisingly old technique. They're like from the forties neural nets. So they've been around a long time. But yeah, essentially you create this sort of fake brain, I hate to call it that because it sounds like weird and science fictiony, but you create this network of neurons essentially and you feed all of the English language or basically all the text on the internet through it and you see if you can get it to produce weights that like accurately predict for a given sentence what word will come next. Yeah. And so then what is happening with GPT3 generally speaking is you give it a prompt, it takes that prompt, it generates a word and then it runs back through and then it generates the next word and then it's like recursive isn't quite quite the right word, but essentially at every step it's taking all of the text, it's already generated, running it through GPT3 and then predicting the next word and then sort of taking all of it and starting over what happens on top of GPT3 to create the chat functionality is actually two more steps.
So they've got this sort of neural net predictive model and then they train it with humans. So humans interact with the chat, they grade it and sort of give the chat feedback. So that's more like supervised learning. So they're getting the model gets like a labeled pair, right? Here's some text, here's what a human thought of it and then they build a model on top of that, a reinforcement model essentially that predicts what the human grade will be. So like eventually humans are out of the loop because they've built this sort of third model that predicts the human feedback that chat G B T will get.
Melissa:
Cool. Okay. And so that is the last piece that we're talking about on top of it, that's what we interact with. If we go to like chat GPT on open AI and start typing into it. Okay, so that's like the reinforcement learning model that you were talking about so we can be like, nope, you're wrong <laugh>. Yes, just what some people doing. Okay. So one thing that I think is good to clarify and I think like a lot of people in tech know this but not everybody and we kind of got into this in class is your algorithms and your GPT3 and everything like that, it's only as good as the data that you actually put into it. How does that work? How do you feed data into systems? How do you have to think about the type of data that would go in there and what should people know about that when they're thinking about is machine learning right? For solving my problem?
Christina:
Yeah, so I talk with students about this a lot but also fellow founders as they're trying to think about whether or not ML is a good fit for their business. So there's a couple different ways I think about is ML good for this problem? And I think the first and most basic question is how is this model gonna learn? So what decision is it that you need it to make and then how are you going to train it to make that decision? And I think getting good training data is actually like a surprisingly high hurdle in a lot of cases for chat GPT3 obviously like you can use that out of the box, they've already done the training for you. But for a lot of algorithms, if you want say a model to determine whether or not a comment should be moderated out of your social network, the machine needs to be able to learn how to make that judgment call in some way.
That's kind of the first question is how complex is the decision? Do you have good data that a machine could learn from essentially? And do you have enough of it? That's really like the first sort of yes no branch. The second is none of these are like disqualifying, but they're just like things to think about. I think the next thing to ask yourself is does the model have to produce a definitive answer And like how high are the stakes? So if an e-commerce site generates personalized recommendations and like I think the couches are ugly, let's not so bad, but if like a financial institution is deciding whether or not I'm a credit risk, I think you need to be a lot more thoughtful and cautious about the data that you're feeding into that model cuz you end up with like lots of ethical considerations. And I think product considerations too, right?
At some point like you feed bad data into a model, you're gonna get bad predictions and that's gonna have downstream consequences. There's a couple other things to think about. Is this a good fit? So like how quickly do you need the answer to come back? Are you doing online inference? Which means that it's got actually some fairly serious infrastructure requirements. Can there be a fairly long lag like a couple of seconds or hours that's a little bit easier from an infrastructure perspective? And then can you have a a human in the loop? So let's say you're like doing moderation, right? It's very hard to like get the exact balance between precision and recall, right and moderation. But if you can just take the gray area stuff and hand it off to a human and sort of alleviate some of the pressure on a model.
So if you're looking at a problem where it's like look we have tons of training data, the data's really high quality. Either the stakes are fairly low or there can be a human in the loop. If we feel like we've got the right infrastructure to sort of feed a model, then that's great, that's a really good fit. Once those things aren't true, it's still possible fraud. The stakes are really high and the latency requirements are really low and the quality of the data needs to be really high. But obviously that all still happens in an automated way. It just means your model is gonna take a lot more work and fine tuning.
Melissa:
So when you say you have to have enough data to train this too, how do you know if it's enough or not?
Christina:
The like technical answer is lame and it's like you build the model and look it out of sample performance. The answer to that is gonna come on much more of like a case by case basis though. So it's hard to give you like a heuristic or like a framework. Some models are gonna need sort of more data and some are gonna need less. Some of it depends on the complexity of the decision. So it's just like it's a little bit hard to ballpark. But I think once you've narrowed in on the problem you wanna solve and roughly the right approach, there's usually pretty good literature out there on like how much training data will we need to make this work.
Melissa:
When you talk about a complex decision versus a not complex decision, whatever's going into that, what's an example of a question that's pretty versus one that's got more complexity in it With ml,
Christina:
Let me give you two examples. So I was at HBS and I was chatting with a couple founders and I kind of got, I got the like is ML gonna work for me from two very different ideas? So one was, I'm gonna obscure away some of the details cuz I think they're actually building these businesses. But it's basically like could you across a variety of platforms given a bunch of labeled training data, make a decision about whether or not to accept some text, right? A comment essentially that's a great fit, it's great because it's text, the text isn't that long and the decision is just yes or no and the stakes are pretty low. And then the other question I got was could you feed a fairly complex legal document into a model and have it pull out essentially like tech specs where those tech specs would then become the basis of a bunch of like fairly serious money intensive plans that felt like a bad fit for a bunch of reasons. Yes you could go back and look at like previous contracts into technical specifications, but the contracts were all very custom. They're long, they're complex, the stakes are really high. Like if you make a mistake in translating, then your specs are gonna be wrong and that means things are gonna get built wrong And it didn't seem like there was a great way to have a human in the loop on a like a sort of regular basis. So that felt like a much worse fit to me.
Melissa:
Okay, so what I'm hearing from you too is that the higher the complexity, the more you probably need a human in there to double check these types of things and it brings down the risk as well to make sure that it's right.
Christina:
It does. It also sort of increases the latency, right? Okay. Once a human's looking at it, it sort of knocks it out of the running for being a part of your product where you know a user clicks on something and then sees something, right? Yeah. It's gotta be like, yeah to think
Melissa:
About that would be like a bad thing if you were doing like automatic fraud detection like somebody's using your card across the country and you need to shut it off immediately before the charge goes through. You don't wanna waste 40 minutes trying to get a person there because a person took your money and left already. Yes. Okay.
Christina:
Yeah. And so I've never done financial fraud to be clear, but my general sense of how that works is they air on the side of like saying no and then you can appeal, which of course has downside scenarios as well, right? Like it turns out that you are actually standing at the rental counter and you can't rent a car. But
Melissa:
Yeah, it's bad when it doesn't work because it's you.
Christina:
Yeah. But I think that's a good thing to think about in general with ml and I think it tends to be neglected by people who are new to doing ML in product is like your model is absolutely gonna be wrong sometimes more sometimes than you are probably care to admit what happens when the model's wrong. Like what are you gonna do? And I think you have to plan for that as you're thinking about how to incorporate models and ML more generally into your product.
Melissa:
Yeah and I think that's a really good point too about ethical consideration. So for those of you listening out there, Christina and I would teach class on ethics at hbs and this is where we started getting into these topics about ml. And in the one example that we use, the one case that we use it is basically healthcare system that's out there. It's not healthcare, it's a software system used by healthcare let's say to predict if somebody is an opioid risk or not. Basically the doctors don't wanna fill prescriptions or won't actually treat people who are at high risk for abusing opioids because they have all these crazy legal ramifications of actually doing that. So a software company built an algorithm to predict if people were opioid risks or not and it turns out <laugh> in this one case that we were looking at, this woman kept getting denied the medication that she needed.
She had all these doctors turn her away and she was not on opioids. She was like, what is going on? This is absolutely insane. And it turns out it was because a algorithm and a model was actually doing these predictions so they were feeding it all the information, but one of the edge cases they didn't consider when doing all this training data and feeding it in was what happens if you actually prescribe opioids for your pet through the pet owner's name? And that's what was happening. So her pet was on this medication, this pain medication, the pet's like 14 years old, bad hip dysplasia, really suffering. So he is getting all this pain medication but you can't prescribe something to just a dog <laugh>. So it went through her and she'd pick it up at the pharmacy and eventually it fed into the data that she was getting all these opioids and she might be abusing it.
So she got cut off from all of her doctors and in this case it's a bad ethical consideration for that because the product managers are not picking up and it starts to make you think right about how do we do ml, where do we use it, why do these things happen? But getting back to the point of what Christina was talking about, nobody was thinking about what happens when things go wrong, what could be the edge cases where somebody was not actually in opioid risk, what could come up with a false positive? And that is incredibly important when we're actually training ML data, like Christine was saying.
Christina:
Yeah for sure. Obviously in things like finance and healthcare, like the stakes are really high as we've talked about like human in the loop is like a good way to deal with some of that where you're just taking the sort of high risk models, un very uncertain and you're just passing it away from a model. You can also do some things where the stakes are like lower and you don't have to provide an answer when your model's not sure, just don't show product recommendations. That's a very easy fix generally speaking like if you know that you don't want the model to return an answer, if the answer's not very certain, tell your data scientist that work with them to figure out what the right threshold is for when the model should return and answer and when it should tell you. It just doesn't have a very good prediction.
Melissa:
If you're a product manager working in ML and trying to train an do this type of data, what conversations should you be having with your data scientist? How should you be working with them to make sure that you are taking into account all these edge cases and getting the right training data and getting the right answers. What do you do on a day-to-day basis?
Christina:
I think in some ways it starts in the same way that I think a lot of products should, right? Where you are making sure that there's a lot of clarity and agreement between you and your team about like what is the problem you're trying to solve? Why are you building this model? What do you think it will do for your customers or for your business? That kind of clarity. I mean I think it's helpful in all product work. I think it's especially helpful in machine learning and data science because the folks building the models are necessarily making some judgment calls as they go. I mean it's helpful for them to understand what you're trying to accomplish. So I think that's the first thing. The other things that you should be talking about with them that's just helpful for them to know as they're building the model is we kind of alluded to precision and recall, but that gets to the idea of your model's gonna generate false negatives and false positives.
How do you wanna tune it? Are you trying to avoid false negatives at all costs or like avoid false positives at all costs? Do you wanna hit somewhere in the middle? So like thinking about how you wanna tune it I think is a good one. That's something to discuss with them. Explain abilities is another one. So some kinds of models are notoriously hard to explain what's going on under the hood. Neural nets are a good example of that. Not really sure why neural nets work or like what exactly about them makes them so powerful. So like do you need to be able to explain the answer that you produce? That's a good thing for them to know. And then I think the last one kind of goes more to like infrastructure requirements, but how much latency can there be in the answer? How fast does the model need to run?
Where does it need to run? Cuz I think once you get into situations where it's okay, the model needs to run effectively in real time and we need to be able to serve an answer to our customers and a couple hundred milliseconds, that leads you to like a very different technical architecture than okay, like we can run this model like every couple minutes and the answers can take up to a couple minutes or hours. Those just like very different technical architectures. So those are the things I would think about upfront. And then we also, we talked a little bit at the beginning too about what did I feel like the difference was between doing machine learning product work as you go on a more ongoing basis versus sort of more traditional product work?
Melissa:
Yeah. What were those different areas that you look at? So obviously one you're working with the data scientists, that's new for a lot of people for sure. Also, here's a good question. I'm sure people are out there like what does a data scientist actually do? Because I'm sure people have worked with developers, but there's probably a lot of people out there who are like what do data scientists do on a day-to-day basis? How are they involved in the team? How do you bring them into a agile cadence or however you're gonna work with them?
Christina:
Yeah, so in a lot of companies there's two very distinct roles, right? So you have data scientists and machine learning engineers and sometimes those are like two different teams of people. And then in some cases, and this is what we did at TripAdvisor, we hired people that had sort of both skill sets and so sort of the life of a model is the first thing you're trying to do is assemble good training data. It turns out for a lot of teams this takes just an enormous amount of time and effort and heartache. We thought a lot about that at TripAdvisor. It is a lot of that thinking got translated into causal. So a lot of this sort of horribleness is why Jeff and I are now building causal. But anyway, you assembled training data so that you've got something to build the model on and then there is a bunch of work to just sort of build the model itself.
So they're trying new techniques, they're doing different runs with different weights, they're trying different kinds of algorithms, just sort of seeing like what the results come back as. And then at some point that sort of work is done and then every team is different. But this is a very common handoff point between like a data scientist and machine learning engineer model's done and now it needs to go run somewhere in production and there's a bunch of work involved there. So whatever training data the model relies on now that data needs to be assembled and processed and cleaned sort of regularly so the model can be retrained. So that's part of it. The model needs to be able to run in production. So sometimes that means like writing it into a different language or just deploying it, maybe it's still in Python and they're just deploying it somewhere and then actually like hooking it into the product. And so Jeff's team covered that end-to-end. That's not uncommon but it's also not uncommon for two different teams to handle those sort of two kinds of work.
Melissa:
Okay, interesting. So then you started talking about it as well, but what are the biggest differences that you found from being a product manager for some kind of normal UX focused type team with workflows and all that versus somebody who works on machine learning?
Christina:
Yeah, I thought the cadence was very different. So there is more downtime. I didn't feel like I have downtime, so let me think about how to put this, there's more time upfront where a data scientist is like experimenting and building the model and figuring out the data. And so there's just like a lot more exploratory time where you're figuring out what's gonna work before you transition into how's it gonna get deployed in production. So in some ways that mimics the kind of work you might be doing, like exploratory pro product work where it's customer research or working with design and figuring out what the UX is gonna be, but it's got a very tone feel cause you're sort of working with a data scientist rather than like a designer and a UX researcher. So that the cadence was different. That's the first thing. The second thing for me was returns to downtime or maybe a 20 time I guess is what Google calls it, but time for your data scientists and machine learning engineers to keep up with what's going on in the field and like learn about new techniques, write papers, go to conferences.
I felt like our team learned a ton from the community and brought it back to TripAdvisor in a way that was hugely valuable to us and to the company. So just like building that kind of slack time into the team's cadence is a good one. And then the last is more product focused but it's, you know, how are you gonna QA your model? So if you're Qing like a front end change, you're like clicking around and seeing if you can find edge cases and you can just look at the product in dev and figure out if it's working most of the time. But models are a little bit different. You can do a little bit of that but there's so many cases, there's so many ways it could work or not work. It's almost impossible to do that in a way that is like manual.
So think about like what kind of telemetry do you need in place? Like what do you need to be tracking? What data do you wanna have available to you? And then usually one of the things that I was doing when a model was newly deployed was frankly like running a bunch of queries, right? And being like okay, where are times I wanna know all the things we've recommended to a user. Make sure you're tracking that and what are some things that just never seem to get clicked on and like why? Or like what are search queries that just nobody ever succeeds when they query. That's like what's going on? So make sure you've got the telemetry in place that will help you find edge cases in production so you can go investigate them.
Melissa:
I think that's really important. And I also think people's heads are probably exploding right now when they heard you say downtime for the engineers to go to conferences and not be on a weekly sprint schedule where they have to deliver a full fledged algorithm every single week. So here's a question I'm sure people are looking at, does an agile cadence really work for machine learning when you're building out these algorithms and testing them at the beginning?
Christina:
So we didn't do agile. I think it would've been hard for us. Neither Jeff nor myself particularly like it, we did something that looked a lot more like conbon. I think it might work if you treat the exploratory part of model building as another, you know like they talk about like dual track agile where you treat the exploratory part of model building in the same way that you treat the exploratory customer research UX track and agile. But I think you need to recognize that there is real technical risk models might not work. They're gonna learn things as they go. It's hard to just like point a new model and have it fit neatly into a sprint. So if you do agile and you're really committed to that lifestyle, that's fine. You're just gonna have to think a little bit about how this different way of working is gonna fit into that.
Melissa:
What you just said I think is really important for people listening and going, well why can't you do this? Like with the sprint, like Agile's a way of working When we think about agile risk, and you just said something that really hit home for me on this. When we think about doing biweekly sprints and putting it out to the customer and checking those types of things, what we're really looking for is reducing the risk that it's not the thing that the customer wants and at a certain point of machine learning, the risk is there because it is can machine learning give the optimal answer that a customer wants? And we can test that, but we can test that without building a whole machine learning model. We can just say pretend there is a working machine learning. And I think a lot of people get started with this in machine learning when they're doing startups and stuff, but it's like show 'em an answer.
Did it work? Yay, real time. Cool. But then on the back end you're like okay, let's automate this and let's actually build a machine learning learning model that does this regularly. And then you've got that technical risk that you were talking about where it's not, is this what my customer expects? It is also like is this what a human expects but like is this correct? Is it giving the right answer? Is it like actually doing what it's supposed to be doing? And that's a different type of risk than just is this what a customer expects at the end of the day?
Christina:
For sure. Yeah and I, I think the idea of technical risk is maybe something that you just don't encounter quite as much. Cuz I mean for most of the non ML stuff I built a tripad advisor. Like we're not wondering if we could build a different login flow like we definitely could.
Melissa:
Yeah, exactly. You're like if if you know you could do it, just do it.
That makes a lot of sense. Okay, so when you are thinking about all of these ML technologies too, I'm curious to hear a lot of people we're talking about or robots gonna take over the world, is chat GPT going to replace everybody's jobs? You know, you're dealing with a lot of this, it calls a labs, you're making tools for people to do this too, right? What's your opinion on the state of where machine learning is going and this is gonna replace everybody's jobs today? Like how far are we with it? Like how good is this technology that's out there right now?
Christina:
Yeah, I'll say first to like we did a lot of natural language work at TripAdvisor and I, I'm really amazed by how far language models have come in the last five or six years. Robotics is like an an interesting like other case it turns out that hands are actually like very difficult to make. And so robotics is I think proceeding in a way that feels very different than maybe what we had all guessed. I don't know that anyone have guessed that it would be like easier to replace our brains than our hands, but it does kind of seem like that's how it's shaking out. So I don't know if robots are gonna come and kill us all. Like if they do, you should know future robots that like I'm a nice person and you should share me. But I think look Chad Chb t's magical and I really don't wanna like, it's incredible.
I don't wanna diminish anyone's joy in it. It does have some like really interesting limitations though it is absolutely untethered from reality. Sometimes it does not know how far away Paris and Toronto are. So it, how truthful is it as like an interesting thing? And I think that's gonna be actually like a really tough hurdle. I think if we've learned anything from 10 years of having newspapers fact checked, is this true or false is actually actually like a tough judgment call to make. The sort of broader question is what do I think about artificial general intelligence? I would put myself in the camp of like, I think it's probably coming. I would give myself even money on being alive when it does. So I think it's probably further off than we think.
Melissa:
It's interesting too that a lot of people don't understand that in GPT3 and chat GPT, they're trained on like what's just prevalent on the internet. So it's just as smart as the collective internet. Yes. However you wanna interpret that <laugh>. But it's not like it's only reading science books from people who've studied this research for years. It's actually digesting everything that's been written on the internet.
Christina:
<Laugh>. Oh yeah, cat videos, dumb message boards, like it's all in there,
Melissa:
Which is funny. So I, I feel like a lot of people need to remember that and you, you brought that up, but I thought that was a very good point. It doesn't know how far away Paris and Toronto are and it also will probably spit out a bunch of untrue facts that people wrote on the internet and acted like it's true. So all your robotic brains out there are pretty much just as smart as the stuff that you feed into it.
Christina:
That is true. But I, you know, if you're listening and you're very worried that we're all going to be murdered by ai, AI alignment is a field and I'm sure they're looking for smart people and you could get into it and you know, prevent our collective death.
Melissa:
So yes, please do that. Please do that. <Laugh>. Well thank you so much Christina for joining us today. If people wanna learn more about you, learn what you're doing at Causal, where could they go?
Christina:
So I'm on Twitter. I would say I am sporadically active on Twitter, but if you wanna chat with me, my dms are open. So I am at Seki, C E E P S K I. I'm gonna spell it again cause I think I got it wrong. C E E P S K I. It's what my brother calls me. I don't know. I picked it 10 years ago.
Melissa:
Yeah, that's why I have Lissie Jean. It's what my mom calls me <laugh>.
Christina:
I love it. And you can find me on LinkedIn. There aren't that many Christina Pawlikowski out there. I think I'll be easy to find there. Can check out what we're building at causal if you want more detail. And we are at causallabs.io.
Melissa:
Great. Well thank you so much for listening to this episode of the Product Thinking Podcast and if you have any questions for us, we are taking more questions for our next Dear Melissa episode, which is next week. So please go to dear melissa.com and tell me what you have questions about, related to any of this or anything in product management. And we will see you next time.