Insight ON The Product Manager Mindset: Why Prompting Isn't the Path to AI ROI

Audio transcript:

The Product Manager Mindset: Why Prompting Isn't the Path to AI ROI

Jillian Viner:

You actually advocate for employees to adopt a product manager mindset. Why is that important as a skill set to develop? Now

Melissa Valentine:

Product managers are really good at like, sort of systematically inventorying, like a set of opportunities. Like what's the most annoying thing in my job? What's the biggest piece of friction? The second thing that product managers are really good at is basically matching something about the technology feature to the opportunity that they discovered. Product managers know that you don't like stay in design mode. You like experiment 'cause that's like where you learn the best.

Jillian:

Welcome to Insight on, I'm your host Jillian Weiner, and if you're making technology decisions that impact people budgets and outcomes, you are in the right place. What does it take to get AI out of experimentation and into a place where you're actually seeing returns? It's a lot more than change management and we're gonna get into it today with our guest, Melissa Valentine. She's a professor at Stanford University and a senior fellow at the Stanford Institute for Human-Centered Artificial Intelligence and the co-author of National bestseller Flash Teams. Let's go. Welcome. It's really good to have you here.

Melissa:

Thank you.

Jillian:

You have an impressive resume to say the least. Thank you. Uh, you've earned a master's from New York University, a PhD from Harvard Business School. You're a tenured professor in the Department of Management, science and Engineering at Stanford University. A senior fellow at the Stanford Institute for Human-Centered Artificial Intelligence. And you're a co-author of a national bestseller, flash Games . So

Melissa:

When you put it that way, , I mean Bra does sound pretty good. Bra. Thank you.

Jillian:

. Um, but I have to say, I think the most important credential for our listeners to know about you is that you are not a performative thought leader.

Melissa:

Oh.

Jillian:

You're not giving strategic advice based on really shallow insights. You are a field researcher.

Melissa:

That's right. Yeah. Thanks for saying that. That's true. Yeah. Yeah. Sometimes I'll like start my talks like that. I'll be like, the usefulness of what I do is called grounded research and it means I go to like, right where they're building the ai and I'm like, is this working? How is this working? And it, it does seem like really useful to have like a non, I mean, I think like a lot of people have a lot of lala to offer right now, but like to have like a really like non hyped way of talking about what's happening with ai. Yes. Yes. Because I feel like organizational leaders have such a big job to transform so many things right now. So I feel like if they can really truly understand like what it takes, like what's hard about this, what does it actually take? Yeah. I feel like I, I feel like grounded research will give them, I mean honestly just like compassion for how hard it is, and then like confidence that like it being hard for you is 'cause it's hard. Yes. It's because this is actually a really difficult thing that we're all trying to do together.

Jillian:

I am one of countless leaders that are experiencing exactly that. Oh yeah. So I

Melissa:

Appreciate

Jillian:

It's real, that empathy. It's very real. It's very real. It's real. And we talk about change management very broadly, but like we're gonna get a little bit more tactical about that. And so I think that's a really great point. And you have been embedded in organizations, different types of organizations too. Gig economies, uh, enterprise marketing teams mm-hmm . Hospitals. Like you've kind of seen it all. Yeah. And like you said, you've really seen the actual impact once technology meets the organizational structure. Mm-hmm . Yeah. And so that we're not getting Melissa Valentine's hot take . We're getting your observations. It's true. Through research. Yeah. Totally. So that's important. The other thing I'm gonna hedge is that this is a conversation that every functional and business leader needs to hear and not one that every business is ready for. Hmm. So I'll float that out there and I'm gonna ask you first the question that I think many business leaders are a little bit tired and maybe frustrated with asking mm-hmm . Which is why are they not seeing the ROI from AI and don't give me the fluffy nonsense , like give me the non-obvious reason. If it's such a transformational technology, where's the

Melissa:

Proof? Yeah, yeah. No, I, that's, that's real. And um, in fact, I was, I was telling a group recently, um, like I think the experience that we're all having is like with our individual use of ai. So for example, somebody that I was interviewing at one of my companies was showing me this, this new, um, this new use case that she had come up with. And it was based on having like the split tab in Chrome. Have you seen that you can have like two chrome views open together and then like Gemini is sitting right next to it. Mm-hmm . So she had two different spreadsheets, open Gemini's right there. And then Gemini like consolidated the spreadsheets and came up with these insights and like it actually worked. So it was like two days ago she told me that and I was like, no way. And then I went home and tried it and I was like, oh, alright, well .

Melissa:

And I think we all have that experience that like, like from time to time it blows our minds mm-hmm . And we're like, this is transformative. This is gonna change everything. Mm-hmm . And then you also go to work and you're like, this is still also a very frustrating . Like, it's also like none of the systems work. Yeah. Logging in takes me three tries, , like nothing works. The internet's down. Hey, also like, where's the data? Hey, also the version control. You know what I mean? Yeah. So there's like the reality of organizational life and the sense that we all have of this like transformative technology. So I think one, so I think there's like a lot of different things that are needed, like a lot of innovation all around this technology that's needed to like, make it actually get to impact. Um, one bit of research that we did recently I think is, um, helpful to answer this question.

Melissa:

Um, and it has to do with like how people on the front line are adopting. And what we were seeing is that I think it's like pretty easy for people to get stuck in the prompt window. Mm-hmm . So if you're just like, you know, copying something into the prompt window or you're just like working right in the prompt window, you can get a lot of cool stuff that you can brainstorm while, like there's a lot that can happen there. But what we, what we like found in this research is that, um, the high impact use cases are when you can basically do an integrated workflow where you find a way so that you're not copying information into the prompt window mm-hmm . And then copying it out of the prompt window. 'cause you haven't like saved time there. Correct. So this is in a, this, this is in if, if people wanna like read more, this is in the Harvard Business Review article that just came out and we kind of go into depth of like what it takes to actually like, set up an integrated workflow.

Melissa:

Like how you like, figure out what's a really high value task. How do you match that to the right technology? So for example, you mentioned notebook lm, right? So like, are are you needing to use just Gemini? Do you need to create a gem where you have like a custom gem set up mm-hmm . Do you need notebook lm? So you have to like match the right kind of wrapper of the LLM. Yeah. Um, and then you have to figure out how to like integrate that flow of information. And like when you do that, then you actually have save time, which is

Jillian:

A harder task than it sounds. And we're gonna get into that. Yeah. Before we get that deep, the dominant narrative that we still hear around AI right now is this focus on efficiency and time savings. And you maybe see that individual level, maybe not, sometimes it's just a better output. Right. But that does seem to be the, the mission that organization organizations are going after. Is that efficiency now you're studying how AI is actually changing work. Do you buy that narrative? Like do you think that's the right metric for organizations to be focused on?

Melissa:

I mean, I think organizations care so much about efficiency and like it does feel like there's a lot of efficiencies to be had. So it seems like near term that seems like a fine thing to focus on. Mm-hmm . Um, I, there's this, um, great graphic, um, that I've seen that's called like the Turing trap. And like in this graphic, it basically shows tasks that humans used to do that machines can do. But then it shows like this big space of like tasks that we can now do that humans plus machines can now do. So that like the idea of the Turing trap is we're like stuck trying to like replicate humans. Um, that's what we're doing with AI right now. Some of the efficiency stuff is like just replacing human tasks and like why not? Like it's, I mean, some of the stuff that we're replacing is just like us moving information around and that's like, who cares if a machine does that for us?

Melissa:

Right? mm-hmm . Um, but I think maybe this is what you're getting at. It seems like the frontier, like it seems like we could push the questions of like, what could we solve? Like what problems could we actually solve once we get the efficiency stuff dialed in. Mm-hmm. Like, what's the frontier of like, what could we take on, what could we do now that we can't do before? And it does seem like the intelligence that is like available here with these tools plus what we as a human society are able to do. I mean, it does seem like we could do something much cooler. Yeah.

Jillian:

Yeah. That's a big mind shift though. Yeah. It's a really big mind shift. Yeah. At the organizational level. Even at the individual level.

Melissa:

Yeah. I mean, and I always have to like joke as an academic, it's really easy for me to like be like, you know what we should do because like business leaders actually have like, they have like business, they need to be efficient mm-hmm . You know what I mean? So it is decently hard to be doing your job. Mm-hmm . Transforming your job, thinking about the future of humanity . Yeah. Which is, which is why we have professors who don't have business metrics . So I'm like, what if we cured cancer guys? , what if we, what if we fix the healthcare system?

Jillian:

But you can get there faster if you actually know how to set up the teams and the instruments to make that happen. Exactly.

Melissa:

Yes. Yes. If your information systems are integrated, what can we do? Yeah, I think that, I mean, I do think it isn't, I do think we could all start to think like that.

Jillian:

So early on it felt like the secret to AI success was being like really good at prompting. Like having the magical prompt that gave you a good outcome. You have a very different thesis on this and you actually advocate for employees to adopt a product manager mindset. And I think that's a framing that people maybe haven't heard before. Mm-hmm . And I think that's what you're getting at too, about thinking differently. So explain that to me a bit more. What does that mean to have a product manager mindset and why is that important as a skill set to develop now?

Melissa:

Yeah. To, and I'm, I'm not like against prompting. Um, I will say like I have my own way of like interacting with the tools and they're very good. They're very responsive. So for example, I was interacting with, I don't remember which one, let's say Gemini, but I like asked it a question and it like way over answered the question and I just typed settle down . And then I came back and it was like, you're right. That was totally over the top . And just say like, you don't have to actually be like super, super careful with them. They're very powerful tools. Anyways. , is that what you're asking? No, I'm just kidding. Um, yeah, so, so this is based on, uh, research that I, um, have been doing with my PhD student, Amanda Pret. Um, and yeah, we've been doing, um, a lot of like interviews and observations.

Melissa:

Um, this is at Google and we started to see that there were, that there was like some set of people that were getting to like really high impact use cases. So, and since we had all these interviews and all these observations, we were able to kind of analyze like, what was it, what was it taking, like what did it take for them to actually get to like a really high impact use case. Mm-hmm . Um, and credit to Amanda, she's the one who like, after analyzing all of the data, she was like, oh, that's like product management. And once she said it, I was like, oh, I can't unsee that. That's true. Um, I don't know a lot about product management, but I learned about this from her. So basically product managers are really good at like, sort of systematically inventorying like a set of opportunities.

Melissa:

So for you in your own work, that would be the difference between like, here's Gemini, what can I do with this tool? Um, compared to being like, what's the hardest thing right now? Like, what's the most annoying thing in my job? Mm. What's the biggest piece of friction? What is a repetitive task I take? So you would look at your opportunities differently. The second thing that product managers are really good at is basically matching something about the technology feature to the opportunity that they discovered. So we have all these examples in the article that people can read, but it's like just, it's like not staying stuck on just Gemini, it's like recognizing you can create a gem or you can set up notebook, lm mm-hmm . And you'll wanna match kind of like what that user interface or that like data integration. You'll wanna match what that is good at with your problem.

Melissa:

So product managers are really good at good at that. Um, product managers know that you don't like stay in design mode. You like experiment 'cause that's like where you learn the best. So we saw, um, that was like really an, a part of people getting to good use cases. Um, and then the last one is the integration is like really having an eye towards mi doing a lot of copy paste. Is this a workflow where the information just kind of goes smoothly? I can, I can give you an example of that. I see. Yeah. Yeah. So, so one of the, um, people that we interviewed was a manager who spent a lot of time kind of like synthesizing her team updates and then sending them to her executives. Um, what she did is she created a custom gem, and this is the part that's important is she had her team upload their updates to the gem and then the gem would send her the synthesize update. So instead of you see that she like knocked out one whole piece of the workflow.

Jillian:

Yeah. She cut out like five steps and put it into one piece.

Melissa:

Yeah. And that's, that was not technology. That was just like kind of her thinking in terms of integration Yeah. Workflow integration. Yeah.

Jillian:

I'm thinking of like that cheesy nineties commercials of like, there's gotta be a better way. Yeah.

Melissa:

,

Jillian:

And that's really what you're saying Yeah. Is like, don't ask what can this technology do? Examine your workday and be like, what is the thing that is annoying? It's it's complicated. Repetitive. It's repetitive. It's eating

Melissa:

Time. It's no value here. Yeah. Yeah.

Jillian:

Yeah. It's a lot of those tasks that even if you had an intern, you wouldn't pun to the intern because it di there's still some nuance there. Yeah. Like you have to filter it through. And so thinking about breaking down, what is that problem statement? Mm-hmm . Mm-hmm . And then tra like looking at the technologies. I think this is the other advantage of being exposed to multiple LLMs, which not everybody has the luxury of. Yeah. Here at Insight we have the luxury of testing out. We have, you know, Google, Gemini and copilot. Like we have it all. Yeah, yeah,

Melissa:

Yeah. Because

Jillian:

You do find out what platform or what l LM is better for what type of task. But then the other thing that you said that I really love is you kind of build like your MVP Yeah. You start somewhere, it may not be perfect at go, but it gives you a foundation to build from.

Melissa:

Yeah. And I think it, yeah. It really gives people a sense of like, it gives them like a good mental model of like what's possible.

Jillian:

Yeah. Yeah.

Melissa:

So like hands-on tinkering, like what's possible. And it does feel like there's just like such an unlock the first time you see it take on a, like a big chunky task for you. Like you see what's possible, you're like, oh. And then you like learn how to tinker, like in a way that you wouldn't know how to pre-design it. Yeah.

Jillian:

I'm curious, have you had to deal with this or observed this where you have folks in the org that like, okay, that sounds great, but it's gonna take me, you know, five days, 10 hours to just build the thing. I could just do the task.

Melissa:

Yeah.

Jillian:

, how do you help those people get over that initial, like, yeah, you gotta build the thing, it's gonna take time, but then ultimately it's gonna save you time. Like how do you convince them that that's worth it?

Melissa:

I mean that's, that is a real dilemma. I mean, it is true. Like the, the first, the time that you take to like step out and like reconfigure your workflow, you could do the task probably. That's true. . Yeah, probably that's true. Um, I think what I've seen is, I mean, a couple of thoughts that I have, um, I think that there's a real value to like demoing and modeling mm-hmm . So like within a team, um, like a manager can have 15 minutes at the beginning of every meeting and have somebody show their workflow and be like, this is what I tried. This is the tool that didn't work. Here's what it took to make it work. And I think if you really demo it and you actually like show the tool working and you show the output mm-hmm . I think that like lowers the threshold for people

Jillian:

Get out of PowerPoints, just go do it and show Yeah, that's right. Show, show the message work.

Melissa:

That's right. That's right. Yeah. Just, uh, scream. Share. Yeah. Yeah.

Jillian:

What about more of the psychological concerns there? Because they may be using the time commitment as an excuse, but the rest of the story is, it's, it's kind of a value trade off. Mm-hmm . Like, this is something that my viewpoints, my taste, you know, I've had to qualify whether or not this is good to ship or like it requires my overview. And you're kind of like saying that no, like the AI can kind of take this piece. Like how do you mm-hmm . Overcome that?

Melissa:

Yeah. I mean, I think that's real. I appreciate that you're kind of like slowing down that moment and kind of like reminding us that's what people are experiencing oftentimes with these tools. Um, and some people just kind of automatically love to tinker, so they're gonna jump, jump in and just like tinker, tinker, tinker. And we can learn a lot from it. But the workforce also, envi involves people that you're describing. Right. And they have a lot of pride in what they're doing. Mm-hmm. And they have ex enjoy doing the tasks that they've been doing for a long time, the way that they've been doing it. Um, I think I don't have like a strong, like I don't have a strong like thou shell on it, . I think like what I've seen is, um, like when people have the opportunity to try for themselves mm-hmm . Then they can, they can kind of like pick which part of it feels valuable to them. Hmm. Um, I mean I think it's also just like a moment of like, so much change right now. Yeah. And I, you know, it's like in some ways like we can't really like save ourselves or like save the workforce from how much change there is. It feels useful to just kind of be transparent as you are. And to me it feels useful for leaders to just kind of name it. Like, we're going through a lot of change right now. Mm-hmm . Talk about it. What's it, what's it like for you? What's going on here?

Jillian:

Yeah. And by the way, this isn't the first time in history. We've gone through something quite this dramatic . I mean, I know you have examples of like the steam engine, but even not going that far back, there was computers

Melissa:

Yeah. And some people the internet Totally.

Jillian:

Yeah. Intensive training for computers, others didn't. Yes. Yes, yes,

Melissa:

Yes. One of my favorite books, um, is called, uh, in the Age of the Smart Machine and I reread it recently and it's, um, when factories become digitized mm-hmm . And it is that same moment, like people used to be very hands-on with the machines and the equipment and the hardware, and then now they're like at dashboards and they're having to read dashboards instead of touching the stuff. Yeah. And like similar for us, like we used to create the first draft of everything and now we're evaluating something that AI created the first time. It's a change. It is a big change. Yeah. Yeah.

Jillian:

So if companies accept, or, or leaders can start kind of pushing out this mental model of being more of a product manager and addressing your problems, that's still very much I ai at the individual level. Mm-hmm

Melissa:

. That's right.

Jillian:

So it's still kind of like that lower scale that you're saying is you're not gonna at the enterprise level experience that what are sort of the levels of AI maturity that, that are out there? And where do you think most organizations are right now from your observations?

Melissa:

Yeah, I think my sense is that this, these moments of like frontline experimentation and like innovation and transformation, I do think that those are super valuable. Um, because it does like, invite everyone to get involved. Um, I think there's like so much learning and kind of hands-on educating that happens for people when they're like hands-on with the tools. So I think there's a way in which I hope that companies don't skip that. I hope that they do have a moment where they're inviting everybody into the experimentation, um, to the transformation of frontline work. It just, it feels like being part of the process I think is like, useful for people. Um, and, um, there are also different kinds of use cases that you're gonna set up where you're getting different kinds of scale where you like manage it differently and you, you scale it really differently.

Melissa:

Um, so for example, a lot of, um, companies are doing that with like their help pages or like their, their, they're like elevated chatbots, right? Where like you get all of the documentation and then you have like an agent that is like making, like drawing on that documentation interact with your customers. Like that's pretty common at this point. Um, and that does change the role of like customer service people and how they're interacting with agents, um, or interacting with customers. So, um, we have this other article, um, where we kind of analyze like what it takes, like what's the change management, what's the kind of digital governance, what's the workforce change that happens to set something like that up. Mm-hmm . Um, I do think that more and more use cases are gonna look like that, not like more and more companies are going to have domains where they resource something at the like company level versus the like frontline transformation. So when you set something like that up, it does, it takes like a lot of data work. You have to get your data ready, you have to set up this like new digital governance. You have to like train people differently and it fundamentally changes the jobs that business people are doing. Mm-hmm . So somebody maybe used to be like creating all the marketing content and now they're like evaluating what an agent recommends is the content. And that's, it's really different. It's like a really different job. It's scary.

Melissa:

Same more. Yeah.

Jillian:

.

Melissa:

Yeah. So you would not enjoy that ?

Jillian:

Well it's, um, again, this isn't necessarily new in history, but it's new to us because you are, you're changing fundamentally what people do and the purpose that they serve in an organization. Yeah. And we're doing it in real time. So you're kind as a, as a leader doing that change management, you are trying to navigate people who are working in the system of yesterday while trying to move them along into the system of tomorrow. Yeah. And there's so many things that are happening simultaneously. It's, it's the actual skills that need to be learned. Mm-hmm . It's the shifting of how do I even think about the work that I do and the value that I bring to the organization mm-hmm . And then just how do I even reconcile a new way of working with the way that our systems are still working. I mean Yeah. I'm sure you've seen a lot of clashes

Melissa:

That's right in

Jillian:

Yeah. The organizational structure of trying to make this work.

Melissa:

Yeah. It's really true. And I like that you're sort of like calling out the different aspects of change that people are facing. Um, and one that I, I think I'm hearing you say is like, we have like identity and like meaning around the way we used to do stuff mm-hmm . So Yeah. Maybe I got into marketing because I like to creatively create marketing copy . You know what I mean? Yes. And that like, now all of a sudden you're sending me a report of like what the agent said to 10 other marketers and I have to be like, good answer, bad answer. Good answer. Bad answer. . And I'm like, is this my job now? . Right. And maybe for some people that's fantastic and maybe that frees up time for other stuff. Um, and it is a big change mm-hmm . And maybe not everybody wants it. Maybe people didn't pick it.

Melissa:

Yeah. Yeah. So like what's the leadership moment there? I think it's, I think with so much, with so much change, I think leaders have a lot to offer in terms of transparency and like offering kind of like a container for people around what's happening. Mm-hmm . And a, a container, like I'm thinking of this is I have kids and with a container, like you sort of tell somebody about, you're like, here's who we are, here's what's happening and here's why we're gonna be okay. And it's like, it's useful to actually just like call that out right now Yeah. For people.

Jillian:

Yeah. 'cause that is the, the sort of the philosophical element of all this mm-hmm . Mm-hmm

Melissa:

.

Jillian:

The tactical is also, I mean, you, you need both. Right? Totally. So I wanna press you a little bit more on, on the tactical. Yeah. And sort of like the, the level of AI integration we get, uh, you know, people coming up with use cases to help their individual work. Maybe they find ways to automate systems within their small scope. Mm-hmm . What's the next level to that? Mm-hmm . Maybe not like, 'cause I think you mentioned one that's like the large scale organization they're gonna invest in, but there's probably a imagine like a middle level in there. Yeah. What does that look like?

Melissa:

, I was just talking about this with, um, with a group. Yeah. It's in some ways you're framing the current research frontier. So, um, like we have frontline transformation happening. What I'm seeing in the companies that I'm studying is, is it, it is kind of like rolling up to the team level at this point. Like we are seeing team changes mm-hmm . So I can say something about that in a moment. And then with these enterprise use cases, maybe you have agents doing this, like one set of tasks, right? Like maybe all of the financial transactions in this one system mm-hmm . Now can be done. Right. And so like that's a big change there. But that's also pretty local. So we have like, kind of local transformation happening. We have these like local adjunct use cases set up. And I think what you're saying, and I think rightly is, you know, we have this whole organization, there's so much in the middle, there's so much coordination across all of this stuff. Like what happens there mm-hmm . Right. And so I'll be back next year to

Jillian:

Tell you

Melissa:

because

Jillian:

We'll figure

Melissa:

That out. Yeah. I think it's, I think, I mean truly like, I think that's what I'm seeing in companies is that I haven't seen that happen yet. I haven't seen that, I don't wanna say solved 'cause it's not like inevitable that it needs to get solved. But I just, I haven't seen people building there or changing there or transforming there. It's like hard enough to get these like local agentic cases with all the data work. Mm-hmm . It's hard enough to get people to un to like, it's hard enough to transform the front line. So all this like middle complexity, I think that's coming, but I, yeah. I don't think it's here yet. Yeah.

Jillian:

Let me make this even messier. Okay. . Okay. Um, 'cause I wanna talk about the human and the structural cost of all this change. Okay. AI is expanding what individuals are capable of. Mm-hmm . Which is I think the most exciting thing about ai mm-hmm . Mm-hmm . Um, it's, and in many ways it's blurring traditional roles. Mm-hmm . I think you have a very strong opinion about role clarity and how important that is. It's important. But that is getting blurry uhhuh. Um, you've got non-technical teammates who are vibe coding. You've got people who wouldn't call themselves writers who are making like production ready copy, arguable, . Um, and now you're gonna have agents enter the workforce and you've got people managing hybrid teams of humans agents or maybe agents only. So Melissa, are we watching the org chart Die ? And should we mourn it? ?

Melissa:

That's so funny. Um, , I don't know if you've, so I have, um, a joke that I start most of my talks with where I'll be like, I'll be like, I'm a senior fellow at Stanford AI and I wanna tell you about the most important invention of the last 200 years. And then I'll be like the org chart and like, it sounds like I'm gonna say ai, but I say the org chart. So it is funny that you're asking, 'cause I'm such a fan of the org chart. In fact I love the org chart and I don't think it's going anywhere. I do think it's changing quite a lot. Um, I'll I'll just tell you my hot take and hopefully everyone can like write in and argue with me. ,

Jillian:

. Let's do it. Let's make it spicy. I know.

Melissa:

It's, it's, here's the tea. I think like what I, so to me the org chart is like ultimately about coordination mm-hmm . And it is a structure that we invented as a human society. 'cause it has these nice properties that allow us to coordinate in certain ways. Um, we do really well with hierarchy. We as human people do really well with hierarchy. Um, and I don't mean like the social kind where people are like, do you know it's not about like status ranking mm-hmm . Stuff like that mm-hmm . Um, but centralization of decision making. Mm-hmm . Is actually extremely useful. , it's actually extremely useful. So as an information processing structure, the org chart is elegant and there's like a lot of kind of like mathematical and economic theory behind centralization of decision making and what that allows groups to do. Um, so what I would love to see is for us to be able to recognize the usefulness of that kind of decision making structure, centralized information processing, centralized decision making.

Melissa:

When is that useful mm-hmm . And then how do we take that and augment it with all of these data tools. Um, so I have a paper that's called, um, the algorithm and the org chart. I dunno if you saw that . I did. Yeah. That's, yeah, that's part of my, so in that paper I do talk about like what is useful about the org chart and the way that it will conflict with data products. So you, I mean, you do have to be really thoughtful about org charts are good for people and people brains. There's like a lot of economic theory on our bounded rationality, on our information processing, difficulties on collective decision making. There's like a lot that goes on there in groups making decisions together mm-hmm . That the org chart is really helpful for. And then also, um, all of these data tools are so powerful and your data tools should not be constrained by your org chart. Right. Your org chart is for your people. , it helps with accountability, it helps with governance, it helps with so many things. It helps with relationships, it helps with coordination. And then your data products need to be governed differently. So I think that's like, I think that's where we like run into trouble is if you're trying to govern your data products using a decision tree or like an org chart that's a mistake. Should do separate those out

Jillian:

Agents. Yeah. Where do agents fit in that then? Are they part of the human org chart? Because in some cases agents will be making decisions mm-hmm . And someone has to be accountable for those decisions, particularly when those decisions are the wrong ones.

Melissa:

I'm gonna, I'm gonna give you another hot take. Okay. And

Jillian:

I don't, I told them at the top we weren't getting your hot takes. We were getting your observable research . It's

Melissa:

True. We've like ventured into the world. Like Yeah. My hot takes, I think, um, one of my current, um, my current, actually, I can't even say it that strongly. I'll just say I always like side eye when I see like a one-to-one replacement of a human with an AI agent and an org chart, I always side eye that. And I'm like, that's not how that's gonna work

Jillian:

At all. . I think that's music to a lot of people's ears. . Yeah.

Melissa:

So it's not a hot take. It's

Jillian:

Not a hot well preaching to the choir. Maybe not a hot take, but a comforting one because I think that is a, a, a real fear that people have. Like

Melissa:

The one to one-To-one.

Jillian:

Yeah. Yeah. Like, an agent is going to take over my job or I'm just going to be working with a team of agents and mm-hmm . There's podcasts about that and it is alarming. . Oh, .

Melissa:

Yeah. I think like, the thing that seems useful to me is to like look at like human information processing and then look at agent information processing and just recognize that they're fundamentally different phenomenon. So you should not govern them the same way. Do not just like plop. Like it's not a one-to-one substitution mm-hmm . Even for an agent. It's silly, honestly, which is not me saying that we shouldn't use agents. Yeah. We just structure them differently. 'cause they process information so differently than us. Yeah. Yeah. The org chart. Like, there's all these like Nobel prizes about why we have org charts, why? And it has everything to do with like bounded rationality, collective decision making, like human limitations. Okay. That's what org charts are for. And agents have other limitations, but they don't have the same limitations as us. So if you, like, if you were to set up specialization and hierarchy for your agents, I mean, I have, I've, I've seen that that's the way you have like orchestrator agents. I think it's just useful to, to recognize like, what is this structure for mm-hmm . Like what are you, like what problem are you solving? What is this structure for?

Jillian:

So embrace that AI can help your employees do more and variety of work. And even as employees, we should embrace AI to expand our own capabilities, but respect that the org chart provides a navigational map for work. We know where decision needs to get made. We know that once we do something, it goes to this person. It helps the flow of information go.

Melissa:

That's right. Yeah. Yeah. I think, yeah, it's like recognize what the structure is in service of mm-hmm . And the org chart is in service of humans. And you can use specialization and hierarchy for agents for sure. But like, you would just be asking questions like, um, so like I, one, I think there's like a lot of questions as to like how we visualize agents in relationship with humans going forward. And like, I think agents can take on a lot of tasks, for example mm-hmm . They can take on a lot of tasks and then oftentimes you'll need to have like a human in the loop to evaluate those tasks. Like what's the right way to visualize that? I don't know that the right way to visualize that is like in an org chart. I don't know that that's the right way to visualize that. That's the right way to like think about that.

Melissa:

Yeah. I think that's mostly what I'm saying is just like recognize what is the actual relationship between the human and this like, team of agents. Yeah. And if it's evaluation, like do you structure that? Do you visualize that hierarchically, you know, like what does it do if you visualize it hierarchically? What do you do if you visualize it as a loop? Like it's like a UI question. It's a human computer interaction question. Like how does it, what does the visualization kind of signal to the user in terms of their right behavior if all you're doing is evaluating

Jillian:

For a lot of organizations, I think even the Asian conversation still feels like sci-fi Oh yeah. ,

Melissa:

Like,

Jillian:

Not to go too deep in this, but like, do you have a, a use case that you've seen of an agentic workforce that's actually,

Melissa:

That's actually working, functioning

Jillian:

And working? Yeah. Mm-hmm .

Melissa:

Yeah. So like when I was give, when I was doing research and kind of giving talks last year, I was, I sort of felt like agents were overhyped. Um, this year I am seeing like systems that are actually working. So you can sort of imagine like in a financial services firm, um, like maybe you could record all of the like, um, transactions or something like that and that could be enough of a data set. And it's like a small enough set. It's like a small enough task that you could actually train agents to basically kind of troubleshoot all of those transactions. It's like a closed enough data set, it's a closed enough problem that you, I they, in that case you can get to like 99% agentic mm-hmm . Um, and then the problem is that like the 1% of exceptions are really complex and that tends to be when companies have set up something like that where something agentic is like 99% agentic, then the 1% of like exceptional use cases do require like a human to kind of get in there and be like, what went wrong? And it's really complex. And then sometimes the human doesn't have enough com have enough context to like really figure out what happened. Got it. So does that make it less sci-fi ?

Jillian:

Um, you don't necessarily have research around this, but it's something that I really wanna get your take on because I think it's something that is not getting enough attention right now mm-hmm . And probably will be, AI lets us do work in hours or minutes that used to take us days if not weeks. Mm-hmm . And in the moment there's a real dopamine hit when you can do that, you're like powering through these tasks that are like these big things and, and it's exciting that you like, solve the problem that quickly, but like, we're compressing deep focus work in these short sprints. And when you're stacking those cognitively demanding sessions back to back, like there is a cognitive load that happens. And I wondered if you've observed this in some of your research, if you're feeling it as a researcher, if you go through those bouts, is it something that we are paying enough attention to, is something that organizations are paying enough attention to?

Melissa:

Yeah, I've, I've seen, I I've really seen what you're describing. I do think it is a future of work design challenge or problem we're gonna have to figure out. Um, yeah. 'cause our, our work days, like the scope of our roles were created when our minds doing everything. And now, like a lot of that like hard focus work is like automatically done as you say. And all we're doing is like evaluating mm-hmm . But if you're evaluating something that is that complex and like, that's like, you're just constantly in that mode. Mm-hmm . Um, will you burn out? Is it exhausting? Will you still think well Right. Probably not. Like, probably we'll have to, probably that will require a big change in job design and role design. Um, I've heard people bring that problem up. I have not yet seen solutions. So that's a call to listeners if anybody has like, examples

Jillian:

Work week

Melissa:

that's for

Jillian:

Thumb. Put that up there. That's

Melissa:

Right. A four hour

Jillian:

Work week. Four hour work week. Yeah. Yeah. Wasn't that utopia that we were promised? That's right. . Um, perfect. We opened the conversation by challenging the narrative that AI is going to help companies just reduce cost, re re like increase efficiency, save time. You have really helped draw a bigger picture here that putting AI in the organization is really about role design, rethinking the organizational structure. How do you change the mind of leaders, particularly like A CFO who is focused on like the bottom line narrative? Hmm. Why is it an important distinction to make that this isn't just about cost savings or time savings? Like this is truly an organizational transformation.

Melissa:

Yeah. I think for the C FFO is like an interesting one. I think. Um, I think like I would think of all of this as kind of like a yes. And like the CFO is doing the right thing for that role for right now. Like, we really want them on what they're doing. We really want them on cost savings. Good job. CFO, , , like keep after it . And then also, um, I think it's, um, I mean I, you can hear like, I kind of keep returning back to this. I think it's just like inviting people along on the journey. So I think what I would wanna do with the CFO is I would wanna like spend 30 minutes and be like, what's the most annoying part of your job? And then have you ever tried AI for that? Mm. And then like, have them actually try it and have them like, think about it because I think the more hands-on people are, the more that they have like a really informed mental model of like what it could do for them, what they don't like about it.

Melissa:

So yeah, I think that's what I would do. And then I would kind of like just trust the CFO, like once they have that kind of like, hands-on experience with it to tell us like, how is this tool gonna be great for your role? Like, how can you do what you're doing so well for us? Like, how is this gonna help you do what you're doing so well first even better. And then once you've got that tool helping you do that, what else is possible? Like maybe you wanted to like, uh, I'm just, I don't, I'm not a CFO . not I'm gonna like, embarrass myself, but like, do you wanna like audit 20 years of, you know what I mean? Like what are the analyses you always wanted to do that you couldn't do before? Mm-hmm . Like, what, what, what's the unlock here? Like, what's possible? Yeah.

Jillian:

Melissa, thank you so much. I feel like we've barely scratched the surface. You've done such incredible research around this. We will put your articles and a link to your book in the show notes. Cool. Thanks. Um, thank you for joining us.

Melissa:

Yeah. Thank you. This is really fun.

Speaker 3:

Thanks for listening to this episode of Insight on If today's conversation sparked an idea or raised the challenge you're facing, head to insight.com. You'll find the resources, case studies, and real world solutions to help you lead with clarity. If you found this episode to be helpful, be sure to follow insight on, leave a review and share it with a colleague. It's how we grow the conversation and help more leaders make better tech decisions. Discover more@insight.com. The views and opinions expressed in this podcast are of those of the hosts and the guests, and do not necessarily reflect on the official policy or position of insight or its affiliates. This content is for informational purposes only, should not be considered as professional or legal advice.