Defining What Good Looks Like

00:00:03: Hi folks, this is All

00:00:05: Things

00:00:06: Product with Petra Wille

00:00:07: and Theresa Corks.

00:00:09: And we're so happy you are here!

00:00:17: Teresa... You wrote a piece on AI Evols few weeks ago?

00:00:25: I saw there's part in it which i found particularly interesting Which was the part about what good looks like.

00:00:34: Can we maybe double click a bit of that Because I know AI Avalds, we recorded an episode on that.

00:00:41: It's the catchy title but i thought The particularly valuable part is a part of what good looks like.

00:00:48: Yeah...I think this was also the part that hasn't changed.

00:00:50: So for product teams when were trying to decide What To Build We're defining a feature even if we fall back to traditional rules.

00:01:03: If the product manager's writing a Product Requirements document, if the team is doing cross-functional discovery in their story mapping and trying to figure out like what it that were building?

00:01:13: A lot of work is What Does Good Look

00:01:15: Like?,

00:01:16: What Is The Good Version Of This Feature Looks

00:01:19: Like?,

00:01:19: And What Does Success Look Like?

00:01:20: Maybe this really just outcomes in disguise?

00:01:23: how do know our feature will be successful?

00:01:28: I think with AI evals, it's really easy to focus on.

00:01:30: like this is a new skill and absolutely as the new skill.

00:01:33: It's how we measure um The quality of non-deterministic outputs.

00:01:38: But i think that heart of activity Like what is good?

00:01:42: And How do you put definition around?

00:01:44: Good isn't New.

00:01:46: so When i wrote my article some people pushed back and were like Evals measured technical performance.

00:01:57: This is actually really analogous to me, To the mistakes we made when we first started instrumenting our products.

00:02:03: So that very first analytics We put in our product for things like time yeah speed performance Like.

00:02:09: of course these things matter But they're not sufficient right?

00:02:13: Like we need to do more and measure customer behavior Are we getting a value creation moment When you define what good looks like the customer did something that created value for the customer.

00:02:26: When we're talking about analytics, where measuring Did That Happen?

00:02:30: Do We Create Value For The Customer?

00:02:32: And hopefully our knowledge of the customer and what they need is influencing that definition of that value creation moment in those

00:02:41: metrics.".

00:02:42: I think with evals like it's a same thing.

00:02:45: sure i can just write some evals With no Knowledge Of The Customer whatsoever.

00:02:51: Was this answer helpful?

00:02:52: And I'm going to make my own judgment of whether it was helpful or not.

00:02:55: But ultimately, the customer should decide if it's helpful and so the product team's job is to understand the customer well enough to define a definition of helpful in measurement-of-helpful that defines what good looks like and hopefully what good look likes from point view with customers.

00:03:13: If you find it wildly fascinating how In this transformation, we are repeating the same pattern that we saw in past transformations again.

00:03:27: So it is very fruitful these days to have these conversations with people who went through various of the mobile transformation and user-generated content transformations or several of these transformations because folks like you find far easier to spot these like, yeah I like what you're currently doing with this evils but you can go one step further and i would advise you do so because ive already been there in the past when we made the exact same.

00:03:56: I would not call it mistake!

00:03:58: It's a bit of an onion model off.

00:04:01: first of all if u do easy things then obvious thing is really helpful.

00:04:07: Inspects are kindof style which is fine So i wouldn't blame people.

00:04:13: fifth IT transformation.

00:04:16: But yeah, I like to see that things are happening Again, and that coming back to the basics.

00:04:22: And coming back towards what has worked as mental models in the last I don't know twenty years is really so helpful.

00:04:29: So That's why like your framing In particular about this What Really Is The Measurable Impact That Our Features And Product Is Having?

00:04:36: How Is It Contributing To The Bigger Company Strategy?

00:04:40: And To The Business Success?

00:04:41: Ideally So i Really Like That Part.

00:04:44: Yeah!

00:04:48: It's easy to get distracted by like, oh there is new language.

00:04:51: There are terms I don't understand and a new methodology or tactic And we glom onto the shiny and new and maybe hard and complex.

00:05:02: We're looking for what's the simplest version that can be understood.

00:05:07: Maybe because this started with machine learning engineers and it felt very engineering topic.

00:05:15: it's natural that we made this mistake of like, oh let's just measure technology performance.

00:05:23: But I think this is the argument for a cross-functional approach because of course the engineers are going to think about from an technology standpoint

00:05:32: and nobody can blame them for that.

00:05:34: by the way No!

00:05:35: That's their job right?

00:05:36: And so hopefully We have The business mindset represented in the user mindset represented When you're building an AI product, just like any feature.

00:05:48: Let's drop the AI.

00:05:51: We have to look at is it creating value for business or customer?

00:05:56: Are those things aligned and we can put metrics in place that measure both of them.

00:06:02: For example with Feature Adoption we measured activation metrics which usually, by the way is a value creation metric for the customer.

00:06:11: So do we create value for the customers and then measure things like retention?

00:06:14: Which is did you create value from

00:06:15: business?".

00:06:17: And I think with AI products... We don't always know.

00:06:21: maybe if they have enough experience yet to say your activation metrics should look this and your retention metrics should be like that.

00:06:29: But if you're building an AI product and don't know what customer value is, at least have a theory of how that's going to create value for your business.

00:06:37: That's the problem!

00:06:40: If we think about evals as just metrics or instrumenting our products... To me it becomes really obvious that we should not just instrument technology performance but also tech-instrument customer values and business values.

00:06:54: And the eval suite should cover all those things.

00:06:57: Agreed and that might be hard in the beginning because when I think back to the times, When i was working for this big social network here in Germany which is called thing.

00:07:09: And engagements came up so people were liking stuff for the first time we had new things to measure Because engagement wasn't a thing prior period, right?

00:07:21: And I think there will be similar things happening with AI.

00:07:23: There will be new metrics coming up that we now able to measure or the better metrics to keep track of in times off AIs.

00:07:32: so i think all of them happen.

00:07:34: but this would not change the fundamentals for what you just said.

00:07:39: it's about user value and business values if those two have enough overlap.

00:07:48: Yeah, actually engagement thinking about the metric engagement is a great analogy for some of this stuff.

00:07:53: so For people that have ever tried to measure Engagement usually what happens?

00:07:58: Is we start with our really crude metric like monthly active users and then maybe as We get better instrumented products.

00:08:05: We might look at Like daily active users And then People came up with this idea Of a sticky product and they measured Daily Active Users divided by Monthly active users.

00:08:14: So already we see like a metric evolving.

00:08:17: But then the really good teams realize not all engagement is equal and we actually should measure highly valuable engagement.

00:08:27: So if I am on Facebook, And i just scroll all day maybe that's less valuable than If I post.

00:08:35: yeah...I mean they have an ad business so They want you to scroll All Day.

00:08:38: but humor me in this example right?

00:08:40: For a lot of products or Actually here's A better Example.

00:08:44: let's take something like DocuSign.

00:08:46: You receive a document to sign and you're engaged.

00:08:50: Do you sign it?

00:08:51: That's good, but better engagement is that encourages you To now send your own document for signing via.

00:08:59: so we So you could imagine for that team They might measure engagement as both actions any behavior in the product And then eventually they realize like their real activation metric Is getting you just send a document.

00:09:13: So now we started with these universal metrics monthly active usage.

00:09:18: We evolved them to be a little better still.

00:09:20: Universal daily divided by Monthly is still something anybody could track.

00:09:25: and then eventually.

00:09:26: We realized the best metrics

00:09:28: are our business.

00:09:30: To the value creation moment of our Business,

00:09:33: yeah And

00:09:33: I think were seeing this same thing with evals.

00:09:35: so there's a whole bunch of evil vendors that will provide off-the-shelf evals.

00:09:40: They'll just come up with these generic definitions.

00:09:43: concise, the AI was concise in its answer.

00:09:47: The I was helpful and it's answer there was friendly In its answer.

00:09:51: And help is my new NPS by the way.

00:09:54: Yeah, yeah

00:09:55: i was helpful in the end.

00:09:56: its answer It's my new nps thing.

00:09:58: so what's hard about this?

00:10:00: Is like let's just take conciseness.

00:10:03: If im a student This is an example used to my article if Im a student and I'm at a physics class and I am starting To learn astrophysics for first time Don't want a concise answer.

00:10:13: I want an answer that like walks me through it and teaches Me something.

00:10:17: yeah, if i'm an astrophysics professor?

00:10:21: I don't want teaching example!

00:10:23: I want a precise answer right.

00:10:25: so even conciseness the definition of what good looks Like changes on the context which we're going to

00:10:36: contextualization and hyperpartisanalization.

00:10:39: in times of AI

00:10:43: We're going through the same learning curve that we went with regular analytics.

00:10:48: And so I guess my key thing here is like, we've always had to define what good looks like.

00:10:53: when you talk about AI products don't outsource what good look like.

00:10:56: really think in your context of how does it look?

00:11:01: Yeah So In My Context It's Not That It Has To Do A Lot With AI Evils But I often talk about what good looks like as well because in my leadership coaching that has a lot to do with role definitions.

00:11:15: So, What is the Good Product Manager?

00:11:17: What Is The Good Product Leadership Person?

00:11:19: But Product Leader or What Is A Good Product Organization Looks Like.

00:11:23: so It's Smooth.

00:11:24: This Is Deficient Low Friction Low Conflict.

00:11:28: We Talked About Conflicted Just Recently.

00:11:30: Maybe That's A Good Time To Link To That Episode But it's a very familiar concept to me.

00:11:37: To always think about, okay what is the smallest common then?

00:11:43: nominator of goodness?

00:11:46: Yeah across-the-board when it comes to roles... When it comes processes.. When it come to organizations and products & features & their success.

00:11:55: I think its'a very familiar pattern for both you&I in our work right?

00:12:00: I think it's even more important now because there is so much punishment if you use AI on automized stuff and look at pattern, add an organization that is not in a good shape.

00:12:15: And for product that isn't in the good shape or attributed well but people don't have clear idea of where we are headed then why?

00:12:23: People do not know what user group we're serving etc... it's more important than ever to get your stuff in order, I'd say on all levels organization level people level product level.

00:12:38: Yeah you know what's interesting is that like What's In the back of my head Is this like intertwined concept Of Like This Just The Same As Setting Outcomes?

00:12:47: It's The Same as Saying Being Explicit About Our Expectations So That We Can See When We Get It Wrong.

00:12:56: so we teach Start With An Outcome.

00:12:59: What are you trying to achieve?

00:13:00: It's hard to achieve something if you don't set out to achieve it, right.

00:13:04: True!

00:13:05: Um...it is the same thing like we define what a good product manager looks like If we do not define with designer.

00:13:10: look at organizational level which by way of context specific A Good Product Manager in one organization may not be a Good Product Manager In another organization.

00:13:21: Like how did expect people to succeed if were not explicit about that?

00:13:26: Yes And I can make the same analogy of like, how do we expect our feature or product to succeed if were not explicit about her expectations for it.

00:13:37: Yeah and this is just different scopes.

00:13:40: This has always been their work.

00:13:42: How do you define what good looks like?

00:13:46: Try to achieve that bar then iterate when fall short?

00:13:57: twelve at twenty-four months, is this?

00:14:01: in the beginning all of it seemed so new and revolutionary.

00:14:05: But The Last Human's So Much Of The Work That I Do Is The Basic Work That i've Always Done.

00:14:12: Yeah!

00:14:13: It's really defining these what good looks like moment more ruthlessly than ever I'd say doubling down on directional clarity with.

00:14:24: And why is it?

00:14:25: Because there's an increasing pain that you're automizing stuff, not yet in good shape with the use of AI just quicker and faster.

00:14:36: It gets to your output.

00:14:40: but you need to optimize process first before jumping into the realm of experimentation.

00:14:48: Yeah I saw someone post on this.

00:14:55: Right now we're automating a bunch of stuff that shouldn't even be done in the first place.

00:15:00: And it's true.

00:15:02: Some of this I think is just a natural learning curve.

00:15:05: Yeah, and I'm gonna equate back to what.

00:15:08: We were just talking about like I Think that Cal Newport has a concept about this Of Like.

00:15:17: how much of the work that we do doesn't actually matter?

00:15:22: Wow, this has just got pretty depressing.

00:15:23: But like how much of the work that we do?

00:15:26: Like doesn't actually have an impact and could we just stop doing those

00:15:30: things?".

00:15:31: And so

00:15:32: already...

00:15:33: Already were not good at asking these questions unlike reflecting on.

00:15:36: why am I doing it?

00:15:38: is what's my expectation about being explicit about that?

00:15:42: So there's a time when you tie back to outcomes in measurements.

00:15:45: What would i expect happen if I did this?

00:15:48: Am I making sure its what happens?

00:15:50: Then if we don't do that and then we automate it?

00:15:53: now, It's like we're automating a bunch of things.

00:15:55: That I Have no impact because when we did the thing had no impact.

00:16:00: yeah whoo?

00:16:01: I loved it Teresa i'm glad We unpacked this topic.

00:16:07: I'd say

00:16:08: Yeah, I think well all wrap up by just saying Like I Think um we Just talked about This in The context Of products We Talked About In The Contexts of Organizational Rules.

00:16:17: Now We're Even Talking About it in the Concept Context of like the work you're doing moment to moment.

00:16:23: I think like, the habit that underlies metrics outcomes is really just what do you expect to happen?

00:16:32: Can you be explicit about that?

00:16:33: so then- Yeah understand

00:16:34: a world and...

00:16:36: The assumptions around it which has good critical thinking!

00:16:40: Yeah i love that.

00:16:42: thank you so much Teresa.

00:16:44: Thanks Petra.