#606 AI Can Generate Code. It Still Can’t Replace Engineering Judgment | Jason Li

Mehmet: [00:00:00] Hello, and welcome back to a new episode of "The CTO Show with Mehmet." Today, I'm very pleased, joining me from the US, Jason Li. He's the CTO of Laurel. We're gonna talk about Laurel in details. Jason is building something really interesting, which is, you know, time management and, you know, we're gonna talk about also what's the problems behind this and how AI is able to solve these problems.

But before this, as my audience know, I like to keep it to my guests to introduce themselves. So Jason, again, thank you for joining me here today. Tell us a bit more about you, your background, your journey. You have a very like, uh, you know, great background, but I will leave to you to, to tell us more about this, and then we can dive directly into our discussion today.

So the floor is yours. 

Jason: Thank you. Thank you. Thank you for having me on. I'm excited to be here. Um, so yeah, background for me, um, I've been working in, in tech, especially tech startups for quite a while. Um, I spent a good chunk of the time in enterprise software, so I was at a CRM startup for a bit and was at [00:01:00] Salesforce.

Um, then I was at a company called Ironclad, which was contracting software. Um, and now, as you mentioned, I'm at, I'm at Laurel. Um, so really I've kind of s- seen and spent a lot of time in that journey from early stage B2B software to, to, to larger corporations o- of that size, uh, doing that kind of work. Um, and obviously right now going through that transition into, you know, very AI focused, AI native in, in this legal tech space that is so, so hot in that intersection right now as well.

Um, and then, yeah, I'm based out in the, the Bay Area and, and, uh, yeah. 

Mehmet: Great. Thank you again, Jason, for being here with me today. Now, Laurel started it originally as per my researches, like, as automated timekeeping. Yeah. But, you know, now the messaging kind of evolved, and this is very normal, into time intelligence.

Like, walk us through this, you know, uh, what does it mean from, you know, product perspective? 

Jason: Yeah, yeah, yeah. So timekeeping, like, that, that product, that focus, you can think of it [00:02:00] as there's a bunch of people in the world who bill for time, right? You think about lawyers, you think about accountants. They, they bill by, per the hour, right?

Um, and if you're not in that area, you may not realize how antiquated that is. Like, people effectively will write down notepads or write somewhere, like, "I spent six minutes doing this thing," or, "I spent 12 minutes doing this thing." And then they go back and write, like, they spent hours in a day or, you know, hours at the end of the week going, filling out those bills, right?

And so, like, that's, that's why automated timekeeping is so impactful. However, on technology side, if you think about how you'd solve that problem, which is what Laurel has done, is like... And we actually kind of track what you're doing and give visibility and surface that activity back up to you so you can build those time sheets.

That kind of technology is relevant for everyone, right? Like, I would venture a guess the vast majority of people who listen to this, and knowledge workers, et cetera, you know, you spend a lot of time working on the computer. But you wouldn't, you don't actually know where you spend your time, right? I mean, you would love to, right?

And so effectively our technology allows you to know where you spend your time so that you can, in the, in our initial case is to bill for time, but also just get that visibility [00:03:00] to do whatever else makes sense. You know, invest strategically here because you're spending time, more time there than you want or whatever else.

So 

Mehmet: yeah. Right. Now, most of people when they think about time tracking, they think about it as an administrative problem, right? Yep. So, right? Purely. You frame that into an operational intelligence problem actually. Mm-hmm. So what, w- what do you think the market understand too late about time data? Like, or, or as, is, are they still missing that, I would say?

Jason: Yeah, yeah. I mean, I think it's just- It's one of those things that I think if you talk to people is actually very obvious, but it's like so far from what actually people are used to that people don't think about it, right? Like if you go tell people, "You should know where you spend your time," everyone would be like, "Obviously I should know where I spend my time."

Like that's not a, that's not a, a huge insight. However, it is kind of is in the sense that no one is like, "Oh, wait, how am I gonna get a better insight of what we're, we're doing," right? Like, and it's funny 'cause like there's all kinds of work. You think about software engineering, like, like a lot of what we talk about.

You know, people [00:04:00] do, people do, uh, agile, they do points, they track how much a team can do, et cetera. So you try to get some sense of like where are you spending your time in terms of work, but you miss so many different things. And so it's really almost like people are so default, like used to not having a solution to that problem even though everyone agrees that that's something they would want.

Um, it's really being across that, that, that barrier of like, you know what? Actually we really could just do that, right? And if you could, you could do all these things, right? Um, yeah. 

Mehmet: Yeah. It's 

Jason: funny. 

Mehmet: Now Jason, s- something interesting because, you know, when we, we, we look today at the AI landscape, I mean all the companies, the first thing they tell you like we're trying to replace the work, right?

Yeah. Now from, for, for Lore it like seems like it's more focusing on understanding the work itself. Mm-hmm. And, you know, in, in my opinion, this is a huge differentiation because- Um, like in, I work a little bit in cyber security, I work in network visibility back in the days, and we used to say like we cannot actually, you know, [00:05:00] protect what we don't see.

Right? Yep, 

Jason: exactly. 

Mehmet: Now to me, like do, do you think like this distinction is, is like applicable for you at Laurel where if I come today as a consultant and an AI, you know, quote unquote- Yeah ... of course, consultant. An AI. You know, so the first thing I'm gonna start to tell the client maybe after a while, "Oh, you know, like you, let's put this AI tool here, let's put generative AI there," and so on.

So do you see what you're doing trying to understand actually where is the time spent as a differentiation, as a moat actually, versus, you know, what the other AI startups are trying to do? 

Jason: 100%, and I think your analogy to, to the, what you measure and what you go against is, is perfectly apt, is like that, like, yeah, I honestly, I'm a human.

I'd rather not replace human work. I'd rather be like, "Hey, where can we find the most leverage for, for the work that we do?" And I, you know, I feel comfortable saying, I know I'm gonna guess most people do, is like, "Hey, there's definitely leverage to be had." Like where do we spend our time? Where we could be better [00:06:00] spent and, and be more effective?

And like, so a couple examples is like in our space we s- we, a lot of our folks we sell to is legal and accounting because they do bill for time. Um, and that's where we started. Um, and there's examples there where, for example, senior partners at those firms, a lot of their job is getting new clients, right?

And so they, if they spend a lot of their time on administrative work, that's wasted time, right? And so if we can find obviously where they're spending time doing time entry, but even beyond that where they're like doing some internal thing that isn't adding leverage for their business, hey, they should find tooling or they should hire somebody to go do that so they can do the biggest additive thing for the business.

And right now they don't have visibility to that, and we can help them do that. But that's true for many, many things. And like a good example, y- we're talking about other AI companies as well, is like, is actually AI spend itself, right? Like everyone is buying some AI tool to do XYZ, and honestly, a lot of the times people don't have really good insight if it's actually adding the leverage that they hope, right?

Like they're, they're perfectly willing to make the spend, but they don't have good measurements of like did it actually save them time? Did it actually get the results they want? And I think that's one of the things we [00:07:00] can also help with is like, okay, hey, if you, if you got this AI tool 'cause you think it's gonna, you know, reduce time here, we can help see, see that for you, right?

We can be like, "Okay, hey, you know, you're not actually spending as much time there. Great." Or, "It hasn't made a difference," or, "It shifted over here, you have a new bottleneck. It's over here, you gotta go solve that now." 

Mehmet: Right. Now, y-you mentioned actually, uh, it's good you mentioned the, the kind of the verticals you try to, to, to, you know, uh, position your product to legal and accounting firms.

Now, we know there there's a high sensitivity on compliance. 

Jason: Yep. 

Mehmet: There's a lot of sensitive data over there. So, a-and here I'm asking you as a CTO, of course, like, uh, you know, how is the experience of building AI systems that professionals like working in these firms, legal, accounting, finance, um- You know, like how to convey the message for them that, you know, "Gentlemen, you know, you can, you can rely on this system.

It, [00:08:00] it, you can, it can be trusted." Because, you know, at the end of the day, you're touching on maybe some, I would say, economically sensitive workflows for them. So, you know, I'm, I'm curious to know, like, how do you position that when, when you talk to, to these firms? 

Jason: You mean like building trust, especially in the AI-based features 

Mehmet: or- Yes.

Yeah. Yeah, because, you know, there's a lot of talk about how we can trust AI- Yep ... whether like from both security perspective, compliance, and so on. 

Jason: Yeah, yeah, yeah. So I think, I think you're touching on, like, both security and compliance, but there's also, like, the model accuracy itself. 

Mehmet: Right. Yes. And, uh, it's- Exactly.

It is a con- Reliability. 

Jason: Yeah, yeah, exactly. And, like, uh, even prior to this, Laura, I was at a company, Ironclad. We did contract, contracting, right? And we would ingest contracts and try to understand them as well, and so the same, same questions came up. And I think on the security compliance thing, I think a lot of that is honestly doing the same kind of work that you, you know, you've done in, in, in your past career and, and just we've done in the past of like, okay, hey, let's create the great guardrails, the right environments, et cetera, to, to minimize risk and mitigate [00:09:00] risk and get, get to the risk profile they're comfortable with.

And that's everything from, like, data, data residency questions, encryption questions, et cetera, to, to kind of contain that to some degree. Obviously, AI adds an interesting other, other bit. Um, and as an aside, as you go into, as you mentioned, these, these kind of verticals and these spaces that are more sensitive, you get more painful experiences, right?

You have to do, as a vendor, as us, you have to do more work, right? Of like, okay, hey, we're gonna have to be a little more complex in our infrastructure to handle this, or, you know, we have to run another region to doing that. Um, quick aside, funny thing from my perspective as CTO is like, um, in this new world of AI, like, there's a lot of questions on what your mode is and what, what are you-- how are you gonna differentiate.

I actually think a lot of this kind of painful work is great actually, right? This painful work for th- this kind of stuff actually is, is a lot least easy to copy. You can actually copy a UI to go do something, but, like, a lot of this stuff just takes time and you gotta do the right thing and, and that, that there's effort to be had to, to go solve that.

So I think there's that. And then on the performance bit, like does it do what you expect it to do, uh, for AI? That one's an interesting one because I think it's [00:10:00] a... To me, it's a mix of a few things. Like, one, it's actually a good thing to remind everyone who's worked in this is, like, humans are fallible, right?

It's not like the humans who were doing this work before were perfect, right? And so, like, obviously you want to-- like, the, the goal is not to be imperfect as a software, right? But, like, you're, you're, you're, you're going there. So then it's like, okay, hey, if that's the case, like what do we wanna do to make it successful?

And then a lot of it, and I've really enjoyed talking to our design leader here too about this, is like- I actually think a lot with like AI features, it's so much like in partnership with design. It's not just like, does the model do this thing? 'Cause it's not gonna be 100%, 100% of the time, right? That's just like that, that pretty much is, is, is impossible, right?

Um, and so I think it's so much about how do you combine the technology to do the thing you work on and the experience you give, so that gives confidence, right? Like y- if for example, you have an experience that's presented to you as, as, "Hey, we have 100% confidence," and then when you're wrong, like that's, that's very different than you're like, you're presenting something like, "Hey, we're pretty sure this is right.

This should save you time. You should double check it and then confirm." And like that's a completely different [00:11:00] experience, right? And I think to this point of like, hey, when people are super sensitive about something, do, do that, and you might do it differently. And like, yeah. So I think it's so much like how do you, how do you tailor the experience to, to the models you have, the performance they have, so that the user like builds trust, right?

In, in doing that. And that's the same thing honestly you do with people too, right? Like the, the, the same, same, same things apply. 

Mehmet: Right. Now, in terms of, you know, like any other AI, um, you know, solution that you need to deploy, I've heard a lot and I've seen also, uh, personally, is that, you know, these systems can work as good as, you know, the systems around them are good, right?

Jason: Yeah. 

Mehmet: So are companies getting better in, you know, fixing their stuff before actually deploying these AI systems internally? Or are we still, you know, struggling, you know, in doing some pre-war before making, you know, the actual solution, [00:12:00] um, act as we are expecting it to act? 

Jason: My, my quick answer as you were saying that was like, "No."

We're not getting- And I said that because I think, I think... So I think, yeah, so yeah, I think people are getting better. They're getting more sophisticated and all of that jazz. But I do think, I do think there's a couple things. I think people in working in tech, we live in a bubble. We live in a bubble of...

You know, obviously AI has, has escaped the bubble and everyone's talking about AI, but I think people's, um, where they are on that curve, you know, the vast majority of people in, even in people who are at companies that are buying AI tooling, is not, they're not at the forefront of the tooling, right? And so, um, and so you see that, right?

Effectively you see s- those same lessons being learned again and again and, and that's a thing. Right. And, and even the slower adopters often have, you know, naturally have more things that, that are resistant to, to some of that. Um, so I think, I think you do see, I do see, y- I think you do see some of that improvement in some areas where there are people like, "Okay, hey, I've done this a few times.

I know actually the thing we need to go solve is provide the right [00:13:00] data, you know, set up the right context, change how we work." Actually, that's a huge part of it, right? Mm-hmm. You're gonna have to change your own processes just, like, as people to, to do it as well. Um, and so I think you see that, and I think that's great, but oftentimes you don't.

As an aside, I think that also means that as a lot of AI companies, for a lot of you, if you sp- especially if you wanna go really broad and not just the early adopters, you do have to meet people where they are. You can't just assume that they're gonna know all this stuff and do this stuff on their own.

You kind of have to, you have to meet them. Um- 

Mehmet: Yeah ... 

Jason: so. 

Mehmet: Yeah. So, so Jason, you, you kind of touched on, on the following question I'm gonna ask you, but, you know, if we can go a little bit more in details. Yeah. Now, um, so every company today that they want to implement, like any other solution, but especially now with the AI, because there is a lot of hype.

Let's, let's be, you know, honest about it. There's a lot of hype. 

Jason: Yeah, yeah, yeah. Sure. 

Mehmet: But, but there's a lot of good solutions also as well. So now, um- [00:14:00] Very few companies, you know, in my opinion, and this is something again from previous experiences not related necessarily to the AI, can actually measure, you know, ca- the ROI of their solutions, right?

Yeah. Right. So, so how is that applicable on measuring AI productivity? Um, you know, is it like more difficult? Is it like something you can show? And the reason I'm asking you because there's a lot of talks currently, and I shared that on my socials also as well, you know, the, the licensing model of- Yeah

all software is changing based on outcomes, right? So, so no more- Yeah ... seat-based. Yep. I mean, it's, it's not going, but I mean, everyone's talking about, you know. Yeah, yeah. Yeah. Yeah. So, so when it comes to AI productivity, like how do you, how do you measure it? 

Jason: Yeah, yeah. I mean, I think, I don't know which one you're asking, but I, I, I'm both on the buyer and the vendor side, so- 

Mehmet: Okay.

Um, 

Jason: on, on- 

Mehmet: You can answer on both. 

Jason: [00:15:00] Yeah, yeah. I, I'll take a stab at both. I think on, so on the vendor side for us, because our primary product, our first product is this timekeeping product, the plus there is like we actually help people find more time. Um, and so what we have happens with our customers is they end up billing for more time.

They save time in prepping the bills. They have fewer compliance issues of their bills getting rejected. And so that, that nicely is if you bill for time, that is very tied to economic value, right? Like if you bill for more time, you don't get rejected, blah, blah, blah, our customers make more money, right?

So that's, it's almost quite beautiful for us in terms of that ROI. Um, obviously the pricing models, there's still a lot of work to be done is like who's gonna do p- pra- per seat, who does whatever, who does platform prices, who does result pricing. Um, but at the very least for us that there is that connection at the end of the day of like, if you use us, you will make more money than you pay us, and we'll all be happy, right?

And so I think that's, that is nice and like, like you're saying, not everybody, not all businesses, not all products have that, that tie very, uh, clear. I think a great example of that is me on the buyer side, right? Like, you know, we buy all kinds of AI [00:16:00] tools. If you think about on the software development side specifically, we have everything from Anthropic to we use Codex, we use, uh, we use every, potential coding agent that we might, might imagine.

And I think that one I think is, is really fascinating and I think maybe I'm just too old, but like, I feel like a lot of this is actually just surfaces the same challenges that have always existed, um, but are just now, you know, even more focused, right? Like- Engineering, software engineering productivity has always been very difficult to me- to measure, right?

There's been like a... There's a small vertical about trying to solve this. Everyone's trying to figure this out. No one's done a great job. And so I think that's going back to, like, being able to tie that to ROI. Like, as a buyer, we use all these tools that should help us. Um, and unfortunately I don't think we still now have a great measurement as, like, how much did it help us?

And like, and it, I think it's been a lot of, you know, the jobs of CTOs, of engineering leaders to, like, have a feel for that and, like, everyone has their own system. Um, but honestly it's kind of a intuition and people kind of have some [00:17:00] system plus intuition on it. Um, I guess plus is it's good for our jobs, you know, and the job security in, in the fuzziness of that measurement.

Um, but it... I think that's, that's a continual challenge, I think. Mm-hmm. Is that I don't, I don't think, like... Like, I, I, I don't know. I can tell you how much we spend on some of these things. I don't know if we're sh- I can't tell you the dollar ROI of those things yet, and obviously we'd all love that. Um, but I'm not sure that we're terribly close to that yet.

And I, like I said, I don't think that's unique to AI. It's always been true, and it's always been hard to measure 'cause it's just like what you're measuring is, is so far out. Um, but yeah. 

Mehmet: A- are we still in experimental phase, you would say, Jason? 

Jason: Yes. I think very, very much so, right? Like, don't think, like, you dive into the software development stuff, right?

Is like a year ago we were all using Cursor, like ID-based, like, AI stuff. Now everyone's doing agentic code all the time. I was just talking to somebody and he was asked... Actually, a CTO was considering taking a job, and he was like, "Does anyone actually write the code anymore?" I was like, "Nope. No one actually writes the code directly anymore.

It's all through agents, et cetera." Um, and so, like, that's, that's only, like, been the [00:18:00] last, whatever, six, nine months or whatever that's been happening, right? And so I think everything is changing and everyone's finding new challenge. Like, one, one thing on my mind is, like, I think with everyone using agentic code, like we're creating code quickly.

Like everybody is. Mm-hmm. Right? Um, but everyone's running into new bottlenecks, right? Like the code review, the, the quality side of it is like, honestly hasn't been as well solved and so we're all running into bottlenecks there. My personal hypothesis, and I don't know what the solution is, is that a lot of our, our systems like PRs and code reviews, et cetera, are...

don't make sense in this current. Like they were, they- 

Mehmet: Mm. 

Jason: Like, they, like they- My, my maybe hot take is code reviews were never good. They were like ter- always a terrible process and they were super leaky. Um, but they, they worked because you didn't generate that much code and the people who wrote code, like, knew it well, and so that you could do a final check.

They caught some stuff, but it was super leaky, honestly, but like caught some stuff. But it doesn't really work when most of the code is written by agents and, like, it's coming at a volume that's super high, and code reviews have too much of a responsibility for catching things, right? Like, that they weren't really built for.

So like I do, I like to this experiment... Sorry, I'm going way [00:19:00] tangent on your question, experimental point. But like- It's okay ... it does feel like there's just a change in fundamentally how we work to, to solve that, right? And when everyone's trying different things, we're trying different things, but like it does feel like there's a, a shift in that, that normal flow because just the math is so different now in terms of where the time is spent.

Mehmet: Great input because you helped me in crafting better my next question, and sorry if it gonna look like a loaded one. 

Jason: No, please. 

Mehmet: I, I, I don't... Yeah. But because, you know, very related to each other. So you mentioned about, you know, not writing code anymore. I, I think we've seen a lot of people talking about it.

Anthropic CEO himself, like- Yeah ... he talked about it. Now- I'm gonna ask you like two related questions here. I'm gonna like try to, to, you know, put them into one. How this affecting, you know, the team structure from engineering team perspective? 

Jason: Yeah. 

Mehmet: And how that is also shaping leadership, [00:20:00] technical leadership, right?

Yeah. So, uh, because everyone has access to, I believe, sort of a, again, quote-unquote co-pilot or like- Yeah ... whatever solution you're using. Um, so, so, so h- how, how the measurement, how, how, you know, the management and the measurement of, of the performance of the engineers- 

Jason: Yeah, yeah ... 

Mehmet: get sorted out now. 

Jason: Yeah, yeah, yeah.

No, it's, it's, it's really interesting. Um, I might actually add one thing to that too, um- Sure ... at the end, but like, yeah, no, I think it's, it's really interesting. I was actually have a very similar conversation recently too. It's like on, on how to measure performance of engineers. I think the funny thing is at the end of the day, we're gonna measure it in the same ways that we always have.

We always had other like, like if you think about career ladders, et cetera, that you would build up to what it is. But at the end of the day, we are going to like what is the value that people dev- that's delivering to the company, right? Like, that's actually the, you, you do like we do a lot of special architecture and code delivery, blah, blah, blah.

But really all of that, you [00:21:00] know, if you didn't deliver value to the company in doing that, like that, you know, that's, that's not, that's not the performance we're looking for. And so I think in some ways, like, because we don't have those easy, you know, piece- pieces as much anymore, there's a little bit of we kinda have to step back to like, all right, is this team or this person delivering value?

And that's both in terms of releasing new stuff, but also it works and like we're able to manage it going forward. And so I think there's, there's that kind of, uh, stepping back and like analyzing that. As an aside, I actually think that's part of the reason why there's so much challenge with junior people versus more senior people.

Like more senior engineers, you are always looking at a, a little bit of a longer t- longer scale, long timescale. Like did you add value over time type of deal. Um, and actually this kind of fits a little bit nicer 'cause you're not like, I don't really... I- you have a certain amount of trust of the code quality, et cetera, and you see the result of like, does it have issues over time?

With more junior people, it's like, did you do this task? And now that task is kind of largely done by AI and so like it's harder to be like, are you, are you developing or not? Um, on the management piece, I think, um- I think one thing is just managers are more hands-on. I think a lot of managers [00:22:00] were hands-on before, but it's definitely more of a requirement now, right?

'Cause like it's just very possible to do it and like it's almost a miss if you're, if you're not. And so I think there's just more people, uh, working with agents, writing code or doing tooling and like, you can... That like hour of time as a manager to go do something on the side, now you can actually go and do something when before it's really hard to jump into enough context and go do.

Um, it is... One thing I find really interesting though is, so I think our management, like span of control, people who manage it probably has grown a bit just 'cause it, it fits there. However, like I think there's a lot of talk about what the ratios are of like product people to engineers to designers, et cetera.

Um, or like what size pod you should have. 

Mehmet: Mm-hmm. 

Jason: I'm not tied to this, but I current, current information, like current experience that we're seeing is like, I'm actually not sure, um, we should do this like two engineers to one PM, like these really tiny pods that people are, some people are trying to do.

And part of it is like, once again, unrelated to AI stuff, right? Like if you're running software, you probably have some on-call schedule, right? If your on-call schedule is two people, like that's, that's, that's no good for anybody, right? You kind of [00:23:00] need a certain group of people. The scope that a, let's call it 5% team or whatever, 10% team is doing, is obviously much larger than it was before, but I think there's still some organizational benefit of grouping a set of people together.

And in some ways it's actually kind of nice when you have enough, big enough group with now a huge scope You can really give them a key company goal and they can own the whole thing, right? Actually, that's really motivating and clear. It's like, "Okay, my job is to solve this." Like, it doesn't... You know, I can go, I have a really large remit to go do that.

Um, and in some ways that's actually, that's actually really clean. So yeah. 

Mehmet: I think one important point you just mentioned, Jason, um, and, uh, again, um, this is, I learned it by, uh, curiosity I would say back when- Mm-hmm ... I was working full time. So when people think about, you know, the engineering team, they think only about the people who are coding, designing- Yeah

and of course there is the product managers and, and so on. But they forget that, you know, you have to have, you know, a group of, you know, people and I know like you [00:24:00] usually do it in shifts- Yeah ... where these engineers they have to be on call in case like a bug shows, you know, like something- Yeah. You know, a- and people they think, "Oh, like now with AI we can shrink the engineering organization.

We don't need a lot of people." Like say, by the way, it's sa- it's same happening not only in, in, in, in, uh, engineering, I think it's happening in other departments as well. Yeah. Which is, I think what you just mentioned is a wake up call. Like, okay, we ca- AI cannot solve all your problems like this in, in- Yeah

in one shot, right? So we, we need to get onto it. Now, what I want to ask you from engineering perspective, with AI now generating the codes, I never applied for, uh, f- to be honest with you, to a job, although like I, I used to like to code, you know, 20 years ago. Yeah. But I didn't have the chance to get. But you know what I was hearing, you know, during interviews and you, you go to the skills and you try to see like who got the be- now with AI, right?

Yeah. So what kind of skills you as, as, as a leader you look for- Yeah ... um, so these guys, you know, like they can differentiate themselves, they [00:25:00] can add value to the organization? Yeah. 

Jason: Um, first off, I will say I don't have a great answer there and I, I don't think anyone does. It's very hard right now. Um- That's 

Mehmet: fair enough.

Jason: Yeah, yeah. No. And it is actually what, what I was wanting to add to is like, yeah, no, this is, it's very hard, right? And there are some similarities, for sure, right? Like a lot of the, the system design, can you think clearly and talk clearly about something is still there, right? Um, 'cause like- You know what, what I think we've seen time and time again is you need to have that clarity of thought so you can guide the agents to do the thing that you think is the right thing, right?

So you still do need that, like, given a problem, here's what we need to go and do. Um, it does raise the importance of, like, the ability to read and understand code, right? That was always important, but, like, that's almost, that's almost- Right ... more important now, right? Um, I was just kind of describing it as almost like with, with agent code, it's almost like there's like everyone's jumping to legacy code bases all the time, right?

'Cause it's, like, a ton of code that there's, there's not somebody there who wrote it that you're trying to figure it out. You kind of like always, like, doing archeology to figure it out. And so you have to really be able to understand that and, and figure that out. And so, like, I think there's actually [00:26:00] a lot of those, those core skills that are super relevant.

Funny, funny tangent. Um, so I have, I have four young kids, and so, like, one of the things I've been thinking about, like, and what, what, what should I, what, what skills should my kids be learning in this future that I don't even know what it's gonna be? Um, and I think this is related to interviewing is like, um, you know what?

I actually still want them to learn how to code, um, the same the way like, uh, you know, we learned arithmetic growing up. Like- Yep ... I don't have to do arithmetic, but doing arithmetic is good for building intuition for, for how things work and, and figuring out how to apply math. And I think of it similar here as like understanding software deep- deeply gives you intuition even if you're not gonna write about, like, how it should be done or where there might be problems or how to think about things.

And I think that applies to interviewing too, is like, okay, hey, how do you, how do you test and find and understand those key things so that they have the right intuition? 'Cause yes, anybody can write the prompt if you go build me an app that does X, right? But, like, do you have the right intuition about software to be like, okay, hey, if you're, if you're building this kind [00:27:00] of app, you're worried about this kind of scale problem, or you're worried about these kinds of ways it might break down.

And like, that's, that's the intuition we're looking for because that, that's, that, that, that almost translation gap of like what is our company problem to what is the technical thing the agent might build? Like, that's, that's what that person at the keyboard doing needs to do. Um, and so that, that's really that understanding.

I'm not sure that we, we as us, this company or the industry have figured out like the system to go apply to go do that. But w- we're, we're at least trying to lean on some of the things we have and then do some amount of like, hey, as part of the interview, you, you can use AI to do parts of the interview and that's great.

That's, that's the expectation. Um, but let's figure out how to, like, pull apart your, your contribution to that versus what- 

Mehmet: Mm-hmm ... did 

Jason: as well. So. 

Mehmet: Right. Um, I, I agree with you on keep teaching, you know, the next generation about coding just for the simple reason. It's not, like, about m- maybe, yeah, they will, they will not have to write a single line of code in the future.

Or actually you don't have to even [00:28:00] today. Yeah. We know this, uh, if you want. But I think, you know, for me always what the reason I used to like and still I like, you know, to play sometimes with, with coding is the logic, right? Yeah. So, so it, it, it, it, it, it taught me, you know, how to think in logic and then how do you structure things.

It's not-- it's more than just, you know, the, um, you know, the, the, the, the program itself or the code itself- Yeah ... how it works. It's, like, about how I can structure things. 

Jason: Yeah. 

Mehmet: And I think, I hope someone does this experiment and we try, for example, especially with these vibe coding, um- Yeah ... tools, if we try to get someone who have, like, little bit of coding background to type a prompt and, you know, and then fix the output- Yeah

versus someone non-technical. And I'm not saying this because non-technical people cannot do it. They can, and of course they might have, like-- they might describe it from business perspective better than a technical person. Yeah. But still, if you want to get a better result, I actually, I tried, you know, I, I cannot [00:29:00] claim it's an experiment, but I, I tried, you know, to write the prompt- Yeah

in different ways, more, one more like as a kind of- Yeah ... a technical guy, the other one more a business guy. Of course, I saw the difference, right? And then when you try to fix things later, it makes the life much, much easy. 

Jason: Yeah. 

Mehmet: Now, the reason I'm asking, you know, and I brought this here, you kind of touched on it, but now how big is the AI hallucination noise?

Uh, of course we, and we just mentioned, and you just mentioned, uh, Jason, it's about, you know, the... So, so we, we know these AI tools, they are chatty. They l- they like to talk. I, I call them, they are, they are talkative. Yeah. Yeah. Uh, we can see from ChatGPT and the other tools, you ask for something, they throw in front of you- 

Jason: Yeah

Mehmet: hundreds of, you know, nonsense things sometime. Yeah. Now, if, if we take this into code perspective, how much of that is noise? How much of that is technical [00:30:00] depth, which is very dangerous because I remember when I started the podcast, I didn't know even what a technical depth is at that time. I mean, from, you know- 

Jason: Yeah, yeah, 

Mehmet: yeah

structuring your thing. So, and then I, I start to understand from people I was, you know, interviewing, oh my God, like if actually, and this still by the way, AI generated code wasn't that much in 2023. It was, like, still early days. Yeah. And then I start to think, oh my God, like now if we are building these whole modules using AI- How much risk is there, Jason, still?

And do you think, like, we will be able to solve the problem that is caused by AI-generated code? 

Jason: Yeah, yeah. On the last question, I think yes. I'm a very human optimist, is that- Good ... we create our problems and we'll solve our problems. Like, we're gonna create some big problems, but we'll, we'll solve them. So that, I, I think we will solve them w- you know, maybe some challenges.

But I, I, yes, I think for sure there's a technical debt issue, [00:31:00] uh, that exists and that keeps growing, uh, with this. And I think there's actually, like, an interesting spectrum, right? There's the, "Hey-" You create some prompt, like you said, like especially if you're either naive on the technical side or not, not being very diligent about it, you can, it can just like be explosive, right?

And like there can be versions of this where you're like, and anybody with some technical skill can glance at it and be like, "That's crazy." Um, and so hopefully you have things to stop that, right? Um, and, you know, hopefully some systems be like, "Okay, hey, yeah, we should, we should redo this. We should have the right system prompt or whatever to, to manage it, and we should check it," blah, blah, blah.

But actually, I think the more insidious ones are the smaller ones, right? The ones that are like, "Hey, that's okay. It's not how I would've done it, but it's okay." Um, and so then I think the challenge is that probably happens a lot, right? 'Cause then everyone's like, "Okay, either I could iterate a few more times to get the prompt and the code exactly how we want, or I could not and get it, get it out and go into the next thing."

And because everyone's generating so much code, you actually do that a lot. And the whole team does it a lot, right? Um, and so I do think you end up having this, [00:32:00] uh, maybe drift from the quality you might otherwise require. Um, and, and I think that, that, that definitely creates this like tech debt challenge.

And the other part too is also the, the expertise challenge, right? Is like when you were as an engineer writing the code yourself, you just like deeply understood it 'cause y- you're hands-on keyboard typing it through and see. It just creates a certain depth of understanding. When you're doing it with agents, yeah, you're in it, but you're never quite in it quite as much, and so you don't quite understand it as quite as, quite as well.

And so not only do you have this code that may not be exactly the highest quality that you would want it, but also you don't know it as deeply as, as you're used to. So when something comes up, you don't have the perfect intuition about how it works. And so yeah, I do think there's a bunch of challenges.

I do... One thing I, I'm very curious about is with AI, in theory, you should be able to do big rewrites more often, right? If that makes sense. Mm-hmm. Right? Like general, general role in software engineering is like, don't do big rewrites. It's a terrible idea. You're gonna, you're gonna spend a lot of time and it's gonna be really awful, right?

Um, but I think there's a world where [00:33:00] now when code generation is much cheaper, is it actually much more possible to be like, "Hey, this thing sorta works, but we need to re-architect it. We should throw it away and build it again." And I think that, that is actually one of the things I'm interested to explore to solve this tech debt issue is like, hey, it's gonna happen.

It's pretty much unsolvable to grow right now, but are we willing to take the steps to like, "Okay, hey, cool. We, we've learned what we need to do with the product. Let's redo it this way." And not only can we do that relatively quickly because we need agents to help us, but two, we can actually build the test harnesses and the other things to make sure it works so this rewrite is a lot less scary.

And so like actually, like internally we've talked about, hey, for sometimes we have a tech review, if I could actually review some, some idea. Normally you, you do all the like whiteboarding and boxes of like what you're gonna do. Sometimes we're like, "You should build a prototype of a thing that kinda works- Because, like, that's much cheaper.

You can do that even before we have a, a group meeting about it, and if we decide we're not gonna do it and rebuild it, that's not a big deal either. And so, like, I wonder if that's, like, a mentality shift of like, hey, actually, if we treat the code as much more throwaway, um, [00:34:00] and we're build- the code around it to be able to replace it more comfortably, if is that the path to solve this tech debt issue of like, hey, we're just willing...

It's gonna happen, and it always has happened, but it's happening faster, but we're just willing to go and like, yeah, we're gonna go and go and fix it now 'cause we have this context and we can, you know, craft the right way to, to think about it going forward. Um, and, and use the tools. I think that's actually the big thing, is you have to be able to use AI tools to tr- address the problems that AI is creating 'cause, like, we just won't have the bandwidth otherwise.

And so it's like, okay, hey, how do we go find, find that opportunity? 

Mehmet: Right. I gotta ask, like, something which I didn't plan to ask, but because w- we, you know, the conversation brought us here. Yeah. Is there any difference, Jason, I'm asking you from pure technical perspective, using the AI

code generators, you know, directly from, let's say, Anthropic, like, you know, Cloud Code or, uh, Codex in case of, uh, you know, OpenAI [00:35:00] or, uh, I forget what the Google one. It's Gemini. It's Gemini. It's one of the Gemini. Yeah, 

Jason: yeah. Yeah, yeah. 

Mehmet: And, and I saw like now xAI, they have something also as well, versus the wrappers, right?

Yeah. So, so there's a lot of, you know, they just wrap it, you know, like similar where you see an AI application. Is-- Are there really any, you know, differences? Because I'm asking you this from an experience that recently I had is sometimes I believe when you use the directly the model provider, you get more solid results rather than when...

But, but when you go on the design side, you know, of course the others they're better. And I, I, you know- Yeah ... this, this podcast is, is not, um, sponsored by anyone, so any name I mention- ... there's no affil- Yeah ... there's no affiliation. So let's say if you go to Lovable, Replit, you know, these kinds of things, the design is awesome.

If you go to Anthropic, if you go to, uh, Cloud Code or, uh, [00:36:00] you know, the Codex from, from, um, uh, OpenAI, I'm sure... It's not I'm sure. I am sure because I tried a few things and from functionality perspective, I can feel it's more robust than, you know, using a- Yeah ... third party tool. Anything, you know, experience you can share here, like or any lights you can shed?

Jason: Yeah. I mean, I think like is the, the fundamental question of like, you know, the differences between the direct from the model providers and, and other tools. I mean, like I think the funny thing is a lot of this is a black box. It's like it's hard to tell. You know, I think there's, there's always, if you like look at forums, there's whispers.

It's like, "Hey, did they nerf the, nerf the, you know, 5.2 model encoders?" Whatever. Like there's all these questions all the time. 

Mehmet: Yeah, yeah, yeah. 

Jason: And we honestly, we don't know. Like we're, we're on the outside. We, we don't really know. Um- I, I do think-- I actually, I think the biggest thing is this answer's always constantly changing.

Like, this space is changing so fast, and so, like, the right model for the right issues, the right tooling to choose, et cetera, kind of changes all the time. And like, I do [00:37:00] think it's pretty clear that, um, it's an interesting combination where I think some wrappers, um, are nice 'cause they, they help... I think part of it is like these wrappers are helpful 'cause they might solve a part of the problem that makes certain things easier, right?

Like for example, making a really simple, nice looking site probably easier than these levelables, you know, uh, vaults, whatevers of the world. Um, that's true. Um, if you're gonna go build a full-fledged app, you, you know, you're probably still gonna go lean back on Claude Code or Codex or whatever, and that'll make the most sense.

Um, but there is some real value in that stuff. Like, for example, we also use, um, Devin, which is like a thing by Cognition that as far as I can tell, I think it's probably a wrapper on one of these models, but they did some niceties of like, oh, they hook up to all these other systems easily, like Linear, and they run the agent in the cloud for you, et cetera.

It's like we could do all of that, but it's nice that they did it for us, right? And that, that's the same thing with these wrappers is like, hey, if you're technical enough, you can set up something in Supabase and Netlify, whatever, to go set up a website. But if you're not or you don't wanna bother, like there, there's value in that.

And so I think there's, there's something there. [00:38:00] Um, obviously that doesn't apply for everybody, right? Like if you're a software company, like there's a lot of stuff you should just do in-house 'cause it's so core to what you do. Um, but like it's right now, at least to me, it seems like with those wrappers, it's like the ecosystem and the value of the ecosystem they're building around it is a real value.

And then as the buyer of it, you have to decide like, do you actually care about the ecosystem or not? And you don't necessarily care about it all, but like that, I think that's, that to me is the question. And I think the really tricky part too is like, like it just changes so quickly. Um- Right ... it's a funny thing with all these AI tools is like it does feel like you just have to avoid any long-term contracts 'cause like they all change so fast

And so like you have to be ready to be like, "Oh, how we wanna build software is actually completely different now. We have to be ready to, to, to move there if need be." So. 

Mehmet: Yeah. A-and to your point, we're dealing with black boxes and- Uh, it's their nature, by the way, because- Yeah ... at the end of the day, you know, the, the way they-- I don't want to go into that now, but I mean, just on a high level so for the audience to understand.

So the way any LLM tools try, [00:39:00] they, they do guesses on what's the next- Yeah ... token is, and there's a lot of, you know, if, if you like math and you like, uh, probability theories, like you, you you might be- Yeah ... able to understand that. But yeah, so, so I, and I, I tried a couple of times to the same model, same version, to put the same question, and I get slightly different answers sometimes.

Yeah. And I be-believe when you are doing the coding, you never know, right? So- 

Jason: Yeah, 

Mehmet: yeah. 

Jason: Yeah. So, so- I think that's the best thing about this whole new space is like in software, we've been so used to it being super logical. I've always told people software's actually really easy, right? 'Cause one plus one always equals two.

Exactly. You have to build the software system, but always some smart human came up with it, so you can kind of guess at how it works 'cause like just like logically- Yeah ... simply. Now with LLMs, like that's not true. Like it's all stochastic. It's got kind of random like and so like when you build a product, it doesn't always work, but maybe it's actually a great product.

It just only works 99.9% of the time. You just have to accept that 0.1% is wrong. Um, but then also like when we use these tools to go build it, it's very interesting 'cause we're using these stochastic tools, these agent builders to go build deterministic software, and it's an interesting like... [00:40:00] And then of course, the problems we're working on are, are different every time.

So it's like very hard to get a good sense for like The performance of different models and different tools, uh, for sure, but also just like it's just changing how we think, right? Of, um- 

Mehmet: Absolutely. 

Jason: Like, you know- Absolutely ... what's a bug versus what is, you know, just part of the, the cost of doing business.

So. 

Mehmet: I again, no affiliation, but I w- I have to give credit to, to, to, uh, Anthropic on, uh, the cloud code part. Yeah. Yeah. Because, you know, one of the things, and of course, this is because of the memory, I always like, you know, part of my instructions, like anything you give me, explain your rational, right? Like ex- Yeah.

Yeah, yeah ... explain how you c- how you come to it. Because I, I tried that with the others, don't get me wrong, but the best explanation I get is, is, is from, from Anthropic, you know, the, its super product. Like even if you try- Yeah ... a- as hobbyist like myself, you know, I, I would, I would recommend it. Um, Jason, as we are always coming to an end, like final thoughts maybe, and where people can get in touch.[00:41:00] 

Jason: Oh, yeah. Um, get in touch, I think LinkedIn is probably the easiest, uh, for me. Name is Jason Li. Um, the, uh, final thoughts. I don't know. It's, I guess the biggest thing is like, like we're kind of touching on, there's so, like, so much unknowns. Like I, I, there's most of what I've said is like my best hypothesis, but who, who knows?

And like, but that's also actually the fun of it, right? Like, a- actually, the way I've des- I mentioned at the beginning, like I've, I've worked in B2B software for a while. Um, it's great, it's challenging, et cetera, but it also become pretty well-defined. Like, you go do these things and like this is what you go and do, so playbook to do this.

A lot of AI's really thrown up, I'm sure for many different people, it's like kind of thrown up a lot of the, the things that you should do up in the air, and so that's actually quite fun though, right? You're kind of rediscovering like what is the right answer? How do you go think about this, and what do you do?

What, what lessons apply from years ago, and what needs to be re- rethought? Um, and I think that's just like really interesting, right? And I, you know, I assume you're, you're seeing a lot of that as you have conversations and stuff. But like it's just a really interesting time where a lot [00:42:00] of things are changing.

Not everything will change, um, but a lot of things will change. Um, and trying to understanding what changes and what they change to and what doesn't change is, is just quite fun. Yeah, it's good. So. 

Mehmet: O- opportunities also for- Yeah ... you know, startups, right? I believe. 

Jason: 100%. 100. Tons of opportunities for startups or individuals to, for, for, for everything.

Um, and yeah, there's just like tons of Yeah. With a, with a rapid change, I feel like there's always an opportunity both for individual and companies to go and seize on 

Mehmet: something. Yeah. You know, like, just as a follow-up on this, like, I, I get surprised, especially in the past, I would say six to nine months because of the guests, of course, and you know- Yeah

you know, people they, they, you know, um, when they show interest to be here, okay, I, wow, like see how much AI, like it created new categories or like the same, maybe not a category yet, but I mean, problems that- Yeah ... actually it's, it's using AI to solve another AI problem. Yeah. So there's a, so there is a lot of opportunities plus I believe there's a lot [00:43:00] of, you know, areas we didn't touch still yet to solve it with AI, I mean- Yeah

in, in, in the real life. And yeah, so, so it's great to, to, to be in, uh, in such times. Yeah. Jason, really I enjoyed. Of course, like, you know, I gotta put your LinkedIn profile in, in the, in the show notes. Of course, Laurel website will also be, will be available there. Uh, and thank you for, for the time today. I really enjoyed the discussion.

We are in exciting times. This is my takeaway and there's a lot of things still to, to uncover, uh, especially when it comes to AI, especially when we talk about, you know, ROI and showing the value. And of course, one of the main topics we discussed today is about like how engineering team structures are changing and the role of the leadership in this.

So thank you again, Jason. As I said, the links will be available in the show notes if you're listening on your favorite podcasting app. If you're watching this on YouTube, you will find them in description. And this is how I end my episode. This is for the audience. You know, as I say always, if you just [00:44:00] discovered this podcast by luck, thank you for passing by.

I hope you enjoyed. If you did so, give me a favor and try to, you know, subscribe, share it with as much people as you can because we're trying to, you know, spread the knowledge and spread, you know, all, all the good and nice stuff like Jason brought to us a lot of, uh, great, uh, topics today. So just share that.

And if you are one of the people who keeps coming again and again, thank you very much for your support. Thank you very much for keep following. And a big thank you for everyone who still mention us, tell other people about us. This is huge for me. Because of this, for the first time, and we're recording in the end of May, so in May, the podcast managed to get first time ever in North America, in Canada, we were in the top 200 podcast chart.

This is big for me. Like we were always in Europe somewhere, in Asia, in, in the Middle East where I'm based. So we managed to get into North America finally, and this is cannot happen by itself. So [00:45:00] this is again, thank you for everyone who is listening and tuning in. And as I say always, we will be again a new episode very soon.

Thank you. Bye-bye.