1
00:01:40.420 –> 00:01:51.769
Jeffrey Palermo: Well, welcome, welcome to today’s… today’s training. I’m excited about the topic, and excited about the… the really large group of people who are interested in it, so…
2
00:01:52.050 –> 00:01:58.030
Jeffrey Palermo: I’m gonna try to get through the content as quick as I can to leave some time for discussion and questions.
3
00:01:58.170 –> 00:02:07.569
Jeffrey Palermo: So… The, the topic at hand is, DevOps environment, and talking about what artificial intelligence has changed.
4
00:02:07.850 –> 00:02:16.339
Jeffrey Palermo: Because we all know that, that the industry with, with AI, a lot of things are changing and.
5
00:02:16.340 –> 00:02:27.920
Jeffrey Palermo: When it comes to DevOps Environments, that is the infrastructure and automation and process pieces that allow us to make changes to software, know that the change is good.
6
00:02:27.920 –> 00:02:32.900
Jeffrey Palermo: And be able to promote it across environments to various audiences, and run it
7
00:02:33.020 –> 00:02:50.459
Jeffrey Palermo: so that our customers and software users can use it in a stable fashion, and keep it running, and know what it’s doing, and keep it stable, and respond to incidents. All those things, the technical capabilities to allow us to do that, that’s kind of the DevOps environment, soup to nuts.
8
00:02:50.860 –> 00:02:57.319
Jeffrey Palermo: So, Let’s, let’s… let’s dive into it.
9
00:03:01.350 –> 00:03:17.510
Jeffrey Palermo: So, this is our… this is our agenda for today, our kind of our, our roadmap, as it were, and we’re going to talk about the things that are stable, what AI has broken, and a few… a few views on
10
00:03:17.650 –> 00:03:28.320
Jeffrey Palermo: trusting what AI does, and what should we depend on AI for, and then we’ll talk about the actual AI DevOps environment. That is not just the.
11
00:03:28.880 –> 00:03:30.260
Jeffrey Palermo: Not just the…
12
00:03:30.860 –> 00:03:41.080
Jeffrey Palermo: DevOps environment that we’ve had, but what does it mean to have an AI DevOps environment, where we are pushing on the
13
00:03:41.290 –> 00:03:48.909
Jeffrey Palermo: on what AI-driven development can do for us, and how it needs to… how our DevOps environments need to be up-leveled
14
00:03:49.040 –> 00:03:52.679
Jeffrey Palermo: to, I’ll call them, an AI DevOps environment, okay?
15
00:03:53.640 –> 00:04:00.520
Jeffrey Palermo: And we’ll have… I’ll show some, show some… short demos. Alright, so…
16
00:04:03.490 –> 00:04:23.179
Jeffrey Palermo: this is kind of how we see, we see the past. You know, continuous integration, came along a little over 20 years ago. DevOps became a mainstream practice, and there’s some… the principles that… that stayed stable, and
17
00:04:23.400 –> 00:04:29.390
Jeffrey Palermo: The engineering practices with extreme programming, we could even put 1999
18
00:04:29.790 –> 00:04:47.950
Jeffrey Palermo: on this, on this chart, because test-driven development and automated builds, automated testing in general, you know, that goes way back. And then the topics of continuous integration and trunk-based development and feature flags and…
19
00:04:48.060 –> 00:04:53.640
Jeffrey Palermo: Kevin D’Ephemeral Environments and observability and all these things.
20
00:04:54.000 –> 00:05:03.280
Jeffrey Palermo: have come along, and now we have AI-driven development, and what… what is stable, and what stays the same. So.
21
00:05:03.920 –> 00:05:14.450
Jeffrey Palermo: Let’s… as we dive into that, let’s just kind of reflect on some of the things… some of the things that are changing. Here’s some… some stats, and… and…
22
00:05:14.450 –> 00:05:24.449
Jeffrey Palermo: There’s some articles on DevOps.com, Octopus.com also has some articles and some studies and some surveys. So.
23
00:05:24.690 –> 00:05:37.870
Jeffrey Palermo: in general, code generation, at least from the survey, seems to be speeding up industry-wide by 40.2%. And I can see some people would say, no, no, no, coding generation, for me, has sped up by
24
00:05:38.080 –> 00:05:52.829
Jeffrey Palermo: You know, an order of magnitude, you know, it obviously has sped up faster in some areas and slower in other areas, but this is just kind of what some of the surveys we’re seeing industry-wide. It’s already… the averages are already up 40%.
25
00:05:53.390 –> 00:06:11.150
Jeffrey Palermo: As far as using AI for code generation, and we’re using AI in code reviews, 37%, we’re using AI in CI and CD for 13%, and using it for deployment decisions, 6%, so a little bit less there.
26
00:06:11.230 –> 00:06:15.140
Jeffrey Palermo: Just some numbers, from an adoption perspective.
27
00:06:15.460 –> 00:06:18.370
Jeffrey Palermo: And…
28
00:06:19.960 –> 00:06:33.949
Jeffrey Palermo: We sometimes think, okay, we’re changing code at a faster rate, and so we need to… we need to review the code faster, and team-based manual reviews are the bottleneck, and…
29
00:06:33.950 –> 00:06:41.169
Jeffrey Palermo: And, and maybe that’s the case in, in, in the short term, but really the true bottleneck is.
30
00:06:41.540 –> 00:06:51.569
Jeffrey Palermo: what pace of changes your customers can absorb, and are you going at the natural pace of business? And…
31
00:06:52.310 –> 00:07:08.659
Jeffrey Palermo: And also, do we have any batching? Do we have… do we have approved pull requests or approved changes that haven’t even been deployed to the customers yet? Alright, that’s… that’s where the bottlenecks, that’s where it, seems to…
32
00:07:08.950 –> 00:07:10.350
Jeffrey Palermo: Seems to mount up.
33
00:07:11.020 –> 00:07:17.850
Jeffrey Palermo: Alright, and there’s a few… a few voices I want to kind of highlight that are talking about
34
00:07:18.140 –> 00:07:25.099
Jeffrey Palermo: what we should trust when it comes to AI-generated changes. I want to highlight a few of them.
35
00:07:26.740 –> 00:07:44.740
Jeffrey Palermo: And, one… one is, DHH, the creator of Ruby on Rails, and, co-founder of 37 Signals, and, Uncle Bob Martin, Robert C. Martin, and then also the research team at Octopus Deploy, and…
36
00:07:45.110 –> 00:07:49.570
Jeffrey Palermo: And, they say a little bit, you know, some different things, and here’s some…
37
00:07:49.600 –> 00:08:03.199
Jeffrey Palermo: some paraphrasing, you know, DHH says that agents are going to work more like a team than a pair programmer, and just last week, if you want to go on YouTube, there’s a really good talk at the Rails conference.
38
00:08:03.210 –> 00:08:16.710
Jeffrey Palermo: And, about… about the approach. And then, Uncle Bob Martin says that he no longer reads every line of code, just does spot checks, and the research team at Octopus Deploy
39
00:08:16.710 –> 00:08:23.949
Jeffrey Palermo: It talks about enforcing the design with custom static code analysis and linters and structural tests.
40
00:08:23.950 –> 00:08:31.870
Jeffrey Palermo: And, and, and there’s a whole lot of, there’s a whole lot of commentary on, on,
41
00:08:32.150 –> 00:08:47.460
Jeffrey Palermo: on this issue, and I want to kind of go back in time. I pulled out… pulled out some, you know, some more quotes from DHH, just because it’s interesting, his journey, he’s talked a lot about this, and my journey’s kind of been similar.
42
00:08:47.650 –> 00:09:01.430
Jeffrey Palermo: from January just to September this year, and how we’ve gone from thinking that, you know what, let me use AI to assist in making these changes.
43
00:09:01.700 –> 00:09:13.989
Jeffrey Palermo: Versus, no, I’m gonna have… I’m gonna build a system so that AI can… can make all of the changes. Because me handcrafting a line of code now is…
44
00:09:14.170 –> 00:09:18.359
Jeffrey Palermo: Slower is provably slower than I could go otherwise now.
45
00:09:18.600 –> 00:09:20.869
Jeffrey Palermo: We’re gonna have an industry debate here.
46
00:09:21.190 –> 00:09:23.490
Jeffrey Palermo: Over this next period of time.
47
00:09:23.660 –> 00:09:35.209
Jeffrey Palermo: Because… because the thought of, you know, some… some programmers who’ve been doing it… I mean, I’ve been a programmer since 1997. Some programmers… the thought of… wait a minute.
48
00:09:35.390 –> 00:09:50.130
Jeffrey Palermo: You mean I’m not gonna have my I’m not gonna be in my IDE cockpit for eight hours a day. I’m not gonna be doing you know my resharper autocompletes. I’m my my fingers aren’t gonna be dancing over the keyboard, I’m not gonna be left alone for hours on end in a constant state of flow.
49
00:09:50.140 –> 00:10:09.279
Jeffrey Palermo: And, I’m not going to be handcrafting these lines of code. I’m not going to be… I’m not going to be celebrating this new language feature that just got announced and geeking out with my friends on how, you know, now the way to build this syntax is slightly different. Let’s talk about that. I’m not going to be able to argue about tabs versus spaces.
50
00:10:09.390 –> 00:10:20.930
Jeffrey Palermo: I mean, that I think a lot of people are gonna go through the five stages of grief because our whole industry is changing where the the work of a software engineer.
51
00:10:21.120 –> 00:10:27.979
Jeffrey Palermo: Is morphing from hand-typing syntax characters, and… and…
52
00:10:28.100 –> 00:10:31.340
Jeffrey Palermo: The mechanics are changing. Now we’re going to be…
53
00:10:31.740 –> 00:10:36.360
Jeffrey Palermo: We’re still… we still have to cause the software to do the right things.
54
00:10:36.370 –> 00:10:54.519
Jeffrey Palermo: But the mechanics of how we do that are changing. And, I don’t believe for one second… some of the naysayers say that, oh, we’re gonna have fewer programmer jobs. Exactly… it’s the opposite. I mean, every time… every time some type of work or some type of.
55
00:10:54.530 –> 00:11:10.930
Jeffrey Palermo: material has become cheaper, the demand for it explodes. I mean, just look in history, look at coal for heating, look at oil, look at transportation. As soon as there’s a breakthrough.
56
00:11:11.230 –> 00:11:16.290
Jeffrey Palermo: And the per unit cost of a material is cheaper, and that material being a line of code.
57
00:11:16.700 –> 00:11:28.949
Jeffrey Palermo: there are so many more people that think, you know what? These software projects and these software teams, they’re way too expensive for me to think that, you know, I should build a piece of custom software, and instead I turn to
58
00:11:28.950 –> 00:11:37.850
Jeffrey Palermo: custom off-the-shelf software, and I… and I, I bought a piece of software, or I licensed a piece of software. Well, now, the per-unit cost is down.
59
00:11:38.110 –> 00:11:51.079
Jeffrey Palermo: maybe I should build something for myself, because I hate all these software vendors anyway. And so, my prediction is that the demand for software is going to explode, and… and because the
60
00:11:51.080 –> 00:11:59.919
Jeffrey Palermo: per line of code cost is going down, and so those of us that… our entire career is about delivering custom software.
61
00:11:59.950 –> 00:12:06.999
Jeffrey Palermo: Man, we, for the next decade, we are gonna be busier than we ever have been.
62
00:12:07.940 –> 00:12:12.550
Jeffrey Palermo: So that’s my prediction, that… that, we…
63
00:12:12.780 –> 00:12:29.490
Jeffrey Palermo: those of us in the business of delivering custom software, and those of you on this call, you are going to be busier… far… the doomsayers are saying, oh, there’s going to be fewer jobs. Like, there’s going to be a kind of a reshuffling of where the jobs are and who’s doing the jobs, but…
64
00:12:29.750 –> 00:12:33.109
Jeffrey Palermo: We’re gonna have explosion of demand.
65
00:12:33.320 –> 00:12:51.169
Jeffrey Palermo: So that’s… that’s where I think… anyway, that’s an aside. I want to bring up, Uncle Bob Martin. I had him on my podcast just a couple weeks ago, the AI DevOps Podcast. You can search YouTube, or just go to aidevopspodcast.clear-measure.com.
66
00:12:51.280 –> 00:12:59.459
Jeffrey Palermo: Also on Apple Podcast, Android, Google Play. You can find find them pretty much anywhere. And it was a great interview.
67
00:12:59.840 –> 00:13:05.129
Jeffrey Palermo: And, and Bob is a really fascinating, fascinating guy to talk to, and
68
00:13:05.150 –> 00:13:21.449
Jeffrey Palermo: I mean, he’s been… he’s… his career started back in the… in the 70s, and so he’s seen more transitions than I’ve seen. I mean, early in micro, I saw the internet transition, and that’s… that’s very instructive to see the parallels, but he’s seeing the… the, the mainframe and the microcomputer
69
00:13:21.760 –> 00:13:32.370
Jeffrey Palermo: transition as well, and and basically, you know, his view of it is that the same things that slow down,
70
00:13:33.220 –> 00:13:48.940
Jeffrey Palermo: team-based programmers slow down the AI agents. And so messy code is one of those things. Code that has no pattern, code that has no consistency, it slows them down. And so, some of the things that are stable is we still need
71
00:13:49.320 –> 00:13:51.760
Jeffrey Palermo: We still need well-organized
72
00:13:52.250 –> 00:14:10.350
Jeffrey Palermo: easy-to-understand code, because remember, we… these AI models, they’re… they’re basically designed mimicking the biology of the structure of our brain. And… and so we’ve been able to kind of simulate language, and, you know, C-sharp and TypeScript.
73
00:14:10.490 –> 00:14:14.530
Jeffrey Palermo: They’re all… they’re just different languages, and so we built a machine
74
00:14:14.720 –> 00:14:34.699
Jeffrey Palermo: that is able to simulate language and work with language, and so if our… if the language that we store is messy, well, this digital brain that’s kind of been designed the way our own biology works for that portion of it, the same things are gonna… are gonna be hard. One thing…
75
00:14:35.020 –> 00:14:37.000
Jeffrey Palermo: Well, I don’t want to get ahead of myself.
76
00:14:39.310 –> 00:14:42.919
Jeffrey Palermo: The, the, Octopus Deploy lens.
77
00:14:43.040 –> 00:14:58.870
Jeffrey Palermo: is, is that there’s… there’s really… there’s really three concepts, or progression of concepts. One is… one is context, the other is, constraints, and the other is, entropy management. And so, the context is…
78
00:14:59.070 –> 00:15:04.900
Jeffrey Palermo: You know, documents in the code, and,
79
00:15:07.810 –> 00:15:17.790
Jeffrey Palermo: builds, and scripts, and structure, and anything that can go into the AI context, I like… I like to put my, architecture decision records.
80
00:15:18.060 –> 00:15:36.550
Jeffrey Palermo: in a docs folder, and that way they can be pulled into the, into the context. And then architectural constraints are builds and pull request checks, things where they are scripts, where if they don’t pass, then we’re not… we’re not done with the changes on this… on this branch.
81
00:15:37.120 –> 00:15:55.700
Jeffrey Palermo: And those need to have teeth. Linters, static code analyzers, and then… and then entropy, management, and that is how to keep the code consistent, how to keep the design patterns, how to keep the architecture, things like checking to make sure that we’re not accidentally just introducing new NuGet packages.
82
00:15:55.700 –> 00:16:00.039
Jeffrey Palermo: If we don’t intend to actually make an architectural shift, do we have
83
00:16:00.060 –> 00:16:14.900
Jeffrey Palermo: Do we have the architectural diagrams of the intended structure in the source control repository so that when there’s a change that’s made, it can be evaluated against the chosen architectural structure?
84
00:16:15.030 –> 00:16:16.410
Jeffrey Palermo: So…
85
00:16:16.720 –> 00:16:33.989
Jeffrey Palermo: when it comes to the focus of a software engineer, we are… we are trying to set up an environment so that a good change can be made to the software with the shortest cycle time as possible, alright? Now.
86
00:16:34.310 –> 00:16:48.970
Jeffrey Palermo: I brought up DHH, Uncle Bob Martin, and the Octopus Deploy Research Team, but they’re really saying the same thing. They’re really saying the exact same thing. And if you look at the different lenses, it’s saying that, you know what?
87
00:16:49.380 –> 00:16:53.350
Jeffrey Palermo: We have the capacity now to make changes much more frequently.
88
00:16:53.450 –> 00:17:01.449
Jeffrey Palermo: We need to have a development environment that will catch problems quickly.
89
00:17:01.580 –> 00:17:02.380
Jeffrey Palermo: Okay.
90
00:17:02.480 –> 00:17:08.699
Jeffrey Palermo: And we need to elevate, we as the actual software engineers, the people in the software team.
91
00:17:08.700 –> 00:17:28.269
Jeffrey Palermo: We need to be able to elevate above that system and design that system and observe that system so that we can constantly fine tune it and see where the bottlenecks are and keep it moving. So different ways to look at it. And I could probably pull any number of people who are really analyzing it and at the forefront of analysis of it.
92
00:17:28.530 –> 00:17:45.410
Jeffrey Palermo: And the same themes are coming out, is that we need to put together this DevOps environment where before it’s been all about CICD and builds and deploy. Builds, deploy, observability. Yeah, those are great.
93
00:17:45.580 –> 00:17:51.139
Jeffrey Palermo: But now… in… with the AI lens, we’re thinking, okay.
94
00:17:51.820 –> 00:17:56.080
Jeffrey Palermo: How many of these steps, how many of these activities can be
95
00:17:56.260 –> 00:18:11.480
Jeffrey Palermo: completely automated, or 80% automated, or how much of the work is compatible with high levels of automation, and how do I make that happen? And given that I have these steps being automated.
96
00:18:11.730 –> 00:18:25.490
Jeffrey Palermo: What type of automated steps can I put into the process so that if there’s a mistake or a problem that happens, I can detect it right there on the spot and not let that change move downstream.
97
00:18:25.710 –> 00:18:28.630
Jeffrey Palermo: As opposed to letting the problem move downstream.
98
00:18:28.890 –> 00:18:38.450
Jeffrey Palermo: So, our pipelines need new stages and new bits of automation, not just… not just…
99
00:18:38.640 –> 00:18:51.449
Jeffrey Palermo: make the old ones faster. And so, I just wanted to kind of show a little bit about, how I see that the AI DevOps environment and some of the concepts that we’re helping our clients with, and
100
00:18:51.750 –> 00:18:55.399
Jeffrey Palermo: And the lens here is, we’re using Git for source control.
101
00:18:56.420 –> 00:19:02.510
Jeffrey Palermo: two different build servers, we’re using GitHub Actions for some of the build servers, we’re using Codefresh for some of the builds.
102
00:19:02.510 –> 00:19:17.560
Jeffrey Palermo: We’re using Octopus for the deployments, we’re using Azure for the environments. Now, that just happens to be what we’re using, what you see, but I mean, these are architectural categories, so do these things work with AWS? Yes. Do these things work with
103
00:19:17.560 –> 00:19:22.620
Jeffrey Palermo: You know, fill in the blank, you know, insert your build server here. Yes, it does.
104
00:19:22.650 –> 00:19:29.170
Jeffrey Palermo: But, you need something in each of these categories, just to have a complete…
105
00:19:32.470 –> 00:19:34.570
Jeffrey Palermo: So, you need… you…
106
00:19:34.980 –> 00:19:46.599
Jeffrey Palermo: some of the… some of the old things, before AI, you absolutely still need these things. You need a build server, you need version control, you need Octopus Deploy for deployments. By the way.
107
00:19:47.200 –> 00:20:03.819
Jeffrey Palermo: In the simple cases, you can use all kinds of tools for deployments, but once you have a lot of pieces of software to deploy, or once you have a lot of environments to deploy to, or a lot of different ways to deploy environments, the configuration management just becomes daunting.
108
00:20:03.820 –> 00:20:09.200
Jeffrey Palermo: And… and you’ll… you’ll start to feel the pain. That’s when… that’s when you know it’s time to graduate up.
109
00:20:09.200 –> 00:20:17.009
Jeffrey Palermo: to Octopus Deploy, just kind of going to a best-of-breed tool that really takes all the complexity and simplifies it down.
110
00:20:17.010 –> 00:20:29.890
Jeffrey Palermo: If you have, you know, a single application, and you’ve used any number of just regular automation tools and are using scripts to just deploy, well, that’s fine for your environment, but at some point, when you say, man.
111
00:20:29.890 –> 00:20:38.760
Jeffrey Palermo: this is just a mess, I got scripts all over the place, or how am I keeping track of environments, or something happens in the environments, and this is… my runbooks are just manual…
112
00:20:38.760 –> 00:20:43.280
Jeffrey Palermo: I wish I could automate… well, that’s when you should look to Octopus Deploy, that just…
113
00:20:43.360 –> 00:20:50.960
Jeffrey Palermo: basically organizes everything. That’s… that’s an aside. And then Azure, we need it… we need our actual hosting environment.
114
00:20:51.190 –> 00:21:09.860
Jeffrey Palermo: And that is Azure. And really, for the modern world, most of our stuff is going to be in the platform-as-a-service, serverless offerings, and then largely more and more, just across everything, is going to Kubernetes, with, with, with…
115
00:21:10.250 –> 00:21:15.429
Jeffrey Palermo: other factors on top. I’m not going to talk deeply about this today, but
116
00:21:16.140 –> 00:21:22.079
Jeffrey Palermo: And I’ll probably do a training dedicated to this at some point in the future. Actually, if you want to send me an email.
117
00:21:22.080 –> 00:21:44.800
Jeffrey Palermo: and just tell me if you’d like to have this put on the schedule. If you’ve never looked at the GitOps pattern, that’s G-I-T-O-P-S, if you’ve never looked at that pattern, then you probably just want to look it up and see what it is, because in a world where more and more changes are going to happen faster and faster, we also need to have a
118
00:21:44.800 –> 00:21:50.830
Jeffrey Palermo: automated DevOps pipeline for incremental changes to our environments.
119
00:21:50.830 –> 00:22:04.379
Jeffrey Palermo: Just like we have incremental… automated incremental changes to our applications. Things as simple as, hey, you know what, I need to go… I need… I need this particular, this particular running process in production to have
120
00:22:04.420 –> 00:22:06.270
Jeffrey Palermo: 3 parallel…
121
00:22:06.270 –> 00:22:30.570
Jeffrey Palermo: instances versus two parallel instances, or we’re looking at some of the logs, and we need to scale up and give a particular container some more CPU, or some more memory, or some more storage. I mean, those are just little tiny environment tweaks that happen all the time, and yes, you can automate them with scripts and Terraform and Bicep and whatnot.
122
00:22:30.740 –> 00:22:38.940
Jeffrey Palermo: But if you don’t have an actual environment to make changes, the hosting environment, that is a software system.
123
00:22:39.140 –> 00:22:45.930
Jeffrey Palermo: And… and so, think about it. GitOps gives you a DevOps pipeline
124
00:22:46.080 –> 00:22:56.989
Jeffrey Palermo: for your infrastructure, for your running infrastructure, so that you can… and you literally have a, I’m getting an aside here, you literally have a Git repository
125
00:22:57.190 –> 00:23:10.430
Jeffrey Palermo: that has, the design of your environments where you can say, hey, this particular… this particular container now needs, you know, two CPUs versus one and a half CPUs equivalent.
126
00:23:10.430 –> 00:23:26.099
Jeffrey Palermo: And you make a commit to that repository, and then that pipeline kicks off, and that change is made in production, zero downtime. And that’s where… that’s for Kubernetes, that’s where Argo CD come in.
127
00:23:26.100 –> 00:23:43.310
Jeffrey Palermo: and those kinds of technologies layered on top. Of course, you know, you’ve got some built-in things with Azure App Service and Azure Container Apps and whatnot. And then observability, we’ve needed observability for a long time, that stays the same. One thing that is absolutely stable
128
00:23:43.590 –> 00:23:45.950
Jeffrey Palermo: Absolutely stable in the AI world.
129
00:23:46.800 –> 00:23:52.130
Jeffrey Palermo: is… small, incremental changes.
130
00:23:52.320 –> 00:24:00.900
Jeffrey Palermo: And there was big industry conversation back in 2001 when the manifesto for Agile software development came around, and one of the big things.
131
00:24:01.180 –> 00:24:09.970
Jeffrey Palermo: was… Being able to continuously change software, and smaller changes, and it was a big idea to say.
132
00:24:09.970 –> 00:24:22.690
Jeffrey Palermo: Our unit of project delivery was gonna go from multiple months down to 1 month, down to 2 weeks, and then, of course, in 2007, there’s a big body of work
133
00:24:22.690 –> 00:24:25.729
Jeffrey Palermo: Led by Daniel Vacanti.
134
00:24:25.730 –> 00:24:38.310
Jeffrey Palermo: around Kanban and you know that that body of work pushed single piece flow and the notion of you know what we ought to be able to make a single change.
135
00:24:38.540 –> 00:24:46.639
Jeffrey Palermo: to the system, and flow that through. We don’t need to group it with anything else, we don’t need to batch it with anything else, we need to be able to make a single change.
136
00:24:47.070 –> 00:24:50.980
Jeffrey Palermo: I predict that that concept, will…
137
00:24:51.510 –> 00:24:57.399
Jeffrey Palermo: Flow to its natural lower limit from a, you know, from a mathematical.
138
00:24:57.590 –> 00:25:08.250
Jeffrey Palermo: limits perspective, and we… we need to have… well, I would say the AI DevOps environment facilitates The…
139
00:25:08.720 –> 00:25:12.490
Jeffrey Palermo: Absolute smallest change possible.
140
00:25:12.600 –> 00:25:17.719
Jeffrey Palermo: So that anytime we decide something needs to be different.
141
00:25:18.190 –> 00:25:28.579
Jeffrey Palermo: there is nothing that forces us to wait on anything else. From as simple as a spelling mistake, if I… if we see there’s some, you know, exported report or a screen.
142
00:25:28.810 –> 00:25:35.380
Jeffrey Palermo: That has a spelling error, We, we should be able to dispatch to our
143
00:25:35.670 –> 00:25:45.580
Jeffrey Palermo: AI DevOps environment that, okay, we… let’s… let’s go ahead and fix this spelling mistake right now. Now, when our teammates
144
00:25:46.230 –> 00:25:47.260
Jeffrey Palermo: are…
145
00:25:47.980 –> 00:26:01.969
Jeffrey Palermo: doing every single branch and every single pull request, well, guess what? That spelling error… that correction has to wait, and it has to wait on when a person is going to be available. It has to wait because
146
00:26:02.400 –> 00:26:17.720
Jeffrey Palermo: our team is a finite constraint with a finite capacity, and so we have to prioritize things, but when we have AI software factory concepts baked in.
147
00:26:17.940 –> 00:26:25.400
Jeffrey Palermo: to our environment, and our DevOps environment is up-leveled to fully support AI, then…
148
00:26:25.540 –> 00:26:31.779
Jeffrey Palermo: If there’s a… if there’s a change, and we have clarity on what it needs to be, for example, that spelling error.
149
00:26:32.210 –> 00:26:43.989
Jeffrey Palermo: Then we define a item, a work item on our work tracking board. Hey, we need to fix that spelling error. We dispatch it into our AI DevOps environment and.
150
00:26:44.600 –> 00:26:55.400
Jeffrey Palermo: it happens. All of the… all of the steps that we would do manually are still done to the same standard, and… and that’s… that’s what…
151
00:26:56.500 –> 00:27:02.089
Jeffrey Palermo: Every software team in the industry has to transition to.
152
00:27:02.640 –> 00:27:10.550
Jeffrey Palermo: Those who don’t… are… are… Going to end up.
153
00:27:11.830 –> 00:27:16.280
Jeffrey Palermo: I hate the word falling behind, because then it’s a comparison.
154
00:27:16.460 –> 00:27:18.780
Jeffrey Palermo: Every business is trying to compete.
155
00:27:18.990 –> 00:27:31.680
Jeffrey Palermo: And every… every business executive team is choosing what to invest in. And I don’t know of a single executive team or chief financial officer that
156
00:27:32.270 –> 00:27:37.970
Jeffrey Palermo: Is happy with the amount of software projects that.
157
00:27:38.060 –> 00:27:49.629
Jeffrey Palermo: is able to be done with the current software investment. And if there’s any option for getting more return on that investment, then it’s going to be taken.
158
00:27:49.680 –> 00:28:04.800
Jeffrey Palermo: And so, I am not predicting that… this is not a cost-cutting thing. This is a competition thing. The leading companies already treat software delivery as a weapon, not as a cost
159
00:28:04.950 –> 00:28:21.110
Jeffrey Palermo: not as a cost-cutting control. Those companies that just treat software as a cost that’s meant to be minimized are not trying to compete based on software. They just have some software, and they’re trying to limp along and keep it barely running, spending as little money as possible.
160
00:28:21.210 –> 00:28:40.329
Jeffrey Palermo: But… but all of you, just by… you’re… you’re coming to… you’re coming to trainings like this, you probably already work for a company that treats software delivery as a competitive weapon, as a sword, not a shield, and so… so your organizations are looking for what is possible, what you can do.
161
00:28:40.340 –> 00:28:48.180
Jeffrey Palermo: And… When you can make, types of changes 100% automated?
162
00:28:48.200 –> 00:28:50.759
Jeffrey Palermo: Then, that’s a competitive advantage.
163
00:28:50.760 –> 00:29:13.079
Jeffrey Palermo: And, yeah, you know what? In 10 years, this is going to be old hat, and of course this is how we do it. Of course it’s all automated. Of course software engineers are in the process of designing software and making sure that we design and deliver software that does the right things, but the mechanics by which we do it
164
00:29:13.080 –> 00:29:19.060
Jeffrey Palermo: are changing, and our DevOps environment has to support these new primitives.
165
00:29:21.030 –> 00:29:31.159
Jeffrey Palermo: So… AI doesn’t plug itself into… into a pipeline in just one place, but it kind of has places
166
00:29:32.020 –> 00:29:39.500
Jeffrey Palermo: everywhere. So, when… Forget the spelling, forget the spelling change.
167
00:29:39.680 –> 00:29:54.850
Jeffrey Palermo: When we plan, yeah, we interactively use the agents. We need to just come up with a plan. And then when we make changes, we need to build and verify. We need to release and promote across environments. Then we need to operate with good telemetry.
168
00:29:54.850 –> 00:29:59.779
Jeffrey Palermo: The OpenTelemetry standard has proven to be a really, really good one so that we know what the software is doing.
169
00:30:00.120 –> 00:30:09.700
Jeffrey Palermo: So, there’s opportunities for leveraging AI automation everywhere. But let me… let me kind of show you some examples. Let me… I have some demos prepared.
170
00:30:11.330 –> 00:30:24.050
Jeffrey Palermo: So, let me… Let’s see here… I’m just gonna go forward… Shown some new architectural elements.
171
00:30:24.430 –> 00:30:30.150
Jeffrey Palermo: And… I’m actually going to… Switch over…
172
00:30:31.820 –> 00:30:35.869
Jeffrey Palermo: Let me switch over to… hold on just a second.
173
00:30:40.890 –> 00:30:46.760
Jeffrey Palermo: Sorry, I need to… my apologies while I switch over to my demo… area.
174
00:30:48.830 –> 00:30:49.620
Jeffrey Palermo: Okay.
175
00:30:51.530 –> 00:30:52.640
Jeffrey Palermo: Let me…
176
00:30:53.180 –> 00:31:05.969
Jeffrey Palermo: Okay, so here’s… I have some demos. So, this is how we approach the opportunity of AI-driven development and what it means to get in place an AI DevOps environment. So, the whole world has gone through the progression
177
00:31:06.000 –> 00:31:13.809
Jeffrey Palermo: of learning and thinking and assisting and monitoring and delegating, and then AI ownership. So, I’m gonna talk about the transition
178
00:31:13.810 –> 00:31:34.209
Jeffrey Palermo: from 5 to 6, where we can literally put in place a system that will take ownership of a change and make it happen. And I’m talking about all the classes. Once you understand exactly what you want to do, which, by the way, there’s a lot of engineering work that goes into figuring out what you want to do.
179
00:31:34.210 –> 00:31:39.749
Jeffrey Palermo: Okay? That’s the hard part! And so you need to figure out what you want to do.
180
00:31:39.970 –> 00:31:57.509
Jeffrey Palermo: Once you’ve figured that out, okay, now computer automation of all kinds comes into… comes into play. But you have to decide what you want to do. This, digital AI brain that is… that is artificial intelligence, it’s still artificial.
181
00:31:57.570 –> 00:32:11.459
Jeffrey Palermo: it… it cannot and it will not decide what you want to do, alright? So, that’s where… that’s where the hard work comes in. But once you decide what to do, there’s 3 different
182
00:32:12.230 –> 00:32:15.810
Jeffrey Palermo: There’s 3 different,
183
00:32:17.710 –> 00:32:19.190
Jeffrey Palermo: Actually.
184
00:32:19.900 –> 00:32:24.600
Jeffrey Palermo: Let me get my… proper screen up.
185
00:32:26.840 –> 00:32:27.620
Jeffrey Palermo: Okay.
186
00:32:28.590 –> 00:32:47.399
Jeffrey Palermo: There’s 3 modes that I want to highlight. One mode is as a thinking partner. We’re all used to that. We have… we have an agent that’s up on the screen, whether it’s Copilot, or Claude, or Codex, or Grok, or Cursor, or Grokbot, or IBM Bob, or any of these things.
187
00:32:47.490 –> 00:32:58.449
Jeffrey Palermo: a thinking partner. We’re interactively using it, researching it, it’s doing really quick internet searches and looking through our documents and all that, and we’ve got it connected with Microsoft 365 and Google Drive and all these things.
188
00:32:58.600 –> 00:33:08.819
Jeffrey Palermo: And our CRM, and we’re… it’s helping us think so that we can… we can decide for ourselves what we want to do. Alright, we’re always… we’re gonna… that’s… I think that’s gonna continue on.
189
00:33:08.820 –> 00:33:26.600
Jeffrey Palermo: And that’s absolutely useful. The other is an interactive agent where once we decide to do something, it’s a little bit complex. We know enough to take a couple of steps forward. And so it’s something that we dispatch using some of the skills that we develop, some of the AI tool skills that we develop.
190
00:33:26.600 –> 00:33:47.370
Jeffrey Palermo: And… but we’re gonna watch it. We’re gonna watch it every step of the way, we’re not gonna quite put it on auto, and we’re gonna kind of shepherd it along, because there’s some bits of ambiguity, and we kind of need to monitor it, so we put it off on the side monitor, and we keep it up, and we call… one of our popular skills is feature loop dispatch.
191
00:33:47.940 –> 00:34:00.059
Jeffrey Palermo: And then another agent is the factory monitoring agent, and… or you could call it your AI DevOps monitoring agent. And that is an agent that’s operating
192
00:34:00.170 –> 00:34:03.069
Jeffrey Palermo: A completely automated flow.
193
00:34:03.500 –> 00:34:05.020
Jeffrey Palermo: that… that has
194
00:34:05.310 –> 00:34:24.319
Jeffrey Palermo: That has some connections to your work tracking board, your builds, your deployments, your observability data syncs, your, you know, application insights, and… and you can say… you can just create a work item for fixing that spelling error, and you can just say, go.
195
00:34:24.320 –> 00:34:35.520
Jeffrey Palermo: And if it’s something simple, and you feel confident that it can auto-merge, even, then you could label it for auto-merge, and if it passes all of the builds, if it passes all of the gates.
196
00:34:35.889 –> 00:34:54.500
Jeffrey Palermo: then it’s good to go, flow it all the way through, and queue it up for production deployment. Or you could identify it as, you know what, I want to go all the way to pull request, and then wait there, let me put eyes on it before the merge, or anywhere in between. Totally, you know, totally your choice, because you’re going to have work items
197
00:34:54.500 –> 00:34:57.690
Jeffrey Palermo: That are all kinds of different levels and size.
198
00:34:58.070 –> 00:35:14.079
Jeffrey Palermo: But, there’s this automation that is going, and your factory monitoring agent is just… is just making sure that that continues to go. And so, let me… within GitHub projects, here is a…
199
00:35:14.890 –> 00:35:30.650
Jeffrey Palermo: Here’s a quick demo that I’ll play, and this is what it looks like. Now, I have sped up this big time, because it, you know, slow demos are boring, but we just dispatched some work items, using the
200
00:35:30.650 –> 00:35:33.010
Jeffrey Palermo: Factory loop dispatch, even interactively.
201
00:35:33.010 –> 00:35:52.400
Jeffrey Palermo: And we can see that, you know, going through UX design and technical design and test design, those happen pretty quickly, and then development takes longer because there’s more going on. There’s more branching and coding and running the private build and running tests, and oh, there was a test failure, or there was a static code analysis failure. So that tends to happen. That’s what…
202
00:35:52.400 –> 00:36:03.050
Jeffrey Palermo: That type of experience where things are just moving across the board because you have the automation happening that is a part of the AI DevOps environment.
203
00:36:03.620 –> 00:36:10.720
Jeffrey Palermo: Now, I do want to, do want to show… I do wanna…
204
00:36:11.470 –> 00:36:17.510
Jeffrey Palermo: Show one other concept that, that’s happening here.
205
00:36:18.220 –> 00:36:21.670
Jeffrey Palermo: Let’s see here, let me fast forward a little bit…
206
00:36:22.710 –> 00:36:32.160
Jeffrey Palermo: Okay, right about here, I’m fast-forwarding it. It was working on development, and there’s another work item that just pops up in conceptual definition.
207
00:36:32.260 –> 00:36:38.990
Jeffrey Palermo: One of the elements that I contend needs to be a part of any AI DevOps environment is a
208
00:36:39.090 –> 00:36:41.389
Jeffrey Palermo: self-improvement routine.
209
00:36:41.560 –> 00:36:49.559
Jeffrey Palermo: And so, in this, in this, sample environment, of our, our AI software factory.
210
00:36:49.580 –> 00:37:03.250
Jeffrey Palermo: set up, and whether the right term is AI DevOps, or an AI software factory, or some other word that comes out of the blue, it’s the concepts that matter. That we’re gonna have automation
211
00:37:03.250 –> 00:37:11.229
Jeffrey Palermo: And we’re gonna be doing a lot of builds, running the tests a lot, and so one of the routines, every time
212
00:37:11.600 –> 00:37:27.989
Jeffrey Palermo: every time there’s a new… at certain stages, and so I’ve got it configured every time there’s a new pull request, every time there’s a new pull request merged, I have a separate AI session do an analysis of the logs, and suggest opportunities
213
00:37:27.990 –> 00:37:40.379
Jeffrey Palermo: for reducing the cycle time of the various builds and automated checks and the pull request cycle. And so there was one that was identified and it…
214
00:37:41.570 –> 00:37:46.530
Jeffrey Palermo: And… No, actually, that’s not the one. Let me go back.
215
00:37:48.250 –> 00:37:50.389
Jeffrey Palermo: I think I had the wrong video, hold on one second.
216
00:37:50.610 –> 00:37:51.569
Jeffrey Palermo: Let me get the right one.
217
00:37:54.300 –> 00:37:56.099
Jeffrey Palermo: Last name field…
218
00:37:58.810 –> 00:38:08.580
Jeffrey Palermo: There we go, this is the one, okay. So, sorry, you’re probably confused. So, a work item pops up, and it’s called Build Workflow Concurrency Groups Are Branch Scoped.
219
00:38:08.990 –> 00:38:22.679
Jeffrey Palermo: Canceling in-flight master runs during merge bursts. Okay, so in this case, we dispatched 10 different work items at the same time, and so you think if there’s 10 people trying to merge to master at the same time, there’s going to be contention.
220
00:38:22.680 –> 00:38:38.609
Jeffrey Palermo: And so, in this case, we had some branch scoping in our workflow rules, and it was causing some of the in-flight builds on master to be canceled. And so, here’s a suggested enhancement by analyzing the logs, and
221
00:38:38.610 –> 00:38:53.499
Jeffrey Palermo: And you can set up your own system for high, you know, high confidence approvals versus ones that, oh yeah, it’s a slam dunk, it’s a low-risk thing, go ahead and develop it. But this enhancement is put on the board just as a regular work item, and then the
222
00:38:53.500 –> 00:39:02.429
Jeffrey Palermo: AI DevOps environment, the AI software factory portion of it, runs it through the same process, because it’s going to be committing to that repository.
223
00:39:02.430 –> 00:39:09.409
Jeffrey Palermo: And it’s gonna work it, it’s gonna create a pull request, and it’s gonna make sure that the pull request works just the same, and…
224
00:39:09.610 –> 00:39:12.290
Jeffrey Palermo: Then it’s gonna… gonna move it along, okay?
225
00:39:12.860 –> 00:39:26.280
Jeffrey Palermo: All right, so that’s the self-improvement cycle. This is a screenshot of the actual work item that was identified from that self-improvement session, and I kind of showed you the…
226
00:39:26.860 –> 00:39:29.009
Jeffrey Palermo: Showed you the,
227
00:39:30.860 –> 00:39:44.900
Jeffrey Palermo: process. But basically, when you run through things, you want to come back to the board, and you want to see a whole bunch of items that are finished, that are ready for whatever you define as the personal step.
228
00:39:44.900 –> 00:39:56.160
Jeffrey Palermo: And so in this case, I’ve got it configured to do everything necessary all the way to the UAT environment, and let’s leave these in the user experience testing column, meaning
229
00:39:56.160 –> 00:40:07.839
Jeffrey Palermo: They’re all… they’re all merged to master, we’ve got… we’re deployed to the UAT environment, and now they’re in UX testing because we need to set our eyeballs on them to make sure that we’re happy to move them forward.
230
00:40:07.840 –> 00:40:17.809
Jeffrey Palermo: Alright? Now, if you want certain… a certain class of item to go all the way to production once you get a little bit mature, or if you want it to stop at the pull request stage, that’s totally up to you. Totally up to you.
231
00:40:18.310 –> 00:40:34.150
Jeffrey Palermo: All right. And, oh, by the way, as we’re going along, you can go ahead and put some questions in the chat, and I’ll come back to it. We’re almost, we’re almost to the point of discussion. But, another aspect that you’re going to want in an AI DevOps environment is
232
00:40:34.850 –> 00:40:36.700
Jeffrey Palermo: Some tools to know.
233
00:40:37.010 –> 00:40:47.329
Jeffrey Palermo: What a feature was, or… or some quick way to eyeball a feature to know that you’re happy with it without having to
234
00:40:47.870 –> 00:40:54.999
Jeffrey Palermo: Check out the branch locally, spin up the entire app, get to the proper screen, and so…
235
00:40:55.380 –> 00:41:13.320
Jeffrey Palermo: it… in our automation somewhere, we are going to want to… to shortcut this. Here’s an example of how it might be shortcutted. Alright, so we have a work item to add to our sample work orders application. I want a sortable… I want some sortable column headers.
236
00:41:13.340 –> 00:41:29.610
Jeffrey Palermo: And so, we have a work item that’s, that is sortable column headers. Let me play this quick video. And we’re gonna scroll down, and we can see that the AI Software Factory has done a lot of work on this work item, and it’s completely implemented.
237
00:41:29.610 –> 00:41:37.470
Jeffrey Palermo: And by the way, we use all the different AI tools, and so you’ll see that we have, you know.
238
00:41:37.700 –> 00:41:41.309
Jeffrey Palermo: IBM Bobs, the AI tool from IBM.
239
00:41:41.420 –> 00:41:45.680
Jeffrey Palermo: In this particular example, but at the bottom.
240
00:41:46.300 –> 00:41:58.189
Jeffrey Palermo: At the bottom, we have a step that we’ve identified that’s a demo video, and we have a link to a video where if we click on that link, then we can
241
00:41:58.190 –> 00:42:09.690
Jeffrey Palermo: we can literally see that feature in action, and we can download it. See, I’ve downloaded it in the browser, and I can pop it up. So, I am going to,
242
00:42:09.970 –> 00:42:14.410
Jeffrey Palermo: Let’s see here… I’ve already done that…
243
00:42:14.560 –> 00:42:21.839
Jeffrey Palermo: The next slide is the video. What I’m gonna do, I’m gonna stop share and reshare with audio, because I want you to hear
244
00:42:22.210 –> 00:42:30.019
Jeffrey Palermo: I want you to hear what this does. So I’m gonna reshare with audio and share the sound.
245
00:42:30.410 –> 00:42:40.760
Jeffrey Palermo: There we go… and screen number 2… Okay, if you turn on your speakers… well, your speakers are already on… and let’s just listen to some of this.
246
00:42:44.150 –> 00:42:51.249
Audio shared by Jeffrey Palermo: Issue 9524 adds sortable column headers to the work order list. Title and room have been promoted from plain text to column headers.
247
00:42:52.770 –> 00:42:53.790
Jeffrey Palermo: Oh, let me keep going.
248
00:42:53.790 –> 00:42:58.309
Audio shared by Jeffrey Palermo: to clickable sort buttons, matching the existing status and due date controls.
249
00:42:58.960 –> 00:43:10.700
Audio shared by Jeffrey Palermo: Clicking the title header for the first time sorts the list ascending The header highlights in blue and shows an up arrow Here, Alpha appears before Zebra, and the new room column is visible on the right
250
00:43:11.420 –> 00:43:14.429
Audio shared by Jeffrey Palermo: A second click reverses the order to descending.
251
00:43:14.810 –> 00:43:28.309
Jeffrey Palermo: Okay, I’m gonna stop that real quick. So, now obviously this… the environment that this AI session was using is a smaller screen real estate, so it doesn’t… it’s not 1080p, obviously, but it’s enough
252
00:43:28.310 –> 00:43:36.449
Jeffrey Palermo: for us to see that, oh, okay, the title, that’s what that little… the down arrow, the up arrow. Okay, it gives… and it’s…
253
00:43:36.490 –> 00:43:41.080
Jeffrey Palermo: It’s there so that it can give me the confidence that it actually
254
00:43:41.410 –> 00:43:46.240
Jeffrey Palermo: was implemented where, if I’m happy with it, for example, if it was a spelling mistake.
255
00:43:46.350 –> 00:43:50.799
Jeffrey Palermo: This tactic would give me just enough confidence to say, yeah, I’m good, I don’t need to…
256
00:43:51.190 –> 00:44:01.590
Jeffrey Palermo: do any more testing. Or, it gives me enough information for me to say, okay, that’s 80% there, but you know what? We still need a little tweak.
257
00:44:01.610 –> 00:44:12.099
Jeffrey Palermo: before I’m actually happy with it. But either way, it’s a shortcut for you to do. And by the way, creating this video is just a prompt, and it’s totally automated.
258
00:44:12.100 –> 00:44:22.760
Jeffrey Palermo: So, you know, you can make them as long or short, you know, we’re not only title, but also the room column, so it’s showing the room column, alright? So…
259
00:44:22.910 –> 00:44:25.310
Jeffrey Palermo: Things like this, you want to shorten the cycle.
260
00:44:25.310 –> 00:44:25.890
Audio shared by Jeffrey Palermo: 95.
261
00:44:25.890 –> 00:44:27.070
Jeffrey Palermo: As much as possible.
262
00:44:27.100 –> 00:44:37.489
Jeffrey Palermo: Another is ephemeral Environments and our DevOps environment, we added a screen to our AI software factory where
263
00:44:37.490 –> 00:44:49.690
Jeffrey Palermo: for any release number that is not yet in production, I can just click on that release number, click the open, and on demand, it will give me an environment that is running
264
00:44:49.700 –> 00:45:05.260
Jeffrey Palermo: running that particular exact version number. And, you know, under the covers, this environment is… with the cloud environments, or with Kubernetes, you have plenty of flexibility on the infrastructure side, but you can click on open, and you’re gonna see
265
00:45:05.350 –> 00:45:23.340
Jeffrey Palermo: you’re gonna see the environment running. Or, if you’ve done some things and you want to reset the environment because you need to reset the test data, well, there’s a reset button. And any number of people can be testing any number of different version numbers in parallel. In the age of AI-driven development.
266
00:45:23.340 –> 00:45:27.639
Jeffrey Palermo: We have to obliterate the constraint of named.
267
00:45:27.640 –> 00:45:37.860
Jeffrey Palermo: test environments. We have to have as many parallel test environments as we need to in order to validate things with no blockages, with no waiting.
268
00:45:38.210 –> 00:45:41.640
Jeffrey Palermo: Okay, let me… gonna play another demo video here.
269
00:45:41.810 –> 00:45:46.479
Jeffrey Palermo: And so, we’re gonna… we’re going to,
270
00:45:47.130 –> 00:45:53.649
Jeffrey Palermo: use this work orders application. This is the manual way to work the application, but…
271
00:45:53.700 –> 00:46:01.969
Jeffrey Palermo: Every application can have a chatbot inside, and we can type in certain things, like, hey, Gertie needs to clean the organ.
272
00:46:01.970 –> 00:46:14.669
Jeffrey Palermo: And this AI chatbot inside the application will be using the application’s own MCP server interface in order to do multiple transactions. Let me forward along the…
273
00:46:15.240 –> 00:46:30.819
Jeffrey Palermo: Let me fast forward. There we go. And so we can see that multiple work orders were created with particular due dates where how many clicks would it have taken the user to create 10 different work orders? Okay, so that’s something in an AI driven world.
274
00:46:31.060 –> 00:46:35.820
Jeffrey Palermo: There’s all kinds of things that require multiple transactions. Well.
275
00:46:35.860 –> 00:46:52.030
Jeffrey Palermo: by using the MCP server concept and embedding chat agents directly in the application, we can… it’s a new usage paradigm. And of course, we can see that we have a lot of brand new Mow the Grass assigned to Groundskeeper Willie.
276
00:46:52.030 –> 00:46:59.860
Jeffrey Palermo: Nate Gray, with those due dates. So those transactions, those work orders were automatically created. How many… how many clicks did it save us?
277
00:46:59.910 –> 00:47:06.089
Jeffrey Palermo: Now, that’s… that’s an interesting… that’s an interesting aspect of the…
278
00:47:09.270 –> 00:47:16.630
Jeffrey Palermo: application itself, but… When the functionality of our application exposes.
279
00:47:16.740 –> 00:47:26.590
Jeffrey Palermo: And… interface that is accessible by an AI tool. Yes, we can put a chat interface directly in the UI of the application.
280
00:47:26.920 –> 00:47:29.830
Jeffrey Palermo: But when we have that API service.
281
00:47:30.560 –> 00:47:43.599
Jeffrey Palermo: we have options. If some of our customers are used to using their own AI tools, it literally doesn’t matter. They could be using Copilot, Codex, Grok, Claude, IBM Bob.
282
00:47:43.790 –> 00:47:49.460
Jeffrey Palermo: Grokbot, and they could be using Pi, they could be using.
283
00:47:49.550 –> 00:48:05.000
Jeffrey Palermo: open code, you know, any of the harnesses, and guess what? They can just register our application as an MCP server, and use our application with their chosen interface. Why? Because their chosen interface
284
00:48:05.000 –> 00:48:13.529
Jeffrey Palermo: is integrated with their organization’s data. Think Microsoft 365, their email, their calendar, their Google Drive.
285
00:48:13.810 –> 00:48:14.980
Jeffrey Palermo: And now.
286
00:48:15.560 –> 00:48:26.510
Jeffrey Palermo: We’ve allowed our application to be just as easily integrated into their workflow as… as other software as a service providers.
287
00:48:26.530 –> 00:48:41.939
Jeffrey Palermo: And so, when we do that, then our users get used to using our application more on an automated basis. It doesn’t… we don’t care if they’re using our interface, we really don’t. We care that they’re using our asset.
288
00:48:42.280 –> 00:48:49.980
Jeffrey Palermo: The software system is the asset. The user interface is just what we had to build over the last
289
00:48:50.130 –> 00:49:05.239
Jeffrey Palermo: multiple decades in order to allow the user to interface with the system. User interface. Well, now we have more options for building an interface to the user. And if you think about it.
290
00:49:05.600 –> 00:49:21.080
Jeffrey Palermo: How many applications have you already stopped using and instead you are using your own AI tool? I used to use weather apps. You think I’m ever going to use a weather app again? No.
291
00:49:21.150 –> 00:49:33.869
Jeffrey Palermo: I’m not! So, an MCP… an MCP API is for when you need it available Over… over the,
292
00:49:34.000 –> 00:49:37.679
Jeffrey Palermo: Over the internet, and you don’t have the opportunity to…
293
00:49:37.680 –> 00:49:51.749
Jeffrey Palermo: install anything locally on the user’s computer. If you do have the opportunity to install something on the user’s computer, well, this is where a command line interface client comes in, because you can provide a whole lot more utility
294
00:49:51.750 –> 00:50:00.330
Jeffrey Palermo: If you can have a little bit of code running on the user’s computer, and that’s why that’s why command line clients exist. So if you can do that, then that’s a whole nother level.
295
00:50:00.540 –> 00:50:09.630
Jeffrey Palermo: For DevOps, we’ve always had, nested cycles, where, you know, the coding, the code, the code run cycle.
296
00:50:09.630 –> 00:50:29.149
Jeffrey Palermo: has to be the fastest, and then the design-to-test cycle is the next fastest. These are concentric circles, concentric cycles. The outer cycles will never go faster, will never have a shorter cycle time than the cycles in between. So that continues to be the case. And if we want our… if we want
297
00:50:29.150 –> 00:50:42.130
Jeffrey Palermo: to leverage AI automation with the coding, then our DevOps process for the minimum cycle time of our builds and testing has to speed up. It has to speed up. A 10-minute build.
298
00:50:42.440 –> 00:50:57.189
Jeffrey Palermo: that’s gonna hold us back. I mean, think about it. If AI can make, you know, a dozen changes, but then I have to wait a dozen times the build, because builds have to run one after the other, or have some contention.
299
00:50:57.490 –> 00:50:59.559
Jeffrey Palermo: We just have to think about that, okay?
300
00:50:59.790 –> 00:51:15.839
Jeffrey Palermo: We also need to measure. When we are going to be generating and validating changes a lot faster in our AI DevOps environment, we have to measure. And what I’m showing you on the screen is just… is our Clear Measure Intelligence Scorecard tool that we use, that we use.
301
00:51:16.050 –> 00:51:27.739
Jeffrey Palermo: for our own internal measurement, and a lot of status reports of… let me stop here. A lot of status reports or reports from work tracking
302
00:51:27.830 –> 00:51:34.040
Jeffrey Palermo: Systems start at the work item, and then you can trace through the history.
303
00:51:34.080 –> 00:51:47.989
Jeffrey Palermo: It’s a much… it’s a much more natural way to report status when you start at the number, and you know, you know what? We had this many items of new work come in this week, and we had this many items of
304
00:51:48.290 –> 00:51:50.189
Jeffrey Palermo: This many items…
305
00:51:50.440 –> 00:51:56.240
Jeffrey Palermo: of work that were delivered. And then you got mean time to delivery, and other things.
306
00:51:56.240 –> 00:52:10.459
Jeffrey Palermo: and then be able to click on that number and say, well, what work items were they? And that’s where we click on a number, and then we have multiple work items, and this happens to be connected to GitHub projects at the moment, and you can see a whole bunch of sample
307
00:52:10.630 –> 00:52:29.739
Jeffrey Palermo: changes. It’s really easy to run through an API change. And we can click on one, and 9118, and we click on it, and it goes straight over to the GitHub issue. And you can, if it’s Azure DevOps, you can click on straight over there, it’s just a URL. And then we can, we can look at everything that’s happened, in that
308
00:52:29.920 –> 00:52:39.740
Jeffrey Palermo: in that work item. So, you need to have that visibility and traceability from your metrics, from your numbers, back to the source item so that you can really understand what’s going on.
309
00:52:40.780 –> 00:52:44.969
Jeffrey Palermo: And you really want to be able to trend
310
00:52:45.080 –> 00:53:01.529
Jeffrey Palermo: what your team does over time. And this is our version that’s inside Microsoft Excel, so you don’t have to install anything, you literally just install a… it’s a ribbon bar on Excel, and then it pulls straight from Azure DevOps
311
00:53:01.660 –> 00:53:12.890
Jeffrey Palermo: into your Excel spreadsheet, and then you can see and trend over time, then you can use Excel charts. And it’s a really easy way to get started with, if you’re using Azure DevOps.
312
00:53:12.910 –> 00:53:22.300
Jeffrey Palermo: It’s a real easy way of getting started with measuring how much new work comes in per week, and how much goes out. You can also do it by sprint, if you want.
313
00:53:22.350 –> 00:53:33.210
Jeffrey Palermo: But that’s just you. The point is, in an AI DevOps environment, you have to have a good way to know what your metrics are. Okay.
314
00:53:34.380 –> 00:53:36.150
Jeffrey Palermo: Alright, and…
315
00:53:40.940 –> 00:53:49.160
Jeffrey Palermo: Alright, so… New architectural elements, and we’ll close up here. New architectural elements for an AI DevOps environment.
316
00:53:49.280 –> 00:53:52.799
Jeffrey Palermo: I wanna… I wanna highlight some.
317
00:53:52.960 –> 00:53:58.110
Jeffrey Palermo: And… Just a second… okay.
318
00:53:59.290 –> 00:54:03.720
Jeffrey Palermo: First is your agent identity.
319
00:54:04.780 –> 00:54:14.229
Jeffrey Palermo: These AI tools make it really easy to authorize integration with something using my own username or tokens from my own accounts. As we build this automation.
320
00:54:14.750 –> 00:54:22.460
Jeffrey Palermo: We’ve learned over the years that when we put a piece of software in production, or any non-production environment to run.
321
00:54:22.590 –> 00:54:34.480
Jeffrey Palermo: we give it a service account. We give it its own identity. And we give that service account, that identity, that token, just the permissions that are necessary for that piece of software to run.
322
00:54:35.130 –> 00:54:41.870
Jeffrey Palermo: These… automated… AI agents are the same thing.
323
00:54:42.010 –> 00:54:52.870
Jeffrey Palermo: They are pieces of software that are running. We need to give them service accounts, we need to give them their own identity, and we need to give that account just the
324
00:54:52.870 –> 00:55:07.749
Jeffrey Palermo: just the permissions that are necessary for it to run. That’s really, really important. So, even though you might have tested something and got it working with your identity, you know, you don’t want the AI tool to exercise all of the authority
325
00:55:07.770 –> 00:55:10.060
Jeffrey Palermo: of your identity.
326
00:55:10.420 –> 00:55:14.699
Jeffrey Palermo: Alright, then… Then there’s the harness, and by the way.
327
00:55:14.710 –> 00:55:26.129
Jeffrey Palermo: We’ve tested every AI tool out there, and just about all of them will work for different unattended, unattended automation
328
00:55:26.130 –> 00:55:39.250
Jeffrey Palermo: tasks. And so, it’s about making sure that the Git repositories have enough context to be automatically discovered by the AI session that you’re wanting to do something productive.
329
00:55:39.390 –> 00:55:47.489
Jeffrey Palermo: And so, I don’t like the term context engineering. However, we have to make it… we have to.
330
00:55:47.490 –> 00:56:03.869
Jeffrey Palermo: make it easily discoverable by the AI tool enough information so that the right… so that the right work is done. And then verification gates, your builds, they’re… they were… they have been important before AI, and they are even more important now.
331
00:56:04.260 –> 00:56:13.670
Jeffrey Palermo: static code analyzers and tests. If you were a little bit lukewarm on test-driven development and having full automated test coverage.
332
00:56:14.940 –> 00:56:22.559
Jeffrey Palermo: These things are now prerequisites for AI-driven development. They’re absolutely prerequisites.
333
00:56:22.810 –> 00:56:39.450
Jeffrey Palermo: And then delivery scorecard. If you’re going to be doing… if you’re going to be automating some things, and doing more of some things, you want to know where your bottleneck is, and where your bottleneck is moving. And so you have to measure. And then finally, the human final gate.
334
00:56:39.880 –> 00:56:41.520
Jeffrey Palermo: you want to…
335
00:56:41.630 –> 00:56:57.640
Jeffrey Palermo: You want to decide where you want to set your eyeballs on. You can’t set your eyeballs on everything, and so you have to make that decision. For each type of work, where do you want the automation to stop and wait for somebody? And for other types of work.
336
00:56:58.040 –> 00:57:09.610
Jeffrey Palermo: Do you want to have that stop and wait for somebody be at a different place? For the spelling error, you know, maybe you want it to go all the way to the release queue and have a person just glance at it before going to production.
337
00:57:09.610 –> 00:57:24.569
Jeffrey Palermo: For something that’s, you know, brand new screens, brand new database tables, maybe you want the stop and wait for a person’s eyeballs at the pull request before it’s merged into master. But you have to make those decisions, because not all changes are created equal.
338
00:57:24.570 –> 00:57:25.330
Jeffrey Palermo: All right.
339
00:57:26.420 –> 00:57:44.609
Jeffrey Palermo: And by the way, regardless of these decisions we make, the only thing that matters is getting new capabilities out to our customers. Because anything else is work in process, and we know that WIP is waste. And so until it’s actually delivering value to our customers.
340
00:57:45.680 –> 00:57:50.140
Jeffrey Palermo: We haven’t… there’s no opportunity to get a return on that investment.
341
00:57:51.510 –> 00:57:52.340
Jeffrey Palermo: Okay.
342
00:57:52.680 –> 00:58:00.960
Jeffrey Palermo: So, let me… Go on… Okay.
343
00:58:01.260 –> 00:58:08.279
Jeffrey Palermo: There are… let’s see here… Go back…
344
00:58:16.430 –> 00:58:25.269
Jeffrey Palermo: Hold on a second, let me… this off… Yes, this is the final.
345
00:58:29.680 –> 00:58:30.440
Jeffrey Palermo: Okay.
346
00:58:31.050 –> 00:58:37.719
Jeffrey Palermo: Sorry about that, just getting the right… moving back to the other slide deck. Okay, so, follow-up.
347
00:58:37.730 –> 00:58:55.439
Jeffrey Palermo: Any questions anyone has about this, I’m here to help. Our mission is to empower software delivery, and we have a training coming up, our .NET AI Boot Camp, coming up February 1st through 3rd, 2027, so that’s 3, 4 months away.
348
00:58:55.560 –> 00:59:01.590
Jeffrey Palermo: February 1st through 3rd, you can go to our website, and Kaylee’s gonna put a link to that in the chat.
349
00:59:01.960 –> 00:59:13.760
Jeffrey Palermo: And, sign up for it. We keep it small, it is for experienced software engineers, and, our last one had, had…
350
00:59:14.160 –> 00:59:33.550
Jeffrey Palermo: the third day completely dedicated to AI, now everything is AI. All of the same… all of the same principles and engineering tactics, but we are in the AI-driven world. And so, that is… that is the new normal. We have to be able to do a good job
351
00:59:33.640 –> 00:59:43.690
Jeffrey Palermo: And we have to deliver valuable software as fast as possible. By the way, if as fast as possible means that it’s broken, then it’s not valuable software.
352
00:59:43.810 –> 00:59:52.659
Jeffrey Palermo: So, quality is first, valuable software, and then as fast as possible, while keeping the software valuable and not breaking it.
353
00:59:52.880 –> 00:59:54.789
Jeffrey Palermo: So there’s a balance there.
354
00:59:55.440 –> 01:00:10.009
Jeffrey Palermo: But quality… quality is paramount, and then, and then tactics for… for shortening cycle time. So, send me an email, I love doing that. Also, if… most of you probably already listened to my AI DevOps podcast that comes out every week.
355
01:00:10.450 –> 01:00:18.639
Jeffrey Palermo: You can find it anywhere you get your podcasts. And I think I answered the questions in line.
356
01:00:18.760 –> 01:00:24.450
Jeffrey Palermo: So, this is a… this is a… this is a big topic. I know it’s a big topic.
357
01:00:24.660 –> 01:00:37.579
Jeffrey Palermo: Another announcement, if you lead a software team, if you’re a software architect, and you lead other software engineers, every month we have a AI architect forum. It’s literally a meeting, it’s a peer group.
358
01:00:37.580 –> 01:00:44.929
Jeffrey Palermo: an architect user group of sorts, and that’s on our website. So, if you know someone who would love to.
359
01:00:44.930 –> 01:00:56.819
Jeffrey Palermo: to have a peer group of other software architects who are running software projects and leading other software engineers, then that could be for you, and that’s on our website. So.
360
01:00:56.920 –> 01:01:12.500
Jeffrey Palermo: Thank you very much. Thank you for those of you who kind of streamed in some questions, just to fill in some gaps, I always appreciate that. And send me an email with any questions, or if you want me to set my eyes on what you’re doing, and maybe give you some pointers.
361
01:01:12.500 –> 01:01:18.799
Jeffrey Palermo: I love doing that. You don’t… you don’t have to sign a contract with us just to get me to look at something.
362
01:01:18.800 –> 01:01:27.609
Jeffrey Palermo: I actually love… I love, I love, kind of, bouncing ideas off of other people that are doing really interesting things, so…
363
01:01:30.240 –> 01:01:35.400
Jeffrey Palermo: All right, well… Thank you so much for attending the training, and
364
01:01:35.550 –> 01:01:36.970
Jeffrey Palermo: Have a… have a great day.