Agile Conversations with Jeffrey Fredrick

In this episode of Semaphore Uncut, we welcome Jeffrey Fredrick, Co-Author of Agile Conversations, internationally recognized expert in software development.

🎧 What we talked about:

• How CITCON Conference was born
• How to bring business metrics to microservices
• Technology adoption lifecycle
• Is Agile dead?
• The art of productive conflict and the Four R’s framework

Listen to our entire conversation in this podcast episode, and check out my favorite parts in the episode highlights!

You can also get Semaphore Uncut on Apple Podcasts, Spotify, Google Podcasts, Stitcher, and more.

Edited Transcript

Darko (00:01): Hello, and welcome to Semaphore Uncut, a podcast for developers about building great products. Today, I’m excited to welcome Jeffrey Fredrick. Jeffrey, thank you so much for joining us.

Jeffrey (00:11): And thank you for having me here, Darko.

Darko (00:13): Great. Please, just go ahead and introduce yourself.

Jeffrey (00:16): Sure. I think most people who know me would know me either from the podcast that I have with Douglas Squirrel. We have a podcast called Troubleshooting Agile and we’re co-authors of a book called Agile Conversations. People who are a bit older and have been in the industry longer, might know me from my work on CruiseControl, which was one of the early open source continuous integration tools. And also I’ve been the co-organizer of continuous integration and testing conference, CITCON. So I think those are the things that people would most know me for.

How CITCON Started

Darko: Great. And before we move to what you’re working on right now, maybe we can take a step back and maybe talk a little bit about CITCON and potentially even before that, what led you to working on CruiseControl?

Jeffrey: So I actually looked for a role where I could build that kind of system at a company, and I found a couple of companies that were interested in that and I went to one of them. Before I started building the system, though, I looked around to see if there was already an open source project that I could leverage, and that’s when I found CruiseControl, so this is 2001. I became an active contributor and later committed there. And that’s pretty much directly what led to my working on the CITCON Conference that was founded by myself and PJ, PJ being the originator of the CruiseControl project, one of the founders of the project.

Jeffrey: When it was coming up to the five-year anniversary of CruiseControl, and just remember, this is the very, very early days of automated CI. And we were all very much figuring out how to make the most of this kind of tooling, this kind of approach. He approached me to say, “Would you be interested in starting a conference on this? And we invite everyone else who was involved in CI tooling at the time.” And so we did that. We invited a bunch of people who were leading other open-source projects at the time, largely open-source, but there was also some commercial people there. So people who were using maybe cruisecontrol.net or Damage Control, which very few of your listeners will have heard unless they’re old like I am. We also invited people from a company called Urban Code that made one of the first commercial tools called Anthill, and they had a professional version of Anthill Pro. So we invited them.

Jeffrey: So it started as kind of like an inside baseball who were the really early people involved in continuous integration and the related field of testing. So we also invited people who were kind of skeptics of this automated approach. And so we invited, for example, James Shore, who people might know from his Art of Agile book, or his videos on how to do unit testing. He’d written a blog called “Why I Hate CruiseControl”. We thought that was too good of an opportunity to pass up, so we invited him. I say invited because the format of the conference was very different than a traditional conference. My view is that conferences really should be for conferring, they should be for talking with your colleagues. But I think a lot of conferences you kind of show up and it’s a bunch of talking heads in a room and there’s very limited time to actually interact with people.

Jeffrey: So sometimes people say, like the best conversations, the best part of a conference might be the hallway conversations. So PJ had the idea, he said “I could have the conference in what’s called an open space style, also known as an unconference.” And this is something he had had firsthand experience with. Bruce Eckel and Martin Fowler had run a series of conferences in open space format. And PJ had gone to one of these and was just blown away by the open space format. It’s a fantastic way of connecting people in the community, and I think that’s one of things I’ve been most excited with about the CITCON Сonference.

Darko: I can contribute to this. So recently CITCON was in virtual format, and is still running. I had the pleasure to attend a couple of years ago and I was blown away by the format, and for the people to just maybe visualize so Jeffrey and PJ make all the attendees sit in a circle, introduce yourself. There is a whiteboard; you write down on sticky notes what are the topics, and then there are people who group them together and an agenda for the next day is formed.

What Is End-to-End Continuous Integration?

Darko: Moving from here, there are a couple of things that we touched upon that you’re really interested in. One of the things being end-to-end CI/CD. Can you maybe elaborate on that, how you have seen that work in practice or made that work in practice, and introduce us to the idea?

Jeffrey: Sure. So end-to-end continuous integration was a term we kind of came up CITCON, probably around 2007-2008. What would happen is people have been doing continuous integration for a few years now, and the scope of what people brought into their CI tools had grown. Originally, it might be just compiling the code and then compiling in unit testing, and then adding functional tests, and adding code coverage tools, and adding stack analysis. And people were asking, “Well, how far can we go with this? What would be the limit?” And some people would begin experimenting with, “well, actually we will deploy from our CI system out to production.”

Today the idea of automated deployment from your build tool is not uncommon, people expect that from CI. But that was a new thing back at that time, it was kind of being pioneered. There’s a connection between this end-to-end continuous integration and DevOps. In fact, Patrick Debois attended CITCON the year before he started the DevOps Days series of conferences. People were trying to solve similar problems, how do we get everything automated and how do we have everything in the system where we have the reproducibility and also the visibility.

For me, a real touchpoint might be the work done at IMVU. Timothy Fitz wrote a blog post called “Doing the Impossible Fifty Times a Day”, where they talked about how they went from a commit to having something running in production, and not just running, but also being monitored. So they would do a partial deployment, and then monitor the business metrics, the health of the cluster, and if after being deployed the metrics looked good on that cluster they would then continue the deployment out to the full system. But if things didn’t look good, they would stop the rollout so they could investigate to figure out what went wrong. And that, I thought, was a really great idea. Beyond even what most people do today, I think that a lot of times people have lost this idea of actually having monitoring tied in as part of your deployment pipeline.

And even that term, the deployment pipeline, is one that has developed over time with the idea that we are going to be promoting things from our development build, into a test environment, into production. To me, that’s been a longterm touchpoint. How do we have visibility from commit all the way through to production, and actually what’s happening in production? And this idea of getting visibility of the business impact, so the work we’re doing, tied back to the commits we’re making. For me, that’s what I’m talking about with end-to-end.

Further on Microservices

Darko: To push metrics, to be a first-class citizen, you have to split things into microservices. And there, as you have many small individual components that are running, some things become easier and there is a whole set of things that become harder. Having metrics that really follow the whole life cycle of that individual microservice is very important. It also introduces a possibility of various other kinds of deployments, and you can test something out in part of instruction, which is available in other tools also. It something that would relates to this, what you were describing?

Jeffrey: I think there’s a similarity. I think microservices makes it more obvious to people the benefit of having metrics, but I think that tends to be focused very often on the technical metrics because people are first saying, “Is this working?” Now, that’s good, it’s nothing wrong with that. To me, the idea of bringing the business metrics into it is really important.

Another blind spot that people have developed as they’ve developed good technical practices of making sure that code is tested. You can still end up with the problem that the developers are only caring that it works in development, they’re not always thinking about, “well, how will we know it’s working in production?” This, I think, is one of the ideas that devops is supposed to address. If you do that, it leads very naturally to the idea of a rich mindset of monitoring and metrics.

Technology Adoption Lifecycle

Darko: From this long and rich career you have behind you, something that was a major topic of discussion a couple of years ago, at least in the circles that I was following, is that “you build it, you run it.” Talking with customers these days, has become the norm, at least for the teams that are very technical and mature and embraced CI/CD and DevOps fully.

Jeffrey: So which means about 15% of companies, but it’s normal for them.

Darko: Yes, luckily they are our customers. So I think that we have a privilege of working with such great teams. You actually answered my question, I wanted to ask the progression of that “you build it, you run it” as you have witnessed over the years, obviously it’s reached 15%.

Jeffrey: Yeah, and I’m kind of spit balling it. I’m using something called the technology adoption lifecycle, which I first learned from the book “Crossing the Chasm”. The technology adoption lifecycle divides the population of people in any market into different groups. So there’s the innovators, the people who invent new processes. Then you have the early adopters, and these are people who are always seeking to be better. Then you get what’s called the early majority, then the late majority, and then the laggards, so that’s the rest of the 85%.

Agile Technology Adoption Lifecylce

To give it a comparison, I think you can look at the spread of Agile since 2001. At the time the Agile manifesto was written, the dominant Agile methodology was extreme programming, and so I think that was something that was picked up by the early adopters. I think in about the mid 2000’s Agile crossed the chasm into the early majority, and when it did it came through as scrum. And so it moved from a focus on technical practices to focusing more on project management. And now as we’re getting to the late majority, this is where you see the rise of these sorts of things like SAFE, the scaled agile framework, where they’re saying “This is the standard way to approach this.” That spread of Agile, how different segments of the market have picked up Agile in a different way, and basically mean something different, I think that’s true micro chasm for all the different sub pieces.

Darko: Unfortunately, you’re probably right, it is going to go that direction. Do you see that continuous integration, continuous delivery, devops, that it is adopted by roughly 15% of the companies and you would still say that it’s in the early adopters category?

Jeffrey: I think this has moved beyond it. Continuous integration and devops have both moved well past the early adopter phase, but it means something different. A lot of how it’s been adopted by the mainstream, CI is about tools, not about technical practices. It’s not about developers committing frequently.

Is Agile Dead?

Jeffrey: Alistair Cockburn quoted something and posted on Twitter a couple of years ago, there was a lot of talk about Agile being dead, and he was rejecting that. I think he’s right. Agile is not dead, there’s still millions of knowledge workers out there that Agile way of working haven’t reached them. It’s going to. Now, I think what happens is long time Agile people look at how things get transformed and they think, “oh, that’s not what we had in mind, that’s not what we meant”. But I’ll just say, “Look, you have a choice. When you have an idea, one of two things will happen. It will be ignored, or it’ll be misunderstood. So those are your options.” So Agile being misunderstood is not a sign of failure, it’s a sign of success. Because the alternative is it dies a death in a lonely dark corner.

We haven’t achieved all the promises of Agile, there’s still this large gap between the 15% early adopters and people who are really embracing all the practices, getting the best results.

Darko: I think that the majority of people who are practicing continuous integration actually are pretty far away from that idea that we should be integrating continuously. But something else which is great on the other side, and let me explain that through an example. Just two weeks ago, we had a conversation with a customer and their pipeline is actually great. They get the feedback on all of their tests in under eight minutes, and that’s their goal. They have 55 jobs running in parallel, executing their testing in parallel, and they have a team of 60 engineers.

They want to grow to a team of 100 engineers by the end of the year. One of their concerns is they want to maintain that feedback loop of under eight minutes that they set and maintain the speed of deployment, how many times are they deployed per day. And they can deploy 10+ times per day, which is amazing. So in a way you could say that continuous integration is happening, maybe in a bit different form and it’s not the main focus, because the main focus is to deploy many times per day and have things shipped. However, feature branches that can diverge from main master are a reality and vast majority of people from what we are seeing are using feature branches to defer the integration. It’s still good, there are still many benefits to it.

Jeffrey: You know, it’d be easy to mock them say, “Oh, they sound like they’re almost where IMVU was was 13 years ago.” Which were they were deploying to production from commits 50 times a day with a team that was, I think, around 100 people. So they’re approaching that. That’s still huge progress, and so I think it’s great that that kind of practice is becoming mainstream. So yeah, I think what I hear you describing is a success story.

Darko: Yeah, I also think so. Majority of people are actually struggling to get there and to maintain that position.

Agile Conversations

Jeffrey: I would make one bridge from this work on CI to my more recent work on conversations, because I think that’s how my career has evolved from a real focus on tooling and process to a focus on conversation and collaboration. And the reason is because in my experience at CITCON for over a decade, was that I kept finding frustrated technologists who wanted to be doing better and were frustrated that companies weren’t adopting these practices. What I realized over time was that the limiting factor of a company to improve was the quality of conversations they were having, and that’s where I’ve put my focus in the book. “Agile Conversations” is a book not limited to Agile, it’s really for any time you have humans collaborating. The question is how do we have productive conversations, including productive conflict? And there’s real skills about making that happen, and these are learnable skills.

My view is if you’re at a team, and you’re feeling that frustration of “why aren’t we better?”, The way forward isn’t to say, “well, let’s go and research more technology. Let’s go research more tools.” Go ahead and figure out how to have productive conversations first.

Darko: I would want to ask for advice in that area, and let me give you an example. Over 10 years ago, our company was a handful of engineers, and then more engineers came by the nature of our situation. The majority of those engineers were very young and perhaps lacking soft skills (not to insult anyone).

It is something that has to be taught and it’s part of the culture, and the bigger the organization is, the harder it is to move the ship in that direction. So I, and I think many of listeners would be interested probably to hear some advice in that area, how to level up that level of communication.

The Art of Constructive Conflict

Jeffrey: Our intention with the book is to have very practical hands-on exercises that you can do. The main advice I have is to first make the point of productive conflict the explicit goal. What do you mean productive conflict? We’re saying here is the value of a team comes from the diversity of opinions and diversity of knowledge. We want to have a group of people where we don’t agree at the start, because if we did all agree we’re really not getting the benefits of a team. Everyone agrees to this in theory, but they don’t know how to navigate it in practice, and that’s where the skills come in. So once you’ve agreed, and everyone typically does agree that you want the value of diversity, then you say, “Well, if we’re going to get that value, we also need to work on the skills.” And I mean work. Have practice sessions with pen and paper to do conversation analysis.

The Four R’s Framework

Jeffrey: In the book we introduce a framework called the four R’s, which is you have a conversation that you record, you reflect on the conversation in a structured way, you revise your conversation, and then you role play it. So you get practice thinking in a different way, constructing conversations differently, and speaking. And by developing this through practice, you start to develop different habits of thought and communication, and that can make a tremendous difference. And this is something that I’ve experienced, and this is what was behind the book was having done this at the company I was at. We actually went through and did sessions for the whole company, we did exercises with the whole company and offsite. We also had small group sessions for everyone in the company so that everyone got practice. And the important thing I hear was two things. One is establishing it as an espoused norm and value at the company level, and the second was following that up with hands-on practice time. I think those two things together were what led to a really dramatic change in culture to much better collaboration.

Darko: So this point I wanted to check, you mentioned the podcast. I’m not familiar what are the topics? Is it continuation of conversation that you started in a book, and expansion on that? Or…

Jeffrey: We started the podcast while writing the book, and because our goal, again, was to be very practical, the idea of troubleshooting Agile was to help talk about symptoms that people might see and then talk about how you can troubleshoot the problems and what you might do to solve them. In my career I’ve done most jobs you can do in a software company, everything from technical support, and QA, and development, and engineering manager, product manager, product owner, VP of development, VP of product management, CTO, managing director. The only thing I haven’t really done is have a quota as a salesperson, but I’ve done sales engineering and sales support and evangelism, marketing, all that kind of stuff. And Squirrel is a very active technical consulting CTO, but he really does a lot more than that now. He really specializes in helping, as he would say, software teams become wildly profitable, and he works very intensely with software teams and software companies to make radical changes in their productivity in a matter of weeks.

So that’s the background. He’s worked with hundreds of companies, I’ve worked with fewer but a long period of time, I’ve had exposure to lots of companies, partly through CITCON, and partially through my roles at different tool companies. We bring a lot of experience in having seen a lot of different scenarios and bring that then to discussion in our podcast.

It doesn’t tie directly to the book in that sense, although there’s a lot of commonality. There’s conversations you need to be happening often come up. So it’s less intense. It’s more than what you get in the book, but there’s some overlap.

Darko: Thank you so much, it was a great conversation. We’ll make sure to link to CITCON, recommendations, your book, and the podcast. Some great material there for sure. Thank you again. Good luck.