The 1st Crisis in Software Engineering, the Roots of Agile and the Short Story of DevOps

For some of us, the interest we have in software engineering started when we played our first video game when we were young, for others it started with coding bootcamps and others started having an interest in software engineering in college.

Whether you started this or that way, there’s a common thing that most of us share, we initially started with the intention of building visible code whether it’s a game, a website, or a mobile application.

At this early stage of our understanding of software engineering, you very rarely hear of people who want to develop APIs, manage networks or maintain production systems... In the beginning, we were charmed by the light and colors shining out of our screens but with time, driven by wonder and curiosity, we start asking questions on what’s really running inside, how it’s running, and more importantly how to create a similar thing.

We started seeking the answers and this is when we discovered a new fascinating world of possibilities: The backend that powers the frontend. The technologies that everybody uses but no one sees. The foundation of the base.

From there, choices and possibilities diverged. Many chose to stay in development as is the case with backend developers. Others have chosen to go into infrastructure and networks like system and network engineers. Each took a different track, but these tracks met once again with DevOps.

I’m your host Kassandra Russel and today, we're going to talk about the short story of DevOps, but before that we’re going to travel back in time to the first crisis in software engineering, passing through agile methodologies, and arriving to DevOps.

In 1968 and 1969, The NATO Science Committee sponsored two conferences on software engineering. Many believe these conferences marked the official start of the profession of software engineering.

But like in every evolving field, the crisis is inevitable - software engineering has known crises too.

The term "software crisis" was coined by some attendees at the same NATO Software Engineering Conferences.

Edsger Dijkstra, a dutch systems scientist who, during the same period, advocated a new programming style to improve the quality of programs, said once:

The software crisis was linked to the growing complexity of software and hardware and negatively struck the industry. These impacts were witnessed in different ways: Projects running over-budget and over-time, becoming more and more unmanageable, software with low-quality, difficult to maintain, and overall software projects were never delivered in many cases.

To solve these problems, the community has turned its attention towards processes, methodologies, and philosophies.

Various approaches have been developed and adopted over the last few decades to improve software quality like object-oriented programming. Still, large software projects were always vulnerable to unanticipated problems.

In 1986, Fred Brooks, the man behind the known statement "Adding manpower to a late software project makes it later", published an article called No Silver Bullet. He argued that there is no one silver bullet.

The search for a single key to success never worked.

A few years later, during the 90s with the rise of the Internet, when computer users went from hundreds to millions, the demand for software in many small organizations expanded along with the need for inexpensive solutions. This led to the growth of simple methodologies such as agile methodologies.

Agile software development comprises various approaches and advocates common practices like adaptive planning, continual improvement, evolutionary development, and early releases.

What actually happened during the 1990s is a transition from the dominant heavyweight methods to a number of lightweight software development methods. It was a reaction to the inefficiency of the status quo.

Different teams in different countries came close to the same conclusion “iterative and incremental development works well in comparison to heavyweight development models”.

The same approach was called with multiple names and applied with subtle differences: Rapid Application Development, Extreme Programming, Feature Driven Development or Scrum.

In 2001, when seventeen software developers met and discussed the similarities of these approaches they ended up by publishing and signing the Manifesto for Agile Software Development agreeing on 12 similarities and proclaiming that they value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to Change over following a plan.

Jim Highsmith, one of the initiators of the Agile movement stated that:

If we take a look at one of these lightweight methodologies like Extreme Programming, we’ll find out many artifacts and routines that we still follow today in many development teams and projects like: pair programming, unit testing, and standup meetings.

The incredible technological growth and the fact that software started being considered as a commodity did not only made the old development approaches obsolete but pushed the new ones to some of their limits. Agile didn’t have room for another important stakeholder: the system administrator.

In reality, due to the different nature of responsibilities, a system administrator’s way of perceiving things is different from the developer’s one. While the developer usually focuses on change and features, the system administrator’s concerns are more focused on stability and durability.

In 2008, at the Agile Conference in Toronto, a software developer called Andrew Shafer posts a notice of a “birds of a feather” session entitled “The Agile Infrastructure”. One person decided to attend it, Patrick Debois, a system administrator who experienced the benefits of Agile development. Noticing that the topic was of no interest, Andrew skips his own session but Debois tracks him down for a conversation. As a result of their hallway discussion, they created the Agile Systems Administration Group.

The main concern of this group was finding an answer to a self-questioning issue: How to [sound] build an agile infrastructure that supports agile development.

In 2009, when Patrick was watching the O’Reilly Velocity conference on Twitter, he had the idea of creating a similar conference. He took the first 3 letters of “development” and “operations”, added Days, and called the conference DevOpsDays.

This is how the first DevOpsDays was organized in Belgium in 2009.

During the same event, on the ongoing conversation on Twitter, Patrick used the hashtag #DevOps as a shortened name of DevOpsDays and this is how this new movement has been known as DevOps ever since.

A large community of developers, testing, and operation engineers embraced DevOps which gave birth to a worldwide community of adopters and contributors.

DevOps has its agile roots for sure, but there are some differences.

The Agile process focuses on constant changes while DevOps focuses on constant testing and delivery.

Agile leverages the shift-left principles while DevOps leverages both shifts left and right principles.

DevOps brought about fundamental changes with regard to culture: Collaboration, information flow, shared responsibilities, learning from failure, and cross-functional ways of working became the norm.

Aside from the fundamental shifts in culture, automation, measurement, and sharing. DevOps also brought new toolchains, technologies, and paradigms like CI/CD, infrastructure as code, elastic and immutable infrastructure.

During a meetup presentation about scaling up vs scaling out, Bill Baker, a software engineer, used an analogy of pets and cattle to simply explain the differences. Randy Bias, another software engineer used and developed the same analogy to explain to customers how cloud-based systems are different:

Cloud computing made building infrastructure as code easier, it allowed us to automate most of the manual tasks and enabled collaboration between the teams building the application and the teams maintaining the infrastructure.

The automation made reviewing, testing and bug hunting happens at an earlier phase of the software development lifecycle. This was referred to as a new concept: “shifting left”.

Shifting left made quality assurance cheaper because fixing issues in the production environment cost a lot more than in the development stages.

The movement never stopped adopting new technologies and paradigms. This vibrant ecosystem is changing a lot and every day there’s a new thing to learn.

That was a short story of DevOps. In the upcoming episodes, we’ll discuss many other interesting topics about basic concepts like CI/CD but also newer ones like GitOps. If you don't want to miss these topics, visit faun.dev/podcast and subscribe.

If you have suggestions, use the hashtag #AskFAUN on Twitter and Instagram and let us hear from you. Stay in touch!