Adam Gordon Bell on Better Builds with Earthly

What’s in the episode:

• Introduction to Earthly
• When do you want to start using Earthly?
• What technologies Earthly supports
• Earthly vs Bazel
• How to start with Earthly
• CoRecursive podcast
• Is being a late adopter a bad thing?

Edited Transcript

Darko Fabijan: Welcome to Semaphore Uncut, a podcast for developers about building great products. Today, I’m excited to welcome Adam Gordon Bell. Adam, thank you so much for joining us. Please go ahead and introduce yourself.

Adam Gordon Bell: Thanks for inviting me! I’m excited. I work at Earthly, and I’m a developer. I’m working in developer relations now. I also have a podcast about software development, and I live in Peterborough, Canada.

Introduction to Earthly

Darko Fabijan: For those of us who don’t know, can you please give us an intro to Earthly?

Adam Gordon Bell: I’ll start with the story. Vlad, the founder of Earthly, worked at Google. Internally, they have this build tool called Blaze, which has been open sourced as Bazel. If you work at Google or have a really large monorepo, it’s a great tool chain for building software.

But it only works in a specific context where you have a team that support it, you’re using languages that support it, you have a monorepo with things vendored. It doesn’t necessarily work great outside of this context. 

Read also: Beyond Docker with Earthly

Vlad wanted to come up with a way to build a tool like that, that supported normal development. The way that Bazel works is it uses a file system sandbox to ensure that you don’t have any dependencies that are undeclared between each build step, to make the builds reproducible. So, Vlad saw that containers exist and are also a way of sandboxing things.

So he built Earthly. Earthly is a build tool. It’s like a combination between a Makefile and Dockerfiles. You put targets and describe the build steps within them, and you use it to build your software in a way that is reproducible and parallel.

Earthly is a build tool. It’s like a combination between a Makefile and Dockerfiles.
-Adam Gordon Bell, Developer Advocate at Earthly

When Do You Want to Start Using Earthly?

Darko Fabijan: For people who are just getting started with Earthly, what would be some trigger points that they might experience in their workflow and their needs?

Adam Gordon Bell: One would be, you want to be able to make changes to your build but it’s a slow process because you have to change something in the build script and then send it off, have it run and see what’s wrong. That can be slow. Earthly allows you to abstract all those tasks into something you can run locally because it’s containerized. 

Another common point is, you have a team that uses more than one language. Once you need to work across more than one language tool, one way to deal with that is writing a bash script or a Makefile.

Earthly is another way. The way we see a lot of people enter is they want to have this way to do things across more than one tool. Or they want to be able to do something locally that previously they could only do in CI or some staging environment.

Another way people join Earthly is coming from multi-stage Dockerfiles. They can get really gnarly. A lot of people come to Earthly because it’s a really nice way to make a multi-stage Docker build.

What Technologies Does Earthly Support?

Darko Fabijan: One of the use cases that I’ve seen for Docker containers that the developers seem happy about is Docker Compose, when it’s going to boot a couple of containers for you and connect them over DNS. Does Earthly work in conjunction with Docker Compose or it has its own way of solving that problem?

Adam Gordon Bell: Earthly supports Docker Compose files. You can use Earthly to build a Docker container but you can also use Docker containers within Earthly and Docker Compose files as well. 

Here’s an example. I was helping the Phoenix project, which is an Elixir web framework, work on their CI process. In the CI process for Phoenix they want to make calls that access a database. It’s an integration test, and they want to test across several databases. 

So, they have a Docker Compose that starts up like Postgres, SQL Server and MySQL, and then they build their code and then they run tests again. Within Earthly, you just specify those as dependencies and then you say docker run, you give the Compose file. And then within that context you run your tests, and then when you exit that scope then the Docker Compose shuts down. So, that’s how you could use it with Docker Compose.

Comparing Earthly to Bazel

Darko Fabijan: You mentioned Bazel. Can you compare Earthly to Bazel?

Adam Gordon Bell: Our origin story is based around Bazel. Bazel builds monorepos in a distributed way with lots of caching, and it does it very well. To do that, it makes certain trade-offs. For instance, the support for the language that you want to build needs to be baked into the tool. Right now, it supports the languages that were commonly used at Google, like Python and Java. If you’re using another tool, then… if it’s not built-in support for it, then you’re going to have a hard time.

Also, you need to declare inputs and outputs to these tools, and that can be kind of verbose. It’s designed to be an amazing build tool for a place where you can have dedicated people to work on it and understand the tool. 

Earthly is a tool designed to have some of the same principles but to approach it very differently. With Earthly, you can use whatever tool you’re currently using. We tried to give people the benefits of a big tool but that would be more approachable for everyday people.

With Earthly, you can use whatever tools you’re currently using. We tried to give people the benefits of a big tool but that would be more approachable for everyday people.
-Adam Gordon Bell, Developer Advocate at Earthly

Another thing is Earthly supports polyrepos. You pull down your dependencies from this Git repository, build that, and then pull down from here, build that. And it doesn’t all have to be in one, big monorepo.

Read also: What Is Monorepo (And Should You Use It)?

How to Start with Earthly

Darko Fabijan: If someone wants to try Earthly today, what do you suggest as some first steps to getting to know Earthly?

Adam Gordon Bell: Go to Earthly.dev. There’s a landing page that will explain some of the steps. You can download Earthly, and if you’re on Mac, just do brew install earthly. Then you can take a look at some of the examples. 

Darko Fabijan: Yeah, when you watch that video on the landing page, it becomes pretty clear how it works, and it’s very very approachable. Earthly is a CLI tool that you can use. You mentioned that you approach some of the tools, like the very classic tools from the Unix ecosystem, or you reapproach them with a fresh eye. Can you talk more about that?

Adam Gordon Bell: In this role, I’m helping people build software. Earthly is very much built around a developer-first approach to building things that’s based in the command line and not a big YAML file but something more declarative.

Earthly is very much built around a developer-first approach to building things that’s based in the command line and not a big YAML file.
-Adam Gordon Bell, Developer Advocate at Earthly

It made me start to think about other command line tools that are classically used and sometimes get used in builds. And there’s tools that are really useful but I never really knew how to use. 

So, the first one was JQ, a command line tool for dealing with JavaScript. When I didn’t know how to use it, I would go to StackOverflow and end up copying and pasting something from there.

I spent time trying to understand how JQ works and writing it down in a way that would explain it to other people. If I spent half a day learning it, think of the dividends that would pay. I wrote an article about it, called An Introduction to JQ.

The takeaway: learn how to use your tools.

Darko Fabijan: Exactly! I have many regrets that back in the day when I had much more time, I would learn something to a shallow extent. I’d read only the first two or three chapters from a book, I’d get the idea. Then I’d end up limping for the next 10 years of my career because I hadn’t spent the time learning. 

I had a similar experience with other tools which are not my primary tools, like Bash. I know bits here and there but there are a couple of syntax-y things that I just don’t know. As my career progressed, I started helping other developers develop and move on. I started noticing the same in other people, that I wasn’t the only one leaving the last two or three chapters. 

So I ended up suggesting to people to pick those essential tools that will not age or are going to age well, and invest. As you have put nicely, one will get dividends over time.

Adam Gordon Bell: That’s interesting to think about. Similarly to the experience you had with Bash, I had the same experience with Ack. When I wrote the JQ article, somebody was like, “This is really great. I feel like JQ is like Ack. I never quite knew how to use it.”  And I was like, “Yeah, I have no idea how to use it. I’ve copied some things once in a while.” 

I think you hit the nail on the head by saying it’s something to do with tools that are a bit older, because there’s less risk. If something’s still around 10 years later, there’s less risk it’s going to go away. 

Speaking of tools, if something’s still around 10 years later, there’s less risk it’s going to go way.
-Adam Gordon Bell, Developer Advocate at Earthly

CoRecursive Podcast

Darko Fabijan: Another thing that you do that I find super amazing is your podcast. You put a lot of care into it, and it’s very interesting to listen to. The guests are also amazing. Can you please tell our listeners about the origins? 

Adam Gordon Bell: My podcast is called CoRecursive. I started doing podcasts as a guest host for a couple of other podcasts, one of which is Software Engineering Radio. They have a training process for how you do the podcasts and there’s a lot of preparation ahead of time. You pick a topic and you make sure that you get all the facts and answers out of people. So I did that and started my own podcast in parallel to it.

I was talking to people a lot about functional programming as I was a Scala programmer at the time. I would just ask people, what is this? What is that? 

But over time, I felt that the parts of the podcast I really liked were more about people’s experiences. Instead of explaining what X is, tell me when you first encountered X. What was it like? What happened? What were you wearing? Just really getting somebody’s personal story. So I’ve just kind of doubled down on that.

I really like it. I recall being a software developer and being at lunch, and something had blown up yesterday and somebody would get called in, and then they kind of tell you the story. “Oh, the web server was down,” and then somebody is like, “Oh, did you look, was it this?” And, “Oh, I looked. It wasn’t that.” The drama, I guess, of somebody recounting these stories instead of explaining to you, “Oh, what is a web server?” So, I’ve just been doubling down on this concept. Now my podcast is very story focused. The most recent episode was about this guy, Paul Lutus, and he was known as the Oregon Hermit. He built software for the Apple II from a cabin he built himself in the woods. It’s a very strange story. It’s super interesting.

Adam Gordon Bell: There’s things to learn from people’s stories. His story about how he worked remotely in the woods has some things to teach us about how to work remotely. He also thinks that developers have lost touch with product and the business side. He wants developers to be more fully rounded people. But it’s hard to understand that until you hear his story about how he built the software. I like to focus on the experience, the experiences of software development.

Darko Fabijan: Another thing that fascinated me was the episode about the SQLite and what are the origins of it and how it’s actually connected to the warship, or it’s a battleship story. It might be very inspiring for folks who are just getting started in the area.

Also, there is that generational difference. There are people who started coding in the ’80s, and their view on all the current tools and trends and everything is quite different from the people who just started a decade ago. And that’s why I also found your guests interesting, because a couple of folks that I checked out are really, let’s say, seniors.

Adam Gordon Bell: Yeah, like a seniors’… both in terms of title and in terms of seniors’ discounts at a restaurant.

Darko Fabijan: I find that inspiring. A vast majority of people that I worked with were maybe a decade older than me. That’s one of the reasons that it’s more interesting to me to talk with folks who have seen industry progress over a couple of decades.

Is Being a Late Adopter a Good Thing?

Adam Gordon Bell: Yeah. It’s interesting how things change really fast, but some things don’t change that much. I mean, you were talking about learning how bash works. I mean, that’s not new, right?

Darko Fabijan: Yeah, absolutely. So I would say, someone learned the best all the way back in the early ’90s and covered Postgres in the early ’90s or just SQL in depth. Now, the guy or girl is just fine even today.

Adam Gordon Bell: It’s a tricky thing. At some point I remember learning Angular 1, and I don’t think it was ever useful for me besides the couple of months when I used Angular 1, because it turned over very quickly. So you got to get beneath that surface level and beneath the hot new thing and look to the things that have stood the test of time.

Darko Fabijan: Yeah. There is that piece of mesh in the library, it’s called Envoy. It’s made by Matt Klein. He was one of the first people that I talked to who very strongly expressed that he is a late adopter. He wants to be a late adopter. He explained the number of benefits of that. I mean, as I finished university, I started working and all that. You’re always looking at the news and what’s the hot new thing, hot new thing, hot new thing. And whilst the majority of people are kind of like that, “Okay, what’s next?” 

I think there are also huge benefits in looking the other way. Maybe you need to wait for the first 10 years of your career, that you run through a couple of things. And then at that point, find what are those stable components of your career that you are going to invest, and you will be safe building software which is super stable and is going to stand the test of time.

Adam Gordon Bell: Matt is one of the investors into Earthly. He’s a very talented developer, for certain. I heard somebody else say, who was also a slow adopter, that sometimes they have to learn less than people who are fast adopters. I forgot who this was but they were saying, they missed Ruby. Everybody was excited about Ruby, and they were like, “Oh, we’ll wait. I’ll wait on this. I’ll wait on this,” and then Ruby fell out of popularity and he’s like, “Oh, I didn’t have to learn that.”

People love Ruby so I don’t want to diss it, but sometimes these things fade. If you move slow enough, you don’t even have to learn them.

Darko Fabijan: Also, in the first days there’s less documentation. The stability, the quality of everything is lower. There are less people. And they’re also exciting times but they also tend to be quite rough.

The Downsides to Being a Slow Adopter

Adam Gordon Bell: There is another side where you could make a name for yourself as a developer by going to the newest language and there’s probably packages missing, and you can develop these packages, right? You could… whatever. You’re the best at making a hash map or a dictionary. And each new language needs some sort of concurrent hash map and you could bring that to them. So there’s downsides, I guess, to being slow at adopting things.

The Challenges When Building Earthly

Darko Fabijan: So, we talked through these couple of approaches, how to pick the right tools and how in-depth to go with them. Can you share your experiences when you were building Earthly? What are some of the challenges that you faced?

Adam Gordon Bell: Earthly is being built in the open, it’s on GitHub. One of the challenges we have is what features we add. So, Earthly is like a syntax for describing a build. It’s just kind of like a small programming language. And people ask for features, but if you just add every feature you’re not thinking through what the end state will be and you can get to a weird place. 

So, we spent a lot of time thinking about this. For instance, somebody asked for this feature, using host mounts. That would make builds more performant, at least in their case, but it could hurt the reproducibility. It would be possible to have something not build the same on everybody’s environment, so should we add that? 

So, we came up with this list of things that we value in the product, and we try to basically consider whether a feature helps these values or kind of hinders them. So, if our number one value was performance, then that feature that hurts reproducibility but does help performance, we would certainly add it. But actually, we value reproducibility above speed. So, we did not put that feature in.

Adam Gordon Bell: And so, we came up with this list. We’re actually going to be writing a blog post about it. And this idea came from Bryan Cantrill, who said that every software platform has things they value more than others. We value both speed and reproducibility, but the thing is you need to decide what’s more important to you. 

And that order actually ends up being a lot about what your programming language, what your product, becomes. One thing we value a lot is versatility, so we want people to be able to bring whatever programming language, whatever build things they’re using, and use it within Earthly. 

So, we put that as our very top value. Bazel, that we were discussing earlier, puts reproducibility at the very top; the build should always be byte for byte perfect. So, that means they can’t have this versatility. They can’t just allow people to use any given build tool. They need to kind of gate that, right? And so, the way you stack these software values really informs what your product becomes.

Adam Gordon Bell: We’re writing this up, but we pick versatility as our top goal, so we want people to bring whatever they want and use it to build things. And then we picked approachability as our next. So, builds are often a thing where you don’t spend all day looking at it. It’s just once in a while you need to go and change the build. 

So for us, that means it’s important that the build should be straightforward to understand. If there’s a feature that’s super powerful but it takes a bit to follow what it does, we would rather not include that, because we know that people don’t spend their whole days in builds. It’s a once in a while thing, so it needs to be easy to understand at first blush. 

That’s a very long way of saying, the biggest challenge we face is how to rank these trade-offs between the values that features kind of embody in your product. And I think we figured it out. I mean, thanks to Bryan Cantrill with this idea that you need to put the values of your product first.

Darko Fabijan: That’s very interesting. We have been facing backlogs, which would take a century to execute all the time, and it’s a real struggle. There are a lot of great ideas, but there is a limited amount of time, but you can end up somewhere you really don’t want to go. And you just look around and you see that maybe three years ago you took the wrong street.

Adam Gordon Bell: Yeah. And if I look at the Semaphore website, it talks a lot about speed, right? Speed’s important in CI, so I would guess that, that one’s really high for you guys, right? If there’s something that would slow things down, but whatever other benefits, forget about it. Speed is where we’re focused.

Darko Fabijan: Okay. Thank you so much for your time. Good luck with Earthly and your podcast. It’s a really amazing podcast, so go and check it out.

Adam Gordon Bell: Thank you. Yeah. You can find me on Twitter, I guess: @adamgordonbell. Or yeah, in your podcast player: Adam Gordon Bell and the CoRecursive podcast. And Earthly is on GitHub or at earthly.dev.