A History of DevOps, and How You Can Too

The Future of Containers in the Enterprise

February 10, 2016

New York

A History of DevOps, and How You Can Too

Fantastic to be here, it is actually >> [APPLAUSE] >> It is the speaker's prerogative to change titles and change slides up to the moment of the presentation—it would perhaps be bad form to be changing it during the presentation—and so if you were looking at your schedule, you may be seeing this title doesn't map up to what it is, but it is on screen there we go, and so we've been talking a lot about DevOps here, we've been talking about the correlation of DevOps to containers, and it would be a shame of us to go all day without actually going through the history of DevOps and the opportunities that it can create, but also its failures.

This is a history of DevOps and how you can to. Thank you, I'm Casey. I'm actually a Director of product for Joyent where my role is actually to look for what we are doing, the best practices of what we are doing and what we dream of doing with infrastructure and with the future of our apps and then making those things.

Building what we have at Joyent to be, to make those things a possibility, or at least a possible reality for what we are building. So just for a little bit of background the marketing folks forced me to put slides like this in here. Joyent is a software company first. We build infrastructure automation software, datacenter automation software, perhaps we can call it container native datacenter automation software.

We have been doing this since 2006, container native since 2006, how we were using containers before the introduction of Docker is a whole other conversation of history about virtualization. We love telling histories at Joyent, but we're not gonna do that one right now. But we operate a public cloud as well, and I think we are unique among companies offering the exact same software that powers our public cloud as open source software that you can use to download and run enterprises of every size.

So, you wanna run it in your own data center, wanna run it in a public cloud where we manage the datacenter, either way, you want support for your own datacenter, your own private cloud implementation we definitely offer that, you wanna run open source try out the latest stuff we definitely offer that as well.

But back to the dreams that we have as developers and operators and realistically I think we were promised jet-packs. This is in fact, this is 2016, we were promised jetpacks, we want applications that are self operating. So we want apps and workflows that can be tested and automated in CI/CD. We want apps and workflows that work just as well on our laptops just as they do in the cloud, whether that cloud is on prem or in a public cloud.

We want apps and workflows that aren't married to any specific infrastructure provider or scheduler. In short, we want the kind of apps that we were promised with the DevOps philosophy. And so what we wanna talk about here is "what is DevOps?" So all this—we gotta get the history of DevOps and where did that start? Well of course the classic history is that it started with a computer, the computer eventually begat the Internet, the Internet begat the cloud and that begat a whole lot of really lousy lousy clip art.

>> [LAUGH] >> As well as a rush of new developers to the cloud building distributed applications for their first time, and new users for those applications and that begat a whole lot of fail, a whole lot of fail. So, it was from this this crucible that DevOps was born. >> [LAUGH] >> Now, watching DevOps when it's working well is kinda like watching Bruce Lee play ping pong with nunchucks. Not everybody can do it, and it's not necessarily the correct tool but gosh, it is a thing of grace and beauty when it happens.

And this is powerful. Unfortunately, as beautiful as it is when it works, it doesn't always work, right? DevOps is a process not a pattern and despite all the marketing around it, it's not a product. You can't introduce DevOps into an organization just by dy declaring it so. And so despite it being now six or seven or eight years since the introduction of DevOps—I think it's seven years since we defined it in Wikipedia—we're still making the same mistakes, and the DevOps philosophy has not delivered on it's promise, I would say.

And what do I mean by that? Consider this particular promoted tweet that I saw just last week. Now the problem here isn't that it takes 40 minutes to provision a server, down from 90 days, that's great. I assure you, I love that. The problem is in an age where we need automated applications, these applications need automated deploys.

It's not the servers that matter, right? If we're spending 40 minutes to deploy a server, that is totally the wrong problem to solve. Certainly it's taking too long, but we're thinking about servers not services.In the age that we want, we're talking about services. So if DevOps isn't working the way we want, where can we find inspiration? So I would say Conway's Game of Life from the 1970's introduced cellular simulators, automata, that follow a specific set of rules by which those cells can self-operate.

There is something kind of interesting going on here that we can look at, especially if we fast forward to 1988 and the Morris Worm. This was a project that was claimed to supposed to map the Internet, but actually what it was doing is taking advantage of a set of vulnerabilities in services on hosts on the Internet to deploy an application on them that is operating.

Now I think this is actually sort of the first example of the cloud in practice, and amazingly what it means is we actually have, perhaps the first example of a cloud application, the Morris Worm, actually demonstrated a lot of the characteristics that we want in our modern apps.

But science fiction is also really exciting about this. So, of course independence day demonstrated applications that not only can self-deploy, they're of course infrastructure and OS independent, which is kind of amazing. >> [LAUGH] >> And at least this is an application in the service of good, because when it comes to the alternative it's a little scary.

Now I have to use what is actually the second worst Terminator movie of all time to demonstrate one of the most pervasive examples of self operating applications in cinema, and that is of course Skynet, an application that deploys with a button click, scales easily and all of that, basically self-operates, but most importantly here, Skynet gives me an answer for how I can describe what I do to my parents, to my family.

As a person who cares how to build self operating applications, I can tell my family I'm building Sky-net and you can too. And so the lesson here isn't that Hollywood will always get these things wrong, and it will. But that instead, the lesson here is that there is a history of the applications that we want, real history.

The Morris Worm demonstrated using the cloud in a way that is very very interesting to us as an emblem of the applications we want to build. So it has some characteristics that are worth noting. First of all, it is self assembling. Of course it's self-aware and self-healing in how it goes through here.

It's infrastructure independent. It actually moved across different hosts, different operating systems, and there was no scheduler. It was totally scheduler independent. These are characteristics that we wanna pay attention to, I think for the applications that we want to build as modern DevOps practice.

So what are we doing wrong? How are we going wrong here? What in our current practice is working or not working? Well, the first problem is despite all this time doing DevOps, we're still struggling with the very, very, very common problems of "it works on my machine." And no matter what we want to do instead, this is actually probably the primary thing that everybody who is trying to practice DevOps is doing.

So we gotta get past that first thing. Then when we're not working on that particular problem, we're most often working on trying to automate, so configuration management tools are the hammer of DevOps. If we don't have them implemented we the dream of having them implemented and we want to use them everywhere.

So this is I think really—configuration management is DevOps for a lot of people, but setting them up is complex. And they're actually not universally used. And even where they are used in production very aggressively, we don't see them used typically for developers on our laptops.

And those differences matter. Those differences actually exacerbate the problem that keeps us busy most of the time. But worst than that, just as bad as the different usage between laptop and production is this common pattern of separating the configuration management code and templates and such from the application itself. By keeping those things in separate repos, what we end up doing is losing track of the changes as the application changes and requires configuration management changes those things are typically happening in separate repos, and developer often times don't have visibility into what's changing there. So they're not running it on their laptops, the developers don't have the visibility to that.

If the developers don't have visibility to the operational changes, the operators probably don't have visibility to the application changes that require those things. So we continue to exacerbate this problem. But another thing that's a little bit harder to see as we enter this age of containerized applications What we end up doing is configuration management solutions that are host centric—server centric—are not necessarily good solutions for application centric—service centric—tools.

So one of the things that we latched onto as a pattern is of course the 12 factor app. And some of us may quibble with some of the details, and there are good arguments about that, and then of course the other side is a lot of it is pretty self evident, obvious to a lot of developers.

But its a fine overall set of instructions to developers. Unfortunately, there's no instruction to operators about how to run these applications. So we continue this problem of it works on my machine. >> [LAUGH] >> Now, [LAUGH] Docker images, and there's something genius about Docker images, they may solve that works on my machine problem by making everything a dependency of the application including the operating system.

But until we come up with patterns that work for Docker in production, what we end up with actually it's just a whole bunch of developers saying Docker works on my machine. So DevOps, we're struggling here with a lot of our DevOps. However, even though there's a lot that does not work, there are some things that do, and I think there's new hope here.

First of all, the Docker API and Terraform both have two different approaches to something that I think is really revolutionary, but it probably slips through without notice. And that is, in here we're looking at the Docker run syntax. It combines arguments to the infrastructure as we provision this particular container—arguments to the infrastructure for that container with arguments to the application—combining those two things is pretty genius.

It allows us to think of and control our application components and separate those from the servers. So we're thinking about apps, we're not thinking about servers here. Here's a similar approach—actually this is looking at Docker compose syntax, and here we see both of those things together in the same space.

Thinking about services rather than servers, controlling and provisioning applications in one step is huge. In fact I would argue that it's actually like—it actually brings to this question of modern applications the same thing that calculus brought to orbital mechanics, right? During building modern applications as contribute during a modern applications without using tools like these would be like trying to calculate orbital mechanics with algebra.

It doesn't really work.
It's very slow. You maybe able to do it, but it's a lot more work and a lot less predictable than you think. Another thing that I think is emerging here as a winning pattern, of course, is infrastructure that supports these things, so I mentioned Docker a few times, and I think this is slowly coming together and importantly here we're seeing infrastructure that will run Docker containers in production in a way that especially useful for modern applications.

The specific capability I have in mind here, we can see it in implementations of Docker swarm, we have networking across hosts, MANTL, which is an implementation of Mesos that does some really exciting things, and (of course I'm biased) I think Triton is the leader in this space in what it does.

In fact, lets take a look at what Triton does here for networking specifically. eth0 is a layer 2 private DLAN, right. And this allows our containers to communicate with each other in a private space that nobody else can see besides my containers owned by me as a customer. Because if I asked for it for this particular container, I also have eth1 which gives another virtual NIC that allows this container direct access to the Internet.

More importantly, it allows the Internet direct access to this container. So the containers that I have in this particular application, multiple containers, they can all speak to each other with direct IPs. We don't have to hop through the host, we don't have to do anything with port mapping, every container gets it's own virtualized NIC, and we don't have any of those port conflicts.

On Triton what we do is we actually take advantage of hardware in the NIC to maximize the performance, both minimize latency and maximize throughput for this so we can push wire speed through these containers really easily, but however it's implemented, the reality of virtualized networking is here with us for our containers, and that's really important. Network simplicity is critical to our self operating apps, because how can they interact with the rest of our system if they can't self identify, and directly connect to all their components.

So that's an important piece. Another fantastic thing is the availability—common availability of service catalogs. Consul is my favorite I think it's far and away ahead of all the others, and this important development makes it possible to solve for this discovery problem. So service catalogs, in the process of service discovery, make it possible for application components to find each other, it's how my app can find the database that it has to speak to, and how the front end proxy can find the app it has to speak to.

Automating that process is a requirement for repeatable automated testable deploys and scaling. Now service catalogs are just one piece of it, you have to have a consumer of those service catalogs, and what we're seeing is an emergence of applications that are aware of service catalogs.

So Traefik is an interesting new proxy that uses a service catalog like Consul to automatically identify the back ends that it's supposed to connect to. And there are a couple of libraries and other applications that are emerging to do this. Containerbuddy can actually do this for almost any app. I cannot, I have not been able to think of an app that this will not work for.

And so when you bring—by moving the discovery into the app, it makes it possible to automate those apps without needing to set up configuration management systems, and it's ideal for containers because it focuses on the services not the servers they're on. Right, again, the separation of services from servers.

When combined with the new generation of infrastructure that I described a moment ago, this means that we can deploy our applications quickly and easily without preparing the host for them. More importantly, this means that we can run those applications on our laptops and in prod using the exact same configuration and discovery strategies. That means that every environment we go into can behave exactly the same. We don't have to worry about differences in those different environments causing more "works on my machine" problems.

And because the configuration management and discovery strategy is in the application—and this is critical—it means the developers and operators have visibility to all the changes that are needed to run that application. That visibility simplifies operations significantly. So this isn't just a talk we've been working with this pieces for almost a year we started work shortly after making our commitment to Docker compatibility when we saw the new possibilities that Docker was making—some of the things I pointed out earlier. We built this, worked with a number of applications to implement this, and if you've been watching you may have even seen some of our blog posts about this.

One of them implements a complete application, all of the pieces here with automated discovery. Automated discovery configuration, all of the self operating systems that we're talking about—in fact, consider those characteristics for a moment. If this list looks to you like the same list that we saw earlier when we were talking about the Morris Worm, it is. Right? These are the characteristics of self operating applications, and we have a pattern to build them. We have the infrastructure, we have the software we need to build legitimate real apps that can self-operate.

Now of course the advantage to all this, right, is that we get all of these easy, repeatable, testable, deploys that we can move from environment to environment. But importantly, by keeping all that automation code in the same repo with the application we are increasing visibility as I say and the opportunity for participation by all. We are also making that application and it's operation entirely self-documenting.

So [inaudible] what we get is cross infrastructure, scheduler independent, one click deploy, easy scaling applications. Right. So we have all of these pieces. All we're missing is a name. So we considered a lot of things Velcro is one of our favorites. It does a lot of what we want in terms of it goes together easily without a whole lot of pre-coordination, and it looks especially awesome when you zoom in deep and do it in macro. But it's a really closely held trademark, and some of the details are actually not the metaphor that we're looking for.

We considered a lot of other very very dumb names that I'm not gonna get into, but what we did get to is the autopilot pattern, which says pretty much exactly what we want. In an airplane or a boat—any vehicle where you have an autopilot, the pilot, the captain of this vehicle is in charge of everything, right.

What the autopilot does is automates the boring repetitive tasks that are related to the micro-operation of it. The pilot is still responsible for the operation of the aircraft. The pilot is still setting the heading, the course and speed. The autopilot is what automates making that happen without a lot of minor corrections along the way.

So the autopilot pattern actually says a lot of exactly what we want, and again it's by automating these repetitive and boring operational tasks as code in the application. I think that's critical, that's what the pattern is. But I'll tell you what, I think a demo is worth 1,000 slides and so for this I wanna bring up Tim Gross, another Joyent project manager whose working with me.

Before Tim joined Joyent he was head of DevOps, at a streaming media startup and running Docker in production as early as 2013 so he has more than a little history in the space and the demo especially is interesting. We introduced a blueprint some time ago related to—and many of our blueprints since have depended on a NoSQL database called Couchbase.

We got some criticism because Couchbase actually—it's a modern application according to some people, and it automates many of the custom management things that we're looking at. What happened was people challenged us, how can we do that with applications that are in common use everywhere today? And I think we have an answer.

We took that challenge and Tim I think is gonna show you.


Casey Bisson: Product Lead, Joyent

Tim's Demo: Applications on Autopilot