The Evolving Container Ecosystem: What’s Next?

The Future of Containers in the Enterprise

February 10, 2016

New York

The Evolving Container Ecosystem: What’s Next?

So this is super panel, we got the All Stars here. A lot of you have been in Docker for a really long time. These are the, it's awkward to call you a greybeard of Docker. >> [LAUGH] >> Do you prefer a different, I don't know - >> No, I get it. >> [LAUGH] >> All right, Jessie is a total Docker grey beard at this point, and we wanna talk about the evolving container ecosystem, and what's next, where are we going? So maybe I wanna go through, let's go through and maybe give me a quick introduction about how you got into Docker, how you got into the container the ecosystem, and what you see coming 12 to 18 months.

Jessie, you wanna take it away?>> Sure, yeah. Previous to Docker I was actually using LXC containers here in New York start out to allow data scientists to get their analytical models into the cloud, and they would just spin up like LXC container and then I switched to Docker, and it was really bad at the time because it was not even 1.0, and I started contributing patches.

Where do I see it going? That's really hard, I guess just more of the ability cuz right now Docker itself is single host. More of the ability to have multiple hosts by default maybe. >> I got a better way to phrase this question, what angers you the most? Anger is the true mother of invention. >> Yeah. >>[LAUGH] >> Sowhat angers you the most about the state of the art right now? >> Oh man, so many things anger me - >. There we go! >> [LAUGH] >> We went from, I can't think of anything, to I can't think of only one thing. >> Logging, monitoring, there's so many problems with like everything with that.

Kind of off the top of my head, the first thing I can think of. >> Just overwhelmed with rage? >> Yeah >> Great >> [LAUGH] >> Exciting future. Dustin, what do you say? >> What angers me? Networking and storage really. How do those fit into containers? I think that's what we're seeing in the enterprise as far as when people try to bring containers into production. To post real workloads, you've gotta have a serious network and a serious storage discussion.

I got into Docker, I met Ben and Nick, Ben Golub and Nick Stinemates, we co-presented at a meetup in Austin, Texas, where I live, a couple years ago actually. And then, Docker was something I'd heard about, and that was sort of the coming out party, so to speak, and from that point on we were all systems go to really make the Docker in Ubuntu experience outstanding, and the Ubuntu in Docker experience outstanding.

And that's really been our focus. >> So you led with the anger, and ended with the romance? >> That's right. >> [CROSSTALK] >> I thought you were going to describe the night in Austin - >> [LAUGH] >> Take it away! >> How did you get into it, and what angers you? >> How'd I get into it? Originally, I was actually not interested in Docker.

I looked at the website and was like, eh this doesn't look all that cool. Then I checked it out again and I thought, wow! This could actually be kinda interesting. This can solve some little niche edge cases, and what hooked me was that ability to just instantly get an environment of some OS that I'm not running without spinning up a VM. Spinning up a VM is a very expensive operation on your local workstation, and it is on the cloud too, but that's kinda amortized, right?>> So when you say an OS you're not running, so you mean like ->> [CROSSTALK] >>I'm running CentOS, but now I'm running Ubuntu in my—>> Yeah, my daily driver is Gentoo, and I write a lot of scripts. I actually mostly maintain the, build scripts for Docker itself, and testing those in multiple environments was a challenge before Docker.

>> Interesting. >> Like differences between different Linux distributions, just the difference between running it in CentOS versus running it in Debian or Ubuntu. Even between Debian and Ubuntu, they share a base, but they're fairly different on the little subtle things that you wouldn't expect.

So— >> Do subtle things ever cause problems? I didn't think they caused it. >> No, no. Subtle things are never a problem— >> And it's statically linked so it all just works, right? >> Yeah, right? Yeah. So that was what got me kinda hooked, was this ability to just spin up an environment instantly, and be able to test my script and be done and that environment is gone, and I don't lose half of my available RAM for a VM that I used for a second to test a script.

>> It's interesting again because the history there, it's always interesting in that, I think it was a great asset of Linux to allow the distros to flourish. But now you're trying to like actually, can we prune it back down a little bit? >> [LAUGH] Yeah, yeah. >> Yeah that's cool.

What angers you? So what's the? >> Everything angers me, everything, everywhere. I'm a distribution packager, so I hate everything. >> [LAUGH] >> That's the way it goes, right? >> So those that actually have kids, have you been so angry at technology that you've actually shouted at your kids? I feel bad, I've done that a couple of times, I'm like, I'm sorry Daddy's not mad at you.

Daddy's not mad at you at all. >> I've done that to my wife, but not—>> There you go! right! exactly. You're the dunderhead! I'm like actually no! >> Yeah. >> No, actually Debian is just in bed, yeah right. >> [LAUGH] >> Yeah, yeah, there's some really angry Debian packagers sometimes.

Yeah, so I think what angers me the most specifically in the container ecosystem is that it's just not quite mature enough. We're getting there, we're a lot further than we were the last time we had this panel in New York. We're a lot further than we were. It's quite a bit more mature now.

>> It's kinda pubescence, isn't it? I kinda feel like it's like that old Simpsons about fluffy bunny having hair where there wasn't hair before, and— >> [LAUGH] >> I feel like we're in that awkward phase of like, wow! I didn't realize my body was gonna grow that long, and did I take that too far? Was that inappropriate? >> No, that was perfect! >> [LAUGH] >> You're really good at finding the metaphor that I was looking for.

>> [LAUGH] >> So ->> Or the one you were trying to avoid. >> Yeah, to unwrap that a little bit, just specific tools for like what should I use to monitor my containers? How do I monitor usage over, okay I've got 30 hosts and they're all running 20 containers a piece, how do I know what I have running? Questions like that. We're getting there, there are tools for that now that we didn't have at the beginning, and it's been a long rocky ride, and I think we're getting there.

>> I feel we're getting there too. So Ilan, what's? >> Yeah, so my first exposure to Docker was, I was at Ooyala, it was an online video startup in the Bay Area. And we decided we needed an internal platform as a service offering for our engineers to help get code out faster. And the team that was tasked with building that chose to do that on really, really early on Docker. And so we got to have lots of fun experiences and challenges with that, and since then I've moved onto Datadog, we're at that today and we get to do a lot of fun monitoring of things running in containers using our platform, and that's my exposure with, how I got started with Docker, and containers in general.

It's hard to think of what hasn't sorta been said here, and I think in general, I don't know that I'd say anger, but what annoys me tends to be things that are, just sort of rough edges, where things are getting so close to working exactly the way everybody wants them to, and then once in awhile like you said, you hit like an awkward unsanded edge, and you're like, oh! that splinter kinda hurt.

>> [LAUGH] >> And so moving on from that, but in general I think that's a sign of progress, the fact that we're getting to the point where we expect everything to just go—we just expect everything to work and when it doesn't, that's the surprise to us that angers us. >> We actually forgot when our face was full of the splinters.

>> Exactly. >> [LAUGH] >> It's like you might get angry at X today, but back in the day, how many days did you spend trying to get that video driver working, and now it just works 90% of the time, so. >> Yeah, totally! And X used to be this huge frustrating thing that now everyone just kinda of like, works.

>> Ctrl-Backspace, how many times again gets it working. >> Yeah. >> Start xtrussing. >> The kind of the classic was either trussing, or xtrussing your X Server, forget what was going on. It's like well, that's obviously that's the end of you, because the X Server's output, the X Server's now stopped waiting to output something—it's always a bit sheepish going to the next person asking them to kill your session. I'm sorry.

Stephen, how did you get into Docker, and how did you get into containers? >> So previously I worked at a company called Iron.io, which let you execute people's code in different languages, and the biggest thing was dependency hell. So as they started adopting Docker, and people would give us Dockerfiles, and we would execute their code in these Dockerfiles, it sort of just went away, and that was beautiful.

But those were ephemeral processes, so started looking at ClusterHQ, and started concentrating on stateful applications. I think seven out of the top ten Docker images are stateful applications, but they're very poorly documented on how to run them into production, so— >> Seven of the top ten Docker apps— >> They're like Postgres, like Mongo— >> Oh, right.

>> Might be seven out of the top 20, >> Right. >> Or something like that, but there's very little documentation on how to run this in production, and there's low confidence right now on doing that. People are experimenting with it, but it sorta angers me that people aren't confident enough to run like a Mongo cluster, or Postgres master slave in Docker >> Or maybe it angers you that they are confident enough to run a Mongo cluster - >> [LAUGH] >> I bet you can't wait for that one.

>> [LAUGH] >> The biggest part of that would be managing the state that goes along with a container. >> Right. >> So, to echo Dustin's point, state is a big piece of applications. In the ephemeral instances people will connect to like RDS instances, stuff like that and hard code their keys into their images, so it's a little bit gnarly and— >> And state is kinda the elephant in the room, isn't it? We kinda like to pretend like, oh 12-Factor, great.

But there needs to be a state somewhere, right? And someone has to keep track of state and ->> Always state. >> Always state. Yeah. >> To take this from a sandbox environment to something that you use in production, that's really key. >> So, how do you deal with state? Let's talk about state for a second.

>> So, I thought I would take this opportunity to perhaps stroke your ego a bit, and tell you that Ubuntu— >> Careful, it'll bite you. >> [LAUGH] >>You may want you to back off! >> We'll see, we'll see. It's been a theme of the day I think, and looking at some of the smart lessons we learned from Solaris over the years, and how that's made it's way into containers and Linux in general.

Ubuntu 16.04, which ships in a little over a month is actually building ZFS directly into Ubuntu. >> Yes! >> So we're shipping ZFS ->> [LAUGH] >> [APPLAUSE] >> And- >> My god! Wow! Are you sure? Don't rush into that clean water, you don't wanna! It's kinda fun to have cholera! Something everyone we can talk about! >> [LAUGH] >> I think that's an important part of the story, man! ZFS brings really advanced snapshotting and a fantastic experience.

zfs.ko is part of every Ubuntu install as of 16.04, apt install zfsutils will get you the user space. LXD, the machine container hypervisor that's installed as part of Ubuntu sets up ZFS underneath it by default, and we'd love to see that in Docker as well. >> That is awesome! That is great to hear.

From sort of my own perspective, it allows you, once you have that layer below you that you can trust, you can do a lot more interesting things. So that's— >> Absolutely. >> Great stuff. >> And to assure you, that's stroking Bonwick's ego not mine, by the way. >> [LAUGH] >> I'm cool, that's fine.

How else do people deal with state? ZFS is gonna become an important piece of the puzzle. We certainly believe that, I've been using that in production for a long time. >> ClusterHQ has their open source library called Flocker, which enables Docker users to better failover, and have more resilient volumes to their Docker containers.

We're trying to concentrate on state across the entire development workflow though, through realizing that every day developer who pulls down the latest Docker image won't have the latest dataset that should go with that image. We have this open source tool called dvol, which is very similar to Github where you can push and pull a volume up from something very similar to a Docker hub.

Right now we're calling that Volume Hub, and we're trying to improve that workflow across, right now it's development and production environments, but then in between there's testing and staging environments. So when you're running your staging tests, you wanna make sure that you run those tests against the latest version of your dataset, or something very similar to your production dataset.

What we've seen is a lot of people are transferring files on Dropbox, Google Drive, trying to just share volumes in every which way. Saving data inside of a Docker image, which is pretty nasty, so hopefully improving this workflow for developers. >> What could possibly go wrong? >> [LAUGH] >> Like a 2GB images is a joy.

>> And there's no password in that data. >> Yeah, there is none. >> None at all. >> This actually gets to another interesting issue about secrets, so can someone give me the secret to secrets? Cuz maybe we can— >> That actually might be my most pain point, and I totally forgot about it, it infuriates me.

>> Is secrets? >> Yeah. >> It's not really a container's fault, or Dockers fault, it seems to be kind of an intractable problem, right? So how do we square this? Maybe we tell people, and what's the state of the art today? >> Right now it's just like people doing crazy hacks that are really bad.

>> Can we rank how bad? So everyone's sinning, but some of us are actually committing like murder one, and some of us are just jaywalking. So it would be good to know, can you help me? >> Yeah, murder one I would say would be just throwing it in the container then pushing it up to the hub.

>> Do that, you will not walk, you will be convicted by your peers if you sow secrets in the image, that's murder one, don't do that. >> Not even attempting, just doing it. >> That's wrong, that's wrong. You were not raised right if that's, okay, fine. >> Then there's like the whole build arguments that we added, I think two versions ago, it's kind of like an environment variable, and people are using those for secrets even though we say, don't use them for secrets, don't use them for secrets, don't use them for secrets because there is nothing hidden about it. It is literally there in the history of your image, and yet people are like oh! I'm gonna use this build environment variable, and [SOUND] There you go! Plain text.

>> So that's maybe conspiracy? >> And then push it to the hub. >> No, no, no, no, no, wow! So that's in the same bucket then, where it's like your secrets are now, let's not call them secrets anymore because that's not what it is, it was a secret. What are some of the other things? What's the right way to do it? If someone wants to do it, I wanna do it the right way, what's the right way to do it? >> The only cool thing that I will say is just approved, because I'm so picky. I saw CloudFlare has some cool secrets thing with certificates in their Red October stuff, and I think they're hopefully going to release something, or write a blogpost on it. But that's the only thing that I've seen with regards to like decrypting on run like a secret stored in an image that I think is actually viable.

>> So the only viable thing is something that hasn't been announced yet? Okay, so this is like the secret war, you have a plan to end the war in Vietnam basically, okay. >> There is no plan though. [LAUGH] >> All right, so that was the problem with it. >> [LAUGH] >> Okay, I'm sorry.

>> I find it really amusing that at the last panel we had here in New York >> Yeah. >>We talked about secrets, and we don't really have a better answer today than we did then. >> Isn't a more intractable problem? Cuz the thing is it's not just like an implementation problem, right? >> Yeah.

>> It's not like, oh by the way, like go fix this bug.
>> Yeah. >> Right? You've got a fundamental problem, you've got this immutable infrastructure, you don't wanna put it in there. In some ways you need to actually defer to some other substrate to actually say, okay I'm gonna inject this secret, I the trusted thing am gonna inject the secret into you. I don't know what the— >> Yes, I think we saw quite a bit of this in- >> Stop tittering, that was not meant to be vulgar! >> [LAUGH] >> Get your minds outta the gutter! Just because a man makes a couple of pubescent references— >> [LAUGH] >> All of a sudden everything that comes out of my mouth is filth.

>> [LAUGH] >> Moving along. >> Moving right along. Yeah, I think we saw quite a bit of this in the early days of Amazon AWS with the problems associated with AMI, just thousands, millions of instances all cloned from various sources with very little provenance. >> Right. >> Within Ubuntu as we started working with each of the major clouds to ensure that, the Ubuntu that's in that image is one that is the generic signed Ubuntu image produced by Canonical. We had to develop a technology called CloudInit, which is what injects metadata.

>> It's the trusted thing. >> Exactly! >> Cuz we have to trust something, right? I think it's what it really boils down to. >> And CloudInit pulls information from metadata, and then configures that instance, and that could mean SSH keys or SSL keys, or passphrases, or something like that.

>> So the thing that kinda sucks about that, it is kind of intractable, aren't we gonna have a slightly different solution for every kind of substrate that you're running on, am I missing something? >> It seems like the container, the things that launch our containers and know who we are, are the things that will know whether or not you're trusted enough to have these credentials.

So similar like when you launch in Amazon, and eventually they came up with IAM profiles and IAM roles, an idea that they could inject short lived keys into you that way, and I imagine, I don't know, maybe that's where the Kubernetes, Mesos, whatever thing that's launching you, maybe that's something that can eventually be the source of that identity— >> It will need a metadata service that provides that information to the container that runs, and that container would need a process by which it consumes that metadata and adapts it, and applies it to the container.

>> The issue is not an implementation issue, it's kind of an agreement on what other implementations are gonna look like, it's a standards issue almost. I can say the S-word. Yeah, I know! I'm sorry! >> [LAUGH] >> I love meetings like these. >> You've just guaranteed that the problem won't be solved for eight years.

>> [LAUGH] >> Yeah. So Ilan, honestly you're kinda wearing the CNCF hat, it's kind of a good problem maybe the OCI to pick up. I'm not sure, but it would be a good problem for one of these bodies to kinda pick up and solve. How do you deal with secrets? If it's a secret that's cool! >> [LAUGH] >> I don't know that we have anything.

We have a Docker-based agent that you use to deploy DataDog when you wanna monitor your containers, or the host. And I don't know that I want to admit to which of the crimes we commit— >> [LAUGH] >> If you need your lawyer present like— >> [LAUGH] >> In all seriousness, as Jessie was saying, I don't know there's a good solution today, so.

>> So definitely that's a big one that we gotta go tackle. So kinda pumping back to state a little bit, for those of who are actually developing—or actually let me ask a question first. So the rise of microservices, and the rise of Docker, and then the rise of say, Node and Go, what's coming first? What's leading? What's pulling? These things all seem to be kind of one gigantic avalanche of change.

What's leading versus following, or maybe are they reinforcing each other? Am I wrong to even connect these things? >> They're interconnected at different layers. Go, is a fundamental change to the way we think about writing software, and it's a pretty profound change. And I think you've seen a lot of the smartest, best new software being written to solve big network problems being implemented in Go, and that's beautiful.

I don't think that's at the same layer though as say as Docker, or Kubernetes, Mesos. So I think hierarchically you've got your building tool chain, and you build a bunch of solutions from that. Docker is incredible infrastructure for solving that microservices problem, but it takes a Swarm, or a Kubernetes, or a Mesos or something on top of that, that can then orchestrate all this, those containers.

Leading versus following, right now it seems like a sort of bunch of cats running [LAUGH] in various directions. >> [LAUGH] >> A bunch of like wet cats. >> Yeah. >> It's like the cat that falls into the bathtub when it's full. >> Indeed. >> Tianon, what's your take >> I think Dustin's spot on there.

It's definitely a lot of cats. I think there is a lot of, [BLANK_AUDIO] Man! I just lost the word I was looking for. A lot of working together on these things. I think it's not really as much a competition, as it is we're all in this together, we're all using the same kernel features at the end of the day.

The way we orchestrate those is the only real differentiator, and so I think it would be extremely foolish of us to try and discount one another. Does that makes sense? >> Totally. So it is open source, and the fact that all these things are open source, it's amazing, right? >> Yeah, that's fantastic! >>It's like literally the whole ecosystem is open source.

>> It's funny to think if you're not doing open source today you're doing it wrong. It's been around for as long as we've been around like pushing the open source thing, and Microsoft is doing open source now? >> What! What is that! >> Microsoft is on Github! >> [CROSSTALK] >> Am I the only one who gets uncomfortable, like a little bit nervous and fidgety it's like, why is dad at this party? >> [LAUGH] >> But dad's rocking out, I gotta admit he's got some moves! You're like wow! Okay! Running like Core COR, when you're talking about running ZFS, are you running ZFS, Ubuntu 16, ZFS. You can run Core COR, you got like literally, we are through the looking glass.

We're running Core COR on Ubuntu on ZFS - >> Kind of unreal. >> It is kinda unreal. >> It's the golden age of open source. >> It is. >> Let's enjoy it. >> It is. It is actually—so part of me in my darker moments, cuz hope it's not an golden age, right? I hope it's - >> It's a really long golden age.

>> Because if we see kinda of a re-rise of proprietary software, and not to get too dark, but I'm a little worried that service, that proprietary services are kind of becoming the new proprietary software. Where you've got kind of people, instead of having software lock-in, it's service lock-in, am I? >> No doubt! GitHub's not open source, Gmail is not open source, your SaaSes, your Twitter is not open source, Facebook is not open source. So yeah, you're right. The SaaSes are still not open source, maybe one day we'll get there.

There are kinda crappy open source alternatives to each of them.
>> [LAUGH] >> Right hey don't worry, each of those has crappy open source alternative! There's like a gnu Twitter. >> [LAUGH] >> Is it like, Stallman, are you trolling me? Like some foot-eating Twitter. >> [LAUGH] >> But yeah, you're right, I'd love to see those communities come along and start implementing, or the companies behind those who love open source for a lot of things start open sourcing more of their infrastructure.

>> Well, I think it just shows you how much things have changed even in the last five years, where we have seen that, yes open source has been around for a long time obviously, but it's like we are really accelerating, just the fact that all these folks are open. So that obviously creates a lot opportunities, what problems does that create, having everyone be open, if any? Maybe it's all 100% win.

>> Bikesheds. >> Bikesheds, yeah. >> [LAUGH] >>Business models. >> Yeah. >> Figuring out a sustainable business model. >> Yes, has anyone figured that out? I can ask them to leave, it's okay. How do you make money off open source? >> [LAUGH] >> I probably would compare— >> [LAUGH] >> Heroku is a good example of a closed source platform that people love, and there's all these open source-like Heroku clones that people have made, but none of them have taken off.

And for a good reason cuz there's a ton of immense value that they've created by being able to take in money and hire engineers to work specifically on their platform. If they open sourced it, I don't know if Heroku would be as great as it is. >> Yeah, I think business models are a real challenge.

>> I don't know that the value in Heroku is that they're proprietary. It's that they've built a service, and that takes away all of the pains of running all of these other open source PaaSes for you. >> Right. >> And we're sitting on stage here with folks from Docker, who've got their roots in dotCloud and know there's a lot of, that you guys now you've got, you know the Tutum folks.

There's clearly a lot of value in taking those pains away from folks who could run these things open source in-house, but would rather not, and rather focus on their business value. And I think that is one place where you can make money in open source world. >> So I agree with you cuz I think people will pay for a service, right? And so our model is that at Joyent, like our stack is all open source, and then we monetize it with on prem software that we sell support for, it's still open, same open thing.

At one point they were like, people at Joyent were like, there were very few people who thought this, but like, do we have to actually build it for them, why don't we let them build it themselves? Come on! >> [LAUGH] >> Just build it for them! I think we're making it really easy for people to deploy things themselves, but then we monetize it with the actual service.

One question I've got, if that's a model that works, and to your PaaS comment, Stephen about Heroku. To me Docker makes it a lot easier for a developer to generate a PaaS that someone else can use. Is that fair? >> Yeah. >> I wonder cuz I think the big problem with the PaaS is that you outgrow it.

It's like it's great when you were small, but now that my app is getting crushed and I just race around, and now we're gonna make this thing fly. >> There's a couple of general purpose PaaSes available. Cloud Foundry. I call Kubernetes and Mesos sort of PaaS-y, PaaS-like. >> Are they a PaaS? >> Orchestrating application containers to me— >> It starts looking like a PaaS. >> It's absolutely a PaaS, the lines get super blurry, right? >> But what I've seen ->> To just your bikeshed point. It's bikeshedding time! >> [LAUGH] >> But for those three or four general purpose PaaS solutions, I could name a hundred bespoke PaaSes that I've seen behind the firewall in infrastructure that are based on Docker, or LXC, or Let Me Contain That For You, or some other container-type solution. The bespoke PaaS is like the in-house software.

I think I've seen statistics from IBM that said 90% of all the software in the world is sitting behind a firewall somewhere that was custom-written by some institution. >> Okay, so let me ask you this. I personally wonder if we're not gonna see, if the bespoke software is gonna skyrocket.

My reasoning for this is that software is becoming literacy, we're actually winning, and I don't know about the schools you went to, but where I went to school— >> If you can call that winning. >> If you can call it winning. Well no, but it's interesting we're really democratizing it.

Yes the peasants are gonna be able to read! Yes, I know that! But we have to brace for that. But I think that the, sorry peasants. >> [LAUGH] >> They were very excited about the ability to read. In all honesty, the most popular course at these schools is the introduction to computer science course. And you'd think that, that coupled with the fact that all this stuff is open source, coupled with the fact that software is eating the world, when you add all that up, and you get a lot more people developing software that's suited to their purpose.

>> Software used to be scarce. Who remembers going to Babbage's to buy software? >> I worked at Babbage's! >> [LAUGH] >> I worked at Babbage's. >> Software used to be a really precious thing that you had to— >> I was a greeter at Babbage's. >> [LAUGH] >> Wear my little tie, and then I would try to shepherd people over to the ComPyrus.

I was a ComPyrus salesman when I was 15! >> There you go! I was buying Zork. But software is not scarce anymore, Git clone, whatever you want and do that over, and over thousands of times, and there's so much software. There's over 50,000 binary packages in Ubuntu. You can apt install 50,000 unique strings, 53,475, I looked it up before I got here, and that is for something else.

That is just amazing! And you don't go to - >> And that's all it takes to get it running. >> Yeah, that's it. >> [LAUGH] >> Once it's installed, it starts up, and it's running,and it's good to go, that's amazing. >> [CROSSTALK] >> It's the world's first app store. You didn't used have to fire up the Apple app store to get stuff out, just apt get install what have you, or yum, or—>> Exactly.

>> So do you think we're gonna see more? I think we're into more software being written, right? And to me this is like a really important facility that Docker, and the container ecosystem is serving, is actually the democratization of infrastructure, and allowing many more people to write software.

This is actually a very good thing I think, right? Although I reserve the right to be curmudgeonly about it some years from now. Because when you think about it, our ranks are gonna grow enormously over the next decade, and we've gotta be, I think, braced for a whole—I think that containers are—part of the reason for enthusiasm on containers is because they're essential.

We actually couldn't do what we need to do over the next ten years with everyone provisioning their own VMs, I think. Am I stoned? It's been a long day, I've been doing doing drugs since— >> [LAUGH] >> I was wondering what the secret was to your energy. >> [LAUGH] >> It's all pharmacopeia, you should sit down. No, actually I don't do drugs for exactly that reason. You don't wanna see [INAUDIBLE] >> I look at containers in two ways, and I think they solve two very different problems, the replace VMs-problems, and do your microservice, your application architecture smarter-problem.

And that's really the difference between an application container that runs one thing, and one thing really well, single binary, in that application container. And if you need to run another application to make that one better, Apache and MySQL together, you put them in separate Docker containers, and you federate them together.

And you do that at large scale, and you end up with a PaaS, Platform as a Service. The replace-VMs is a slightly different problem. I think it's fundamentally different in that replace-VMs means a system that looks like a Linux system, that boots a Linux system, that runs the supporting infrastructure, the syslogger, and the TTY, and ssh, and Chef or Puppet, or whatever else has to go into that system, replace a VM.

Now it's Linux on top of Linux in most cases, right? When you were running Linux on top of Windows, or Linux on top of VMware or something like that, you needed that full virtualization infrastructure. Today it's Linux on top of Linux let's dedup the kernel, have one kernel to update at the end of the day, and run machine containers.

Machine containers replace VMs, and when you scale those out, and you have a lot of those you end up with infrastructure as a service. So I think it's important to solve the replace my VMs with containers-problem with machine containers and infrastructure as a service, and let's build a smarter, more modern architecture for my software with application containers and platforms as a service.

>> Jessie, is that a reasonable distinction, what's your take on that? >> Yeah, I don't know if you can fully obviously replace a VM because at the end of the day if you wanna run some other OS like Windows on top of Linux, you can't. Containers are great, but it's not a Sandbox yet.

There are ways that you can escape a container, if you're smart enough to do it, and you can find it. And with regards to your kernel, it's easier or harder depending on which one that you're running, and I don't know, I feel like VMs still have a place in the world, but— >> Are you trying to give me a SmartOS layup here? That's like alley oop before - >> [LAUGH] >> I appreciate it if so. >> I do, I do like SmartOS actually.

And like the whole— >> You don't have to say actually when you say that, you don't have to say actually. >> [LAUGH] >> People are always shocked - >> I know, but you just didn't have to say actually. That's all I'm saying! It's not you, it's great thing, just embrace the love. >> [LAUGH] >> But let's talk about that for a little bit because obviously with the caveat this is our angle.

So we've taken the Linux System Call Table and put it on top of the SmartOS kernel, that allows you to run all of your different Linux variants, including Alpine. Am I the only Alpine lover? You must be an Alpine lover. >> I do like Alpine. >> No actually! No actually! >> [LAUGH] >> You say Alpine, and it's just like obviously you're in love with Alpine and so am I, that's fine.

But like there's no actually there! See we need to get, all right, so the SmartOS love we gotta upgrade to the Alpine love. All right, but sorry. >> [LAUGH] >> You love Alpine? >> Totally. >> Totally. I think Alpine is, nothing against our great friends at Canonical and Ubuntu, but I think that Alpine is really interesting. Those of you don't know Alpine, super-thin layer, based on not glibc. So there are alternative libcs. So if you want like the truly alternative, exactly! Ilan is just like! Neckbeard or like facial tats.

I'm not sure what's that like, that ecosystem is. >> [LAUGH] >> So it uses musl and then BusyBox, and it ends up being ridiculously lean. Yeah I find— >> It's great. >> Yeah, it's great! It is great, no actuallys, it is actually great. Anyway from our perspective we're running those Linux binaries on top of the SmartOS kernel, but on top of the Linux Kernel, you do have issues where you can escape.

I hope like we've seen ZFS in Ubuntu, I hope we're gonna be able to knock down some of these security issues. >> Yeah, if you're running like a grsec and you are using now our setcom profile you just eliminated a shit ton of crap that can happen in containers. Because I actually was one of the people who helped on the default profile, and I went through the entire Linux kernel mailing list for all known vulnerabilities for the past few years, and those are all blocked.

So you just like killed a bunch, but I cannot promise things like the sandboxing, cuz it's just hard. >> But a lot progress is being made? >> Yeah. >> Yeah that's great. >> If I may, this kinda goes back to our previous point about we're all in this together. So I think having multiple competing implementations of this container runtime helps with this problem because we're all looking at the same issues from a slightly different angle, and so they're more likely to be discovered and thus fixed, and we all benefit from that.

>> And I think this is the power of open source as well, that by presenting that, there's a democracy of ideas, which is very important to kind of allow some of these checks, and allow for many things to get better. Even if you're only using me to make Linux kernel better, but that's fine, that's good. >> The security has to improve to really succeed in production.

Cgroups of course is where it's starts, sepcomp and grsec are absolutely steps in the right direction. User namespaces and unprivileged containers, I think are essential to making this work. Ensuring that even a compromise inside of a container results in the user breaking out of that container, and being an unprivileged user on the outside of the host is incredibly powerful.

Personally back to things that make me angry, one thing that makes me angry is that I have to sudo Docker anything. The fact that I have to run Docker commands as root, that those Docker containers have to run as root, I really want those to run as unprivileged users. That's the world that we work in with LXT right now, and I'd love to see it, and I think it would be incredibly important to see Docker moving to that direction.

And then the last thing is - >> Last week, Docker 1.10, fixed that. >> Excellent! >> So you can set on the daemon, run user namespaces and it's done. >> Excellent. >> [CROSSTALK] >> It's not on by default yet. >> Right. But it's getting there. >> Yeah. >> [CROSSTALK] >> Good, excellent. >> Sorry, I didn't mean to derail you.

>> [LAUGH] >> That's perfect, good to know everybody should know that. >> Yeah. >> Don't sudo anything if you don't have to. >> I guess speaking on security, there's an issue of people not updating their images for security loopholes, and there is a fellow at CoreOS that was talking about Clair.

Yeah, oh CoreOS, I think. They created Clair, which analyzed Docker images, like 80% of them still have Heartbleed in them. So I think secrets, and how people are building their images is another behavioral security problem that the Docker community hasn't really had to face upfront yet. And it's probably going to take something major for us to learn our lessons as a community.

It hasn't happened yet.
>> The concern is that the vulns that are in extant images, like Heartbleed that have not been updated, and to a certain degree it's a challenging problem because it's not necessarily a container problem, it's a problem that's being facilitated by containers.

>> Yep. >> So it's kind of a nasty one. Ilan, how do you? >> You now have how many containers, I think Datadog did a study and people are running on average four containers per host in production as we were saying. That means that now you have 4x the number of Heartbleed, the number of SSL libraries to update.

Like you said I don't think it's a problem that containers have caused, it's just a situation now you have to be, as an administrator, as an application owner, you now need to understand your stack in four or five, six different places, and knowing that you now have to update all of these different pieces. So it's more to keep track of, I don't know that there's an answer to it beyond what we've learned in our classic systems administration, know what you're running and be ready to respond to security issues around it. >> Well, and this is the peril of, we have democratized the infrastructure, which is basically a win, but there is this peril over here, that we've gotta go address somehow.

>> Luckily we've made it easy to build all of these things, and so rebuilding a container with newer libraries isn't necessarily the end of the world. >> And hopefully easier to balance these microservices. >> Right, well yeah. >> And since we've democratized it, we haven't really trained all these developers on how to be good system admins, which is a huge problem I think.

>> But it's a practice that doesn't really get taught in computer science education, right? We talk about algorithms, we talk about data structures, we talk about distributed systems, at what point do you talk about good data hygiene, or good system hygiene? I don't think source control came up once in my entire CS program, right? [LAUGH] I was the odd one using CVS somewhere, like why are you doing that? And these are things that just don't come up in education, and it's even worse now that, and it's gonna be aneven a bigger problem now that like you said, you got everybody can just do this— >> Right, we need to kind of change the way we think about the way people are educated. >> But on the flipside of that I wonder how many mySQL VM snapshots there are in the world that are also vulnerable to Heartbleed, and we just have no idea because there's no visibility into that.

These people aren't publishing their Docker image for mySQL so we have no idea. >> By offering some transparency maybe we can actually take what is a pre-existing problem—I do think a lot of the problems with the container ecosystem today are pre-existing problems that have been made worse just by use. These are pre-existing problems.

>> Multiplication. >> Yeah, right, right. So we've got a couple minutes left, I want to give folks an opportunity to ask questions, if other folks had any questions. Yes. [BLANK_AUDIO] Right, so the question is, 80% of these containers are vulnerable, but are these things even using SSL in a server capacity, and how do we even figure out what that means? Yeah? >> I think the answer is why are you even having OpenSSL on your container then? >> Why do we even have OpenSSL? Can we just ask, like come on! Roll on Libra SSL please, can we please! >> But even with Libra SSL, we'll still have the same problem, vulnerabilities still happen. We're still human beings, we still make mistakes, whether we meant to, or not is debatable. But we still have this problem of vulnerabilities come up, things we didn't intend for our code to do, it's doing in the wild and it's not a good thing, and we need to upgrade.

>> And maybe I guess Jessie to your point is that you should be really—maybe the problem is that you have these containers that are not narrowly tailored. So to your point, yes they're not using a SSL, but if you're not using SSL, why is it there? Of course, you're saying you maybe using it as a client.

And maybe that's where we're gonna see, certainly like Alpine allows you to really, really, really tightly tailor things. Is that something we're gonna see going forward? >> You can certainly auto-update Ubuntu. If you apt install unattended upgrades, and turn that on, security updates will be installed automatically as soon as Ubuntu makes them available.

That doesn't quite work inside of Docker containers. >> [LAUGH] >> Because Docker doesn't run Cron or anything like that, but that's our sensible default in machine containers. When you treat a machine like a virtual machine, and unattended upgrades is on automatically installed and turned on, you're protected from that.

It would be nice to see those updates applied at the Docker level as well. >> Okay. >> I think that's good hygiene. >> So we're just about running out of time, if folks have burning questions I'll be sure we get to that. If not, is there a question back there? So I wanna wrap up by—one of the things that I've been wondering with so many different everythings coming up and seemingly a new one all the time, I've wondered when we're gonna to hit peak confusion in the container space.

So where are we relative to the peak confusion? Are we at peak confusion? Past peak confusion? Not yet at peak confusion? Is confusion gonna grow in 2016, or are we gonna get clarity? Is 2016 gonna be the year of clarity? Jessie? >> I feel like about five more schedulers, some more UI apps, and then we're there.

>> Five more schedulers, okay. So I'm just gonna count down, so when we hit four I'll be, come on! one more! And we are done! please! Okay, five more schedulers. Dustin? >> I put us about 2012 in the cloud world. So about 2012, I think most people had started to grok what cloud meant to different people. But I think we've come quite a bit further, when we say cloud we know there's different types of cloud.

>> Okay >> And we're about there in containers. I think we've rounded the bend. >> Rounded the bend? >> Rounded the bend. >> Past peak confusion? >> Past peak confusion, still have work to do. >> Okay. >> On containers specifically? >> Container ecosystem, the whole. >> The container ecosystem in general, yeah I think I'd agree on, just in general, generally I think that we're on a trajectory that's going straight up.

>> Confusion pointed right to the roof. >> Yep. >> Okay. >> Especially because of that point you made earlier about the, we're teaching the peasants to read, and- >> You don't have to repeat that one if you don't want to. >> [LAUGH] >> No need to tweet! Please don't tweet that one! I hate Twitter.

Unfortunately when I say things controversial, they shouldn't fit in 140 characters. >> [LAUGH] >> To illustrate what I mean a little better, when way, way back the Bible wasn't readable by normal people. You had to be an actual priest in a church to be able to read the Bible, and so no one could understand what was being said unless it was told to you by a preacher.

We got a lot more confusion after we could read the Bible. >> [LAUGH] >> Right? Because we could actually debate back and forth. >> You have just out-metaphored me, touche! >> [LAUGH] >> I'm going through like, we're gonna have Lutheranism, we're gonna have the Reformation, we're gonna have— >> [CROSSTALK] >> Yeah, oh my god! So we've got a religious war is what we got coming up! >> The Inquisition.

>> The inquisition, like there's all sorts of ugly, that's a hell like wow! Okay. >> So your next talk is gonna be about be all about that? >> That is- >> Cool [CROSSTALK] okay, my mind's blown. Peak confusion? >> I hope we're plateauing for a little bit on confusion, but I think it will probably pick up at some point here soon, so we should— >> Oh, so we're doing okay? >> We're finally starting to collaborate, You mentioned— >> So it's like the eye of the confusion storm is blowing over? We've got like the other, okay. >> For a while we're gonna work together on standards, and then there'll be a couple of folks that are like, well, these standards, I can build a better one, or there's too many of these standards, I'll build a new one. Give it a year, and then we'll be back up on - >> Okay, so we're gonna take a breather on confusion? But don't leave it— >> I'll leave it though, I think that with every community the number one thing we need to work on is education around what we are offering in the tooling.

Whether it's events like this, or online training, what have you, on ramps for people trying to become part of this ecosystem. As you said, folks that have not been systems administrators, or developers before are now starting to become them, let's make sure that when they show up, there's a getting started guide, and put them on the right track from day one.

It's a lot easier to, as they say a stitch in time prevents nine type of thing, let's do that. >> What's apt to produce confusion. I agree it's not actually a weather system, it's religious warfare. >> [LAUGH] >> Stephen, what do you think? >> I'm gonna go home and run Mezos on top of Kubernetes. >> [LAUGH] >> That's running a Docker Swarm cluster inside of it, all on SmartOS.

>> Running it all on SmartOS, running Core COR, let's do it! Let's end the cycle. >> I'm pretty confused right now, I think we're past peak confusion. The frameworks have started to mature, I don't see other players really coming in at this point with drastically new ideas, at least in the next year, or so.

>> Consolidation, yeah. >> Until we've seen these players play out, these players play out. So I think we're headed towards a lot of people shooting themselves in the foot, though. There's a lot of tools, they don't know how to plug them together correctly, they don't know the best practices. So I think- >> Good news less confusion, bad news more physical pain. >> Past peak confusion towards a boom!, or something. >> All right, and we've got like the confusion singularity happening.

That is great! Folks, join me in thanking the panel, thank you guys. >> [APPLAUSE]

Speakers:

Jessica Frazelle: Core Maintainer, Docker

Dustin Kirkland: Ubuntu Product Manager, Canonical

Tianon Gravi: SVP Operations, InfoSiftr

Ilan Rabinovitch: Director, Technical Community & Evangelism, Datadog

Stephen Nguyen: Developer Evangelist, ClusterHQ

Borja Burgos: CEO and Co-founder, Tutum

Moderator:

Bryan Cantrill: CTO, Joyent