Docker & Microservices: A Perfect Couple

The Future of Containers in the Enterprise

February 10, 2016

New York

Docker & Microservices: A Perfect Couple

So I thought we could come, actually so three of us that have actually done the panel very similar to this before, and we were joking that we were going to take this show on the road, but Scott has joined us and you're like the new band member here I like it. So anyway, before we kinda talk about running Node in Docker, running apps in containers, I do feel like we're kinda mastering that, like we kinda know how to do that now, and what we're moving towards now is this idea of like a whole datacenter as an operating system, a whole cluster is the OS, but we're kinda in the early days of it, go back ten years and we didn't have something nice and curated on the single host operating system, we now have like this systemd experience, I know you've lots of feelings about that, but we had like it was the init script era, we had a init, we had scripts, we had bash, we had scripts running all over the place by alphabet—yeah, that system boots by running scripts an rc directory in alphabetical order, you're like, oh yeah that makes total sense on how to boot the system.

And if you look back from now it kinda of looks scary. But I was gonna say is what we're in now is that era of the cluster, we've got like scripts running all over the place, we've got—like there is no highly curated experience. And I think that part of the scariness in embracing the kind of Docker container world is moving from a highly curated single host experience, to this kind of open world and my kind of thing that I was gonna throw out is I think Node is actually a great candidate for architecting those systems.

I know Joyent does this, you wanna even— >> Yeah obviously Node is very well suited to kinda of glueing these things together, it's a combination of features, but the use of a asynchrony—and people kind of debate this. "It's the fact that it's JavaScript." I think JavaScript is important but I think it's the fact that asynchrony is so well treated, from our perspective.

>> I also think one of the things This is what we discovered when we [INAUDIBLE] >> I don't think he's hot. Oh - >> Start again. >> Your mic's fallen. >> Oh yeah, yeah. Oh wow. >> That was awkward. >> That was awkward, right? >> That's what that sound was. >> [LAUGH] >> Okay, where was I? Let's start again.

So the module ecosystem around Node is done right. So when we started writing systems in Node back in 2011, whenever it was, we were naturally driven to write small components, and just small things that did things well in cooperating process. Now we weren't deploying them in containers at that time.

But the module structure was right, and so it's a natural fit as you say just to move that modular structure into containers, it just fits perfectly. >> Yeah, modules are a form of code reuse and what we're seeing now is these containers or running these small apps as services is just another form of that code reuse.

>> Right , absolutely >> Yeah, yeah >> I was thinking about this earlier how this all played together and how at Dow Jones we're actually really early in the cycle with Docker. When I came to Container Summit last year we were just experimenting with it. Now we actually have half a dozen of apps that are in production.

And I was like, well how is Node or any of microservices and how do they all play together, are they key to each other and it wasn't that alone, it was like all the little pieces. Like moving to Node was a big step in architeching our applications in a way that works better with containers moving— >> Yeah.

>> moving to the cloud was a step and I feel like each piece is very incremental. >> Yeah. >> So I do a lot of prototyping and some of it is on the product end customer facing, but some of it's on the internal side, and I finally use Node more and more for like the glue bits in between components.

So, there's many ways. You could have it like participating in the cluster orchestration, you can have it participating in cluster management but more and more, I actually use it in band so it's like I can put a proxy layer in Node, but I control all the logic there so if I wanted to do service discovery with it, that's no problem.

If I wanted to do service discovery via etcd, I pull in the etcd module and then I have full control over it. I'm not kind of, something I call config file meta programming. I'm like, my hands are really literally on the request. I can do whatever I want with it. >> Yeah I think that's exactly it.

And I think that, Node—you know it's funny a lot of us got into Node, because okay we'll just prototype this real quick in Node >> Yeah. >> And then if we need to, if we that doesn't perform well because it's JavaScript, or what have you, we'll rewrite it. And I can't tell you how many of those components we just put into production, and they've worked great and we've not needed to go—the number of things we've needed to go rewrite is essentially an extremely low single digit number and most things that actually—I mean I never would have thought—our DHCP server is written in Node.

I never would have thought that that would be—but it's a gem. >> I pulled that module the other day, we are talking about TJs module, DHCP—sorry I'm talking DNS. >> You can take—our DNS server also. >> I needed to hijack the DNS at some point, and I was like well, I could try and figure out what the config files script is, or I could just write a DNS server and once I'd written it I was like, that was super easy.

Like I could add anything to this, I could have that participate in like your zookeeper etcd, and then serve DNS that way, or just serve it to where I wanted things to go and like it was super easy to like get going. Having said that, I work at a company that we have a lot of Java and Ruby, and one thing I found is that we are familiar with Node, and we are like yeah, you just do this, asynch programming. And that's non-trivial for a lot of people. And if you're coming from, if you're coming from say architecturing enterprise Java, a lot of the patterns that work well there you can't actually transfer to Node.

I'm not saying those are bad, and I'm not saying the Node ones are better, what I'm saying like they're such different platforms. >> Right absolutely I mean when you talk about enterprise patterns, I mean that's just wrong. I mean if you look at the code bases that get produced using those enterprise patterns, they're enormous and they're difficult to grok and people have real problems with them. When you come across to Node, and you start doing things, just do one simple thing with one service.

It's very liberating. I was on that journey as well. I used to write a lot of Java code and coming to Node was just a liberating experience. I'd never go back okay [LAUGH] >> Strong feelings right there >> Absolutely >> How about you >> I mean same thing like those half dozen application that are going, like we don't have—we have older systems that are doing orchestration, but we don't have a lot of stuff around Docker orchestration and we're not even at the the point to do multi-container stuff where we deploy this whole service where we have like eight microservices which are very well orchestrated. What Node allowed us to do is what we have now inside of Cloud Formation. We run a simple Node like when I have chef or something more complicated but ultimately the Node app is really simple because all of our Node apps are that way, so just plucking that out and putting it inside of a Docker container and then telling Cloud Formation to run that instead of chef.

>> You're using it there as like a infrastructure management tool. >> Like we are definitely not gaining all the benefits right but it helps us manage infrastructure, it helps speed up scaling. There are distinct benefits and obviously the developer work flow is certainly easier than we had before, and we didn't have to do anything.

I'm not saying our developer work flow is hard, but we had to build all of that out to make AWS infrastructure management and configuration management as easy as we could for the developer. With Docker, we can just throw a bunch of that stuff we wrote out because it's there. >> Yeah, and that's part of the promise of Docker the ability to actually effect this kind of dream of the small system philosophy, and what you're talking about of coming to node and never going back to the kind of world of monstrosities >> But it's not like discovering Node, for me it's almost rediscovering the way people use to write systems, if you look at what is the most successful system on the planet probably, it's the Unix kind of a environment. >> [CROSSTALK] >> It's been around for a long time, ps, ls, all these little commands, it survived because it's good.

>> It is the single most important revolution in system software is the transition—we we're talking about at the ball last night cuz I get to. >> I learn what multics was>> Yeah, do you actually not know that? Are you just placating me with that? If you are, please do. It's >> [CROSS_TALK] >> To your credit I will say that when you find out someone doesn't know something, often people will react like, how could you not know that.

But you are like "I get to tell this story!" >> [CROSSTALK] >> I love that. >> Multics was this giant functional but very large system, and Unix was a reaction to that system. Unix is actually, it's a pun, it's a play on a eunuch on the castrati and they're taking a very large multics and making it much smaller, and it is such an important revolution I think. And Go is the same way. I think Go and Node I think are siblings in this kind of modern infrastructure because they effect that small systems philosophy.

>> Did you think that we are in the multics?. Are we about to hit a revolution with this datacenter OSes, or are we past that revolution? Was Docker that revolution or is there like super something coming. I think that Docker is a very big piece of it in terms of bringing it to a much broader audience.

Just like with Unix, actually cuz you don't wanna overly romanticize Unix the implementation. >> Mm-hm. >> If you go back to the code, there are some people that kind of reminisce like "oh Unix is only five thousand lines." It actually sucked. I love Unix. It was very important but there were so many error conditions that it would just give up and die. And I think we're in a similar space with containers where the idea, the abstraction is right but there's a lot of the surround that needs to be made.

>> This is where I am a little bit out of my depth and you're just gonna like the bumpers on the side of a bowling arley you're gonna like, when I start going off the rails just pull me back on, but I was gonna say it seems like we're following a lot of similar patterns in how we architected single host systems in the cluster and by which I mean how we did code reuse and shared libraries and like and like linking, for example. Like I asked, I think someone at your company once who works on kernels, and I said how does the linker work? Cuz I was trying to learn this, and I think he just shook his head like it was dark magic and walked away. And— >> it is a miracle whenever it runs, by the way >> and if anyone would like to put me aside and explain it all to me I would love to hear it, but I think we're experiencing a similar thing in the cluster, and one of the problems that a lot of people run into is like, I have all these services and I need like this service to go here and there, like I want internal people who are testing the edge of this newly deployed system to like take this rough path, but I want my enterprise customers to go down the happy path and you kinda roll them forward incrementally.

But that's like kind of hard to do, and we're still figuring out all those tools. So our linker engineers for the cluster are still hard at work .>> Yeah it's nota solved problem, right, not by any means. And I think we'll see a lot tuning developing in this phase— >> And I think Node will be—a lot of people will be experimenting in both Node and Go at writing these kind of init scripts for the cluster and these pieces.

>> Yeah I think all these tools are amazing. Docker is really good. For the past like three to five years, certainly in the enterprise, it's probably going on longer outside. The tools in isolation have been really great, but putting them all together, it hasn't worked yet - >> We're still figuring that part out.

>> So that will be the next thing. So for the past three years it just been the individual tools I think now what we need is standards around that. >> So given that we now just like frightened everybody, I was thinking okay how do we now convince people to use containers and what I came to was would you go back 10 years and say hey don't get into computers until systemd arrives like it's just not worth it. >> [LAUGH] >> That doesn't make sense. Stay in computing until systemd arrives. >> What I mean is like don't wait until there's a highly curated experience because you're just gonna be left behind like this is where we're matching now and you have all these really great tools for kinda carving out exactly what you want— >> I think the best practices aren't there, but the tools are definitely at a place now where anyone can quickly build something, we use a bunch of different things, and I actually just put a presentation together for management and one of the key slides in there is to say we only pick Lego technologies, right we only look at—which is one of the reasons we love HashiCorp, right?>> Composeable pieces.

>> Let's take this piece out which is the workflow, and then us ourselves and a bunch of other people write plugins so—because that's what you need to do in the enterprise, that's whole thing, like stuff will just churn, like even if we get to this magical solution, then immediately after that someone will start working on the next thing and building out those tools and that workflow, so… >> One pattern I love in Node is that often in Node there's really asmall module so it will be a module that does like one thing.

All right I talked to AWS I'm like I wanna give you some information. What I like about that is you can just use lot of them because if one of them ends up being totally broken, actually not that hard to replace it. You can write your own, it's just you're kinda leveraging the fact that other people have done a certain amount of work And you may outgrow it.

>> You don't have to tear down the whole thing, you generally just replace the API or— >> It's the same thing if your building microservices, get rid of one if it's not working, change it, swap it but you don't have to grok an entire company codebase in order to make that change. You're changing one small thing, right? >> I think it highlights a power of Node, and to be fair in Go as well, which is what I call the library philosophy versus the framework philosophy. And I think that part—again, it's the small system, big system mentality, the framework mentality "I the omniscient framework control all and a magical and I will call out to you in these certain contexts," and on one hand it makes it really quick to be able to—rails for example classic—J2EE, classic examples of frameworks and on one hand it makes it kinda quick to build something functional. On the other hand, when you actually wanna go extend that when you wanna make that robust, it can be really challenging because the framework is in control. Whereas Node has a very clear library mentality where you are in control of the code flow, and you pull the tools out of the tool box that you need, and as a result like, yeah you pulled out this tool and actually it turns out like you don't like it very much or it's not working for you or, as we discovered when we did a license audit, it's like the one thing that's GPLD in the thousands and thousands of Node modules, okay you need to be replaced. But you can do that without disrupting the system because it's not a framework, you're not changing the brain stem of your application because your application has been composed out of these library models.

>> You've sold—everyone here is writing an email right now, we must switch to Node, and we're gonna—we gotta warn them, though, what's the bumpy roads that they're gonna hit and just to assure them, yeah you're gonna hit this bumpy road but it gets better on the other side. I have a few that I can say I just curious if you guys have a view.

Error handling, I'm just gonna throw that out. Error handling in Node needs a little bit of work I'd say. >> Compute intensive stuff, there is one right there. It's very easy to tie up the event loop. Don't do that. Use the appropriate tool for the job, Node is the appropriate tool in a lot of cases, but for computation intensive things, it's not. Use C or something else.

Well not strictly error handling but another thing Docker-related and both of them is even if you do a great job—we were talking about Node error-handling before, and how stuff can just get swallow, even if you manage to avoid that problem and you might write that out to a log in your Docker Container and it might disappear before its gets collected or whatever if you use the wrong approach >> Like writing the log to the like an ephemeral file system? >> Yeah.

Right. >> If it crushes then it's— >> So you have to pick the right tools for the job and process to make sure it's fundamentally different. >> I would think that with error handling, I think you need to be deliberate and disciplined. I think that if you people kind of complain about callback hell but to me you need to get through callback hell, understand it and you need to generate the discipline around it.

Because if callback hell is the problem you've got a much deeper problem in that your application doesn't really understand what's going on. And Node is—and JavaScript too, JavaScript is much more subtle than people realize, it is actually Lisp in C's clothing and you need to understand what a closure is and it's like sorry it's kind of a tough concept to understand when variables bind and when they bind, and what that scope means.

But if we don't grind through that now you're just gonna be in for a world of hurt later on. >> It's kind of an easy bit at the beginning but then to kind of get into your—a lot of people are like I can't do anything complex. Well you haven't crossed that hill yet. You can do a lot of easy stuff right away, then there's this hill and different languages have these at different points >> Right >> Once you cross that hill you kinda stop having those problems. >> And JavaScript, I think the advantage of JavaScript is that because it is a failed state, it is the failed state of languages, in that it seems nobody is actually is actually in charge, and it's like glue-sniffing teenagers wielding Kalashnikovs come barreling through the language and all of a sudden man does not triple equal man.

I just found that out recently, man does not triple equal man. What are you serious? >> Having done some numerical stuff where I needed that to exploit that back. >> [CROSSTALK] >> You're the guy that exploited the fact that man does not triple equal man? >> I believe I used it at matlab, and I think they have the same situation.

>> Let me be the confessional booth, if you wanna actually. I mean I'm glad you feel this is a safe space, and we can provide that that's great. >> [CROSSTALK] >> I think that JavaScript on the one hand it is maddening, the lack of design of JavaScript and so many of these things like why, why? >> I think I can answer that.

The why is that they don't wanna break the old Web, and we don't wanna move forward and like there's a lot of sites that are working and we don't wanna, "I'm gonna change this and then change this," And you break the old without necessarily benefiting the new. Just like don't do those things.

I understand >> [CROSSTALK] >> I totally agree. I think that the JavaScript has more of those >> [CROSSTALK] >> When JavaScript was a week old, they didn't wanna break it's like the bad decision made on Monday, we don't wanna break on Friday, okay. >> [CROSSTALK] >> The thing is that because it is not opinionated, it's very flexible >> Uh-huh >> And one of the things actually, there's a little bit of a refugee movement from Go into node because people complained that I'm writing the same code over and over over and over and over again in Go, and actually with Node it's a little more of free for all.

And if it's like if I—except it's a failed state in like if you get knifed in a failed state, like that's your problem, don't call the cops. I guess like - >>[CROSSTALK] >> But I'm the good cop of, Node.js, though, what is that implying? >> You're the good cop in a failed state.

>> [INAUDIBLE] I got into Go after doing Node for a while, not got into it but like tried to dip my toe into it, and that's exactly the reason why I backed out, I was like no that's too cold and I got right back out of it, because I was like I don't understand why isn't there this large module ecosystem, and why do they encourage me to write things over and over again.

I didn't buy it. >> I think I agree with you on the whole groking closures. You have to do that, otherwise, what we find is people sometimes bring that object-oriented baggage from another language and then try to implement that again in JavaScript which is kind of the wrong approach.

>> And having said that, I went back to doing some work in Java the other day and my brain was in full asynchronous mode. No, no, I like- >> Don't they have like ->> [CROSSTALK] >> I let go, I admit it but I brought my fully asynchronous brain and in Node you often set up some handlers and then you kick things off.

But that doesn't work when you blocking on the handler and I'd just totally written this Java in code totally incorrectly so you can't, really bring it over and you can't really bring it back either they're just different modes. >> [COUGH] Modes>> Yeah, yeah. >> Its not impossible, you just have to invest time in making that switch.

You have to understand that it's going to be learning some new stuff, and learning new architechural patterns, and learning new language patterns, and you just kind of move over and it works, it works totally fine. [BLANK_AUDIO] >> Yeah, drop the mic. >>[CROSSTALK] >> I'm sorry, it does work and I think then you are reduced to some of these challenges—I do think because its a bit of a free for all I think that the more discipline you go in with the better your results will be.

>> Yep >> So, I mean should be using JSLint, you should be doing things—there are things like, why does the language does allow you to be like yeah I assumed you want to eat the handgun, that's why you put it in your mouth. I know. You knew I was drunk at the time, but it will allow you to do things that are obviously wrong and if you do actually need—I think that's what we have found, is that you've gotta pretty assertive on tooling. And actually, I think things like TypeScript are actually interesting.

I don't know, I mean we are not using TypeScript, but I think it's kind of—I think it's actually probably everyone's attitude of TypeScript is they think it's interesting but they are not using it themselves. >> Pretty much. >> Facebook was working on a like static analyzer for like untyped JavaScript try to use it with Node I think it needs like a small layer of compatibility, but the more like—true static analysis is really hard to do in JavaScript when it is completely un-typed, but you can see the whole program and in Node when you are like all your modules are local that is actually possible, you can actually infer a lot of information I think we'll actually get some really great tooling out of there we'll be like I think you need a property on that I think you are calling this wrong I think that static analysis and types go together but one doesn't necessarily imply the other.

>> This weakness is also kind of a perverse strength because like in lots of other languages developers will be like, you know .Net they write their code and then let me write some tests, write some BS tests but really not comprehensive and you try to encourage them because it's a good practice that's going to help the quality of their code, it's going to see them into production with less failure in Node you go to the same developer and he's not going to do that.

>> If you want people to drive slower don't put an airbag put like a big spike. >> That's why I said "perverse strength." >> but I do think there is a little bit of you have to be very self aware and I think error handling is a good point, because I think that we are—because there are different kinds of errors in the world, and when we're developing distributed systems, a lot of the errors we need to deal with are not programmatic errors.

There are operational errors. And operational errors and programmatic errors need to be dealt with completely differently. >> You wanna give like one line about those differences. >> A programmatic error is this code is broken. >> You typed the variable name wrong. >> You typed the variable name wrong and now we've got a type error coming up.

You tried to do a reference undefined. >> So it's never gonna work >> It's never gonna work and the idea of like oh well let's catch that exception and try it again. It's a bug in the software. >> Operational error would be like you tried to reach out over the network and the host wasn't available. Your program is written correctly, but there's something beyond your control, an exceptional stage, you actually It's like a well defined outcome for you, they weren't there. Like DNS failed, or I tried to open a port and it didn't work.

>> And you need to deal with these things and think of this things totally differently, and these are not—the fact that we even use the same word for them is unfortunate because these are totally different concepts, and the way you need to handle them and deal with them is totally different, I think the language doesn't—and even Node doesn't necessarily help you because Node doesn't necessarily help you clearly differentiate those.

>> There's a long thread about this right now with promises, which I will spare everyone in the audience from getting into but it's a very active area that we're trying to improve for Node quiet frankly. >> I think we all touched on something interesting differently, and that is that the microservices, Node, Docker all these small composable things, what happens is they make infrastructure super—a lot more accessible to developers who couldn't kind of envision or din't have the access or the view of the whole system, so it's a big boon for them, but it scares the crap out of ops guys who in our old world had complete visibility on understanding and control, and now everything is broken up and pulled away from them.

So I think there is a big adjustment on both sides.
>> There's a big adjustment I think that we have to, we kinda have to develop this cross-operational empathy. We have to understand—developers need to understand what production looks like, and production needs to understand what development looks like.

>> And as you go to these clusters the whole Netflix, Chaos, Monkey, that just becomes like you have to adopt it in some form, and it turns out that now, I forgot to put an error handler there and it crashed, well that was gonna happen at some point anyway. Like every app is gonna, it could be a hardware issue, you could write the most perfect program ever and something goes wrong at an area you can't control and it crushes, you already have to account for that, so I think it changes how detrimental the error handling situation is, because you have the account for every node possibly going away or network partitions, or all sorts of madness to be engineering in these clusters and it follows the same grain.

>> And this whole thing forces you to pretty much forces you to automate a lot more than you had to before which is also super beneficial like somebody had talked earlier about constantly being called the blocker, right? And we get that complaint on the team I'm on now, it's like a DevOps team and we costantly get that complaint, but I have to come back right away and say, yeah but it happened in integration or staging, that problem used to happen in production for you, so there's kind of indirect benefits as well to moving to this model which these tools promote.

[BLANK_AUDIO]. Drop the mic. Now we scared everyone from using Node, we scared everyone from, but I mean I personally like using Node to prototype and even roll things in better like these glue pieces, and I would say more and more—I'm always kind of amazed how quickly I can go, like Node streams, like if you're getting into Node and you're feeling up, if you really wanna harness like the power, the true power, get into streaming, because streaming is kinda crazy, like all of a sudden you're just like wow, I can proxy anything to anything, I can proxy DNS over a web socket and then it's gonna go through like an HTTP tunnel and then it's gonna go here and then send it to etc, like you can just start doing all these things and once you get ahold of that, like you look at these clusters and you stop seeing like, oh I can apply this framework here and that framework there, you just the protocols.

It's like in the Matrix where you look and you just see the code now, you don't see the people and I find that I think in protocols a lot more, I'm like, well you know, if I can just get that http request to upgrade to a duplex socket, that would solve my problem. >> That has always been the case, the more developer—before any of this ever occurred, it was always my philosophy that the more a developer understands about ops the better the developer they are, and that is generally all an operational concern but now like I said before these tools are making that very transparent and accessible to developers, which I think it's a great thing.

>> You think in terms of the messages as well in these systems right, what message is going where and you'v almost got like a language for the system expressing the messages right. >> So what are some cool modules people should check out. >> For microservices development is it? >> It can either be for orchestration or kind of like glue in the middle like proxying, or even just a handy module that you're using to develop your app in a cluster ecosystem.

>> Right, we have a really cool module we put out called fuge, it's a development tool for local development of microservices, so that's my recommendation. Check that out. >> F-U-G-E right. >> F-U-G-E, yeah absolutely. Because one of the problems we experienced, is suddenly you've got things decoupled in a whole bunch of services you wanna tie all that up locally, you do development so having a rapid environment or workbench for microservices is what we are trying to achieve there, so absolutely check it out.

>> Peter gave a cool talk at Node Interactive on demoing fuge in videos. Check it out it's pretty cool. I don't wanna speak for you, but what you're allowing people to do is allow people to develop microservices—kind of a family of microservices quickly, which is definitely a challenge when you've got a distributed system.

>> Absolutely. It's born out of experience with projects, again your point of automation is we were scripting, and scripting, okay three strikes done write a tool, automate, then move on. >> Like I said I'm doing most DevOps these days, and the glue for me is how do I get the software development to pipeline from writing the code out to production smoother, and you've heard it 1000 times today but all of the Hashicorp tools.

Right, like they're pretty much invaluable to us more than any—like Jenkins or whatever you use is great, Chef or Salt or whatever you use is great, but Hashicorp is the only one out there that like end to ends packing up my images, how do I ship them, how do I secure it. Now I have this distributed cluster, I have all these secrets freaking everywhere, what do I do with them? Vaults. Like they've just attacked every problem in this cool independent tool that's supper pluggable, so if you're doing any of this stuff just read—and their docs are great.

So just literally hit every product on their site, and it's all open source and free except for Atlas. >> Totally I go with these guys, in terms of specific modules that we like to use, so we're big on restify, we wrote restify at Joyent, restify allows you to build your own HTTP-based microservice.

What we found was that the way of doing this prior to restify, Express and so on, was really more designed to serve up a web app, like someone sitting in a browser and that was really how it was optimized, and what we wanted to do was really pure HTTP and so we did that with restify, and the important thing we did restify from our perspective is that we—when you only focus on a microservice, then you begin to invest in things like tooling, debugging support, and so on. So we've built in DTrace support into restify, we built in debugging support into restify, such that we can walk up to a service and if we wanna know where it's spending its time in its routes, we can light up a DTrace enabling, and then like know exactly where we are actually spending time. >> And I think it's worth mentioning debugging Linux core dumps, and that's actually pretty amazing.

>> Yeah so this is where you get to our bias, coming to Node really from the perspective of we're a bunch of kernel developers and so. >> No linkers though. >> What's that? >> No, no linkers though. >> I will learn you some linker. You'll find that it's like a miracle birth to be able to link a—when you run a binary the horrific things that need to happen to make that binary happen.

I will show you the horrors. You can come up river with me.
>> This is gonna be like the after show at the happy hour. >> I think for us, one of the problems, and actually what is still a nasty problem in Node, is things run out of memory, programs run out of memory and it's really hard because—people kind of like to blame the GC, it's not a garbage collection issue.

It's a garbage creation issue. >> [LAUGH] - >> It's the fact that, why isn't the garbage man taking my garbage? Because it's all in your living room— >> Still in your house - >> Yeah. It's still in your house. That would be breaking and entering, and I agree it smells terrible and it's rancid, but they don't know to break into your house and throw it out.

So I mean, and that's the problem, and without understanding closures it is really, even if you do understand closures, it is super easy to accidentally hold on to a reference of something that holds on to a huge tree of objects—>> Part of this is that V8 is taking JavaScript and actually making it quite performant.

I mean anyone who's using Chrome on the web, or an Android phone, like you are hopping to test the performance of Node, because we're inheriting the speediness of that— >> I like the way you're thinking of it. You're not actually using a browser you are just like guinea pig for Node.>> [CROSSTALK] >> Yeah.

Now I totally forgot what I was gonna say but to do that, you know they're making a lot of speed optimizations, and one of these is like you don't always realize what the GC is holding on to, what references because I think you link to this one in the same scope and that has another like there's weird implications sometimes.

>> It's nasty. And what we have found is that what you have is an application that's running out of memory, and running out of memory is still—running out of memory and running disk space—these are the two problems we are not gonna outrun in this business. We are still running into—these are like the problems and the reason these are really nasty problems is because the failure modes are bad.

When you run out of memory and run out of disk space, you don't tend to just blow up. You're just like, I can do work give me work and you're actually not responding and we don't do well. If we actually get the hardware failure, we deal well with that, but we don't do well running out of memory.

What we found is that we really needed to invest in the tooling to be able to take a core dump of that process and then. >> Just like…I don't know what a core dump is, let's say what's a core dump? >> A core dump is simply a static image of that, the process has died, we are going to take a static snapshot of its address space and dump it to disk. It's an idea that's so old that core comes from magnetic core memory so this is like super old.

I'm glad we don't call it a mercury delay dump, which is literally the technology that immediately pre-dates core dumps are mercury delay lines. That's how old this is, a very, very old idea. >> What I like about it is you can actually get this file and you can load it up, and you can actually look and say actually print out what JavaScript object are in my memory and you can even start seeing what's holding on to what, and that's, and I don't know like V8 is making a lot of optimizations, so often what you wrote in your JavaScript file is not actually what's running, it kinda changed the form a little bit and the part that for me is pure magic is that you're able to undo that, I'm looking like—I'm not looking at like whatever line of assembly code got complied out from this optimized function, you're like that was actually this line of the JavaScript, and this is the function, this is the constructor and here's all the arguments and I just see magic again.

>> I'm glad you appreciate it, basically have gone into the raw sewage and telling you what people ate for dinner which is a stinky task, and it's really, really hard but it's incredibly invaluable and because we use that time and time again to nail these really nasty problems, these are production-level problems and we can kindof debug them asynchronously, so very important for us.

So restified and Bunyan would be the other one I'd mention in terms of modules. A Bunyan is a loging module, its like a log for J but done in kind of a a small systems philosophy. Again, built in DTrace probes, built in debugging which has been invaluable for us. To be able to walk into a microservice that's in a bad mood, and is not doing work, or not doing work acceptably, and be able to light up all it's Bunyan probes and say okay show me all of your debugging output without having to restart the thing.

I can't tell you the number of problems we walk up just and just nail like that. >> I think we should maybe take a few questions if there's some, got a question right down here. >> [INAUDIBLE]. >> So I'm not sure what you mean—[INAUDIBLE] >> So JavaScript probably is the wrong tool, if you're actively managing your own memory, you're gonna find that you and the VM are going to be working at cross purposes, and there are ways that you can do it, but if you're managing things that statically, you probably don't wanna be doing JavaScript, for something like that, if it's high performance.

In our experience, maybe you wanna do it in Node, and you do it with a plugin and you have the plugin explicitly manage some memory, which you guys have probably done as well. >> Yes, absolutely. >> [INAUDIBLE] >> Well you actually can't because the JavaScript heap size, the V8 heap size—>> [CROSSTALK] >> It's also worth clarifying, so every time you create a JavaScript object that's going on to the JavaScript heap, and that has, what's the limit right now? >> So the limit was 1.2, you can kinda goose it up to like six point something, but that doesn't include buffers, and it's worth like whatever, calling that out.

So data that it's buffering off of like an HTTP request, I don't think that counts towards that limit. >> So it's a low memory limit. I think it has actually been an asset for Node, because if you hit that limit there's something wrong with the way you're architecting, I don't know what— >> [CROSSTALK] >> It's kinda of a barrier.

>> [CROSSTALK] >> Right? >> Right, yeah, you don't have a kind of 28 GB heap that you might find on a JVM. >> Yeah, we're not running like we ran in Java, that's most of the memory limits like inside of process managers and stuff that we do have set up is way lower than that, you might be somewhere between 128 and 512 megabytes.

Because if you're running outside of that you're probably writing too vague of a service, or you shouldn't be using Node. >> I think, that's the key. The interesting thing is like, no your service is too big. I am killing you not because we don't have memory available for you, but because I think you are an addict and it's just gonna go up your arm.

>> So there's like >> [LAUGH] When you're kind of a— are you calling the JVM an addict? >> [LAUGH] >> I'm just saying like don't blame the JVM money, cuz you're not gonna see it again >> [LAUGH] >>The last time the JVM spent the night at my house, I woke up and the silverware was gone again.>> So there's kind of a value, an area where you're scaling where if you could just throw more memory on the single host, that would solve the problem and that's one criticism I hear, and why a lot of I guess people don't—Node isn't investing a lot of time solving their problem because soon you'll hit another layer where you need to go multi host and then you're back—so why not just do it a little sooner? I guess is the solution.

>> Think about your problem now, and find your leak too. If your're leaking let's figure out where your leak is. >> That's just chaos engineering. >> Treat that as a plus. That's a feature not a bug. >> Right. >> Do we have any other questions? >> [INAUDIBLE] >> This is where I'm a victim of my own experience.

I left C++ for dead in 1996 assuming that it would die of it's wounds, somehow I actually blame Google for this. Google actually took C++ into it's house and then like somehow brought it back to full health, and it's now destroying the village again. So I'm not sure how that happened.

>> [LAUGH] >> You obviously can, and I think C++ is probably a fine fit. I think of the challenge with C++ is gonna be the agility and gonna be the speed. I mean the common theme for today has been developer speed and I don't think I've ever heard anyone accuse C++ of having made them go faster.

>> On one note, if you get into native addons for Node, it's actually fairly easy to integrate with like any C-style library, and that's one benefit. >> [INAUDIBLE] >> You can actually pass your objects from JavaScript into like—you gotta take your JS object, turn into like a C-style object, and then send it to wherever you want and you'll get it back cuz it's all running the same process space.

>> This is actually very important point so Jacob's point is, you can actually make that choice in the small in your microservice, and when I said we had a microservice that we needed to rewrite, we actually didn't rewrite it. That's actually not accurate. What we actually did is we took one component that actually did need a big chunk of memory, and we made that an add on and then left the rest of it. >> Kind of keep the logic in JS and put the meaty bits in C++.

>> We're out of time, but thank you everyone for coming. >> Hope everyone learned something thank you very much. >> What's the next stop of the band, Minneapolis? >> [CROSSTALK] >> Come on. >> Okay, thank you very much. >> All right. >> Thanks guys. >> [APPLAUSE]


Scott Rahner: Engineering & Productivity Lead, Dow Jones

Bryan Cantrill: CTO, Joyent

Peter Elger: Director of Engineering, nearForm


Jacob Groundwater: Senior Software Engineer, New Relic