Jake Moshenko and Joey Schorr didn’t set out to build a business around container image storage.
When the two left their jobs on the APIs team at Google for the startup life, their fledgling business, DevTable, focused on creating developer tools that would allow people to code in the browser. “As part of that,” Moshenko explains, “people wanted to spin up development servers, which required that we adopt a containerization strategy. And at the time, this was like Docker super-early, pre-release version.” As such, there was no Docker private registry, no Docker hub—and DevTable itself needed a place to store and manage its private container images.
“We said, ‘Hey, if we need this, I bet some other people need this,’” says Moshenko. “So we went ahead and built in about a month, crazy fast.” They introduced the first version of this container registry, which they called Quay, at a Docker meetup in New York City in October 2013. It was almost an aside during a presentation about DevTable. Says Moshenko: “Our very first ticket was someone complaining that they couldn’t figure out how to pay us. We had forgotten to put the production Stripe keys into the app.”
Quite simply, they hadn’t anticipated the immediate, positive response. “We’d been fighting tooth and nail for every customer on [DevTable] before that,” says Moshenko. “Joey and I were both first-time entrepreneurs so we really didn’t know what traction was supposed to feel like. So it was like, ‘Oh, maybe it is just a slog. Maybe all the users are right around the corner.’ But then when Quay happened, we had the instant traction, and we were like, ‘Oh! This is what a business feels like. Got it!’”
The pair had hit upon a huge need in the space of storing binaries, and the customers rushed in. “We got to switch from building the things that we thought people might want to building the things that people were telling us they want,” says Moshenko. “The thing that we built Quay for was to store those build server images for DevTable. But we actually never got around to using Quay as part of DevTable, because we were just so busy with the inbound requests. Eventually DevTable was shut down so we could focus entirely on Quay.”
For about a year, the company—still just Moshenko and Schorr—grew organically. “We built it for what we thought the need was, and then people asked for basically all of the features that you’ve come to expect in a business app nowadays,” says Moshenko. That meant adding organizations where everybody could collaborate, team support, and audit logging so users could see what was happening on the system.
Still bootstrapping their business, the pair were working toward profitability when two things happened in their space in 2014. First, a competitor emerged. And CoreOS came calling.
Joining Forces with CoreOS
Alex Polvi, the CEO of CoreOS, had actually been at the same Docker meetup where Quay was launched. He was there pitching his own product, an operating system for running containers. Back then, Moshenko says, “We had a chat with Alex and said, ‘Hey, where are you telling people to store their images?’ And at the time there wasn’t a really good story. Joey had kept in touch with Alex, and about a year later, it just made sense for us to join forces.”
As part of CoreOS, Quay works in tandem with Tectonic, which is CoreOS’s commercially-supported, self-driving version of Kubernetes, with the overarching mission of securing the back end of the internet. Quay has expanded beyond the SaaS product that was being used by startups and other smaller businesses, and developed Quay Enterprise, an on-premise edition that is targeted to larger customers like eBay that have their own infrastructure and keep everything on their own data centers.
“If you’re a bank or a media company, and you have regulatory or security needs, then you’re probably not going to be using a public SaaS to store all of your binary data,” says Moshenko. “So we’ve really invested a lot in the Quay Enterprise experience, and let people use it in their infrastructure, whatever it happens to be. We’re building out all of the connectors and adaptors and giving them the best possible experience.”
The virtual machines that we start on the Packet instances start in like 6 seconds, and they’re ready to do a build after 30 seconds, so it’s way, way faster than our EC2 cluster. The other thing is that because we’re paying for the underlying machines and not the virtual machines, we step around the problem with EC2’s billing model where they charge you for an entire hour, regardless of how long you use the machine. So we just kind of found the cost sweet spot as well as the user experience sweet spot when we switched to Packet.
The Journey from AWS to Packet
Quay has been run almost entirely on Amazon since the beginning, but one of the company’s early decisions was to add a build cluster in their infrastructure. “Building a continuous integration system that’s multitenant and that allows other people to give you code to build and run is a scary security nightmare,” Moshenko says. “And so from day one, we’ve had to use all of these kind of standard strategies to isolate code and make sure that malicious code isn’t allowed to do anything that it shouldn’t be allowed to do.”
After experimenting with using Digital Ocean droplets (which, at the time, didn't start reliably), Linux user namespaces (which wasn’t close enough to the real native build that they wanted), and an Amazon Elastic Compute Cloud (EC2) machine spun up for every build (which was too expensive), the Quay team signed on with Packet last year to run their builds as virtual machines on Kubernetes. “We needed to give the Docker engines that are doing our builds on a very native feeling so that they can be as effective as possible but not make any compromises on security,” Moshenko says. “So what we’re essentially doing with Packet is building our own GCE [Google Compute Engine].”
And this system’s performance exceeded their expectations. “The virtual machines that we start on the Packet instances start in like 6 seconds, and they’re ready to do a build after 30 seconds, so it’s way, way faster than our EC2 cluster,” says Moshenko. “The other thing is that because we’re paying for the underlying machines and not the virtual machines, we step around the problem with EC2’s billing model where they charge you for an entire hour, regardless of how long you use the machine. So we just kind of found the cost sweet spot as well as the user experience sweet spot when we switched to Packet.”
In fact, what Quay is currently doing isn’t possible on any other virtualized cloud provider; only Packet supports nested virtualization. “If you were to run a virtual machine on EC2, you have to run it in pure software virtualization, which causes a 60x speed decrease,” says Moshenko. “Because Packet is giving us the actual bare metal machines, we still have that first layer of virtualization extensions that we can use to isolate our guest machines from one another. None of the other cloud providers provide that. We could have used the classic rack space style infrastructure provider, but we really like having the API to spin machines up and down without making a phone call or without signing any contracts. It’s like an Amazon-style experience but with the bare metal machines.”
(Moshenko also points out that during Amazon’s S3 service outage on Feb. 28, “most of our website was down, but our build cluster was happy! It was our yearly reminder about why we don’t want there only to be Amazon in the world.”)
Looking Towards the Future
So what’s next for Quay? “Everybody always gives the platform story,” says Moshenko. “I don’t really want to be a platform, but we naturally tend that way.” One of their first steps in that direction was Clair, an open-source container image security analyzer, which peers into the container and matches the images up against known vulnerabilities.
Looking ahead, he says, “We have this whole concept at CoreOS called operators, which is software that runs other software. Keep your eyes peeled on the Tectonic stack and on the work that we’re going to do around operators because it’s going to blow people’s minds. I’m really excited.”