Is Deploying to Production 50x/Day a GOOD Idea?

I had the pleasure of attending the Continuous Deployment Breakfast with Eric Ries last week. Eric Ries, Develop Feature Increment - Verify on Buildbot - Live in Productioncreator of the Lean Startup methodology, Co-Founded IMVU who’s known for deploying code to production as many as 50 times per day!

In Agile, there’s this notion that there’s someone “out there” (e.g., the Product Owner) who has all the answers about what the product should do. However, Lean Startup says no one has all the answers. This, of course, makes sense in a startup where where the customer is To Be Determined. But I wonder if it doesn’t hold a bit of truth even for more established products… does one person really hold all the answers?

And so – if we’re not sure what the right thing to build is, our most important task becomes learning.

What’s the fastest way to learn if the product is going to make sense for people? Get it in their hands! Of course, Agile recognizes this – which is why we work in short, 2 week iterations. But Continuous Deployment says why do we have to wait 2 weeks? Wouldn’t it be better if we could get continuous feedback on what’s working and what isn’t? Wasn’t that actually the original idea behind agile (ala eXtreme Programming) – that the customer was supposed to be sitting in the room with the development team? We don’t hear about that as much today because it’s proven so impractical in most situations. But maybe Continuous Deployment offers a better way.

50 times per day?!

I realize it sounds pretty – well, extreme. But I think that might be because the way we work today isn’t set up for it. I keep thinking back to that crazy, radical concept of “daily builds” in the late 90’s. This notion that we should integrate everyone’s code every day (zomg!). We’d work on these huge chunks of functionality that took a month or more to code. At any time within that month our code could be in any random state of working or not. It didn’t matter because we were just trying to get it working by the end of the month. But the idea of integrating that code with everyone else’s every day – well that was crazy. We weren’t working like that. In Lean-speak, we were working with month-long batch sizes.

Agile has brought our batch sizes down so we can fit several within our 2 week sprints. And, as we’ve seen, lowering our batch size has helped us work faster, be more flexible, and get more feedback so we’re better aligned to real customer needs. But still, continuous deployment is asking us to drop our batch sizes even smaller – where each developer actually arrives at “done” one or more times in a day. Is it crazy? Or is it just taking agile to the obvious next step?

But… isn’t Deploying everybody’s changes right into Production a really terrible idea?

Yes! Probably. So we have to build what Eric calls an Immune System. This starts with what we’d normally do for Continuous Integration. As soon as code is checked in, it’s integrated with the rest of the system and if the build or any of the automated tests break, the new code is rejected.

Okay, great, but the code might pass all its automated tests and still have a fundamental problem, right? Eric likes to give the example of turning the Check Out button to be white text on a white background, “effectively turning your startup from a business into a hobby.” We need more then just automated tests to make sure everything is working. We need to also be monitoring user’s behavior on the site to make sure everything is copasetic.

When we’re totally focused on learning, we want to be monitoring anyway – is anyone visiting our site? What are they doing there? Are they buying anything? Think Big Visible Charts – but about what our customers are doing rather than what the developers are doing (because, in the end, all that really matters is what the customers are doing, right?). So add some alerts if anything goes wonky (say check outs suddenly drop to 0) and now our immune system includes not only our automated tests but also evaluating user activity to make sure it’s still healthy.

Okay, but… is it going to be Usable?

Do we seriously want to throw changes out there that a human being hasn’t looked at yet?

According to Eric, 80% of the changes we make are behind the scenes, so not even visible to end users.

For the remaining 20%, yes – we do want a person to look at them. But, in the case of a brand new product, wouldn’t we prefer our early adopters be the ones to look at it? Ya know, the ones we’re trying to learn from so we can have a successful product. Or, maybe there’s a certain customer that specifically asked for this change, or provided input on it’s specs. What if you could selectively release it to just that person and get their thoughts on it. Wouldn’t that be more useful?

Release is a Marketing Term.

If there’s one thing to make the entire company crazy, it’s releases. Getting a release out on time, making sure sales doesn’t “oversell” what it can do, making sure it actually works in production and doesn’t break a bunch of other things.

At IMVU, Eric said most all of their initial deployments of features were done only to a small percentage of customers – customers who they could talk to to get feedback from – so they could learn if that feature made sense.

Then, once they’d validated/tweaked the feature into what it should be, then marketing could decide when they wanted to deploy it to everyone. This totally eliminates the stress around releases because: we know exactly what the feature will do, we know it’s ready, and – most importantly – we know it works with real users in production!

So, what do you think? Can deploying to production 50x/day be a Good idea?


6 responses to “Is Deploying to Production 50x/Day a GOOD Idea?”

  1. Yves Hanoulle Avatar
    Yves Hanoulle

    Yes it’s a good idea. See the whole Devops movement.

    Will you break your product?
    At some point it might happen. Hopefully pretty early on.
    Anyway, the fact that you are able to deploy that fast, you can fix it fast too.
    When you deploy directly life, your developers will care even more about quality. (You don’t want to be the one that killed your life product)

    At Flickr they talk about +/- 30 uploads per day.
    They make decisions on algorithms on their live website via A/B testing: 1/3 of customers get old code, 1/3 get new code option A, 1/3 get new code option B. They have ton’s of logging behind and they decide based on the actually behavior of the code what is the best solution.


  2. Abby Fichtner Avatar
    Abby Fichtner

    Thanks, Yves!

    Great points and really interesting about Flickr.


  3. I don’t know ANY end-users who want their applications changing 50X a year, let alone 50X a day! Who has time to give feedback on a daily basis? This sounds like something that’s appealing to the developers, but I don’t see the value for users, and I see a LOT of risk.

  4. jdtangney Avatar

    We’ve been doing Continuous Delivery (aka Continuous Release) for a while now, and we really, REALLY love it! And our users are very pleased with the pace at which improvements become available. Naturally, have a battery of Continuous Integration tests just to start with. Then we have what we call “release tests” that run against the (clustered) app that’s about to go live. Occasionally, we have to hide some work in progress so that we can check it in without breaking the UX. But it’s all good.

  5. The Abbot of Unreason Avatar
    The Abbot of Unreason

    I think it’s easier to let the customer think like a customer instead of a deployment team trying to think like one. I wonder if we’re not starting to see a level of tolerance from customers that wasn’t supposed to be there in the 90s. Nobody ever says “perpetual beta” any more, not because it went away but because it’s a normal way of doing things now.

  6. CuriousityForLife Avatar

    As long as deploying 50x a day won’t exceed budgeted costs and are still within prescribed efficiency levels I guess they are okay!! What firm or team in their right minds wouldn’t want to be this productive right?

    Western Digital My Book Live Deals

Leave a Reply

Your email address will not be published. Required fields are marked *