Posted on
by

When IT Says No: Creating Fast Feature Flow
Gene Kim
@RealGeneKim
  • How many of you had the problem where you had a great idea, something that can help the business, only to have IT say, “well, maybe you can have it in 2016, when the planning calendar opens up?” (Lots of hands go up.)
  • Where Did the High Performers Come From?
    • Non-commissioned officers.
    • Chemical engineers
    • Auditors
  • What do they have in common?
    • Rigor and discipline!
  • When we wrote Visible Ops, we saw a downward spiral
    • Fragile applications are prone to failure
    • A long time required to figure out what went wrong
    • Detection comes from a salesperson who said “why are the banner ads being shown upside down”?
    • Too much firefighting and unplanned
    • Planned work cannot get done
    • Frustrated customers leave
    • Market share goes down.
    • Business misses Wall Street commitments
  • The key aha: This isn’t an IT ops problem, this is a business risk.
  • Mission: Figure out how to break the IT core chronic conflict
    • Every IT organization is pressured 
  • Tribes needed:
    • Ops
    • Dev
    • Op Security
    • Design
  • Velocity talk in 2009 at Flickr: 10 deploys per day
    • Dev and ops working together
    • Ops who think like devs
    • Devs who think like ops
  • In 2011, Amazon doing a maximum of 1,079 deployments per HOUR.
    • That’s 11 seconds per deployment.
  • If your company can deploy at most once a month, how can you compete against someone who can deploy daily or hourly?
  • DevOps is a real movement
    • I would never do another startup without employing devops principles
    • It’s happening in enterprises, government, and non-profits.
  • The Three Ways
  • The First Way
    • Systems Thinking: Left to Right
    • Never Pass defects to downstream work centers
    • Never allow local optimization to create global degradation
    • Eradicate blockages in the flow
    • Outcomes
      • Faster cycle times
  • The Second Way
    • Amplify feedback loops (right to left)
      • Expose visual data everyone can see how their decisions affect the entire system.
    • Outcomes
  • The Third Way
    • Culture of Continual Experimentation and Learning
      • Foster a culture that rewards
        • Experimentation (taking risks) and learning from failure
          • Jared Spool story of Intuit, where the CEO, in a monthly ceremony, gives a lifepreserver to the person who took the biggest risk, and they share their knowledge of what they learned.
        • Repetition is the prerequisite to mastery
      • Outcomes
        • 15 minutes
  • Prescriptive
    • Meeting the DevOps Leadership Team
      • Typically led by Dev, QA, IT Ops, and Product Management/Design
    • Agile Sprints
      • typically one week to one month
      • at the end of each sprint, team should have potentially deliverable product
      • But where this breaks down, is that typically dev uses up all the time in the project, leaving none operations or testing
    • Help Dev and Ops Build Code and Environments
      • Dev and Ops work together in Sprint 0 and 1 to create code and environments
        • Create environment that Dev deployed into
      • Security must integrate security testing into continuous testing through automation. If it takes 2 to 3 weeks to perform a security check, it won’t fit into the agile process, and it will be marginalized.
    • Keep Shrinking Batch Sizes
      • Waterfall projects often have cycle time of one year
      • Sprints have cycle time of 1 or 2 weeks
      • When IT Operations work is sufficiently fast and capable (e.g. it’s a < 1 hr process) we may decide to decouple from sprint boundaries. 
        • Now we don’t have to wait two weeks for a feature to go out.
        • And the deployments get real small: we push out a single feature. 
          • This is lower risk than pushing out hundreds of features together.
    • IT Operations Increases Process Rigor
  • Letters To Stakeholders
    • Development:
      • Be aware of the downstream effects of your actions
        • Unplanned work comes at the expense of planned work (features)
        • When we take shortcuts at the front of the line, it has an amplified effect downstream.
        • Technical debt retards feature throughput
        • Environment matters as much as code
    • QA
      • Ensure test plans cover not only code functionality, but also:
        • suitability of the environment the code runs in
        • The end-to-end deployment process
      • Help find variance
        • functionality, performance, configuration
        • duration, wait time, errors
    • Operations
      • Expect and tolerate failure (use Chaos Monkey)
      • See: “5 Lessons We’ve Learned Using AWS”
      • “The best way to avoid failure is to fail constantly”
      • Harden the production environment
      • Have scheduled drills to “crash the data center”
      • Create your “chaos monkeys”
    • Product Management:
      • Marty Cagan: Led product mgmt organization at eBay
      • He inherited the organization at eBay when they were suffering from chronic outages.
      • “you must take 20% of dev cycles to paying down technical debt.”
    • Designers
      • Help IT Operations codify their work and requirements into great and ever increasing library of user stories
      • Realize that IT processes are likely the largest impediment preventing your great ideas from making it to market
      • By working on the processes of how code gets into production, you can remove the impediments, and get more