The Evolution of Continuous Delivery at Scale @ Linkedin
5 likes1,411 views
The document outlines the evolution of LinkedIn's software development and continuous delivery practices from 2007 to 2014, highlighting growth from 30 to 1800 developers and the transition from monolithic to independent deployments. Key improvements included automated testing, enhanced developer ownership, and the implementation of a trunk-based development process, which enabled faster and more reliable releases. It emphasizes the significance of culture, tooling, and processes in achieving successful software delivery at scale.
The Evolution of Continuous Delivery at Scale @ Linkedin
- 1. THE EVOLUTION OF CONTINUOUS DELIVERY AT SCALE QCon SF Nov 2014 Jason Toy [email protected] 1
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /cd-linkedin
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. How did we evolve our solution to allow developers to quickly iterate on creating product as LinkedIn engineering grew from 30 to 1800 technologists? 2 ?
- 5. We will be talking about that evolution today. 3 • How we have improved developer productivity and the release pipeline • The pitfalls we’ve seen • How we’ve tackled them • What it took • What we have learned
- 6. 4 What have we accomplished as we scaled?? • Scaling: From 2007 to Today • 5 services -> 550+ services • 30 -> 1800+ technologists • 13 million members -> 332 million members • At the same time • Monolithic deployments to prod once every several weeks -> Independent deployments when ready • Manual -> Automated commit to production pipeline • Faster iterations on the technology stack
- 7. 5 LinkedIn 2007 • ~30 developers, 5-10 services • Trunk based development • Testing • Mostly manual • Nightly regressions: automated junit, manual functional • Release (Every couple weeks) • Create branch and deployment ordering • Rehearse deployment, run tests in staging • Site downtime to push release (All eng + ops party)
- 8. Problems in 2007 • Testing and Development • Trunk stability: large changes, manual/local/nightly testing • Codebase increasing in size • Release • Infrequent, and time consuming 6
- 9. LinkedIn 2008-2011 • ~ 300 developers, ~300 services • Branch based development, merge for release • Testing • Added automated ‘Feature Branch Readiness’ • Before merge prove branch had 0 test failures / issues • Release (Every couple weeks) • Exactly as before: • Create, rehearse, and execute a deployment ordering. 7
- 10. Improvements in 2008-2011 • Branches supported more developers • More automated testing 8
- 11. Tradeoff: Branch Hell • Qualifying 20-40 branches • Stabilizing release branch hard • Point of friction: fragile/flaky/unmaintained tests • Impact: • frustrating process became power struggle 9
- 12. Problem: Deployment Hell • Monolithic change with 29 levels of ordering • Must fix forward: too complex to rollback • Manual prod deployment did not scale: • Dangerous, painful, and long (2 days) • Impact: • Operations very expensive and distracting • Missing a release became expensive to developers • More hotfixes and alternative process created 10
- 13. Linkedin 2011: The Turning Point • Company-wide Project Inversion • Build a well defined release process • Move to trunk development • Automated deployment process • Build the tooling to support this! • Enforcing good engineering practices. • No more isolated development (no branches) • No backwards incompatible changes • Remove deployment dependencies • Simplify architecture (complexity a cascading effect) • Code must be able to go out at any time 11
- 14. LinkedIn 2011 • ~ 600 developers ~250 services • Trunk based development • Testing: • Mostly automated • Source code validation: post commit test automation • Artifact validation: automated jobs in the test environment • Release: • On your own timeline per service • One button to push to deploy to testing or prod 12
- 15. How did we make this work? (A mixture of people, process, and tooling) 13 ?
- 16. Commit Pipeline • Pre/Post commit (PCX) machinery • On each commit, tests are run • Focused test effort: scope based on change set • Automated remediation: either block or rollback • Small team maintains machinery and stability • Creates new artifact upon success • Working Copy Test • PCX machinery to test local changes before commit • Great for qualifying massive/horizontal changes 14
- 17. Shared Test Environment • Continuously test artifacts with automated jobs • Stability treated in the same respect as trunk • Can test local changes against environment 15
- 18. Deployment vs Release • New distinction: • Deployment (new change to the site) • Trunk must be deployable at all times • Release (new feature for customers) • Feature exposure ramped through configs • Predictable schedule for releasing change • Product teams can release functionality at will without interfering with change 16
- 19. Deployment Process • Deployment Sequence: 1. Canary Deployment (New!) 2. Full rollout 3. Ramp feature exposure (New!) 4. Problem? Revert step. (New!) • No deployment dependencies allowed • Fully automated • Owners / Auto nominate deployment or rollback • All the deployment / rollback information is in plans 17
- 20. People • Everyone had to be willing to change • Greater engineering responsibility • No backwards incompatible changes • Rethink architecture, practices (piecewise features) • In return gave ownership of products and quality back to engineers • Release on your own schedule • Local decision making • You are responsible for your quality, not a central team • You own a piece of the codebase not a branch (acls) 18
- 21. Tooling • Acls for code review • Pre/Post commit CI framework / pipeline • CRT: Change Request Tracker • Developer commit lifecycle management • Deployment automation plans / Canaries • Performance • i.e. Evaluate canaries on things like exceptions • Test Manager • Manage automated tests (mostly in test environment) • Monitoring for environment / service stability • Config changes to ramp features 19
- 22. Improvements in 2011 • No merge hell • Find failures faster • Keep testing sane and automated • Independent and easy deployment and release • Create greater ownership • More control over, responsible for your decisions • Breaking the barriers: Easier to work with others 20
- 23. Challenges in 2011 (Overcame) • Breakages immediately affect others, so find and remove failures fast • Pre and post commit automation • Hard to save off work in progress • Break down your feature into commits that are safe to push to production. Use configs to ramp 21
- 24. Problems in 2011 • Monolithic Codebase • Not flexible enough to accommodate • Acquisitions • Exploration • Iterations needed to be even faster (non global block) • Ownership could be clearer • Of code • Of failures • Developer and code base grew significantly (again) 22
- 25. Multiproduct • ~1500 products ~1800 devs ~550 services • Ecosystem of smaller individual products each with an individual release cycle • Can depend on artifacts from other products • Uniform process of lifecycle and tasks • Abstractions allow us to build generic tooling to accommodate a variety of technologies and products • Lifecycle / tasks (i.e. build, test, deploy) owner defined • Testing and Release mostly the same • During your postcommit we test everything that depends on you – to ensure you aren’t breaking anything 23
- 26. Improvements with Multiproduct • No monolithic codebase • Flexible • Easier, faster to validate and not block 24
- 27. Challenges with Multiproduct • Architecture • Versioning Hell • Circular Dependencies • How to work across many products • How to work with others • Give people full control (no central police) 25
- 28. Conclusion: Key Successes • 0 Test Failures • Multitude of automated testing options • Automated, independent, frequent deployments • Distinguish between Deployments and Release • More accountability and ownership for teams 26
- 29. Conclusion: Takeaways • Notice any trends? • Validate fast, early, often • Simplify • Build the tooling to succeed • Creating more digestible pieces, giving more control to owners • It’s all a matter of tradeoffs and priorities • They change over time • Ours seem to be getting better! • It’s not only about technology: culture matters • Change, Ownership, Craftsmanship • People, process, technology • Invest in improvements, and stick with it 27
- 30. Thanks! 28
- 31. Questions? 29
- 32. 30
- 33. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/cd- linkedin