SlideShare a Scribd company logo
5
Most read
7
Most read
12
Most read
Chrome Release Cycle




 Author: Anthony Laforge (laforge@chromium.org)
Our Philosophy

• Think of any major website, even the fancy Web 2.HTML5
  ones... do they have version numbers?
• We took the same approach to our client software as an
  online web service.
• That is... we treat releases as a means of getting features out
  to users and not goals in and of themselves.
• It's about flow.
How Our Users Work

• Users sign up for Chrome release channels based on the
  level of stability they want.
• Updates happen automatically in the background.
• New features and functionality just flow in without any effort
  on the user's part.
How the Chrome Team Works


• Developers work almost exclusively on a single central trunk
  (possible due to our try bots and continuous build
  infrastructure)
• Our branches stem from points on the trunk
• We stabilize branches by pulling in changes from trunk (i.e.
  everything lands on trunk first)
Where we started (v1-4) - On Paper
In practice, for that final beta...we'd

• Cut a branch (when we thought we had everything)
• Merge about ~500 patches (since we didn't have everything)
• Spend weeks stabilizing and re-stabilizing issues.
• Ended up working 1-3 months to get a release out the door,
  always certainly missing our 13 week plan.
• And at the end finally shipping something we were happy
  with... but that left us pretty drained (i.e. the bad flow)
The chain of events

• We'd hold releases, sometimes our branch points, until
  features were done, which meant...
• Lots of changes got queued up on the trunk, which caused...
• Engineers merging lots of changes to the branch, which led
  to...
• Instability on the branch that we had to deal with and that
  meant...
That chain of events...

• Long lived branches, which...
• Made supporting the branch (i.e. merging changes) more
  difficult as we drifted further from the trunk...
• All of which led to...
   o Unpredictable release cycles, with date targets we could
     never hit, and....
   o Engineers who were always in a rush to get their features
     into "this" release since we couldn't make any promises
     about the schedule
Which led me to think...

• I need a long vacation.
• There has to be a better way.
• What's causing things not to flow smoothly?




                               ...Yes...In that order.
Primary Goals
• Shorten the release cycle and reduce the life span of a
  branch (make merges easier)
• Make releases more predictable and easier to scope
• Reduce the strain on the entire team




   Good                              Bad
Goal to Plan - Getting down the cycle

• Originally set out a plan to simplify the 12 week cycle, 6
  weeks on dev, 6 weeks on beta, and then to stable.
• Once diagrammed, using blocks to represent the weeks,
  realized that with 3 channels you could have two overlapping
  releases running at once, which could get us a stable release
  ~ every 6 weeks.
Block diagrams
It was a start, but it wasn't the full answer

 • Sure, turning the wheel faster would make some things, like
   merges, easier.
 • But without addressing the scope problems, the flow wouldn't
   be any better and would continue to interrupt releases.
 • Things wouldn't be any more predictable and the whole cycle
   would just fall apart (more puddles, less of a stream).
Goal to Plan - Controlling Scope



• The pace of the schedule sets the boundaries for the amount
  of work that can be completed.
• It's important to have specific points in the schedule to review
  features and cut scope.
• Establish clear expectations (and engineering practice) to
  developers that any features not ready to ship at branch will
  be disabled (i.e. we only cut post branch, never add).
Plan to Pitch
• Reached out to the various cross functional groups on the
  Chrome team, who would all be impacted, before
  approaching our executives.
   o Engineering, QA, Product, Marketing, Support,
      Localization, Security, etc...
• There were a lot of concerns to address, but the exercise
  forced a lot of thought on the implications of the schedule
  change.
• It took time, but it made the pitch easier to present to the
  leadership and the rest of the team.
Concerns
• Would large feature development still be possible?

  "Yes, engineers would have to work behind flags, however
  they can work for as many releases as they need to and can
  remove the flag when they are done."
• Can the engineers keep up?

  "Their pace won't need to change, since features can be
  disabled there should be no milestone pressure, things ship
  when they are ready."
• What would a world look like where we didn't base our
  marketing on releases?

  "We market features, not releases."
And so we implemented it

       11 week overlapping schedules.
Our General Rules

• The branch point is the end of our development cycle
• Features only get disabled post branch point
• Features should be engineered so that they can be disabled
  easily (1 patch)
• Only stability, security, and critical regressions can ever
  block a release
Things we struggled with
• Overcoming team inertia (particularly when we cut scope)
• Saying no consistently to the team
• Fighting the urge to return to date driven management
  decisions
• We initially didn't solve the trunk->dev->branch time to patch
  problem (daily channel helped).
• Setting the right burn down point (i.e. branch point).
• Disabling features.
Key Lessons


• Having clear and concise rules is important.
• Predictable schedules cut down on communication costs and
  team confusion.
• It takes work and fore-thought to disable features.
• Diligent feature tracking is important.
Conclusion
• Speed alone isn't always the right answer, it's about keeping
  things running smoothly.
• For us, scope was getting in the way of that goal.




   We basically wanted to operate more like trains leaving Grand Central Station
(regularly scheduled and always on time), and less like taxis leaving the Bronx (ad
                             hoc and unpredictable).
Ad

Recommended

Introduzione a Docker
Introduzione a Docker
Roberto Messora
 
Docker Security workshop slides
Docker Security workshop slides
Docker, Inc.
 
Docker and the Linux Kernel
Docker and the Linux Kernel
Docker, Inc.
 
Tutoriel GIT
Tutoriel GIT
Francois ANDRE
 
Docker 101: An Introduction
Docker 101: An Introduction
POSSCON
 
Docker on Docker
Docker on Docker
Docker, Inc.
 
Deep Dive into the AOSP
Deep Dive into the AOSP
Dr. Ketan Parmar
 
Platform as reflection of values: Joyent, node.js, and beyond
Platform as reflection of values: Joyent, node.js, and beyond
bcantrill
 
Docker Swarm 0.2.0
Docker Swarm 0.2.0
Docker, Inc.
 
A deep dive into Android OpenSource Project(AOSP)
A deep dive into Android OpenSource Project(AOSP)
Siji Sunny
 
Docker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker Networking
Docker, Inc.
 
presentation on Docker
presentation on Docker
Virendra Ruhela
 
Apresentação UX e UI - Webdesign - Aula 07
Apresentação UX e UI - Webdesign - Aula 07
Renato Melo
 
VMware Tanzu Kubernetes Connect
VMware Tanzu Kubernetes Connect
VMware Tanzu
 
An Insight into the Linux Booting Process
An Insight into the Linux Booting Process
Hardeep Bhurji
 
[C++ Korea 2nd Seminar] Ranges for The Cpp Standard Library
[C++ Korea 2nd Seminar] Ranges for The Cpp Standard Library
DongMin Choi
 
Guaranteeing Memory Safety in Rust
Guaranteeing Memory Safety in Rust
nikomatsakis
 
Android SDK Tutorial | Edureka
Android SDK Tutorial | Edureka
Edureka!
 
Momenti Seminar - 5 Years of RosettaStone
Momenti Seminar - 5 Years of RosettaStone
Chris Ohk
 
U Boot or Universal Bootloader
U Boot or Universal Bootloader
Satpal Parmar
 
Project meeting: Android Graphics Architecture Overview
Project meeting: Android Graphics Architecture Overview
Yu-Hsin Hung
 
Version control system
Version control system
Aryman Gautam
 
Kubernetes in Docker
Kubernetes in Docker
Docker, Inc.
 
Docker advance topic
Docker advance topic
Kalkey
 
Android Synopsis
Android Synopsis
Niraj Rahi
 
Affordances (propiciação) em interfaces digitais
Affordances (propiciação) em interfaces digitais
Rodrigo Freese Gonzatto
 
Docker introduction
Docker introduction
Phuc Nguyen
 
DevOps - Cultura e Filosofia
DevOps - Cultura e Filosofia
Jônatan Gouveia
 
Chrome发布周期
Chrome发布周期
36Kr.com
 
White Paper: Release This! - Tools for a Smooth Release Cycle
White Paper: Release This! - Tools for a Smooth Release Cycle
Perforce
 

More Related Content

What's hot (20)

Docker Swarm 0.2.0
Docker Swarm 0.2.0
Docker, Inc.
 
A deep dive into Android OpenSource Project(AOSP)
A deep dive into Android OpenSource Project(AOSP)
Siji Sunny
 
Docker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker Networking
Docker, Inc.
 
presentation on Docker
presentation on Docker
Virendra Ruhela
 
Apresentação UX e UI - Webdesign - Aula 07
Apresentação UX e UI - Webdesign - Aula 07
Renato Melo
 
VMware Tanzu Kubernetes Connect
VMware Tanzu Kubernetes Connect
VMware Tanzu
 
An Insight into the Linux Booting Process
An Insight into the Linux Booting Process
Hardeep Bhurji
 
[C++ Korea 2nd Seminar] Ranges for The Cpp Standard Library
[C++ Korea 2nd Seminar] Ranges for The Cpp Standard Library
DongMin Choi
 
Guaranteeing Memory Safety in Rust
Guaranteeing Memory Safety in Rust
nikomatsakis
 
Android SDK Tutorial | Edureka
Android SDK Tutorial | Edureka
Edureka!
 
Momenti Seminar - 5 Years of RosettaStone
Momenti Seminar - 5 Years of RosettaStone
Chris Ohk
 
U Boot or Universal Bootloader
U Boot or Universal Bootloader
Satpal Parmar
 
Project meeting: Android Graphics Architecture Overview
Project meeting: Android Graphics Architecture Overview
Yu-Hsin Hung
 
Version control system
Version control system
Aryman Gautam
 
Kubernetes in Docker
Kubernetes in Docker
Docker, Inc.
 
Docker advance topic
Docker advance topic
Kalkey
 
Android Synopsis
Android Synopsis
Niraj Rahi
 
Affordances (propiciação) em interfaces digitais
Affordances (propiciação) em interfaces digitais
Rodrigo Freese Gonzatto
 
Docker introduction
Docker introduction
Phuc Nguyen
 
DevOps - Cultura e Filosofia
DevOps - Cultura e Filosofia
Jônatan Gouveia
 
Docker Swarm 0.2.0
Docker Swarm 0.2.0
Docker, Inc.
 
A deep dive into Android OpenSource Project(AOSP)
A deep dive into Android OpenSource Project(AOSP)
Siji Sunny
 
Docker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker Networking
Docker, Inc.
 
Apresentação UX e UI - Webdesign - Aula 07
Apresentação UX e UI - Webdesign - Aula 07
Renato Melo
 
VMware Tanzu Kubernetes Connect
VMware Tanzu Kubernetes Connect
VMware Tanzu
 
An Insight into the Linux Booting Process
An Insight into the Linux Booting Process
Hardeep Bhurji
 
[C++ Korea 2nd Seminar] Ranges for The Cpp Standard Library
[C++ Korea 2nd Seminar] Ranges for The Cpp Standard Library
DongMin Choi
 
Guaranteeing Memory Safety in Rust
Guaranteeing Memory Safety in Rust
nikomatsakis
 
Android SDK Tutorial | Edureka
Android SDK Tutorial | Edureka
Edureka!
 
Momenti Seminar - 5 Years of RosettaStone
Momenti Seminar - 5 Years of RosettaStone
Chris Ohk
 
U Boot or Universal Bootloader
U Boot or Universal Bootloader
Satpal Parmar
 
Project meeting: Android Graphics Architecture Overview
Project meeting: Android Graphics Architecture Overview
Yu-Hsin Hung
 
Version control system
Version control system
Aryman Gautam
 
Kubernetes in Docker
Kubernetes in Docker
Docker, Inc.
 
Docker advance topic
Docker advance topic
Kalkey
 
Android Synopsis
Android Synopsis
Niraj Rahi
 
Affordances (propiciação) em interfaces digitais
Affordances (propiciação) em interfaces digitais
Rodrigo Freese Gonzatto
 
Docker introduction
Docker introduction
Phuc Nguyen
 
DevOps - Cultura e Filosofia
DevOps - Cultura e Filosofia
Jônatan Gouveia
 

Similar to Chrome release cycle (20)

Chrome发布周期
Chrome发布周期
36Kr.com
 
White Paper: Release This! - Tools for a Smooth Release Cycle
White Paper: Release This! - Tools for a Smooth Release Cycle
Perforce
 
The Evolution of Continuous Delivery at Scale @ Linkedin
The Evolution of Continuous Delivery at Scale @ Linkedin
C4Media
 
2012 - A Release Odyssey
2012 - A Release Odyssey
Ernest Mueller
 
Pusheando en master, que es gerundio
Pusheando en master, que es gerundio
Isidro José López Martínez
 
Branching and Merging Practices
Branching and Merging Practices
Rajesh Kumar
 
ETCA_8
ETCA_8
PMI2011
 
Party of One
Party of One
Kirill Grouchnikov
 
A real-life overview of Agile and Scrum
A real-life overview of Agile and Scrum
mtoppa
 
Agile Product Owner
Agile Product Owner
Aaron Sanders
 
Rilasciamo rilasciamo
Rilasciamo rilasciamo
Francesco Mapelli
 
Release This! - Tools for a Smooth Release Cycle
Release This! - Tools for a Smooth Release Cycle
Perforce
 
What the music of the 1980s taught me about shipping software
What the music of the 1980s taught me about shipping software
Michael Ewins
 
Scaling Up Lookout
Scaling Up Lookout
Lookout
 
Release Engineering
Release Engineering
GDG Odessa
 
Continuous delivery the french way Agile Cambridge 2014
Continuous delivery the french way Agile Cambridge 2014
Dimitri Baeli
 
A real-life overview of Agile workflow practices
A real-life overview of Agile workflow practices
mtoppa
 
Doing agile with an ISO-20000 Telco (AgilePT 2015)
Doing agile with an ISO-20000 Telco (AgilePT 2015)
Manuel Padilha
 
Introduction to Scrum
Introduction to Scrum
Bixlabs
 
Timeboxed releases - Peter Antman
Timeboxed releases - Peter Antman
manssandstrom
 
Chrome发布周期
Chrome发布周期
36Kr.com
 
White Paper: Release This! - Tools for a Smooth Release Cycle
White Paper: Release This! - Tools for a Smooth Release Cycle
Perforce
 
The Evolution of Continuous Delivery at Scale @ Linkedin
The Evolution of Continuous Delivery at Scale @ Linkedin
C4Media
 
2012 - A Release Odyssey
2012 - A Release Odyssey
Ernest Mueller
 
Branching and Merging Practices
Branching and Merging Practices
Rajesh Kumar
 
A real-life overview of Agile and Scrum
A real-life overview of Agile and Scrum
mtoppa
 
Release This! - Tools for a Smooth Release Cycle
Release This! - Tools for a Smooth Release Cycle
Perforce
 
What the music of the 1980s taught me about shipping software
What the music of the 1980s taught me about shipping software
Michael Ewins
 
Scaling Up Lookout
Scaling Up Lookout
Lookout
 
Release Engineering
Release Engineering
GDG Odessa
 
Continuous delivery the french way Agile Cambridge 2014
Continuous delivery the french way Agile Cambridge 2014
Dimitri Baeli
 
A real-life overview of Agile workflow practices
A real-life overview of Agile workflow practices
mtoppa
 
Doing agile with an ISO-20000 Telco (AgilePT 2015)
Doing agile with an ISO-20000 Telco (AgilePT 2015)
Manuel Padilha
 
Introduction to Scrum
Introduction to Scrum
Bixlabs
 
Timeboxed releases - Peter Antman
Timeboxed releases - Peter Antman
manssandstrom
 
Ad

Recently uploaded (20)

" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
Fwdays
 
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
Edge AI and Vision Alliance
 
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
caoyixuan2019
 
Agentic AI for Developers and Data Scientists Build an AI Agent in 10 Lines o...
Agentic AI for Developers and Data Scientists Build an AI Agent in 10 Lines o...
All Things Open
 
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdf
AmirStern2
 
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Safe Software
 
The Growing Value and Application of FME & GenAI
The Growing Value and Application of FME & GenAI
Safe Software
 
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Priyanka Aash
 
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
Earley Information Science
 
Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft development
shyamraj55
 
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdf
Muhammad Yuga Nugraha
 
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?
Shashi Sathyanarayana, Ph.D
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 
Cyber Defense Matrix Workshop - RSA Conference
Cyber Defense Matrix Workshop - RSA Conference
Priyanka Aash
 
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
pcprocore
 
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Josef Weingand
 
Quantum AI: Where Impossible Becomes Probable
Quantum AI: Where Impossible Becomes Probable
Saikat Basu
 
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FME
Safe Software
 
Python Conference Singapore - 19 Jun 2025
Python Conference Singapore - 19 Jun 2025
ninefyi
 
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
Priyanka Aash
 
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
Fwdays
 
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
Edge AI and Vision Alliance
 
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
caoyixuan2019
 
Agentic AI for Developers and Data Scientists Build an AI Agent in 10 Lines o...
Agentic AI for Developers and Data Scientists Build an AI Agent in 10 Lines o...
All Things Open
 
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdf
AmirStern2
 
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Safe Software
 
The Growing Value and Application of FME & GenAI
The Growing Value and Application of FME & GenAI
Safe Software
 
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Priyanka Aash
 
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
Earley Information Science
 
Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft development
shyamraj55
 
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdf
Muhammad Yuga Nugraha
 
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?
Shashi Sathyanarayana, Ph.D
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 
Cyber Defense Matrix Workshop - RSA Conference
Cyber Defense Matrix Workshop - RSA Conference
Priyanka Aash
 
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
pcprocore
 
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Josef Weingand
 
Quantum AI: Where Impossible Becomes Probable
Quantum AI: Where Impossible Becomes Probable
Saikat Basu
 
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FME
Safe Software
 
Python Conference Singapore - 19 Jun 2025
Python Conference Singapore - 19 Jun 2025
ninefyi
 
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
Priyanka Aash
 
Ad

Chrome release cycle

  • 1. Chrome Release Cycle Author: Anthony Laforge ([email protected])
  • 2. Our Philosophy • Think of any major website, even the fancy Web 2.HTML5 ones... do they have version numbers? • We took the same approach to our client software as an online web service. • That is... we treat releases as a means of getting features out to users and not goals in and of themselves. • It's about flow.
  • 3. How Our Users Work • Users sign up for Chrome release channels based on the level of stability they want. • Updates happen automatically in the background. • New features and functionality just flow in without any effort on the user's part.
  • 4. How the Chrome Team Works • Developers work almost exclusively on a single central trunk (possible due to our try bots and continuous build infrastructure) • Our branches stem from points on the trunk • We stabilize branches by pulling in changes from trunk (i.e. everything lands on trunk first)
  • 5. Where we started (v1-4) - On Paper
  • 6. In practice, for that final beta...we'd • Cut a branch (when we thought we had everything) • Merge about ~500 patches (since we didn't have everything) • Spend weeks stabilizing and re-stabilizing issues. • Ended up working 1-3 months to get a release out the door, always certainly missing our 13 week plan. • And at the end finally shipping something we were happy with... but that left us pretty drained (i.e. the bad flow)
  • 7. The chain of events • We'd hold releases, sometimes our branch points, until features were done, which meant... • Lots of changes got queued up on the trunk, which caused... • Engineers merging lots of changes to the branch, which led to... • Instability on the branch that we had to deal with and that meant...
  • 8. That chain of events... • Long lived branches, which... • Made supporting the branch (i.e. merging changes) more difficult as we drifted further from the trunk... • All of which led to... o Unpredictable release cycles, with date targets we could never hit, and.... o Engineers who were always in a rush to get their features into "this" release since we couldn't make any promises about the schedule
  • 9. Which led me to think... • I need a long vacation. • There has to be a better way. • What's causing things not to flow smoothly? ...Yes...In that order.
  • 10. Primary Goals • Shorten the release cycle and reduce the life span of a branch (make merges easier) • Make releases more predictable and easier to scope • Reduce the strain on the entire team Good Bad
  • 11. Goal to Plan - Getting down the cycle • Originally set out a plan to simplify the 12 week cycle, 6 weeks on dev, 6 weeks on beta, and then to stable. • Once diagrammed, using blocks to represent the weeks, realized that with 3 channels you could have two overlapping releases running at once, which could get us a stable release ~ every 6 weeks.
  • 13. It was a start, but it wasn't the full answer • Sure, turning the wheel faster would make some things, like merges, easier. • But without addressing the scope problems, the flow wouldn't be any better and would continue to interrupt releases. • Things wouldn't be any more predictable and the whole cycle would just fall apart (more puddles, less of a stream).
  • 14. Goal to Plan - Controlling Scope • The pace of the schedule sets the boundaries for the amount of work that can be completed. • It's important to have specific points in the schedule to review features and cut scope. • Establish clear expectations (and engineering practice) to developers that any features not ready to ship at branch will be disabled (i.e. we only cut post branch, never add).
  • 15. Plan to Pitch • Reached out to the various cross functional groups on the Chrome team, who would all be impacted, before approaching our executives. o Engineering, QA, Product, Marketing, Support, Localization, Security, etc... • There were a lot of concerns to address, but the exercise forced a lot of thought on the implications of the schedule change. • It took time, but it made the pitch easier to present to the leadership and the rest of the team.
  • 16. Concerns • Would large feature development still be possible? "Yes, engineers would have to work behind flags, however they can work for as many releases as they need to and can remove the flag when they are done." • Can the engineers keep up? "Their pace won't need to change, since features can be disabled there should be no milestone pressure, things ship when they are ready." • What would a world look like where we didn't base our marketing on releases? "We market features, not releases."
  • 17. And so we implemented it 11 week overlapping schedules.
  • 18. Our General Rules • The branch point is the end of our development cycle • Features only get disabled post branch point • Features should be engineered so that they can be disabled easily (1 patch) • Only stability, security, and critical regressions can ever block a release
  • 19. Things we struggled with • Overcoming team inertia (particularly when we cut scope) • Saying no consistently to the team • Fighting the urge to return to date driven management decisions • We initially didn't solve the trunk->dev->branch time to patch problem (daily channel helped). • Setting the right burn down point (i.e. branch point). • Disabling features.
  • 20. Key Lessons • Having clear and concise rules is important. • Predictable schedules cut down on communication costs and team confusion. • It takes work and fore-thought to disable features. • Diligent feature tracking is important.
  • 21. Conclusion • Speed alone isn't always the right answer, it's about keeping things running smoothly. • For us, scope was getting in the way of that goal. We basically wanted to operate more like trains leaving Grand Central Station (regularly scheduled and always on time), and less like taxis leaving the Bronx (ad hoc and unpredictable).