July 1st, 2024

How Google migrated billions of lines of code from Perforce to Piper

Google migrated billions of lines of code from Perforce to Piper over four years to scale and reduce risks. Challenges included dependencies and seamless migration, but the transition was successful, improving operational efficiency.

Read original articleLink Icon
How Google migrated billions of lines of code from Perforce to Piper

Google successfully migrated billions of lines of code from Perforce to Piper, a process that took over four years. The decision to move away from Perforce was driven by the need to scale and reduce operational risks. The migration involved creating a new distributed system called Piper, based on Google's infrastructure, and cutting over all traffic to the new system. The project faced challenges such as dependencies on Perforce APIs and the need for a seamless migration process. Despite the risks involved, the migration was completed without any loss of state or impact on Google's production instances. The successful cutover to Piper not only reduced operational risk but also unblocked new systems due to increased traffic support. The migration to Piper showcased Google's engineering daring and scrappiness in tackling complex challenges.

Link Icon 4 comments
By @kccqzy - 5 months
> In an ideal world, a version control system should be strictly internal facing and be able to fall over without impacting live traffic; however the Piper team kept discovering that this wasn’t the case.

This is something that surprised me while I was at Google too. Of course a source code version control system shouldn't be a dependency of production, but it turns out it is just so convenient to go from (1) use your source code system to store configuration so that they are versioned and tracked; (2) have tools to read configuration and apply them automatically, Terraform style; (3) have tools to modify said configuration automatically instead of manually. Voila your source version control system suddenly becomes a NoSQL database.

By @floating-io - 5 months
The more interesting story here IMO is the seemingly extreme prevalence of NIH syndrome at Google, and whether or not that was a smart investment for Google compared to, say, contributing to GitLab or something.

I would love to hear the reasoning behind going custom, and if that ended up truly being worth the investment.

Or maybe I'm misremembering the timeline on when certain things became available. shrug

By @rty32 - 5 months
I have been reading about Google's monorepo setup and source control system, and my answer to the question in the title is simple: they had money and people. It is hard to think any other company that is not of Google's scale would be able to create their in-house source control system and complete a migration. This especially feels true after my company failed migrate from Perforce to git. You don't just build your source control, you build an entire ecosystem around it. And you need (very well-paid, smart) people to do that.
By @readthenotes1 - 5 months
"Jeff Dean, now Alphabet's chief scientist, personally stopped by the room that day to check in on the progress, helping boost morale, and further emphasizing the critical nature of the project."

That is such a cringy, fearful, statement...