Google transfers 1.2 EB of data every day using Effingo
Google's Effingo is a data transfer tool that moves 1.2 exabytes daily, managing 14 terabytes per second while prioritizing critical tasks and aiming for improved integration and performance.
Read original article
Google has introduced Effingo, a sophisticated data transfer tool that enables the company to move an impressive 1.2 exabytes of data daily. This tool is designed to address the challenges of bandwidth limitations and the speed of light, which affect data replication across global systems. Effingo operates on Google's proprietary Colossus filesystem and is deployed across numerous clusters, managing data transfers at a rate of 14 terabytes per second. The system prioritizes data transfer jobs based on their importance, ensuring critical tasks, such as disaster recovery, receive the necessary resources. Effingo's architecture includes a control plane for managing data copy lifecycles and a data plane responsible for the actual transfer. Additionally, it utilizes a tool called Bandwidth Enforcer to allocate network capacity based on service priority. Despite its efficiency, Effingo experiences a mean global backlog of 12 million files, equating to about eight petabytes of data. Google plans to enhance Effingo's integration with resource management systems and improve its performance during cross-datacenter transfers.
- Google’s Effingo transfers 1.2 exabytes of data daily.
- The tool balances infrastructure efficiency with user needs.
- Effingo operates on the Colossus filesystem and manages transfers at 14 terabytes per second.
- It prioritizes critical data transfer jobs to ensure resource allocation.
- Google aims to improve Effingo's integration and performance in future updates.
Related
How Google migrated billions of lines of code from Perforce to Piper
Google migrated billions of lines of code from Perforce to Piper over four years to scale and reduce risks. Challenges included dependencies and seamless migration, but the transition was successful, improving operational efficiency.
Google's carbon emissions surge nearly 50% due to AI energy demand
Google's 2024 environmental report reveals a 50% rise in carbon emissions from 2019, hindering its 2030 net-zero goal. Increased data center energy use and AI demand are key contributors. Microsoft also faces similar challenges.
Google emissions jump nearly 50% over five years as AI use surges
Google's greenhouse gas emissions surged by 48% in 2023 due to AI data center expansion. Despite challenges, Google aims for net zero emissions by 2030 through clean energy deals and supply chain improvements.
Google's greenhouse gas emissions jump 48% in five years
Google's greenhouse gas emissions surged by 48% in 2023 due to AI data center expansion, challenging its 2030 "net zero" goal. Despite obstacles, Google is committed to emission reduction and clean energy strategies.
Humans.txt
Google is a diverse global company with engineers, designers, and researchers constantly updating using various tools. Career opportunities are available at careers.google.com for interested individuals.
8 commentsIt is open access, so you should be able to see the PDF.
There is a youtube video summarizing the main points: https://www.youtube.com/watch?v=hBME9Q6RPZ8.
I’m being sarcastic, if this wasn’t obvious.
So that's how much data they collect from their users every day. /s
Related
How Google migrated billions of lines of code from Perforce to Piper
Google migrated billions of lines of code from Perforce to Piper over four years to scale and reduce risks. Challenges included dependencies and seamless migration, but the transition was successful, improving operational efficiency.
Google's carbon emissions surge nearly 50% due to AI energy demand
Google's 2024 environmental report reveals a 50% rise in carbon emissions from 2019, hindering its 2030 net-zero goal. Increased data center energy use and AI demand are key contributors. Microsoft also faces similar challenges.
Google emissions jump nearly 50% over five years as AI use surges
Google's greenhouse gas emissions surged by 48% in 2023 due to AI data center expansion. Despite challenges, Google aims for net zero emissions by 2030 through clean energy deals and supply chain improvements.
Google's greenhouse gas emissions jump 48% in five years
Google's greenhouse gas emissions surged by 48% in 2023 due to AI data center expansion, challenging its 2030 "net zero" goal. Despite obstacles, Google is committed to emission reduction and clean energy strategies.
Humans.txt
Google is a diverse global company with engineers, designers, and researchers constantly updating using various tools. Career opportunities are available at careers.google.com for interested individuals.