Predicting the future of distributed systems
Object storage is increasingly integrated into transactional and analytical systems, enhancing reliability. Future programming models may shift code management to infrastructure, despite skepticism about the sustainability of emerging technologies.
Read original articlesignificant ways to reduce perceived risks and demonstrate clear value. The evolution of distributed systems is marked by the integration of object storage into transactional and analytical frameworks, which is seen as a step-change in value. However, the adoption of new programming models remains challenging due to the perception of high investment risks and the difficulty in identifying one-way-door versus two-way-door decisions. Object storage has matured and is increasingly utilized across various systems, offering features that enhance reliability and simplicity. The future of programming models may involve a shift towards extracting code from applications into infrastructure, allowing for better security, management, and portability. This transition could lead to a more efficient development process, where auxiliary code is managed independently, thus reducing the burden on developers. Despite the potential benefits, skepticism remains regarding the longevity and viability of emerging technologies, as well as the control engineers have over their stacks. The path forward is complex, but the opportunities for innovation in distributed systems and programming models are significant.
- Object storage is becoming integral to both transactional and analytical systems.
- The distinction between one-way-door and two-way-door decisions is crucial for technology adoption.
- New programming models may shift code management from applications to infrastructure.
- There is skepticism about the sustainability of emerging technologies in distributed systems.
- The evolution of programming models could enhance security and portability of business logic.
Related
A Eulogy for DevOps
DevOps, introduced in 2007 to improve development and operations collaboration, faced challenges like centralized risks and communication issues. Despite advancements like container adoption, obstacles remain in managing complex infrastructures.
Programmers Should Never Trust Anyone, Not Even Themselves
Programmers are warned to stay cautious and skeptical in software development. Abstractions simplify but can fail, requiring verification and testing to mitigate risks and improve coding reliability and skills.
Is it time to version observability?
The article outlines the transition from Observability 1.0 to 2.0, highlighting structured logs for better data analysis, improved debugging, and enhanced software development, likening its impact to virtualization.
Distributed == Relational
The article explores how distributed systems can utilize relational database principles, advocating for parallel data gathering, triggers for function invocations, and user-friendly alternatives to SQL for efficient software development.
Predicting the Future of Distributed Systems
Object storage is increasingly integrated into transactional and analytical systems, enhancing reliability. Organizations face challenges in adopting new programming models due to perceived investment risks and uncertainty about technology longevity.
For OLAP use cases with real-time data ingestion requirements, object-storage-only approach also leads to write amplification. Therefore, I don't think that architectures like Apache Pinot, Apache Paimon, and Apache Druid are going anywhere.
Another problem with "open table formats" like Iceberg, Hudi, and Delta Lake is their slow innovation speed.
I've recently argued about this at greater length here: https://engineeringideas.substack.com/p/the-future-of-olap-t...
I think we're probably too early to build this today. Ray is used at my current job for scaling subroutines in our distributed job system. It's the closest I've seen.
Related
A Eulogy for DevOps
DevOps, introduced in 2007 to improve development and operations collaboration, faced challenges like centralized risks and communication issues. Despite advancements like container adoption, obstacles remain in managing complex infrastructures.
Programmers Should Never Trust Anyone, Not Even Themselves
Programmers are warned to stay cautious and skeptical in software development. Abstractions simplify but can fail, requiring verification and testing to mitigate risks and improve coding reliability and skills.
Is it time to version observability?
The article outlines the transition from Observability 1.0 to 2.0, highlighting structured logs for better data analysis, improved debugging, and enhanced software development, likening its impact to virtualization.
Distributed == Relational
The article explores how distributed systems can utilize relational database principles, advocating for parallel data gathering, triggers for function invocations, and user-friendly alternatives to SQL for efficient software development.
Predicting the Future of Distributed Systems
Object storage is increasingly integrated into transactional and analytical systems, enhancing reliability. Organizations face challenges in adopting new programming models due to perceived investment risks and uncertainty about technology longevity.