Scaling Rails and Postgres to Users at Microsoft: Lessons and Takeaways
Andrew Atkinson discusses scaling Ruby on Rails applications with PostgreSQL, emphasizing ActiveRecord optimization, Sidekiq for background jobs, and addressing challenges like high concurrency and seasonal traffic in educational software.
Read original article
Andrew Atkinson, a consultant at StepChange, discusses strategies for scaling Ruby on Rails applications using PostgreSQL to accommodate millions of users, drawing from his experience at Microsoft with the Flip app. He emphasizes the importance of optimizing ActiveRecord queries, enhancing database efficiency, and utilizing Sidekiq for background job processing. Atkinson highlights the challenges faced when scaling applications, particularly in educational software, which experiences seasonal traffic fluctuations. He notes that relational databases like PostgreSQL provide strong data guarantees but can struggle with high concurrency during peak usage, especially with frequent updates, such as view counts in a social media context. The interview underscores the need for developers to understand the implications of their design choices on database performance, particularly when using an Object-Relational Mapping (ORM) like ActiveRecord, which abstracts database interactions. Atkinson's insights aim to help developers navigate the complexities of scaling applications effectively.
- Andrew Atkinson shares insights on scaling Rails applications with PostgreSQL based on his experience at Microsoft.
- Key strategies include optimizing ActiveRecord queries and using Sidekiq for background processing.
- Challenges include managing high concurrency and frequent updates in relational databases.
- Understanding the balance between application development needs and database performance is crucial.
- Seasonal traffic patterns in educational software require careful planning for scalability.
Related
Difference between running Postgres for yourself and for others
The post compares self-managed PostgreSQL with managing it for others, focusing on provisioning, backup/restore, HA, and security. It addresses complexities in provisioning, backup strategies, HA setup, and security measures for external users.
Rails Is Good Enough
Onur Ozer discusses the lasting significance of Ruby on Rails, highlighting its use by startups and large companies, its productivity, and its ability to simplify web application development over two decades.
Does PostgreSQL respond to the challenge of analytical queries?
PostgreSQL has advanced in handling analytical queries with foreign data wrappers and partitioning, improving efficiency through optimizer enhancements, while facing challenges in pruning and statistical data. Ongoing community discussions aim for further improvements.
I'm Back, Ruby on Rails
The author reassesses Ruby on Rails, praising its stability, built-in features, and supportive community, while highlighting its advantages for rapid development and deployment, making it suitable for startups.
Just use Postgres
Ethan McCue recommends Postgres as the default database for web applications due to its reliability and ease of use, contrasting it with SQLite and various NoSQL and proprietary databases.
- Several commenters emphasize the importance of caching and optimizing read operations to improve performance.
- There is a debate on the scalability of Rails compared to other frameworks, with some suggesting alternatives like Elixir/Phoenix for better performance.
- Some users advocate for vertical scaling of PostgreSQL without the need for microservices or additional tools.
- Concerns are raised about the inherent limitations of Ruby on Rails for handling high traffic and large user bases.
- Recommendations for resources, such as documentation and books on scaling Rails and PostgreSQL, are shared among commenters.
13 commentsNo need to microservice or sync read replicas even (unless you are making a game). No load balancers. Just up the RAM and CPU up to TB levels for heavy real world apps (99% of you wont ever run into this issue)
Seriously its so create scalable backend services with postgrest, rpc, triggers, v8, even queues now all in Postgres. You dont even need cloud. Even a mildly RAM'd VPS will do for most apps.
got rid of redis, kubernetes, rabbitmq, bunch of SaaS tools. I just do everything on Postgres and scale vertically.
One server. No serverless. No microservice or load handlers. It's sooo easy.
While I was mostly living out of the "High Availability, Load Balancing, and Replication" chapter, I couldn't help but poke around and found the docs to be excellent in general. Highly recommend checking them out.
1. try to make most things static-ish reads and cache generic stuff, e.g. most things became non-user specific HTML that got cached as SSI via nginx or memcached
2. move dynamic content to services to load after static-ish main content, e.g. comments, likes, etc. would be loaded via JSON after the page load
3. Move write operations to microservices, i.e. creating new content and changes to DB become mostly deferrable background operations
I guess the strategy was to do as much serving of content without dipping into ruby layer except for write or infrequent reads that would update cache.
[1] High Performance PostgreSQL for Rails Reliable, Scalable, Maintainable Database Applications by Andrew Atkinson:
https://pragprog.com/titles/aapsql/high-performance-postgres...
[1] https://github.com/lfittl/activerecord-clean-db-structure/is...
IDE smartness (auto complete, refactoring), compile error instead of runtime, clear APIs...
Kotlin is a pretty nice "Type-safe Ruby" to me nowadays.
I am not sure why are we boliling the oceans for the sake of a language like Ruby and a framework like Rails. I love those to death but Amazons approach is much better (or it used to be): you can't make a service for 10.000+ users in anything else than: C++, Java (probably Rust as well nowadays).
For millions of users the CPU cost difference probably justifies the rewrite cost.
Related
Difference between running Postgres for yourself and for others
The post compares self-managed PostgreSQL with managing it for others, focusing on provisioning, backup/restore, HA, and security. It addresses complexities in provisioning, backup strategies, HA setup, and security measures for external users.
Rails Is Good Enough
Onur Ozer discusses the lasting significance of Ruby on Rails, highlighting its use by startups and large companies, its productivity, and its ability to simplify web application development over two decades.
Does PostgreSQL respond to the challenge of analytical queries?
PostgreSQL has advanced in handling analytical queries with foreign data wrappers and partitioning, improving efficiency through optimizer enhancements, while facing challenges in pruning and statistical data. Ongoing community discussions aim for further improvements.
I'm Back, Ruby on Rails
The author reassesses Ruby on Rails, praising its stability, built-in features, and supportive community, while highlighting its advantages for rapid development and deployment, making it suitable for startups.
Just use Postgres
Ethan McCue recommends Postgres as the default database for web applications due to its reliability and ease of use, contrasting it with SQLite and various NoSQL and proprietary databases.