September 21st, 2024

Saving $10k/Month on Analytics – Snowplow Serverless Alternative

Agon Data has implemented a serverless analytics solution, saving $10,000 monthly, ensuring data ownership, and utilizing tools like Buz and AWS Kinesis, with plans for optimization and potential open-sourcing.

Read original articleLink Icon
Saving $10k/Month on Analytics – Snowplow Serverless Alternative

Agon Data has successfully implemented a serverless alternative to traditional analytics solutions like Snowplow, resulting in significant cost savings of approximately $10,000 per month. The shift to a cloud-native, serverless architecture has provided a scalable and flexible analytics platform while ensuring full data ownership. The team noted that many existing analytics providers have increased their prices due to organizational bloat and high operational costs, making them less viable for smaller organizations. Their new infrastructure leverages tools such as Buz for data collection, AWS Kinesis for streaming, and Benthos for data movement, all while maintaining compatibility with the Snowplow data model. The deployment process has been streamlined using SST, allowing for automated infrastructure management and schema updates. Initial costs for the new setup are projected to be under $100 per month, a stark contrast to previous expenses. The team expresses gratitude for the open-source community, which has facilitated the development of their data pipeline. Future plans include optimizing the system further and potentially open-sourcing their solution for broader use.

- Agon Data has reduced analytics costs by approximately $10,000 per month with a serverless solution.

- The new infrastructure ensures data ownership and scalability while maintaining compatibility with the Snowplow data model.

- Tools like Buz, AWS Kinesis, and Benthos are utilized for data collection and movement.

- The deployment process is automated using SST, enhancing efficiency and reliability.

- Future plans include further optimization and potential open-sourcing of their data stack.

Link Icon 4 comments
By @robertclaus - 7 months
We ran into similar issues with SAAS log aggregation a few years ago. It's scarily easy to get into a "this is just what service X costs" mindset with an FTE's worth of bills going out every month.
By @mannyv - 7 months
It's not hard to build an analytics pipeline if you pay attention. Ours is basically free due to how our fastly account is structured.

The flow is:

https request to fastly

Synthetic request that logs to fastlylogs

Ship logfile to s3

Process with lambda

This is basically freeish except for the RDS instance.