A tale of using chaos engineering at scale to keep our systems resilient

In a recent blog post, Tines software engineer Shayon Mukherjee discussed lessons learned from a Redis cluster upgrade incident that affected customer workflows. The upgrade, intended to enhance platform resilience, revealed a hidden bug when the monitoring system flagged an API outage shortly after the process began. The issue stemmed from a failure in the persistent connection of a dedicated listener thread to Redis, which went undetected and led to service degradation.

The incident highlighted a critical vulnerability in the webhook system's architecture, specifically the lack of robust error handling for the listener thread during connectivity issues. This oversight underscored the importance of comprehensive testing and a holistic understanding of system dependencies.

Despite the stress of the situation, the team viewed the incident as an opportunity for improvement. They recognized the need for a reconciliation loop for the singleton thread to enhance resilience and committed to conducting periodic chaos testing to uncover hidden vulnerabilities.

Mukherjee emphasized that incidents, while challenging, are essential for understanding system resilience and improving service quality. Each incident provides valuable insights that can lead to better designs and practices, reinforcing the idea that embracing chaos can ultimately lead to clarity and progress in complex systems.