Using Rust to corrode insane Python run-times

Vortexa faced significant performance issues with a Python task that took 30 hours to process GPS signals from ships, impacting their QA feedback cycle. The task involved filtering these signals through multiple polygons, and profiling revealed that a substantial portion of the runtime was spent in matplotlib for point-in-polygon calculations. The initial attempt to optimize using geopandas resulted in a tenfold increase in runtime, prompting a reevaluation of the approach. The team decided to develop a custom library in Rust, leveraging PyO3 for integration with Python. This new solution involved creating a Python class in Rust that processed geojson strings and numpy arrays, significantly reducing the number of library calls and utilizing integer-based math for efficiency. The results were impressive; the Rust implementation reduced processing time from 29.8 seconds to just 2.9 seconds, achieving a 24x improvement over the original matplotlib approach. The overall task duration decreased from 30 hours to 6 hours, enhancing the speed of model updates and QA processes. While the introduction of Rust added complexity to the codebase, the targeted optimization proved beneficial, maintaining the integrity of the existing business logic. Vortexa continues to seek talent for roles in Python, data science, Java, and Rust, reflecting their commitment to tackling ongoing technical challenges.