When Bloom filters don't bloom (2020)
The author explored Bloom filters for IP spoofing but faced performance issues due to memory access times. A simpler hash table approach ultimately provided better performance for large datasets.
Read original article
The blog post discusses the author's exploration of Bloom filters while working on a project related to IP spoofing. The author initially attempted to use traditional methods for deduplicating a massive dataset of IP addresses but found them inefficient. Bloom filters, which are probabilistic data structures designed to test membership and allow for some false positives, seemed like a suitable solution. However, the author encountered performance issues, particularly related to memory access times, as the data structure did not fit into the CPU's cache. Profiling revealed that a significant amount of CPU time was spent on memory fetches, leading to slower performance than expected. The author experimented with reducing the number of hash functions used in the Bloom filter but found that this increased memory requirements significantly without improving speed. Ultimately, the author shifted to a simpler hash table approach, which provided better performance and memory access patterns. The key takeaway is that while advanced data structures like Bloom filters can be useful, they may not be optimal for large datasets that exceed cache sizes, highlighting the importance of cache-optimized algorithms in modern computing.
- Bloom filters are efficient for membership testing but can struggle with large datasets that exceed CPU cache sizes.
- Memory access patterns significantly impact performance, with random access being much slower than sequential access.
- A simpler hash table approach can outperform more complex data structures in certain scenarios.
- Profiling tools are essential for identifying performance bottlenecks in data processing tasks.
- Optimizing for reduced memory accesses is often more beneficial than merely minimizing memory usage.
Related
`noexcept` affects libstdc++'s `unordered_set`
The article examines the impact of the `noexcept` specifier on `std::unordered_set` performance in libstdc++, highlighting optimization opportunities and advocating for improvements to handle hash function efficiency better.
Immutable Data Structures in Qdrant
Qdrant's article highlights the benefits of immutable data structures in vector databases, improving performance and memory efficiency, while addressing challenges through mutable segments and techniques like perfect hashing and defragmentation.
Do low-level optimizations matter? Faster quicksort with cmov (2020)
The article emphasizes the importance of low-level optimizations in sorting algorithms, highlighting how modern CPU features and minimizing conditional branches can enhance performance, particularly with the new `swap_if` primitive.
The Treacherous Optimization (2006)
The author attempts to optimize Hex Fiend's string searching to surpass grep's performance but initially fails. They adopt grep's optimization technique, achieving slight improvements while questioning the trade-offs involved.
High-Performance Binary Search
Optimized binary search algorithms can be up to four times faster than std::lower_bound by eliminating branching and improving memory layout, enhancing cache performance and data locality for specific contexts.
4 comments cat file.txt | perl -ne 'print unless $x{$_}++'You sure about that?
If the entire network is healthy. Yes, and maybe by two(?) transatlantic cables is not the efficient way to serve a page. NYC is probably closer to Italy by network topology than is Brazil.
But if CF had a data center in Italy and one in Germany, there are very valid reasons for Italian traffic to end up in Germany. Like a grid failure, flooding. Or fire. So how far you can take this and still meet SLAs is a little tricky.
With a good implementation it would be roughly 10 times faster, eg a SIMD implementation. For example here https://github.com/FastFilter/fastfilter_cpp/blob/master/src... (there are others)
Related
`noexcept` affects libstdc++'s `unordered_set`
The article examines the impact of the `noexcept` specifier on `std::unordered_set` performance in libstdc++, highlighting optimization opportunities and advocating for improvements to handle hash function efficiency better.
Immutable Data Structures in Qdrant
Qdrant's article highlights the benefits of immutable data structures in vector databases, improving performance and memory efficiency, while addressing challenges through mutable segments and techniques like perfect hashing and defragmentation.
Do low-level optimizations matter? Faster quicksort with cmov (2020)
The article emphasizes the importance of low-level optimizations in sorting algorithms, highlighting how modern CPU features and minimizing conditional branches can enhance performance, particularly with the new `swap_if` primitive.
The Treacherous Optimization (2006)
The author attempts to optimize Hex Fiend's string searching to surpass grep's performance but initially fails. They adopt grep's optimization technique, achieving slight improvements while questioning the trade-offs involved.
High-Performance Binary Search
Optimized binary search algorithms can be up to four times faster than std::lower_bound by eliminating branching and improving memory layout, enhancing cache performance and data locality for specific contexts.