Visualizing Entropy in Binary Files (2012)

The article discusses a method for visualizing entropy in binary files, particularly using space-filling curves and a color function that measures local entropy. Entropy, in this context, refers to the disorder within a data set, with low entropy indicating uniformity and high entropy indicating a diverse representation of symbols. This visualization technique is particularly useful for reverse engineers and penetration testers, as it helps identify compressed data and cryptographic material within binaries. The author employs the Shannon entropy measure to calculate byte entropy over a sliding window, producing a "local entropy" value for each byte.

Using the OSX ksh binary as an example, the article illustrates how different sections of the binary exhibit varying levels of entropy. Notably, certain sections contain cryptographic material, such as code signing hashes and certificates, which are essential for security audits. The article also highlights a cautionary example where structured data is misinterpreted as high entropy due to the Shannon measure's focus on symbol frequency. The author explains that some sections of the ksh binary, which appear to have high entropy, actually serve as maps for character encoding translations. The code used for generating the visualizations is available on GitHub, and the author has created an interactive tool for binary visualization at binvis.io.