Plaintext is not a great format for (system) logs
Using plain text for system logs poses challenges due to rich metadata. Approaches include augmenting logs with metadata, storing it separately, or discarding it. Tools can help manage metadata, making logs more structured than plain text. JSON, though text-based, may not be plain text.
Read original articleThe blog post discusses the limitations of using plain text for storing system logs, regardless of the tool being used. It highlights that log messages often come with rich metadata that can be challenging to handle in plain text format. The post outlines three approaches to dealing with metadata in logs: augmenting log messages with metadata in text format, storing metadata by implication in separate files, or discarding the metadata altogether. It points out that as systems attach more metadata to log messages, storing logs in plain text can lead to clutter, complexity, or loss of metadata. The post suggests that relying on tools to manipulate metadata for readability can result in logs becoming more structured than plain text. Ultimately, it mentions that while technically JSON is text, it may not qualify as plain text due to its structured nature.
Related
Advanced text features and PDF
The post explores complex text features in PDFs, covering Unicode, glyph representation, kerning, and font challenges. It emphasizes tools like Harfbuzz and CapyPDF for accurate text handling in PDFs.
Why *not* parse `ls` (and what to do instead)
Parsing 'ls' output in Unix systems is discouraged due to challenges with special characters. Shell globs and for loops are recommended for reliable file handling, criticizing 'ls' parsing for its limitations.
Structured logs are the way to start
Structured logs are crucial for system insight, aiding search and aggregation. Despite storage challenges, prioritizing indexing and retention strategies is key. Valuable lessons can be gleaned from email for software processes.
The Eternal Truth of Markdown
Markdown, a simplified code alternative to HTML, enables diverse document formats from plain text. Despite lacking standardization, it thrives for its adaptability and simplicity, appealing to writers and programmers alike.
Writing HTML by hand is easier than debugging your static site generator
The blog author discusses challenges of static site generators versus manual HTML coding, citing setup complexities and advocating for simplicity, stability, and control in website management. Emphasizes static data benefits.
I think text is really awesome though. You can read it with any editor and there are a zillion easy to learn Linux tools like grep, cut, awk...etc for interactively creating a oneliner in a short amount of time that you can use to get what you want out of the text. I used to have a bunch of those when I was in charge of some applications on a Linux server. Text is very universal and I'm not just talking about log files.
For that matter, journalctl is terrible, cryptic, and difficult to use.
In my opinion, plain text wins for everything, very portable and people can read with very little knowledge.
And yeah, sure, you want to use structured logging. So, in addition to the greppable log message, include a Magic Separator Character (say, \t) that you treat as 'end-of-line' in human-oriented processing, and have your key-value-pair structured data following that for automated tooling to have its way with. Or, be really creative with [key=value] blocks that are both human- and machine-readable.
Some of my most painful logging experiences were having to extract application logs from binary blobs created by a proprietary Windows tool (no, not the main Windows event log: that's bad, but at least documented). Even the most recent versions of the official viewer just crashed, the vendor was not interested in fixing that (since we were not a customer, just a third party in need of log data, but even an offer of a modest payment was met with indifference...), and the actual format turned out to be byzantine beyond words.
So, yeah, give me plain text all day, every day...
perhaps some llm-powered awk command generator would be useful?
Related
Advanced text features and PDF
The post explores complex text features in PDFs, covering Unicode, glyph representation, kerning, and font challenges. It emphasizes tools like Harfbuzz and CapyPDF for accurate text handling in PDFs.
Why *not* parse `ls` (and what to do instead)
Parsing 'ls' output in Unix systems is discouraged due to challenges with special characters. Shell globs and for loops are recommended for reliable file handling, criticizing 'ls' parsing for its limitations.
Structured logs are the way to start
Structured logs are crucial for system insight, aiding search and aggregation. Despite storage challenges, prioritizing indexing and retention strategies is key. Valuable lessons can be gleaned from email for software processes.
The Eternal Truth of Markdown
Markdown, a simplified code alternative to HTML, enables diverse document formats from plain text. Despite lacking standardization, it thrives for its adaptability and simplicity, appealing to writers and programmers alike.
Writing HTML by hand is easier than debugging your static site generator
The blog author discusses challenges of static site generators versus manual HTML coding, citing setup complexities and advocating for simplicity, stability, and control in website management. Emphasizes static data benefits.