August 19th, 2024

Show HN: SiteOne Crawler – in-depth website analyzer and exporter (open-source)

SiteOne Crawler is a multifunctional tool for developers and website owners, compatible with multiple operating systems, offering website analysis, offline generation, report sending, and customizable command line options.

Read original articleLink Icon
Show HN: SiteOne Crawler – in-depth website analyzer and exporter (open-source)

The SiteOne Crawler is a multifunctional tool designed for developers, website owners, and consultants, compatible with Windows, macOS, and Linux. It offers various functionalities, including website crawling to analyze and report issues, generating offline versions of websites, creating sitemaps, and sending reports via email. Key features include a crawler that assesses website status codes and response times, a Dev/DevOps assistant for stress testing and cache warming, an analyzer for error identification and statistics, and a reporter for sending HTML reports. The tool also allows for offline website generation and sitemap creation. Installation is straightforward, with ready-to-use releases available on GitHub, and it can be operated via command line with customizable options. Users are encouraged to check permissions for crawling websites, as some may have restrictions in their `robots.txt` files. Additional resources, including documentation and tutorial videos, are available to assist users in maximizing the tool's capabilities.

- SiteOne Crawler is compatible with Windows, macOS, and Linux.

- It provides features for website analysis, offline generation, and report sending.

- Users can run the crawler via command line with various customization options.

- Documentation and tutorial videos are available for user support.

- Users should verify permissions for crawling websites to comply with restrictions.

Link Icon 1 comments
By @ablation - 4 months
This is actually pretty neat, so thank you. I think the output reports could perhaps benefit from some nicer formatting - font is a little hard on the eye? But it's fast, detailed, and I could see this being useful to a wide variety of people.