A web scraping CLI made for AI that is idempotent

The GitHub repository "Scrape It Now" is a web scraping tool that emphasizes efficiency and scalability. It features a decoupled architecture utilizing Azure Queue Storage for task management, enabling idempotent operations that allow for parallel execution without re-scraping unchanged pages. Scraped data is stored in Azure Blob Storage, and the tool includes ad-blocking capabilities to minimize network costs. It effectively handles dynamic content by using Playwright to load JavaScript, while also ensuring user anonymity through random user agents and viewport sizes. Additionally, it automatically creates a searchable index of the scraped content using Azure AI Search. To use the tool, users can run jobs to scrape websites or index scraped content by setting the necessary environment variables for Azure services. Advanced users can simplify configuration by sourcing environment variables from a `.env` file. This tool is particularly beneficial for developers aiming to scrape and index web content efficiently while utilizing Azure's cloud infrastructure.

- "Scrape It Now" is designed for efficient and scalable web scraping.

- It features ad-blocking and dynamic content handling with Playwright.

- The tool supports idempotent operations for parallel scraping.

- Scraped data is stored in Azure Blob Storage and indexed using Azure AI Search.

- Users can configure the tool easily with environment variables or a `.env` file.