A FLOSS platform for data analysis pipelines that you probably haven't heard of

Arvados is an open-source platform designed for managing and processing large volumes of data, ranging from terabytes to petabytes. It features a content addressable storage system called Keep, which ensures high reliability and throughput for file management. Keep allows users to create collections of data without the need for reorganization or duplication, and it operates on various filesystems and object stores. The platform also includes Crunch, an orchestration system that runs Common Workflow Language (CWL) workflows, maintaining data provenance and reproducibility while optimizing costs in cloud environments. Arvados emphasizes security and compliance with data protection regulations, offering features such as access tokens, data encryption, and integration with external authentication systems like Active Directory and Google accounts. Users can interact with Arvados through a web application called Workbench, a command line interface, or via a RESTful API with available SDKs for multiple programming languages. This flexibility allows for easy integration with existing infrastructures and enhances user experience in querying, browsing, and visualizing data.

- Arvados is an open-source platform for managing large datasets.

- Key components include Keep for storage and Crunch for workflow orchestration.

- The platform ensures data security and compliance with regulations.

- Users can access Arvados through a web application, command line, or API.

- SDKs are available for various programming languages to facilitate integration.