Repeatability: As Difficult as it is Important
The article emphasizes the importance of repeatability in experimental systems research, highlighting challenges in achieving it, particularly with open-source software, and recommending tools and documentation to aid replication.
Read original article
The article discusses the critical importance of repeatability in experimental systems research, emphasizing that reproducibility should be a fundamental aspect of scientific inquiry. It highlights the challenges faced by the experimental systems research community in achieving repeatability, particularly in the context of open-source software. While there have been improvements, such as the introduction of ACM badging policies that encourage the sharing of research artifacts, technical barriers remain significant. The complexity of modern systems, which often rely on multiple integrated components, complicates the replication process. The author suggests that to facilitate repeatability, blueprints for system replication should be executable and require minimal assumptions from users. Tools like Ansible are recommended for their transparency and flexibility, allowing users to replicate systems without excessive complexity. The article concludes by stressing the need for thorough documentation and example executions to guide users in replicating research findings effectively.
- Repeatability is essential for validating experimental results in systems research.
- Open-source software can lower barriers to replication but introduces technical challenges.
- Executable blueprints and minimal assumptions are crucial for facilitating system replication.
- Tools like Ansible can help manage complexity in replicating systems.
- Comprehensive documentation and examples are necessary to support users in the replication process.
Related
DevOps: The Funeral
The article explores Devops' evolution, emphasizing reproducibility in system administration. It critiques mislabeling cloud sysadmins as Devops practitioners and questions the industry's shift towards new approaches like Platform Engineering. It warns against neglecting automation and reproducibility principles.
Reproducibility in Disguise
Reproducibility in software development is supported by tools like Bazel, addressing lifecycle challenges. Vendor dependencies for reproducibility face complexity, leading to proposed solutions like vendoring all dependencies for control.
Our Software Dependency Problem (2019)
The article highlights the increasing reliance on software dependencies in programming, emphasizing the need for thorough inspection and better tools to assess associated risks and ensure software security.
Build systems, not heroes
The article highlights the need for system-based approaches in enterprise programming to reduce risks from personnel changes, ensure consistent quality, and improve workflows, despite potential resistance from individuals.
Irreproducible Results
The article highlights declining reproducibility in scientific experiments, particularly in biological sciences, due to biases favoring positive results. Experts recommend open-source databases to document all experimental outcomes for improved reliability.
4 commentsI always provide gists, repos, and playgrounds, and they are as standalone as possible (no dependencies). I think that it is very important that readers be able to exactly reproduce what I’m saying, and that they are able to understand it completely.
I don’t think anyone actually cares (except me), but I’m anal. I do it for myself.
Most tutorials that I take are pretty slapdash, when it comes to supporting artifacts. People just throw them together. Sometimes, they are still working on them, as the class is starting, but I still learn what I need, so it probably isn’t mandatory that the class material be circumspect.
[UPDATE] I just noticed that HN formats link pairs, where I have a link, followed by another one in parentheses, in parallel columns, when displayed on small screens. Cool stuff.
[0] https://littlegreenviper.com/series/universal-links/ (https://github.com/LittleGreenViper/ImplementingUniversalLin...)
[1] https://littlegreenviper.com/series/bluetooth/ (https://github.com/ChrisMarshallNY/ITCB-master)
[2] https://littlegreenviper.com/miscellany/leaving-a-legacy/ (https://github.com/LittleGreenViper/Ozymandias)
[3] https://littlegreenviper.com/series/swiftwater/
It doesn't matter if you're using Python, Rust, C, Haskell, bash or some fresh hell written in PHP 4, perl and a fork of Netscape from 1999.
The answer is Nix. The Nix way _solves_ dependency management.
WGPU.rs, Rust's cross-platform graphics library, just reduced performance by 21%. They didn't notice. While WGPU supports Windows, MacOS, Linux, and Android, it started as WebGPU support in browsers. So it comes from web dev land, the land of "move fast and break things", "agile", and not worrying too much about performance until things get really bad.
This is part of why so many game projects have abandoned Rust. The graphics stack, four years on, still isn't very good.
[1] https://www.reddit.com/r/rust_gamedev/comments/1giujbk/wgpu_...
git clone URL
cd PROJECTNAME
For Rust,
cargo test
The old days of
./autoconf
make
Python and Javascript have too many build and install systems and much worse versioning problems.
Related
DevOps: The Funeral
The article explores Devops' evolution, emphasizing reproducibility in system administration. It critiques mislabeling cloud sysadmins as Devops practitioners and questions the industry's shift towards new approaches like Platform Engineering. It warns against neglecting automation and reproducibility principles.
Reproducibility in Disguise
Reproducibility in software development is supported by tools like Bazel, addressing lifecycle challenges. Vendor dependencies for reproducibility face complexity, leading to proposed solutions like vendoring all dependencies for control.
Our Software Dependency Problem (2019)
The article highlights the increasing reliance on software dependencies in programming, emphasizing the need for thorough inspection and better tools to assess associated risks and ensure software security.
Build systems, not heroes
The article highlights the need for system-based approaches in enterprise programming to reduce risks from personnel changes, ensure consistent quality, and improve workflows, despite potential resistance from individuals.
Irreproducible Results
The article highlights declining reproducibility in scientific experiments, particularly in biological sciences, due to biases favoring positive results. Experts recommend open-source databases to document all experimental outcomes for improved reliability.