July 24th, 2024

Hiding Linux Processes with Bind Mounts

The article explains a technique for hiding Linux processes using bind mounts, making them undetectable by standard tools. It highlights the method's implications for both offensive and defensive cybersecurity strategies.

Read original articleLink Icon
Hiding Linux Processes with Bind Mounts

The article discusses a technique for hiding Linux processes using bind mounts, inspired by a previous blog post by Stephan Berger. The method involves overlaying the /proc/PID directory of a process with a bind mount of another process's /proc/PID directory, effectively concealing the "evil" process from standard monitoring tools like `ps`. The author notes that processes with low PIDs and names in square brackets, which are typically kernel threads, can serve as effective covers for hidden processes.

By executing a bind mount command, the hidden process becomes indistinguishable from the kernel threads, making it difficult for administrators to detect. The article also highlights the importance of monitoring the /proc/mounts file to identify any suspicious bind mounts that may indicate process hiding. The author provides a script to automate the detection of such mounts and suggests that while the technique is clever, it is not foolproof, as there are ways to uncover hidden processes.

The discussion emphasizes the dual nature of this technique, serving as a tool for red teamers (offensive security) and a challenge for blue team analysts (defensive security). The article concludes by acknowledging the inspiration from Berger's work and encourages further exploration of this method in the context of cybersecurity.

Related

Htop explained – everything you can see in htop on Linux (2019)

Htop explained – everything you can see in htop on Linux (2019)

This article explains htop, a Linux system monitoring tool. It covers uptime, load average, processes, memory usage, and more. It details htop's display, load averages, process IDs, procfs, and process tree structure. Practical examples are provided for system analysis.

The weirdest QNX bug I've ever encountered

The weirdest QNX bug I've ever encountered

The author encountered a CPU usage bug in a QNX system's 'ps' utility due to a 15-year-old bug. Debugging revealed a race condition, leading to code modifications and a shift towards open-source solutions.

Userland Rootkits Are Lame

Userland Rootkits Are Lame

Userland rootkits like Symbiote pose a threat on Linux systems by injecting libraries to hide information. Countermeasures include using statically linked binaries for immunity, examining memory layouts, and comparing file lists.

Exploring Novel File System Objects for Data-Only Attacks on Linux Systems

Exploring Novel File System Objects for Data-Only Attacks on Linux Systems

The study explores data-only attacks on Linux systems, identifying critical file system objects for exploitation without requiring Kernel Address Space Layout Randomization. It presents novel exploit strategies and evaluates them against real-world vulnerabilities.

Linux 6.11 To Allow Tightening Of /proc/[PID]/mem Access For Better Security

Linux 6.11 To Allow Tightening Of /proc/[PID]/mem Access For Better Security

Linux 6.11 will introduce a security feature tightening access to /proc/[pid]/mem files, proposed by Christian Brauner. It aims to restrict writes unless the current process ptraces to the task, addressing past exploits. Brauner emphasizes balancing security with legitimate use cases like debugging tools.

Link Icon 10 comments
By @jordemort - 10 months
If you have the privilege to pull off bind-mounting on top of a particular process directory in /proc you should also be able to mount something on top of /proc itself that will present a false /proc/mounts and conceal your trickery. I experimented a bit and wasn't able to bind a single file on top of /proc/mounts and I seem to be having trouble getting procfs to participate in an overlayfs, but if you were really dedicated you could construct a whole fake /proc by hand, or create something using FUSE to create a /proc that redacts anything you want, including mountpoints.
By @o11c - 10 months
If you're relying on such superficial disguising, just put your process name in brackets directly, no mount privileges required.

As a general rule, if malware gets root privilege, no other process on the same system (container, at least) should be expected to be able to detect it.

By @bpoyner - 10 months
I'm so old I remember when ps would directly get the processes via system calls to the kernel and none of this reading from /proc.
By @efitz - 10 months
The abstraction of representing everything as files is often leaky (and often clunky).
By @amelius - 10 months
What I want to hide is mounts. With containers, snap (squashfs), etc., the number of mounts on a modern Linux system is growing to ridiculous numbers. And why should they all be visible to normal users?
By @anyfoo - 10 months
Coincidentally, I just used a veritable litany of bind mounts this week. I did not attempt to hide anything, though one of those bind mounts exists in order to fake something. I needed this to be able to properly run some x86 software under Rosetta 2 in an arm64 Ubuntu VM.

I wasn't happy with how handling packages for multiple architectures seems to work in Ubuntu/Debian: It looks like you basically have to choose what package you want to install with each architecture, because if you try to install a package for both architectures, they almost always conflict with each other (trying to occupy the same files).

So instead, I made an x86 chroot. I used "debootstrap" to populate the x86 chroot with an x86 Ubuntu base system, and schroot so that a regular non-root user could use that chroot. That way, I can install any x86 package I want without conflicting with the surrounding VM by just chroot'ing into it and regularly typing e.g. "apt install emacs".

But both because a lot of software needs it, and for interop with the surrounding VM (unix domain sockets etc.), I had to create a bunch of bind mounts to bring shared "state" directories into the x86 chroot:

    mount --make-private --bind /proc /data/x86/proc
    mount --make-private --bind /proc /data/x86/sys
    ...
I did this for at least /proc, /sys, /run, /dev, /dev/pts and /home. Depending on how much you want to bring the environments together, you could also add /run, /tmp, and others.

I also added bind mounts for individual files:

   mount --make-private --bind /etc/passwd /data/x86/etc/passwd
   mount --make-private --bind /etc/shadow /data/x86/etc/shadow
   mount --make-private --bind /etc/group /data/x86/etc/group
so that my x86 environment has the same user/group database.

Finally, and I think this is the most interesting piece, some x86 software I tried to run did not work because it tried to parse /proc/cpuinfo for some x86 CPU features. Rosetta 2 implements those, but of course /proc/cpuinfo in the system describes the actual (virtualized) ARM CPUs, which is of course not what x86 software expects.

So, I crafted my own fake cpuinfo text file that looked roughly like it would look in a real x86 environment (I did not put much effort in it, just copied an output from a random real x86 machine and adjusted the number of CPUs), and bind mounted that into the x86 /proc overlay:

   mount --make-private --bind /data/fake-x86-cpuinfo.txt /data/x86/proc/cpuinfo
Now, /proc/cpuinfo inside the chroot (so, /data/x86/proc/cpuinfo) would give the fake cpuinfo and make the x86 software that parses it happy. This is also where the --make-private that I applied to /data/x86/proc earlier becomes important: Without that, that last command would not just overlay /data/x86/proc/cpuinfo, but also /proc/cpuinfo itself, now in turn making arm64 software potentially unhappy. With --make-private on the /proc bind mount, it effectively becomes a separate filesystem in that regard (only).

Finally, the biggest hurdle I had was getting systemd to properly mount all of these in the right order at startup. systemd parallelizes the mounts in fstab (and mounts generally). But if e.g. the /data/x86/proc mount does not happen after both /proc and /data/x86 have been mounted already, you effectively get an empty directory (you can easily work out yourself why that results in all improper cases). This was even more complicated because I use ZFS, and so /data/x86 gets mounted by zfs.mount-service. After much fiddling, I gave up. No combination of "x-systemd.requires=<mountpoint>", "x-systemd.after=zfs-mount.service" and whatever else in the fstab options really fully did the right thing.

I resorted to having a shell script that just runs the bind mounts in the correct order.

By @Szpadel - 10 months
in Linux you can rename your own process name, many applications do that to name their threads or to describe what they are doing

isn't it just much easier to rename our process to looks like kernel thread or some other process? we could easily escape process tree by forking child and exiting parent (we would get adopted by init)

By @awaythrow999 - 10 months
Interesting but

>> My guess is that nobody is going to notice this unless they are specifically looking for this technique.

But having two identical PIDs is a pretty weak cloak. Even more so when reducing terminal clutter e.g. run "ps | grep procname" ... anyone not completely asleep is bound to notice it.

By @zamalek - 10 months
If you're looking for naughty processes you could start with:

    mkdir /tmp/foo
    mount -t proc none /tmp/foo