Ext4 100% Full? How Data Can Still Be Written

Hey guys, ever seen that dreaded message where your Ext4 filesystem is showing 100% full, but weirdly, your apps like your bittorrent client can still, like, write stuff? Yeah, it’s a super common and frankly, pretty confusing situation, especially when you're running something like Proxmox and think you’ve hit a hard limit. You look at df -h and it screams /dev/sdb1 is at a whopping 100% /hdd-scratch. That’s enough to make anyone panic, right? You’re probably thinking, “How in the heck is it still letting me download these massive files? Is this some kind of magic, or am I about to lose everything?” Well, before you start pulling your hair out, let’s dive into why this happens and what’s really going on under the hood with your trusty Ext4 filesystem. It turns out, that 100% full notification isn’t always the absolute, end-all be-all of your storage capacity. There are nuances, especially with how Linux and filesystems manage space. We’ll break down the concepts of reserved blocks, inodes, and how deleted files can still take up space, making sure you get a clear picture so you can manage your storage like a pro. So, grab your favorite beverage, and let’s unravel this storage mystery together. We're gonna get to the bottom of this, and by the end, you'll be way more confident in understanding your disk usage, even when it looks like it's defying the laws of physics.

Understanding the Nuances of Ext4 Fullness

So, let's get this straight, when your Ext4 filesystem reports 100% full, it doesn’t always mean there’s literally zero bytes left for new data. This is a super important distinction, especially when you’re dealing with large storage drives like that 3.6T /hdd-scratch you’re seeing. One of the main culprits behind this apparent contradiction is reserved blocks. In Ext4, a certain percentage of the disk space (usually 5% by default) is reserved for the superuser (root). This is a safety net, guys, designed to prevent a full filesystem from completely crashing the system. Imagine if your OS or critical system processes couldn't write even tiny logs because a user-space application filled up the disk – chaos! This reservation ensures that the system can continue to function, write essential logs, and perform administrative tasks, even when the regular user space sees 100% utilization. So, while your bittorrent client might be struggling or eventually stop, the system itself maintains a bit of breathing room. This reserved space is a lifesaver, but it means that the available space for regular users and applications is actually less than the total capacity reported by df -h. Another common reason, and something that often catches people off guard, is the existence of deleted files. When you delete a file in Linux, the filesystem usually just marks the blocks that file occupied as free. The actual data isn’t immediately wiped clean. However, the file's entry in the directory is removed, and the space is made available for new data. The catch? If a process still has that file open when it’s deleted, the filesystem won't actually reclaim the space until the process closes the file handle. This is super common with download clients, media servers, or anything that might be actively streaming or processing files. Even though the file appears deleted from your perspective (you can't see it in ls), the operating system still holds a reference to it, and thus, the space isn't truly freed up until that reference is released. This is a crucial point, and understanding it can save you a lot of headaches when troubleshooting disk space issues. We'll dig deeper into how to identify and clear these lingering deleted files in the next section. It's all about peeling back the layers of the filesystem to see what's really happening beyond the simple percentage shown.

Inodes and Hidden Space Hogs

Alright, let’s talk about another sneaky factor that can make your Ext4 filesystem appear full even when you think you have space: inodes. You might be thinking, “Wait, I’ve deleted a bunch of files, why is it still full?” Well, it’s not just about the raw data blocks; it’s also about the metadata that describes those files. Every single file, directory, symbolic link, and even device file on an Ext4 filesystem has an inode. This inode stores crucial information like the file’s permissions, ownership, timestamps, and, importantly, pointers to the actual data blocks on the disk. When you create a file, you consume one inode. When you delete a file, you free up its inode and its data blocks (unless, as we discussed, it's still open). Now, here’s the kicker: Ext4 filesystems are created with a fixed number of inodes, based on the size of the partition and the chosen block size. If you have a massive drive, like your 3.6T /hdd-scratch, you likely have a ton of inodes available. However, if you were storing a huge number of very small files – think millions of tiny configuration files, logs, or thumbnails – you could actually run out of inodes before you run out of disk space. This would mean you can’t create any new files, even if there's plenty of raw data capacity left. Your df -h might show, say, 90% full, but if df -i (which shows inode usage) is at 100%, you're effectively out of space for new files. While this might not be your exact scenario if you’re filling up with large torrents, it’s a vital concept to grasp for general filesystem health. The reverse can also be true: if you have a lot of free inodes but are filling up with large files, your data space will hit 100% first. For your specific situation, where df -h shows 100% but writing continues, the inode count is less likely to be the primary bottleneck for large files, but it's good to keep in mind for future troubleshooting. The real issue often lies in those deleted files that are still held open. Processes like your bittorrent client might have initiated downloads, and even if you’ve stopped the torrent or deleted the download entry within the client, the underlying operating system might still have file handles open for those data chunks. This is a common behaviour; the client might be writing to temporary files, or it might have cached file descriptors. Until those specific processes are restarted or the file handles are explicitly released by the OS, the space associated with those files (even if they appear deleted) remains occupied. It’s a bit like having books checked out from a library; even if you’ve returned them to your house, they’re not truly back on the shelf until you physically put them back. We’ll explore how to identify and purge these ghost files in the next section, which is often the key to reclaiming that usable space.

Identifying and Reclaiming Space: The Practical Steps

Okay, so we understand why your Ext4 filesystem might show 100% full but still allow writes, it’s time to get practical and actually reclaim that space. The first thing you should absolutely do, especially if you suspect open file handles are the culprit, is to identify processes that have deleted files open. The command lsof +L1 is your best friend here. This powerful utility lists open files, and the +L1 option specifically filters for files that have been deleted but are still being held open by a process. You’ll get a list of process IDs (PIDs), the commands they’re running, and the names of these 'ghost' files. Be cautious, guys! Don't just blindly kill processes. Understand what they are. If it’s your bittorrent client that’s the offender, you might be able to simply restart the client. This usually forces it to close all its file handles, and the OS will then reclaim the space. If it’s a system process you’re unsure about, a quick Google search for the PID or command name is wise. In many cases, a graceful restart of the offending service or application is the safest bet. If restarting the application isn't an option or doesn't free up space, you might need to consider rebooting the server. A reboot will terminate all running processes and ensure all file handles are closed, forcing the filesystem to release all truly deleted data. However, this is often a last resort. Another thing to check is the reserved blocks. While you generally shouldn't mess with the default 5% reservation unless you have a very specific reason, you can check and even reduce it if absolutely necessary. You can see the current reservation percentage with tune2fs -l /dev/sdb1 | grep 'Reserved block count'. To change it (use with extreme caution!), you'd use tune2fs -m <percentage> /dev/sdb1. Lowering this too much can impact system stability, so it’s usually better to focus on deleting actual data or clearing open files first. Finally, always keep an eye on your inode usage using df -i. If df -i shows you're hitting 100% on inode usage, you have a different problem – too many small files. In that case, you'd need to identify and remove large numbers of small files, which is a separate troubleshooting task. For your immediate situation, focus on lsof +L1 and restarting services. By understanding these underlying mechanisms and using the right tools, you can effectively manage your disk space and prevent those confusing