Rsync Exclusion Secrets: Files Vs. Folders & Deletions

Hey there, Plastik Magazine crew! Ever been neck-deep in a Linux backup, thinking you've got your rsync commands all figured out, only to find it doing something completely unexpected? We've all been there, trust me. Rsync is a phenomenal tool for syncing files and directories, a true powerhouse in any Bash guru's toolkit. It’s super efficient, reliable, and incredibly flexible, making it the go-to utility for everything from simple local copies to complex remote backups across networks. But, like any powerful tool, it comes with its own quirks, especially when you start diving into its exclusion rules and the --delete flags. This is where things can get a little confusing, particularly when you’re trying to exclude a specific file but keep a directory that happens to share a similar name segment. You're trying to keep your backups lean and clean, ensuring only relevant data gets transferred, but suddenly rsync seems to have a mind of its own, excluding things you want and including things you don't. It's like trying to navigate a maze blindfolded, right?

Today, guys, we’re going to unravel one of those head-scratching rsync mysteries: why it might exclude a file using --delete and --delete-excluded, but seemingly not exclude a subdirectory with a similar name, or even worse, delete something you explicitly wanted to keep. We're talking about situations where you want to exclude a file named core from your backups, but you definitely want to keep those crucial foo/bar/core/baz.txt files safely tucked away. It sounds simple on the surface, but the devil, as always, is in the details of rsync's powerful (and sometimes picky) pattern matching. This isn't just a niche problem; understanding these nuances is absolutely critical for anyone relying on rsync for robust, accurate, and truly reliable backups. We’ll break down the core concepts, expose the subtle differences in exclusion patterns, and equip you with the knowledge to craft rsync commands that do exactly what you want, every single time. Get ready to master your backups, because by the end of this article, you’ll be an rsync exclusion wizard, effortlessly distinguishing between files and folders, and ensuring your data integrity is always top-notch. Let's dive in and make your rsync game stronger than ever!

The Rsync Exclusion Rulebook: How It Really Works

Alright, let's kick things off by getting into the nitty-gritty of rsync's exclusion rules. Understanding how rsync interprets your --exclude patterns is fundamental to mastering this awesome tool. At its heart, rsync uses a flexible pattern-matching system, similar to what you might find in your shell, but with a few crucial distinctions. When you use --exclude='pattern', rsync evaluates that pattern against the relative paths of files and directories within your source. This is a key point, folks – it's all about the path relative to the root of the transfer. So, if your source is /home/user/mydata and you're excluding temp/, rsync will look for /home/user/mydata/temp/. Simple enough, right?

The patterns themselves can include wildcards like * (matches zero or more characters), ? (matches a single character), and [] (matches a range or set of characters). These are your standard globbing patterns, super useful for general exclusions. For instance, --exclude='*.tmp' will catch all temporary files, and --exclude='photos/202[0-9]/' can exclude entire directories for specific years. However, the real magic, and often the source of confusion, comes with how rsync distinguishes between files and directories using these patterns. A pattern that ends with a forward slash (/) is explicitly designed to match only directories. So, --exclude='build/' will exclude any directory named build and all its contents, but it will not exclude a file named build. Conversely, a pattern without a trailing slash, like --exclude='core', will match both files AND directories named core. This distinction is paramount for our discussion today. If you tell rsync to exclude core, and you have a file named core and a directory named foo/bar/core/, then both will be considered for exclusion by default because the pattern core matches both types of entries.

Furthermore, rsync processes these exclusion patterns in a specific order, and the placement of your exclude options matters significantly, especially when combined with --include (which we'll touch on later). Each pattern is tested against the path as rsync traverses the directory tree. The first pattern that matches usually wins, unless you're using more complex filter rules. This means if you have a general --exclude rule and then a more specific --include rule, the order in which they appear in your command can drastically change the outcome. Mastering this order and the nuances of trailing slashes is your secret weapon against unexpected rsync behavior. Keep in mind that rsync also has special pattern characters for matching at the root of the transfer, such as /- for patterns that match entries only at the source directory root level, which gives you even more granular control for top-level files or directories. By understanding these core principles – relative paths, wildcard usage, the significance of the trailing slash, and pattern matching order – you're laying a solid foundation for crafting rsync commands that precisely reflect your backup strategy, no more guesswork involved!

Unpacking --delete and --delete-excluded Magic

Now, let's talk about the --delete and --delete-excluded flags, which can be both incredibly powerful and, if misunderstood, the source of significant headaches – or worse, data loss! When you're running rsync to synchronize a source to a destination, its primary job is to make the destination look exactly like the source. The --delete option is crucial for achieving this perfect mirror. Essentially, --delete tells rsync, "Hey, if a file or directory exists on the destination but doesn't exist on the source, go ahead and delete it from the destination." This is super useful for cleaning up old files that have been removed from your source, ensuring your destination is always a true reflection of your current data. Without --delete, rsync would only add or update files, leaving behind any files that are no longer present in the source, potentially cluttering your backups with stale data. Imagine your website's image gallery, where old images are removed; --delete ensures those old images don't linger on your backup server, saving space and keeping things tidy. It's a fantastic way to maintain a lean, accurate mirror of your active data.

Now, add --delete-excluded into the mix, and things get even more interesting. This flag takes the --delete concept a step further by saying, "Not only delete files from the destination that are missing from the source, but also delete files from the destination that match any of my exclusion patterns, even if they do exist on the source." This is where our current puzzle lies. If you've got an --exclude='core' pattern, and there's an old core file or directory on your destination that matches this pattern, --delete-excluded will actively remove it. The catch here is that --delete-excluded operates based on the same exclusion rules that rsync uses to decide what not to transfer from the source. So, if a file or directory on the destination matches an exclusion pattern, rsync effectively treats it as if it "doesn't belong," and --delete-excluded cleans it up.

The critical aspect here is the order of operations and how rsync applies these rules. First, rsync builds a list of files and directories from the source that should be transferred, after applying all --exclude and --include rules. Then, it compares this list with the files and directories already present on the destination. If --delete is active, any files on the destination that aren't on the source's "to-be-transferred" list are candidates for deletion. If --delete-excluded is also active, then any files on the destination that match any of your specified exclusion patterns are also added to that deletion list. This means that if you're using --exclude='core' and --delete-excluded, and you have a core directory on your destination, rsync will see that core matches your exclusion pattern, and --delete-excluded will wipe it out, along with all its contents. This is a common pitfall! It's super important to realize that --delete-excluded doesn't just apply to files that would have been excluded from the source, but to any item on the destination that matches an exclusion pattern. So, when you're trying to selectively exclude a file but keep a directory with the same name, --delete-excluded can be your nemesis if your exclusion patterns aren't precise enough. It's a powerful broom, but it sweeps broadly if you don't define your boundaries carefully. This is why understanding the nuances of how exclusion patterns interact with these deletion flags is not just a good idea, it's essential for maintaining the integrity and completeness of your backups. Always test your commands with --dry-run first to avoid any unwanted surprises!

The Core Conflict: When Files and Directories Share a Name

Alright, let's get down to the real conundrum that brings many a rsync user to the brink of despair: when you have a file named core and a directory named core (e.g., foo/bar/core/) somewhere in your source tree, and you want to treat them differently. Your goal, just like our original scenario, is to exclude the file core from your backups, but include and back up the directory foo/bar/core/ and its contents, like baz.txt. This is where rsync's pattern matching, combined with the --delete-excluded flag, can get super tricky if you don't know the specific incantations.

As we discussed, a simple --exclude='core' pattern is ambiguous to rsync. Because it doesn't end with a trailing slash, rsync interprets core as matching both a file named core and a directory named core. So, if you run rsync -av --delete --delete-excluded --exclude='core' source/ dest/, what happens? Rsync will find the file source/core and correctly exclude it from the transfer. If dest/core exists from a previous backup, --delete-excluded will likely remove it. Now, for source/foo/bar/core/baz.txt, rsync will also process the source/foo/bar/core/ directory. Since --exclude='core' matches the directory core as well, rsync will exclude the entire source/foo/bar/core/ directory and all its contents from the transfer. This means baz.txt won't get backed up, and if dest/foo/bar/core/ exists, --delete-excluded will happily wipe it out. This is the exact opposite of what you want!

The problem isn't that rsync isn't excluding the subdirectory. The problem is that the pattern --exclude='core' is too broad and does exclude both the file and the directory. The user's initial assumption,