Synchronizing files between directories is a common task for managing backups, ensuring consistency across multiple storage locations, or simply keeping data organized.

While there are many tools available to do this, creating a Python script to handle directory synchronization offers flexibility and control.

This guide will walk you through a Python script designed to synchronize files between two directories.

You can download the source code for the basic and expanded version here:

Get Source Code

You can also follow along with the video version:

Introduction to the Script

The script begins by importing several essential Python libraries.

These include os for interacting with the operating system, shutil for high-level file operations, filecmp for comparing files, argparse for parsing command-line arguments, and tqdm for displaying progress bars during lengthy operations.

These libraries work together to create a robust solution for directory synchronization.

import os
import shutil
import filecmp
import argparse
from tqdm import tqdm

The scripts uses mainly Python built-in modules, but for the progress bar is uses the tqdm library, which needs to the installed with:

pip install tqdm

Checking and Preparing Directories

Before starting the synchronization, the script needs to check if the source directory exists.

If the destination directory doesn't exist, the script will create it.

This step is important to make sure the synchronization process can run smoothly without any issues caused by missing directories.

# Function to check if the source and destination directories exist
def check_directories(src_dir, dst_dir):
    # Check if the source directory exists
    if not os.path.exists(src_dir):
        print(f"\nSource directory '{src_dir}' does not exist.")
        return False
    # Create the destination directory if it does not exist
    if not os.path.exists(dst_dir):
        os.makedirs(dst_dir)
        print(f"\nDestination directory '{dst_dir}' created.")
    return True

The check_directories function makes sure that both the source and destination directories are ready for synchronization. Here's how it works:

  • The function uses os.path.exists() to check if the directories exist.
  • If the source directory is missing, the script tells the user and stops running.
  • If the destination directory is missing, the script creates it automatically using os.makedirs(). This ensures that the necessary directory structure is in place.
Are you tired of writing the same old Python code? Want to take your programming skills to the next level? Look no further! This book is the ultimate resource for beginners and experienced Python developers alike.

Get "Python's Magic Methods - Beyond __init__ and __str__"

Magic methods are not just syntactic sugar, they're powerful tools that can significantly improve the functionality and performance of your code. With this book, you'll learn how to use these tools correctly and unlock the full potential of Python.

Synchronizing Files Between Directories

The main job of the script is to synchronize files between the source and destination directories.

The sync_directories function handles this task by first going through the source directory to gather a list of all files and subdirectories.

The os.walk function helps by generating file names in the directory tree, allowing the script to capture every file and folder within the source directory.

# Function to synchronize files between two directories
def sync_directories(src_dir, dst_dir, delete=False):
    # Get a list of all files and directories in the source directory
    files_to_sync = []
    for root, dirs, files in os.walk(src_dir):
        for directory in dirs:
            files_to_sync.append(os.path.join(root, directory))
        for file in files:
            files_to_sync.append(os.path.join(root, file))

    # Iterate over each file in the source directory with a progress bar
    with tqdm(total=len(files_to_sync), desc="Syncing files", unit="file") as pbar:
        # Iterate over each file in the source directory
        for source_path in files_to_sync:
            # Get the corresponding path in the replica directory
            replica_path = os.path.join(dst_dir, os.path.relpath(source_path, src_dir))

            # Check if path is a directory and create it in the replica directory if it does not exist
            if os.path.isdir(source_path):
                if not os.path.exists(replica_path):
                    os.makedirs(replica_path)
            # Copy all files from the source directory to the replica directory
            else:
                # Check if the file exists in the replica directory and if it is different from the source file
                if not os.path.exists(replica_path) or not filecmp.cmp(source_path, replica_path, shallow=False):
                    # Set the description of the progress bar and print the file being copied
                    pbar.set_description(f"Processing '{source_path}'")
                    print(f"\nCopying {source_path} to {replica_path}")

                    # Copy the file from the source directory to the replica directory
                    shutil.copy2(source_path, replica_path)

            # Update the progress bar
            pbar.update(1)

Once the list of files and directories is compiled, the script uses a progress bar provided by tqdm to give the user feedback on the synchronization process.

For each file and directory in the source, the script calculates the corresponding path in the destination.

  • If the path is a directory, the script ensures it exists in the destination.
  • If the path is a file, the script checks whether the file already exists in the destination and whether it is identical to the source file.
  • If the file is missing or different, the script copies it to the destination.

This way, the script keeps the destination directory up-to-date with the source directory.

Cleaning Up Extra Files

The script also has an optional feature to delete files in the destination directory that are not in the source directory.

This is controlled by a --delete flag that the user can set.

If this flag is used, the script goes through the destination directory and compares each file and folder to the source.

If it finds anything in the destination that isn't in the source, the script deletes it.

This ensures that the destination directory is an exact copy of the source directory.

    # Clean up files in the destination directory that are not in the source directory, if delete flag is set
    if delete:
        # Get a list of all files in the destination directory
        files_to_delete = []
        for root, dirs, files in os.walk(dst_dir):
            for directory in dirs:
                files_to_delete.append(os.path.join(root, directory))
            for file in files:
                files_to_delete.append(os.path.join(root, file))

        # Iterate over each file in the destination directory with a progress bar
        with tqdm(total=len(files_to_delete), desc="Deleting files", unit="file") as pbar:
            # Iterate over each file in the destination directory
            for replica_path in files_to_delete:
                # Check if the file exists in the source directory
                source_path = os.path.join(src_dir, os.path.relpath(replica_path, dst_dir))
                if not os.path.exists(source_path):
                    # Set the description of the progress bar
                    pbar.set_description(f"Processing '{replica_path}'")
                    print(f"\nDeleting {replica_path}")

                    # Check if the path is a directory and remove it
                    if os.path.isdir(replica_path):
                        shutil.rmtree(replica_path)
                    else:
                        # Remove the file from the destination directory
                        os.remove(replica_path)

                # Update the progress bar
                pbar.update(1)

This part of the script uses similar techniques as the synchronization process.

It uses os.walk() to gather files and directories and tqdm to show progress.

The shutil.rmtree() function is used to remove directories, while os.remove() handles individual files.

Running the Script

The script is designed to be run from the command line, with arguments specifying the source and destination directories.

The argparse module makes it easy to handle these arguments, allowing users to simply provide the necessary paths and options when running the script.

# Main function to parse command line arguments and synchronize directories
if __name__ == "__main__":
    # Parse command line arguments
    parser = argparse.ArgumentParser(description="Synchronize files between two directories.")
    parser.add_argument("source_directory", help="The source directory to synchronize from.")
    parser.add_argument("destination_directory", help="The destination directory to synchronize to.")
    parser.add_argument("-d", "--delete", action="store_true",
                        help="Delete files in destination that are not in source.")
    args = parser.parse_args()

    # If the delete flag is set, print a warning message
    if args.delete:
        print("\nExtraneous files in the destination will be deleted.")

    # Check the source and destination directories
    if not check_directories(args.source_directory, args.destination_directory):
        exit(1)

    # Synchronize the directories
    sync_directories(args.source_directory, args.destination_directory, args.delete)
    print("\nSynchronization complete.")

The main function brings everything together.

It processes the command-line arguments, checks the directories, and then performs the synchronization.

If the --delete flag is set, it also handles the cleanup of extra files.

Examples

Let's see some examples of how to run the script with the different options.

Source to Destination

python file_sync.py d:\sync d:\sync_copy 

Destination directory 'd:\sync2' created.
Processing 'd:\sync\video.mp4':   0%|                                                                                                   | 0/5 [00:00<?, ?file/s]
Copying d:\sync\video.mp4 to d:\sync2\video.mp4
Processing 'd:\sync\video_final.mp4':  20%|██████████████████▌                                                                          | 1/5 [00:00<?, ?file/s] 
Copying d:\sync\video_final.mp4 to d:\sync2\video_final.mp4
Processing 'd:\sync\video_single - Copy (2).mp4':  40%|████████████████████████████████▍                                                | 2/5 [00:00<?, ?file/s] 
Copying d:\sync\video_single - Copy (2).mp4 to d:\sync2\video_single - Copy (2).mp4
Processing 'd:\sync\video_single - Copy.mp4':  60%|█████████████████████████████████████████████▌                              | 3/5 [00:00<00:00, 205.83file/s]
Copying d:\sync\video_single - Copy.mp4 to d:\sync2\video_single - Copy.mp4
Processing 'd:\sync\video_single.mp4':  80%|██████████████████████████████████████████████████████████████████▍                | 4/5 [00:00<00:00, 274.44file/s] 
Copying d:\sync\video_single.mp4 to d:\sync2\video_single.mp4
Processing 'd:\sync\video_single.mp4': 100%|███████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 343.05file/s] 

Synchronization complete.

Source to Destination with Cleanup of Extra Files

python file_sync.py d:\sync d:\sync_copy -d


Extraneous files in the destination will be deleted.
Syncing files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 63.29file/s]
Deleting files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<?, ?file/s] 

Synchronization complete.

Expanding the Script with Advanced Features

This expanded version not only copies files between two directories but also offers additional options to customize the synchronization to your specific needs.

Below, we explore the key features and enhancements included in this version of the script.

Excluding Files and Directories

One of the most significant enhancements is the ability to exclude specific files or directories from synchronization based on patterns.

By using the --exclude option, users can specify patterns that match files or folders they wish to skip.

This is particularly useful for ignoring temporary files, cache directories, or other non-essential data that should not be synchronized.

The script uses the fnmatch module to handle pattern matching, making it easy to filter out unwanted items.

Handling Hidden Files and Directories

Another valuable feature is the ability to control the synchronization of hidden files and directories.

By default, the script ignores hidden files and directories (those starting with a dot . in Unix-based systems) during synchronization.

However, users can include these hidden items by enabling the --include-hidden option.

This provides flexibility depending on whether hidden files are considered part of the essential data set.

Dry Run Mode for Safe Execution

Mistakes during synchronization can lead to the loss or overwriting of important data.

To mitigate this risk, the script includes a --dry-run mode.

When this option is activated, the script simulates the synchronization process without making any actual changes to the files or directories.

This allows users to review what actions the script would take, such as copying or deleting files, before committing to these changes.

The dry run feature is invaluable for testing and ensuring that the synchronization process behaves as expected.

Sample code

# Main function to parse command line arguments and synchronize directories
if __name__ == "__main__":
    # Parse command line arguments
    parser = argparse.ArgumentParser(description="Synchronize files between two directories.")
    parser.add_argument("source_directory", help="The source directory to synchronize from.")
    parser.add_argument("destination_directory", help="The destination directory to synchronize to.")
    parser.add_argument("-d", "--delete", action="store_true",
                        help="Delete files in destination that are not in source.")
    parser.add_argument("--dry-run", action="store_true",
                        help="Perform a dry run, i.e., do not actually modify any files.")
    parser.add_argument("--exclude", nargs='*',
                        help="List of patterns to exclude from synchronization (e.g., '*.tmp', '*/__pycache__').")
    parser.add_argument("--include-hidden", action="store_true",
                        help="Include hidden files and directories.")
    args = parser.parse_args()

    # If the delete flag is set, print a warning message
    if args.delete:
        print("\nExtraneous files in the destination will be deleted.")

    # If the dry-run flag is set, print a message
    if args.dry_run:
        print("\nPerforming a dry run, no files will be modified.")

    # Check the source and destination directories
    if not check_directories(args.source_directory, args.destination_directory):
        exit(1)

    # Synchronize the directories
    sync_directories(args.source_directory, args.destination_directory, args.delete, args.dry_run, args.exclude, args.include_hidden)
    print("\nSynchronization complete.")

You can download the source code for the basic and expanded version here:

Get Source Code

Conclusion

This Python script offers a powerful and flexible way to synchronize files between two directories.

It uses key libraries like os, shutil, and filecmp, and enhances the user experience with tqdm for tracking progress.

This ensures that your data is consistently and efficiently synchronized.

Whether you're maintaining backups or ensuring consistency across storage locations, this script can be a valuable tool in your toolkit.

Tagged in:

Python