Synchronizing Files Between Two Directories Using Python

Synchronizing files between directories is a common task for managing backups, ensuring consistency across multiple storage locations, or simply keeping data organized.

While there are many tools available to do this, creating a Python script to handle directory synchronization offers flexibility and control.

This guide will walk you through a Python script designed to synchronize files between two directories.

You can download the source code for the basic and expanded version here:

You can also follow along with the video version:

Introduction to the Script

The script begins by importing several essential Python libraries.

These include os for interacting with the operating system, shutil for high-level file operations, filecmp for comparing files, argparse for parsing command-line arguments, and tqdm for displaying progress bars during lengthy operations.

These libraries work together to create a robust solution for directory synchronization.

import os
import shutil
import filecmp
import argparse
from tqdm import tqdm

The scripts uses mainly Python built-in modules, but for the progress bar is uses the tqdm library, which needs to the installed with:

pip install tqdm

Checking and Preparing Directories

Before starting the synchronization, the script needs to check if the source directory exists.

If the destination directory doesn't exist, the script will create it.

This step is important to make sure the synchronization process can run smoothly without any issues caused by missing directories.

# Function to check if the source and destination directories exist
def check_directories(src_dir, dst_dir):
    # Check if the source directory exists
    if not os.path.exists(src_dir):
        print(f"\nSource directory '{src_dir}' does not exist.")
        return False
    # Create the destination directory if it does not exist
    if not os.path.exists(dst_dir):
        os.makedirs(dst_dir)
        print(f"\nDestination directory '{dst_dir}' created.")
    return True

The check_directories function makes sure that both the source and destination directories are ready for synchronization. Here's how it works:

• The function uses os.path.exists() to check if the directories exist.
• If the source directory is missing, the script tells the user and stops running.
• If the destination directory is missing, the script creates it automatically using os.makedirs(). This ensures that the necessary directory structure is in place.

Synchronizing Files Between Directories

The main job of the script is to synchronize files between the source and destination directories.

The sync_directories function handles this task by first going through the source directory to gather a list of all files and subdirectories.

The os.walk function helps by generating file names in the directory tree, allowing the script to capture every file and folder within the source directory.

# Function to synchronize files between two directories
def sync_directories(src_dir, dst_dir, delete=False):
    # Get a list of all files and directories in the source directory
    files_to_sync = []
    for root, dirs, files in os.walk(src_dir):
        for directory in dirs:
            files_to_sync.append(os.path.join(root, directory))
        for file in files:
            files_to_sync.append(os.path.join(root, file))

    # Iterate over each file in the source directory with a progress bar
    with tqdm(total=len(files_to_sync), desc="Syncing files", unit="file") as pbar:
        # Iterate over each file in the source directory
        for source_path in files_to_sync:
            # Get the corresponding path in the replica directory
            replica_path = os.path.join(dst_dir, os.path.relpath(source_path, src_dir))

            # Check if path is a directory and create it in the replica directory if it does not exist
            if os.path.isdir(source_path):
                if not os.path.exists(replica_path):
                    os.makedirs(replica_path)
            # Copy all files from the source directory to the replica directory
            else:
                # Check if the file exists in the replica directory and if it is different from the source file
                if not os.path.exists(replica_path) or not filecmp.cmp(source_path, replica_path, shallow=False):
                    # Set the description of the progress bar and print the file being copied
                    pbar.set_description(f"Processing '{source_path}'")
                    print(f"\nCopying {source_path} to {replica_path}")

                    # Copy the file from the source directory to the replica directory
                    shutil.copy2(source_path, replica_path)

            # Update the progress bar
            pbar.update(1)

Once the list of files and directories is compiled, the script uses a progress bar provided by tqdm to give the user feedback on the synchronization process.

For each file and directory in the source, the script calculates the corresponding path in the destination.

• If the path is a directory, the script ensures it exists in the destination.
• If the path is a file, the script checks whether the file already exists in the destination and whether it is identical to the source file.
• If the file is missing or different, the script copies it to the destination.

This way, the script keeps the destination directory up-to-date with the source directory.

Cleaning Up Extra Files

The script also has an optional feature to delete files in the destination directory that are not in the source directory.

This is controlled by a --delete flag that the user can set.

If this flag is used, the script goes through the destination directory and compares each file and folder to the source.

If it finds anything in the destination that isn't in the source, the script deletes it.

This ensures that the destination directory is an exact copy of the source directory.