Client-Side Deduplication

Deduplication is an approach based on reusing identical data in several places.

The new backup format is based on client-side deduplication. Deduplication on a local computer brings the following benefits:

• Client-side deduplication is much faster than server-side deduplication
• Absence of internet connection issues: data is deduplicated locally
• Significant decrease in internet traffic
• Ability to purge unnecessary data
• Storage costs: the server deduplication database constantly grows, which can significantly increase expenses. Client-side deduplication uses only local capacity.

How It Works

The first backup is always full. In most cases, it is enough to have a full backup with subsequent incremental backups. Thus, after a full backup, the next backup plan executions are usually incremental and depend on full backup and previous incremental backups as well.

The new backup format accounts for full backup plan independence, so each separate backup plan has its own deduplication database. Moreover, backup plan generations (generation is a sequence of full and incremental backups that follow this full one) also have their own deduplication databases.

Once a backup plan is run, Backup for macOS reads backup data in batches that are multiples (2x, 4x, ...) of the block size. Once a block is read, it is compared with deduplication database records. If a block is not found, it is delivered to storage and assigned a block ID, which becomes a new deduplication database record. The block scanning continues, and if a block matches any of the deduplication database records, a block with such ID is not backed up.

This approach significantly decreases the backup size, especially in virtual environments with a large number of identical blocks.