Client-Side Deduplication

Deduplication is an approach that is based on multiple usage of identical data in several places.

The new backup format is based on the client-side deduplication. Deduplication on a local computer brings the following benefits:

• Client-side deduplication is much faster compared to a server deduplication
• Absence of internet connection issues: data is deduplicated locally
• Significant decrease in internet traffic
• Ability to purge an unnecessary data
• Storage costs: server deduplication database constantly grows, so this can cause a significant expense increase. Client-side deduplication uses local capacities only

How It Works

The first backup is always full. In most cases, it is enough to have a full backup with subsequent incremental backups. Thus, after a full backup, the next backup plan executions are usually incremental and depend on full backup and previous incremental backups as well.

The new backup format reckons for a full backup plan independence, so each separate backup plan has its own deduplication database. Moreover, backup plan generations (generation is a sequence of full and incremental backups that follow this full one) also have their own deduplication databases.

Once a backup plan is run, Backup for Linux reads backup data in batches multiple (2x, 4x,...) to block size. Once a block is read, it is compared with deduplication database records. If a block is not found, it is delivered to storage and is assigned with a block ID, which becomes a new deduplication database record. The block scanning continues, and if a block matches any of the deduplication database records, a block with such ID is not backed up.

This approach significantly decreases a backup size, especially in virtual environments with a large number of identical blocks.