See my video below which outlines a high level overview of the backup process and how the variable segment length, source and global deduplication works in an EMC Avamar solution.

Here are the different / common types of data deduplication found in the marketplace today.

<strong>File-level</strong> (the most common)

- File level identifies duplicate files within a volume, and commonly uses a hashing technique to take the file name, size, last modified date, and other similar metadata information to make a unique hash for that file.

- If two copies of this exist, and a user opens and changes a single letter or adds a period to it, the hash becomes unique, and this file will not be de-duplicated.

This is what you’d find on an EMC Celerra or VNX CIFS share. This is good for environments with lots of file system type data and users that tend to share data between them. Our customers seem to be in the 20-25% space savings range using this technology on their VNX / Celerra CIFS shares, but your mileage may vary.

Fixed Block Level (also called fixed length)
<ul>
	<li>Common in snapshot/replication technologies, it breaks a file into a fixed number of bits, and if they’re the same, that segment is de-duplicated</li>
</ul>
If you have two files that break down into the following bits, the <span style="color: #ff0000">RED</span> sections would be de-duplicated.

<span style="color: #ff0000">1111</span> 1111 0001

<span style="color: #ff0000">1111</span> 1110 0000

This will usually have a higher deduplication perecentage than File level, as files of different types that have identical segments (sub-file level) will still see some level of deduplication.

Variable block level (what Avamar uses)
<ul>
	<li>• This uses an intelligent method to determine segment size based on the data itself to determine length size.</li>
	<li>• Provides much greater granularity in finding duplicate data</li>
</ul>
&nbsp;

Avamar’s intelligence allows it to look at a file, read the header and say “this is a X type of file, so I’m going to break it into these variable segments”.

&nbsp;

This provides the greatest level of deduplication.

EMC Avamar uses variable block length deduplication, compression, and hashing to achieve the incredibly high levels of efficiency we see customers happily enjoying every day.

One thing I forgot to mention in the video is when the process gets to the point where it checks to see if the hash of the data exists on the Avamar server (step 6), this is the "global" part of the global deduplication magic that is EMC Avamar.

http://www.youtube.com/watch?v=EbqW_016nVM

For the record, my Macbook Air keeps autocorrecting "deduplication" into "reduplication" in case you see that in any of these posts.