To understand how dedupe works in an EMC Avamar server, you must first understand the methods of data deduplication commonly found today. Below is a brief explanation of the top 3.

<strong>File-level</strong> (the most common)

- File level identifies duplicate files within a volume, and commonly uses a hashing technique to take the file name, size, last modified date, and other similar metadata information to make a unique hash for that file.

- If two copies of this exist, and a user opens and changes a single letter or adds a period to it, the hash becomes unique, and this file will not be de-duplicated.

This is what you’d find on an EMC Celerra or VNX CIFS share. This is good for environments with lots of file system type data and users that tend to share data between them. Our customers seem to be in the 20-25% space savings range using this technology on their VNX / Celerra CIFS shares, but your mileage may vary.

<strong>Fixed Block Level</strong> (also called fixed length)

- Common in snapshot/replication technologies, it breaks a file into a fixed number of bits, and if they’re the same, that segment is de-duplicated

If you have two files that break down into the following bits, the <span style="color: #ff0000">RED</span> sections would be de-duplicated based on a 4-bit block length.

<span style="color: #ff0000">1111</span> 1111 0001 / <span style="color: #ff0000">1111</span> 1110 0000

This will usually have a higher deduplication perecentage than File level, as files of different types that have identical segments (sub-file level) will still see some level of deduplication. But this isn't the best solution.

<strong>Variable block level (what Avamar uses)</strong>

- This uses an intelligent method to determine segment size based on the data itself to determine length size.

- Provides much greater granularity in finding duplicate data

Avamar’s intelligence allows it to look at a file, read the header and say “this is an X type of file, so I’m going to break it into these variable segments”. This provides the greatest level of deduplication. So with that foundation

Note: Most of the following process happens on the CLIENT itself, which is where "source dedupe" happens. Keep this in mind when reading through this post and watching the video at the bottom. The benefit of this is faster backups. and substantially less consumption of bandwidth, which is ideal in environments that have smaller WAN pipes between offices.

When a file is backed up to an Avamar server, the file is hashed using its meta data (file size, date modified, etc.), and this 20-byte SHA1 hash is stored in the "file cache" file locally on the client. When this client goes to backup its data, this file is loaded into RAM, and each file is then hashed, and compared to the hashes in the file. If it matches, it means the file has already been backed up and has not been modified, so the next file is processed. Because this is done in memory, it happens VERY quickly, and it's common to see 95-98% of data on a traditional file system be satisfied against the file cache.

Assume the file HAS been changed, and therefore the hash of the file no longer matches the hash in the file cache. From here, the data will be broken into variable segment lengths through Avamar's <a href="http://patft.uspto.gov/netacgi/nph-Parser?Sect2=PTO1&amp;Sect2=HITOFF&amp;p=1&amp;u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&amp;r=1&amp;f=G&amp;l=50&amp;d=PALL&amp;RefSrch=yes&amp;Query=PN%2F6810398">patented sticky byte factoring process</a>. These segments are then compressed and hashed, and the resulting hash is written to a HASH CACHE file locally on the client. This file is also loaded into RAM when a backup is initiated.

When the client begins it backup, each of these segments will be compared against hashes in the hash cache, and if it matches any hash (which could be from a different file), it will not send the data to the Avamar system, because it knows that specific piece of data has already been backed up. If Avamar finds the specific segment has not been backed up (the hash doesn't exist in the hash cache), it will then take that hash and send ONLY the hash to the Avamar grid itself, and check to see if that segment has been backed up by ANY OTHER client in the environment. This reduces bandwidth, as only the hash gets transmitted, not the data itself, and is where the "global" dedupe magic happens.

http://www.youtube.com/watch?v=EbqW_016nVM

For the record, my Macbook Air keeps autocorrecting "deduplication" into "reduplication" in case you see that in any of these posts.