“I don’t know why, but there is a misconception among practitioners about the technologies we all refer to as compression and deduplication. They aren’t oil and water. They’re more like two different flavors of water.

A few of my past comments about the topic can be found here:
http://wikibon.org/wiki/v/Pitfalls_of_compressing_online_storage

I wrote, “De-dupe (of the type the storage industry now markets) IS a form of compression. Traditional “dedupe” (aka compression) occurs within a file (e.g. gif, jpg, zip, tar, etc) where the data resides along with its dictionary. In contrast, storage-level de-dupe is executed across files and repositories using an external “dictionary” managed separately from the files. That is to say, compression is little more than ultra-granular deduplication occurring within a file.”

Obviously, it is always possible to further compress data that has been “deduplicated” by looking for redundancy at a more granular level than that which was used to deduplicate the original data. It is no different, in practice, than turning up the level of compression or deduplication from low to high such that comparisons are made between decreasingly smaller strings.

So, Mike, figures such as 50% compression, 20% deduplication and a combined 55% (dedupe and compression) are absolutely meaningless without additional context. The outcome of each test would depend on the base (and relative) levels of compression/deduplication. Crank up dedupe (using smaller or variable length strings) and the benefit from further compression will drop. Dial back the dedupe and the benefits from further compression will jump. A useful (if imperfect) analogy: It’s a bit like compressing a JPG in Photoshop using the “low” setting, then compressing the output file a second time using the “high” setting…albeit lossy.”

To-date Mike has not approved the comment to appear on his blog, nor has he replied to it. Frankly, there’s not much he can say.

It’s definitely Milk & Cookies.

BTW – It’s a shame there is no industry standard benchmarks for primary storage compression/dedup. This would allow potential IT buyers to compare performance and data reduction at the same time.

The next issue is does it work with your backups? Now if your backing up to tape the question is, what ends up on tape and can it be found easily enough should you need to do a recovery? Given the fact that IT has invested heavily over the last couple of years, and they all tell you to decompress your data before you backup – the question is – does it work with deduplicaiton? There is only one technology that does this today because it does random access compression and that is a part of the IP of Storwize. So the new question is, does it make sense to stop doing deduplication for the segment of your data that is 4x or more than your primary with all the fulls and incrementals using a compression technology on primary that may only save you 50% of the primary footprint and may be slow? Again, there may be cases where that is an okay solution, as with every answer in IT – it depends.