Yesterday The Storage Alchemist at Storwize posted a complaint about Tom’s discussion of compression and deduplication. We certainly aren’t savaging compression technologies — I think perhaps it’s clearer to consider our points not so much as a criticism of compression, but as a list of concerns regarding bump-in-the-wire optimization appliances. We absolutely agree with Steve the Alchemist that data compression and data deduplication are two technologies that complement one another well — we use both in our Enterprise Archive Value NAS and Cloud Storage offerings , and we make it possible for our partners to compress (if they so choose) when using our Albireo SDK.

I’ll comment on his technical concerns.

Compression and deduplication are very similar in that they identify and eliminate redundant data, but the scope of this duplicate identification is vastly different. Traditional compression works on a small window of data and with short duplicate segments so that the compression tables fit efficiently in a very small amount of memory. Storwize may not be using a 64 KB window, but I imagine the order of magnitude is about right… and that’s not a criticism of their technology at all. In fact, the way Storwize manages data in chunks so that they can maintain performance is very clever.

Calling deduplication lossy is nonsense; both compression and dedupe replace redundant data with references to other instances of that data, just at different scales as I note above. Unlike Ocarina’s NFO, which frighteningly throws away actual content, both dedupe and traditional compression return the original bitstream. Tom’s point was that Albireo embedded dedupe leverages existing file and block system concepts to make those references so no interaction with our software is required on read, while a compression appliance modifies the data format before it reaches the storage array, which creates data lock-in. Take away the appliance, and the storage is full of uninterpretable data. That’s a concern for storage vendors and users alike.

As to the chart, when you look at this as ‘embedded dedupe’ vs. ‘appliance-mediated compression’, you can see why Tom says that appliance compression alters the data, and Albireo dedupe does not require ‘rehydration’. As for ‘optimizes block’, I haven’t yet seen Storwize’s block optimzation products, so I can’t comment, but I do wonder how they make the space saved to compression available to the user? We agree that savings are absolutely data dependent. In general, deduplication alone offers more savings than compression alone, and both together give the best results by far. Perhaps we can work together to ensure Albireo and Storwize yield optimal results?

I’ll approve, but in general it is poor form to put a link in someone’s comments back to your blog. – Farley had posted something on his blog about good linking practices. I would be more than happy to write a response that links to you but no sweat. And I also already added some comments on his post. Thanks for the update Mike.

Steve

Jered posted a follow up blog to address some of the technical points you discuss above. Rather than fill up your comment section, you can view them here: http://blog.permabit.com/index.php/2010/06/compression-and-dedupe-redux/

Regards,
Mike

Remember what Alexis de Toqueville said: “America is great because she is good.” I have great faith in market forces driving “good”, customer centric behavior.

Keep charging.

Tom

First, thanks for your note. I know there is some sarcasm and trust me I am no CEO and what you have done with Permabit is nothing short of great so I am glad the comments were taking in the manner in which the were intended.

I actually like the dialog around what customers are looking for very much. I would agree with you 100% that customers don’t want vendor lock in. The real question is what do the OEM’s want? I would say the do care about the customer BUT I would also think that behind the scenes they do want vendor lock in because they want customers to keep buying their stuff. I also don’t think this is a bad strategy if you are a storage vendor. You need a competitive edge and if partnering with a provider who provides a unique solution can give you that edge, that is a good model.

I’ll leave the announcement thing alone for now and just say, best of luck and I hope they can come out with solutions soon. And as you are supportive of compression, I too am very supportive of the total, overall capacity optimization strategy, AS LONG AS – the end users get everything they require (and I believe you believe the same thing as well).

Thanks again Tom!
Steve

You know, I am spending a lot of time meeting with OEM customers and prospects and my blogs (usually written while flying) generally reflect what I hear from them. They fear data lock-in and they want to do what is right for their customers in terms of data optimization and data safety while maintaining storage performance and features.

We are a supplier and partner to OEMs. They will make product announcements on their own timelines.

We are highly supportive of compression owned, operated and deployed by the storage vendor, because it makes sense and that is what OEMs tell me they are implementing. Isn’t that supportive of the future Storwize business model?

Cheers,

Tom

To be fair, I did post to Tom’s blog as well with some questions about his numbers and reasoning. It remains to be seen if the post will be approved, and if he’ll respond to it. Stay tuned.

Among other things, I told him that his statement “Compression alters the underlying data structures and requires compression and decompression of data.” makes no sense at all.

Couple of points – I actually think we are in close agreement unless I missed something. The first line of 3rd paragraph, the meat of the post, I talk about the fact that the two technologies are complementary. I was trying to point out that the posts that are still trying to ‘compare’ the two are a waste of time. However, if you are going to compare, let me at least shed some light on the facts.

Your point regarding lossless and lossy are also right on. I would put them both in the same category, however if you review Tom’s chart – to say that compression is lossy when it is not by definition is a gross misrepresentation of the facts. I was merely trying to point out this fact and how could dedupelication be less lossy than compression?

Anyway, as I said in the beginning of the post, I hate playing the defensive, no one usually wins – I was just trying to set the record straight.

Thanks again, as always for reading and commenting and making the post better.

Steve

I’m not sure how many times I can possibly write this, but clearly more than a dozen times on various storage blogs is it enough: Dedupe and compression are essentially two peas from the same pod.

For the most part, compression is the term used to describe data reduction applied WITHIN individual files (this includes such types as ZIP, SIT, RAR and TAR etc which DO NOT look for commonality across files), and dedupe is the term used to describe data reduction as it is applied ACROSS multiple files and types of data. I prefer to refer to the two by application: intrafile and interfile compression. It’s neat, simple and most people will quickly understand the difference. The two are complementary.

I ask that you please not refer to dedupe as “lossy”. Dedupe (as I have seen it implemented to-date) is no more “lossy” than lossless compression. Today’s dedupe, like lossless compression, allows the EXACT original data to be reconstructed from the compressed data. Any opinion to the contrary is a gross misrepresentation of the facts.

Using the term lossy in the context of dedupe will only confuse readers. Lossy data compression allows an approximation of the original data to be reconstructed in exchange for far superior compression rates. And yes, it takes a lot more computational horsepower and time to execute lossy compression at tolerable speeds.

Your thoughts?