Great conversation on Linked In about deduplication and compression for storage efficiency in the Data Storage Professionals Group.  Help the storage community answer this question:

Does anyone has any experience in NAS de-duplication at filesystem level, like NetApps. Does it really work? I concerns/limitations?

What a great name, Snowball.  And much like information can behave like a "snowball" turning into an "avalanche", Snowball VFX has been able to control the avalanche of data utilizing the IBM Real-time Compression technology.  Here in this video customer case study, Yoni Cohen, Founder of Snowball talks about the value of Real-time Compression in his environment.

Additionally, we have captured from all of our Media & Entertainment customers what they think about the Real-time Compression technology.  To see the value proposition, check out our Media & Entertainment Portal through our good friends at TechValidate.

By the way, we couldn't have captured this amazing video without the help of the folks at Snowball for all of their footage they let us use as well as the folks at MediaBoss TV who helped us create this video.

Over the past few weeks I have witnessed the proverbial mudslinging that takes place in the blogosphere when marketing feathers are ruffled.  Most recently I was reading Rich Anderson of The StorageSavvy Blog.  The article was "Compression better than Dedup?  NetApp Confirms!"

I have to agree with Rich on many fronts.  First, "When all you have is a hammer, everything is a nail."  Rich points out vendors have to sell "what's in the bag" so it is conceivable that all problems look like they can be solved with their solution.  If you look back over the last few years NTAP has always had a "me too" reputation.  Whatever the industry has, they have one too and its better.  For the last few years, while competing against Storwize, they have pulled the EMC tactic of trying to stall a market by saying, "We have optimization for primary storage with deduplication."   The reality is, you can't use it in real time, it is a resource hog, and again Rich mentions, the only use case it works well on in primary storage is VMware (and that is ONLY IF the customer stores their data outside the .vmdk file otherwise compression is much better).  Now that NTAP has compression their story has changed saying that compression on primary storage is better for most use cases.  Duh!  The folks at Storwize (now IBM Real-time Compression) have been saying that for years.  Why, deduplication is great for repetitive data sets, i.e. backup, not primary storage.  There just isn't that much repetitive data in primary storage.  Again, NTAP is trying to stall the market saying they have "in-line" compression for primary storage.  Sorry guys, not good enough.  In-line is NOT Real-time.  Rich also points out that the key characteristics of storage for customers are capacity and performance.  Patrick Rogers of NTAP has said publically that compression WILL indeed impact performance and that they even have a tool that will tell you how much performance will be impacted.  While NTAP may say compression is "free", we all know nothing worth having in life is free, you get what you pay for.  If you need the performance to do compression you are going to have to perform a major upgrade to  your filer in order to just be able to perform compression let alone try to do compression in real time.  No real savings there.

It’s a simple fact that embedding optimization technology into the array you are optimizing is the right direction, it is just not ready for prime time today for a number of key reasons.  If you can't implement the technology in the array and preserve performance, availability, price and feature set, then its not a viable solution.  I know that every answer in IT is "It depends", and I am sure there are some use cases where the solution is "good enough" for some customers who meet all the proper criteria, the reality is, it is not ready for the general public.

On the other hand, if you can off load the work from the array in an appliance model, preserve performance and ensure 100% transparency to applications, infrastructure and processes then that is a real solution.   This is exactly what the real-time compression technology does from IBM.  Yes it is an appliance that sits in your CIFS/NFS storage infrastructure.  However, the benefits of the time based compression technology, along with the ability, by offloading the array from doing the compression, to maintain and in some cases increase performance, and the fact that putting the technology in place requires no change to your applications, no changes to your storage infrastructure and no changes to your downstream processes means this solution is the top primary storage optimization solution on the market.  It's not a big surprise.  First the technology comes from an Israeli startup, some of the smartest minds in storage / high tech.  Second, most great solutions start out as an appliance.  They are easy to deploy and allow you to get an understanding of how the technology really works in multiple use cases before trying to embed the solution with limited information.

Below are the top 10 reasons to deploy a Real-time Compression appliance for NAS from IBM over NTAP compression technology:

Of the customers running the technology in their environment over 57% of these customers have rated the Real-time Compression technology an "Excellent value – one of the top values in my data center" or better.

I am guessing by David Vaughn's visceral reaction that Rich hit a nerve that is more than just the typical old NTAP / EMC battling.  I don't think anyone ever told David that usually when you have such a reaction to something, you must be trying to hide something.  In this case, while NTAP has great products, they fall short when it comes to optimization technology.  Also, would you want your migration process to look like this?

Today IBM is making one of the most significant storage announcements in the last 10 years.  (Steve Duplessie is quoted as saying “…last 20 years.” but announcing that they were basically getting out of the storage business in 2000 by selling their Mylex division was pretty significant.)

Today IBM has made it abundantly clear that they are back in the storage business.  It makes a lot of sense actually.  Servers have been commoditized to about 3% margins and services is a body business (to make more money you need more bodies).  Storage is the only place these days to add significant ‘value’ to the infrastructure eco-system.

Storage software – or as I like to refer to them, storage services allow vendors to add more value to commodity hardware by providing very useful capabilities such as thin provisioning, virtualization, snapshots, replication, and optimization solutions such as real-time compression.

However this story is as much about the naming of the product as it is the product itself.

Let’s rewind a bit.  It is September 1^st in Tel-Aviv.  IBM has just completed the Storwize acquisition and we are having our leadership meetings discussing the integration when the new management team informs us that: “You know, we really like the Storwize name a lot.”  “We like the name so much that we decided we want to use it for a totally separate and new product.”

As new employees we two choices:

1. Complain and argue about how this will confuse the market
2. Salute the flag and move on

Having branded products and technology before, I didn’t really see this as an issue.  Sales of course didn’t care for it.  My feeling was, that even though the new Storwize V7000 doesn’t support the new IBM Real-time Compression (which now you have it, our new name – not flashy but does describe exactly what we do), it is the most modular storage architecture designed today and I am quite sure, that because the value in this ‘platform’ is really about storage services (software), IBM Real-time Compression could eventually make it on to this platform.  That being said, the amount of noise IBM is going to make regarding the new Storwize will drive the old Storwize sales team to a number of new opportunities.  Sure, we will have to work harder to vet them as the new V7000 is a block device and IBM Real-time Compression is an appliance for NAS today but that is okay.  We will now have exposure to a great deal of customers to tell our story.

After 5 months of trying to name their new product, IBM, at a high level had decided that Storwize IS going to be the new name of the new product which left us to find a new name for our product.  So, we saluted the flag and began, as new people to IBM, to ‘work the process to re-name’ our product in 10 days.  After two days, it was clear why IBM couldn’t name their product.  After dealing with, well lets just say ‘all the blue tape’, naming a product inside IBM is not easy.

While it was always our strategy to have the industry, which means analysts, create a category within storage optimization solutions that was a separate segment than compression called real-time compression.  Wikibon blog started this with the development of CORE – Capacity Optimization Ratio Efficiency.  While a heated discussion, all of the influencers who weighed in and would agree that when you can do compression in real-time it is much different than any post process methodology (and quite frankly better, but that is my bias opinion).

However, being short on time and needing to come up with a name we have ‘branded’ real-time compression, IBM Real-time Compression.  It’s hard for the analysts to now create a category with only one company in it with their product name the name of the category but that is life I guess.  Kinda reminds me of when STK decided to trademark ILM – how’s that working out for them?

Anyway, here we are now, IBM Real-time Compression offering customers up to 80% storage efficiency without performance degradation and totally transparent to all applications, servers, networks, storage and downstream storage processes.  (Sound familiar?)

As for the new Storwize V7000 – I am actually pretty excited to see what IBM will be doing with the new product.  The product is focused at the largest segment of the market, the mid-market, and is a serious ‘out of the shoot’ solution for customers who care about ease of storage management and making the most of their storage.

IBM may not do a lot of outbound product marketing but they do quite a bit of outbound marketing around how CIOs look at the world they have to manage.  Being outside of IBM and hearing messages around Smarter Planet, I never really thought much about it.  Being inside and now talking to CIOs about what Smarter Planet means to them (the digital explosion making it possible for them to be more competitive but having to manage all that storage makes sense).  Then at a storage layer, talking about Storage Efficiency also is how customers are thinking about storage.  All I know is, call it what you want, IBM Real-time Compression is THE most effective technology for allowing customers to do more with their storage footprint than any other storage provider so take your “storage guarantees” and well, go redo your math – you can’t beat real-time compression when it comes to storage efficiency.  (We could call it, “…chateaubriand in a NASA approved, space saving powdered package!” – thanks Steve)

Video created by MediaBoss Studios

(BTW: In case you didn't get it Storwize is now IBM Real-time Compression)!

Storage is elastic.  How do I know you ask?  Yesterday I visited a customer who is using the Storwize product to do Real-time Compression on their primary storage.  The customer is Allianz and has been using the product for over a year.  They see 75% compression on their users home directory data.  To give you an idea, Allianz is an insurance company and generates TONS of spreadsheets, 14TB worth of spreadsheets (okay, not all 14TB is spreadsheets but you get the picture).

Prior to Allianz purchasing the Storwize technology, Allianz didn’t have great data management practices.  Users store data in their home directories and there is really no discipline around deleting or cleaning up files so data just grows.  Additionally, storage isn’t really budgeted for.  Overall IT is but at a storage level, they just purchase some when the need some.

Again, prior to the Storwize technology, Allianz had their primary storage and a backup to tape at their local site.  They then replicated the data to their remote site and also performed a backup to tape.

Allianz has an overall IT mission to reduce spend by 10% per year.  The thing to think about is that this 10% could come from a lot of places including data management.

Once the Storwize technology was installed the first things they saw were:

• 75% capacity optimization
• Better data management capabilities through Storwize reporting
• The ability to keep more data on line and available for faster recoveries
• No change in any of their existing storage processes

Storwize allowed Allianz to compress their primary storage 75% and then leverage the snapshot feature on their NTAP to create compressed snapshots seamlessly for quick recovery.  Additionally, Allianz now does local backups to disk instead of tape reducing the amount of time it takes to do local backups and providing them with on line backup data for much faster recoveries which is much easier to manage.  Allianz then replicates 75% less data on a daily basis to their remote site which saves on replication costs, then at the remote site they are able to perform the same snapshots and backup to disk that they do at the local site in a very timely manner providing them with a highly efficient DR site.  Then they create one backup to tape to archive their data.  It is important to note that the Storwize technology fit into their environment with no re-architecture to any of their applications, servers, networks or storage nor did any of their IT processes change when it came to backups and recoveries - they worked seamlessly with the compressed data as if it were non-compressed data.

The point of the piece well is two fold.  One, customers are using the Storwize technology to successfully manage their storage growth and drive down costs.  By performing more snapshots and backups to disk they are able to perform faster recoveries which save them time, make them more competitive by being able to respond faster and they haven’t had to purchase storage for two years.

The second point is that disk is elastic.  The more space you give users, it is proven that they will just use the space.  Now instead of one local copy and a backup to tape, Allianz uses the extra disk space to store more data that allows them to be much more effective at business continuity.  Give users the space, and they will find a way to use it.

This was indeed true when data deduplication became popular.  Every vendor I spoke with five years ago that sold disk for backups claimed they would sell less disk for backups.  The reality was because it saved time in the backup process customers did more backups to disk.  Because it reduced the cost of disk and brought it in line with tape – customers stored data on disk vs. tape.  Because it could replicate data very efficiently customers bought more disk for a second site to keep data in a DR site.  If a customer would have bough 10TB of disk for disk based backups before, they still bought 10TB and kept more backup data on-line for a greater variation of recovery point objectives.  All viable business cases.  Now if a customer isn’t doing data deduplication – well they are one pink slip away from a deduplicated recovery from disk.

Customers use the disk to their advantage which goes to show that if a customer has a budget of $1M they will spend the $1M – how it gets divided up may be different.  For example in the Storwize case, instead of buying $1M of disk, customers now buy $700K of disk and $300K of Storwize and use that disk 5 times more efficiently.

As for Allianz, Daniel Gill said “Storwize was a no brainer.”  Not only does it help him save his company money, it makes them a much better IT shop by having more data readily available when their users need it.

Between time off with the family this summer and all the work required to get done between 'signing' a deal to be acquired and 'closing' a deal to get acquired, the blog has been a bit slow.  But I am here now to tell you it is official.  Storwize is now Storwize, an IBM company.

As for myself, I am looking forward to the work of integrating the Storwize Technology into the IBM Storage portfolio.  The Storwize group will live under the STG organization under Brian Truskowski.  There is a new ground swell taking head at IBM these days all around storage efficiency.  To get a better understanding, please have a look at my new colleague, Tony Pearson's blog discussing storage efficiency.  My job will be now to evangelize how IT now needs to take a look at all of the available storage "services" (clones, snapshots, thin provisioning, replication, compression, deduplication, etc...) can help to create an overall storage solution that allows them to reduce their over all $/TB on not only capital expense, but also on operational expense.

Lets face it, data growth isn't slowing down and there is never a one size fits all solution for storage.  The great part about being a part of IBM now is that we have all the tools to pick from to architect a data storage solution, end to end, that allows customers to reduce their overall $/TB for both primary as well as secondary storage and make that storage much more efficient and work for the end user.

This is going to be an exciting time.  I am also anxious to continue the Storage Alchemist blog.  EMC, under the guise of Polly Pearson and Chuck Hollis taught me that social media is great, but social media done right, in a collaborative and thoughtful way can drive influence.  I join some of the best bloggers around from IBM.  (I have added Tony's "Inside System Storage" - It is a great read.)

The basics that we know about today with regard to the acquisition is that this is good for Storwize employees as they get to continue working on a great technology that they have matured over the last 5 years and will continue to mature.  This acquisition is great for IBM and Storwize customers as for IBM it adds the ONLY real-time compression technology to their storage portfolio driving storage efficiency.  And for Storwize customers, you now have the backing of a global 2000 company that is committed to continued development of an already great product as well as now have the resources to go after new features and functionality that often comes at the hands of limited budgets in a stratup.

For IBM resellers and partners, hold on to your socks because now you have one of the most competitive technologies in the market that will allow you to 1) provide a great service to your customer through efficient storage and not just sell customers more disk drives (I have talked to over 600 customers in the last 2 years, customers don't want more disk drives, they want smarter ones) 2) a competitive, value added solution that allows you to be much more competitive in selling your clients better solutions that aren't wrapped around infrastructure, but around business solutions that allow your customers to be more successful and happier and you to make more margins.

Seems like a win for all.

Over the course of the next few weeks we will be learning more as a collective team on what the roadmap looks like for IBM Storwize but we are all excited.  Stay tuned.

This blog will be taking a turn to more on storage technology can turned into IT Gold!