Some interesting data came out of this panel.  There were probably over 150 people in the audience.  It was a well-attended session.  Also, Dave is VERY good about asking the audience questions.  Let me start by making sure we all know where everyone sits at the “storage efficiency table” that was on the panel.

• Larry Freeman is from NetApp – they claim, and I believe them, that they have 10 storage efficiency technologies that are embedded into WAFL
• Craig Nunes main focus on the panel was ‘zero reclamation’ to optimize storage
• I have a Real-time Compression drum I am beating
• Jarred Floyed focuses on data deduplication

Here are some questions and answers Dave got when speaking to the audience:

One last comment too that Larry Freeman stated was that NetApp has 10 storage optimization / efficiency technologies embedded into WAFL.  The interesting thing is that NetApp gets a report from 150,000 systems that "report in" over the weekend and they have collected statistics that users only use 3.2 out of these 10 efficiency technologies on average.

When the question was posed "why" is it that you have technology, that is “free” (more on that later) that can help you better optimize your storage, why they don’t turn it on?  To which the end user response was that they didn't want any change in the infrastructure that could require a change in their processes.  (This means that transparency is another thing that an appliance has to ensure it has.)

I also want to publicly commend Larry for a good deal of his comments and his honesty.  See Larry and I are technologists and sometimes we see the marketing arms of our companies sometimes stretch the truth to get what they want in the market by way of some FUD.  My example is this.  The RtC technology does provide high compression, for the life of the file without performance degradation.  Well, as you can see, this is what customers want give the response from the audience.  However, our friends at NetApp will say “yes, but our compression is free” (like that is supposed to be better).  Anyway, Larry addressed this in the same way I do.  He said that “you don’t get anything for free and there are tradeoffs, specifically in this case around performance”.  Now, I am not saying this to say “see, I told you so” I am saying it because we, as technologists, want to give the user the best answer.  I also admitted that an embedded strategy for technology is the right way to go, IF you can accomplish what the end users’ needs are, performance in this case.

Look, there is a reason why startups exist and why appliances exist.  Storwize didn’t invent compression; they invented a real-time platform that allowed compression to happen in real time.  We do “compression off load”.  We take the work off the array and put it on the appliance – how could that not be faster.  And yes, one day it will be embedded and there will be some other new great optimization technology that will start as an appliance and slowly find its way into the array.  It is the evolution of technology.

At the end of the day, here is what I have to say.  Every end user buys their storage for a reason.  It may be performance; it may be capacity who knows.  What I ask each user to do is to consider their BUSINESS needs for their storage and apply the RIGHT storage optimization technologies for their given environments.  There is not a “one size fits all” approach (this is why NetApp customers don’t choose to turn on all 10 optimization / efficiency technologies) but there are a handful that can help you get the most out of your storage.

“Storage Efficiency” has become a big topic over the past 12 months.  There are a number of new technologies that have come out in the last few years that are helping to deal with storage growth.  We all know that data is the root of the decisions that drive business today.  The more data you have, hopefully, the better decisions you can make to drive your business to success.  The question is, “what is the value (and hence the cost) of the infrastructure to create that success?”  What we do know is that the ability to put more data in a highly efficient footprint can give your company a competitive edge.  There are five technologies that can help an IT organization create an efficient storage infrastructure.  These are:

1)      Tiering

2)      Virtualization

3)      Thin Provisioning

4)      Compression

5)      Deduplication

It is also important to point out that there are some semantics when talking about storage efficiency, specifically between efficiency and optimization technologies.  I think it is useful to attempt to define these as they lead us to picking the right solutions for what we are trying to accomplish.  For the purpose of this post, efficiency will relate to making existing capacity more useful and optimization will mean making more capacity out of existing capacity.

Using these definitions, technologies such as Tiering, Virtualization and Thin Provisioning are efficiency technologies.  These technologies help to utilize the existing capacity that you have.

Tiering is technology that is used on about 10% of your data or less.  It is used to move data that requires higher performance to flash storage.  Good tiering technology analyzes data access patterns and moves the most active data to the highest performing disk.  It doesn’t really change the amount of physical capacity that is required; it just changes what type of capacity is required and allows IT to make sure data is operating as fast and efficiently as possible.

Virtualization technology allows IT to make sure disk utilization is used as efficiently as possible.   Until recently storage utilization rates were around 50%.  By leveraging virtualization technology, IT can group pools of storage so they don’t need to purchase capacity needlessly.  Virtualization can be used on 50% to 60% of your storage but it doesn’t change your physical capacity infrastructure requirements and at most allows users to take advantage of 20% to 40% of their capacity that they once didn’t access.

Similar to Virtualization technology, thin provisioning technology also can be used on 50% to 60% of your capacity however, thin provisioning technology gives IT about 10% to 40% of their capacity back.  Thin Provisioning helps IT manage their existing capacity and their utilization by being able to make capacity available to users much easier again however it doesn’t change the amount of physical storage infrastructure required.

Optimization technologies help IT to better manage their physical storage footprint.  Optimization technologies optimize existing infrastructure by allowing users to put more capacity in the physical same space.  The two technologies that are currently used today are data deduplication and real-time compression.

Optimization technologies are a bit tricky.  There is a balance that is required between optimization and performance and availability.  At the end of the day, IT chooses the storage it buys with two very important characteristics in mind, performance and availability.  Optimization technologies can not affect these characteristics.  It is for this reason that data deduplication really isn’t ready for “prime time” on primary, active storage.  Data deduplication creates too much of a performance impact on primary, active data.  Today, data deduplication could be used on about 10% to 15% of the primary, less active capacity that is in the data center and only provides about 30% to 50% overall optimization.  In other words deduplication technology can impact the physical infrastructure by as much as 10%, meaning IT may not need to buy as much physical capacity.

Real-time compression, on the other hand, has one of the most dramatic affects on primary storage capacity.  Real-time compression can be used on as much as 85% of the storage footprint and can compress data between 50% and 80%.  That said Real-time compression could have IT purchase as much as 70% less overall storage capacity.  Real-time compression also does not affect the main characteristics for which users buy storage (performance and availability).  IT could have as much as 70% less footprint but keep the same amount of data or more on-line.  Additionally, IT can now purchase storage opportunistically without having to have such a dramatic impact on their infrastructure, process or budgets.  This allows companies to keep more capacity on line and available to help companies do more analytics on more capacity and become more competitive.

When deciding which storage efficiency technology will have a more effective impact on your overall environment and budget, start with optimization technologies and start to get the data growth under control.  Adding value to the line of business that can drive revenue with more data will make you a hero and your business more successful.

I’m sure by now most of you all have read the fury lately over Tom Georgens’ comments about the future of storage tiering.  A number of folks who have ‘tiering’ technology reacted with disdain (see a list on Storagerap).  Some wondered how a storage visionary like Tom could turn his back on technology that helps people save money in storage.  Some even suggested that this is just marketing to overcome deficiency in the NetApp product line.  However, one applauded Tom for understanding how the real world deploys storage.  All good points, but I have my own theory on storage teiring...

I want to come right out and say I think that storage tiering is an incredibly smart concept.  (Now that that is off the table…) I would also say that much like the prediction that tape is ‘dead’ (I guess Data Domain didn’t get that memo), storage tiering, while it can’t be dead, because in reality, it never actually was, nor do I think it will be for a very long time.  Let’s look at the facts:

First, HSM never really went anywhere.  There is not mass adoption of HSM technology.  Second, tiering is not a technology issue.  Humans are lazy.  What do I mean?  HSM / Tiering or whatever you want to call it depends on policy.  IT can’t get any two groups in a company to decide on anything other than storage is too expensive.  When I speak to well respected people in IT the ‘real world’ (my dad), they tell me it is too difficult to get organizations to agree on when data can be archived in order to save money (and that is what this is all about really).  Finally, IT processes get in the way of a good tiering strategy.  Getting data to go one way is easy – move data to cheaper and cheaper tiers of storage until it vanishes.  Try getting it back.  That takes a lot of management tools and integration and costs just as much as doing nothing.

Remember back 8 or 10 years ago when blogs didn’t exist, and magazines did?  On the back page of one of those storage trade rags, I recall that Steve Duplessie picked Tom as the ‘Smartest Storage-Guy of 200x.’  I can’t remember which year, and Google isn’t helping, but that isn’t the point.  Tom is too smart to say something as frivolous as ‘tiering is dead’ by mistake.  He is also not a marketing guy.  So I’m placing my bets that his point is, lets help users utilize storage the way they like to consume it, simply.

The most basic value proposition for tiering is to save money inside the domain of the storage array. Tiering moves data to lower cost disk technology according to a pre-defined policy.  If your policy reflected the last time some data was accessed, tiering software would put your most active data on your highest tier of storage, perhaps SSD, and your ‘stale’ data moves to SATA.  For that luxury, you get:

a)      To fight with all the organizations within the company to decide on a policy as to when it is actually okay to move the data

b)      To spend money on a vendor’s tiering software, and pay maintenance fees, and learn how to use new software.

c)       Hope that the application doesn’t throw you a curve and want the SATA data quickly, because then you need to hurry and move it back to SSD, which would be inefficient and could be prone to error (at least historically it has).

So I think what Tom is challenging you to think about is, are you spending that money on tiering software wisely?  Vendors will tell you that it pays for itself, but does it really?  Despite the efforts of all tiered solutions to be truly autonomic, the reality is that they can’t replace a person’s decision making process, and if you could get all of your data to tier the way every organization would want then tiering would be a disruption to your process.  Additionally, I haven’t heard of a vendor offering a heterogeneous tiering solution, and not many customers buy all their storage from one vendor (as much as EMC would like this) so in the end, there really isn’t one good product available to do storage tiering so you would need many.  If this is the case, then you need people to manage all the software and tiering policies.  I thought we said this was supposed to save us money?

The hidden OPEX associated with figuring out how tiering works from each vendor in your environment will ultimately make you take pause before you deploy.  Maybe that is too much complexity to deal with for the benefit you get.

Part of the reason this discussion reared it’s ugly head has more to do with marketing than anything else.  EMC launched flash drives last year and told customers that “Capacity pricing is no longer about $/GB but $/GB/IO.”  (Of course, if you can’t sell on the rules of the game, change the rules.)  The problem is, no matter what the rules are, budgets are finite.  Selling customers on SSD (higher margin drives) meant that if users were going to buy these drives, they could only afford to put the data requiring the highest performance there so they would need to move data to a lower tier.  EMC said, “right, so we can also sell you FAST to help you with the tiering”.  The problem is, as Mark points out, it is 1.0, doesn’t work well yet and besides, for all the reasons we outlined above with regard to human nature, it really just isn’t going to take off (though I am sure the marketing group at EMC will ‘show’ otherwise).

On the other hand HDDs and Flash keep getting cheaper, so you might convince yourself that you are just fine riding that disk cost curve and working on other pressing matters, rather than deploying new tiering software.  If you take pause, maybe everyone else will as well, which means that maybe today's hype on tiering will never will be deployed widely across the industry.  Is it possible that this is what Tom was thinking?

To me, it boils down to something I’ve said many times:  new technology is easy to introduce into the data center, but new process is not.  Tiering runs the risk of disrupting process, which means buying behavior will be slow.  Plus, maybe there are other ways to reduce cost rather than using tiering.

For example, as Duplessie points out in his blog Random Thoughts for a Friday, he points out that “primary storage data reduction is going to be an in vogue conversation by the end of this year”.  So if Real-time, random access compression to primary storage can give IT what they want, significantly cheaper storage, no performance impact, maintain high availability, and be agnostic to any storage (heterogeneous) why wouldn’t they do that versus try to figure out storage tiering?  Primary storage compression takes the notion of storage tiering as a requirement and pushes it out 5 years and who knows, by then, it may work and be automated.

Now can't we all just get along and have No More Tiers?