There was a great post a couple of weeks ago, with Tom Coughlin as a contributing editor, on Forbes’ news site about the floods that hit Thailand and how it will affect the disk drive market.  The great thing about the article is it truly highlights that necessity is the mother of invention.  What do I mean by that?  Over the past few “storage efficiency” has been a big topic with vendors.  Helping customers “do more with less”, especially in these stringent economic times, is key to the vitality of a number of businesses.  Technologies such as storage virtualization and thin provisioning have helped customers to slow their storage spend and get better utilization out of their existing storage.  Once customers have moved their utilization rates from 35% to 65% or 70%, time comes when new storage needs to be acquired to keep up with the growth of data.  The issue comes when there are no more disk drives to be acquired.  Due to the floods in Thailand, analysts predict that the storage industry could be 50 to 60 million units shy of the demand this quarter.  This does two things:

1)      Drives the price of disk higher, at a time when the expectation is to spend less for disk

2)      Has IT getting more creative on how they use and deploy their storage

It is the later that I want to focus on as paying more for disk is not necessarily the best option.  It is important to note that data grows for one reason, business does not stop, it needs to keep going and it is what is driving the demand on the data.

In the Forbes piece Tom talks about “a surge in new technologies because of this disk shortage” but he doesn’t cover some of the most innovative technologies that are available to help customers.  I would agree with Tom that we “could” see a surge in SSD but that would be short lived do to both supply and cost as well as a surge in tape, but these aren’t really “new technologies”.

New technologies for primary storage optimization can and will play a key role in helping IT be more productive with their existing capacity.  New technology such as Real-time Compression can help customers get back up to 80% of their existing storage capacity without losing any of their current capabilities or changing any of their data management processes.  The technology seamlessly integrates into your storage environment and compresses your data 50% to 80% (depending upon data type).  It also fits into IT’s existing data management practices without having to change anything.  No change is required to any of the applications, snapshots stay the same, replication stays the same even backup works without having to change anything in the environment.  And while some vendors may say “you can’t deduplicate compressed data”, you actually can deduplicate data written with Real-time Compression.

The Real-time Compression technology is truly a “new” technology that can expect to surge in this environment.  IT can deploy this technology and expect:

1)      Up to 80% compression on their primary storage

1. This means they can defer adding new capacity until the HDD market comes back and disk prices stabilize

2)      See up to 80% optimization in each of their downstream processes that use disk

1. Meaning up to 80% less capacity for snapshots
2. Meaning up to 80% less capacity for replication
3. Meaning up to 80% less capacity for backups

(In each of these cases, each process uses disk so there is a tremendous savings by just compressing the primary copy of the data)

3)      The technology will be transparent to their existing infrastructure

In addition, Real-time Compression can cut your cost per TB by a factor of your compression ratio (50% compression is a 2:1 cost reduction in your $/TB cost).  It is also the case, if you are looking to SSD for performance, you can now afford to spend some money on SSD or more money on SSD given the new cost model.

Now, the “new” technology does need to be efficient and fit into a customer’s existing infrastructure seamlessly or it isn’t really useful.  Asking IT to change their processes can be just as costly as purchasing new capacity in the long run.  I mention this because in a related story, NetApp is also fearful about what the HDD shortage will do for their business.  I find this ironic.  On a recent panel I was on at SNW with Larry Freeman of NetApp, he told the audience that NetApp filers have these “new” technologies “built in” to their WAFL file system, in fact they have 10 “storage efficiency” features built in to WAFL.  He went on to say that on a weekly basis they get reports from a number of systems in the field that “report in” on how customers are using their systems.  On average customers use only 3 of the 10 features.  When we polled the audience to ask them why, they said that while the feature may help them save space, they impact other areas of their operation.  Maybe it impacts system performance, maybe it impacts backup so they can’t use the feature.

The moral of the story is that I do believe that new technologies are going to “surge” (as Tom states) in his piece, because IT will need other alternatives to the shortage of disk drive that are available and the higher prices.  In addition, this will force IT to look at their environment to identify how to be more efficient with their storage environment as stuff like the flood could come up again and affect the supply and demand of HDD.  But the right technologies that not only help with storage capacity as well as data growth needs to be the answer to the challenge.  The best technologies fit into IT’s existing infrastructure and makes it more efficient overall.

I’m sure by now most of you all have read the fury lately over Tom Georgens’ comments about the future of storage tiering.  A number of folks who have ‘tiering’ technology reacted with disdain (see a list on Storagerap).  Some wondered how a storage visionary like Tom could turn his back on technology that helps people save money in storage.  Some even suggested that this is just marketing to overcome deficiency in the NetApp product line.  However, one applauded Tom for understanding how the real world deploys storage.  All good points, but I have my own theory on storage teiring...

I want to come right out and say I think that storage tiering is an incredibly smart concept.  (Now that that is off the table…) I would also say that much like the prediction that tape is ‘dead’ (I guess Data Domain didn’t get that memo), storage tiering, while it can’t be dead, because in reality, it never actually was, nor do I think it will be for a very long time.  Let’s look at the facts:

First, HSM never really went anywhere.  There is not mass adoption of HSM technology.  Second, tiering is not a technology issue.  Humans are lazy.  What do I mean?  HSM / Tiering or whatever you want to call it depends on policy.  IT can’t get any two groups in a company to decide on anything other than storage is too expensive.  When I speak to well respected people in IT the ‘real world’ (my dad), they tell me it is too difficult to get organizations to agree on when data can be archived in order to save money (and that is what this is all about really).  Finally, IT processes get in the way of a good tiering strategy.  Getting data to go one way is easy – move data to cheaper and cheaper tiers of storage until it vanishes.  Try getting it back.  That takes a lot of management tools and integration and costs just as much as doing nothing.

Remember back 8 or 10 years ago when blogs didn’t exist, and magazines did?  On the back page of one of those storage trade rags, I recall that Steve Duplessie picked Tom as the ‘Smartest Storage-Guy of 200x.’  I can’t remember which year, and Google isn’t helping, but that isn’t the point.  Tom is too smart to say something as frivolous as ‘tiering is dead’ by mistake.  He is also not a marketing guy.  So I’m placing my bets that his point is, lets help users utilize storage the way they like to consume it, simply.

The most basic value proposition for tiering is to save money inside the domain of the storage array. Tiering moves data to lower cost disk technology according to a pre-defined policy.  If your policy reflected the last time some data was accessed, tiering software would put your most active data on your highest tier of storage, perhaps SSD, and your ‘stale’ data moves to SATA.  For that luxury, you get:

a)      To fight with all the organizations within the company to decide on a policy as to when it is actually okay to move the data

b)      To spend money on a vendor’s tiering software, and pay maintenance fees, and learn how to use new software.

c)       Hope that the application doesn’t throw you a curve and want the SATA data quickly, because then you need to hurry and move it back to SSD, which would be inefficient and could be prone to error (at least historically it has).

So I think what Tom is challenging you to think about is, are you spending that money on tiering software wisely?  Vendors will tell you that it pays for itself, but does it really?  Despite the efforts of all tiered solutions to be truly autonomic, the reality is that they can’t replace a person’s decision making process, and if you could get all of your data to tier the way every organization would want then tiering would be a disruption to your process.  Additionally, I haven’t heard of a vendor offering a heterogeneous tiering solution, and not many customers buy all their storage from one vendor (as much as EMC would like this) so in the end, there really isn’t one good product available to do storage tiering so you would need many.  If this is the case, then you need people to manage all the software and tiering policies.  I thought we said this was supposed to save us money?

The hidden OPEX associated with figuring out how tiering works from each vendor in your environment will ultimately make you take pause before you deploy.  Maybe that is too much complexity to deal with for the benefit you get.

Part of the reason this discussion reared it’s ugly head has more to do with marketing than anything else.  EMC launched flash drives last year and told customers that “Capacity pricing is no longer about $/GB but $/GB/IO.”  (Of course, if you can’t sell on the rules of the game, change the rules.)  The problem is, no matter what the rules are, budgets are finite.  Selling customers on SSD (higher margin drives) meant that if users were going to buy these drives, they could only afford to put the data requiring the highest performance there so they would need to move data to a lower tier.  EMC said, “right, so we can also sell you FAST to help you with the tiering”.  The problem is, as Mark points out, it is 1.0, doesn’t work well yet and besides, for all the reasons we outlined above with regard to human nature, it really just isn’t going to take off (though I am sure the marketing group at EMC will ‘show’ otherwise).

On the other hand HDDs and Flash keep getting cheaper, so you might convince yourself that you are just fine riding that disk cost curve and working on other pressing matters, rather than deploying new tiering software.  If you take pause, maybe everyone else will as well, which means that maybe today's hype on tiering will never will be deployed widely across the industry.  Is it possible that this is what Tom was thinking?

To me, it boils down to something I’ve said many times:  new technology is easy to introduce into the data center, but new process is not.  Tiering runs the risk of disrupting process, which means buying behavior will be slow.  Plus, maybe there are other ways to reduce cost rather than using tiering.

For example, as Duplessie points out in his blog Random Thoughts for a Friday, he points out that “primary storage data reduction is going to be an in vogue conversation by the end of this year”.  So if Real-time, random access compression to primary storage can give IT what they want, significantly cheaper storage, no performance impact, maintain high availability, and be agnostic to any storage (heterogeneous) why wouldn’t they do that versus try to figure out storage tiering?  Primary storage compression takes the notion of storage tiering as a requirement and pushes it out 5 years and who knows, by then, it may work and be automated.

Now can't we all just get along and have No More Tiers?

When it comes to cost, the average cost per terabyte for SSD is still 8x the cost of standard FC disk.  It is for this reason that companies such as EMC are pulling the proverbial wool over the customers eyes by telling customers that the formula they need to consider when it comes to storage costs is no longer about dollars per gigabyte but dollars per gigabyte per disk I/O.  And while this is an important and useful formula, the overall cost is still critical end users.  By performing compression before the data gets to the SSD you can, at a minimum, reduce the cost by 2x and in some cases 10x (depending upon your compression ratio) and if compression is being performed in real-time, the storage performance can even increase (imagine that, increasing your SSD performance and cutting the cost in half!)

The next issue with SSD is the issue with write failures over time.  Weather this is a myth or an issue that is blown out of proportion it still comes up with customers.  The two characteristics that are important to customers when buying storage are performance (noted above as a function of dollars per I/O) and availability.  If there is the slightest chance that SSD will loose data, I don't care how fast it is, I can't use it.  Now, if real-time compression sits in front of SSD drives, and does the compression before the data is stored on disk then the disk I/O is cut by 50% or more which extends the life of the disk.  Less I/O means less less reads and writes from the SSD which in turn means greater long term value of the SSD.

I keep saying 'real-time' compression here.  It is important to keep in mind that if you don't do real-time compression then you do the basic industry compression which will negatively impact the MTBF of an SSD.  The reason: because with 'traditional compression' it is post process meaning it compressed the data once the data is stored.  This means that not only is there a full write for every file, there is a full read for every file and then a subsequent, albeit smaller write for the compressed file.  This is a lot more I/O that would be seen in non-compressed environments.

The bottom line, if you want to save money and extend the life of your solid state disk drives, you need to be doing real-time compression.