Storage Tiers – Take 3
I find myself in a true quandary. First, I have true admiration for my good friend and fellow blogger 3Par Farley and never feel comfortable being on the other side of the coin from him. Second, I find myself agreeing, to a degree, with Jon Toigo (who still uses crazy permalinks and considers Novell a serious storage player. What is up with that?).
I’m sure by now most of you all have read the fury lately over Tom Georgens’ comments about the future of storage tiering. A number of folks who have ‘tiering’ technology reacted with disdain (see a list on Storagerap). Some wondered how a storage visionary like Tom could turn his back on technology that helps people save money in storage. Some even suggested that this is just marketing to overcome deficiency in the NetApp product line. However, one applauded Tom for understanding how the real world deploys storage. All good points, but I have my own theory on storage teiring…
I want to come right out and say I think that storage tiering is an incredibly smart concept. (Now that that is off the table…) I would also say that much like the prediction that tape is ‘dead’ (I guess Data Domain didn’t get that memo), storage tiering, while it can’t be dead, because in reality, it never actually was, nor do I think it will be for a very long time. Let’s look at the facts:
First, HSM never really went anywhere. There is not mass adoption of HSM technology. Second, tiering is not a technology issue. Humans are lazy. What do I mean? HSM / Tiering or whatever you want to call it depends on policy. IT can’t get any two groups in a company to decide on anything other than storage is too expensive. When I speak to well respected people in IT the ‘real world’ (my dad), they tell me it is too difficult to get organizations to agree on when data can be archived in order to save money (and that is what this is all about really). Finally, IT processes get in the way of a good tiering strategy. Getting data to go one way is easy – move data to cheaper and cheaper tiers of storage until it vanishes. Try getting it back. That takes a lot of management tools and integration and costs just as much as doing nothing.
Remember back 8 or 10 years ago when blogs didn’t exist, and magazines did? On the back page of one of those storage trade rags, I recall that Steve Duplessie picked Tom as the ‘Smartest Storage-Guy of 200x.’ I can’t remember which year, and Google isn’t helping, but that isn’t the point. Tom is too smart to say something as frivolous as ‘tiering is dead’ by mistake. He is also not a marketing guy. So I’m placing my bets that his point is, lets help users utilize storage the way they like to consume it, simply.
The most basic value proposition for tiering is to save money inside the domain of the storage array. Tiering moves data to lower cost disk technology according to a pre-defined policy. If your policy reflected the last time some data was accessed, tiering software would put your most active data on your highest tier of storage, perhaps SSD, and your ‘stale’ data moves to SATA. For that luxury, you get:
a) To fight with all the organizations within the company to decide on a policy as to when it is actually okay to move the data
b) To spend money on a vendor’s tiering software, and pay maintenance fees, and learn how to use new software.
c) Hope that the application doesn’t throw you a curve and want the SATA data quickly, because then you need to hurry and move it back to SSD, which would be inefficient and could be prone to error (at least historically it has).
So I think what Tom is challenging you to think about is, are you spending that money on tiering software wisely? Vendors will tell you that it pays for itself, but does it really? Despite the efforts of all tiered solutions to be truly autonomic, the reality is that they can’t replace a person’s decision making process, and if you could get all of your data to tier the way every organization would want then tiering would be a disruption to your process. Additionally, I haven’t heard of a vendor offering a heterogeneous tiering solution, and not many customers buy all their storage from one vendor (as much as EMC would like this) so in the end, there really isn’t one good product available to do storage tiering so you would need many. If this is the case, then you need people to manage all the software and tiering policies. I thought we said this was supposed to save us money?
The hidden OPEX associated with figuring out how tiering works from each vendor in your environment will ultimately make you take pause before you deploy. Maybe that is too much complexity to deal with for the benefit you get.
Part of the reason this discussion reared it’s ugly head has more to do with marketing than anything else. EMC launched flash drives last year and told customers that “Capacity pricing is no longer about $/GB but $/GB/IO.” (Of course, if you can’t sell on the rules of the game, change the rules.) The problem is, no matter what the rules are, budgets are finite. Selling customers on SSD (higher margin drives) meant that if users were going to buy these drives, they could only afford to put the data requiring the highest performance there so they would need to move data to a lower tier. EMC said, “right, so we can also sell you FAST to help you with the tiering”. The problem is, as Mark points out, it is 1.0, doesn’t work well yet and besides, for all the reasons we outlined above with regard to human nature, it really just isn’t going to take off (though I am sure the marketing group at EMC will ‘show’ otherwise).
On the other hand HDDs and Flash keep getting cheaper, so you might convince yourself that you are just fine riding that disk cost curve and working on other pressing matters, rather than deploying new tiering software. If you take pause, maybe everyone else will as well, which means that maybe today’s hype on tiering will never will be deployed widely across the industry. Is it possible that this is what Tom was thinking?
To me, it boils down to something I’ve said many times: new technology is easy to introduce into the data center, but new process is not. Tiering runs the risk of disrupting process, which means buying behavior will be slow. Plus, maybe there are other ways to reduce cost rather than using tiering.
For example, as Duplessie points out in his blog Random Thoughts for a Friday, he points out that “primary storage data reduction is going to be an in vogue conversation by the end of this year”. So if Real-time, random access compression to primary storage can give IT what they want, significantly cheaper storage, no performance impact, maintain high availability, and be agnostic to any storage (heterogeneous) why wouldn’t they do that versus try to figure out storage tiering? Primary storage compression takes the notion of storage tiering as a requirement and pushes it out 5 years and who knows, by then, it may work and be automated.
Now can’t we all just get along and have No More Tiers? 
Hey Steve,
Storage tiering is not the same as ILM or HSM. The concept of moving data to lower cost tiers when it becomes less interesting might never work.
Tiering is about moving data to high performance tiers when additional performance is needed from storage. Tiering is about performance acceleration. It’s about getting the lowest cost for IOPS – when IOPS are needed – as well as achieving lower costs for data storage when performance demands are relaxed.
The question is: how do you know what data needs to be promoted to higher performance tiers? The answer comes from monitoring I/O levels in the array. This is a lot more direct than asking an administrator what data needs to be moved. Storage tiering can assume all data should be located on lower performance storage until activity levels demand that it be moved to higher performance storage.
Tiering does not have to have an archiving component, nor does it depend on making lifecycle policy decisions before you have the information you need to determine what those policies should be. In fact, I think data management and tiering should not be linked. Archiving and lifecycle management functions are application or content management features, not storage features.
Storage level functions should be driven to providing optimal cost and performance for production opportunities. Confusing data management with storage will probably always deliver confusing results.
I think you’re confusing ILM with tiering and HSM. Tiering could be classed as something as simple as placing data on the most appropriate tier of storage at the time of allocation. If we follow this model, then databases could go on expensive storage, File data on cheaper, less performing storage.
The evolution of this static tiering model is to allow misplaced data to be moved. Maybe the data owner didn’t advise the storage team of their requirements – or maybe they didn’t even know. After an application goes live, performance metrics show the data could have been placed elsewhere.
Now scale that up. Rather than having a few LUNs to evaluate, I’ve 1000′s. This isn’t a human solvable problem. It isn’t about lazy humans, its about the inability for humans to analyse large volumes of data.
Now, look at our data in more granular detail. The LUN level was good, but the sub-LUN level is even better, once we realise that “hot” data doesn’t fully occupy a single LUN.
It’s pretty easy to see how we’ve arrived at our current position. Placing data on the right tier of storage saves money. Automating the process allows us to realise the savings. The “future” is letting the array determine the best workload mix and basing tiering decisions on policy. This still means having data at rest on a tier of storage at some stage.
Now, to the subject of reducing the working set of active data, that’s another whole discussion.
Mark,
I would agree with you on your points, however this is not how vendors are selling ‘tiering’ today except perhaps 3Par. Remember, I was at EMC when FAST came out. At the end of the day, and it has been said multiple times, tiering is about saving money. Not many IT folks (at least the ones I talk to) are thinking about tiering as a function of I/O and in the same breath saving money. IT wants performance, they want flash, they can’t afford it so vendors tell them, use less flash and more SATA to off set the cost. This is how tireing is being sold. So my reply to this is, give the customer what they want. Cut the price of all disk in half (at a minimum) with compression and keep all the other processes the same.
It would be great if a storage system were smart enough to see data being accessed in a manner such that it needed to be on SSD and then moved it without disruption. I don’t hear many customers talking about this though and I don’t know how moving data around affects things like backup processes.
Chris,
Thanks for the reply. A lot of folks are talking about tiering, ILM, HSM. I would agree with Mark’s comment. True tiering shouldn’t be about HSM OR ILM but it is today because this is what vendors are telling their customers. It’s just a fact.
Steve
To reference your reply to Marc, have a look at this post: http://www.thestoragearchitect.com/2009/10/21/enteprise-computing-automated-tiering-why-move-the-data/
I think it partly addresses the comment in your last paragraph. Moving data between tiers isn’t hard if you’re actively reading/writing the data already.
Chris
Let’s not put the cart before the horse.
There’s no point in talking about a storage strategy until you understand what’s going on at the business level with information and the way that information is managed, consumed and put out to pasture. Most of the storage types I have met understand bits, bytes, blocks, volumes and LUNs like nobody’s business, but they know VERY little about the way information is created, shared and consumed. Which explains why many of their products have proven to be inefficient or inadequate in the context of modern information management.
I STRONGLY disagree with most of Marc’s response. Hell I don’t even know where to begin explaining my issues with it. I could spend all day writing a detailed response, but I’m not going to bother.
I will write this…
Tiering is about service levels. On that much I would think we would all agree. Understand, however, that there is a direct and inextricable relationship between information stewards (in business) and information custodians (in IT) and the bond is storage real estate. Establishing realistic, sustainable, manageable storage service levels requires collaboration and a knowledge of corporate information assets that goes well beyond monitoring I/O levels in an array.
Should any storage industry pundits wish to debate the topic, I’d love to participate along with some practitioners from the information management side of the house. Any takers?
Joe,
Thanks for the response. As far as disagreeing with Marc, well, I guess that is fine, but he hits on some important points. To your point, You are actually making my point. Tiering is a about service levels and who decides these levels? IT? No, lines of business managers do and they can’t agree on anything other than – make my storage costs go down. I spent some time with some of these practitioners this weekend, and the only thing they did agree on was storage is too expensive.
So, in the vane of sounding self serving, compress it all and ALL of this goes away. (Well at least for a few years.)
Thanks again for your comments.
I would agree with you that part of this discussion is related to service levels. Ultimately, IT should provide a service to the business. If they can deliver that service by reducing costs through tiering, then that’s great. Ultimately, service levels and service catalog should be abstracted from the technology used to deliver them. That way, the business can get on with what they do best and so can the IT teams. Whilst it’s true that this approach isn’t always followed, what is true is that tiering – not just slapping a load of read cache into your hardware – can help achieve this.
Chris
Precisely, Chris. Information stewards could care less how it’s all accomplished underneath the covers as long as the system as a whole supports their objectives
What are a few of the things that they want? They want…
* to store information in a way that makes sense to them.
* to hold onto it for as long as they require.
* to preserve and protect not just the information assets, but also any defined relationships between them.
* to find and retrieve it in a way that makes sense to them.
* to retrieve it with speed appropriate to the asset’s importance.
* for it to be usable/viewable/reusable once retrieved. (this is one of the ticking time bombs of storage)
* for the system to “watch their backs” and alert them when something is amiss, out of whack, out of compliance, or at risk.
* to automate that which can be safely automated without fear of failures that might lead to harsh penalties.
* and to do all of this affordably over the lifetime of their assets. (ahhh, the mythical holy grail)
Steve, we know how people operate in the real world once they brush problems under the rug. Out of sight, out of mind. Leave it for the next guy to clean up. Compression (at list intra-file compression) may save space today by brushing the issues under the rug for now, but eventually some other poor schmuck who inherits the mess is going to have to clean it up. And millions of compressed assets MAY make his/her job a lot more difficult and resource intensive.
I prefer to shame the companies into making change sooner rather than later by explaining that the problem is only going to get more expensive as time passes and open them up to even more risk. Sadly, even that strategy can fail. If only their customers knew what goes on behind the scenes.
There’s a reason every single one of my presentations end with a slide that says “Pay me now or pay me later”. Eventually, they all pay the piper.
Joe,
You seem to think that storage products should also be ILM products. WAKE UP! Storage and ILM are different and people should think hard about how much overlap there should be.
Chris makes the point, “Ultimately, service levels and service catalog should be abstracted from the technology used to deliver them.” I think you agreed with that, but I’m not sure.
The costs of managing storage and managing data are related, but different. The key to running a cost effective operation is understanding which ways to turn the dials for each without screwing up the other. Storage tiering is certainly not for everybody, but it can certainly give IT organizations great options for reducing the cost of storage performance – without impacting ILM goals at all.
NAS inherently has a lot more overlap between storage and ILM than SAN does because NAS manages files, directories and their metadata. It is a lot of lifting – and the way I see it – the biggest strength and weakness of NAS.
I quietly read all the comments, and I realized that I almost agree with all of them
, although Joe said: “I STRONGLY disagree with most of Marc’s response”.
I think that Marc did great academic job trying to identify the terms ILM, HSM and tiering and the differences between them.
When it comes to engineering we need to identify our new feature and emphasize our intention that this feature is a function of IOPs, and yes it can help you to save money if you use your resources wisely.
Steve said that he agree with Marc but when it comes to the “real world” sales persons introduce tiering as HSM.
I think the source of the confusion is because generally, unused data or “cold data” requires less IOPs and this is the cross-road between tiering and HSM.
Sales persons not always understand all the technical details and they are generally driven by the new buzzword in market, and maybe they do understand but they don’t really care,as far as it answer the ‘right’ questions:
Can it help the customer to decrease costs, can it helps him to better utilize his resources (or shell I ask , can it helps me to sell the product?) and the answers to all questions is YES.
I would like to quote a joke:
An engineer, a physicist, and a mathematician were on a train heading north, and had just crossed the border into Scotland.
• The engineer looked out of the window and said “Look!
Scottish sheep are black!”
• The physicist said, “No, no. Some Scottish sheep are
black.”
• The mathematician looked irritated. “There is at least
one field, containing at least one sheep, of which at
least one side is black.”
________________________________________
variants:
•The statistician : “It’s not significant. We only know there’s one black sheep”
•The computer ‘scientist’ : “Oh, no! A special case!”
Mark is the mathematician
(all in a good spirit)
To make the distinction better Marc said the wonderful sentence:
“Storage tiering can assume all data should be located on lower performance storage until activity levels demand that it be moved to higher performance storage.”
This is the theory, now to reality, do you have free high performance storage (which generally is very expensive) and you don’t use it.
And when it needs to be copied into the high performance storage, what then Marc? How do you identify that activity levels demand it to be moved to higher performance storage? just because at that specific day someone used the data intensively?
You don’t mean to monitor it at real time and copy the data at the background,do you? And what if the user will never use it again is it going to stay at the high performance storage? or copies back to the lower level?
What you are suggesting in that sentence is called cache and what you are actually saying I putting the cache on the high performance storage.
In any way that you look at it: Storage administration needs planning!
Because it is a technical discussion I would like to bring my point of view:
The administrator should locate/manage his storage according to the organization ILM paradigm.
Yes, it has nothing with Storage product, but teh storage products should give him the capability to do so.
I don’t believe that fully automated tiering will be acceptable in the market, people wants to know where the data is located and don’t like that ‘someone’ change the location behind their back (even if it is for good purposes). However, static tiering and semi automated tiering can and should be part of the view.
I will explain:
In the storage world today almost all the players talks about virtualization , user generally create Pools of the same tiering and locate applications and data accordingly.
As a rule of thumb, high transactional DB that hold the core of the buisness will be located at the high performance storage pools – this is what I called Static tiering.
BTW: even if the database taht manage teh “workers gaming zone” requiers better IOPs rate then the DB that manage the core of the buisness , we will still locate the core of the buisness on the higher we have. so it is not all about IOPs.
No one would like that the storage management tool will move data from one pool to another automatically.
Now regarding semi automated tiering:
According to policy based rules the user can define that the storage allocation for snapshots or backups for example, will be automatically taken from Pools which are defined as tier B.
That’s it:
It might be that tiering automation will be acceptable only if it will be marketing as cache level solution (remember teh L1 , L2).
Dror
re: It would be great if a storage system were smart enough to see data being accessed in a manner such that it needed to be on SSD and then moved it without disruption. I don’t hear many customers talking about this though and I don’t know how moving data around affects things like backup processes.
Check out Compellent. Been doing that for some time now and getting super high customer sat on the reality of that technology.
Clearly, Georgens comments are self-interested. He would like tiering to go away so he says it will. It’s a common marketing strategy to say something is true in an effort to make it so. Georgens knows his company is being assaulted on many fronts by companies like Compellent.
Tiering will happen because different data has different SLA requirements. Commonly accessed data will live on flash. Infrequently accessed data will live on cheap disks. Never accessed data will live on spun down disks. Further, the most successful tiering will be automatic and spun down disks will spin up on demand.
Marc,
Didn’t want you to think I forgot about you. I’ll address your response later this week, partly here for those who may be following this thread and more thoroughly on our blog.
It’s much too easy for some things to slip off of one’s radar. I had forgotten about this thread until I recently participated in a similar discussion.
In a nutshell…
Marc, I am in agreement with Chris’s point about abstraction of service levels.
What I suggest is that we collectively work toward improving communication between information management and storage applications, just as we need to improve communication between information stewards (business) and custodians (IT). Until then we’ll continue to create systems in which the left hand hasn’t a clue what the right hand is doing.