George Crump posed an interesting question when he asked if primary storage deduplication will kill archive and backup.  It is a great question, and one we should explore.  If you don’t want to read my ramblings, my short answer is no. 

There is a lot more to archive and backup than simply storing a lot of data, something deduplication has proven it can do well.  Backed up and archived data needs to be cataloged, indexes and managed through its life or retention.  That’s one of the reasons we don’t use tar and dump commands much these days.  Snapshots can remove much of the recovery burden from alternate storage devices.  I have seen customers recover almost all single file restores from snapshots.  But they never served as a replacement for backups.   As George said, we sleep better at night when copies of our data are on different systems.  There are lots of reasons for that.  We all worry about a bad firmware load.  If you have all your data on one array (or replicated to an identical one) a bad firmware release could wipe you out.  And of course there are physical failures. No matter how well designed a system is, something external can happen.  In the years I was in the field, I heard some unbelievable external failure stories where an non-IT event started the failure.   (Maybe I should start collecting them).

This leads me to conclude that proper architecture of a data storage environment includes dissimilar storage devices.  Your backup and DR copies need to be independent from production data, to prevent a cascading failure getting every copy.  For archive, the first copy could be on the primary storage platform, but the redundant copies (and all good archive systems maintain a minimum of 2 copies of the data) need the same.  It could be as easy as Spectra nTier disk and Spectra T-Series tape, or it could be more complex.  What it won't be is a single disk array for primary storage, archive and data protection.

The high importance of data protection is top of mind these days – specifically in light of some high profile cases of data loss in the UK. News of some potentially impending legislation this side of the pond has again drawn attention to the issue of how companies look after customer data.

With LTO-5 drives released in our T-Series libraries it’s another trusted innovation milestone for us at Spectra. We keep our innovation conveyor cranking along here. As Spectra’s customers trust us to deliver valued tape innovation, millions trust LTO tape (> 3.3M drives sold, > 140M cartridges sold) with their data assets, now and for the long term. 

http://www.spectralogic.com/LTO-5

I am amazed at how easy it is for people to consume all of their available storage space—and how quickly we can do it.  Just looking at my own storage situation, it is remarkable how much “stuff” I have to store; a snow blower, coolers, lots of car parts, bike parts, camping gear and who knows what else.   My old storage shed was recently hit by a tree, so I had to replace it this month.  I purchased the biggest one allowed by city zoning regulations, and already…. I realize I should have gone bigger. 

The Active Archive Alliance formation was just announced last week. I wanted to take the opportunity to tell you how the concept originated, evolved and solidified, and why Spectra Logic is participating as a founding member.

Spectra Logic is excited to announce our second product of the year award for the Spectra T-Finity tape library. Brian Grainger of Spectra attended the second annual Data Intensive Computing Environment (DICE) Data Intensive Impact Awards event yesterday and accepted the award.

Cheers to our development team. We appreciate the recognition for our continued innovation and dedication to the high performance computing market.

 

Part One or Two

One of the interesting things I have discovered since I have been talking with so many HPC customers is that the term “backup” is seldom used.  You might ask if they aren’t doing traditional backups, then why would we, a backup solutions provider,  want to talk to them. Well, first you need to fully understand the difference between backup and archive.  Archive is a word you will hear more often in the HPC and M&E environments, especially if there is data in excess of the petabyte range and large files that aren’t accessed frequently but need to be kept indefinitely. 

In this blog, which is the first of a two part series, I will provide some fundamental information that can help you differentiate backup from archive.  In the subsequent blog, part two, we will peel the covers back on the process that is different from backup and archive and similar to the traditional HSM (Hierarchical Storage Management). This information will prove to be valuable for those HPC or other data intensive customers who may claim that they don’t do backups.  Stay tuned for more on this subject later.

The differences between backup and archive:

Backup: simply refers to the creation of a copy of data and storing it somewhere for restoration in the event the original version of the data was compromised in some way.  We evangelize the concept of backups because we know, and most customers realize, that data can accidentally be deleted, corruption could occur, data loss, or even worse, a natural disaster could wipe out the entire data center.

Backup is simply safeguarding or protecting the data that is being used by duplicating that data.  This is usually done in a rotating cycle or through schedules including: daily incremental which are kept for seven days, a weekly full kept for a month, a monthly full kept for a year and a yearly full kept for seven years.  Although this process has proven effective and most of the backup applications on the market today are ideal for doing this, problems occur when you start having multiple copies of the same data consuming a lot more hardware than necessary, not to mention the associated costs of running and managing that hardware. 

With backup – think business continuity

One of the key differences when comparing backup strategies to  archiving, is the difficulty of singling out select files for long term retention.  Everything in the backup gets lumped into the large full backup at the end of the year or seven years and called an “archive”.  It may in fact be called an archive but a recovery would function more like a backup recovery, which could be very costly and time consuming.  Backup strategies are more for business continuity purposes and not necessary for long term archiving.

With archive – think long-term retention

Archive: The main difference between an archive and a backup is that an archive refers to a single collection of records or data that is designated for long-term retention.  When the data is moved from the production environment to the archive environment it is tagged or indexed by metadata that assists in quickly locating that particular file or chunk of data through a search mechanism.  This process and the sophisticated software that performs it make locating a single file much more efficient than it would be in a traditional backup.  An archive is generally found in a common file system structure and the determination of where the file is located is a function of file system.  The file system may have several different storage devices that the archived data is stored on based on a number of attributes such as size, type, last accessed, etc.  This system could be a combination of expensive disk, such as fiber channel, less expensive disk, such as SATA or SAS and tape.  The key is how the data is “structured.”  In most cases, the data may never be accessed again, but it is necessary to keep it for historical purposes, regulatory compliance or unplanned event.  The goal with creating an archive is to keep it separate from the backup rotation cycle.  It is recommended that a separate copy of the archived data be made and kept in a separate location so there are at least two copies of the final archive.

Many environments will include both backup and archive.  Through the use of sophisticated software features that are available today, customers can establish policies that determine type, size, age, last accessed, remaining disk space and other characteristics of stored data that can automate the process of deciding whether to keep the data in the backup cycle or move it to the archive pool.

These two functions can be performed within a single library in separate partitions.  The software can then provide notification of what tapes need to be exported based on the function that was performed on those tapes, backup or archive.  I have seen numbers as high as 80% indicating how much data is duplicated within a storage infrastructure because the differences between backup and archive aren’t fully understood.  At the end of the day, knowing the difference and the benefits of backup and archive technologies, when to use them and how to balance the the two functions in an environment can drastically reduce the amount of redundancy, complexity and storage operating costs.

In Part Two of this discussion, we will look at how archives that contain production data, no matter how old or infrequently accessed, can still be retrieved online using high density and high speed tape systems and secondary disk systems.  Stay tuned for my next post which will look at enduring access to data. 

Want to talk more? I’ll be in Dearborn Michigan at the IDC HPC User Forum and DICE Alliance 2010 events next week. Contact me at jimm@spectralogic.com.

Dear Ms. Meade:
My data crunching company generates 2-3 TB of data per customer, and I need to store that somehow. However, I don’t have room for a tape library. The only thing I can think to do is put the data on some hard drives using Linux-based RAID software, then put the disk in a safety deposit box. Do you have any other suggestions?”
Sincerely,
Short on Space

Dear Ms. Space:
You have money and room for terabytes of disk storage, which you will squirrel away in a pretty large safety deposit box, but not a couple of dollars and rack units for a small library? Hmmm.

In pondering a polite answer to this question, Ms. Meade called to mind something similar posted on Slashdot, and is heartened that several intelligent points were discussed in that context. Being one to always encourage others in the path of light, Ms. Meade will summarize these intelligent comments and add to them.

The Short Version, by the way, in case you are averse to reading: Buy an LTO-4 tape drive and LTO-4 tapes. Forget the disk.

The Long Version: As fond as Ms. Meade is of disk, especially Spectra nTier disk, Ms. Meade understands that disk’s greatest asset is the speed at which it retrieves data—NOT its use for secure offline data storage.

Tape is cheaper than disk, even the disk to which you are likely referring. The Tape Equation: you can buy an LTO-4 tape drive for around $1400 (and likely for less), and at $40 per tape, store 800 GB of data; with these and a little compression, you are two tapes away from serious, long-term storage.  Assuming that you have more than a handful of customers annually, this pays for itself pretty rapidly, compared to purchasing cheap (and risky) jbod.

At $100/TB per hard drive and twenty customers each with 2 TB of compressed data, annually the company must shell out $4,000 per year. If, instead, the firm purchases a tape drive and LTO media, your costs are under that in just the first year-- about $2,000 for tape, and another $1,400 for the drive. You’ve paid for the drive in one year. After that, you save $60/TB. (That translates to thousands of dollars annually.)

You may want to consider a tape library, which truly are not space- or budget-hogs. Libraries such as the 4U Spectra T50e may be worth the space and time simply in convenience. This depends a great deal on your business volumes and staffing, and Ms. Meade acknowledges constraints due to current recessionary times. However, to emphasize the point: a relatively lightweight investment such as the purchase of a small library can automate data protection—and most companies that deal in data understand that their business also mandates data-caretaking.

For those unenlightened few who say that LTO tape is not a wise choice because eventually new technologies replace older ones, please consider that LTO has been around this past decade and shows no sign of going away. No migration will be necessary for years to come, given that current generations of LTO tape technology read data on tape that is two generations back, and write one generation back. With new generations about every 3 years, and giving the mobility of today’s clientele, the lifespan of at least 6-10 years is likely sufficient for your business requirements.

Frankly, the issue truly cries out for tape, and Ms. Meade is glad to add her voice to those doing the crying out.
 

Dear Ms. Meade,
I am charged with architecting a backup system without any single points of failure. Obviously, tape is SO failure-prone that I am not including it at all. How do you think I should configure such a system?
Sincerely,
Tape is Doomed

Dear Doomed,
You are doomed if you rely solely on disk for your data backup.  A possible interpretation of your question may be “How much disk does it take to replace tape?”  The answer is “unobtainium”—that is, you can’t replace tape using disk.

Further, the very concept of single point of failure is terribly funny in a terribly dark way. Failure is inevitable, unless you plan to address human imperfection? What about acts of natural and man-made disaster that may affect the national power grid? Switch problems? What about loose screws, including any screwed-up (or self-perceived screwed over) employee?

Instead, consider asking a question that does have an answer—“How can I reliably protect data?” The answer is “disk and tape.”

Ms. Meade is a major fan of disk with RAID 6, offered in Spectra’s nTier disk. With RAID 6, up to three disks can fail without affecting data integrity. Go disk and go RAID. However, disk (even with RAID 6) can’t be considered failure-proof because it has its own Achilles’ heel (aka single point of failure): the RAID controller. You can have all the data you want on all the spinning disk you want—but if the controller fails, the brains are gone, and the bits and bytes you’ve carefully protected are toast. Whither goest the RAID controller, so goeth the data. Dead controller= permanently decomposed data. So disk alone, even with the marvels of RAID, is not enough to provide true disaster recovery and continuity of operations.

Further, please note that your information about tape as failure-prone is completely wrong. Tape is, it turns out, incredibly reliable.  With tape’s reliability increase of 700% over the last decade, multiple layers of ECC protection, and smart Spectra libraries tracking media and drive health, tape meets and beats disk in terms of reliability. If you’re worried about a single point of failure,  make sure you get two tape drives. Consider the T950 and T-Finity libraries’ global spare feature—which is an installed drive that can be directed to take over in case of a drive failure.

Ms. Meade admits that she is curious about the pointy-haired boss who directed you to create the no single point of failure unobtanium backup environment….