Cas-des-attaques-intelligentes , 9. Acharya, M. Franklin, and S. DOI : Avizienis, J. Laprie, and B.
|Published (Last):||14 March 2016|
|PDF File Size:||18.29 Mb|
|ePub File Size:||8.82 Mb|
|Price:||Free* [*Free Regsitration Required]|
Enterprises and consumers today face the problem of storing and managing an ever-increasing amount of data on non-volatile data storage systems such as hard disk drives. One promising direction in computer storage systems is to harness the collective storage capacity of massive commodity computers to form a large distributed data storage system.
When designing such distributed data storage system an important factor to consider is data reliability. Once data is stored a user typically does not want or cannot afford to lose any of the stored data.
Unfortunately, the data management chain is prone to failures at various links that can result in permanent data loss or a temporary unavailability of the data.
For example, any one of a number of individual components of a massive distributed data storage system may fail for a variety of reasons. Hard drive failures, computer motherboard failures, memory problems, network cable problems, loose connections such as a loose hard drive cable, memory cable, or network cable , power supply problems, and so forth can occur leaving the data inaccessible.
For distributed data storage systems to be useful in practice, proper redundancy schemes must be implemented to provide high reliability, availability and survivability. One type of redundancy scheme is replication, whereby data is replicated two, three, or more times to different computers in the system. As long as any one of the replica is accessible, the data is available. Most distributed data storage systems use replication for simplified system design and low access overhead.
One problem, however, with the replication technique is that the cost of storing a duplication of data can become prohibitively expense. Large storage cost directly translates into high cost in hardware hard drives and associated machines , as well as the cost to operate the storage system, which includes the power for the machine, cooling, and maintenance. For example, if the data is replicated three times then the associated costs of storing the data are tripled.
One way to decrease this storage cost is by using another type of redundancy scheme called erasure resilient coding ERC. Erasure resilient coding enables lossless data recovery notwithstanding loss of information during storage or transmission. At retrieval time the n data blocks are retrieved to recover the original data piece.
Currently, the main use of the ERC technique in distributed data storage systems is in the form of large peer-to-peer P2P systems. A protection group is often used in ERC to provide an added measure of protection to the data. Typically, each of the n data blocks is placed in a single protection group. One problem, however, with using the ERC technique in distributed data storage systems is that because the data is fragmented and stored in a plurality of blocks multiple protection groups cannot be created.
Another problem is that when a data block is modified each of the data blocks belonging to the same protection group must also be modified. In other words, whenever a data block is written or read then all the other data blocks in the protection group also must be modified.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The erasure resilient coding ERC distributed data storage system and method includes using ERC in a distributed data storage environment to achieve the same level of reliability as data replication with much less hardware.
The system and method use software instead of hardware to improve data reliability and survivability. More specifically, the system and method allows the formation of multiple protection groups that contain a plurality of data blocks. The multiple protection groups are formed both across and within storage nodes.
Because of the unique read and write operations based on erasure resilient coding, the reading and writing of each data block can be performed independent of other data blocks in the same protection group. The ERC distributed data storage system and method also achieves load balancing over the ERC distributed data storage system.
In particular, an original data piece is segmented into a plurality of data blocks, including original data blocks and ERC data blocks. The system includes several storage nodes that store both types of data blocks. The system and method interleaves original data blocks and ERC data blocks among the storage nodes so that the load is balanced between nodes. In some embodiments, this balancing is achieved by dispersing the data blocks such that each storage node performs approximately the same number of read and write operations.
In other embodiments, the balancing is achieved by ensuring that each storage node contains a relatively equal number of original data blocks and ERC data blocks. The ERC distributed data storage system and method reads and writes a data block independent of other data blocks with the same protection group. The unique write operation is capable of a true write operation when there is an existing data block or an append operation when there is not an existing data block. In the first case, the write operation replaces an old data block with a new data block and performs Galois field arithmetic on the new and old data blocks.
Further mathematical operations are performed, including a mathematical transform using erasure resilient coding and a second Galois field arithmetic operation. The resultant transformed data block is written to each of the storage nodes containing the old data block. In the second case, there is no old data block and the new data block is appended to either the front or back of the data after being mathematically processed as described above.
The unique read operation of the ERC distributed data storage system and method is capable of recovering a data block in a variety of ways. First, any data block that is live and fresh on its storage node is directly read out of the node. Second, if the data block is stale then a search is made for one protection group having all the live data blocks of the original data piece. Stale means that a failure has occurred or that the machine is in the process of recovering from such a failure.
If such a protection group is found, then a distributed read and ERC decoding are performed to recover the data block. Third, if such a protection group cannot be found then another type of decoding is performed to attempt to recover the data block. It should be noted that alternative embodiments are possible, and that steps and elements discussed herein may be changed, added, or eliminated, depending on the particular embodiment.
These alternative embodiments include alternative steps and alternative elements that may be used, and structural changes that may be made, without departing from the scope of the invention. Referring now to the drawings in which like reference numbers represent corresponding parts throughout:. In the following description of the erasure resilient coding ERC distributed data storage system and method reference is made to the accompanying drawings, which form a part thereof, and in which is shown by way of illustration a specific example whereby the ERC distributed data storage system and method may be practiced.
It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the claimed subject matter. Referring to FIG. Each of these storage nodes is a computing device. The storage nodes are in communication with an index server through a network As explained in detail below, the index server contains an index table not shown for tracking the location of data blocks dispersed throughout the storage nodes.
The data blocks are fragments of an original piece of data or data piece. As shown in FIG. It should be noted that FIG. The ERC distributed data storage system may be implemented on various types of processing systems, such as on a central processing unit CPU or multi-core processing systems. It should be noted that the storage node computing device may include a single processor such as a desktop or laptop computer or several processor and computers connected to each other.
The input to the ERC distributed data storage system is an original data piece box As explained in detail below, the ERC distributed data storage system processes this original data piece, breaks the original data piece into several blocks, performs ERC processing on the blocks to obtain multiple protected blocks of the original data piece box , and stores the multiple protected blocks such that even in the event of failures the original data piece can be recovered.
The ERC distributed data storage system includes software or program modules for execution on the storage node computing device In particular, the ERC distributed data storage system includes a multiple protection group module and a data block allocation module The multiple protection group module generates multiple protection groups for the original data piece The data block allocation module allocates both original data blocks and ERC data blocks among the storage nodes such the computation load is equally balanced between the storage nodes.
The ERC distributed data storage system also includes a data write module and a data read module The data write module appends a data block to storage nodes by performing mathematical modifications to the data blocks and replacing the old data block with the new data block. The data read module recovers data by determining whether a data block is alive or dead on a storage node and acting accordingly. The operation of each of these modules will be discussed in detail below.
In general, the ERC distributed data storage method processes an original data piece to ensure that the data piece can be reconstructed even in the case of hardware failures. More specifically, the ERC distributed data storage method begins by inputting an original data piece box The original data piece then is segmented into a plurality of data block box These plurality of data blocks include original data blocks and ERC data blocks.
The ERC data blocks are obtained using erasure resilient coding, which is well known in the art. The method then forms multiple protection group each having more than a single data block box These multiple protection groups add another layer of data reliability. Next, each of the plurality of data blocks can be written independently of other data blocks in the same protection group box Similarly, each of the plurality of data blocks also can be read independently of other data blocks in the same protection group box The read and write operations can be used to reconstruct the data piece on demand box Once requested, the method reconstructs the data piece and outputs a reconstructed data piece box Each of the mentioned above will now be discussed in further detail.
In particular, details of the multiple protection group module , the data block allocation module , the data write module , and the data read module will be discussed to further clarify the details of the ERC distributed data storage system and method. In general, the module generates multiple protection groups for protecting data.
The operation begins by inputting original data blocks and ERC data blocks box Some multiple protection groups are formed within storage nodes box , while some multiple protection groups are formed across storage nodes box The output of the module are the multiple protection groups box The example of FIG. Note that in the example of FIG.
It should be noted that each storage node cluster can be data centers located in different geographic locations. The idea behind forming the protection groups is that there are a plurality of data blocks that can be located on different storage nodes.
A key concept is that the erasure chunks are interleaved into each data center or storage node cluster. This alleviates the need to dedicate one data center or machine to erasure coding only.
The 3D planes of the array are labeled as A, B, C. The A-B face is shown labeled, where X i,j represents an original data block, Y i,j represents an erasure-coded data chunk within the same data center or storage node cluster, and Z i,j represents an erasure-coded data chunk across the data center or storage node cluster. It should be noted that the Z i,j 's are interleaved with each line such that there is no need to dedicate any single data center or storage node cluster for erasure coding.
Examples of protection groups for this data are shown by the dashed lines.
Optimizing Cauchy Reed-Solomon Codes for P2P Storage Cloud
Skip to Main Content. A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. Use of this web site signifies your agreement to the terms and conditions. Personal Sign In.
Erasure Resilient Transmission (ERT)
Skip to search form Skip to main content You are currently offline. Some features of the site may not work correctly. An m; n; b; r -erasure-resilient coding scheme consists of an encoding algorithm and a decoding algorithm with the following properties. The encoding algorithm produces a set of n packets each containing b bits from a message of m packets containing b bits. The decoding algorithm is able to recover the message from any set of r packets.
We apologize for the inconvenience...
US8051362B2 - Distributed data storage using erasure resilient coding - Google Patents