5A-4
An alternative method for reliably managing large files.
○河井 裕（高エネルギー加速器研究機構），Hasan Adil（University of Liverpool, Liverpool），佐々木節（高エネルギー加速器研究機構）
The volume of digital data and the size of an individual file are increasing due to the introduction of high-resolution images, high-definition audio-visual files etc. The reliable storage of such large files essentially becomes problematic with traditional methods as replication which would require large storage units to hold multiple copies of the file. In addition, managing the file storage requires the validation of the fixity (or checksum) values.
In this note we describe a method of managing large files in the integrated Rule Oriented Data management System (iRODS) by splitting them into smaller units in an traceable manner and managing the smaller units. Each unit contains its own metadata that describes its original location as well as its checksum value. We also describe the tools developed to demonstrate the method that allow the file to be split before ingestion into iRODS and assembled after extraction from iRODS. We also make use of the Resource Namespace Service (RNS) to store metadata information and allow the large file to be discovered in Grid systems.
We believe this approach results in more reliability with fewer replicas than in the case of large-file replication as different sub-file units can be stored on different storage systems reducing the risk due to hardware failures.