[Gluster-devel] Harddisk economy alternatives

[Gordan Bobic]

On 09/11/2011 19:09, Magnus Näslund wrote:
> On 11/09/2011 06:51 PM, Gordan Bobic wrote:
>>
>> My main concern with such data volumes would be the error rates of
>> modern disks. If your FS doesn't have automatic checking and block level
>> checksums, you will suffer data corruption, silent or otherwise. Quality
>> of modern disks is pretty appaling these days. One of my experiences is
>> here:
>> http://www.altechnative.net/?p=120
>> but it is by no means the only one.
>>
>
> Interesting read, and I agree that raid data corruption and hard disk
> untrustworthiness issues being a huge problem. To combat this we're
> thinking of using a crude health checking utility that would use
> checksum files, on top of whatever we end up using (glusterfs or
> otherwise). These scripts would be specific to our application, and file
> based.
>
> In glusterfs I believe that it would be possible to do the checksum
> checking locally on the nodes, since the underlying filesystem is
> accessible?

In some cases. If you are using striping, then that gets potentially 
tricky. If you are using straight mirroring, then yes, you could easily 
just check the underlying files. However, that would mean manually 
correcting things. ZFS will check this for you every time a file is 
accessed or scrubbed and auto-correct the corrupted blocks, so no 
corrupted data ever gets served.

>> Currently the only FS that meets all of my reliability criteria is ZFS
>> (and the linux port works quite well now), and it has saved me from data
>> corruption, silent and otherwise, a number of times by now, in cases
>> where normal RAID wouldn't have helped.
>>
>
> We're using OpenSolaris+ZFS today in production, if glusterfs works well
> on OpenSolaris that might very well be what we end up with.

I have no idea whether glfs works well on Solaris. It's worth trying, 
but given how much effort Emanuel has put into porting it to (Net)BSD, 
it may well not "just work", but perhaps one of the developers will be 
able to clarify whether glfs is tested/supported/expected to work on 
Solaris.

> We're a linux-shop, but we settled for OpenSolaris on ZFS alone.
> Are you running glusterfs on Solaris or/and Linux in production?

On Linux.

You may be interested to have a look here:

http://groups.google.com/a/zfsonlinux.org/group/zfs-discuss/browse_thread/thread/4d88218d6c8f67f0/78a7b633dd66157a?hl=en&lnk=gst&q=glusterfs#78a7b633dd66157a

>> Anyway, to summarize:
>> 1) With large volumes of data, you need something other than the disk's
>> sector checksums to keep your data correct, i.e. a checksum checking FS.
>> If you don't, expect to see silent data corruption sooner or later.
>
> The silent corruption case can be mitigated an application specific way
> for us, as described above. Having that automatically using ZFS is
> definately interesting in several ways. Does glusterfs have (or plan to
> have) a scrubbing-like functionality that checks the data?

I'd be interested to hear the developers' thoughts on this, but I think 
this would be extremely expensive. Doing "ls -laR" to check/auto-heal 
all files is already very time consuming on large file systems. 
Calculating md5s of all files as well would be orders of magnitude more 
expensive.

Gordan