[Gluster-users] Why does this setup not survive a node crash?
Burnash, James
jburnash at knight.com
Fri Mar 18 14:15:53 UTC 2011
I remembered this from an earlier experience using a VMWare client - it's in the GlusterFS documentation - I can't find the link at the moment ...
James Burnash, Unix Engineering
-----Original Message-----
From: Mohit Anchlia [mailto:mohitanchlia at gmail.com]
Sent: Wednesday, March 16, 2011 2:26 PM
To: Burnash, James; gluster-users at gluster.org
Subject: Re: [Gluster-users] Why does this setup not survive a node crash?
Thanks! Also, how did you find that you need to disable
--disable-direct-io-mode?
On Wed, Mar 16, 2011 at 11:19 AM, Burnash, James <jburnash at knight.com> wrote:
> Yes - the "long" response time went to about 18 seconds from over a minute (at least).
>
> Whether or not this is generally a good idea is something I'll let the devs and the list respond to.
>
>
> James Burnash, Unix Engineering
>
> -----Original Message-----
> From: Mohit Anchlia [mailto:mohitanchlia at gmail.com]
> Sent: Wednesday, March 16, 2011 2:17 PM
> To: Burnash, James; gluster-users at gluster.org
> Subject: Re: [Gluster-users] Why does this setup not survive a node crash?
>
> Thanks for sharing! Did it make it any faster after changing
> network.ping-timeout to 10 secs?
>
> On Wed, Mar 16, 2011 at 11:07 AM, Burnash, James <jburnash at knight.com> wrote:
>> So - answering myself with the (apparent) solution. The configuration IS correct as shown - the problems were elsewhere.
>>
>> Primary cause for this seems to be performing the gluster native client mount on a virtual machine WITHOUT using the " -O --disable-direct-io-mode" parameter.
>>
>> So I was mounting like this:
>>
>> mount -t glusterfs jc1letgfs5:/test-pfs-ro1 /test-pfs2
>>
>> When I should have been doing this:
>>
>> mount -t glusterfs -O --disable-direct-io-mode jc1letgfs5:/test-pfs-ro1 /test-pfs2
>>
>> Secondly, I changed the volume parameter "network.ping-timeout" from its default of 43 to 10 seconds, in order to get faster recovery from a downed storage node:
>>
>> gluster volume set pfs-rw1 network.ping-timeout 10
>>
>> This configuration now survives the loss of either node of the two storage server mirrors. There is a noticeable delay before commands on the mount point complete the first time a command is issued after one of the nodes have gone done - but then they return at the same speed as when all nodes were present.
>>
>> Thanks especially to all who helped, and Anush who helped me troubleshoot it from a different angle.
>>
>> James Burnash, Unix Engineering
>>
>> -----Original Message-----
>> From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James
>> Sent: Friday, March 11, 2011 11:31 AM
>> To: gluster-users at gluster.org
>> Subject: Re: [Gluster-users] Why does this setup not survive a node crash?
>>
>> Could anyone else please take a peek at this an sanity check my configuration. I'm quite frankly at a loss and tremendously under the gun ...
>>
>> Thanks in advance to any kind souls.
>>
>> James Burnash, Unix Engineering
>>
>> -----Original Message-----
>> From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James
>> Sent: Thursday, March 10, 2011 3:55 PM
>> To: gluster-users at gluster.org
>> Subject: [Gluster-users] Why does this setup not survive a node crash?
>>
>> Perhaps someone will see immediately, given the data below, why this configuration will not survive a crash of one node - it appears that any node crashed out of this set will cause gluster native clients to hang until the node comes back.
>>
>> Given (2) initial storage servers (CentOS 5.5, Gluster 3.1.1):
>>
>> Starting out by creating a Replicated-Distributed pair with this command:
>> gluster volume create test-pfs-ro1 replica 2 jc1letgfs5:/export/read-only/g01 jc1letgfs6:/export/read-only/g01 jc1letgfs5:/export/read-only/g02 jc1letgfs6:/export/read-only/g02
>>
>> Which ran fine (thought I did not attempt to crash 1 of the pair)
>>
>> And then adding (2) more servers, identically configured, with this command:
>> gluster volume add-brick test-pfs-ro1 jc1letgfs7:/export/read-only/g01 jc1letgfs8:/export/read-only/g01 jc1letgfs7:/export/read-only/g02 jc1letgfs8:/export/read-only/g02
>> Add Brick successful
>>
>> root at jc1letgfs5:~# gluster volume info
>>
>> Volume Name: test-pfs-ro1
>> Type: Distributed-Replicate
>> Status: Started
>> Number of Bricks: 4 x 2 = 8
>> Transport-type: tcp
>> Bricks:
>> Brick1: jc1letgfs5:/export/read-only/g01
>> Brick2: jc1letgfs6:/export/read-only/g01
>> Brick3: jc1letgfs5:/export/read-only/g02
>> Brick4: jc1letgfs6:/export/read-only/g02
>> Brick5: jc1letgfs7:/export/read-only/g01
>> Brick6: jc1letgfs8:/export/read-only/g01
>> Brick7: jc1letgfs7:/export/read-only/g02
>> Brick8: jc1letgfs8:/export/read-only/g02
>>
>> And this volfile info out of the log file /var/log/glusterfs/etc-glusterd-mount-test-pfs-ro1.log:
>>
>> [2011-03-10 14:38:26.310807] W [dict.c:1204:data_to_str] dict: @data=(nil)
>> Given volfile:
>> +------------------------------------------------------------------------------+
>> 1: volume test-pfs-ro1-client-0
>> 2: type protocol/client
>> 3: option remote-host jc1letgfs5
>> 4: option remote-subvolume /export/read-only/g01
>> 5: option transport-type tcp
>> 6: end-volume
>> 7:
>> 8: volume test-pfs-ro1-client-1
>> 9: type protocol/client
>> 10: option remote-host jc1letgfs6
>> 11: option remote-subvolume /export/read-only/g01
>> 12: option transport-type tcp
>> 13: end-volume
>> 14:
>> 15: volume test-pfs-ro1-client-2
>> 16: type protocol/client
>> 17: option remote-host jc1letgfs5
>> 18: option remote-subvolume /export/read-only/g02
>> 19: option transport-type tcp
>> 20: end-volume
>> 21:
>> 22: volume test-pfs-ro1-client-3
>> 23: type protocol/client
>> 24: option remote-host jc1letgfs6
>> 25: option remote-subvolume /export/read-only/g02
>> 26: option transport-type tcp
>> 27: end-volume
>> 28:
>> 29: volume test-pfs-ro1-client-4
>> 30: type protocol/client
>> 31: option remote-host jc1letgfs7
>> 32: option remote-subvolume /export/read-only/g01
>> 33: option transport-type tcp
>> 34: end-volume
>> 35:
>> 36: volume test-pfs-ro1-client-5
>> 37: type protocol/client
>> 38: option remote-host jc1letgfs8
>> 39: option remote-subvolume /export/read-only/g01
>> 40: option transport-type tcp
>> 41: end-volume
>> 42:
>> 43: volume test-pfs-ro1-client-6
>> 44: type protocol/client
>> 45: option remote-host jc1letgfs7
>> 46: option remote-subvolume /export/read-only/g02
>> 47: option transport-type tcp
>> 48: end-volume
>> 49:
>> 50: volume test-pfs-ro1-client-7
>> 51: type protocol/client
>> 52: option remote-host jc1letgfs8
>> 53: option remote-subvolume /export/read-only/g02
>> 54: option transport-type tcp
>> 55: end-volume
>> 56:
>> 57: volume test-pfs-ro1-replicate-0
>> 58: type cluster/replicate
>> 59: subvolumes test-pfs-ro1-client-0 test-pfs-ro1-client-1
>> 60: end-volume
>> 61:
>> 62: volume test-pfs-ro1-replicate-1
>> 63: type cluster/replicate
>> 64: subvolumes test-pfs-ro1-client-2 test-pfs-ro1-client-3
>> 65: end-volume
>> 66:
>> 67: volume test-pfs-ro1-replicate-2
>> 68: type cluster/replicate
>> 69: subvolumes test-pfs-ro1-client-4 test-pfs-ro1-client-5
>> 70: end-volume
>> 71:
>> 72: volume test-pfs-ro1-replicate-3
>> 73: type cluster/replicate
>> 74: subvolumes test-pfs-ro1-client-6 test-pfs-ro1-client-7
>> 75: end-volume
>> 76:
>> 77: volume test-pfs-ro1-dht
>> 78: type cluster/distribute
>> 79: subvolumes test-pfs-ro1-replicate-0 test-pfs-ro1-replicate-1 test-pfs-ro1-replicate-2 test-pfs-ro1-replicate-3
>> 80: end-volume
>> 81:
>> 82: volume test-pfs-ro1-write-behind
>> 83: type performance/write-behind
>> 84: subvolumes test-pfs-ro1-dht
>> 85: end-volume
>> 86:
>> 87: volume test-pfs-ro1-read-ahead
>> 88: type performance/read-ahead
>> 89: subvolumes test-pfs-ro1-write-behind
>> 90: end-volume
>> 91:
>> 92: volume test-pfs-ro1-io-cache
>> 93: type performance/io-cache
>> 94: subvolumes test-pfs-ro1-read-ahead
>> 95: end-volume
>> 96:
>> 97: volume test-pfs-ro1-quick-read
>> 98: type performance/quick-read
>> 99: subvolumes test-pfs-ro1-io-cache
>> 100: end-volume
>> 101:
>> 102: volume test-pfs-ro1-stat-prefetch
>> 103: type performance/stat-prefetch
>> 104: subvolumes test-pfs-ro1-quick-read
>> 105: end-volume
>> 106:
>> 107: volume test-pfs-ro1
>> 108: type debug/io-stats
>> 109: subvolumes test-pfs-ro1-stat-prefetch
>> 110: end-volume
>>
>> Any input would be greatly appreciated. I'm working beyond my deadline already, and I'm guessing that I'm not seeing the forest for the trees here.
>>
>> James Burnash, Unix Engineering
>>
>>
>> DISCLAIMER:
>> This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission.
>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list