[Gluster-users] Why does this setup not survive a node crash?

Mohit Anchlia mohitanchlia at gmail.com
Thu Mar 10 21:12:39 UTC 2011


My guess is that is waiting for tcp socket timeout to occur. It could
be implementation problem in glusterfs and probably need to be fixed I
am guessing.

Can you instead try to do this, Leave the node up but issue kill -9 on
all ps -eaf|grep gluster processes?

On Thu, Mar 10, 2011 at 12:55 PM, Burnash, James <jburnash at knight.com> wrote:
> Perhaps someone will see immediately, given the data below, why this configuration will not survive a crash of one node - it appears that any node crashed out of this set will cause gluster native clients to hang until the node comes back.
>
> Given (2) initial storage servers (CentOS 5.5, Gluster 3.1.1):
>
> Starting out by creating a Replicated-Distributed pair with this command:
> gluster volume create test-pfs-ro1 replica 2 jc1letgfs5:/export/read-only/g01 jc1letgfs6:/export/read-only/g01 jc1letgfs5:/export/read-only/g02 jc1letgfs6:/export/read-only/g02
>
> Which ran fine (thought I did not attempt to crash 1 of the pair)
>
> And then adding (2) more servers, identically configured, with this command:
> gluster volume add-brick test-pfs-ro1 jc1letgfs7:/export/read-only/g01 jc1letgfs8:/export/read-only/g01 jc1letgfs7:/export/read-only/g02 jc1letgfs8:/export/read-only/g02
> Add Brick successful
>
> root at jc1letgfs5:~# gluster volume info
>
> Volume Name: test-pfs-ro1
> Type: Distributed-Replicate
> Status: Started
> Number of Bricks: 4 x 2 = 8
> Transport-type: tcp
> Bricks:
> Brick1: jc1letgfs5:/export/read-only/g01
> Brick2: jc1letgfs6:/export/read-only/g01
> Brick3: jc1letgfs5:/export/read-only/g02
> Brick4: jc1letgfs6:/export/read-only/g02
> Brick5: jc1letgfs7:/export/read-only/g01
> Brick6: jc1letgfs8:/export/read-only/g01
> Brick7: jc1letgfs7:/export/read-only/g02
> Brick8: jc1letgfs8:/export/read-only/g02
>
> And this volfile info out of the log file /var/log/glusterfs/etc-glusterd-mount-test-pfs-ro1.log:
>
> [2011-03-10 14:38:26.310807] W [dict.c:1204:data_to_str] dict: @data=(nil)
> Given volfile:
> +------------------------------------------------------------------------------+
>  1: volume test-pfs-ro1-client-0
>  2:     type protocol/client
>  3:     option remote-host jc1letgfs5
>  4:     option remote-subvolume /export/read-only/g01
>  5:     option transport-type tcp
>  6: end-volume
>  7:
>  8: volume test-pfs-ro1-client-1
>  9:     type protocol/client
>  10:     option remote-host jc1letgfs6
>  11:     option remote-subvolume /export/read-only/g01
>  12:     option transport-type tcp
>  13: end-volume
>  14:
>  15: volume test-pfs-ro1-client-2
>  16:     type protocol/client
>  17:     option remote-host jc1letgfs5
>  18:     option remote-subvolume /export/read-only/g02
>  19:     option transport-type tcp
>  20: end-volume
>  21:
>  22: volume test-pfs-ro1-client-3
>  23:     type protocol/client
>  24:     option remote-host jc1letgfs6
>  25:     option remote-subvolume /export/read-only/g02
>  26:     option transport-type tcp
>  27: end-volume
>  28:
>  29: volume test-pfs-ro1-client-4
>  30:     type protocol/client
>  31:     option remote-host jc1letgfs7
>  32:     option remote-subvolume /export/read-only/g01
>  33:     option transport-type tcp
>  34: end-volume
>  35:
> 36: volume test-pfs-ro1-client-5
>  37:     type protocol/client
>  38:     option remote-host jc1letgfs8
>  39:     option remote-subvolume /export/read-only/g01
>  40:     option transport-type tcp
>  41: end-volume
>  42:
>  43: volume test-pfs-ro1-client-6
>  44:     type protocol/client
>  45:     option remote-host jc1letgfs7
>  46:     option remote-subvolume /export/read-only/g02
>  47:     option transport-type tcp
>  48: end-volume
>  49:
>  50: volume test-pfs-ro1-client-7
>  51:     type protocol/client
>  52:     option remote-host jc1letgfs8
>  53:     option remote-subvolume /export/read-only/g02
>  54:     option transport-type tcp
>  55: end-volume
>  56:
>  57: volume test-pfs-ro1-replicate-0
>  58:     type cluster/replicate
>  59:     subvolumes test-pfs-ro1-client-0 test-pfs-ro1-client-1
>  60: end-volume
>  61:
>  62: volume test-pfs-ro1-replicate-1
>  63:     type cluster/replicate
>  64:     subvolumes test-pfs-ro1-client-2 test-pfs-ro1-client-3
>  65: end-volume
>  66:
>  67: volume test-pfs-ro1-replicate-2
>  68:     type cluster/replicate
>  69:     subvolumes test-pfs-ro1-client-4 test-pfs-ro1-client-5
>  70: end-volume
>  71:
>  72: volume test-pfs-ro1-replicate-3
>  73:     type cluster/replicate
>  74:     subvolumes test-pfs-ro1-client-6 test-pfs-ro1-client-7
>  75: end-volume
>  76:
>  77: volume test-pfs-ro1-dht
>  78:     type cluster/distribute
>  79:     subvolumes test-pfs-ro1-replicate-0 test-pfs-ro1-replicate-1 test-pfs-ro1-replicate-2 test-pfs-ro1-replicate-3
>  80: end-volume
>  81:
>  82: volume test-pfs-ro1-write-behind
>  83:     type performance/write-behind
>  84:     subvolumes test-pfs-ro1-dht
>  85: end-volume
>  86:
>  87: volume test-pfs-ro1-read-ahead
>  88:     type performance/read-ahead
>  89:     subvolumes test-pfs-ro1-write-behind
>  90: end-volume
>  91:
>  92: volume test-pfs-ro1-io-cache
>  93:     type performance/io-cache
>  94:     subvolumes test-pfs-ro1-read-ahead
>  95: end-volume
>  96:
>  97: volume test-pfs-ro1-quick-read
>  98:     type performance/quick-read
>  99:     subvolumes test-pfs-ro1-io-cache
> 100: end-volume
> 101:
> 102: volume test-pfs-ro1-stat-prefetch
> 103:     type performance/stat-prefetch
> 104:     subvolumes test-pfs-ro1-quick-read
> 105: end-volume
> 106:
> 107: volume test-pfs-ro1
> 108:     type debug/io-stats
> 109:     subvolumes test-pfs-ro1-stat-prefetch
> 110: end-volume
>
> Any input would be greatly appreciated. I'm working beyond my deadline already, and I'm guessing that I'm not seeing the forest for the trees here.
>
> James Burnash, Unix Engineering
>
>
> DISCLAIMER:
> This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission.
> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>



More information about the Gluster-users mailing list