[Gluster-users] volume start: gv01: failed: Quorum not met. Volume operation not allowed.
Alex K
rightkicktech at gmail.com
Wed Apr 11 15:54:16 UTC 2018
On Wed, Apr 11, 2018 at 4:35 AM, TomK <tomkcpr at mdevsys.com> wrote:
> On 4/9/2018 2:45 AM, Alex K wrote:
> Hey Alex,
>
> With two nodes, the setup works but both sides go down when one node is
> missing. Still I set the below two params to none and that solved my issue:
>
> cluster.quorum-type: none
> cluster.server-quorum-type: none
>
> yes this disables quorum so as to avoid the issue. Glad that this helped.
Bare in in mind though that it is easier to face split-brain issues with
quorum is disabled, that's why 3 nodes at least are recommended. Just to
note that I have also a 2 node cluster which is running without issues for
long time.
> Thank you for that.
>
> Cheers,
> Tom
>
> Hi,
>>
>> You need 3 nodes at least to have quorum enabled. In 2 node setup you
>> need to disable quorum so as to be able to still use the volume when one of
>> the nodes go down.
>>
>> On Mon, Apr 9, 2018, 09:02 TomK <tomkcpr at mdevsys.com <mailto:
>> tomkcpr at mdevsys.com>> wrote:
>>
>> Hey All,
>>
>> In a two node glusterfs setup, with one node down, can't use the
>> second
>> node to mount the volume. I understand this is expected behaviour?
>> Anyway to allow the secondary node to function then replicate what
>> changed to the first (primary) when it's back online? Or should I
>> just
>> go for a third node to allow for this?
>>
>> Also, how safe is it to set the following to none?
>>
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>>
>>
>> [root at nfs01 /]# gluster volume start gv01
>> volume start: gv01: failed: Quorum not met. Volume operation not
>> allowed.
>> [root at nfs01 /]#
>>
>>
>> [root at nfs01 /]# gluster volume status
>> Status of volume: gv01
>> Gluster process TCP Port RDMA Port
>> Online Pid
>> ------------------------------------------------------------
>> ------------------
>> Brick nfs01:/bricks/0/gv01 N/A N/A N
>> N/A
>> Self-heal Daemon on localhost N/A N/A Y
>> 25561
>>
>> Task Status of Volume gv01
>> ------------------------------------------------------------
>> ------------------
>> There are no active volume tasks
>>
>> [root at nfs01 /]#
>>
>>
>> [root at nfs01 /]# gluster volume info
>>
>> Volume Name: gv01
>> Type: Replicate
>> Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: nfs01:/bricks/0/gv01
>> Brick2: nfs02:/bricks/0/gv01
>> Options Reconfigured:
>> transport.address-family: inet
>> nfs.disable: on
>> performance.client-io-threads: off
>> nfs.trusted-sync: on
>> performance.cache-size: 1GB
>> performance.io-thread-count: 16
>> performance.write-behind-window-size: 8MB
>> performance.readdir-ahead: on
>> client.event-threads: 8
>> server.event-threads: 8
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>> [root at nfs01 /]#
>>
>>
>>
>>
>> ==> n.log <==
>> [2018-04-09 05:08:13.704156] I [MSGID: 100030]
>> [glusterfsd.c:2556:main]
>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version
>> 3.13.2 (args: /usr/sbin/glusterfs --process-name fuse
>> --volfile-server=nfs01 --volfile-id=/gv01 /n)
>> [2018-04-09 05:08:13.711255] W [MSGID: 101002]
>> [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family'
>> is
>> deprecated, preferred is 'transport.address-family', continuing with
>> correction
>> [2018-04-09 05:08:13.728297] W [socket.c:3216:socket_connect]
>> 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Protocol not
>> available"
>> [2018-04-09 05:08:13.729025] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread
>> with index 1
>> [2018-04-09 05:08:13.737757] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread
>> with index 2
>> [2018-04-09 05:08:13.738114] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread
>> with index 3
>> [2018-04-09 05:08:13.738203] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread
>> with index 4
>> [2018-04-09 05:08:13.738324] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread
>> with index 5
>> [2018-04-09 05:08:13.738330] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread
>> with index 6
>> [2018-04-09 05:08:13.738655] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread
>> with index 7
>> [2018-04-09 05:08:13.738742] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread
>> with index 8
>> [2018-04-09 05:08:13.739460] W [MSGID: 101174]
>> [graph.c:363:_log_if_unknown_option] 0-gv01-readdir-ahead: option
>> 'parallel-readdir' is not recognized
>> [2018-04-09 05:08:13.739787] I [MSGID: 114020] [client.c:2360:notify]
>> 0-gv01-client-0: parent translators are ready, attempting connect on
>> transport
>> [2018-04-09 05:08:13.747040] W [socket.c:3216:socket_connect]
>> 0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not
>> available"
>> [2018-04-09 05:08:13.747372] I [MSGID: 114020] [client.c:2360:notify]
>> 0-gv01-client-1: parent translators are ready, attempting connect on
>> transport
>> [2018-04-09 05:08:13.747883] E [MSGID: 114058]
>> [client-handshake.c:1571:client_query_portmap_cbk] 0-gv01-client-0:
>> failed to get the port number for remote subvolume. Please run
>> 'gluster
>> volume status' on server to see if brick process is running.
>> [2018-04-09 05:08:13.748026] I [MSGID: 114018]
>> [client.c:2285:client_rpc_notify] 0-gv01-client-0: disconnected from
>> gv01-client-0. Client process will keep trying to connect to glusterd
>> until brick's port is available
>> [2018-04-09 05:08:13.748070] W [MSGID: 108001]
>> [afr-common.c:5391:afr_notify] 0-gv01-replicate-0: Client-quorum is
>> not met
>> [2018-04-09 05:08:13.754493] W [socket.c:3216:socket_connect]
>> 0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not
>> available"
>> Final graph:
>> +-----------------------------------------------------------
>> -------------------+
>> 1: volume gv01-client-0
>> 2: type protocol/client
>> 3: option ping-timeout 42
>> 4: option remote-host nfs01
>> 5: option remote-subvolume /bricks/0/gv01
>> 6: option transport-type socket
>> 7: option transport.address-family inet
>> 8: option username 916ccf06-dc1d-467f-bc3d-f00a7449618f
>> 9: option password a44739e0-9587-411f-8e6a-9a6a4e46156c
>> 10: option event-threads 8
>> 11: option transport.tcp-user-timeout 0
>> 12: option transport.socket.keepalive-time 20
>> 13: option transport.socket.keepalive-interval 2
>> 14: option transport.socket.keepalive-count 9
>> 15: option send-gids true
>> 16: end-volume
>> 17:
>> 18: volume gv01-client-1
>> 19: type protocol/client
>> 20: option ping-timeout 42
>> 21: option remote-host nfs02
>> 22: option remote-subvolume /bricks/0/gv01
>> 23: option transport-type socket
>> 24: option transport.address-family inet
>> 25: option username 916ccf06-dc1d-467f-bc3d-f00a7449618f
>> 26: option password a44739e0-9587-411f-8e6a-9a6a4e46156c
>> 27: option event-threads 8
>> 28: option transport.tcp-user-timeout 0
>> 29: option transport.socket.keepalive-time 20
>> 30: option transport.socket.keepalive-interval 2
>> 31: option transport.socket.keepalive-count 9
>> 32: option send-gids true
>> 33: end-volume
>> 34:
>> 35: volume gv01-replicate-0
>> 36: type cluster/replicate
>> 37: option afr-pending-xattr gv01-client-0,gv01-client-1
>> 38: option quorum-type auto
>> 39: option use-compound-fops off
>> 40: subvolumes gv01-client-0 gv01-client-1
>> 41: end-volume
>> 42:
>> 43: volume gv01-dht
>> 44: type cluster/distribute
>> 45: option lock-migration off
>> 46: subvolumes gv01-replicate-0
>> 47: end-volume
>> 48:
>> 49: volume gv01-write-behind
>> 50: type performance/write-behind
>> 51: option cache-size 8MB
>> 52: subvolumes gv01-dht
>> 53: end-volume
>> 54:
>> 55: volume gv01-read-ahead
>> 56: type performance/read-ahead
>> 57: subvolumes gv01-write-behind
>> 58: end-volume
>> 59:
>> 60: volume gv01-readdir-ahead
>> 61: type performance/readdir-ahead
>> 62: option parallel-readdir off
>> 63: option rda-request-size 131072
>> 64: option rda-cache-limit 10MB
>> 65: subvolumes gv01-read-ahead
>> 66: end-volume
>> 67:
>> 68: volume gv01-io-cache
>> 69: type performance/io-cache
>> 70: option cache-size 1GB
>> 71: subvolumes gv01-readdir-ahead
>> 72: end-volume
>> 73:
>> 74: volume gv01-quick-read
>> 75: type performance/quick-read
>> 76: option cache-size 1GB
>> 77: subvolumes gv01-io-cache
>> 78: end-volume
>> 79:
>> 80: volume gv01-open-behind
>> 81: type performance/open-behind
>> 82: subvolumes gv01-quick-read
>> 83: end-volume
>> 84:
>> 85: volume gv01-md-cache
>> 86: type performance/md-cache
>> 87: subvolumes gv01-open-behind
>> 88: end-volume
>> 89:
>> 90: volume gv01
>> 91: type debug/io-stats
>> 92: option log-level INFO
>> 93: option latency-measurement off
>> 94: option count-fop-hits off
>> 95: subvolumes gv01-md-cache
>> 96: end-volume
>> 97:
>> 98: volume meta-autoload
>> 99: type meta
>> 100: subvolumes gv01
>> 101: end-volume
>> 102:
>> +-----------------------------------------------------------
>> -------------------+
>> [2018-04-09 05:08:13.922631] E [socket.c:2374:socket_connect_finish]
>> 0-gv01-client-1: connection to 192.168.0.119:24007
>> <http://192.168.0.119:24007> failed (No route to
>>
>> host); disconnecting socket
>> [2018-04-09 05:08:13.922690] E [MSGID: 108006]
>> [afr-common.c:5164:__afr_handle_child_down_event] 0-gv01-replicate-0:
>> All subvolumes are down. Going offline until atleast one of them comes
>> back up.
>> [2018-04-09 05:08:13.926201] I [fuse-bridge.c:4205:fuse_init]
>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24
>> kernel 7.22
>> [2018-04-09 05:08:13.926245] I [fuse-bridge.c:4835:fuse_graph_sync]
>> 0-fuse: switched to graph 0
>> [2018-04-09 05:08:13.926518] I [MSGID: 108006]
>> [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes
>> up
>> [2018-04-09 05:08:13.926671] E [MSGID: 101046]
>> [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null
>> [2018-04-09 05:08:13.926762] E [fuse-bridge.c:4271:fuse_first_lookup]
>> 0-fuse: first lookup on root failed (Transport endpoint is not
>> connected)
>> [2018-04-09 05:08:13.927207] I [MSGID: 108006]
>> [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes
>> up
>> [2018-04-09 05:08:13.927262] E [MSGID: 101046]
>> [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null
>> [2018-04-09 05:08:13.927301] W
>> [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
>> 00000000-0000-0000-0000-000000000001: failed to resolve (Transport
>> endpoint is not connected)
>> [2018-04-09 05:08:13.927339] E [fuse-bridge.c:900:fuse_getatt
>> r_resume]
>> 0-glusterfs-fuse: 2: GETATTR 1 (00000000-0000-0000-0000-000000000001)
>> resolution failed
>> [2018-04-09 05:08:13.931497] I [MSGID: 108006]
>> [afr-common.c:5444:afr_local_init] 0-gv01-replicate-0: no subvolumes
>> up
>> [2018-04-09 05:08:13.931558] E [MSGID: 101046]
>> [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null
>> [2018-04-09 05:08:13.931599] W
>> [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
>> 00000000-0000-0000-0000-000000000001: failed to resolve (Transport
>> endpoint is not connected)
>> [2018-04-09 05:08:13.931623] E [fuse-bridge.c:900:fuse_getatt
>> r_resume]
>> 0-glusterfs-fuse: 3: GETATTR 1 (00000000-0000-0000-0000-000000000001)
>> resolution failed
>> [2018-04-09 05:08:13.937258] I [fuse-bridge.c:5093:fuse_thread_proc]
>> 0-fuse: initating unmount of /n
>> [2018-04-09 05:08:13.938043] W [glusterfsd.c:1393:cleanup_and_exit]
>> (-->/lib64/libpthread.so.0(+0x7e25) [0x7fb80b05ae25]
>> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560b52471675]
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560b5247149b] ) 0-:
>> received signum (15), shutting down
>> [2018-04-09 05:08:13.938086] I [fuse-bridge.c:5855:fini] 0-fuse:
>> Unmounting '/n'.
>> [2018-04-09 05:08:13.938106] I [fuse-bridge.c:5860:fini] 0-fuse:
>> Closing
>> fuse connection to '/n'.
>>
>> ==> glusterd.log <==
>> [2018-04-09 05:08:15.118078] W [socket.c:3216:socket_connect]
>> 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not
>> available"
>>
>> ==> glustershd.log <==
>> [2018-04-09 05:08:15.282192] W [socket.c:3216:socket_connect]
>> 0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not
>> available"
>> [2018-04-09 05:08:15.289508] W [socket.c:3216:socket_connect]
>> 0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not
>> available"
>>
>>
>>
>>
>>
>>
>>
>> --
>> Cheers,
>> Tom K.
>> ------------------------------------------------------------
>> -------------------------
>>
>> Living on earth is expensive, but it includes a free trip around the
>> sun.
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>
> --
> Cheers,
> Tom K.
> ------------------------------------------------------------
> -------------------------
>
> Living on earth is expensive, but it includes a free trip around the sun.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180411/4e48c8fa/attachment.html>
More information about the Gluster-users
mailing list