[Gluster-users] brick is down but gluster volume status says it's fine

Alastair Neil ajneil.tech at gmail.com
Tue Oct 24 17:43:55 UTC 2017


gluster version 3.10.6, replica 3 volume, daemon is present but does not
appear to be functioning

peculiar behaviour.  If I kill the glusterfs brick daemon and restart
glusterd then the brick becomes available - but one of my other volumes
bricks on the same server goes down in the same way it's like wack-a-mole.

any ideas?


[root at gluster-2 bricks]# glv status digitalcorpora

> Status of volume: digitalcorpora
> Gluster process                             TCP Port  RDMA Port  Online
> Pid
>
> ------------------------------------------------------------------------------
> Brick gluster-2:/export/brick7/digitalcorpo
> ra                                          49156     0          Y
> 125708
> Brick gluster1.vsnet.gmu.edu:/export/brick7
> /digitalcorpora                             49152     0          Y
> 12345
> Brick gluster0:/export/brick7/digitalcorpor
> a                                           49152     0          Y
> 16098
> Self-heal Daemon on localhost               N/A       N/A        Y
> 126625
> Self-heal Daemon on gluster1                N/A       N/A        Y
> 15405
> Self-heal Daemon on gluster0                N/A       N/A        Y
> 18584
>
> Task Status of Volume digitalcorpora
>
> ------------------------------------------------------------------------------
> There are no active volume tasks
>
> [root at gluster-2 bricks]# glv heal digitalcorpora info
> Brick gluster-2:/export/brick7/digitalcorpora
> Status: Transport endpoint is not connected
> Number of entries: -
>
> Brick gluster1.vsnet.gmu.edu:/export/brick7/digitalcorpora
> /.trashcan
> /DigitalCorpora/hello2.txt
> /DigitalCorpora
> Status: Connected
> Number of entries: 3
>
> Brick gluster0:/export/brick7/digitalcorpora
> /.trashcan
> /DigitalCorpora/hello2.txt
> /DigitalCorpora
> Status: Connected
> Number of entries: 3
>
> [2017-10-24 17:18:48.288505] W [glusterfsd.c:1360:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7e25) [0x7f6f83c9de25]
> -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x55a148eeb135]
> -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55a148eeaf5b] ) 0-:
> received signum (15), shutting down
> [2017-10-24 17:18:59.270384] I [MSGID: 100030] [glusterfsd.c:2503:main]
> 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.10.6
> (args: /usr/sbin/glusterfsd -s gluster-2 --volfile-id
> digitalcorpora.gluster-2.export-brick7-digitalcorpora -p
> /var/lib/glusterd/vols/digitalcorpora/run/gluster-2-export-brick7-digitalcorpora.pid
> -S /var/run/gluster/f8e0b3393e47dc51a07c6609f9b40841.socket --brick-name
> /export/brick7/digitalcorpora -l
> /var/log/glusterfs/bricks/export-brick7-digitalcorpora.log --xlator-option
> *-posix.glusterd-uuid=032c17f5-8cc9-445f-aa45-897b5a066b43 --brick-port
> 49154 --xlator-option digitalcorpora-server.listen-port=49154)
> [2017-10-24 17:18:59.285279] I [MSGID: 101190]
> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index 1
> [2017-10-24 17:19:04.611723] I
> [rpcsvc.c:2237:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured
> rpc.outstanding-rpc-limit with value 64
> [2017-10-24 17:19:04.611815] W [MSGID: 101002]
> [options.c:954:xl_opt_validate] 0-digitalcorpora-server: option
> 'listen-port' is deprecated, preferred is 'transport.socket.listen-port',
> continuing with correction
> [2017-10-24 17:19:04.615974] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-digitalcorpora-server: option
> 'rpc-auth.auth-glusterfs' is not recognized
> [2017-10-24 17:19:04.616033] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-digitalcorpora-server: option
> 'rpc-auth.auth-unix' is not recognized
> [2017-10-24 17:19:04.616070] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-digitalcorpora-server: option
> 'rpc-auth.auth-null' is not recognized
> [2017-10-24 17:19:04.616134] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-digitalcorpora-server: option
> 'auth-path' is not recognized
> [2017-10-24 17:19:04.616177] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-digitalcorpora-server: option
> 'ping-timeout' is not recognized
> [2017-10-24 17:19:04.616203] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-/export/brick7/digitalcorpora:
> option 'rpc-auth-allow-insecure' is not recognized
> [2017-10-24 17:19:04.616215] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-/export/brick7/digitalcorpora:
> option 'auth.addr./export/brick7/digitalcorpora.allow' is not recognized
> [2017-10-24 17:19:04.616226] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-/export/brick7/digitalcorpora:
> option 'auth-path' is not recognized
> [2017-10-24 17:19:04.616237] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-/export/brick7/digitalcorpora:
> option 'auth.login.b17f2513-7d9c-4174-a0c5-de4a752d46ca.password' is not
> recognized
> [2017-10-24 17:19:04.616248] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-/export/brick7/digitalcorpora:
> option 'auth.login./export/brick7/digitalcorpora.allow' is not recognized
> [2017-10-24 17:19:04.616283] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-digitalcorpora-quota: option
> 'timeout' is not recognized
> [2017-10-24 17:19:04.616367] W [MSGID: 101174]
> [graph.c:361:_log_if_unknown_option] 0-digitalcorpora-trash: option
> 'brick-path' is not recognized
> Final graph:
>
> +------------------------------------------------------------------------------+
>   1: volume digitalcorpora-posix
>   2:     type storage/posix
>   3:     option glusterd-uuid 032c17f5-8cc9-445f-aa45-897b5a066b43
>   4:     option directory /export/brick7/digitalcorpora
>   5:     option volume-id 61efe58a-ae5b-4d8b-b9f9-67829867c442
>   6:     option brick-uid 36
>   7:     option brick-gid 36
>   8: end-volume
>   9:
>  10: volume digitalcorpora-trash
>  11:     type features/trash
>  12:     option trash-dir .trashcan
>  13:     option brick-path /export/brick7/digitalcorpora
>  14:     option trash-internal-op off
>  15:     subvolumes digitalcorpora-posix
>  16: end-volume
>  17:
>  18: volume digitalcorpora-changetimerecorder
>  19:     type features/changetimerecorder
>  20:     option db-type sqlite3
>  21:     option hot-brick off
>  22:     option db-name digitalcorpora.db
>  23:     option db-path /export/brick7/digitalcorpora/.glusterfs/
>  24:     option record-exit off
>  25:     option ctr_link_consistency off
>  26:     option ctr_lookupheal_link_timeout 300
>  27:     option ctr_lookupheal_inode_timeout 300
>  28:     option record-entry on
>  29:     option ctr-enabled off
>  30:     option record-counters off
>  31:     option ctr-record-metadata-heat off
>  32:     option sql-db-cachesize 12500
>  33:     option sql-db-wal-autocheckpoint 25000
>  34:     subvolumes digitalcorpora-trash
>  35: end-volume
>  36:
>  37: volume digitalcorpora-changelog
>  38:     type features/changelog
>  39:     option changelog-brick /export/brick7/digitalcorpora
>  40:     option changelog-dir
> /export/brick7/digitalcorpora/.glusterfs/changelogs
>  41:     option changelog-barrier-timeout 120
>  42:     subvolumes digitalcorpora-changetimerecorder
>  43: end-volume
>  44:
>  45: volume digitalcorpora-bitrot-stub
>  46:     type features/bitrot-stub
>  47:     option export /export/brick7/digitalcorpora
>  48:     subvolumes digitalcorpora-changelog
>  49: end-volume
>  50:
>  51: volume digitalcorpora-access-control
>  52:     type features/access-control
>  53:     subvolumes digitalcorpora-bitrot-stub
>  54: end-volume
>  55:
>  56: volume digitalcorpora-locks
>  57:     type features/locks
>  58:     subvolumes digitalcorpora-access-control
>  59: end-volume
>  60:
>  61: volume digitalcorpora-worm
>  62:     type features/worm
>  63:     option worm off
>  64:     option worm-file-level off
>  65:     subvolumes digitalcorpora-locks
>  66: end-volume
>  67:
>  68: volume digitalcorpora-read-only
>  69:     type features/read-only
>  70:     option read-only off
>  71:     subvolumes digitalcorpora-worm
>  72: end-volume
>  73:
>  74: volume digitalcorpora-leases
>  75:     type features/leases
>  76:     option leases off
>  77:     subvolumes digitalcorpora-read-only
>  78: end-volume
>  79:
>  80: volume digitalcorpora-upcall
>  81:     type features/upcall
>  82:     option cache-invalidation off
>  83:     subvolumes digitalcorpora-leases
>  84: end-volume
>  85:
>  86: volume digitalcorpora-io-threads
>  87:     type performance/io-threads
>  88:     subvolumes digitalcorpora-upcall
>  89: end-volume
>  90:
>  91: volume digitalcorpora-marker
>  92:     type features/marker
>  93:     option volume-uuid 61efe58a-ae5b-4d8b-b9f9-67829867c442
>  94:     option timestamp-file
> /var/lib/glusterd/vols/digitalcorpora/marker.tstamp
>  95:     option quota-version 0
>  96:     option xtime off
>  97:     option gsync-force-xtime off
>  98:     option quota off
>  99:     option inode-quota off
> 100:     subvolumes digitalcorpora-io-threads
> 101: end-volume
> 102:
> 103: volume digitalcorpora-barrier
> 104:     type features/barrier
> 105:     option barrier disable
> 106:     option barrier-timeout 120
> 107:     subvolumes digitalcorpora-marker
> 108: end-volume
> 109:
> 110: volume digitalcorpora-index
> 111:     type features/index
> 112:     option index-base /export/brick7/digitalcorpora/.glusterfs/indices
> 113:     option xattrop-dirty-watchlist trusted.afr.dirty
> 114:     option xattrop-pending-watchlist trusted.afr.digitalcorpora-
> 115:     subvolumes digitalcorpora-barrier
> 116: end-volume
> 117:
> 118: volume digitalcorpora-quota
> 119:     type features/quota
> 120:     option volume-uuid digitalcorpora
> 121:     option server-quota off
> 122:     option timeout 0
> 123:     option deem-statfs off
> 124:     subvolumes digitalcorpora-index
> 125: end-volume
> 126:
> 127: volume digitalcorpora-io-stats
> 128:     type debug/io-stats
> 129:     option unique-id /export/brick7/digitalcorpora
> 130:     option log-level WARNING
> 131:     option latency-measurement off
> 132:     option count-fop-hits off
> 133:     subvolumes digitalcorpora-quota
> 134: end-volume
> 135:
> 136: volume /export/brick7/digitalcorpora
> 137:     type performance/decompounder
> 138:     option rpc-auth-allow-insecure on
> 139:     option auth.addr./export/brick7/digitalcorpora.allow
> 129.174.125.204,129.174.93.204
> 140:     option auth-path /export/brick7/digitalcorpora
> 141:     option auth.login.b17f2513-7d9c-4174-a0c5-de4a752d46ca.password
> 6c007ad0-b5a2-4564-8464-300f8317e5c7
> 142:     option auth.login./export/brick7/digitalcorpora.allow
> b17f2513-7d9c-4174-a0c5-de4a752d46ca
> 143:     subvolumes digitalcorpora-io-stats
> 144: end-volume
> 145:
> 146: volume digitalcorpora-server
> 147:     type protocol/server
> 148:     option transport.socket.listen-port 49154
> 149:     option rpc-auth.auth-glusterfs on
> 150:     option rpc-auth.auth-unix on
> 151:     option rpc-auth.auth-null on
> 152:     option transport-type tcp
> 153:     option transport.address-family inet
> 154:     option auth.login./export/brick7/digitalcorpora.allow
> b17f2513-7d9c-4174-a0c5-de4a752d46ca
> 155:     option auth.login.b17f2513-7d9c-4174-a0c5-de4a752d46ca.password
> 6c007ad0-b5a2-4564-8464-300f8317e5c7
> 156:     option auth-path /export/brick7/digitalcorpora
> 157:     option auth.addr./export/brick7/digitalcorpora.allow
> 129.174.125.204,129.174.93.204
> 158:     option ping-timeout 42
> 159:     option transport.socket.keepalive 1
> 160:     option rpc-auth-allow-insecure on
> 161:     option transport.tcp-user-timeout 0
> 162:     option transport.socket.keepalive-time 20
> 163:     option transport.socket.keepalive-interval 2
> 164:     option transport.socket.keepalive-count 9
> 165:     subvolumes /export/brick7/digitalcorpora
> 166: end-volume
> 167:
>
> +------------------------------------------------------------------------------+
> [2017-10-24 17:22:21.438620] W [socket.c:593:__socket_rwv] 0-glusterfs:
> readv on 129.174.126.87:24007 failed (No data available)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171024/d8acd6bd/attachment.html>


More information about the Gluster-users mailing list