[Gluster-users] split-brain
Bernhard Dübi
1linuxengineer at gmail.com
Fri Mar 17 21:50:18 UTC 2017
Hi,
I have a situation
the volume logfile reports a possible split-brain but when I try to heal it
fails because the file is not in split-brain. Any ideas?
root at chastcvtprd04:~# uname -a
Linux chastcvtprd04 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux
root at chastcvtprd04:~# dpkg -l | grep gluster
ii glusterfs-client 3.8.8-ubuntu1~xenial1
amd64 clustered file-system (client package)
ii glusterfs-common 3.8.8-ubuntu1~xenial1
amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.8.8-ubuntu1~xenial1
amd64 clustered file-system (server package)
root at chastcvtprd04:~# gluster volume info Server_Legal_01
Volume Name: Server_Legal_01
Type: Replicate
Volume ID: ce6426a8-6fd2-4f34-bb04-7817cea598fb
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: chastcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick
Brick2: chglbcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick
Options Reconfigured:
auth.allow: 127.0.0.1,10.30.28.43,10.30.28.44
nfs.disable: on
performance.readdir-ahead: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
features.bitrot: on
features.scrub: Active
root at chastcvtprd04:~# ls -l
/data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
-rw-rw-rw- 1 root simpana 27357050 Jan 11 21:21
/data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
root at chastcvtprd04:~# sha256sum
/data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
sha256sum:
/data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046:
Input/output error
root at chastcvtprd04:~# gluster volume heal Server_Legal_01 info
Brick chastcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick
Status: Connected
Number of entries: 0
Brick chglbcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick
Status: Connected
Number of entries: 0
root at chastcvtprd04:~# getfattr -d -e hex -m .
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
getfattr: Removing leading '/' from absolute path names
# file:
data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.bad-file=0x3100
trusted.bit-rot.signature=0x010500000000000000164947a36f9e22b5dcdfb4cf69d809396302de9d6a9b3f266532e40ddc45db04
trusted.bit-rot.version=0x0500000000000000585818d80008b694
trusted.gfid=0xbfdfe21a1af3474ba6a4bc0e17edb529
root at chglbcvtprd04:~# getfattr -d -e hex -m .
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
getfattr: Removing leading '/' from absolute path names
# file:
data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.bad-file=0x3100
trusted.bit-rot.signature=0x010500000000000000164947a36f9e22b5dcdfb4cf69d809396302de9d6a9b3f266532e40ddc45db04
trusted.bit-rot.version=0x050000000000000058580d5f0001543b
trusted.gfid=0xbfdfe21a1af3474ba6a4bc0e17edb529
root at chastcvtprd04:~# sha256sum
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
6334395ab5eae6c9956ca82b76e2f0124aff3eb3a50ef6ebccbdccad65525f73
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
root at chglbcvtprd04:~# sha256sum
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
6334395ab5eae6c9956ca82b76e2f0124aff3eb3a50ef6ebccbdccad65525f73
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
root at chastcvtprd04:~# ls -li
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
1163752 -rw-rw-rw- 2 root simpana 27357050 Jan 11 21:21
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
root at chastcvtprd04:~# find /data/glusterfs/Server_Legal/2I-1-32/brick -inum
1163752
/data/glusterfs/Server_Legal/2I-1-32/brick/.glusterfs/bf/df/bfdfe21a-1af3-474b-a6a4-bc0e17edb529
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
root at chglbcvtprd04:~# ls -li
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
12886039523 -rw-rw-rw- 2 root simpana 27357050 Jan 11 21:21
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
root at chglbcvtprd04:~# find /data/glusterfs/Server_Legal/2I-1-32/brick
-inum 12886039523
/data/glusterfs/Server_Legal/2I-1-32/brick/.glusterfs/bf/df/bfdfe21a-1af3-474b-a6a4-bc0e17edb529
/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
root at chastcvtprd04:/var/log/glusterfs# ls -lrt *Server_Legal_01*.log
-rw------- 1 root root 2024748 Mar 17 20:50
data-simpana-Data-Server_Legal_01.log
-rw------- 1 root root 251700 Mar 17 20:54 glfsheal-Server_Legal_01.log
root at chastcvtprd04:/var/log/glusterfs#
root at chastcvtprd04:/var/log/glusterfs# tail -100
data-simpana-Data-Server_Legal_01.log
[2017-03-17 19:12:36.122616] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. [Input/output
error]
[2017-03-17 19:12:36.122949] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)
[2017-03-17 19:12:36.123099] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:12:36.123385] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:12:36.123407] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. [Input/output
error]
[2017-03-17 19:12:36.137285] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:12:36.137498] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080
(c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]
[2017-03-17 19:12:36.137837] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080
(c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]
[2017-03-17 19:12:36.137875] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9440,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:12:36.137917] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 321134: READ => -1
gfid=c5a0afdf-3114-4b2f-9664-b00255a08b1b fd=0x7f9dfb0f9440 (Input/output
error)
[2017-03-17 19:12:36.137994] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)
[2017-03-17 19:12:36.138172] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:12:36.138484] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:12:36.138505] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. [Input/output
error]
[2017-03-17 19:12:36.138802] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)
[2017-03-17 19:12:36.138989] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:12:36.139241] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:12:36.139246] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. [Input/output
error]
[2017-03-17 19:12:36.139793] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:12:36.139966] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080
(c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]
[2017-03-17 19:12:36.140245] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080
(c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]
[2017-03-17 19:12:36.140274] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9440,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:12:36.140316] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 321135: READ => -1
gfid=c5a0afdf-3114-4b2f-9664-b00255a08b1b fd=0x7f9dfb0f9440 (Input/output
error)
[2017-03-17 19:12:47.802878] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)
[2017-03-17 19:12:47.803128] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:12:47.803342] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:12:47.803373] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. [Input/output
error]
[2017-03-17 19:12:47.803524] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)
[2017-03-17 19:12:47.803692] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:12:47.803999] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:12:47.804020] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. [Input/output
error]
[2017-03-17 19:12:47.804576] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:12:47.804746] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070
(4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]
[2017-03-17 19:12:47.805046] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070
(4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]
[2017-03-17 19:12:47.805078] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f94cc,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:12:47.805120] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 332890: READ => -1
gfid=4619a256-3787-4c06-ac57-33dfb174456e fd=0x7f9dfb0f94cc (Input/output
error)
[2017-03-17 19:12:47.805177] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)
[2017-03-17 19:12:47.805353] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:12:47.805690] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:12:47.805712] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. [Input/output
error]
[2017-03-17 19:12:47.806462] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)
[2017-03-17 19:12:47.806663] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:12:47.806944] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:12:47.806950] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. [Input/output
error]
[2017-03-17 19:12:47.807486] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:12:47.807651] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070
(4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]
[2017-03-17 19:12:47.807993] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070
(4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]
[2017-03-17 19:12:47.808044] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f94cc,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:12:47.808086] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 332891: READ => -1
gfid=4619a256-3787-4c06-ac57-33dfb174456e fd=0x7f9dfb0f94cc (Input/output
error)
[2017-03-17 19:11:20.346125] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
6de87045-ece9-40e8-bdf3-7dad12de561f: split-brain observed. [Input/output
error]
The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn]
0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event
generation 2 for gfid 6de87045-ece9-40e8-bdf3-7dad12de561f. (Possible
split-brain)" repeated 3 times between [2017-03-17 19:11:20.343014] and
[2017-03-17 19:11:20.346996]
[2017-03-17 19:12:59.771690] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc. (Possible split-brain)
[2017-03-17 19:12:59.771978] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:12:59.772209] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:12:59.772228] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. [Input/output
error]
[2017-03-17 19:12:59.772958] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. [Input/output
error]
[2017-03-17 19:12:59.773509] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:12:59.773677] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116
(1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]
[2017-03-17 19:12:59.774045] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116
(1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]
[2017-03-17 19:12:59.774132] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:12:59.774218] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 345004: READ => -1
gfid=1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc fd=0x7f9dfb0f9184 (Input/output
error)
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk]
0-Server_Legal_01-client-1: remote operation failed [Input/output error]"
repeated 3 times between [2017-03-17 19:12:59.772209] and [2017-03-17
19:12:59.775988]
[2017-03-17 19:12:59.775996] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. [Input/output
error]
[2017-03-17 19:12:59.776538] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:12:59.776814] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116
(1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]
[2017-03-17 19:12:59.777067] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116
(1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]
[2017-03-17 19:12:59.777131] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 345006: READ => -1
gfid=1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc fd=0x7f9dfb0f9184 (Input/output
error)
[2017-03-17 19:12:59.774854] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. [Input/output
error]
The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn]
0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event
generation 2 for gfid 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc. (Possible
split-brain)" repeated 3 times between [2017-03-17 19:12:59.771690] and
[2017-03-17 19:12:59.775548]
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk]
0-Server_Legal_01-client-0: remote operation failed [Input/output error]"
repeated 3 times between [2017-03-17 19:12:59.771978] and [2017-03-17
19:12:59.775719]
[2017-03-17 19:12:59.777105] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:26:51.275897] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
bfdfe21a-1af3-474b-a6a4-bc0e17edb529. (Possible split-brain)
[2017-03-17 19:26:51.276390] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing GETXATTR on gfid
bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output
error]
[2017-03-17 19:26:51.276427] W [fuse-bridge.c:3441:fuse_xattr_cbk]
0-glusterfs-fuse: 1154312: GETXATTR((null))
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 => -1
(Input/output error)
[2017-03-17 19:49:56.076750] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
bfdfe21a-1af3-474b-a6a4-bc0e17edb529. (Possible split-brain)
[2017-03-17 19:49:56.077000] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:49:56.077259] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:49:56.077294] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output
error]
[2017-03-17 19:49:56.078283] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output
error]
[2017-03-17 19:49:56.078337] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output
error]
[2017-03-17 19:49:56.088463] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:49:56.088539] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:49:56.088830] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
[2017-03-17 19:49:56.088867] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
[2017-03-17 19:49:56.089012] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
[2017-03-17 19:49:56.089060] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:49:56.089081] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
[2017-03-17 19:49:56.089107] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 2491526: READ => -1
gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 (Input/output
error)
[2017-03-17 19:49:56.089124] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:49:56.089165] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 2491525: READ => -1
gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 (Input/output
error)
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk]
0-Server_Legal_01-client-1: remote operation failed [Input/output error]"
repeated 5 times between [2017-03-17 19:49:56.077259] and [2017-03-17
19:49:56.090419]
[2017-03-17 19:49:56.090426] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output
error]
[2017-03-17 19:49:56.091034] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:49:56.091201] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
[2017-03-17 19:49:56.091519] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
[2017-03-17 19:49:56.091578] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 2491534: READ => -1
gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 (Input/output
error)
The message "E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output
error]" repeated 2 times between [2017-03-17 19:49:56.077294] and
[2017-03-17 19:49:56.089723]
The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn]
0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event
generation 2 for gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529. (Possible
split-brain)" repeated 5 times between [2017-03-17 19:49:56.076750] and
[2017-03-17 19:49:56.089951]
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk]
0-Server_Legal_01-client-0: remote operation failed [Input/output error]"
repeated 5 times between [2017-03-17 19:49:56.077000] and [2017-03-17
19:49:56.090086]
[2017-03-17 19:49:56.091548] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @
Server_Legal_01-replicate-0 [Input/output error]
root at chastcvtprd04:/var/log/glusterfs#
root at chastcvtprd04:/var/log/glusterfs# tail -100
glfsheal-Server_Legal_01.log
[2017-03-17 19:19:32.393846] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:19:32.561091] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:19:32.561904] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-0
[2017-03-17 19:19:32.562494] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 63686173-7463-7674-7072-6430342d3334 (0)
[2017-03-17 19:23:55.983126] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 63686173-7463-7674-7072-6430342d3939 (0) coming up
[2017-03-17 19:23:55.983162] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:23:56.003121] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:23:56.003303] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:23:56.004036] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:23:56.004251] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:23:56.004269] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:23:56.004307] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-0' came back up; going online.
[2017-03-17 19:23:56.004329] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:23:56.004367] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:23:56.005341] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:23:56.005862] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:23:56.005877] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:23:56.099082] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:23:56.099922] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-0
[2017-03-17 19:23:56.100670] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 63686173-7463-7674-7072-6430342d3939 (0)
[2017-03-17 19:34:07.952797] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 63686173-7463-7674-7072-6430342d3331 (0) coming up
[2017-03-17 19:34:07.957727] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:34:07.958670] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:34:07.958853] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:34:07.959549] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:34:07.959819] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:34:07.959832] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:34:07.959865] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-0' came back up; going online.
[2017-03-17 19:34:07.959885] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:34:07.960019] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:34:07.960957] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:34:07.961445] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:34:07.961457] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:34:08.130678] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:34:08.131439] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-0
[2017-03-17 19:34:08.132110] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 63686173-7463-7674-7072-6430342d3331 (0)
[2017-03-17 19:36:40.730984] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 63686173-7463-7674-7072-6430342d3234 (0) coming up
[2017-03-17 19:36:40.731023] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:36:40.732963] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:36:40.733160] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:36:40.733704] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:36:40.733952] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:36:40.733974] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:36:40.734012] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-0' came back up; going online.
[2017-03-17 19:36:40.734034] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:36:40.734119] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:36:40.735092] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:36:40.735594] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:36:40.735607] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:36:40.935215] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:36:40.936135] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-0
[2017-03-17 19:36:40.936856] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 63686173-7463-7674-7072-6430342d3234 (0)
[2017-03-17 19:44:08.745876] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 63686173-7463-7674-7072-6430342d3335 (0) coming up
[2017-03-17 19:44:08.745911] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:44:08.748918] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:44:08.749122] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:44:08.749677] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:44:08.749980] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:44:08.750000] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:44:08.750047] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-0' came back up; going online.
[2017-03-17 19:44:08.750071] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:44:08.750145] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:44:08.751171] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:44:08.751669] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:44:08.751682] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:44:08.871739] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:44:08.872568] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-0
[2017-03-17 19:44:08.873269] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 63686173-7463-7674-7072-6430342d3335 (0)
[2017-03-17 19:51:04.343728] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 63686173-7463-7674-7072-6430342d3231 (0) coming up
[2017-03-17 19:51:04.343761] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:51:04.345547] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:51:04.345702] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:51:04.346288] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:51:04.346666] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:51:04.346681] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:51:04.346724] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-0' came back up; going online.
[2017-03-17 19:51:04.346756] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:51:04.346788] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:51:04.347909] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:51:04.348419] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:51:04.348432] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:51:04.528253] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:51:04.529286] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-0
[2017-03-17 19:51:04.530379] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 63686173-7463-7674-7072-6430342d3231 (0)
[2017-03-17 19:54:04.941563] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 63686173-7463-7674-7072-6430342d3232 (0) coming up
[2017-03-17 19:54:04.941606] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:54:04.942498] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:54:04.942753] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:54:04.943498] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:54:04.943737] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:54:04.943823] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:54:04.943848] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:54:04.943893] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-0' came back up; going online.
[2017-03-17 19:54:04.943914] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:54:04.944870] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:54:04.945439] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:54:04.945460] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:54:05.292434] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:54:05.293381] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-0
[2017-03-17 19:54:05.294255] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 63686173-7463-7674-7072-6430342d3232 (0)
root at chastcvtprd04:/var/log/glusterfs#
root at chglbcvtprd04:/var/log/glusterfs# ls -lrt *Server_Legal_01*.log
-rw------- 1 root root 240660 Mar 17 20:52 glfsheal-Server_Legal_01.log
-rw------- 1 root root 7788358 Mar 17 20:53
data-simpana-Data-Server_Legal_01.log
root at chglbcvtprd04:/var/log/glusterfs# tail -100
data-simpana-Data-Server_Legal_01.log
[2017-03-17 19:48:39.804078] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:48:39.804249] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:48:39.804253] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
53f03480-7281-48c1-8daf-51cd3099b47f: split-brain observed. [Input/output
error]
[2017-03-17 19:48:39.804716] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:48:39.804900] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030
(53f03480-7281-48c1-8daf-51cd3099b47f) [Input/output error]
[2017-03-17 19:48:39.805156] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030
(53f03480-7281-48c1-8daf-51cd3099b47f) [Input/output error]
[2017-03-17 19:48:39.805185] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:48:39.805224] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7535988: READ => -1
gfid=53f03480-7281-48c1-8daf-51cd3099b47f fd=0x7f1a94002328 (Input/output
error)
[2017-03-17 19:47:45.522943] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
ddc01be9-1c4a-4535-adc3-91a61e326e3e: split-brain observed. [Input/output
error]
The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn]
0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event
generation 2 for gfid ddc01be9-1c4a-4535-adc3-91a61e326e3e. (Possible
split-brain)" repeated 3 times between [2017-03-17 19:47:45.520414] and
[2017-03-17 19:47:45.523127]
[2017-03-17 19:49:44.398208] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
e09bffec-f6f4-432f-8f68-ec3520c6f211. (Possible split-brain)
[2017-03-17 19:49:44.398446] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:49:44.398693] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:49:44.398720] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. [Input/output
error]
[2017-03-17 19:49:44.399501] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. [Input/output
error]
[2017-03-17 19:49:44.400061] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:49:44.400258] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044
(e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]
[2017-03-17 19:49:44.400507] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044
(e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]
[2017-03-17 19:49:44.400537] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:49:44.400585] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7538116: READ => -1
gfid=e09bffec-f6f4-432f-8f68-ec3520c6f211 fd=0x7f1a94002328 (Input/output
error)
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk]
0-Server_Legal_01-client-0: remote operation failed [Input/output error]"
repeated 3 times between [2017-03-17 19:49:44.398693] and [2017-03-17
19:49:44.401700]
[2017-03-17 19:49:44.401709] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. [Input/output
error]
[2017-03-17 19:49:44.402194] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:49:44.402372] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044
(e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]
[2017-03-17 19:49:44.402627] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044
(e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]
[2017-03-17 19:49:44.402678] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7538117: READ => -1
gfid=e09bffec-f6f4-432f-8f68-ec3520c6f211 fd=0x7f1a94002328 (Input/output
error)
[2017-03-17 19:49:44.402654] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:50:47.215179] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk]
0-Server_Legal_01-client-1: remote operation failed [Input/output error]"
repeated 4 times between [2017-03-17 19:49:44.398446] and [2017-03-17
19:50:47.215449]
[2017-03-17 19:50:47.215623] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:50:47.215659] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. [Input/output
error]
[2017-03-17 19:50:47.215974] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)
[2017-03-17 19:50:47.216138] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:50:47.216339] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:50:47.216355] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. [Input/output
error]
[2017-03-17 19:50:47.216802] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:50:47.217006] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005
(e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]
[2017-03-17 19:50:47.217221] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005
(e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]
[2017-03-17 19:50:47.217254] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:50:47.217298] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7538180: READ => -1
gfid=e71812cf-725d-4e95-a471-59cf099055e5 fd=0x7f1a94002440 (Input/output
error)
[2017-03-17 19:50:47.217382] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)
[2017-03-17 19:50:47.217572] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:50:47.217779] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:50:47.217799] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. [Input/output
error]
[2017-03-17 19:50:47.217966] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)
[2017-03-17 19:50:47.218181] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:50:47.218389] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:50:47.218394] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. [Input/output
error]
[2017-03-17 19:50:47.218842] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:50:47.219020] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005
(e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]
[2017-03-17 19:50:47.219200] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005
(e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]
[2017-03-17 19:50:47.219231] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:50:47.219270] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7538181: READ => -1
gfid=e71812cf-725d-4e95-a471-59cf099055e5 fd=0x7f1a94002440 (Input/output
error)
[2017-03-17 19:49:44.401142] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. [Input/output
error]
The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn]
0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event
generation 2 for gfid e09bffec-f6f4-432f-8f68-ec3520c6f211. (Possible
split-brain)" repeated 3 times between [2017-03-17 19:49:44.398208] and
[2017-03-17 19:49:44.401301]
[2017-03-17 19:51:13.064446] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
2532bb1c-6241-4fd0-92df-30059be2be17. (Possible split-brain)
[2017-03-17 19:51:13.064691] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:51:13.065358] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:51:13.065389] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. [Input/output
error]
[2017-03-17 19:51:13.066393] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. [Input/output
error]
[2017-03-17 19:51:13.066873] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:51:13.067147] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110
(2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]
[2017-03-17 19:51:13.067281] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110
(2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]
[2017-03-17 19:51:13.067366] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:51:13.067416] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7538199: READ => -1
gfid=2532bb1c-6241-4fd0-92df-30059be2be17 fd=0x7f1a94002440 (Input/output
error)
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk]
0-Server_Legal_01-client-0: remote operation failed [Input/output error]"
repeated 3 times between [2017-03-17 19:51:13.065358] and [2017-03-17
19:51:13.068580]
[2017-03-17 19:51:13.068586] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. [Input/output
error]
[2017-03-17 19:51:13.069092] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:51:13.069276] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110
(2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]
[2017-03-17 19:51:13.069538] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110
(2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]
[2017-03-17 19:51:13.069624] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7538200: READ => -1
gfid=2532bb1c-6241-4fd0-92df-30059be2be17 fd=0x7f1a94002440 (Input/output
error)
[2017-03-17 19:51:13.069597] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:52:14.972489] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk]
0-Server_Legal_01-client-1: remote operation failed [Input/output error]"
repeated 4 times between [2017-03-17 19:51:13.064691] and [2017-03-17
19:52:14.972716]
[2017-03-17 19:52:14.972967] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:52:14.973008] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. [Input/output
error]
[2017-03-17 19:52:14.973861] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)
[2017-03-17 19:52:14.974022] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:52:14.974239] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:52:14.974263] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. [Input/output
error]
[2017-03-17 19:52:14.974724] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:52:14.974893] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062
(5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]
[2017-03-17 19:52:14.975112] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062
(5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]
[2017-03-17 19:52:14.975146] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a940020f8,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:52:14.975192] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7538225: READ => -1
gfid=5752e22b-5e89-4543-b7d7-da2a22af09df fd=0x7f1a940020f8 (Input/output
error)
[2017-03-17 19:52:14.975266] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)
[2017-03-17 19:52:14.975463] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:52:14.975654] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:52:14.975685] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. [Input/output
error]
[2017-03-17 19:52:14.975860] W [MSGID: 108008]
[afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable
subvolume -1 found with event generation 2 for gfid
5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)
[2017-03-17 19:52:14.976067] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1:
remote operation failed [Input/output error]
[2017-03-17 19:52:14.976240] W [MSGID: 114031]
[client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0:
remote operation failed [Input/output error]
[2017-03-17 19:52:14.976248] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid
5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. [Input/output
error]
[2017-03-17 19:52:14.976742] W [MSGID: 108027]
[afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no
read subvols for (null)
[2017-03-17 19:52:14.976933] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062
(5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]
[2017-03-17 19:52:14.977122] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0:
remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062
(5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]
[2017-03-17 19:52:14.977152] E [MSGID: 109089]
[dht-helper.c:1297:dht_migration_complete_check_task]
0-Server_Legal_01-dht: failed to open the fd (0x7f1a940020f8,
flags=0100000) on file
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 @
Server_Legal_01-replicate-0 [Input/output error]
[2017-03-17 19:52:14.977193] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 7538226: READ => -1
gfid=5752e22b-5e89-4543-b7d7-da2a22af09df fd=0x7f1a940020f8 (Input/output
error)
[2017-03-17 19:51:13.067915] E [MSGID: 108008]
[afr-read-txn.c:80:afr_read_txn_refresh_done]
0-Server_Legal_01-replicate-0: Failing READ on gfid
2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. [Input/output
error]
The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn]
0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event
generation 2 for gfid 2532bb1c-6241-4fd0-92df-30059be2be17. (Possible
split-brain)" repeated 3 times between [2017-03-17 19:51:13.064446] and
[2017-03-17 19:51:13.068212]
root at chglbcvtprd04:/var/log/glusterfs#
root at chglbcvtprd04:/var/log/glusterfs# tail -100
glfsheal-Server_Legal_01.log
[2017-03-17 19:02:46.687131] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:02:46.778952] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:02:46.779801] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-1
[2017-03-17 19:02:46.780649] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 6368676c-6263-7674-7072-6430342d3334 (0)
[2017-03-17 19:12:50.450406] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3139 (0) coming up
[2017-03-17 19:12:50.450442] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:12:50.451271] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:12:50.451783] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:12:50.452408] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:12:50.453418] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:12:50.454094] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:12:50.454109] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:12:50.454151] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-0' came back up; going online.
[2017-03-17 19:12:50.454510] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:12:50.459877] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:12:50.460109] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:12:50.460123] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:12:50.594044] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:12:50.594904] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-1
[2017-03-17 19:12:50.595728] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 6368676c-6263-7674-7072-6430342d3139 (0)
[2017-03-17 19:22:39.620877] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3337 (0) coming up
[2017-03-17 19:22:39.620916] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:22:39.621777] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:22:39.622293] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:22:39.622741] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:22:39.622912] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:22:39.622991] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:22:39.623011] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:22:39.623057] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-1' came back up; going online.
[2017-03-17 19:22:39.623080] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:22:39.623953] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:22:39.625582] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:22:39.625599] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:22:39.721866] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:22:39.722617] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-1
[2017-03-17 19:22:39.723187] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 6368676c-6263-7674-7072-6430342d3337 (0)
[2017-03-17 19:32:36.816124] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3130 (0) coming up
[2017-03-17 19:32:36.827326] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:32:36.877385] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:32:36.877896] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:32:36.878285] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:32:36.878345] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:32:36.878567] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:32:36.878587] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:32:36.878626] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-1' came back up; going online.
[2017-03-17 19:32:36.878646] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:32:36.879552] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:32:36.892670] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:32:36.892690] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:32:37.030054] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:32:37.030893] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-1
[2017-03-17 19:32:37.031642] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 6368676c-6263-7674-7072-6430342d3130 (0)
[2017-03-17 19:42:49.185410] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3339 (0) coming up
[2017-03-17 19:42:49.193109] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:42:49.194028] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:42:49.194544] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:42:49.194997] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:42:49.195080] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:42:49.195233] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:42:49.195249] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:42:49.195287] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-1' came back up; going online.
[2017-03-17 19:42:49.195307] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:42:49.196118] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:42:49.196576] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:42:49.196592] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:42:49.359040] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:42:49.359995] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-1
[2017-03-17 19:42:49.360629] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 6368676c-6263-7674-7072-6430342d3339 (0)
[2017-03-17 19:52:42.886903] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3439 (0) coming up
[2017-03-17 19:52:42.887194] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 19:52:42.888075] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 19:52:42.888592] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 19:52:42.888974] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 19:52:42.889131] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:52:42.889475] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:52:42.889499] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:52:42.889542] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-1' came back up; going online.
[2017-03-17 19:52:42.889574] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 19:52:42.889891] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 19:52:42.896888] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 19:52:42.896926] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 19:52:43.024347] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 19:52:43.025326] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-1
[2017-03-17 19:52:43.025934] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 6368676c-6263-7674-7072-6430342d3439 (0)
[2017-03-17 20:02:39.027898] I [MSGID: 104045] [glfs-master.c:91:notify]
0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3337 (0) coming up
[2017-03-17 20:02:39.028098] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-0: parent translators are ready, attempting
connect on transport
[2017-03-17 20:02:39.029060] I [MSGID: 114020] [client.c:2356:notify]
0-Server_Legal_01-client-1: parent translators are ready, attempting
connect on transport
[2017-03-17 20:02:39.029750] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-1: changing port to 49658 (from 0)
[2017-03-17 20:02:39.029976] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-Server_Legal_01-client-0: changing port to 49635 (from 0)
[2017-03-17 20:02:39.030502] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 20:02:39.030780] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Connected to Server_Legal_01-client-1, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 20:02:39.030801] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 20:02:39.030841] I [MSGID: 108005]
[afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume
'Server_Legal_01-client-1' came back up; going online.
[2017-03-17 20:02:39.030873] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-1: Server lk version = 1
[2017-03-17 20:02:39.030900] I [MSGID: 114057]
[client-handshake.c:1447:select_server_supported_programs]
0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2017-03-17 20:02:39.031378] I [MSGID: 114046]
[client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Connected to Server_Legal_01-client-0, attached to remote volume
'/data/glusterfs/Server_Legal/2I-1-32/brick'.
[2017-03-17 20:02:39.031392] I [MSGID: 114047]
[client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2017-03-17 20:02:39.147618] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-Server_Legal_01-client-0: Server lk version = 1
[2017-03-17 20:02:39.148469] I [MSGID: 108031]
[afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0:
selecting local read_child Server_Legal_01-client-1
[2017-03-17 20:02:39.148990] I [MSGID: 104041]
[glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to
graph 6368676c-6263-7674-7072-6430342d3337 (0)
root at chglbcvtprd04:/var/log/glusterfs#
Healing
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_066
failed: File not in split-brain.
Volume heal failed.
Healing
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_034
failed: File not in split-brain.
Volume heal failed.
Healing
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293457/SFILE_CONTAINER_021
failed: File not in split-brain.
Volume heal failed.
Healing
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_046
failed: File not in split-brain.
Volume heal failed.
Healing
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293559/SFILE_CONTAINER_005
failed: File not in split-brain.
Volume heal failed.
Healing
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9288763/SFILE_CONTAINER_030
failed: File not in split-brain.
Volume heal failed.
Healing
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_014
failed: File not in split-brain.
Volume heal failed.
Healing
/Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9288068/SFILE_CONTAINER_004
failed: File not in split-brain.
Volume heal failed.
Regards
Bernhard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170317/44bb4956/attachment.html>
More information about the Gluster-users
mailing list