[Gluster-users] split-brain

[Ravishankar N]

SFILE_CONTAINER_080 is the one which seems to be in split-brain. 
SFILE_CONTAINER_046, for which you have provided the getfattr output, 
hard links etc doesn't seem to be in split-brain.  We do see that the 
fops on SFILE_CONTAINER_046 are failing on the client translator itself 
due to EIO:

[2017-03-17 19:49:56.088867] E [MSGID: 114031] 
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
remote operation failed. Path: 
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.089012] E [MSGID: 114031] 
[client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
remote operation failed. Path: 
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

which is  why the sha256sum on the mount gave EIO.  And that is because 
the file seems to be corrupt on both bricks because the 
'trusted.bit-rot.bad-file' xattr is set.

Did you write to the files directly on the backend? What is interesting 
is that the sha256sum is same on both the bricks despite being both 
marked as bad by bitrot.

-Ravi


On 03/18/2017 03:20 AM, Bernhard Dübi wrote:
> Hi,
>
> I have a situation
>
> the volume logfile reports a possible split-brain but when I try to 
> heal it fails because the file is not in split-brain. Any ideas?
>
>
> root at chastcvtprd04:~# uname -a
>
> Linux chastcvtprd04 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 
> UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>
> root at chastcvtprd04:~# dpkg -l | grep gluster
>
> ii glusterfs-client 3.8.8-ubuntu1~xenial1              amd64        
> clustered file-system (client package)
>
> ii glusterfs-common 3.8.8-ubuntu1~xenial1              amd64        
> GlusterFS common libraries and translator modules
>
> ii glusterfs-server 3.8.8-ubuntu1~xenial1              amd64        
> clustered file-system (server package)
>
> root at chastcvtprd04:~# gluster volume info Server_Legal_01
>
> Volume Name: Server_Legal_01
>
> Type: Replicate
>
> Volume ID: ce6426a8-6fd2-4f34-bb04-7817cea598fb
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 1 x 2 = 2
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: chastcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick
>
> Brick2: chglbcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick
>
> Options Reconfigured:
>
> auth.allow: 127.0.0.1,10.30.28.43,10.30.28.44
>
> nfs.disable: on
>
> performance.readdir-ahead: on
>
> diagnostics.latency-measurement: on
>
> diagnostics.count-fop-hits: on
>
> features.bitrot: on
>
> features.scrub: Active
>
> root at chastcvtprd04:~# ls -l 
> /data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> -rw-rw-rw- 1 root simpana 27357050 Jan 11 21:21 
> /data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> root at chastcvtprd04:~# sha256sum 
> /data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> sha256sum: 
> /data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046: 
> Input/output error
>
> root at chastcvtprd04:~# gluster volume heal Server_Legal_01 info
>
> Brick chastcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick
>
> Status: Connected
>
> Number of entries: 0
>
> Brick chglbcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick
>
> Status: Connected
>
> Number of entries: 0
>
> root at chastcvtprd04:~# getfattr -d -e hex -m . 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: 
> data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.bit-rot.bad-file=0x3100
>
> trusted.bit-rot.signature=0x010500000000000000164947a36f9e22b5dcdfb4cf69d809396302de9d6a9b3f266532e40ddc45db04
>
> trusted.bit-rot.version=0x0500000000000000585818d80008b694
>
> trusted.gfid=0xbfdfe21a1af3474ba6a4bc0e17edb529
>
> root at chglbcvtprd04:~# getfattr -d -e hex -m . 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: 
> data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.bit-rot.bad-file=0x3100
>
> trusted.bit-rot.signature=0x010500000000000000164947a36f9e22b5dcdfb4cf69d809396302de9d6a9b3f266532e40ddc45db04
>
> trusted.bit-rot.version=0x050000000000000058580d5f0001543b
>
> trusted.gfid=0xbfdfe21a1af3474ba6a4bc0e17edb529
>
> root at chastcvtprd04:~# sha256sum 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> 6334395ab5eae6c9956ca82b76e2f0124aff3eb3a50ef6ebccbdccad65525f73 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> root at chglbcvtprd04:~# sha256sum 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> 6334395ab5eae6c9956ca82b76e2f0124aff3eb3a50ef6ebccbdccad65525f73 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> root at chastcvtprd04:~# ls -li 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> 1163752 -rw-rw-rw- 2 root simpana 27357050 Jan 11 21:21 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> root at chastcvtprd04:~# find /data/glusterfs/Server_Legal/2I-1-32/brick 
> -inum 1163752
>
> /data/glusterfs/Server_Legal/2I-1-32/brick/.glusterfs/bf/df/bfdfe21a-1af3-474b-a6a4-bc0e17edb529
>
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> root at chglbcvtprd04:~# ls -li 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> 12886039523 -rw-rw-rw- 2 root simpana 27357050 Jan 11 21:21 
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> root at chglbcvtprd04:~# find /data/glusterfs/Server_Legal/2I-1-32/brick 
> -inum 12886039523
>
> /data/glusterfs/Server_Legal/2I-1-32/brick/.glusterfs/bf/df/bfdfe21a-1af3-474b-a6a4-bc0e17edb529
>
> /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
>
> root at chastcvtprd04:/var/log/glusterfs# ls -lrt *Server_Legal_01*.log
>
> -rw------- 1 root root 2024748 Mar 17 20:50 
> data-simpana-Data-Server_Legal_01.log
>
> -rw------- 1 root root  251700 Mar 17 20:54 glfsheal-Server_Legal_01.log
>
> root at chastcvtprd04:/var/log/glusterfs#
>
> root at chastcvtprd04:/var/log/glusterfs# tail -100 
> data-simpana-Data-Server_Legal_01.log
>
> [2017-03-17 19:12:36.122616] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:36.122949] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)
>
> [2017-03-17 19:12:36.123099] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:36.123385] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:36.123407] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:36.137285] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:12:36.137498] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 
> (c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]
>
> [2017-03-17 19:12:36.137837] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 
> (c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]
>
> [2017-03-17 19:12:36.137875] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9440, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:12:36.137917] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 321134: READ => -1 
> gfid=c5a0afdf-3114-4b2f-9664-b00255a08b1b fd=0x7f9dfb0f9440 
> (Input/output error)
>
> [2017-03-17 19:12:36.137994] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)
>
> [2017-03-17 19:12:36.138172] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:36.138484] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:36.138505] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:36.138802] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)
>
> [2017-03-17 19:12:36.138989] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:36.139241] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:36.139246] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:36.139793] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:12:36.139966] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 
> (c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]
>
> [2017-03-17 19:12:36.140245] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 
> (c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]
>
> [2017-03-17 19:12:36.140274] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9440, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:12:36.140316] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 321135: READ => -1 
> gfid=c5a0afdf-3114-4b2f-9664-b00255a08b1b fd=0x7f9dfb0f9440 
> (Input/output error)
>
> [2017-03-17 19:12:47.802878] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)
>
> [2017-03-17 19:12:47.803128] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:47.803342] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:47.803373] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:47.803524] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)
>
> [2017-03-17 19:12:47.803692] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:47.803999] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:47.804020] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:47.804576] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:12:47.804746] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 
> (4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]
>
> [2017-03-17 19:12:47.805046] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 
> (4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]
>
> [2017-03-17 19:12:47.805078] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f94cc, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:12:47.805120] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 332890: READ => -1 
> gfid=4619a256-3787-4c06-ac57-33dfb174456e fd=0x7f9dfb0f94cc 
> (Input/output error)
>
> [2017-03-17 19:12:47.805177] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)
>
> [2017-03-17 19:12:47.805353] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:47.805690] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:47.805712] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:47.806462] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)
>
> [2017-03-17 19:12:47.806663] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:47.806944] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:47.806950] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:47.807486] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:12:47.807651] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 
> (4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]
>
> [2017-03-17 19:12:47.807993] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 
> (4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]
>
> [2017-03-17 19:12:47.808044] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f94cc, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:12:47.808086] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 332891: READ => -1 
> gfid=4619a256-3787-4c06-ac57-33dfb174456e fd=0x7f9dfb0f94cc 
> (Input/output error)
>
> [2017-03-17 19:11:20.346125] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 6de87045-ece9-40e8-bdf3-7dad12de561f: split-brain observed. 
> [Input/output error]
>
> The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 
> 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with 
> event generation 2 for gfid 6de87045-ece9-40e8-bdf3-7dad12de561f. 
> (Possible split-brain)" repeated 3 times between [2017-03-17 
> 19:11:20.343014] and [2017-03-17 19:11:20.346996]
>
> [2017-03-17 19:12:59.771690] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc. (Possible split-brain)
>
> [2017-03-17 19:12:59.771978] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:59.772209] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:12:59.772228] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:59.772958] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:59.773509] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:12:59.773677] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 
> (1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]
>
> [2017-03-17 19:12:59.774045] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 
> (1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]
>
> [2017-03-17 19:12:59.774132] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:12:59.774218] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 345004: READ => -1 
> gfid=1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc fd=0x7f9dfb0f9184 
> (Input/output error)
>
> The message "W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output 
> error]" repeated 3 times between [2017-03-17 19:12:59.772209] and 
> [2017-03-17 19:12:59.775988]
>
> [2017-03-17 19:12:59.775996] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:12:59.776538] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:12:59.776814] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 
> (1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]
>
> [2017-03-17 19:12:59.777067] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 
> (1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]
>
> [2017-03-17 19:12:59.777131] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 345006: READ => -1 
> gfid=1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc fd=0x7f9dfb0f9184 
> (Input/output error)
>
> [2017-03-17 19:12:59.774854] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. 
> [Input/output error]
>
> The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 
> 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with 
> event generation 2 for gfid 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc. 
> (Possible split-brain)" repeated 3 times between [2017-03-17 
> 19:12:59.771690] and [2017-03-17 19:12:59.775548]
>
> The message "W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output 
> error]" repeated 3 times between [2017-03-17 19:12:59.771978] and 
> [2017-03-17 19:12:59.775719]
>
> [2017-03-17 19:12:59.777105] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:26:51.275897] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> bfdfe21a-1af3-474b-a6a4-bc0e17edb529. (Possible split-brain)
>
> [2017-03-17 19:26:51.276390] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing GETXATTR on gfid 
> bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:26:51.276427] W [fuse-bridge.c:3441:fuse_xattr_cbk] 
> 0-glusterfs-fuse: 1154312: GETXATTR((null)) 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
> => -1 (Input/output error)
>
> [2017-03-17 19:49:56.076750] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> bfdfe21a-1af3-474b-a6a4-bc0e17edb529. (Possible split-brain)
>
> [2017-03-17 19:49:56.077000] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:49:56.077259] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:49:56.077294] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:49:56.078283] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:49:56.078337] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:49:56.088463] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:49:56.088539] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:49:56.088830] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
> (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
>
> [2017-03-17 19:49:56.088867] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
> (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
>
> [2017-03-17 19:49:56.089012] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
> (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
>
> [2017-03-17 19:49:56.089060] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:49:56.089081] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
> (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
>
> [2017-03-17 19:49:56.089107] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 2491526: READ => -1 
> gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 
> (Input/output error)
>
> [2017-03-17 19:49:56.089124] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:49:56.089165] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 2491525: READ => -1 
> gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 
> (Input/output error)
>
> The message "W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output 
> error]" repeated 5 times between [2017-03-17 19:49:56.077259] and 
> [2017-03-17 19:49:56.090419]
>
> [2017-03-17 19:49:56.090426] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:49:56.091034] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:49:56.091201] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
> (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
>
> [2017-03-17 19:49:56.091519] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 
> (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]
>
> [2017-03-17 19:49:56.091578] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 2491534: READ => -1 
> gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 
> (Input/output error)
>
> The message "E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. 
> [Input/output error]" repeated 2 times between [2017-03-17 
> 19:49:56.077294] and [2017-03-17 19:49:56.089723]
>
> The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 
> 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with 
> event generation 2 for gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529. 
> (Possible split-brain)" repeated 5 times between [2017-03-17 
> 19:49:56.076750] and [2017-03-17 19:49:56.089951]
>
> The message "W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output 
> error]" repeated 5 times between [2017-03-17 19:49:56.077000] and 
> [2017-03-17 19:49:56.090086]
>
> [2017-03-17 19:49:56.091548] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> root at chastcvtprd04:/var/log/glusterfs#
>
> root at chastcvtprd04:/var/log/glusterfs# tail -100 
> glfsheal-Server_Legal_01.log
>
> [2017-03-17 19:19:32.393846] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:19:32.561091] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:19:32.561904] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-0
>
> [2017-03-17 19:19:32.562494] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 63686173-7463-7674-7072-6430342d3334 (0)
>
> [2017-03-17 19:23:55.983126] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 63686173-7463-7674-7072-6430342d3939 (0) coming up
>
> [2017-03-17 19:23:55.983162] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:23:56.003121] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:23:56.003303] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:23:56.004036] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:23:56.004251] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:23:56.004269] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:23:56.004307] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-0' came back up; going online.
>
> [2017-03-17 19:23:56.004329] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:23:56.004367] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:23:56.005341] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:23:56.005862] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:23:56.005877] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:23:56.099082] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:23:56.099922] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-0
>
> [2017-03-17 19:23:56.100670] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 63686173-7463-7674-7072-6430342d3939 (0)
>
> [2017-03-17 19:34:07.952797] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 63686173-7463-7674-7072-6430342d3331 (0) coming up
>
> [2017-03-17 19:34:07.957727] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:34:07.958670] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:34:07.958853] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:34:07.959549] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:34:07.959819] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:34:07.959832] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:34:07.959865] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-0' came back up; going online.
>
> [2017-03-17 19:34:07.959885] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:34:07.960019] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:34:07.960957] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:34:07.961445] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:34:07.961457] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:34:08.130678] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:34:08.131439] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-0
>
> [2017-03-17 19:34:08.132110] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 63686173-7463-7674-7072-6430342d3331 (0)
>
> [2017-03-17 19:36:40.730984] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 63686173-7463-7674-7072-6430342d3234 (0) coming up
>
> [2017-03-17 19:36:40.731023] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:36:40.732963] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:36:40.733160] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:36:40.733704] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:36:40.733952] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:36:40.733974] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:36:40.734012] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-0' came back up; going online.
>
> [2017-03-17 19:36:40.734034] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:36:40.734119] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:36:40.735092] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:36:40.735594] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:36:40.735607] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:36:40.935215] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:36:40.936135] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-0
>
> [2017-03-17 19:36:40.936856] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 63686173-7463-7674-7072-6430342d3234 (0)
>
> [2017-03-17 19:44:08.745876] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 63686173-7463-7674-7072-6430342d3335 (0) coming up
>
> [2017-03-17 19:44:08.745911] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:44:08.748918] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:44:08.749122] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:44:08.749677] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:44:08.749980] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:44:08.750000] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:44:08.750047] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-0' came back up; going online.
>
> [2017-03-17 19:44:08.750071] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:44:08.750145] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:44:08.751171] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:44:08.751669] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:44:08.751682] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:44:08.871739] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:44:08.872568] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-0
>
> [2017-03-17 19:44:08.873269] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 63686173-7463-7674-7072-6430342d3335 (0)
>
> [2017-03-17 19:51:04.343728] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 63686173-7463-7674-7072-6430342d3231 (0) coming up
>
> [2017-03-17 19:51:04.343761] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:51:04.345547] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:51:04.345702] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:51:04.346288] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:51:04.346666] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:51:04.346681] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:51:04.346724] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-0' came back up; going online.
>
> [2017-03-17 19:51:04.346756] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:51:04.346788] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:51:04.347909] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:51:04.348419] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:51:04.348432] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:51:04.528253] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:51:04.529286] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-0
>
> [2017-03-17 19:51:04.530379] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 63686173-7463-7674-7072-6430342d3231 (0)
>
> [2017-03-17 19:54:04.941563] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 63686173-7463-7674-7072-6430342d3232 (0) coming up
>
> [2017-03-17 19:54:04.941606] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:54:04.942498] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:54:04.942753] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:54:04.943498] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:54:04.943737] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:54:04.943823] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:54:04.943848] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:54:04.943893] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-0' came back up; going online.
>
> [2017-03-17 19:54:04.943914] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:54:04.944870] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:54:04.945439] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:54:04.945460] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:54:05.292434] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:54:05.293381] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-0
>
> [2017-03-17 19:54:05.294255] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 63686173-7463-7674-7072-6430342d3232 (0)
>
> root at chastcvtprd04:/var/log/glusterfs#
>
> root at chglbcvtprd04:/var/log/glusterfs# ls -lrt *Server_Legal_01*.log
>
> -rw------- 1 root root  240660 Mar 17 20:52 glfsheal-Server_Legal_01.log
>
> -rw------- 1 root root 7788358 Mar 17 20:53 
> data-simpana-Data-Server_Legal_01.log
>
> root at chglbcvtprd04:/var/log/glusterfs# tail -100 
> data-simpana-Data-Server_Legal_01.log
>
> [2017-03-17 19:48:39.804078] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:48:39.804249] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:48:39.804253] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 53f03480-7281-48c1-8daf-51cd3099b47f: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:48:39.804716] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:48:39.804900] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030 
> (53f03480-7281-48c1-8daf-51cd3099b47f) [Input/output error]
>
> [2017-03-17 19:48:39.805156] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030 
> (53f03480-7281-48c1-8daf-51cd3099b47f) [Input/output error]
>
> [2017-03-17 19:48:39.805185] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:48:39.805224] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7535988: READ => -1 
> gfid=53f03480-7281-48c1-8daf-51cd3099b47f fd=0x7f1a94002328 
> (Input/output error)
>
> [2017-03-17 19:47:45.522943] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> ddc01be9-1c4a-4535-adc3-91a61e326e3e: split-brain observed. 
> [Input/output error]
>
> The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 
> 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with 
> event generation 2 for gfid ddc01be9-1c4a-4535-adc3-91a61e326e3e. 
> (Possible split-brain)" repeated 3 times between [2017-03-17 
> 19:47:45.520414] and [2017-03-17 19:47:45.523127]
>
> [2017-03-17 19:49:44.398208] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> e09bffec-f6f4-432f-8f68-ec3520c6f211. (Possible split-brain)
>
> [2017-03-17 19:49:44.398446] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:49:44.398693] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:49:44.398720] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:49:44.399501] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:49:44.400061] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:49:44.400258] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 
> (e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]
>
> [2017-03-17 19:49:44.400507] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 
> (e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]
>
> [2017-03-17 19:49:44.400537] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:49:44.400585] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7538116: READ => -1 
> gfid=e09bffec-f6f4-432f-8f68-ec3520c6f211 fd=0x7f1a94002328 
> (Input/output error)
>
> The message "W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output 
> error]" repeated 3 times between [2017-03-17 19:49:44.398693] and 
> [2017-03-17 19:49:44.401700]
>
> [2017-03-17 19:49:44.401709] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:49:44.402194] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:49:44.402372] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 
> (e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]
>
> [2017-03-17 19:49:44.402627] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 
> (e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]
>
> [2017-03-17 19:49:44.402678] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7538117: READ => -1 
> gfid=e09bffec-f6f4-432f-8f68-ec3520c6f211 fd=0x7f1a94002328 
> (Input/output error)
>
> [2017-03-17 19:49:44.402654] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:50:47.215179] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)
>
> The message "W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output 
> error]" repeated 4 times between [2017-03-17 19:49:44.398446] and 
> [2017-03-17 19:50:47.215449]
>
> [2017-03-17 19:50:47.215623] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:50:47.215659] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:50:47.215974] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)
>
> [2017-03-17 19:50:47.216138] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:50:47.216339] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:50:47.216355] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:50:47.216802] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:50:47.217006] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 
> (e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]
>
> [2017-03-17 19:50:47.217221] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 
> (e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]
>
> [2017-03-17 19:50:47.217254] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:50:47.217298] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7538180: READ => -1 
> gfid=e71812cf-725d-4e95-a471-59cf099055e5 fd=0x7f1a94002440 
> (Input/output error)
>
> [2017-03-17 19:50:47.217382] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)
>
> [2017-03-17 19:50:47.217572] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:50:47.217779] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:50:47.217799] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:50:47.217966] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)
>
> [2017-03-17 19:50:47.218181] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:50:47.218389] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:50:47.218394] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:50:47.218842] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:50:47.219020] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 
> (e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]
>
> [2017-03-17 19:50:47.219200] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 
> (e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]
>
> [2017-03-17 19:50:47.219231] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:50:47.219270] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7538181: READ => -1 
> gfid=e71812cf-725d-4e95-a471-59cf099055e5 fd=0x7f1a94002440 
> (Input/output error)
>
> [2017-03-17 19:49:44.401142] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. 
> [Input/output error]
>
> The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 
> 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with 
> event generation 2 for gfid e09bffec-f6f4-432f-8f68-ec3520c6f211. 
> (Possible split-brain)" repeated 3 times between [2017-03-17 
> 19:49:44.398208] and [2017-03-17 19:49:44.401301]
>
> [2017-03-17 19:51:13.064446] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 2532bb1c-6241-4fd0-92df-30059be2be17. (Possible split-brain)
>
> [2017-03-17 19:51:13.064691] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:51:13.065358] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:51:13.065389] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:51:13.066393] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:51:13.066873] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:51:13.067147] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 
> (2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]
>
> [2017-03-17 19:51:13.067281] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 
> (2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]
>
> [2017-03-17 19:51:13.067366] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:51:13.067416] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7538199: READ => -1 
> gfid=2532bb1c-6241-4fd0-92df-30059be2be17 fd=0x7f1a94002440 
> (Input/output error)
>
> The message "W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output 
> error]" repeated 3 times between [2017-03-17 19:51:13.065358] and 
> [2017-03-17 19:51:13.068580]
>
> [2017-03-17 19:51:13.068586] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:51:13.069092] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:51:13.069276] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 
> (2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]
>
> [2017-03-17 19:51:13.069538] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 
> (2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]
>
> [2017-03-17 19:51:13.069624] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7538200: READ => -1 
> gfid=2532bb1c-6241-4fd0-92df-30059be2be17 fd=0x7f1a94002440 
> (Input/output error)
>
> [2017-03-17 19:51:13.069597] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:52:14.972489] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)
>
> The message "W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output 
> error]" repeated 4 times between [2017-03-17 19:51:13.064691] and 
> [2017-03-17 19:52:14.972716]
>
> [2017-03-17 19:52:14.972967] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:52:14.973008] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:52:14.973861] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)
>
> [2017-03-17 19:52:14.974022] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:52:14.974239] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:52:14.974263] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:52:14.974724] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:52:14.974893] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 
> (5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]
>
> [2017-03-17 19:52:14.975112] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 
> (5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]
>
> [2017-03-17 19:52:14.975146] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a940020f8, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:52:14.975192] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7538225: READ => -1 
> gfid=5752e22b-5e89-4543-b7d7-da2a22af09df fd=0x7f1a940020f8 
> (Input/output error)
>
> [2017-03-17 19:52:14.975266] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)
>
> [2017-03-17 19:52:14.975463] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:52:14.975654] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:52:14.975685] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:52:14.975860] W [MSGID: 108008] 
> [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: 
> Unreadable subvolume -1 found with event generation 2 for gfid 
> 5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)
>
> [2017-03-17 19:52:14.976067] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-1: remote operation failed [Input/output error]
>
> [2017-03-17 19:52:14.976240] W [MSGID: 114031] 
> [client-rpc-fops.c:1494:client3_3_fstat_cbk] 
> 0-Server_Legal_01-client-0: remote operation failed [Input/output error]
>
> [2017-03-17 19:52:14.976248] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 
> 5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. 
> [Input/output error]
>
> [2017-03-17 19:52:14.976742] W [MSGID: 108027] 
> [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: 
> no read subvols for (null)
>
> [2017-03-17 19:52:14.976933] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 
> (5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]
>
> [2017-03-17 19:52:14.977122] E [MSGID: 114031] 
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: 
> remote operation failed. Path: 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 
> (5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]
>
> [2017-03-17 19:52:14.977152] E [MSGID: 109089] 
> [dht-helper.c:1297:dht_migration_complete_check_task] 
> 0-Server_Legal_01-dht: failed to open the fd (0x7f1a940020f8, 
> flags=0100000) on file 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 @ 
> Server_Legal_01-replicate-0 [Input/output error]
>
> [2017-03-17 19:52:14.977193] W [fuse-bridge.c:2228:fuse_readv_cbk] 
> 0-glusterfs-fuse: 7538226: READ => -1 
> gfid=5752e22b-5e89-4543-b7d7-da2a22af09df fd=0x7f1a940020f8 
> (Input/output error)
>
> [2017-03-17 19:51:13.067915] E [MSGID: 108008] 
> [afr-read-txn.c:80:afr_read_txn_refresh_done] 
> 0-Server_Legal_01-replicate-0: Failing READ on gfid 
> 2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. 
> [Input/output error]
>
> The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 
> 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with 
> event generation 2 for gfid 2532bb1c-6241-4fd0-92df-30059be2be17. 
> (Possible split-brain)" repeated 3 times between [2017-03-17 
> 19:51:13.064446] and [2017-03-17 19:51:13.068212]
>
> root at chglbcvtprd04:/var/log/glusterfs#
>
> root at chglbcvtprd04:/var/log/glusterfs# tail -100 
> glfsheal-Server_Legal_01.log
>
> [2017-03-17 19:02:46.687131] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:02:46.778952] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:02:46.779801] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-1
>
> [2017-03-17 19:02:46.780649] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 6368676c-6263-7674-7072-6430342d3334 (0)
>
> [2017-03-17 19:12:50.450406] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 6368676c-6263-7674-7072-6430342d3139 (0) coming up
>
> [2017-03-17 19:12:50.450442] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:12:50.451271] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:12:50.451783] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:12:50.452408] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:12:50.453418] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:12:50.454094] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:12:50.454109] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:12:50.454151] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-0' came back up; going online.
>
> [2017-03-17 19:12:50.454510] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:12:50.459877] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:12:50.460109] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:12:50.460123] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:12:50.594044] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:12:50.594904] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-1
>
> [2017-03-17 19:12:50.595728] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 6368676c-6263-7674-7072-6430342d3139 (0)
>
> [2017-03-17 19:22:39.620877] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 6368676c-6263-7674-7072-6430342d3337 (0) coming up
>
> [2017-03-17 19:22:39.620916] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:22:39.621777] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:22:39.622293] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:22:39.622741] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:22:39.622912] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:22:39.622991] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:22:39.623011] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:22:39.623057] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-1' came back up; going online.
>
> [2017-03-17 19:22:39.623080] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:22:39.623953] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:22:39.625582] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:22:39.625599] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:22:39.721866] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:22:39.722617] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-1
>
> [2017-03-17 19:22:39.723187] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 6368676c-6263-7674-7072-6430342d3337 (0)
>
> [2017-03-17 19:32:36.816124] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 6368676c-6263-7674-7072-6430342d3130 (0) coming up
>
> [2017-03-17 19:32:36.827326] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:32:36.877385] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:32:36.877896] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:32:36.878285] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:32:36.878345] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:32:36.878567] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:32:36.878587] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:32:36.878626] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-1' came back up; going online.
>
> [2017-03-17 19:32:36.878646] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:32:36.879552] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:32:36.892670] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:32:36.892690] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:32:37.030054] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:32:37.030893] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-1
>
> [2017-03-17 19:32:37.031642] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 6368676c-6263-7674-7072-6430342d3130 (0)
>
> [2017-03-17 19:42:49.185410] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 6368676c-6263-7674-7072-6430342d3339 (0) coming up
>
> [2017-03-17 19:42:49.193109] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:42:49.194028] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:42:49.194544] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:42:49.194997] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:42:49.195080] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:42:49.195233] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:42:49.195249] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:42:49.195287] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-1' came back up; going online.
>
> [2017-03-17 19:42:49.195307] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:42:49.196118] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:42:49.196576] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:42:49.196592] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:42:49.359040] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:42:49.359995] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-1
>
> [2017-03-17 19:42:49.360629] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 6368676c-6263-7674-7072-6430342d3339 (0)
>
> [2017-03-17 19:52:42.886903] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 6368676c-6263-7674-7072-6430342d3439 (0) coming up
>
> [2017-03-17 19:52:42.887194] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:52:42.888075] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 19:52:42.888592] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 19:52:42.888974] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 19:52:42.889131] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:52:42.889475] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:52:42.889499] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:52:42.889542] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-1' came back up; going online.
>
> [2017-03-17 19:52:42.889574] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 19:52:42.889891] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 19:52:42.896888] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 19:52:42.896926] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 19:52:43.024347] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 19:52:43.025326] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-1
>
> [2017-03-17 19:52:43.025934] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 6368676c-6263-7674-7072-6430342d3439 (0)
>
> [2017-03-17 20:02:39.027898] I [MSGID: 104045] 
> [glfs-master.c:91:notify] 0-gfapi: New graph 
> 6368676c-6263-7674-7072-6430342d3337 (0) coming up
>
> [2017-03-17 20:02:39.028098] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-0: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 20:02:39.029060] I [MSGID: 114020] [client.c:2356:notify] 
> 0-Server_Legal_01-client-1: parent translators are ready, attempting 
> connect on transport
>
> [2017-03-17 20:02:39.029750] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-1: changing port to 49658 (from 0)
>
> [2017-03-17 20:02:39.029976] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 
> 0-Server_Legal_01-client-0: changing port to 49635 (from 0)
>
> [2017-03-17 20:02:39.030502] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 20:02:39.030780] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 20:02:39.030801] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-1: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 20:02:39.030841] I [MSGID: 108005] 
> [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: 
> Subvolume 'Server_Legal_01-client-1' came back up; going online.
>
> [2017-03-17 20:02:39.030873] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-1: Server lk version = 1
>
> [2017-03-17 20:02:39.030900] I [MSGID: 114057] 
> [client-handshake.c:1447:select_server_supported_programs] 
> 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num 
> (1298437), Version (330)
>
> [2017-03-17 20:02:39.031378] I [MSGID: 114046] 
> [client-handshake.c:1223:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, 
> attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.
>
> [2017-03-17 20:02:39.031392] I [MSGID: 114047] 
> [client-handshake.c:1234:client_setvolume_cbk] 
> 0-Server_Legal_01-client-0: Server and Client lk-version numbers are 
> not same, reopening the fds
>
> [2017-03-17 20:02:39.147618] I [MSGID: 114035] 
> [client-handshake.c:202:client_set_lk_version_cbk] 
> 0-Server_Legal_01-client-0: Server lk version = 1
>
> [2017-03-17 20:02:39.148469] I [MSGID: 108031] 
> [afr-common.c:2152:afr_local_discovery_cbk] 
> 0-Server_Legal_01-replicate-0: selecting local read_child 
> Server_Legal_01-client-1
>
> [2017-03-17 20:02:39.148990] I [MSGID: 104041] 
> [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched 
> to graph 6368676c-6263-7674-7072-6430342d3337 (0)
>
> root at chglbcvtprd04:/var/log/glusterfs#
>
> Healing 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_066 
> failed: File not in split-brain.
>
> Volume heal failed.
>
> Healing 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_034 
> failed: File not in split-brain.
>
> Volume heal failed.
>
> Healing 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293457/SFILE_CONTAINER_021 
> failed: File not in split-brain.
>
> Volume heal failed.
>
> Healing 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_046 
> failed: File not in split-brain.
>
> Volume heal failed.
>
> Healing 
> /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293559/SFILE_CONTAINER_005 
> failed: File not in split-brain.
>
> Volume heal failed.
>
> Healing 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9288763/SFILE_CONTAINER_030 
> failed: File not in split-brain.
>
> Volume heal failed.
>
> Healing 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_014 
> failed: File not in split-brain.
>
> Volume heal failed.
>
> Healing 
> /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9288068/SFILE_CONTAINER_004 
> failed: File not in split-brain.
>
> Volume heal failed.
>
>
>
> Regards
>
> Bernhard
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170320/b4ea825e/attachment-0001.html>