[Gluster-users] glusterfs under high load failing?
Pranith Kumar Karampuri
pkarampu at redhat.com
Wed Nov 5 03:02:42 UTC 2014
On 11/04/2014 10:07 PM, Roman wrote:
> But why it could take too long? :)
because fsync needs to write all the data that is written immediately to
disk which could take order of seconds.
Pranith
>
> 2014-11-04 11:52 GMT+02:00 Pranith Kumar Karampuri
> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>>:
>
>
> On 11/04/2014 03:20 PM, Roman wrote:
>> Hello,
>>
>> some news on this?
> I thought it is working fine for you, after setting the option?
> The delay is happening because fsync takes too long to complete on
> the brick.
>
> Pranith
>>
>> 2014-10-21 12:38 GMT+03:00 Roman <romeo.r at gmail.com
>> <mailto:romeo.r at gmail.com>>:
>>
>> root at stor2:~# gluster volume info HA-testvol-1T
>>
>> Volume Name: HA-testvol-1T
>> Type: Replicate
>> Volume ID: 224a294c-d22a-4b63-8441-2b293527cbab
>> Status: Started
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: stor1:/exports/testvol/1T
>> Brick2: stor2:/exports/testvol/1T
>> Options Reconfigured:
>> diagnostics.count-fop-hits: on
>> diagnostics.latency-measurement: on
>> cluster.ensure-durability: off
>> nfs.disable: 1
>> network.ping-timeout: 10
>>
>>
>> and this of course.
>>
>> 2014-10-21 12:37 GMT+03:00 Roman <romeo.r at gmail.com
>> <mailto:romeo.r at gmail.com>>:
>>
>> Hi,
>> well, this time it was a lot faster and no warnings from VM-s
>>
>> root at glstor-cli:/srv/gfs/HA-testvol-1T# dd if=/dev/zero
>> of=900G-disk bs=2G count=450 iflag=fullblock
>> 450+0 records in
>> 450+0 records out
>> 966367641600 bytes (966 GB) copied, 5292.48 s, 183 MB/s
>>
>> root at stor2:~# gluster volume profile HA-testvol-1T info
>> Brick: stor2:/exports/testvol/1T
>> --------------------------------
>> Cumulative Stats:
>> Block Size: 4b+ 4096b+
>> 65536b+
>> No. of Reads: 0 0
>> 0
>> No. of Writes: 1 4
>> 5
>>
>> Block Size: 131072b+
>> No. of Reads: 0
>> No. of Writes: 7454715
>> %-latency Avg-latency Min-Latency Max-Latency No.
>> of calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 10 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 29 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 70 RELEASEDIR
>> 0.00 115.00 us 115.00 us 115.00 us
>> 1 SETXATTR
>> 0.00 81.50 us 62.00 us 101.00 us
>> 2 SETATTR
>> 0.00 57.44 us 27.00 us 108.00 us
>> 9 STATFS
>> 0.00 32.22 us 10.00 us 76.00 us
>> 18 FSTAT
>> 0.00 40.73 us 13.00 us 64.00 us
>> 15 FLUSH
>> 0.00 48.22 us 19.00 us 166.00 us
>> 18 OPEN
>> 0.00 38.24 us 18.00 us 87.00 us
>> 29 GETXATTR
>> 0.00 43.29 us 22.00 us 146.00 us
>> 42 ENTRYLK
>> 0.00 46.10 us 22.00 us 81.00 us
>> 40 READDIR
>> 0.00 226.36 us 184.00 us 282.00 us
>> 11 CREATE
>> 0.00 68.77 us 27.00 us 119.00 us
>> 70 OPENDIR
>> 0.00 6378.00 us 6378.00 us 6378.00 us
>> 1 FSYNC
>> 0.00 149.32 us 22.00 us 375.00 us
>> 84 READDIRP
>> 0.00 32.07 us 8.00 us 218.00 us
>> 1094 FINODELK
>> 0.03 7272.05 us 12.00 us 699192.00 us
>> 97 INODELK
>> 0.06 4326.87 us 20.00 us 834845.00 us
>> 294 LOOKUP
>> 0.07 150574.00 us 74.00 us 1504668.00 us
>> 10 UNLINK
>> 1.70 51392.90 us 20.00 us 5756079.00 us
>> 704 FXATTROP
>> 98.13 280.28 us 71.00 us 2507482.00 us
>> 7454725 WRITE
>>
>> Duration: 11974 seconds
>> Data Read: 0 bytes
>> Data Written: 977105055751 bytes
>>
>> Interval 1 Stats:
>> Block Size: 4b+ 4096b+
>> 65536b+
>> No. of Reads: 0 0
>> 0
>> No. of Writes: 1 4
>> 5
>>
>> Block Size: 131072b+
>> No. of Reads: 0
>> No. of Writes: 7454715
>> %-latency Avg-latency Min-Latency Max-Latency No.
>> of calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 10 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 29 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 70 RELEASEDIR
>> 0.00 115.00 us 115.00 us 115.00 us
>> 1 SETXATTR
>> 0.00 81.50 us 62.00 us 101.00 us
>> 2 SETATTR
>> 0.00 57.44 us 27.00 us 108.00 us
>> 9 STATFS
>> 0.00 32.22 us 10.00 us 76.00 us
>> 18 FSTAT
>> 0.00 40.73 us 13.00 us 64.00 us
>> 15 FLUSH
>> 0.00 48.22 us 19.00 us 166.00 us
>> 18 OPEN
>> 0.00 38.24 us 18.00 us 87.00 us
>> 29 GETXATTR
>> 0.00 43.29 us 22.00 us 146.00 us
>> 42 ENTRYLK
>> 0.00 46.10 us 22.00 us 81.00 us
>> 40 READDIR
>> 0.00 226.36 us 184.00 us 282.00 us
>> 11 CREATE
>> 0.00 68.77 us 27.00 us 119.00 us
>> 70 OPENDIR
>> 0.00 6378.00 us 6378.00 us 6378.00 us
>> 1 FSYNC
>> 0.00 149.32 us 22.00 us 375.00 us
>> 84 READDIRP
>> 0.00 32.07 us 8.00 us 218.00 us
>> 1094 FINODELK
>> 0.03 7272.05 us 12.00 us 699192.00 us
>> 97 INODELK
>> 0.06 4326.87 us 20.00 us 834845.00 us
>> 294 LOOKUP
>> 0.07 150574.00 us 74.00 us 1504668.00 us
>> 10 UNLINK
>> 1.70 51392.90 us 20.00 us 5756079.00 us
>> 704 FXATTROP
>> 98.13 280.28 us 71.00 us 2507482.00 us
>> 7454725 WRITE
>>
>> Duration: 11948 seconds
>> Data Read: 0 bytes
>> Data Written: 977105055751 bytes
>>
>> Brick: stor1:/exports/testvol/1T
>> --------------------------------
>> Cumulative Stats:
>> Block Size: 4b+ 4096b+
>> 65536b+
>> No. of Reads: 1 0
>> 0
>> No. of Writes: 1 4
>> 5
>>
>> Block Size: 131072b+
>> No. of Reads: 0
>> No. of Writes: 7454715
>> %-latency Avg-latency Min-Latency Max-Latency No.
>> of calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 10 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 29 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 70 RELEASEDIR
>> 0.00 37.00 us 37.00 us 37.00 us
>> 1 STAT
>> 0.00 140.00 us 140.00 us 140.00 us
>> 1 READ
>> 0.00 151.00 us 151.00 us 151.00 us
>> 1 SETXATTR
>> 0.00 87.00 us 76.00 us 98.00 us
>> 2 SETATTR
>> 0.00 33.13 us 13.00 us 65.00 us
>> 15 FLUSH
>> 0.00 28.83 us 12.00 us 97.00 us
>> 18 FSTAT
>> 0.00 66.00 us 26.00 us 104.00 us
>> 9 STATFS
>> 0.00 55.33 us 19.00 us 105.00 us
>> 18 OPEN
>> 0.00 41.55 us 23.00 us 91.00 us
>> 29 GETXATTR
>> 0.00 50.52 us 22.00 us 80.00 us
>> 40 READDIR
>> 0.00 53.12 us 22.00 us 149.00 us
>> 42 ENTRYLK
>> 0.00 238.00 us 187.00 us 283.00 us
>> 11 CREATE
>> 0.00 74.44 us 22.00 us 121.00 us
>> 70 OPENDIR
>> 0.00 6660.00 us 6660.00 us 6660.00 us
>> 1 FSYNC
>> 0.00 98.08 us 20.00 us 281.00 us
>> 294 LOOKUP
>> 0.06 143484.30 us 80.00 us 1433636.00 us
>> 10 UNLINK
>> 0.46 117012.34 us 14.00 us 4468461.00 us
>> 97 INODELK
>> 1.98 74942.61 us 20.00 us 5196144.00 us
>> 646 FXATTROP
>> 3.68 83834.87 us 10.00 us 4469758.00 us
>> 1072 FINODELK
>> 93.82 307.76 us 62.00 us 2507005.00 us
>> 7454725 WRITE
>>
>> Duration: 11972 seconds
>> Data Read: 7 bytes
>> Data Written: 977105055751 bytes
>>
>> Interval 1 Stats:
>> Block Size: 4b+ 4096b+
>> 65536b+
>> No. of Reads: 1 0
>> 0
>> No. of Writes: 1 4
>> 5
>>
>> Block Size: 131072b+
>> No. of Reads: 0
>> No. of Writes: 7454715
>> %-latency Avg-latency Min-Latency Max-Latency No.
>> of calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 10 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 29 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 70 RELEASEDIR
>> 0.00 37.00 us 37.00 us 37.00 us
>> 1 STAT
>> 0.00 140.00 us 140.00 us 140.00 us
>> 1 READ
>> 0.00 151.00 us 151.00 us 151.00 us
>> 1 SETXATTR
>> 0.00 87.00 us 76.00 us 98.00 us
>> 2 SETATTR
>> 0.00 33.13 us 13.00 us 65.00 us
>> 15 FLUSH
>> 0.00 28.83 us 12.00 us 97.00 us
>> 18 FSTAT
>> 0.00 66.00 us 26.00 us 104.00 us
>> 9 STATFS
>> 0.00 55.33 us 19.00 us 105.00 us
>> 18 OPEN
>> 0.00 41.55 us 23.00 us 91.00 us
>> 29 GETXATTR
>> 0.00 50.52 us 22.00 us 80.00 us
>> 40 READDIR
>> 0.00 53.12 us 22.00 us 149.00 us
>> 42 ENTRYLK
>> 0.00 238.00 us 187.00 us 283.00 us
>> 11 CREATE
>> 0.00 74.44 us 22.00 us 121.00 us
>> 70 OPENDIR
>> 0.00 6660.00 us 6660.00 us 6660.00 us
>> 1 FSYNC
>> 0.00 98.08 us 20.00 us 281.00 us
>> 294 LOOKUP
>> 0.06 143484.30 us 80.00 us 1433636.00 us
>> 10 UNLINK
>> 0.46 117012.34 us 14.00 us 4468461.00 us
>> 97 INODELK
>> 1.98 74942.61 us 20.00 us 5196144.00 us
>> 646 FXATTROP
>> 3.68 83834.87 us 10.00 us 4469758.00 us
>> 1072 FINODELK
>> 93.82 307.76 us 62.00 us 2507005.00 us
>> 7454725 WRITE
>>
>> Duration: 11948 seconds
>> Data Read: 7 bytes
>> Data Written: 977105055751 bytes
>>
>> got something interesting? :)
>>
>> 2014-10-21 9:21 GMT+03:00 Roman <romeo.r at gmail.com
>> <mailto:romeo.r at gmail.com>>:
>>
>> Hi,
>>
>> I'm sorry, I had no time to make tests yesterday. Am
>> starting to do them now. So soon I'll reply.
>>
>> 2014-10-18 19:29 GMT+03:00 Roman <romeo.r at gmail.com
>> <mailto:romeo.r at gmail.com>>:
>>
>> This volume is now being tested by my collegue
>> for windows purposes.
>> I will create new one on monday and will test
>> with parameters you've sent me.
>>
>> 2014-10-17 17:36 GMT+03:00 Pranith Kumar
>> Karampuri <pkarampu at redhat.com
>> <mailto:pkarampu at redhat.com>>:
>>
>> Roman,
>> Everything in the logs look okay to me,
>> except the following profile number:
>> 3.91 1255944.81 us 127.00 us
>> 23397532.00 us 189 FSYNC
>>
>> It seems that at least one of the fsyncs is
>> taking almost 23 seconds to complete.
>> According to all the data you gave till now,
>> I feel this is the only thing I feel could
>> have done it. To test this bit, could you
>> turn off the following option using and try
>> again?
>>
>> gluster volume set <volname>
>> cluster.ensure-durability off
>>
>> Let me know what happened. I am extremely
>> curious to here about it.
>>
>> Pranith
>>
>> On 10/17/2014 12:04 PM, Roman wrote:
>>> mount
>>>
>>> [2014-10-13 17:36:56.758654] I
>>> [glusterfsd.c:1959:main]
>>> 0-/usr/sbin/glusterfs: Started running
>>> /usr/sbin/glusterfs version 3.5.2
>>> (/usr/sbin/glusterfs --direct-io-mode=enable
>>> --fuse-mountopts=default_permissions,allow_other,max_read=131072
>>> --volfile-server=stor1
>>> --volfile-server=stor2
>>> --volfile-id=HA-WIN-TT-1T
>>> --fuse-mountopts=default_permissions,allow_other,max_read=131072
>>> /srv/nfs/HA-WIN-TT-1T)
>>> [2014-10-13 17:36:56.762162] I
>>> [socket.c:3561:socket_init] 0-glusterfs: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:36:56.762223] I
>>> [socket.c:3576:socket_init] 0-glusterfs:
>>> using system polling thread
>>> [2014-10-13 17:36:56.766686] I
>>> [dht-shared.c:311:dht_init_regex]
>>> 0-HA-WIN-TT-1T-dht: using regex
>>> rsync-hash-regex = ^\.(.+)\.[^.]+$
>>> [2014-10-13 17:36:56.768887] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-WIN-TT-1T-client-1: SSL support is NOT
>>> enabled
>>> [2014-10-13 17:36:56.768939] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-WIN-TT-1T-client-1: using system
>>> polling thread
>>> [2014-10-13 17:36:56.769280] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-WIN-TT-1T-client-0: SSL support is NOT
>>> enabled
>>> [2014-10-13 17:36:56.769294] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-WIN-TT-1T-client-0: using system
>>> polling thread
>>> [2014-10-13 17:36:56.769336] I
>>> [client.c:2294:notify]
>>> 0-HA-WIN-TT-1T-client-0: parent translators
>>> are ready, attempting connect on transport
>>> [2014-10-13 17:36:56.769829] I
>>> [client.c:2294:notify]
>>> 0-HA-WIN-TT-1T-client-1: parent translators
>>> are ready, attempting connect on transport
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>> 1: volume HA-WIN-TT-1T-client-0
>>> 2: type protocol/client
>>> 3: option remote-host stor1
>>> 4: option remote-subvolume /exports/NFS-WIN/1T
>>> 5: option transport-type socket
>>> 6: option ping-timeout 10
>>> 7: option send-gids true
>>> 8: end-volume
>>> 9:
>>> 10: volume HA-WIN-TT-1T-client-1
>>> 11: type protocol/client
>>> 12: option remote-host stor2
>>> 13: option remote-subvolume /exports/NFS-WIN/1T
>>> 14: option transport-type socket
>>> 15: option ping-timeout 10
>>> 16: option send-gids true
>>> 17: end-volume
>>> 18:
>>> 19: volume HA-WIN-TT-1T-replicate-0
>>> 20: type cluster/replicate
>>> 21: subvolumes HA-WIN-TT-1T-client-0
>>> HA-WIN-TT-1T-client-1
>>> 22: end-volume
>>> 23:
>>> 24: volume HA-WIN-TT-1T-dht
>>> 25: type cluster/distribute
>>> 26: subvolumes HA-WIN-TT-1T-replicate-0
>>> 27: end-volume
>>> 28:
>>> 29: volume HA-WIN-TT-1T-write-behind
>>> 30: type performance/write-behind
>>> 31: subvolumes HA-WIN-TT-1T-dht
>>> 32: end-volume
>>> 33:
>>> 34: volume HA-WIN-TT-1T-read-ahead
>>> 35: type performance/read-ahead
>>> 36: subvolumes HA-WIN-TT-1T-write-behind
>>> 37: end-volume
>>> 38:
>>> 39: volume HA-WIN-TT-1T-io-cache
>>> 40: type performance/io-cache
>>> 41: subvolumes HA-WIN-TT-1T-read-ahead
>>> 42: end-volume
>>> 43:
>>> 44: volume HA-WIN-TT-1T-quick-read
>>> 45: type performance/quick-read
>>> 46: subvolumes HA-WIN-TT-1T-io-cache
>>> 47: end-volume
>>> 48:
>>> 49: volume HA-WIN-TT-1T-open-behind
>>> 50: type performance/open-behind
>>> 51: subvolumes HA-WIN-TT-1T-quick-read
>>> 52: end-volume
>>> 53:
>>> 54: volume HA-WIN-TT-1T-md-cache
>>> 55: type performance/md-cache
>>> 56: subvolumes HA-WIN-TT-1T-open-behind
>>> 57: end-volume
>>> 58:
>>> 59: volume HA-WIN-TT-1T
>>> 60: type debug/io-stats
>>> 61: option latency-measurement off
>>> 62: option count-fop-hits off
>>> 63: subvolumes HA-WIN-TT-1T-md-cache
>>> 64: end-volume
>>> 65:
>>> +------------------------------------------------------------------------------+
>>> [2014-10-13 17:36:56.770718] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-1: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:36:56.771378] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-0: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:36:56.772008] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-1: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:36:56.772083] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-0: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:36:56.772338] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Connected to
>>> 10.250.0.2:49160 <http://10.250.0.2:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:36:56.772361] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:36:56.772424] I
>>> [afr-common.c:4131:afr_notify]
>>> 0-HA-WIN-TT-1T-replicate-0: Subvolume
>>> 'HA-WIN-TT-1T-client-1' came back up; going
>>> online.
>>> [2014-10-13 17:36:56.772463] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Connected to
>>> 10.250.0.1:49160 <http://10.250.0.1:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:36:56.772477] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:36:56.779099] I
>>> [fuse-bridge.c:4977:fuse_graph_setup]
>>> 0-fuse: switched to graph 0
>>> [2014-10-13 17:36:56.779338] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server lk version = 1
>>> [2014-10-13 17:36:56.779367] I
>>> [fuse-bridge.c:3914:fuse_init]
>>> 0-glusterfs-fuse: FUSE inited with protocol
>>> versions: glusterfs 7.22 kernel 7.17
>>> [2014-10-13 17:36:56.779438] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server lk version = 1
>>> [2014-10-13 17:37:02.010942] I
>>> [fuse-bridge.c:4818:fuse_thread_proc]
>>> 0-fuse: unmounting /srv/nfs/HA-WIN-TT-1T
>>> [2014-10-13 17:37:02.011296] W
>>> [glusterfsd.c:1095:cleanup_and_exit]
>>> (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
>>> [0x7fc7b7672e6d]
>>> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)
>>> [0x7fc7b7d20b50]
>>> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5)
>>> [0x7fc7b95add55]))) 0-: received signum
>>> (15), shutting down
>>> [2014-10-13 17:37:02.011316] I
>>> [fuse-bridge.c:5475:fini] 0-fuse: Unmounting
>>> '/srv/nfs/HA-WIN-TT-1T'.
>>> [2014-10-13 17:37:31.133036] W
>>> [socket.c:522:__socket_rwv]
>>> 0-HA-WIN-TT-1T-client-0: readv on
>>> 10.250.0.1:49160 <http://10.250.0.1:49160>
>>> failed (No data available)
>>> [2014-10-13 17:37:31.133110] I
>>> [client.c:2229:client_rpc_notify]
>>> 0-HA-WIN-TT-1T-client-0: disconnected from
>>> 10.250.0.1:49160 <http://10.250.0.1:49160>.
>>> Client process will keep trying to connect
>>> to glusterd until brick's port is available
>>> [2014-10-13 17:37:33.317437] W
>>> [socket.c:522:__socket_rwv]
>>> 0-HA-WIN-TT-1T-client-1: readv on
>>> 10.250.0.2:49160 <http://10.250.0.2:49160>
>>> failed (No data available)
>>> [2014-10-13 17:37:33.317478] I
>>> [client.c:2229:client_rpc_notify]
>>> 0-HA-WIN-TT-1T-client-1: disconnected from
>>> 10.250.0.2:49160 <http://10.250.0.2:49160>.
>>> Client process will keep trying to connect
>>> to glusterd until brick's port is available
>>> [2014-10-13 17:37:33.317496] E
>>> [afr-common.c:4168:afr_notify]
>>> 0-HA-WIN-TT-1T-replicate-0: All subvolumes
>>> are down. Going offline until atleast one of
>>> them comes back up.
>>> [2014-10-13 17:37:42.045604] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-0: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:37:42.046177] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-0: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:37:42.048863] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Connected to
>>> 10.250.0.1:49160 <http://10.250.0.1:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:37:42.048883] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:37:42.048897] I
>>> [client-handshake.c:1314:client_post_handshake]
>>> 0-HA-WIN-TT-1T-client-0: 1 fds open -
>>> Delaying child_up until they are re-opened
>>> [2014-10-13 17:37:42.049299] W
>>> [client-handshake.c:980:client3_3_reopen_cbk] 0-HA-WIN-TT-1T-client-0:
>>> reopen on
>>> <gfid:b00e322a-7bae-479f-91e0-1fd77c73692b>
>>> failed (Stale NFS file handle)
>>> [2014-10-13 17:37:42.049328] I
>>> [client-handshake.c:936:client_child_up_reopen_done]
>>> 0-HA-WIN-TT-1T-client-0: last fd
>>> open'd/lock-self-heal'd - notifying CHILD-UP
>>> [2014-10-13 17:37:42.049360] I
>>> [afr-common.c:4131:afr_notify]
>>> 0-HA-WIN-TT-1T-replicate-0: Subvolume
>>> 'HA-WIN-TT-1T-client-0' came back up; going
>>> online.
>>> [2014-10-13 17:37:42.049446] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server lk version = 1
>>> [2014-10-13 17:37:45.087592] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-1: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:37:45.088132] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-1: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:37:45.088343] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Connected to
>>> 10.250.0.2:49160 <http://10.250.0.2:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:37:45.088360] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:37:45.088373] I
>>> [client-handshake.c:1314:client_post_handshake]
>>> 0-HA-WIN-TT-1T-client-1: 1 fds open -
>>> Delaying child_up until they are re-opened
>>> [2014-10-13 17:37:45.088681] W
>>> [client-handshake.c:980:client3_3_reopen_cbk] 0-HA-WIN-TT-1T-client-1:
>>> reopen on
>>> <gfid:b00e322a-7bae-479f-91e0-1fd77c73692b>
>>> failed (Stale NFS file handle)
>>> [2014-10-13 17:37:45.088697] I
>>> [client-handshake.c:936:client_child_up_reopen_done]
>>> 0-HA-WIN-TT-1T-client-1: last fd
>>> open'd/lock-self-heal'd - notifying CHILD-UP
>>> [2014-10-13 17:37:45.088819] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server lk version = 1
>>> [2014-10-13 17:37:54.601822] I
>>> [glusterfsd.c:1959:main]
>>> 0-/usr/sbin/glusterfs: Started running
>>> /usr/sbin/glusterfs version 3.5.2
>>> (/usr/sbin/glusterfs --direct-io-mode=enable
>>> --fuse-mountopts=default_permissions,allow_other,max_read=131072
>>> --volfile-server=stor1
>>> --volfile-server=stor2
>>> --volfile-id=HA-WIN-TT-1T
>>> --fuse-mountopts=default_permissions,allow_other,max_read=131072
>>> /srv/nfs/HA-WIN-TT-1T)
>>> [2014-10-13 17:37:54.604972] I
>>> [socket.c:3561:socket_init] 0-glusterfs: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:37:54.605034] I
>>> [socket.c:3576:socket_init] 0-glusterfs:
>>> using system polling thread
>>> [2014-10-13 17:37:54.609219] I
>>> [dht-shared.c:311:dht_init_regex]
>>> 0-HA-WIN-TT-1T-dht: using regex
>>> rsync-hash-regex = ^\.(.+)\.[^.]+$
>>> [2014-10-13 17:37:54.611421] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-WIN-TT-1T-client-1: SSL support is NOT
>>> enabled
>>> [2014-10-13 17:37:54.611466] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-WIN-TT-1T-client-1: using system
>>> polling thread
>>> [2014-10-13 17:37:54.611808] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-WIN-TT-1T-client-0: SSL support is NOT
>>> enabled
>>> [2014-10-13 17:37:54.611821] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-WIN-TT-1T-client-0: using system
>>> polling thread
>>> [2014-10-13 17:37:54.611862] I
>>> [client.c:2294:notify]
>>> 0-HA-WIN-TT-1T-client-0: parent translators
>>> are ready, attempting connect on transport
>>> [2014-10-13 17:37:54.612354] I
>>> [client.c:2294:notify]
>>> 0-HA-WIN-TT-1T-client-1: parent translators
>>> are ready, attempting connect on transport
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>> 1: volume HA-WIN-TT-1T-client-0
>>> 2: type protocol/client
>>> 3: option remote-host stor1
>>> 4: option remote-subvolume /exports/NFS-WIN/1T
>>> 5: option transport-type socket
>>> 6: option ping-timeout 10
>>> 7: option send-gids true
>>> 8: end-volume
>>> 9:
>>> 10: volume HA-WIN-TT-1T-client-1
>>> 11: type protocol/client
>>> 12: option remote-host stor2
>>> 13: option remote-subvolume /exports/NFS-WIN/1T
>>> 14: option transport-type socket
>>> 15: option ping-timeout 10
>>> 16: option send-gids true
>>> 17: end-volume
>>> 18:
>>> 19: volume HA-WIN-TT-1T-replicate-0
>>> 20: type cluster/replicate
>>> 21: subvolumes HA-WIN-TT-1T-client-0
>>> HA-WIN-TT-1T-client-1
>>> 22: end-volume
>>> 23:
>>> 24: volume HA-WIN-TT-1T-dht
>>> 25: type cluster/distribute
>>> 26: subvolumes HA-WIN-TT-1T-replicate-0
>>> 27: end-volume
>>> 28:
>>> 29: volume HA-WIN-TT-1T-write-behind
>>> 30: type performance/write-behind
>>> 31: subvolumes HA-WIN-TT-1T-dht
>>> 32: end-volume
>>> 33:
>>> 34: volume HA-WIN-TT-1T-read-ahead
>>> 35: type performance/read-ahead
>>> 36: subvolumes HA-WIN-TT-1T-write-behind
>>> 37: end-volume
>>> 38:
>>> 39: volume HA-WIN-TT-1T-io-cache
>>> 40: type performance/io-cache
>>> 41: subvolumes HA-WIN-TT-1T-read-ahead
>>> 42: end-volume
>>> 43:
>>> 44: volume HA-WIN-TT-1T-quick-read
>>> 45: type performance/quick-read
>>> 46: subvolumes HA-WIN-TT-1T-io-cache
>>> 47: end-volume
>>> 48:
>>> 49: volume HA-WIN-TT-1T-open-behind
>>> 50: type performance/open-behind
>>> 51: subvolumes HA-WIN-TT-1T-quick-read
>>> 52: end-volume
>>> 53:
>>> 54: volume HA-WIN-TT-1T-md-cache
>>> 55: type performance/md-cache
>>> 56: subvolumes HA-WIN-TT-1T-open-behind
>>> 57: end-volume
>>> 58:
>>> 59: volume HA-WIN-TT-1T
>>> 60: type debug/io-stats
>>> 61: option latency-measurement off
>>> 62: option count-fop-hits off
>>> 63: subvolumes HA-WIN-TT-1T-md-cache
>>> 64: end-volume
>>> 65:
>>> +------------------------------------------------------------------------------+
>>> [2014-10-13 17:37:54.613137] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-0: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:37:54.613521] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-1: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:37:54.614228] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-0: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:37:54.614399] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-1: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:37:54.614483] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Connected to
>>> 10.250.0.1:49160 <http://10.250.0.1:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:37:54.614499] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:37:54.614557] I
>>> [afr-common.c:4131:afr_notify]
>>> 0-HA-WIN-TT-1T-replicate-0: Subvolume
>>> 'HA-WIN-TT-1T-client-0' came back up; going
>>> online.
>>> [2014-10-13 17:37:54.614625] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server lk version = 1
>>> [2014-10-13 17:37:54.614709] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Connected to
>>> 10.250.0.2:49160 <http://10.250.0.2:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:37:54.614724] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:37:54.621318] I
>>> [fuse-bridge.c:4977:fuse_graph_setup]
>>> 0-fuse: switched to graph 0
>>> [2014-10-13 17:37:54.621545] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server lk version = 1
>>> [2014-10-13 17:37:54.621617] I
>>> [fuse-bridge.c:3914:fuse_init]
>>> 0-glusterfs-fuse: FUSE inited with protocol
>>> versions: glusterfs 7.22 kernel 7.17
>>> [2014-10-13 17:38:25.951778] W
>>> [client-rpc-fops.c:4235:client3_3_flush]
>>> 0-HA-WIN-TT-1T-client-0:
>>> (b00e322a-7bae-479f-91e0-1fd77c73692b)
>>> remote_fd is -1. EBADFD
>>> [2014-10-13 17:38:25.951827] W
>>> [client-rpc-fops.c:4235:client3_3_flush]
>>> 0-HA-WIN-TT-1T-client-1:
>>> (b00e322a-7bae-479f-91e0-1fd77c73692b)
>>> remote_fd is -1. EBADFD
>>> [2014-10-13 17:38:25.966963] I
>>> [fuse-bridge.c:4818:fuse_thread_proc]
>>> 0-fuse: unmounting /srv/nfs/HA-WIN-TT-1T
>>> [2014-10-13 17:38:25.967174] W
>>> [glusterfsd.c:1095:cleanup_and_exit]
>>> (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
>>> [0x7ffec893de6d]
>>> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)
>>> [0x7ffec8febb50]
>>> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5)
>>> [0x7ffeca878d55]))) 0-: received signum
>>> (15), shutting down
>>> [2014-10-13 17:38:25.967194] I
>>> [fuse-bridge.c:5475:fini] 0-fuse: Unmounting
>>> '/srv/nfs/HA-WIN-TT-1T'.
>>> [2014-10-13 17:40:21.500514] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:40:21.517782] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:40:21.524056] I
>>> [dht-shared.c:311:dht_init_regex]
>>> 0-HA-WIN-TT-1T-dht: using regex
>>> rsync-hash-regex = ^\.(.+)\.[^.]+$
>>> [2014-10-13 17:40:21.528430] I
>>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk]
>>> 0-glusterfs: No change in volfile, continuing
>>>
>>> glusterfshd stor1
>>>
>>> 2014-10-13 17:38:17.203360] I
>>> [glusterfsd.c:1959:main]
>>> 0-/usr/sbin/glusterfs: Started running
>>> /usr/sbin/glusterfs version 3.5.2
>>> (/usr/sbin/glusterfs -s localhost
>>> --volfile-id gluster/glustershd -p
>>> /var/lib/glusterd/glustershd/run/glustershd.pid
>>> -l /var/log/glusterfs/glustershd.log -S
>>> /var/run/75bbc77a676bde0d0afe20f40dc9e3e1.socket
>>> --xlator-option
>>> *replicate*.node-uuid=e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3)
>>> [2014-10-13 17:38:17.204958] I
>>> [socket.c:3561:socket_init]
>>> 0-socket.glusterfsd: SSL support is NOT enabled
>>> [2014-10-13 17:38:17.205016] I
>>> [socket.c:3576:socket_init]
>>> 0-socket.glusterfsd: using system polling thread
>>> [2014-10-13 17:38:17.205188] I
>>> [socket.c:3561:socket_init] 0-glusterfs: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:17.205209] I
>>> [socket.c:3576:socket_init] 0-glusterfs:
>>> using system polling thread
>>> [2014-10-13 17:38:17.207840] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> adding option 'node-uuid' for volume
>>> 'HA-2TB-TT-Proxmox-cluster-replicate-0' with
>>> value 'e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3'
>>> [2014-10-13 17:38:17.209433] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:17.209448] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: using
>>> system polling thread
>>> [2014-10-13 17:38:17.209625] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:17.209634] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: using
>>> system polling thread
>>> [2014-10-13 17:38:17.209652] I
>>> [client.c:2294:notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: parent
>>> translators are ready, attempting connect on
>>> transport
>>> [2014-10-13 17:38:17.210241] I
>>> [client.c:2294:notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: parent
>>> translators are ready, attempting connect on
>>> transport
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>> 1: volume HA-2TB-TT-Proxmox-cluster-client-0
>>> 2: type protocol/client
>>> 3: option remote-host stor1
>>> 4: option remote-subvolume
>>> /exports/HA-2TB-TT-Proxmox-cluster/2TB
>>> 5: option transport-type socket
>>> 6: option username
>>> 59c66122-55c1-4c28-956e-6189fcb1aff5
>>> 7: option password
>>> 34b79afb-a93c-431b-900a-b688e67cdbc9
>>> 8: option ping-timeout 10
>>> 9: end-volume
>>> 10:
>>> 11: volume HA-2TB-TT-Proxmox-cluster-client-1
>>> 12: type protocol/client
>>> 13: option remote-host stor2
>>> 14: option remote-subvolume
>>> /exports/HA-2TB-TT-Proxmox-cluster/2TB
>>> 15: option transport-type socket
>>> 16: option username
>>> 59c66122-55c1-4c28-956e-6189fcb1aff5
>>> 17: option password
>>> 34b79afb-a93c-431b-900a-b688e67cdbc9
>>> 18: option ping-timeout 10
>>> 19: end-volume
>>> 20:
>>> 21: volume
>>> HA-2TB-TT-Proxmox-cluster-replicate-0
>>> 22: type cluster/replicate
>>> 23: option node-uuid
>>> e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
>>> 24: option background-self-heal-count 0
>>> 25: option metadata-self-heal on
>>> 26: option data-self-heal on
>>> 27: option entry-self-heal on
>>> 28: option self-heal-daemon on
>>> 29: option iam-self-heal-daemon yes
>>> 30: subvolumes
>>> HA-2TB-TT-Proxmox-cluster-client-0
>>> HA-2TB-TT-Proxmox-cluster-client-1
>>> 31: end-volume
>>> 32:
>>> 33: volume glustershd
>>> 34: type debug/io-stats
>>> 35: subvolumes
>>> HA-2TB-TT-Proxmox-cluster-replicate-0
>>> 36: end-volume
>>> 37:
>>> +------------------------------------------------------------------------------+
>>> [2014-10-13 17:38:17.210709] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>> changing port to 49159 (from 0)
>>> [2014-10-13 17:38:17.211008] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Using
>>> Program GlusterFS 3.3, Num (1298437),
>>> Version (330)
>>> [2014-10-13 17:38:17.211170] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>> Connected to 10.250.0.1:49159
>>> <http://10.250.0.1:49159>, attached to
>>> remote volume
>>> '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
>>> [2014-10-13 17:38:17.211195] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Server
>>> and Client lk-version numbers are not same,
>>> reopening the fds
>>> [2014-10-13 17:38:17.211250] I
>>> [afr-common.c:4131:afr_notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> Subvolume
>>> 'HA-2TB-TT-Proxmox-cluster-client-0' came
>>> back up; going online.
>>> [2014-10-13 17:38:17.211297] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Server
>>> lk version = 1
>>> [2014-10-13 17:38:17.211656] I
>>> [afr-self-heald.c:1690:afr_dir_exclusive_crawl]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> Another crawl is in progress for
>>> HA-2TB-TT-Proxmox-cluster-client-0
>>> [2014-10-13 17:38:17.211661] E
>>> [afr-self-heald.c:1479:afr_find_child_position]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> getxattr failed on
>>> HA-2TB-TT-Proxmox-cluster-client-1 -
>>> (Transport endpoint is not connected)
>>> [2014-10-13 17:38:17.216327] E
>>> [afr-self-heal-data.c:1611:afr_sh_data_open_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> open of
>>> <gfid:65381af4-8e0b-4721-8214-71d29dcf5237>
>>> failed on child
>>> HA-2TB-TT-Proxmox-cluster-client-1
>>> (Transport endpoint is not connected)
>>> [2014-10-13 17:38:17.217372] E
>>> [afr-self-heal-data.c:1611:afr_sh_data_open_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> open of
>>> <gfid:65381af4-8e0b-4721-8214-71d29dcf5237>
>>> failed on child
>>> HA-2TB-TT-Proxmox-cluster-client-1
>>> (Transport endpoint is not connected)
>>> [2014-10-13 17:38:19.226057] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1:
>>> changing port to 49159 (from 0)
>>> [2014-10-13 17:38:19.226704] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Using
>>> Program GlusterFS 3.3, Num (1298437),
>>> Version (330)
>>> [2014-10-13 17:38:19.226896] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1:
>>> Connected to 10.250.0.2:49159
>>> <http://10.250.0.2:49159>, attached to
>>> remote volume
>>> '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
>>> [2014-10-13 17:38:19.226916] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Server
>>> and Client lk-version numbers are not same,
>>> reopening the fds
>>> [2014-10-13 17:38:19.227031] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Server
>>> lk version = 1
>>> [2014-10-13 17:38:25.933950] W
>>> [glusterfsd.c:1095:cleanup_and_exit]
>>> (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
>>> [0x7f1a7c03ce6d]
>>> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)
>>> [0x7f1a7c6eab50]
>>> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5)
>>> [0x7f1a7df77d55]))) 0-: received signum
>>> (15), shutting down
>>> [2014-10-13 17:38:26.942918] I
>>> [glusterfsd.c:1959:main]
>>> 0-/usr/sbin/glusterfs: Started running
>>> /usr/sbin/glusterfs version 3.5.2
>>> (/usr/sbin/glusterfs -s localhost
>>> --volfile-id gluster/glustershd -p
>>> /var/lib/glusterd/glustershd/run/glustershd.pid
>>> -l /var/log/glusterfs/glustershd.log -S
>>> /var/run/75bbc77a676bde0d0afe20f40dc9e3e1.socket
>>> --xlator-option
>>> *replicate*.node-uuid=e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3)
>>> [2014-10-13 17:38:26.944548] I
>>> [socket.c:3561:socket_init]
>>> 0-socket.glusterfsd: SSL support is NOT enabled
>>> [2014-10-13 17:38:26.944584] I
>>> [socket.c:3576:socket_init]
>>> 0-socket.glusterfsd: using system polling thread
>>> [2014-10-13 17:38:26.944689] I
>>> [socket.c:3561:socket_init] 0-glusterfs: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:26.944701] I
>>> [socket.c:3576:socket_init] 0-glusterfs:
>>> using system polling thread
>>> [2014-10-13 17:38:26.946667] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> adding option 'node-uuid' for volume
>>> 'HA-2TB-TT-Proxmox-cluster-replicate-0' with
>>> value 'e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3'
>>> [2014-10-13 17:38:26.946684] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-WIN-TT-1T-replicate-0: adding option
>>> 'node-uuid' for volume
>>> 'HA-WIN-TT-1T-replicate-0' with value
>>> 'e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3'
>>> [2014-10-13 17:38:26.948783] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:26.948809] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: using
>>> system polling thread
>>> [2014-10-13 17:38:26.949118] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:26.949134] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: using
>>> system polling thread
>>> [2014-10-13 17:38:26.951698] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-WIN-TT-1T-client-1: SSL support is NOT
>>> enabled
>>> [2014-10-13 17:38:26.951715] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-WIN-TT-1T-client-1: using system
>>> polling thread
>>> [2014-10-13 17:38:26.951921] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-WIN-TT-1T-client-0: SSL support is NOT
>>> enabled
>>> [2014-10-13 17:38:26.951932] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-WIN-TT-1T-client-0: using system
>>> polling thread
>>> [2014-10-13 17:38:26.951959] I
>>> [client.c:2294:notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: parent
>>> translators are ready, attempting connect on
>>> transport
>>> [2014-10-13 17:38:26.952612] I
>>> [client.c:2294:notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: parent
>>> translators are ready, attempting connect on
>>> transport
>>> [2014-10-13 17:38:26.952862] I
>>> [client.c:2294:notify]
>>> 0-HA-WIN-TT-1T-client-0: parent translators
>>> are ready, attempting connect on transport
>>> [2014-10-13 17:38:26.953447] I
>>> [client.c:2294:notify]
>>> 0-HA-WIN-TT-1T-client-1: parent translators
>>> are ready, attempting connect on transport
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>> 1: volume HA-2TB-TT-Proxmox-cluster-client-0
>>> 2: type protocol/client
>>> 3: option remote-host stor1
>>> 4: option remote-subvolume
>>> /exports/HA-2TB-TT-Proxmox-cluster/2TB
>>> 5: option transport-type socket
>>> 6: option username
>>> 59c66122-55c1-4c28-956e-6189fcb1aff5
>>> 7: option password
>>> 34b79afb-a93c-431b-900a-b688e67cdbc9
>>> 8: option ping-timeout 10
>>> 9: end-volume
>>> 10:
>>> 11: volume HA-2TB-TT-Proxmox-cluster-client-1
>>> 12: type protocol/client
>>> 13: option remote-host stor2
>>> 14: option remote-subvolume
>>> /exports/HA-2TB-TT-Proxmox-cluster/2TB
>>> 15: option transport-type socket
>>> 16: option username
>>> 59c66122-55c1-4c28-956e-6189fcb1aff5
>>> 17: option password
>>> 34b79afb-a93c-431b-900a-b688e67cdbc9
>>> 18: option ping-timeout 10
>>> 19: end-volume
>>> 20:
>>> 21: volume
>>> HA-2TB-TT-Proxmox-cluster-replicate-0
>>> 22: type cluster/replicate
>>> 23: option node-uuid
>>> e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
>>> 24: option background-self-heal-count 0
>>> 25: option metadata-self-heal on
>>> 26: option data-self-heal on
>>> 27: option entry-self-heal on
>>> 28: option self-heal-daemon on
>>> 29: option iam-self-heal-daemon yes
>>> 30: subvolumes
>>> HA-2TB-TT-Proxmox-cluster-client-0
>>> HA-2TB-TT-Proxmox-cluster-client-1
>>> 31: end-volume
>>> 32:
>>> 33: volume HA-WIN-TT-1T-client-0
>>> 34: type protocol/client
>>> 35: option remote-host stor1
>>> 36: option remote-subvolume /exports/NFS-WIN/1T
>>> 37: option transport-type socket
>>> 38: option username
>>> 101b907c-ff21-47da-8ba6-37e2920691ce
>>> 39: option password
>>> f4f29094-891f-4241-8736-5e3302ed8bc8
>>> 40: option ping-timeout 10
>>> 41: end-volume
>>> 42:
>>> 43: volume HA-WIN-TT-1T-client-1
>>> 44: type protocol/client
>>> 45: option remote-host stor2
>>> 46: option remote-subvolume /exports/NFS-WIN/1T
>>> 47: option transport-type socket
>>> 48: option username
>>> 101b907c-ff21-47da-8ba6-37e2920691ce
>>> 49: option password
>>> f4f29094-891f-4241-8736-5e3302ed8bc8
>>> 50: option ping-timeout 10
>>> 51: end-volume
>>> 52:
>>> 53: volume HA-WIN-TT-1T-replicate-0
>>> 54: type cluster/replicate
>>> 55: option node-uuid
>>> e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
>>> 56: option background-self-heal-count 0
>>> 57: option metadata-self-heal on
>>> 58: option data-self-heal on
>>> 59: option entry-self-heal on
>>> 60: option self-heal-daemon on
>>> 61: option iam-self-heal-daemon yes
>>> 62: subvolumes HA-WIN-TT-1T-client-0
>>> HA-WIN-TT-1T-client-1
>>> 63: end-volume
>>> 64:
>>> 65: volume glustershd
>>> 66: type debug/io-stats
>>> 67: subvolumes
>>> HA-2TB-TT-Proxmox-cluster-replicate-0
>>> HA-WIN-TT-1T-replicate-0
>>> 68: end-volume
>>> 69:
>>> +------------------------------------------------------------------------------+
>>> [2014-10-13 17:38:26.954036] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>> changing port to 49159 (from 0)
>>> [2014-10-13 17:38:26.954308] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-0: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:38:26.954741] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Using
>>> Program GlusterFS 3.3, Num (1298437),
>>> Version (330)
>>> [2014-10-13 17:38:26.954815] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-0: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:38:26.954999] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>> Connected to 10.250.0.1:49159
>>> <http://10.250.0.1:49159>, attached to
>>> remote volume
>>> '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
>>> [2014-10-13 17:38:26.955017] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Server
>>> and Client lk-version numbers are not same,
>>> reopening the fds
>>> [2014-10-13 17:38:26.955073] I
>>> [afr-common.c:4131:afr_notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> Subvolume
>>> 'HA-2TB-TT-Proxmox-cluster-client-0' came
>>> back up; going online.
>>> [2014-10-13 17:38:26.955127] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Server
>>> lk version = 1
>>> [2014-10-13 17:38:26.955151] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Connected to
>>> 10.250.0.1:49160 <http://10.250.0.1:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:38:26.955161] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:38:26.955226] I
>>> [afr-common.c:4131:afr_notify]
>>> 0-HA-WIN-TT-1T-replicate-0: Subvolume
>>> 'HA-WIN-TT-1T-client-0' came back up; going
>>> online.
>>> [2014-10-13 17:38:26.955297] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server lk version = 1
>>> [2014-10-13 17:38:26.955583] I
>>> [afr-self-heald.c:1690:afr_dir_exclusive_crawl]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> Another crawl is in progress for
>>> HA-2TB-TT-Proxmox-cluster-client-0
>>> [2014-10-13 17:38:26.955589] E
>>> [afr-self-heald.c:1479:afr_find_child_position]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> getxattr failed on
>>> HA-2TB-TT-Proxmox-cluster-client-1 -
>>> (Transport endpoint is not connected)
>>> [2014-10-13 17:38:26.955832] I
>>> [afr-self-heald.c:1690:afr_dir_exclusive_crawl]
>>> 0-HA-WIN-TT-1T-replicate-0: Another crawl is
>>> in progress for HA-WIN-TT-1T-client-0
>>> [2014-10-13 17:38:26.955858] E
>>> [afr-self-heald.c:1479:afr_find_child_position]
>>> 0-HA-WIN-TT-1T-replicate-0: getxattr failed
>>> on HA-WIN-TT-1T-client-1 - (Transport
>>> endpoint is not connected)
>>> [2014-10-13 17:38:26.964913] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1:
>>> changing port to 49159 (from 0)
>>> [2014-10-13 17:38:26.965553] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Using
>>> Program GlusterFS 3.3, Num (1298437),
>>> Version (330)
>>> [2014-10-13 17:38:26.965794] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1:
>>> Connected to 10.250.0.2:49159
>>> <http://10.250.0.2:49159>, attached to
>>> remote volume
>>> '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
>>> [2014-10-13 17:38:26.965815] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Server
>>> and Client lk-version numbers are not same,
>>> reopening the fds
>>> [2014-10-13 17:38:26.965968] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Server
>>> lk version = 1
>>> [2014-10-13 17:38:26.967510] I
>>> [afr-self-heald.c:1690:afr_dir_exclusive_crawl]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> Another crawl is in progress for
>>> HA-2TB-TT-Proxmox-cluster-client-0
>>> [2014-10-13 17:38:27.971374] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-1: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:38:27.971940] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-1: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:38:27.975460] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Connected to
>>> 10.250.0.2:49160 <http://10.250.0.2:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:38:27.975481] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:38:27.976656] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server lk version = 1
>>> [2014-10-13 17:41:05.390992] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:41:05.408292] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:41:05.412221] I
>>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk]
>>> 0-glusterfs: No change in volfile, continuing
>>> [2014-10-13 17:41:05.417388] I
>>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk]
>>> 0-glusterfs: No change in volfile, continuing
>>> root at stor1:~#
>>>
>>> glusterfshd stor2
>>>
>>> [2014-10-13 17:38:28.992891] I
>>> [glusterfsd.c:1959:main]
>>> 0-/usr/sbin/glusterfs: Started running
>>> /usr/sbin/glusterfs version 3.5.2
>>> (/usr/sbin/glusterfs -s localhost
>>> --volfile-id gluster/glustershd -p
>>> /var/lib/glusterd/glustershd/run/glustershd.pid
>>> -l /var/log/glusterfs/glustershd.log -S
>>> /var/run/b1494ca4d047df6e8590d7080131908f.socket
>>> --xlator-option
>>> *replicate*.node-uuid=abf9e3a7-eb91-4273-acdf-876cd6ba1fe3)
>>> [2014-10-13 17:38:28.994439] I
>>> [socket.c:3561:socket_init]
>>> 0-socket.glusterfsd: SSL support is NOT enabled
>>> [2014-10-13 17:38:28.994476] I
>>> [socket.c:3576:socket_init]
>>> 0-socket.glusterfsd: using system polling thread
>>> [2014-10-13 17:38:28.994581] I
>>> [socket.c:3561:socket_init] 0-glusterfs: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:28.994594] I
>>> [socket.c:3576:socket_init] 0-glusterfs:
>>> using system polling thread
>>> [2014-10-13 17:38:28.996569] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> adding option 'node-uuid' for volume
>>> 'HA-2TB-TT-Proxmox-cluster-replicate-0' with
>>> value 'abf9e3a7-eb91-4273-acdf-876cd6ba1fe3'
>>> [2014-10-13 17:38:28.996585] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-WIN-TT-1T-replicate-0: adding option
>>> 'node-uuid' for volume
>>> 'HA-WIN-TT-1T-replicate-0' with value
>>> 'abf9e3a7-eb91-4273-acdf-876cd6ba1fe3'
>>> [2014-10-13 17:38:28.998463] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:28.998483] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: using
>>> system polling thread
>>> [2014-10-13 17:38:28.998695] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:28.998707] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: using
>>> system polling thread
>>> [2014-10-13 17:38:29.000506] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-WIN-TT-1T-client-1: SSL support is NOT
>>> enabled
>>> [2014-10-13 17:38:29.000520] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-WIN-TT-1T-client-1: using system
>>> polling thread
>>> [2014-10-13 17:38:29.000723] I
>>> [socket.c:3561:socket_init]
>>> 0-HA-WIN-TT-1T-client-0: SSL support is NOT
>>> enabled
>>> [2014-10-13 17:38:29.000734] I
>>> [socket.c:3576:socket_init]
>>> 0-HA-WIN-TT-1T-client-0: using system
>>> polling thread
>>> [2014-10-13 17:38:29.000762] I
>>> [client.c:2294:notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: parent
>>> translators are ready, attempting connect on
>>> transport
>>> [2014-10-13 17:38:29.001064] I
>>> [client.c:2294:notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: parent
>>> translators are ready, attempting connect on
>>> transport
>>> [2014-10-13 17:38:29.001639] I
>>> [client.c:2294:notify]
>>> 0-HA-WIN-TT-1T-client-0: parent translators
>>> are ready, attempting connect on transport
>>> [2014-10-13 17:38:29.001877] I
>>> [client.c:2294:notify]
>>> 0-HA-WIN-TT-1T-client-1: parent translators
>>> are ready, attempting connect on transport
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>> 1: volume HA-2TB-TT-Proxmox-cluster-client-0
>>> 2: type protocol/client
>>> 3: option remote-host stor1
>>> 4: option remote-subvolume
>>> /exports/HA-2TB-TT-Proxmox-cluster/2TB
>>> 5: option transport-type socket
>>> 6: option username
>>> 59c66122-55c1-4c28-956e-6189fcb1aff5
>>> 7: option password
>>> 34b79afb-a93c-431b-900a-b688e67cdbc9
>>> 8: option ping-timeout 10
>>> 9: end-volume
>>> 10:
>>> 11: volume HA-2TB-TT-Proxmox-cluster-client-1
>>> 12: type protocol/client
>>> 13: option remote-host stor2
>>> 14: option remote-subvolume
>>> /exports/HA-2TB-TT-Proxmox-cluster/2TB
>>> 15: option transport-type socket
>>> 16: option username
>>> 59c66122-55c1-4c28-956e-6189fcb1aff5
>>> 17: option password
>>> 34b79afb-a93c-431b-900a-b688e67cdbc9
>>> 18: option ping-timeout 10
>>> 19: end-volume
>>> 20:
>>> 21: volume
>>> HA-2TB-TT-Proxmox-cluster-replicate-0
>>> 22: type cluster/replicate
>>> 23: option node-uuid
>>> abf9e3a7-eb91-4273-acdf-876cd6ba1fe3
>>> 24: option background-self-heal-count 0
>>> 25: option metadata-self-heal on
>>> 26: option data-self-heal on
>>> 27: option entry-self-heal on
>>> 28: option self-heal-daemon on
>>> 29: option iam-self-heal-daemon yes
>>> 30: subvolumes
>>> HA-2TB-TT-Proxmox-cluster-client-0
>>> HA-2TB-TT-Proxmox-cluster-client-1
>>> 31: end-volume
>>> 32:
>>> 33: volume HA-WIN-TT-1T-client-0
>>> 34: type protocol/client
>>> 35: option remote-host stor1
>>> 36: option remote-subvolume /exports/NFS-WIN/1T
>>> 37: option transport-type socket
>>> 38: option username
>>> 101b907c-ff21-47da-8ba6-37e2920691ce
>>> 39: option password
>>> f4f29094-891f-4241-8736-5e3302ed8bc8
>>> 40: option ping-timeout 10
>>> 41: end-volume
>>> 42:
>>> 43: volume HA-WIN-TT-1T-client-1
>>> 44: type protocol/client
>>> 45: option remote-host stor2
>>> 46: option remote-subvolume /exports/NFS-WIN/1T
>>> 47: option transport-type socket
>>> 48: option username
>>> 101b907c-ff21-47da-8ba6-37e2920691ce
>>> 49: option password
>>> f4f29094-891f-4241-8736-5e3302ed8bc8
>>> 50: option ping-timeout 10
>>> 51: end-volume
>>> 52:
>>> 53: volume HA-WIN-TT-1T-replicate-0
>>> 54: type cluster/replicate
>>> 55: option node-uuid
>>> abf9e3a7-eb91-4273-acdf-876cd6ba1fe3
>>> 56: option background-self-heal-count 0
>>> 57: option metadata-self-heal on
>>> 58: option data-self-heal on
>>> 59: option entry-self-heal on
>>> 60: option self-heal-daemon on
>>> 61: option iam-self-heal-daemon yes
>>> 62: subvolumes HA-WIN-TT-1T-client-0
>>> HA-WIN-TT-1T-client-1
>>> 63: end-volume
>>> 64:
>>> 65: volume glustershd
>>> 66: type debug/io-stats
>>> 67: subvolumes
>>> HA-2TB-TT-Proxmox-cluster-replicate-0
>>> HA-WIN-TT-1T-replicate-0
>>> 68: end-volume
>>> 69:
>>> +------------------------------------------------------------------------------+
>>> [2014-10-13 17:38:29.002743] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1:
>>> changing port to 49159 (from 0)
>>> [2014-10-13 17:38:29.003027] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-1: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:38:29.003290] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>> changing port to 49159 (from 0)
>>> [2014-10-13 17:38:29.003334] I
>>> [rpc-clnt.c:1729:rpc_clnt_reconfig]
>>> 0-HA-WIN-TT-1T-client-0: changing port to
>>> 49160 (from 0)
>>> [2014-10-13 17:38:29.003922] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Using
>>> Program GlusterFS 3.3, Num (1298437),
>>> Version (330)
>>> [2014-10-13 17:38:29.004023] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-1: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:38:29.004139] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Using
>>> Program GlusterFS 3.3, Num (1298437),
>>> Version (330)
>>> [2014-10-13 17:38:29.004202] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1:
>>> Connected to 10.250.0.2:49159
>>> <http://10.250.0.2:49159>, attached to
>>> remote volume
>>> '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
>>> [2014-10-13 17:38:29.004217] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Server
>>> and Client lk-version numbers are not same,
>>> reopening the fds
>>> [2014-10-13 17:38:29.004266] I
>>> [afr-common.c:4131:afr_notify]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> Subvolume
>>> 'HA-2TB-TT-Proxmox-cluster-client-1' came
>>> back up; going online.
>>> [2014-10-13 17:38:29.004318] I
>>> [client-handshake.c:1677:select_server_supported_programs]
>>> 0-HA-WIN-TT-1T-client-0: Using Program
>>> GlusterFS 3.3, Num (1298437), Version (330)
>>> [2014-10-13 17:38:29.004368] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Connected to
>>> 10.250.0.2:49160 <http://10.250.0.2:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:38:29.004383] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:38:29.004429] I
>>> [afr-common.c:4131:afr_notify]
>>> 0-HA-WIN-TT-1T-replicate-0: Subvolume
>>> 'HA-WIN-TT-1T-client-1' came back up; going
>>> online.
>>> [2014-10-13 17:38:29.004483] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-1: Server
>>> lk version = 1
>>> [2014-10-13 17:38:29.004506] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-1: Server lk version = 1
>>> [2014-10-13 17:38:29.004526] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>> Connected to 10.250.0.1:49159
>>> <http://10.250.0.1:49159>, attached to
>>> remote volume
>>> '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
>>> [2014-10-13 17:38:29.004535] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Server
>>> and Client lk-version numbers are not same,
>>> reopening the fds
>>> [2014-10-13 17:38:29.004613] I
>>> [client-handshake.c:1462:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Connected to
>>> 10.250.0.1:49160 <http://10.250.0.1:49160>,
>>> attached to remote volume '/exports/NFS-WIN/1T'.
>>> [2014-10-13 17:38:29.004626] I
>>> [client-handshake.c:1474:client_setvolume_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server and Client
>>> lk-version numbers are not same, reopening
>>> the fds
>>> [2014-10-13 17:38:29.004731] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-2TB-TT-Proxmox-cluster-client-0: Server
>>> lk version = 1
>>> [2014-10-13 17:38:29.004796] I
>>> [client-handshake.c:450:client_set_lk_version_cbk]
>>> 0-HA-WIN-TT-1T-client-0: Server lk version = 1
>>> [2014-10-13 17:38:29.005291] I
>>> [afr-self-heald.c:1690:afr_dir_exclusive_crawl]
>>> 0-HA-WIN-TT-1T-replicate-0: Another crawl is
>>> in progress for HA-WIN-TT-1T-client-1
>>> [2014-10-13 17:38:29.005303] I
>>> [afr-self-heald.c:1690:afr_dir_exclusive_crawl]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> Another crawl is in progress for
>>> HA-2TB-TT-Proxmox-cluster-client-1
>>> [2014-10-13 17:38:29.005443] I
>>> [afr-self-heald.c:1690:afr_dir_exclusive_crawl]
>>> 0-HA-2TB-TT-Proxmox-cluster-replicate-0:
>>> Another crawl is in progress for
>>> HA-2TB-TT-Proxmox-cluster-client-1
>>> [2014-10-13 17:41:05.427867] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:41:05.443271] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:41:05.444111] I
>>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk]
>>> 0-glusterfs: No change in volfile, continuing
>>> [2014-10-13 17:41:05.444807] I
>>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk]
>>> 0-glusterfs: No change in volfile, continuing
>>>
>>> brick stor2
>>>
>>> [2014-10-13 17:38:17.213386] W
>>> [glusterfsd.c:1095:cleanup_and_exit]
>>> (-->/lib/x86_64-linux-gnu/libc.so.6(+0x462a0) [0x7f343271f2a0]
>>> (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_wrap+0x12)
>>> [0x7f343371db12]
>>> (-->/usr/sbin/glusterfsd(glusterfs_handle_terminate+0x15)
>>> [0x7f3434790dd5]))) 0-: received signum
>>> (15), shutting down
>>> [2014-10-13 17:38:26.957312] I
>>> [glusterfsd.c:1959:main]
>>> 0-/usr/sbin/glusterfsd: Started running
>>> /usr/sbin/glusterfsd version 3.5.2
>>> (/usr/sbin/glusterfsd -s stor2 --volfile-id
>>> HA-WIN-TT-1T.stor2.exports-NFS-WIN-1T -p
>>> /var/lib/glusterd/vols/HA-WIN-TT-1T/run/stor2-exports-NFS-WIN-1T.pid
>>> -S
>>> /var/run/91514691033d00e666bb151f9c771a26.socket
>>> --brick-name /exports/NFS-WIN/1T -l
>>> /var/log/glusterfs/bricks/exports-NFS-WIN-1T.log
>>> --xlator-option
>>> *-posix.glusterd-uuid=abf9e3a7-eb91-4273-acdf-876cd6ba1fe3
>>> --brick-port 49160 --xlator-option
>>> HA-WIN-TT-1T-server.listen-port=49160)
>>> [2014-10-13 17:38:26.958864] I
>>> [socket.c:3561:socket_init]
>>> 0-socket.glusterfsd: SSL support is NOT enabled
>>> [2014-10-13 17:38:26.958899] I
>>> [socket.c:3576:socket_init]
>>> 0-socket.glusterfsd: using system polling thread
>>> [2014-10-13 17:38:26.959003] I
>>> [socket.c:3561:socket_init] 0-glusterfs: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:26.959015] I
>>> [socket.c:3576:socket_init] 0-glusterfs:
>>> using system polling thread
>>> [2014-10-13 17:38:26.961860] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-WIN-TT-1T-server: adding option
>>> 'listen-port' for volume
>>> 'HA-WIN-TT-1T-server' with value '49160'
>>> [2014-10-13 17:38:26.961878] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-WIN-TT-1T-posix: adding option
>>> 'glusterd-uuid' for volume
>>> 'HA-WIN-TT-1T-posix' with value
>>> 'abf9e3a7-eb91-4273-acdf-876cd6ba1fe3'
>>> [2014-10-13 17:38:26.965032] I
>>> [rpcsvc.c:2127:rpcsvc_set_outstanding_rpc_limit]
>>> 0-rpc-service: Configured
>>> rpc.outstanding-rpc-limit with value 64
>>> [2014-10-13 17:38:26.965075] W
>>> [options.c:888:xl_opt_validate]
>>> 0-HA-WIN-TT-1T-server: option 'listen-port'
>>> is deprecated, preferred is
>>> 'transport.socket.listen-port', continuing
>>> with correction
>>> [2014-10-13 17:38:26.965097] I
>>> [socket.c:3561:socket_init]
>>> 0-tcp.HA-WIN-TT-1T-server: SSL support is
>>> NOT enabled
>>> [2014-10-13 17:38:26.965105] I
>>> [socket.c:3576:socket_init]
>>> 0-tcp.HA-WIN-TT-1T-server: using system
>>> polling thread
>>> [2014-10-13 17:38:26.965602] W
>>> [graph.c:329:_log_if_unknown_option]
>>> 0-HA-WIN-TT-1T-quota: option 'timeout' is
>>> not recognized
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>> 1: volume HA-WIN-TT-1T-posix
>>> 2: type storage/posix
>>> 3: option glusterd-uuid
>>> abf9e3a7-eb91-4273-acdf-876cd6ba1fe3
>>> 4: option directory /exports/NFS-WIN/1T
>>> 5: option volume-id
>>> 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
>>> 6: end-volume
>>> 7:
>>> 8: volume HA-WIN-TT-1T-changelog
>>> 9: type features/changelog
>>> 10: option changelog-brick /exports/NFS-WIN/1T
>>> 11: option changelog-dir
>>> /exports/NFS-WIN/1T/.glusterfs/changelogs
>>> 12: subvolumes HA-WIN-TT-1T-posix
>>> 13: end-volume
>>> 14:
>>> 15: volume HA-WIN-TT-1T-access-control
>>> 16: type features/access-control
>>> 17: subvolumes HA-WIN-TT-1T-changelog
>>> 18: end-volume
>>> 19:
>>> 20: volume HA-WIN-TT-1T-locks
>>> 21: type features/locks
>>> 22: subvolumes HA-WIN-TT-1T-access-control
>>> 23: end-volume
>>> 24:
>>> 25: volume HA-WIN-TT-1T-io-threads
>>> 26: type performance/io-threads
>>> 27: subvolumes HA-WIN-TT-1T-locks
>>> 28: end-volume
>>> 29:
>>> 30: volume HA-WIN-TT-1T-index
>>> 31: type features/index
>>> 32: option index-base
>>> /exports/NFS-WIN/1T/.glusterfs/indices
>>> 33: subvolumes HA-WIN-TT-1T-io-threads
>>> 34: end-volume
>>> 35:
>>> 36: volume HA-WIN-TT-1T-marker
>>> 37: type features/marker
>>> 38: option volume-uuid
>>> 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
>>> 39: option timestamp-file
>>> /var/lib/glusterd/vols/HA-WIN-TT-1T/marker.tstamp
>>> 40: option xtime off
>>> 41: option gsync-force-xtime off
>>> 42: option quota off
>>> 43: subvolumes HA-WIN-TT-1T-index
>>> 44: end-volume
>>> 45:
>>> 46: volume HA-WIN-TT-1T-quota
>>> 47: type features/quota
>>> 48: option volume-uuid HA-WIN-TT-1T
>>> 49: option server-quota off
>>> 50: option timeout 0
>>> 51: option deem-statfs off
>>> 52: subvolumes HA-WIN-TT-1T-marker
>>> 53: end-volume
>>> 54:
>>> 55: volume /exports/NFS-WIN/1T
>>> 56: type debug/io-stats
>>> 57: option latency-measurement off
>>> 58: option count-fop-hits off
>>> 59: subvolumes HA-WIN-TT-1T-quota
>>> 60: end-volume
>>> 61:
>>> 62: volume HA-WIN-TT-1T-server
>>> 63: type protocol/server
>>> 64: option transport.socket.listen-port 49160
>>> 65: option rpc-auth.auth-glusterfs on
>>> 66: option rpc-auth.auth-unix on
>>> 67: option rpc-auth.auth-null on
>>> 68: option transport-type tcp
>>> 69: option
>>> auth.login./exports/NFS-WIN/1T.allow
>>> 101b907c-ff21-47da-8ba6-37e2920691ce
>>> 70: option
>>> auth.login.101b907c-ff21-47da-8ba6-37e2920691ce.password
>>> f4f29094-891f-4241-8736-5e3302ed8bc8
>>> 71: option
>>> auth.addr./exports/NFS-WIN/1T.allow *
>>> 72: subvolumes /exports/NFS-WIN/1T
>>> 73: end-volume
>>> 74:
>>> +------------------------------------------------------------------------------+
>>> [2014-10-13 17:38:27.985048] I
>>> [server-handshake.c:575:server_setvolume]
>>> 0-HA-WIN-TT-1T-server: accepted client from
>>> stor1-14362-2014/10/13-17:38:26:938194-HA-WIN-TT-1T-client-1-0-0
>>> (version: 3.5.2)
>>> [2014-10-13 17:38:28.988700] I
>>> [server-handshake.c:575:server_setvolume]
>>> 0-HA-WIN-TT-1T-server: accepted client from
>>> glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-1-0-1
>>> (version: 3.5.2)
>>> [2014-10-13 17:38:29.004121] I
>>> [server-handshake.c:575:server_setvolume]
>>> 0-HA-WIN-TT-1T-server: accepted client from
>>> stor2-15494-2014/10/13-17:38:28:989227-HA-WIN-TT-1T-client-1-0-0
>>> (version: 3.5.2)
>>> [2014-10-13 17:38:38.515315] I
>>> [server-handshake.c:575:server_setvolume]
>>> 0-HA-WIN-TT-1T-server: accepted client from
>>> glstor-cli-23823-2014/10/13-17:37:54:595571-HA-WIN-TT-1T-client-1-0-0
>>> (version: 3.5.2)
>>> [2014-10-13 17:39:09.872223] I
>>> [server.c:520:server_rpc_notify]
>>> 0-HA-WIN-TT-1T-server: disconnecting
>>> connectionfrom
>>> glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-1-0-1
>>> [2014-10-13 17:39:09.872299] I
>>> [client_t.c:417:gf_client_unref]
>>> 0-HA-WIN-TT-1T-server: Shutting down
>>> connection
>>> glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-1-0-1
>>> [2014-10-13 17:41:05.427810] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:41:05.443234] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:41:05.445049] I
>>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk]
>>> 0-glusterfs: No change in volfile, continuing
>>> root at stor2:~#
>>>
>>> brick stor1
>>>
>>> [2014-10-13 17:38:24.900066] I
>>> [glusterfsd.c:1959:main]
>>> 0-/usr/sbin/glusterfsd: Started running
>>> /usr/sbin/glusterfsd version 3.5.2
>>> (/usr/sbin/glusterfsd -s stor1 --volfile-id
>>> HA-WIN-TT-1T.stor1.exports-NFS-WIN-1T -p
>>> /var/lib/glusterd/vols/HA-WIN-TT-1T/run/stor1-exports-NFS-WIN-1T.pid
>>> -S
>>> /var/run/02580c93278849804f3f34f7ed8314b2.socket
>>> --brick-name /exports/NFS-WIN/1T -l
>>> /var/log/glusterfs/bricks/exports-NFS-WIN-1T.log
>>> --xlator-option
>>> *-posix.glusterd-uuid=e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
>>> --brick-port 49160 --xlator-option
>>> HA-WIN-TT-1T-server.listen-port=49160)
>>> [2014-10-13 17:38:24.902022] I
>>> [socket.c:3561:socket_init]
>>> 0-socket.glusterfsd: SSL support is NOT enabled
>>> [2014-10-13 17:38:24.902077] I
>>> [socket.c:3576:socket_init]
>>> 0-socket.glusterfsd: using system polling thread
>>> [2014-10-13 17:38:24.902214] I
>>> [socket.c:3561:socket_init] 0-glusterfs: SSL
>>> support is NOT enabled
>>> [2014-10-13 17:38:24.902239] I
>>> [socket.c:3576:socket_init] 0-glusterfs:
>>> using system polling thread
>>> [2014-10-13 17:38:24.906698] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-WIN-TT-1T-server: adding option
>>> 'listen-port' for volume
>>> 'HA-WIN-TT-1T-server' with value '49160'
>>> [2014-10-13 17:38:24.906731] I
>>> [graph.c:254:gf_add_cmdline_options]
>>> 0-HA-WIN-TT-1T-posix: adding option
>>> 'glusterd-uuid' for volume
>>> 'HA-WIN-TT-1T-posix' with value
>>> 'e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3'
>>> [2014-10-13 17:38:24.908378] I
>>> [rpcsvc.c:2127:rpcsvc_set_outstanding_rpc_limit]
>>> 0-rpc-service: Configured
>>> rpc.outstanding-rpc-limit with value 64
>>> [2014-10-13 17:38:24.908435] W
>>> [options.c:888:xl_opt_validate]
>>> 0-HA-WIN-TT-1T-server: option 'listen-port'
>>> is deprecated, preferred is
>>> 'transport.socket.listen-port', continuing
>>> with correction
>>> [2014-10-13 17:38:24.908472] I
>>> [socket.c:3561:socket_init]
>>> 0-tcp.HA-WIN-TT-1T-server: SSL support is
>>> NOT enabled
>>> [2014-10-13 17:38:24.908485] I
>>> [socket.c:3576:socket_init]
>>> 0-tcp.HA-WIN-TT-1T-server: using system
>>> polling thread
>>> [2014-10-13 17:38:24.909105] W
>>> [graph.c:329:_log_if_unknown_option]
>>> 0-HA-WIN-TT-1T-quota: option 'timeout' is
>>> not recognized
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>> 1: volume HA-WIN-TT-1T-posix
>>> 2: type storage/posix
>>> 3: option glusterd-uuid
>>> e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
>>> 4: option directory /exports/NFS-WIN/1T
>>> 5: option volume-id
>>> 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
>>> 6: end-volume
>>> 7:
>>> 8: volume HA-WIN-TT-1T-changelog
>>> 9: type features/changelog
>>> 10: option changelog-brick /exports/NFS-WIN/1T
>>> 11: option changelog-dir
>>> /exports/NFS-WIN/1T/.glusterfs/changelogs
>>> 12: subvolumes HA-WIN-TT-1T-posix
>>> 13: end-volume
>>> 14:
>>> 15: volume HA-WIN-TT-1T-access-control
>>> 16: type features/access-control
>>> 17: subvolumes HA-WIN-TT-1T-changelog
>>> 18: end-volume
>>> 19:
>>> 20: volume HA-WIN-TT-1T-locks
>>> 21: type features/locks
>>> 22: subvolumes HA-WIN-TT-1T-access-control
>>> 23: end-volume
>>> 24:
>>> 25: volume HA-WIN-TT-1T-io-threads
>>> 26: type performance/io-threads
>>> 27: subvolumes HA-WIN-TT-1T-locks
>>> 28: end-volume
>>> 29:
>>> 30: volume HA-WIN-TT-1T-index
>>> 31: type features/index
>>> 32: option index-base
>>> /exports/NFS-WIN/1T/.glusterfs/indices
>>> 33: subvolumes HA-WIN-TT-1T-io-threads
>>> 34: end-volume
>>> 35:
>>> 36: volume HA-WIN-TT-1T-marker
>>> 37: type features/marker
>>> 38: option volume-uuid
>>> 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
>>> 39: option timestamp-file
>>> /var/lib/glusterd/vols/HA-WIN-TT-1T/marker.tstamp
>>> 40: option xtime off
>>> 41: option gsync-force-xtime off
>>> 42: option quota off
>>> 43: subvolumes HA-WIN-TT-1T-index
>>> 44: end-volume
>>> 45:
>>> 46: volume HA-WIN-TT-1T-quota
>>> 47: type features/quota
>>> 48: option volume-uuid HA-WIN-TT-1T
>>> 49: option server-quota off
>>> 50: option timeout 0
>>> 51: option deem-statfs off
>>> 52: subvolumes HA-WIN-TT-1T-marker
>>> 53: end-volume
>>> 54:
>>> 55: volume /exports/NFS-WIN/1T
>>> 56: type debug/io-stats
>>> 57: option latency-measurement off
>>> 58: option count-fop-hits off
>>> 59: subvolumes HA-WIN-TT-1T-quota
>>> 60: end-volume
>>> 61:
>>> 62: volume HA-WIN-TT-1T-server
>>> 63: type protocol/server
>>> 64: option transport.socket.listen-port 49160
>>> 65: option rpc-auth.auth-glusterfs on
>>> 66: option rpc-auth.auth-unix on
>>> 67: option rpc-auth.auth-null on
>>> 68: option transport-type tcp
>>> 69: option
>>> auth.login./exports/NFS-WIN/1T.allow
>>> 101b907c-ff21-47da-8ba6-37e2920691ce
>>> 70: option
>>> auth.login.101b907c-ff21-47da-8ba6-37e2920691ce.password
>>> f4f29094-891f-4241-8736-5e3302ed8bc8
>>> 71: option
>>> auth.addr./exports/NFS-WIN/1T.allow *
>>> 72: subvolumes /exports/NFS-WIN/1T
>>> 73: end-volume
>>> 74:
>>> +------------------------------------------------------------------------------+
>>> [2014-10-13 17:38:25.933796] I
>>> [server-handshake.c:575:server_setvolume]
>>> 0-HA-WIN-TT-1T-server: accepted client from
>>> glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-0-0-1
>>> (version: 3.5.2)
>>> [2014-10-13 17:38:26.954924] I
>>> [server-handshake.c:575:server_setvolume]
>>> 0-HA-WIN-TT-1T-server: accepted client from
>>> stor1-14362-2014/10/13-17:38:26:938194-HA-WIN-TT-1T-client-0-0-0
>>> (version: 3.5.2)
>>> [2014-10-13 17:38:28.991488] I
>>> [server-handshake.c:575:server_setvolume]
>>> 0-HA-WIN-TT-1T-server: accepted client from
>>> stor2-15494-2014/10/13-17:38:28:989227-HA-WIN-TT-1T-client-0-0-0
>>> (version: 3.5.2)
>>> [2014-10-13 17:38:38.502056] I
>>> [server-handshake.c:575:server_setvolume]
>>> 0-HA-WIN-TT-1T-server: accepted client from
>>> glstor-cli-23823-2014/10/13-17:37:54:595571-HA-WIN-TT-1T-client-0-0-0
>>> (version: 3.5.2)
>>> [2014-10-13 17:39:09.858784] I
>>> [server.c:520:server_rpc_notify]
>>> 0-HA-WIN-TT-1T-server: disconnecting
>>> connectionfrom
>>> glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-0-0-1
>>> [2014-10-13 17:39:09.858863] I
>>> [client_t.c:417:gf_client_unref]
>>> 0-HA-WIN-TT-1T-server: Shutting down
>>> connection
>>> glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-0-0-1
>>> [2014-10-13 17:41:05.390918] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:41:05.408236] I
>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt:
>>> Volume file changed
>>> [2014-10-13 17:41:05.414813] I
>>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk]
>>> 0-glusterfs: No change in volfile, continuing
>>>
>>>
>>> seems to be the right part of logs :)
>>>
>>>
>>> 2014-10-15 18:24 GMT+03:00 Pranith Kumar
>>> Karampuri <pkarampu at redhat.com
>>> <mailto:pkarampu at redhat.com>>:
>>>
>>>
>>> On 10/14/2014 01:20 AM, Roman wrote:
>>>> ok. done.
>>>> this time there were no disconnects, at
>>>> least all of vms are working, but got
>>>> some mails from VM about IO writes again.
>>>>
>>>> WARNINGs: Read IO Wait time is 1.45
>>>> (outside range [0:1]).
>>> This warning says 'Read IO wait' and
>>> there is not a single READ operation
>>> that came to gluster. Wondering why that
>>> is :-/. Any clue? There is at least one
>>> write which took 3 seconds according to
>>> the stats. At least one synchronization
>>> operation (FINODELK) took 23 seconds.
>>> Could you give logs of this run? for
>>> mount, glustershd, bricks.
>>>
>>> Pranith
>>>
>>>>
>>>> here is the output
>>>>
>>>> root at stor1:~# gluster volume profile
>>>> HA-WIN-TT-1T info
>>>> Brick: stor1:/exports/NFS-WIN/1T
>>>> --------------------------------
>>>> Cumulative Stats:
>>>> Block Size: 131072b+ 262144b+
>>>> No. of Reads: 0 0
>>>> No. of Writes: 7372798 1
>>>> %-latency Avg-latency Min-Latency
>>>> Max-Latency No. of calls Fop
>>>> --------- ----------- -----------
>>>> ----------- ------------ ----
>>>> 0.00 0.00 us 0.00 us 0.00 us 25
>>>> RELEASE
>>>> 0.00 0.00 us 0.00 us 0.00 us 16
>>>> RELEASEDIR
>>>> 0.00 64.00 us 52.00 us 76.00 us
>>>> 2 ENTRYLK
>>>> 0.00 73.50 us 51.00 us 96.00 us
>>>> 2 FLUSH
>>>> 0.00 68.43 us 30.00 us 135.00 us
>>>> 7 STATFS
>>>> 0.00 54.31 us 44.00 us 109.00 us
>>>> 16 OPENDIR
>>>> 0.00 50.75 us 16.00 us 74.00 us
>>>> 24 FSTAT
>>>> 0.00 47.77 us 19.00 us 119.00 us
>>>> 26 GETXATTR
>>>> 0.00 59.21 us 21.00 us 89.00 us
>>>> 24 OPEN
>>>> 0.00 59.39 us 22.00 us 296.00 us
>>>> 28 READDIR
>>>> 0.00 4972.00 us 4972.00 us 4972.00
>>>> us 1 CREATE
>>>> 0.00 97.42 us 19.00 us 184.00 us
>>>> 62 LOOKUP
>>>> 0.00 89.49 us 20.00 us 656.00 us
>>>> 324 FXATTROP
>>>> 3.91 1255944.81 us 127.00 us
>>>> 23397532.00 us 189 FSYNC
>>>> 7.40 3406275.50 us 17.00 us
>>>> 23398013.00 us 132 INODELK
>>>> 34.96 94598.02 us 8.00 us
>>>> 23398705.00 us 22445 FINODELK
>>>> 53.73 442.66 us 79.00 us 3116494.00
>>>> us 7372799 WRITE
>>>>
>>>> Duration: 7813 seconds
>>>> Data Read: 0 bytes
>>>> Data Written: 966367641600 bytes
>>>>
>>>> Interval 0 Stats:
>>>> Block Size: 131072b+ 262144b+
>>>> No. of Reads: 0 0
>>>> No. of Writes: 7372798 1
>>>> %-latency Avg-latency Min-Latency
>>>> Max-Latency No. of calls Fop
>>>> --------- ----------- -----------
>>>> ----------- ------------ ----
>>>> 0.00 0.00 us 0.00 us 0.00 us 25
>>>> RELEASE
>>>> 0.00 0.00 us 0.00 us 0.00 us 16
>>>> RELEASEDIR
>>>> 0.00 64.00 us 52.00 us 76.00 us
>>>> 2 ENTRYLK
>>>> 0.00 73.50 us 51.00 us 96.00 us
>>>> 2 FLUSH
>>>> 0.00 68.43 us 30.00 us 135.00 us
>>>> 7 STATFS
>>>> 0.00 54.31 us 44.00 us 109.00 us
>>>> 16 OPENDIR
>>>> 0.00 50.75 us 16.00 us 74.00 us
>>>> 24 FSTAT
>>>> 0.00 47.77 us 19.00 us 119.00 us
>>>> 26 GETXATTR
>>>> 0.00 59.21 us 21.00 us 89.00 us
>>>> 24 OPEN
>>>> 0.00 59.39 us 22.00 us 296.00 us
>>>> 28 READDIR
>>>> 0.00 4972.00 us 4972.00 us 4972.00
>>>> us 1 CREATE
>>>> 0.00 97.42 us 19.00 us 184.00 us
>>>> 62 LOOKUP
>>>> 0.00 89.49 us 20.00 us 656.00 us
>>>> 324 FXATTROP
>>>> 3.91 1255944.81 us 127.00 us
>>>> 23397532.00 us 189 FSYNC
>>>> 7.40 3406275.50 us 17.00 us
>>>> 23398013.00 us 132 INODELK
>>>> 34.96 94598.02 us 8.00 us
>>>> 23398705.00 us 22445 FINODELK
>>>> 53.73 442.66 us 79.00 us 3116494.00
>>>> us 7372799 WRITE
>>>>
>>>> Duration: 7813 seconds
>>>> Data Read: 0 bytes
>>>> Data Written: 966367641600 bytes
>>>>
>>>> Brick: stor2:/exports/NFS-WIN/1T
>>>> --------------------------------
>>>> Cumulative Stats:
>>>> Block Size: 131072b+ 262144b+
>>>> No. of Reads: 0 0
>>>> No. of Writes: 7372798 1
>>>> %-latency Avg-latency Min-Latency
>>>> Max-Latency No. of calls Fop
>>>> --------- ----------- -----------
>>>> ----------- ------------ ----
>>>> 0.00 0.00 us 0.00 us 0.00 us 25
>>>> RELEASE
>>>> 0.00 0.00 us 0.00 us 0.00 us 16
>>>> RELEASEDIR
>>>> 0.00 61.50 us 46.00 us 77.00 us
>>>> 2 ENTRYLK
>>>> 0.00 82.00 us 67.00 us 97.00 us
>>>> 2 FLUSH
>>>> 0.00 265.00 us 265.00 us 265.00 us
>>>> 1 CREATE
>>>> 0.00 57.43 us 30.00 us 85.00 us
>>>> 7 STATFS
>>>> 0.00 61.12 us 37.00 us 107.00 us
>>>> 16 OPENDIR
>>>> 0.00 44.04 us 12.00 us 86.00 us
>>>> 24 FSTAT
>>>> 0.00 41.42 us 24.00 us 96.00 us
>>>> 26 GETXATTR
>>>> 0.00 45.93 us 24.00 us 133.00 us
>>>> 28 READDIR
>>>> 0.00 57.17 us 25.00 us 147.00 us
>>>> 24 OPEN
>>>> 0.00 145.28 us 31.00 us 288.00 us
>>>> 32 READDIRP
>>>> 0.00 39.50 us 10.00 us 152.00 us
>>>> 132 INODELK
>>>> 0.00 330.97 us 20.00 us 14280.00 us
>>>> 62 LOOKUP
>>>> 0.00 79.06 us 19.00 us 851.00 us
>>>> 430 FXATTROP
>>>> 0.02 29.32 us 7.00 us 28154.00 us
>>>> 22568 FINODELK
>>>> 7.80 1313096.68 us 125.00 us
>>>> 23281862.00 us 189 FSYNC
>>>> 92.18 397.92 us 76.00 us 1838343.00
>>>> us 7372799 WRITE
>>>>
>>>> Duration: 7811 seconds
>>>> Data Read: 0 bytes
>>>> Data Written: 966367641600 bytes
>>>>
>>>> Interval 0 Stats:
>>>> Block Size: 131072b+ 262144b+
>>>> No. of Reads: 0 0
>>>> No. of Writes: 7372798 1
>>>> %-latency Avg-latency Min-Latency
>>>> Max-Latency No. of calls Fop
>>>> --------- ----------- -----------
>>>> ----------- ------------ ----
>>>> 0.00 0.00 us 0.00 us 0.00 us 25
>>>> RELEASE
>>>> 0.00 0.00 us 0.00 us 0.00 us 16
>>>> RELEASEDIR
>>>> 0.00 61.50 us 46.00 us 77.00 us
>>>> 2 ENTRYLK
>>>> 0.00 82.00 us 67.00 us 97.00 us
>>>> 2 FLUSH
>>>> 0.00 265.00 us 265.00 us 265.00 us
>>>> 1 CREATE
>>>> 0.00 57.43 us 30.00 us 85.00 us
>>>> 7 STATFS
>>>> 0.00 61.12 us 37.00 us 107.00 us
>>>> 16 OPENDIR
>>>> 0.00 44.04 us 12.00 us 86.00 us
>>>> 24 FSTAT
>>>> 0.00 41.42 us 24.00 us 96.00 us
>>>> 26 GETXATTR
>>>> 0.00 45.93 us 24.00 us 133.00 us
>>>> 28 READDIR
>>>> 0.00 57.17 us 25.00 us 147.00 us
>>>> 24 OPEN
>>>> 0.00 145.28 us 31.00 us 288.00 us
>>>> 32 READDIRP
>>>> 0.00 39.50 us 10.00 us 152.00 us
>>>> 132 INODELK
>>>> 0.00 330.97 us 20.00 us 14280.00 us
>>>> 62 LOOKUP
>>>> 0.00 79.06 us 19.00 us 851.00 us
>>>> 430 FXATTROP
>>>> 0.02 29.32 us 7.00 us 28154.00 us
>>>> 22568 FINODELK
>>>> 7.80 1313096.68 us 125.00 us
>>>> 23281862.00 us 189 FSYNC
>>>> 92.18 397.92 us 76.00 us 1838343.00
>>>> us 7372799 WRITE
>>>>
>>>> Duration: 7811 seconds
>>>> Data Read: 0 bytes
>>>> Data Written: 966367641600 bytes
>>>>
>>>> does it make something more clear?
>>>>
>>>> 2014-10-13 20:40 GMT+03:00 Roman
>>>> <romeo.r at gmail.com
>>>> <mailto:romeo.r at gmail.com>>:
>>>>
>>>> i think i may know what was an
>>>> issue. There was an iscsitarget
>>>> service runing, that was exporting
>>>> this generated block device. so
>>>> maybe my collegue Windows server
>>>> picked it up and mountd :) I'll if
>>>> it will happen again.
>>>>
>>>> 2014-10-13 20:27 GMT+03:00 Roman
>>>> <romeo.r at gmail.com
>>>> <mailto:romeo.r at gmail.com>>:
>>>>
>>>> So may I restart the volume and
>>>> start the test, or you need
>>>> something else from this issue?
>>>>
>>>> 2014-10-13 19:49 GMT+03:00
>>>> Pranith Kumar Karampuri
>>>> <pkarampu at redhat.com
>>>> <mailto:pkarampu at redhat.com>>:
>>>>
>>>>
>>>> On 10/13/2014 10:03 PM,
>>>> Roman wrote:
>>>>> hmm,
>>>>> seems like another strange
>>>>> issue? Seen this before.
>>>>> Had to restart the volume
>>>>> to get my empty space back.
>>>>> root at glstor-cli:/srv/nfs/HA-WIN-TT-1T#
>>>>> ls -l
>>>>> total 943718400
>>>>> -rw-r--r-- 1 root root
>>>>> 966367641600 Oct 13 16:55 disk
>>>>> root at glstor-cli:/srv/nfs/HA-WIN-TT-1T#
>>>>> rm disk
>>>>> root at glstor-cli:/srv/nfs/HA-WIN-TT-1T#
>>>>> df -h
>>>>> Filesystem Size Used
>>>>> Avail Use% Mounted on
>>>>> rootfs 282G 1.1G
>>>>> 266G 1% /
>>>>> udev 10M 0 10M
>>>>> 0% /dev
>>>>> tmpfs 1.4G 228K 1.4G
>>>>> 1% /run
>>>>> /dev/disk/by-uuid/c62ee3c0-c0e5-44af-b0cd-7cb3fbcc0fba
>>>>> 282G 1.1G 266G 1% /
>>>>> tmpfs 5.0M 0 5.0M
>>>>> 0% /run/lock
>>>>> tmpfs 5.2G 0 5.2G
>>>>> 0% /run/shm
>>>>> stor1:HA-WIN-TT-1T
>>>>> 1008G 901G 57G 95%
>>>>> /srv/nfs/HA-WIN-TT-1T
>>>>>
>>>>> no file, but size is still
>>>>> 901G.
>>>>> Both servers show the same.
>>>>> Do I really have to
>>>>> restart the volume to fix
>>>>> that?
>>>> IMO this can happen if
>>>> there is an fd leak.
>>>> open-fd is the only
>>>> variable that can change
>>>> with volume restart. How do
>>>> you re-create the bug?
>>>>
>>>> Pranith
>>>>
>>>>>
>>>>> 2014-10-13 19:30 GMT+03:00
>>>>> Roman <romeo.r at gmail.com
>>>>> <mailto:romeo.r at gmail.com>>:
>>>>>
>>>>> Sure.
>>>>> I'll let it to run for
>>>>> this night .
>>>>>
>>>>> 2014-10-13 19:19
>>>>> GMT+03:00 Pranith
>>>>> Kumar Karampuri
>>>>> <pkarampu at redhat.com
>>>>> <mailto:pkarampu at redhat.com>>:
>>>>>
>>>>> hi Roman,
>>>>> Do you think
>>>>> we can run this
>>>>> test again? this
>>>>> time, could you
>>>>> enable 'gluster
>>>>> volume profile
>>>>> <volname> start',
>>>>> do the same test.
>>>>> Provide output of
>>>>> 'gluster volume
>>>>> profile <volname>
>>>>> info' and logs
>>>>> after the test?
>>>>>
>>>>> Pranith
>>>>>
>>>>> On 10/13/2014
>>>>> 09:45 PM, Roman wrote:
>>>>>> Sure !
>>>>>>
>>>>>> root at stor1:~#
>>>>>> gluster volume info
>>>>>>
>>>>>> Volume Name:
>>>>>> HA-2TB-TT-Proxmox-cluster
>>>>>> Type: Replicate
>>>>>> Volume ID:
>>>>>> 66e38bde-c5fa-4ce2-be6e-6b2adeaa16c2
>>>>>> Status: Started
>>>>>> Number of Bricks:
>>>>>> 1 x 2 = 2
>>>>>> Transport-type: tcp
>>>>>> Bricks:
>>>>>> Brick1:
>>>>>> stor1:/exports/HA-2TB-TT-Proxmox-cluster/2TB
>>>>>> Brick2:
>>>>>> stor2:/exports/HA-2TB-TT-Proxmox-cluster/2TB
>>>>>> Options Reconfigured:
>>>>>> nfs.disable: 0
>>>>>> network.ping-timeout:
>>>>>> 10
>>>>>>
>>>>>> Volume Name:
>>>>>> HA-WIN-TT-1T
>>>>>> Type: Replicate
>>>>>> Volume ID:
>>>>>> 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
>>>>>> Status: Started
>>>>>> Number of Bricks:
>>>>>> 1 x 2 = 2
>>>>>> Transport-type: tcp
>>>>>> Bricks:
>>>>>> Brick1:
>>>>>> stor1:/exports/NFS-WIN/1T
>>>>>> Brick2:
>>>>>> stor2:/exports/NFS-WIN/1T
>>>>>> Options Reconfigured:
>>>>>> nfs.disable: 1
>>>>>> network.ping-timeout:
>>>>>> 10
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2014-10-13 19:09
>>>>>> GMT+03:00 Pranith
>>>>>> Kumar Karampuri
>>>>>> <pkarampu at redhat.com
>>>>>> <mailto:pkarampu at redhat.com>>:
>>>>>>
>>>>>> Could you
>>>>>> give your
>>>>>> 'gluster
>>>>>> volume info'
>>>>>> output?
>>>>>>
>>>>>> Pranith
>>>>>>
>>>>>> On 10/13/2014
>>>>>> 09:36 PM,
>>>>>> Roman wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've got
>>>>>>> this kind of
>>>>>>> setup
>>>>>>> (servers run
>>>>>>> replica)
>>>>>>>
>>>>>>>
>>>>>>> @ 10G backend
>>>>>>> gluster storage1
>>>>>>> gluster storage2
>>>>>>> gluster client1
>>>>>>>
>>>>>>> @1g backend
>>>>>>> other
>>>>>>> gluster clients
>>>>>>>
>>>>>>> Servers got
>>>>>>> HW RAID5
>>>>>>> with SAS disks.
>>>>>>>
>>>>>>> So today
>>>>>>> I've desided
>>>>>>> to create a
>>>>>>> 900GB file
>>>>>>> for iscsi
>>>>>>> target that
>>>>>>> will be
>>>>>>> located @
>>>>>>> glusterfs
>>>>>>> separate
>>>>>>> volume,
>>>>>>> using dd
>>>>>>> (just a
>>>>>>> dummy file
>>>>>>> filled with
>>>>>>> zeros, bs=1G
>>>>>>> count 900)
>>>>>>> For the
>>>>>>> first of all
>>>>>>> the process
>>>>>>> took pretty
>>>>>>> lots of
>>>>>>> time, the
>>>>>>> writing
>>>>>>> speed was
>>>>>>> 130 MB/sec
>>>>>>> (client port
>>>>>>> was 2 gbps,
>>>>>>> servers
>>>>>>> ports were
>>>>>>> running @
>>>>>>> 1gbps).
>>>>>>> Then it
>>>>>>> reported
>>>>>>> something
>>>>>>> like
>>>>>>> "endpoint is
>>>>>>> not
>>>>>>> connected"
>>>>>>> and all of
>>>>>>> my VMs on
>>>>>>> the other
>>>>>>> volume
>>>>>>> started to
>>>>>>> give me IO
>>>>>>> errors.
>>>>>>> Servers load
>>>>>>> was around
>>>>>>> 4,6 (total
>>>>>>> 12 cores)
>>>>>>>
>>>>>>> Maybe it was
>>>>>>> due to
>>>>>>> timeout of 2
>>>>>>> secs, so
>>>>>>> I've made it
>>>>>>> a big
>>>>>>> higher, 10 sec.
>>>>>>>
>>>>>>> Also during
>>>>>>> the dd image
>>>>>>> creation
>>>>>>> time, VMs
>>>>>>> very often
>>>>>>> reported me
>>>>>>> that their
>>>>>>> disks are
>>>>>>> slow like
>>>>>>>
>>>>>>> WARNINGs:
>>>>>>> Read IO Wait
>>>>>>> time is
>>>>>>> -0.02
>>>>>>> (outside
>>>>>>> range [0:1]).
>>>>>>>
>>>>>>> Is 130MB
>>>>>>> /sec is the
>>>>>>> maximum
>>>>>>> bandwidth
>>>>>>> for all of
>>>>>>> the volumes
>>>>>>> in total?
>>>>>>> That why
>>>>>>> would we
>>>>>>> need 10g
>>>>>>> backends?
>>>>>>>
>>>>>>> HW Raid
>>>>>>> local speed
>>>>>>> is 300
>>>>>>> MB/sec, so
>>>>>>> it should
>>>>>>> not be an
>>>>>>> issue. any
>>>>>>> ideas or mby
>>>>>>> any advices?
>>>>>>>
>>>>>>>
>>>>>>> Maybe some1
>>>>>>> got
>>>>>>> optimized
>>>>>>> sysctl.conf
>>>>>>> for 10G backend?
>>>>>>>
>>>>>>> mine is
>>>>>>> pretty
>>>>>>> simple,
>>>>>>> which can be
>>>>>>> found from
>>>>>>> googling.
>>>>>>>
>>>>>>>
>>>>>>> just to
>>>>>>> mention:
>>>>>>> those VM-s
>>>>>>> were
>>>>>>> connected
>>>>>>> using
>>>>>>> separate
>>>>>>> 1gbps
>>>>>>> intraface,
>>>>>>> which means,
>>>>>>> they should
>>>>>>> not be
>>>>>>> affected by
>>>>>>> the client
>>>>>>> with 10g
>>>>>>> backend.
>>>>>>>
>>>>>>>
>>>>>>> logs are
>>>>>>> pretty
>>>>>>> useless,
>>>>>>> they just
>>>>>>> say this
>>>>>>> during the
>>>>>>> outage
>>>>>>>
>>>>>>>
>>>>>>> [2014-10-13
>>>>>>> 12:09:18.392910]
>>>>>>> W
>>>>>>> [client-handshake.c:276:client_ping_cbk]
>>>>>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>>>>>> timer must
>>>>>>> have expired
>>>>>>>
>>>>>>> [2014-10-13
>>>>>>> 12:10:08.389708]
>>>>>>> C
>>>>>>> [client-handshake.c:127:rpc_client_ping_timer_expired]
>>>>>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>>>>>> server
>>>>>>> 10.250.0.1:49159
>>>>>>> <http://10.250.0.1:49159>
>>>>>>> has not
>>>>>>> responded in
>>>>>>> the last 2
>>>>>>> seconds,
>>>>>>> disconnecting.
>>>>>>>
>>>>>>> [2014-10-13
>>>>>>> 12:10:08.390312]
>>>>>>> W
>>>>>>> [client-handshake.c:276:client_ping_cbk]
>>>>>>> 0-HA-2TB-TT-Proxmox-cluster-client-0:
>>>>>>> timer must
>>>>>>> have expired
>>>>>>>
>>>>>>> so I decided
>>>>>>> to set the
>>>>>>> timout a bit
>>>>>>> higher.
>>>>>>>
>>>>>>> So it seems
>>>>>>> to me, that
>>>>>>> under high
>>>>>>> load
>>>>>>> GlusterFS is
>>>>>>> not useable?
>>>>>>> 130 MB/s is
>>>>>>> not that
>>>>>>> much to get
>>>>>>> some kind of
>>>>>>> timeouts or
>>>>>>> makeing the
>>>>>>> systme so
>>>>>>> slow, that
>>>>>>> VM-s feeling
>>>>>>> themselves bad.
>>>>>>>
>>>>>>> Of course,
>>>>>>> after the
>>>>>>> disconnection,
>>>>>>> healing
>>>>>>> process was
>>>>>>> started, but
>>>>>>> as VM-s lost
>>>>>>> connection
>>>>>>> to both of
>>>>>>> servers, it
>>>>>>> was pretty
>>>>>>> useless,
>>>>>>> they could
>>>>>>> not run
>>>>>>> anymore. and
>>>>>>> BTW, when u
>>>>>>> load the
>>>>>>> server with
>>>>>>> such huge
>>>>>>> job (dd of
>>>>>>> 900GB),
>>>>>>> healing
>>>>>>> process goes
>>>>>>> soooooo slow :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best regards,
>>>>>>> Roman.
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Roman.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Roman.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Roman.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Roman.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Roman.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Roman.
>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Roman.
>>
>>
>>
>>
>> --
>> Best regards,
>> Roman.
>>
>>
>>
>>
>> --
>> Best regards,
>> Roman.
>>
>>
>>
>>
>> --
>> Best regards,
>> Roman.
>>
>>
>>
>>
>> --
>> Best regards,
>> Roman.
>>
>>
>>
>>
>> --
>> Best regards,
>> Roman.
>
>
>
>
> --
> Best regards,
> Roman.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141105/9f4b8bd1/attachment-0001.html>
More information about the Gluster-users
mailing list