[Gluster-users] Gluster 3.7.0 released
Ted Miller
tmiller at sonsetsolutions.org
Wed May 27 14:03:03 UTC 2015
responses below
Ted Miller
On 5/26/2015 12:01 AM, Atin Mukherjee wrote:
>
> On 05/26/2015 03:12 AM, Ted Miller wrote:
>> From: Niels de Vos <ndevos at redhat.com>
>> Sent: Monday, May 25, 2015 4:44 PM
>>
>> On Mon, May 25, 2015 at 06:49:26PM +0000, Ted Miller wrote:
>>> ________________________________
>>> From: Humble Devassy Chirammal <humble.devassy at gmail.com>
>>> Sent: Monday, May 18, 2015 9:37 AM
>>> Hi All,
>>>
>>> GlusterFS 3.7.0 RPMs for RHEL, CentOS, Fedora and packages for Debian are available at download.gluster.org<http://download.gluster.org> [1].
>>>
>>> [1] http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/
>>>
>>> --Humble
>>>
>>>
>>> On Thu, May 14, 2015 at 2:49 PM, Vijay Bellur <vbellur at redhat.com<mailto:vbellur at redhat.com>> wrote:
>>>
>>> Hi All,
>>>
>>> I am happy to announce that Gluster 3.7.0 is now generally available. 3.7.0 contains several
>>>
>>> [snip]
>>>
>>> Cheers,
>>> Vijay
>>>
>>> [snip]
>>>
>>> What happened to packages for RHEL/Centos 5? I have the (probably
>>> unusual--added gluster to existing servers) setup of running a replica
>>> 3 cluster where two nodes run on Centos 6 and one is still on Centos
>>> 5. This is a personal setup, and I have been using
>>> http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-5/x86_64/repodata/repomod.xml
>>> as my repo. It has worked fine for a while, but this time the two
>>> Centos 6 nodes updated to 3.7, but the Centos 5 node got left behind
>>> at 3.6.3.
>> Packages for RHEL/CentOS-5 are not available yet. These will follow
>> later. Thare are some changes needed to be able to build the packages on
>> EL5. Because we are currently stabilizing our CI/regression tests, we do
>> not merge any other changes. Until we provide packages in our
>> repository, you could apply patch http://review.gluster.org/10803
>> yourself and build the EL5 version. I expect that we will do a release
>> in 2-3 weeks which will have EL5 RPMs too.
>>
>> I have no idea about the problem below, it sounds like something the
>> GlusterD developers could help with.
>>
>> Niels
>>
>>> Command 'gluster volume status' on the C5 machine makes everything
>>> look fine:
>>>
>>> Status of volume: ISO2
>>> Gluster process Port Online Pid
>>> ------------------------------------------------------------------------------
>>> Brick 10.x.x.2:/bricks/01/iso2 49162 Y 4679
>>> Brick 10.x.x.4:/bricks/01/iso2 49183 Y 6447
>>> Brick 10.x.x.9:/bricks/01/iso2 49169 Y 1985
>>>
>>> But the same command on either of the C6 machines shows the C5 machine
>>> (10.x.x.2) missing in action (though it does recognize that there are
>>> NFS and heal daemons there):
>>>
>>> Status of volume: ISO2
>>> Gluster process TCP Port RDMA Port Online Pid
>>> ------------------------------------------------------------------------------
>>> Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447
>>> Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985
>>> NFS Server on localhost 2049 0 Y 2279
>>> Self-heal Daemon on localhost N/A N/A Y 2754
>>> NFS Server on 10.41.65.2 2049 0 Y 4757
>>> Self-heal Daemon on 10.41.65.2 N/A N/A Y 4764
>>> NFS Server on 10.41.65.4 2049 0 Y 6543
>>> Self-heal Daemon on 10.41.65.4 N/A N/A Y 6551
>>>
>>> So, is this just an oversight (I hope), or has support for C5 been dropped?
>>> If support for C5 is gone, how do I downgrade my Centos6 machines back
>>> to 3.6.x? (I know how to change the repo, but the actual sequence of
>>> yum commands and gluster commands is unknown to me).
> Could you attach the glusterd log file of 10.x.x.2 machine
attached as etc-glusterfs-glusterd.vol.log.newer.2, starting from last
machine reboot
> and the node from where you triggered volume status.
attached as etc-glusterfs-glusterd.vol.log.newer4 starting same time as .2 log
> Could you also share gluster volume info output of all the nodes?
I have several volumes, so I chose the one that shows up first on the listings:
*from 10.41.65.2:*
[root at office2 /var/log/glusterfs]$ gluster volume info
Volume Name: ISO2
Type: Replicate
Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.41.65.2:/bricks/01/iso2
Brick2: 10.41.65.4:/bricks/01/iso2
Brick3: 10.41.65.9:/bricks/01/iso2
[root at office2 /var/log/glusterfs]$ gluster volume status ISO2
Status of volume: ISO2
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick 10.41.65.2:/bricks/01/iso2 49162 Y 4463
Brick 10.41.65.4:/bricks/01/iso2 49183 Y 6447
Brick 10.41.65.9:/bricks/01/iso2 49169 Y 1985
NFS Server on localhost 2049 Y 4536
Self-heal Daemon on localhost N/A Y 4543
NFS Server on 10.41.65.9 2049 Y 2279
Self-heal Daemon on 10.41.65.9 N/A Y 2754
NFS Server on 10.41.65.4 2049 Y 6543
Self-heal Daemon on 10.41.65.4 N/A Y 6551
Task Status of Volume ISO2
------------------------------------------------------------------------------
There are no active volume tasks
[root at office2 ~]$ gluster peer status
Number of Peers: 2
Hostname: 10.41.65.9
Uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
State: Peer in Cluster (Connected)
Hostname: 10.41.65.4
Uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c
State: Peer in Cluster (Connected)
*from 10.41.65.4:*
[root at office4b ~]# gluster volume info ISO2
Volume Name: ISO2
Type: Replicate
Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.41.65.2:/bricks/01/iso2
Brick2: 10.41.65.4:/bricks/01/iso2
Brick3: 10.41.65.9:/bricks/01/iso2
[root at office4b ~]# gluster volume status ISO2
Status of volume: ISO2
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447
Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985
NFS Server on localhost 2049 0 Y 6543
Self-heal Daemon on localhost N/A N/A Y 6551
NFS Server on 10.41.65.2 2049 0 Y 4536
Self-heal Daemon on 10.41.65.2 N/A N/A Y 4543
NFS Server on 10.41.65.9 2049 0 Y 2279
Self-heal Daemon on 10.41.65.9 N/A N/A Y 2754
Task Status of Volume ISO2
------------------------------------------------------------------------------
There are no active volume tasks
[root at office4b ~]# gluster peer status
Number of Peers: 2
Hostname: 10.41.65.2
Uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57
State: Peer in Cluster (Connected)
Hostname: 10.41.65.9
Uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
State: Peer in Cluster (Connected)
*from 10.41.65.9:*
[root at office9 ~]$ gluster volume info ISO2
Volume Name: ISO2
Type: Replicate
Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.41.65.2:/bricks/01/iso2
Brick2: 10.41.65.4:/bricks/01/iso2
Brick3: 10.41.65.9:/bricks/01/iso2
[root at office9 ~]$ gluster volume status ISO2
Status of volume: ISO2
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447
Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985
NFS Server on localhost 2049 0 Y 2279
Self-heal Daemon on localhost N/A N/A Y 2754
NFS Server on 10.41.65.2 2049 0 Y 4536
Self-heal Daemon on 10.41.65.2 N/A N/A Y 4543
NFS Server on 10.41.65.4 2049 0 Y 6543
Self-heal Daemon on 10.41.65.4 N/A N/A Y 6551
Task Status of Volume ISO2
------------------------------------------------------------------------------
There are no active volume tasks
[root at office9 ~]$ gluster peer status
Number of Peers: 2
Hostname: 10.41.65.2
Uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57
State: Peer in Cluster (Connected)
Hostname: 10.41.65.4
Uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c
State: Peer in Cluster (Connected)
>>> Ted Miller
>>> Elkhart, IN, USA
>>
>> Thanks for the information. As long as I know it is coming, I can improvise and hang on.
>>
>> I am assuming that the problem with the .2 machine not being seen is a result of running a cluster with a version split.
>>
>> Ted Miller
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
--
*Ted Miller*, Design Engineer
*SonSet Solutions*
(formerly HCJB Global Technology Center)
my desk +1 574.970.4272
receptionist +1 574.972.4252
http://sonsetsolutions.org
/Technology for abundant life!/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150527/040d40f8/attachment.html>
-------------- next part --------------
[2015-05-25 14:51:53.898279] W [socket.c:620:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (No data available)
[2015-05-25 14:51:53.898403] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer cf2ae9c7-833e-4a73-a996-e72158011c69, in Peer in Cluster state, has disconnected from glusterd.
[2015-05-25 14:51:53.898800] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISO2 not held
[2015-05-25 14:51:53.899083] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISOold not held
[2015-05-25 14:51:53.899356] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM-eng01 not held
[2015-05-25 14:51:53.899647] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM1 not held
[2015-05-25 14:51:53.899958] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM2 not held
[2015-05-25 14:51:53.900230] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM3 not held
[2015-05-25 14:51:53.900498] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VMold not held
[2015-05-25 14:51:53.900767] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol export1 not held
[2015-05-25 14:51:53.901034] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol home1 not held
[2015-05-25 14:51:53.901304] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol host_share1 not held
[2015-05-25 14:51:53.901591] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol pics not held
[2015-05-25 14:51:53.901861] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol video01 not held
[2015-05-25 14:52:03.883612] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600
[2015-05-25 14:52:04.100889] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:04.153955] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0
[2015-05-25 14:52:04.214125] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c
[2015-05-25 14:52:04.214210] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-05-25 14:52:04.253289] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c
[2015-05-25 14:52:05.876100] W [socket.c:620:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (Connection reset by peer)
[2015-05-25 14:52:05.876432] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1d7)[0x315680e887] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x315680e98e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x315680ea31] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd2)[0x315680f152] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2015-05-25 14:52:04.163362 (xid=0xe)
[2015-05-25 14:52:05.876479] E [glusterd-rpc-ops.c:619:__glusterd_friend_update_cbk] 0-management: RPC Error
[2015-05-25 14:52:05.876515] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2015-05-25 14:52:05.876706] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1d7)[0x315680e887] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x315680e98e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x315680ea31] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd2)[0x315680f152] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: forced unwinding frame type(Gluster MGMT Handshake) op(MGMT-VERS(1)) called at 2015-05-25 14:52:04.166562 (xid=0xf)
[2015-05-25 14:52:05.876751] E [glusterd-handshake.c:1680:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later
[2015-05-25 14:52:05.877044] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISO2 not held
[2015-05-25 14:52:05.877336] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISOold not held
[2015-05-25 14:52:05.877610] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM-eng01 not held
[2015-05-25 14:52:05.877879] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM1 not held
[2015-05-25 14:52:05.878149] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM2 not held
[2015-05-25 14:52:05.878418] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM3 not held
[2015-05-25 14:52:05.878686] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VMold not held
[2015-05-25 14:52:05.878955] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol export1 not held
[2015-05-25 14:52:05.879235] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol home1 not held
[2015-05-25 14:52:05.879538] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol host_share1 not held
[2015-05-25 14:52:05.880094] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol pics not held
[2015-05-25 14:52:05.880366] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol video01 not held
[2015-05-25 14:52:08.846394] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600
[2015-05-25 14:52:10.144090] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:10.156233] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0
[2015-05-25 14:52:10.161604] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c
[2015-05-25 14:52:10.161697] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:10.161767] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-05-25 14:52:10.474292] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69, host: 10.41.65.9, port: 0
[2015-05-25 14:52:10.528558] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:10.528639] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-05-25 14:52:10.813342] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c
[2015-05-25 14:52:10.813441] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-05-25 14:52:10.997945] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:05.876801] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer cf2ae9c7-833e-4a73-a996-e72158011c69, in Peer in Cluster state, has disconnected from glusterd.
-------------- next part --------------
[2015-05-25 14:51:53.902180] W [socket.c:642:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (No data available)
[2015-05-25 14:51:53.960843] I [MSGID: 106004] [glusterd-handler.c:4809:__glusterd_peer_rpc_notify] 0-management: Peer <10.41.65.9> (<cf2ae9c7-833e-4a73-a996-e72158011c69>), in state <Peer in Cluster>, has disconnected from glusterd.
[2015-05-25 14:51:53.963643] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISO2 not held
[2015-05-25 14:51:53.963793] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISOold not held
[2015-05-25 14:51:53.963927] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM-eng01 not held
[2015-05-25 14:51:53.964057] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM1 not held
[2015-05-25 14:51:53.964187] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM2 not held
[2015-05-25 14:51:53.964316] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM3 not held
[2015-05-25 14:51:53.964452] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VMold not held
[2015-05-25 14:51:53.964587] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol export1 not held
[2015-05-25 14:51:53.964716] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol home1 not held
[2015-05-25 14:51:53.964845] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol host_share1 not held
[2015-05-25 14:51:53.964974] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol pics not held
[2015-05-25 14:51:53.965103] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol video01 not held
[2015-05-25 14:52:03.879790] I [glusterd-handshake.c:1151:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600
[2015-05-25 14:52:04.097083] I [glusterd-handler.c:2422:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:04.161852] I [glusterd-handler.c:3580:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0
[2015-05-25 14:52:04.249180] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57
[2015-05-25 14:52:04.249236] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-05-25 14:52:04.249279] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57
[2015-05-25 14:52:05.871153] W [socket.c:642:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (Connection reset by peer)
[2015-05-25 14:52:05.871411] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f066ce7fa87] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f066ce7fb9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f066ce7fc6b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f066ce8022f] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2015-05-25 14:52:04.210183 (xid=0xa)
[2015-05-25 14:52:05.871431] E [glusterd-rpc-ops.c:653:__glusterd_friend_update_cbk] 0-management: RPC Error
[2015-05-25 14:52:05.871441] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2015-05-25 14:52:05.871525] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f066ce7fa87] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f066ce7fb9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f066ce7fc6b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f066ce8022f] ))))) 0-management: forced unwinding frame type(Gluster MGMT Handshake) op(MGMT-VERS(1)) called at 2015-05-25 14:52:04.249148 (xid=0xb)
[2015-05-25 14:52:05.871545] E [glusterd-handshake.c:1740:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later
[2015-05-25 14:52:05.871564] I [MSGID: 106004] [glusterd-handler.c:4809:__glusterd_peer_rpc_notify] 0-management: Peer <10.41.65.9> (<cf2ae9c7-833e-4a73-a996-e72158011c69>), in state <Peer in Cluster>, has disconnected from glusterd.
[2015-05-25 14:52:05.871701] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISO2 not held
[2015-05-25 14:52:05.871832] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISOold not held
[2015-05-25 14:52:05.871962] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM-eng01 not held
[2015-05-25 14:52:05.872091] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM1 not held
[2015-05-25 14:52:05.872248] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM2 not held
[2015-05-25 14:52:05.872385] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM3 not held
[2015-05-25 14:52:05.872517] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VMold not held
[2015-05-25 14:52:05.872646] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol export1 not held
[2015-05-25 14:52:05.872774] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol home1 not held
[2015-05-25 14:52:05.872903] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol host_share1 not held
[2015-05-25 14:52:05.873032] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol pics not held
[2015-05-25 14:52:05.873167] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol video01 not held
[2015-05-25 14:52:08.842559] I [glusterd-handshake.c:1151:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600
[2015-05-25 14:52:09.710802] I [glusterd-rpc-ops.c:463:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69, host: 10.41.65.9, port: 0
[2015-05-25 14:52:10.155216] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57
[2015-05-25 14:52:10.155267] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-05-25 14:52:10.737330] I [glusterd-handler.c:2422:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:10.766113] I [glusterd-handler.c:3580:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0
[2015-05-25 14:52:10.852943] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57
[2015-05-25 14:52:10.852988] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:10.853013] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-05-25 14:52:10.853061] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
[2015-05-25 14:52:10.853079] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-05-25 14:52:11.047970] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69
More information about the Gluster-users
mailing list