[Gluster-users] VM disks corruption on 3.7.11
Kevin Lemonnier
lemonnierk at ulrar.net
Wed May 25 15:58:47 UTC 2016
There, re-created the VM from scratch, and still got the same errors.
Attached are the logs, I created the VM on node 50, worked fine. I tried
to reboot it and start my import again, still worked fine. I powered off the
VM, then started it again on node 2, rebooted it a bunch and just got the error
as usual, just attached a screen of the VM's console, might help.
I can see that everytime the VM powers down, glusterFS complains about an inode still
active, might it be the problem ?
Thanks for the help !
On Wed, May 25, 2016 at 04:10:02PM +0200, Kevin Lemonnier wrote:
> Just did that, below is the output.
> Didn't seem to move after the boot, and no new lines when the I/O errors appeared.
> Also, as mentionned I tried moving the disk on NFS and had the exact same errors,
> so it doesn't look like it's a libgfapi problem ..
> I should probably re-create the VM, maybe the errors from this night corrupted
> the disk and I now get errors unrelated to the original issue.
>
> Let me re-create the VM from scratch and try to reproduce the problem with
> the logs enabled, maybe it'll be more informative than this !
>
>
> [2016-05-25 13:56:30.851493] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e733339-3635-3033-2e69-702d34362d31 (0) coming up
> [2016-05-25 13:56:30.851553] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport
> [2016-05-25 13:56:30.852130] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport
> [2016-05-25 13:56:30.852650] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport
> [2016-05-25 13:56:30.852909] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0)
> [2016-05-25 13:56:30.853434] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0)
> [2016-05-25 13:56:30.853484] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0)
> [2016-05-25 13:56:30.854182] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
> [2016-05-25 13:56:30.854398] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
> [2016-05-25 13:56:30.854441] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
> [2016-05-25 13:56:30.861931] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'.
> [2016-05-25 13:56:30.861965] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds
> [2016-05-25 13:56:30.862073] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-2' came back up; going online.
> [2016-05-25 13:56:30.862139] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1
> [2016-05-25 13:56:30.865451] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'.
> [2016-05-25 13:56:30.865485] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
> [2016-05-25 13:56:30.865757] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1
> [2016-05-25 13:56:30.865826] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'.
> [2016-05-25 13:56:30.865841] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
> [2016-05-25 13:56:30.888604] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1
> [2016-05-25 13:56:30.890388] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-2
> [2016-05-25 13:56:30.890731] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 6e733339-3635-3033-2e69-702d34362d31 (0)
>
>
>
> On Wed, May 25, 2016 at 02:48:27PM +0530, Krutika Dhananjay wrote:
> > Also, it seems Lindsay knows a way to get the gluster client logs when
> > using proxmox and libgfapi.
> > Would it be possible for you to get that sorted with Lindsay's help before
> > recreating this issue next time
> > and share the glusterfs client logs from all the nodes when you do hit the
> > issue?
> > It is critical for some of the debugging we do. :)
> >
> > -Krutika
> > On Wed, May 25, 2016 at 2:38 PM, Krutika Dhananjay <kdhananj at redhat.com>
> > wrote:
> >
> > Hi Kevin,
> >
> > If you actually ran into a 'read-only filesystem' issue, then it could
> > possibly because of a bug in AFR
> > that Pranith recently fixed.
> > To confirm if that is indeed the case, could you tell meA if you saw
> > the pause after a brick (single brick) was
> > down while IO was going on?
> >
> > -Krutika
> > On Wed, May 25, 2016 at 1:28 PM, Kevin Lemonnier <lemonnierk at ulrar.net>
> > wrote:
> >
> > >A A Whats the underlying filesystem under the bricks?
> >
> > I use XFS, I read that was recommended. What are you using ?
> > Since yours seems to work, I'm not opposed to changing !
> > --
> > Kevin Lemonnier
> > PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
> --
> Kevin Lemonnier
> PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
--
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
-------------- next part --------------
[2016-05-25 14:15:51.986216] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e733339-3635-3033-2e69-702d34362d31 (0) coming up
[2016-05-25 14:15:51.986267] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport
[2016-05-25 14:15:51.986794] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport
[2016-05-25 14:15:51.987164] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport
[2016-05-25 14:15:51.987476] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0)
[2016-05-25 14:15:51.987779] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0)
[2016-05-25 14:15:51.987846] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0)
[2016-05-25 14:15:51.988602] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 14:15:51.988751] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 14:15:51.988897] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 14:15:51.989164] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 14:15:51.989190] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 14:15:51.989248] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 14:15:51.989265] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 14:15:51.989267] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online.
[2016-05-25 14:15:51.989310] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 14:15:51.989323] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 14:15:51.989380] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1
[2016-05-25 14:15:51.989488] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1
[2016-05-25 14:15:52.008578] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1
[2016-05-25 14:15:52.009952] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-2
[2016-05-25 14:15:52.010405] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 6e733339-3635-3033-2e69-702d34362d31 (0)
[2016-05-25 14:53:58.342467] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-0: current graph is no longer active, destroying rpc_client
[2016-05-25 14:53:58.342546] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-1: current graph is no longer active, destroying rpc_client
[2016-05-25 14:53:58.342577] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-2: current graph is no longer active, destroying rpc_client
[2016-05-25 14:53:58.342605] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-0: disconnected from gluster-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 14:53:58.342623] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-1: disconnected from gluster-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 14:53:58.342681] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-2: disconnected from gluster-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 14:53:58.342683] W [MSGID: 108001] [afr-common.c:4090:afr_notify] 0-gluster-replicate-0: Client-quorum is not met
[2016-05-25 14:53:58.342729] E [MSGID: 108006] [afr-common.c:4046:afr_notify] 0-gluster-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2016-05-25 14:53:58.343159] W [inode.c:1792:inode_table_destroy] (-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3f9) [0x79401d003239] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x4e) [0x79401cd7484e] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xe3) [0x79401cd74753] ) 0-gfapi: Active inode(0x7940020002cc) with refcount(1) found during cleanup
[2016-05-25 14:53:58.344168] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=84 max=439 total=439
[2016-05-25 14:53:58.345653] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=156 max=444 total=1051
[2016-05-25 14:53:58.348407] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=108 max=33 total=17888
[2016-05-25 14:53:58.348468] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=588 max=111 total=401948
[2016-05-25 14:53:58.348488] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=124 max=111 total=401948
[2016-05-25 14:53:58.348508] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=1300 max=64 total=173435
[2016-05-25 14:53:58.348525] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=588 max=111 total=402594
[2016-05-25 14:53:58.348670] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=124 max=111 total=402594
[2016-05-25 14:53:58.348691] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=1300 max=64 total=174080
[2016-05-25 14:53:58.348709] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=588 max=62 total=426158
[2016-05-25 14:53:58.348722] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=124 max=62 total=426158
[2016-05-25 14:53:58.348737] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=1300 max=62 total=197641
[2016-05-25 14:53:58.348753] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-replicate-0: size=10524 max=243 total=453393
[2016-05-25 14:53:58.349278] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=1148 max=0 total=0
[2016-05-25 14:53:58.349438] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=2284 max=111 total=343736
[2016-05-25 14:53:58.349672] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-shard: size=1396 max=110 total=115585
[2016-05-25 14:53:58.349693] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-readdir-ahead: size=52 max=0 total=0
[2016-05-25 14:53:58.349713] I [io-stats.c:2951:fini] 0-gluster: io-stats translator unloaded
[2016-05-25 14:53:58.350395] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2
[2016-05-25 14:53:58.350407] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1
[2016-05-25 14:57:19.258341] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e733339-3635-3033-2e69-702d34362d31 (0) coming up
[2016-05-25 14:57:19.258393] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport
[2016-05-25 14:57:19.258937] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport
[2016-05-25 14:57:19.259336] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport
[2016-05-25 14:57:19.259691] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0)
[2016-05-25 14:57:19.259944] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0)
[2016-05-25 14:57:19.260495] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0)
[2016-05-25 14:57:19.260854] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 14:57:19.260954] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 14:57:19.261137] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 14:57:19.261371] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 14:57:19.261394] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 14:57:19.261415] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 14:57:19.261432] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 14:57:19.261472] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online.
[2016-05-25 14:57:19.261575] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 14:57:19.261597] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 14:57:19.261602] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1
[2016-05-25 14:57:19.261647] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1
[2016-05-25 14:57:19.272598] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1
[2016-05-25 14:57:19.274117] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-2
[2016-05-25 14:57:19.274445] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 6e733339-3635-3033-2e69-702d34362d31 (0)
[2016-05-25 15:29:42.577939] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-0: current graph is no longer active, destroying rpc_client
[2016-05-25 15:29:42.577977] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-1: current graph is no longer active, destroying rpc_client
[2016-05-25 15:29:42.577994] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-2: current graph is no longer active, destroying rpc_client
[2016-05-25 15:29:42.577991] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-0: disconnected from gluster-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 15:29:42.578032] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-1: disconnected from gluster-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 15:29:42.578043] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-2: disconnected from gluster-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 15:29:42.578079] W [MSGID: 108001] [afr-common.c:4090:afr_notify] 0-gluster-replicate-0: Client-quorum is not met
[2016-05-25 15:29:42.578112] E [MSGID: 108006] [afr-common.c:4046:afr_notify] 0-gluster-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2016-05-25 15:29:42.578342] W [inode.c:1792:inode_table_destroy] (-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3f9) [0x71a50b99e239] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x4e) [0x71a50b70f84e] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xe3) [0x71a50b70f753] ) 0-gfapi: Active inode(0x71a4f0c002cc) with refcount(1) found during cleanup
[2016-05-25 15:29:42.579064] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=84 max=464 total=464
[2016-05-25 15:29:42.580114] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=156 max=466 total=1362
[2016-05-25 15:29:42.582010] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=108 max=32 total=28137
[2016-05-25 15:29:42.582026] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=588 max=59 total=394992
[2016-05-25 15:29:42.582035] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=124 max=59 total=394992
[2016-05-25 15:29:42.582045] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=1300 max=59 total=148576
[2016-05-25 15:29:42.582132] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=588 max=60 total=395060
[2016-05-25 15:29:42.582141] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=124 max=60 total=395060
[2016-05-25 15:29:42.582149] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=1300 max=59 total=148641
[2016-05-25 15:29:42.582159] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=588 max=36 total=432612
[2016-05-25 15:29:42.582185] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=124 max=36 total=432612
[2016-05-25 15:29:42.582216] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=1300 max=36 total=186181
[2016-05-25 15:29:42.582296] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-replicate-0: size=10524 max=125 total=513483
[2016-05-25 15:29:42.582586] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=1148 max=0 total=0
[2016-05-25 15:29:42.582664] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=2284 max=53 total=397149
[2016-05-25 15:29:42.582787] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-shard: size=1396 max=53 total=151134
[2016-05-25 15:29:42.582800] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-readdir-ahead: size=52 max=0 total=0
[2016-05-25 15:29:42.582819] I [io-stats.c:2951:fini] 0-gluster: io-stats translator unloaded
[2016-05-25 15:29:42.583446] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1
[2016-05-25 15:29:42.583441] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2
-------------- next part --------------
[2016-05-25 15:30:28.098039] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 73322e70-6172-6170-6861-726d617a656e (0) coming up
[2016-05-25 15:30:28.098100] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport
[2016-05-25 15:30:28.098744] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport
[2016-05-25 15:30:28.099261] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport
[2016-05-25 15:30:28.100732] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0)
[2016-05-25 15:30:28.100808] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0)
[2016-05-25 15:30:28.100880] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0)
[2016-05-25 15:30:28.101713] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 15:30:28.101959] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 15:30:28.102139] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 15:30:28.102150] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 15:30:28.102175] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 15:30:28.102248] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online.
[2016-05-25 15:30:28.102287] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1
[2016-05-25 15:30:28.102400] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 15:30:28.102413] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 15:30:28.102605] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1
[2016-05-25 15:30:28.102654] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 15:30:28.102671] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 15:30:28.118768] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1
[2016-05-25 15:30:28.120203] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-0
[2016-05-25 15:30:28.120644] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 73322e70-6172-6170-6861-726d617a656e (0)
[2016-05-25 15:41:56.201463] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-0: current graph is no longer active, destroying rpc_client
[2016-05-25 15:41:56.201533] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-1: current graph is no longer active, destroying rpc_client
[2016-05-25 15:41:56.201566] I [MSGID: 114021] [client.c:2115:notify] 0-gluster-client-2: current graph is no longer active, destroying rpc_client
[2016-05-25 15:41:56.201576] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-0: disconnected from gluster-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 15:41:56.201587] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-1: disconnected from gluster-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 15:41:56.201632] W [MSGID: 108001] [afr-common.c:4090:afr_notify] 0-gluster-replicate-0: Client-quorum is not met
[2016-05-25 15:41:56.201638] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-2: disconnected from gluster-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-25 15:41:56.201670] E [MSGID: 108006] [afr-common.c:4046:afr_notify] 0-gluster-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2016-05-25 15:41:56.207335] W [inode.c:1792:inode_table_destroy] (-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3f9) [0x64a06250d239] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x4e) [0x64a06227e84e] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xe3) [0x64a06227e753] ) 0-gfapi: Active inode(0x64a0478002cc) with refcount(1) found during cleanup
[2016-05-25 15:41:56.207969] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=84 max=338 total=338
[2016-05-25 15:41:56.209033] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=156 max=340 total=877
[2016-05-25 15:41:56.210896] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gfapi: size=108 max=38 total=14869
[2016-05-25 15:41:56.210922] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=588 max=25 total=173570
[2016-05-25 15:41:56.210932] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=124 max=25 total=173570
[2016-05-25 15:41:56.210945] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-0: size=1300 max=24 total=77362
[2016-05-25 15:41:56.211045] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=588 max=28 total=156927
[2016-05-25 15:41:56.211057] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=124 max=28 total=156927
[2016-05-25 15:41:56.211068] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-1: size=1300 max=27 total=60719
[2016-05-25 15:41:56.211079] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=588 max=26 total=156954
[2016-05-25 15:41:56.211117] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=124 max=26 total=156954
[2016-05-25 15:41:56.211151] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-client-2: size=1300 max=25 total=60745
[2016-05-25 15:41:56.211257] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-replicate-0: size=10524 max=60 total=207019
[2016-05-25 15:41:56.211549] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=1148 max=0 total=0
[2016-05-25 15:41:56.211630] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-dht: size=2284 max=21 total=159841
[2016-05-25 15:41:56.211771] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-shard: size=1396 max=20 total=63784
[2016-05-25 15:41:56.211788] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-gluster-readdir-ahead: size=52 max=0 total=0
[2016-05-25 15:41:56.211801] I [io-stats.c:2951:fini] 0-gluster: io-stats translator unloaded
[2016-05-25 15:41:56.211967] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1
[2016-05-25 15:41:56.211963] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2
[2016-05-25 15:42:58.096640] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 73322e70-6172-6170-6861-726d617a656e (0) coming up
[2016-05-25 15:42:58.096687] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport
[2016-05-25 15:42:58.097266] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport
[2016-05-25 15:42:58.097518] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0)
[2016-05-25 15:42:58.097815] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport
[2016-05-25 15:42:58.098087] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 15:42:58.098299] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0)
[2016-05-25 15:42:58.098554] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 15:42:58.098573] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 15:42:58.098633] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online.
[2016-05-25 15:42:58.098673] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1
[2016-05-25 15:42:58.098790] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0)
[2016-05-25 15:42:58.098898] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 15:42:58.099345] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 15:42:58.099360] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 15:42:58.099459] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 15:42:58.099533] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1
[2016-05-25 15:42:58.101238] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 15:42:58.101269] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 15:42:58.123883] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1
[2016-05-25 15:42:58.125043] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-0
[2016-05-25 15:42:58.125633] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 73322e70-6172-6170-6861-726d617a656e (0)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: VM.png
Type: image/png
Size: 48505 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160525/01977111/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160525/01977111/attachment.sig>
More information about the Gluster-users
mailing list