[Gluster-users] VM disks corruption on 3.7.11

Wed May 25 14:10:02 UTC 2016

Just did that, below is the output.
Didn't seem to move after the boot, and no new lines when the I/O errors appeared.
Also, as mentionned I tried moving the disk on NFS and had the exact same errors,
so it doesn't look like it's a libgfapi problem ..
I should probably re-create the VM, maybe the errors from this night corrupted
the disk and I now get errors unrelated to the original issue.

Let me re-create the VM from scratch and try to reproduce the problem with
the logs enabled, maybe it'll be more informative than this !

[2016-05-25 13:56:30.851493] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e733339-3635-3033-2e69-702d34362d31 (0) coming up
[2016-05-25 13:56:30.851553] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport
[2016-05-25 13:56:30.852130] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport
[2016-05-25 13:56:30.852650] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport
[2016-05-25 13:56:30.852909] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0)
[2016-05-25 13:56:30.853434] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0)
[2016-05-25 13:56:30.853484] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49152 (from 0)
[2016-05-25 13:56:30.854182] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 13:56:30.854398] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 13:56:30.854441] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-25 13:56:30.861931] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 13:56:30.861965] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 13:56:30.862073] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-2' came back up; going online.
[2016-05-25 13:56:30.862139] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1
[2016-05-25 13:56:30.865451] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 13:56:30.865485] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 13:56:30.865757] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1
[2016-05-25 13:56:30.865826] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'.
[2016-05-25 13:56:30.865841] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-25 13:56:30.888604] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1
[2016-05-25 13:56:30.890388] I [MSGID: 108031] [afr-common.c:1900:afr_local_discovery_cbk] 0-gluster-replicate-0: selecting local read_child gluster-client-2
[2016-05-25 13:56:30.890731] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-gluster: switched to graph 6e733339-3635-3033-2e69-702d34362d31 (0)

On Wed, May 25, 2016 at 02:48:27PM +0530, Krutika Dhananjay wrote:
>    Also, it seems Lindsay knows a way to get the gluster client logs when
>    using proxmox and libgfapi.
>    Would it be possible for you to get that sorted with Lindsay's help before
>    recreating this issue next time
>    and share the glusterfs client logs from all the nodes when you do hit the
>    issue?
>    It is critical for some of the debugging we do. :)
> 
>    -Krutika
>    On Wed, May 25, 2016 at 2:38 PM, Krutika Dhananjay <kdhananj at redhat.com>
>    wrote:
> 
>      Hi Kevin,
> 
>      If you actually ran into a 'read-only filesystem' issue, then it could
>      possibly because of a bug in AFR
>      that Pranith recently fixed.
>      To confirm if that is indeed the case, could you tell meA  if you saw
>      the pause after a brick (single brick) was
>      down while IO was going on?
> 
>      -Krutika
>      On Wed, May 25, 2016 at 1:28 PM, Kevin Lemonnier <lemonnierk at ulrar.net>
>      wrote:
> 
>        >A  A  Whats the underlying filesystem under the bricks?
> 
>        I use XFS, I read that was recommended. What are you using ?
>        Since yours seems to work, I'm not opposed to changing !
>        --
>        Kevin Lemonnier
>        PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
>        _______________________________________________
>        Gluster-users mailing list
>        Gluster-users at gluster.org
>        http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160525/db340fb3/attachment.sig>