<div dir="ltr"><div><div>The newly introduced "SEEK" fop seems to be failing at the bricks.<br><br></div>Adding Niels for his inputs/help.<br></div><div><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 8, 2017 at 3:43 PM, Alessandro Briosi <span dir="ltr"><<a href="mailto:ab1@metalit.com" target="_blank">ab1@metalit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
I have sporadic VM going down which files are on gluster FS.<br>
<br>
If I look at the gluster logs the only events I find are:<br>
/var/log/glusterfs/bricks/<wbr>data-brick2-brick.log<br>
<br>
[2017-05-08 09:51:17.661697] I [MSGID: 115036]<br>
[server.c:548:server_rpc_<wbr>notify] 0-datastore2-server: disconnecting<br>
connection from<br>
srvpve2-9074-2017/05/04-14:12:<wbr>53:301448-datastore2-client-0-<wbr>0-0<br>
[2017-05-08 09:51:17.661697] I [MSGID: 115036]<br>
[server.c:548:server_rpc_<wbr>notify] 0-datastore2-server: disconnecting<br>
connection from<br>
srvpve2-9074-2017/05/04-14:12:<wbr>53:367950-datastore2-client-0-<wbr>0-0<br>
[2017-05-08 09:51:17.661810] W [inodelk.c:399:pl_inodelk_log_<wbr>cleanup]<br>
0-datastore2-server: releasing lock on<br>
66d9eefb-ee55-40ad-9f44-<wbr>c55d1e809006 held by {client=0x7f4c7c004880,<br>
pid=0 lk-owner=5c7099efc97f0000}<br>
[2017-05-08 09:51:17.661810] W [inodelk.c:399:pl_inodelk_log_<wbr>cleanup]<br>
0-datastore2-server: releasing lock on<br>
a8d82b3d-1cf9-45cf-9858-<wbr>d8546710b49c held by {client=0x7f4c840f31d0,<br>
pid=0 lk-owner=5c7019fac97f0000}<br>
[2017-05-08 09:51:17.661835] I [MSGID: 115013]<br>
[server-helpers.c:293:do_fd_<wbr>cleanup] 0-datastore2-server: fd cleanup on<br>
/images/201/vm-201-disk-2.<wbr>qcow2<br>
[2017-05-08 09:51:17.661838] I [MSGID: 115013]<br>
[server-helpers.c:293:do_fd_<wbr>cleanup] 0-datastore2-server: fd cleanup on<br>
/images/201/vm-201-disk-1.<wbr>qcow2<br>
[2017-05-08 09:51:17.661953] I [MSGID: 101055]<br>
[client_t.c:415:gf_client_<wbr>unref] 0-datastore2-server: Shutting down<br>
connection srvpve2-9074-2017/05/04-14:12:<wbr>53:301448-datastore2-client-0-<wbr>0-0<br>
[2017-05-08 09:51:17.661953] I [MSGID: 101055]<br>
[client_t.c:415:gf_client_<wbr>unref] 0-datastore2-server: Shutting down<br>
connection srvpve2-9074-2017/05/04-14:12:<wbr>53:367950-datastore2-client-0-<wbr>0-0<br>
[2017-05-08 10:01:06.210392] I [MSGID: 115029]<br>
[server-handshake.c:692:<wbr>server_setvolume] 0-datastore2-server: accepted<br>
client from<br>
srvpve2-162483-2017/05/08-10:<wbr>01:06:189720-datastore2-<wbr>client-0-0-0<br>
(version: 3.8.11)<br>
[2017-05-08 10:01:06.237433] E [MSGID: 113107] [posix.c:1079:posix_seek]<br>
0-datastore2-posix: seek failed on fd 18 length 42957209600 [No such<br>
device or address]<br>
[2017-05-08 10:01:06.237463] E [MSGID: 115089]<br>
[server-rpc-fops.c:2007:<wbr>server_seek_cbk] 0-datastore2-server: 18: SEEK-2<br>
(a8d82b3d-1cf9-45cf-9858-<wbr>d8546710b49c) ==> (No such device or address)<br>
[No such device or address]<br>
[2017-05-08 10:01:07.019974] I [MSGID: 115029]<br>
[server-handshake.c:692:<wbr>server_setvolume] 0-datastore2-server: accepted<br>
client from<br>
srvpve2-162483-2017/05/08-10:<wbr>01:07:3687-datastore2-client-<wbr>0-0-0<br>
(version: 3.8.11)<br>
[2017-05-08 10:01:07.041967] E [MSGID: 113107] [posix.c:1079:posix_seek]<br>
0-datastore2-posix: seek failed on fd 19 length 859136720896 [No such<br>
device or address]<br>
[2017-05-08 10:01:07.041992] E [MSGID: 115089]<br>
[server-rpc-fops.c:2007:<wbr>server_seek_cbk] 0-datastore2-server: 18: SEEK-2<br>
(66d9eefb-ee55-40ad-9f44-<wbr>c55d1e809006) ==> (No such device or address)<br>
[No such device or address]<br>
<br>
The strange part is that I cannot seem to find any other error.<br>
If I restart the VM everything works as expected (it stopped at ~9.51<br>
UTC and was started at ~10.01 UTC) .<br>
<br>
This is not the first time that this happened, and I do not see any<br>
problems with networking or the hosts.<br>
<br>
Gluster version is 3.8.11<br>
this is the incriminated volume (though it happened on a different one too)<br>
<br>
Volume Name: datastore2<br>
Type: Replicate<br>
Volume ID: c95ebb5f-6e04-4f09-91b9-<wbr>bbbe63d83aea<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x (2 + 1) = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: srvpve2g:/data/brick2/brick<br>
Brick2: srvpve3g:/data/brick2/brick<br>
Brick3: srvpve1g:/data/brick2/brick (arbiter)<br>
Options Reconfigured:<br>
nfs.disable: on<br>
performance.readdir-ahead: on<br>
transport.address-family: inet<br>
<br>
Any hint on how to dig more deeply into the reason would be greatly<br>
appreciated.<br>
<br>
Alessandro<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div>