<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 05/11/2017 05:49 PM, Niels de Vos
wrote:<br>
</div>
<blockquote
cite="mid:20170511121924.GD11257@ndevos-x240.usersys.redhat.com"
type="cite">
<pre wrap="">On Wed, May 10, 2017 at 09:08:03PM +0530, Pranith Kumar Karampuri wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Wed, May 10, 2017 at 7:11 PM, Niels de Vos <a class="moz-txt-link-rfc2396E" href="mailto:ndevos@redhat.com"><ndevos@redhat.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Wed, May 10, 2017 at 04:08:22PM +0530, Pranith Kumar Karampuri wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Tue, May 9, 2017 at 7:40 PM, Niels de Vos <a class="moz-txt-link-rfc2396E" href="mailto:ndevos@redhat.com"><ndevos@redhat.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">...
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">client from
srvpve2-162483-2017/05/08-10:01:06:189720-datastore2-client-0-0-0
(version: 3.8.11)
[2017-05-08 10:01:06.237433] E [MSGID: 113107]
</pre>
</blockquote>
</blockquote>
<pre wrap="">[posix.c:1079:posix_seek]
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">0-datastore2-posix: seek failed on fd 18 length 42957209600 [No
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">such
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">device or address]
</pre>
</blockquote>
</blockquote>
<pre wrap="">
The SEEK procedure translates to lseek() in the posix xlator. This can
return with "No suck device or address" (ENXIO) in only one case:
ENXIO whence is SEEK_DATA or SEEK_HOLE, and the file offset is
beyond the end of the file.
This means that an lseek() was executed where the current offset of the
filedescriptor was higher than the size of the file. I'm not sure how
that could happen... Sharding prevents using SEEK at all atm.
...
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">The strange part is that I cannot seem to find any other error.
If I restart the VM everything works as expected (it stopped at
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">~9.51
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">UTC and was started at ~10.01 UTC) .
This is not the first time that this happened, and I do not see any
problems with networking or the hosts.
Gluster version is 3.8.11
this is the incriminated volume (though it happened on a different
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">one
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">too)
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">
Volume Name: datastore2
Type: Replicate
Volume ID: c95ebb5f-6e04-4f09-91b9-bbbe63d83aea
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: srvpve2g:/data/brick2/brick
Brick2: srvpve3g:/data/brick2/brick
Brick3: srvpve1g:/data/brick2/brick (arbiter)
Options Reconfigured:
nfs.disable: on
performance.readdir-ahead: on
transport.address-family: inet
Any hint on how to dig more deeply into the reason would be greatly
appreciated.
</pre>
</blockquote>
</blockquote>
<pre wrap="">
Probably the problem is with SEEK support in the arbiter functionality.
Just like with a READ or a WRITE on the arbiter brick, SEEK can only
succeed on bricks where the files with content are located. It does not
look like arbiter handles SEEK, so the offset in lseek() will likely be
higher than the size of the file on the brick (empty, 0 size file). I
don't know how the replication xlator responds on an error return from
SEEK on one of the bricks, but I doubt it likes it.
</pre>
</blockquote>
<pre wrap="">
inode-read fops don't get sent to arbiter brick. So this won't happen.
</pre>
</blockquote>
<pre wrap="">
Yes, I see that the arbiter xlator returns on reads without going to the
bricks. Should that not be done for seek as well? It's the first time I
actually looked at the code of the arbiter xlator, so I might well be
misunderstanding how it works :)
</pre>
</blockquote>
<pre wrap="">
inode-read fops are the fops which read some information from the inode.
Like stat/getxattr/read. Even seek falls in that category. It is not sent
on arbiter brick...
</pre>
</blockquote>
<pre wrap="">
What confuses me is that the arbiter xlator defines the following FOPs
in xlators/features/arbiter/src/arbiter.c:</pre>
</blockquote>
AFR has a list of readable subvols on which all read related FOPS
are wound. For arbiter volumes, we mark the arbiter as non-readable
during lookup cbk.<br>
So any read FOP is not wound to arbiter anymore. This change was
made at a later stage after arbiter_readv was coded initially to
send an error. So in the current code, arbiter_readv should never
get hit.<br>
<br>
<blockquote
cite="mid:20170511121924.GD11257@ndevos-x240.usersys.redhat.com"
type="cite">
<pre wrap="">
struct xlator_fops fops = {
.lookup = arbiter_lookup,
.readv = arbiter_readv,
.truncate = arbiter_truncate,
.writev = arbiter_writev,
.ftruncate = arbiter_ftruncate,
.fallocate = arbiter_fallocate,
.discard = arbiter_discard,
.zerofill = arbiter_zerofill,
};
To go back to the error message:
[posix.c:1079:posix_seek] 0-datastore2-posix: seek failed on fd 18 length 42957209600 [No such device or address]
We need to know on which brick this occurs to confirm that is was not
sent on the arbiter brick somehow.</pre>
</blockquote>
<br>
This is what Alessandro said earlier in the thread:<br>
<br>
"Also the seek errors where there before when there was no arbiter
(only 2 replica)."<br>
<blockquote
cite="mid:20170511121924.GD11257@ndevos-x240.usersys.redhat.com"
type="cite">
<pre wrap="">
Thanks,
Niels
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users">http://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<p><br>
</p>
</body>
</html>