<div dir="ltr"><div dir="ltr">Hi Jose,<div><br></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 4, 2021 at 1:58 PM José Manuel Blanco <<a href="mailto:josemanuel.blanco@uva.es">josemanuel.blanco@uva.es</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><font size="+1"><font face="monospace">Hi all,</font></font></p>
<p><font size="+1"><font face="monospace">We have a problem with
tasks (written in PHP) doing a lot of file renaming/moving
(even several renames per second).</font></font></p>
<p><font size="+1"><font face="monospace">The pattern is always the
same: the task rename files with variable filename to the SAME
final file (that is: the destination filename is ALWAYS the
same).</font></font></p>
<p><font size="+1"><font face="monospace">Problem: according to
glusterfs and task logs, some rename are successful but other
don't and we don't understand why or what is causing the error
in the failing renames because the error is "File exists", but
the rename supossedly must address these situations correctly
and "overwrite" the destination file if it exists (it uses the
PHP rename() function)<br>
</font></font></p>
<p><font size="+1"><font face="monospace">More info:<br>
</font></font></p>
<p><font size="+1"><font face="monospace">- We use a
distributed-replicated volume in a 3 node cluster</font></font></p>
<blockquote>
<p><font size="+1"><font face="monospace">gluster volume info
moodle-cv<br>
<br>
</font></font><font face="monospace">Volume Name: moodle-cv</font><br>
<font face="monospace">Type: Distributed-Replicate</font><br>
<font face="monospace">Volume ID:
1eef9714-3943-4d77-b42f-6b1144389c56</font><br>
<font face="monospace">Status: Started</font><br>
<font face="monospace">Snapshot Count: 0</font><br>
<font face="monospace">Number of Bricks: 5 x (2 + 1) = 15</font><br>
<font face="monospace">Transport-type: tcp</font><br>
<font face="monospace">Bricks:</font><br>
<font face="monospace">Brick1:
moodle2017-n2:/glusterfs/moodle-cv/brick1/datos</font><br>
<font face="monospace">Brick2:
moodle2017-n3:/glusterfs/moodle-cv/brick1/replica</font><br>
<font face="monospace">Brick3:
moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter (arbiter)</font><br>
<font face="monospace">Brick4:
moodle2017-n3:/glusterfs/moodle-cv/brick2/datos</font><br>
<font face="monospace">Brick5:
moodle2017-n2:/glusterfs/moodle-cv/brick2/replica</font><br>
<font face="monospace">Brick6:
moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter (arbiter)</font><br>
<font face="monospace">Brick7:
moodle2017-n2:/glusterfs/moodle-cv/brick3/datos</font><br>
<font face="monospace">Brick8:
moodle2017-n3:/glusterfs/moodle-cv/brick3/replica</font><br>
<font face="monospace">Brick9:
moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter (arbiter)</font><br>
<font face="monospace">Brick10:
moodle2017-n3:/glusterfs/moodle-cv/brick4/datos</font><br>
<font face="monospace">Brick11:
moodle2017-n2:/glusterfs/moodle-cv/brick4/replica</font><br>
<font face="monospace">Brick12:
moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter (arbiter)</font><br>
<font face="monospace">Brick13:
moodle2017-n2:/glusterfs/moodle-cv/brick5/datos</font><br>
<font face="monospace">Brick14:
moodle2017-n3:/glusterfs/moodle-cv/brick5/replica</font><br>
<font face="monospace">Brick15:
moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter (arbiter)</font><br>
<font face="monospace">Options Reconfigured:</font><br>
<font face="monospace">nfs.disable: on</font><br>
<font face="monospace">storage.fips-mode-rchecksum: on</font><br>
<font face="monospace">performance.open-behind: off</font><br>
<font face="monospace">performance.lazy-open: no</font><br>
<font face="monospace">cluster.self-heal-daemon: enable</font><br>
</p>
</blockquote>
<p><font size="+1"><font face="monospace">- The nodes use Oracle
Linux 7.9 (RedHat clone) and GlusterFS 8.5</font></font></p>
<p><font size="+1"><font face="monospace">- The tasks use the PHP
rename() function<br>
</font></font></p>
<p><font size="+1"><font face="monospace">- The tasks renaming files
are executed IN ONE NODE OF THE CLUSTER that mounts the volume
using the FUSE client</font></font></p>
<p><font size="+1"><font face="monospace">- At the end of the
message I paste a (very) little fragment of the volume log</font></font></p>
<p><font size="+1"><font face="monospace"><br>
</font></font></p>
<p><font size="+1"><font face="monospace">Any ideas of the possible
cause of the problem and/or suggestions to avoid it?</font></font></p></div></blockquote><div><br></div><div>Most probably the issue is caused by special internal files needed by Gluster that are used to reference the correct location of a file in a distributed volume (they are referred as linkto files). I would say the error happens because those files already exist when it's expected that they don't exist (I think it's a similar problem as <a href="https://github.com/gluster/glusterfs/issues/1723">https://github.com/gluster/glusterfs/issues/1723</a>).</div><div><br></div><div>Are you using FUSE mounts ? are you doing the renames from more than one client ?</div><div><br></div><div>Regards,</div><div><br></div><div>Xavi</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p><font size="+1"><font face="monospace">Regards.</font></font></p>
<p><font size="+1"><font face="monospace"><br>
</font></font></p>
<p><font size="+1"><font face="monospace">This is the fragment of
the log:</font></font></p>
<p><font size="+1"><font face="monospace">* I've "separated" the
lines by "rename operation" but all the lines appear together
in the log<br>
</font></font></p>
<blockquote>
<p><font face="monospace">--->SUCCESSFUL RENAME:</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.478321] I [MSGID:
109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp
(4a875eac-5389-400c-9359-458e151054f7)
(hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
(11fdf019-16f8-48ae-bd1f-05b670b29ec1)
(hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2)</font></p>
<p><font face="monospace">--->UNSUCCESSFUL RENAME: <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.497845] I [MSGID:
109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
(350698df-35d9-4489-90cc-faa004bfc861)
(hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
(4a875eac-5389-400c-9359-458e151054f7)
(hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.503271] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-0: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.503339] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-1: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.503393] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-2: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.504997] W
[fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
781438412:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
=> -1 (File exists)</font></p>
<p><font face="monospace">--->UNSUCCESSFUL RENAME:<br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.729314] I [MSGID:
109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
(5bff6629-80b8-4d9c-8e37-abfa7691124e)
(hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
(4a875eac-5389-400c-9359-458e151054f7)
(hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.735631] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-5: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.735709] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-4: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.735747] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-3: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.737356] W
[fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
781438861:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
=> -1 (File exists)</font><font size="+1"><font face="monospace"><br>
</font></font></p>
<p><font face="monospace">--->SUCCESSFUL RENAME:</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.799776] I [MSGID:
109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp
(82725bab-b888-4d24-b239-31b86f50425f)
(hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
(4a875eac-5389-400c-9359-458e151054f7)
(hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) <br>
</font></p>
<p><font face="monospace">--->UNSUCCESFULL RENAME:<br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.963545] I [MSGID:
109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
(b3c5435c-f79b-402f-ba00-0e78dcdb6cd2)
(hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
(82725bab-b888-4d24-b239-31b86f50425f)
(hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.968665] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-5: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.968801] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-4: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.968847] W [MSGID:
114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
2-moodle-cv-client-3: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
{errno=17}, {error=File exists}] <br>
</font></p>
<p><font face="monospace">[2021-09-03 09:04:02.970438] W
[fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
781439391:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
=> -1 (File exists)</font><font size="+1"><font face="monospace"><br>
</font></font></p>
</blockquote>
<p><font size="+1"><font face="monospace"><br>
</font></font></p>
<div>-- <br>
<br>
<table>
<tbody>
<tr>
<td width="100"><a style="float:left;margin-right:1em" href="http://www.uva.es" target="_blank"> </a>
<div><a style="float:left;margin-right:1em" href="http://www.uva.es" target="_blank"> </a><a href="http://www.uva.es" target="_blank">
<img alt="Escudo UVa" src="https://webmail.uva.es/images/logo_UVa_firma.gif" width="80" height="80">
</a> </div>
</td>
<td>
<p><span>Jose Manuel Blanco</span><br>
<br>
<span>Analista Sistemas</span> · <span><a href="mailto:josemanuel.blanco@uva.es" target="_blank">josemanuel.blanco@uva.es</a></span><br>
<span>Tfno.: (+34) 983 18 6411</span></p>
</td>
</tr>
<tr>
<td colspan="2">
<p><span>Universidad de Valladolid</span><br>
<span>Servicio de las TIC</span><br>
<br>
<span>Edificio Alfonso VIII · C/ Real de Burgos s/n<br>
47011 · Valladolid · España (<em>Spain</em>)</span></p>
<p>| <a href="https://www.facebook.com/Universidad-de-Valladolid-187763507920209/" target="_blank">Facebook</a> | <a href="https://twitter.com/uva_es" target="_blank">Twitter</a>
| <a href="http://www.uva.es" target="_blank">UVa</a> </p>
</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>
<p>Este mensaje puede contener información confidencial,
sometida al secreto profesional, cuya divulgación no
está permitida por la ley. Si usted no es su
destinatario, por favor, notifíquelo al remitente y
borre este correo de su sistema. A los efectos de la
protección de datos y el RGPD, consulte: <a href="http://www.uva.es/protecciondedatos/" target="_blank">protección de datos en la UVa</a>. El
emisor no garantiza la integridad, rapidez o seguridad
del presente correo, ni se responsabiliza de posibles
perjuicios derivados de la captura, incorporaciones de
virus o cualesquiera otras manipulaciones efectuadas por
terceros. <span>Piensa en verde: lee en
la pantalla.</span></p>
<p>This message may contain confidential information
covered by the obligation of professional secrecy, the
disclosure of which would be contrary to the law. If you
are not the intended recipient, please advise the sender
and delete this e-mail from your system. For the
purposes of data protection and GDPR, read: <a href="http://www.uva.es/protecciondedatos/" target="_blank">protección de datos en la UVa</a>. The
sender does not guarantee the integrity, the accuracy,
the swift delivery or the security of this email
transmission, and assumes no responsibility for any
possible damage incurred through data capture, virus
incorporation or any manipulation carried out by third
parties. <span>Think green: read on the
screen.</span></p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div></div>