[Gluster-users] Problem with massive file renaming in glusterfs volume

José Manuel Blanco josemanuel.blanco at uva.es
Sat Sep 4 11:07:04 UTC 2021 

Hi all,

We have a problem with tasks (written in PHP) doing a lot of file
renaming/moving (even several renames per second).

The pattern is always the same: the task rename files with variable
filename to the SAME final file (that is: the destination filename is
ALWAYS the same).

Problem: according to glusterfs and task logs, some rename are
successful but other don't and we don't understand why or what is
causing the error in the failing renames because the error is "File
exists", but the rename supossedly must address these situations
correctly and "overwrite" the destination file if it exists (it uses the
PHP rename() function)

More info:

- We use a distributed-replicated volume in a 3 node cluster

    gluster volume info moodle-cv
     
    Volume Name: moodle-cv
    Type: Distributed-Replicate
    Volume ID: 1eef9714-3943-4d77-b42f-6b1144389c56
    Status: Started
    Snapshot Count: 0
    Number of Bricks: 5 x (2 + 1) = 15
    Transport-type: tcp
    Bricks:
    Brick1: moodle2017-n2:/glusterfs/moodle-cv/brick1/datos
    Brick2: moodle2017-n3:/glusterfs/moodle-cv/brick1/replica
    Brick3: moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter (arbiter)
    Brick4: moodle2017-n3:/glusterfs/moodle-cv/brick2/datos
    Brick5: moodle2017-n2:/glusterfs/moodle-cv/brick2/replica
    Brick6: moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter (arbiter)
    Brick7: moodle2017-n2:/glusterfs/moodle-cv/brick3/datos
    Brick8: moodle2017-n3:/glusterfs/moodle-cv/brick3/replica
    Brick9: moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter (arbiter)
    Brick10: moodle2017-n3:/glusterfs/moodle-cv/brick4/datos
    Brick11: moodle2017-n2:/glusterfs/moodle-cv/brick4/replica
    Brick12: moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter (arbiter)
    Brick13: moodle2017-n2:/glusterfs/moodle-cv/brick5/datos
    Brick14: moodle2017-n3:/glusterfs/moodle-cv/brick5/replica
    Brick15: moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter (arbiter)
    Options Reconfigured:
    nfs.disable: on
    storage.fips-mode-rchecksum: on
    performance.open-behind: off
    performance.lazy-open: no
    cluster.self-heal-daemon: enable

- The nodes use Oracle Linux 7.9 (RedHat clone) and GlusterFS 8.5

- The tasks use the PHP rename() function

- The tasks renaming files are executed IN ONE NODE OF THE CLUSTER that
mounts the volume using the FUSE client

- At the end of the message I paste a (very) little fragment of the
volume log


Any ideas of the possible cause of the problem and/or suggestions to
avoid it?

Regards.


This is the fragment of the log:

* I've "separated" the lines by "rename operation" but all the lines
appear together in the log

    --->SUCCESSFUL RENAME:

    [2021-09-03 09:04:02.478321] I [MSGID: 109066]
    [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp
    (4a875eac-5389-400c-9359-458e151054f7)
    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
    (11fdf019-16f8-48ae-bd1f-05b670b29ec1)
    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2)

    --->UNSUCCESSFUL RENAME: 

    [2021-09-03 09:04:02.497845] I [MSGID: 109066]
    [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
    (350698df-35d9-4489-90cc-faa004bfc861)
    (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) =>
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
    (4a875eac-5389-400c-9359-458e151054f7)
    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

    [2021-09-03 09:04:02.503271] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-0:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.503339] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-1:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.503393] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-2:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.504997] W [fuse-bridge.c:2500:fuse_rename_cbk]
    0-glusterfs-fuse: 781438412:
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
    ->
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
    => -1 (File exists)

    --->UNSUCCESSFUL RENAME:

    [2021-09-03 09:04:02.729314] I [MSGID: 109066]
    [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
    (5bff6629-80b8-4d9c-8e37-abfa7691124e)
    (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
    (4a875eac-5389-400c-9359-458e151054f7)
    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

    [2021-09-03 09:04:02.735631] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.735709] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.735747] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.737356] W [fuse-bridge.c:2500:fuse_rename_cbk]
    0-glusterfs-fuse: 781438861:
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
    ->
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
    => -1 (File exists)

    --->SUCCESSFUL RENAME:

    [2021-09-03 09:04:02.799776] I [MSGID: 109066]
    [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp
    (82725bab-b888-4d24-b239-31b86f50425f)
    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
    (4a875eac-5389-400c-9359-458e151054f7)
    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

    --->UNSUCCESFULL RENAME:

    [2021-09-03 09:04:02.963545] I [MSGID: 109066]
    [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
    (b3c5435c-f79b-402f-ba00-0e78dcdb6cd2)
    (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
    (82725bab-b888-4d24-b239-31b86f50425f)
    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

    [2021-09-03 09:04:02.968665] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.968801] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.968847] W [MSGID: 114031]
    [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3:
    remote operation failed.
    [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
    {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
    {errno=17}, {error=File exists}]

    [2021-09-03 09:04:02.970438] W [fuse-bridge.c:2500:fuse_rename_cbk]
    0-glusterfs-fuse: 781439391:
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
    ->
    /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
    => -1 (File exists)


-- 

<http://www.uva.es>
<http://www.uva.es>Escudo UVa <http://www.uva.es>
	

Jose Manuel Blanco

Analista Sistemas · josemanuel.blanco at uva.es
<mailto:josemanuel.blanco at uva.es>
Tfno.: (+34) 983 18 6411

Universidad de Valladolid
Servicio de las TIC

Edificio Alfonso VIII · C/ Real de Burgos s/n
47011 · Valladolid · España (/Spain/)

| Facebook
<https://www.facebook.com/Universidad-de-Valladolid-187763507920209/> |
Twitter <https://twitter.com/uva_es> | UVa <http://www.uva.es>

Este mensaje puede contener información confidencial, sometida al
secreto profesional, cuya divulgación no está permitida por la ley. Si
usted no es su destinatario, por favor, notifíquelo al remitente y borre
este correo de su sistema. A los efectos de la protección de datos y el
RGPD, consulte: protección de datos en la UVa
<http://www.uva.es/protecciondedatos/>. El emisor no garantiza la
integridad, rapidez o seguridad del presente correo, ni se
responsabiliza de posibles perjuicios derivados de la captura,
incorporaciones de virus o cualesquiera otras manipulaciones efectuadas
por terceros. Piensa en verde: lee en la pantalla.

This message may contain confidential information covered by the
obligation of professional secrecy, the disclosure of which would be
contrary to the law. If you are not the intended recipient, please
advise the sender and delete this e-mail from your system. For the
purposes of data protection and GDPR, read: protección de datos en la
UVa <http://www.uva.es/protecciondedatos/>. The sender does not
guarantee the integrity, the accuracy, the swift delivery or the
security of this email transmission, and assumes no responsibility for
any possible damage incurred through data capture, virus incorporation
or any manipulation carried out by third parties. Think green: read on
the screen.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210904/01b47534/attachment.html>

More information about the Gluster-users mailing list