[Gluster-users] Problem with massive file renaming in glusterfs volume

Mon Sep 6 07:40:06 UTC 2021

Hi Jose,

On Sat, Sep 4, 2021 at 1:58 PM José Manuel Blanco <josemanuel.blanco at uva.es>
wrote:

> Hi all,
>
> We have a problem with tasks (written in PHP) doing a lot of file
> renaming/moving (even several renames per second).
>
> The pattern is always the same: the task rename files with variable
> filename to the SAME final file (that is: the destination filename is
> ALWAYS the same).
>
> Problem: according to glusterfs and task logs, some rename are successful
> but other don't and we don't understand why or what is causing the error in
> the failing renames because the error is "File exists", but the rename
> supossedly must address these situations correctly and "overwrite" the
> destination file if it exists (it uses the PHP rename() function)
>
> More info:
>
> - We use a distributed-replicated volume in a 3 node cluster
>
> gluster volume info moodle-cv
>
> Volume Name: moodle-cv
> Type: Distributed-Replicate
> Volume ID: 1eef9714-3943-4d77-b42f-6b1144389c56
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 5 x (2 + 1) = 15
> Transport-type: tcp
> Bricks:
> Brick1: moodle2017-n2:/glusterfs/moodle-cv/brick1/datos
> Brick2: moodle2017-n3:/glusterfs/moodle-cv/brick1/replica
> Brick3: moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter (arbiter)
> Brick4: moodle2017-n3:/glusterfs/moodle-cv/brick2/datos
> Brick5: moodle2017-n2:/glusterfs/moodle-cv/brick2/replica
> Brick6: moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter (arbiter)
> Brick7: moodle2017-n2:/glusterfs/moodle-cv/brick3/datos
> Brick8: moodle2017-n3:/glusterfs/moodle-cv/brick3/replica
> Brick9: moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter (arbiter)
> Brick10: moodle2017-n3:/glusterfs/moodle-cv/brick4/datos
> Brick11: moodle2017-n2:/glusterfs/moodle-cv/brick4/replica
> Brick12: moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter (arbiter)
> Brick13: moodle2017-n2:/glusterfs/moodle-cv/brick5/datos
> Brick14: moodle2017-n3:/glusterfs/moodle-cv/brick5/replica
> Brick15: moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter (arbiter)
> Options Reconfigured:
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> performance.open-behind: off
> performance.lazy-open: no
> cluster.self-heal-daemon: enable
>
> - The nodes use Oracle Linux 7.9 (RedHat clone) and GlusterFS 8.5
>
> - The tasks use the PHP rename() function
>
> - The tasks renaming files are executed IN ONE NODE OF THE CLUSTER that
> mounts the volume using the FUSE client
>
> - At the end of the message I paste a (very) little fragment of the volume
> log
>
>
> Any ideas of the possible cause of the problem and/or suggestions to avoid
> it?
>

Most probably the issue is caused by special internal files needed by
Gluster that are used to reference the correct location of a file in a
distributed volume (they are referred as linkto files). I would say the
error happens because those files already exist when it's expected that
they don't exist (I think it's a similar problem as
https://github.com/gluster/glusterfs/issues/1723).

Are you using FUSE mounts ? are you doing the renames from more than one
client ?

Regards,

Xavi

Regards.
>
>
> This is the fragment of the log:
>
> * I've "separated" the lines by "rename operation" but all the lines
> appear together in the log
>
> --->SUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.478321] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (11fdf019-16f8-48ae-bd1f-05b670b29ec1)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2)
>
> --->UNSUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.497845] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
> (350698df-35d9-4489-90cc-faa004bfc861)
> (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> [2021-09-03 09:04:02.503271] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-0: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.503339] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-1: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.503393] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-2: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.504997] W [fuse-bridge.c:2500:fuse_rename_cbk]
> 0-glusterfs-fuse: 781438412:
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
> ->
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> => -1 (File exists)
>
> --->UNSUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.729314] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
> (5bff6629-80b8-4d9c-8e37-abfa7691124e)
> (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> [2021-09-03 09:04:02.735631] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.735709] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.735747] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.737356] W [fuse-bridge.c:2500:fuse_rename_cbk]
> 0-glusterfs-fuse: 781438861:
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
> ->
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> => -1 (File exists)
>
> --->SUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.799776] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp
> (82725bab-b888-4d24-b239-31b86f50425f)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> --->UNSUCCESFULL RENAME:
>
> [2021-09-03 09:04:02.963545] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
> (b3c5435c-f79b-402f-ba00-0e78dcdb6cd2)
> (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (82725bab-b888-4d24-b239-31b86f50425f)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> [2021-09-03 09:04:02.968665] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.968801] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.968847] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3: remote
> operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.970438] W [fuse-bridge.c:2500:fuse_rename_cbk]
> 0-glusterfs-fuse: 781439391:
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
> ->
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> => -1 (File exists)
>
>
> --
>
> <http://www.uva.es>
> <http://www.uva.es> [image: Escudo UVa] <http://www.uva.es>
>
> Jose Manuel Blanco
>
> Analista Sistemas · josemanuel.blanco at uva.es
> Tfno.: (+34) 983 18 6411
>
> Universidad de Valladolid
> Servicio de las TIC
>
> Edificio Alfonso VIII · C/ Real de Burgos s/n
> 47011 · Valladolid · España (*Spain*)
>
> | Facebook
> <https://www.facebook.com/Universidad-de-Valladolid-187763507920209/> |
> Twitter <https://twitter.com/uva_es> | UVa <http://www.uva.es>
>
> Este mensaje puede contener información confidencial, sometida al secreto
> profesional, cuya divulgación no está permitida por la ley. Si usted no es
> su destinatario, por favor, notifíquelo al remitente y borre este correo de
> su sistema. A los efectos de la protección de datos y el RGPD, consulte: protección
> de datos en la UVa <http://www.uva.es/protecciondedatos/>. El emisor no
> garantiza la integridad, rapidez o seguridad del presente correo, ni se
> responsabiliza de posibles perjuicios derivados de la captura,
> incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por
> terceros. Piensa en verde: lee en la pantalla.
>
> This message may contain confidential information covered by the
> obligation of professional secrecy, the disclosure of which would be
> contrary to the law. If you are not the intended recipient, please advise
> the sender and delete this e-mail from your system. For the purposes of
> data protection and GDPR, read: protección de datos en la UVa
> <http://www.uva.es/protecciondedatos/>. The sender does not guarantee the
> integrity, the accuracy, the swift delivery or the security of this email
> transmission, and assumes no responsibility for any possible damage
> incurred through data capture, virus incorporation or any manipulation
> carried out by third parties. Think green: read on the screen.
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210906/67093b3c/attachment.html>