[Gluster-users] Problem with massive file renaming in glusterfs volume
José Manuel Blanco
josemanuel.blanco at uva.es
Mon Sep 13 12:50:37 UTC 2021
Hello Xavier,
Sorry: these days I've been busy with other tasks :-(
More info about the problem and your questions:
- The rename/move errors appear not only in the glusterfs node that
execute tasks renaming files BUT ALSO in ALL clients of the glusterfs volume
- All clients (and even the glusterfs node executing the failing
renames) mount the glusterfs volume using the fuse client
Regards.
<http://www.uva.es>
<http://www.uva.es>Escudo UVa <http://www.uva.es>
Jose Manuel Blanco
Analista Sistemas · josemanuel.blanco at uva.es
<mailto:josemanuel.blanco at uva.es>
Tfno.: (+34) 983 18 6411
Universidad de Valladolid
Servicio de las TIC
Edificio Alfonso VIII · C/ Real de Burgos s/n
47011 · Valladolid · España (/Spain/)
| Facebook
<https://www.facebook.com/Universidad-de-Valladolid-187763507920209/> |
Twitter <https://twitter.com/uva_es> | UVa <http://www.uva.es>
Este mensaje puede contener información confidencial, sometida al
secreto profesional, cuya divulgación no está permitida por la ley. Si
usted no es su destinatario, por favor, notifíquelo al remitente y borre
este correo de su sistema. A los efectos de la protección de datos y el
RGPD, consulte: protección de datos en la UVa
<http://www.uva.es/protecciondedatos/>. El emisor no garantiza la
integridad, rapidez o seguridad del presente correo, ni se
responsabiliza de posibles perjuicios derivados de la captura,
incorporaciones de virus o cualesquiera otras manipulaciones efectuadas
por terceros. Piensa en verde: lee en la pantalla.
This message may contain confidential information covered by the
obligation of professional secrecy, the disclosure of which would be
contrary to the law. If you are not the intended recipient, please
advise the sender and delete this e-mail from your system. For the
purposes of data protection and GDPR, read: protección de datos en la
UVa <http://www.uva.es/protecciondedatos/>. The sender does not
guarantee the integrity, the accuracy, the swift delivery or the
security of this email transmission, and assumes no responsibility for
any possible damage incurred through data capture, virus incorporation
or any manipulation carried out by third parties. Think green: read on
the screen.
El 6/9/21 a las 9:40, Xavi Hernandez escribió:
> Hi Jose,
>
> On Sat, Sep 4, 2021 at 1:58 PM José Manuel Blanco
> <josemanuel.blanco at uva.es <mailto:josemanuel.blanco at uva.es>> wrote:
>
> Hi all,
>
> We have a problem with tasks (written in PHP) doing a lot of file
> renaming/moving (even several renames per second).
>
> The pattern is always the same: the task rename files with
> variable filename to the SAME final file (that is: the destination
> filename is ALWAYS the same).
>
> Problem: according to glusterfs and task logs, some rename are
> successful but other don't and we don't understand why or what is
> causing the error in the failing renames because the error is
> "File exists", but the rename supossedly must address these
> situations correctly and "overwrite" the destination file if it
> exists (it uses the PHP rename() function)
>
> More info:
>
> - We use a distributed-replicated volume in a 3 node cluster
>
> gluster volume info moodle-cv
>
> Volume Name: moodle-cv
> Type: Distributed-Replicate
> Volume ID: 1eef9714-3943-4d77-b42f-6b1144389c56
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 5 x (2 + 1) = 15
> Transport-type: tcp
> Bricks:
> Brick1: moodle2017-n2:/glusterfs/moodle-cv/brick1/datos
> Brick2: moodle2017-n3:/glusterfs/moodle-cv/brick1/replica
> Brick3: moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter
> (arbiter)
> Brick4: moodle2017-n3:/glusterfs/moodle-cv/brick2/datos
> Brick5: moodle2017-n2:/glusterfs/moodle-cv/brick2/replica
> Brick6: moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter
> (arbiter)
> Brick7: moodle2017-n2:/glusterfs/moodle-cv/brick3/datos
> Brick8: moodle2017-n3:/glusterfs/moodle-cv/brick3/replica
> Brick9: moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter
> (arbiter)
> Brick10: moodle2017-n3:/glusterfs/moodle-cv/brick4/datos
> Brick11: moodle2017-n2:/glusterfs/moodle-cv/brick4/replica
> Brick12: moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter
> (arbiter)
> Brick13: moodle2017-n2:/glusterfs/moodle-cv/brick5/datos
> Brick14: moodle2017-n3:/glusterfs/moodle-cv/brick5/replica
> Brick15: moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter
> (arbiter)
> Options Reconfigured:
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> performance.open-behind: off
> performance.lazy-open: no
> cluster.self-heal-daemon: enable
>
> - The nodes use Oracle Linux 7.9 (RedHat clone) and GlusterFS 8.5
>
> - The tasks use the PHP rename() function
>
> - The tasks renaming files are executed IN ONE NODE OF THE CLUSTER
> that mounts the volume using the FUSE client
>
> - At the end of the message I paste a (very) little fragment of
> the volume log
>
>
> Any ideas of the possible cause of the problem and/or suggestions
> to avoid it?
>
>
> Most probably the issue is caused by special internal files needed by
> Gluster that are used to reference the correct location of a file in a
> distributed volume (they are referred as linkto files). I would say
> the error happens because those files already exist when it's expected
> that they don't exist (I think it's a similar problem
> as https://github.com/gluster/glusterfs/issues/1723
> <https://github.com/gluster/glusterfs/issues/1723>).
>
> Are you using FUSE mounts ? are you doing the renames from more than
> one client ?
>
> Regards,
>
> Xavi
>
> Regards.
>
>
> This is the fragment of the log:
>
> * I've "separated" the lines by "rename operation" but all the
> lines appear together in the log
>
> --->SUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.478321] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (11fdf019-16f8-48ae-bd1f-05b670b29ec1)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2)
>
> --->UNSUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.497845] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
> (350698df-35d9-4489-90cc-faa004bfc861)
> (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> [2021-09-03 09:04:02.503271] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-0: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.503339] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-1: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.503393] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-2: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.504997] W
> [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
> 781438412:
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
> ->
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> => -1 (File exists)
>
> --->UNSUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.729314] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
> (5bff6629-80b8-4d9c-8e37-abfa7691124e)
> (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> [2021-09-03 09:04:02.735631] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-5: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.735709] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-4: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.735747] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-3: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.737356] W
> [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
> 781438861:
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
> ->
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> => -1 (File exists)
>
> --->SUCCESSFUL RENAME:
>
> [2021-09-03 09:04:02.799776] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp
> (82725bab-b888-4d24-b239-31b86f50425f)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (4a875eac-5389-400c-9359-458e151054f7)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> --->UNSUCCESFULL RENAME:
>
> [2021-09-03 09:04:02.963545] I [MSGID: 109066]
> [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
> (b3c5435c-f79b-402f-ba00-0e78dcdb6cd2)
> (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> (82725bab-b888-4d24-b239-31b86f50425f)
> (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
>
> [2021-09-03 09:04:02.968665] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-5: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.968801] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-4: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.968847] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
> 2-moodle-cv-client-3: remote operation failed.
> [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
> {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
> {errno=17}, {error=File exists}]
>
> [2021-09-03 09:04:02.970438] W
> [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
> 781439391:
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
> ->
> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
> => -1 (File exists)
>
>
> --
>
> <http://www.uva.es>
> <http://www.uva.es>Escudo UVa <http://www.uva.es>
>
>
> Jose Manuel Blanco
>
> Analista Sistemas · josemanuel.blanco at uva.es
> <mailto:josemanuel.blanco at uva.es>
> Tfno.: (+34) 983 18 6411
>
> Universidad de Valladolid
> Servicio de las TIC
>
> Edificio Alfonso VIII · C/ Real de Burgos s/n
> 47011 · Valladolid · España (/Spain/)
>
> | Facebook
> <https://www.facebook.com/Universidad-de-Valladolid-187763507920209/>
> | Twitter <https://twitter.com/uva_es> | UVa <http://www.uva.es>
>
> Este mensaje puede contener información confidencial, sometida al
> secreto profesional, cuya divulgación no está permitida por la
> ley. Si usted no es su destinatario, por favor, notifíquelo al
> remitente y borre este correo de su sistema. A los efectos de la
> protección de datos y el RGPD, consulte: protección de datos en la
> UVa <http://www.uva.es/protecciondedatos/>. El emisor no garantiza
> la integridad, rapidez o seguridad del presente correo, ni se
> responsabiliza de posibles perjuicios derivados de la captura,
> incorporaciones de virus o cualesquiera otras manipulaciones
> efectuadas por terceros. Piensa en verde: lee en la pantalla.
>
> This message may contain confidential information covered by the
> obligation of professional secrecy, the disclosure of which would
> be contrary to the law. If you are not the intended recipient,
> please advise the sender and delete this e-mail from your system.
> For the purposes of data protection and GDPR, read: protección de
> datos en la UVa <http://www.uva.es/protecciondedatos/>. The sender
> does not guarantee the integrity, the accuracy, the swift delivery
> or the security of this email transmission, and assumes no
> responsibility for any possible damage incurred through data
> capture, virus incorporation or any manipulation carried out by
> third parties. Think green: read on the screen.
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> <https://meet.google.com/cpu-eiue-hvk>
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> https://lists.gluster.org/mailman/listinfo/gluster-users
> <https://lists.gluster.org/mailman/listinfo/gluster-users>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210913/d5743766/attachment.html>
More information about the Gluster-users
mailing list