<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office"><head><!--[if gte mso 9]><xml><o:OfficeDocumentSettings><o:AllowPNG/><o:PixelsPerInch>96</o:PixelsPerInch></o:OfficeDocumentSettings></xml><![endif]--></head><body>
Hi,<div>Have you checked inode usage (df -i <span style="-webkit-text-size-adjust: auto;">/lvbackups/brick ) ?</span><span style="-webkit-text-size-adjust: auto;"></span></div><div><span style="-webkit-text-size-adjust: auto;"><br></span></div><div><span style="-webkit-text-size-adjust: auto;">Best Regards,</span></div><div><span style="-webkit-text-size-adjust: auto;">Strahil Nikolov</span></div><div><br><p class="yahoo-quoted-begin" style="font-size: 15px; color: #715FFA; padding-top: 15px; margin-top: 0">On Tuesday, May 2, 2023, 3:05 AM, brandon@thinkhuge.net wrote:</p><blockquote class="iosymail"><div dir="ltr">Hi Gluster users,<br></div><div dir="ltr"><br></div><div dir="ltr">We are seeing 'error=No space left on device' issue and hoping someone might<br></div><div dir="ltr">could advise?<br></div><div dir="ltr"><br></div><div dir="ltr">We are using a 12 node glusterfs v10.4 distributed vsftpd backup cluster for<br></div><div dir="ltr">years (not new) and recently 2 weeks ago upgraded to v9 > v10.4. I do not<br></div><div dir="ltr">know if the upgrade is related to this new issue.<br></div><div dir="ltr"><br></div><div dir="ltr">We are seeing a new issue 'error=No space left on device' error below on<br></div><div dir="ltr">multiple gluster v10.4 nodes in the logs. At the moment seeing it in the<br></div><div dir="ltr">logs for about half (5 out of 12) of the nodes. The issue will go away if<br></div><div dir="ltr">we reboot all the glusterfs nodes but, backups take a little over 2 days to<br></div><div dir="ltr">complete each weekend and the issue returns after about 1 day of backups<br></div><div dir="ltr">running and before the backup cycle is complete. It has been happening the<br></div><div dir="ltr">last 2 weekends we have run backups to these nodes.<br></div><div dir="ltr"><br></div><div dir="ltr">#example log msg from /var/log/glusterfs/home-volbackups.log<br></div><div dir="ltr">[2023-05-01 21:43:15.450502 +0000] W [MSGID: 114031]<br></div><div dir="ltr">[client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-volbackups-client-18:<br></div><div dir="ltr">remote operation failed. [{errno=28}, {error=No space left on device}]<br></div><div dir="ltr"><br></div><div dir="ltr">Each glusterfs node has a single brick and mounts a single distributed<br></div><div dir="ltr">volume as a glusterfs client locally and receives backup files to the volume<br></div><div dir="ltr">each weekend. <br></div><div dir="ltr"><br></div><div dir="ltr">We distribute the ftp upload load between the servers through a combination<br></div><div dir="ltr">of /etc/hosts entries and AWS weighted dns.<br></div><div dir="ltr"><br></div><div dir="ltr">We have 91 TB available on the volume though and each of the 12 nodes have<br></div><div dir="ltr">4-11 TB free so we are nowhere near out of space on any node?<br></div><div dir="ltr"><br></div><div dir="ltr">We have already tried the setting change from 'cluster.min-free-disk: 1%' to<br></div><div dir="ltr">'cluster.min-free-disk: 1GB' and rebooted all the gluster nodes to refresh<br></div><div dir="ltr">them and it happened again. That was mentioned in this doc<br></div><div dir="ltr"><a href="https://access.redhat.com/solutions/276483" target="_blank">https://access.redhat.com/solutions/276483</a> as an idea.<br></div><div dir="ltr"><br></div><div dir="ltr">Does anyone know what we might check next?<br></div><div dir="ltr"><br></div><div dir="ltr">glusterfs-server-10.4-1.el8s.x86_64<br></div><div dir="ltr">glusterfs-fuse-10.4-1.el8s.x86_64<br></div><div dir="ltr"><br></div><div dir="ltr">Here is the info (hostnames changed) below.<br></div><div dir="ltr"><br></div><div dir="ltr">[root@nybaknode1 ~]# gluster volume status volbackups detail<br></div><div dir="ltr">Status of volume: volbackups<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode9.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 60039<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1664<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquot<br></div><div dir="ltr">a<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 6.1TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108974976<br></div><div dir="ltr">Free Inodes : 3108881513<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode11.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 52682<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 2076<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquot<br></div><div dir="ltr">a<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 10.1TB<br></div><div dir="ltr">Total Disk Space : 43.5TB<br></div><div dir="ltr">Inode Count : 4672138432<br></div><div dir="ltr">Free Inodes : 4672039743<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode2.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 56722<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1761<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquot<br></div><div dir="ltr">a<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 6.6TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108921344<br></div><div dir="ltr">Free Inodes : 3108827241<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode3.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 53098<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1601<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquo<br></div><div dir="ltr">ta<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 6.4TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108921344<br></div><div dir="ltr">Free Inodes : 3108827312<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode4.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 51476<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1633<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquo<br></div><div dir="ltr">ta<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 6.9TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108921344<br></div><div dir="ltr">Free Inodes : 3108826837<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode12.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 50224<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1966<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquot<br></div><div dir="ltr">a<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 9.9TB<br></div><div dir="ltr">Total Disk Space : 43.5TB<br></div><div dir="ltr">Inode Count : 4671718976<br></div><div dir="ltr">Free Inodes : 4671620385<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode5.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 55270<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1666<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquo<br></div><div dir="ltr">ta<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 7.4TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108921344<br></div><div dir="ltr">Free Inodes : 3108827575<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode6.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 53106<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1688<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 7.6TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108921344<br></div><div dir="ltr">Free Inodes : 3108827668<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode7.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 56734<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1655<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquo<br></div><div dir="ltr">ta<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 4.0TB<br></div><div dir="ltr">Total Disk Space : 14.4TB<br></div><div dir="ltr">Inode Count : 1546333376<br></div><div dir="ltr">Free Inodes : 1546245572<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode8.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 60208<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1754<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=128,noquot<br></div><div dir="ltr">a<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 7.0TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108921344<br></div><div dir="ltr">Free Inodes : 3108827378<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode10.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 53237<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1757<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquot<br></div><div dir="ltr">a<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 10.5TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108921344<br></div><div dir="ltr">Free Inodes : 3108828289<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick : Brick nybaknode1.example.net:/lvbackups/brick<br></div><div dir="ltr">TCP Port : 54446<br></div><div dir="ltr">RDMA Port : 0<br></div><div dir="ltr">Online : Y<br></div><div dir="ltr">Pid : 1685<br></div><div dir="ltr">File System : xfs<br></div><div dir="ltr">Device : /dev/mapper/vgbackups-lvbackups<br></div><div dir="ltr">Mount Options :<br></div><div dir="ltr">rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=128,swidth=128,noquot<br></div><div dir="ltr">a<br></div><div dir="ltr">Inode Size : 512<br></div><div dir="ltr">Disk Space Free : 11.2TB<br></div><div dir="ltr">Total Disk Space : 29.0TB<br></div><div dir="ltr">Inode Count : 3108921344<br></div><div dir="ltr">Free Inodes : 3108828701<br></div><div dir="ltr"><br></div><div dir="ltr">[root@nybaknode1 ~]#<br></div><div dir="ltr"><br></div><div dir="ltr">[root@nybaknode1 ~]# gluster volume status volbackups<br></div><div dir="ltr">Status of volume: volbackups<br></div><div dir="ltr">Gluster process TCP Port RDMA Port Online Pid<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Brick nybaknode9.example.net:/lvbackups/b<br></div><div dir="ltr">rick 60039 0 Y<br></div><div dir="ltr">1664<br></div><div dir="ltr">Brick nybaknode11.example.net:/lvbackups/<br></div><div dir="ltr">brick 52682 0 Y<br></div><div dir="ltr">2076<br></div><div dir="ltr">Brick nybaknode2.example.net:/lvbackups/b<br></div><div dir="ltr">rick 56722 0 Y<br></div><div dir="ltr">1761<br></div><div dir="ltr">Brick nybaknode3.example.net:/lvbackups/b<br></div><div dir="ltr">rick 53098 0 Y<br></div><div dir="ltr">1601<br></div><div dir="ltr">Brick nybaknode4.example.net:/lvbackups/b<br></div><div dir="ltr">rick 51476 0 Y<br></div><div dir="ltr">1633<br></div><div dir="ltr">Brick nybaknode12.example.net:/lvbackups/<br></div><div dir="ltr">brick 50224 0 Y<br></div><div dir="ltr">1966<br></div><div dir="ltr">Brick nybaknode5.example.net:/lvbackups/b<br></div><div dir="ltr">rick 55270 0 Y<br></div><div dir="ltr">1666<br></div><div dir="ltr">Brick nybaknode6.example.net:/lvbackups/b<br></div><div dir="ltr">rick 53106 0 Y<br></div><div dir="ltr">1688<br></div><div dir="ltr">Brick nybaknode7.example.net:/lvbackups/b<br></div><div dir="ltr">rick 56734 0 Y<br></div><div dir="ltr">1655<br></div><div dir="ltr">Brick nybaknode8.example.net:/lvbackups/b<br></div><div dir="ltr">rick 60208 0 Y<br></div><div dir="ltr">1754<br></div><div dir="ltr">Brick nybaknode10.example.net:/lvbackups/<br></div><div dir="ltr">brick 53237 0 Y<br></div><div dir="ltr">1757<br></div><div dir="ltr">Brick nybaknode1.example.net:/lvbackups/b<br></div><div dir="ltr">rick 54446 0 Y<br></div><div dir="ltr">1685<br></div><div dir="ltr"><br></div><div dir="ltr">Task Status of Volume volbackups<br></div><div dir="ltr">----------------------------------------------------------------------------<br></div><div dir="ltr">--<br></div><div dir="ltr">Task : Rebalance<br></div><div dir="ltr">ID : df687907-fee4-4a46-9d23-2cfb38cb17cd<br></div><div dir="ltr">Status : completed<br></div><div dir="ltr"><br></div><div dir="ltr">[root@nybaknode1 ~]#<br></div><div dir="ltr"><br></div><div dir="ltr">[root@nybaknode1 ~]# gluster volume info volbackups<br></div><div dir="ltr"><br></div><div dir="ltr">Volume Name: volbackups<br></div><div dir="ltr">Type: Distribute<br></div><div dir="ltr">Volume ID: cd40794d-ab74-4706-a0bc-3e95bb8c63a2<br></div><div dir="ltr">Status: Started<br></div><div dir="ltr">Snapshot Count: 0<br></div><div dir="ltr">Number of Bricks: 12<br></div><div dir="ltr">Transport-type: tcp<br></div><div dir="ltr">Bricks:<br></div><div dir="ltr">Brick1: nybaknode9.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick2: nybaknode11.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick3: nybaknode2.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick4: nybaknode3.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick5: nybaknode4.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick6: nybaknode12.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick7: nybaknode5.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick8: nybaknode6.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick9: nybaknode7.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick10: nybaknode8.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick11: nybaknode10.example.net:/lvbackups/brick<br></div><div dir="ltr">Brick12: nybaknode1.example.net:/lvbackups/brick<br></div><div dir="ltr">Options Reconfigured:<br></div><div dir="ltr">cluster.min-free-disk: 1GB<br></div><div dir="ltr">nfs.disable: on<br></div><div dir="ltr">transport.address-family: inet<br></div><div dir="ltr">performance.cache-max-file-size: 2MB<br></div><div dir="ltr">diagnostics.brick-log-level: WARNING<br></div><div dir="ltr">diagnostics.brick-sys-log-level: WARNING<br></div><div dir="ltr">client.event-threads: 16<br></div><div dir="ltr">performance.client-io-threads: on<br></div><div dir="ltr">performance.io-thread-count: 32<br></div><div dir="ltr">server.event-threads: 16<br></div><div dir="ltr">performance.cache-size: 256MB<br></div><div dir="ltr">[root@nybaknode1 ~]#<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">________<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Community Meeting Calendar:<br></div><div dir="ltr"><br></div><div dir="ltr">Schedule -<br></div><div dir="ltr">Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br></div><div dir="ltr">Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br></div><div dir="ltr">Gluster-users mailing list<br></div><div dir="ltr"><a ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br></div><div dir="ltr"><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></div><blockquote></blockquote></blockquote></div>
</body></html>