From cuculovic at mdpi.com Mon Jul 1 11:52:15 2019 From: cuculovic at mdpi.com (Milos Cuculovic) Date: Mon, 1 Jul 2019 13:52:15 +0200 Subject: [Gluster-users] ls slow on a GlusterFS mounted filesystem Message-ID: <46CEA21B-7499-420B-9656-B47630F805A0@mdpi.com> We have two replicated GlusterFS servers. When trying to run ls on any of the GlusterFS mounted directories, it takes long time to process. For example, on a irectory with 10 files and 10 subdirectories, it takes 5sec to do the ls. Here is the volume config info: Volume Name: storage2 Type: Replicate Volume ID: 8af504b7-8a95-463b-8578-9a969ae060ff Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: storage3:/data/data-cluster Brick2: storage4:/data/data-cluster Options Reconfigured: storage.build-pgfid: on cluster.readdir-optimize: off storage.fips-mode-rchecksum: on client.event-threads: 12 server.event-threads: 12 nfs.disable: on transport.address-family: inet auth.allow: 10.10.0.* features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.stat-prefetch: on performance.readdir-ahead: on performance.cache-refresh-timeout: 1 network.compression.compression-level: -1 network.compression: off cluster.min-free-disk: 2% performance.cache-size: 1GB features.bitrot: on features.scrub: Active performance.client-io-threads: on features.scrub-freq: monthly features.scrub-throttle: normal Any idea on how to improve this? Looking forward to hearing from you. Milos From nbalacha at redhat.com Tue Jul 2 06:55:02 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Tue, 2 Jul 2019 12:25:02 +0530 Subject: [Gluster-users] Removing subvolume from dist/rep volume In-Reply-To: <20190628142454.GX19805@sherohman.org> References: <20190625091330.GU19805@sherohman.org> <20190628142454.GX19805@sherohman.org> Message-ID: Hi Dave, Yes, files in split brain are not migrated as we cannot figure out which is the good copy. Adding Ravi to look at this and see what can be done. Also adding Krutika as this is a sharded volume. The files with the "---------T" permissions are internal files and can be ignored. Ravi and Krutika, please take a look at the other files. Regards, Nithya On Fri, 28 Jun 2019 at 19:56, Dave Sherohman wrote: > On Thu, Jun 27, 2019 at 12:17:10PM +0530, Nithya Balachandran wrote: > > There are some edge cases that may prevent a file from being migrated > > during a remove-brick. Please do the following after this: > > > > 1. Check the remove-brick status for any failures. If there are any, > > check the rebalance log file for errors. > > 2. Even if there are no failures, check the removed bricks to see if > any > > files have not been migrated. If there are any, please check that > they are > > valid files on the brick and copy them to the volume from the brick > to the > > mount point. > > Well, looks like I hit one of those edge cases. Probably because of > some issues around a reboot last September which left a handful of files > in a state where self-heal identified them as needing to be healed, but > incapable of actually healing them. (Check the list archives for > "Kicking a stuck heal", posted on Sept 4, if you want more details.) > > So I'm getting 9 failures on the arbiter (merlin), 8 on one data brick > (gandalf), and 3 on the other (saruman). Looking in > /var/log/gluster/palantir-rebalance.log, I see those numbers of > > migrate file failed: /.shard/291e9749-2d1b-47af-ad53-3a09ad4e64c6.229: > failed to lock file on palantir-replicate-1 [Stale file handle] > > errors. > > Also, merlin has four errors, and gandalf has one, of the form: > > Gfid mismatch detected for > /0f500288-ff62-4f0b-9574-53f510b4159f.2898>, > 9f00c0fe-58c3-457e-a2e6-f6a006d1cfc6 on palantir-client-7 and > 08bb7cdc-172b-4c21-916a-2a244c095a3e on palantir-client-1. > > There are no gfid mismatches recorded on saruman. All of the gfid > mismatches are for and (on > saruman) appear to correspond to 0-byte files (e.g., > .shard/0f500288-ff62-4f0b-9574-53f510b4159f.2898, in the case of the > gfid mismatch quoted above). > > For both types of errors, all affected files are in .shard/ and have > UUID-style names, so I have no idea which actual files they belong to. > File sizes are generally either 0 bytes or 4M (exactly), although one of > them has a size slightly larger than 3M. So I'm assuming they're chunks > of larger files (which would be almost all the files on the volume - > it's primarily holding disk image files for kvm servers). > > Web searches generally seem to consider gfid mismatches to be a form of > split-brain, but `gluster volume heal palantir info split-brain` shows > "Number of entries in split-brain: 0" for all bricks, including those > bricks which are reporting gfid mismatches. > > > Given all that, how do I proceed with cleaning up the stale handle > issues? I would guess that this will involve somehow converting the > shard filename to a "real" filename, then shutting down the > corresponding VM and maybe doing some additional cleanup. > > And then there's the gfid mismatches. Since they're for 0-byte files, > is it safe to just ignore them on the assumption that they only hold > metadata? Or do I need to do some kind of split-brain resolution on > them (even though gluster says no files are in split-brain)? > > > Finally, a listing of /var/local/brick0/data/.shard on saruman, in case > any of the information it contains (like file sizes/permissions) might > provide clues to resolving the errors: > > --- cut here --- > root at saruman:/var/local/brick0/data/.shard# ls -l > total 63996 > -rw-rw---- 2 root libvirt-qemu 0 Sep 17 2018 > 0f500288-ff62-4f0b-9574-53f510b4159f.2864 > -rw-rw---- 2 root libvirt-qemu 0 Sep 17 2018 > 0f500288-ff62-4f0b-9574-53f510b4159f.2868 > -rw-rw---- 2 root libvirt-qemu 0 Sep 17 2018 > 0f500288-ff62-4f0b-9574-53f510b4159f.2879 > -rw-rw---- 2 root libvirt-qemu 0 Sep 17 2018 > 0f500288-ff62-4f0b-9574-53f510b4159f.2898 > -rw------- 2 root libvirt-qemu 4194304 May 17 14:42 > 291e9749-2d1b-47af-ad53-3a09ad4e64c6.229 > -rw------- 2 root libvirt-qemu 4194304 Jun 24 09:10 > 291e9749-2d1b-47af-ad53-3a09ad4e64c6.925 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 26 12:54 > 2df12cb0-6cf4-44ae-8b0a-4a554791187e.266 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 26 16:30 > 2df12cb0-6cf4-44ae-8b0a-4a554791187e.820 > -rw-r--r-- 2 root libvirt-qemu 4194304 Jun 17 20:22 > 323186b1-6296-4cbe-8275-b940cc9d65cf.27466 > -rw-r--r-- 2 root libvirt-qemu 4194304 Jun 27 05:01 > 323186b1-6296-4cbe-8275-b940cc9d65cf.32575 > -rw-r--r-- 2 root libvirt-qemu 3145728 Jun 11 13:23 > 323186b1-6296-4cbe-8275-b940cc9d65cf.3448 > ---------T 2 root libvirt-qemu 0 Jun 28 14:26 > 4cd094f4-0344-4660-98b0-83249d5bd659.22998 > -rw------- 2 root libvirt-qemu 4194304 Mar 13 2018 > 6cdd2e5c-f49e-492b-8039-239e71577836.1302 > ---------T 2 root libvirt-qemu 0 Jun 28 13:22 > 7530a2d1-d6ec-4a04-95a2-da1f337ac1ad.47131 > ---------T 2 root libvirt-qemu 0 Jun 28 13:22 > 7530a2d1-d6ec-4a04-95a2-da1f337ac1ad.52615 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 27 08:56 > 8fefae99-ed2a-4a8f-ab87-aa94c6bb6e68.100 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 27 11:29 > 8fefae99-ed2a-4a8f-ab87-aa94c6bb6e68.106 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 28 02:35 > 8fefae99-ed2a-4a8f-ab87-aa94c6bb6e68.137 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Nov 4 2018 > 9544617c-901c-4613-a94b-ccfad4e38af1.165 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Nov 4 2018 > 9544617c-901c-4613-a94b-ccfad4e38af1.168 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Nov 5 2018 > 9544617c-901c-4613-a94b-ccfad4e38af1.193 > -rw-rw-r-- 2 root libvirt-qemu 4194304 Nov 6 2018 > 9544617c-901c-4613-a94b-ccfad4e38af1.3800 > ---------T 2 root libvirt-qemu 0 Jun 28 15:02 > b48a5934-5e5b-4918-8193-6ff36f685f70.46559 > -rw-rw---- 2 root libvirt-qemu 0 Oct 12 2018 > c5bde2f2-3361-4d1a-9c88-28751ef74ce6.3568 > -rw-r--r-- 2 root libvirt-qemu 4194304 Apr 13 2018 > c953c676-152d-4826-80ff-bd307fa7f6e5.10724 > -rw-r--r-- 2 root libvirt-qemu 4194304 Apr 11 2018 > c953c676-152d-4826-80ff-bd307fa7f6e5.3101 > --- cut here --- > > -- > Dave Sherohman > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From khiremat at redhat.com Tue Jul 2 07:07:58 2019 From: khiremat at redhat.com (Kotresh Hiremath Ravishankar) Date: Tue, 2 Jul 2019 12:37:58 +0530 Subject: [Gluster-users] Exception in Geo-Replication In-Reply-To: References: Message-ID: You should be looking into the other log file (changes-.log) for actual failure. In your case "changes-home-sas-gluster-data-code-misc.log" On Tue, Jul 2, 2019 at 12:33 PM deepu srinivasan wrote: > Any Update on this issue ? > > On Mon, Jul 1, 2019 at 4:19 PM deepu srinivasan > wrote: > >> Hi >> I am getting this exception while starting geo-replication. Please help. >> >> [2019-07-01 10:48:02.445475] E [repce(agent >> /home/sas/gluster/data/code-misc):122:worker] : call failed: >> >> Traceback (most recent call last): >> >> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, >> in worker >> >> res = getattr(self.obj, rmeth)(*in_data[2:]) >> >> File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", >> line 41, in register >> >> return Changes.cl_register(cl_brick, cl_dir, cl_log, cl_level, >> retries) >> >> File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", >> line 45, in cl_register >> >> cls.raise_changelog_err() >> >> File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", >> line 29, in raise_changelog_err >> >> raise ChangelogException(errn, os.strerror(errn)) >> >> ChangelogException: [Errno 21] Is a directory >> >> [2019-07-01 10:48:02.446341] E [repce(worker >> /home/sas/gluster/data/code-misc):214:__call__] RepceClient: call failed >> call=31023:140523296659264:1561978082.44 method=register >> error=ChangelogException >> >> [2019-07-01 10:48:02.446654] E [resource(worker >> /home/sas/gluster/data/code-misc):1268:service_loop] GLUSTER: Changelog >> register failed error=[Errno 21] Is a directory >> > -- Thanks and Regards, Kotresh H R -------------- next part -------------- An HTML attachment was scrubbed... URL: From cuculovic at mdpi.com Wed Jul 3 08:24:52 2019 From: cuculovic at mdpi.com (Milos Cuculovic) Date: Wed, 3 Jul 2019 10:24:52 +0200 Subject: [Gluster-users] ls slow on a GlusterFS mounted filesystem In-Reply-To: <46CEA21B-7499-420B-9656-B47630F805A0@mdpi.com> References: <46CEA21B-7499-420B-9656-B47630F805A0@mdpi.com> Message-ID: <7BC5FD15-4659-4EC7-A5A0-B85951578B65@mdpi.com> Any idea? - Kindest regards, Milos Cuculovic IT Manager --- MDPI AG Postfach, CH-4020 Basel, Switzerland Office: St. Alban-Anlage 66, 4052 Basel, Switzerland Tel. +41 61 683 77 35 Fax +41 61 302 89 18 Email: cuculovic at mdpi.com Skype: milos.cuculovic.mdpi Disclaimer: The information and files contained in this message are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error, please notify me and delete this message from your system. You may not copy this message in its entirety or in part, or disclose its contents to anyone. > On 1 Jul 2019, at 13:52, Milos Cuculovic wrote: > > We have two replicated GlusterFS servers. When trying to run ls on any of the GlusterFS mounted directories, it takes long time to process. > For example, on a irectory with 10 files and 10 subdirectories, it takes 5sec to do the ls. > > Here is the volume config info: > > Volume Name: storage2 > Type: Replicate > Volume ID: 8af504b7-8a95-463b-8578-9a969ae060ff > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: storage3:/data/data-cluster > Brick2: storage4:/data/data-cluster > Options Reconfigured: > storage.build-pgfid: on > cluster.readdir-optimize: off > storage.fips-mode-rchecksum: on > client.event-threads: 12 > server.event-threads: 12 > nfs.disable: on > transport.address-family: inet > auth.allow: 10.10.0.* > features.cache-invalidation: on > features.cache-invalidation-timeout: 600 > performance.stat-prefetch: on > performance.readdir-ahead: on > performance.cache-refresh-timeout: 1 > network.compression.compression-level: -1 > network.compression: off > cluster.min-free-disk: 2% > performance.cache-size: 1GB > features.bitrot: on > features.scrub: Active > performance.client-io-threads: on > features.scrub-freq: monthly > features.scrub-throttle: normal > > Any idea on how to improve this? > > Looking forward to hearing from you. > > Milos -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.melnik at tucha.ua Wed Jul 3 08:39:41 2019 From: v.melnik at tucha.ua (Vladimir Melnik) Date: Wed, 3 Jul 2019 11:39:41 +0300 Subject: [Gluster-users] Extremely low performance - am I doing something wrong? Message-ID: <20190703083941.GL1375@shagomer.uplink.tucha13.net> Dear colleagues, I have a lab with a bunch of virtual machines (the virtualization is provided by KVM) running on the same physical host. 4 of these VMs are working as a GlusterFS cluster and there's one more VM that works as a client. I'll specify all the packages' versions in the ending of this message. I created 2 volumes - one is having type "Distributed-Replicate" and another one is "Distribute". The problem is that both of volumes are showing really poor performance. Here's what I see on the client: $ mount | grep gluster 10.13.1.16:storage1 on /mnt/glusterfs1 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) 10.13.1.16:storage2 on /mnt/glusterfs2 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s The distributed one shows a bit better performance than the distributed-replicated one, but it's still poor. :-( The disk storage itself is OK, here's what I see on each of 4 GlusterFS servers: for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s The network between all 5 VMs is OK, they all are working on the same physical host. Can't understand, what am I doing wrong. :-( Here's the detailed info about the volumes: Volume Name: storage1 Type: Distributed-Replicate Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32 Status: Started Snapshot Count: 0 Number of Bricks: 2 x (2 + 1) = 6 Transport-type: tcp Bricks: Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1 Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2 Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3 Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4 Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off Volume Name: storage2 Type: Distribute Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067 Status: Started Snapshot Count: 0 Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2 Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2 Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2 Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2 Options Reconfigured: transport.address-family: inet nfs.disable: on The OS is CentOS Linux release 7.6.1810. The packages I'm using are: glusterfs-6.3-1.el7.x86_64 glusterfs-api-6.3-1.el7.x86_64 glusterfs-cli-6.3-1.el7.x86_64 glusterfs-client-xlators-6.3-1.el7.x86_64 glusterfs-fuse-6.3-1.el7.x86_64 glusterfs-libs-6.3-1.el7.x86_64 glusterfs-server-6.3-1.el7.x86_64 kernel-3.10.0-327.el7.x86_64 kernel-3.10.0-514.2.2.el7.x86_64 kernel-3.10.0-957.12.1.el7.x86_64 kernel-3.10.0-957.12.2.el7.x86_64 kernel-3.10.0-957.21.3.el7.x86_64 kernel-tools-3.10.0-957.21.3.el7.x86_64 kernel-tools-libs-3.10.0-957.21.3.el7.x86_6 Please, be so kind as to help me to understand, did I do it wrong or that's quite normal performance of GlusterFS? Thanks in advance! From v.melnik at tucha.ua Wed Jul 3 10:20:00 2019 From: v.melnik at tucha.ua (Vladimir Melnik) Date: Wed, 3 Jul 2019 13:20:00 +0300 Subject: [Gluster-users] Extremely low performance - am I doing something wrong? In-Reply-To: <20190703083941.GL1375@shagomer.uplink.tucha13.net> References: <20190703083941.GL1375@shagomer.uplink.tucha13.net> Message-ID: <20190703102000.GM1375@shagomer.uplink.tucha13.net> Just to be exact - here are the results of iperf3 measurements between the client and one of servers: $ iperf3 -c gluster1 Connecting to host gluster1, port 5201 [ 4] local 10.13.16.1 port 33156 connected to 10.13.1.16 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 77.8 MBytes 652 Mbits/sec 3 337 KBytes [ 4] 1.00-2.00 sec 89.7 MBytes 752 Mbits/sec 0 505 KBytes [ 4] 2.00-3.00 sec 103 MBytes 862 Mbits/sec 2 631 KBytes [ 4] 3.00-4.00 sec 104 MBytes 870 Mbits/sec 1 741 KBytes [ 4] 4.00-5.00 sec 98.8 MBytes 828 Mbits/sec 1 834 KBytes [ 4] 5.00-6.00 sec 101 MBytes 849 Mbits/sec 0 923 KBytes [ 4] 6.00-7.00 sec 102 MBytes 860 Mbits/sec 0 1005 KBytes [ 4] 7.00-8.00 sec 106 MBytes 890 Mbits/sec 0 1.06 MBytes [ 4] 8.00-9.00 sec 109 MBytes 913 Mbits/sec 0 1.13 MBytes [ 4] 9.00-10.00 sec 109 MBytes 912 Mbits/sec 0 1.20 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1000 MBytes 839 Mbits/sec 7 sender [ 4] 0.00-10.00 sec 998 MBytes 837 Mbits/sec receiver iperf Done. $ iperf3 -c gluster1 -R Connecting to host gluster1, port 5201 Reverse mode, remote host gluster1 is sending [ 4] local 10.13.16.1 port 33160 connected to 10.13.1.16 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 58.8 MBytes 492 Mbits/sec [ 4] 1.00-2.00 sec 80.1 MBytes 673 Mbits/sec [ 4] 2.00-3.00 sec 83.8 MBytes 703 Mbits/sec [ 4] 3.00-4.00 sec 95.6 MBytes 800 Mbits/sec [ 4] 4.00-5.00 sec 102 MBytes 858 Mbits/sec [ 4] 5.00-6.00 sec 101 MBytes 850 Mbits/sec [ 4] 6.00-7.00 sec 102 MBytes 860 Mbits/sec [ 4] 7.00-8.00 sec 107 MBytes 898 Mbits/sec [ 4] 8.00-9.00 sec 106 MBytes 893 Mbits/sec [ 4] 9.00-10.00 sec 108 MBytes 904 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 949 MBytes 796 Mbits/sec 6 sender [ 4] 0.00-10.00 sec 946 MBytes 794 Mbits/sec receiver iperf Done. So, the bandwidth seems to be OK too. On Wed, Jul 03, 2019 at 11:39:41AM +0300, Vladimir Melnik wrote: > Dear colleagues, > > I have a lab with a bunch of virtual machines (the virtualization is > provided by KVM) running on the same physical host. 4 of these VMs are > working as a GlusterFS cluster and there's one more VM that works as a > client. I'll specify all the packages' versions in the ending of this > message. > > I created 2 volumes - one is having type "Distributed-Replicate" and > another one is "Distribute". The problem is that both of volumes are > showing really poor performance. > > Here's what I see on the client: > $ mount | grep gluster > 10.13.1.16:storage1 on /mnt/glusterfs1 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > 10.13.1.16:storage2 on /mnt/glusterfs2 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s > > The distributed one shows a bit better performance than the > distributed-replicated one, but it's still poor. :-( > > The disk storage itself is OK, here's what I see on each of 4 GlusterFS > servers: > for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s > > The network between all 5 VMs is OK, they all are working on the same > physical host. > > Can't understand, what am I doing wrong. :-( > > Here's the detailed info about the volumes: > Volume Name: storage1 > Type: Distributed-Replicate > Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32 > Status: Started > Snapshot Count: 0 > Number of Bricks: 2 x (2 + 1) = 6 > Transport-type: tcp > Bricks: > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1 > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2 > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3 > Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4 > Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > > Volume Name: storage2 > Type: Distribute > Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067 > Status: Started > Snapshot Count: 0 > Number of Bricks: 4 > Transport-type: tcp > Bricks: > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2 > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2 > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2 > Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2 > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > > The OS is CentOS Linux release 7.6.1810. The packages I'm using are: > glusterfs-6.3-1.el7.x86_64 > glusterfs-api-6.3-1.el7.x86_64 > glusterfs-cli-6.3-1.el7.x86_64 > glusterfs-client-xlators-6.3-1.el7.x86_64 > glusterfs-fuse-6.3-1.el7.x86_64 > glusterfs-libs-6.3-1.el7.x86_64 > glusterfs-server-6.3-1.el7.x86_64 > kernel-3.10.0-327.el7.x86_64 > kernel-3.10.0-514.2.2.el7.x86_64 > kernel-3.10.0-957.12.1.el7.x86_64 > kernel-3.10.0-957.12.2.el7.x86_64 > kernel-3.10.0-957.21.3.el7.x86_64 > kernel-tools-3.10.0-957.21.3.el7.x86_64 > kernel-tools-libs-3.10.0-957.21.3.el7.x86_6 > > Please, be so kind as to help me to understand, did I do it wrong or > that's quite normal performance of GlusterFS? > > Thanks in advance! > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- V.Melnik From hunter86_bg at yahoo.com Wed Jul 3 15:55:09 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Wed, 03 Jul 2019 18:55:09 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? Message-ID: Check the following link (4.1) for the optimal gluster volume settings. They are quite safe. Gluster provides a group called virt (/var/lib/glusterd/groups/virt) and can be applied via 'gluster volume set VOLNAME group virt' Then try again. Best Regards, Strahil NikolovOn Jul 3, 2019 11:39, Vladimir Melnik wrote: > > Dear colleagues, > > I have a lab with a bunch of virtual machines (the virtualization is > provided by KVM) running on the same physical host. 4 of these VMs are > working as a GlusterFS cluster and there's one more VM that works as a > client. I'll specify all the packages' versions in the ending of this > message. > > I created 2 volumes - one is having type "Distributed-Replicate" and > another one is "Distribute". The problem is that both of volumes are > showing really poor performance. > > Here's what I see on the client: > $ mount | grep gluster > 10.13.1.16:storage1 on /mnt/glusterfs1 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > 10.13.1.16:storage2 on /mnt/glusterfs2 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s > > The distributed one shows a bit better performance than the > distributed-replicated one, but it's still poor. :-( > > The disk storage itself is OK, here's what I see on each of 4 GlusterFS > servers: > for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s > > The network between all 5 VMs is OK, they all are working on the same > physical host. > > Can't understand, what am I doing wrong. :-( > > Here's the detailed info about the volumes: > Volume Name: storage1 > Type: Distributed-Replicate > Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32 > Status: Started > Snapshot Count: 0 > Number of Bricks: 2 x (2 + 1) = 6 > Transport-type: tcp > Bricks: > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1 > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2 > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3 > Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4 > Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > > Volume Name: storage2 > Type: Distribute > Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067 > Status: Started > Snapshot Count: 0 > Number of Bricks: 4 > Transport-type: tcp > Bricks: > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2 > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2 > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2 > Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2 > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > > The OS is CentOS Linux release 7.6.1810. The packages I'm using are: > glusterfs-6.3-1.el7.x86_64 > glusterfs-api-6.3-1.el7.x86_64 > glusterfs-cli-6.3-1.el7.x86_64 > glusterfs-client-xlators-6.3-1.el7.x86_64 > glusterfs-fuse-6.3-1.el7.x86_64 > glusterfs-libs-6.3-1.el7.x86_64 > glusterfs-server-6.3-1.el7.x86_64 > kernel-3.10.0-327.el7.x86_64 > kernel-3.10.0-514.2.2.el7.x86_64 > kernel-3.10.0-957.12.1.el7.x86_64 > kernel-3.10.0-957.12.2.el7.x86_64 > kernel-3.10.0-957.21.3.el7.x86_64 > kernel-tools-3.10.0-957.21.3.el7.x86_64 > kernel-tools-libs-3.10.0-957.21.3.el7.x86_6 > > Please, be so kind as to help me to understand, did I do it wrong or > that's quite normal performance of GlusterFS? > > Thanks in advance! > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From nico at furyweb.fr Wed Jul 3 16:10:38 2019 From: nico at furyweb.fr (nico at furyweb.fr) Date: Wed, 3 Jul 2019 18:10:38 +0200 (CEST) Subject: [Gluster-users] Parallel process hang on gluster volume In-Reply-To: <1119765369.1998.1561103327489.JavaMail.zimbra@furyweb.fr> References: <1119765369.1998.1561103327489.JavaMail.zimbra@furyweb.fr> Message-ID: <67699782.8645.1562170238208.JavaMail.zimbra@furyweb.fr> Am I alone having this problem ? ----- Mail original ----- De: nico at furyweb.fr ?: "gluster-users" Envoy?: Vendredi 21 Juin 2019 09:48:47 Objet: [Gluster-users] Parallel process hang on gluster volume I encounterd an issue on production servers using GlusterFS servers 5.1 and clients 4.1.5 when several process write at the same time on a gluster volume. With more than 48 process writes on the volume at the same time, they are blocked in D state (uninterruptible sleep), I guess some volume settings have to be tuned but can't figure out which. The client is using op-version 40100 on this volume Below are volume info, volume settings and ps output on blocked processes. _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users From nico at furyweb.fr Wed Jul 3 16:13:43 2019 From: nico at furyweb.fr (nico at furyweb.fr) Date: Wed, 3 Jul 2019 18:13:43 +0200 (CEST) Subject: [Gluster-users] What is TCP/4007 for ? In-Reply-To: <850101223.444.1561050100860.JavaMail.zimbra@furyweb.fr> References: <850101223.444.1561050100860.JavaMail.zimbra@furyweb.fr> Message-ID: <1378653908.8654.1562170423160.JavaMail.zimbra@furyweb.fr> Does anybody have info about this port, is it mandatory, is there any way to disable it if so ? ----- Mail original ----- De: nico at furyweb.fr ?: "gluster-users" Envoy?: Jeudi 20 Juin 2019 19:01:40 Objet: [Gluster-users] What is TCP/4007 for ? I have several Gluster clients behind firewalls and opened 24007:24008,49152:49241 to gluster servers. I recently found that TCP/4007 is blocked from clients to servers but found this port reference nowhere on google search. On server side there's no process listening on TCP/4007 so I would like to disable it on client side, maybe a volume setting should be initialized. Gluster servers are 5.1 Volumes are replica 3 with arbiter Clients are 3.12.15 (Wheezy) or 4.1.5 (Jessie/Stretch) I thank anyone able to give me some information. _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users From v.melnik at tucha.ua Wed Jul 3 16:18:08 2019 From: v.melnik at tucha.ua (Vladimir Melnik) Date: Wed, 3 Jul 2019 19:18:08 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: References: Message-ID: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> Thank you, it helped a little: $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep copied 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s But 14-15 MB/s is still quite far from the actual storage's performance (200-3000 MB/s). :-( Here's full configuration dump (just in case): Option Value ------ ----- cluster.lookup-unhashed on cluster.lookup-optimize on cluster.min-free-disk 10% cluster.min-free-inodes 5% cluster.rebalance-stats off cluster.subvols-per-directory (null) cluster.readdir-optimize off cluster.rsync-hash-regex (null) cluster.extra-hash-regex (null) cluster.dht-xattr-name trusted.glusterfs.dht cluster.randomize-hash-range-by-gfid off cluster.rebal-throttle normal cluster.lock-migration off cluster.force-migration off cluster.local-volume-name (null) cluster.weighted-rebalance on cluster.switch-pattern (null) cluster.entry-change-log on cluster.read-subvolume (null) cluster.read-subvolume-index -1 cluster.read-hash-mode 1 cluster.background-self-heal-count 8 cluster.metadata-self-heal off cluster.data-self-heal off cluster.entry-self-heal off cluster.self-heal-daemon on cluster.heal-timeout 600 cluster.self-heal-window-size 1 cluster.data-change-log on cluster.metadata-change-log on cluster.data-self-heal-algorithm full cluster.eager-lock enable disperse.eager-lock on disperse.other-eager-lock on disperse.eager-lock-timeout 1 disperse.other-eager-lock-timeout 1 cluster.quorum-type auto cluster.quorum-count (null) cluster.choose-local off cluster.self-heal-readdir-size 1KB cluster.post-op-delay-secs 1 cluster.ensure-durability on cluster.consistent-metadata no cluster.heal-wait-queue-length 128 cluster.favorite-child-policy none cluster.full-lock yes diagnostics.latency-measurement off diagnostics.dump-fd-stats off diagnostics.count-fop-hits off diagnostics.brick-log-level INFO diagnostics.client-log-level INFO diagnostics.brick-sys-log-level CRITICAL diagnostics.client-sys-log-level CRITICAL diagnostics.brick-logger (null) diagnostics.client-logger (null) diagnostics.brick-log-format (null) diagnostics.client-log-format (null) diagnostics.brick-log-buf-size 5 diagnostics.client-log-buf-size 5 diagnostics.brick-log-flush-timeout 120 diagnostics.client-log-flush-timeout 120 diagnostics.stats-dump-interval 0 diagnostics.fop-sample-interval 0 diagnostics.stats-dump-format json diagnostics.fop-sample-buf-size 65535 diagnostics.stats-dnscache-ttl-sec 86400 performance.cache-max-file-size 0 performance.cache-min-file-size 0 performance.cache-refresh-timeout 1 performance.cache-priority performance.cache-size 32MB performance.io-thread-count 16 performance.high-prio-threads 16 performance.normal-prio-threads 16 performance.low-prio-threads 32 performance.least-prio-threads 1 performance.enable-least-priority on performance.iot-watchdog-secs (null) performance.iot-cleanup-disconnected-reqsoff performance.iot-pass-through false performance.io-cache-pass-through false performance.cache-size 128MB performance.qr-cache-timeout 1 performance.cache-invalidation false performance.ctime-invalidation false performance.flush-behind on performance.nfs.flush-behind on performance.write-behind-window-size 1MB performance.resync-failed-syncs-after-fsyncoff performance.nfs.write-behind-window-size1MB performance.strict-o-direct off performance.nfs.strict-o-direct off performance.strict-write-ordering off performance.nfs.strict-write-ordering off performance.write-behind-trickling-writeson performance.aggregate-size 128KB performance.nfs.write-behind-trickling-writeson performance.lazy-open yes performance.read-after-open yes performance.open-behind-pass-through false performance.read-ahead-page-count 4 performance.read-ahead-pass-through false performance.readdir-ahead-pass-through false performance.md-cache-pass-through false performance.md-cache-timeout 1 performance.cache-swift-metadata true performance.cache-samba-metadata false performance.cache-capability-xattrs true performance.cache-ima-xattrs true performance.md-cache-statfs off performance.xattr-cache-list performance.nl-cache-pass-through false features.encryption off network.frame-timeout 1800 network.ping-timeout 42 network.tcp-window-size (null) client.ssl off network.remote-dio enable client.event-threads 4 client.tcp-user-timeout 0 client.keepalive-time 20 client.keepalive-interval 2 client.keepalive-count 9 network.tcp-window-size (null) network.inode-lru-limit 16384 auth.allow * auth.reject (null) transport.keepalive 1 server.allow-insecure on server.root-squash off server.all-squash off server.anonuid 65534 server.anongid 65534 server.statedump-path /var/run/gluster server.outstanding-rpc-limit 64 server.ssl off auth.ssl-allow * server.manage-gids off server.dynamic-auth on client.send-gids on server.gid-timeout 300 server.own-thread (null) server.event-threads 4 server.tcp-user-timeout 42 server.keepalive-time 20 server.keepalive-interval 2 server.keepalive-count 9 transport.listen-backlog 1024 transport.address-family inet performance.write-behind on performance.read-ahead off performance.readdir-ahead on performance.io-cache off performance.open-behind on performance.quick-read off performance.nl-cache off performance.stat-prefetch on performance.client-io-threads on performance.nfs.write-behind on performance.nfs.read-ahead off performance.nfs.io-cache off performance.nfs.quick-read off performance.nfs.stat-prefetch off performance.nfs.io-threads off performance.force-readdirp true performance.cache-invalidation false performance.global-cache-invalidation true features.uss off features.snapshot-directory .snaps features.show-snapshot-directory off features.tag-namespaces off network.compression off network.compression.window-size -15 network.compression.mem-level 8 network.compression.min-size 0 network.compression.compression-level -1 network.compression.debug false features.default-soft-limit 80% features.soft-timeout 60 features.hard-timeout 5 features.alert-time 86400 features.quota-deem-statfs off geo-replication.indexing off geo-replication.indexing off geo-replication.ignore-pid-check off geo-replication.ignore-pid-check off features.quota off features.inode-quota off features.bitrot disable debug.trace off debug.log-history no debug.log-file no debug.exclude-ops (null) debug.include-ops (null) debug.error-gen off debug.error-failure (null) debug.error-number (null) debug.random-failure off debug.error-fops (null) nfs.disable on features.read-only off features.worm off features.worm-file-level off features.worm-files-deletable on features.default-retention-period 120 features.retention-mode relax features.auto-commit-period 180 storage.linux-aio off storage.batch-fsync-mode reverse-fsync storage.batch-fsync-delay-usec 0 storage.owner-uid -1 storage.owner-gid -1 storage.node-uuid-pathinfo off storage.health-check-interval 30 storage.build-pgfid off storage.gfid2path on storage.gfid2path-separator : storage.reserve 1 storage.health-check-timeout 10 storage.fips-mode-rchecksum off storage.force-create-mode 0000 storage.force-directory-mode 0000 storage.create-mask 0777 storage.create-directory-mask 0777 storage.max-hardlinks 100 features.ctime on config.gfproxyd off cluster.server-quorum-type server cluster.server-quorum-ratio 0 changelog.changelog off changelog.changelog-dir {{ brick.path }}/.glusterfs/changelogs changelog.encoding ascii changelog.rollover-time 15 changelog.fsync-interval 5 changelog.changelog-barrier-timeout 120 changelog.capture-del-path off features.barrier disable features.barrier-timeout 120 features.trash off features.trash-dir .trashcan features.trash-eliminate-path (null) features.trash-max-filesize 5MB features.trash-internal-op off cluster.enable-shared-storage disable locks.trace off locks.mandatory-locking off cluster.disperse-self-heal-daemon enable cluster.quorum-reads no client.bind-insecure (null) features.shard on features.shard-block-size 64MB features.shard-lru-limit 16384 features.shard-deletion-rate 100 features.scrub-throttle lazy features.scrub-freq biweekly features.scrub false features.expiry-time 120 features.cache-invalidation off features.cache-invalidation-timeout 60 features.leases off features.lease-lock-recall-timeout 60 disperse.background-heals 8 disperse.heal-wait-qlength 128 cluster.heal-timeout 600 dht.force-readdirp on disperse.read-policy gfid-hash cluster.shd-max-threads 8 cluster.shd-wait-qlength 10000 cluster.shd-wait-qlength 10000 cluster.locking-scheme granular cluster.granular-entry-heal no features.locks-revocation-secs 0 features.locks-revocation-clear-all false features.locks-revocation-max-blocked 0 features.locks-monkey-unlocking false features.locks-notify-contention no features.locks-notify-contention-delay 5 disperse.shd-max-threads 1 disperse.shd-wait-qlength 1024 disperse.cpu-extensions auto disperse.self-heal-window-size 1 cluster.use-compound-fops off performance.parallel-readdir off performance.rda-request-size 131072 performance.rda-low-wmark 4096 performance.rda-high-wmark 128KB performance.rda-cache-limit 10MB performance.nl-cache-positive-entry false performance.nl-cache-limit 10MB performance.nl-cache-timeout 60 cluster.brick-multiplex off cluster.max-bricks-per-process 250 disperse.optimistic-change-log on disperse.stripe-cache 4 cluster.halo-enabled False cluster.halo-shd-max-latency 99999 cluster.halo-nfsd-max-latency 5 cluster.halo-max-latency 5 cluster.halo-max-replicas 99999 cluster.halo-min-replicas 2 features.selinux on cluster.daemon-log-level INFO debug.delay-gen off delay-gen.delay-percentage 10% delay-gen.delay-duration 100000 delay-gen.enable disperse.parallel-writes on features.sdfs off features.cloudsync off features.ctime on ctime.noatime on feature.cloudsync-storetype (null) features.enforce-mandatory-lock off What do you think, are there any other knobs worth to be turned? Thanks! On Wed, Jul 03, 2019 at 06:55:09PM +0300, Strahil wrote: > Check the following link (4.1) for the optimal gluster volume settings. > They are quite safe. > > Gluster provides a group called virt (/var/lib/glusterd/groups/virt) and can be applied via 'gluster volume set VOLNAME group virt' > > Then try again. > > Best Regards, > Strahil NikolovOn Jul 3, 2019 11:39, Vladimir Melnik wrote: > > > > Dear colleagues, > > > > I have a lab with a bunch of virtual machines (the virtualization is > > provided by KVM) running on the same physical host. 4 of these VMs are > > working as a GlusterFS cluster and there's one more VM that works as a > > client. I'll specify all the packages' versions in the ending of this > > message. > > > > I created 2 volumes - one is having type "Distributed-Replicate" and > > another one is "Distribute". The problem is that both of volumes are > > showing really poor performance. > > > > Here's what I see on the client: > > $ mount | grep gluster > > 10.13.1.16:storage1 on /mnt/glusterfs1 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > 10.13.1.16:storage2 on /mnt/glusterfs2 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s > > > > The distributed one shows a bit better performance than the > > distributed-replicated one, but it's still poor. :-( > > > > The disk storage itself is OK, here's what I see on each of 4 GlusterFS > > servers: > > for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s > > > > The network between all 5 VMs is OK, they all are working on the same > > physical host. > > > > Can't understand, what am I doing wrong. :-( > > > > Here's the detailed info about the volumes: > > Volume Name: storage1 > > Type: Distributed-Replicate > > Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32 > > Status: Started > > Snapshot Count: 0 > > Number of Bricks: 2 x (2 + 1) = 6 > > Transport-type: tcp > > Bricks: > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1 > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2 > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > > Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3 > > Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4 > > Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > > Options Reconfigured: > > transport.address-family: inet > > nfs.disable: on > > performance.client-io-threads: off > > > > Volume Name: storage2 > > Type: Distribute > > Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067 > > Status: Started > > Snapshot Count: 0 > > Number of Bricks: 4 > > Transport-type: tcp > > Bricks: > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2 > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2 > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2 > > Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2 > > Options Reconfigured: > > transport.address-family: inet > > nfs.disable: on > > > > The OS is CentOS Linux release 7.6.1810. The packages I'm using are: > > glusterfs-6.3-1.el7.x86_64 > > glusterfs-api-6.3-1.el7.x86_64 > > glusterfs-cli-6.3-1.el7.x86_64 > > glusterfs-client-xlators-6.3-1.el7.x86_64 > > glusterfs-fuse-6.3-1.el7.x86_64 > > glusterfs-libs-6.3-1.el7.x86_64 > > glusterfs-server-6.3-1.el7.x86_64 > > kernel-3.10.0-327.el7.x86_64 > > kernel-3.10.0-514.2.2.el7.x86_64 > > kernel-3.10.0-957.12.1.el7.x86_64 > > kernel-3.10.0-957.12.2.el7.x86_64 > > kernel-3.10.0-957.21.3.el7.x86_64 > > kernel-tools-3.10.0-957.21.3.el7.x86_64 > > kernel-tools-libs-3.10.0-957.21.3.el7.x86_6 > > > > Please, be so kind as to help me to understand, did I do it wrong or > > that's quite normal performance of GlusterFS? > > > > Thanks in advance! > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users -- V.Melnik From atumball at redhat.com Wed Jul 3 17:11:07 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Wed, 3 Jul 2019 22:41:07 +0530 Subject: [Gluster-users] What is TCP/4007 for ? In-Reply-To: <1378653908.8654.1562170423160.JavaMail.zimbra@furyweb.fr> References: <850101223.444.1561050100860.JavaMail.zimbra@furyweb.fr> <1378653908.8654.1562170423160.JavaMail.zimbra@furyweb.fr> Message-ID: I am not aware of port 4007 usage in entire Gluster. Not aware of any dependent projects too. -Amar On Wed, Jul 3, 2019 at 9:44 PM wrote: > Does anybody have info about this port, is it mandatory, is there any way > to disable it if so ? > > ----- Mail original ----- > De: nico at furyweb.fr > ?: "gluster-users" > Envoy?: Jeudi 20 Juin 2019 19:01:40 > Objet: [Gluster-users] What is TCP/4007 for ? > > I have several Gluster clients behind firewalls and opened > 24007:24008,49152:49241 to gluster servers. > > I recently found that TCP/4007 is blocked from clients to servers but > found this port reference nowhere on google search. > > On server side there's no process listening on TCP/4007 so I would like to > disable it on client side, maybe a volume setting should be initialized. > > Gluster servers are 5.1 > Volumes are replica 3 with arbiter > Clients are 3.12.15 (Wheezy) or 4.1.5 (Jessie/Stretch) > > I thank anyone able to give me some information. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Wed Jul 3 17:59:24 2019 From: hunter86_bg at yahoo.com (Strahil Nikolov) Date: Wed, 3 Jul 2019 17:59:24 +0000 (UTC) Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> References: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> Message-ID: <1834286573.1635103.1562176764092@mail.yahoo.com> Can you try with a fresh replica volume with 'virt' group applied ? Best Regards,Strahil Nikolov ? ?????, 3 ??? 2019 ?., 19:18:18 ?. ???????+3, Vladimir Melnik ??????: Thank you, it helped a little: $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep copied 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s But 14-15 MB/s is still quite far from the actual storage's performance (200-3000 MB/s). :-( Here's full configuration dump (just in case): Option? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Value ------? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ----- cluster.lookup-unhashed? ? ? ? ? ? ? ? on cluster.lookup-optimize? ? ? ? ? ? ? ? on cluster.min-free-disk? ? ? ? ? ? ? ? ? 10% cluster.min-free-inodes? ? ? ? ? ? ? ? 5% cluster.rebalance-stats? ? ? ? ? ? ? ? off cluster.subvols-per-directory? ? ? ? ? (null) cluster.readdir-optimize? ? ? ? ? ? ? ? off cluster.rsync-hash-regex? ? ? ? ? ? ? ? (null) cluster.extra-hash-regex? ? ? ? ? ? ? ? (null) cluster.dht-xattr-name? ? ? ? ? ? ? ? ? trusted.glusterfs.dht cluster.randomize-hash-range-by-gfid? ? off cluster.rebal-throttle? ? ? ? ? ? ? ? ? normal cluster.lock-migration? ? ? ? ? ? ? ? ? off cluster.force-migration? ? ? ? ? ? ? ? off cluster.local-volume-name? ? ? ? ? ? ? (null) cluster.weighted-rebalance? ? ? ? ? ? ? on cluster.switch-pattern? ? ? ? ? ? ? ? ? (null) cluster.entry-change-log? ? ? ? ? ? ? ? on cluster.read-subvolume? ? ? ? ? ? ? ? ? (null) cluster.read-subvolume-index? ? ? ? ? ? -1 cluster.read-hash-mode? ? ? ? ? ? ? ? ? 1 cluster.background-self-heal-count? ? ? 8 cluster.metadata-self-heal? ? ? ? ? ? ? off cluster.data-self-heal? ? ? ? ? ? ? ? ? off cluster.entry-self-heal? ? ? ? ? ? ? ? off cluster.self-heal-daemon? ? ? ? ? ? ? ? on cluster.heal-timeout? ? ? ? ? ? ? ? ? ? 600 cluster.self-heal-window-size? ? ? ? ? 1 cluster.data-change-log? ? ? ? ? ? ? ? on cluster.metadata-change-log? ? ? ? ? ? on cluster.data-self-heal-algorithm? ? ? ? full cluster.eager-lock? ? ? ? ? ? ? ? ? ? ? enable disperse.eager-lock? ? ? ? ? ? ? ? ? ? on disperse.other-eager-lock? ? ? ? ? ? ? on disperse.eager-lock-timeout? ? ? ? ? ? 1 disperse.other-eager-lock-timeout? ? ? 1 cluster.quorum-type? ? ? ? ? ? ? ? ? ? auto cluster.quorum-count? ? ? ? ? ? ? ? ? ? (null) cluster.choose-local? ? ? ? ? ? ? ? ? ? off cluster.self-heal-readdir-size? ? ? ? ? 1KB cluster.post-op-delay-secs? ? ? ? ? ? ? 1 cluster.ensure-durability? ? ? ? ? ? ? on cluster.consistent-metadata? ? ? ? ? ? no cluster.heal-wait-queue-length? ? ? ? ? 128 cluster.favorite-child-policy? ? ? ? ? none cluster.full-lock? ? ? ? ? ? ? ? ? ? ? yes diagnostics.latency-measurement? ? ? ? off diagnostics.dump-fd-stats? ? ? ? ? ? ? off diagnostics.count-fop-hits? ? ? ? ? ? ? off diagnostics.brick-log-level? ? ? ? ? ? INFO diagnostics.client-log-level? ? ? ? ? ? INFO diagnostics.brick-sys-log-level? ? ? ? CRITICAL diagnostics.client-sys-log-level? ? ? ? CRITICAL diagnostics.brick-logger? ? ? ? ? ? ? ? (null) diagnostics.client-logger? ? ? ? ? ? ? (null) diagnostics.brick-log-format? ? ? ? ? ? (null) diagnostics.client-log-format? ? ? ? ? (null) diagnostics.brick-log-buf-size? ? ? ? ? 5 diagnostics.client-log-buf-size? ? ? ? 5 diagnostics.brick-log-flush-timeout? ? 120 diagnostics.client-log-flush-timeout? ? 120 diagnostics.stats-dump-interval? ? ? ? 0 diagnostics.fop-sample-interval? ? ? ? 0 diagnostics.stats-dump-format? ? ? ? ? json diagnostics.fop-sample-buf-size? ? ? ? 65535 diagnostics.stats-dnscache-ttl-sec? ? ? 86400 performance.cache-max-file-size? ? ? ? 0 performance.cache-min-file-size? ? ? ? 0 performance.cache-refresh-timeout? ? ? 1 performance.cache-priority performance.cache-size? ? ? ? ? ? ? ? ? 32MB performance.io-thread-count? ? ? ? ? ? 16 performance.high-prio-threads? ? ? ? ? 16 performance.normal-prio-threads? ? ? ? 16 performance.low-prio-threads? ? ? ? ? ? 32 performance.least-prio-threads? ? ? ? ? 1 performance.enable-least-priority? ? ? on performance.iot-watchdog-secs? ? ? ? ? (null) performance.iot-cleanup-disconnected-reqsoff performance.iot-pass-through? ? ? ? ? ? false performance.io-cache-pass-through? ? ? false performance.cache-size? ? ? ? ? ? ? ? ? 128MB performance.qr-cache-timeout? ? ? ? ? ? 1 performance.cache-invalidation? ? ? ? ? false performance.ctime-invalidation? ? ? ? ? false performance.flush-behind? ? ? ? ? ? ? ? on performance.nfs.flush-behind? ? ? ? ? ? on performance.write-behind-window-size? ? 1MB performance.resync-failed-syncs-after-fsyncoff performance.nfs.write-behind-window-size1MB performance.strict-o-direct? ? ? ? ? ? off performance.nfs.strict-o-direct? ? ? ? off performance.strict-write-ordering? ? ? off performance.nfs.strict-write-ordering? off performance.write-behind-trickling-writeson performance.aggregate-size? ? ? ? ? ? ? 128KB performance.nfs.write-behind-trickling-writeson performance.lazy-open? ? ? ? ? ? ? ? ? yes performance.read-after-open? ? ? ? ? ? yes performance.open-behind-pass-through? ? false performance.read-ahead-page-count? ? ? 4 performance.read-ahead-pass-through? ? false performance.readdir-ahead-pass-through? false performance.md-cache-pass-through? ? ? false performance.md-cache-timeout? ? ? ? ? ? 1 performance.cache-swift-metadata? ? ? ? true performance.cache-samba-metadata? ? ? ? false performance.cache-capability-xattrs? ? true performance.cache-ima-xattrs? ? ? ? ? ? true performance.md-cache-statfs? ? ? ? ? ? off performance.xattr-cache-list performance.nl-cache-pass-through? ? ? false features.encryption? ? ? ? ? ? ? ? ? ? off network.frame-timeout? ? ? ? ? ? ? ? ? 1800 network.ping-timeout? ? ? ? ? ? ? ? ? ? 42 network.tcp-window-size? ? ? ? ? ? ? ? (null) client.ssl? ? ? ? ? ? ? ? ? ? ? ? ? ? ? off network.remote-dio? ? ? ? ? ? ? ? ? ? ? enable client.event-threads? ? ? ? ? ? ? ? ? ? 4 client.tcp-user-timeout? ? ? ? ? ? ? ? 0 client.keepalive-time? ? ? ? ? ? ? ? ? 20 client.keepalive-interval? ? ? ? ? ? ? 2 client.keepalive-count? ? ? ? ? ? ? ? ? 9 network.tcp-window-size? ? ? ? ? ? ? ? (null) network.inode-lru-limit? ? ? ? ? ? ? ? 16384 auth.allow? ? ? ? ? ? ? ? ? ? ? ? ? ? ? * auth.reject? ? ? ? ? ? ? ? ? ? ? ? ? ? (null) transport.keepalive? ? ? ? ? ? ? ? ? ? 1 server.allow-insecure? ? ? ? ? ? ? ? ? on server.root-squash? ? ? ? ? ? ? ? ? ? ? off server.all-squash? ? ? ? ? ? ? ? ? ? ? off server.anonuid? ? ? ? ? ? ? ? ? ? ? ? ? 65534 server.anongid? ? ? ? ? ? ? ? ? ? ? ? ? 65534 server.statedump-path? ? ? ? ? ? ? ? ? /var/run/gluster server.outstanding-rpc-limit? ? ? ? ? ? 64 server.ssl? ? ? ? ? ? ? ? ? ? ? ? ? ? ? off auth.ssl-allow? ? ? ? ? ? ? ? ? ? ? ? ? * server.manage-gids? ? ? ? ? ? ? ? ? ? ? off server.dynamic-auth? ? ? ? ? ? ? ? ? ? on client.send-gids? ? ? ? ? ? ? ? ? ? ? ? on server.gid-timeout? ? ? ? ? ? ? ? ? ? ? 300 server.own-thread? ? ? ? ? ? ? ? ? ? ? (null) server.event-threads? ? ? ? ? ? ? ? ? ? 4 server.tcp-user-timeout? ? ? ? ? ? ? ? 42 server.keepalive-time? ? ? ? ? ? ? ? ? 20 server.keepalive-interval? ? ? ? ? ? ? 2 server.keepalive-count? ? ? ? ? ? ? ? ? 9 transport.listen-backlog? ? ? ? ? ? ? ? 1024 transport.address-family? ? ? ? ? ? ? ? inet performance.write-behind? ? ? ? ? ? ? ? on performance.read-ahead? ? ? ? ? ? ? ? ? off performance.readdir-ahead? ? ? ? ? ? ? on performance.io-cache? ? ? ? ? ? ? ? ? ? off performance.open-behind? ? ? ? ? ? ? ? on performance.quick-read? ? ? ? ? ? ? ? ? off performance.nl-cache? ? ? ? ? ? ? ? ? ? off performance.stat-prefetch? ? ? ? ? ? ? on performance.client-io-threads? ? ? ? ? on performance.nfs.write-behind? ? ? ? ? ? on performance.nfs.read-ahead? ? ? ? ? ? ? off performance.nfs.io-cache? ? ? ? ? ? ? ? off performance.nfs.quick-read? ? ? ? ? ? ? off performance.nfs.stat-prefetch? ? ? ? ? off performance.nfs.io-threads? ? ? ? ? ? ? off performance.force-readdirp? ? ? ? ? ? ? true performance.cache-invalidation? ? ? ? ? false performance.global-cache-invalidation? true features.uss? ? ? ? ? ? ? ? ? ? ? ? ? ? off features.snapshot-directory? ? ? ? ? ? .snaps features.show-snapshot-directory? ? ? ? off features.tag-namespaces? ? ? ? ? ? ? ? off network.compression? ? ? ? ? ? ? ? ? ? off network.compression.window-size? ? ? ? -15 network.compression.mem-level? ? ? ? ? 8 network.compression.min-size? ? ? ? ? ? 0 network.compression.compression-level? -1 network.compression.debug? ? ? ? ? ? ? false features.default-soft-limit? ? ? ? ? ? 80% features.soft-timeout? ? ? ? ? ? ? ? ? 60 features.hard-timeout? ? ? ? ? ? ? ? ? 5 features.alert-time? ? ? ? ? ? ? ? ? ? 86400 features.quota-deem-statfs? ? ? ? ? ? ? off geo-replication.indexing? ? ? ? ? ? ? ? off geo-replication.indexing? ? ? ? ? ? ? ? off geo-replication.ignore-pid-check? ? ? ? off geo-replication.ignore-pid-check? ? ? ? off features.quota? ? ? ? ? ? ? ? ? ? ? ? ? off features.inode-quota? ? ? ? ? ? ? ? ? ? off features.bitrot? ? ? ? ? ? ? ? ? ? ? ? disable debug.trace? ? ? ? ? ? ? ? ? ? ? ? ? ? off debug.log-history? ? ? ? ? ? ? ? ? ? ? no debug.log-file? ? ? ? ? ? ? ? ? ? ? ? ? no debug.exclude-ops? ? ? ? ? ? ? ? ? ? ? (null) debug.include-ops? ? ? ? ? ? ? ? ? ? ? (null) debug.error-gen? ? ? ? ? ? ? ? ? ? ? ? off debug.error-failure? ? ? ? ? ? ? ? ? ? (null) debug.error-number? ? ? ? ? ? ? ? ? ? ? (null) debug.random-failure? ? ? ? ? ? ? ? ? ? off debug.error-fops? ? ? ? ? ? ? ? ? ? ? ? (null) nfs.disable? ? ? ? ? ? ? ? ? ? ? ? ? ? on features.read-only? ? ? ? ? ? ? ? ? ? ? off features.worm? ? ? ? ? ? ? ? ? ? ? ? ? off features.worm-file-level? ? ? ? ? ? ? ? off features.worm-files-deletable? ? ? ? ? on features.default-retention-period? ? ? 120 features.retention-mode? ? ? ? ? ? ? ? relax features.auto-commit-period? ? ? ? ? ? 180 storage.linux-aio? ? ? ? ? ? ? ? ? ? ? off storage.batch-fsync-mode? ? ? ? ? ? ? ? reverse-fsync storage.batch-fsync-delay-usec? ? ? ? ? 0 storage.owner-uid? ? ? ? ? ? ? ? ? ? ? -1 storage.owner-gid? ? ? ? ? ? ? ? ? ? ? -1 storage.node-uuid-pathinfo? ? ? ? ? ? ? off storage.health-check-interval? ? ? ? ? 30 storage.build-pgfid? ? ? ? ? ? ? ? ? ? off storage.gfid2path? ? ? ? ? ? ? ? ? ? ? on storage.gfid2path-separator? ? ? ? ? ? : storage.reserve? ? ? ? ? ? ? ? ? ? ? ? 1 storage.health-check-timeout? ? ? ? ? ? 10 storage.fips-mode-rchecksum? ? ? ? ? ? off storage.force-create-mode? ? ? ? ? ? ? 0000 storage.force-directory-mode? ? ? ? ? ? 0000 storage.create-mask? ? ? ? ? ? ? ? ? ? 0777 storage.create-directory-mask? ? ? ? ? 0777 storage.max-hardlinks? ? ? ? ? ? ? ? ? 100 features.ctime? ? ? ? ? ? ? ? ? ? ? ? ? on config.gfproxyd? ? ? ? ? ? ? ? ? ? ? ? off cluster.server-quorum-type? ? ? ? ? ? ? server cluster.server-quorum-ratio? ? ? ? ? ? 0 changelog.changelog? ? ? ? ? ? ? ? ? ? off changelog.changelog-dir? ? ? ? ? ? ? ? {{ brick.path }}/.glusterfs/changelogs changelog.encoding? ? ? ? ? ? ? ? ? ? ? ascii changelog.rollover-time? ? ? ? ? ? ? ? 15 changelog.fsync-interval? ? ? ? ? ? ? ? 5 changelog.changelog-barrier-timeout? ? 120 changelog.capture-del-path? ? ? ? ? ? ? off features.barrier? ? ? ? ? ? ? ? ? ? ? ? disable features.barrier-timeout? ? ? ? ? ? ? ? 120 features.trash? ? ? ? ? ? ? ? ? ? ? ? ? off features.trash-dir? ? ? ? ? ? ? ? ? ? ? .trashcan features.trash-eliminate-path? ? ? ? ? (null) features.trash-max-filesize? ? ? ? ? ? 5MB features.trash-internal-op? ? ? ? ? ? ? off cluster.enable-shared-storage? ? ? ? ? disable locks.trace? ? ? ? ? ? ? ? ? ? ? ? ? ? off locks.mandatory-locking? ? ? ? ? ? ? ? off cluster.disperse-self-heal-daemon? ? ? enable cluster.quorum-reads? ? ? ? ? ? ? ? ? ? no client.bind-insecure? ? ? ? ? ? ? ? ? ? (null) features.shard? ? ? ? ? ? ? ? ? ? ? ? ? on features.shard-block-size? ? ? ? ? ? ? 64MB features.shard-lru-limit? ? ? ? ? ? ? ? 16384 features.shard-deletion-rate? ? ? ? ? ? 100 features.scrub-throttle? ? ? ? ? ? ? ? lazy features.scrub-freq? ? ? ? ? ? ? ? ? ? biweekly features.scrub? ? ? ? ? ? ? ? ? ? ? ? ? false features.expiry-time? ? ? ? ? ? ? ? ? ? 120 features.cache-invalidation? ? ? ? ? ? off features.cache-invalidation-timeout? ? 60 features.leases? ? ? ? ? ? ? ? ? ? ? ? off features.lease-lock-recall-timeout? ? ? 60 disperse.background-heals? ? ? ? ? ? ? 8 disperse.heal-wait-qlength? ? ? ? ? ? ? 128 cluster.heal-timeout? ? ? ? ? ? ? ? ? ? 600 dht.force-readdirp? ? ? ? ? ? ? ? ? ? ? on disperse.read-policy? ? ? ? ? ? ? ? ? ? gfid-hash cluster.shd-max-threads? ? ? ? ? ? ? ? 8 cluster.shd-wait-qlength? ? ? ? ? ? ? ? 10000 cluster.shd-wait-qlength? ? ? ? ? ? ? ? 10000 cluster.locking-scheme? ? ? ? ? ? ? ? ? granular cluster.granular-entry-heal? ? ? ? ? ? no features.locks-revocation-secs? ? ? ? ? 0 features.locks-revocation-clear-all? ? false features.locks-revocation-max-blocked? 0 features.locks-monkey-unlocking? ? ? ? false features.locks-notify-contention? ? ? ? no features.locks-notify-contention-delay? 5 disperse.shd-max-threads? ? ? ? ? ? ? ? 1 disperse.shd-wait-qlength? ? ? ? ? ? ? 1024 disperse.cpu-extensions? ? ? ? ? ? ? ? auto disperse.self-heal-window-size? ? ? ? ? 1 cluster.use-compound-fops? ? ? ? ? ? ? off performance.parallel-readdir? ? ? ? ? ? off performance.rda-request-size? ? ? ? ? ? 131072 performance.rda-low-wmark? ? ? ? ? ? ? 4096 performance.rda-high-wmark? ? ? ? ? ? ? 128KB performance.rda-cache-limit? ? ? ? ? ? 10MB performance.nl-cache-positive-entry? ? false performance.nl-cache-limit? ? ? ? ? ? ? 10MB performance.nl-cache-timeout? ? ? ? ? ? 60 cluster.brick-multiplex? ? ? ? ? ? ? ? off cluster.max-bricks-per-process? ? ? ? ? 250 disperse.optimistic-change-log? ? ? ? ? on disperse.stripe-cache? ? ? ? ? ? ? ? ? 4 cluster.halo-enabled? ? ? ? ? ? ? ? ? ? False cluster.halo-shd-max-latency? ? ? ? ? ? 99999 cluster.halo-nfsd-max-latency? ? ? ? ? 5 cluster.halo-max-latency? ? ? ? ? ? ? ? 5 cluster.halo-max-replicas? ? ? ? ? ? ? 99999 cluster.halo-min-replicas? ? ? ? ? ? ? 2 features.selinux? ? ? ? ? ? ? ? ? ? ? ? on cluster.daemon-log-level? ? ? ? ? ? ? ? INFO debug.delay-gen? ? ? ? ? ? ? ? ? ? ? ? off delay-gen.delay-percentage? ? ? ? ? ? ? 10% delay-gen.delay-duration? ? ? ? ? ? ? ? 100000 delay-gen.enable disperse.parallel-writes? ? ? ? ? ? ? ? on features.sdfs? ? ? ? ? ? ? ? ? ? ? ? ? off features.cloudsync? ? ? ? ? ? ? ? ? ? ? off features.ctime? ? ? ? ? ? ? ? ? ? ? ? ? on ctime.noatime? ? ? ? ? ? ? ? ? ? ? ? ? on feature.cloudsync-storetype? ? ? ? ? ? (null) features.enforce-mandatory-lock? ? ? ? off What do you think, are there any other knobs worth to be turned? Thanks! On Wed, Jul 03, 2019 at 06:55:09PM +0300, Strahil wrote: > Check the following link (4.1)? for the optimal gluster volume settings. > They are quite safe. > > Gluster? provides a group called? virt (/var/lib/glusterd/groups/virt)? and can be applied via? 'gluster volume set VOLNAME group virt' > > Then try again. > > Best Regards, > Strahil NikolovOn Jul 3, 2019 11:39, Vladimir Melnik wrote: > > > > Dear colleagues, > > > > I have a lab with a bunch of virtual machines (the virtualization is > > provided by KVM) running on the same physical host. 4 of these VMs are > > working as a GlusterFS cluster and there's one more VM that works as a > > client. I'll specify all the packages' versions in the ending of this > > message. > > > > I created 2 volumes - one is having type "Distributed-Replicate" and > > another one is "Distribute". The problem is that both of volumes are > > showing really poor performance. > > > > Here's what I see on the client: > > $ mount | grep gluster > > 10.13.1.16:storage1 on /mnt/glusterfs1 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > 10.13.1.16:storage2 on /mnt/glusterfs2 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s > > > > The distributed one shows a bit better performance than the > > distributed-replicated one, but it's still poor. :-( > > > > The disk storage itself is OK, here's what I see on each of 4 GlusterFS > > servers: > > for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s > > 10+0 records in > > 10+0 records out > > 10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s > > > > The network between all 5 VMs is OK, they all are working on the same > > physical host. > > > > Can't understand, what am I doing wrong. :-( > > > > Here's the detailed info about the volumes: > > Volume Name: storage1 > > Type: Distributed-Replicate > > Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32 > > Status: Started > > Snapshot Count: 0 > > Number of Bricks: 2 x (2 + 1) = 6 > > Transport-type: tcp > > Bricks: > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1 > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2 > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > > Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3 > > Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4 > > Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > > Options Reconfigured: > > transport.address-family: inet > > nfs.disable: on > > performance.client-io-threads: off > > > > Volume Name: storage2 > > Type: Distribute > > Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067 > > Status: Started > > Snapshot Count: 0 > > Number of Bricks: 4 > > Transport-type: tcp > > Bricks: > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2 > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2 > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2 > > Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2 > > Options Reconfigured: > > transport.address-family: inet > > nfs.disable: on > > > > The OS is CentOS Linux release 7.6.1810. The packages I'm using are: > > glusterfs-6.3-1.el7.x86_64 > > glusterfs-api-6.3-1.el7.x86_64 > > glusterfs-cli-6.3-1.el7.x86_64 > > glusterfs-client-xlators-6.3-1.el7.x86_64 > > glusterfs-fuse-6.3-1.el7.x86_64 > > glusterfs-libs-6.3-1.el7.x86_64 > > glusterfs-server-6.3-1.el7.x86_64 > > kernel-3.10.0-327.el7.x86_64 > > kernel-3.10.0-514.2.2.el7.x86_64 > > kernel-3.10.0-957.12.1.el7.x86_64 > > kernel-3.10.0-957.12.2.el7.x86_64 > > kernel-3.10.0-957.21.3.el7.x86_64 > > kernel-tools-3.10.0-957.21.3.el7.x86_64 > > kernel-tools-libs-3.10.0-957.21.3.el7.x86_6 > > > > Please, be so kind as to help me to understand, did I do it wrong or > > that's quite normal performance of GlusterFS? > > > > Thanks in advance! > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users -- V.Melnik -------------- next part -------------- An HTML attachment was scrubbed... URL: From nico at furyweb.fr Wed Jul 3 18:20:31 2019 From: nico at furyweb.fr (nico at furyweb.fr) Date: Wed, 3 Jul 2019 20:20:31 +0200 (CEST) Subject: [Gluster-users] What is TCP/4007 for ? In-Reply-To: References: <850101223.444.1561050100860.JavaMail.zimbra@furyweb.fr> <1378653908.8654.1562170423160.JavaMail.zimbra@furyweb.fr> Message-ID: <78027737.8768.1562178031772.JavaMail.zimbra@furyweb.fr> Thanks for reply, I can show this usage with lsof on some 3.12.15 clients : # ps -ef | awk '/[g]lusterfs/ {print $2}' | xargs -n 1 lsof -p | grep -w 4007 glusterfs 28011 root 7u IPv4 205739452 0t0 TCP traitNetVM1:49148->glusterVMa:4007 (SYN_SENT) glusterfs 49535 root 7u IPv4 205739406 0t0 TCP traitNetVM1:49150->glusterVMa:4007 (SYN_SENT) glusterfs 92903 root 6u IPv4 205738745 0t0 TCP traitNetVM1:49151->glusterVMa:4007 (SYN_SENT) glusterfs 96465 root 6u IPv4 205739411 0t0 TCP traitNetVM1:49149->glusterVMa:4007 (SYN_SENT) ----- Mail original ----- De: "Amar Tumballi Suryanarayan" ?: nico at furyweb.fr Cc: "gluster-users" Envoy?: Mercredi 3 Juillet 2019 19:11:07 Objet: Re: [Gluster-users] What is TCP/4007 for ? I am not aware of port 4007 usage in entire Gluster. Not aware of any dependent projects too. -Amar On Wed, Jul 3, 2019 at 9:44 PM < [ mailto:nico at furyweb.fr | nico at furyweb.fr ] > wrote: Does anybody have info about this port, is it mandatory, is there any way to disable it if so ? ----- Mail original ----- De: [ mailto:nico at furyweb.fr | nico at furyweb.fr ] ?: "gluster-users" < [ mailto:gluster-users at gluster.org | gluster-users at gluster.org ] > Envoy?: Jeudi 20 Juin 2019 19:01:40 Objet: [Gluster-users] What is TCP/4007 for ? I have several Gluster clients behind firewalls and opened 24007:24008,49152:49241 to gluster servers. I recently found that TCP/4007 is blocked from clients to servers but found this port reference nowhere on google search. On server side there's no process listening on TCP/4007 so I would like to disable it on client side, maybe a volume setting should be initialized. Gluster servers are 5.1 Volumes are replica 3 with arbiter Clients are 3.12.15 (Wheezy) or 4.1.5 (Jessie/Stretch) I thank anyone able to give me some information. _______________________________________________ Gluster-users mailing list [ mailto:Gluster-users at gluster.org | Gluster-users at gluster.org ] [ https://lists.gluster.org/mailman/listinfo/gluster-users | https://lists.gluster.org/mailman/listinfo/gluster-users ] _______________________________________________ Gluster-users mailing list [ mailto:Gluster-users at gluster.org | Gluster-users at gluster.org ] [ https://lists.gluster.org/mailman/listinfo/gluster-users | https://lists.gluster.org/mailman/listinfo/gluster-users ] -- Amar Tumballi (amarts) From v.melnik at tucha.ua Wed Jul 3 18:44:57 2019 From: v.melnik at tucha.ua (Vladimir Melnik) Date: Wed, 3 Jul 2019 21:44:57 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: <1834286573.1635103.1562176764092@mail.yahoo.com> References: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> <1834286573.1635103.1562176764092@mail.yahoo.com> Message-ID: <20190703184457.GR1375@shagomer.uplink.tucha13.net> Thank you, I tried to do that. Created a new volume: $ gluster volume create storage2 \ replica 3 \ arbiter 1 \ transport tcp \ gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2/brick1 \ gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2/brick2 \ gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2/brick_arbiter \ gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2/brick3 \ gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2/brick4 \ gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2/brick_arbiter Changed the volume's settings: $ gluster volume set storage2 group virt Started the volume: $ gluster volume start storage2 And the same thing: $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done 2>&1 | grep copied 10485760 bytes (10 MB) copied, 0.988662 s, 10.6 MB/s 10485760 bytes (10 MB) copied, 0.768863 s, 13.6 MB/s 10485760 bytes (10 MB) copied, 0.828568 s, 12.7 MB/s 10485760 bytes (10 MB) copied, 0.84322 s, 12.4 MB/s 10485760 bytes (10 MB) copied, 0.812504 s, 12.9 MB/s On Wed, Jul 03, 2019 at 05:59:24PM +0000, Strahil Nikolov wrote: > Can you try with a fresh replica volume with 'virt' group applied ? > Best Regards,Strahil Nikolov > ? ?????, 3 ??? 2019 ?., 19:18:18 ?. ???????+3, Vladimir Melnik ??????: > > Thank you, it helped a little: > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep copied > 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s > 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s > 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s > 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s > 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s > > But 14-15 MB/s is still quite far from the actual storage's performance (200-3000 MB/s). :-( > > Here's full configuration dump (just in case): > > Option? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Value > ------? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ----- > cluster.lookup-unhashed? ? ? ? ? ? ? ? on > cluster.lookup-optimize? ? ? ? ? ? ? ? on > cluster.min-free-disk? ? ? ? ? ? ? ? ? 10% > cluster.min-free-inodes? ? ? ? ? ? ? ? 5% > cluster.rebalance-stats? ? ? ? ? ? ? ? off > cluster.subvols-per-directory? ? ? ? ? (null) > cluster.readdir-optimize? ? ? ? ? ? ? ? off > cluster.rsync-hash-regex? ? ? ? ? ? ? ? (null) > cluster.extra-hash-regex? ? ? ? ? ? ? ? (null) > cluster.dht-xattr-name? ? ? ? ? ? ? ? ? trusted.glusterfs.dht > cluster.randomize-hash-range-by-gfid? ? off > cluster.rebal-throttle? ? ? ? ? ? ? ? ? normal > cluster.lock-migration? ? ? ? ? ? ? ? ? off > cluster.force-migration? ? ? ? ? ? ? ? off > cluster.local-volume-name? ? ? ? ? ? ? (null) > cluster.weighted-rebalance? ? ? ? ? ? ? on > cluster.switch-pattern? ? ? ? ? ? ? ? ? (null) > cluster.entry-change-log? ? ? ? ? ? ? ? on > cluster.read-subvolume? ? ? ? ? ? ? ? ? (null) > cluster.read-subvolume-index? ? ? ? ? ? -1 > cluster.read-hash-mode? ? ? ? ? ? ? ? ? 1 > cluster.background-self-heal-count? ? ? 8 > cluster.metadata-self-heal? ? ? ? ? ? ? off > cluster.data-self-heal? ? ? ? ? ? ? ? ? off > cluster.entry-self-heal? ? ? ? ? ? ? ? off > cluster.self-heal-daemon? ? ? ? ? ? ? ? on > cluster.heal-timeout? ? ? ? ? ? ? ? ? ? 600 > cluster.self-heal-window-size? ? ? ? ? 1 > cluster.data-change-log? ? ? ? ? ? ? ? on > cluster.metadata-change-log? ? ? ? ? ? on > cluster.data-self-heal-algorithm? ? ? ? full > cluster.eager-lock? ? ? ? ? ? ? ? ? ? ? enable > disperse.eager-lock? ? ? ? ? ? ? ? ? ? on > disperse.other-eager-lock? ? ? ? ? ? ? on > disperse.eager-lock-timeout? ? ? ? ? ? 1 > disperse.other-eager-lock-timeout? ? ? 1 > cluster.quorum-type? ? ? ? ? ? ? ? ? ? auto > cluster.quorum-count? ? ? ? ? ? ? ? ? ? (null) > cluster.choose-local? ? ? ? ? ? ? ? ? ? off > cluster.self-heal-readdir-size? ? ? ? ? 1KB > cluster.post-op-delay-secs? ? ? ? ? ? ? 1 > cluster.ensure-durability? ? ? ? ? ? ? on > cluster.consistent-metadata? ? ? ? ? ? no > cluster.heal-wait-queue-length? ? ? ? ? 128 > cluster.favorite-child-policy? ? ? ? ? none > cluster.full-lock? ? ? ? ? ? ? ? ? ? ? yes > diagnostics.latency-measurement? ? ? ? off > diagnostics.dump-fd-stats? ? ? ? ? ? ? off > diagnostics.count-fop-hits? ? ? ? ? ? ? off > diagnostics.brick-log-level? ? ? ? ? ? INFO > diagnostics.client-log-level? ? ? ? ? ? INFO > diagnostics.brick-sys-log-level? ? ? ? CRITICAL > diagnostics.client-sys-log-level? ? ? ? CRITICAL > diagnostics.brick-logger? ? ? ? ? ? ? ? (null) > diagnostics.client-logger? ? ? ? ? ? ? (null) > diagnostics.brick-log-format? ? ? ? ? ? (null) > diagnostics.client-log-format? ? ? ? ? (null) > diagnostics.brick-log-buf-size? ? ? ? ? 5 > diagnostics.client-log-buf-size? ? ? ? 5 > diagnostics.brick-log-flush-timeout? ? 120 > diagnostics.client-log-flush-timeout? ? 120 > diagnostics.stats-dump-interval? ? ? ? 0 > diagnostics.fop-sample-interval? ? ? ? 0 > diagnostics.stats-dump-format? ? ? ? ? json > diagnostics.fop-sample-buf-size? ? ? ? 65535 > diagnostics.stats-dnscache-ttl-sec? ? ? 86400 > performance.cache-max-file-size? ? ? ? 0 > performance.cache-min-file-size? ? ? ? 0 > performance.cache-refresh-timeout? ? ? 1 > performance.cache-priority > performance.cache-size? ? ? ? ? ? ? ? ? 32MB > performance.io-thread-count? ? ? ? ? ? 16 > performance.high-prio-threads? ? ? ? ? 16 > performance.normal-prio-threads? ? ? ? 16 > performance.low-prio-threads? ? ? ? ? ? 32 > performance.least-prio-threads? ? ? ? ? 1 > performance.enable-least-priority? ? ? on > performance.iot-watchdog-secs? ? ? ? ? (null) > performance.iot-cleanup-disconnected-reqsoff > performance.iot-pass-through? ? ? ? ? ? false > performance.io-cache-pass-through? ? ? false > performance.cache-size? ? ? ? ? ? ? ? ? 128MB > performance.qr-cache-timeout? ? ? ? ? ? 1 > performance.cache-invalidation? ? ? ? ? false > performance.ctime-invalidation? ? ? ? ? false > performance.flush-behind? ? ? ? ? ? ? ? on > performance.nfs.flush-behind? ? ? ? ? ? on > performance.write-behind-window-size? ? 1MB > performance.resync-failed-syncs-after-fsyncoff > performance.nfs.write-behind-window-size1MB > performance.strict-o-direct? ? ? ? ? ? off > performance.nfs.strict-o-direct? ? ? ? off > performance.strict-write-ordering? ? ? off > performance.nfs.strict-write-ordering? off > performance.write-behind-trickling-writeson > performance.aggregate-size? ? ? ? ? ? ? 128KB > performance.nfs.write-behind-trickling-writeson > performance.lazy-open? ? ? ? ? ? ? ? ? yes > performance.read-after-open? ? ? ? ? ? yes > performance.open-behind-pass-through? ? false > performance.read-ahead-page-count? ? ? 4 > performance.read-ahead-pass-through? ? false > performance.readdir-ahead-pass-through? false > performance.md-cache-pass-through? ? ? false > performance.md-cache-timeout? ? ? ? ? ? 1 > performance.cache-swift-metadata? ? ? ? true > performance.cache-samba-metadata? ? ? ? false > performance.cache-capability-xattrs? ? true > performance.cache-ima-xattrs? ? ? ? ? ? true > performance.md-cache-statfs? ? ? ? ? ? off > performance.xattr-cache-list > performance.nl-cache-pass-through? ? ? false > features.encryption? ? ? ? ? ? ? ? ? ? off > network.frame-timeout? ? ? ? ? ? ? ? ? 1800 > network.ping-timeout? ? ? ? ? ? ? ? ? ? 42 > network.tcp-window-size? ? ? ? ? ? ? ? (null) > client.ssl? ? ? ? ? ? ? ? ? ? ? ? ? ? ? off > network.remote-dio? ? ? ? ? ? ? ? ? ? ? enable > client.event-threads? ? ? ? ? ? ? ? ? ? 4 > client.tcp-user-timeout? ? ? ? ? ? ? ? 0 > client.keepalive-time? ? ? ? ? ? ? ? ? 20 > client.keepalive-interval? ? ? ? ? ? ? 2 > client.keepalive-count? ? ? ? ? ? ? ? ? 9 > network.tcp-window-size? ? ? ? ? ? ? ? (null) > network.inode-lru-limit? ? ? ? ? ? ? ? 16384 > auth.allow? ? ? ? ? ? ? ? ? ? ? ? ? ? ? * > auth.reject? ? ? ? ? ? ? ? ? ? ? ? ? ? (null) > transport.keepalive? ? ? ? ? ? ? ? ? ? 1 > server.allow-insecure? ? ? ? ? ? ? ? ? on > server.root-squash? ? ? ? ? ? ? ? ? ? ? off > server.all-squash? ? ? ? ? ? ? ? ? ? ? off > server.anonuid? ? ? ? ? ? ? ? ? ? ? ? ? 65534 > server.anongid? ? ? ? ? ? ? ? ? ? ? ? ? 65534 > server.statedump-path? ? ? ? ? ? ? ? ? /var/run/gluster > server.outstanding-rpc-limit? ? ? ? ? ? 64 > server.ssl? ? ? ? ? ? ? ? ? ? ? ? ? ? ? off > auth.ssl-allow? ? ? ? ? ? ? ? ? ? ? ? ? * > server.manage-gids? ? ? ? ? ? ? ? ? ? ? off > server.dynamic-auth? ? ? ? ? ? ? ? ? ? on > client.send-gids? ? ? ? ? ? ? ? ? ? ? ? on > server.gid-timeout? ? ? ? ? ? ? ? ? ? ? 300 > server.own-thread? ? ? ? ? ? ? ? ? ? ? (null) > server.event-threads? ? ? ? ? ? ? ? ? ? 4 > server.tcp-user-timeout? ? ? ? ? ? ? ? 42 > server.keepalive-time? ? ? ? ? ? ? ? ? 20 > server.keepalive-interval? ? ? ? ? ? ? 2 > server.keepalive-count? ? ? ? ? ? ? ? ? 9 > transport.listen-backlog? ? ? ? ? ? ? ? 1024 > transport.address-family? ? ? ? ? ? ? ? inet > performance.write-behind? ? ? ? ? ? ? ? on > performance.read-ahead? ? ? ? ? ? ? ? ? off > performance.readdir-ahead? ? ? ? ? ? ? on > performance.io-cache? ? ? ? ? ? ? ? ? ? off > performance.open-behind? ? ? ? ? ? ? ? on > performance.quick-read? ? ? ? ? ? ? ? ? off > performance.nl-cache? ? ? ? ? ? ? ? ? ? off > performance.stat-prefetch? ? ? ? ? ? ? on > performance.client-io-threads? ? ? ? ? on > performance.nfs.write-behind? ? ? ? ? ? on > performance.nfs.read-ahead? ? ? ? ? ? ? off > performance.nfs.io-cache? ? ? ? ? ? ? ? off > performance.nfs.quick-read? ? ? ? ? ? ? off > performance.nfs.stat-prefetch? ? ? ? ? off > performance.nfs.io-threads? ? ? ? ? ? ? off > performance.force-readdirp? ? ? ? ? ? ? true > performance.cache-invalidation? ? ? ? ? false > performance.global-cache-invalidation? true > features.uss? ? ? ? ? ? ? ? ? ? ? ? ? ? off > features.snapshot-directory? ? ? ? ? ? .snaps > features.show-snapshot-directory? ? ? ? off > features.tag-namespaces? ? ? ? ? ? ? ? off > network.compression? ? ? ? ? ? ? ? ? ? off > network.compression.window-size? ? ? ? -15 > network.compression.mem-level? ? ? ? ? 8 > network.compression.min-size? ? ? ? ? ? 0 > network.compression.compression-level? -1 > network.compression.debug? ? ? ? ? ? ? false > features.default-soft-limit? ? ? ? ? ? 80% > features.soft-timeout? ? ? ? ? ? ? ? ? 60 > features.hard-timeout? ? ? ? ? ? ? ? ? 5 > features.alert-time? ? ? ? ? ? ? ? ? ? 86400 > features.quota-deem-statfs? ? ? ? ? ? ? off > geo-replication.indexing? ? ? ? ? ? ? ? off > geo-replication.indexing? ? ? ? ? ? ? ? off > geo-replication.ignore-pid-check? ? ? ? off > geo-replication.ignore-pid-check? ? ? ? off > features.quota? ? ? ? ? ? ? ? ? ? ? ? ? off > features.inode-quota? ? ? ? ? ? ? ? ? ? off > features.bitrot? ? ? ? ? ? ? ? ? ? ? ? disable > debug.trace? ? ? ? ? ? ? ? ? ? ? ? ? ? off > debug.log-history? ? ? ? ? ? ? ? ? ? ? no > debug.log-file? ? ? ? ? ? ? ? ? ? ? ? ? no > debug.exclude-ops? ? ? ? ? ? ? ? ? ? ? (null) > debug.include-ops? ? ? ? ? ? ? ? ? ? ? (null) > debug.error-gen? ? ? ? ? ? ? ? ? ? ? ? off > debug.error-failure? ? ? ? ? ? ? ? ? ? (null) > debug.error-number? ? ? ? ? ? ? ? ? ? ? (null) > debug.random-failure? ? ? ? ? ? ? ? ? ? off > debug.error-fops? ? ? ? ? ? ? ? ? ? ? ? (null) > nfs.disable? ? ? ? ? ? ? ? ? ? ? ? ? ? on > features.read-only? ? ? ? ? ? ? ? ? ? ? off > features.worm? ? ? ? ? ? ? ? ? ? ? ? ? off > features.worm-file-level? ? ? ? ? ? ? ? off > features.worm-files-deletable? ? ? ? ? on > features.default-retention-period? ? ? 120 > features.retention-mode? ? ? ? ? ? ? ? relax > features.auto-commit-period? ? ? ? ? ? 180 > storage.linux-aio? ? ? ? ? ? ? ? ? ? ? off > storage.batch-fsync-mode? ? ? ? ? ? ? ? reverse-fsync > storage.batch-fsync-delay-usec? ? ? ? ? 0 > storage.owner-uid? ? ? ? ? ? ? ? ? ? ? -1 > storage.owner-gid? ? ? ? ? ? ? ? ? ? ? -1 > storage.node-uuid-pathinfo? ? ? ? ? ? ? off > storage.health-check-interval? ? ? ? ? 30 > storage.build-pgfid? ? ? ? ? ? ? ? ? ? off > storage.gfid2path? ? ? ? ? ? ? ? ? ? ? on > storage.gfid2path-separator? ? ? ? ? ? : > storage.reserve? ? ? ? ? ? ? ? ? ? ? ? 1 > storage.health-check-timeout? ? ? ? ? ? 10 > storage.fips-mode-rchecksum? ? ? ? ? ? off > storage.force-create-mode? ? ? ? ? ? ? 0000 > storage.force-directory-mode? ? ? ? ? ? 0000 > storage.create-mask? ? ? ? ? ? ? ? ? ? 0777 > storage.create-directory-mask? ? ? ? ? 0777 > storage.max-hardlinks? ? ? ? ? ? ? ? ? 100 > features.ctime? ? ? ? ? ? ? ? ? ? ? ? ? on > config.gfproxyd? ? ? ? ? ? ? ? ? ? ? ? off > cluster.server-quorum-type? ? ? ? ? ? ? server > cluster.server-quorum-ratio? ? ? ? ? ? 0 > changelog.changelog? ? ? ? ? ? ? ? ? ? off > changelog.changelog-dir? ? ? ? ? ? ? ? {{ brick.path }}/.glusterfs/changelogs > changelog.encoding? ? ? ? ? ? ? ? ? ? ? ascii > changelog.rollover-time? ? ? ? ? ? ? ? 15 > changelog.fsync-interval? ? ? ? ? ? ? ? 5 > changelog.changelog-barrier-timeout? ? 120 > changelog.capture-del-path? ? ? ? ? ? ? off > features.barrier? ? ? ? ? ? ? ? ? ? ? ? disable > features.barrier-timeout? ? ? ? ? ? ? ? 120 > features.trash? ? ? ? ? ? ? ? ? ? ? ? ? off > features.trash-dir? ? ? ? ? ? ? ? ? ? ? .trashcan > features.trash-eliminate-path? ? ? ? ? (null) > features.trash-max-filesize? ? ? ? ? ? 5MB > features.trash-internal-op? ? ? ? ? ? ? off > cluster.enable-shared-storage? ? ? ? ? disable > locks.trace? ? ? ? ? ? ? ? ? ? ? ? ? ? off > locks.mandatory-locking? ? ? ? ? ? ? ? off > cluster.disperse-self-heal-daemon? ? ? enable > cluster.quorum-reads? ? ? ? ? ? ? ? ? ? no > client.bind-insecure? ? ? ? ? ? ? ? ? ? (null) > features.shard? ? ? ? ? ? ? ? ? ? ? ? ? on > features.shard-block-size? ? ? ? ? ? ? 64MB > features.shard-lru-limit? ? ? ? ? ? ? ? 16384 > features.shard-deletion-rate? ? ? ? ? ? 100 > features.scrub-throttle? ? ? ? ? ? ? ? lazy > features.scrub-freq? ? ? ? ? ? ? ? ? ? biweekly > features.scrub? ? ? ? ? ? ? ? ? ? ? ? ? false > features.expiry-time? ? ? ? ? ? ? ? ? ? 120 > features.cache-invalidation? ? ? ? ? ? off > features.cache-invalidation-timeout? ? 60 > features.leases? ? ? ? ? ? ? ? ? ? ? ? off > features.lease-lock-recall-timeout? ? ? 60 > disperse.background-heals? ? ? ? ? ? ? 8 > disperse.heal-wait-qlength? ? ? ? ? ? ? 128 > cluster.heal-timeout? ? ? ? ? ? ? ? ? ? 600 > dht.force-readdirp? ? ? ? ? ? ? ? ? ? ? on > disperse.read-policy? ? ? ? ? ? ? ? ? ? gfid-hash > cluster.shd-max-threads? ? ? ? ? ? ? ? 8 > cluster.shd-wait-qlength? ? ? ? ? ? ? ? 10000 > cluster.shd-wait-qlength? ? ? ? ? ? ? ? 10000 > cluster.locking-scheme? ? ? ? ? ? ? ? ? granular > cluster.granular-entry-heal? ? ? ? ? ? no > features.locks-revocation-secs? ? ? ? ? 0 > features.locks-revocation-clear-all? ? false > features.locks-revocation-max-blocked? 0 > features.locks-monkey-unlocking? ? ? ? false > features.locks-notify-contention? ? ? ? no > features.locks-notify-contention-delay? 5 > disperse.shd-max-threads? ? ? ? ? ? ? ? 1 > disperse.shd-wait-qlength? ? ? ? ? ? ? 1024 > disperse.cpu-extensions? ? ? ? ? ? ? ? auto > disperse.self-heal-window-size? ? ? ? ? 1 > cluster.use-compound-fops? ? ? ? ? ? ? off > performance.parallel-readdir? ? ? ? ? ? off > performance.rda-request-size? ? ? ? ? ? 131072 > performance.rda-low-wmark? ? ? ? ? ? ? 4096 > performance.rda-high-wmark? ? ? ? ? ? ? 128KB > performance.rda-cache-limit? ? ? ? ? ? 10MB > performance.nl-cache-positive-entry? ? false > performance.nl-cache-limit? ? ? ? ? ? ? 10MB > performance.nl-cache-timeout? ? ? ? ? ? 60 > cluster.brick-multiplex? ? ? ? ? ? ? ? off > cluster.max-bricks-per-process? ? ? ? ? 250 > disperse.optimistic-change-log? ? ? ? ? on > disperse.stripe-cache? ? ? ? ? ? ? ? ? 4 > cluster.halo-enabled? ? ? ? ? ? ? ? ? ? False > cluster.halo-shd-max-latency? ? ? ? ? ? 99999 > cluster.halo-nfsd-max-latency? ? ? ? ? 5 > cluster.halo-max-latency? ? ? ? ? ? ? ? 5 > cluster.halo-max-replicas? ? ? ? ? ? ? 99999 > cluster.halo-min-replicas? ? ? ? ? ? ? 2 > features.selinux? ? ? ? ? ? ? ? ? ? ? ? on > cluster.daemon-log-level? ? ? ? ? ? ? ? INFO > debug.delay-gen? ? ? ? ? ? ? ? ? ? ? ? off > delay-gen.delay-percentage? ? ? ? ? ? ? 10% > delay-gen.delay-duration? ? ? ? ? ? ? ? 100000 > delay-gen.enable > disperse.parallel-writes? ? ? ? ? ? ? ? on > features.sdfs? ? ? ? ? ? ? ? ? ? ? ? ? off > features.cloudsync? ? ? ? ? ? ? ? ? ? ? off > features.ctime? ? ? ? ? ? ? ? ? ? ? ? ? on > ctime.noatime? ? ? ? ? ? ? ? ? ? ? ? ? on > feature.cloudsync-storetype? ? ? ? ? ? (null) > features.enforce-mandatory-lock? ? ? ? off > > What do you think, are there any other knobs worth to be turned? > > Thanks! > > On Wed, Jul 03, 2019 at 06:55:09PM +0300, Strahil wrote: > > Check the following link (4.1)? for the optimal gluster volume settings. > > They are quite safe. > > > > Gluster? provides a group called? virt (/var/lib/glusterd/groups/virt)? and can be applied via? 'gluster volume set VOLNAME group virt' > > > > Then try again. > > > > Best Regards, > > Strahil NikolovOn Jul 3, 2019 11:39, Vladimir Melnik wrote: > > > > > > Dear colleagues, > > > > > > I have a lab with a bunch of virtual machines (the virtualization is > > > provided by KVM) running on the same physical host. 4 of these VMs are > > > working as a GlusterFS cluster and there's one more VM that works as a > > > client. I'll specify all the packages' versions in the ending of this > > > message. > > > > > > I created 2 volumes - one is having type "Distributed-Replicate" and > > > another one is "Distribute". The problem is that both of volumes are > > > showing really poor performance. > > > > > > Here's what I see on the client: > > > $ mount | grep gluster > > > 10.13.1.16:storage1 on /mnt/glusterfs1 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > 10.13.1.16:storage2 on /mnt/glusterfs2 type fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s > > > > > > The distributed one shows a bit better performance than the > > > distributed-replicated one, but it's still poor. :-( > > > > > > The disk storage itself is OK, here's what I see on each of 4 GlusterFS > > > servers: > > > for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s > > > > > > The network between all 5 VMs is OK, they all are working on the same > > > physical host. > > > > > > Can't understand, what am I doing wrong. :-( > > > > > > Here's the detailed info about the volumes: > > > Volume Name: storage1 > > > Type: Distributed-Replicate > > > Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32 > > > Status: Started > > > Snapshot Count: 0 > > > Number of Bricks: 2 x (2 + 1) = 6 > > > Transport-type: tcp > > > Bricks: > > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1 > > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2 > > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > > > Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3 > > > Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4 > > > Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter) > > > Options Reconfigured: > > > transport.address-family: inet > > > nfs.disable: on > > > performance.client-io-threads: off > > > > > > Volume Name: storage2 > > > Type: Distribute > > > Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067 > > > Status: Started > > > Snapshot Count: 0 > > > Number of Bricks: 4 > > > Transport-type: tcp > > > Bricks: > > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > Options Reconfigured: > > > transport.address-family: inet > > > nfs.disable: on > > > > > > The OS is CentOS Linux release 7.6.1810. The packages I'm using are: > > > glusterfs-6.3-1.el7.x86_64 > > > glusterfs-api-6.3-1.el7.x86_64 > > > glusterfs-cli-6.3-1.el7.x86_64 > > > glusterfs-client-xlators-6.3-1.el7.x86_64 > > > glusterfs-fuse-6.3-1.el7.x86_64 > > > glusterfs-libs-6.3-1.el7.x86_64 > > > glusterfs-server-6.3-1.el7.x86_64 > > > kernel-3.10.0-327.el7.x86_64 > > > kernel-3.10.0-514.2.2.el7.x86_64 > > > kernel-3.10.0-957.12.1.el7.x86_64 > > > kernel-3.10.0-957.12.2.el7.x86_64 > > > kernel-3.10.0-957.21.3.el7.x86_64 > > > kernel-tools-3.10.0-957.21.3.el7.x86_64 > > > kernel-tools-libs-3.10.0-957.21.3.el7.x86_6 > > > > > > Please, be so kind as to help me to understand, did I do it wrong or > > > that's quite normal performance of GlusterFS? > > > > > > Thanks in advance! > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > -- > V.Melnik > -- V.Melnik From filonov at hkl.hms.harvard.edu Wed Jul 3 19:16:45 2019 From: filonov at hkl.hms.harvard.edu (Dmitry Filonov) Date: Wed, 3 Jul 2019 15:16:45 -0400 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> References: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> Message-ID: Well, if your network is limited to 100MB/s then it doesn't matter if storage is capable of doing 300+MB/s. But 15 MB/s is still way less than 100 MB/s P.S. just tried on my gluster and found out that am getting ~15MB/s on replica 3 volume on SSDs and... 2MB/s on replica 3 volume on HDDs. Something to look at next week. -- Dmitry Filonov Linux Administrator SBGrid Core | Harvard Medical School 250 Longwood Ave, SGM-114 Boston, MA 02115 On Wed, Jul 3, 2019 at 12:18 PM Vladimir Melnik wrote: > Thank you, it helped a little: > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M > count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep > copied > 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s > 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s > 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s > 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s > 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s > > But 14-15 MB/s is still quite far from the actual storage's performance > (200-3000 MB/s). :-( > > Here's full configuration dump (just in case): > > Option Value > ------ ----- > cluster.lookup-unhashed on > cluster.lookup-optimize on > cluster.min-free-disk 10% > cluster.min-free-inodes 5% > cluster.rebalance-stats off > cluster.subvols-per-directory (null) > cluster.readdir-optimize off > cluster.rsync-hash-regex (null) > cluster.extra-hash-regex (null) > cluster.dht-xattr-name trusted.glusterfs.dht > cluster.randomize-hash-range-by-gfid off > cluster.rebal-throttle normal > cluster.lock-migration off > cluster.force-migration off > cluster.local-volume-name (null) > cluster.weighted-rebalance on > cluster.switch-pattern (null) > cluster.entry-change-log on > cluster.read-subvolume (null) > cluster.read-subvolume-index -1 > cluster.read-hash-mode 1 > cluster.background-self-heal-count 8 > cluster.metadata-self-heal off > cluster.data-self-heal off > cluster.entry-self-heal off > cluster.self-heal-daemon on > cluster.heal-timeout 600 > cluster.self-heal-window-size 1 > cluster.data-change-log on > cluster.metadata-change-log on > cluster.data-self-heal-algorithm full > cluster.eager-lock enable > disperse.eager-lock on > disperse.other-eager-lock on > disperse.eager-lock-timeout 1 > disperse.other-eager-lock-timeout 1 > cluster.quorum-type auto > cluster.quorum-count (null) > cluster.choose-local off > cluster.self-heal-readdir-size 1KB > cluster.post-op-delay-secs 1 > cluster.ensure-durability on > cluster.consistent-metadata no > cluster.heal-wait-queue-length 128 > cluster.favorite-child-policy none > cluster.full-lock yes > diagnostics.latency-measurement off > diagnostics.dump-fd-stats off > diagnostics.count-fop-hits off > diagnostics.brick-log-level INFO > diagnostics.client-log-level INFO > diagnostics.brick-sys-log-level CRITICAL > diagnostics.client-sys-log-level CRITICAL > diagnostics.brick-logger (null) > diagnostics.client-logger (null) > diagnostics.brick-log-format (null) > diagnostics.client-log-format (null) > diagnostics.brick-log-buf-size 5 > diagnostics.client-log-buf-size 5 > diagnostics.brick-log-flush-timeout 120 > diagnostics.client-log-flush-timeout 120 > diagnostics.stats-dump-interval 0 > diagnostics.fop-sample-interval 0 > diagnostics.stats-dump-format json > diagnostics.fop-sample-buf-size 65535 > diagnostics.stats-dnscache-ttl-sec 86400 > performance.cache-max-file-size 0 > performance.cache-min-file-size 0 > performance.cache-refresh-timeout 1 > performance.cache-priority > performance.cache-size 32MB > performance.io-thread-count 16 > performance.high-prio-threads 16 > performance.normal-prio-threads 16 > performance.low-prio-threads 32 > performance.least-prio-threads 1 > performance.enable-least-priority on > performance.iot-watchdog-secs (null) > performance.iot-cleanup-disconnected-reqsoff > performance.iot-pass-through false > performance.io-cache-pass-through false > performance.cache-size 128MB > performance.qr-cache-timeout 1 > performance.cache-invalidation false > performance.ctime-invalidation false > performance.flush-behind on > performance.nfs.flush-behind on > performance.write-behind-window-size 1MB > performance.resync-failed-syncs-after-fsyncoff > performance.nfs.write-behind-window-size1MB > performance.strict-o-direct off > performance.nfs.strict-o-direct off > performance.strict-write-ordering off > performance.nfs.strict-write-ordering off > performance.write-behind-trickling-writeson > performance.aggregate-size 128KB > performance.nfs.write-behind-trickling-writeson > performance.lazy-open yes > performance.read-after-open yes > performance.open-behind-pass-through false > performance.read-ahead-page-count 4 > performance.read-ahead-pass-through false > performance.readdir-ahead-pass-through false > performance.md-cache-pass-through false > performance.md-cache-timeout 1 > performance.cache-swift-metadata true > performance.cache-samba-metadata false > performance.cache-capability-xattrs true > performance.cache-ima-xattrs true > performance.md-cache-statfs off > performance.xattr-cache-list > performance.nl-cache-pass-through false > features.encryption off > network.frame-timeout 1800 > network.ping-timeout 42 > network.tcp-window-size (null) > client.ssl off > network.remote-dio enable > client.event-threads 4 > client.tcp-user-timeout 0 > client.keepalive-time 20 > client.keepalive-interval 2 > client.keepalive-count 9 > network.tcp-window-size (null) > network.inode-lru-limit 16384 > auth.allow * > auth.reject (null) > transport.keepalive 1 > server.allow-insecure on > server.root-squash off > server.all-squash off > server.anonuid 65534 > server.anongid 65534 > server.statedump-path /var/run/gluster > server.outstanding-rpc-limit 64 > server.ssl off > auth.ssl-allow * > server.manage-gids off > server.dynamic-auth on > client.send-gids on > server.gid-timeout 300 > server.own-thread (null) > server.event-threads 4 > server.tcp-user-timeout 42 > server.keepalive-time 20 > server.keepalive-interval 2 > server.keepalive-count 9 > transport.listen-backlog 1024 > transport.address-family inet > performance.write-behind on > performance.read-ahead off > performance.readdir-ahead on > performance.io-cache off > performance.open-behind on > performance.quick-read off > performance.nl-cache off > performance.stat-prefetch on > performance.client-io-threads on > performance.nfs.write-behind on > performance.nfs.read-ahead off > performance.nfs.io-cache off > performance.nfs.quick-read off > performance.nfs.stat-prefetch off > performance.nfs.io-threads off > performance.force-readdirp true > performance.cache-invalidation false > performance.global-cache-invalidation true > features.uss off > features.snapshot-directory .snaps > features.show-snapshot-directory off > features.tag-namespaces off > network.compression off > network.compression.window-size -15 > network.compression.mem-level 8 > network.compression.min-size 0 > network.compression.compression-level -1 > network.compression.debug false > features.default-soft-limit 80% > features.soft-timeout 60 > features.hard-timeout 5 > features.alert-time 86400 > features.quota-deem-statfs off > geo-replication.indexing off > geo-replication.indexing off > geo-replication.ignore-pid-check off > geo-replication.ignore-pid-check off > features.quota off > features.inode-quota off > features.bitrot disable > debug.trace off > debug.log-history no > debug.log-file no > debug.exclude-ops (null) > debug.include-ops (null) > debug.error-gen off > debug.error-failure (null) > debug.error-number (null) > debug.random-failure off > debug.error-fops (null) > nfs.disable on > features.read-only off > features.worm off > features.worm-file-level off > features.worm-files-deletable on > features.default-retention-period 120 > features.retention-mode relax > features.auto-commit-period 180 > storage.linux-aio off > storage.batch-fsync-mode reverse-fsync > storage.batch-fsync-delay-usec 0 > storage.owner-uid -1 > storage.owner-gid -1 > storage.node-uuid-pathinfo off > storage.health-check-interval 30 > storage.build-pgfid off > storage.gfid2path on > storage.gfid2path-separator : > storage.reserve 1 > storage.health-check-timeout 10 > storage.fips-mode-rchecksum off > storage.force-create-mode 0000 > storage.force-directory-mode 0000 > storage.create-mask 0777 > storage.create-directory-mask 0777 > storage.max-hardlinks 100 > features.ctime on > config.gfproxyd off > cluster.server-quorum-type server > cluster.server-quorum-ratio 0 > changelog.changelog off > changelog.changelog-dir {{ brick.path > }}/.glusterfs/changelogs > changelog.encoding ascii > changelog.rollover-time 15 > changelog.fsync-interval 5 > changelog.changelog-barrier-timeout 120 > changelog.capture-del-path off > features.barrier disable > features.barrier-timeout 120 > features.trash off > features.trash-dir .trashcan > features.trash-eliminate-path (null) > features.trash-max-filesize 5MB > features.trash-internal-op off > cluster.enable-shared-storage disable > locks.trace off > locks.mandatory-locking off > cluster.disperse-self-heal-daemon enable > cluster.quorum-reads no > client.bind-insecure (null) > features.shard on > features.shard-block-size 64MB > features.shard-lru-limit 16384 > features.shard-deletion-rate 100 > features.scrub-throttle lazy > features.scrub-freq biweekly > features.scrub false > features.expiry-time 120 > features.cache-invalidation off > features.cache-invalidation-timeout 60 > features.leases off > features.lease-lock-recall-timeout 60 > disperse.background-heals 8 > disperse.heal-wait-qlength 128 > cluster.heal-timeout 600 > dht.force-readdirp on > disperse.read-policy gfid-hash > cluster.shd-max-threads 8 > cluster.shd-wait-qlength 10000 > cluster.shd-wait-qlength 10000 > cluster.locking-scheme granular > cluster.granular-entry-heal no > features.locks-revocation-secs 0 > features.locks-revocation-clear-all false > features.locks-revocation-max-blocked 0 > features.locks-monkey-unlocking false > features.locks-notify-contention no > features.locks-notify-contention-delay 5 > disperse.shd-max-threads 1 > disperse.shd-wait-qlength 1024 > disperse.cpu-extensions auto > disperse.self-heal-window-size 1 > cluster.use-compound-fops off > performance.parallel-readdir off > performance.rda-request-size 131072 > performance.rda-low-wmark 4096 > performance.rda-high-wmark 128KB > performance.rda-cache-limit 10MB > performance.nl-cache-positive-entry false > performance.nl-cache-limit 10MB > performance.nl-cache-timeout 60 > cluster.brick-multiplex off > cluster.max-bricks-per-process 250 > disperse.optimistic-change-log on > disperse.stripe-cache 4 > cluster.halo-enabled False > cluster.halo-shd-max-latency 99999 > cluster.halo-nfsd-max-latency 5 > cluster.halo-max-latency 5 > cluster.halo-max-replicas 99999 > cluster.halo-min-replicas 2 > features.selinux on > cluster.daemon-log-level INFO > debug.delay-gen off > delay-gen.delay-percentage 10% > delay-gen.delay-duration 100000 > delay-gen.enable > disperse.parallel-writes on > features.sdfs off > features.cloudsync off > features.ctime on > ctime.noatime on > feature.cloudsync-storetype (null) > features.enforce-mandatory-lock off > > What do you think, are there any other knobs worth to be turned? > > Thanks! > > On Wed, Jul 03, 2019 at 06:55:09PM +0300, Strahil wrote: > > Check the following link (4.1) for the optimal gluster volume settings. > > They are quite safe. > > > > Gluster provides a group called virt (/var/lib/glusterd/groups/virt) > and can be applied via 'gluster volume set VOLNAME group virt' > > > > Then try again. > > > > Best Regards, > > Strahil NikolovOn Jul 3, 2019 11:39, Vladimir Melnik > wrote: > > > > > > Dear colleagues, > > > > > > I have a lab with a bunch of virtual machines (the virtualization is > > > provided by KVM) running on the same physical host. 4 of these VMs are > > > working as a GlusterFS cluster and there's one more VM that works as a > > > client. I'll specify all the packages' versions in the ending of this > > > message. > > > > > > I created 2 volumes - one is having type "Distributed-Replicate" and > > > another one is "Distribute". The problem is that both of volumes are > > > showing really poor performance. > > > > > > Here's what I see on the client: > > > $ mount | grep gluster > > > 10.13.1.16:storage1 on /mnt/glusterfs1 type > fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > 10.13.1.16:storage2 on /mnt/glusterfs2 type > fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp > bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp > bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s > > > > > > The distributed one shows a bit better performance than the > > > distributed-replicated one, but it's still poor. :-( > > > > > > The disk storage itself is OK, here's what I see on each of 4 > GlusterFS > > > servers: > > > for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M > count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s > > > 10+0 records in > > > 10+0 records out > > > 10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s > > > > > > The network between all 5 VMs is OK, they all are working on the same > > > physical host. > > > > > > Can't understand, what am I doing wrong. :-( > > > > > > Here's the detailed info about the volumes: > > > Volume Name: storage1 > > > Type: Distributed-Replicate > > > Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32 > > > Status: Started > > > Snapshot Count: 0 > > > Number of Bricks: 2 x (2 + 1) = 6 > > > Transport-type: tcp > > > Bricks: > > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1 > > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2 > > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter > (arbiter) > > > Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3 > > > Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4 > > > Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter > (arbiter) > > > Options Reconfigured: > > > transport.address-family: inet > > > nfs.disable: on > > > performance.client-io-threads: off > > > > > > Volume Name: storage2 > > > Type: Distribute > > > Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067 > > > Status: Started > > > Snapshot Count: 0 > > > Number of Bricks: 4 > > > Transport-type: tcp > > > Bricks: > > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > Options Reconfigured: > > > transport.address-family: inet > > > nfs.disable: on > > > > > > The OS is CentOS Linux release 7.6.1810. The packages I'm using are: > > > glusterfs-6.3-1.el7.x86_64 > > > glusterfs-api-6.3-1.el7.x86_64 > > > glusterfs-cli-6.3-1.el7.x86_64 > > > glusterfs-client-xlators-6.3-1.el7.x86_64 > > > glusterfs-fuse-6.3-1.el7.x86_64 > > > glusterfs-libs-6.3-1.el7.x86_64 > > > glusterfs-server-6.3-1.el7.x86_64 > > > kernel-3.10.0-327.el7.x86_64 > > > kernel-3.10.0-514.2.2.el7.x86_64 > > > kernel-3.10.0-957.12.1.el7.x86_64 > > > kernel-3.10.0-957.12.2.el7.x86_64 > > > kernel-3.10.0-957.21.3.el7.x86_64 > > > kernel-tools-3.10.0-957.21.3.el7.x86_64 > > > kernel-tools-libs-3.10.0-957.21.3.el7.x86_6 > > > > > > Please, be so kind as to help me to understand, did I do it wrong or > > > that's quite normal performance of GlusterFS? > > > > > > Thanks in advance! > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > -- > V.Melnik > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From csirotic at evoqarchitecture.com Wed Jul 3 19:39:39 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Wed, 3 Jul 2019 15:39:39 -0400 Subject: [Gluster-users] What is the right way to bring down a Glusterfs server for maintenance? References: <901a0448-3c8f-623d-1f5b-b7653623bba8@evoqarchitecture.com> Message-ID: I have a replica 3 cluster, 3 nodes with bricks and 2 "client" nodes, that run the VMs through a mount of the data on the bricks. Now, one of the bricks need maintenance and I will need to shut it down for about 15 minutes. I didn't find any information on what I am suposed to do. If I get this right, I am suposed to remove the brick completely from the cluster and add them again when the maintenance is finished ? Carl From mabi at protonmail.ch Wed Jul 3 19:54:16 2019 From: mabi at protonmail.ch (mabi) Date: Wed, 03 Jul 2019 19:54:16 +0000 Subject: [Gluster-users] GlusterFS FUSE client on BSD Message-ID: Hello, Is there a way to mount a GlusterFS volume using FUSE on an BSD machine such as OpenBSD? If not, what is the alternative, I guess NFS? Regards, M. From jstrunk at redhat.com Wed Jul 3 19:56:49 2019 From: jstrunk at redhat.com (John Strunk) Date: Wed, 3 Jul 2019 15:56:49 -0400 Subject: [Gluster-users] What is the right way to bring down a Glusterfs server for maintenance? In-Reply-To: References: <901a0448-3c8f-623d-1f5b-b7653623bba8@evoqarchitecture.com> Message-ID: Nope. Just: * Ensure all volumes are fully healed so you don't run into split brain * Go ahead and shutdown the server needing maintenance ** If you just want gluster down on that sever node: stop glusterd and kill the glusterfs bricks, then do what you need to do ** If you just want to power off: then shutdown -h as usual (don't worry about stopping gluster) When you bring the server back up, glusterd and the bricks should start, and the bricks should heal from the 2 replicas that remained up during maintenance. -John On Wed, Jul 3, 2019 at 3:48 PM Carl Sirotic wrote: > I have a replica 3 cluster, 3 nodes with bricks and 2 "client" nodes, > that run the VMs through a mount of the data on the bricks. > > Now, one of the bricks need maintenance and I will need to shut it down > for about 15 minutes. > > I didn't find any information on what I am suposed to do. > > If I get this right, I am suposed to remove the brick completely from > the cluster and add them again when the maintenance is finished ? > > > Carl > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From csirotic at evoqarchitecture.com Wed Jul 3 20:07:43 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Wed, 3 Jul 2019 16:07:43 -0400 Subject: [Gluster-users] What is the right way to bring down a Glusterfs server for maintenance? In-Reply-To: References: <901a0448-3c8f-623d-1f5b-b7653623bba8@evoqarchitecture.com> <529e6227-10a9-c458-5b1b-625a2e4ef614@evoqarchitecture.com> Message-ID: Very good, I will give this a try. Thank you. Carl On 2019-07-03 3:56 p.m., John Strunk wrote: > Nope. Just: > * Ensure all volumes are fully healed so you don't run into split brain > * Go ahead and shutdown the server needing maintenance > ** If you just want gluster down on that sever node: stop glusterd and > kill the glusterfs bricks, then do what you need to do > ** If you just want to power off: then shutdown -h as usual (don't > worry about stopping gluster) > > When you bring the server back up, glusterd and the bricks should > start, and the bricks should heal from the 2 replicas that remained up > during maintenance. > > -John > > > On Wed, Jul 3, 2019 at 3:48 PM Carl Sirotic > > > wrote: > > I have a replica 3 cluster, 3 nodes with bricks and 2 "client" nodes, > that run the VMs through a mount of the data on the bricks. > > Now, one of the bricks need maintenance and I will need to shut it > down > for about 15 minutes. > > I didn't find any information on what I am suposed to do. > > If I get this right, I am suposed to remove the brick completely from > the cluster and add them again when the maintenance is finished ? > > > Carl > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.melnik at tucha.ua Wed Jul 3 20:15:08 2019 From: v.melnik at tucha.ua (Vladimir Melnik) Date: Wed, 3 Jul 2019 23:15:08 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: References: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> Message-ID: <20190703201508.GA5499@shagomer.uplink.tucha13.net> Indeed, I wouldn't be surprised if I had around 80-100 MB/s, but 10-15 MB/s is really few. :-( Even when I mount a filesystem on the same GlusterFS node, I have the following result: 10485760 bytes (10 MB) copied, 0.409856 s, 25.6 MB/s 10485760 bytes (10 MB) copied, 0.38967 s, 26.9 MB/s 10485760 bytes (10 MB) copied, 0.466758 s, 22.5 MB/s 10485760 bytes (10 MB) copied, 0.412075 s, 25.4 MB/s 10485760 bytes (10 MB) copied, 0.381626 s, 27.5 MB/s At the same time on the same node when I'm writing directly to the disk: 10485760 bytes (10 MB) copied, 0.0326612 s, 321 MB/s 10485760 bytes (10 MB) copied, 0.0302878 s, 346 MB/s 10485760 bytes (10 MB) copied, 0.0352449 s, 298 MB/s 10485760 bytes (10 MB) copied, 0.0316872 s, 331 MB/s 10485760 bytes (10 MB) copied, 0.0333189 s, 315 MB/s Can't explain it to myself. Are replica-3 volumes really so slow? On Wed, Jul 03, 2019 at 03:16:45PM -0400, Dmitry Filonov wrote: > Well, if your network is limited to 100MB/s then it doesn't matter if > storage is capable of doing 300+MB/s. > But 15 MB/s is still way less than 100 MB/s > > P.S. just tried on my gluster and found out that am getting ~15MB/s on > replica 3 volume on SSDs and... 2MB/s on replica 3 volume on HDDs. > Something to look at next week. > > > > -- > Dmitry Filonov > Linux Administrator > SBGrid Core | Harvard Medical School > 250 Longwood Ave, SGM-114 > Boston, MA 02115 > > > On Wed, Jul 3, 2019 at 12:18 PM Vladimir Melnik wrote: > > > Thank you, it helped a little: > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M > > count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep > > copied > > 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s > > 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s > > 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s > > 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s > > 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s > > > > But 14-15 MB/s is still quite far from the actual storage's performance > > (200-3000 MB/s). :-( > > > > Here's full configuration dump (just in case): > > > > Option Value > > ------ ----- > > cluster.lookup-unhashed on > > cluster.lookup-optimize on > > cluster.min-free-disk 10% > > cluster.min-free-inodes 5% > > cluster.rebalance-stats off > > cluster.subvols-per-directory (null) > > cluster.readdir-optimize off > > cluster.rsync-hash-regex (null) > > cluster.extra-hash-regex (null) > > cluster.dht-xattr-name trusted.glusterfs.dht > > cluster.randomize-hash-range-by-gfid off > > cluster.rebal-throttle normal > > cluster.lock-migration off > > cluster.force-migration off > > cluster.local-volume-name (null) > > cluster.weighted-rebalance on > > cluster.switch-pattern (null) > > cluster.entry-change-log on > > cluster.read-subvolume (null) > > cluster.read-subvolume-index -1 > > cluster.read-hash-mode 1 > > cluster.background-self-heal-count 8 > > cluster.metadata-self-heal off > > cluster.data-self-heal off > > cluster.entry-self-heal off > > cluster.self-heal-daemon on > > cluster.heal-timeout 600 > > cluster.self-heal-window-size 1 > > cluster.data-change-log on > > cluster.metadata-change-log on > > cluster.data-self-heal-algorithm full > > cluster.eager-lock enable > > disperse.eager-lock on > > disperse.other-eager-lock on > > disperse.eager-lock-timeout 1 > > disperse.other-eager-lock-timeout 1 > > cluster.quorum-type auto > > cluster.quorum-count (null) > > cluster.choose-local off > > cluster.self-heal-readdir-size 1KB > > cluster.post-op-delay-secs 1 > > cluster.ensure-durability on > > cluster.consistent-metadata no > > cluster.heal-wait-queue-length 128 > > cluster.favorite-child-policy none > > cluster.full-lock yes > > diagnostics.latency-measurement off > > diagnostics.dump-fd-stats off > > diagnostics.count-fop-hits off > > diagnostics.brick-log-level INFO > > diagnostics.client-log-level INFO > > diagnostics.brick-sys-log-level CRITICAL > > diagnostics.client-sys-log-level CRITICAL > > diagnostics.brick-logger (null) > > diagnostics.client-logger (null) > > diagnostics.brick-log-format (null) > > diagnostics.client-log-format (null) > > diagnostics.brick-log-buf-size 5 > > diagnostics.client-log-buf-size 5 > > diagnostics.brick-log-flush-timeout 120 > > diagnostics.client-log-flush-timeout 120 > > diagnostics.stats-dump-interval 0 > > diagnostics.fop-sample-interval 0 > > diagnostics.stats-dump-format json > > diagnostics.fop-sample-buf-size 65535 > > diagnostics.stats-dnscache-ttl-sec 86400 > > performance.cache-max-file-size 0 > > performance.cache-min-file-size 0 > > performance.cache-refresh-timeout 1 > > performance.cache-priority > > performance.cache-size 32MB > > performance.io-thread-count 16 > > performance.high-prio-threads 16 > > performance.normal-prio-threads 16 > > performance.low-prio-threads 32 > > performance.least-prio-threads 1 > > performance.enable-least-priority on > > performance.iot-watchdog-secs (null) > > performance.iot-cleanup-disconnected-reqsoff > > performance.iot-pass-through false > > performance.io-cache-pass-through false > > performance.cache-size 128MB > > performance.qr-cache-timeout 1 > > performance.cache-invalidation false > > performance.ctime-invalidation false > > performance.flush-behind on > > performance.nfs.flush-behind on > > performance.write-behind-window-size 1MB > > performance.resync-failed-syncs-after-fsyncoff > > performance.nfs.write-behind-window-size1MB > > performance.strict-o-direct off > > performance.nfs.strict-o-direct off > > performance.strict-write-ordering off > > performance.nfs.strict-write-ordering off > > performance.write-behind-trickling-writeson > > performance.aggregate-size 128KB > > performance.nfs.write-behind-trickling-writeson > > performance.lazy-open yes > > performance.read-after-open yes > > performance.open-behind-pass-through false > > performance.read-ahead-page-count 4 > > performance.read-ahead-pass-through false > > performance.readdir-ahead-pass-through false > > performance.md-cache-pass-through false > > performance.md-cache-timeout 1 > > performance.cache-swift-metadata true > > performance.cache-samba-metadata false > > performance.cache-capability-xattrs true > > performance.cache-ima-xattrs true > > performance.md-cache-statfs off > > performance.xattr-cache-list > > performance.nl-cache-pass-through false > > features.encryption off > > network.frame-timeout 1800 > > network.ping-timeout 42 > > network.tcp-window-size (null) > > client.ssl off > > network.remote-dio enable > > client.event-threads 4 > > client.tcp-user-timeout 0 > > client.keepalive-time 20 > > client.keepalive-interval 2 > > client.keepalive-count 9 > > network.tcp-window-size (null) > > network.inode-lru-limit 16384 > > auth.allow * > > auth.reject (null) > > transport.keepalive 1 > > server.allow-insecure on > > server.root-squash off > > server.all-squash off > > server.anonuid 65534 > > server.anongid 65534 > > server.statedump-path /var/run/gluster > > server.outstanding-rpc-limit 64 > > server.ssl off > > auth.ssl-allow * > > server.manage-gids off > > server.dynamic-auth on > > client.send-gids on > > server.gid-timeout 300 > > server.own-thread (null) > > server.event-threads 4 > > server.tcp-user-timeout 42 > > server.keepalive-time 20 > > server.keepalive-interval 2 > > server.keepalive-count 9 > > transport.listen-backlog 1024 > > transport.address-family inet > > performance.write-behind on > > performance.read-ahead off > > performance.readdir-ahead on > > performance.io-cache off > > performance.open-behind on > > performance.quick-read off > > performance.nl-cache off > > performance.stat-prefetch on > > performance.client-io-threads on > > performance.nfs.write-behind on > > performance.nfs.read-ahead off > > performance.nfs.io-cache off > > performance.nfs.quick-read off > > performance.nfs.stat-prefetch off > > performance.nfs.io-threads off > > performance.force-readdirp true > > performance.cache-invalidation false > > performance.global-cache-invalidation true > > features.uss off > > features.snapshot-directory .snaps > > features.show-snapshot-directory off > > features.tag-namespaces off > > network.compression off > > network.compression.window-size -15 > > network.compression.mem-level 8 > > network.compression.min-size 0 > > network.compression.compression-level -1 > > network.compression.debug false > > features.default-soft-limit 80% > > features.soft-timeout 60 > > features.hard-timeout 5 > > features.alert-time 86400 > > features.quota-deem-statfs off > > geo-replication.indexing off > > geo-replication.indexing off > > geo-replication.ignore-pid-check off > > geo-replication.ignore-pid-check off > > features.quota off > > features.inode-quota off > > features.bitrot disable > > debug.trace off > > debug.log-history no > > debug.log-file no > > debug.exclude-ops (null) > > debug.include-ops (null) > > debug.error-gen off > > debug.error-failure (null) > > debug.error-number (null) > > debug.random-failure off > > debug.error-fops (null) > > nfs.disable on > > features.read-only off > > features.worm off > > features.worm-file-level off > > features.worm-files-deletable on > > features.default-retention-period 120 > > features.retention-mode relax > > features.auto-commit-period 180 > > storage.linux-aio off > > storage.batch-fsync-mode reverse-fsync > > storage.batch-fsync-delay-usec 0 > > storage.owner-uid -1 > > storage.owner-gid -1 > > storage.node-uuid-pathinfo off > > storage.health-check-interval 30 > > storage.build-pgfid off > > storage.gfid2path on > > storage.gfid2path-separator : > > storage.reserve 1 > > storage.health-check-timeout 10 > > storage.fips-mode-rchecksum off > > storage.force-create-mode 0000 > > storage.force-directory-mode 0000 > > storage.create-mask 0777 > > storage.create-directory-mask 0777 > > storage.max-hardlinks 100 > > features.ctime on > > config.gfproxyd off > > cluster.server-quorum-type server > > cluster.server-quorum-ratio 0 > > changelog.changelog off > > changelog.changelog-dir {{ brick.path > > }}/.glusterfs/changelogs > > changelog.encoding ascii > > changelog.rollover-time 15 > > changelog.fsync-interval 5 > > changelog.changelog-barrier-timeout 120 > > changelog.capture-del-path off > > features.barrier disable > > features.barrier-timeout 120 > > features.trash off > > features.trash-dir .trashcan > > features.trash-eliminate-path (null) > > features.trash-max-filesize 5MB > > features.trash-internal-op off > > cluster.enable-shared-storage disable > > locks.trace off > > locks.mandatory-locking off > > cluster.disperse-self-heal-daemon enable > > cluster.quorum-reads no > > client.bind-insecure (null) > > features.shard on > > features.shard-block-size 64MB > > features.shard-lru-limit 16384 > > features.shard-deletion-rate 100 > > features.scrub-throttle lazy > > features.scrub-freq biweekly > > features.scrub false > > features.expiry-time 120 > > features.cache-invalidation off > > features.cache-invalidation-timeout 60 > > features.leases off > > features.lease-lock-recall-timeout 60 > > disperse.background-heals 8 > > disperse.heal-wait-qlength 128 > > cluster.heal-timeout 600 > > dht.force-readdirp on > > disperse.read-policy gfid-hash > > cluster.shd-max-threads 8 > > cluster.shd-wait-qlength 10000 > > cluster.shd-wait-qlength 10000 > > cluster.locking-scheme granular > > cluster.granular-entry-heal no > > features.locks-revocation-secs 0 > > features.locks-revocation-clear-all false > > features.locks-revocation-max-blocked 0 > > features.locks-monkey-unlocking false > > features.locks-notify-contention no > > features.locks-notify-contention-delay 5 > > disperse.shd-max-threads 1 > > disperse.shd-wait-qlength 1024 > > disperse.cpu-extensions auto > > disperse.self-heal-window-size 1 > > cluster.use-compound-fops off > > performance.parallel-readdir off > > performance.rda-request-size 131072 > > performance.rda-low-wmark 4096 > > performance.rda-high-wmark 128KB > > performance.rda-cache-limit 10MB > > performance.nl-cache-positive-entry false > > performance.nl-cache-limit 10MB > > performance.nl-cache-timeout 60 > > cluster.brick-multiplex off > > cluster.max-bricks-per-process 250 > > disperse.optimistic-change-log on > > disperse.stripe-cache 4 > > cluster.halo-enabled False > > cluster.halo-shd-max-latency 99999 > > cluster.halo-nfsd-max-latency 5 > > cluster.halo-max-latency 5 > > cluster.halo-max-replicas 99999 > > cluster.halo-min-replicas 2 > > features.selinux on > > cluster.daemon-log-level INFO > > debug.delay-gen off > > delay-gen.delay-percentage 10% > > delay-gen.delay-duration 100000 > > delay-gen.enable > > disperse.parallel-writes on > > features.sdfs off > > features.cloudsync off > > features.ctime on > > ctime.noatime on > > feature.cloudsync-storetype (null) > > features.enforce-mandatory-lock off > > > > What do you think, are there any other knobs worth to be turned? > > > > Thanks! > > > > On Wed, Jul 03, 2019 at 06:55:09PM +0300, Strahil wrote: > > > Check the following link (4.1) for the optimal gluster volume settings. > > > They are quite safe. > > > > > > Gluster provides a group called virt (/var/lib/glusterd/groups/virt) > > and can be applied via 'gluster volume set VOLNAME group virt' > > > > > > Then try again. > > > > > > Best Regards, > > > Strahil NikolovOn Jul 3, 2019 11:39, Vladimir Melnik > > wrote: > > > > > > > > Dear colleagues, > > > > > > > > I have a lab with a bunch of virtual machines (the virtualization is > > > > provided by KVM) running on the same physical host. 4 of these VMs are > > > > working as a GlusterFS cluster and there's one more VM that works as a > > > > client. I'll specify all the packages' versions in the ending of this > > > > message. > > > > > > > > I created 2 volumes - one is having type "Distributed-Replicate" and > > > > another one is "Distribute". The problem is that both of volumes are > > > > showing really poor performance. > > > > > > > > Here's what I see on the client: > > > > $ mount | grep gluster > > > > 10.13.1.16:storage1 on /mnt/glusterfs1 type > > fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > > > 10.13.1.16:storage2 on /mnt/glusterfs2 type > > fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp > > bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s > > > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp > > bs=1M count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s > > > > > > > > The distributed one shows a bit better performance than the > > > > distributed-replicated one, but it's still poor. :-( > > > > > > > > The disk storage itself is OK, here's what I see on each of 4 > > GlusterFS > > > > servers: > > > > for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M > > count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s > > > > 10+0 records in > > > > 10+0 records out > > > > 10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s > > > > > > > > The network between all 5 VMs is OK, they all are working on the same > > > > physical host. > > > > > > > > Can't understand, what am I doing wrong. :-( > > > > > > > > Here's the detailed info about the volumes: > > > > Volume Name: storage1 > > > > Type: Distributed-Replicate > > > > Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32 > > > > Status: Started > > > > Snapshot Count: 0 > > > > Number of Bricks: 2 x (2 + 1) = 6 > > > > Transport-type: tcp > > > > Bricks: > > > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1 > > > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2 > > > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter > > (arbiter) > > > > Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3 > > > > Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4 > > > > Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter > > (arbiter) > > > > Options Reconfigured: > > > > transport.address-family: inet > > > > nfs.disable: on > > > > performance.client-io-threads: off > > > > > > > > Volume Name: storage2 > > > > Type: Distribute > > > > Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067 > > > > Status: Started > > > > Snapshot Count: 0 > > > > Number of Bricks: 4 > > > > Transport-type: tcp > > > > Bricks: > > > > Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > > Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > > Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > > Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2 > > > > Options Reconfigured: > > > > transport.address-family: inet > > > > nfs.disable: on > > > > > > > > The OS is CentOS Linux release 7.6.1810. The packages I'm using are: > > > > glusterfs-6.3-1.el7.x86_64 > > > > glusterfs-api-6.3-1.el7.x86_64 > > > > glusterfs-cli-6.3-1.el7.x86_64 > > > > glusterfs-client-xlators-6.3-1.el7.x86_64 > > > > glusterfs-fuse-6.3-1.el7.x86_64 > > > > glusterfs-libs-6.3-1.el7.x86_64 > > > > glusterfs-server-6.3-1.el7.x86_64 > > > > kernel-3.10.0-327.el7.x86_64 > > > > kernel-3.10.0-514.2.2.el7.x86_64 > > > > kernel-3.10.0-957.12.1.el7.x86_64 > > > > kernel-3.10.0-957.12.2.el7.x86_64 > > > > kernel-3.10.0-957.21.3.el7.x86_64 > > > > kernel-tools-3.10.0-957.21.3.el7.x86_64 > > > > kernel-tools-libs-3.10.0-957.21.3.el7.x86_6 > > > > > > > > Please, be so kind as to help me to understand, did I do it wrong or > > > > that's quite normal performance of GlusterFS? > > > > > > > > Thanks in advance! > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users at gluster.org > > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > > V.Melnik > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users > > -- V.Melnik From lists at localguru.de Wed Jul 3 20:59:09 2019 From: lists at localguru.de (Marcus Schopen) Date: Wed, 03 Jul 2019 22:59:09 +0200 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: References: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> Message-ID: <4d15bd26ae6fceeeb7dbf604eafbf9cb94947a7f.camel@localguru.de> Hi, Am Mittwoch, den 03.07.2019, 15:16 -0400 schrieb Dmitry Filonov: > Well, if your network is limited to 100MB/s then it doesn't matter if > storage is capable of doing 300+MB/s. > But 15 MB/s is still way less than 100 MB/s What network is recommended in the backend, 10 Gigabit or better more? Ciao! Marcus From guy.boisvert at ingtegration.com Wed Jul 3 21:01:37 2019 From: guy.boisvert at ingtegration.com (Guy Boisvert) Date: Wed, 03 Jul 2019 17:01:37 -0400 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: <4d15bd26ae6fceeeb7dbf604eafbf9cb94947a7f.camel@localguru.de> References: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> <4d15bd26ae6fceeeb7dbf604eafbf9cb94947a7f.camel@localguru.de> Message-ID: <99b1a5a3-ccca-47ce-b620-e35241145290@ingtegration.com> Yeah, 10 Gbps is affordable these days, even 25 Gbps!? Wouldn't go lower than 10 Gbps. ?Get BlueMail for Android ? On Jul 3, 2019, 16:59, at 16:59, Marcus Schopen wrote: >Hi, > >Am Mittwoch, den 03.07.2019, 15:16 -0400 schrieb Dmitry Filonov: >> Well, if your network is limited to 100MB/s then it doesn't matter if >> storage is capable of doing 300+MB/s. >> But 15 MB/s is still way less than 100 MB/s > >What network is recommended in the backend, 10 Gigabit or better more? > >Ciao! >Marcus > > >_______________________________________________ >Gluster-users mailing list >Gluster-users at gluster.org >https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.melnik at tucha.ua Wed Jul 3 21:16:40 2019 From: v.melnik at tucha.ua (Vladimir Melnik) Date: Thu, 4 Jul 2019 00:16:40 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: <99b1a5a3-ccca-47ce-b620-e35241145290@ingtegration.com> References: <20190703161808.GQ1375@shagomer.uplink.tucha13.net> <4d15bd26ae6fceeeb7dbf604eafbf9cb94947a7f.camel@localguru.de> <99b1a5a3-ccca-47ce-b620-e35241145290@ingtegration.com> Message-ID: <20190703211640.GA11972@shagomer.uplink.tucha13.net> OK, I tweaked the virtualization parameters and now I have ~10 Gbit/s between all the nodes. $ iperf3 -c 10.13.1.16 Connecting to host 10.13.1.16, port 5201 [ 4] local 10.13.1.17 port 47242 connected to 10.13.1.16 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.42 GBytes 12.2 Gbits/sec 0 1.86 MBytes [ 4] 1.00-2.00 sec 1.54 GBytes 13.3 Gbits/sec 0 2.53 MBytes [ 4] 2.00-3.00 sec 1.37 GBytes 11.8 Gbits/sec 0 2.60 MBytes [ 4] 3.00-4.00 sec 1.25 GBytes 10.7 Gbits/sec 0 2.70 MBytes [ 4] 4.00-5.00 sec 1.30 GBytes 11.1 Gbits/sec 0 2.81 MBytes [ 4] 5.00-6.00 sec 1.55 GBytes 13.3 Gbits/sec 0 2.86 MBytes [ 4] 6.00-7.00 sec 1.46 GBytes 12.6 Gbits/sec 0 2.92 MBytes [ 4] 7.00-8.00 sec 1.41 GBytes 12.1 Gbits/sec 0 2.97 MBytes [ 4] 8.00-9.00 sec 1.39 GBytes 12.0 Gbits/sec 0 2.98 MBytes [ 4] 9.00-10.00 sec 1.46 GBytes 12.5 Gbits/sec 0 3.00 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 14.2 GBytes 12.2 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 14.1 GBytes 12.2 Gbits/sec receiver iperf Done. $ iperf3 -c 10.13.1.16 -R Connecting to host 10.13.1.16, port 5201 Reverse mode, remote host 10.13.1.16 is sending [ 4] local 10.13.1.17 port 47246 connected to 10.13.1.16 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 1.63 GBytes 14.0 Gbits/sec [ 4] 1.00-2.00 sec 1.63 GBytes 14.0 Gbits/sec [ 4] 2.00-3.00 sec 1.56 GBytes 13.4 Gbits/sec [ 4] 3.00-4.00 sec 1.24 GBytes 10.7 Gbits/sec [ 4] 4.00-5.00 sec 1.51 GBytes 13.0 Gbits/sec [ 4] 5.00-6.00 sec 1.40 GBytes 12.0 Gbits/sec [ 4] 6.00-7.00 sec 1.49 GBytes 12.8 Gbits/sec [ 4] 7.00-8.00 sec 1.58 GBytes 13.6 Gbits/sec [ 4] 8.00-9.00 sec 1.45 GBytes 12.4 Gbits/sec [ 4] 9.00-10.00 sec 1.47 GBytes 12.6 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 15.0 GBytes 12.9 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 15.0 GBytes 12.9 Gbits/sec receiver iperf Done. Looks good, right? Let's see the what it has given to us... $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/tmp/test.tmp bs=1M count=10 oflag=sync; rm -f /mnt/tmp/test.tmp; } done 2>&1 | grep copied 10485760 bytes (10 MB) copied, 0.403512 s, 26.0 MB/s 10485760 bytes (10 MB) copied, 0.354702 s, 29.6 MB/s 10485760 bytes (10 MB) copied, 0.386806 s, 27.1 MB/s 10485760 bytes (10 MB) copied, 0.405671 s, 25.8 MB/s 10485760 bytes (10 MB) copied, 0.426986 s, 24.6 MB/s So, the network can do ~10 Gbit/s, the disk can do ~2 Gbit/s, the GlusterFS can do ~0.2 Gbit/s. Am I the only one so lucky? :-) Does anyone else observe the samephenomenon? On Wed, Jul 03, 2019 at 05:01:37PM -0400, Guy Boisvert wrote: > Yeah, 10 Gbps is affordable these days, even 25 Gbps!? Wouldn't go lower than 10 Gbps. > > ?Get BlueMail for Android ? > > On Jul 3, 2019, 16:59, at 16:59, Marcus Schopen wrote: > >Hi, > > > >Am Mittwoch, den 03.07.2019, 15:16 -0400 schrieb Dmitry Filonov: > >> Well, if your network is limited to 100MB/s then it doesn't matter if > >> storage is capable of doing 300+MB/s. > >> But 15 MB/s is still way less than 100 MB/s > > > >What network is recommended in the backend, 10 Gigabit or better more? > > > >Ciao! > >Marcus > > > > > >_______________________________________________ > >Gluster-users mailing list > >Gluster-users at gluster.org > >https://lists.gluster.org/mailman/listinfo/gluster-users From hunter86_bg at yahoo.com Thu Jul 4 04:16:21 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Thu, 04 Jul 2019 07:16:21 +0300 Subject: [Gluster-users] What is the right way to bring down a Glusterfs server for maintenance? Message-ID: It's a replica 3 , right. In my ovirt - I just run poweroff, but you can stop the glusterd.service and run this script: /usr/share/glusterfs/scripts/stop-all-gluster-processes.sh I think plain shutdown is enough. After powering off , you can force a heal. I have created my own functions in .bashrc - 1 for full heal (using sharding here) and the second to check the status: gluster-heal-all() { for i in $(gluster volume list) do gluster volume heal $i full done } gluster-heal-info() { for i in $(gluster volume list) do gluster volume heal $i info summary echo;echo sleep 2 done } P.S.: Even if you don't trigger the heal - gluster has internal mechanism to do it for you. Best Regards, Strahil NikolovOn Jul 3, 2019 22:39, Carl Sirotic wrote: > > I have a replica 3 cluster, 3 nodes with bricks and 2 "client" nodes, > that run the VMs through a mount of the data on the bricks. > > Now, one of the bricks need maintenance and I will need to shut it down > for about 15 minutes. > > I didn't find any information on what I am suposed to do. > > If I get this right, I am suposed to remove the brick completely from > the cluster and add them again when the maintenance is finished ? > > > Carl > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From hunter86_bg at yahoo.com Thu Jul 4 04:18:21 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Thu, 04 Jul 2019 07:18:21 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? Message-ID: I think , it'related to the sync type of oflag. Do you have a raid controller on each brick , to immediate take the data into the cache ? Best Regards, Strahil NikolovOn Jul 3, 2019 23:15, Vladimir Melnik wrote: > > Indeed, I wouldn't be surprised if I had around 80-100 MB/s, but 10-15 > MB/s is really few. :-( > > Even when I mount a filesystem on the same GlusterFS node, I have the > following result: > 10485760 bytes (10 MB) copied, 0.409856 s, 25.6 MB/s > 10485760 bytes (10 MB) copied, 0.38967 s, 26.9 MB/s > 10485760 bytes (10 MB) copied, 0.466758 s, 22.5 MB/s > 10485760 bytes (10 MB) copied, 0.412075 s, 25.4 MB/s > 10485760 bytes (10 MB) copied, 0.381626 s, 27.5 MB/s > > At the same time on the same node when I'm writing directly to the disk: > 10485760 bytes (10 MB) copied, 0.0326612 s, 321 MB/s > 10485760 bytes (10 MB) copied, 0.0302878 s, 346 MB/s > 10485760 bytes (10 MB) copied, 0.0352449 s, 298 MB/s > 10485760 bytes (10 MB) copied, 0.0316872 s, 331 MB/s > 10485760 bytes (10 MB) copied, 0.0333189 s, 315 MB/s > > Can't explain it to myself. Are replica-3 volumes really so slow? > > On Wed, Jul 03, 2019 at 03:16:45PM -0400, Dmitry Filonov wrote: > > Well, if your network is limited to 100MB/s then it doesn't matter if > > storage is capable of doing 300+MB/s. > > But 15 MB/s is still way less than 100 MB/s > > > > P.S. just tried on my gluster and found out that am getting ~15MB/s on > > replica 3 volume on SSDs and... 2MB/s on replica 3 volume on HDDs. > > Something to look at next week. > > > > > > > > -- > > Dmitry Filonov > > Linux Administrator > > SBGrid Core | Harvard Medical School > > 250 Longwood Ave, SGM-114 > > Boston, MA 02115 > > > > > > On Wed, Jul 3, 2019 at 12:18 PM Vladimir Melnik wrote: > > > > > Thank you, it helped a little: > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M > > > count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep > > > copied > > > 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s > > > 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s > > > 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s > > > 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s > > > 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s > > > > > > But 14-15 MB/s is still quite far from the actual storage's performance > > > (200-3000 MB/s). :-( > > > > > > Here's full configuration dump (just in case): > > > > > > Option????????????????????????????????? Value > > > ------????????????????????????????????? ----- > > > cluster.lookup-unhashed???????????????? on > > > cluster.lookup-optimize???????????????? on > > > cluster.min-free-disk?????????????????? 10% > > > cluster.min-free-inodes???????????????? 5% > > > cluster.rebalance-stats???????????????? off > > > cluster.subvols-per-directory?????????? (null) > > > cluster.readdir-optimize??????????????? off > > > cluster.rsync-hash-regex??????????????? (null) > > > cluster.extra-hash-regex??????????????? (null) > > > cluster.dht-xattr-name????????????????? trusted.glusterfs.dht > > > cluster.randomize-hash-range-by-gfid??? off > > > cluster.rebal-throttle????????????????? normal > > > cluster.lock-migration????????????????? off > > > cluster.force-migration???????????????? off > > > cluster.local-volume-name?????????????? (null) > > > cluster.weighted-rebalance????????????? on > > > cluster.switch-pattern????????????????? (null) > > > cluster.entry-change-log??????????????? on > > > cluster.read-subvolume????????????????? (null) > > > cluster.read-subvolume-index??????????? -1 > > > cluster.read-hash-mode????????????????? 1 > > > cluster.background-self-heal-count????? 8 > > > cluster.metadata-self-heal????????????? off > > > cluster.data-self-heal????????????????? off > > > cluster.entry-self-heal???????????????? off > > > cluster.self-heal-daemon??????????????? on > > > cluster.heal-timeout??????????????????? 600 > > > cluster.self-heal-window-size?????????? 1 > > > cluster.data-change-log???????????????? on > > > cluster.metadata-change-log???????????? on > > > cluster.data-self-heal-algorithm??????? full > > > cluster.eager-lock????????????????????? enable > > > disperse.eager-lock???????????????????? on > > > disperse.other-eager-lock?????????????? on > > > disperse.eager-lock-timeout???????????? 1 > > > disperse.other-eager-lock-timeout?????? 1 > > > cluster.quorum-type???????????????????? auto > > > cluster.quorum-count??????????????????? (null) > > > cluster.choose-local??????????????????? off > > > cluster.self-heal-readdir-size????????? 1KB > > > cluster.post-op-delay-secs????????????? 1 > > > cluster.ensure-durability?????????????? on > > > cluster.consistent-metadata???????????? no > > > cluster.heal-wait-queue-length????????? 128 > > > cluster.favorite-child-policy?????????? none > > > cluster.full-lock?????????????????????? yes > > > diagnostics.latency-measurement???????? off > > > diagnostics.dump-fd-stats?????????????? off > > > diagnostics.count-fop-hits????????????? off > > > diagnostics.brick-log-level???????????? INFO > > > diagnostics.client-log-level??????????? INFO > > > diagnostics.brick-sys-log-level???????? CRITICAL > > > diagnostics.client-sys-log-level??????? CRITICAL > > > diagnostics.brick-logger??????????????? (null) > > > diagnostics.client-logger?????????????? (null) > > > diagnostics.brick-log-format??????????? (null) > > > diagnostics.client-log-format?????? From v.melnik at tucha.ua Thu Jul 4 09:28:01 2019 From: v.melnik at tucha.ua (Vladimir Melnik) Date: Thu, 4 Jul 2019 12:28:01 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: References: Message-ID: <20190704092801.GB11972@shagomer.uplink.tucha13.net> All 4 virtual machines working as nodes of the cluster are located on the same physical server. The server has 6 SSD-modules and a RAID-controller with a BBU. RAID level is 10, write-back cache is enabled. Moreover, each node of the GlusterFS cluster shows normal performance when it writes to the disk where the brick resides even with oflag=sync: > 10485760 bytes (10 MB) copied, 0.0326612 s, 321 MB/s > 10485760 bytes (10 MB) copied, 0.0302878 s, 346 MB/s > 10485760 bytes (10 MB) copied, 0.0352449 s, 298 MB/s > 10485760 bytes (10 MB) copied, 0.0316872 s, 331 MB/s > 10485760 bytes (10 MB) copied, 0.0333189 s, 315 MB/s So, the disk is OK and the network is OK, I'm 100% sure. Seems to be a GlusterFS-related issue. Either something needs to be tweaked or it's a normal performance for a replica-3 cluster. On Thu, Jul 04, 2019 at 07:18:21AM +0300, Strahil wrote: > I think , it'related to the sync type of oflag. > Do you have a raid controller on each brick , to immediate take the data into the cache ? > > Best Regards, > Strahil NikolovOn Jul 3, 2019 23:15, Vladimir Melnik wrote: > > > > Indeed, I wouldn't be surprised if I had around 80-100 MB/s, but 10-15 > > MB/s is really few. :-( > > > > Even when I mount a filesystem on the same GlusterFS node, I have the > > following result: > > 10485760 bytes (10 MB) copied, 0.409856 s, 25.6 MB/s > > 10485760 bytes (10 MB) copied, 0.38967 s, 26.9 MB/s > > 10485760 bytes (10 MB) copied, 0.466758 s, 22.5 MB/s > > 10485760 bytes (10 MB) copied, 0.412075 s, 25.4 MB/s > > 10485760 bytes (10 MB) copied, 0.381626 s, 27.5 MB/s > > > > At the same time on the same node when I'm writing directly to the disk: > > 10485760 bytes (10 MB) copied, 0.0326612 s, 321 MB/s > > 10485760 bytes (10 MB) copied, 0.0302878 s, 346 MB/s > > 10485760 bytes (10 MB) copied, 0.0352449 s, 298 MB/s > > 10485760 bytes (10 MB) copied, 0.0316872 s, 331 MB/s > > 10485760 bytes (10 MB) copied, 0.0333189 s, 315 MB/s > > > > Can't explain it to myself. Are replica-3 volumes really so slow? > > > > On Wed, Jul 03, 2019 at 03:16:45PM -0400, Dmitry Filonov wrote: > > > Well, if your network is limited to 100MB/s then it doesn't matter if > > > storage is capable of doing 300+MB/s. > > > But 15 MB/s is still way less than 100 MB/s > > > > > > P.S. just tried on my gluster and found out that am getting ~15MB/s on > > > replica 3 volume on SSDs and... 2MB/s on replica 3 volume on HDDs. > > > Something to look at next week. > > > > > > > > > > > > -- > > > Dmitry Filonov > > > Linux Administrator > > > SBGrid Core | Harvard Medical School > > > 250 Longwood Ave, SGM-114 > > > Boston, MA 02115 > > > > > > > > > On Wed, Jul 3, 2019 at 12:18 PM Vladimir Melnik wrote: > > > > > > > Thank you, it helped a little: > > > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M > > > > count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep > > > > copied > > > > 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s > > > > 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s > > > > 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s > > > > 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s > > > > 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s > > > > > > > > But 14-15 MB/s is still quite far from the actual storage's performance > > > > (200-3000 MB/s). :-( > > > > > > > > Here's full configuration dump (just in case): > > > > > > > > Option????????????????????????????????? Value > > > > ------????????????????????????????????? ----- > > > > cluster.lookup-unhashed???????????????? on > > > > cluster.lookup-optimize???????????????? on > > > > cluster.min-free-disk?????????????????? 10% > > > > cluster.min-free-inodes???????????????? 5% > > > > cluster.rebalance-stats???????????????? off > > > > cluster.subvols-per-directory?????????? (null) > > > > cluster.readdir-optimize??????????????? off > > > > cluster.rsync-hash-regex??????????????? (null) > > > > cluster.extra-hash-regex??????????????? (null) > > > > cluster.dht-xattr-name????????????????? trusted.glusterfs.dht > > > > cluster.randomize-hash-range-by-gfid??? off > > > > cluster.rebal-throttle????????????????? normal > > > > cluster.lock-migration????????????????? off > > > > cluster.force-migration???????????????? off > > > > cluster.local-volume-name?????????????? (null) > > > > cluster.weighted-rebalance????????????? on > > > > cluster.switch-pattern????????????????? (null) > > > > cluster.entry-change-log??????????????? on > > > > cluster.read-subvolume????????????????? (null) > > > > cluster.read-subvolume-index??????????? -1 > > > > cluster.read-hash-mode????????????????? 1 > > > > cluster.background-self-heal-count????? 8 > > > > cluster.metadata-self-heal????????????? off > > > > cluster.data-self-heal????????????????? off > > > > cluster.entry-self-heal???????????????? off > > > > cluster.self-heal-daemon??????????????? on > > > > cluster.heal-timeout??????????????????? 600 > > > > cluster.self-heal-window-size?????????? 1 > > > > cluster.data-change-log???????????????? on > > > > cluster.metadata-change-log???????????? on > > > > cluster.data-self-heal-algorithm??????? full > > > > cluster.eager-lock????????????????????? enable > > > > disperse.eager-lock???????????????????? on > > > > disperse.other-eager-lock?????????????? on > > > > disperse.eager-lock-timeout???????????? 1 > > > > disperse.other-eager-lock-timeout?????? 1 > > > > cluster.quorum-type???????????????????? auto > > > > cluster.quorum-count??????????????????? (null) > > > > cluster.choose-local??????????????????? off > > > > cluster.self-heal-readdir-size????????? 1KB > > > > cluster.post-op-delay-secs????????????? 1 > > > > cluster.ensure-durability?????????????? on > > > > cluster.consistent-metadata???????????? no > > > > cluster.heal-wait-queue-length????????? 128 > > > > cluster.favorite-child-policy?????????? none > > > > cluster.full-lock?????????????????????? yes > > > > diagnostics.latency-measurement???????? off > > > > diagnostics.dump-fd-stats?????????????? off > > > > diagnostics.count-fop-hits????????????? off > > > > diagnostics.brick-log-level???????????? INFO > > > > diagnostics.client-log-level??????????? INFO > > > > diagnostics.brick-sys-log-level???????? CRITICAL > > > > diagnostics.client-sys-log-level??????? CRITICAL > > > > diagnostics.brick-logger??????????????? (null) > > > > diagnostics.client-logger?????????????? (null) > > > > diagnostics.brick-log-format??????????? (null) > > > > diagnostics.client-log-format?????? -- V.Melnik From hunter86_bg at yahoo.com Thu Jul 4 22:09:58 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Fri, 05 Jul 2019 01:09:58 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? Message-ID: So your glusterfs is virtual... I think that Red Hat mention about VMs on gluster (https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/configuring_red_hat_enterprise_virtualization_with_red_hat_gluster_storage/optimizing_virtual_machines_on_red_hat_storage_volumes) , but nothing about virtualized gluster. What tune profile do you use on the gluster cluster ? Can you cycle through high-performance , latency-performance & network-performance (all nodes on same tuned profile before going to the next)? What disk scheduler are you using ? Try with 'noop'/'none' (depends if multique is enabled) on all nodes. Note: 'elevator=none' kernel parameter doesn't work (actually it doesn't exist at all). Use udev rules or manually change it. Best Regards, Strahil NikolovOn Jul 4, 2019 12:28, Vladimir Melnik wrote: > > All 4 virtual machines working as nodes of the cluster are located on > the same physical server. The server has 6 SSD-modules and a > RAID-controller with a BBU. RAID level is 10, write-back cache is > enabled. Moreover, each node of the GlusterFS cluster shows normal > performance when it writes to the disk where the brick resides even with > oflag=sync: > > 10485760 bytes (10 MB) copied, 0.0326612 s, 321 MB/s > > 10485760 bytes (10 MB) copied, 0.0302878 s, 346 MB/s > > 10485760 bytes (10 MB) copied, 0.0352449 s, 298 MB/s > > 10485760 bytes (10 MB) copied, 0.0316872 s, 331 MB/s > > 10485760 bytes (10 MB) copied, 0.0333189 s, 315 MB/s > > So, the disk is OK and the network is OK, I'm 100% sure. > > Seems to be a GlusterFS-related issue. Either something needs to be > tweaked or it's a normal performance for a replica-3 cluster. > > On Thu, Jul 04, 2019 at 07:18:21AM +0300, Strahil wrote: > > I think , it'related to the sync type of oflag. > > Do you have a raid controller on each brick , to immediate take the data into the cache ? > > > > Best Regards, > > Strahil NikolovOn Jul 3, 2019 23:15, Vladimir Melnik wrote: > > > > > > Indeed, I wouldn't be surprised if I had around 80-100 MB/s, but 10-15 > > > MB/s is really few. :-( > > > > > > Even when I mount a filesystem on the same GlusterFS node, I have the > > > following result: > > > 10485760 bytes (10 MB) copied, 0.409856 s, 25.6 MB/s > > > 10485760 bytes (10 MB) copied, 0.38967 s, 26.9 MB/s > > > 10485760 bytes (10 MB) copied, 0.466758 s, 22.5 MB/s > > > 10485760 bytes (10 MB) copied, 0.412075 s, 25.4 MB/s > > > 10485760 bytes (10 MB) copied, 0.381626 s, 27.5 MB/s > > > > > > At the same time on the same node when I'm writing directly to the disk: > > > 10485760 bytes (10 MB) copied, 0.0326612 s, 321 MB/s > > > 10485760 bytes (10 MB) copied, 0.0302878 s, 346 MB/s > > > 10485760 bytes (10 MB) copied, 0.0352449 s, 298 MB/s > > > 10485760 bytes (10 MB) copied, 0.0316872 s, 331 MB/s > > > 10485760 bytes (10 MB) copied, 0.0333189 s, 315 MB/s > > > > > > Can't explain it to myself. Are replica-3 volumes really so slow? > > > > > > On Wed, Jul 03, 2019 at 03:16:45PM -0400, Dmitry Filonov wrote: > > > > Well, if your network is limited to 100MB/s then it doesn't matter if > > > > storage is capable of doing 300+MB/s. > > > > But 15 MB/s is still way less than 100 MB/s > > > > > > > > P.S. just tried on my gluster and found out that am getting ~15MB/s on > > > > replica 3 volume on SSDs and... 2MB/s on replica 3 volume on HDDs. > > > > Something to look at next week. > > > > > > > > > > > > > > > > -- > > > > Dmitry Filonov > > > > Linux Administrator > > > > SBGrid Core | Harvard Medical School > > > > 250 Longwood Ave, SGM-114 > > > > Boston, MA 02115 > > > > > > > > > > > > On Wed, Jul 3, 2019 at 12:18 PM Vladimir Melnik wrote: > > > > > > > > > Thank you, it helped a little: > > > > > > > > > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M > > > > > count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep > > > > > copied > > > > > 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s > > > > > 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s > > > > > 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s > > > > > 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s > > > > > 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s > > > > > > > > > > But 14-15 MB/s is still quite far from the actual storage's performance > > > > > (200-3000 MB/s). :-( > > > > > > > > > > Here's full configuration dump (just in case): > > > > > > > > > > Option????????????????????????????????? Value > > > > > ------????????????????????????????????? ----- > > > > > cluster.lookup-unhashed???????????????? on > > > > > cluster.lookup-optimize???????????????? on > > > > > cluster.min-free-disk?????????????????? 10% > > > > > cluster.min-free-inodes???????????????? 5% > > > > > cluster.rebalance-stats???????????????? off > > > > > cluster.subvols-per-directory?????????? (null) > > > > > cluster.readdir-optimize??????????????? off > > > > > cluster.rsync-hash-regex??????????????? (null) > > > > > cluster.extra-hash-regex??????????????? (null) > > > > > cluster.dht-xattr-name????????????????? trusted.glusterfs.dht > > > > > cluster.randomize-hash-range-by-gfid??? off > > > > > cluster.rebal-throttle????????????????? normal > > > > > cluster.lock-migration????????????????? off > > > > > cluster.force-migration???????????????? off > > > > > cluster.local-volume-name?????????????? (null) > > > > > cluster.weighted-rebalance????????????? on > > > > > cluster.switch-pattern????????????????? (null) > > > > > cluster.entry-change-log??????????????? on > > > > > cluster.read-subvolume????????????????? (null) > > > > > cluster.read-subvolume-index??????????? -1 > > > > > cluster.read-hash-mode????????????????? 1 > > > > > cluster.background-self-heal-count????? 8 > > > > > cluster.metadata-self-heal????????????? off > > > > > cluster.data-self-heal????????????????? off > > > > > cluster.entry-self-heal???????????????? off > > > > > cluster.self-heal-d From nbalacha at redhat.com Fri Jul 5 06:09:52 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Fri, 5 Jul 2019 11:39:52 +0530 Subject: [Gluster-users] Parallel process hang on gluster volume In-Reply-To: <67699782.8645.1562170238208.JavaMail.zimbra@furyweb.fr> References: <1119765369.1998.1561103327489.JavaMail.zimbra@furyweb.fr> <67699782.8645.1562170238208.JavaMail.zimbra@furyweb.fr> Message-ID: Did you see this behaviour with previous Gluster versions? Regards, Nithya On Wed, 3 Jul 2019 at 21:41, wrote: > Am I alone having this problem ? > > ----- Mail original ----- > De: nico at furyweb.fr > ?: "gluster-users" > Envoy?: Vendredi 21 Juin 2019 09:48:47 > Objet: [Gluster-users] Parallel process hang on gluster volume > > I encounterd an issue on production servers using GlusterFS servers 5.1 > and clients 4.1.5 when several process write at the same time on a gluster > volume. > > With more than 48 process writes on the volume at the same time, they are > blocked in D state (uninterruptible sleep), I guess some volume settings > have to be tuned but can't figure out which. > > The client is using op-version 40100 on this volume > Below are volume info, volume settings and ps output on blocked processes. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From nico at furyweb.fr Fri Jul 5 06:30:29 2019 From: nico at furyweb.fr (nico at furyweb.fr) Date: Fri, 5 Jul 2019 08:30:29 +0200 (CEST) Subject: [Gluster-users] Parallel process hang on gluster volume In-Reply-To: References: <1119765369.1998.1561103327489.JavaMail.zimbra@furyweb.fr> <67699782.8645.1562170238208.JavaMail.zimbra@furyweb.fr> Message-ID: <1074995430.9817.1562308229695.JavaMail.zimbra@furyweb.fr> I compared 4.1.5 and 3.12.15, there's no pb with 3.12.15 client. Regards, Nicolas. De: "Nithya Balachandran" ?: nico at furyweb.fr Cc: "gluster-users" Envoy?: Vendredi 5 Juillet 2019 08:09:52 Objet: Re: [Gluster-users] Parallel process hang on gluster volume Did you see this behaviour with previous Gluster versions? Regards, Nithya On Wed, 3 Jul 2019 at 21:41, < [ mailto:nico at furyweb.fr | nico at furyweb.fr ] > wrote: Am I alone having this problem ? ----- Mail original ----- De: [ mailto:nico at furyweb.fr | nico at furyweb.fr ] ?: "gluster-users" < [ mailto:gluster-users at gluster.org | gluster-users at gluster.org ] > Envoy?: Vendredi 21 Juin 2019 09:48:47 Objet: [Gluster-users] Parallel process hang on gluster volume I encounterd an issue on production servers using GlusterFS servers 5.1 and clients 4.1.5 when several process write at the same time on a gluster volume. With more than 48 process writes on the volume at the same time, they are blocked in D state (uninterruptible sleep), I guess some volume settings have to be tuned but can't figure out which. The client is using op-version 40100 on this volume Below are volume info, volume settings and ps output on blocked processes. _______________________________________________ Gluster-users mailing list [ mailto:Gluster-users at gluster.org | Gluster-users at gluster.org ] [ https://lists.gluster.org/mailman/listinfo/gluster-users | https://lists.gluster.org/mailman/listinfo/gluster-users ] _______________________________________________ Gluster-users mailing list [ mailto:Gluster-users at gluster.org | Gluster-users at gluster.org ] [ https://lists.gluster.org/mailman/listinfo/gluster-users | https://lists.gluster.org/mailman/listinfo/gluster-users ] -------------- next part -------------- An HTML attachment was scrubbed... URL: From mr.anteater at gmail.com Fri Jul 5 09:10:05 2019 From: mr.anteater at gmail.com (Adam C) Date: Fri, 5 Jul 2019 10:10:05 +0100 Subject: [Gluster-users] Isolated Client faster than Server+Client Message-ID: Hi, I have been doing some testing of GlusterFS. I have 2 servers running Gluster 6.3 (although same happening in version 3). 1 Server has 32gb ram the other 4gb. Volume is type Replicated. On both servers I also have the volume mounted using the Fuse client and when I run a small of copy 100 x 1kb files it takes about 20 seconds to complete. The strange thing is if I spin up a 3rd server and set it up as a client on the volume the same test completes in 2.7 seconds, almost 7.5 times faster. Are there any known reasons for this behaviour? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pasechny at iskrauraltel.ru Fri Jul 5 11:53:07 2019 From: Pasechny at iskrauraltel.ru (Pasechny Alexey) Date: Fri, 5 Jul 2019 11:53:07 +0000 Subject: [Gluster-users] Geo-replication does not send filesystem changes Message-ID: <538D094457940F4BAA41127FA1578A9A95F648D4@IUT-EXCHANGE.iskrauraltel.local> Hi everyone, I have a problem with native geo-replication setup. It successfully starts, makes initial sync but does not send any filesystem data changes afterward. I'm using CentOS 7.6.1810 with official glusterfs-6.3-1.el7 build on top of ZFS on Linux. It is a single Master node with single brick and the same Slave node. # "gluster vol geo-rep status" command gives the following output MASTER NODE = gfs-alfa1 MASTER VOL = cicd MASTER BRICK = /zdata/cicd/brick SLAVE USER = root SLAVE = gfs-alfa2::cicd SLAVE NODE = gfs-alfa2 STATUS = Active CRAWL STATUS = Changelog Crawl LAST_SYNCED = 2019-07-05 12:08:17 ENTRY = 0 DATA = 0 META = 0 FAILURES = 0 CHECKPOINT TIME = 2019-07-05 12:13:46 CHECKPOINT COMPLETED = No I enabled DEBUG level log for gsyncd.log but did not get any error messages from it. Full log is available here: https://pastebin.com/pXL4dBhZ On both brick I disabled ctime feature because it is incompatible with old versions of gfs clients, enabling this feature does not help too. # gluster volume info Volume Name: cicd Type: Distribute Volume ID: 8f959a35-c7ab-4484-a1e8-9fa8e3a713b4 Status: Started Snapshot Count: 0 Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: gfs-alfa1:/zdata/cicd/brick Options Reconfigured: nfs.disable: on transport.address-family: inet features.ctime: off geo-replication.indexing: on geo-replication.ignore-pid-check: on changelog.changelog: on # gluster volume get cicd rollover-time Option Value ------ ----- changelog.rollover-time 15 # gluster volume get cicd fsync-interval Option Value ------ ----- changelog.fsync-interval 5 Could someone help me with debug of this geo-rep setup? Thank you! BR, Alexey ________________________________ ? ?????? ??????????? ????? ? ?? ???? ?? ????????? ????? ??????????? ?????????? ????????????????? ?????????, ??????????????? ????????????? ?????? ??? ????????. ???? ?? ?? ?????? ???????? ?? ??????????????? ??? ??? ?????????, ??????????, ??????????????? ??????????????? ?? ???? ???????????, ? ???? ????????? ?????????? ??????????. ??????????? ??? ???????????? ????, ?? ??????????? ????????? ?????, ?????? ??????????? ?????????????????? ????????????, ?????????????, ??????????, ??????????????, ?????????? ??? ????????????? ???? ???????? ?????????? ????? ?????. This e-mail and any attachments may contain confidential and/or privileged information and is intended solely for the addressee. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised use, review, retransmissions, dissemination, copying or other use of this information by persons or entities other than the intended recipient is strictly prohibited. From khiremat at redhat.com Fri Jul 5 12:20:14 2019 From: khiremat at redhat.com (Kotresh Hiremath Ravishankar) Date: Fri, 5 Jul 2019 17:50:14 +0530 Subject: [Gluster-users] Geo-replication does not send filesystem changes In-Reply-To: <538D094457940F4BAA41127FA1578A9A95F648D4@IUT-EXCHANGE.iskrauraltel.local> References: <538D094457940F4BAA41127FA1578A9A95F648D4@IUT-EXCHANGE.iskrauraltel.local> Message-ID: The session is moved from "history crawl" to "changelog crawl". After this point, there are no changelogs to be synced as per the logs. Please checking in ".processing" directories if there are any pending changelogs to be synced at "/var/lib/misc/gluster/gsyncd///.processing" If there are no pending changelogs, then please check if the brick is up. On Fri, Jul 5, 2019 at 5:29 PM Pasechny Alexey wrote: > Hi everyone, > > I have a problem with native geo-replication setup. It successfully > starts, makes initial sync but does not send any filesystem data changes > afterward. > I'm using CentOS 7.6.1810 with official glusterfs-6.3-1.el7 build on top > of ZFS on Linux. > It is a single Master node with single brick and the same Slave node. > > # "gluster vol geo-rep status" command gives the following output > MASTER NODE = gfs-alfa1 > MASTER VOL = cicd > MASTER BRICK = /zdata/cicd/brick > SLAVE USER = root > SLAVE = gfs-alfa2::cicd > SLAVE NODE = gfs-alfa2 > STATUS = Active > CRAWL STATUS = Changelog Crawl > LAST_SYNCED = 2019-07-05 12:08:17 > ENTRY = 0 > DATA = 0 > META = 0 > FAILURES = 0 > CHECKPOINT TIME = 2019-07-05 12:13:46 > CHECKPOINT COMPLETED = No > > I enabled DEBUG level log for gsyncd.log but did not get any error > messages from it. Full log is available here: > https://pastebin.com/pXL4dBhZ > On both brick I disabled ctime feature because it is incompatible with old > versions of gfs clients, enabling this feature does not help too. > > # gluster volume info > Volume Name: cicd > Type: Distribute > Volume ID: 8f959a35-c7ab-4484-a1e8-9fa8e3a713b4 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 > Transport-type: tcp > Bricks: > Brick1: gfs-alfa1:/zdata/cicd/brick > Options Reconfigured: > nfs.disable: on > transport.address-family: inet > features.ctime: off > geo-replication.indexing: on > geo-replication.ignore-pid-check: on > changelog.changelog: on > > # gluster volume get cicd rollover-time > Option Value > ------ ----- > changelog.rollover-time 15 > > # gluster volume get cicd fsync-interval > Option Value > ------ ----- > changelog.fsync-interval 5 > > Could someone help me with debug of this geo-rep setup? > Thank you! > > BR, Alexey > > > ________________________________ > > ? ?????? ??????????? ????? ? ?? ???? ?? ????????? ????? ??????????? > ?????????? ????????????????? ?????????, ??????????????? ????????????? > ?????? ??? ????????. ???? ?? ?? ?????? ???????? ?? ??????????????? ??? ??? > ?????????, ??????????, ??????????????? ??????????????? ?? ???? ???????????, > ? ???? ????????? ?????????? ??????????. ??????????? ??? ???????????? ????, > ?? ??????????? ????????? ?????, ?????? ??????????? ?????????????????? > ????????????, ?????????????, ??????????, ??????????????, ?????????? ??? > ????????????? ???? ???????? ?????????? ????? ?????. > > This e-mail and any attachments may contain confidential and/or privileged > information and is intended solely for the addressee. If you are not the > intended recipient (or have received this e-mail in error) please notify > the sender immediately and destroy this e-mail. Any unauthorised use, > review, retransmissions, dissemination, copying or other use of this > information by persons or entities other than the intended recipient is > strictly prohibited. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- Thanks and Regards, Kotresh H R -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pasechny at iskrauraltel.ru Fri Jul 5 15:22:03 2019 From: Pasechny at iskrauraltel.ru (Pasechny Alexey) Date: Fri, 5 Jul 2019 15:22:03 +0000 Subject: [Gluster-users] Geo-replication does not send filesystem changes In-Reply-To: References: <538D094457940F4BAA41127FA1578A9A95F648D4@IUT-EXCHANGE.iskrauraltel.local> Message-ID: <538D094457940F4BAA41127FA1578A9A95F64DB9@IUT-EXCHANGE.iskrauraltel.local> Thank you for reply Kotresh! I found the root of the issue. I started over geo-rep setup and erased the geo-replication.indexing on Master. Replication worked fine if gluster volume is mounted natively or via nfs-ganesha server. But when I tried to make a change on a brick locally it did not go to Slave node. # mount | grep zdata zdata on /zdata type zfs (rw,nosuid,noexec,noatime,xattr,noacl) zdata/cicd on /zdata/cicd type zfs (rw,nosuid,noexec,noatime,xattr,posixacl) Then I mounted glusterfs locally and local changes started to go to Slave node too. BR, Alexey From: Kotresh Hiremath Ravishankar [mailto:khiremat at redhat.com] Sent: Friday, July 05, 2019 5:20 PM To: Pasechny Alexey Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Geo-replication does not send filesystem changes The session is moved from "history crawl" to "changelog crawl". After this point, there are no changelogs to be synced as per the logs. Please checking in ".processing" directories if there are any pending changelogs to be synced at "/var/lib/misc/gluster/gsyncd///.processing" If there are no pending changelogs, then please check if the brick is up. On Fri, Jul 5, 2019 at 5:29 PM Pasechny Alexey > wrote: Hi everyone, I have a problem with native geo-replication setup. It successfully starts, makes initial sync but does not send any filesystem data changes afterward. I'm using CentOS 7.6.1810 with official glusterfs-6.3-1.el7 build on top of ZFS on Linux. It is a single Master node with single brick and the same Slave node. # "gluster vol geo-rep status" command gives the following output MASTER NODE = gfs-alfa1 MASTER VOL = cicd MASTER BRICK = /zdata/cicd/brick SLAVE USER = root SLAVE = gfs-alfa2::cicd SLAVE NODE = gfs-alfa2 STATUS = Active CRAWL STATUS = Changelog Crawl LAST_SYNCED = 2019-07-05 12:08:17 ENTRY = 0 DATA = 0 META = 0 FAILURES = 0 CHECKPOINT TIME = 2019-07-05 12:13:46 CHECKPOINT COMPLETED = No I enabled DEBUG level log for gsyncd.log but did not get any error messages from it. Full log is available here: https://pastebin.com/pXL4dBhZ On both brick I disabled ctime feature because it is incompatible with old versions of gfs clients, enabling this feature does not help too. # gluster volume info Volume Name: cicd Type: Distribute Volume ID: 8f959a35-c7ab-4484-a1e8-9fa8e3a713b4 Status: Started Snapshot Count: 0 Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: gfs-alfa1:/zdata/cicd/brick Options Reconfigured: nfs.disable: on transport.address-family: inet features.ctime: off geo-replication.indexing: on geo-replication.ignore-pid-check: on changelog.changelog: on # gluster volume get cicd rollover-time Option Value ------ ----- changelog.rollover-time 15 # gluster volume get cicd fsync-interval Option Value ------ ----- changelog.fsync-interval 5 Could someone help me with debug of this geo-rep setup? Thank you! BR, Alexey ________________________________ ? ?????? ??????????? ????? ? ?? ???? ?? ????????? ????? ??????????? ?????????? ????????????????? ?????????, ??????????????? ????????????? ?????? ??? ????????. ???? ?? ?? ?????? ???????? ?? ??????????????? ??? ??? ?????????, ??????????, ??????????????? ??????????????? ?? ???? ???????????, ? ???? ????????? ?????????? ??????????. ??????????? ??? ???????????? ????, ?? ??????????? ????????? ?????, ?????? ??????????? ?????????????????? ????????????, ?????????????, ??????????, ??????????????, ?????????? ??? ????????????? ???? ???????? ?????????? ????? ?????. This e-mail and any attachments may contain confidential and/or privileged information and is intended solely for the addressee. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised use, review, retransmissions, dissemination, copying or other use of this information by persons or entities other than the intended recipient is strictly prohibited. _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -- Thanks and Regards, Kotresh H R -------------- next part -------------- An HTML attachment was scrubbed... URL: From wkmail at bneit.com Fri Jul 5 17:17:24 2019 From: wkmail at bneit.com (wkmail) Date: Fri, 5 Jul 2019 10:17:24 -0700 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: <20190704092801.GB11972@shagomer.uplink.tucha13.net> References: <20190704092801.GB11972@shagomer.uplink.tucha13.net> Message-ID: On 7/4/2019 2:28 AM, Vladimir Melnik wrote: > So, the disk is OK and the network is OK, I'm 100% sure. > > Seems to be a GlusterFS-related issue. Either something needs to be > tweaked or it's a normal performance for a replica-3 cluster. There is more to it than Gluster on that particular test. I have some addititional datapoints, since those numbers seemed low given the long time I have played with Gluster (first install was 3.3) So I ran that exact test on some locally mounted hard drive sets (mdadm RAID1- spinning metal) on Cent7 (stock)? and U18(stock) and got the following: No Gluster involved. # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 oflag=sync; rm -f ./test.tmp; } done 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.0144 s, 10.3 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.791071 s, 13.3 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.832186 s, 12.6 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.80427 s, 13.0 MB/s 10+0 records in That was reproducible over several machines with different CPUs that we have in production. Performance is about 20%? better when 7200 rpm drives were involved or when no RAID was involved but never above 18 MB/s Performance is also MUCH better when I use oflag=direct (roughly 2x) However, on a U18 VM Host testbed machine that has a seperate SSD swap disk I get the following, even though I am writing the test.tmp file to the metal. # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 oflag=sync; rm -f ./test.tmp; } done 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0949153 s, 110 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0605883 s, 173 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0582863 s, 180 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0604369 s, 173 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0598746 s, 175 MB/s So something else is going on with that particular test. Clearly, buffers, elevators, cache etc count despite the oflag setting. For the record on the Gluster Fuse Mount (2x+1arb Volume)? on that VM Host I do get reduced performance part of that is due to the gluster network being 2x1G using teaming on that testbed, so there is a network bottleneck. # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 oflag=sync; rm -f ./test.tmp; } done 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.693351 s, 15.1 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.349881 s, 30.0 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.339699 s, 30.9 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.34202 s, 30.7 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.337904 s, 31.0 MB/s So the gluster fuse mount negates the advantage of that SSD swap disk along with the obvious network bottleneck. But clearly we have to all agree on same valid test. -wk From hunter86_bg at yahoo.com Fri Jul 5 19:13:47 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Fri, 05 Jul 2019 22:13:47 +0300 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? Message-ID: I don't know what you are trying to test, but I'm sure this test doesn't show anything meaningful. Have you tested with your apps' workload ? I have done your test and I get aprox 20MB/s, but I can asure you that the performance is way better in my VMs. Best Regards, Strahil NikolovOn Jul 5, 2019 20:17, wkmail wrote: > > > On 7/4/2019 2:28 AM, Vladimir Melnik wrote: > > So, the disk is OK and the network is OK, I'm 100% sure. > > > > Seems to be a GlusterFS-related issue. Either something needs to be > > tweaked or it's a normal performance for a replica-3 cluster. > > There is more to it than Gluster on that particular test. > > I have some addititional datapoints, since those numbers seemed low > given the long time I have played with Gluster (first install was 3.3) > > So I ran that exact test on some locally mounted hard drive sets (mdadm > RAID1- spinning metal) on Cent7 (stock)? and U18(stock) and got the > following: > > No Gluster involved. > > # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 > oflag=sync; rm -f ./test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.0144 s, 10.3 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.791071 s, 13.3 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.832186 s, 12.6 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.80427 s, 13.0 MB/s > 10+0 records in > > That was reproducible over several machines with different CPUs that we > have in production. > > Performance is about 20%? better when 7200 rpm drives were involved or > when no RAID was involved but never above 18 MB/s > > Performance is also MUCH better when I use oflag=direct (roughly 2x) > > However, on a U18 VM Host testbed machine that has a seperate SSD swap > disk I get the following, even though I am writing the test.tmp file to > the metal. > > # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 > oflag=sync; rm -f ./test.tmp; } done > > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0949153 s, 110 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0605883 s, 173 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0582863 s, 180 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0604369 s, 173 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0598746 s, 175 MB/s > > So something else is going on with that particular test. Clearly, > buffers, elevators, cache etc count despite the oflag setting. > > For the record on the Gluster Fuse Mount (2x+1arb Volume)? on that VM > Host I do get reduced performance > > part of that is due to the gluster network being 2x1G using teaming on > that testbed, so there is a network bottleneck. > > # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 > oflag=sync; rm -f ./test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.693351 s, 15.1 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.349881 s, 30.0 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.339699 s, 30.9 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.34202 s, 30.7 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.337904 s, 31.0 MB/s > > So the gluster fuse mount negates the advantage of that SSD swap disk > along with the obvious network bottleneck. > > But clearly we have to all agree on same valid test. > > -wk > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From wkmail at bneit.com Fri Jul 5 19:28:48 2019 From: wkmail at bneit.com (wkmail) Date: Fri, 5 Jul 2019 12:28:48 -0700 Subject: [Gluster-users] Extremely low performance - am I doing somethingwrong? In-Reply-To: References: Message-ID: <58b10f35-84df-153a-ca26-450cf10fdf22@bneit.com> well if you are addressing me, that was the point of my post re the original posters complaint. If his chosen test gets lousy or inconsistent results on non-gluster setups then its hard to complain about gluster absent the known Gluster issues (i.e. network bandwidth, fuse context switching, etc) there is more involved there. and yes, my performance IS better inside the VMs because even though you use the oflag for sync or direct, KVM/Qemu still caches stuff underneath the qcow2 image. So this is hist test on an active Gluster Rep2 + arb KVM setup run within a qcow2 image that is doing real work. # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 oflag=sync; rm -f ./test.tmp; } done 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0206228 s, 508 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0152477 s, 688 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0149008 s, 704 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.014808 s, 708 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0147982 s, 709 MB/s On 7/5/2019 12:13 PM, Strahil wrote: > I don't know what you are trying to test, but I'm sure this test doesn't show anything meaningful. > Have you tested with your apps' workload ? > > I have done your test and I get aprox 20MB/s, but I can asure you that the performance is way better in my VMs. > > Best Regards, > Strahil NikolovOn Jul 5, 2019 20:17, wkmail wrote: >> >> On 7/4/2019 2:28 AM, Vladimir Melnik wrote: >>> So, the disk is OK and the network is OK, I'm 100% sure. >>> >>> Seems to be a GlusterFS-related issue. Either something needs to be >>> tweaked or it's a normal performance for a replica-3 cluster. >> There is more to it than Gluster on that particular test. >> >> I have some addititional datapoints, since those numbers seemed low >> given the long time I have played with Gluster (first install was 3.3) >> >> So I ran that exact test on some locally mounted hard drive sets (mdadm >> RAID1- spinning metal) on Cent7 (stock)? and U18(stock) and got the >> following: >> >> No Gluster involved. >> >> # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 >> oflag=sync; rm -f ./test.tmp; } done >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB) copied, 1.0144 s, 10.3 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB) copied, 0.791071 s, 13.3 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB) copied, 0.832186 s, 12.6 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB) copied, 0.80427 s, 13.0 MB/s >> 10+0 records in >> >> That was reproducible over several machines with different CPUs that we >> have in production. >> >> Performance is about 20%? better when 7200 rpm drives were involved or >> when no RAID was involved but never above 18 MB/s >> >> Performance is also MUCH better when I use oflag=direct (roughly 2x) >> >> However, on a U18 VM Host testbed machine that has a seperate SSD swap >> disk I get the following, even though I am writing the test.tmp file to >> the metal. >> >> # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 >> oflag=sync; rm -f ./test.tmp; } done >> >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.0949153 s, 110 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.0605883 s, 173 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.0582863 s, 180 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.0604369 s, 173 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.0598746 s, 175 MB/s >> >> So something else is going on with that particular test. Clearly, >> buffers, elevators, cache etc count despite the oflag setting. >> >> For the record on the Gluster Fuse Mount (2x+1arb Volume)? on that VM >> Host I do get reduced performance >> >> part of that is due to the gluster network being 2x1G using teaming on >> that testbed, so there is a network bottleneck. >> >> # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 >> oflag=sync; rm -f ./test.tmp; } done >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.693351 s, 15.1 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.349881 s, 30.0 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.339699 s, 30.9 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.34202 s, 30.7 MB/s >> 10+0 records in >> 10+0 records out >> 10485760 bytes (10 MB, 10 MiB) copied, 0.337904 s, 31.0 MB/s >> >> So the gluster fuse mount negates the advantage of that SSD swap disk >> along with the obvious network bottleneck. >> >> But clearly we have to all agree on same valid test. >> >> -wk >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users From aspandey at redhat.com Mon Jul 8 06:52:54 2019 From: aspandey at redhat.com (Ashish Pandey) Date: Mon, 8 Jul 2019 02:52:54 -0400 (EDT) Subject: [Gluster-users] Gluster Community Meeting (APAC friendly hours) Message-ID: <1917478196.26602423.1562568774768.JavaMail.zimbra@redhat.com> The following is a new meeting request: Subject: Gluster Community Meeting (APAC friendly hours) Organizer: "Ashish Pandey" Location: https://bluejeans.com/836554017 Time: Tuesday, July 9, 2019, 11:30:00 AM - 12:30:00 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Invitees: gluster-users at gluster.org; gluster-devel at gluster.org; aspandey at redhat.com *~*~*~*~*~*~*~*~*~* Bridge: https://bluejeans.com/836554017 Minutes meeting: https://hackmd.io/Keo9lk_yRMK24QTEo7qr7g Previous Meeting notes: https://github.com/gluster/community/meetings Flash talk: Amar would like to talk about glusterfs 8.0 and its roadmap. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: meeting.ics Type: text/calendar Size: 3058 bytes Desc: not available URL: From aspandey at redhat.com Tue Jul 9 12:55:01 2019 From: aspandey at redhat.com (Ashish Pandey) Date: Tue, 9 Jul 2019 08:55:01 -0400 (EDT) Subject: [Gluster-users] Gluster Community Meeting : 2019-07-09 In-Reply-To: <781545948.26833790.1562676847855.JavaMail.zimbra@redhat.com> Message-ID: <184783736.26834446.1562676901798.JavaMail.zimbra@redhat.com> Hi All, Today, we had Gluster Community Meeting and the minutes of meeting can be found on following link - https://github.com/gluster/community/blob/master/meetings/2019-07-09-Community_meeting.md --- Ashish -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Tue Jul 9 15:30:58 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Tue, 9 Jul 2019 21:00:58 +0530 Subject: [Gluster-users] [Announcement] Gluster Community Update Message-ID: Hello Gluster community, Today marks a new day in the 26-year history of Red Hat. IBM has finalized its acquisition of Red Hat , which will operate as a distinct unit within IBM moving forward. What does this mean for Red Hat?s contributions to the Gluster project? In short, nothing. Red Hat always has and will continue to be a champion for open source and projects like Gluster. IBM is committed to Red Hat?s independence and role in open source software communities so that we can continue this work without interruption or changes. Our mission, governance, and objectives remain the same. We will continue to execute the existing project roadmap. Red Hat associates will continue to contribute to the upstream in the same ways they have been. And, as always, we will continue to help upstream projects be successful and contribute to welcoming new members and maintaining the project. We will do this together, with the community, as we always have. If you have questions or would like to learn more about today?s news, I encourage you to review the list of materials below. Red Hat CTO Chris Wright will host an online Q&A session in the coming days where you can ask questions you may have about what the acquisition means for Red Hat and our involvement in open source communities. Details will be announced on the Red Hat blog . - Press release - Chris Wright blog - Red Hat and IBM: Accelerating the adoption of open source - FAQ on Red Hat Community Blog Amar Tumballi, Maintainer, Lead, Gluster Community. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcus.pedersen at slu.se Tue Jul 9 11:02:36 2019 From: marcus.pedersen at slu.se (Marcus =?iso-8859-1?Q?Peders=E9n?=) Date: Tue, 9 Jul 2019 13:02:36 +0200 Subject: [Gluster-users] Snapshots inconsistent state In-Reply-To: <20190708092938.GA19321@slu.se> References: <20190708092938.GA19321@slu.se> Message-ID: <20190709110236.GA12450@slu.se> Hi all, I have a problem with snapshots on my gluster system. Running centos7 and gluster 4.1.3. I am running a distributed-replicated system: 2 x (2 + 1) I needed to remove some snapshots so I used: gluster snapshot delete I missed that one of the snapshots were activated. It took a long while before the command finished and it finished with an error. The problem now is that this snaphot is in an inconsistent state. The snapshot is missing on node1 and on the rest of the nodes (node2-6) it is activated and exist. If I try to activate, deactivate or delete on node1 I get the error Snapshot does not exist. The snapshot exists in /run/gluster/snaps But it is not mounted. If I try deactivate or delete on any of the other nodes I get: Pre Validation failed on node1 If I try to activate it on any other node I get: Already activated. How do I solve these issue? Many thanks in advance!! Regards Marcus -- ************************************************** * Marcus Peders?n * * System administrator * ************************************************** * Interbull Centre * * ================ * * Department of Animal Breeding & Genetics ? SLU * * Box 7023, SE-750 07 * * Uppsala, Sweden * ************************************************** * Visiting address: * * Room 55614, Ulls v?g 26, Ultuna * * Uppsala * * Sweden * * * * Tel: +46-(0)18-67 1962 * * * ************************************************** * ISO 9001 Bureau Veritas No SE004561-1 * ************************************************** --- N?r du skickar e-post till SLU s? inneb?r detta att SLU behandlar dina personuppgifter. F?r att l?sa mer om hur detta g?r till, klicka h?r E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here From dave at sherohman.org Wed Jul 10 09:41:06 2019 From: dave at sherohman.org (Dave Sherohman) Date: Wed, 10 Jul 2019 04:41:06 -0500 Subject: [Gluster-users] Removing subvolume from dist/rep volume In-Reply-To: References: <20190625091330.GU19805@sherohman.org> <20190628142454.GX19805@sherohman.org> Message-ID: <20190710094106.7pnq5lv5oemmex4a@sherohman.org> Thanks for bringing in the relevant experts on this one, Nithya. Since my last mail to the list, I've tried - Deleting some old disk image files from the volume, in hopes that the problem chunks might belong to them - Running a "gluster volume heal full" - Stopping the remove-brick operation and then running it again to retry Unfortunately, none of this has had any effect. It didn't even remove a single error from the list of problems. Would it be reasonable to proceed by attempting to identify which disk image file(s) the problem chunks belong to, then shut down the corresponding VMs, copy the files (accepting which ever side of the apparent split-brain gluster chooses for the copy - the VM has been running in this state for 8 months, so it doesn't seem to be causing problems), restart each VM on the new copy, and then, after a suitable period to be sure everything is ok, delete the original image file, taking the problem chunks with it? Or would gluster just throw an error when I try to make the copy and refuse to do it? If that is a viable approach (or even if it isn't, really), how do I determine which actual file these problem chunks belong to? Or should I come at it in a completely different way? Also, if this is a split-brain issue (which I agree it seems to be), why doesn't "heal info" see it? # gluster volume heal palantir info split-brain | grep split Number of entries in split-brain: 0 Number of entries in split-brain: 0 Number of entries in split-brain: 0 Number of entries in split-brain: 0 Number of entries in split-brain: 0 Number of entries in split-brain: 0 Number of entries in split-brain: 0 Number of entries in split-brain: 0 Number of entries in split-brain: 0 On Tue, Jul 02, 2019 at 12:25:02PM +0530, Nithya Balachandran wrote: > Hi Dave, > > Yes, files in split brain are not migrated as we cannot figure out which is > the good copy. Adding Ravi to look at this and see what can be done. > Also adding Krutika as this is a sharded volume. > > The files with the "---------T" permissions are internal files and can be > ignored. Ravi and Krutika, please take a look at the other files. > > Regards, > Nithya > > > On Fri, 28 Jun 2019 at 19:56, Dave Sherohman wrote: > > > On Thu, Jun 27, 2019 at 12:17:10PM +0530, Nithya Balachandran wrote: > > > There are some edge cases that may prevent a file from being migrated > > > during a remove-brick. Please do the following after this: > > > > > > 1. Check the remove-brick status for any failures. If there are any, > > > check the rebalance log file for errors. > > > 2. Even if there are no failures, check the removed bricks to see if > > any > > > files have not been migrated. If there are any, please check that > > they are > > > valid files on the brick and copy them to the volume from the brick > > to the > > > mount point. > > > > Well, looks like I hit one of those edge cases. Probably because of > > some issues around a reboot last September which left a handful of files > > in a state where self-heal identified them as needing to be healed, but > > incapable of actually healing them. (Check the list archives for > > "Kicking a stuck heal", posted on Sept 4, if you want more details.) > > > > So I'm getting 9 failures on the arbiter (merlin), 8 on one data brick > > (gandalf), and 3 on the other (saruman). Looking in > > /var/log/gluster/palantir-rebalance.log, I see those numbers of > > > > migrate file failed: /.shard/291e9749-2d1b-47af-ad53-3a09ad4e64c6.229: > > failed to lock file on palantir-replicate-1 [Stale file handle] > > > > errors. > > > > Also, merlin has four errors, and gandalf has one, of the form: > > > > Gfid mismatch detected for > > /0f500288-ff62-4f0b-9574-53f510b4159f.2898>, > > 9f00c0fe-58c3-457e-a2e6-f6a006d1cfc6 on palantir-client-7 and > > 08bb7cdc-172b-4c21-916a-2a244c095a3e on palantir-client-1. > > > > There are no gfid mismatches recorded on saruman. All of the gfid > > mismatches are for and (on > > saruman) appear to correspond to 0-byte files (e.g., > > .shard/0f500288-ff62-4f0b-9574-53f510b4159f.2898, in the case of the > > gfid mismatch quoted above). > > > > For both types of errors, all affected files are in .shard/ and have > > UUID-style names, so I have no idea which actual files they belong to. > > File sizes are generally either 0 bytes or 4M (exactly), although one of > > them has a size slightly larger than 3M. So I'm assuming they're chunks > > of larger files (which would be almost all the files on the volume - > > it's primarily holding disk image files for kvm servers). > > > > Web searches generally seem to consider gfid mismatches to be a form of > > split-brain, but `gluster volume heal palantir info split-brain` shows > > "Number of entries in split-brain: 0" for all bricks, including those > > bricks which are reporting gfid mismatches. > > > > > > Given all that, how do I proceed with cleaning up the stale handle > > issues? I would guess that this will involve somehow converting the > > shard filename to a "real" filename, then shutting down the > > corresponding VM and maybe doing some additional cleanup. > > > > And then there's the gfid mismatches. Since they're for 0-byte files, > > is it safe to just ignore them on the assumption that they only hold > > metadata? Or do I need to do some kind of split-brain resolution on > > them (even though gluster says no files are in split-brain)? > > > > > > Finally, a listing of /var/local/brick0/data/.shard on saruman, in case > > any of the information it contains (like file sizes/permissions) might > > provide clues to resolving the errors: > > > > --- cut here --- > > root at saruman:/var/local/brick0/data/.shard# ls -l > > total 63996 > > -rw-rw---- 2 root libvirt-qemu 0 Sep 17 2018 > > 0f500288-ff62-4f0b-9574-53f510b4159f.2864 > > -rw-rw---- 2 root libvirt-qemu 0 Sep 17 2018 > > 0f500288-ff62-4f0b-9574-53f510b4159f.2868 > > -rw-rw---- 2 root libvirt-qemu 0 Sep 17 2018 > > 0f500288-ff62-4f0b-9574-53f510b4159f.2879 > > -rw-rw---- 2 root libvirt-qemu 0 Sep 17 2018 > > 0f500288-ff62-4f0b-9574-53f510b4159f.2898 > > -rw------- 2 root libvirt-qemu 4194304 May 17 14:42 > > 291e9749-2d1b-47af-ad53-3a09ad4e64c6.229 > > -rw------- 2 root libvirt-qemu 4194304 Jun 24 09:10 > > 291e9749-2d1b-47af-ad53-3a09ad4e64c6.925 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 26 12:54 > > 2df12cb0-6cf4-44ae-8b0a-4a554791187e.266 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 26 16:30 > > 2df12cb0-6cf4-44ae-8b0a-4a554791187e.820 > > -rw-r--r-- 2 root libvirt-qemu 4194304 Jun 17 20:22 > > 323186b1-6296-4cbe-8275-b940cc9d65cf.27466 > > -rw-r--r-- 2 root libvirt-qemu 4194304 Jun 27 05:01 > > 323186b1-6296-4cbe-8275-b940cc9d65cf.32575 > > -rw-r--r-- 2 root libvirt-qemu 3145728 Jun 11 13:23 > > 323186b1-6296-4cbe-8275-b940cc9d65cf.3448 > > ---------T 2 root libvirt-qemu 0 Jun 28 14:26 > > 4cd094f4-0344-4660-98b0-83249d5bd659.22998 > > -rw------- 2 root libvirt-qemu 4194304 Mar 13 2018 > > 6cdd2e5c-f49e-492b-8039-239e71577836.1302 > > ---------T 2 root libvirt-qemu 0 Jun 28 13:22 > > 7530a2d1-d6ec-4a04-95a2-da1f337ac1ad.47131 > > ---------T 2 root libvirt-qemu 0 Jun 28 13:22 > > 7530a2d1-d6ec-4a04-95a2-da1f337ac1ad.52615 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 27 08:56 > > 8fefae99-ed2a-4a8f-ab87-aa94c6bb6e68.100 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 27 11:29 > > 8fefae99-ed2a-4a8f-ab87-aa94c6bb6e68.106 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 28 02:35 > > 8fefae99-ed2a-4a8f-ab87-aa94c6bb6e68.137 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Nov 4 2018 > > 9544617c-901c-4613-a94b-ccfad4e38af1.165 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Nov 4 2018 > > 9544617c-901c-4613-a94b-ccfad4e38af1.168 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Nov 5 2018 > > 9544617c-901c-4613-a94b-ccfad4e38af1.193 > > -rw-rw-r-- 2 root libvirt-qemu 4194304 Nov 6 2018 > > 9544617c-901c-4613-a94b-ccfad4e38af1.3800 > > ---------T 2 root libvirt-qemu 0 Jun 28 15:02 > > b48a5934-5e5b-4918-8193-6ff36f685f70.46559 > > -rw-rw---- 2 root libvirt-qemu 0 Oct 12 2018 > > c5bde2f2-3361-4d1a-9c88-28751ef74ce6.3568 > > -rw-r--r-- 2 root libvirt-qemu 4194304 Apr 13 2018 > > c953c676-152d-4826-80ff-bd307fa7f6e5.10724 > > -rw-r--r-- 2 root libvirt-qemu 4194304 Apr 11 2018 > > c953c676-152d-4826-80ff-bd307fa7f6e5.3101 > > --- cut here --- > > > > -- > > Dave Sherohman > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users > > -- Dave Sherohman From hgowtham at redhat.com Thu Jul 11 09:09:54 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Thu, 11 Jul 2019 14:39:54 +0530 Subject: [Gluster-users] Announcing Gluster release 6.3 Message-ID: Hi, The Gluster community is pleased to announce the release of Gluster 6.3 (packages available at [1]). Release notes for the release can be found at [2]. Major changes, features and limitations addressed in this release: None Thanks, Gluster community [1] Packages for 6.2: https://download.gluster.org/pub/gluster/glusterfs/6/6.3/ [2] Release notes for 6.2: https://docs.gluster.org/en/latest/release-notes/6.3/ -- Regards, Hari Gowtham. From hunter86_bg at yahoo.com Thu Jul 11 17:45:42 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Thu, 11 Jul 2019 20:45:42 +0300 Subject: [Gluster-users] [ovirt-users] Update 4.2.8 --> 4.3.5 Message-ID: I'm addding gluster-users as I'm not sure if you can go gluster v3 -> v6 directly. Theoretically speaking , there should be no problem - but I don't know if you will observe any issues. @Gluster-users, Can someone share their thoughts about v3 to v6 migration ? Best Regards, Strahil NikolovOn Jul 11, 2019 14:05, Christoph K?hler wrote: > > Hello! > > We have a 4.2.8 environment with some managed gluster-volumes as storage > domains and we want to go up to 4.3.5. > > How is that procedure especially with the gluster nodes in ovirt that > are running 3.12.15? My fear is on the jump to gluster 6. Do the cluster > work if the first node (of three) is upgraded? And what about the > sequence - first the hypervisors or first gluster nodes? > > Is there anyone who had done this? > > Greetings! > Christoph K?hler > _______________________________________________ > Users mailing list -- users at ovirt.org > To unsubscribe send an email to users-leave at ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users at ovirt.org/message/VWFS6YUKA77VP5DWGV7SBYGZREZDJMO7/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From avishwan at redhat.com Tue Jul 16 02:47:53 2019 From: avishwan at redhat.com (Aravinda) Date: Tue, 16 Jul 2019 08:17:53 +0530 Subject: [Gluster-users] Configure geo replication after volume extension In-Reply-To: References: Message-ID: <2381fe580401a71a31a6e1c78ce9a0dd3fa8d85c.camel@redhat.com> On Master run `gluster-georep-sshkey generate` and Geo-rep create command with force option to redistribute the keys from new nodes to all Slave nodes. On Tue, 2019-07-16 at 00:49 +0530, deepu srinivasan wrote: > Hi Users > Aim: I intend to extend my volume which is basically a 1x3 volume > which is in geo-replication > > Steps I did: I stopped the geo-replication and I extended the volume > 1x3 to 2x3 both the master side and slave side. I did the rebalancing > on the master side alone. Restarted the geo-replication and some > error pops. > > What happened: Some error popped out while starting geo-replication. > > How do I do it properly? Any wiki regardin.g it? > Have attached the georeplication error in the mail for reference > > > > -- regards Aravinda From avishwan at redhat.com Tue Jul 16 02:53:31 2019 From: avishwan at redhat.com (Aravinda) Date: Tue, 16 Jul 2019 08:23:31 +0530 Subject: [Gluster-users] Webhook for georep status change In-Reply-To: References: Message-ID: <42aba399f641cf1620124999bca4bf5b004a6b50.camel@redhat.com> `GEOREP_CHECKPOINT_COMPLETED` event/webhook is available when Events API feature is used. Below document is missing that event details, I will add it to the documentation. https://docs.gluster.org/en/latest/Administrator%20Guide/Events%20APIs/ On Mon, 2019-07-15 at 16:04 +0530, deepu srinivasan wrote: > Hi Users > Is there a webhook that can be configured for a change in georep > status change.Like from "checkpoint not completed" to "checkpoint > completed"? -- regards Aravinda From sabose at redhat.com Tue Jul 16 06:48:32 2019 From: sabose at redhat.com (Sahina Bose) Date: Tue, 16 Jul 2019 12:18:32 +0530 Subject: [Gluster-users] [ovirt-users] Update 4.2.8 --> 4.3.5 In-Reply-To: References: Message-ID: On Thu, Jul 11, 2019 at 11:15 PM Strahil wrote: > I'm addding gluster-users as I'm not sure if you can go gluster v3 -> v6 > directly. > > Theoretically speaking , there should be no problem - but I don't know > if you will observe any issues. > > @Gluster-users, > > Can someone share their thoughts about v3 to v6 migration ? > > Best Regards, > Strahil Nikolov > On Jul 11, 2019 14:05, Christoph K?hler wrote: > > Hello! > > We have a > 4.2.8 environment with some managed gluster-volumes as storage > domains > and we want to go up to 4.3.5. > > How is that procedure especially with > the gluster nodes in ovirt that > are running 3.12.15? My fear is on the > jump to gluster 6. Do the cluster > work if the first node (of three) is > upgraded? And what about the > sequence - first the hypervisors or first > gluster nodes? Since this is not a hyperconverged deployment - the gluster servers needs to be upgraded first , and then the client i.e hypervisors. > > Is there anyone who had done this? > > Greetings! > Christoph K?hler > > _______________________________________________ > Users mailing list -- > users at ovirt.org > To unsubscribe send an email to users-leave at ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt > Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List > Archives: > https://lists.ovirt.org/archives/list/users at ovirt.org/message/VWFS6YUKA77VP5DWGV7SBYGZREZDJMO7/ > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Tue Jul 16 06:51:11 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Tue, 16 Jul 2019 12:21:11 +0530 Subject: [Gluster-users] [ovirt-users] Update 4.2.8 --> 4.3.5 In-Reply-To: References: Message-ID: There were some issues with gluster v3.12.x to v5.x, but we haven't heard any major problems with v3.12.x to v6.x version, other than https://bugzilla.redhat.com/show_bug.cgi?id=1727682 which also seems to be all resolved now. Regards, Amar On Thu, Jul 11, 2019 at 11:16 PM Strahil wrote: > I'm addding gluster-users as I'm not sure if you can go gluster v3 -> v6 > directly. > > Theoretically speaking , there should be no problem - but I don't know > if you will observe any issues. > > @Gluster-users, > > Can someone share their thoughts about v3 to v6 migration ? > > Best Regards, > Strahil Nikolov > On Jul 11, 2019 14:05, Christoph K?hler wrote: > > Hello! > > We have a > 4.2.8 environment with some managed gluster-volumes as storage > domains > and we want to go up to 4.3.5. > > How is that procedure especially with > the gluster nodes in ovirt that > are running 3.12.15? My fear is on the > jump to gluster 6. Do the cluster > work if the first node (of three) is > upgraded? And what about the > sequence - first the hypervisors or first > gluster nodes? > > Is there anyone who had done this? > > Greetings! > > Christoph K?hler > _______________________________________________ > Users > mailing list -- users at ovirt.org > To unsubscribe send an email to > users-leave at ovirt.org > Privacy Statement: > https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List > Archives: > https://lists.ovirt.org/archives/list/users at ovirt.org/message/VWFS6YUKA77VP5DWGV7SBYGZREZDJMO7/ > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.danti at assyoma.it Tue Jul 16 09:46:57 2019 From: g.danti at assyoma.it (Gionatan Danti) Date: Tue, 16 Jul 2019 11:46:57 +0200 Subject: [Gluster-users] Graceful gluster server retire/poweroff Message-ID: <03ab8dc3-d9b7-2a68-763b-64f6ef8ac3e4@assyoma.it> Hi list, I have a replica 3 test cluster and I have a question about how clients behave to an host shutdown. If I suddenly switch off one of the gluster server, the connected clients see a ~42s stall in I/O: this is expected, as it is the default client timeout. However, it is possible to *gracefully* shutdown a running gluster server, *without* impacting a running client? Ie: putting the affected server in a "maintenance mode" of sort. Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From avishwan at redhat.com Tue Jul 16 13:00:34 2019 From: avishwan at redhat.com (Aravinda) Date: Tue, 16 Jul 2019 18:30:34 +0530 Subject: [Gluster-users] Configure geo replication after volume extension In-Reply-To: References: <2381fe580401a71a31a6e1c78ce9a0dd3fa8d85c.camel@redhat.com> Message-ID: <21d77d21bf08c00862c6e7c3bb34b363233166f5.camel@redhat.com> Rebalance is not dependent on Geo-replication. New workers will start with Hybrid crawl and syncs the files created by Rebalance. Rebalance on slave side will not affect Geo-replication since it syncs through mount. On Tue, 2019-07-16 at 16:26 +0530, deepu srinivasan wrote: > Thanks a lot, Aravindha. > How should the volume rebalance should be done? Should it be done on > either side i.e master and slave side or only master side? > > > On Tue, Jul 16, 2019 at 8:18 AM Aravinda wrote: > > On Master run `gluster-georep-sshkey generate` and Geo-rep create > > command with force option to redistribute the keys from new nodes > > to > > all Slave nodes. > > > > > > On Tue, 2019-07-16 at 00:49 +0530, deepu srinivasan wrote: > > > Hi Users > > > Aim: I intend to extend my volume which is basically a 1x3 volume > > > which is in geo-replication > > > > > > Steps I did: I stopped the geo-replication and I extended the > > volume > > > 1x3 to 2x3 both the master side and slave side. I did the > > rebalancing > > > on the master side alone. Restarted the geo-replication and some > > > error pops. > > > > > > What happened: Some error popped out while starting geo- > > replication. > > > > > > How do I do it properly? Any wiki regardin.g it? > > > Have attached the georeplication error in the mail for reference > > > > > > > > > > > > -- regards Aravinda From avishwan at redhat.com Tue Jul 16 13:07:06 2019 From: avishwan at redhat.com (Aravinda) Date: Tue, 16 Jul 2019 18:37:06 +0530 Subject: [Gluster-users] Data Rate and Files transfered in Geo Replication In-Reply-To: References: Message-ID: On Tue, 2019-07-16 at 13:24 +0530, deepu srinivasan wrote: > Hi Users > Is there a way to know the stats of geo-replication like the transfer > rate, amount of data and number of the file transferred in Geo- > Replication? Last synced time column in status output represents the time till data is synced to Slave from Master. Sync performance metrics are not yet available(Issue: https://github.com/gluster/glusterfs/issues/515) > Also, can I know the estimated time left for a checkpoint? Checkpoint time - Last synced time(in status output) will provide pending sync. But estimated time left is not yet available. -- regards Aravinda http://aravindavk.in From g.danti at assyoma.it Tue Jul 16 14:32:29 2019 From: g.danti at assyoma.it (Gionatan Danti) Date: Tue, 16 Jul 2019 16:32:29 +0200 Subject: [Gluster-users] Graceful gluster server retire/poweroff In-Reply-To: References: <03ab8dc3-d9b7-2a68-763b-64f6ef8ac3e4@assyoma.it> Message-ID: <9a1c8960-c08f-11d7-b92b-6437035df6f5@assyoma.it> On 16/07/2019 15:27, Ravishankar N wrote: > Yes, if you simply pkill the gluster brick processes of the node before > switching it off, you won't observe the hang on the clients because they > will receive the disconnect notification immediately. But before that, > you would need to check if there are no pending heals etc. You can use > the script [1] which does all these checks in the graceful mode. Hi Ravi, thanks for your reply. I tried killing the glusterfsd process and I confirm that the client does not see any pause, indeed. I was thinking that during the shutdown procedure systemd would kill the processes by itself; which it *does*, but only if the glusterfsd.service is enabled/started. By default, the systemd service installed on CentOS 7 + Gluster 6.0 SIG does not start glusterfsd.service and so, in the shutdown phase, it does not stop it. Thanks very much for your information. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From hunter86_bg at yahoo.com Wed Jul 17 15:20:17 2019 From: hunter86_bg at yahoo.com (Strahil Nikolov) Date: Wed, 17 Jul 2019 15:20:17 +0000 (UTC) Subject: [Gluster-users] Graceful gluster server retire/poweroff In-Reply-To: <9a1c8960-c08f-11d7-b92b-6437035df6f5@assyoma.it> References: <03ab8dc3-d9b7-2a68-763b-64f6ef8ac3e4@assyoma.it> <9a1c8960-c08f-11d7-b92b-6437035df6f5@assyoma.it> Message-ID: <1666501861.1548609.1563376817764@mail.yahoo.com> Hi Ravi, Can you clarify which script do you use ?Usually I rely on oVirt to shutdown the system properly , but sometimes I have used '/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh' which kills everything (including fuse mounts). Best Regards,Strahil Nikolov? ? ???????, 16 ??? 2019 ?., 17:32:44 ?. ???????+3, Gionatan Danti ??????: On 16/07/2019 15:27, Ravishankar N wrote: > Yes, if you simply pkill the gluster brick processes of the node before > switching it off, you won't observe the hang on the clients because they > will receive the disconnect notification immediately. But before that, > you would need to check if there are no pending heals etc. You can use > the script [1] which does all these checks in the graceful mode. Hi Ravi, thanks for your reply. I tried killing the glusterfsd process and I confirm that the client does not see any pause, indeed. I was thinking that during the shutdown procedure systemd would kill the processes by itself; which it *does*, but only if the glusterfsd.service is enabled/started. By default, the systemd service installed on CentOS 7 + Gluster 6.0 SIG does not start glusterfsd.service and so, in the shutdown phase, it does not stop it. Thanks very much for your information. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewb at uvic.ca Wed Jul 17 15:30:54 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Wed, 17 Jul 2019 08:30:54 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr Message-ID: Hello, I've just noticed one brick in my 7 node distribute volume is missing the trusted.glusterfs.dht xattr...? How can I fix this? I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. All of the other nodes are fine, but gluster07 from the list below does not have the attribute. $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . --absolute-names -n trusted.glusterfs.dht -e hex /mnt/raid6-storage/storage" ... gluster05 | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 gluster03 | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 gluster04 | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff gluster06 | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 gluster02 | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 gluster07 | FAILED | rc=1 >> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such attributenon-zero return code gluster01 | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x0000000100000000555555547ffffffd Here are all of the attr's from the brick: [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ # file: /mnt/raid6-storage/storage/ security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 And here is the volume information: [root at gluster07 ~]# gluster volume info storage Volume Name: storage Type: Distribute Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 Status: Started Snapshot Count: 0 Number of Bricks: 7 Transport-type: tcp Bricks: Brick1: 10.0.231.50:/mnt/raid6-storage/storage Brick2: 10.0.231.51:/mnt/raid6-storage/storage Brick3: 10.0.231.52:/mnt/raid6-storage/storage Brick4: 10.0.231.53:/mnt/raid6-storage/storage Brick5: 10.0.231.54:/mnt/raid6-storage/storage Brick6: 10.0.231.55:/mnt/raid6-storage/storage Brick7: 10.0.231.56:/mnt/raid6-storage/storage Options Reconfigured: changelog.changelog: on features.quota-deem-statfs: on features.read-only: off features.inode-quota: on features.quota: on performance.readdir-ahead: on nfs.disable: on geo-replication.indexing: on geo-replication.ignore-pid-check: on transport.address-family: inet Thanks, -Matthew From felix.koelzow at gmx.de Thu Jul 18 07:27:56 2019 From: felix.koelzow at gmx.de (=?UTF-8?Q?Felix_K=c3=b6lzow?=) Date: Thu, 18 Jul 2019 09:27:56 +0200 Subject: [Gluster-users] Share Experience with Productive Gluster Setup <> Message-ID: <25e55f2e-9ef9-f732-7230-9d1605d67804@gmx.de> Dear Gluster-Community, we try to implement a gluster setup in a productive environment. During the development process, we found this nice thread regarding to gluster storage experience: https://forums.overclockers.com.au/threads/glusterfs-800tb-and-growing.1078674/ This thread seems to be in the early retirement. Thus, i try to reanimate this thread to generate a concise gluster real life experience for further readers. Unfortunately, I don't have the given permissions by the admin to post this message, so? I attached the message below. I plan to write news/updates within this thread (link above). Regards Felix >>> Should be me first post <<< Dear Community, thank you very much for such a very informative talk regarding gluster and gluster real life experience. We also plan to build a gluster system and I thought it is maybe a good idea to reanimate this little old thread to unify the experience for further readers. The main idea is to discuss our current setup (this exists only in our minds actually) and? share our experience with the community if we really go ahead with gluster and real hardware. * Requirements: - Current Volume: about 55TB - Future Volume: REALLY unpredictable for several reasons ? - could be 120 TB in 5 years or ? - could be 500 TB in 5 years - Files: ? - well known office files ? - research files ??? - larger text files (> 10GB) ??? - many pictures from experimental investigations (currently 500 GB - 1.5TB) for each experiment ??? - large HDF5 files - Clients: ? - Windows 98,2000,XP,7,10; Linux (Arch, Ubuntu, CentOS), MacOS - Some data must be immediately available in 10 - 20 years (data cannot exclusively moved to tape storage, must available on hot storage) - Use Redhat Gluster Storage for production environment - Currently planed: Backup-Side using CentOS8 + tape storage - timeline: if gluster is realized, it should be productive in end of this year * Support - It is very likely that we are going use Redhat Gluster Storage ? and corresponding support. * Planned Architecture: - Distributed Replicated (Replica 3-Setup) Volume with two 60 TB Bricks per node and 500GB LVM Cache PCIe NVMe, i.e. 120TB Total Storage * Scenario - It is planned that we run 3 nodes with 2 volumes within a distributed replicated setup - Also a dispersed volume could be possible?! - Using LVM Cache - two volumes ? - one volume for office data (almost for pdf, MS Office :-/,...) ? - one volume for research data (see above) - The Gluster Cluster uses a geo-replication to a single node that is ? located in a different building. Thus, we have a asynchronus native ? backup (realized to due snapshots) that is easily to restore. ? Snapshots will be retained for 7 days. - The native backup server and the server for compression/deduplication ? are located next to each other (for no network bottleneck reasons). ? Bacula/Bareos should be used for weekly, monthly and yearly backups. - Yearly and certain monthly backups are archived on tape-storage - Tapes will be stored at two different locations * Hardware: **? Gluster Storage Server:? GEH 4u Supermicro | CSE-847BE1C-R1K28LPB - Mainboard X11DPi-NT - 2x Intel Xeon LGA 3647 Silver, 8 Cores - 96 GB RAM, DDR4 2666 ECC - Adaptec ASR 8805 Raid-Controller - Two smaller enterprise SSD 2.5" for os - 2x Intel Optane ca. 500GB PCIe NMVe for LVM Cache (Software Raid 0) - 4x 10Gbps RJ45 for Network - 24x 6TB Drives (2 Bricks, 12 Drives per Brick, HW-Raid6 per Brick) ** Server for Native Backup (1 Node Geo-Replication) - Mainboard X11DPi-NT - 2x Intel Xeon LGA 3647 Silver, 8 Cores - 96 GB RAM, DDR4 2666 ECC - Adaptec ASR 8805 Raid-Controller (IT-Mode) - Two smaller enterprise SSD 2.5" for os (xfs) - 4x 10Gbps RJ45 for Network - 12x 10 TB Drives (planned for ZFSonLinux, raidz2) - ZFS compression on the fly, no deduplication ** Server for Compressed/Deduplicated Backup - Mainboard X11DPi-NT - 2x Intel Xeon LGA 3647 Silver, 8 Cores - 192 GB RAM, DDR4 2666 ECC (considering ZFS RAM consumption during deduplication) - Adaptec ASR 8805 Raid-Controller (IT-Mode) - Two smaller enterprise SSD 2.5" for os (xfs) - 4x 10Gbps RJ45 for Network - 12x 6 TB Drives (planned for ZFSonLinux, raidz2) ** Tape-Storage - Hardware not defined yet * Software/OS **? Gluster Storage Server - Running RHEL8 and REDHAT Gluster Storage - With Support ** Server for Native Backup - Running CentOS 8 (not released yet) - Running COMMUNITY Gluster ** Server for Compressed/Deduplicated Backup - Running CentOS 8 (not released yet) - Backup using Bareos or Bacula (without Support, but 'personal' training planned) - Deduplication using ZFSonLinux * Volumes ** Linux Clients and Linux Computing Environment - Using FUSE ** Windows Clients / MacOS - CTDB * Questions: 1. Any hints/remarks to the considered hardware? 2. Does anybody realized a comparable setup and is able to share performance results? 3. Is ZFSonLinux really ready for production for this scenario. 4. Is it reliable to mix Gluster Versions, i.e. production system on Redhat Gluster Storage, Native Backup Server on Community Gluster The aim here is really to share experience and we try to document our experience and performance tests if we are really going ahead with gluster. Every remarks/hints are very welcome. I think this is enough for the first post. Feel free to ask! Thanks to image-ascii converter, the shorted setup could look like this. :-/::--./:? Geo-Replication?????? ::--:/-:::/.// ::-????? `.??????? `.`-????? .` `. - ? /:/.- `? ` `` `` ``? /:/`` `` `.:` `:``:`:` ` `......-::::::.-.:::/::-:::::::-----::.-::-:::::::--.::.....` d/h.sy-o/:yo::+-s+/o.hh-++///s/s:/s::s/soo/o`s:+s-o+: ? s+s/? Production System? Building A???? y-h:os :.??????????????????????? ----://--:-: `..???? s/o-+o???? Backup System Building B o--/:yy/ ? -/:`/-/:...//-sh.//? :` .-`//`/:-::::.:--: /` `-? Master???????????????? ..`----`.-.-????????????? Slave :????? s/o-+o:++-/o-+o-h/+--+:o/.////`o-::o o///+`//o--/:yy/ ??????????????? `` .. .``?? `??????????????????????????????????????? `.` : ..`+:/+/+//+:`??????????????????????????????????? ://:+:-+: : .. ``````````???????????????????????????????????? ````````` : ?????? ``???????? ` `???? `?????????????????? `` ..???????????????????????????????????? ` ` ```` `? ``?????? ` : ?????? //-//:::::`::o-::::/::???? : ::-.::::` +/.:--:. / ..???????????????????????????????????? /`::/:+-+/.-:/:::-v ?? \ : / :-::-:-:.--.`+:+-:--::: ?????? `.`.``.``` ````.```...???? -...``.-.`? ...-.... . ..???????????????????????????????????? .`.`.`....````...`.` ` \|/?????????????????????????? .`.-./..-.`. ..-.-`--.- .`````````````````````````-:`````````````````````````. ..`````````````````````````/`````````````````````````- ??????????????????????????????? -``` ..?????????????????????? ..-???????? .... : ..- ??????????????????????????????? -``` ..?????????????????????? ..-???????? .``` : ``- ??????????????? ` ` -``.``````````````````````--``````````````````````.``- ..`-```````````````````````/``````````````````````..`- ?????????????? `+:+:::-/--????? -? . ..????????????????????? .? -???????? .` . :????????????????????? `. - ??????????????? ....``.-`.????? -? . ..????????????????????? .? -???????? .` . -h:???????????????????? `. - ??????????????????????????????? -? . ..????????????????????? .? -???????? .` :-:``:-::-:.:----------:o:-:.:------:---::--`-::. - ` ??????????????????????????????? -? . ..????????????????????? .? -???????? .` //+ `..--:`:------..-------`:-------.---.`` ./+` - `/-::---- ``````````````.` ??????????????????????????????? -? . ..????????????????????? .? -???????? .` //+ .:`?????????????????? ./+` -? .....-.. /`````````````:--` ???????????? `????????????????? - /--`.------.-------.---:/---.-------.------.`--/? -???????? .` //+??????????????????? +s-?????????????????? ./+` -?????????????????? :???????????? -` .-` ???????????? o-:--./-//:-/.-:?? -? +/: ````````````````.+o.`. .````````````?? :/+? -???????? .` //+ ``` ```````````````` ```````` ``???? ./+` - :???????????? -`?? .- ???????????? -.``...`........?? -? +//?? `....? Gluster Storage Node 1.`....`?? :/+? -???????? .` //+ --://+.///////::///////.+///////-///:-. ./+` - :???????????? -.??? `-- ??????????????????????????????? - :..`.-..............-..::..-`.............-.`..:? -???????? .` //+? +////+./?? Native Backup Server? /////: ./+` -?????????????????? :??? ./:..-:-.-//-...../ ??????????????????????????????? -? . ..????????????????????? .? -???????? .` //+ -:///+.///////::///////.+///////-////:. ./+` - :??? ..-::----:/-`???? : ??????????????????????????????? -? . ..????????????????????? .? -???????? .` //+???? `.. .......``....... ........`..`??? ./+` - +-...-....-.-.....``-..+ ??????????????? ``????????????? - /--`.--------------.---:/--:.--------------.`--/? -???????? .` -..`````````````````````:.```````````````````..-. -???????????????? ``+------/:------/-.:---/y ??????????????? /-/:/:/--/`???? -? +/:???? ``````````` ``.oo.`` ````````````??? :/+? -???????? .` . -`???????? ``..........--.:..................`: ``..`-`??????? : ??????????????? ` ``...:`.`???? -? +/:? ``....? Gluster Storage Node 2......``? :/+? -???????? .` . .-.........``????????? `. -?????????????????? : --:.::`??????? : ??????????????????????????????? - :..``..................::...`...............`..:? -???????? .` .?????????????????????? `-???????????????????? `. -?????????????????? o...`.- `...`...````.`./ ?? ................???????????? -? . ..????????????????????? .? -???????? .` . `-???????????????????? `. - s.://-/-+-+::./:/.::./-s ?? :???????????? .:-.?????????? -? . ..????????????????????? .? -???????? .` . +y`??????????????????? `. -?????????????????? : `???????????? ./ ?? :???????????? .- `-.???????? - /--`.---::--:::::::.::::/:::.:::::::--::---.`--/? -???????? .` ::/`.:::::/.::::::::::::/o::./:::::::-:::::-`-:/. -?????????????????? :????????????????????? : ?? :???????????? .-?? `-.?????? -? +/:??? ```````````` ```++``` ````````````??? :/+? -???????? .` //+ ``.---`-------..-------`--------`--.``? ./+` - :????????????????????? : ?? :???????????? .-??? `.-.???? -? +/:? ``...... Gluster Storage Node 3.....``? :/+? -???????? .` //+ -/`?????????????????? ./+` - /``````````````````````/ ?? :????????????? ......../???? - :..``..............`........`..............``..:? -???????? .` //+??????????????????? /o.?????????????????? ./+` - ` `???? ` `?????? ```````````````````````` ?? :????????????????????? :???? - .????????????????????????????????????????????? .? -???????? .` //+??? ``.. .......``....... ........`..`??? ./+` - :-//-:/:::: ?? :????????????????????? :???? - .????????????????????????????????????????????? .? -???????? .` //+? -:://+.///////::///////.+///////-//:::. ./+` - ` `.``.````?????? :.............--` ?? :????????????????????? :? `+s/ .????????????????????????????????????????????? .? -???????? .` //+? +////+???? Backup - Server??? //-/////: ./+` -?????????????????? :???????????? -.-.` ?? :? ..` ``````````````? :..`--- .????????????????????????????????????????????? .? -???????? .` //+? -::// Compression / Deduplication? ::-. ./+` -?????????????????? :???????????? -` `-. ?? :? :/::/::/::::::.:::? :???? - .????????????????????????????????????????????? .? -???????? .` //+`````...`................`........`...````-/+` -?????????????????? :???????????? -`?? `-. ?? :? `-..-.`..```````.`? :???? - .????????????????????????????????????????????? .? -???????? .` -..`````````````````````:`````````````````````... -?????????????????? :???????????? -.`````.-- ?? : --/-://`:-/:::/:/:/` :???? - .????????????????????????????????????????????? .? -???????? .` .?????????????????????? :????????????????????? `. -?????????????????? :????????????? ````````/ ?? :? ```..` `..`.`.````? :???? - .????????????????????????????????????????????? .? -???????? .` .????????????????????? :s.???????????????????? `. -?????????????????? :??????? `???????????? : ?? :??? //::::-/::::.:??? :???? - .????????????????????????????????????????????? .? -???????? .` .???????????? -/++++/. .+ `-/++++:`??????????? `. -?????????????????? :?????? `/::::::?????? : ?? :??? .``````.-:```???? :???? - .????????????????????????????????????????????? .? -???????? .` .?????????? `s/:o?? .+o` -s/+:? `-o/?????????? `. - ````````````:???? Tape Storage???? : ?? :????????????????????? :???? - .????????????????????????????????????????????? .? -???????? .` .?????????? y:? :++```+o d.? ++/ `.y:/s........--./........``````.````:???? :/:-::./-::/` : ?? :?? `????????????????? :???? - .????????????????????????????????????????????? .? -???????? .` .?????????? h.? /sy--//y`d?? soh`//o+.:??????? `. -?????????????????? :?????? Building B???? : ?? :? .:/-:-/-+-/./::::.? :???? - .????????????????????????????????????????????? .? -???????? .` .?????????? :s..s`?? .y- o+`-+`?? -y`????????? `. -?????????????????? :????????????????????? : ?? :? `-.`...`.........`? :???? - .????????????????????????????????????????????? .? -???????? .` .??????????? .+o+::/ods+++hys/-:/o+`?????????? `. -?????????????????? :????????????????????? : ?? :? `++-:--::-./:-:-/`? : -``-``````````````````````````````````````````````-``- ..`-```````````````-::.````````.-::.``````````````..`: :????????????????????? : ?? :?? ....:-...--....-?? :???? - `??????????????????????????????????????????????? ``- .``` ``-?????????????????? :????????????????????? : ?? :???? `:./:: .//:-???? : -...?????????????????????????????????????????????? --- .... ..:?????????????????? :????????????????????? : ?? /......:/::/..::::...../ -````````````````````````````````````````````````````- ..```````````````````````````````````````````````````- /....................../ From sdeepugd at gmail.com Mon Jul 1 10:49:45 2019 From: sdeepugd at gmail.com (deepu srinivasan) Date: Mon, 01 Jul 2019 10:49:45 -0000 Subject: [Gluster-users] Exception in Geo-Replication Message-ID: Hi I am getting this exception while starting geo-replication. Please help. [2019-07-01 10:48:02.445475] E [repce(agent /home/sas/gluster/data/code-misc):122:worker] : call failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 41, in register return Changes.cl_register(cl_brick, cl_dir, cl_log, cl_level, retries) File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 45, in cl_register cls.raise_changelog_err() File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 29, in raise_changelog_err raise ChangelogException(errn, os.strerror(errn)) ChangelogException: [Errno 21] Is a directory [2019-07-01 10:48:02.446341] E [repce(worker /home/sas/gluster/data/code-misc):214:__call__] RepceClient: call failed call=31023:140523296659264:1561978082.44 method=register error=ChangelogException [2019-07-01 10:48:02.446654] E [resource(worker /home/sas/gluster/data/code-misc):1268:service_loop] GLUSTER: Changelog register failed error=[Errno 21] Is a directory -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdeepugd at gmail.com Tue Jul 2 07:03:12 2019 From: sdeepugd at gmail.com (deepu srinivasan) Date: Tue, 02 Jul 2019 07:03:12 -0000 Subject: [Gluster-users] Exception in Geo-Replication In-Reply-To: References: Message-ID: Any Update on this issue ? On Mon, Jul 1, 2019 at 4:19 PM deepu srinivasan wrote: > Hi > I am getting this exception while starting geo-replication. Please help. > > [2019-07-01 10:48:02.445475] E [repce(agent > /home/sas/gluster/data/code-misc):122:worker] : call failed: > > Traceback (most recent call last): > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in > worker > > res = getattr(self.obj, rmeth)(*in_data[2:]) > > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", > line 41, in register > > return Changes.cl_register(cl_brick, cl_dir, cl_log, cl_level, > retries) > > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", > line 45, in cl_register > > cls.raise_changelog_err() > > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", > line 29, in raise_changelog_err > > raise ChangelogException(errn, os.strerror(errn)) > > ChangelogException: [Errno 21] Is a directory > > [2019-07-01 10:48:02.446341] E [repce(worker > /home/sas/gluster/data/code-misc):214:__call__] RepceClient: call failed > call=31023:140523296659264:1561978082.44 method=register > error=ChangelogException > > [2019-07-01 10:48:02.446654] E [resource(worker > /home/sas/gluster/data/code-misc):1268:service_loop] GLUSTER: Changelog > register failed error=[Errno 21] Is a directory > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdeepugd at gmail.com Tue Jul 2 07:53:45 2019 From: sdeepugd at gmail.com (deepu srinivasan) Date: Tue, 02 Jul 2019 07:53:45 -0000 Subject: [Gluster-users] Exception in Geo-Replication In-Reply-To: References: Message-ID: Thank you kotresh On Tue, Jul 2, 2019 at 12:38 PM Kotresh Hiremath Ravishankar < khiremat at redhat.com> wrote: > You should be looking into the other log file (changes-.log) > for actual failure. > In your case "changes-home-sas-gluster-data-code-misc.log" > > On Tue, Jul 2, 2019 at 12:33 PM deepu srinivasan > wrote: > >> Any Update on this issue ? >> >> On Mon, Jul 1, 2019 at 4:19 PM deepu srinivasan >> wrote: >> >>> Hi >>> I am getting this exception while starting geo-replication. Please help. >>> >>> [2019-07-01 10:48:02.445475] E [repce(agent >>> /home/sas/gluster/data/code-misc):122:worker] : call failed: >>> >>> Traceback (most recent call last): >>> >>> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, >>> in worker >>> >>> res = getattr(self.obj, rmeth)(*in_data[2:]) >>> >>> File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", >>> line 41, in register >>> >>> return Changes.cl_register(cl_brick, cl_dir, cl_log, cl_level, >>> retries) >>> >>> File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", >>> line 45, in cl_register >>> >>> cls.raise_changelog_err() >>> >>> File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", >>> line 29, in raise_changelog_err >>> >>> raise ChangelogException(errn, os.strerror(errn)) >>> >>> ChangelogException: [Errno 21] Is a directory >>> >>> [2019-07-01 10:48:02.446341] E [repce(worker >>> /home/sas/gluster/data/code-misc):214:__call__] RepceClient: call failed >>> call=31023:140523296659264:1561978082.44 method=register >>> error=ChangelogException >>> >>> [2019-07-01 10:48:02.446654] E [resource(worker >>> /home/sas/gluster/data/code-misc):1268:service_loop] GLUSTER: Changelog >>> register failed error=[Errno 21] Is a directory >>> >> > > -- > Thanks and Regards, > Kotresh H R > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hawk at tbi.univie.ac.at Thu Jul 4 00:11:43 2019 From: hawk at tbi.univie.ac.at (Cron Daemon) Date: Thu, 4 Jul 2019 02:11:43 +0200 (CEST) Subject: [Gluster-users] Cron rsync -aq --delete /scr/orion/software/linux/ /scratch2/linux/ Message-ID: <20190704001154.AECBE8002869@denobula.tbi.univie.ac.at> file has vanished: "/scr/orion/software/linux/fedora/updates/30/Everything/x86_64/Packages/q/.qt5-qtdeclarative-devel-5.12.4-1.fc30.x86_64.rpm.h3U0S8" rsync warning: some files vanished before they could be transferred (code 24) at main.c(1189) [sender=3.1.3] From adam at yoobra.com Thu Jul 4 20:53:54 2019 From: adam at yoobra.com (Adam Cox) Date: Thu, 4 Jul 2019 21:53:54 +0100 Subject: [Gluster-users] Isolated Client faster than Server+Client Message-ID: Hi, I have been doing some testing of GlusterFS. I have 2 servers running Gluster 6.3 (although same happening in version 3). 1 Server has 32gb ram the other 4gb. On both servers I also have the volume mounted using the Fuse client and when I run a small of copy 100 x 1kb files it takes about 20 seconds to complete. The strange thing is if I spin up a 3rd server and set it up as a client on the volume the same test completes in 2.7 seconds, almost 7.5 times faster. Are there any known reasons for this behaviour? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdeepugd at gmail.com Mon Jul 15 10:34:11 2019 From: sdeepugd at gmail.com (deepu srinivasan) Date: Mon, 15 Jul 2019 16:04:11 +0530 Subject: [Gluster-users] Webhook for georep status change Message-ID: Hi Users Is there a webhook that can be configured for a change in georep status change.Like from "checkpoint not completed" to "checkpoint completed"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdeepugd at gmail.com Mon Jul 15 19:19:44 2019 From: sdeepugd at gmail.com (deepu srinivasan) Date: Tue, 16 Jul 2019 00:49:44 +0530 Subject: [Gluster-users] Configure geo replication after volume extension Message-ID: Hi Users Aim: I intend to extend my volume which is basically a 1x3 volume which is in geo-replication Steps I did: I stopped the geo-replication and I extended the volume 1x3 to 2x3 both the master side and slave side. I did the rebalancing on the master side alone. Restarted the geo-replication and some error pops. What happened: Some error popped out while starting geo-replication. How do I do it properly? Any wiki regardin.g it? Have attached the georeplication error in the mail for reference -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-16 at 12.44.55 AM.png Type: image/png Size: 85499 bytes Desc: not available URL: From sdeepugd at gmail.com Tue Jul 16 07:54:40 2019 From: sdeepugd at gmail.com (deepu srinivasan) Date: Tue, 16 Jul 2019 13:24:40 +0530 Subject: [Gluster-users] Data Rate and Files transfered in Geo Replication Message-ID: Hi Users Is there a way to know the stats of geo-replication like the transfer rate, amount of data and number of the file transferred in Geo-Replication? Also, can I know the estimated time left for a checkpoint? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdeepugd at gmail.com Tue Jul 16 10:56:50 2019 From: sdeepugd at gmail.com (deepu srinivasan) Date: Tue, 16 Jul 2019 16:26:50 +0530 Subject: [Gluster-users] Configure geo replication after volume extension In-Reply-To: <2381fe580401a71a31a6e1c78ce9a0dd3fa8d85c.camel@redhat.com> References: <2381fe580401a71a31a6e1c78ce9a0dd3fa8d85c.camel@redhat.com> Message-ID: Thanks a lot, Aravindha. How should the volume rebalance should be done? Should it be done on either side i.e master and slave side or only master side? On Tue, Jul 16, 2019 at 8:18 AM Aravinda wrote: > On Master run `gluster-georep-sshkey generate` and Geo-rep create > command with force option to redistribute the keys from new nodes to > all Slave nodes. > > > On Tue, 2019-07-16 at 00:49 +0530, deepu srinivasan wrote: > > Hi Users > > Aim: I intend to extend my volume which is basically a 1x3 volume > > which is in geo-replication > > > > Steps I did: I stopped the geo-replication and I extended the volume > > 1x3 to 2x3 both the master side and slave side. I did the rebalancing > > on the master side alone. Restarted the geo-replication and some > > error pops. > > > > What happened: Some error popped out while starting geo-replication. > > > > How do I do it properly? Any wiki regardin.g it? > > Have attached the georeplication error in the mail for reference > > > > > > > > > -- > regards > Aravinda > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ranaraya at redhat.com Tue Jul 16 13:27:37 2019 From: ranaraya at redhat.com (Ravishankar N) Date: Tue, 16 Jul 2019 18:57:37 +0530 Subject: [Gluster-users] Graceful gluster server retire/poweroff In-Reply-To: <03ab8dc3-d9b7-2a68-763b-64f6ef8ac3e4@assyoma.it> References: <03ab8dc3-d9b7-2a68-763b-64f6ef8ac3e4@assyoma.it> Message-ID: On 16/07/19 3:16 PM, Gionatan Danti wrote: > Hi list, > I have a replica 3 test cluster and I have a question about how > clients behave to an host shutdown. > > If I suddenly switch off one of the gluster server, the connected > clients see a ~42s stall in I/O: this is expected, as it is the > default client timeout. > > However, it is possible to *gracefully* shutdown a running gluster > server, *without* impacting a running client? Ie: putting the affected > server in a "maintenance mode" of sort. Yes, if you simply pkill the gluster brick processes of the node before switching it off, you won't observe the hang on the clients because they will receive the disconnect notification immediately. But before that, you would need to check if there are no pending heals etc. You can use the script [1] which does all these checks in the graceful mode. HTH, Ravi [1] https://github.com/gluster/glusterfs/blob/master/extras/stop-all-gluster-processes.sh > > Thanks. > From ranaraya at redhat.com Thu Jul 18 00:30:45 2019 From: ranaraya at redhat.com (Ravishankar N) Date: Thu, 18 Jul 2019 06:00:45 +0530 Subject: [Gluster-users] Graceful gluster server retire/poweroff In-Reply-To: <1666501861.1548609.1563376817764@mail.yahoo.com> References: <03ab8dc3-d9b7-2a68-763b-64f6ef8ac3e4@assyoma.it> <9a1c8960-c08f-11d7-b92b-6437035df6f5@assyoma.it> <1666501861.1548609.1563376817764@mail.yahoo.com> Message-ID: <452d6394-d6da-e6aa-7d9f-71f7866c388c@redhat.com> On 17/07/19 8:50 PM, Strahil Nikolov wrote: > Hi Ravi, > > Can you clarify which script do you use ? > Usually I rely on oVirt to shutdown the system properly , but > sometimes I have used > '/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh' which > kills everything (including fuse mounts). This is the same script I referred to in my reply (https://github.com/gluster/glusterfs/blob/master/extras/stop-all-gluster-processes.sh) . Not sure why the email is not delivered to the mailing list. Regards, Ravi > > Best Regards, > Strahil Nikolov > > > ? ???????, 16 ??? 2019 ?., 17:32:44 ?. ???????+3, Gionatan Danti > ??????: > > > On 16/07/2019 15:27, Ravishankar N wrote: > > Yes, if you simply pkill the gluster brick processes of the node before > > switching it off, you won't observe the hang on the clients because > they > > will receive the disconnect notification immediately. But before that, > > you would need to check if there are no pending heals etc. You can use > > the script [1] which does all these checks in the graceful mode. > > Hi Ravi, > thanks for your reply. I tried killing the glusterfsd process and I > confirm that the client does not see any pause, indeed. > > I was thinking that during the shutdown procedure systemd would kill the > processes by itself; which it *does*, but only if the glusterfsd.service > is enabled/started. > > By default, the systemd service installed on CentOS 7 + Gluster 6.0 SIG > does not start glusterfsd.service and so, in the shutdown phase, it does > not stop it. > > Thanks very much for your information. > > -- > Danti Gionatan > Supporto Tecnico > Assyoma S.r.l. - www.assyoma.it > email: g.danti at assyoma.it - > info at assyoma.it > GPG public key ID: FF5F32A8 > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From srakonde at redhat.com Thu Jul 18 08:15:28 2019 From: srakonde at redhat.com (Sanju Rakonde) Date: Thu, 18 Jul 2019 13:45:28 +0530 Subject: [Gluster-users] Memory leak in gluster 5.4 In-Reply-To: References: Message-ID: Christian, To debug memory leaks, we need periodic statedumps of respective processes. Please provide the statedumps. I suspect that you are hitting https://bugzilla.redhat.com/show_bug.cgi?id=1694612. This bug is addressed in release-5.6 Thanks, Sanju On Thu, Jul 18, 2019 at 1:30 PM Christian Meyer wrote: > Hi everyone! > > I'm using a Gluster 5.4 Setup with 3 Nodes with three volumes > (replicated) (one is the gluster shared storage). > Each node has 64GB of RAM. > Over the time of ~2 month the memory consumption of glusterd grow > linear. An the end glusterd used ~45% of RAM the brick processes > together ~43% of RAM. > I think this is a memory leak. > > I made a coredump of the processes (glusterd, bricks) (zipped ~500MB), > hope this will help you to find the problem. > > Could you please have a look on it? > > Download Coredumps: > https://s3.eu-central-1.amazonaws.com/glusterlogs/gluster_coredump.zip > > Kind regards > > Christian > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -- Thanks, Sanju -------------- next part -------------- An HTML attachment was scrubbed... URL: From amukherj at redhat.com Thu Jul 18 08:16:59 2019 From: amukherj at redhat.com (Atin Mukherjee) Date: Thu, 18 Jul 2019 13:46:59 +0530 Subject: [Gluster-users] Glusterd2 for production In-Reply-To: References: Message-ID: No, it's not production ready. On Thu, Jul 18, 2019 at 1:33 PM deepu srinivasan wrote: > Hi Users > Is it safe to use glusterd2 for production? > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From mscherer at redhat.com Thu Jul 18 08:38:43 2019 From: mscherer at redhat.com (Michael Scherer) Date: Thu, 18 Jul 2019 10:38:43 +0200 Subject: [Gluster-users] Regarding the recent arrival of emails on list Message-ID: <6285ca09acc6fc87d133527ec9ffd6bdde600124.camel@redhat.com> Hi, while working on https://bugzilla.redhat.com/show_bug.cgi?id=1730962, I did notice that no one was moderating the list since end of April, so I decided to clean the queue (around 200 to 300 mails, most non legitimate, but some where). That's why old emails came suddenly, as I figured that even if this was old, some might be useful for archives or something. Sorry for not having looked earlier, it seems we might need to redistribute the responsibility regarding this task since a few people who were doing it likely moved on from the project. -- Michael Scherer Sysadmin, Community Infrastructure -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: From aspandey at redhat.com Thu Jul 18 11:26:15 2019 From: aspandey at redhat.com (Ashish Pandey) Date: Thu, 18 Jul 2019 07:26:15 -0400 (EDT) Subject: [Gluster-users] Share Experience with Productive Gluster Setup <> In-Reply-To: <25e55f2e-9ef9-f732-7230-9d1605d67804@gmx.de> References: <25e55f2e-9ef9-f732-7230-9d1605d67804@gmx.de> Message-ID: <1427476165.722921.1563449175601.JavaMail.zimbra@redhat.com> Hi Felix, As I don't have much expertise on hardware side, I would not comment on that segment. Based on your "Requirements", I would say that it looks very much feasible. As you are talking about storing large files, I would say that disperse volume could be a good choice. However, disperse volumes may not give you good performance for frequent random IO's. If the read/writes are sequential than this could be the better choice. I also don't think that mixing up gluster versions for the same storage solution would be a smart thing to do. BTW, pictures were not visible on this mail. In case you want to share your ideas and discuss these things with other gluster community users, you can participate in community meeting. Meeting Details - * APAC friendly hours * Every 2nd and 4th Tuesday at 11:30AM IST * Bridge: https://bluejeans.com/836554017 * NA/EMEA * Every 1st and 3rd Tuesday at 01:00 PM EDT * Bridge: https://bluejeans.com/486278655 --- Ashish ----- Original Message ----- From: "Felix K?lzow" To: Gluster-users at Gluster.org Sent: Thursday, July 18, 2019 12:57:56 PM Subject: [Gluster-users] Share Experience with Productive Gluster Setup <> Dear Gluster-Community, we try to implement a gluster setup in a productive environment. During the development process, we found this nice thread regarding to gluster storage experience: https://forums.overclockers.com.au/threads/glusterfs-800tb-and-growing.1078674/ This thread seems to be in the early retirement. Thus, i try to reanimate this thread to generate a concise gluster real life experience for further readers. Unfortunately, I don't have the given permissions by the admin to post this message, so I attached the message below. I plan to write news/updates within this thread (link above). Regards Felix >>> Should be me first post <<< Dear Community, thank you very much for such a very informative talk regarding gluster and gluster real life experience. We also plan to build a gluster system and I thought it is maybe a good idea to reanimate this little old thread to unify the experience for further readers. The main idea is to discuss our current setup (this exists only in our minds actually) and share our experience with the community if we really go ahead with gluster and real hardware. * Requirements: - Current Volume: about 55TB - Future Volume: REALLY unpredictable for several reasons - could be 120 TB in 5 years or - could be 500 TB in 5 years - Files: - well known office files - research files - larger text files (> 10GB) - many pictures from experimental investigations (currently 500 GB - 1.5TB) for each experiment - large HDF5 files - Clients: - Windows 98,2000,XP,7,10; Linux (Arch, Ubuntu, CentOS), MacOS - Some data must be immediately available in 10 - 20 years (data cannot exclusively moved to tape storage, must available on hot storage) - Use Redhat Gluster Storage for production environment - Currently planed: Backup-Side using CentOS8 + tape storage - timeline: if gluster is realized, it should be productive in end of this year * Support - It is very likely that we are going use Redhat Gluster Storage and corresponding support. * Planned Architecture: - Distributed Replicated (Replica 3-Setup) Volume with two 60 TB Bricks per node and 500GB LVM Cache PCIe NVMe, i.e. 120TB Total Storage * Scenario - It is planned that we run 3 nodes with 2 volumes within a distributed replicated setup - Also a dispersed volume could be possible?! - Using LVM Cache - two volumes - one volume for office data (almost for pdf, MS Office :-/,...) - one volume for research data (see above) - The Gluster Cluster uses a geo-replication to a single node that is located in a different building. Thus, we have a asynchronus native backup (realized to due snapshots) that is easily to restore. Snapshots will be retained for 7 days. - The native backup server and the server for compression/deduplication are located next to each other (for no network bottleneck reasons). Bacula/Bareos should be used for weekly, monthly and yearly backups. - Yearly and certain monthly backups are archived on tape-storage - Tapes will be stored at two different locations * Hardware: ** Gluster Storage Server: GEH 4u Supermicro | CSE-847BE1C-R1K28LPB - Mainboard X11DPi-NT - 2x Intel Xeon LGA 3647 Silver, 8 Cores - 96 GB RAM, DDR4 2666 ECC - Adaptec ASR 8805 Raid-Controller - Two smaller enterprise SSD 2.5" for os - 2x Intel Optane ca. 500GB PCIe NMVe for LVM Cache (Software Raid 0) - 4x 10Gbps RJ45 for Network - 24x 6TB Drives (2 Bricks, 12 Drives per Brick, HW-Raid6 per Brick) ** Server for Native Backup (1 Node Geo-Replication) - Mainboard X11DPi-NT - 2x Intel Xeon LGA 3647 Silver, 8 Cores - 96 GB RAM, DDR4 2666 ECC - Adaptec ASR 8805 Raid-Controller (IT-Mode) - Two smaller enterprise SSD 2.5" for os (xfs) - 4x 10Gbps RJ45 for Network - 12x 10 TB Drives (planned for ZFSonLinux, raidz2) - ZFS compression on the fly, no deduplication ** Server for Compressed/Deduplicated Backup - Mainboard X11DPi-NT - 2x Intel Xeon LGA 3647 Silver, 8 Cores - 192 GB RAM, DDR4 2666 ECC (considering ZFS RAM consumption during deduplication) - Adaptec ASR 8805 Raid-Controller (IT-Mode) - Two smaller enterprise SSD 2.5" for os (xfs) - 4x 10Gbps RJ45 for Network - 12x 6 TB Drives (planned for ZFSonLinux, raidz2) ** Tape-Storage - Hardware not defined yet * Software/OS ** Gluster Storage Server - Running RHEL8 and REDHAT Gluster Storage - With Support ** Server for Native Backup - Running CentOS 8 (not released yet) - Running COMMUNITY Gluster ** Server for Compressed/Deduplicated Backup - Running CentOS 8 (not released yet) - Backup using Bareos or Bacula (without Support, but 'personal' training planned) - Deduplication using ZFSonLinux * Volumes ** Linux Clients and Linux Computing Environment - Using FUSE ** Windows Clients / MacOS - CTDB * Questions: 1. Any hints/remarks to the considered hardware? 2. Does anybody realized a comparable setup and is able to share performance results? 3. Is ZFSonLinux really ready for production for this scenario. 4. Is it reliable to mix Gluster Versions, i.e. production system on Redhat Gluster Storage, Native Backup Server on Community Gluster The aim here is really to share experience and we try to document our experience and performance tests if we are really going ahead with gluster. Every remarks/hints are very welcome. I think this is enough for the first post. Feel free to ask! Thanks to image-ascii converter, the shorted setup could look like this. :-/::--./: Geo-Replication ::--:/-:::/.// ::- `. `.`- .` `. - /:/.- ` ` `` `` `` /:/`` `` `.:` `:``:`:` ` `......-::::::.-.:::/::-:::::::-----::.-::-:::::::--.::.....` d/h.sy-o/:yo::+-s+/o.hh-++///s/s:/s::s/soo/o`s:+s-o+: s+s/ Production System Building A y-h:os :. ----://--:-: `.. s/o-+o Backup System Building B o--/:yy/ -/:`/-/:...//-sh.// :` .-`//`/:-::::.:--: /` `- Master ..`----`.-.- Slave : s/o-+o:++-/o-+o-h/+--+:o/.////`o-::o o///+`//o--/:yy/ `` .. .`` ` `.` : ..`+:/+/+//+:` ://:+:-+: : .. `````````` ````````` : `` ` ` ` `` .. ` ` ```` ` `` ` : //-//:::::`::o-::::/:: : ::-.::::` +/.:--:. / .. /`::/:+-+/.-:/:::-v \ : / :-::-:-:.--.`+:+-:--::: `.`.``.``` ````.```... -...``.-.` ...-.... . .. .`.`.`....````...`.` ` \|/ .`.-./..-.`. ..-.-`--.- .`````````````````````````-:`````````````````````````. ..`````````````````````````/`````````````````````````- -``` .. ..- .... : ..- -``` .. ..- .``` : ``- ` ` -``.``````````````````````--``````````````````````.``- ..`-```````````````````````/``````````````````````..`- `+:+:::-/-- - . .. . - .` . : `. - ....``.-`. - . .. . - .` . -h: `. - - . .. . - .` :-:``:-::-:.:----------:o:-:.:------:---::--`-::. - ` - . .. . - .` //+ `..--:`:------..-------`:-------.---.`` ./+` - `/-::---- ``````````````.` - . .. . - .` //+ .:` ./+` - .....-.. /`````````````:--` ` - /--`.------.-------.---:/---.-------.------.`--/ - .` //+ +s- ./+` - : -` .-` o-:--./-//:-/.-: - +/: ````````````````.+o.`. .```````````` :/+ - .` //+ ``` ```````````````` ```````` `` ./+` - : -` .- -.``...`........ - +// `.... Gluster Storage Node 1.`....` :/+ - .` //+ --://+.///////::///////.+///////-///:-. ./+` - : -. `-- - :..`.-..............-..::..-`.............-.`..: - .` //+ +////+./ Native Backup Server /////: ./+` - : ./:..-:-.-//-...../ - . .. . - .` //+ -:///+.///////::///////.+///////-////:. ./+` - : ..-::----:/-` : - . .. . - .` //+ `.. .......``....... ........`..` ./+` - +-...-....-.-.....``-..+ `` - /--`.--------------.---:/--:.--------------.`--/ - .` -..`````````````````````:.```````````````````..-. - ``+------/:------/-.:---/y /-/:/:/--/` - +/: ``````````` ``.oo.`` ```````````` :/+ - .` . -` ``..........--.:..................`: ``..`-` : ` ``...:`.` - +/: ``.... Gluster Storage Node 2......`` :/+ - .` . .-.........`` `. - : --:.::` : - :..``..................::...`...............`..: - .` . `- `. - o...`.- `...`...````.`./ ................ - . .. . - .` . `- `. - s.://-/-+-+::./:/.::./-s : .:-. - . .. . - .` . +y` `. - : ` ./ : .- `-. - /--`.---::--:::::::.::::/:::.:::::::--::---.`--/ - .` ::/`.:::::/.::::::::::::/o::./:::::::-:::::-`-:/. - : : : .- `-. - +/: ```````````` ```++``` ```````````` :/+ - .` //+ ``.---`-------..-------`--------`--.`` ./+` - : : : .- `.-. - +/: ``...... Gluster Storage Node 3.....`` :/+ - .` //+ -/` ./+` - /``````````````````````/ : ......../ - :..``..............`........`..............``..: - .` //+ /o. ./+` - ` ` ` ` ```````````````````````` : : - . . - .` //+ ``.. .......``....... ........`..` ./+` - :-//-:/:::: : : - . . - .` //+ -:://+.///////::///////.+///////-//:::. ./+` - ` `.``.```` :.............--` : : `+s/ . . - .` //+ +////+ Backup - Server //-/////: ./+` - : -.-.` : ..` `````````````` :..`--- . . - .` //+ -::// Compression / Deduplication ::-. ./+` - : -` `-. : :/::/::/::::::.::: : - . . - .` //+`````...`................`........`...````-/+` - : -` `-. : `-..-.`..```````.` : - . . - .` -..`````````````````````:`````````````````````... - : -.`````.-- : --/-://`:-/:::/:/:/` : - . . - .` . : `. - : ````````/ : ```..` `..`.`.```` : - . . - .` . :s. `. - : ` : : //::::-/::::.: : - . . - .` . -/++++/. .+ `-/++++:` `. - : `/:::::: : : .``````.-:``` : - . . - .` . `s/:o .+o` -s/+: `-o/ `. - ````````````: Tape Storage : : : - . . - .` . y: :++```+o d. ++/ `.y:/s........--./........``````.````: :/:-::./-::/` : : ` : - . . - .` . h. /sy--//y`d soh`//o+.: `. - : Building B : : .:/-:-/-+-/./::::. : - . . - .` . :s..s` .y- o+`-+` -y` `. - : : : `-.`...`.........` : - . . - .` . .+o+::/ods+++hys/-:/o+` `. - : : : `++-:--::-./:-:-/` : -``-``````````````````````````````````````````````-``- ..`-```````````````-::.````````.-::.``````````````..`: : : : ....:-...--....- : - ` ``- .``` ``- : : : `:./:: .//:- : -... --- .... ..: : : /......:/::/..::::...../ -````````````````````````````````````````````````````- ..```````````````````````````````````````````````````- /....................../ _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From dijuremo at gmail.com Thu Jul 18 12:23:39 2019 From: dijuremo at gmail.com (Diego Remolina) Date: Thu, 18 Jul 2019 08:23:39 -0400 Subject: [Gluster-users] Memory leak in gluster 5.4 In-Reply-To: References: Message-ID: Are you using FUSE mounts to access the file system? I have observed the same issue all the way from 3.x series to 4.x series. I have to reboot servers monthly, at most every 45 days because gluster keeps eating the ram. As you can see, 30 days in we are right at around 30GB of RAM used. [image: image.png] Diego On Thu, Jul 18, 2019 at 4:15 AM Sanju Rakonde wrote: > Christian, > > To debug memory leaks, we need periodic statedumps of respective > processes. Please provide the statedumps. > > I suspect that you are hitting > https://bugzilla.redhat.com/show_bug.cgi?id=1694612. This bug is > addressed in release-5.6 > > Thanks, > Sanju > > On Thu, Jul 18, 2019 at 1:30 PM Christian Meyer > wrote: > >> Hi everyone! >> >> I'm using a Gluster 5.4 Setup with 3 Nodes with three volumes >> (replicated) (one is the gluster shared storage). >> Each node has 64GB of RAM. >> Over the time of ~2 month the memory consumption of glusterd grow >> linear. An the end glusterd used ~45% of RAM the brick processes >> together ~43% of RAM. >> I think this is a memory leak. >> >> I made a coredump of the processes (glusterd, bricks) (zipped ~500MB), >> hope this will help you to find the problem. >> >> Could you please have a look on it? >> >> Download Coredumps: >> https://s3.eu-central-1.amazonaws.com/glusterlogs/gluster_coredump.zip >> >> Kind regards >> >> Christian >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > > > -- > Thanks, > Sanju > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 107885 bytes Desc: not available URL: From matthewb at uvic.ca Thu Jul 18 14:20:25 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Thu, 18 Jul 2019 07:20:25 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: Message-ID: Hi Nithya, No - it was added about a year and a half ago. I have tried re-mounting the volume on the server, but it didn't add the attr: [root at gluster07 ~]# umount /storage/ [root at gluster07 ~]# cat /etc/fstab | grep "/storage" 10.0.231.56:/storage /storage glusterfs defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 [root at gluster07 ~]# mount /storage/ [root at gluster07 ~]# df -h /storage/ Filesystem??????????? Size? Used Avail Use% Mounted on 10.0.231.56:/storage? 255T? 194T?? 62T? 77% /storage [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ # file: /mnt/raid6-storage/storage/ security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 Thanks, ?-Matthew On 7/17/19 10:04 PM, Nithya Balachandran wrote: > Hi Matthew, > > Was this node/brick added to the volume recently? If yes, try mounting > the volume on a fresh mount point - that should create the xattr on > this as well. > > Regards, > Nithya > > On Wed, 17 Jul 2019 at 21:01, Matthew Benstead > wrote: > > Hello, > > I've just noticed one brick in my 7 node distribute volume is missing > the trusted.glusterfs.dht xattr...? How can I fix this? > > I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. > > All of the other nodes are fine, but gluster07 from the list below > does > not have the attribute. > > $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . > --absolute-names -n trusted.glusterfs.dht -e hex > /mnt/raid6-storage/storage" > ... > gluster05 | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 > > gluster03 | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 > > gluster04 | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff > > gluster06 | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 > > gluster02 | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 > > gluster07 | FAILED | rc=1 >> > /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such > attributenon-zero return code > > gluster01 | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000555555547ffffffd > > Here are all of the attr's from the brick: > > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ > # file: /mnt/raid6-storage/storage/ > security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 > trusted.gfid=0x00000000000000000000000000000001 > trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 > trusted.glusterfs.quota.dirty=0x3000 > trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 > trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 > > > And here is the volume information: > > [root at gluster07 ~]# gluster volume info storage > > Volume Name: storage > Type: Distribute > Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 > Status: Started > Snapshot Count: 0 > Number of Bricks: 7 > Transport-type: tcp > Bricks: > Brick1: 10.0.231.50:/mnt/raid6-storage/storage > Brick2: 10.0.231.51:/mnt/raid6-storage/storage > Brick3: 10.0.231.52:/mnt/raid6-storage/storage > Brick4: 10.0.231.53:/mnt/raid6-storage/storage > Brick5: 10.0.231.54:/mnt/raid6-storage/storage > Brick6: 10.0.231.55:/mnt/raid6-storage/storage > Brick7: 10.0.231.56:/mnt/raid6-storage/storage > Options Reconfigured: > changelog.changelog: on > features.quota-deem-statfs: on > features.read-only: off > features.inode-quota: on > features.quota: on > performance.readdir-ahead: on > nfs.disable: on > geo-replication.indexing: on > geo-replication.ignore-pid-check: on > transport.address-family: inet > > Thanks, > ?-Matthew > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From corochoone at gmail.com Thu Jul 18 15:56:22 2019 From: corochoone at gmail.com (=?UTF-8?B?0JLQuNC60YLQvtGAINCS0LjRgdC70L7QsdC+0LrQvtCy?=) Date: Thu, 18 Jul 2019 18:56:22 +0300 Subject: [Gluster-users] Fuse vs NFS In-Reply-To: References: Message-ID: Hello. Seriously? You REALLY thinks, that FUSE (in user space) will be faster then kernel NFS? With best wishes. Victor 2019-06-07 7:29 GMT+03:00, Sudheer Singh : > Hi , > > I was doing perf testing and found out fuse mount much slower than NFS > mount. I was curious to know what community recommends, mount volumes as > fuse or NFS? > > -- > Thanks, > Sudheer > From alan.orth at gmail.com Thu Jul 18 15:57:31 2019 From: alan.orth at gmail.com (Alan Orth) Date: Thu, 18 Jul 2019 18:57:31 +0300 Subject: [Gluster-users] CentOS RPMs for version 5.7 Message-ID: Dear list, I noticed that the release notes for Gluster version 5.7 were updated two weeks ago?, but there are no RPMs for CentOS yet?. Did someone forget to push a button or is there some other process we are waiting for? Thank you, ? https://github.com/gluster/glusterfs/blob/release-5/doc/release-notes/5.7.md ? http://mirror.centos.org/centos/7/storage/x86_64/gluster-5/ -- Alan Orth alan.orth at gmail.com https://picturingjordan.com https://englishbulgaria.net https://mjanja.ch "In heaven all the interesting people are missing." ?Friedrich Nietzsche -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgowtham at redhat.com Thu Jul 18 16:15:53 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Thu, 18 Jul 2019 21:45:53 +0530 Subject: [Gluster-users] CentOS RPMs for version 5.7 In-Reply-To: References: Message-ID: Hi Alan, We came across a python dependency issue which broke 5.7. The issue has been fixed and we are doing 5.8 by skipping 5.7. 5.8 has been tagged, the rpms will be made available soon. Do keep an eye on the list for the announcement of 5.8. Sorry for the inconvenience. Regards, Hari. On Thu, 18 Jul, 2019, 9:27 PM Alan Orth, wrote: > Dear list, > > I noticed that the release notes for Gluster version 5.7 were updated two > weeks ago?, but there are no RPMs for CentOS yet?. Did someone forget to > push a button or is there some other process we are waiting for? > > Thank you, > > ? > https://github.com/gluster/glusterfs/blob/release-5/doc/release-notes/5.7.md > ? http://mirror.centos.org/centos/7/storage/x86_64/gluster-5/ > -- > Alan Orth > alan.orth at gmail.com > https://picturingjordan.com > https://englishbulgaria.net > https://mjanja.ch > "In heaven all the interesting people are missing." ?Friedrich Nietzsche > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Fri Jul 19 04:00:15 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Fri, 19 Jul 2019 09:30:15 +0530 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: Message-ID: Hi Mathew, Could you send me the log file for this mount point after doing the following: 1. gluster v set client-log-level DEBUG 2. unmount and remount the volume 3. stat 4. gluster v set client-log-level INFO Regards, Nithya On Thu, 18 Jul 2019 at 19:49, Matthew Benstead wrote: > Hi Nithya, > > No - it was added about a year and a half ago. I have tried re-mounting > the volume on the server, but it didn't add the attr: > > [root at gluster07 ~]# umount /storage/ > [root at gluster07 ~]# cat /etc/fstab | grep "/storage" > 10.0.231.56:/storage /storage glusterfs > defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 > [root at gluster07 ~]# mount /storage/ > [root at gluster07 ~]# df -h /storage/ > Filesystem Size Used Avail Use% Mounted on > 10.0.231.56:/storage 255T 194T 62T 77% /storage > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ > # file: /mnt/raid6-storage/storage/ > > security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 > trusted.gfid=0x00000000000000000000000000000001 > > trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 > trusted.glusterfs.quota.dirty=0x3000 > > trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 > trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 > > Thanks, > -Matthew > > On 7/17/19 10:04 PM, Nithya Balachandran wrote: > > Hi Matthew, > > Was this node/brick added to the volume recently? If yes, try mounting the > volume on a fresh mount point - that should create the xattr on this as > well. > > Regards, > Nithya > > On Wed, 17 Jul 2019 at 21:01, Matthew Benstead wrote: > >> Hello, >> >> I've just noticed one brick in my 7 node distribute volume is missing >> the trusted.glusterfs.dht xattr...? How can I fix this? >> >> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >> >> All of the other nodes are fine, but gluster07 from the list below does >> not have the attribute. >> >> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . >> --absolute-names -n trusted.glusterfs.dht -e hex >> /mnt/raid6-storage/storage" >> ... >> gluster05 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >> >> gluster03 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >> >> gluster04 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >> >> gluster06 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >> >> gluster02 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >> >> gluster07 | FAILED | rc=1 >> >> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >> attributenon-zero return code >> >> gluster01 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >> >> Here are all of the attr's from the brick: >> >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ >> # file: /mnt/raid6-storage/storage/ >> >> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.gfid=0x00000000000000000000000000000001 >> >> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >> trusted.glusterfs.quota.dirty=0x3000 >> >> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >> >> >> And here is the volume information: >> >> [root at gluster07 ~]# gluster volume info storage >> >> Volume Name: storage >> Type: Distribute >> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 7 >> Transport-type: tcp >> Bricks: >> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >> Options Reconfigured: >> changelog.changelog: on >> features.quota-deem-statfs: on >> features.read-only: off >> features.inode-quota: on >> features.quota: on >> performance.readdir-ahead: on >> nfs.disable: on >> geo-replication.indexing: on >> geo-replication.ignore-pid-check: on >> transport.address-family: inet >> >> Thanks, >> -Matthew >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rightkicktech at gmail.com Fri Jul 19 04:23:16 2019 From: rightkicktech at gmail.com (Alex K) Date: Fri, 19 Jul 2019 07:23:16 +0300 Subject: [Gluster-users] Fuse vs NFS In-Reply-To: References: Message-ID: On Thu, Jul 18, 2019, 11:06 Sudheer Singh wrote: > Hi , > > I was doing perf testing and found out fuse mount much slower than NFS > mount. I was curious to know what community recommends, mount volumes as > fuse or NFS? > You may need to consider libgfapi instead of fuse. I have seen better performance with it then nfs. > > -- > Thanks, > Sudheer > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From srakonde at redhat.com Fri Jul 19 06:03:16 2019 From: srakonde at redhat.com (Sanju Rakonde) Date: Fri, 19 Jul 2019 11:33:16 +0530 Subject: [Gluster-users] Memory leak in gluster 5.4 In-Reply-To: References: Message-ID: Diego, please raise a bug and share the periodic statedumps of processes which have memory leaks. On Thu, Jul 18, 2019 at 5:53 PM Diego Remolina wrote: > Are you using FUSE mounts to access the file system? I have observed the > same issue all the way from 3.x series to 4.x series. I have to reboot > servers monthly, at most every 45 days because gluster keeps eating the ram. > > As you can see, 30 days in we are right at around 30GB of RAM used. > > [image: image.png] > > Diego > > On Thu, Jul 18, 2019 at 4:15 AM Sanju Rakonde wrote: > >> Christian, >> >> To debug memory leaks, we need periodic statedumps of respective >> processes. Please provide the statedumps. >> >> I suspect that you are hitting >> https://bugzilla.redhat.com/show_bug.cgi?id=1694612. This bug is >> addressed in release-5.6 >> >> Thanks, >> Sanju >> >> On Thu, Jul 18, 2019 at 1:30 PM Christian Meyer >> wrote: >> >>> Hi everyone! >>> >>> I'm using a Gluster 5.4 Setup with 3 Nodes with three volumes >>> (replicated) (one is the gluster shared storage). >>> Each node has 64GB of RAM. >>> Over the time of ~2 month the memory consumption of glusterd grow >>> linear. An the end glusterd used ~45% of RAM the brick processes >>> together ~43% of RAM. >>> I think this is a memory leak. >>> >>> I made a coredump of the processes (glusterd, bricks) (zipped ~500MB), >>> hope this will help you to find the problem. >>> >>> Could you please have a look on it? >>> >>> Download Coredumps: >>> https://s3.eu-central-1.amazonaws.com/glusterlogs/gluster_coredump.zip >>> >>> Kind regards >>> >>> Christian >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> -- >> Thanks, >> Sanju >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > -- Thanks, Sanju -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 107885 bytes Desc: not available URL: From mauro.tridici at cmcc.it Fri Jul 19 09:56:54 2019 From: mauro.tridici at cmcc.it (Mauro Tridici) Date: Fri, 19 Jul 2019 11:56:54 +0200 Subject: [Gluster-users] "du" and "df -hT" commands output mismatch Message-ID: <287791D2-4775-4CB1-9AA1-9C160839C2D2@cmcc.it> Dear All, I?m experiencing again a problem with gluster file system quota. The ?df -hT /tier2/CSP/sp1? command output is different from the ?du -ms? command executed against the same folder. [root at s01 manual]# df -hT /tier2/CSP/sp1 Filesystem Type Size Used Avail Use% Mounted on s01-stg:tier2 fuse.glusterfs 25T 22T 3.5T 87% /tier2 [root at s01 sp1]# du -ms /tier2/CSP/sp1 14TB /tier2/CSP/sp1 In the past, I used successfully the quota_fsck_new-6.py script in order to detect the SIZE_MISMATCH occurrences and fix them. Unfortunately, the number of sub-directories and files saved in /tier2/CSP/sp1 grew so much and the list of SIZE_MISMATCH entries is very long. Is there a faster way to correct the mismatching outputs? Could you please help me to solve, if it is possible, this issue? Thank you in advance, Mauro -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgowtham at redhat.com Fri Jul 19 10:48:40 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Fri, 19 Jul 2019 16:18:40 +0530 Subject: [Gluster-users] "du" and "df -hT" commands output mismatch In-Reply-To: <287791D2-4775-4CB1-9AA1-9C160839C2D2@cmcc.it> References: <287791D2-4775-4CB1-9AA1-9C160839C2D2@cmcc.it> Message-ID: Hi Mauro, The fsck script is the fastest way to resolve the issue. The other way would be to disable quota and once the crawl for disable is done, we have to enable and set the limits again. In this way, the crawl happens twice and hence its slow. On Fri, Jul 19, 2019 at 3:27 PM Mauro Tridici wrote: > > Dear All, > > I?m experiencing again a problem with gluster file system quota. > The ?df -hT /tier2/CSP/sp1? command output is different from the ?du -ms? command executed against the same folder. > > [root at s01 manual]# df -hT /tier2/CSP/sp1 > Filesystem Type Size Used Avail Use% Mounted on > s01-stg:tier2 fuse.glusterfs 25T 22T 3.5T 87% /tier2 > > [root at s01 sp1]# du -ms /tier2/CSP/sp1 > 14TB /tier2/CSP/sp1 > > In the past, I used successfully the quota_fsck_new-6.py script in order to detect the SIZE_MISMATCH occurrences and fix them. > Unfortunately, the number of sub-directories and files saved in /tier2/CSP/sp1 grew so much and the list of SIZE_MISMATCH entries is very long. > > Is there a faster way to correct the mismatching outputs? > Could you please help me to solve, if it is possible, this issue? > > Thank you in advance, > Mauro > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- Regards, Hari Gowtham. From mauro.tridici at cmcc.it Fri Jul 19 12:12:30 2019 From: mauro.tridici at cmcc.it (Mauro Tridici) Date: Fri, 19 Jul 2019 14:12:30 +0200 Subject: [Gluster-users] "du" and "df -hT" commands output mismatch In-Reply-To: References: <287791D2-4775-4CB1-9AA1-9C160839C2D2@cmcc.it> Message-ID: <176E4EC2-DB3E-4C33-8EB4-8D09DCB11599@cmcc.it> Hi Hari, thank you very much for the fast answer. I think that the we will try to solve the issue disabling and enabling quota. So, if I understand I have to do the following actions: - save on my notes the current quota limits; - disable quota using "gluster volume quota /tier2 disable? command; - wait a while for the crawl (question: how can I understand that crawl is terminated!? how logn should I wait?); - enable quota using "gluster volume quota /tier2 enable?; - set again the previous quota limits. Is this correct? Many thanks for your support, Mauro > On 19 Jul 2019, at 12:48, Hari Gowtham wrote: > > Hi Mauro, > > The fsck script is the fastest way to resolve the issue. > The other way would be to disable quota and once the crawl for disable > is done, we have to enable and set the limits again. > In this way, the crawl happens twice and hence its slow. > > On Fri, Jul 19, 2019 at 3:27 PM Mauro Tridici wrote: >> >> Dear All, >> >> I?m experiencing again a problem with gluster file system quota. >> The ?df -hT /tier2/CSP/sp1? command output is different from the ?du -ms? command executed against the same folder. >> >> [root at s01 manual]# df -hT /tier2/CSP/sp1 >> Filesystem Type Size Used Avail Use% Mounted on >> s01-stg:tier2 fuse.glusterfs 25T 22T 3.5T 87% /tier2 >> >> [root at s01 sp1]# du -ms /tier2/CSP/sp1 >> 14TB /tier2/CSP/sp1 >> >> In the past, I used successfully the quota_fsck_new-6.py script in order to detect the SIZE_MISMATCH occurrences and fix them. >> Unfortunately, the number of sub-directories and files saved in /tier2/CSP/sp1 grew so much and the list of SIZE_MISMATCH entries is very long. >> >> Is there a faster way to correct the mismatching outputs? >> Could you please help me to solve, if it is possible, this issue? >> >> Thank you in advance, >> Mauro >> >> >> >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Regards, > Hari Gowtham. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewb at uvic.ca Fri Jul 19 15:31:28 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Fri, 19 Jul 2019 08:31:28 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: Message-ID: <1504eb87-91c2-20b5-8971-5cb06fa2da5a@uvic.ca> Hi Nithya, I've completed this, and attached the log. [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ # file: /mnt/raid6-storage/storage/ security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d31d426000cee76 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size.2=0x00001b73f79d92000000000000764c7a000000000005ce79 trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 [root at gluster07 ~]# gluster volume get storage client-log-level Option????????????????????????????????? Value?????????????????????????????????? ------????????????????????????????????? -----?????????????????????????????????? diagnostics.client-log-level??????????? INFO??????????????????????????????????? [root at gluster07 ~]# gluster volume set storage client-log-level DEBUG volume set: success [root at gluster07 ~]# umount /storage/ umount: /storage/: not mounted [root at gluster07 ~]# mount /storage/ [root at gluster07 ~]# stat /storage/ ? File: ?/storage/? ? Size: 4096????? ??? Blocks: 8????????? IO Block: 131072 directory Device: 2bh/43d??? Inode: 1?????????? Links: 6 Access: (0755/drwxr-xr-x)? Uid: (??? 0/??? root)?? Gid: (??? 0/??? root) Context: system_u:object_r:fusefs_t:s0 Access: 2019-07-19 08:01:17.458527914 -0700 Modify: 2018-07-04 15:18:05.947569905 -0700 Change: 2019-07-19 08:01:10.350465738 -0700 ?Birth: - [root at gluster07 ~]# gluster volume set storage client-log-level INFO volume set: success [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ # file: /mnt/raid6-storage/storage/ security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d31dafd000a5aa3 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size.2=0x00001b73f79da0000000000000764c7a000000000005ce79 trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 Thanks, ?-Matthew On 7/18/19 9:00 PM, Nithya Balachandran wrote: > Hi Mathew, > > Could you send me the log file for this mount point after doing the > following: > > 1. gluster v set client-log-level DEBUG > 2. unmount and remount the volume > 3. stat > 4. gluster v set client-log-level INFO > > > > Regards, > Nithya > > On Thu, 18 Jul 2019 at 19:49, Matthew Benstead > wrote: > > Hi Nithya, > > No - it was added about a year and a half ago. I have tried > re-mounting the volume on the server, but it didn't add the attr: > > [root at gluster07 ~]# umount /storage/ > [root at gluster07 ~]# cat /etc/fstab | grep "/storage" > 10.0.231.56:/storage /storage glusterfs > defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 > [root at gluster07 ~]# mount /storage/ > [root at gluster07 ~]# df -h /storage/ > Filesystem??????????? Size? Used Avail Use% Mounted on > 10.0.231.56:/storage? 255T? 194T?? 62T? 77% /storage > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ > # file: /mnt/raid6-storage/storage/ > security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 > trusted.gfid=0x00000000000000000000000000000001 > trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 > trusted.glusterfs.quota.dirty=0x3000 > trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 > trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 > > Thanks, > ?-Matthew > > On 7/17/19 10:04 PM, Nithya Balachandran wrote: >> Hi Matthew, >> >> Was this node/brick added to the volume recently? If yes, try >> mounting the volume on a fresh mount point - that should create >> the xattr on this as well. >> >> Regards, >> Nithya >> >> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead > > wrote: >> >> Hello, >> >> I've just noticed one brick in my 7 node distribute volume is >> missing >> the trusted.glusterfs.dht xattr...? How can I fix this? >> >> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >> >> All of the other nodes are fine, but gluster07 from the list >> below does >> not have the attribute. >> >> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a >> "getfattr -m . >> --absolute-names -n trusted.glusterfs.dht -e hex >> /mnt/raid6-storage/storage" >> ... >> gluster05 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >> >> gluster03 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >> >> gluster04 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >> >> gluster06 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >> >> gluster02 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >> >> gluster07 | FAILED | rc=1 >> >> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >> attributenon-zero return code >> >> gluster01 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >> >> Here are all of the attr's from the brick: >> >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ >> # file: /mnt/raid6-storage/storage/ >> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.gfid=0x00000000000000000000000000000001 >> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >> trusted.glusterfs.quota.dirty=0x3000 >> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >> >> >> And here is the volume information: >> >> [root at gluster07 ~]# gluster volume info storage >> >> Volume Name: storage >> Type: Distribute >> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 7 >> Transport-type: tcp >> Bricks: >> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >> Options Reconfigured: >> changelog.changelog: on >> features.quota-deem-statfs: on >> features.read-only: off >> features.inode-quota: on >> features.quota: on >> performance.readdir-ahead: on >> nfs.disable: on >> geo-replication.indexing: on >> geo-replication.ignore-pid-check: on >> transport.address-family: inet >> >> Thanks, >> ?-Matthew >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [2019-07-19 15:01:17.493010] D [MSGID: 0] [io-threads.c:842:__iot_workers_scale] 0-storage-io-threads: scaled threads to 1 (queue_size=0/1) [2019-07-19 15:01:17.493212] D [MSGID: 0] [quick-read.c:1183:check_cache_size_ok] 0-storage-quick-read: Max cache size is 67271393280 [2019-07-19 15:01:17.493298] D [MSGID: 0] [io-cache.c:1595:check_cache_size_ok] 0-storage-io-cache: Max cache size is 67271393280 [2019-07-19 15:01:17.493323] D [MSGID: 0] [options.c:1227:xlator_option_init_size_uint64] 0-storage-readdir-ahead: option rda-request-size using set value 131072 [2019-07-19 15:01:17.493339] D [MSGID: 0] [options.c:1227:xlator_option_init_size_uint64] 0-storage-readdir-ahead: option rda-cache-limit using set value 10MB [2019-07-19 15:01:17.493348] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-readdir-ahead: option parallel-readdir using set value off [2019-07-19 15:01:17.493441] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-dht: option lock-migration using set value off [2019-07-19 15:01:17.493457] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-dht: option force-migration using set value off [2019-07-19 15:01:17.493509] D [MSGID: 0] [dht-shared.c:356:dht_init_regex] 0-storage-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2019-07-19 15:01:17.493599] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-storage-client-6: option ping-timeout using set value 42 [2019-07-19 15:01:17.493599] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2019-07-19 15:01:17.493628] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-storage-client-6: option remote-subvolume using set value /mnt/raid6-storage/storage [2019-07-19 15:01:17.493643] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-client-6: option send-gids using set value true [2019-07-19 15:01:17.493668] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-storage-client-6: defaulting frame-timeout to 30mins [2019-07-19 15:01:17.493678] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-storage-client-6: setting ping-timeout to 42 [2019-07-19 15:01:17.493688] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.3/rpc-transport/socket.so [2019-07-19 15:01:17.493793] D [socket.c:4464:socket_init] 0-storage-client-6: Configued transport.tcp-user-timeout=0 [2019-07-19 15:01:17.493806] D [socket.c:4482:socket_init] 0-storage-client-6: Reconfigued transport.keepalivecnt=9 [2019-07-19 15:01:17.493813] D [socket.c:4167:ssl_setup_connection_params] 0-storage-client-6: SSL support on the I/O path is NOT enabled [2019-07-19 15:01:17.493819] D [socket.c:4170:ssl_setup_connection_params] 0-storage-client-6: SSL support for glusterd is NOT enabled [2019-07-19 15:01:17.493828] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-storage-client-6: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-07-19 15:01:17.493837] D [MSGID: 0] [client.c:2543:client_init_rpc] 0-storage-client-6: client init successful [2019-07-19 15:01:17.493863] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-storage-client-5: option ping-timeout using set value 42 [2019-07-19 15:01:17.493882] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-storage-client-5: option remote-subvolume using set value /mnt/raid6-storage/storage [2019-07-19 15:01:17.493895] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-client-5: option send-gids using set value true [2019-07-19 15:01:17.493904] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-storage-client-5: defaulting frame-timeout to 30mins [2019-07-19 15:01:17.493911] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-storage-client-5: setting ping-timeout to 42 [2019-07-19 15:01:17.493922] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.3/rpc-transport/socket.so [2019-07-19 15:01:17.494015] D [socket.c:4464:socket_init] 0-storage-client-5: Configued transport.tcp-user-timeout=0 [2019-07-19 15:01:17.494025] D [socket.c:4482:socket_init] 0-storage-client-5: Reconfigued transport.keepalivecnt=9 [2019-07-19 15:01:17.494032] D [socket.c:4167:ssl_setup_connection_params] 0-storage-client-5: SSL support on the I/O path is NOT enabled [2019-07-19 15:01:17.494037] D [socket.c:4170:ssl_setup_connection_params] 0-storage-client-5: SSL support for glusterd is NOT enabled [2019-07-19 15:01:17.494044] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-storage-client-5: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-07-19 15:01:17.494051] D [MSGID: 0] [client.c:2543:client_init_rpc] 0-storage-client-5: client init successful [2019-07-19 15:01:17.494077] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-storage-client-4: option ping-timeout using set value 42 [2019-07-19 15:01:17.494087] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-storage-client-4: option remote-subvolume using set value /mnt/raid6-storage/storage [2019-07-19 15:01:17.494103] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-client-4: option send-gids using set value true [2019-07-19 15:01:17.494112] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-storage-client-4: defaulting frame-timeout to 30mins [2019-07-19 15:01:17.494118] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-storage-client-4: setting ping-timeout to 42 [2019-07-19 15:01:17.494126] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.3/rpc-transport/socket.so [2019-07-19 15:01:17.494214] D [socket.c:4464:socket_init] 0-storage-client-4: Configued transport.tcp-user-timeout=0 [2019-07-19 15:01:17.494222] D [socket.c:4482:socket_init] 0-storage-client-4: Reconfigued transport.keepalivecnt=9 [2019-07-19 15:01:17.494228] D [socket.c:4167:ssl_setup_connection_params] 0-storage-client-4: SSL support on the I/O path is NOT enabled [2019-07-19 15:01:17.494233] D [socket.c:4170:ssl_setup_connection_params] 0-storage-client-4: SSL support for glusterd is NOT enabled [2019-07-19 15:01:17.494239] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-storage-client-4: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-07-19 15:01:17.494249] D [MSGID: 0] [client.c:2543:client_init_rpc] 0-storage-client-4: client init successful [2019-07-19 15:01:17.494274] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-storage-client-3: option ping-timeout using set value 42 [2019-07-19 15:01:17.494283] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-storage-client-3: option remote-subvolume using set value /mnt/raid6-storage/storage [2019-07-19 15:01:17.494297] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-client-3: option send-gids using set value true [2019-07-19 15:01:17.494312] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-storage-client-3: defaulting frame-timeout to 30mins [2019-07-19 15:01:17.494318] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-storage-client-3: setting ping-timeout to 42 [2019-07-19 15:01:17.494327] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.3/rpc-transport/socket.so [2019-07-19 15:01:17.494444] D [socket.c:4464:socket_init] 0-storage-client-3: Configued transport.tcp-user-timeout=0 [2019-07-19 15:01:17.494454] D [socket.c:4482:socket_init] 0-storage-client-3: Reconfigued transport.keepalivecnt=9 [2019-07-19 15:01:17.494460] D [socket.c:4167:ssl_setup_connection_params] 0-storage-client-3: SSL support on the I/O path is NOT enabled [2019-07-19 15:01:17.494466] D [socket.c:4170:ssl_setup_connection_params] 0-storage-client-3: SSL support for glusterd is NOT enabled [2019-07-19 15:01:17.494473] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-storage-client-3: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-07-19 15:01:17.494479] D [MSGID: 0] [client.c:2543:client_init_rpc] 0-storage-client-3: client init successful [2019-07-19 15:01:17.494520] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-storage-client-2: option ping-timeout using set value 42 [2019-07-19 15:01:17.494532] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-storage-client-2: option remote-subvolume using set value /mnt/raid6-storage/storage [2019-07-19 15:01:17.494545] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-client-2: option send-gids using set value true [2019-07-19 15:01:17.494556] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-storage-client-2: defaulting frame-timeout to 30mins [2019-07-19 15:01:17.494562] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-storage-client-2: setting ping-timeout to 42 [2019-07-19 15:01:17.494585] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.3/rpc-transport/socket.so [2019-07-19 15:01:17.494683] D [socket.c:4464:socket_init] 0-storage-client-2: Configued transport.tcp-user-timeout=0 [2019-07-19 15:01:17.494692] D [socket.c:4482:socket_init] 0-storage-client-2: Reconfigued transport.keepalivecnt=9 [2019-07-19 15:01:17.494699] D [socket.c:4167:ssl_setup_connection_params] 0-storage-client-2: SSL support on the I/O path is NOT enabled [2019-07-19 15:01:17.494709] D [socket.c:4170:ssl_setup_connection_params] 0-storage-client-2: SSL support for glusterd is NOT enabled [2019-07-19 15:01:17.494717] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-storage-client-2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-07-19 15:01:17.494725] D [MSGID: 0] [client.c:2543:client_init_rpc] 0-storage-client-2: client init successful [2019-07-19 15:01:17.494757] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-storage-client-1: option ping-timeout using set value 42 [2019-07-19 15:01:17.494770] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-storage-client-1: option remote-subvolume using set value /mnt/raid6-storage/storage [2019-07-19 15:01:17.494784] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-client-1: option send-gids using set value true [2019-07-19 15:01:17.494792] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-storage-client-1: defaulting frame-timeout to 30mins [2019-07-19 15:01:17.494800] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-storage-client-1: setting ping-timeout to 42 [2019-07-19 15:01:17.494811] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.3/rpc-transport/socket.so [2019-07-19 15:01:17.494904] D [socket.c:4464:socket_init] 0-storage-client-1: Configued transport.tcp-user-timeout=0 [2019-07-19 15:01:17.494912] D [socket.c:4482:socket_init] 0-storage-client-1: Reconfigued transport.keepalivecnt=9 [2019-07-19 15:01:17.494918] D [socket.c:4167:ssl_setup_connection_params] 0-storage-client-1: SSL support on the I/O path is NOT enabled [2019-07-19 15:01:17.494924] D [socket.c:4170:ssl_setup_connection_params] 0-storage-client-1: SSL support for glusterd is NOT enabled [2019-07-19 15:01:17.494930] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-storage-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-07-19 15:01:17.494945] D [MSGID: 0] [client.c:2543:client_init_rpc] 0-storage-client-1: client init successful [2019-07-19 15:01:17.494997] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-storage-client-0: option ping-timeout using set value 42 [2019-07-19 15:01:17.495012] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-storage-client-0: option remote-subvolume using set value /mnt/raid6-storage/storage [2019-07-19 15:01:17.495028] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-storage-client-0: option send-gids using set value true [2019-07-19 15:01:17.495042] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-storage-client-0: defaulting frame-timeout to 30mins [2019-07-19 15:01:17.495050] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-storage-client-0: setting ping-timeout to 42 [2019-07-19 15:01:17.495058] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.3/rpc-transport/socket.so [2019-07-19 15:01:17.495164] D [socket.c:4464:socket_init] 0-storage-client-0: Configued transport.tcp-user-timeout=0 [2019-07-19 15:01:17.495173] D [socket.c:4482:socket_init] 0-storage-client-0: Reconfigued transport.keepalivecnt=9 [2019-07-19 15:01:17.495179] D [socket.c:4167:ssl_setup_connection_params] 0-storage-client-0: SSL support on the I/O path is NOT enabled [2019-07-19 15:01:17.495185] D [socket.c:4170:ssl_setup_connection_params] 0-storage-client-0: SSL support for glusterd is NOT enabled [2019-07-19 15:01:17.495192] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-storage-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2019-07-19 15:01:17.495198] D [MSGID: 0] [client.c:2543:client_init_rpc] 0-storage-client-0: client init successful [2019-07-19 15:01:17.495235] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-6: option 'transport.address-family' is not recognized [2019-07-19 15:01:17.495247] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-6: option 'transport.tcp-user-timeout' is not recognized [2019-07-19 15:01:17.495261] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-6: option 'transport.socket.keepalive-time' is not recognized [2019-07-19 15:01:17.495272] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-6: option 'transport.socket.keepalive-interval' is not recognized [2019-07-19 15:01:17.495282] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-6: option 'transport.socket.keepalive-count' is not recognized [2019-07-19 15:01:17.495298] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-5: option 'transport.address-family' is not recognized [2019-07-19 15:01:17.495311] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-5: option 'transport.tcp-user-timeout' is not recognized [2019-07-19 15:01:17.495321] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-5: option 'transport.socket.keepalive-time' is not recognized [2019-07-19 15:01:17.495336] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-5: option 'transport.socket.keepalive-interval' is not recognized [2019-07-19 15:01:17.495347] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-5: option 'transport.socket.keepalive-count' is not recognized [2019-07-19 15:01:17.495362] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-4: option 'transport.address-family' is not recognized [2019-07-19 15:01:17.495372] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-4: option 'transport.tcp-user-timeout' is not recognized [2019-07-19 15:01:17.495385] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-4: option 'transport.socket.keepalive-time' is not recognized [2019-07-19 15:01:17.495395] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-4: option 'transport.socket.keepalive-interval' is not recognized [2019-07-19 15:01:17.495404] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-4: option 'transport.socket.keepalive-count' is not recognized [2019-07-19 15:01:17.495419] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-3: option 'transport.address-family' is not recognized [2019-07-19 15:01:17.495437] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-3: option 'transport.tcp-user-timeout' is not recognized [2019-07-19 15:01:17.495449] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-3: option 'transport.socket.keepalive-time' is not recognized [2019-07-19 15:01:17.495459] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-3: option 'transport.socket.keepalive-interval' is not recognized [2019-07-19 15:01:17.495472] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-3: option 'transport.socket.keepalive-count' is not recognized [2019-07-19 15:01:17.495489] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-2: option 'transport.address-family' is not recognized [2019-07-19 15:01:17.495499] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-2: option 'transport.tcp-user-timeout' is not recognized [2019-07-19 15:01:17.495508] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-2: option 'transport.socket.keepalive-time' is not recognized [2019-07-19 15:01:17.495519] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-2: option 'transport.socket.keepalive-interval' is not recognized [2019-07-19 15:01:17.495529] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-2: option 'transport.socket.keepalive-count' is not recognized [2019-07-19 15:01:17.495548] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-1: option 'transport.address-family' is not recognized [2019-07-19 15:01:17.495558] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-1: option 'transport.tcp-user-timeout' is not recognized [2019-07-19 15:01:17.495568] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-1: option 'transport.socket.keepalive-time' is not recognized [2019-07-19 15:01:17.495578] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-1: option 'transport.socket.keepalive-interval' is not recognized [2019-07-19 15:01:17.495588] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-1: option 'transport.socket.keepalive-count' is not recognized [2019-07-19 15:01:17.495603] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-0: option 'transport.address-family' is not recognized [2019-07-19 15:01:17.495613] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-0: option 'transport.tcp-user-timeout' is not recognized [2019-07-19 15:01:17.495624] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-0: option 'transport.socket.keepalive-time' is not recognized [2019-07-19 15:01:17.495642] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-0: option 'transport.socket.keepalive-interval' is not recognized [2019-07-19 15:01:17.495658] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-storage-client-0: option 'transport.socket.keepalive-count' is not recognized [2019-07-19 15:01:17.495673] D [fuse-bridge.c:5332:notify] 0-fuse: got event 12 on graph 0 [2019-07-19 15:01:17.495691] I [MSGID: 114020] [client.c:2354:notify] 0-storage-client-0: parent translators are ready, attempting connect on transport [2019-07-19 15:01:17.498183] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.50 (port-24007) for hostname: 10.0.231.50 and port: 24007 [2019-07-19 15:01:17.498203] D [socket.c:3221:socket_fix_ssl_opts] 0-storage-client-0: disabling SSL for portmapper connection [2019-07-19 15:01:17.498375] I [MSGID: 114020] [client.c:2354:notify] 0-storage-client-1: parent translators are ready, attempting connect on transport [2019-07-19 15:01:17.498496] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-0: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.498905] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.50:24007: ping timer event already removed [2019-07-19 15:01:17.498985] D [MSGID: 0] [client-handshake.c:1394:server_has_portmap] 0-storage-client-0: detected portmapper on server [2019-07-19 15:01:17.499085] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-0: Ping latency is 0ms [2019-07-19 15:01:17.499161] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-storage-client-0: changing port to 49154 (from 0) [2019-07-19 15:01:17.499192] D [socket.c:2927:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:13) (non-SSL) [2019-07-19 15:01:17.499207] D [MSGID: 0] [client.c:2271:client_rpc_notify] 0-storage-client-0: got RPC_CLNT_DISCONNECT [2019-07-19 15:01:17.499221] D [MSGID: 0] [client.c:2312:client_rpc_notify] 0-storage-client-0: disconnected (skipped notify) [2019-07-19 15:01:17.500802] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.51 (port-24007) for hostname: 10.0.231.51 and port: 24007 [2019-07-19 15:01:17.500818] D [socket.c:3221:socket_fix_ssl_opts] 0-storage-client-1: disabling SSL for portmapper connection [2019-07-19 15:01:17.500930] I [MSGID: 114020] [client.c:2354:notify] 0-storage-client-2: parent translators are ready, attempting connect on transport [2019-07-19 15:01:17.503047] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.50 (port-24007) for hostname: 10.0.231.50 and port: 24007 [2019-07-19 15:01:17.503434] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-07-19 15:01:17.503486] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-1: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.503694] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.51:24007: ping timer event already removed [2019-07-19 15:01:17.503733] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-0: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.503888] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.50:49154: ping timer event already removed [2019-07-19 15:01:17.503936] D [MSGID: 0] [client-handshake.c:1394:server_has_portmap] 0-storage-client-1: detected portmapper on server [2019-07-19 15:01:17.504040] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-1: Ping latency is 0ms [2019-07-19 15:01:17.504065] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-0: Ping latency is 0ms [2019-07-19 15:01:17.504081] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-storage-client-1: changing port to 49155 (from 0) [2019-07-19 15:01:17.504100] D [socket.c:2927:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:15) (non-SSL) [2019-07-19 15:01:17.504111] D [MSGID: 0] [client.c:2271:client_rpc_notify] 0-storage-client-1: got RPC_CLNT_DISCONNECT [2019-07-19 15:01:17.504120] D [MSGID: 0] [client.c:2312:client_rpc_notify] 0-storage-client-1: disconnected (skipped notify) [2019-07-19 15:01:17.505440] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.52 (port-24007) for hostname: 10.0.231.52 and port: 24007 [2019-07-19 15:01:17.505456] D [socket.c:3221:socket_fix_ssl_opts] 0-storage-client-2: disabling SSL for portmapper connection [2019-07-19 15:01:17.505564] I [MSGID: 114020] [client.c:2354:notify] 0-storage-client-3: parent translators are ready, attempting connect on transport [2019-07-19 15:01:17.507649] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.51 (port-24007) for hostname: 10.0.231.51 and port: 24007 [2019-07-19 15:01:17.507787] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-07-19 15:01:17.507839] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-storage-client-0: Connected to storage-client-0, attached to remote volume '/mnt/raid6-storage/storage'. [2019-07-19 15:01:17.507855] D [MSGID: 0] [client-handshake.c:946:client_post_handshake] 0-storage-client-0: No fds to open - notifying all parents child up [2019-07-19 15:01:17.507925] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-2: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.508103] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.52:24007: ping timer event already removed [2019-07-19 15:01:17.508141] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-1: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.508277] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.51:49155: ping timer event already removed [2019-07-19 15:01:17.508319] D [MSGID: 0] [client-handshake.c:1394:server_has_portmap] 0-storage-client-2: detected portmapper on server [2019-07-19 15:01:17.508370] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-storage-dht: subvolume 'storage-client-0': avail_percent is: 24.00 and avail_space is: 9879221805056 and avail_inodes is: 99.00 [2019-07-19 15:01:17.508430] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-2: Ping latency is 0ms [2019-07-19 15:01:17.508452] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-1: Ping latency is 0ms [2019-07-19 15:01:17.508467] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-storage-client-2: changing port to 49155 (from 0) [2019-07-19 15:01:17.508486] D [socket.c:2927:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:14) (non-SSL) [2019-07-19 15:01:17.508497] D [MSGID: 0] [client.c:2271:client_rpc_notify] 0-storage-client-2: got RPC_CLNT_DISCONNECT [2019-07-19 15:01:17.508505] D [MSGID: 0] [client.c:2312:client_rpc_notify] 0-storage-client-2: disconnected (skipped notify) [2019-07-19 15:01:17.510077] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.53 (port-24007) for hostname: 10.0.231.53 and port: 24007 [2019-07-19 15:01:17.510095] D [socket.c:3221:socket_fix_ssl_opts] 0-storage-client-3: disabling SSL for portmapper connection [2019-07-19 15:01:17.510202] I [MSGID: 114020] [client.c:2354:notify] 0-storage-client-4: parent translators are ready, attempting connect on transport [2019-07-19 15:01:17.512424] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.52 (port-24007) for hostname: 10.0.231.52 and port: 24007 [2019-07-19 15:01:17.512574] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-07-19 15:01:17.512621] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-storage-client-1: Connected to storage-client-1, attached to remote volume '/mnt/raid6-storage/storage'. [2019-07-19 15:01:17.512640] D [MSGID: 0] [client-handshake.c:946:client_post_handshake] 0-storage-client-1: No fds to open - notifying all parents child up [2019-07-19 15:01:17.512697] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-3: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.512895] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.53:24007: ping timer event already removed [2019-07-19 15:01:17.512934] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-2: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.513066] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.52:49155: ping timer event already removed [2019-07-19 15:01:17.513115] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-storage-dht: subvolume 'storage-client-1': avail_percent is: 23.00 and avail_space is: 9383555014656 and avail_inodes is: 99.00 [2019-07-19 15:01:17.513141] D [MSGID: 0] [client-handshake.c:1394:server_has_portmap] 0-storage-client-3: detected portmapper on server [2019-07-19 15:01:17.513204] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-3: Ping latency is 0ms [2019-07-19 15:01:17.513226] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-2: Ping latency is 0ms [2019-07-19 15:01:17.513253] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-storage-client-3: changing port to 49153 (from 0) [2019-07-19 15:01:17.513273] D [socket.c:2927:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:15) (non-SSL) [2019-07-19 15:01:17.513283] D [MSGID: 0] [client.c:2271:client_rpc_notify] 0-storage-client-3: got RPC_CLNT_DISCONNECT [2019-07-19 15:01:17.513292] D [MSGID: 0] [client.c:2312:client_rpc_notify] 0-storage-client-3: disconnected (skipped notify) [2019-07-19 15:01:17.515002] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.54 (port-24007) for hostname: 10.0.231.54 and port: 24007 [2019-07-19 15:01:17.515021] D [socket.c:3221:socket_fix_ssl_opts] 0-storage-client-4: disabling SSL for portmapper connection [2019-07-19 15:01:17.515159] I [MSGID: 114020] [client.c:2354:notify] 0-storage-client-5: parent translators are ready, attempting connect on transport [2019-07-19 15:01:17.517280] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.53 (port-24007) for hostname: 10.0.231.53 and port: 24007 [2019-07-19 15:01:17.517434] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-07-19 15:01:17.517481] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-storage-client-2: Connected to storage-client-2, attached to remote volume '/mnt/raid6-storage/storage'. [2019-07-19 15:01:17.517493] D [MSGID: 0] [client-handshake.c:946:client_post_handshake] 0-storage-client-2: No fds to open - notifying all parents child up [2019-07-19 15:01:17.517540] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-4: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.517730] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.54:24007: ping timer event already removed [2019-07-19 15:01:17.517780] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-3: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.517906] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.53:49153: ping timer event already removed [2019-07-19 15:01:17.517948] D [MSGID: 0] [client-handshake.c:1394:server_has_portmap] 0-storage-client-4: detected portmapper on server [2019-07-19 15:01:17.517994] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-storage-dht: subvolume 'storage-client-2': avail_percent is: 23.00 and avail_space is: 9500197941248 and avail_inodes is: 99.00 [2019-07-19 15:01:17.518053] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-4: Ping latency is 0ms [2019-07-19 15:01:17.518075] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-3: Ping latency is 0ms [2019-07-19 15:01:17.518090] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-storage-client-4: changing port to 49153 (from 0) [2019-07-19 15:01:17.518109] D [socket.c:2927:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:14) (non-SSL) [2019-07-19 15:01:17.518120] D [MSGID: 0] [client.c:2271:client_rpc_notify] 0-storage-client-4: got RPC_CLNT_DISCONNECT [2019-07-19 15:01:17.518128] D [MSGID: 0] [client.c:2312:client_rpc_notify] 0-storage-client-4: disconnected (skipped notify) [2019-07-19 15:01:17.519815] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.55 (port-24007) for hostname: 10.0.231.55 and port: 24007 [2019-07-19 15:01:17.519833] D [socket.c:3221:socket_fix_ssl_opts] 0-storage-client-5: disabling SSL for portmapper connection [2019-07-19 15:01:17.519952] I [MSGID: 114020] [client.c:2354:notify] 0-storage-client-6: parent translators are ready, attempting connect on transport [2019-07-19 15:01:17.522022] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.54 (port-24007) for hostname: 10.0.231.54 and port: 24007 [2019-07-19 15:01:17.522159] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-07-19 15:01:17.522200] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-storage-client-3: Connected to storage-client-3, attached to remote volume '/mnt/raid6-storage/storage'. [2019-07-19 15:01:17.522210] D [MSGID: 0] [client-handshake.c:946:client_post_handshake] 0-storage-client-3: No fds to open - notifying all parents child up [2019-07-19 15:01:17.522285] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-5: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.522463] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.55:24007: ping timer event already removed [2019-07-19 15:01:17.522500] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-4: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.522621] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.54:49153: ping timer event already removed [2019-07-19 15:01:17.522667] D [MSGID: 0] [client-handshake.c:1394:server_has_portmap] 0-storage-client-5: detected portmapper on server [2019-07-19 15:01:17.522706] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-storage-dht: subvolume 'storage-client-3': avail_percent is: 23.00 and avail_space is: 9281959972864 and avail_inodes is: 99.00 [2019-07-19 15:01:17.522770] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-5: Ping latency is 0ms [2019-07-19 15:01:17.522792] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-4: Ping latency is 0ms [2019-07-19 15:01:17.522806] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-storage-client-5: changing port to 49155 (from 0) [2019-07-19 15:01:17.522825] D [socket.c:2927:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:15) (non-SSL) [2019-07-19 15:01:17.522836] D [MSGID: 0] [client.c:2271:client_rpc_notify] 0-storage-client-5: got RPC_CLNT_DISCONNECT [2019-07-19 15:01:17.522845] D [MSGID: 0] [client.c:2312:client_rpc_notify] 0-storage-client-5: disconnected (skipped notify) [2019-07-19 15:01:17.524583] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.56 (port-24007) for hostname: 10.0.231.56 and port: 24007 [2019-07-19 15:01:17.524601] D [socket.c:3221:socket_fix_ssl_opts] 0-storage-client-6: disabling SSL for portmapper connection Final graph: +------------------------------------------------------------------------------+ 1: volume storage-client-0 2: type protocol/client 3: option opversion 50000 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key /storage 7: option client-version 5.3 8: option process-name fuse 9: option process-uuid CTX_ID:d0875391-ca96-40d7-9350-63f487ef1210-GRAPH_ID:0-PID:23336-HOST:gluster07.pcic.uvic.ca-PC_NAME:storage-client-0-RECON_NO:-0 10: option fops-version 1298437 11: option ping-timeout 42 12: option remote-host 10.0.231.50 13: option remote-subvolume /mnt/raid6-storage/storage 14: option transport-type socket 15: option transport.address-family inet 16: option username affc2c53-f158-48fa-8191-2ae39ced889e 17: option password e4177b59-b486-4551-ba0c-d5768a3be3e0 18: option transport.tcp-user-timeout 0 19: option transport.socket.keepalive-time 20 20: option transport.socket.keepalive-interval 2 21: option transport.socket.keepalive-count 9 22: option send-gids true 23: end-volume 24: 25: volume storage-client-1 26: type protocol/client 27: option opversion 50000 28: option clnt-lk-version 1 29: option volfile-checksum 0 30: option volfile-key /storage 31: option client-version 5.3 32: option process-name fuse 33: option process-uuid CTX_ID:d0875391-ca96-40d7-9350-63f487ef1210-GRAPH_ID:0-PID:23336-HOST:gluster07.pcic.uvic.ca-PC_NAME:storage-client-1-RECON_NO:-0 34: option fops-version 1298437 35: option ping-timeout 42 36: option remote-host 10.0.231.51 37: option remote-subvolume /mnt/raid6-storage/storage 38: option transport-type socket 39: option transport.address-family inet 40: option username affc2c53-f158-48fa-8191-2ae39ced889e 41: option password e4177b59-b486-4551-ba0c-d5768a3be3e0 42: option transport.tcp-user-timeout 0 43: option transport.socket.keepalive-time 20 44: option transport.socket.keepalive-interval 2 45: option transport.socket.keepalive-count 9 46: option send-gids true 47: end-volume 48: 49: volume storage-client-2 50: type protocol/client 51: option opversion 50000 52: option clnt-lk-version 1 53: option volfile-checksum 0 54: option volfile-key /storage 55: option client-version 5.3 56: option process-name fuse 57: option process-uuid CTX_ID:d0875391-ca96-40d7-9350-63f487ef1210-GRAPH_ID:0-PID:23336-HOST:gluster07.pcic.uvic.ca-PC_NAME:storage-client-2-RECON_NO:-0 58: option fops-version 1298437 59: option ping-timeout 42 60: option remote-host 10.0.231.52 61: option remote-subvolume /mnt/raid6-storage/storage 62: option transport-type socket 63: option transport.address-family inet 64: option username affc2c53-f158-48fa-8191-2ae39ced889e 65: option password e4177b59-b486-4551-ba0c-d5768a3be3e0 66: option transport.tcp-user-timeout 0 67: option transport.socket.keepalive-time 20 68: option transport.socket.keepalive-interval 2 69: option transport.socket.keepalive-count 9 70: option send-gids true 71: end-volume 72: 73: volume storage-client-3 74: type protocol/client 75: option opversion 50000 76: option clnt-lk-version 1 77: option volfile-checksum 0 78: option volfile-key /storage 79: option client-version 5.3 80: option process-name fuse 81: option process-uuid CTX_ID:d0875391-ca96-40d7-9350-63f487ef1210-GRAPH_ID:0-PID:23336-HOST:gluster07.pcic.uvic.ca-PC_NAME:storage-client-3-RECON_NO:-0 82: option fops-version 1298437 83: option ping-timeout 42 84: option remote-host 10.0.231.53 85: option remote-subvolume /mnt/raid6-storage/storage 86: option transport-type socket 87: option transport.address-family inet 88: option username affc2c53-f158-48fa-8191-2ae39ced889e 89: option password e4177b59-b486-4551-ba0c-d5768a3be3e0 90: option transport.tcp-user-timeout 0 91: option transport.socket.keepalive-time 20 92: option transport.socket.keepalive-interval 2 93: option transport.socket.keepalive-count 9 94: option send-gids true 95: end-volume 96: 97: volume storage-client-4 98: type protocol/client 99: option opversion 50000 100: option clnt-lk-version 1 101: option volfile-checksum 0 102: option volfile-key /storage 103: option client-version 5.3 104: option process-name fuse 105: option process-uuid CTX_ID:d0875391-ca96-40d7-9350-63f487ef1210-GRAPH_ID:0-PID:23336-HOST:gluster07.pcic.uvic.ca-PC_NAME:storage-client-4-RECON_NO:-0 106: option fops-version 1298437 107: option ping-timeout 42 108: option remote-host 10.0.231.54 109: option remote-subvolume /mnt/raid6-storage/storage 110: option transport-type socket 111: option transport.address-family inet 112: option username affc2c53-f158-48fa-8191-2ae39ced889e 113: option password e4177b59-b486-4551-ba0c-d5768a3be3e0 114: option transport.tcp-user-timeout 0 115: option transport.socket.keepalive-time 20 116: option transport.socket.keepalive-interval 2 117: option transport.socket.keepalive-count 9 118: option send-gids true 119: end-volume 120: 121: volume storage-client-5 122: type protocol/client 123: option ping-timeout 42 124: option remote-host 10.0.231.55 125: option remote-subvolume /mnt/raid6-storage/storage 126: option transport-type socket 127: option transport.address-family inet 128: option username affc2c53-f158-48fa-8191-2ae39ced889e 129: option password e4177b59-b486-4551-ba0c-d5768a3be3e0 130: option transport.tcp-user-timeout 0 131: option transport.socket.keepalive-time 20 132: option transport.socket.keepalive-interval 2 133: option transport.socket.keepalive-count 9 134: option send-gids true 135: end-volume 136: 137: volume storage-client-6 138: type protocol/client 139: option ping-timeout 42 140: option remote-host 10.0.231.56 141: option remote-subvolume /mnt/raid6-storage/storage 142: option transport-type socket 143: option transport.address-family inet 144: option username affc2c53-f158-48fa-8191-2ae39ced889e 145: option password e4177b59-b486-4551-ba0c-d5768a3be3e0 146: option transport.tcp-user-timeout 0 147: option transport.socket.keepalive-time 20 148: option transport.socket.keepalive-interval 2 149: option transport.socket.keepalive-count 9 150: option send-gids true 151: end-volume 152: 153: volume storage-dht 154: type cluster/distribute 155: option lock-migration off 156: option force-migration off 157: subvolumes storage-client-0 storage-client-1 storage-client-2 storage-client-3 storage-client-4 storage-client-5 storage-client-6 158: end-volume 159: 160: volume storage-write-behind 161: type performance/write-behind 162: subvolumes storage-dht 163: end-volume 164: 165: volume storage-read-ahead 166: type performance/read-ahead 167: subvolumes storage-write-behind 168: end-volume 169: 170: volume storage-readdir-ahead 171: type performance/readdir-ahead 172: option parallel-readdir off 173: option rda-request-size 131072 174: option rda-cache-limit 10MB 175: subvolumes storage-read-ahead 176: end-volume 177: 178: volume storage-io-cache 179: type performance/io-cache 180: subvolumes storage-readdir-ahead 181: end-volume 182: 183: volume storage-quick-read 184: type performance/quick-read 185: subvolumes storage-io-cache 186: end-volume 187: 188: volume storage-open-behind 189: type performance/open-behind 190: subvolumes storage-quick-read 191: end-volume 192: 193: volume storage-md-cache 194: type performance/md-cache 195: subvolumes storage-open-behind 196: end-volume 197: 198: volume storage-io-threads 199: type performance/io-threads 200: subvolumes storage-md-cache 201: end-volume 202: 203: volume storage 204: type debug/io-stats 205: option log-level DEBUG 206: option latency-measurement off 207: option count-fop-hits off 208: subvolumes storage-io-threads 209: end-volume 210: 211: volume meta-autoload 212: type meta 213: subvolumes storage 214: end-volume 215: +------------------------------------------------------------------------------+ [2019-07-19 15:01:17.525571] D [glusterfsd-mgmt.c:2839:glusterfs_mgmt_pmap_signin] 0-fsd-mgmt: portmapper signin arguments not given [2019-07-19 15:01:17.525624] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-storage-client-4: Connected to storage-client-4, attached to remote volume '/mnt/raid6-storage/storage'. [2019-07-19 15:01:17.525646] D [MSGID: 0] [client-handshake.c:946:client_post_handshake] 0-storage-client-4: No fds to open - notifying all parents child up [2019-07-19 15:01:17.525724] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-6: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.525919] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.56:24007: ping timer event already removed [2019-07-19 15:01:17.525988] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-storage-dht: subvolume 'storage-client-4': avail_percent is: 25.00 and avail_space is: 10120701075456 and avail_inodes is: 99.00 [2019-07-19 15:01:17.526024] D [MSGID: 0] [client-handshake.c:1394:server_has_portmap] 0-storage-client-6: detected portmapper on server [2019-07-19 15:01:17.526080] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-6: Ping latency is 0ms [2019-07-19 15:01:17.526127] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-storage-client-6: changing port to 49154 (from 0) [2019-07-19 15:01:17.526162] D [socket.c:2927:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:14) (non-SSL) [2019-07-19 15:01:17.526180] D [MSGID: 0] [client.c:2271:client_rpc_notify] 0-storage-client-6: got RPC_CLNT_DISCONNECT [2019-07-19 15:01:17.526194] D [MSGID: 0] [client.c:2312:client_rpc_notify] 0-storage-client-6: disconnected (skipped notify) [2019-07-19 15:01:17.527075] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.55 (port-24007) for hostname: 10.0.231.55 and port: 24007 [2019-07-19 15:01:17.527208] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-07-19 15:01:17.527287] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-5: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.527439] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.55:49155: ping timer event already removed [2019-07-19 15:01:17.527521] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-5: Ping latency is 0ms [2019-07-19 15:01:17.528010] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-storage-client-5: Connected to storage-client-5, attached to remote volume '/mnt/raid6-storage/storage'. [2019-07-19 15:01:17.528024] D [MSGID: 0] [client-handshake.c:946:client_post_handshake] 0-storage-client-5: No fds to open - notifying all parents child up [2019-07-19 15:01:17.528193] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-storage-dht: subvolume 'storage-client-5': avail_percent is: 24.00 and avail_space is: 9646427553792 and avail_inodes is: 99.00 [2019-07-19 15:01:17.529409] D [MSGID: 0] [common-utils.c:535:gf_resolve_ip6] 0-resolver: returning ip-10.0.231.56 (port-24007) for hostname: 10.0.231.56 and port: 24007 [2019-07-19 15:01:17.529578] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-07-19 15:01:17.529592] D [MSGID: 0] [client.c:2260:client_rpc_notify] 0-storage-client-6: got RPC_CLNT_CONNECT [2019-07-19 15:01:17.529785] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fa4e29eafbb] (--> /lib64/libgfrpc.so.0(+0x125cb)[0x7fa4e27b95cb] (--> /lib64/libgfrpc.so.0(+0x12d91)[0x7fa4e27b9d91] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7fa4e27b660b] (--> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x13f99)[0x7fa4d4e5cf99] ))))) 0-: 10.0.231.56:49154: ping timer event already removed [2019-07-19 15:01:17.529851] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-storage-client-6: Ping latency is 0ms [2019-07-19 15:01:17.530328] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-storage-client-6: Connected to storage-client-6, attached to remote volume '/mnt/raid6-storage/storage'. [2019-07-19 15:01:17.530343] D [MSGID: 0] [client-handshake.c:946:client_post_handshake] 0-storage-client-6: No fds to open - notifying all parents child up [2019-07-19 15:01:17.530404] D [fuse-bridge.c:5332:notify] 0-fuse: got event 5 on graph 0 [2019-07-19 15:01:17.530559] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-storage-dht: subvolume 'storage-client-6': avail_percent is: 23.00 and avail_space is: 9361142747136 and avail_inodes is: 99.00 [2019-07-19 15:01:17.532232] D [fuse-bridge.c:4919:fuse_get_mount_status] 0-fuse: mount status is 0 [2019-07-19 15:01:17.532356] D [fuse-bridge.c:4206:fuse_init] 0-glusterfs-fuse: Detected support for FUSE_AUTO_INVAL_DATA. Enabling fopen_keep_cache automatically. [2019-07-19 15:01:17.532384] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2019-07-19 15:01:17.532397] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0 [2019-07-19 15:01:17.532455] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-storage-io-threads: LOOKUP scheduled as fast fop [2019-07-19 15:01:17.532580] D [MSGID: 0] [dht-common.c:3454:dht_do_fresh_lookup] 0-storage-dht: Calling fresh lookup for / on storage-client-0 [2019-07-19 15:01:17.533478] D [MSGID: 0] [dht-common.c:3013:dht_lookup_cbk] 0-storage-dht: fresh_lookup returned for / with op_ret 0 [2019-07-19 15:01:17.534244] D [MSGID: 0] [dht-common.c:1354:dht_lookup_dir_cbk] 0-storage-dht: Internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:17.534805] D [logging.c:1998:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [dht-common.c:1354:dht_lookup_dir_cbk] 0-storage-dht: Internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 " repeated 6 times between [2019-07-19 15:01:17.534244] and [2019-07-19 15:01:17.534563] [2019-07-19 15:01:17.534804] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-storage-io-threads: LOOKUP scheduled as fast fop [2019-07-19 15:01:17.535444] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:17.535511] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:17.535552] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:17.535580] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:17.535664] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:17.535680] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:17.535697] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:17.535714] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:17.535793] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:17.535813] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:17.535834] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:17.535846] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:17.535859] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:17.535875] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:19.371655] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-storage-io-threads: LOOKUP scheduled as fast fop [2019-07-19 15:01:19.372470] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:19.372505] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:19.372528] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:19.372576] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:19.372605] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:19.372621] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:19.372701] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:19.372719] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:19.372752] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:19.372773] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:19.372828] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:19.372841] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:19.372857] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:19.372876] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:20.898527] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-storage-io-threads: LOOKUP scheduled as fast fop [2019-07-19 15:01:20.899369] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:20.899400] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:20.899448] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:20.899488] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:20.899512] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:20.899534] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:20.899562] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:20.899596] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:20.899634] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:20.899648] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:20.899753] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:20.899768] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:20.899787] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:20.899809] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:27.358805] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-storage-io-threads: LOOKUP scheduled as fast fop [2019-07-19 15:01:27.359673] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:27.359705] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:27.359741] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:27.359771] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:27.359834] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:27.359852] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:27.359962] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:27.359991] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:27.360036] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:27.360051] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:27.360088] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:27.360102] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:27.361112] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-storage-dht: revalidate lookup of / returned with op_ret 0 [2019-07-19 15:01:27.361130] D [MSGID: 0] [dht-common.c:1651:dht_revalidate_cbk] 0-storage-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-07-19 15:01:27.361207] D [MSGID: 0] [io-threads.c:372:iot_schedule] 0-storage-io-threads: STATFS scheduled as fast fop [2019-07-19 15:01:27.455734] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2019-07-19 15:01:27.455790] D [rpc-clnt-ping.c:302:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2019-07-19 15:01:27.542660] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2019-07-19 15:01:27.542721] D [rpc-clnt-ping.c:302:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2019-07-19 15:01:27.550102] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/protocol/client.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550336] D [logging.c:1998:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/protocol/client.so: undefined symbol: xlator_api. Fall back to old symbols" repeated 6 times between [2019-07-19 15:01:27.550102] and [2019-07-19 15:01:27.550306] [2019-07-19 15:01:27.550336] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550388] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550413] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550441] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550476] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550509] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/performance/open-behind.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550541] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/performance/io-threads.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550566] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550620] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.3/xlator/meta.so: undefined symbol: xlator_api. Fall back to old symbols [2019-07-19 15:01:27.550646] D [MSGID: 0] [graph.c:847:is_graph_topology_equal] 0-glusterfsd-mgmt: graphs are equal [2019-07-19 15:01:27.550655] D [MSGID: 0] [graph.c:897:glusterfs_volfile_reconfigure] 0-glusterfsd-mgmt: Only options have changed in the new graph [2019-07-19 15:01:27.550676] I [MSGID: 0] [options.c:1240:xlator_option_reconf_int32] 0-storage-client-0: option ping-timeout using set value 42 [2019-07-19 15:01:27.550711] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-client-0: option send-gids using set value true [2019-07-19 15:01:27.550726] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-client-0: reconfigured [2019-07-19 15:01:27.550746] I [MSGID: 0] [options.c:1240:xlator_option_reconf_int32] 0-storage-client-1: option ping-timeout using set value 42 [2019-07-19 15:01:27.550776] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-client-1: option send-gids using set value true [2019-07-19 15:01:27.550790] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-client-1: reconfigured [2019-07-19 15:01:27.550808] I [MSGID: 0] [options.c:1240:xlator_option_reconf_int32] 0-storage-client-2: option ping-timeout using set value 42 [2019-07-19 15:01:27.550838] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-client-2: option send-gids using set value true [2019-07-19 15:01:27.550850] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-client-2: reconfigured [2019-07-19 15:01:27.550862] I [MSGID: 0] [options.c:1240:xlator_option_reconf_int32] 0-storage-client-3: option ping-timeout using set value 42 [2019-07-19 15:01:27.550878] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-client-3: option send-gids using set value true [2019-07-19 15:01:27.550885] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-client-3: reconfigured [2019-07-19 15:01:27.550894] I [MSGID: 0] [options.c:1240:xlator_option_reconf_int32] 0-storage-client-4: option ping-timeout using set value 42 [2019-07-19 15:01:27.550910] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-client-4: option send-gids using set value true [2019-07-19 15:01:27.550917] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-client-4: reconfigured [2019-07-19 15:01:27.550926] I [MSGID: 0] [options.c:1240:xlator_option_reconf_int32] 0-storage-client-5: option ping-timeout using set value 42 [2019-07-19 15:01:27.550942] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-client-5: option send-gids using set value true [2019-07-19 15:01:27.550949] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-client-5: reconfigured [2019-07-19 15:01:27.550973] I [MSGID: 0] [options.c:1240:xlator_option_reconf_int32] 0-storage-client-6: option ping-timeout using set value 42 [2019-07-19 15:01:27.550996] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-client-6: option send-gids using set value true [2019-07-19 15:01:27.551003] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-client-6: reconfigured [2019-07-19 15:01:27.551039] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-dht: option lock-migration using set value off [2019-07-19 15:01:27.551053] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-dht: option force-migration using set value off [2019-07-19 15:01:27.551088] D [MSGID: 0] [dht-shared.c:356:dht_init_regex] 0-storage-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2019-07-19 15:01:27.551103] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-dht: reconfigured [2019-07-19 15:01:27.551120] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-write-behind: reconfigured [2019-07-19 15:01:27.551133] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-read-ahead: reconfigured [2019-07-19 15:01:27.551142] I [MSGID: 0] [options.c:1242:xlator_option_reconf_size_uint64] 0-storage-readdir-ahead: option rda-request-size using set value 131072 [2019-07-19 15:01:27.551154] I [MSGID: 0] [options.c:1242:xlator_option_reconf_size_uint64] 0-storage-readdir-ahead: option rda-cache-limit using set value 10MB [2019-07-19 15:01:27.551163] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage-readdir-ahead: option parallel-readdir using set value off [2019-07-19 15:01:27.551172] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-readdir-ahead: reconfigured [2019-07-19 15:01:27.551245] D [logging.c:1998:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2019-07-19 15:01:27.551243] D [MSGID: 0] [io-cache.c:1595:check_cache_size_ok] 0-storage-io-cache: Max cache size is 67271393280 [2019-07-19 15:01:27.551244] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-io-cache: reconfigured [2019-07-19 15:01:27.551313] D [logging.c:1998:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2019-07-19 15:01:27.551312] D [MSGID: 0] [quick-read.c:1183:check_cache_size_ok] 0-storage-quick-read: Max cache size is 67271393280 [2019-07-19 15:01:27.551313] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-quick-read: reconfigured [2019-07-19 15:01:27.551342] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-open-behind: reconfigured [2019-07-19 15:01:27.551372] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-md-cache: reconfigured [2019-07-19 15:01:27.551394] D [MSGID: 0] [options.c:1115:xlator_reconfigure_rec] 0-storage-io-threads: reconfigured [2019-07-19 15:01:27.551405] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage: option count-fop-hits using set value off [2019-07-19 15:01:27.551415] I [MSGID: 0] [options.c:1245:xlator_option_reconf_bool] 0-storage: option latency-measurement using set value off [2019-07-19 15:01:27.551434] I [MSGID: 0] [options.c:1236:xlator_option_reconf_str] 0-storage: option log-level using set value INFO [2019-07-19 15:01:27.551475] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing From hunter86_bg at yahoo.com Fri Jul 19 17:20:45 2019 From: hunter86_bg at yahoo.com (Strahil Nikolov) Date: Fri, 19 Jul 2019 17:20:45 +0000 (UTC) Subject: [Gluster-users] Fuse vs NFS In-Reply-To: References: Message-ID: <1498228262.2281356.1563556845469@mail.yahoo.com> It depends of your needs.The Gluster's inbuilt NFS server is no longer recommended.I think you should try NFS Ganesha, which uses libgfapi communication channel.If your app can use libgfappi - you can try that approach. Best Regards,Strahil Nikolov ? ?????????, 18 ??? 2019 ?., 11:06:00 ?. ???????+3, Sudheer Singh ??????: Hi , I was doing perf testing and found out fuse mount much slower than NFS mount. I was curious to know what community recommends, mount volumes as fuse or NFS? --Thanks,Sudheer_______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbellur at redhat.com Sat Jul 20 06:42:32 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Fri, 19 Jul 2019 23:42:32 -0700 Subject: [Gluster-users] Fuse vs NFS In-Reply-To: References: Message-ID: On Thu, Jul 18, 2019 at 1:06 AM Sudheer Singh wrote: > Hi , > > I was doing perf testing and found out fuse mount much slower than NFS > mount. I was curious to know what community recommends, mount volumes as > fuse or NFS? > Performance depends on several factors like: 1. Network latency between clients & servers 2. Caching on the client-side 3. Total number of context switches and memory copies between kernel and userspace. 4. Workload that is being exercised on the mount point. 5. Resources on the client & servers. It is to be noted that with FUSE, the client talks to servers directly and in the case of NFS client, the requests reach the gluster servers through the NFS server which acts as a gateway. Hence for most operations that bypass the cache (client side cache or cache in NFS sever), NFS will have one more network hop than FUSE. If the context switch and memory copy overheads are lesser than the latency added by the additional network hop with NFS, FUSE will be faster than NFS. Typically for workloads that perform large sequential writes or reads (> 64 KB block size), you would expect the performance to be better in FUSE. For writes with small record sizes and some metadata operations that can be answered by cached content, NFS can be faster. Also to note is that with FUSE, replication, distribution, erasure coding etc. happen on the client. With NFS, all these functions happen on the server. So, the hardware resources available on client & servers also have a bearing on overall performance. Hence, in essence, the performance that you get from gluster is not just dependent on the access protocol but also on other factors like the ones alluded above. Thanks, Vijay -- > Thanks, > Sudheer > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgowtham at redhat.com Mon Jul 22 07:16:06 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Mon, 22 Jul 2019 12:46:06 +0530 Subject: [Gluster-users] "du" and "df -hT" commands output mismatch In-Reply-To: <176E4EC2-DB3E-4C33-8EB4-8D09DCB11599@cmcc.it> References: <287791D2-4775-4CB1-9AA1-9C160839C2D2@cmcc.it> <176E4EC2-DB3E-4C33-8EB4-8D09DCB11599@cmcc.it> Message-ID: Hi, Yes the above mentioned steps are right. The way to find if the crawl is still happening is to grep for quota_crawl in the processes that are still running. # ps aux | grep quota_crawl As long as this process is alive, the crawl is happening. Note: crawl does take a lot of time as well. And it happens twice. On Fri, Jul 19, 2019 at 5:42 PM Mauro Tridici wrote: > > Hi Hari, > > thank you very much for the fast answer. > I think that the we will try to solve the issue disabling and enabling quota. > So, if I understand I have to do the following actions: > > - save on my notes the current quota limits; > - disable quota using "gluster volume quota /tier2 disable? command; > - wait a while for the crawl (question: how can I understand that crawl is terminated!? how logn should I wait?); > - enable quota using "gluster volume quota /tier2 enable?; > - set again the previous quota limits. > > Is this correct? > > Many thanks for your support, > Mauro > > > > On 19 Jul 2019, at 12:48, Hari Gowtham wrote: > > Hi Mauro, > > The fsck script is the fastest way to resolve the issue. > The other way would be to disable quota and once the crawl for disable > is done, we have to enable and set the limits again. > In this way, the crawl happens twice and hence its slow. > > On Fri, Jul 19, 2019 at 3:27 PM Mauro Tridici wrote: > > > Dear All, > > I?m experiencing again a problem with gluster file system quota. > The ?df -hT /tier2/CSP/sp1? command output is different from the ?du -ms? command executed against the same folder. > > [root at s01 manual]# df -hT /tier2/CSP/sp1 > Filesystem Type Size Used Avail Use% Mounted on > s01-stg:tier2 fuse.glusterfs 25T 22T 3.5T 87% /tier2 > > [root at s01 sp1]# du -ms /tier2/CSP/sp1 > 14TB /tier2/CSP/sp1 > > In the past, I used successfully the quota_fsck_new-6.py script in order to detect the SIZE_MISMATCH occurrences and fix them. > Unfortunately, the number of sub-directories and files saved in /tier2/CSP/sp1 grew so much and the list of SIZE_MISMATCH entries is very long. > > Is there a faster way to correct the mismatching outputs? > Could you please help me to solve, if it is possible, this issue? > > Thank you in advance, > Mauro > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > -- > Regards, > Hari Gowtham. > > > -- Regards, Hari Gowtham. From mauro.tridici at cmcc.it Mon Jul 22 07:42:04 2019 From: mauro.tridici at cmcc.it (Mauro Tridici) Date: Mon, 22 Jul 2019 09:42:04 +0200 Subject: [Gluster-users] "du" and "df -hT" commands output mismatch In-Reply-To: References: <287791D2-4775-4CB1-9AA1-9C160839C2D2@cmcc.it> <176E4EC2-DB3E-4C33-8EB4-8D09DCB11599@cmcc.it> Message-ID: <05385746-58F0-4FFE-BE49-3CFED6C919A2@cmcc.it> Hi Hari, I hope that the crawl will run at most for a couple of days. Do you know if there is a way to solve the issue definitely ? GlusterFS version is 3.12.14. You can find below some additional info. Volume Name: tier2 Type: Distributed-Disperse Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c Status: Started Snapshot Count: 0 Number of Bricks: 12 x (4 + 2) = 72 Transport-type: tcp Many thanks, Mauro > On 22 Jul 2019, at 09:16, Hari Gowtham wrote: > > Hi, > Yes the above mentioned steps are right. > The way to find if the crawl is still happening is to grep for > quota_crawl in the processes that are still running. > # ps aux | grep quota_crawl > As long as this process is alive, the crawl is happening. > > Note: crawl does take a lot of time as well. And it happens twice. > > On Fri, Jul 19, 2019 at 5:42 PM Mauro Tridici wrote: >> >> Hi Hari, >> >> thank you very much for the fast answer. >> I think that the we will try to solve the issue disabling and enabling quota. >> So, if I understand I have to do the following actions: >> >> - save on my notes the current quota limits; >> - disable quota using "gluster volume quota /tier2 disable? command; >> - wait a while for the crawl (question: how can I understand that crawl is terminated!? how logn should I wait?); >> - enable quota using "gluster volume quota /tier2 enable?; >> - set again the previous quota limits. >> >> Is this correct? >> >> Many thanks for your support, >> Mauro >> >> >> >> On 19 Jul 2019, at 12:48, Hari Gowtham wrote: >> >> Hi Mauro, >> >> The fsck script is the fastest way to resolve the issue. >> The other way would be to disable quota and once the crawl for disable >> is done, we have to enable and set the limits again. >> In this way, the crawl happens twice and hence its slow. >> >> On Fri, Jul 19, 2019 at 3:27 PM Mauro Tridici wrote: >> >> >> Dear All, >> >> I?m experiencing again a problem with gluster file system quota. >> The ?df -hT /tier2/CSP/sp1? command output is different from the ?du -ms? command executed against the same folder. >> >> [root at s01 manual]# df -hT /tier2/CSP/sp1 >> Filesystem Type Size Used Avail Use% Mounted on >> s01-stg:tier2 fuse.glusterfs 25T 22T 3.5T 87% /tier2 >> >> [root at s01 sp1]# du -ms /tier2/CSP/sp1 >> 14TB /tier2/CSP/sp1 >> >> In the past, I used successfully the quota_fsck_new-6.py script in order to detect the SIZE_MISMATCH occurrences and fix them. >> Unfortunately, the number of sub-directories and files saved in /tier2/CSP/sp1 grew so much and the list of SIZE_MISMATCH entries is very long. >> >> Is there a faster way to correct the mismatching outputs? >> Could you please help me to solve, if it is possible, this issue? >> >> Thank you in advance, >> Mauro >> >> >> >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >> -- >> Regards, >> Hari Gowtham. >> >> >> > > > -- > Regards, > Hari Gowtham. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgowtham at redhat.com Mon Jul 22 08:28:31 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Mon, 22 Jul 2019 13:58:31 +0530 Subject: [Gluster-users] "du" and "df -hT" commands output mismatch In-Reply-To: <05385746-58F0-4FFE-BE49-3CFED6C919A2@cmcc.it> References: <287791D2-4775-4CB1-9AA1-9C160839C2D2@cmcc.it> <176E4EC2-DB3E-4C33-8EB4-8D09DCB11599@cmcc.it> <05385746-58F0-4FFE-BE49-3CFED6C919A2@cmcc.it> Message-ID: As of now we don't have way to solve it indefinitely. There may be a number of ways accounting mismatch can happen. To solve each way, we need to identify how it happened (the IOs that went through, their order and the timing) with this we need to understand what change is necessary and implement that. This has to done every time we come across an issue that can cause accounting mismatch. Most of the changes might affect the performance. That is a down side. And we don't have a way to collect the above necessary information. To do the above requirements, we don't have enough bandwidth. If anyone from the community is interested, they can contribute to it. We are here to help with them out. On Mon, Jul 22, 2019 at 1:12 PM Mauro Tridici wrote: > > Hi Hari, > > I hope that the crawl will run at most for a couple of days. > Do you know if there is a way to solve the issue definitely ? > > GlusterFS version is 3.12.14. > You can find below some additional info. > > Volume Name: tier2 > Type: Distributed-Disperse > Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c > Status: Started > Snapshot Count: 0 > Number of Bricks: 12 x (4 + 2) = 72 > Transport-type: tcp > > Many thanks, > Mauro > > On 22 Jul 2019, at 09:16, Hari Gowtham wrote: > > Hi, > Yes the above mentioned steps are right. > The way to find if the crawl is still happening is to grep for > quota_crawl in the processes that are still running. > # ps aux | grep quota_crawl > As long as this process is alive, the crawl is happening. > > Note: crawl does take a lot of time as well. And it happens twice. > > On Fri, Jul 19, 2019 at 5:42 PM Mauro Tridici wrote: > > > Hi Hari, > > thank you very much for the fast answer. > I think that the we will try to solve the issue disabling and enabling quota. > So, if I understand I have to do the following actions: > > - save on my notes the current quota limits; > - disable quota using "gluster volume quota /tier2 disable? command; > - wait a while for the crawl (question: how can I understand that crawl is terminated!? how logn should I wait?); > - enable quota using "gluster volume quota /tier2 enable?; > - set again the previous quota limits. > > Is this correct? > > Many thanks for your support, > Mauro > > > > On 19 Jul 2019, at 12:48, Hari Gowtham wrote: > > Hi Mauro, > > The fsck script is the fastest way to resolve the issue. > The other way would be to disable quota and once the crawl for disable > is done, we have to enable and set the limits again. > In this way, the crawl happens twice and hence its slow. > > On Fri, Jul 19, 2019 at 3:27 PM Mauro Tridici wrote: > > > Dear All, > > I?m experiencing again a problem with gluster file system quota. > The ?df -hT /tier2/CSP/sp1? command output is different from the ?du -ms? command executed against the same folder. > > [root at s01 manual]# df -hT /tier2/CSP/sp1 > Filesystem Type Size Used Avail Use% Mounted on > s01-stg:tier2 fuse.glusterfs 25T 22T 3.5T 87% /tier2 > > [root at s01 sp1]# du -ms /tier2/CSP/sp1 > 14TB /tier2/CSP/sp1 > > In the past, I used successfully the quota_fsck_new-6.py script in order to detect the SIZE_MISMATCH occurrences and fix them. > Unfortunately, the number of sub-directories and files saved in /tier2/CSP/sp1 grew so much and the list of SIZE_MISMATCH entries is very long. > > Is there a faster way to correct the mismatching outputs? > Could you please help me to solve, if it is possible, this issue? > > Thank you in advance, > Mauro > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > -- > Regards, > Hari Gowtham. > > > > > > -- > Regards, > Hari Gowtham. > > > -- Regards, Hari Gowtham. From ernestclyde at sav25.com Mon Jul 22 09:13:49 2019 From: ernestclyde at sav25.com (Ernest Clyde A. Chua) Date: Mon, 22 Jul 2019 17:13:49 +0800 Subject: [Gluster-users] change voulme type replica 3 to a distributed replicated volume Message-ID: Good day everyone, Currently we have a replica of 3 servers at the near future it will reach its maximum capacity and we are planning to add 1 server to increase the capacity by?changing the current volume type to distributed replicated. can you give me some guide or instructions on how i can accomplish this. Thanks. The entire content of this email message and any attachments to it are confidential and are intended solely for use of the individual or entity named as addressee and recipient. If you have received it by mistake, you must neither take any action based upon its contents, nor copy, disseminate or show it to anyone. Make sure to immediately notify the sender by email and follow with its deletion from your system. This email may contain personal information thus unauthorized dissemination, reproduction, or processing of information is strictly prohibited and subject to the Republic Act No. 10173 also known as the Data Privacy Act of 2012. By replying to this email containing any personal information, you agree to comply with the Privacy Act and therefore gives consent to SAV25 Data Systems for the processing of personal data included in your email reply. SAV25 Data Systems cannot be held responsible for the content of this email, nor can it be responsible for the consequences of the act ions taken based on the information we have provided in this email unless otherwise subsequently confirmed by SAV25 Data Systems the information found in email in writing. The integrity and security of this message cannot be guaranteed on the Internet although SAV25 Data Systems have put efforts into ensuring that the message is error and virus-free. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email. From mauro.tridici at cmcc.it Mon Jul 22 11:45:52 2019 From: mauro.tridici at cmcc.it (Mauro Tridici) Date: Mon, 22 Jul 2019 13:45:52 +0200 Subject: [Gluster-users] "du" and "df -hT" commands output mismatch In-Reply-To: References: <287791D2-4775-4CB1-9AA1-9C160839C2D2@cmcc.it> <176E4EC2-DB3E-4C33-8EB4-8D09DCB11599@cmcc.it> <05385746-58F0-4FFE-BE49-3CFED6C919A2@cmcc.it> Message-ID: <21D1D909-2941-4AFC-AFDC-F11BA6D5538B@cmcc.it> Hello Hari, thank you very much for the explanation. Regards, Mauro > On 22 Jul 2019, at 10:28, Hari Gowtham wrote: > > As of now we don't have way to solve it indefinitely. > There may be a number of ways accounting mismatch can happen. > To solve each way, we need to identify how it happened (the IOs that > went through, their order and the timing) > with this we need to understand what change is necessary and implement that. > This has to done every time we come across an issue that can cause > accounting mismatch. > Most of the changes might affect the performance. That is a down side. > And we don't have a way to collect the above necessary information. > > To do the above requirements, we don't have enough bandwidth. > If anyone from the community is interested, they can contribute to it. > We are here to help with them out. > > On Mon, Jul 22, 2019 at 1:12 PM Mauro Tridici wrote: >> >> Hi Hari, >> >> I hope that the crawl will run at most for a couple of days. >> Do you know if there is a way to solve the issue definitely ? >> >> GlusterFS version is 3.12.14. >> You can find below some additional info. >> >> Volume Name: tier2 >> Type: Distributed-Disperse >> Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 12 x (4 + 2) = 72 >> Transport-type: tcp >> >> Many thanks, >> Mauro >> >> On 22 Jul 2019, at 09:16, Hari Gowtham wrote: >> >> Hi, >> Yes the above mentioned steps are right. >> The way to find if the crawl is still happening is to grep for >> quota_crawl in the processes that are still running. >> # ps aux | grep quota_crawl >> As long as this process is alive, the crawl is happening. >> >> Note: crawl does take a lot of time as well. And it happens twice. >> >> On Fri, Jul 19, 2019 at 5:42 PM Mauro Tridici wrote: >> >> >> Hi Hari, >> >> thank you very much for the fast answer. >> I think that the we will try to solve the issue disabling and enabling quota. >> So, if I understand I have to do the following actions: >> >> - save on my notes the current quota limits; >> - disable quota using "gluster volume quota /tier2 disable? command; >> - wait a while for the crawl (question: how can I understand that crawl is terminated!? how logn should I wait?); >> - enable quota using "gluster volume quota /tier2 enable?; >> - set again the previous quota limits. >> >> Is this correct? >> >> Many thanks for your support, >> Mauro >> >> >> >> On 19 Jul 2019, at 12:48, Hari Gowtham wrote: >> >> Hi Mauro, >> >> The fsck script is the fastest way to resolve the issue. >> The other way would be to disable quota and once the crawl for disable >> is done, we have to enable and set the limits again. >> In this way, the crawl happens twice and hence its slow. >> >> On Fri, Jul 19, 2019 at 3:27 PM Mauro Tridici wrote: >> >> >> Dear All, >> >> I?m experiencing again a problem with gluster file system quota. >> The ?df -hT /tier2/CSP/sp1? command output is different from the ?du -ms? command executed against the same folder. >> >> [root at s01 manual]# df -hT /tier2/CSP/sp1 >> Filesystem Type Size Used Avail Use% Mounted on >> s01-stg:tier2 fuse.glusterfs 25T 22T 3.5T 87% /tier2 >> >> [root at s01 sp1]# du -ms /tier2/CSP/sp1 >> 14TB /tier2/CSP/sp1 >> >> In the past, I used successfully the quota_fsck_new-6.py script in order to detect the SIZE_MISMATCH occurrences and fix them. >> Unfortunately, the number of sub-directories and files saved in /tier2/CSP/sp1 grew so much and the list of SIZE_MISMATCH entries is very long. >> >> Is there a faster way to correct the mismatching outputs? >> Could you please help me to solve, if it is possible, this issue? >> >> Thank you in advance, >> Mauro >> >> >> >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >> -- >> Regards, >> Hari Gowtham. >> >> >> >> >> >> -- >> Regards, >> Hari Gowtham. >> >> >> > > > -- > Regards, > Hari Gowtham. From matthewb at uvic.ca Mon Jul 22 16:29:29 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Mon, 22 Jul 2019 09:29:29 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: Message-ID: Hi Nithya, Over the weekend there was a maintenance window and the version of gluster was updated from 5.3 to 5.6, and the entire cluster was rebooted, but the dht xattr is still missing: [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ # file: /mnt/raid6-storage/storage/ security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d35e1ab000a0ba6 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size.2=0x00001b6a366e74000000000000763ca3000000000005ce66 trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 [root at gluster07 ~]# rpm -q glusterfs glusterfs-5.6-1.el7.x86_64 [root at gluster07 ~]# uname -r 3.10.0-957.21.3.el7.x86_64 Thanks, ?-Matthew -- Matthew Benstead System Administrator Pacific Climate Impacts Consortium University of Victoria, UH1 PO Box 1800, STN CSC Victoria, BC, V8W 2Y2 Phone: +1-250-721-8432 Email: matthewb at uvic.ca On 7/18/19 7:20 AM, Matthew Benstead wrote: > Hi Nithya, > > No - it was added about a year and a half ago. I have tried > re-mounting the volume on the server, but it didn't add the attr: > > [root at gluster07 ~]# umount /storage/ > [root at gluster07 ~]# cat /etc/fstab | grep "/storage" > 10.0.231.56:/storage /storage glusterfs > defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 > [root at gluster07 ~]# mount /storage/ > [root at gluster07 ~]# df -h /storage/ > Filesystem??????????? Size? Used Avail Use% Mounted on > 10.0.231.56:/storage? 255T? 194T?? 62T? 77% /storage > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ > # file: /mnt/raid6-storage/storage/ > security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 > trusted.gfid=0x00000000000000000000000000000001 > trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 > trusted.glusterfs.quota.dirty=0x3000 > trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 > trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 > > Thanks, > ?-Matthew > > On 7/17/19 10:04 PM, Nithya Balachandran wrote: >> Hi Matthew, >> >> Was this node/brick added to the volume recently? If yes, try >> mounting the volume on a fresh mount point - that should create the >> xattr on this as well. >> >> Regards, >> Nithya >> >> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead > > wrote: >> >> Hello, >> >> I've just noticed one brick in my 7 node distribute volume is missing >> the trusted.glusterfs.dht xattr...? How can I fix this? >> >> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >> >> All of the other nodes are fine, but gluster07 from the list >> below does >> not have the attribute. >> >> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr >> -m . >> --absolute-names -n trusted.glusterfs.dht -e hex >> /mnt/raid6-storage/storage" >> ... >> gluster05 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >> >> gluster03 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >> >> gluster04 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >> >> gluster06 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >> >> gluster02 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >> >> gluster07 | FAILED | rc=1 >> >> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >> attributenon-zero return code >> >> gluster01 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >> >> Here are all of the attr's from the brick: >> >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ >> # file: /mnt/raid6-storage/storage/ >> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.gfid=0x00000000000000000000000000000001 >> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >> trusted.glusterfs.quota.dirty=0x3000 >> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >> >> >> And here is the volume information: >> >> [root at gluster07 ~]# gluster volume info storage >> >> Volume Name: storage >> Type: Distribute >> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 7 >> Transport-type: tcp >> Bricks: >> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >> Options Reconfigured: >> changelog.changelog: on >> features.quota-deem-statfs: on >> features.read-only: off >> features.inode-quota: on >> features.quota: on >> performance.readdir-ahead: on >> nfs.disable: on >> geo-replication.indexing: on >> geo-replication.ignore-pid-check: on >> transport.address-family: inet >> >> Thanks, >> ?-Matthew >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgowtham at redhat.com Tue Jul 23 04:30:20 2019 From: hgowtham at redhat.com (hgowtham at redhat.com) Date: Tue, 23 Jul 2019 04:30:20 +0000 Subject: [Gluster-users] Invitation: nvitation: Gluster Community Meeting (APAC friendly hours... @ Tue Jul 23, 2019 10am - 11am (IST) (gluster-users@gluster.org) Message-ID: <00000000000080e746058e51a63c@google.com> You have been invited to the following event. Title: nvitation: Gluster Community Meeting (APAC friendly hours) @ Tue July 23, 2019 11:30am - 12:30pm (IST) Hi all, This is the biweekly Gluster community meeting that is hosted to collaborate and make the community better. Please do join the discussion. Bridge: https://bluejeans.com/836554017 Minutes meeting: https://hackmd.io/zoI6PYIrTbWUpchGPZ_pBg?both Previous Meeting notes: https://github.com/gluster/community Regards, Hari. When: Tue Jul 23, 2019 10am ? 11am India Standard Time - Kolkata Calendar: gluster-users at gluster.org Who: * hgowtham at redhat.com - organizer * gluster-users at gluster.org * gluster-devel at gluster.org Event details: https://www.google.com/calendar/event?action=VIEW&eid=M2RuOGlmMmgyMHJqdmE4MzMzb20wcHRpa3IgZ2x1c3Rlci11c2Vyc0BnbHVzdGVyLm9yZw&tok=MTkjaGdvd3RoYW1AcmVkaGF0LmNvbTgxODBiMzBjNGJiY2IyZjU5ZmQ2NmIyMzEwYTM0ZDkyNTYxZjk5ZmE&ctz=Asia%2FKolkata&hl=en&es=0 Invitation from Google Calendar: https://www.google.com/calendar/ You are receiving this courtesy email at the account gluster-users at gluster.org because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn more at https://support.google.com/calendar/answer/37135#forwarding -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 1932 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: invite.ics Type: application/ics Size: 1973 bytes Desc: not available URL: From hgowtham at redhat.com Tue Jul 23 04:33:03 2019 From: hgowtham at redhat.com (hgowtham at redhat.com) Date: Tue, 23 Jul 2019 04:33:03 +0000 Subject: [Gluster-users] Updated invitation: Invitation: Gluster Community Meeting (APAC friendly hour... @ Tue Jul 23, 2019 11:30am - 12:25pm (IST) (gluster-users@gluster.org) Message-ID: <000000000000387489058e51b07c@google.com> This event has been changed. Title: Invitation: Gluster Community Meeting (APAC friendly hours) @ Tue July 23, 2019 11:30am - 12:30pm (IST) (changed) Hi all, This is the biweekly Gluster community meeting that is hosted to collaborate and make the community better. Please do join the discussion. Bridge: https://bluejeans.com/836554017 Minutes meeting: https://hackmd.io/zoI6PYIrTbWUpchGPZ_pBg?both Previous Meeting notes: https://github.com/gluster/community Regards, Hari. When: Tue Jul 23, 2019 11:30am ? 12:25pm India Standard Time - Kolkata (changed) Calendar: gluster-users at gluster.org Who: * hgowtham at redhat.com - organizer * gluster-users at gluster.org * gluster-devel at gluster.org * jae.park at thryv.com Event details: https://www.google.com/calendar/event?action=VIEW&eid=M2RuOGlmMmgyMHJqdmE4MzMzb20wcHRpa3IgZ2x1c3Rlci11c2Vyc0BnbHVzdGVyLm9yZw&tok=MTkjaGdvd3RoYW1AcmVkaGF0LmNvbTgxODBiMzBjNGJiY2IyZjU5ZmQ2NmIyMzEwYTM0ZDkyNTYxZjk5ZmE&ctz=Asia%2FKolkata&hl=en&es=0 Invitation from Google Calendar: https://www.google.com/calendar/ You are receiving this courtesy email at the account gluster-users at gluster.org because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn more at https://support.google.com/calendar/answer/37135#forwarding -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 2078 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: invite.ics Type: application/ics Size: 2121 bytes Desc: not available URL: From hgowtham at redhat.com Tue Jul 23 06:54:03 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Tue, 23 Jul 2019 12:24:03 +0530 Subject: [Gluster-users] Minutes of Gluster Community Meeting (APAC) 23rd July 2019 Message-ID: Hi, The minutes of the meeting are as follows: Recording of this meeting- - https://bluejeans.com/s/s1Zma ------- ### Attendance Name (#gluster-dev alias) - company * Ashish Pandey (_apandey) - Redhat * Rishubh Jain (risjain) - Redhat * Susant Palai (spalai) - Redhat * hari gowtham (hgowtham) - Red Hat * Sheetal Pamecha (spamecha) - Red Hat * Shwetha Acharya (sacharya) - Red Hat * Amar Tumballi (amarts/@tumballi) * Sunny Kumar (sunnyk) - Red Hat * Rinku Kothya (rinku) - Red Hat * Hamza * Khalid * Sanju Rakonde (srakonde) - RedHat * Deepshikha (dkhandel) - Red Hat * Pranith Kumar (pranithk) - Red Hat * Sunil Kumar (skumar) - Red Hat * Kotresh HR (kotreshhr) - RedHat * David Spisla - Gluster User * Rafi KC - Red Hat * Prasanna Kumar Kalever (pkalever) - RedHat * Karthik Subrahmanya (ksubrahm) - Red Hat * Arjun Sharma - Red Hat * Kaustav Majumder - Red Hat ### User stories Felix, is trying to setup a production server. asked for ideas to set it up. storing large media and research files using replica/disperse vol. a users issue was addressed with a patch. duplicate entry in volfile. [bug which talks about issue](https://bugzilla.redhat.com/1730953) ### Community * Project metrics: | Metrics | Value | | ------------------------- | -------- | |[Coverity](https://scan.coverity.com/projects/gluster-glusterfs) | 71 | |[Clang Scan](https://build.gluster.org/job/clang-scan/lastBuild/) | 59 | |[Test coverage](https://build.gluster.org/job/line-coverage/lastCompletedBuild/Line_20Coverage_20Report/)| 69.7 (13-07-2019) | |New Bugs in last 14 days
[master](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&f1=creation_ts&o1=greaterthan&product=GlusterFS&query_format=advanced&v1=-14d&version=mainline)
[7.x](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&f1=creation_ts&list_id=10353290&o1=greaterthan&product=GlusterFS&query_format=advanced&v1=-14d&version=7)
[ 6.x](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&f1=creation_ts&o1=greaterthan&product=GlusterFS&query_format=advanced&v1=-14d&version=6)
[ 5.x](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&f1=creation_ts&o1=greaterthan&product=GlusterFS&query_format=advanced&v1=-14d&version=5) |
12
2
5
0 | |[Gluster User Queries in last 14 days](https://lists.gluster.org/pipermail/gluster-users/2019-April/thread.html) | 17 | |[Total Bugs](https://bugzilla.redhat.com/report.cgi?x_axis_field=bug_status&y_axis_field=component&z_axis_field=&no_redirect=1&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&bug_status=__open__&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&deadlinefrom=&deadlineto=&bug_id=&bug_id_type=anyexact&votes=&votes_type=greaterthaneq&emailtype1=substring&email1=&emailtype2=substring&email2=&emailtype3=substring&email3=&chfieldvalue=&chfieldfrom=&chfieldto=Now&j_top=AND&f1=noop&o1=noop&v1=&format=table&action=wrap&product=GlusterFS) | 335 | |[Total Github issues](https://github.com/gluster/glusterfs/issues) | 380 | * Any release updates? * 4.1.10, 5.8 and 6.4's packaging is nearly done. will release it in a day or two. * 4.1.10 will be EOLed * Release 7, merged some patches and some are pending due to centos regression failing. * Blocker issues across the project? * we have fixed the python issue with 5.7 and are working on 5.8 * Notable thread form mailing list * webhook for geo rep by Aravinda ### Conferences / Meetups * [Developers' Conference - August 2-3, 2019](https://devconf.info/in) - Important dates: CFP Closed Schedule Announcement: https://devconfin19.sched.com/ Event Open for Registration : https://devconfin19.eventbrite.com Last Date of Registration: 31st July, 2019 Event dates: Aug 2nd, 3rd Venue: Christ University - Bengaluru, India Talks related to gluster: Ashish: Thin Arbiter volume Aravinda: Rethinking Gluster Management using k8s Ravi: Strategies for Replication in Distributed systems Mugdha: selenium for automation in RHHI ### GlusterFS - v7.0 and beyond * Proposal - https://docs.google.com/presentation/d/1rtn38S4YBe77KK5IjczWmoAR-ZSO-i3tNHg9pAH8Wt8/edit?usp=sharing * Proposed Plan: - GlusterFS-7.0 (July 1st) - Stability, Automation - Only - GlusterFS-8.0 (Nov 1st) - - Plan for Fedora 31/RHEL8.2 - GlusterFS-9.0 (March 1st, 2020) Reflink, io_uring, and similar improvements. ### Developer focus * Any design specs to discuss? nothing ### Component status * Arbiter - nothing * AFR - metadata split brain has been fixed. gfid split brain is being worked on. * DHT - nothing * EC - data corruption worked by Xavi and Pranith. * FUSE - Nithya sent a few patches to invalidate inode. mail to discuss this * POSIX - nothing * DOC - Man page bugs are open need to be looked into. glusterfs-8 doc has to be looked into(RDMA has to be removed) Gluster v status has RDMA which has to be looked into as well . tier has to be removed from man page. * Geo Replication - The mount broker blocker issue is fixed. The [patch](https://review.gluster.org/#/c/glusterfs/+/23089/) is merged. Needs backport to release branches. * libglusterfs - nothing * Management Daemon : glusterd1 - nothing new. glusterd_volinfo_find() optimization * Snapshot - nothing * NFS - * thin-arbiter - performance improvements. ### Flash Talk Gluster * Typical 5 min talk about Gluster with up to 5 more minutes for questions * For this meeting lets talk about Roadmap suggestions. - https://hackmd.io/JtfYZr49QeGaNIlTvQNsaA ### Recent Blog posts / Document updates * https://medium.com/@sheetal.pamecha08/https-medium-com-sheetal-pamecha08-one-good-way-to-start-contributing-to-open-source-static-analysers-16543eeeb138 * https://pkalever.wordpress.com/2019/07/02/expanding-gluster-block-volume/ * https://medium.com/@tumballi/glusters-management-in-k8s-13020a561962 * https://aravindavk.in/blog/gluster-and-k8s-portmap/ ### Gluster Friday Five * Every friday we release this, which basically covers highlight of week in gluster.Also you can find more videos in youtube link. https://www.youtube.com/channel/UCfilWh0JA5NfCjbqq1vsBVA ### Host * Who will host next meeting? - Host will need to send out the agenda 24hr - 12hrs in advance to mailing list, and also make sure to send the meeting minutes. - Host will need to reach out to one user at least who can talk about their usecase, their experience, and their needs. - Host needs to send meeting minutes as PR to http://github.com/gluster/community Sunny will host the next meeting. ### Notetaker * Who will take notes from the next meeting? ### RoundTable [Amar] Road map has to be worked on for the next 6 months to be sent by Maintainers. ### Action Items on host * Check-in Minutes of meeting for this meeting -- Regards, Hari Gowtham. From g.danti at assyoma.it Tue Jul 23 08:35:42 2019 From: g.danti at assyoma.it (Gionatan Danti) Date: Tue, 23 Jul 2019 10:35:42 +0200 Subject: [Gluster-users] Recommended gluster stable version Message-ID: Hi list, I have a question about recommended gluster stable version for using to host virtual disk images. From my understanding, current RHGS uses the latest 3.x gluster branch. This is also the same version provided by default in RHEL/CentOS; [root at localhost ~]# yum info glusterfs.x86_64 ... Name : glusterfs Arch : x86_64 Version : 3.12.2 Release : 18.el7 Size : 542 k Repo : base/7/x86_64 ... At the same time, CentOS SIG enables version 4.0, 4.1, 5.x and 6.x. Discarding the 4.0 version (a short term stable release), what gluster version should be used for maximum reliability and supportability, even in the face of some features missing? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From kkeithle at redhat.com Tue Jul 23 11:21:30 2019 From: kkeithle at redhat.com (Kaleb Keithley) Date: Tue, 23 Jul 2019 07:21:30 -0400 Subject: [Gluster-users] Recommended gluster stable version In-Reply-To: References: Message-ID: On Tue, Jul 23, 2019 at 4:36 AM Gionatan Danti wrote: > Hi list, > I have a question about recommended gluster stable version for using to > host virtual disk images. > > From my understanding, current RHGS uses the latest 3.x gluster branch. > This is also the same version provided by default in RHEL/CentOS; > > [root at localhost ~]# yum info glusterfs.x86_64 > ... > Name : glusterfs > Arch : x86_64 > Version : 3.12.2 > Release : 18.el7 > Size : 542 k > Repo : base/7/x86_64 > ... > > At the same time, CentOS SIG enables version 4.0, 4.1, 5.x and 6.x. > That's true, but the glusterfs-3.12.x packages are still available on the CentOS mirrors. I'm not sure why the centos-release-gluster312 package has been removed from the mirrors, but you can still get it from https://cbs.centos.org/koji/packageinfo?packageID=6530 It seems odd to me that centos-release-gluster312 would have been removed when RHEL is still shipping ? for a little while longer anyway ? glusterfs-3.12 (client side) packages. -- Kaleb -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndevos at redhat.com Tue Jul 23 12:15:55 2019 From: ndevos at redhat.com (Niels de Vos) Date: Tue, 23 Jul 2019 14:15:55 +0200 Subject: [Gluster-users] Recommended gluster stable version In-Reply-To: References: Message-ID: <20190723121555.GA2437@ndevos-x270> On Tue, Jul 23, 2019 at 07:21:30AM -0400, Kaleb Keithley wrote: > On Tue, Jul 23, 2019 at 4:36 AM Gionatan Danti wrote: > > > Hi list, > > I have a question about recommended gluster stable version for using to > > host virtual disk images. > > > > From my understanding, current RHGS uses the latest 3.x gluster branch. > > This is also the same version provided by default in RHEL/CentOS; > > > > [root at localhost ~]# yum info glusterfs.x86_64 > > ... > > Name : glusterfs > > Arch : x86_64 > > Version : 3.12.2 > > Release : 18.el7 > > Size : 542 k > > Repo : base/7/x86_64 > > ... > > > > At the same time, CentOS SIG enables version 4.0, 4.1, 5.x and 6.x. > > > > > That's true, but the glusterfs-3.12.x packages are still available on the > CentOS mirrors. > > I'm not sure why the centos-release-gluster312 package has been removed > from the mirrors, but you can still get it from > https://cbs.centos.org/koji/packageinfo?packageID=6530 > > It seems odd to me that centos-release-gluster312 would have been removed > when RHEL is still shipping ? for a little while longer anyway ? > glusterfs-3.12 (client side) packages. The release-312 branch is not maintained anymore, GlusterFS 3.12 is End-Of-Life. See https://www.gluster.org/release-schedule/ for details. Of course vendors of productized GlusterFS releases are free to maintain their downstream forks and provide support for that. The 3.12 packages are still available on the CentOS mirrors on request of the oVirt project. We (Gluster Community) do not recommend installing 3.12 anymore as there will not be any further bugfixes for it. Niels From g.danti at assyoma.it Tue Jul 23 12:15:56 2019 From: g.danti at assyoma.it (Gionatan Danti) Date: Tue, 23 Jul 2019 14:15:56 +0200 Subject: [Gluster-users] Recommended gluster stable version In-Reply-To: References: Message-ID: On 23/07/2019 13:21, Kaleb Keithley wrote: > That's true, but the glusterfs-3.12.x packages are still available on > the CentOS mirrors. > > I'm not sure why the centos-release-gluster312 package has been removed > from the mirrors, but you can still get it from > https://cbs.centos.org/koji/packageinfo?packageID=6530 > > It seems odd to me that?centos-release-gluster312 would have been > removed when RHEL is still shipping ? for a little while longer anyway ? > glusterfs-3.12 (client side) packages. Gluster 3.12.x *is* available con CentOS: it is directly in the base repository (the "yum info" of the previous post was issued on a CentOS 7 machine). Is that version (3.12.x) the one used on RHGS? For maximum stability, on what version should I rely? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From hgowtham at redhat.com Tue Jul 23 12:23:14 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Tue, 23 Jul 2019 17:53:14 +0530 Subject: [Gluster-users] Announcing Gluster release 6.4 Message-ID: Hi, The Gluster community is pleased to announce the release of Gluster 6.4 (packages available at [1]). Release notes for the release can be found at [2]. Major changes, features and limitations addressed in this release: None Thanks, Gluster community [1] Packages for 6.4: https://download.gluster.org/pub/gluster/glusterfs/6/6.4/ [2] Release notes for 6.4: https://docs.gluster.org/en/latest/release-notes/6.4/ -- Regards, Hari Gowtham. From hgowtham at redhat.com Tue Jul 23 12:25:11 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Tue, 23 Jul 2019 17:55:11 +0530 Subject: [Gluster-users] Announcing Gluster release 4.1.10 Message-ID: Hi, The Gluster community is pleased to announce the release of Gluster 4.1.10 (packages available at [1]). Release notes for the release can be found at [2]. Major changes, features and limitations addressed in this release: None NOTE: This is the last release for 4.1 series. Thanks, Gluster community [1] Packages for 4.1.10: https://download.gluster.org/pub/gluster/glusterfs/4/4.1.10/ [2] Release notes for 4.1.10: https://docs.gluster.org/en/latest/release-notes/4.1.10/ -- Regards, Hari Gowtham. From hunter86_bg at yahoo.com Tue Jul 23 17:26:01 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Tue, 23 Jul 2019 20:26:01 +0300 Subject: [Gluster-users] Recommended gluster stable version Message-ID: Hi Kaleb, I would recommend you v6. Curently I'm running hyperconverged setup with oVirt (latest) + gluster v6.3 and everything is quite well. Also, keep in mind there multiple enhancements adopted with v6 and if any issue arise - the dev will prioritize it. Also, v6 will have longer support compared to the other options, which means that you will have plenty of time before you have to think about migration. Best Regards, Strahil NikolovOn Jul 23, 2019 14:21, Kaleb Keithley wrote: > > > > On Tue, Jul 23, 2019 at 4:36 AM Gionatan Danti wrote: >> >> Hi list, >> I have a question about recommended gluster stable version for using to >> host virtual disk images. >> >> ?From my understanding, current RHGS uses the latest 3.x gluster branch. >> This is also the same version provided by default in RHEL/CentOS; >> >> [root at localhost ~]# yum info glusterfs.x86_64 >> ... >> Name? ? ? ? : glusterfs >> Arch? ? ? ? : x86_64 >> Version? ? ?: 3.12.2 >> Release? ? ?: 18.el7 >> Size? ? ? ? : 542 k >> Repo? ? ? ? : base/7/x86_64 >> ... >> >> At the same time, CentOS SIG enables version 4.0, 4.1, 5.x and 6.x. > > > > That's true, but the glusterfs-3.12.x packages are still available on the CentOS mirrors. > > I'm not sure why the centos-release-gluster312 package has been removed from the mirrors, but you can still get it from?https://cbs.centos.org/koji/packageinfo?packageID=6530 > > It seems odd to me that?centos-release-gluster312 would have been removed when RHEL is still shipping ? for a little while longer anyway ? glusterfs-3.12 (client side) packages. > > -- > > Kaleb > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dcunningham at voisonics.com Tue Jul 23 23:47:50 2019 From: dcunningham at voisonics.com (David Cunningham) Date: Wed, 24 Jul 2019 11:47:50 +1200 Subject: [Gluster-users] Thin-arbiter questions In-Reply-To: References: <757816852.16925254.1557123682731.JavaMail.zimbra@redhat.com> <645227359.16980056.1557131647054.JavaMail.zimbra@redhat.com> <2006209001.22227657.1560230122663.JavaMail.zimbra@redhat.com> Message-ID: Hello, I see references to fixes for thin-arbiter on the 6.x release notes. Does that mean thin-arbiter is ready to use on glusterfs 6 please? On Wed, 19 Jun 2019 at 17:42, Hari Gowtham wrote: > The committing should happen a little earlier than 10th. The tagging is > the one which decides if the patch makes it to the release or not and > tagging happens before 10th. We do announce the tagging date for each > release in the mailing list. You can keep an eye on that to know the dates. > And committing in master won't be enough for it to make it to a release. If > it has to be a part of release 6 then after being committed into master we > have to back port it to the release 6 branch and it should get committed in > that particular branch as well. Only then it will be a part of the package > released for that branch. > > > On Wed, 19 Jun, 2019, 4:24 AM David Cunningham, > wrote: > >> Hi Hari, >> >> Thanks for that information. So if I understand correctly, if >> thin-arbiter is committed to the master branch by the 10th July, then it >> should be in the CentOS package fairly soon afterwards? >> >> I have a customer asking when we can use it, hence the questions. Thank >> you. >> >> >> On Tue, 18 Jun 2019 at 17:24, Hari Gowtham wrote: >> >>> Hi David, >>> >>> Once a feature is added to the master branch, we have to back port it to >>> the release 5, 6 and other such branches which are active. And these >>> release branches will be tagged every month around 10th. So if an feature >>> has been back ported to the particular release branch before tagging, then >>> it will be a part of the tagging. And this tag is the one used for creating >>> packaging. This is the procedure for CentOS, Fedora and Debian. >>> >>> Regards, >>> Hari. >>> >>> On Tue, 18 Jun, 2019, 4:06 AM David Cunningham, < >>> dcunningham at voisonics.com> wrote: >>> >>>> Hi Ashish, >>>> >>>> Thanks for that. I guess it's not your responsibility, but do you know >>>> how often it typically takes for new versions to reach the CentOS package >>>> system after being released? >>>> >>>> >>>> On Tue, 11 Jun 2019 at 17:15, Ashish Pandey >>>> wrote: >>>> >>>>> Hi David, >>>>> >>>>> It should be any time soon as we are in last phase of patch reviews. >>>>> You can follow this patch - >>>>> https://review.gluster.org/#/c/glusterfs/+/22612/ >>>>> >>>>> --- >>>>> Ashish >>>>> >>>>> ------------------------------ >>>>> *From: *"David Cunningham" >>>>> *To: *"Ashish Pandey" >>>>> *Cc: *"gluster-users" >>>>> *Sent: *Tuesday, June 11, 2019 9:55:40 AM >>>>> *Subject: *Re: [Gluster-users] Thin-arbiter questions >>>>> >>>>> Hi Ashish and Amar, >>>>> >>>>> Is there any news on when thin-arbiter might be in the regular >>>>> GlusterFS, and the CentOS packages please? >>>>> >>>>> Thanks for your help. >>>>> >>>>> >>>>> On Mon, 6 May 2019 at 20:34, Ashish Pandey >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> *From: *"David Cunningham" >>>>>> *To: *"Ashish Pandey" >>>>>> *Cc: *"gluster-users" >>>>>> *Sent: *Monday, May 6, 2019 1:40:30 PM >>>>>> *Subject: *Re: [Gluster-users] Thin-arbiter questions >>>>>> >>>>>> Hi Ashish, >>>>>> >>>>>> Thank you for the update. Does that mean they're now in the regular >>>>>> Glusterfs? Any idea how long it typically takes the Ubuntu and CentOS >>>>>> packages to be updated with the latest code? >>>>>> >>>>>> No, for regular glusterd, work is still in progress. It will be done >>>>>> soon. >>>>>> I don't have answer for the next question. May be Amar have >>>>>> information regarding this. Adding him in CC. >>>>>> >>>>>> >>>>>> On Mon, 6 May 2019 at 18:21, Ashish Pandey >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I can see that Amar has already committed the changes and those are >>>>>>> visible on >>>>>>> https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/ >>>>>>> >>>>>>> --- >>>>>>> Ashish >>>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------ >>>>>>> *From: *"Strahil" >>>>>>> *To: *"Ashish" , "David" < >>>>>>> dcunningham at voisonics.com> >>>>>>> *Cc: *"gluster-users" >>>>>>> *Sent: *Saturday, May 4, 2019 12:10:01 AM >>>>>>> *Subject: *Re: [Gluster-users] Thin-arbiter questions >>>>>>> >>>>>>> Hi Ashish, >>>>>>> >>>>>>> Can someone commit the doc change I have already proposed ? >>>>>>> At least, the doc will clarify that fact . >>>>>>> >>>>>>> Best Regards, >>>>>>> Strahil Nikolov >>>>>>> On May 3, 2019 05:30, Ashish Pandey wrote: >>>>>>> >>>>>>> Hi David, >>>>>>> >>>>>>> Creation of thin-arbiter volume is currently supported by GD2 only. >>>>>>> The command "glustercli" is available when glusterd2 is running. >>>>>>> We are also working on providing thin-arbiter support on glusted >>>>>>> however, it is not available right now. >>>>>>> https://review.gluster.org/#/c/glusterfs/+/22612/ >>>>>>> >>>>>>> --- >>>>>>> Ashish >>>>>>> >>>>>>> ------------------------------ >>>>>>> *From: *"David Cunningham" >>>>>>> *To: *gluster-users at gluster.org >>>>>>> *Sent: *Friday, May 3, 2019 7:40:03 AM >>>>>>> *Subject: *[Gluster-users] Thin-arbiter questions >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> We are setting up a thin-arbiter and hope someone can help with some >>>>>>> questions. We've been following the documentation from >>>>>>> https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/ >>>>>>> . >>>>>>> >>>>>>> 1. What release of 5.x supports thin-arbiter? We tried a "gluster >>>>>>> volume create" with the --thin-arbiter option on 5.5 and got an >>>>>>> "unrecognized option --thin-arbiter" error. >>>>>>> >>>>>>> 2. The instruction to create a new volume with a thin-arbiter is >>>>>>> clear. How do you add a thin-arbiter to an already existing volume though? >>>>>>> >>>>>>> 3. The documentation suggests running glusterfsd manually to start >>>>>>> the thin-arbiter. Is there a service that can do this instead? I found a >>>>>>> mention of one in >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1579786 but it's not >>>>>>> really documented. >>>>>>> >>>>>>> Thanks in advance for your help, >>>>>>> >>>>>>> -- >>>>>>> David Cunningham, Voisonics Limited >>>>>>> http://voisonics.com/ >>>>>>> USA: +1 213 221 1092 >>>>>>> New Zealand: +64 (0)28 2558 3782 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> David Cunningham, Voisonics Limited >>>>>> http://voisonics.com/ >>>>>> USA: +1 213 221 1092 >>>>>> New Zealand: +64 (0)28 2558 3782 >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> >>>>> >>>>> -- >>>>> David Cunningham, Voisonics Limited >>>>> http://voisonics.com/ >>>>> USA: +1 213 221 1092 >>>>> New Zealand: +64 (0)28 2558 3782 >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> >>>> >>>> -- >>>> David Cunningham, Voisonics Limited >>>> http://voisonics.com/ >>>> USA: +1 213 221 1092 >>>> New Zealand: +64 (0)28 2558 3782 >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> >> >> -- >> David Cunningham, Voisonics Limited >> http://voisonics.com/ >> USA: +1 213 221 1092 >> New Zealand: +64 (0)28 2558 3782 >> > -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspandey at redhat.com Wed Jul 24 03:51:57 2019 From: aspandey at redhat.com (Ashish Pandey) Date: Tue, 23 Jul 2019 23:51:57 -0400 (EDT) Subject: [Gluster-users] Thin-arbiter questions In-Reply-To: References: <2006209001.22227657.1560230122663.JavaMail.zimbra@redhat.com> Message-ID: <1263717773.2661191.1563940317878.JavaMail.zimbra@redhat.com> Hi David, This was the bug fix and it had to go to gluster 6 also in which thin-arbiter is supported with glusterd 2. However, what you were looking for was the support of thin-arbiter in glusterd which is available in gluster 7 only. So now using gluster 7.0, you can create and use thin-arbiter volume just like any other volume with few additional steps. Following is the command you can use - gluster volume create replica 2 thin-arbiter 1 : : : [force] Before that you have to run thin-arbiter process on TA host. The above command you can use only with release 7 and not with older release. Patches which got merged for this change are - https://review.gluster.org/#/c/glusterfs/+/22992/ - release 7 https://review.gluster.org/#/c/glusterfs/+/22612/ - master I am trying to come up with modified document/blog for this asap. --- Ashish ----- Original Message ----- From: "David Cunningham" To: "Hari Gowtham" Cc: "gluster-users" Sent: Wednesday, July 24, 2019 5:17:50 AM Subject: Re: [Gluster-users] Thin-arbiter questions Hello, I see references to fixes for thin-arbiter on the 6.x release notes. Does that mean thin-arbiter is ready to use on glusterfs 6 please? On Wed, 19 Jun 2019 at 17:42, Hari Gowtham < hgowtham at redhat.com > wrote: The committing should happen a little earlier than 10th. The tagging is the one which decides if the patch makes it to the release or not and tagging happens before 10th. We do announce the tagging date for each release in the mailing list. You can keep an eye on that to know the dates. And committing in master won't be enough for it to make it to a release. If it has to be a part of release 6 then after being committed into master we have to back port it to the release 6 branch and it should get committed in that particular branch as well. Only then it will be a part of the package released for that branch. On Wed, 19 Jun, 2019, 4:24 AM David Cunningham, < dcunningham at voisonics.com > wrote:
Hi Hari, Thanks for that information. So if I understand correctly, if thin-arbiter is committed to the master branch by the 10th July, then it should be in the CentOS package fairly soon afterwards? I have a customer asking when we can use it, hence the questions. Thank you. On Tue, 18 Jun 2019 at 17:24, Hari Gowtham < hgowtham at redhat.com > wrote:
Hi David, Once a feature is added to the master branch, we have to back port it to the release 5, 6 and other such branches which are active. And these release branches will be tagged every month around 10th. So if an feature has been back ported to the particular release branch before tagging, then it will be a part of the tagging. And this tag is the one used for creating packaging. This is the procedure for CentOS, Fedora and Debian. Regards, Hari. On Tue, 18 Jun, 2019, 4:06 AM David Cunningham, < dcunningham at voisonics.com > wrote:
Hi Ashish, Thanks for that. I guess it's not your responsibility, but do you know how often it typically takes for new versions to reach the CentOS package system after being released? On Tue, 11 Jun 2019 at 17:15, Ashish Pandey < aspandey at redhat.com > wrote:
Hi David, It should be any time soon as we are in last phase of patch reviews. You can follow this patch - https://review.gluster.org/#/c/glusterfs/+/22612/ --- Ashish From: "David Cunningham" < dcunningham at voisonics.com > To: "Ashish Pandey" < aspandey at redhat.com > Cc: "gluster-users" < gluster-users at gluster.org > Sent: Tuesday, June 11, 2019 9:55:40 AM Subject: Re: [Gluster-users] Thin-arbiter questions Hi Ashish and Amar, Is there any news on when thin-arbiter might be in the regular GlusterFS, and the CentOS packages please? Thanks for your help. On Mon, 6 May 2019 at 20:34, Ashish Pandey < aspandey at redhat.com > wrote:
From: "David Cunningham" < dcunningham at voisonics.com > To: "Ashish Pandey" < aspandey at redhat.com > Cc: "gluster-users" < gluster-users at gluster.org > Sent: Monday, May 6, 2019 1:40:30 PM Subject: Re: [Gluster-users] Thin-arbiter questions Hi Ashish, Thank you for the update. Does that mean they're now in the regular Glusterfs? Any idea how long it typically takes the Ubuntu and CentOS packages to be updated with the latest code? No, for regular glusterd, work is still in progress. It will be done soon. I don't have answer for the next question. May be Amar have information regarding this. Adding him in CC. On Mon, 6 May 2019 at 18:21, Ashish Pandey < aspandey at redhat.com > wrote:
Hi, I can see that Amar has already committed the changes and those are visible on https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/ --- Ashish From: "Strahil" < hunter86_bg at yahoo.com > To: "Ashish" < aspandey at redhat.com >, "David" < dcunningham at voisonics.com > Cc: "gluster-users" < gluster-users at gluster.org > Sent: Saturday, May 4, 2019 12:10:01 AM Subject: Re: [Gluster-users] Thin-arbiter questions Hi Ashish, Can someone commit the doc change I have already proposed ? At least, the doc will clarify that fact . Best Regards, Strahil Nikolov On May 3, 2019 05:30, Ashish Pandey < aspandey at redhat.com > wrote:
Hi David, Creation of thin-arbiter volume is currently supported by GD2 only. The command " glustercli " is available when glusterd2 is running. We are also working on providing thin-arbiter support on glusted however, it is not available right now. https://review.gluster.org/#/c/glusterfs/+/22612/ --- Ashish From: "David Cunningham" < dcunningham at voisonics.com > To: gluster-users at gluster.org Sent: Friday, May 3, 2019 7:40:03 AM Subject: [Gluster-users] Thin-arbiter questions Hello, We are setting up a thin-arbiter and hope someone can help with some questions. We've been following the documentation from https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/ . 1. What release of 5.x supports thin-arbiter? We tried a "gluster volume create" with the --thin-arbiter option on 5.5 and got an "unrecognized option --thin-arbiter" error. 2. The instruction to create a new volume with a thin-arbiter is clear. How do you add a thin-arbiter to an already existing volume though? 3. The documentation suggests running glusterfsd manually to start the thin-arbiter. Is there a service that can do this instead? I found a mention of one in https://bugzilla.redhat.com/show_bug.cgi?id=1579786 but it's not really documented. Thanks in advance for your help, -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782
-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.knezel at gmail.com Wed Jul 24 05:38:55 2019 From: peter.knezel at gmail.com (peter knezel) Date: Wed, 24 Jul 2019 07:38:55 +0200 Subject: [Gluster-users] increased latency causes rapid decrease of ftp transfer from/to glusterfs filesystem Message-ID: Hello all, having installed a 2 glusterfs system on debian 9.x and using glusterfs packages 5.6-1 we tried to transfer via ftp files from/to glusterfs filesystem. While ftp download is around 7.5MB/s, after increasing latency to 10ms (see tc command below) download is decreased rapidly to cca 1.3MB/s. # ping xx.xx.xx.xx 64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=64 time=0.426 ms 64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=64 time=0.443 ms 64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=64 time=0.312 ms 64 bytes from xx.xx.xx.xx: icmp_seq=4 ttl=64 time=0.373 ms 64 bytes from xx.xx.xx.xx: icmp_seq=5 ttl=64 time=0.415 ms ^C --- xx.xx.xx.xx ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4100ms rtt min/avg/max/mdev = 0.312/0.393/0.443/0.053 ms # tc qdisc add dev eth0 root netem delay 10ms # ping xx.xx.xx.xx PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data. 64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=64 time=10.3 ms 64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=64 time=10.3 ms 64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=64 time=10.3 ms 64 bytes from xx.xx.xx.xx: icmp_seq=4 ttl=64 time=10.3 ms 64 bytes from xx.xx.xx.xx: icmp_seq=5 ttl=64 time=10.4 ms 64 bytes from xx.xx.xx.xx: icmp_seq=6 ttl=64 time=10.4 ms ^C --- xx.xx.xx.xx ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5007ms rtt min/avg/max/mdev = 10.304/10.387/10.492/0.138 ms root at server1:~# gluster vol list GVOLUME root at server1:~# gluster vol info Volume Name: GVOLUME Type: Replicate Volume ID: xxx Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: server1.lab:/srv/fs/ftp/brick Brick2: server2.lab:/srv/fs/ftp/brick Options Reconfigured: performance.client-io-threads: off nfs.disable: off transport.address-family: inet features.cache-invalidation: on performance.stat-prefetch: on performance.md-cache-timeout: 60 network.inode-lru-limit: 1048576 cluster.quorum-type: auto performance.cache-max-file-size: 512KB performance.cache-size: 1GB performance.flush-behind: on performance.nfs.flush-behind: on performance.write-behind-window-size: 512KB performance.nfs.write-behind-window-size: 512KB performance.strict-o-direct: off performance.nfs.strict-o-direct: off performance.read-after-open: on performance.io-thread-count: 32 client.event-threads: 4 server.event-threads: 4 performance.write-behind: on performance.read-ahead: on performance.readdir-ahead: on nfs.export-dirs: off nfs.addr-namelookup: off nfs.rdirplus: on features.barrier-timeout: 1 features.trash: off cluster.quorum-reads: true auth.allow: 127.0.0.1,xx.xx.xx.xx,xx.xx.xx.yy auth.reject: all root at server1:~# Can somebody help me to tune/solve this issue? Thanks and kind regards, peterk -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Wed Jul 24 07:24:10 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Wed, 24 Jul 2019 10:24:10 +0300 Subject: [Gluster-users] increased latency causes rapid decrease of ftp transfer from/to glusterfs filesystem Message-ID: Hi Peter, Can you elaborate your issue, as I can't completely understand it. So, you have poor client performance when the lattency between client/gluster server increases almost 20 times (from 0.5 ms to 10 ms) , right ? If I understood you corectly - this type of issue cannot be avoided, as the network lattency is decreasing the communication between client/server and server/server. The same behavior will be observed in SAN , so you nees to have persistent network performance. Of course, there are some client caching techniques that will reduce that impact - but they increase the risk for your data consistency. Best Regards, Strahil NikolovOn Jul 24, 2019 08:38, peter knezel wrote: > > Hello all, > > having installed a 2 glusterfs system on debian 9.x > and using glusterfs packages 5.6-1 we tried to transfer via ftp files from/to > glusterfs filesystem. > > While ftp download is around 7.5MB/s, after increasing latency to 10ms (see tc command below) download is decreased rapidly to cca 1.3MB/s. > > > # ping xx.xx.xx.xx > 64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=64 time=0.426 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=64 time=0.443 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=64 time=0.312 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=4 ttl=64 time=0.373 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=5 ttl=64 time=0.415 ms > ^C > --- xx.xx.xx.xx ping statistics --- > 5 packets transmitted, 5 received, 0% packet loss, time 4100ms > rtt min/avg/max/mdev = 0.312/0.393/0.443/0.053 ms > > # tc qdisc add dev eth0 root netem delay 10ms > # ping xx.xx.xx.xx > PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data. > 64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=64 time=10.3 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=64 time=10.3 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=64 time=10.3 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=4 ttl=64 time=10.3 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=5 ttl=64 time=10.4 ms > 64 bytes from xx.xx.xx.xx: icmp_seq=6 ttl=64 time=10.4 ms > ^C > --- xx.xx.xx.xx ping statistics --- > 6 packets transmitted, 6 received, 0% packet loss, time 5007ms > rtt min/avg/max/mdev = 10.304/10.387/10.492/0.138 ms > > root at server1:~# gluster vol list > GVOLUME > root at server1:~# gluster vol info > > Volume Name: GVOLUME > Type: Replicate > Volume ID: xxx > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: server1.lab:/srv/fs/ftp/brick > Brick2: server2.lab:/srv/fs/ftp/brick > Options Reconfigured: > performance.client-io-threads: off > nfs.disable: off > transport.address-family: inet > features.cache-invalidation: on > performance.stat-prefetch: on > performance.md-cache-timeout: 60 > network.inode-lru-limit: 1048576 > cluster.quorum-type: auto > performance.cache-max-file-size: 512KB > performance.cache-size: 1GB > performance.flush-behind: on > performance.nfs.flush-behind: on > performance.write-behind-window-size: 512KB > performance.nfs.write-behind-window-size: 512KB > performance.strict-o-direct: off > performance.nfs.strict-o-direct: off > performance.read-after-open: on > performance.io-thread-count: 32 > client.event-threads: 4 > server.event-threads: 4 > performance.write-behind: on > performance.read-ahead: on > performance.readdir-ahead: on > nfs.export-dirs: off > nfs.addr-namelookup: off > nfs.rdirplus: on > features.barrier-timeout: 1 > features.trash: off > cluster.quorum-reads: true > auth.allow: 127.0.0.1,xx.xx.xx.xx,xx.xx.xx.yy > auth.reject: all > root at server1:~# > > Can somebody help me to tune/solve this issue? > Thanks and kind regards, > > peterk > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.knezel at gmail.com Wed Jul 24 07:33:33 2019 From: peter.knezel at gmail.com (peter knezel) Date: Wed, 24 Jul 2019 09:33:33 +0200 Subject: [Gluster-users] increased latency causes rapid decrease of ftp transfer from/to glusterfs filesystem Message-ID: Exactly, i wanted to do a simulation of a problem that appears in a real situation where there is a latency around 11ms. (same OS, same gluster package) Can you write more about "client caching techniques that will reduce that impact" ? Are the parameters set ok? Thanks, peterk -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Wed Jul 24 07:44:22 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Wed, 24 Jul 2019 10:44:22 +0300 Subject: [Gluster-users] increased latency causes rapid decrease of ftp transfer from/to glusterfs filesystem Message-ID: Hi Peter, I'm using gluster with ovirt and my settings are more leaning towards data safety. I think you can check the gluster volume tions with 'performance' in it. I guess you can check: 'performance.write-behind-window-size' ,? 'performance.cache-size' , 'performance.flush-behind' & 'performance.stat-prefetch' options. Best Regards, Strahil Nikolov On Jul 24, 2019 10:33, peter knezel wrote: > > Exactly, i wanted to do a simulation of a problem that appears in a real situation > where there is a latency around 11ms. (same OS, same gluster package) > > Can you write more about > > "client caching techniques that will reduce that impact" ? > > Are the parameters set ok? > > Thanks, > > peterk -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.knezel at gmail.com Wed Jul 24 07:55:30 2019 From: peter.knezel at gmail.com (peter knezel) Date: Wed, 24 Jul 2019 09:55:30 +0200 Subject: [Gluster-users] ncreased latency causes rapid decrease of ftp transfer from/to glusterfs filesystem Message-ID: Thanks Strahil for the hints. Should gluster volume be stopped before i am changing/testing these parameters? Kind regards, peterk -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Wed Jul 24 08:04:51 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Wed, 24 Jul 2019 11:04:51 +0300 Subject: [Gluster-users] ncreased latency causes rapid decrease of ftp transfer from/to glusterfs filesystem Message-ID: I have only used one them, but I doubt you need to stop the volume to take effect. Keep in mind that there are a lot of options to play with. Check the following page for description of the option and expected effects: https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/ I would recommend you to start with a small test volume before implementing the changes on prod. Best Regards, Strahil NikolovOn Jul 24, 2019 10:55, peter knezel wrote: > > Thanks Strahil for the hints. > > Should gluster volume be stopped before i am changing/testing these parameters? > > Kind regards, > > peterk > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.knezel at gmail.com Wed Jul 24 08:13:47 2019 From: peter.knezel at gmail.com (peter knezel) Date: Wed, 24 Jul 2019 10:13:47 +0200 Subject: [Gluster-users] increased latency causes rapid decrease of ftp transfer from/to glusterfs filesystem Message-ID: Thanks for the explanation and the link. Since my setup is a test environment, i can change parameters at my will. Kind regards, peterk -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgowtham at redhat.com Wed Jul 24 13:35:47 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Wed, 24 Jul 2019 19:05:47 +0530 Subject: [Gluster-users] Announcing Gluster release 5.8 Message-ID: Hi, The Gluster community is pleased to announce the release of Gluster 5.8 (packages available at [1]). Release notes for the release can be found at [2]. Major changes, features and limitations addressed in this release: https://bugzilla.redhat.com/1728988 was an issue blocking the build. NOTE: The 5.7 was dead on release. Thanks, Gluster community [1] Packages for 5.8: https://download.gluster.org/pub/gluster/glusterfs/5/5.8/ [2] Release notes for 5.8: https://docs.gluster.org/en/latest/release-notes/5.8/ -- Regards, Hari Gowtham. From matthewb at uvic.ca Wed Jul 24 16:43:11 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Wed, 24 Jul 2019 09:43:11 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: Message-ID: So looking more closely at the trusted.glusterfs.dht attributes from the bricks it looks like they cover the entire range... and there is no range left for gluster07. The first 6 bricks range from 0x00000000 to 0xffffffff - so... is there a way to re-calculate what the dht values should be? Each of the bricks should have a gap Gluster05 00000000 -> 2aaaaaa9 Gluster06 2aaaaaaa -> 55555553 Gluster01 55555554 -> 7ffffffd Gluster02 7ffffffe -> aaaaaaa7 Gluster03 aaaaaaa8 -> d5555551 Gluster04 d5555552 -> ffffffff Gluster07 None If we split the range into 7 servers that would be a gap of about 0x24924924 for each server. Now in terms of the gluster07 brick, about 2 years ago the RAID array the brick was stored on became corrupted. I ran the remove-brick force command, then provisioned a new server, ran the add-brick command and then restored the missing files from backup by copying them back to the main gluster mount (not the brick). It looks like prior to that event this was the layout - which would make sense given the equal size of the 7 bricks: gluster02.pcic.uvic.ca | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f gluster05.pcic.uvic.ca | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f gluster04.pcic.uvic.ca | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf gluster03.pcic.uvic.ca | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f gluster01.pcic.uvic.ca | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f gluster07.pcic.uvic.ca | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x000000010000000000000000245fffdf gluster06.pcic.uvic.ca | SUCCESS | rc=0 >> # file: /mnt/raid6-storage/storage trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff Which yields the following: 00000000 -> 245fffdf??? Gluster07 245fffe0 -> 48bfff1f??? Gluster01 48bfff20 -> 6d1ffe5f??? Gluster02 6d1ffe60 -> 917ffd9f??? Gluster03 917ffda0 -> b5dffcdf??? Gluster04 b5dffce0 -> da3ffc1f??? Gluster05 da3ffc20 -> ffffffff??? Gluster06 Is there some way to get back to this? Thanks, ?-Matthew -- Matthew Benstead System Administrator Pacific Climate Impacts Consortium University of Victoria, UH1 PO Box 1800, STN CSC Victoria, BC, V8W 2Y2 Phone: +1-250-721-8432 Email: matthewb at uvic.ca On 7/18/19 7:20 AM, Matthew Benstead wrote: > Hi Nithya, > > No - it was added about a year and a half ago. I have tried > re-mounting the volume on the server, but it didn't add the attr: > > [root at gluster07 ~]# umount /storage/ > [root at gluster07 ~]# cat /etc/fstab | grep "/storage" > 10.0.231.56:/storage /storage glusterfs > defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 > [root at gluster07 ~]# mount /storage/ > [root at gluster07 ~]# df -h /storage/ > Filesystem??????????? Size? Used Avail Use% Mounted on > 10.0.231.56:/storage? 255T? 194T?? 62T? 77% /storage > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ > # file: /mnt/raid6-storage/storage/ > security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 > trusted.gfid=0x00000000000000000000000000000001 > trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 > trusted.glusterfs.quota.dirty=0x3000 > trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 > trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 > > Thanks, > ?-Matthew > > On 7/17/19 10:04 PM, Nithya Balachandran wrote: >> Hi Matthew, >> >> Was this node/brick added to the volume recently? If yes, try >> mounting the volume on a fresh mount point - that should create the >> xattr on this as well. >> >> Regards, >> Nithya >> >> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead > > wrote: >> >> Hello, >> >> I've just noticed one brick in my 7 node distribute volume is missing >> the trusted.glusterfs.dht xattr...? How can I fix this? >> >> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >> >> All of the other nodes are fine, but gluster07 from the list >> below does >> not have the attribute. >> >> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr >> -m . >> --absolute-names -n trusted.glusterfs.dht -e hex >> /mnt/raid6-storage/storage" >> ... >> gluster05 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >> >> gluster03 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >> >> gluster04 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >> >> gluster06 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >> >> gluster02 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >> >> gluster07 | FAILED | rc=1 >> >> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >> attributenon-zero return code >> >> gluster01 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >> >> Here are all of the attr's from the brick: >> >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ >> # file: /mnt/raid6-storage/storage/ >> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.gfid=0x00000000000000000000000000000001 >> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >> trusted.glusterfs.quota.dirty=0x3000 >> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >> >> >> And here is the volume information: >> >> [root at gluster07 ~]# gluster volume info storage >> >> Volume Name: storage >> Type: Distribute >> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 7 >> Transport-type: tcp >> Bricks: >> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >> Options Reconfigured: >> changelog.changelog: on >> features.quota-deem-statfs: on >> features.read-only: off >> features.inode-quota: on >> features.quota: on >> performance.readdir-ahead: on >> nfs.disable: on >> geo-replication.indexing: on >> geo-replication.ignore-pid-check: on >> transport.address-family: inet >> >> Thanks, >> ?-Matthew >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Thu Jul 25 04:30:40 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Thu, 25 Jul 2019 10:00:40 +0530 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: Message-ID: On Wed, 24 Jul 2019 at 22:12, Matthew Benstead wrote: > So looking more closely at the trusted.glusterfs.dht attributes from the > bricks it looks like they cover the entire range... and there is no range > left for gluster07. > > The first 6 bricks range from 0x00000000 to 0xffffffff - so... is there a > way to re-calculate what the dht values should be? Each of the bricks > should have a gap > > Gluster05 00000000 -> 2aaaaaa9 > Gluster06 2aaaaaaa -> 55555553 > Gluster01 55555554 -> 7ffffffd > Gluster02 7ffffffe -> aaaaaaa7 > Gluster03 aaaaaaa8 -> d5555551 > Gluster04 d5555552 -> ffffffff > Gluster07 None > > If we split the range into 7 servers that would be a gap of about > 0x24924924 for each server. > > Now in terms of the gluster07 brick, about 2 years ago the RAID array the > brick was stored on became corrupted. I ran the remove-brick force command, > then provisioned a new server, ran the add-brick command and then restored > the missing files from backup by copying them back to the main gluster > mount (not the brick). > > Did you run a rebalance after performing the add-brick? Without a rebalance/fix-layout , the layout for existing directories on the volume will not be updated to use the new brick as well. That the layout does not include the new brick in the root dir is in itself is not a problem. Do you create a lot of files directly in the root of the volume? If yes, you might want to run a rebalance. Otherwise, if you mostly create files in newly added directories, you can probably ignore this. You can check the layout for directories on the volume and see if they incorporate the brick7. I would expect a lookup on the root to have set an xattr on the brick with an empty layout range . The fact that the xattr does not exist at all on the brick is what I am looking into. It looks like prior to that event this was the layout - which would make > sense given the equal size of the 7 bricks: > > gluster02.pcic.uvic.ca | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f > > gluster05.pcic.uvic.ca | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f > > gluster04.pcic.uvic.ca | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf > > gluster03.pcic.uvic.ca | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f > > gluster01.pcic.uvic.ca | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f > > gluster07.pcic.uvic.ca | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x000000010000000000000000245fffdf > > gluster06.pcic.uvic.ca | SUCCESS | rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff > > Which yields the following: > > 00000000 -> 245fffdf Gluster07 > 245fffe0 -> 48bfff1f Gluster01 > 48bfff20 -> 6d1ffe5f Gluster02 > 6d1ffe60 -> 917ffd9f Gluster03 > 917ffda0 -> b5dffcdf Gluster04 > b5dffce0 -> da3ffc1f Gluster05 > da3ffc20 -> ffffffff Gluster06 > > Is there some way to get back to this? > > Thanks, > -Matthew > > -- > Matthew Benstead > System Administrator > Pacific Climate Impacts Consortium > University of Victoria, UH1 > PO Box 1800, STN CSC > Victoria, BC, V8W 2Y2 > Phone: +1-250-721-8432 > Email: matthewb at uvic.ca > On 7/18/19 7:20 AM, Matthew Benstead wrote: > > Hi Nithya, > > No - it was added about a year and a half ago. I have tried re-mounting > the volume on the server, but it didn't add the attr: > > [root at gluster07 ~]# umount /storage/ > [root at gluster07 ~]# cat /etc/fstab | grep "/storage" > 10.0.231.56:/storage /storage glusterfs > defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 > [root at gluster07 ~]# mount /storage/ > [root at gluster07 ~]# df -h /storage/ > Filesystem Size Used Avail Use% Mounted on > 10.0.231.56:/storage 255T 194T 62T 77% /storage > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ > # file: /mnt/raid6-storage/storage/ > > security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 > trusted.gfid=0x00000000000000000000000000000001 > > trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 > trusted.glusterfs.quota.dirty=0x3000 > > trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 > trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 > > Thanks, > -Matthew > > On 7/17/19 10:04 PM, Nithya Balachandran wrote: > > Hi Matthew, > > Was this node/brick added to the volume recently? If yes, try mounting the > volume on a fresh mount point - that should create the xattr on this as > well. > > Regards, > Nithya > > On Wed, 17 Jul 2019 at 21:01, Matthew Benstead wrote: > >> Hello, >> >> I've just noticed one brick in my 7 node distribute volume is missing >> the trusted.glusterfs.dht xattr...? How can I fix this? >> >> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >> >> All of the other nodes are fine, but gluster07 from the list below does >> not have the attribute. >> >> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . >> --absolute-names -n trusted.glusterfs.dht -e hex >> /mnt/raid6-storage/storage" >> ... >> gluster05 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >> >> gluster03 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >> >> gluster04 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >> >> gluster06 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >> >> gluster02 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >> >> gluster07 | FAILED | rc=1 >> >> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >> attributenon-zero return code >> >> gluster01 | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >> >> Here are all of the attr's from the brick: >> >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ >> # file: /mnt/raid6-storage/storage/ >> >> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.gfid=0x00000000000000000000000000000001 >> >> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >> trusted.glusterfs.quota.dirty=0x3000 >> >> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >> >> >> And here is the volume information: >> >> [root at gluster07 ~]# gluster volume info storage >> >> Volume Name: storage >> Type: Distribute >> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 7 >> Transport-type: tcp >> Bricks: >> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >> Options Reconfigured: >> changelog.changelog: on >> features.quota-deem-statfs: on >> features.read-only: off >> features.inode-quota: on >> features.quota: on >> performance.readdir-ahead: on >> nfs.disable: on >> geo-replication.indexing: on >> geo-replication.ignore-pid-check: on >> transport.address-family: inet >> >> Thanks, >> -Matthew >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewb at uvic.ca Thu Jul 25 20:27:08 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Thu, 25 Jul 2019 13:27:08 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: Message-ID: <722b47be-cb8a-c036-9bd8-4913b6fd7ec6@uvic.ca> Hi Nithya, Hmm... I don't remember if I did, but based on what I'm seeing it sounds like I probably didn't run rebalance or fix-layout. It looks like folders that haven't had any new files created have a dht of 0, while other folders have non-zero values. [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ | grep dht [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/home | grep dht trusted.glusterfs.dht=0x00000000000000000000000000000000 [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/home/matthewb | grep dht trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 If I just run the fix-layout command will it re-create all of the dht values or just the missing ones? Since the brick is already fairly size balanced could I get away with running fix-layout but not rebalance? Or would the new dht layout mean slower accesses since the files may be expected on different bricks? Thanks, ?-Matthew -- Matthew Benstead System Administrator Pacific Climate Impacts Consortium University of Victoria, UH1 PO Box 1800, STN CSC Victoria, BC, V8W 2Y2 Phone: +1-250-721-8432 Email: matthewb at uvic.ca On 7/24/19 9:30 PM, Nithya Balachandran wrote: > > > On Wed, 24 Jul 2019 at 22:12, Matthew Benstead > wrote: > > So looking more closely at the trusted.glusterfs.dht attributes > from the bricks it looks like they cover the entire range... and > there is no range left for gluster07. > > The first 6 bricks range from 0x00000000 to 0xffffffff - so... is > there a way to re-calculate what the dht values should be? Each of > the bricks should have a gap > > Gluster05 00000000 -> 2aaaaaa9 > Gluster06 2aaaaaaa -> 55555553 > Gluster01 55555554 -> 7ffffffd > Gluster02 7ffffffe -> aaaaaaa7 > Gluster03 aaaaaaa8 -> d5555551 > Gluster04 d5555552 -> ffffffff > Gluster07 None > > If we split the range into 7 servers that would be a gap of about > 0x24924924 for each server. > > Now in terms of the gluster07 brick, about 2 years ago the RAID > array the brick was stored on became corrupted. I ran the > remove-brick force command, then provisioned a new server, ran the > add-brick command and then restored the missing files from backup > by copying them back to the main gluster mount (not the brick). > > > Did you run a rebalance after performing the add-brick? Without a > rebalance/fix-layout , the layout for existing directories on the > volume will not? be updated to use the new brick as well. > > That the layout does not include the new brick in the root dir is in > itself is not a problem. Do you create a lot of files directly in the > root of the volume? If yes, you might want to run a rebalance. > Otherwise, if you mostly create files in newly added directories, you > can probably ignore this. You can check the layout for directories on > the volume and see if they incorporate the brick7. > > I would expect a lookup on the root to have set an xattr on the brick > with an empty layout range . The fact that the xattr does not exist at > all on the brick is what I am looking into. > > > It looks like prior to that event this was the layout - which > would make sense given the equal size of the 7 bricks: > > gluster02.pcic.uvic.ca | SUCCESS | > rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f > > gluster05.pcic.uvic.ca | SUCCESS | > rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f > > gluster04.pcic.uvic.ca | SUCCESS | > rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf > > gluster03.pcic.uvic.ca | SUCCESS | > rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f > > gluster01.pcic.uvic.ca | SUCCESS | > rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f > > gluster07.pcic.uvic.ca | SUCCESS | > rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x000000010000000000000000245fffdf > > gluster06.pcic.uvic.ca | SUCCESS | > rc=0 >> > # file: /mnt/raid6-storage/storage > trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff > > Which yields the following: > > 00000000 -> 245fffdf??? Gluster07 > 245fffe0 -> 48bfff1f??? Gluster01 > 48bfff20 -> 6d1ffe5f??? Gluster02 > 6d1ffe60 -> 917ffd9f??? Gluster03 > 917ffda0 -> b5dffcdf??? Gluster04 > b5dffce0 -> da3ffc1f??? Gluster05 > da3ffc20 -> ffffffff??? Gluster06 > > Is there some way to get back to this? > > Thanks, > ?-Matthew > > -- > Matthew Benstead > System Administrator > Pacific Climate Impacts Consortium > University of Victoria, UH1 > PO Box 1800, STN CSC > Victoria, BC, V8W 2Y2 > Phone: +1-250-721-8432 > Email: matthewb at uvic.ca > > On 7/18/19 7:20 AM, Matthew Benstead wrote: >> Hi Nithya, >> >> No - it was added about a year and a half ago. I have tried >> re-mounting the volume on the server, but it didn't add the attr: >> >> [root at gluster07 ~]# umount /storage/ >> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >> 10.0.231.56:/storage /storage glusterfs >> defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >> [root at gluster07 ~]# mount /storage/ >> [root at gluster07 ~]# df -h /storage/ >> Filesystem??????????? Size? Used Avail Use% Mounted on >> 10.0.231.56:/storage? 255T? 194T?? 62T? 77% /storage >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ >> # file: /mnt/raid6-storage/storage/ >> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.gfid=0x00000000000000000000000000000001 >> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >> trusted.glusterfs.quota.dirty=0x3000 >> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >> >> Thanks, >> ?-Matthew >> >> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >>> Hi Matthew, >>> >>> Was this node/brick added to the volume recently? If yes, try >>> mounting the volume on a fresh mount point - that should create >>> the xattr on this as well. >>> >>> Regards, >>> Nithya >>> >>> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead >> > wrote: >>> >>> Hello, >>> >>> I've just noticed one brick in my 7 node distribute volume >>> is missing >>> the trusted.glusterfs.dht xattr...? How can I fix this? >>> >>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>> >>> All of the other nodes are fine, but gluster07 from the list >>> below does >>> not have the attribute. >>> >>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a >>> "getfattr -m . >>> --absolute-names -n trusted.glusterfs.dht -e hex >>> /mnt/raid6-storage/storage" >>> ... >>> gluster05 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>> >>> gluster03 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>> >>> gluster04 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>> >>> gluster06 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>> >>> gluster02 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>> >>> gluster07 | FAILED | rc=1 >> >>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>> attributenon-zero return code >>> >>> gluster01 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>> >>> Here are all of the attr's from the brick: >>> >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>> /mnt/raid6-storage/storage/ >>> # file: /mnt/raid6-storage/storage/ >>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>> trusted.gfid=0x00000000000000000000000000000001 >>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>> trusted.glusterfs.quota.dirty=0x3000 >>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>> >>> >>> And here is the volume information: >>> >>> [root at gluster07 ~]# gluster volume info storage >>> >>> Volume Name: storage >>> Type: Distribute >>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 7 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>> Options Reconfigured: >>> changelog.changelog: on >>> features.quota-deem-statfs: on >>> features.read-only: off >>> features.inode-quota: on >>> features.quota: on >>> performance.readdir-ahead: on >>> nfs.disable: on >>> geo-replication.indexing: on >>> geo-replication.ignore-pid-check: on >>> transport.address-family: inet >>> >>> Thanks, >>> ?-Matthew >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Fri Jul 26 07:02:14 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Fri, 26 Jul 2019 12:32:14 +0530 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: <722b47be-cb8a-c036-9bd8-4913b6fd7ec6@uvic.ca> References: <722b47be-cb8a-c036-9bd8-4913b6fd7ec6@uvic.ca> Message-ID: On Fri, 26 Jul 2019 at 01:56, Matthew Benstead wrote: > Hi Nithya, > > Hmm... I don't remember if I did, but based on what I'm seeing it sounds > like I probably didn't run rebalance or fix-layout. > > It looks like folders that haven't had any new files created have a dht of > 0, while other folders have non-zero values. > > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ | grep dht > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/home | grep dht > trusted.glusterfs.dht=0x00000000000000000000000000000000 > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/home/matthewb | grep dht > trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 > > If I just run the fix-layout command will it re-create all of the dht > values or just the missing ones? > A fix-layout will recalculate the layouts entirely so files all the values will change. No files will be moved. A rebalance will recalculate the layouts like the fix-layout but will also move files to their new locations based on the new layout ranges. This could take a lot of time depending on the number of files/directories on the volume. If you do this, I would recommend that you turn off lookup-optimize until the rebalance is over. > Since the brick is already fairly size balanced could I get away with > running fix-layout but not rebalance? Or would the new dht layout mean > slower accesses since the files may be expected on different bricks? > The first access for a file will be slower. The next one will be faster as the location will be cached in the client's in-memory structures. You may not need to run either a fix-layout or a rebalance if new file creations will be in directories created after the add-brick. Gluster will automatically include all 7 bricks for those directories. Regards, Nithya > Thanks, > -Matthew > > -- > Matthew Benstead > System Administrator > Pacific Climate Impacts Consortium > University of Victoria, UH1 > PO Box 1800, STN CSC > Victoria, BC, V8W 2Y2 > Phone: +1-250-721-8432 > Email: matthewb at uvic.ca > On 7/24/19 9:30 PM, Nithya Balachandran wrote: > > > > On Wed, 24 Jul 2019 at 22:12, Matthew Benstead wrote: > >> So looking more closely at the trusted.glusterfs.dht attributes from the >> bricks it looks like they cover the entire range... and there is no range >> left for gluster07. >> >> The first 6 bricks range from 0x00000000 to 0xffffffff - so... is there a >> way to re-calculate what the dht values should be? Each of the bricks >> should have a gap >> >> Gluster05 00000000 -> 2aaaaaa9 >> Gluster06 2aaaaaaa -> 55555553 >> Gluster01 55555554 -> 7ffffffd >> Gluster02 7ffffffe -> aaaaaaa7 >> Gluster03 aaaaaaa8 -> d5555551 >> Gluster04 d5555552 -> ffffffff >> Gluster07 None >> >> If we split the range into 7 servers that would be a gap of about >> 0x24924924 for each server. >> >> Now in terms of the gluster07 brick, about 2 years ago the RAID array the >> brick was stored on became corrupted. I ran the remove-brick force command, >> then provisioned a new server, ran the add-brick command and then restored >> the missing files from backup by copying them back to the main gluster >> mount (not the brick). >> >> > Did you run a rebalance after performing the add-brick? Without a > rebalance/fix-layout , the layout for existing directories on the volume > will not be updated to use the new brick as well. > > That the layout does not include the new brick in the root dir is in > itself is not a problem. Do you create a lot of files directly in the root > of the volume? If yes, you might want to run a rebalance. Otherwise, if you > mostly create files in newly added directories, you can probably ignore > this. You can check the layout for directories on the volume and see if > they incorporate the brick7. > > I would expect a lookup on the root to have set an xattr on the brick with > an empty layout range . The fact that the xattr does not exist at all on > the brick is what I am looking into. > > > It looks like prior to that event this was the layout - which would make >> sense given the equal size of the 7 bricks: >> >> gluster02.pcic.uvic.ca | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f >> >> gluster05.pcic.uvic.ca | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f >> >> gluster04.pcic.uvic.ca | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf >> >> gluster03.pcic.uvic.ca | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f >> >> gluster01.pcic.uvic.ca | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f >> >> gluster07.pcic.uvic.ca | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x000000010000000000000000245fffdf >> >> gluster06.pcic.uvic.ca | SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff >> >> Which yields the following: >> >> 00000000 -> 245fffdf Gluster07 >> 245fffe0 -> 48bfff1f Gluster01 >> 48bfff20 -> 6d1ffe5f Gluster02 >> 6d1ffe60 -> 917ffd9f Gluster03 >> 917ffda0 -> b5dffcdf Gluster04 >> b5dffce0 -> da3ffc1f Gluster05 >> da3ffc20 -> ffffffff Gluster06 >> >> Is there some way to get back to this? >> >> Thanks, >> -Matthew >> >> -- >> Matthew Benstead >> System Administrator >> Pacific Climate Impacts Consortium >> University of Victoria, UH1 >> PO Box 1800, STN CSC >> Victoria, BC, V8W 2Y2 >> Phone: +1-250-721-8432 >> Email: matthewb at uvic.ca >> On 7/18/19 7:20 AM, Matthew Benstead wrote: >> >> Hi Nithya, >> >> No - it was added about a year and a half ago. I have tried re-mounting >> the volume on the server, but it didn't add the attr: >> >> [root at gluster07 ~]# umount /storage/ >> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >> 10.0.231.56:/storage /storage glusterfs >> defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >> [root at gluster07 ~]# mount /storage/ >> [root at gluster07 ~]# df -h /storage/ >> Filesystem Size Used Avail Use% Mounted on >> 10.0.231.56:/storage 255T 194T 62T 77% /storage >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ >> # file: /mnt/raid6-storage/storage/ >> >> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.gfid=0x00000000000000000000000000000001 >> >> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >> trusted.glusterfs.quota.dirty=0x3000 >> >> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >> >> Thanks, >> -Matthew >> >> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >> >> Hi Matthew, >> >> Was this node/brick added to the volume recently? If yes, try mounting >> the volume on a fresh mount point - that should create the xattr on this as >> well. >> >> Regards, >> Nithya >> >> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead wrote: >> >>> Hello, >>> >>> I've just noticed one brick in my 7 node distribute volume is missing >>> the trusted.glusterfs.dht xattr...? How can I fix this? >>> >>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>> >>> All of the other nodes are fine, but gluster07 from the list below does >>> not have the attribute. >>> >>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . >>> --absolute-names -n trusted.glusterfs.dht -e hex >>> /mnt/raid6-storage/storage" >>> ... >>> gluster05 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>> >>> gluster03 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>> >>> gluster04 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>> >>> gluster06 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>> >>> gluster02 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>> >>> gluster07 | FAILED | rc=1 >> >>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>> attributenon-zero return code >>> >>> gluster01 | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>> >>> Here are all of the attr's from the brick: >>> >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>> /mnt/raid6-storage/storage/ >>> # file: /mnt/raid6-storage/storage/ >>> >>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>> trusted.gfid=0x00000000000000000000000000000001 >>> >>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>> trusted.glusterfs.quota.dirty=0x3000 >>> >>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>> >>> >>> And here is the volume information: >>> >>> [root at gluster07 ~]# gluster volume info storage >>> >>> Volume Name: storage >>> Type: Distribute >>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 7 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>> Options Reconfigured: >>> changelog.changelog: on >>> features.quota-deem-statfs: on >>> features.read-only: off >>> features.inode-quota: on >>> features.quota: on >>> performance.readdir-ahead: on >>> nfs.disable: on >>> geo-replication.indexing: on >>> geo-replication.ignore-pid-check: on >>> transport.address-family: inet >>> >>> Thanks, >>> -Matthew >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewb at uvic.ca Fri Jul 26 21:01:51 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Fri, 26 Jul 2019 14:01:51 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: <722b47be-cb8a-c036-9bd8-4913b6fd7ec6@uvic.ca> Message-ID: Ok thank-you for explaining everything - that makes sense. Currently the brick file systems are pretty evenly distributed so I probably won't run the fix-layout right now. Would this state have any impact on geo-replication? I'm trying to geo-replicate this volume, but am getting a weird error: "Changelog register failed error=[Errno 21] Is a directory" I assume this is related to something else, but I wasn't sure. Thanks, ?-Matthew -- Matthew Benstead System Administrator Pacific Climate Impacts Consortium University of Victoria, UH1 PO Box 1800, STN CSC Victoria, BC, V8W 2Y2 Phone: +1-250-721-8432 Email: matthewb at uvic.ca On 7/26/19 12:02 AM, Nithya Balachandran wrote: > > > On Fri, 26 Jul 2019 at 01:56, Matthew Benstead > wrote: > > Hi Nithya, > > Hmm... I don't remember if I did, but based on what I'm seeing it > sounds like I probably didn't run rebalance or fix-layout. > > It looks like folders that haven't had any new files created have > a dht of 0, while other folders have non-zero values. > > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ | grep dht > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/home | grep dht > trusted.glusterfs.dht=0x00000000000000000000000000000000 > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/home/matthewb | grep dht > trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 > > If I just run the fix-layout command will it re-create all of the > dht values or just the missing ones? > > > A fix-layout will recalculate the layouts entirely so files all the > values will change. No files will be moved. > A rebalance will recalculate the layouts like the fix-layout but will > also move files to their new locations based on the new layout ranges. > This could take a lot of time depending on the number of > files/directories on the volume. If you do this, I would recommend > that you turn off lookup-optimize until the rebalance is over. > ? > > Since the brick is already fairly size balanced could I get away > with running fix-layout but not rebalance? Or would the new dht > layout mean slower accesses since the files may be expected on > different bricks? > > ? > The first access for a file will be slower. The next one will be > faster as the location will be cached in the client's in-memory > structures. > You may not need to run either a fix-layout or a rebalance if new file > creations will be in directories created after the add-brick. Gluster > will automatically include all 7 bricks for those directories. > > Regards, > Nithya > > > Thanks, > ?-Matthew > > -- > Matthew Benstead > System Administrator > Pacific Climate Impacts Consortium > University of Victoria, UH1 > PO Box 1800, STN CSC > Victoria, BC, V8W 2Y2 > Phone: +1-250-721-8432 > Email: matthewb at uvic.ca > > On 7/24/19 9:30 PM, Nithya Balachandran wrote: >> >> >> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead > > wrote: >> >> So looking more closely at the trusted.glusterfs.dht >> attributes from the bricks it looks like they cover the >> entire range... and there is no range left for gluster07. >> >> The first 6 bricks range from 0x00000000 to 0xffffffff - >> so... is there a way to re-calculate what the dht values >> should be? Each of the bricks should have a gap >> >> Gluster05 00000000 -> 2aaaaaa9 >> Gluster06 2aaaaaaa -> 55555553 >> Gluster01 55555554 -> 7ffffffd >> Gluster02 7ffffffe -> aaaaaaa7 >> Gluster03 aaaaaaa8 -> d5555551 >> Gluster04 d5555552 -> ffffffff >> Gluster07 None >> >> If we split the range into 7 servers that would be a gap of >> about 0x24924924 for each server. >> >> Now in terms of the gluster07 brick, about 2 years ago the >> RAID array the brick was stored on became corrupted. I ran >> the remove-brick force command, then provisioned a new >> server, ran the add-brick command and then restored the >> missing files from backup by copying them back to the main >> gluster mount (not the brick). >> >> >> Did you run a rebalance after performing the add-brick? Without a >> rebalance/fix-layout , the layout for existing directories on the >> volume will not? be updated to use the new brick as well. >> >> That the layout does not include the new brick in the root dir is >> in itself is not a problem. Do you create a lot of files directly >> in the root of the volume? If yes, you might want to run a >> rebalance. Otherwise, if you mostly create files in newly added >> directories, you can probably ignore this. You can check the >> layout for directories on the volume and see if they incorporate >> the brick7. >> >> I would expect a lookup on the root to have set an xattr on the >> brick with an empty layout range . The fact that the xattr does >> not exist at all on the brick is what I am looking into. >> >> >> It looks like prior to that event this was the layout - which >> would make sense given the equal size of the 7 bricks: >> >> gluster02.pcic.uvic.ca | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f >> >> gluster05.pcic.uvic.ca | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f >> >> gluster04.pcic.uvic.ca | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf >> >> gluster03.pcic.uvic.ca | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f >> >> gluster01.pcic.uvic.ca | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f >> >> gluster07.pcic.uvic.ca | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x000000010000000000000000245fffdf >> >> gluster06.pcic.uvic.ca | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff >> >> Which yields the following: >> >> 00000000 -> 245fffdf??? Gluster07 >> 245fffe0 -> 48bfff1f??? Gluster01 >> 48bfff20 -> 6d1ffe5f??? Gluster02 >> 6d1ffe60 -> 917ffd9f??? Gluster03 >> 917ffda0 -> b5dffcdf??? Gluster04 >> b5dffce0 -> da3ffc1f??? Gluster05 >> da3ffc20 -> ffffffff??? Gluster06 >> >> Is there some way to get back to this? >> >> Thanks, >> ?-Matthew >> >> -- >> Matthew Benstead >> System Administrator >> Pacific Climate Impacts Consortium >> University of Victoria, UH1 >> PO Box 1800, STN CSC >> Victoria, BC, V8W 2Y2 >> Phone: +1-250-721-8432 >> Email: matthewb at uvic.ca >> >> On 7/18/19 7:20 AM, Matthew Benstead wrote: >>> Hi Nithya, >>> >>> No - it was added about a year and a half ago. I have tried >>> re-mounting the volume on the server, but it didn't add the >>> attr: >>> >>> [root at gluster07 ~]# umount /storage/ >>> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >>> 10.0.231.56:/storage /storage glusterfs >>> defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >>> [root at gluster07 ~]# mount /storage/ >>> [root at gluster07 ~]# df -h /storage/ >>> Filesystem??????????? Size? Used Avail Use% Mounted on >>> 10.0.231.56:/storage? 255T? 194T?? 62T? 77% /storage >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>> /mnt/raid6-storage/storage/ >>> # file: /mnt/raid6-storage/storage/ >>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>> trusted.gfid=0x00000000000000000000000000000001 >>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >>> trusted.glusterfs.quota.dirty=0x3000 >>> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>> >>> Thanks, >>> ?-Matthew >>> >>> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >>>> Hi Matthew, >>>> >>>> Was this node/brick added to the volume recently? If yes, >>>> try mounting the volume on a fresh mount point - that >>>> should create the xattr on this as well. >>>> >>>> Regards, >>>> Nithya >>>> >>>> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead >>>> > wrote: >>>> >>>> Hello, >>>> >>>> I've just noticed one brick in my 7 node distribute >>>> volume is missing >>>> the trusted.glusterfs.dht xattr...? How can I fix this? >>>> >>>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>>> >>>> All of the other nodes are fine, but gluster07 from the >>>> list below does >>>> not have the attribute. >>>> >>>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a >>>> "getfattr -m . >>>> --absolute-names -n trusted.glusterfs.dht -e hex >>>> /mnt/raid6-storage/storage" >>>> ... >>>> gluster05 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>>> >>>> gluster03 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>>> >>>> gluster04 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>>> >>>> gluster06 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>>> >>>> gluster02 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>>> >>>> gluster07 | FAILED | rc=1 >> >>>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>>> attributenon-zero return code >>>> >>>> gluster01 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>>> >>>> Here are all of the attr's from the brick: >>>> >>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d >>>> -e hex >>>> /mnt/raid6-storage/storage/ >>>> # file: /mnt/raid6-storage/storage/ >>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.gfid=0x00000000000000000000000000000001 >>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>>> trusted.glusterfs.quota.dirty=0x3000 >>>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>> >>>> >>>> And here is the volume information: >>>> >>>> [root at gluster07 ~]# gluster volume info storage >>>> >>>> Volume Name: storage >>>> Type: Distribute >>>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>>> Status: Started >>>> Snapshot Count: 0 >>>> Number of Bricks: 7 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>>> Options Reconfigured: >>>> changelog.changelog: on >>>> features.quota-deem-statfs: on >>>> features.read-only: off >>>> features.inode-quota: on >>>> features.quota: on >>>> performance.readdir-ahead: on >>>> nfs.disable: on >>>> geo-replication.indexing: on >>>> geo-replication.ignore-pid-check: on >>>> transport.address-family: inet >>>> >>>> Thanks, >>>> ?-Matthew >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From runmatt at live.com Fri Jul 26 13:20:27 2019 From: runmatt at live.com (Matthew Evans) Date: Fri, 26 Jul 2019 13:20:27 +0000 Subject: [Gluster-users] GlusterFS Changing Hash of Large Files? Message-ID: I've got a new glusterfs 4 node replica cluster running under CentOS 7. All hosts are backed by SSD drives and are connected to a 1Gbps Ethernet network. 3 nodes are running on CentOS 7 under ESXi on the same physical host, 1 is running on CentOS 7 under Hyper-V. I use this for my docker swarm persistent storage and all seems to work well. Yesterday however, I copied a 4GB .ISO file to my volume for a friend to download. I noticed the SHA256 hash of the ISO was altered. I downloaded a fresh copy to my desktop, verified the hash, scp'd it to the local glusterfs host storage and again, re-verified the hash. The moment I copied it to my glusterfs volume, the file hash changed. When my friend downloaded the ISO, his hash matched changed hash. I am new to glusterfs, having deployed this as my first cluster ever about a week ago. Can someone help me work through why this file's hash would be changing? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ravishankar at redhat.com Sat Jul 27 06:04:48 2019 From: ravishankar at redhat.com (Ravishankar N) Date: Sat, 27 Jul 2019 11:34:48 +0530 Subject: [Gluster-users] GlusterFS Changing Hash of Large Files? In-Reply-To: References: Message-ID: On 26/07/19 6:50 PM, Matthew Evans wrote: > I've got a new glusterfs 4 node replica cluster running under CentOS > 7.? All hosts are backed by SSD drives and are connected to a 1Gbps > Ethernet network. 3 nodes are running on CentOS 7 under ESXi on the > same physical host, 1 is running on CentOS 7 under Hyper-V. I use this > for my docker swarm persistent storage and all seems to work well. > > Yesterday however, I copied a 4GB .ISO file to my volume for a friend > to download. I noticed the SHA256 hash of the ISO was altered. I > downloaded a fresh copy to my desktop, verified the hash, scp'd it to > the local glusterfs host storage and again, re-verified the hash. The > moment I copied it to my glusterfs volume, the file hash changed. When > my friend downloaded the ISO, his hash matched changed hash. Can you provide the below details? - glusterfs version -`gluster volume info` > > I am new to glusterfs, having deployed this as my first cluster ever > about a week ago. Can someone help me work through why this file's > hash would be changing? > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From runmatt at live.com Sat Jul 27 13:30:31 2019 From: runmatt at live.com (Matthew Evans) Date: Sat, 27 Jul 2019 13:30:31 +0000 Subject: [Gluster-users] GlusterFS Changing Hash of Large Files? In-Reply-To: References: , Message-ID: Hi Ravishankar - I figured out the issue. The 4th node was showing "online" under 'gluster peer status' as well as 'gluster volume status' - but 'gluster volume status' wasn't showing a TCP port for that 4th node. When I opened 49152 in firewalld and then re-copied the ISO, the hash didn't change. So, now I guess the question would be, why would having one malfunctioning node override 3 functioning nodes and cause a file to be altered? I wasn't performing the initial copy onto the malfunctioning node. matt at docker1:~$ sudo glusterfs --version glusterfs 6.3 matt at docker1:~$ sudo gluster volume info Volume Name: swarm-vols Type: Replicate Volume ID: 0b51e6b3-786e-454e-8a16-89b47e94828a Status: Started Snapshot Count: 0 Number of Bricks: 1 x 4 = 4 Transport-type: tcp Bricks: Brick1: docker1:/gluster/data Brick2: docker2:/gluster/data Brick3: docker3:/gluster/data Brick4: docker4:/gluster/data Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet auth.allow: 10.5.22.* ________________________________ From: Ravishankar N Sent: Saturday, July 27, 2019 2:04 AM To: Matthew Evans ; gluster-users at gluster.org Subject: Re: [Gluster-users] GlusterFS Changing Hash of Large Files? On 26/07/19 6:50 PM, Matthew Evans wrote: I've got a new glusterfs 4 node replica cluster running under CentOS 7. All hosts are backed by SSD drives and are connected to a 1Gbps Ethernet network. 3 nodes are running on CentOS 7 under ESXi on the same physical host, 1 is running on CentOS 7 under Hyper-V. I use this for my docker swarm persistent storage and all seems to work well. Yesterday however, I copied a 4GB .ISO file to my volume for a friend to download. I noticed the SHA256 hash of the ISO was altered. I downloaded a fresh copy to my desktop, verified the hash, scp'd it to the local glusterfs host storage and again, re-verified the hash. The moment I copied it to my glusterfs volume, the file hash changed. When my friend downloaded the ISO, his hash matched changed hash. Can you provide the below details? - glusterfs version -`gluster volume info` I am new to glusterfs, having deployed this as my first cluster ever about a week ago. Can someone help me work through why this file's hash would be changing? _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Sun Jul 28 05:17:35 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Sun, 28 Jul 2019 08:17:35 +0300 Subject: [Gluster-users] GlusterFS Changing Hash of Large Files? Message-ID: I never thought that replica 4 is allowed optiion. I always thought that 3 copies is the maximum. Best Regards, Strahil NikolovOn Jul 27, 2019 16:30, Matthew Evans wrote: > > Hi Ravishankar - I figured out the issue. The 4th node was showing "online" under 'gluster peer status' as well as 'gluster volume status' - but 'gluster volume status' wasn't showing a TCP port for that 4th node. When I opened 49152 in firewalld and then re-copied the ISO, the hash didn't change. > > So, now I guess the question would be, why would having one malfunctioning node override 3 functioning nodes and cause a file to be altered? I wasn't performing the initial copy onto the malfunctioning node. > > matt at docker1:~$ sudo glusterfs --version > glusterfs 6.3 > > matt at docker1:~$ sudo gluster volume info > > Volume Name: swarm-vols > Type: Replicate > Volume ID: 0b51e6b3-786e-454e-8a16-89b47e94828a > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 4 = 4 > Transport-type: tcp > Bricks: > Brick1: docker1:/gluster/data > Brick2: docker2:/gluster/data > Brick3: docker3:/gluster/data > Brick4: docker4:/gluster/data > Options Reconfigured: > performance.client-io-threads: off > nfs.disable: on > transport.address-family: inet > auth.allow: 10.5.22.* > > ________________________________ > From: Ravishankar N > Sent: Saturday, July 27, 2019 2:04 AM > To: Matthew Evans ; gluster-users at gluster.org > Subject: Re: [Gluster-users] GlusterFS Changing Hash of Large Files? > ? > > > On 26/07/19 6:50 PM, Matthew Evans wrote: >> >> I've got a new glusterfs 4 node replica cluster running under CentOS 7.? All hosts are backed by SSD drives and are connected to a 1Gbps Ethernet network.?3 nodes are running on CentOS 7 under ESXi on the same physical host, 1 is running on CentOS 7 under Hyper-V. I use this for my docker swarm persistent storage and all seems to work well. >> >> Yesterday however, I copied a 4GB .ISO file to my volume for a friend to download. I noticed the SHA256 hash of the ISO was altered. I downloaded a fresh copy to my desktop, verified the hash, scp'd it to the local glusterfs host storage and again, re-verified the hash. The moment I copied it to my glusterfs volume, the file hash changed. When my friend downloaded the ISO, his hash matched changed hash. > > Can you provide the below details? > - glusterfs version > > -`gluster volume info` > > >> >> I am new to glusterfs, having deployed this as my first cluster ever about a week ago. Can someone help me work through why this file -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Mon Jul 29 03:43:13 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Mon, 29 Jul 2019 09:13:13 +0530 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: <722b47be-cb8a-c036-9bd8-4913b6fd7ec6@uvic.ca> Message-ID: On Sat, 27 Jul 2019 at 02:31, Matthew Benstead wrote: > Ok thank-you for explaining everything - that makes sense. > > Currently the brick file systems are pretty evenly distributed so I > probably won't run the fix-layout right now. > > Would this state have any impact on geo-replication? I'm trying to > geo-replicate this volume, but am getting a weird error: "Changelog > register failed error=[Errno 21] Is a directory" > It should not. Sunny, can you comment on this? Regards, Nithya > > I assume this is related to something else, but I wasn't sure. > > Thanks, > -Matthew > > -- > Matthew Benstead > System Administrator > Pacific Climate Impacts Consortium > University of Victoria, UH1 > PO Box 1800, STN CSC > Victoria, BC, V8W 2Y2 > Phone: +1-250-721-8432 > Email: matthewb at uvic.ca > On 7/26/19 12:02 AM, Nithya Balachandran wrote: > > > > On Fri, 26 Jul 2019 at 01:56, Matthew Benstead wrote: > >> Hi Nithya, >> >> Hmm... I don't remember if I did, but based on what I'm seeing it sounds >> like I probably didn't run rebalance or fix-layout. >> >> It looks like folders that haven't had any new files created have a dht >> of 0, while other folders have non-zero values. >> >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ | grep dht >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/home | grep dht >> trusted.glusterfs.dht=0x00000000000000000000000000000000 >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/home/matthewb | grep dht >> trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 >> >> If I just run the fix-layout command will it re-create all of the dht >> values or just the missing ones? >> > > A fix-layout will recalculate the layouts entirely so files all the values > will change. No files will be moved. > A rebalance will recalculate the layouts like the fix-layout but will also > move files to their new locations based on the new layout ranges. This > could take a lot of time depending on the number of files/directories on > the volume. If you do this, I would recommend that you turn off > lookup-optimize until the rebalance is over. > > >> Since the brick is already fairly size balanced could I get away with >> running fix-layout but not rebalance? Or would the new dht layout mean >> slower accesses since the files may be expected on different bricks? >> > > The first access for a file will be slower. The next one will be faster as > the location will be cached in the client's in-memory structures. > You may not need to run either a fix-layout or a rebalance if new file > creations will be in directories created after the add-brick. Gluster will > automatically include all 7 bricks for those directories. > > Regards, > Nithya > > >> Thanks, >> -Matthew >> >> -- >> Matthew Benstead >> System Administrator >> Pacific Climate Impacts Consortium >> University of Victoria, UH1 >> PO Box 1800, STN CSC >> Victoria, BC, V8W 2Y2 >> Phone: +1-250-721-8432 >> Email: matthewb at uvic.ca >> On 7/24/19 9:30 PM, Nithya Balachandran wrote: >> >> >> >> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead wrote: >> >>> So looking more closely at the trusted.glusterfs.dht attributes from the >>> bricks it looks like they cover the entire range... and there is no range >>> left for gluster07. >>> >>> The first 6 bricks range from 0x00000000 to 0xffffffff - so... is there >>> a way to re-calculate what the dht values should be? Each of the bricks >>> should have a gap >>> >>> Gluster05 00000000 -> 2aaaaaa9 >>> Gluster06 2aaaaaaa -> 55555553 >>> Gluster01 55555554 -> 7ffffffd >>> Gluster02 7ffffffe -> aaaaaaa7 >>> Gluster03 aaaaaaa8 -> d5555551 >>> Gluster04 d5555552 -> ffffffff >>> Gluster07 None >>> >>> If we split the range into 7 servers that would be a gap of about >>> 0x24924924 for each server. >>> >>> Now in terms of the gluster07 brick, about 2 years ago the RAID array >>> the brick was stored on became corrupted. I ran the remove-brick force >>> command, then provisioned a new server, ran the add-brick command and then >>> restored the missing files from backup by copying them back to the main >>> gluster mount (not the brick). >>> >>> >> Did you run a rebalance after performing the add-brick? Without a >> rebalance/fix-layout , the layout for existing directories on the volume >> will not be updated to use the new brick as well. >> >> That the layout does not include the new brick in the root dir is in >> itself is not a problem. Do you create a lot of files directly in the root >> of the volume? If yes, you might want to run a rebalance. Otherwise, if you >> mostly create files in newly added directories, you can probably ignore >> this. You can check the layout for directories on the volume and see if >> they incorporate the brick7. >> >> I would expect a lookup on the root to have set an xattr on the brick >> with an empty layout range . The fact that the xattr does not exist at all >> on the brick is what I am looking into. >> >> >> It looks like prior to that event this was the layout - which would make >>> sense given the equal size of the 7 bricks: >>> >>> gluster02.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f >>> >>> gluster05.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f >>> >>> gluster04.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf >>> >>> gluster03.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f >>> >>> gluster01.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f >>> >>> gluster07.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x000000010000000000000000245fffdf >>> >>> gluster06.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff >>> >>> Which yields the following: >>> >>> 00000000 -> 245fffdf Gluster07 >>> 245fffe0 -> 48bfff1f Gluster01 >>> 48bfff20 -> 6d1ffe5f Gluster02 >>> 6d1ffe60 -> 917ffd9f Gluster03 >>> 917ffda0 -> b5dffcdf Gluster04 >>> b5dffce0 -> da3ffc1f Gluster05 >>> da3ffc20 -> ffffffff Gluster06 >>> >>> Is there some way to get back to this? >>> >>> Thanks, >>> -Matthew >>> >>> -- >>> Matthew Benstead >>> System Administrator >>> Pacific Climate Impacts Consortium >>> University of Victoria, UH1 >>> PO Box 1800, STN CSC >>> Victoria, BC, V8W 2Y2 >>> Phone: +1-250-721-8432 >>> Email: matthewb at uvic.ca >>> On 7/18/19 7:20 AM, Matthew Benstead wrote: >>> >>> Hi Nithya, >>> >>> No - it was added about a year and a half ago. I have tried re-mounting >>> the volume on the server, but it didn't add the attr: >>> >>> [root at gluster07 ~]# umount /storage/ >>> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >>> 10.0.231.56:/storage /storage glusterfs >>> defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >>> [root at gluster07 ~]# mount /storage/ >>> [root at gluster07 ~]# df -h /storage/ >>> Filesystem Size Used Avail Use% Mounted on >>> 10.0.231.56:/storage 255T 194T 62T 77% /storage >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>> /mnt/raid6-storage/storage/ >>> # file: /mnt/raid6-storage/storage/ >>> >>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>> trusted.gfid=0x00000000000000000000000000000001 >>> >>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >>> trusted.glusterfs.quota.dirty=0x3000 >>> >>> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>> >>> Thanks, >>> -Matthew >>> >>> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >>> >>> Hi Matthew, >>> >>> Was this node/brick added to the volume recently? If yes, try mounting >>> the volume on a fresh mount point - that should create the xattr on this as >>> well. >>> >>> Regards, >>> Nithya >>> >>> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead wrote: >>> >>>> Hello, >>>> >>>> I've just noticed one brick in my 7 node distribute volume is missing >>>> the trusted.glusterfs.dht xattr...? How can I fix this? >>>> >>>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>>> >>>> All of the other nodes are fine, but gluster07 from the list below does >>>> not have the attribute. >>>> >>>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . >>>> --absolute-names -n trusted.glusterfs.dht -e hex >>>> /mnt/raid6-storage/storage" >>>> ... >>>> gluster05 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>>> >>>> gluster03 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>>> >>>> gluster04 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>>> >>>> gluster06 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>>> >>>> gluster02 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>>> >>>> gluster07 | FAILED | rc=1 >> >>>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>>> attributenon-zero return code >>>> >>>> gluster01 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>>> >>>> Here are all of the attr's from the brick: >>>> >>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>>> /mnt/raid6-storage/storage/ >>>> # file: /mnt/raid6-storage/storage/ >>>> >>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.gfid=0x00000000000000000000000000000001 >>>> >>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>>> trusted.glusterfs.quota.dirty=0x3000 >>>> >>>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>> >>>> >>>> And here is the volume information: >>>> >>>> [root at gluster07 ~]# gluster volume info storage >>>> >>>> Volume Name: storage >>>> Type: Distribute >>>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>>> Status: Started >>>> Snapshot Count: 0 >>>> Number of Bricks: 7 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>>> Options Reconfigured: >>>> changelog.changelog: on >>>> features.quota-deem-statfs: on >>>> features.read-only: off >>>> features.inode-quota: on >>>> features.quota: on >>>> performance.readdir-ahead: on >>>> nfs.disable: on >>>> geo-replication.indexing: on >>>> geo-replication.ignore-pid-check: on >>>> transport.address-family: inet >>>> >>>> Thanks, >>>> -Matthew >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ravishankar at redhat.com Mon Jul 29 04:35:27 2019 From: ravishankar at redhat.com (Ravishankar N) Date: Mon, 29 Jul 2019 10:05:27 +0530 Subject: [Gluster-users] GlusterFS Changing Hash of Large Files? In-Reply-To: References: Message-ID: <0c9456ee-7d8c-499e-1eb6-f7abfabe79a0@redhat.com> An even replica count is prone to split-brains with the default quorum settings, so replica 3 is recommended. But a replica 4 setup should not cause any checksum change if one node is unreachable, as long as the reads/writes are done via the gluster mount (and not directly the bricks). I wasn't able to re-create this when I copied a huge file on to a 1x4 volume (glusterfs 6.3) with one of the bricks down. Is this something that you can reproduce? Do you see anything suspicious in the mount or brick logs? Regards, Ravi On 28/07/19 10:47 AM, Strahil wrote: > > I never thought that replica 4? is allowed optiion. I always thought > that 3 copies is the maximum. > > Best Regards, > Strahil Nikolov > > On Jul 27, 2019 16:30, Matthew Evans wrote: > > Hi Ravishankar - I figured out the issue. The 4th node was showing > "online" under 'gluster peer status' as well as 'gluster volume > status' - but 'gluster volume status' wasn't showing a TCP port > for that 4th node. When I opened 49152 in firewalld and then > re-copied the ISO, the hash didn't change. > > So, now I guess the question would be, why would having one > malfunctioning node override 3 functioning nodes and cause a file > to be altered? I wasn't performing the initial copy onto the > malfunctioning node. > > matt at docker1:~$ sudo glusterfs --version > glusterfs 6.3 > > matt at docker1:~$ sudo gluster volume info > > Volume Name: swarm-vols > Type: Replicate > Volume ID: 0b51e6b3-786e-454e-8a16-89b47e94828a > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 4 = 4 > Transport-type: tcp > Bricks: > Brick1: docker1:/gluster/data > Brick2: docker2:/gluster/data > Brick3: docker3:/gluster/data > Brick4: docker4:/gluster/data > Options Reconfigured: > performance.client-io-threads: off > nfs.disable: on > transport.address-family: inet > auth.allow: 10.5.22.* > > ------------------------------------------------------------------------ > *From:* Ravishankar N > *Sent:* Saturday, July 27, 2019 2:04 AM > *To:* Matthew Evans ; gluster-users at gluster.org > > *Subject:* Re: [Gluster-users] GlusterFS Changing Hash of Large > Files? > > > On 26/07/19 6:50 PM, Matthew Evans wrote: > > I've got a new glusterfs 4 node replica cluster running under > CentOS 7.? All hosts are backed by SSD drives and are > connected to a 1Gbps Ethernet network. 3 nodes are running on > CentOS 7 under ESXi on the same physical host, 1 is running on > CentOS 7 under Hyper-V. I use this for my docker swarm > persistent storage and all seems to work well. > > Yesterday however, I copied a 4GB .ISO file to my volume for a > friend to download. I noticed the SHA256 hash of the ISO was > altered. I downloaded a fresh copy to my desktop, verified the > hash, scp'd it to the local glusterfs host storage and again, > re-verified the hash. The moment I copied it to my glusterfs > volume, the file hash changed. When my friend downloaded the > ISO, his hash matched changed hash. > > Can you provide the below details? > - glusterfs version > > -`gluster volume info` > > > > I am new to glusterfs, having deployed this as my first > cluster ever about a week ago. Can someone help me work > through why this file > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sunkumar at redhat.com Mon Jul 29 05:56:59 2019 From: sunkumar at redhat.com (Sunny Kumar) Date: Mon, 29 Jul 2019 11:26:59 +0530 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: <722b47be-cb8a-c036-9bd8-4913b6fd7ec6@uvic.ca> Message-ID: HI Matthew, Can you share geo-rep logs and one more log file (changes-.log) it will help to pinpoint actual reason behind failure. /sunny On Mon, Jul 29, 2019 at 9:13 AM Nithya Balachandran wrote: > > > > On Sat, 27 Jul 2019 at 02:31, Matthew Benstead wrote: >> >> Ok thank-you for explaining everything - that makes sense. >> >> Currently the brick file systems are pretty evenly distributed so I probably won't run the fix-layout right now. >> >> Would this state have any impact on geo-replication? I'm trying to geo-replicate this volume, but am getting a weird error: "Changelog register failed error=[Errno 21] Is a directory" > > > It should not. Sunny, can you comment on this? > > Regards, > Nithya >> >> >> I assume this is related to something else, but I wasn't sure. >> >> Thanks, >> -Matthew >> >> -- >> Matthew Benstead >> System Administrator >> Pacific Climate Impacts Consortium >> University of Victoria, UH1 >> PO Box 1800, STN CSC >> Victoria, BC, V8W 2Y2 >> Phone: +1-250-721-8432 >> Email: matthewb at uvic.ca >> >> On 7/26/19 12:02 AM, Nithya Balachandran wrote: >> >> >> >> On Fri, 26 Jul 2019 at 01:56, Matthew Benstead wrote: >>> >>> Hi Nithya, >>> >>> Hmm... I don't remember if I did, but based on what I'm seeing it sounds like I probably didn't run rebalance or fix-layout. >>> >>> It looks like folders that haven't had any new files created have a dht of 0, while other folders have non-zero values. >>> >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ | grep dht >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/home | grep dht >>> trusted.glusterfs.dht=0x00000000000000000000000000000000 >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/home/matthewb | grep dht >>> trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 >>> >>> If I just run the fix-layout command will it re-create all of the dht values or just the missing ones? >> >> >> A fix-layout will recalculate the layouts entirely so files all the values will change. No files will be moved. >> A rebalance will recalculate the layouts like the fix-layout but will also move files to their new locations based on the new layout ranges. This could take a lot of time depending on the number of files/directories on the volume. If you do this, I would recommend that you turn off lookup-optimize until the rebalance is over. >> >>> >>> Since the brick is already fairly size balanced could I get away with running fix-layout but not rebalance? Or would the new dht layout mean slower accesses since the files may be expected on different bricks? >> >> >> The first access for a file will be slower. The next one will be faster as the location will be cached in the client's in-memory structures. >> You may not need to run either a fix-layout or a rebalance if new file creations will be in directories created after the add-brick. Gluster will automatically include all 7 bricks for those directories. >> >> Regards, >> Nithya >> >>> >>> Thanks, >>> -Matthew >>> >>> -- >>> Matthew Benstead >>> System Administrator >>> Pacific Climate Impacts Consortium >>> University of Victoria, UH1 >>> PO Box 1800, STN CSC >>> Victoria, BC, V8W 2Y2 >>> Phone: +1-250-721-8432 >>> Email: matthewb at uvic.ca >>> >>> On 7/24/19 9:30 PM, Nithya Balachandran wrote: >>> >>> >>> >>> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead wrote: >>>> >>>> So looking more closely at the trusted.glusterfs.dht attributes from the bricks it looks like they cover the entire range... and there is no range left for gluster07. >>>> >>>> The first 6 bricks range from 0x00000000 to 0xffffffff - so... is there a way to re-calculate what the dht values should be? Each of the bricks should have a gap >>>> >>>> Gluster05 00000000 -> 2aaaaaa9 >>>> Gluster06 2aaaaaaa -> 55555553 >>>> Gluster01 55555554 -> 7ffffffd >>>> Gluster02 7ffffffe -> aaaaaaa7 >>>> Gluster03 aaaaaaa8 -> d5555551 >>>> Gluster04 d5555552 -> ffffffff >>>> Gluster07 None >>>> >>>> If we split the range into 7 servers that would be a gap of about 0x24924924 for each server. >>>> >>>> Now in terms of the gluster07 brick, about 2 years ago the RAID array the brick was stored on became corrupted. I ran the remove-brick force command, then provisioned a new server, ran the add-brick command and then restored the missing files from backup by copying them back to the main gluster mount (not the brick). >>>> >>> >>> Did you run a rebalance after performing the add-brick? Without a rebalance/fix-layout , the layout for existing directories on the volume will not be updated to use the new brick as well. >>> >>> That the layout does not include the new brick in the root dir is in itself is not a problem. Do you create a lot of files directly in the root of the volume? If yes, you might want to run a rebalance. Otherwise, if you mostly create files in newly added directories, you can probably ignore this. You can check the layout for directories on the volume and see if they incorporate the brick7. >>> >>> I would expect a lookup on the root to have set an xattr on the brick with an empty layout range . The fact that the xattr does not exist at all on the brick is what I am looking into. >>> >>> >>>> It looks like prior to that event this was the layout - which would make sense given the equal size of the 7 bricks: >>>> >>>> gluster02.pcic.uvic.ca | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f >>>> >>>> gluster05.pcic.uvic.ca | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f >>>> >>>> gluster04.pcic.uvic.ca | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf >>>> >>>> gluster03.pcic.uvic.ca | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f >>>> >>>> gluster01.pcic.uvic.ca | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f >>>> >>>> gluster07.pcic.uvic.ca | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x000000010000000000000000245fffdf >>>> >>>> gluster06.pcic.uvic.ca | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff >>>> >>>> Which yields the following: >>>> >>>> 00000000 -> 245fffdf Gluster07 >>>> 245fffe0 -> 48bfff1f Gluster01 >>>> 48bfff20 -> 6d1ffe5f Gluster02 >>>> 6d1ffe60 -> 917ffd9f Gluster03 >>>> 917ffda0 -> b5dffcdf Gluster04 >>>> b5dffce0 -> da3ffc1f Gluster05 >>>> da3ffc20 -> ffffffff Gluster06 >>>> >>>> Is there some way to get back to this? >>>> >>>> Thanks, >>>> -Matthew >>>> >>>> -- >>>> Matthew Benstead >>>> System Administrator >>>> Pacific Climate Impacts Consortium >>>> University of Victoria, UH1 >>>> PO Box 1800, STN CSC >>>> Victoria, BC, V8W 2Y2 >>>> Phone: +1-250-721-8432 >>>> Email: matthewb at uvic.ca >>>> >>>> On 7/18/19 7:20 AM, Matthew Benstead wrote: >>>> >>>> Hi Nithya, >>>> >>>> No - it was added about a year and a half ago. I have tried re-mounting the volume on the server, but it didn't add the attr: >>>> >>>> [root at gluster07 ~]# umount /storage/ >>>> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >>>> 10.0.231.56:/storage /storage glusterfs defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >>>> [root at gluster07 ~]# mount /storage/ >>>> [root at gluster07 ~]# df -h /storage/ >>>> Filesystem Size Used Avail Use% Mounted on >>>> 10.0.231.56:/storage 255T 194T 62T 77% /storage >>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ >>>> # file: /mnt/raid6-storage/storage/ >>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.gfid=0x00000000000000000000000000000001 >>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >>>> trusted.glusterfs.quota.dirty=0x3000 >>>> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>> >>>> Thanks, >>>> -Matthew >>>> >>>> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >>>> >>>> Hi Matthew, >>>> >>>> Was this node/brick added to the volume recently? If yes, try mounting the volume on a fresh mount point - that should create the xattr on this as well. >>>> >>>> Regards, >>>> Nithya >>>> >>>> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead wrote: >>>>> >>>>> Hello, >>>>> >>>>> I've just noticed one brick in my 7 node distribute volume is missing >>>>> the trusted.glusterfs.dht xattr...? How can I fix this? >>>>> >>>>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>>>> >>>>> All of the other nodes are fine, but gluster07 from the list below does >>>>> not have the attribute. >>>>> >>>>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . >>>>> --absolute-names -n trusted.glusterfs.dht -e hex >>>>> /mnt/raid6-storage/storage" >>>>> ... >>>>> gluster05 | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>>>> >>>>> gluster03 | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>>>> >>>>> gluster04 | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>>>> >>>>> gluster06 | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>>>> >>>>> gluster02 | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>>>> >>>>> gluster07 | FAILED | rc=1 >> >>>>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>>>> attributenon-zero return code >>>>> >>>>> gluster01 | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>>>> >>>>> Here are all of the attr's from the brick: >>>>> >>>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>>>> /mnt/raid6-storage/storage/ >>>>> # file: /mnt/raid6-storage/storage/ >>>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>>> trusted.gfid=0x00000000000000000000000000000001 >>>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>>>> trusted.glusterfs.quota.dirty=0x3000 >>>>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>>> >>>>> >>>>> And here is the volume information: >>>>> >>>>> [root at gluster07 ~]# gluster volume info storage >>>>> >>>>> Volume Name: storage >>>>> Type: Distribute >>>>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>>>> Status: Started >>>>> Snapshot Count: 0 >>>>> Number of Bricks: 7 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>>>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>>>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>>>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>>>> Options Reconfigured: >>>>> changelog.changelog: on >>>>> features.quota-deem-statfs: on >>>>> features.read-only: off >>>>> features.inode-quota: on >>>>> features.quota: on >>>>> performance.readdir-ahead: on >>>>> nfs.disable: on >>>>> geo-replication.indexing: on >>>>> geo-replication.ignore-pid-check: on >>>>> transport.address-family: inet >>>>> >>>>> Thanks, >>>>> -Matthew >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>>> >>>> >>> >> From matthewb at uvic.ca Mon Jul 29 16:46:25 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Mon, 29 Jul 2019 09:46:25 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: References: <722b47be-cb8a-c036-9bd8-4913b6fd7ec6@uvic.ca> Message-ID: <803eea7e-86d0-35cd-e18a-500803bf4fc1@uvic.ca> Hi Sunny, Yes, I have attached the gsyncd.log file. I couldn't find any changes-.log files... Trying to start replication goes faulty right away: [root at gluster01 ~]# rpm -q glusterfs glusterfs-5.6-1.el7.x86_64 [root at gluster01 ~]# uname -r 3.10.0-957.21.3.el7.x86_64 [root at gluster01 ~]# cat /etc/centos-release CentOS Linux release 7.6.1810 (Core) [root at gluster01 ~]# gluster volume geo-replication storage root at 10.0.231.81::pcic-backup start Starting geo-replication session between storage & 10.0.231.81::pcic-backup has been successful [root at gluster01 ~]# gluster volume geo-replication storage root at 10.0.231.81::pcic-backup status ? MASTER NODE??? MASTER VOL??? MASTER BRICK????????????????? SLAVE USER??? SLAVE?????????????????????? SLAVE NODE??? STATUS??? CRAWL STATUS??? LAST_SYNCED????????? ------------------------------------------------------------------------------------------------------------------------------------------------------- 10.0.231.50??? storage?????? /mnt/raid6-storage/storage??? root????????? 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? N/A????????????????? 10.0.231.52??? storage?????? /mnt/raid6-storage/storage??? root????????? 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? N/A????????????????? 10.0.231.54??? storage?????? /mnt/raid6-storage/storage??? root????????? 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? N/A????????????????? 10.0.231.51??? storage?????? /mnt/raid6-storage/storage??? root????????? 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? N/A????????????????? 10.0.231.53??? storage?????? /mnt/raid6-storage/storage??? root????????? 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? N/A????????????????? 10.0.231.55??? storage?????? /mnt/raid6-storage/storage??? root????????? 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? N/A????????????????? 10.0.231.56??? storage?????? /mnt/raid6-storage/storage??? root????????? 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? N/A????????????????? [root at gluster01 ~]# gluster volume geo-replication storage root at 10.0.231.81::pcic-backup stop Stopping geo-replication session between storage & 10.0.231.81::pcic-backup has been successful This is the primary cluster: [root at gluster01 ~]# gluster volume info storage ? Volume Name: storage Type: Distribute Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 Status: Started Snapshot Count: 0 Number of Bricks: 7 Transport-type: tcp Bricks: Brick1: 10.0.231.50:/mnt/raid6-storage/storage Brick2: 10.0.231.51:/mnt/raid6-storage/storage Brick3: 10.0.231.52:/mnt/raid6-storage/storage Brick4: 10.0.231.53:/mnt/raid6-storage/storage Brick5: 10.0.231.54:/mnt/raid6-storage/storage Brick6: 10.0.231.55:/mnt/raid6-storage/storage Brick7: 10.0.231.56:/mnt/raid6-storage/storage Options Reconfigured: features.read-only: off features.inode-quota: on features.quota: on performance.readdir-ahead: on nfs.disable: on geo-replication.indexing: on geo-replication.ignore-pid-check: on transport.address-family: inet features.quota-deem-statfs: on changelog.changelog: on diagnostics.client-log-level: INFO And this is the cluster I'm trying to replicate to: [root at pcic-backup01 ~]# gluster volume info pcic-backup ? Volume Name: pcic-backup Type: Distribute Volume ID: 2890bcde-a023-4feb-a0e5-e8ef8f337d4c Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.0.231.81:/pcic-backup01-zpool/brick Brick2: 10.0.231.82:/pcic-backup02-zpool/brick Options Reconfigured: nfs.disable: on transport.address-family: inet Thanks, ?-Matthew On 7/28/19 10:56 PM, Sunny Kumar wrote: > HI Matthew, > > Can you share geo-rep logs and one more log file > (changes-.log) it will help to pinpoint actual reason > behind failure. > > /sunny > > On Mon, Jul 29, 2019 at 9:13 AM Nithya Balachandran wrote: >> >> >> On Sat, 27 Jul 2019 at 02:31, Matthew Benstead wrote: >>> Ok thank-you for explaining everything - that makes sense. >>> >>> Currently the brick file systems are pretty evenly distributed so I probably won't run the fix-layout right now. >>> >>> Would this state have any impact on geo-replication? I'm trying to geo-replicate this volume, but am getting a weird error: "Changelog register failed error=[Errno 21] Is a directory" >> >> It should not. Sunny, can you comment on this? >> >> Regards, >> Nithya >>> >>> I assume this is related to something else, but I wasn't sure. >>> >>> Thanks, >>> -Matthew >>> >>> -- >>> Matthew Benstead >>> System Administrator >>> Pacific Climate Impacts Consortium >>> University of Victoria, UH1 >>> PO Box 1800, STN CSC >>> Victoria, BC, V8W 2Y2 >>> Phone: +1-250-721-8432 >>> Email: matthewb at uvic.ca >>> >>> On 7/26/19 12:02 AM, Nithya Balachandran wrote: >>> >>> >>> >>> On Fri, 26 Jul 2019 at 01:56, Matthew Benstead wrote: >>>> Hi Nithya, >>>> >>>> Hmm... I don't remember if I did, but based on what I'm seeing it sounds like I probably didn't run rebalance or fix-layout. >>>> >>>> It looks like folders that haven't had any new files created have a dht of 0, while other folders have non-zero values. >>>> >>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ | grep dht >>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/home | grep dht >>>> trusted.glusterfs.dht=0x00000000000000000000000000000000 >>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/home/matthewb | grep dht >>>> trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 >>>> >>>> If I just run the fix-layout command will it re-create all of the dht values or just the missing ones? >>> >>> A fix-layout will recalculate the layouts entirely so files all the values will change. No files will be moved. >>> A rebalance will recalculate the layouts like the fix-layout but will also move files to their new locations based on the new layout ranges. This could take a lot of time depending on the number of files/directories on the volume. If you do this, I would recommend that you turn off lookup-optimize until the rebalance is over. >>> >>>> Since the brick is already fairly size balanced could I get away with running fix-layout but not rebalance? Or would the new dht layout mean slower accesses since the files may be expected on different bricks? >>> >>> The first access for a file will be slower. The next one will be faster as the location will be cached in the client's in-memory structures. >>> You may not need to run either a fix-layout or a rebalance if new file creations will be in directories created after the add-brick. Gluster will automatically include all 7 bricks for those directories. >>> >>> Regards, >>> Nithya >>> >>>> Thanks, >>>> -Matthew >>>> >>>> -- >>>> Matthew Benstead >>>> System Administrator >>>> Pacific Climate Impacts Consortium >>>> University of Victoria, UH1 >>>> PO Box 1800, STN CSC >>>> Victoria, BC, V8W 2Y2 >>>> Phone: +1-250-721-8432 >>>> Email: matthewb at uvic.ca >>>> >>>> On 7/24/19 9:30 PM, Nithya Balachandran wrote: >>>> >>>> >>>> >>>> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead wrote: >>>>> So looking more closely at the trusted.glusterfs.dht attributes from the bricks it looks like they cover the entire range... and there is no range left for gluster07. >>>>> >>>>> The first 6 bricks range from 0x00000000 to 0xffffffff - so... is there a way to re-calculate what the dht values should be? Each of the bricks should have a gap >>>>> >>>>> Gluster05 00000000 -> 2aaaaaa9 >>>>> Gluster06 2aaaaaaa -> 55555553 >>>>> Gluster01 55555554 -> 7ffffffd >>>>> Gluster02 7ffffffe -> aaaaaaa7 >>>>> Gluster03 aaaaaaa8 -> d5555551 >>>>> Gluster04 d5555552 -> ffffffff >>>>> Gluster07 None >>>>> >>>>> If we split the range into 7 servers that would be a gap of about 0x24924924 for each server. >>>>> >>>>> Now in terms of the gluster07 brick, about 2 years ago the RAID array the brick was stored on became corrupted. I ran the remove-brick force command, then provisioned a new server, ran the add-brick command and then restored the missing files from backup by copying them back to the main gluster mount (not the brick). >>>>> >>>> Did you run a rebalance after performing the add-brick? Without a rebalance/fix-layout , the layout for existing directories on the volume will not be updated to use the new brick as well. >>>> >>>> That the layout does not include the new brick in the root dir is in itself is not a problem. Do you create a lot of files directly in the root of the volume? If yes, you might want to run a rebalance. Otherwise, if you mostly create files in newly added directories, you can probably ignore this. You can check the layout for directories on the volume and see if they incorporate the brick7. >>>> >>>> I would expect a lookup on the root to have set an xattr on the brick with an empty layout range . The fact that the xattr does not exist at all on the brick is what I am looking into. >>>> >>>> >>>>> It looks like prior to that event this was the layout - which would make sense given the equal size of the 7 bricks: >>>>> >>>>> gluster02.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f >>>>> >>>>> gluster05.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f >>>>> >>>>> gluster04.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf >>>>> >>>>> gluster03.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f >>>>> >>>>> gluster01.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f >>>>> >>>>> gluster07.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x000000010000000000000000245fffdf >>>>> >>>>> gluster06.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>> # file: /mnt/raid6-storage/storage >>>>> trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff >>>>> >>>>> Which yields the following: >>>>> >>>>> 00000000 -> 245fffdf Gluster07 >>>>> 245fffe0 -> 48bfff1f Gluster01 >>>>> 48bfff20 -> 6d1ffe5f Gluster02 >>>>> 6d1ffe60 -> 917ffd9f Gluster03 >>>>> 917ffda0 -> b5dffcdf Gluster04 >>>>> b5dffce0 -> da3ffc1f Gluster05 >>>>> da3ffc20 -> ffffffff Gluster06 >>>>> >>>>> Is there some way to get back to this? >>>>> >>>>> Thanks, >>>>> -Matthew >>>>> >>>>> -- >>>>> Matthew Benstead >>>>> System Administrator >>>>> Pacific Climate Impacts Consortium >>>>> University of Victoria, UH1 >>>>> PO Box 1800, STN CSC >>>>> Victoria, BC, V8W 2Y2 >>>>> Phone: +1-250-721-8432 >>>>> Email: matthewb at uvic.ca >>>>> >>>>> On 7/18/19 7:20 AM, Matthew Benstead wrote: >>>>> >>>>> Hi Nithya, >>>>> >>>>> No - it was added about a year and a half ago. I have tried re-mounting the volume on the server, but it didn't add the attr: >>>>> >>>>> [root at gluster07 ~]# umount /storage/ >>>>> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >>>>> 10.0.231.56:/storage /storage glusterfs defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >>>>> [root at gluster07 ~]# mount /storage/ >>>>> [root at gluster07 ~]# df -h /storage/ >>>>> Filesystem Size Used Avail Use% Mounted on >>>>> 10.0.231.56:/storage 255T 194T 62T 77% /storage >>>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ >>>>> # file: /mnt/raid6-storage/storage/ >>>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>>> trusted.gfid=0x00000000000000000000000000000001 >>>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >>>>> trusted.glusterfs.quota.dirty=0x3000 >>>>> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >>>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>>> >>>>> Thanks, >>>>> -Matthew >>>>> >>>>> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >>>>> >>>>> Hi Matthew, >>>>> >>>>> Was this node/brick added to the volume recently? If yes, try mounting the volume on a fresh mount point - that should create the xattr on this as well. >>>>> >>>>> Regards, >>>>> Nithya >>>>> >>>>> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead wrote: >>>>>> Hello, >>>>>> >>>>>> I've just noticed one brick in my 7 node distribute volume is missing >>>>>> the trusted.glusterfs.dht xattr...? How can I fix this? >>>>>> >>>>>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>>>>> >>>>>> All of the other nodes are fine, but gluster07 from the list below does >>>>>> not have the attribute. >>>>>> >>>>>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . >>>>>> --absolute-names -n trusted.glusterfs.dht -e hex >>>>>> /mnt/raid6-storage/storage" >>>>>> ... >>>>>> gluster05 | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>>>>> >>>>>> gluster03 | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>>>>> >>>>>> gluster04 | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>>>>> >>>>>> gluster06 | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>>>>> >>>>>> gluster02 | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>>>>> >>>>>> gluster07 | FAILED | rc=1 >> >>>>>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>>>>> attributenon-zero return code >>>>>> >>>>>> gluster01 | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>>>>> >>>>>> Here are all of the attr's from the brick: >>>>>> >>>>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>>>>> /mnt/raid6-storage/storage/ >>>>>> # file: /mnt/raid6-storage/storage/ >>>>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>>>> trusted.gfid=0x00000000000000000000000000000001 >>>>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>>>>> trusted.glusterfs.quota.dirty=0x3000 >>>>>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>>>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>>>> >>>>>> >>>>>> And here is the volume information: >>>>>> >>>>>> [root at gluster07 ~]# gluster volume info storage >>>>>> >>>>>> Volume Name: storage >>>>>> Type: Distribute >>>>>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>>>>> Status: Started >>>>>> Snapshot Count: 0 >>>>>> Number of Bricks: 7 >>>>>> Transport-type: tcp >>>>>> Bricks: >>>>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>>>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>>>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>>>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>>>>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>>>>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>>>>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>>>>> Options Reconfigured: >>>>>> changelog.changelog: on >>>>>> features.quota-deem-statfs: on >>>>>> features.read-only: off >>>>>> features.inode-quota: on >>>>>> features.quota: on >>>>>> performance.readdir-ahead: on >>>>>> nfs.disable: on >>>>>> geo-replication.indexing: on >>>>>> geo-replication.ignore-pid-check: on >>>>>> transport.address-family: inet >>>>>> >>>>>> Thanks, >>>>>> -Matthew >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> -------------- next part -------------- A non-text attachment was scrubbed... Name: gluster01-gsyncd.log Type: text/x-log Size: 8980 bytes Desc: not available URL: From dijuremo at gmail.com Tue Jul 30 00:13:40 2019 From: dijuremo at gmail.com (Diego Remolina) Date: Mon, 29 Jul 2019 20:13:40 -0400 Subject: [Gluster-users] Gluster eating up a lot of ram In-Reply-To: References: Message-ID: Unfortunately statedump crashes on both machines, even freshly rebooted. [root at ysmha01 ~]# gluster --print-statedumpdir /var/run/gluster [root at ysmha01 ~]# gluster v statedump export Segmentation fault (core dumped) [root at ysmha02 ~]# uptime 20:12:20 up 6 min, 1 user, load average: 0.72, 0.52, 0.24 [root at ysmha02 ~]# gluster --print-statedumpdir /var/run/gluster [root at ysmha02 ~]# gluster v statedump export Segmentation fault (core dumped) I rebooted today after 40 days. Gluster was eating up shy of 40GB of RAM out of 64. What would you recommend to be the next step? Diego On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah wrote: > Could you also provide the statedump of the gluster process consuming 44G > ram [1]. Please make sure the statedump is taken when the memory > consumption is very high, like 10s of GBs, otherwise we may not be able to > identify the issue. Also i see that the cache size is 10G is that something > you arrived at, after doing some tests? Its relatively higher than normal. > > [1] > https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump > > On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina wrote: > >> Hi, >> >> I will not be able to test gluster-6rc because this is a production >> environment and it takes several days for memory to grow a lot. >> >> The Samba server is hosting all types of files, small and large from >> small roaming profile type files to bigger files like adobe suite, autodesk >> Revit (file sizes in the hundreds of megabytes). >> >> As I stated before, this same issue was present back with 3.8.x which I >> was running before. >> >> The information you requested: >> >> [root at ysmha02 ~]# gluster v info export >> >> Volume Name: export >> Type: Replicate >> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: 10.0.1.7:/bricks/hdds/brick >> Brick2: 10.0.1.6:/bricks/hdds/brick >> Options Reconfigured: >> performance.stat-prefetch: on >> performance.cache-min-file-size: 0 >> network.inode-lru-limit: 65536 >> performance.cache-invalidation: on >> features.cache-invalidation: on >> performance.md-cache-timeout: 600 >> features.cache-invalidation-timeout: 600 >> performance.cache-samba-metadata: on >> transport.address-family: inet >> server.allow-insecure: on >> performance.cache-size: 10GB >> cluster.server-quorum-type: server >> nfs.disable: on >> performance.io-thread-count: 64 >> performance.io-cache: on >> cluster.lookup-optimize: on >> cluster.readdir-optimize: on >> server.event-threads: 5 >> client.event-threads: 5 >> performance.cache-max-file-size: 256MB >> diagnostics.client-log-level: INFO >> diagnostics.brick-log-level: INFO >> cluster.server-quorum-ratio: 51% >> >> >> >> >> >> >> >> Virus-free. >> www.avast.com >> >> <#m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >> >> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah < >> pgurusid at redhat.com> wrote: >> >>> This high memory consumption is not normal. Looks like it's a memory >>> leak. Is it possible to try it on test setup with gluster-6rc? What is the >>> kind of workload that goes into fuse mount? Large files or small files? We >>> need the following information to debug further: >>> - Gluster volume info output >>> - Statedump of the Gluster fuse mount process consuming 44G ram. >>> >>> Regards, >>> Poornima >>> >>> >>> On Sat, Mar 2, 2019, 3:40 AM Diego Remolina wrote: >>> >>>> I am using glusterfs with two servers as a file server sharing files >>>> via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in >>>> current Centos version of samba. So I am mounting via fuse and exporting >>>> the volume to samba from the mount point. >>>> >>>> Upon initial boot, the server where samba is exporting files climbs up >>>> to ~10GB RAM within a couple hours of use. From then on, it is a constant >>>> slow memory increase. In the past with gluster 3.8.x we had to reboot the >>>> servers at around 30 days . With gluster 4.1.6 we are getting up to 48 >>>> days, but RAM use is at 48GB out of 64GB. Is this normal? >>>> >>>> The particular versions are below, >>>> >>>> [root at ysmha01 home]# uptime >>>> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 >>>> [root at ysmha01 home]# rpm -qa | grep gluster >>>> centos-release-gluster41-1.0-3.el7.centos.noarch >>>> glusterfs-server-4.1.6-1.el7.x86_64 >>>> glusterfs-api-4.1.6-1.el7.x86_64 >>>> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >>>> glusterfs-4.1.6-1.el7.x86_64 >>>> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >>>> glusterfs-fuse-4.1.6-1.el7.x86_64 >>>> glusterfs-libs-4.1.6-1.el7.x86_64 >>>> glusterfs-rdma-4.1.6-1.el7.x86_64 >>>> glusterfs-cli-4.1.6-1.el7.x86_64 >>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>> [root at ysmha01 home]# rpm -qa | grep samba >>>> samba-common-tools-4.8.3-4.el7.x86_64 >>>> samba-client-libs-4.8.3-4.el7.x86_64 >>>> samba-libs-4.8.3-4.el7.x86_64 >>>> samba-4.8.3-4.el7.x86_64 >>>> samba-common-libs-4.8.3-4.el7.x86_64 >>>> samba-common-4.8.3-4.el7.noarch >>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>> [root at ysmha01 home]# cat /etc/redhat-release >>>> CentOS Linux release 7.6.1810 (Core) >>>> >>>> RAM view using top >>>> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie >>>> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 >>>> si, 0.0 st >>>> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >>>> buff/cache >>>> KiB Swap: 0 total, 0 free, 0 used. 3134316 avail >>>> Mem >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>> COMMAND >>>> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 >>>> glusterfsd >>>> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 >>>> glusterfs >>>> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 >>>> glusterfs >>>> >>>> [root at ysmha01 ~]# gluster v status export >>>> Status of volume: export >>>> Gluster process TCP Port RDMA Port >>>> Online Pid >>>> >>>> ------------------------------------------------------------------------------ >>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y >>>> 13986 >>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y >>>> 9953 >>>> Self-heal Daemon on localhost N/A N/A Y >>>> 14485 >>>> Self-heal Daemon on 10.0.1.7 N/A N/A Y >>>> 21934 >>>> Self-heal Daemon on 10.0.1.5 N/A N/A Y >>>> 4598 >>>> >>>> Task Status of Volume export >>>> >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> >>>> >>>> >>>> >>>> Virus-free. >>>> www.avast.com >>>> >>>> <#m_-1483290904248086332_m_-4429654867678350131_m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Tue Jul 30 03:11:59 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Tue, 30 Jul 2019 08:41:59 +0530 Subject: [Gluster-users] Gluster eating up a lot of ram In-Reply-To: References: Message-ID: On Tue, Jul 30, 2019 at 5:44 AM Diego Remolina wrote: > Unfortunately statedump crashes on both machines, even freshly rebooted. > Does sending SIGUSR1 to individual processes also crash? # kill -SIGUSR1 > [root at ysmha01 ~]# gluster --print-statedumpdir > /var/run/gluster > [root at ysmha01 ~]# gluster v statedump export > Segmentation fault (core dumped) > > [root at ysmha02 ~]# uptime > 20:12:20 up 6 min, 1 user, load average: 0.72, 0.52, 0.24 > [root at ysmha02 ~]# gluster --print-statedumpdir > /var/run/gluster > [root at ysmha02 ~]# gluster v statedump export > Segmentation fault (core dumped) > > I rebooted today after 40 days. Gluster was eating up shy of 40GB of RAM > out of 64. > > What would you recommend to be the next step? > > Diego > > On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah > wrote: > >> Could you also provide the statedump of the gluster process consuming 44G >> ram [1]. Please make sure the statedump is taken when the memory >> consumption is very high, like 10s of GBs, otherwise we may not be able to >> identify the issue. Also i see that the cache size is 10G is that something >> you arrived at, after doing some tests? Its relatively higher than normal. >> >> [1] >> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump >> >> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina >> wrote: >> >>> Hi, >>> >>> I will not be able to test gluster-6rc because this is a production >>> environment and it takes several days for memory to grow a lot. >>> >>> The Samba server is hosting all types of files, small and large from >>> small roaming profile type files to bigger files like adobe suite, autodesk >>> Revit (file sizes in the hundreds of megabytes). >>> >>> As I stated before, this same issue was present back with 3.8.x which I >>> was running before. >>> >>> The information you requested: >>> >>> [root at ysmha02 ~]# gluster v info export >>> >>> Volume Name: export >>> Type: Replicate >>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 1 x 2 = 2 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 10.0.1.7:/bricks/hdds/brick >>> Brick2: 10.0.1.6:/bricks/hdds/brick >>> Options Reconfigured: >>> performance.stat-prefetch: on >>> performance.cache-min-file-size: 0 >>> network.inode-lru-limit: 65536 >>> performance.cache-invalidation: on >>> features.cache-invalidation: on >>> performance.md-cache-timeout: 600 >>> features.cache-invalidation-timeout: 600 >>> performance.cache-samba-metadata: on >>> transport.address-family: inet >>> server.allow-insecure: on >>> performance.cache-size: 10GB >>> cluster.server-quorum-type: server >>> nfs.disable: on >>> performance.io-thread-count: 64 >>> performance.io-cache: on >>> cluster.lookup-optimize: on >>> cluster.readdir-optimize: on >>> server.event-threads: 5 >>> client.event-threads: 5 >>> performance.cache-max-file-size: 256MB >>> diagnostics.client-log-level: INFO >>> diagnostics.brick-log-level: INFO >>> cluster.server-quorum-ratio: 51% >>> >>> >>> >>> >>> >>> >>> >>> Virus-free. >>> www.avast.com >>> >>> <#m_3511709704803768245_m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>> >>> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah < >>> pgurusid at redhat.com> wrote: >>> >>>> This high memory consumption is not normal. Looks like it's a memory >>>> leak. Is it possible to try it on test setup with gluster-6rc? What is the >>>> kind of workload that goes into fuse mount? Large files or small files? We >>>> need the following information to debug further: >>>> - Gluster volume info output >>>> - Statedump of the Gluster fuse mount process consuming 44G ram. >>>> >>>> Regards, >>>> Poornima >>>> >>>> >>>> On Sat, Mar 2, 2019, 3:40 AM Diego Remolina wrote: >>>> >>>>> I am using glusterfs with two servers as a file server sharing files >>>>> via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in >>>>> current Centos version of samba. So I am mounting via fuse and exporting >>>>> the volume to samba from the mount point. >>>>> >>>>> Upon initial boot, the server where samba is exporting files climbs up >>>>> to ~10GB RAM within a couple hours of use. From then on, it is a constant >>>>> slow memory increase. In the past with gluster 3.8.x we had to reboot the >>>>> servers at around 30 days . With gluster 4.1.6 we are getting up to 48 >>>>> days, but RAM use is at 48GB out of 64GB. Is this normal? >>>>> >>>>> The particular versions are below, >>>>> >>>>> [root at ysmha01 home]# uptime >>>>> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 >>>>> [root at ysmha01 home]# rpm -qa | grep gluster >>>>> centos-release-gluster41-1.0-3.el7.centos.noarch >>>>> glusterfs-server-4.1.6-1.el7.x86_64 >>>>> glusterfs-api-4.1.6-1.el7.x86_64 >>>>> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >>>>> glusterfs-4.1.6-1.el7.x86_64 >>>>> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >>>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >>>>> glusterfs-fuse-4.1.6-1.el7.x86_64 >>>>> glusterfs-libs-4.1.6-1.el7.x86_64 >>>>> glusterfs-rdma-4.1.6-1.el7.x86_64 >>>>> glusterfs-cli-4.1.6-1.el7.x86_64 >>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>> [root at ysmha01 home]# rpm -qa | grep samba >>>>> samba-common-tools-4.8.3-4.el7.x86_64 >>>>> samba-client-libs-4.8.3-4.el7.x86_64 >>>>> samba-libs-4.8.3-4.el7.x86_64 >>>>> samba-4.8.3-4.el7.x86_64 >>>>> samba-common-libs-4.8.3-4.el7.x86_64 >>>>> samba-common-4.8.3-4.el7.noarch >>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>> [root at ysmha01 home]# cat /etc/redhat-release >>>>> CentOS Linux release 7.6.1810 (Core) >>>>> >>>>> RAM view using top >>>>> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie >>>>> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 >>>>> si, 0.0 st >>>>> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >>>>> buff/cache >>>>> KiB Swap: 0 total, 0 free, 0 used. 3134316 avail >>>>> Mem >>>>> >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>> COMMAND >>>>> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 >>>>> glusterfsd >>>>> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 >>>>> glusterfs >>>>> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 >>>>> glusterfs >>>>> >>>>> [root at ysmha01 ~]# gluster v status export >>>>> Status of volume: export >>>>> Gluster process TCP Port RDMA Port >>>>> Online Pid >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y >>>>> 13986 >>>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y >>>>> 9953 >>>>> Self-heal Daemon on localhost N/A N/A Y >>>>> 14485 >>>>> Self-heal Daemon on 10.0.1.7 N/A N/A Y >>>>> 21934 >>>>> Self-heal Daemon on 10.0.1.5 N/A N/A Y >>>>> 4598 >>>>> >>>>> Task Status of Volume export >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> There are no active volume tasks >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Virus-free. >>>>> www.avast.com >>>>> >>>>> <#m_3511709704803768245_m_-1483290904248086332_m_-4429654867678350131_m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>>> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Tue Jul 30 03:27:31 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Tue, 30 Jul 2019 08:57:31 +0530 Subject: [Gluster-users] Gluster eating up a lot of ram In-Reply-To: References: Message-ID: On Tue, 30 Jul 2019 at 05:44, Diego Remolina wrote: > Unfortunately statedump crashes on both machines, even freshly rebooted. > Do you see any statedump files in /var/run/gluster? This looks more like the gluster cli crashed. > > [root at ysmha01 ~]# gluster --print-statedumpdir > /var/run/gluster > [root at ysmha01 ~]# gluster v statedump export > Segmentation fault (core dumped) > > [root at ysmha02 ~]# uptime > 20:12:20 up 6 min, 1 user, load average: 0.72, 0.52, 0.24 > [root at ysmha02 ~]# gluster --print-statedumpdir > /var/run/gluster > [root at ysmha02 ~]# gluster v statedump export > Segmentation fault (core dumped) > > I rebooted today after 40 days. Gluster was eating up shy of 40GB of RAM > out of 64. > > What would you recommend to be the next step? > > Diego > > On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah > wrote: > >> Could you also provide the statedump of the gluster process consuming 44G >> ram [1]. Please make sure the statedump is taken when the memory >> consumption is very high, like 10s of GBs, otherwise we may not be able to >> identify the issue. Also i see that the cache size is 10G is that something >> you arrived at, after doing some tests? Its relatively higher than normal. >> >> [1] >> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump >> >> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina >> wrote: >> >>> Hi, >>> >>> I will not be able to test gluster-6rc because this is a production >>> environment and it takes several days for memory to grow a lot. >>> >>> The Samba server is hosting all types of files, small and large from >>> small roaming profile type files to bigger files like adobe suite, autodesk >>> Revit (file sizes in the hundreds of megabytes). >>> >>> As I stated before, this same issue was present back with 3.8.x which I >>> was running before. >>> >>> The information you requested: >>> >>> [root at ysmha02 ~]# gluster v info export >>> >>> Volume Name: export >>> Type: Replicate >>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 1 x 2 = 2 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 10.0.1.7:/bricks/hdds/brick >>> Brick2: 10.0.1.6:/bricks/hdds/brick >>> Options Reconfigured: >>> performance.stat-prefetch: on >>> performance.cache-min-file-size: 0 >>> network.inode-lru-limit: 65536 >>> performance.cache-invalidation: on >>> features.cache-invalidation: on >>> performance.md-cache-timeout: 600 >>> features.cache-invalidation-timeout: 600 >>> performance.cache-samba-metadata: on >>> transport.address-family: inet >>> server.allow-insecure: on >>> performance.cache-size: 10GB >>> cluster.server-quorum-type: server >>> nfs.disable: on >>> performance.io-thread-count: 64 >>> performance.io-cache: on >>> cluster.lookup-optimize: on >>> cluster.readdir-optimize: on >>> server.event-threads: 5 >>> client.event-threads: 5 >>> performance.cache-max-file-size: 256MB >>> diagnostics.client-log-level: INFO >>> diagnostics.brick-log-level: INFO >>> cluster.server-quorum-ratio: 51% >>> >>> >>> >>> >>> >>> >>> >>> Virus-free. >>> www.avast.com >>> >>> <#m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>> >>> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah < >>> pgurusid at redhat.com> wrote: >>> >>>> This high memory consumption is not normal. Looks like it's a memory >>>> leak. Is it possible to try it on test setup with gluster-6rc? What is the >>>> kind of workload that goes into fuse mount? Large files or small files? We >>>> need the following information to debug further: >>>> - Gluster volume info output >>>> - Statedump of the Gluster fuse mount process consuming 44G ram. >>>> >>>> Regards, >>>> Poornima >>>> >>>> >>>> On Sat, Mar 2, 2019, 3:40 AM Diego Remolina wrote: >>>> >>>>> I am using glusterfs with two servers as a file server sharing files >>>>> via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in >>>>> current Centos version of samba. So I am mounting via fuse and exporting >>>>> the volume to samba from the mount point. >>>>> >>>>> Upon initial boot, the server where samba is exporting files climbs up >>>>> to ~10GB RAM within a couple hours of use. From then on, it is a constant >>>>> slow memory increase. In the past with gluster 3.8.x we had to reboot the >>>>> servers at around 30 days . With gluster 4.1.6 we are getting up to 48 >>>>> days, but RAM use is at 48GB out of 64GB. Is this normal? >>>>> >>>>> The particular versions are below, >>>>> >>>>> [root at ysmha01 home]# uptime >>>>> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 >>>>> [root at ysmha01 home]# rpm -qa | grep gluster >>>>> centos-release-gluster41-1.0-3.el7.centos.noarch >>>>> glusterfs-server-4.1.6-1.el7.x86_64 >>>>> glusterfs-api-4.1.6-1.el7.x86_64 >>>>> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >>>>> glusterfs-4.1.6-1.el7.x86_64 >>>>> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >>>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >>>>> glusterfs-fuse-4.1.6-1.el7.x86_64 >>>>> glusterfs-libs-4.1.6-1.el7.x86_64 >>>>> glusterfs-rdma-4.1.6-1.el7.x86_64 >>>>> glusterfs-cli-4.1.6-1.el7.x86_64 >>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>> [root at ysmha01 home]# rpm -qa | grep samba >>>>> samba-common-tools-4.8.3-4.el7.x86_64 >>>>> samba-client-libs-4.8.3-4.el7.x86_64 >>>>> samba-libs-4.8.3-4.el7.x86_64 >>>>> samba-4.8.3-4.el7.x86_64 >>>>> samba-common-libs-4.8.3-4.el7.x86_64 >>>>> samba-common-4.8.3-4.el7.noarch >>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>> [root at ysmha01 home]# cat /etc/redhat-release >>>>> CentOS Linux release 7.6.1810 (Core) >>>>> >>>>> RAM view using top >>>>> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie >>>>> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 >>>>> si, 0.0 st >>>>> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >>>>> buff/cache >>>>> KiB Swap: 0 total, 0 free, 0 used. 3134316 avail >>>>> Mem >>>>> >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>> COMMAND >>>>> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 >>>>> glusterfsd >>>>> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 >>>>> glusterfs >>>>> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 >>>>> glusterfs >>>>> >>>>> [root at ysmha01 ~]# gluster v status export >>>>> Status of volume: export >>>>> Gluster process TCP Port RDMA Port >>>>> Online Pid >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y >>>>> 13986 >>>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y >>>>> 9953 >>>>> Self-heal Daemon on localhost N/A N/A Y >>>>> 14485 >>>>> Self-heal Daemon on 10.0.1.7 N/A N/A Y >>>>> 21934 >>>>> Self-heal Daemon on 10.0.1.5 N/A N/A Y >>>>> 4598 >>>>> >>>>> Task Status of Volume export >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> There are no active volume tasks >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Virus-free. >>>>> www.avast.com >>>>> >>>>> <#m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>>> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From dijuremo at gmail.com Tue Jul 30 03:34:23 2019 From: dijuremo at gmail.com (Diego Remolina) Date: Mon, 29 Jul 2019 23:34:23 -0400 Subject: [Gluster-users] Gluster eating up a lot of ram In-Reply-To: References: Message-ID: Will this kill the actual process or simply trigger the dump? Which process should I kill? The brick process in the system or the fuse mount? Diego On Mon, Jul 29, 2019, 23:27 Nithya Balachandran wrote: > > > On Tue, 30 Jul 2019 at 05:44, Diego Remolina wrote: > >> Unfortunately statedump crashes on both machines, even freshly rebooted. >> > > Do you see any statedump files in /var/run/gluster? This looks more like > the gluster cli crashed. > >> >> [root at ysmha01 ~]# gluster --print-statedumpdir >> /var/run/gluster >> [root at ysmha01 ~]# gluster v statedump export >> Segmentation fault (core dumped) >> >> [root at ysmha02 ~]# uptime >> 20:12:20 up 6 min, 1 user, load average: 0.72, 0.52, 0.24 >> [root at ysmha02 ~]# gluster --print-statedumpdir >> /var/run/gluster >> [root at ysmha02 ~]# gluster v statedump export >> Segmentation fault (core dumped) >> >> I rebooted today after 40 days. Gluster was eating up shy of 40GB of RAM >> out of 64. >> >> What would you recommend to be the next step? >> >> Diego >> >> On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah >> wrote: >> >>> Could you also provide the statedump of the gluster process consuming >>> 44G ram [1]. Please make sure the statedump is taken when the memory >>> consumption is very high, like 10s of GBs, otherwise we may not be able to >>> identify the issue. Also i see that the cache size is 10G is that something >>> you arrived at, after doing some tests? Its relatively higher than normal. >>> >>> [1] >>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump >>> >>> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina >>> wrote: >>> >>>> Hi, >>>> >>>> I will not be able to test gluster-6rc because this is a production >>>> environment and it takes several days for memory to grow a lot. >>>> >>>> The Samba server is hosting all types of files, small and large from >>>> small roaming profile type files to bigger files like adobe suite, autodesk >>>> Revit (file sizes in the hundreds of megabytes). >>>> >>>> As I stated before, this same issue was present back with 3.8.x which I >>>> was running before. >>>> >>>> The information you requested: >>>> >>>> [root at ysmha02 ~]# gluster v info export >>>> >>>> Volume Name: export >>>> Type: Replicate >>>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >>>> Status: Started >>>> Snapshot Count: 0 >>>> Number of Bricks: 1 x 2 = 2 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: 10.0.1.7:/bricks/hdds/brick >>>> Brick2: 10.0.1.6:/bricks/hdds/brick >>>> Options Reconfigured: >>>> performance.stat-prefetch: on >>>> performance.cache-min-file-size: 0 >>>> network.inode-lru-limit: 65536 >>>> performance.cache-invalidation: on >>>> features.cache-invalidation: on >>>> performance.md-cache-timeout: 600 >>>> features.cache-invalidation-timeout: 600 >>>> performance.cache-samba-metadata: on >>>> transport.address-family: inet >>>> server.allow-insecure: on >>>> performance.cache-size: 10GB >>>> cluster.server-quorum-type: server >>>> nfs.disable: on >>>> performance.io-thread-count: 64 >>>> performance.io-cache: on >>>> cluster.lookup-optimize: on >>>> cluster.readdir-optimize: on >>>> server.event-threads: 5 >>>> client.event-threads: 5 >>>> performance.cache-max-file-size: 256MB >>>> diagnostics.client-log-level: INFO >>>> diagnostics.brick-log-level: INFO >>>> cluster.server-quorum-ratio: 51% >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Virus-free. >>>> www.avast.com >>>> >>>> <#m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>> >>>> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah < >>>> pgurusid at redhat.com> wrote: >>>> >>>>> This high memory consumption is not normal. Looks like it's a memory >>>>> leak. Is it possible to try it on test setup with gluster-6rc? What is the >>>>> kind of workload that goes into fuse mount? Large files or small files? We >>>>> need the following information to debug further: >>>>> - Gluster volume info output >>>>> - Statedump of the Gluster fuse mount process consuming 44G ram. >>>>> >>>>> Regards, >>>>> Poornima >>>>> >>>>> >>>>> On Sat, Mar 2, 2019, 3:40 AM Diego Remolina >>>>> wrote: >>>>> >>>>>> I am using glusterfs with two servers as a file server sharing files >>>>>> via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in >>>>>> current Centos version of samba. So I am mounting via fuse and exporting >>>>>> the volume to samba from the mount point. >>>>>> >>>>>> Upon initial boot, the server where samba is exporting files climbs >>>>>> up to ~10GB RAM within a couple hours of use. From then on, it is a >>>>>> constant slow memory increase. In the past with gluster 3.8.x we had to >>>>>> reboot the servers at around 30 days . With gluster 4.1.6 we are getting up >>>>>> to 48 days, but RAM use is at 48GB out of 64GB. Is this normal? >>>>>> >>>>>> The particular versions are below, >>>>>> >>>>>> [root at ysmha01 home]# uptime >>>>>> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 >>>>>> [root at ysmha01 home]# rpm -qa | grep gluster >>>>>> centos-release-gluster41-1.0-3.el7.centos.noarch >>>>>> glusterfs-server-4.1.6-1.el7.x86_64 >>>>>> glusterfs-api-4.1.6-1.el7.x86_64 >>>>>> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >>>>>> glusterfs-4.1.6-1.el7.x86_64 >>>>>> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >>>>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >>>>>> glusterfs-fuse-4.1.6-1.el7.x86_64 >>>>>> glusterfs-libs-4.1.6-1.el7.x86_64 >>>>>> glusterfs-rdma-4.1.6-1.el7.x86_64 >>>>>> glusterfs-cli-4.1.6-1.el7.x86_64 >>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>> [root at ysmha01 home]# rpm -qa | grep samba >>>>>> samba-common-tools-4.8.3-4.el7.x86_64 >>>>>> samba-client-libs-4.8.3-4.el7.x86_64 >>>>>> samba-libs-4.8.3-4.el7.x86_64 >>>>>> samba-4.8.3-4.el7.x86_64 >>>>>> samba-common-libs-4.8.3-4.el7.x86_64 >>>>>> samba-common-4.8.3-4.el7.noarch >>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>> [root at ysmha01 home]# cat /etc/redhat-release >>>>>> CentOS Linux release 7.6.1810 (Core) >>>>>> >>>>>> RAM view using top >>>>>> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie >>>>>> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 >>>>>> si, 0.0 st >>>>>> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >>>>>> buff/cache >>>>>> KiB Swap: 0 total, 0 free, 0 used. 3134316 >>>>>> avail Mem >>>>>> >>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>> COMMAND >>>>>> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 >>>>>> glusterfsd >>>>>> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 >>>>>> glusterfs >>>>>> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 >>>>>> glusterfs >>>>>> >>>>>> [root at ysmha01 ~]# gluster v status export >>>>>> Status of volume: export >>>>>> Gluster process TCP Port RDMA Port >>>>>> Online Pid >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y >>>>>> 13986 >>>>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y >>>>>> 9953 >>>>>> Self-heal Daemon on localhost N/A N/A Y >>>>>> 14485 >>>>>> Self-heal Daemon on 10.0.1.7 N/A N/A Y >>>>>> 21934 >>>>>> Self-heal Daemon on 10.0.1.5 N/A N/A Y >>>>>> 4598 >>>>>> >>>>>> Task Status of Volume export >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> There are no active volume tasks >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Virus-free. >>>>>> www.avast.com >>>>>> >>>>>> <#m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Tue Jul 30 03:46:39 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Tue, 30 Jul 2019 09:16:39 +0530 Subject: [Gluster-users] Gluster eating up a lot of ram In-Reply-To: References: Message-ID: On Tue, Jul 30, 2019 at 9:04 AM Diego Remolina wrote: > Will this kill the actual process or simply trigger the dump? > This - kill -SIGUSR1 ... - will deliver signal SIGUSR1 to the process. Glusterfs processes (bricks, client mount) are implemented to do a statedump on receiving SIGUSR1. Which process should I kill? The brick process in the system or the fuse > mount? > All those processes that consume large amounts of memory. > Diego > > On Mon, Jul 29, 2019, 23:27 Nithya Balachandran > wrote: > >> >> >> On Tue, 30 Jul 2019 at 05:44, Diego Remolina wrote: >> >>> Unfortunately statedump crashes on both machines, even freshly rebooted. >>> >> >> Do you see any statedump files in /var/run/gluster? This looks more like >> the gluster cli crashed. >> >>> >>> [root at ysmha01 ~]# gluster --print-statedumpdir >>> /var/run/gluster >>> [root at ysmha01 ~]# gluster v statedump export >>> Segmentation fault (core dumped) >>> >>> [root at ysmha02 ~]# uptime >>> 20:12:20 up 6 min, 1 user, load average: 0.72, 0.52, 0.24 >>> [root at ysmha02 ~]# gluster --print-statedumpdir >>> /var/run/gluster >>> [root at ysmha02 ~]# gluster v statedump export >>> Segmentation fault (core dumped) >>> >>> I rebooted today after 40 days. Gluster was eating up shy of 40GB of RAM >>> out of 64. >>> >>> What would you recommend to be the next step? >>> >>> Diego >>> >>> On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah < >>> pgurusid at redhat.com> wrote: >>> >>>> Could you also provide the statedump of the gluster process consuming >>>> 44G ram [1]. Please make sure the statedump is taken when the memory >>>> consumption is very high, like 10s of GBs, otherwise we may not be able to >>>> identify the issue. Also i see that the cache size is 10G is that something >>>> you arrived at, after doing some tests? Its relatively higher than normal. >>>> >>>> [1] >>>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump >>>> >>>> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I will not be able to test gluster-6rc because this is a production >>>>> environment and it takes several days for memory to grow a lot. >>>>> >>>>> The Samba server is hosting all types of files, small and large from >>>>> small roaming profile type files to bigger files like adobe suite, autodesk >>>>> Revit (file sizes in the hundreds of megabytes). >>>>> >>>>> As I stated before, this same issue was present back with 3.8.x which >>>>> I was running before. >>>>> >>>>> The information you requested: >>>>> >>>>> [root at ysmha02 ~]# gluster v info export >>>>> >>>>> Volume Name: export >>>>> Type: Replicate >>>>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >>>>> Status: Started >>>>> Snapshot Count: 0 >>>>> Number of Bricks: 1 x 2 = 2 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: 10.0.1.7:/bricks/hdds/brick >>>>> Brick2: 10.0.1.6:/bricks/hdds/brick >>>>> Options Reconfigured: >>>>> performance.stat-prefetch: on >>>>> performance.cache-min-file-size: 0 >>>>> network.inode-lru-limit: 65536 >>>>> performance.cache-invalidation: on >>>>> features.cache-invalidation: on >>>>> performance.md-cache-timeout: 600 >>>>> features.cache-invalidation-timeout: 600 >>>>> performance.cache-samba-metadata: on >>>>> transport.address-family: inet >>>>> server.allow-insecure: on >>>>> performance.cache-size: 10GB >>>>> cluster.server-quorum-type: server >>>>> nfs.disable: on >>>>> performance.io-thread-count: 64 >>>>> performance.io-cache: on >>>>> cluster.lookup-optimize: on >>>>> cluster.readdir-optimize: on >>>>> server.event-threads: 5 >>>>> client.event-threads: 5 >>>>> performance.cache-max-file-size: 256MB >>>>> diagnostics.client-log-level: INFO >>>>> diagnostics.brick-log-level: INFO >>>>> cluster.server-quorum-ratio: 51% >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Virus-free. >>>>> www.avast.com >>>>> >>>>> <#m_-1957083142921437393_m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>> >>>>> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah < >>>>> pgurusid at redhat.com> wrote: >>>>> >>>>>> This high memory consumption is not normal. Looks like it's a memory >>>>>> leak. Is it possible to try it on test setup with gluster-6rc? What is the >>>>>> kind of workload that goes into fuse mount? Large files or small files? We >>>>>> need the following information to debug further: >>>>>> - Gluster volume info output >>>>>> - Statedump of the Gluster fuse mount process consuming 44G ram. >>>>>> >>>>>> Regards, >>>>>> Poornima >>>>>> >>>>>> >>>>>> On Sat, Mar 2, 2019, 3:40 AM Diego Remolina >>>>>> wrote: >>>>>> >>>>>>> I am using glusterfs with two servers as a file server sharing files >>>>>>> via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in >>>>>>> current Centos version of samba. So I am mounting via fuse and exporting >>>>>>> the volume to samba from the mount point. >>>>>>> >>>>>>> Upon initial boot, the server where samba is exporting files climbs >>>>>>> up to ~10GB RAM within a couple hours of use. From then on, it is a >>>>>>> constant slow memory increase. In the past with gluster 3.8.x we had to >>>>>>> reboot the servers at around 30 days . With gluster 4.1.6 we are getting up >>>>>>> to 48 days, but RAM use is at 48GB out of 64GB. Is this normal? >>>>>>> >>>>>>> The particular versions are below, >>>>>>> >>>>>>> [root at ysmha01 home]# uptime >>>>>>> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 >>>>>>> [root at ysmha01 home]# rpm -qa | grep gluster >>>>>>> centos-release-gluster41-1.0-3.el7.centos.noarch >>>>>>> glusterfs-server-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-api-4.1.6-1.el7.x86_64 >>>>>>> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >>>>>>> glusterfs-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >>>>>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >>>>>>> glusterfs-fuse-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-libs-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-rdma-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-cli-4.1.6-1.el7.x86_64 >>>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>>> [root at ysmha01 home]# rpm -qa | grep samba >>>>>>> samba-common-tools-4.8.3-4.el7.x86_64 >>>>>>> samba-client-libs-4.8.3-4.el7.x86_64 >>>>>>> samba-libs-4.8.3-4.el7.x86_64 >>>>>>> samba-4.8.3-4.el7.x86_64 >>>>>>> samba-common-libs-4.8.3-4.el7.x86_64 >>>>>>> samba-common-4.8.3-4.el7.noarch >>>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>>> [root at ysmha01 home]# cat /etc/redhat-release >>>>>>> CentOS Linux release 7.6.1810 (Core) >>>>>>> >>>>>>> RAM view using top >>>>>>> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie >>>>>>> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 >>>>>>> si, 0.0 st >>>>>>> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >>>>>>> buff/cache >>>>>>> KiB Swap: 0 total, 0 free, 0 used. 3134316 >>>>>>> avail Mem >>>>>>> >>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>>> COMMAND >>>>>>> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 >>>>>>> glusterfsd >>>>>>> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 >>>>>>> glusterfs >>>>>>> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 >>>>>>> glusterfs >>>>>>> >>>>>>> [root at ysmha01 ~]# gluster v status export >>>>>>> Status of volume: export >>>>>>> Gluster process TCP Port RDMA Port >>>>>>> Online Pid >>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y >>>>>>> 13986 >>>>>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y >>>>>>> 9953 >>>>>>> Self-heal Daemon on localhost N/A N/A Y >>>>>>> 14485 >>>>>>> Self-heal Daemon on 10.0.1.7 N/A N/A Y >>>>>>> 21934 >>>>>>> Self-heal Daemon on 10.0.1.5 N/A N/A Y >>>>>>> 4598 >>>>>>> >>>>>>> Task Status of Volume export >>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> There are no active volume tasks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Virus-free. >>>>>>> www.avast.com >>>>>>> >>>>>>> <#m_-1957083142921437393_m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Tue Jul 30 03:51:37 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Tue, 30 Jul 2019 09:21:37 +0530 Subject: [Gluster-users] Gluster eating up a lot of ram In-Reply-To: References: Message-ID: Hi Diego, Please do the following: gluster v get readdir-ahead If this is enabled, please disable it and see if it helps. There was a leak in the opendir codpath that was fixed in later released. Regards, Nithya On Tue, 30 Jul 2019 at 09:04, Diego Remolina wrote: > Will this kill the actual process or simply trigger the dump? Which > process should I kill? The brick process in the system or the fuse mount? > > Diego > > On Mon, Jul 29, 2019, 23:27 Nithya Balachandran > wrote: > >> >> >> On Tue, 30 Jul 2019 at 05:44, Diego Remolina wrote: >> >>> Unfortunately statedump crashes on both machines, even freshly rebooted. >>> >> >> Do you see any statedump files in /var/run/gluster? This looks more like >> the gluster cli crashed. >> >>> >>> [root at ysmha01 ~]# gluster --print-statedumpdir >>> /var/run/gluster >>> [root at ysmha01 ~]# gluster v statedump export >>> Segmentation fault (core dumped) >>> >>> [root at ysmha02 ~]# uptime >>> 20:12:20 up 6 min, 1 user, load average: 0.72, 0.52, 0.24 >>> [root at ysmha02 ~]# gluster --print-statedumpdir >>> /var/run/gluster >>> [root at ysmha02 ~]# gluster v statedump export >>> Segmentation fault (core dumped) >>> >>> I rebooted today after 40 days. Gluster was eating up shy of 40GB of RAM >>> out of 64. >>> >>> What would you recommend to be the next step? >>> >>> Diego >>> >>> On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah < >>> pgurusid at redhat.com> wrote: >>> >>>> Could you also provide the statedump of the gluster process consuming >>>> 44G ram [1]. Please make sure the statedump is taken when the memory >>>> consumption is very high, like 10s of GBs, otherwise we may not be able to >>>> identify the issue. Also i see that the cache size is 10G is that something >>>> you arrived at, after doing some tests? Its relatively higher than normal. >>>> >>>> [1] >>>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump >>>> >>>> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I will not be able to test gluster-6rc because this is a production >>>>> environment and it takes several days for memory to grow a lot. >>>>> >>>>> The Samba server is hosting all types of files, small and large from >>>>> small roaming profile type files to bigger files like adobe suite, autodesk >>>>> Revit (file sizes in the hundreds of megabytes). >>>>> >>>>> As I stated before, this same issue was present back with 3.8.x which >>>>> I was running before. >>>>> >>>>> The information you requested: >>>>> >>>>> [root at ysmha02 ~]# gluster v info export >>>>> >>>>> Volume Name: export >>>>> Type: Replicate >>>>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >>>>> Status: Started >>>>> Snapshot Count: 0 >>>>> Number of Bricks: 1 x 2 = 2 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: 10.0.1.7:/bricks/hdds/brick >>>>> Brick2: 10.0.1.6:/bricks/hdds/brick >>>>> Options Reconfigured: >>>>> performance.stat-prefetch: on >>>>> performance.cache-min-file-size: 0 >>>>> network.inode-lru-limit: 65536 >>>>> performance.cache-invalidation: on >>>>> features.cache-invalidation: on >>>>> performance.md-cache-timeout: 600 >>>>> features.cache-invalidation-timeout: 600 >>>>> performance.cache-samba-metadata: on >>>>> transport.address-family: inet >>>>> server.allow-insecure: on >>>>> performance.cache-size: 10GB >>>>> cluster.server-quorum-type: server >>>>> nfs.disable: on >>>>> performance.io-thread-count: 64 >>>>> performance.io-cache: on >>>>> cluster.lookup-optimize: on >>>>> cluster.readdir-optimize: on >>>>> server.event-threads: 5 >>>>> client.event-threads: 5 >>>>> performance.cache-max-file-size: 256MB >>>>> diagnostics.client-log-level: INFO >>>>> diagnostics.brick-log-level: INFO >>>>> cluster.server-quorum-ratio: 51% >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Virus-free. >>>>> www.avast.com >>>>> >>>>> <#m_8374238785685214358_m_-3340449949414300599_m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>> >>>>> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah < >>>>> pgurusid at redhat.com> wrote: >>>>> >>>>>> This high memory consumption is not normal. Looks like it's a memory >>>>>> leak. Is it possible to try it on test setup with gluster-6rc? What is the >>>>>> kind of workload that goes into fuse mount? Large files or small files? We >>>>>> need the following information to debug further: >>>>>> - Gluster volume info output >>>>>> - Statedump of the Gluster fuse mount process consuming 44G ram. >>>>>> >>>>>> Regards, >>>>>> Poornima >>>>>> >>>>>> >>>>>> On Sat, Mar 2, 2019, 3:40 AM Diego Remolina >>>>>> wrote: >>>>>> >>>>>>> I am using glusterfs with two servers as a file server sharing files >>>>>>> via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in >>>>>>> current Centos version of samba. So I am mounting via fuse and exporting >>>>>>> the volume to samba from the mount point. >>>>>>> >>>>>>> Upon initial boot, the server where samba is exporting files climbs >>>>>>> up to ~10GB RAM within a couple hours of use. From then on, it is a >>>>>>> constant slow memory increase. In the past with gluster 3.8.x we had to >>>>>>> reboot the servers at around 30 days . With gluster 4.1.6 we are getting up >>>>>>> to 48 days, but RAM use is at 48GB out of 64GB. Is this normal? >>>>>>> >>>>>>> The particular versions are below, >>>>>>> >>>>>>> [root at ysmha01 home]# uptime >>>>>>> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 >>>>>>> [root at ysmha01 home]# rpm -qa | grep gluster >>>>>>> centos-release-gluster41-1.0-3.el7.centos.noarch >>>>>>> glusterfs-server-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-api-4.1.6-1.el7.x86_64 >>>>>>> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >>>>>>> glusterfs-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >>>>>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >>>>>>> glusterfs-fuse-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-libs-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-rdma-4.1.6-1.el7.x86_64 >>>>>>> glusterfs-cli-4.1.6-1.el7.x86_64 >>>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>>> [root at ysmha01 home]# rpm -qa | grep samba >>>>>>> samba-common-tools-4.8.3-4.el7.x86_64 >>>>>>> samba-client-libs-4.8.3-4.el7.x86_64 >>>>>>> samba-libs-4.8.3-4.el7.x86_64 >>>>>>> samba-4.8.3-4.el7.x86_64 >>>>>>> samba-common-libs-4.8.3-4.el7.x86_64 >>>>>>> samba-common-4.8.3-4.el7.noarch >>>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>>> [root at ysmha01 home]# cat /etc/redhat-release >>>>>>> CentOS Linux release 7.6.1810 (Core) >>>>>>> >>>>>>> RAM view using top >>>>>>> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie >>>>>>> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 >>>>>>> si, 0.0 st >>>>>>> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >>>>>>> buff/cache >>>>>>> KiB Swap: 0 total, 0 free, 0 used. 3134316 >>>>>>> avail Mem >>>>>>> >>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>>> COMMAND >>>>>>> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 >>>>>>> glusterfsd >>>>>>> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 >>>>>>> glusterfs >>>>>>> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 >>>>>>> glusterfs >>>>>>> >>>>>>> [root at ysmha01 ~]# gluster v status export >>>>>>> Status of volume: export >>>>>>> Gluster process TCP Port RDMA Port >>>>>>> Online Pid >>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y >>>>>>> 13986 >>>>>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y >>>>>>> 9953 >>>>>>> Self-heal Daemon on localhost N/A N/A Y >>>>>>> 14485 >>>>>>> Self-heal Daemon on 10.0.1.7 N/A N/A Y >>>>>>> 21934 >>>>>>> Self-heal Daemon on 10.0.1.5 N/A N/A Y >>>>>>> 4598 >>>>>>> >>>>>>> Task Status of Volume export >>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> There are no active volume tasks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Virus-free. >>>>>>> www.avast.com >>>>>>> >>>>>>> <#m_8374238785685214358_m_-3340449949414300599_m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dijuremo at gmail.com Tue Jul 30 11:07:11 2019 From: dijuremo at gmail.com (Diego Remolina) Date: Tue, 30 Jul 2019 07:07:11 -0400 Subject: [Gluster-users] Gluster eating up a lot of ram In-Reply-To: References: Message-ID: This option is enabled. In which version has this been patched? This is a file server and disabling readdir-ahead will have a hard impact on performance. [root at ysmha01 ~]# gluster v get export readdir-ahead Option Value ------ ----- performance.readdir-ahead on The guide recommends enabling the setting: https://docs.gluster.org/en/latest/Administrator%20Guide/Accessing%20Gluster%20from%20Windows/ Diego On Mon, Jul 29, 2019 at 11:52 PM Nithya Balachandran wrote: > > Hi Diego, > > Please do the following: > > gluster v get readdir-ahead > > If this is enabled, please disable it and see if it helps. There was a > leak in the opendir codpath that was fixed in later released. > > Regards, > Nithya > > > On Tue, 30 Jul 2019 at 09:04, Diego Remolina wrote: > >> Will this kill the actual process or simply trigger the dump? Which >> process should I kill? The brick process in the system or the fuse mount? >> >> Diego >> >> On Mon, Jul 29, 2019, 23:27 Nithya Balachandran >> wrote: >> >>> >>> >>> On Tue, 30 Jul 2019 at 05:44, Diego Remolina wrote: >>> >>>> Unfortunately statedump crashes on both machines, even freshly rebooted. >>>> >>> >>> Do you see any statedump files in /var/run/gluster? This looks more >>> like the gluster cli crashed. >>> >>>> >>>> [root at ysmha01 ~]# gluster --print-statedumpdir >>>> /var/run/gluster >>>> [root at ysmha01 ~]# gluster v statedump export >>>> Segmentation fault (core dumped) >>>> >>>> [root at ysmha02 ~]# uptime >>>> 20:12:20 up 6 min, 1 user, load average: 0.72, 0.52, 0.24 >>>> [root at ysmha02 ~]# gluster --print-statedumpdir >>>> /var/run/gluster >>>> [root at ysmha02 ~]# gluster v statedump export >>>> Segmentation fault (core dumped) >>>> >>>> I rebooted today after 40 days. Gluster was eating up shy of 40GB of >>>> RAM out of 64. >>>> >>>> What would you recommend to be the next step? >>>> >>>> Diego >>>> >>>> On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah < >>>> pgurusid at redhat.com> wrote: >>>> >>>>> Could you also provide the statedump of the gluster process consuming >>>>> 44G ram [1]. Please make sure the statedump is taken when the memory >>>>> consumption is very high, like 10s of GBs, otherwise we may not be able to >>>>> identify the issue. Also i see that the cache size is 10G is that something >>>>> you arrived at, after doing some tests? Its relatively higher than normal. >>>>> >>>>> [1] >>>>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump >>>>> >>>>> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I will not be able to test gluster-6rc because this is a production >>>>>> environment and it takes several days for memory to grow a lot. >>>>>> >>>>>> The Samba server is hosting all types of files, small and large from >>>>>> small roaming profile type files to bigger files like adobe suite, autodesk >>>>>> Revit (file sizes in the hundreds of megabytes). >>>>>> >>>>>> As I stated before, this same issue was present back with 3.8.x which >>>>>> I was running before. >>>>>> >>>>>> The information you requested: >>>>>> >>>>>> [root at ysmha02 ~]# gluster v info export >>>>>> >>>>>> Volume Name: export >>>>>> Type: Replicate >>>>>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >>>>>> Status: Started >>>>>> Snapshot Count: 0 >>>>>> Number of Bricks: 1 x 2 = 2 >>>>>> Transport-type: tcp >>>>>> Bricks: >>>>>> Brick1: 10.0.1.7:/bricks/hdds/brick >>>>>> Brick2: 10.0.1.6:/bricks/hdds/brick >>>>>> Options Reconfigured: >>>>>> performance.stat-prefetch: on >>>>>> performance.cache-min-file-size: 0 >>>>>> network.inode-lru-limit: 65536 >>>>>> performance.cache-invalidation: on >>>>>> features.cache-invalidation: on >>>>>> performance.md-cache-timeout: 600 >>>>>> features.cache-invalidation-timeout: 600 >>>>>> performance.cache-samba-metadata: on >>>>>> transport.address-family: inet >>>>>> server.allow-insecure: on >>>>>> performance.cache-size: 10GB >>>>>> cluster.server-quorum-type: server >>>>>> nfs.disable: on >>>>>> performance.io-thread-count: 64 >>>>>> performance.io-cache: on >>>>>> cluster.lookup-optimize: on >>>>>> cluster.readdir-optimize: on >>>>>> server.event-threads: 5 >>>>>> client.event-threads: 5 >>>>>> performance.cache-max-file-size: 256MB >>>>>> diagnostics.client-log-level: INFO >>>>>> diagnostics.brick-log-level: INFO >>>>>> cluster.server-quorum-ratio: 51% >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Virus-free. >>>>>> www.avast.com >>>>>> >>>>>> <#m_-4833401328225509760_m_8374238785685214358_m_-3340449949414300599_m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>> >>>>>> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah < >>>>>> pgurusid at redhat.com> wrote: >>>>>> >>>>>>> This high memory consumption is not normal. Looks like it's a memory >>>>>>> leak. Is it possible to try it on test setup with gluster-6rc? What is the >>>>>>> kind of workload that goes into fuse mount? Large files or small files? We >>>>>>> need the following information to debug further: >>>>>>> - Gluster volume info output >>>>>>> - Statedump of the Gluster fuse mount process consuming 44G ram. >>>>>>> >>>>>>> Regards, >>>>>>> Poornima >>>>>>> >>>>>>> >>>>>>> On Sat, Mar 2, 2019, 3:40 AM Diego Remolina >>>>>>> wrote: >>>>>>> >>>>>>>> I am using glusterfs with two servers as a file server sharing >>>>>>>> files via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug >>>>>>>> in current Centos version of samba. So I am mounting via fuse and exporting >>>>>>>> the volume to samba from the mount point. >>>>>>>> >>>>>>>> Upon initial boot, the server where samba is exporting files climbs >>>>>>>> up to ~10GB RAM within a couple hours of use. From then on, it is a >>>>>>>> constant slow memory increase. In the past with gluster 3.8.x we had to >>>>>>>> reboot the servers at around 30 days . With gluster 4.1.6 we are getting up >>>>>>>> to 48 days, but RAM use is at 48GB out of 64GB. Is this normal? >>>>>>>> >>>>>>>> The particular versions are below, >>>>>>>> >>>>>>>> [root at ysmha01 home]# uptime >>>>>>>> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 >>>>>>>> [root at ysmha01 home]# rpm -qa | grep gluster >>>>>>>> centos-release-gluster41-1.0-3.el7.centos.noarch >>>>>>>> glusterfs-server-4.1.6-1.el7.x86_64 >>>>>>>> glusterfs-api-4.1.6-1.el7.x86_64 >>>>>>>> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >>>>>>>> glusterfs-4.1.6-1.el7.x86_64 >>>>>>>> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >>>>>>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >>>>>>>> glusterfs-fuse-4.1.6-1.el7.x86_64 >>>>>>>> glusterfs-libs-4.1.6-1.el7.x86_64 >>>>>>>> glusterfs-rdma-4.1.6-1.el7.x86_64 >>>>>>>> glusterfs-cli-4.1.6-1.el7.x86_64 >>>>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>>>> [root at ysmha01 home]# rpm -qa | grep samba >>>>>>>> samba-common-tools-4.8.3-4.el7.x86_64 >>>>>>>> samba-client-libs-4.8.3-4.el7.x86_64 >>>>>>>> samba-libs-4.8.3-4.el7.x86_64 >>>>>>>> samba-4.8.3-4.el7.x86_64 >>>>>>>> samba-common-libs-4.8.3-4.el7.x86_64 >>>>>>>> samba-common-4.8.3-4.el7.noarch >>>>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>>>> [root at ysmha01 home]# cat /etc/redhat-release >>>>>>>> CentOS Linux release 7.6.1810 (Core) >>>>>>>> >>>>>>>> RAM view using top >>>>>>>> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie >>>>>>>> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 >>>>>>>> si, 0.0 st >>>>>>>> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >>>>>>>> buff/cache >>>>>>>> KiB Swap: 0 total, 0 free, 0 used. 3134316 >>>>>>>> avail Mem >>>>>>>> >>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>>>> COMMAND >>>>>>>> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 >>>>>>>> glusterfsd >>>>>>>> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 >>>>>>>> glusterfs >>>>>>>> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 >>>>>>>> glusterfs >>>>>>>> >>>>>>>> [root at ysmha01 ~]# gluster v status export >>>>>>>> Status of volume: export >>>>>>>> Gluster process TCP Port RDMA Port >>>>>>>> Online Pid >>>>>>>> >>>>>>>> ------------------------------------------------------------------------------ >>>>>>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 >>>>>>>> Y 13986 >>>>>>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 >>>>>>>> Y 9953 >>>>>>>> Self-heal Daemon on localhost N/A N/A Y >>>>>>>> 14485 >>>>>>>> Self-heal Daemon on 10.0.1.7 N/A N/A Y >>>>>>>> 21934 >>>>>>>> Self-heal Daemon on 10.0.1.5 N/A N/A Y >>>>>>>> 4598 >>>>>>>> >>>>>>>> Task Status of Volume export >>>>>>>> >>>>>>>> ------------------------------------------------------------------------------ >>>>>>>> There are no active volume tasks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Virus-free. >>>>>>>> www.avast.com >>>>>>>> >>>>>>>> <#m_-4833401328225509760_m_8374238785685214358_m_-3340449949414300599_m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Tue Jul 30 12:43:57 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Tue, 30 Jul 2019 18:13:57 +0530 Subject: [Gluster-users] Gluster eating up a lot of ram In-Reply-To: References: Message-ID: On Tue, 30 Jul 2019 at 16:37, Diego Remolina wrote: > This option is enabled. In which version has this been patched? This is a > file server and disabling readdir-ahead will have a hard impact on > performance. > This was fixed in 5.3 (https://bugzilla.redhat.com/show_bug.cgi?id=1659676 ). This bug is only relevant if the gluster fuse client is the one that is using up memory. The first thing to do would be to determine which process is using up the memory and to get a statedump. ps should give you the details of the gluster process . Regards, Nithya > > [root at ysmha01 ~]# gluster v get export readdir-ahead > Option Value > > ------ ----- > > performance.readdir-ahead on > > The guide recommends enabling the setting: > > > https://docs.gluster.org/en/latest/Administrator%20Guide/Accessing%20Gluster%20from%20Windows/ > > Diego > > > > On Mon, Jul 29, 2019 at 11:52 PM Nithya Balachandran > wrote: > >> >> Hi Diego, >> >> Please do the following: >> >> gluster v get readdir-ahead >> >> If this is enabled, please disable it and see if it helps. There was a >> leak in the opendir codpath that was fixed in later released. >> >> Regards, >> Nithya >> >> >> On Tue, 30 Jul 2019 at 09:04, Diego Remolina wrote: >> >>> Will this kill the actual process or simply trigger the dump? Which >>> process should I kill? The brick process in the system or the fuse mount? >>> >>> Diego >>> >>> On Mon, Jul 29, 2019, 23:27 Nithya Balachandran >>> wrote: >>> >>>> >>>> >>>> On Tue, 30 Jul 2019 at 05:44, Diego Remolina >>>> wrote: >>>> >>>>> Unfortunately statedump crashes on both machines, even freshly >>>>> rebooted. >>>>> >>>> >>>> Do you see any statedump files in /var/run/gluster? This looks more >>>> like the gluster cli crashed. >>>> >>>>> >>>>> [root at ysmha01 ~]# gluster --print-statedumpdir >>>>> /var/run/gluster >>>>> [root at ysmha01 ~]# gluster v statedump export >>>>> Segmentation fault (core dumped) >>>>> >>>>> [root at ysmha02 ~]# uptime >>>>> 20:12:20 up 6 min, 1 user, load average: 0.72, 0.52, 0.24 >>>>> [root at ysmha02 ~]# gluster --print-statedumpdir >>>>> /var/run/gluster >>>>> [root at ysmha02 ~]# gluster v statedump export >>>>> Segmentation fault (core dumped) >>>>> >>>>> I rebooted today after 40 days. Gluster was eating up shy of 40GB of >>>>> RAM out of 64. >>>>> >>>>> What would you recommend to be the next step? >>>>> >>>>> Diego >>>>> >>>>> On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah < >>>>> pgurusid at redhat.com> wrote: >>>>> >>>>>> Could you also provide the statedump of the gluster process consuming >>>>>> 44G ram [1]. Please make sure the statedump is taken when the memory >>>>>> consumption is very high, like 10s of GBs, otherwise we may not be able to >>>>>> identify the issue. Also i see that the cache size is 10G is that something >>>>>> you arrived at, after doing some tests? Its relatively higher than normal. >>>>>> >>>>>> [1] >>>>>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump >>>>>> >>>>>> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I will not be able to test gluster-6rc because this is a production >>>>>>> environment and it takes several days for memory to grow a lot. >>>>>>> >>>>>>> The Samba server is hosting all types of files, small and large from >>>>>>> small roaming profile type files to bigger files like adobe suite, autodesk >>>>>>> Revit (file sizes in the hundreds of megabytes). >>>>>>> >>>>>>> As I stated before, this same issue was present back with 3.8.x >>>>>>> which I was running before. >>>>>>> >>>>>>> The information you requested: >>>>>>> >>>>>>> [root at ysmha02 ~]# gluster v info export >>>>>>> >>>>>>> Volume Name: export >>>>>>> Type: Replicate >>>>>>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff >>>>>>> Status: Started >>>>>>> Snapshot Count: 0 >>>>>>> Number of Bricks: 1 x 2 = 2 >>>>>>> Transport-type: tcp >>>>>>> Bricks: >>>>>>> Brick1: 10.0.1.7:/bricks/hdds/brick >>>>>>> Brick2: 10.0.1.6:/bricks/hdds/brick >>>>>>> Options Reconfigured: >>>>>>> performance.stat-prefetch: on >>>>>>> performance.cache-min-file-size: 0 >>>>>>> network.inode-lru-limit: 65536 >>>>>>> performance.cache-invalidation: on >>>>>>> features.cache-invalidation: on >>>>>>> performance.md-cache-timeout: 600 >>>>>>> features.cache-invalidation-timeout: 600 >>>>>>> performance.cache-samba-metadata: on >>>>>>> transport.address-family: inet >>>>>>> server.allow-insecure: on >>>>>>> performance.cache-size: 10GB >>>>>>> cluster.server-quorum-type: server >>>>>>> nfs.disable: on >>>>>>> performance.io-thread-count: 64 >>>>>>> performance.io-cache: on >>>>>>> cluster.lookup-optimize: on >>>>>>> cluster.readdir-optimize: on >>>>>>> server.event-threads: 5 >>>>>>> client.event-threads: 5 >>>>>>> performance.cache-max-file-size: 256MB >>>>>>> diagnostics.client-log-level: INFO >>>>>>> diagnostics.brick-log-level: INFO >>>>>>> cluster.server-quorum-ratio: 51% >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Virus-free. >>>>>>> www.avast.com >>>>>>> >>>>>>> <#m_119331930431500301_m_-8568737347882478303_m_-4833401328225509760_m_8374238785685214358_m_-3340449949414300599_m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>>> >>>>>>> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah < >>>>>>> pgurusid at redhat.com> wrote: >>>>>>> >>>>>>>> This high memory consumption is not normal. Looks like it's a >>>>>>>> memory leak. Is it possible to try it on test setup with gluster-6rc? What >>>>>>>> is the kind of workload that goes into fuse mount? Large files or small >>>>>>>> files? We need the following information to debug further: >>>>>>>> - Gluster volume info output >>>>>>>> - Statedump of the Gluster fuse mount process consuming 44G ram. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Poornima >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Mar 2, 2019, 3:40 AM Diego Remolina >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I am using glusterfs with two servers as a file server sharing >>>>>>>>> files via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug >>>>>>>>> in current Centos version of samba. So I am mounting via fuse and exporting >>>>>>>>> the volume to samba from the mount point. >>>>>>>>> >>>>>>>>> Upon initial boot, the server where samba is exporting files >>>>>>>>> climbs up to ~10GB RAM within a couple hours of use. From then on, it is a >>>>>>>>> constant slow memory increase. In the past with gluster 3.8.x we had to >>>>>>>>> reboot the servers at around 30 days . With gluster 4.1.6 we are getting up >>>>>>>>> to 48 days, but RAM use is at 48GB out of 64GB. Is this normal? >>>>>>>>> >>>>>>>>> The particular versions are below, >>>>>>>>> >>>>>>>>> [root at ysmha01 home]# uptime >>>>>>>>> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, >>>>>>>>> 3.00 >>>>>>>>> [root at ysmha01 home]# rpm -qa | grep gluster >>>>>>>>> centos-release-gluster41-1.0-3.el7.centos.noarch >>>>>>>>> glusterfs-server-4.1.6-1.el7.x86_64 >>>>>>>>> glusterfs-api-4.1.6-1.el7.x86_64 >>>>>>>>> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >>>>>>>>> glusterfs-4.1.6-1.el7.x86_64 >>>>>>>>> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >>>>>>>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >>>>>>>>> glusterfs-fuse-4.1.6-1.el7.x86_64 >>>>>>>>> glusterfs-libs-4.1.6-1.el7.x86_64 >>>>>>>>> glusterfs-rdma-4.1.6-1.el7.x86_64 >>>>>>>>> glusterfs-cli-4.1.6-1.el7.x86_64 >>>>>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>>>>> [root at ysmha01 home]# rpm -qa | grep samba >>>>>>>>> samba-common-tools-4.8.3-4.el7.x86_64 >>>>>>>>> samba-client-libs-4.8.3-4.el7.x86_64 >>>>>>>>> samba-libs-4.8.3-4.el7.x86_64 >>>>>>>>> samba-4.8.3-4.el7.x86_64 >>>>>>>>> samba-common-libs-4.8.3-4.el7.x86_64 >>>>>>>>> samba-common-4.8.3-4.el7.noarch >>>>>>>>> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >>>>>>>>> [root at ysmha01 home]# cat /etc/redhat-release >>>>>>>>> CentOS Linux release 7.6.1810 (Core) >>>>>>>>> >>>>>>>>> RAM view using top >>>>>>>>> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 >>>>>>>>> zombie >>>>>>>>> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, >>>>>>>>> 0.8 si, 0.0 st >>>>>>>>> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >>>>>>>>> buff/cache >>>>>>>>> KiB Swap: 0 total, 0 free, 0 used. 3134316 >>>>>>>>> avail Mem >>>>>>>>> >>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM >>>>>>>>> TIME+ COMMAND >>>>>>>>> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 >>>>>>>>> 38626:27 glusterfsd >>>>>>>>> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 >>>>>>>>> 29513:55 glusterfs >>>>>>>>> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 >>>>>>>>> 1590:13 glusterfs >>>>>>>>> >>>>>>>>> [root at ysmha01 ~]# gluster v status export >>>>>>>>> Status of volume: export >>>>>>>>> Gluster process TCP Port RDMA Port >>>>>>>>> Online Pid >>>>>>>>> >>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 >>>>>>>>> Y 13986 >>>>>>>>> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 >>>>>>>>> Y 9953 >>>>>>>>> Self-heal Daemon on localhost N/A N/A >>>>>>>>> Y 14485 >>>>>>>>> Self-heal Daemon on 10.0.1.7 N/A N/A >>>>>>>>> Y 21934 >>>>>>>>> Self-heal Daemon on 10.0.1.5 N/A N/A >>>>>>>>> Y 4598 >>>>>>>>> >>>>>>>>> Task Status of Volume export >>>>>>>>> >>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>> There are no active volume tasks >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Virus-free. >>>>>>>>> www.avast.com >>>>>>>>> >>>>>>>>> <#m_119331930431500301_m_-8568737347882478303_m_-4833401328225509760_m_8374238785685214358_m_-3340449949414300599_m_-7001269052163580460_m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mscherer at redhat.com Wed Jul 31 17:31:12 2019 From: mscherer at redhat.com (Michael Scherer) Date: Wed, 31 Jul 2019 19:31:12 +0200 Subject: [Gluster-users] Ongoing issue with docs.gluster.org Message-ID: <26e3d4c1e1ceeeecb0fb2fcaeff12687eae66251.camel@redhat.com> Hi, people (and nagios) have reported issue with the docs.gluster.org domain. I got texted last night, but the problem solved itself while I was sleeping. It seems to be back, and I restarted the proxy since it seems that readthedocs changed the IP of their load balancer (and it was cached by nginx so slow to propagate) See https://twitter.com/readthedocs/status/1156337277640908801 for the initial report. This is kinda out of the control of the gluster infra team, but do not hesitate to send support, love or money to the volunteers of RTD. We are monitoring the issue. -- Michael Scherer Sysadmin, Community Infrastructure -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: From dcunningham at voisonics.com Wed Jul 31 21:52:09 2019 From: dcunningham at voisonics.com (David Cunningham) Date: Thu, 1 Aug 2019 09:52:09 +1200 Subject: [Gluster-users] Thin-arbiter questions In-Reply-To: <1263717773.2661191.1563940317878.JavaMail.zimbra@redhat.com> References: <2006209001.22227657.1560230122663.JavaMail.zimbra@redhat.com> <1263717773.2661191.1563940317878.JavaMail.zimbra@redhat.com> Message-ID: Hi Ashish, Thank you for that! I see on the release-schedule page that Gluster 7 is in "planning" state, and presumably will be available in a few months. We'll keep a look out for it. On Wed, 24 Jul 2019 at 15:51, Ashish Pandey wrote: > > Hi David, > > This was the bug fix and it had to go to gluster 6 also in which > thin-arbiter is supported with glusterd 2. > However, what you were looking for was the support of thin-arbiter in > glusterd which is available in gluster 7 only. > > So now using gluster 7.0, you can create and use thin-arbiter volume just > like any other volume with few additional steps. > > Following is the command you can use - > > gluster volume create replica 2 thin-arbiter 1 : > : : [force] > > Before that you have to run thin-arbiter process on TA host. > The above command you can use only with release 7 and not with older > release. > > Patches which got merged for this change are - > https://review.gluster.org/#/c/glusterfs/+/22992/ - release 7 > https://review.gluster.org/#/c/glusterfs/+/22612/ - master > > > I am trying to come up with modified document/blog for this asap. > > --- > Ashish > > > ------------------------------ > *From: *"David Cunningham" > *To: *"Hari Gowtham" > *Cc: *"gluster-users" > *Sent: *Wednesday, July 24, 2019 5:17:50 AM > *Subject: *Re: [Gluster-users] Thin-arbiter questions > > Hello, > > I see references to fixes for thin-arbiter on the 6.x release notes. Does > that mean thin-arbiter is ready to use on glusterfs 6 please? > > > > On Wed, 19 Jun 2019 at 17:42, Hari Gowtham wrote: > >> The committing should happen a little earlier than 10th. The tagging is >> the one which decides if the patch makes it to the release or not and >> tagging happens before 10th. We do announce the tagging date for each >> release in the mailing list. You can keep an eye on that to know the dates. >> And committing in master won't be enough for it to make it to a release. If >> it has to be a part of release 6 then after being committed into master we >> have to back port it to the release 6 branch and it should get committed in >> that particular branch as well. Only then it will be a part of the package >> released for that branch. >> >> >> On Wed, 19 Jun, 2019, 4:24 AM David Cunningham, < >> dcunningham at voisonics.com> wrote: >> >>> Hi Hari, >>> >>> Thanks for that information. So if I understand correctly, if >>> thin-arbiter is committed to the master branch by the 10th July, then it >>> should be in the CentOS package fairly soon afterwards? >>> >>> I have a customer asking when we can use it, hence the questions. Thank >>> you. >>> >>> >>> On Tue, 18 Jun 2019 at 17:24, Hari Gowtham wrote: >>> >>>> Hi David, >>>> >>>> Once a feature is added to the master branch, we have to back port it >>>> to the release 5, 6 and other such branches which are active. And these >>>> release branches will be tagged every month around 10th. So if an feature >>>> has been back ported to the particular release branch before tagging, then >>>> it will be a part of the tagging. And this tag is the one used for creating >>>> packaging. This is the procedure for CentOS, Fedora and Debian. >>>> >>>> Regards, >>>> Hari. >>>> >>>> On Tue, 18 Jun, 2019, 4:06 AM David Cunningham, < >>>> dcunningham at voisonics.com> wrote: >>>> >>>>> Hi Ashish, >>>>> >>>>> Thanks for that. I guess it's not your responsibility, but do you know >>>>> how often it typically takes for new versions to reach the CentOS package >>>>> system after being released? >>>>> >>>>> >>>>> On Tue, 11 Jun 2019 at 17:15, Ashish Pandey >>>>> wrote: >>>>> >>>>>> Hi David, >>>>>> >>>>>> It should be any time soon as we are in last phase of patch reviews. >>>>>> You can follow this patch - >>>>>> https://review.gluster.org/#/c/glusterfs/+/22612/ >>>>>> >>>>>> --- >>>>>> Ashish >>>>>> >>>>>> ------------------------------ >>>>>> *From: *"David Cunningham" >>>>>> *To: *"Ashish Pandey" >>>>>> *Cc: *"gluster-users" >>>>>> *Sent: *Tuesday, June 11, 2019 9:55:40 AM >>>>>> *Subject: *Re: [Gluster-users] Thin-arbiter questions >>>>>> >>>>>> Hi Ashish and Amar, >>>>>> >>>>>> Is there any news on when thin-arbiter might be in the regular >>>>>> GlusterFS, and the CentOS packages please? >>>>>> >>>>>> Thanks for your help. >>>>>> >>>>>> >>>>>> On Mon, 6 May 2019 at 20:34, Ashish Pandey >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------ >>>>>>> *From: *"David Cunningham" >>>>>>> *To: *"Ashish Pandey" >>>>>>> *Cc: *"gluster-users" >>>>>>> *Sent: *Monday, May 6, 2019 1:40:30 PM >>>>>>> *Subject: *Re: [Gluster-users] Thin-arbiter questions >>>>>>> >>>>>>> Hi Ashish, >>>>>>> >>>>>>> Thank you for the update. Does that mean they're now in the regular >>>>>>> Glusterfs? Any idea how long it typically takes the Ubuntu and CentOS >>>>>>> packages to be updated with the latest code? >>>>>>> >>>>>>> No, for regular glusterd, work is still in progress. It will be done >>>>>>> soon. >>>>>>> I don't have answer for the next question. May be Amar have >>>>>>> information regarding this. Adding him in CC. >>>>>>> >>>>>>> >>>>>>> On Mon, 6 May 2019 at 18:21, Ashish Pandey >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I can see that Amar has already committed the changes and those are >>>>>>>> visible on >>>>>>>> https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/ >>>>>>>> >>>>>>>> --- >>>>>>>> Ashish >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------ >>>>>>>> *From: *"Strahil" >>>>>>>> *To: *"Ashish" , "David" < >>>>>>>> dcunningham at voisonics.com> >>>>>>>> *Cc: *"gluster-users" >>>>>>>> *Sent: *Saturday, May 4, 2019 12:10:01 AM >>>>>>>> *Subject: *Re: [Gluster-users] Thin-arbiter questions >>>>>>>> >>>>>>>> Hi Ashish, >>>>>>>> >>>>>>>> Can someone commit the doc change I have already proposed ? >>>>>>>> At least, the doc will clarify that fact . >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Strahil Nikolov >>>>>>>> On May 3, 2019 05:30, Ashish Pandey wrote: >>>>>>>> >>>>>>>> Hi David, >>>>>>>> >>>>>>>> Creation of thin-arbiter volume is currently supported by GD2 only. >>>>>>>> The command "glustercli" is available when glusterd2 is running. >>>>>>>> We are also working on providing thin-arbiter support on glusted >>>>>>>> however, it is not available right now. >>>>>>>> https://review.gluster.org/#/c/glusterfs/+/22612/ >>>>>>>> >>>>>>>> --- >>>>>>>> Ashish >>>>>>>> >>>>>>>> ------------------------------ >>>>>>>> *From: *"David Cunningham" >>>>>>>> *To: *gluster-users at gluster.org >>>>>>>> *Sent: *Friday, May 3, 2019 7:40:03 AM >>>>>>>> *Subject: *[Gluster-users] Thin-arbiter questions >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> We are setting up a thin-arbiter and hope someone can help with >>>>>>>> some questions. We've been following the documentation from >>>>>>>> https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/ >>>>>>>> . >>>>>>>> >>>>>>>> 1. What release of 5.x supports thin-arbiter? We tried a "gluster >>>>>>>> volume create" with the --thin-arbiter option on 5.5 and got an >>>>>>>> "unrecognized option --thin-arbiter" error. >>>>>>>> >>>>>>>> 2. The instruction to create a new volume with a thin-arbiter is >>>>>>>> clear. How do you add a thin-arbiter to an already existing volume though? >>>>>>>> >>>>>>>> 3. The documentation suggests running glusterfsd manually to start >>>>>>>> the thin-arbiter. Is there a service that can do this instead? I found a >>>>>>>> mention of one in >>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1579786 but it's not >>>>>>>> really documented. >>>>>>>> >>>>>>>> Thanks in advance for your help, >>>>>>>> >>>>>>>> -- >>>>>>>> David Cunningham, Voisonics Limited >>>>>>>> http://voisonics.com/ >>>>>>>> USA: +1 213 221 1092 >>>>>>>> New Zealand: +64 (0)28 2558 3782 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> David Cunningham, Voisonics Limited >>>>>>> http://voisonics.com/ >>>>>>> USA: +1 213 221 1092 >>>>>>> New Zealand: +64 (0)28 2558 3782 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> David Cunningham, Voisonics Limited >>>>>> http://voisonics.com/ >>>>>> USA: +1 213 221 1092 >>>>>> New Zealand: +64 (0)28 2558 3782 >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> >>>>> >>>>> -- >>>>> David Cunningham, Voisonics Limited >>>>> http://voisonics.com/ >>>>> USA: +1 213 221 1092 >>>>> New Zealand: +64 (0)28 2558 3782 >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> >>> >>> -- >>> David Cunningham, Voisonics Limited >>> http://voisonics.com/ >>> USA: +1 213 221 1092 >>> New Zealand: +64 (0)28 2558 3782 >>> >> > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 -------------- next part -------------- An HTML attachment was scrubbed... URL: