From bugzilla at redhat.com Fri Feb 1 03:17:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:17:34 +0000 Subject: [Bugs] [Bug 1671603] New: flooding of "dict is NULL" logging Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 Bug ID: 1671603 Summary: flooding of "dict is NULL" logging Product: GlusterFS Version: 5 Status: NEW Component: core Keywords: Triaged, ZStream Assignee: bugs at gluster.org Reporter: atumball at redhat.com CC: bugs at gluster.org Depends On: 1313567 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1313567 +++ Description of problem: following logs flood the log files [2016-03-01 10:45:51.688339] W [dict.c:1282:dict_foreach_match] (-->/usr/lib64/libglusterfs.so.0(dict_foreach_match+0x65) [0x7ff139e1e5d5] -->/usr/lib64/glusterfs/3.7.8/xlator/features/index.so(+0x3950) [0x7ff12de49950] -->/usr/lib64/libglusterfs.so.0(dict_foreach_match+0xe1) [0x7ff139e1e651] ) 0-dict: dict|match|action is NULL [Invalid argument] Version-Release number of selected component (if applicable): glusterfs-3.7.8 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: I have seen the older one https://bugzilla.redhat.com/show_bug.cgi?id=1289893 but since i am using the latest version (3.7.8) fix is there. Could this be related to another part of index.c ? --- Additional comment from Red Hat Bugzilla Rules Engine on 2016-03-01 23:14:38 UTC --- This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs?3.1.z' to '?'. If this bug should be proposed for a different release, please manually change the proposed release flag. --- Additional comment from Nithya Balachandran on 2016-03-09 04:16:05 UTC --- This looks like it refers to an upstream release (3.7.8). Changing the product to reflect this. --- Additional comment from evangelos on 2016-03-09 09:18:21 UTC --- is there any update on this ? thank you very much! --- Additional comment from Nithya Balachandran on 2016-03-09 10:43:09 UTC --- Moving this to Anuradha who worked on the original patch. --- Additional comment from evangelos on 2016-04-12 11:20:00 UTC --- is there any update on this ? --- Additional comment from Anuradha on 2016-06-23 10:01:53 UTC --- Hi evangelos, That fix was made in 3.7.5. You say you have been the old issue. Did you upgrade from 3.7.5 to 3.7.8 and are seeing problem or was this volume freshly created based on 3.7.8? As far as I know all the fixes for dict is NULL in index translator are sent. But there is an issue when volfiles are not updated during an upgrade. If you had upgraded the volume, could you please provided the steps that you used to upgrade? Also, could you also verify one thing for me from brick volfiles of you volume? The brick volfiles are supposed to have the following lines: volume test-index type features/index option xattrop-pending-watchlist trusted.afr.test- <--------(1) option xattrop-dirty-watchlist trusted.afr.dirty <--------(2) option index-base /export/test/brick2/.glusterfs/indices subvolumes test-barrier end-volume The two options mentioned above should exist. Otherwise you will see this problem. You can find volfiles at /var/lib/glusterd/vols/. Thanks. --- Additional comment from evangelos on 2016-07-07 19:28:59 UTC --- Hi Anuradha, in the mean time due to various issues we had we decided to downgrade to 3.6 branch so currently i do not have a 3.7 deployment up and running. But thx for the hint i will keep this in mind for the future. In the meantime you can close this bugzilla. thank you --- Additional comment from Anuradha on 2016-07-11 09:09:11 UTC --- Hi Evangelos, Thanks for the update. Closing this bug as per comment#7 Thanks, Anuradha. --- Additional comment from Emerson Gomes on 2019-01-27 15:42:59 UTC --- This error is still reproduceable in 5.3 when upgrading from a 3.x volume. I had to recreate volume from scratch in 5.3 and copy data back in order to avoid it. --- Additional comment from Artem Russakovskii on 2019-01-30 20:23:44 UTC --- I just started seeing this error after upgrading from 4.1 to 5.3. [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] And it floods like crazy with these. --- Additional comment from Emerson Gomes on 2019-01-30 20:33:12 UTC --- I "solved" the issue after upgrading to 5.3 by creating a new volume and moving all data to it. Apparently something is missing on the volumes after upgrade. --- Additional comment from Artem Russakovskii on 2019-01-30 20:37:13 UTC --- I just sent a message to the gluster mailing list about this because that's not how this problem should be resolved. I'm curious to hear what they say. --- Additional comment from Emerson Gomes on 2019-01-30 20:39:04 UTC --- Absolutely. That's the second big issue I had after upgrading. The first one is https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Still unsolved (open for more than 2 months now) --- Additional comment from Artem Russakovskii on 2019-01-30 20:40:29 UTC --- You know, I was *just* going to comment in a follow-up reply about whether the issue here is possibly related to the one you just linked. Seeing tons of those too, though at least the dupes are suppressed. ==> mnt-SITE_data1.log <== [2019-01-30 20:38:20.783713] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] ==> mnt-SITE_data3.log <== The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 413 times between [2019-01-30 20:36:23.881090] and [2019-01-30 20:38:20.015593] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0" repeated 42 times between [2019-01-30 20:36:23.290287] and [2019-01-30 20:38:20.280306] ==> mnt-SITE_data1.log <== The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0" repeated 50 times between [2019-01-30 20:36:22.247367] and [2019-01-30 20:38:19.459789] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 2654 times between [2019-01-30 20:36:22.667327] and [2019-01-30 20:38:20.546355] [2019-01-30 20:38:21.492319] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0 ==> mnt-SITE_data3.log <== [2019-01-30 20:38:22.349689] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0 ==> mnt-SITE_data1.log <== [2019-01-30 20:38:22.762941] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler --- Additional comment from Emerson Gomes on 2019-01-30 20:48:52 UTC --- Yeah, both arrised after upgrading from 3.x to 5.1, persisting in 5.2 and 5.3. The other issue is even more critical. It causes crashes, making the mount point being inacessible ("Transport endpoint is not connected" error) - Requiring a new manual umount/mount. For now I have a crontab entry doing this, but I will have to downgrade if a fix is not issued soon... --- Additional comment from Artem Russakovskii on 2019-01-31 18:00:40 UTC --- Damn, you weren't kidding, I wish I saw these bug reports before I updated from rock solid 4.1. Less than 24 hours after upgrading, I already got a crash that you referenced: [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3" repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31 09:38:03.958061] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 72 times between [2019-01-31 09:37:53.746741] and [2019-01-31 09:38:04.696993] pending frames: frame : type(1) op(READ) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-31 09:38:04 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6] /lib64/libc.so.6(+0x36160)[0x7fccd622d160] /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0] /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa] /lib64/libc.so.6(+0x2e772)[0x7fccd6225772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3] /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559] /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f] --------- Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 [Bug 1313567] flooding of "dict is NULL" logging -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 03:17:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:17:34 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1671603 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 [Bug 1671603] flooding of "dict is NULL" logging -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 03:18:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:18:55 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |atumball at redhat.com Blocks| |1671603 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 [Bug 1671603] flooding of "dict is NULL" logging -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 03:18:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:18:55 +0000 Subject: [Bugs] [Bug 1671603] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1667103 (glusterfs-5.4) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 03:19:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:19:19 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|1671603 | Depends On| |1671603 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 [Bug 1671603] flooding of "dict is NULL" logging -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 03:19:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:19:19 +0000 Subject: [Bugs] [Bug 1671603] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1667103 (glusterfs-5.4) Depends On|1667103 (glusterfs-5.4) | Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 03:21:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:21:53 +0000 Subject: [Bugs] [Bug 1671603] flooding of "dict is NULL" logging & crash of client process In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|flooding of "dict is NULL" |flooding of "dict is NULL" |logging |logging & crash of client | |process -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 03:29:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:29:51 +0000 Subject: [Bugs] [Bug 1671213] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671213 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-01 03:29:51 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22128 (core: move \"dict is NULL\" logs to DEBUG log level) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 03:29:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 03:29:58 +0000 Subject: [Bugs] [Bug 1671217] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671217 Bug 1671217 depends on bug 1671213, which changed state. Bug 1671213 Summary: core: move "dict is NULL" logs to DEBUG log level https://bugzilla.redhat.com/show_bug.cgi?id=1671213 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 04:44:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 04:44:21 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Version|4.1 |mainline -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 05:07:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:07:31 +0000 Subject: [Bugs] [Bug 1671611] New: Unable to delete directories that contain linkto files that point to itself. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671611 Bug ID: 1671611 Summary: Unable to delete directories that contain linkto files that point to itself. Product: GlusterFS Version: 5 Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org Depends On: 1667804 Blocks: 1667556, 1668989 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1667804 +++ Description of problem: A directory containing linkto files that point to itself cannot be deleted. Version-Release number of selected component (if applicable): How reproducible: Consistently Steps to Reproduce: 1. gluster v create tvol 192.168.122.7:/bricks/brick2/tvol-{1..2} 2. gluster v start tvol 3. mount -t glusterfs -s 192.168.122.7:/tvol /mnt/g1 4. cd /mnt/g1 5. mkdir -p dir0/dir1/dir2 6. cd dir0/dir1/dir2 7. for i in {1..100}; do echo "Test file" > tfile-$i; done 8. for i in {1..100}; do mv tfile-$i ntfile-$i; done 9. gluster v remove-brick tvol 192.168.122.7:/bricks/brick2/tvol-2 start Once the remove-brick status shows "completed", 10. gluster v remove-brick tvol 192.168.122.7:/bricks/brick2/tvol-2 stop You should now have only linkto files in 192.168.122.7:/bricks/brick2/tvol-2/dir0/dir1/dir2 and they should all be pointing to tvol-client-0. Manually change the linkto xattr value for every file in brick2 to point to itself, in this case "tvol-client-1"(make sure the string is null terminated). 11. setfattr -n trusted.glusterfs.dht.linkto -v 0x74766f6c2d636c69656e742d3100 /bricks/brick2/tvol-2/dir0/dir1/dir2/ntfile-* 12. Try to delete the directory from the mount point: [root at myserver g1]# rm -rf * Actual results: [root at myserver g1]# rm -rf * rm: cannot remove ?dir0/dir1/dir2?: Directory not empty Expected results: The directory should be deleted as there are no data files inside. Additional info: --- Additional comment from Worker Ant on 2019-01-21 09:50:09 UTC --- REVIEW: https://review.gluster.org/22066 (cluster/dht: Delete invalid linkto files in rmdir) posted (#1) for review on master by N Balachandran --- Additional comment from Worker Ant on 2019-01-22 05:23:04 UTC --- REVIEW: https://review.gluster.org/22066 (cluster/dht: Delete invalid linkto files in rmdir) merged (#2) on master by Amar Tumballi Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. https://bugzilla.redhat.com/show_bug.cgi?id=1668989 [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 05:07:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:07:31 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1671611 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671611 [Bug 1671611] Unable to delete directories that contain linkto files that point to itself. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 05:07:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:07:31 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1671611 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671611 [Bug 1671611] Unable to delete directories that contain linkto files that point to itself. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 05:12:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:12:41 +0000 Subject: [Bugs] [Bug 1671611] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671611 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |nbalacha at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 05:14:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:14:00 +0000 Subject: [Bugs] [Bug 1671611] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671611 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22136 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 05:14:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:14:01 +0000 Subject: [Bugs] [Bug 1671611] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671611 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22136 (cluster/dht: Delete invalid linkto files in rmdir) posted (#1) for review on release-5 by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 05:30:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:30:22 +0000 Subject: [Bugs] [Bug 1669937] Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669937 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Version|4.1 |mainline -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 05:30:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:30:39 +0000 Subject: [Bugs] [Bug 1669937] Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669937 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-01 05:30:39 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22103 (cluster/dht: Remove internal permission bits) merged (#2) on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 05:45:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:45:32 +0000 Subject: [Bugs] [Bug 1662264] thin-arbiter: Check with thin-arbiter file before marking new entry change log In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662264 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21933 (cluster/thin-arbiter: Consider thin-arbiter before marking new entry changelog) merged (#6) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 05:51:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 05:51:19 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #3 from Ravishankar N --- (In reply to David E. Smith from comment #2) > Actually, I ran the cores through strings and grepped for a few things like > passwords -- as you'd expect from a memory dump from a Web server, there's a > log of sensitive information in there. Is there a safe/acceptable way to > send the cores only to developers that can use them, or otherwise not have > to make them publicly available while still letting the Gluster devs benefit > from analyzing them? Perhaps you could upload it to a shared Dropbox folder with view/download access to the red hat email IDs I've CC'ed to this email (including me) to begin with. Note: I upgraded a 1x2 replica volume with 1 fuse client from v4.1.7 to v5.3 and did some basic I/O (kernel untar and iozone) and did not observe any crashes, so maybe this this something that is hit under extreme I/O or memory pressure. :-( -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 06:58:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 06:58:39 +0000 Subject: [Bugs] [Bug 1671637] New: geo-rep: Issue with configparser import Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671637 Bug ID: 1671637 Summary: geo-rep: Issue with configparser import Product: GlusterFS Version: mainline Status: NEW Component: geo-replication Assignee: bugs at gluster.org Reporter: khiremat at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: 'configparser' is backported to python2 and can be installed using pip (pip install configparser). So trying to import 'configparser' first and later 'ConfigParser' can cause issues w.r.t unicode strings. Solution: Always try importing 'ConfigParser' first and then 'configparser'. This solves python2/python3 compat issues. Version-Release number of selected component (if applicable): mainilne -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 06:58:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 06:58:50 +0000 Subject: [Bugs] [Bug 1671637] geo-rep: Issue with configparser import In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671637 Kotresh HR changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |khiremat at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 07:00:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 07:00:46 +0000 Subject: [Bugs] [Bug 1671637] geo-rep: Issue with configparser import In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671637 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22138 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 07:00:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 07:00:47 +0000 Subject: [Bugs] [Bug 1671637] geo-rep: Issue with configparser import In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671637 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22138 (geo-rep: Fix configparser import issue) posted (#1) for review on master by Kotresh HR -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 07:22:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 07:22:33 +0000 Subject: [Bugs] [Bug 1671647] New: Anomalies in python-lint build job Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671647 Bug ID: 1671647 Summary: Anomalies in python-lint build job Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: spamecha at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 08:06:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 08:06:11 +0000 Subject: [Bugs] [Bug 1671647] Anomalies in python-lint build job In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671647 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Comment #0 is|1 |0 private| | CC| |nigelb at redhat.com --- Comment #1 from Nigel Babu --- Can you also paste in a link of where this is happening? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 1 08:32:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 08:32:55 +0000 Subject: [Bugs] [Bug 1665145] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665145 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22139 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 08:32:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 08:32:56 +0000 Subject: [Bugs] [Bug 1665145] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665145 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22139 (readdir-ahead: do not zero-out iatt in fop cbk) posted (#1) for review on release-5 by Ravishankar N -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 09:54:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 09:54:35 +0000 Subject: [Bugs] [Bug 1626085] "glusterfs --process-name fuse" crashes and leads to "Transport endpoint is not connected" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1626085 --- Comment #9 from GCth --- Is there anything else I can do to help fixing this issue? We had to implement monitoring and restarting solution for our glusterfs clusters as they crash frequently, causing open files to be unavailable and dependent applications to stop working correctly. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 10:13:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 10:13:05 +0000 Subject: [Bugs] [Bug 1626085] "glusterfs --process-name fuse" crashes and leads to "Transport endpoint is not connected" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1626085 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(rhb1 at gcth.net) --- Comment #10 from Ravishankar N --- (In reply to GCth from comment #9) > Is there anything else I can do to help fixing this issue? > We had to implement monitoring and restarting solution for our glusterfs > clusters as they crash frequently, causing open files to be unavailable > and dependent applications to stop working correctly. Are all crashes in AFR with the same back trace as in comment #7? What workload are you running on your 4.1 gluster volume? It would be great if you can give a consistent reproducer which we can try on our setup. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 10:23:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 10:23:14 +0000 Subject: [Bugs] [Bug 1665216] Databases crashes on Gluster 5 with the option performance.write-behind enabled In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665216 mhutter changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bugzilla.redhat.com at mhutter | |.net --- Comment #5 from mhutter --- Reproduction case: Exactly as described in the original Ticket. # Prepare gluster volume gluster volume set gluster-pv18 performance.write-behind off # mount the volume mount -t glusterfs :/gluster-pv18 /mnt/gluster-pv18 # start Postgres docker run --name psql-test --rm -v /mnt/gluster-pv18:/var/lib/postgresql/data docker.io/postgres:9.5 # this should work as expected # clean up docker stop psql-test rm -rf /mnt/gluster-pv18/* umount /mnt/gluster-pv18 # enable write-behind gluster volume set gluster-pv18 performance.write-behind on # mount the volume mount -t glusterfs :/gluster-pv18 /mnt/gluster-pv18 # start Postgres docker run --name psql-test --rm -v /mnt/gluster-pv18:/var/lib/postgresql/data docker.io/postgres:9.5 # !!! this will now fail: # creating template1 database in /var/lib/postgresql/data/base/1 ... ok # initializing pg_authid ... LOG: invalid primary checkpoint record # LOG: invalid secondary checkpoint record # PANIC: could not locate a valid checkpoint record # Aborted (core dumped) # child process exited with exit code 134 # initdb: removing contents of data directory "/var/lib/postgresql/data" -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 10:41:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 10:41:10 +0000 Subject: [Bugs] [Bug 1665216] Databases crashes on Gluster 5 with the option performance.write-behind enabled In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665216 mhutter changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(gabisoft at freesurf | |.ch) | --- Comment #6 from mhutter --- Created attachment 1525793 --> https://bugzilla.redhat.com/attachment.cgi?id=1525793&action=edit dump-fuse, gzipped --- Comment #7 from mhutter --- Created attachment 1525794 --> https://bugzilla.redhat.com/attachment.cgi?id=1525794&action=edit strace of initdb (which crashed) Also interesting: while creating the TGZ archive (not on the gluster volume) of all strace files (which were on the gluster volume), a lot of messages like this appeared: tar: strace/initdb.42: file changed as we read it -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 1 13:35:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 01 Feb 2019 13:35:31 +0000 Subject: [Bugs] [Bug 1671733] New: clang-format test is checking contrib files, but rfc.sh skips them Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671733 Bug ID: 1671733 Summary: clang-format test is checking contrib files, but rfc.sh skips them Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: jahernan at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: The clang-format job is testing files inside 'contrib' directory. I think they shouldn't be checked, like rfc.sh already does. Example: https://build.gluster.org/job/clang-format/2868/console Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Execute the job on this patch: https://review.gluster.org/c/glusterfs/+/20636 2. 3. Actual results: The test fails Expected results: The test shouldn't fail because of invalid formatting on files inside 'contrib' directory. Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 2 03:07:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 03:07:52 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #529 from Worker Ant --- REVIEW: https://review.gluster.org/22094 (core: make gf_thread_create() easier to use) merged (#5) on master by Xavi Hernandez -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 2 03:08:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 03:08:22 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #10 from Worker Ant --- REVIEW: https://review.gluster.org/22109 (mount/fuse: expose auto-invalidation as a mount option) merged (#13) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 2 03:09:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 03:09:24 +0000 Subject: [Bugs] [Bug 1658116] python2 to python3 compatibilty issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1658116 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-02 03:09:24 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21845 (glusterfind: python2 to python3 compat) merged (#7) on master by Amar Tumballi -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 2 03:10:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 03:10:13 +0000 Subject: [Bugs] [Bug 1670259] New GFID file recreated in a replica set after a GFID mismatch resolution In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670259 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22112 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 2 03:10:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 03:10:15 +0000 Subject: [Bugs] [Bug 1670259] New GFID file recreated in a replica set after a GFID mismatch resolution In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670259 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-02 03:10:15 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22112 (cluster/dht: Do not use gfid-req in fresh lookup) merged (#7) on master by Amar Tumballi -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 2 03:11:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 03:11:43 +0000 Subject: [Bugs] [Bug 1671647] Anomalies in python-lint build job In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671647 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |high CC| |atumball at redhat.com Severity|unspecified |high --- Comment #2 from Amar Tumballi --- https://build.gluster.org/job/python-lint/ All the latest builds are passing, but if you go inside Console, and watch, there are some exceptions thrown. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 2 20:15:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 20:15:09 +0000 Subject: [Bugs] [Bug 1671603] flooding of "dict is NULL" logging & crash of client process In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 --- Comment #1 from Artem Russakovskii --- The fuse crash happened again yesterday, to another volume. Are there any mount options that could help mitigate this? In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch and restart the mount, which works and recovers the mount point within a minute. Not ideal, but a temporary workaround. By the way, the way to reproduce this "Transport endpoint is not connected" condition for testing purposes is to kill -9 the right "glusterfs --process-name fuse" process. monit check: check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 start program = "/bin/mount /mnt/glusterfs_data1" stop program = "/bin/umount /mnt/glusterfs_data1" if space usage > 90% for 5 times within 15 cycles then alert else if succeeded for 10 cycles then alert stack trace: [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 26 times between [2019-02-01 23:21:20.857333] and [2019-02-01 23:21:56.164427] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-01 23:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 2 20:16:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 20:16:21 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #17 from Artem Russakovskii --- The fuse crash happened again yesterday, to another volume. Are there any mount options that could help mitigate this? In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch and restart the mount, which works and recovers the mount point within a minute. Not ideal, but a temporary workaround. By the way, the way to reproduce this "Transport endpoint is not connected" condition for testing purposes is to kill -9 the right "glusterfs --process-name fuse" process. monit check: check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 start program = "/bin/mount /mnt/glusterfs_data1" stop program = "/bin/umount /mnt/glusterfs_data1" if space usage > 90% for 5 times within 15 cycles then alert else if succeeded for 10 cycles then alert stack trace: [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 26 times between [2019-02-01 23:21:20.857333] and [2019-02-01 23:21:56.164427] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-01 23:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 2 20:16:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 20:16:52 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #29 from Artem Russakovskii --- The fuse crash happened again yesterday, to another volume. Are there any mount options that could help mitigate this? In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch and restart the mount, which works and recovers the mount point within a minute. Not ideal, but a temporary workaround. By the way, the way to reproduce this "Transport endpoint is not connected" condition for testing purposes is to kill -9 the right "glusterfs --process-name fuse" process. monit check: check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 start program = "/bin/mount /mnt/glusterfs_data1" stop program = "/bin/umount /mnt/glusterfs_data1" if space usage > 90% for 5 times within 15 cycles then alert else if succeeded for 10 cycles then alert stack trace: [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 26 times between [2019-02-01 23:21:20.857333] and [2019-02-01 23:21:56.164427] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-01 23:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 2 20:17:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 02 Feb 2019 20:17:15 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #4 from Artem Russakovskii --- The fuse crash happened again yesterday, to another volume. Are there any mount options that could help mitigate this? In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch and restart the mount, which works and recovers the mount point within a minute. Not ideal, but a temporary workaround. By the way, the way to reproduce this "Transport endpoint is not connected" condition for testing purposes is to kill -9 the right "glusterfs --process-name fuse" process. monit check: check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 start program = "/bin/mount /mnt/glusterfs_data1" stop program = "/bin/umount /mnt/glusterfs_data1" if space usage > 90% for 5 times within 15 cycles then alert else if succeeded for 10 cycles then alert stack trace: [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 26 times between [2019-02-01 23:21:20.857333] and [2019-02-01 23:21:56.164427] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-01 23:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Feb 3 03:07:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 03 Feb 2019 03:07:11 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #30 from Milind Changire --- the following line the backtrace which is the topmost line pointing to gluster bits: /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] resolves to: afr-common.c:2203 intersection = alloca0(priv->child_count); ----- NOTE: print-backtrace.sh isn't helping here because the naming convention of rpms have changed -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sun Feb 3 05:57:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 03 Feb 2019 05:57:12 +0000 Subject: [Bugs] [Bug 1671733] clang-format test is checking contrib files, but rfc.sh skips them In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671733 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED CC| |nigelb at redhat.com Assignee|bugs at gluster.org |nigelb at redhat.com --- Comment #1 from Nigel Babu --- Fixing. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Feb 3 11:22:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 03 Feb 2019 11:22:51 +0000 Subject: [Bugs] [Bug 1671733] clang-format test is checking contrib files, but rfc.sh skips them In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671733 --- Comment #2 from Nigel Babu --- Pushed https://review.gluster.org/#/c/build-jobs/+/22143 for review. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sun Feb 3 15:12:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 03 Feb 2019 15:12:43 +0000 Subject: [Bugs] [Bug 1672076] New: chrome / chromium crash on gluster, sqlite issue? Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672076 Bug ID: 1672076 Summary: chrome / chromium crash on gluster, sqlite issue? Product: GlusterFS Version: 5 Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: mjc at avtechpulse.com CC: bugs at gluster.org Target Milestone: --- Classification: Community I run Fedora 29 clients and servers, with user home folders mounted on gluster. This worked fine with Fedora 27 clients, but on F29 clients the chrome and chromium browsers crash. The backtrace info (see below) suggests problems with sqlite. Firefox runs just fine, even though it is an sqlite user too. chromium clients mounted on local drives work fine. - Mike clients: glusterfs-5.3-1.fc29.x86_64, chromium-71.0.3578.98-1.fc29.x86_64 server: glusterfs-server-5.3-1.fc29.x86_64 [root at gluster1 ~]# gluster volume info Volume Name: volume1 Type: Replicate Volume ID: 91ef5aed-94be-44ff-a19d-c41682808159 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gluster1:/gluster/brick1/data Brick2: gluster2:/gluster/brick2/data Options Reconfigured: nfs.disable: on server.allow-insecure: on cluster.favorite-child-policy: mtime [mjc at daisy ~]$ chromium-browser [18826:18826:0130/094436.431828:ERROR:sandbox_linux.cc(364)] InitializeSandbox() called with multiple threads in process gpu-process. [18785:18785:0130/094440.905900:ERROR:x11_input_method_context_impl_gtk.cc(144)] Not implemented reached in virtual void libgtkui::X11InputMethodContextImplGtk::SetSurroundingText(const string16&, const gfx::Range&) Received signal 7 BUS_ADRERR 7fc30e9bd000 #0 0x7fc34b008261 base::debug::StackTrace::StackTrace() #1 0x7fc34b00869b base::debug::(anonymous namespace)::StackDumpSignalHandler() #2 0x7fc34b008cb7 base::debug::(anonymous namespace)::StackDumpSignalHandler() #3 0x7fc3401fe030 #4 0x7fc33f5820f0 __memmove_avx_unaligned_erms #5 0x7fc346099491 unixRead #6 0x7fc3460d2784 readDbPage #7 0x7fc3460d5e4f getPageNormal #8 0x7fc3460d5f01 getPageMMap #9 0x7fc3460958f5 btreeGetPage #10 0x7fc3460ec47b sqlite3BtreeBeginTrans #11 0x7fc3460fd1e8 sqlite3VdbeExec #12 0x7fc3461056af chrome_sqlite3_step #13 0x7fc3464071c7 sql::Statement::StepInternal() #14 0x7fc3464072de sql::Statement::Step() #15 0x555fd21699d7 autofill::AutofillTable::GetAutofillProfiles() #16 0x555fd2160808 autofill::AutofillProfileSyncableService::MergeDataAndStartSyncing() #17 0x555fd1d25207 syncer::SharedChangeProcessor::StartAssociation() #18 0x555fd1d09652 _ZN4base8internal7InvokerINS0_9BindStateIMN6syncer21SharedChangeProcessorEFvNS_17RepeatingCallbackIFvNS3_18DataTypeController15ConfigureResultERKNS3_15SyncMergeResultESA_EEEPNS3_10SyncClientEPNS3_29GenericChangeProcessorFactoryEPNS3_9UserShareESt10unique_ptrINS3_20DataTypeErrorHandlerESt14default_deleteISK_EEEJ13scoped_refptrIS4_ESC_SE_SG_SI_NS0_13PassedWrapperISN_EEEEEFvvEE3RunEPNS0_13BindStateBaseE #19 0x7fc34af4309d base::debug::TaskAnnotator::RunTask() #20 0x7fc34afcda86 base::internal::TaskTracker::RunOrSkipTask() #21 0x7fc34b01b6a2 base::internal::TaskTrackerPosix::RunOrSkipTask() #22 0x7fc34afd07d6 base::internal::TaskTracker::RunAndPopNextTask() #23 0x7fc34afca5e7 base::internal::SchedulerWorker::RunWorker() #24 0x7fc34afcac84 base::internal::SchedulerWorker::RunSharedWorker() #25 0x7fc34b01aa09 base::(anonymous namespace)::ThreadFunc() #26 0x7fc3401f358e start_thread #27 0x7fc33f51d6a3 __GI___clone r8: 00000cbfd93d4a00 r9: 00000000cbfd93d4 r10: 000000000000011c r11: 0000000000000000 r12: 00000cbfd940eb00 r13: 0000000000000000 r14: 0000000000000000 r15: 00000cbfd9336c00 di: 00000cbfd93d4a00 si: 00007fc30e9bd000 bp: 00007fc30faff7e0 bx: 0000000000000800 dx: 0000000000000800 ax: 00000cbfd93d4a00 cx: 0000000000000800 sp: 00007fc30faff788 ip: 00007fc33f5820f0 efl: 0000000000010287 cgf: 002b000000000033 erf: 0000000000000004 trp: 000000000000000e msk: 0000000000000000 cr2: 00007fc30e9bd000 [end of stack trace] Calling _exit(1). Core file will not be generated. And a client mount log is below - although the log is full megabytes of: The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 20178 times between [2019-01-31 13:44:14.962950] and [2019-01-31 13:46:00.013310] and [2019-01-31 13:46:07.470163] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] so I've just shown the start of the log. I guess that's related to https://bugzilla.redhat.com/show_bug.cgi?id=1651246. - Mike Mount log: [2019-01-31 13:44:00.775353] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.3 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=gluster1 --volfile-server=gluster2 --volfile-id=/volume1 /fileserver2) [2019-01-31 13:44:00.817140] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-31 13:44:00.926491] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2019-01-31 13:44:00.928102] I [MSGID: 114020] [client.c:2354:notify] 0-volume1-client-0: parent translators are ready, attempting connect on transport [2019-01-31 13:44:00.931063] I [MSGID: 114020] [client.c:2354:notify] 0-volume1-client-1: parent translators are ready, attempting connect on transport [2019-01-31 13:44:00.932144] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-volume1-client-0: changing port to 49152 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume volume1-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host gluster1 5: option remote-subvolume /gluster/brick1/data 6: option transport-type socket 7: option transport.tcp-user-timeout 0 8: option transport.socket.keepalive-time 20 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-count 9 11: option send-gids true 12: end-volume 13: 14: volume volume1-client-1 15: type protocol/client 16: option ping-timeout 42 17: option remote-host gluster2 18: option remote-subvolume /gluster/brick2/data 19: option transport-type socket 20: option transport.tcp-user-timeout 0 21: option transport.socket.keepalive-time 20 22: option transport.socket.keepalive-interval 2 23: option transport.socket.keepalive-count 9 24: option send-gids true 25: end-volume 26: 27: volume volume1-replicate-0 28: type cluster/replicate 29: option afr-pending-xattr volume1-client-0,volume1-client-1 30: option favorite-child-policy mtime 31: option use-compound-fops off 32: subvolumes volume1-client-0 volume1-client-1 33: end-volume 34: 35: volume volume1-dht 36: type cluster/distribute [2019-01-31 13:44:00.932495] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler 37: option lock-migration off 38: option force-migration off 39: subvolumes volume1-replicate-0 40: end-volume 41: 42: volume volume1-write-behind 43: type performance/write-behind 44: subvolumes volume1-dht 45: end-volume 46: 47: volume volume1-read-ahead 48: type performance/read-ahead 49: subvolumes volume1-write-behind 50: end-volume 51: 52: volume volume1-readdir-ahead 53: type performance/readdir-ahead 54: option parallel-readdir off 55: option rda-request-size 131072 56: option rda-cache-limit 10MB 57: subvolumes volume1-read-ahead 58: end-volume 59: 60: volume volume1-io-cache 61: type performance/io-cache 62: subvolumes volume1-readdir-ahead 63: end-volume 64: 65: volume volume1-quick-read 66: type performance/quick-read 67: subvolumes volume1-io-cache 68: end-volume 69: 70: volume volume1-open-behind 71: type performance/open-behind 72: subvolumes volume1-quick-read 73: end-volume 74: 75: volume volume1-md-cache 76: type performance/md-cache 77: subvolumes volume1-open-behind 78: end-volume 79: 80: volume volume1 81: type debug/io-stats 82: option log-level INFO 83: option latency-measurement off 84: option count-fop-hits off 85: subvolumes volume1-md-cache 86: end-volume 87: 88: volume meta-autoload 89: type meta 90: subvolumes volume1 91: end-volume 92: +------------------------------------------------------------------------------+ [2019-01-31 13:44:00.933375] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-volume1-client-1: changing port to 49152 (from 0) [2019-01-31 13:44:00.933549] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-31 13:44:00.934170] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-volume1-client-0: Connected to volume1-client-0, attached to remote volume '/gluster/brick1/data'. [2019-01-31 13:44:00.934210] I [MSGID: 108005] [afr-common.c:5237:__afr_handle_child_up_event] 0-volume1-replicate-0: Subvolume 'volume1-client-0' came back up; going online. [2019-01-31 13:44:00.935291] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-volume1-client-1: Connected to volume1-client-1, attached to remote volume '/gluster/brick2/data'. [2019-01-31 13:44:00.937661] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.28 [2019-01-31 13:44:00.937691] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0 [2019-01-31 13:44:14.852144] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:14.962950] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-31 13:44:15.038615] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.040956] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.041044] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.041467] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.471018] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.477003] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.482380] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.487047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.603624] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.607726] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] [2019-01-31 13:44:15.607906] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) [0x7fb0e0b49c45] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) [0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) [0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument] -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 02:11:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 02:11:48 +0000 Subject: [Bugs] [Bug 1449773] Finish the installation and freebsd10.3.rht and clean password in jenkins In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1449773 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |CURRENTRELEASE Last Closed|2018-08-29 03:53:37 |2019-02-04 02:11:48 --- Comment #3 from Nigel Babu --- This is now fixed. We build on the internal freebsd builder now. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 02:12:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 02:12:40 +0000 Subject: [Bugs] [Bug 1498151] Move download server to the community cage In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1498151 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-04 02:12:40 --- Comment #11 from Nigel Babu --- This is now complete. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 02:59:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 02:59:25 +0000 Subject: [Bugs] [Bug 1564451] The abandon job for patches should post info in bugzilla that some patch is abandon'd. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1564451 --- Comment #2 from Nigel Babu --- The code for this is written in the bugzilla script, but this needs a Jenkins job to actually call the script. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 03:47:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 03:47:49 +0000 Subject: [Bugs] [Bug 1635688] Keep only the valid (maintained/supported) components in the build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1635688 --- Comment #18 from Worker Ant --- REVIEW: https://review.gluster.org/21877 (glusterd: manage upgrade to current master) merged (#3) on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 03:15:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 03:15:43 +0000 Subject: [Bugs] [Bug 1659394] Maintainer permissions on gluster-mixins project for Ankush In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659394 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED CC| |nigelb at redhat.com Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-04 03:15:43 --- Comment #1 from Nigel Babu --- This has been done for some time. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 04:50:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 04:50:56 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|VERIFIED |RELEASE_PENDING -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 05:14:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 05:14:44 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22120 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 05:14:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 05:14:45 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22120 (inode: Reduce work load of inode_table->lock section) posted (#6) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 05:41:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 05:41:45 +0000 Subject: [Bugs] [Bug 1672155] New: looks like devrpm-fedora jobs are failing due to lack of storage Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672155 Bug ID: 1672155 Summary: looks like devrpm-fedora jobs are failing due to lack of storage Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Severity: urgent Priority: high Assignee: bugs at gluster.org Reporter: atumball at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: https://build.gluster.org/job/devrpm-fedora/ the recent failures seems to be for lack of memory in the system. For example check the failure of https://build.gluster.org/job/devrpm-fedora/14647/console Version-Release number of selected component (if applicable): master How reproducible: 100% Steps to Reproduce: 1. submit a patch to glusterfs. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 05:52:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 05:52:45 +0000 Subject: [Bugs] [Bug 1362129] rename of a file can cause data loss in an replica/arbiter volume configuration In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1362129 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|VERIFIED |RELEASE_PENDING -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 05:53:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 05:53:05 +0000 Subject: [Bugs] [Bug 1654103] Invalid memory read after freed in dht_rmdir_readdirp_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654103 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|VERIFIED |RELEASE_PENDING -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 05:53:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 05:53:07 +0000 Subject: [Bugs] [Bug 1655578] Incorrect usage of local->fd in afr_open_ftruncate_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655578 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|VERIFIED |RELEASE_PENDING -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 05:53:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 05:53:09 +0000 Subject: [Bugs] [Bug 1659439] Memory leak: dict_t leak in rda_opendir In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659439 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|VERIFIED |RELEASE_PENDING -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 05:53:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 05:53:10 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|VERIFIED |RELEASE_PENDING -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 06:27:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 06:27:53 +0000 Subject: [Bugs] [Bug 1670382] parallel-readdir prevents directories and files listing In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670382 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |locbus at gmail.com, | |nbalacha at redhat.com Flags| |needinfo?(locbus at gmail.com) --- Comment #2 from Nithya Balachandran --- Can you clarify that you are doing the following: 1. The files/directories are being created from one gluster client (not directly on the bricks) 2. The files/directories cannot be listed from another client which has mounted the same volume 3. Are the files/directories visible on the client from which they were created? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:21:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:21:30 +0000 Subject: [Bugs] [Bug 1672155] looks like devrpm-fedora jobs are failing due to lack of storage In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672155 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED CC| |nigelb at redhat.com Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-04 07:21:30 --- Comment #1 from Nigel Babu --- This is now fixed. The /home/jenkins/.local folder was consuming a bunch of space as was the mock cache. I've cleared them both out now. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:36:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:36:24 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA Last Closed|2018-10-31 03:11:36 |2019-02-04 07:36:24 --- Comment #45 from errata-xmlrpc --- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0261 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:36:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:36:28 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Red Hat Product Errata | |RHBA-2019:0261 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:25 +0000 Subject: [Bugs] [Bug 1362129] rename of a file can cause data loss in an replica/arbiter volume configuration In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1362129 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA Last Closed| |2019-02-04 07:41:25 --- Comment #27 from errata-xmlrpc --- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:31 +0000 Subject: [Bugs] [Bug 1646892] Portmap entries showing stale brick entries when bricks are down In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1646892 Bug 1646892 depends on bug 1637379, which changed state. Bug 1637379 Summary: Portmap entries showing stale brick entries when bricks are down https://bugzilla.redhat.com/show_bug.cgi?id=1637379 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:33 +0000 Subject: [Bugs] [Bug 1642448] EC volume getting created without any redundant brick In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1642448 Bug 1642448 depends on bug 1597252, which changed state. Bug 1597252 Summary: EC volume getting created without any redundant brick https://bugzilla.redhat.com/show_bug.cgi?id=1597252 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:34 +0000 Subject: [Bugs] [Bug 1654181] glusterd segmentation fault: glusterd_op_ac_brick_op_failed (event=0x7f44e0e63f40, ctx=0x0) at glusterd-op-sm.c:5606 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654181 Bug 1654181 depends on bug 1639476, which changed state. Bug 1639476 Summary: glusterd segmentation fault: glusterd_op_ac_brick_op_failed (event=0x7f44e0e63f40, ctx=0x0) at glusterd-op-sm.c:5606 https://bugzilla.redhat.com/show_bug.cgi?id=1639476 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:35 +0000 Subject: [Bugs] [Bug 1630922] glusterd crashed and core generated at gd_mgmt_v3_unlock_timer_cbk after huge number of volumes were created In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1630922 Bug 1630922 depends on bug 1599220, which changed state. Bug 1599220 Summary: glusterd crashed and core generated at gd_mgmt_v3_unlock_timer_cbk after huge number of volumes were created https://bugzilla.redhat.com/show_bug.cgi?id=1599220 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:35 +0000 Subject: [Bugs] [Bug 1655827] [Glusterd]: Glusterd crash while expanding volumes using heketi In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655827 Bug 1655827 depends on bug 1652466, which changed state. Bug 1652466 Summary: [Glusterd]: Glusterd crash while expanding volumes using heketi https://bugzilla.redhat.com/show_bug.cgi?id=1652466 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:35 +0000 Subject: [Bugs] [Bug 1647074] when peer detach is issued, throw a warning to remount volumes using other cluster IPs before proceeding In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1647074 Bug 1647074 depends on bug 1639568, which changed state. Bug 1639568 Summary: when peer detach is issued, throw a warning to remount volumes using other cluster IPs before proceeding https://bugzilla.redhat.com/show_bug.cgi?id=1639568 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:36 +0000 Subject: [Bugs] [Bug 1615385] glusterd segfault - memcpy () at /usr/include/bits/string3.h:51 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1615385 Bug 1615385 depends on bug 1608507, which changed state. Bug 1608507 Summary: glusterd segfault - memcpy () at /usr/include/bits/string3.h:51 https://bugzilla.redhat.com/show_bug.cgi?id=1608507 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:36 +0000 Subject: [Bugs] [Bug 1654187] [geo-rep]: RFE - Make slave volume read-only while setting up geo-rep (by default) In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654187 Bug 1654187 depends on bug 1643370, which changed state. Bug 1643370 Summary: [geo-rep]: RFE - Make slave volume read-only while setting up geo-rep (by default) https://bugzilla.redhat.com/show_bug.cgi?id=1643370 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:38 +0000 Subject: [Bugs] [Bug 1362129] rename of a file can cause data loss in an replica/arbiter volume configuration In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1362129 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Red Hat Product Errata | |RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:44 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA Last Closed| |2019-02-04 07:41:44 --- Comment #10 from errata-xmlrpc --- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:46 +0000 Subject: [Bugs] [Bug 1665826] [geo-rep]: Directory renames not synced to slave in Hybrid Crawl In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665826 Bug 1665826 depends on bug 1664235, which changed state. Bug 1664235 Summary: [geo-rep]: Directory renames not synced to slave in Hybrid Crawl https://bugzilla.redhat.com/show_bug.cgi?id=1664235 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:46 +0000 Subject: [Bugs] [Bug 1654138] Optimize for virt store fails with distribute volume type In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654138 Bug 1654138 depends on bug 1653613, which changed state. Bug 1653613 Summary: [Dalton] Optimize for virt store fails with distribute volume type https://bugzilla.redhat.com/show_bug.cgi?id=1653613 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:44 +0000 Subject: [Bugs] [Bug 1654103] Invalid memory read after freed in dht_rmdir_readdirp_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654103 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA Last Closed| |2019-02-04 07:41:44 --- Comment #11 from errata-xmlrpc --- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:47 +0000 Subject: [Bugs] [Bug 1667779] glusterd leaks about 1GB memory per day on single machine of storage pool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667779 Bug 1667779 depends on bug 1667169, which changed state. Bug 1667169 Summary: glusterd leaks about 1GB memory per day on single machine of storage pool https://bugzilla.redhat.com/show_bug.cgi?id=1667169 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:48 +0000 Subject: [Bugs] [Bug 1654270] glusterd crashed with seg fault possibly during node reboot while volume creates and deletes were happening In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654270 Bug 1654270 depends on bug 1654161, which changed state. Bug 1654161 Summary: glusterd crashed with seg fault possibly during node reboot while volume creates and deletes were happening https://bugzilla.redhat.com/show_bug.cgi?id=1654161 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:49 +0000 Subject: [Bugs] [Bug 1669382] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669382 Bug 1669382 depends on bug 1668304, which changed state. Bug 1668304 Summary: [RHHI-V] Fuse mount crashed while creating the preallocated image https://bugzilla.redhat.com/show_bug.cgi?id=1668304 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:44 +0000 Subject: [Bugs] [Bug 1655578] Incorrect usage of local->fd in afr_open_ftruncate_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655578 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA Last Closed| |2019-02-04 07:41:44 --- Comment #17 from errata-xmlrpc --- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:49 +0000 Subject: [Bugs] [Bug 1669077] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669077 Bug 1669077 depends on bug 1668304, which changed state. Bug 1668304 Summary: [RHHI-V] Fuse mount crashed while creating the preallocated image https://bugzilla.redhat.com/show_bug.cgi?id=1668304 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:49 +0000 Subject: [Bugs] [Bug 1651322] Incorrect usage of local->fd in afr_open_ftruncate_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651322 Bug 1651322 depends on bug 1655578, which changed state. Bug 1655578 Summary: Incorrect usage of local->fd in afr_open_ftruncate_cbk https://bugzilla.redhat.com/show_bug.cgi?id=1655578 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:50 +0000 Subject: [Bugs] [Bug 1655527] Incorrect usage of local->fd in afr_open_ftruncate_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655527 Bug 1655527 depends on bug 1655578, which changed state. Bug 1655578 Summary: Incorrect usage of local->fd in afr_open_ftruncate_cbk https://bugzilla.redhat.com/show_bug.cgi?id=1655578 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:50 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Red Hat Product Errata | |RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:44 +0000 Subject: [Bugs] [Bug 1659439] Memory leak: dict_t leak in rda_opendir In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659439 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA Last Closed| |2019-02-04 07:41:44 --- Comment #13 from errata-xmlrpc --- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:53 +0000 Subject: [Bugs] [Bug 1662906] Longevity: glusterfsd(brick process) crashed when we do volume creates and deletes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662906 Bug 1662906 depends on bug 1662828, which changed state. Bug 1662828 Summary: Longevity: glusterfsd(brick process) crashed when we do volume creates and deletes https://bugzilla.redhat.com/show_bug.cgi?id=1662828 What |Removed |Added ---------------------------------------------------------------------------- Status|RELEASE_PENDING |CLOSED Resolution|--- |ERRATA -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:54 +0000 Subject: [Bugs] [Bug 1654103] Invalid memory read after freed in dht_rmdir_readdirp_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654103 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Red Hat Product Errata | |RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:54 +0000 Subject: [Bugs] [Bug 1655578] Incorrect usage of local->fd in afr_open_ftruncate_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655578 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Red Hat Product Errata | |RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 07:41:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 07:41:54 +0000 Subject: [Bugs] [Bug 1659439] Memory leak: dict_t leak in rda_opendir In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659439 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Red Hat Product Errata | |RHBA-2019:0263 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 08:48:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 08:48:15 +0000 Subject: [Bugs] [Bug 1670382] parallel-readdir prevents directories and files listing In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670382 Marcin changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(locbus at gmail.com) | --- Comment #3 from Marcin --- Hello Nithya, 1. Yes, the files/directory are being created from Windows2012R2 (samba client) 2. No, the files/directories cannot be listed by another client which has mounted the same volume. 3. No, the files/directories aren't visible on the client from that were created. In addition, I can confirm that they aren't visible, even directly on the brick of the host to which they write data... (the solution is, for example, restarting the host). -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 09:32:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 09:32:44 +0000 Subject: [Bugs] [Bug 1672205] New: [GSS] 'gluster get-state' command fails if volume brick doesn't exist. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672205 Bug ID: 1672205 Summary: [GSS] 'gluster get-state' command fails if volume brick doesn't exist. Product: GlusterFS Version: mainline Status: NEW Component: glusterd Keywords: Improvement Severity: medium Priority: medium Assignee: bugs at gluster.org Reporter: srakonde at redhat.com Depends On: 1669970 Target Milestone: --- Group: private Classification: Community Description of problem: 'gluster get-state' command fails when any brick of a volume is not present or deleted. Instead the command output should report the brick failure. When any brick of a volume is not available or being removed 'gluster get-state' command fails with the following error: 'Failed to get daemon state. Check glusterd log file for more details' The requirement is 'gluster get-state' command should not fail and generate gluster brick's state in the output. For example: cat /var/run/gluster/glusterd_state_XYZ ... Volume3.name: v02 Volume3.id: c194e70d-6738-4ba3-9502-ec5603aab679 Volume3.type: Distributed-Replicate ... ## HERE # Volume3.Brick1.port: N/A or 0 or empty? Volume3.Brick1.rdma_port: 0 Volume3.Brick1.port_registered: N/A or 0 or empty? Volume3.Brick1.status: Failed Volume3.Brick1.spacefree: N/A or 0 or empty? Volume3.Brick1.spacetotal: N/A or 0 or empty? ... This situation can happen in production when a local storage on node is 'broken' or while using heketi with gluster. Volumes are present but bricks are missing. How reproducible: Always Version-Release number of selected component (if applicable): RHGS 3.X Steps to Reproduce: 1. Delete a brick 2. Run command 'gluster get-state' Actual results: Command fails with the below message 'Failed to get daemon state. Check glusterd log file for more details' Expected results: 'gluster get-state'Command should not fail. It should report the faulty brick's state in the output so one can simply identify what is the problem with the volumne. 'gluster get-state' command should return a message regarding that 'faulty brick'. --- Additional comment from Atin Mukherjee on 2019-01-28 15:10:36 IST --- Root cause: from glusterd_get_state () ret = sys_statvfs(brickinfo->path, &brickstat); if (ret) { gf_msg(this->name, GF_LOG_ERROR, errno, GD_MSG_FILE_OP_FAILED, "statfs error: %s ", strerror(errno)); goto out; } memfree = brickstat.f_bfree * brickstat.f_bsize; memtotal = brickstat.f_blocks * brickstat.f_bsize; fprintf(fp, "Volume%d.Brick%d.spacefree: %" PRIu64 "Bytes\n", count_bkp, count, memfree); fprintf(fp, "Volume%d.Brick%d.spacetotal: %" PRIu64 "Bytes\n", count_bkp, count, memtotal); a statfs call is made on the brick path for every bricks of the volumes to calculate the total vs free space. In this case we shouldn't error out on a statfs failure and should report spacefree and spacetotal as unavailable or 0 bytes. --- Additional comment from Atin Mukherjee on 2019-02-04 07:59:34 IST --- We need to have a test coverage to ensure that get-state command should generate an output successfully even if underlying brick(s) of volume(s) in the cluster go bad. --- Additional comment from sankarshan on 2019-02-04 14:48:30 IST --- (In reply to Atin Mukherjee from comment #4) > We need to have a test coverage to ensure that get-state command should > generate an output successfully even if underlying brick(s) of volume(s) in > the cluster go bad. The test coverage flag needs to be set -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 09:33:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 09:33:20 +0000 Subject: [Bugs] [Bug 1672205] [GSS] 'gluster get-state' command fails if volume brick doesn't exist. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672205 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- Group|private | -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 09:33:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 09:33:37 +0000 Subject: [Bugs] [Bug 1672205] 'gluster get-state' command fails if volume brick doesn't exist. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672205 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|[GSS] 'gluster get-state' |'gluster get-state' command |command fails if volume |fails if volume brick |brick doesn't exist. |doesn't exist. -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 09:53:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 09:53:18 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Miklos Szeredi changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(mszeredi at redhat.c | |om) | --- Comment #11 from Miklos Szeredi --- The underlying problem is that auto invalidate cannot differentiate local and remote modification based on mtime alone. What NFS apprently does is refresh attributes immediately after a write (not sure how often it does this, I guess not after each individual write). FUSE maybe should do this if auto invalidation is enabled, but if the filesystem can do its own invalidation, possibly based on better information than c/mtime, then that seem to be a better option. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 10:03:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 10:03:16 +0000 Subject: [Bugs] [Bug 1672205] 'gluster get-state' command fails if volume brick doesn't exist. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672205 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22147 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 10:03:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 10:03:17 +0000 Subject: [Bugs] [Bug 1672205] 'gluster get-state' command fails if volume brick doesn't exist. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672205 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22147 (glusterd: get-state command should not fail if any brick is gone bad) posted (#1) for review on master by Sanju Rakonde -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 10:46:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 10:46:14 +0000 Subject: [Bugs] [Bug 1657744] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657744 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22148 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 10:46:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 10:46:15 +0000 Subject: [Bugs] [Bug 1657744] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657744 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |POST --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22148 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) posted (#1) for review on release-5 by Varsha Rao -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:07:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:07:09 +0000 Subject: [Bugs] [Bug 1657744] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657744 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22149 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:17:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:17:03 +0000 Subject: [Bugs] [Bug 1672248] New: quorum count not updated in nfs-server vol file Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672248 Bug ID: 1672248 Summary: quorum count not updated in nfs-server vol file Product: GlusterFS Version: 5 Status: NEW Component: replicate Assignee: bugs at gluster.org Reporter: varao at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Check the original bug 1657744. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:19:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:19:35 +0000 Subject: [Bugs] [Bug 1672249] New: quorum count value not updated in nfs-server vol file Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672249 Bug ID: 1672249 Summary: quorum count value not updated in nfs-server vol file Product: GlusterFS Version: 4.1 Status: NEW Component: replicate Assignee: bugs at gluster.org Reporter: varao at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Check the original bug 1657744 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:22:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:22:27 +0000 Subject: [Bugs] [Bug 1657744] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657744 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22149 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) posted (#1) for review on release-4.1 by Varsha Rao --- Comment #5 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22149 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) posted (#2) for review on release-4.1 by Varsha Rao -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:22:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:22:28 +0000 Subject: [Bugs] [Bug 1657744] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657744 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Gluster.org Gerrit 22149 | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:22:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:22:29 +0000 Subject: [Bugs] [Bug 1672249] quorum count value not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672249 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22149 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:22:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:22:30 +0000 Subject: [Bugs] [Bug 1672249] quorum count value not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672249 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22149 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) posted (#2) for review on release-4.1 by Varsha Rao -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:33:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:33:38 +0000 Subject: [Bugs] [Bug 1657744] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657744 --- Comment #6 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22148 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) posted (#2) for review on release-5 by Varsha Rao -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:33:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:33:39 +0000 Subject: [Bugs] [Bug 1657744] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657744 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Gluster.org Gerrit 22148 | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:33:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:33:40 +0000 Subject: [Bugs] [Bug 1672248] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672248 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22148 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:33:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:33:41 +0000 Subject: [Bugs] [Bug 1672248] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672248 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22148 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) posted (#2) for review on release-5 by Varsha Rao -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:48:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:48:56 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22150 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 11:48:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:48:57 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #7 from Worker Ant --- REVIEW: https://review.gluster.org/22150 (afr/shd: Cleanup self heal daemon resources during afr fini) posted (#1) for review on master by mohammed rafi kc -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 11:53:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:53:05 +0000 Subject: [Bugs] [Bug 1672258] New: fuse takes memory and doesn't free Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672258 Bug ID: 1672258 Summary: fuse takes memory and doesn't free Product: GlusterFS Version: 4.1 Hardware: x86_64 OS: Linux Status: NEW Component: fuse Assignee: bugs at gluster.org Reporter: redhat at core.ch CC: bugs at gluster.org Target Milestone: --- Classification: Community Created attachment 1526739 --> https://bugzilla.redhat.com/attachment.cgi?id=1526739&action=edit statedump 20190204 Description of problem: Fuse will take daily more and more memory until the swap is full. Then the system won't work properly anymore. We have upgraded to 4.1 at the end of December 2018 and since there we have to restart the gluster cluster and all nodes more or less every 2 weeks, because the memory is taken. We have this situation on all div. gluster clusters. We have gluster 4.1.7 on a ubuntu 16.04.5 LTS (xenial) installed. System-Checks: --- Memory and Swap: free total used free shared buff/cache available Mem: 32834992 31387196 243932 9148 1203864 897800 Swap: 31999996 25951268 6048728 --- top and find out the service and get the status of the service systemctl status data_net.mount ? data_net.mount - Mount System glusterfs on path /data_net from source localhost:/ctgv0 with Loaded: loaded (/etc/systemd/system/data_net.mount; static; vendor preset: enabled) Active: active (mounted) since Fri 2019-02-01 07:51:32 CET; 3 days ago Where: /data_net What: localhost:/ctgv0 Docs: https://oguya.ch/posts/2015-09-01-systemd-mount-partition/ Process: 11256 ExecUnmount=/bin/umount /data_net (code=exited, status=0/SUCCESS) Process: 11257 ExecMount=/bin/mount localhost:/ctgv0 /data_net -t glusterfs -o defaults,_netdev (code= Tasks: 20 Memory: 28.0G CPU: 12h 43min 23.929s CGroup: /system.slice/data_net.mount ?? 7825 /usr/sbin/glusterfs --process-name fuse --volfile-server=localhost --volfile-id=/ctgv ??11337 /usr/sbin/glusterfs --process-name fuse --volfile-server=localhost --volfile-id=/ctgv Feb 01 07:51:32 nucprdstk112 systemd[1]: Mounting coretech: Mount System glusterfs on path /data_net fro Feb 01 07:51:32 nucprdstk112 systemd[1]: Mounted coretech: Mount System glusterfs on path /data_net from Feb 01 08:02:08 nucprdstk112 data_net[7825]: [2019-02-01 07:02:08.392799] C [rpc-clnt-ping.c:166:rpc_cln --- uptime: 12:03:44 up 17 days, 16 min, 1 user, load average: 1.30, 1.03, 1.01 --- Followed this description: https://docs.gluster.org/en/v3/Troubleshooting/troubleshooting-memory/ gluster volume info Volume Name: ctgv0 Type: Replicate Volume ID: 0e70a1ba-2c70-494a-8a85-f757fe77901a Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: nucprdstk112:/var/glusterfs/ctgv0/brick1 Brick2: nucprdstk113:/var/glusterfs/ctgv0/brick2 Brick3: nucprdstk114:/var/glusterfs/ctgv0/brick2 Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet Version-Release number of selected component (if applicable): gluster --version glusterfs 4.1.7 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.5 LTS Release: 16.04 Codename: xenial How reproducible: Steps to Reproduce: 1. Restart server and wait for 1 or 2 weeks 2. 3. Actual results: still takes memory every day. Expected results: free memory Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 11:55:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 11:55:10 +0000 Subject: [Bugs] [Bug 1672258] fuse takes memory and doesn't free In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672258 Ritzo changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |redhat at core.ch --- Comment #1 from Ritzo --- Created attachment 1526740 --> https://bugzilla.redhat.com/attachment.cgi?id=1526740&action=edit statedump 20190201 another statedump file from 1st February Thanks a lot for your advice / support Ritzo -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 12:07:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 12:07:02 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22151 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 12:07:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 12:07:03 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #8 from Worker Ant --- REVIEW: https://review.gluster.org/22151 (afr/shd: Cleanup self heal daemon resources during afr fini) posted (#1) for review on master by mohammed rafi kc -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 12:56:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 12:56:36 +0000 Subject: [Bugs] [Bug 1243991] "gluster volume set group " is not in the help text In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1243991 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-04 12:56:36 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22067 (cli: Added the group option for volume set) merged (#4) on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 14:20:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:20:53 +0000 Subject: [Bugs] [Bug 1672314] New: thin-arbiter: Check with thin-arbiter file before marking new entry change log Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672314 Bug ID: 1672314 Summary: thin-arbiter: Check with thin-arbiter file before marking new entry change log Product: GlusterFS Version: 5 Status: NEW Component: replicate Assignee: bugs at gluster.org Reporter: aspandey at redhat.com CC: bugs at gluster.org Depends On: 1662264 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1662264 +++ Description of problem: In case of creating an entry, if a fop fails on any one of the data bricks, we mark the changelog on that entry on the brick which was successful. For thin arbiter volume before marking this changelog, we should check if the brick on which fop succeeded was the good brick or not. If the bricks was bad according to thin-arbiter file information, we should just continue with postop changelog process. If the brick was good, we should mark the new entry changelog and continuew with postop changelog. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2018-12-27 09:19:25 UTC --- REVIEW: https://review.gluster.org/21933 (cluster/thin-arbiter: Consider thin-arbiter before marking new entry changelog) posted (#1) for review on master by Ashish Pandey --- Additional comment from Worker Ant on 2019-02-01 05:45:32 UTC --- REVIEW: https://review.gluster.org/21933 (cluster/thin-arbiter: Consider thin-arbiter before marking new entry changelog) merged (#6) on master by Amar Tumballi Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1662264 [Bug 1662264] thin-arbiter: Check with thin-arbiter file before marking new entry change log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 14:20:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:20:53 +0000 Subject: [Bugs] [Bug 1662264] thin-arbiter: Check with thin-arbiter file before marking new entry change log In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662264 Ashish Pandey changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672314 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672314 [Bug 1672314] thin-arbiter: Check with thin-arbiter file before marking new entry change log -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 14:21:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:21:14 +0000 Subject: [Bugs] [Bug 1672314] thin-arbiter: Check with thin-arbiter file before marking new entry change log In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672314 Ashish Pandey changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 14:44:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:44:49 +0000 Subject: [Bugs] [Bug 1670303] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-04 14:44:49 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22116 (api: bad GFAPI_4.1.6 block) merged (#2) on release-4.1 by Kaleb KEITHLEY -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 14:44:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:44:49 +0000 Subject: [Bugs] [Bug 1667099] GlusterFS 4.1.8 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667099 Bug 1667099 depends on bug 1670303, which changed state. Bug 1670303 Summary: api: bad GFAPI_4.1.6 block https://bugzilla.redhat.com/show_bug.cgi?id=1670303 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 14:44:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:44:49 +0000 Subject: [Bugs] [Bug 1670307] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670307 Bug 1670307 depends on bug 1670303, which changed state. Bug 1670303 Summary: api: bad GFAPI_4.1.6 block https://bugzilla.redhat.com/show_bug.cgi?id=1670303 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 14:45:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:45:10 +0000 Subject: [Bugs] [Bug 1670307] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670307 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-04 14:45:10 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22117 (api: bad GFAPI_4.1.6 block) merged (#1) on release-5 by Kaleb KEITHLEY -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 14:45:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:45:10 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1670307, which changed state. Bug 1670307 Summary: api: bad GFAPI_4.1.6 block https://bugzilla.redhat.com/show_bug.cgi?id=1670307 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 14:47:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:47:44 +0000 Subject: [Bugs] [Bug 1671217] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671217 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-04 14:47:44 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22129 (core: move \"dict is NULL\" logs to DEBUG log level) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 14:48:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:48:49 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #31 from Worker Ant --- REVIEW: https://review.gluster.org/22135 (socket: don't pass return value from protocol handler to event handler) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 14:50:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:50:30 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #32 from Worker Ant --- REVIEW: https://review.gluster.org/22134 (socket: fix issue when socket write return with EAGAIN) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 14:51:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:51:38 +0000 Subject: [Bugs] [Bug 1671611] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671611 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-04 14:51:38 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22136 (cluster/dht: Delete invalid linkto files in rmdir) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 14:51:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:51:39 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Bug 1668989 depends on bug 1671611, which changed state. Bug 1671611 Summary: Unable to delete directories that contain linkto files that point to itself. https://bugzilla.redhat.com/show_bug.cgi?id=1671611 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 14:53:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:53:29 +0000 Subject: [Bugs] [Bug 1665145] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665145 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-04 14:53:29 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22139 (readdir-ahead: do not zero-out iatt in fop cbk) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 14:53:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 14:53:29 +0000 Subject: [Bugs] [Bug 1670253] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670253 Bug 1670253 depends on bug 1665145, which changed state. Bug 1665145 Summary: Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set https://bugzilla.redhat.com/show_bug.cgi?id=1665145 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 15:08:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 15:08:15 +0000 Subject: [Bugs] [Bug 1669382] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669382 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-04 15:08:15 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22095 (features/shard: Ref shard inode while adding to fsync list) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 4 15:14:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 15:14:57 +0000 Subject: [Bugs] [Bug 1626085] "glusterfs --process-name fuse" crashes and leads to "Transport endpoint is not connected" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1626085 GCth changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ravishankar at redha | |t.com) | |needinfo?(rhb1 at gcth.net) | --- Comment #11 from GCth --- Up until line #17 they are the same, here's another example: Core was generated by `/usr/sbin/glusterfs --process-name fuse --volfile-server=xxxx --'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f8d5e877560 in __gf_free (free_ptr=0x7f8d49a25378) at mem-pool.c:330 330 mem-pool.c: No such file or directory. [Current thread is 1 (Thread 0x7f8d521bf700 (LWP 2217))] (gdb) bt #0 0x00007f8d5e877560 in __gf_free (free_ptr=0x7f8d49a25378) at mem-pool.c:330 #1 0x00007f8d5e842e1e in dict_destroy (this=0x7f8d4994f708) at dict.c:701 #2 0x00007f8d5e842f25 in dict_unref (this=) at dict.c:753 #3 0x00007f8d584330d4 in afr_local_cleanup (local=0x7f8d49a56cc8, this=) at afr-common.c:2091 #4 0x00007f8d5840d584 in afr_transaction_done (frame=, this=) at afr-transaction.c:369 #5 0x00007f8d5841483a in afr_unlock (frame=frame at entry=0x7f8d4995ec08, this=this at entry=0x7f8d54019d40) at afr-lk-common.c:1085 #6 0x00007f8d5840aeca in afr_changelog_post_op_done (frame=frame at entry=0x7f8d4995ec08, this=this at entry=0x7f8d54019d40) at afr-transaction.c:778 #7 0x00007f8d5840e105 in afr_changelog_post_op_do (frame=0x7f8d4995ec08, this=0x7f8d54019d40) at afr-transaction.c:1442 #8 0x00007f8d5840edcf in afr_changelog_post_op_now (frame=0x7f8d4995ec08, this=0x7f8d54019d40) at afr-transaction.c:1512 #9 0x00007f8d5840ef4c in afr_delayed_changelog_wake_up_cbk (data=) at afr-transaction.c:2444 #10 0x00007f8d58410866 in afr_transaction_start (local=local at entry=0x7f8d4cd6ed18, this=this at entry=0x7f8d54019d40) at afr-transaction.c:2847 #11 0x00007f8d58410c89 in afr_transaction (frame=frame at entry=0x7f8d4e643068, this=this at entry=0x7f8d54019d40, type=type at entry=AFR_DATA_TRANSACTION) at afr-transaction.c:2918 #12 0x00007f8d583fcb70 in afr_do_writev (frame=frame at entry=0x7f8d4e245608, this=this at entry=0x7f8d54019d40) at afr-inode-write.c:477 #13 0x00007f8d583fd81d in afr_writev (frame=frame at entry=0x7f8d4e245608, this=this at entry=0x7f8d54019d40, fd=fd at entry=0x7f8d499f3758, vector=0x7f8d4e932b40, count=1, offset=1024, flags=32769, iobref=0x7f8d488cb3b0, xdata=0x0) at afr-inode-write.c:555 #14 0x00007f8d5818cbef in dht_writev (frame=frame at entry=0x7f8d4e29c598, this=, fd=0x7f8d499f3758, vector=vector at entry=0x7f8d521be5c0, count=count at entry=1, off=, flags=32769, iobref=0x7f8d488cb3b0, xdata=0x0) at dht-inode-write.c:223 #15 0x00007f8d53df0b77 in wb_fulfill_head (wb_inode=wb_inode at entry=0x7f8d49a25310, head=0x7f8d49bbcb40) at write-behind.c:1156 #16 0x00007f8d53df0dfb in wb_fulfill (wb_inode=wb_inode at entry=0x7f8d49a25310, liabilities=liabilities at entry=0x7f8d521be720) at write-behind.c:1233 #17 0x00007f8d53df21b6 in wb_process_queue (wb_inode=wb_inode at entry=0x7f8d49a25310) at write-behind.c:1784 #18 0x00007f8d53df233f in wb_fulfill_cbk (frame=frame at entry=0x7f8d49cc15a8, cookie=, this=, op_ret=op_ret at entry=1024, op_errno=op_errno at entry=0, prebuf=prebuf at entry=0x7f8d49c7f8c0, postbuf=, xdata=) at write-behind.c:1105 #19 0x00007f8d5818b31e in dht_writev_cbk (frame=0x7f8d498dfa48, cookie=, this=, op_ret=1024, op_errno=0, prebuf=0x7f8d49c7f8c0, postbuf=0x7f8d49c7f958, xdata=0x7f8d4e65a7e8) at dht-inode-write.c:140 #20 0x00007f8d583fc2b7 in afr_writev_unwind (frame=frame at entry=0x7f8d48374db8, this=this at entry=0x7f8d54019d40) at afr-inode-write.c:234 #21 0x00007f8d583fc83e in afr_writev_wind_cbk (frame=0x7f8d4995ec08, cookie=, this=0x7f8d54019d40, op_ret=, op_errno=, prebuf=, postbuf=0x7f8d521be9d0, xdata=0x7f8d4e6e30d8) at afr-inode-write.c:388 #22 0x00007f8d586c4865 in client4_0_writev_cbk (req=, iov=, count=, myframe=0x7f8d4e621578) at client-rpc-fops_v2.c:685 #23 0x00007f8d5e61c130 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f8d54085540, pollin=pollin at entry=0x7f8d4ea47850) at rpc-clnt.c:755 #24 0x00007f8d5e61c48f in rpc_clnt_notify (trans=0x7f8d54085800, mydata=0x7f8d54085570, event=, data=0x7f8d4ea47850) at rpc-clnt.c:923 #25 0x00007f8d5e618893 in rpc_transport_notify (this=this at entry=0x7f8d54085800, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f8d4ea47850) at rpc-transport.c:525 #26 0x00007f8d59401671 in socket_event_poll_in (notify_handled=true, this=0x7f8d54085800) at socket.c:2504 #27 socket_event_handler (fd=, idx=idx at entry=2, gen=4, data=data at entry=0x7f8d54085800, poll_in=, poll_out=, poll_err=) at socket.c:2905 #28 0x00007f8d5e8ab945 in event_dispatch_epoll_handler (event=0x7f8d521bee8c, event_pool=0x56110317e0b0) at event-epoll.c:591 #29 event_dispatch_epoll_worker (data=0x7f8d5406f7e0) at event-epoll.c:668 #30 0x00007f8d5dacb494 in start_thread (arg=0x7f8d521bf700) at pthread_create.c:333 #31 0x00007f8d5d374acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 All the gluster instances looks similar to the following setup: Type: Distributed-Replicate Volume ID: e9dd963c... Status: Started Snapshot Count: 0 Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.10.11.1:/export/data1 Brick2: 10.10.11.2:/export/data1 Brick3: 10.10.11.3:/export/data1 Brick4: 10.10.11.4:/export/data1 Options Reconfigured: cluster.favorite-child-policy: mtime cluster.self-heal-daemon: enable performance.cache-size: 1GB performance.quick-read: on performance.stat-prefetch: on performance.read-ahead: on performance.readdir-ahead: on auth.allow: 10.*.*.* transport.address-family: inet nfs.disable: on features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.cache-invalidation: on performance.md-cache-timeout: 600 network.inode-lru-limit: 50000 I do not have a reproducer, the gluster instance is 2-5TB of files, mostly small ones, with lots of directories. They reach up to 10M inodes used as reported by df -hi, brick storage is on XFS as recommended. The crash of individual glusterfs process happens once every several days. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 15:16:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 15:16:09 +0000 Subject: [Bugs] [Bug 1626085] "glusterfs --process-name fuse" crashes and leads to "Transport endpoint is not connected" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1626085 --- Comment #12 from GCth --- One more - it's currently: glusterfs 5.3 installed from https://download.gluster.org/pub/gluster/glusterfs/5/LATEST/Debian/stretch/amd64/apt -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 4 16:08:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 04 Feb 2019 16:08:43 +0000 Subject: [Bugs] [Bug 1672248] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672248 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-04 16:08:43 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22148 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) merged (#3) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 02:59:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 02:59:24 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #5 from David E. Smith --- I've added the five of you to our org's Box account; all of you should have invitations to a shared folder, and I'm uploading a few of the cores now. I hope they're of value to you. The binaries are all from the CentOS Storage SIG repo at https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/ . They're all current as of a few days ago: [davidsmith at wuit-s-10882 ~]$ rpm -qa | grep gluster glusterfs-5.3-1.el7.x86_64 glusterfs-client-xlators-5.3-1.el7.x86_64 glusterfs-fuse-5.3-1.el7.x86_64 glusterfs-libs-5.3-1.el7.x86_64 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 04:20:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 04:20:45 +0000 Subject: [Bugs] [Bug 1671637] geo-rep: Issue with configparser import In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671637 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- COMMIT: https://review.gluster.org/22138 committed in master by "Amar Tumballi" with a commit message- geo-rep: Fix configparser import issue 'configparser' is backported to python2 and can be installed using pip (pip install configparser). So trying to import 'configparser' first and later 'ConfigParser' can cause issues w.r.t unicode strings. Always try importing 'ConfigParser' first and then 'configparser'. This solves python2/python3 compat issues. Change-Id: I2a87c3fc46476296b8cb547338f35723518751cc fixes: bz#1671637 Signed-off-by: Kotresh HR -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 5 05:17:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 05:17:23 +0000 Subject: [Bugs] [Bug 1672480] New: Bugs Test Module tests failing on s390x Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Bug ID: 1672480 Summary: Bugs Test Module tests failing on s390x Product: GlusterFS Version: 4.1 Hardware: s390x OS: Linux Status: NEW Component: tests Severity: urgent Assignee: bugs at gluster.org Reporter: abhaysingh1722 at yahoo.in CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Observing test failures for the following test cases:- ./tests/bugs/glusterfs/bug-902610.t ./tests/bugs/posix/bug-1619720.t ./tests/bitrot/bug-1207627-bitrot-scrub-status.t After analyzing the above test failures, we have observed that the hash values for the bricks and files are getting differently calculated on s390x systems as compared to those on x86. As per the documentation given at https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/dht/ , To place a file in a directory, a hash is calculated for the file using both the (containing) directory's unique GFID and the file's name.This hash is then matched to one of the layout assignments, to yield the hashed location. However, on s390x, certain files have hash values that are beyond the hash range of the available bricks. Therefore, these files don't get located in their respective hashed locations. This has been observed in other test cases too. For example, ./tests/bugs/distribute/bug-1161311.t, ./tests/bugs/distribute/bug-1193636.t, ./tests/basic/namespace.t. Is there any workaround to get the correct hashed locations for the files? Version-Release number of selected component (if applicable): v4.1.5 How reproducible: Build Glusterfs v4.1.5 and run the test case with ./run-tests.sh prove -vf Steps to Reproduce: 1. 2. 3. Actual results: Tests FAIL Expected results: Tests should PASS Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 07:05:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 07:05:24 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Red Hat Bugzilla Rules Engine changed: What |Removed |Added ---------------------------------------------------------------------------- Target Release|--- |RHGS 3.4.z Batch Update 4 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 5 11:00:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 11:00:04 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |nbalacha at redhat.com --- Comment #6 from Nithya Balachandran --- (In reply to David E. Smith from comment #5) > I've added the five of you to our org's Box account; all of you should have > invitations to a shared folder, and I'm uploading a few of the cores now. I > hope they're of value to you. > > The binaries are all from the CentOS Storage SIG repo at > https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/ . They're > all current as of a few days ago: > > [davidsmith at wuit-s-10882 ~]$ rpm -qa | grep gluster > glusterfs-5.3-1.el7.x86_64 > glusterfs-client-xlators-5.3-1.el7.x86_64 > glusterfs-fuse-5.3-1.el7.x86_64 > glusterfs-libs-5.3-1.el7.x86_64 Thanks. We will take a look and get back to you. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 12:23:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 12:23:41 +0000 Subject: [Bugs] [Bug 1671647] Anomalies in python-lint build job In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671647 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-05 12:23:41 --- Comment #3 from Nigel Babu --- Ah, this was because we were trying to lint the virtualenv. I've fixed this up in this review and now it should fail correctly: https://review.gluster.org/#/c/build-jobs/+/22155/ -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 14:31:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 14:31:52 +0000 Subject: [Bugs] [Bug 1672656] New: glustereventsd: crash, ABRT report for package glusterfs has reached 100 occurrences Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672656 Bug ID: 1672656 Summary: glustereventsd: crash, ABRT report for package glusterfs has reached 100 occurrences Product: GlusterFS Version: 5 OS: Linux Status: NEW Component: eventsapi Assignee: bugs at gluster.org Reporter: kkeithle at redhat.com Target Milestone: --- Classification: Community Description of problem: https://retrace.fedoraproject.org/faf/reports/bthash/ee9831c192f230a223ebdbecc7ea915aaf92636f/ Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 14:41:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 14:41:05 +0000 Subject: [Bugs] [Bug 1672205] 'gluster get-state' command fails if volume brick doesn't exist. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672205 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- COMMIT: https://review.gluster.org/22147 committed in master by "Atin Mukherjee" with a commit message- glusterd: get-state command should not fail if any brick is gone bad Problem: get-state command will error out, if any of the underlying brick(s) of volume(s) in the cluster go bad. It is expected that get-state command should not error out, but should generate an output successfully. Solution: In glusterd_get_state(), a statfs call is made on the brick path for every bricks of the volumes to calculate the total and free memory available. If any of statfs call fails on any brick, we should not error out and should report total memory and free memory of that brick as 0. This patch also handles a statfs failure scenario in glusterd_store_retrieve_bricks(). fixes: bz#1672205 Change-Id: Ia9e8a1d8843b65949d72fd6809bd21d39b31ad83 Signed-off-by: Sanju Rakonde -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 15:16:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 15:16:51 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #5 from Worker Ant --- REVIEW: https://review.gluster.org/22156 (inode: granular locking) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 16:05:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 16:05:06 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #530 from Worker Ant --- REVIEW: https://review.gluster.org/22157 (fuse: correctly handle setxattr values) posted (#1) for review on master by Xavi Hernandez -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 16:34:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 16:34:47 +0000 Subject: [Bugs] [Bug 1672711] New: Upgrade from glusterfs 3.12 to gluster 4/5 broken Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Bug ID: 1672711 Summary: Upgrade from glusterfs 3.12 to gluster 4/5 broken Product: GlusterFS Version: mainline Status: NEW Component: packaging Severity: urgent Priority: urgent Assignee: bugs at gluster.org Reporter: sabose at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: While updating glusterfs 3.12, run into below error : Error: Package: glusterfs-gnfs-3.12.15-1.el7.x86_64 (@ovirt-4.2-centos-gluster312) Requires: glusterfs(x86-64) = 3.12.15-1.el7 Removing: glusterfs-3.12.15-1.el7.x86_64 (@ovirt-4.2-centos-gluster312) glusterfs(x86-64) = 3.12.15-1.el7 Updated By: glusterfs-5.3-1.el7.x86_64 (ovirt-4.3-centos-gluster5) glusterfs(x86-64) = 5.3-1.el7 Version-Release number of selected component (if applicable): 3.12 How reproducible: Always -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 16:43:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 16:43:45 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(desmith at wustl.edu | |) --- Comment #7 from Nithya Balachandran --- David, Can you try mounting the volume with the option lru-limit=0 and let us know if you still see the crashes? Regards, Nithya -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 5 17:18:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 17:18:35 +0000 Subject: [Bugs] [Bug 1672727] New: Fix timeouts so the tests pass on AWS Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 Bug ID: 1672727 Summary: Fix timeouts so the tests pass on AWS Product: GlusterFS Version: mainline Status: NEW Component: tests Assignee: bugs at gluster.org Reporter: nigelb at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Some test timeouts need a bump on AWS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 17:21:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 17:21:47 +0000 Subject: [Bugs] [Bug 1672727] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22065 (Bump up timeout for tests on AWS) posted (#5) for review on master by Nigel Babu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 5 17:48:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 05 Feb 2019 17:48:01 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-05 17:48:01 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 00:56:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 00:56:04 +0000 Subject: [Bugs] [Bug 1672818] New: GlusterFS 6.0 tracker Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Bug ID: 1672818 Summary: GlusterFS 6.0 tracker Product: GlusterFS Version: 6 Status: NEW Component: core Keywords: Tracking, Triaged Assignee: bugs at gluster.org Reporter: srangana at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Tracker for the release 6.0 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 01:07:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 01:07:47 +0000 Subject: [Bugs] [Bug 1672826] New: Request gerrit dashboard addition for release 6 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672826 Bug ID: 1672826 Summary: Request gerrit dashboard addition for release 6 Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: srangana at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Request a gerrit dashboard for release 6 like the following, - https://review.gluster.org/#/projects/glusterfs,dashboards/dashboard:4-1-dashboard -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 01:08:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 01:08:54 +0000 Subject: [Bugs] [Bug 1672828] New: Restrict gerrit merge permissions for branch release-6 to release owners Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672828 Bug ID: 1672828 Summary: Restrict gerrit merge permissions for branch release-6 to release owners Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: srangana at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Release owners: - srangana at redhat.com Also additionally add Amar Tumballi as well to have merge rights. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 01:18:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 01:18:26 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #531 from Worker Ant --- REVIEW: https://review.gluster.org/22158 (glusterd: Update op-version for release 7) posted (#1) for review on master by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 01:46:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 01:46:52 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #532 from Worker Ant --- REVIEW: https://review.gluster.org/22159 (api: Update all future API versions to rel-6) posted (#1) for review on master by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 03:08:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 03:08:10 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rgowdapp at redhat.com Depends On| |1664934 --- Comment #1 from Raghavendra G --- Bug 1664934 - glusterfs-fuse client not benefiting from page cache on read after write Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 03:08:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 03:08:10 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672818 (glusterfs-6.0) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 03:09:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 03:09:33 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1670718 --- Comment #2 from Raghavendra G --- Bug 1670718 - md-cache should be loaded at a position in graph where it sees stats in write cbk Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670718 [Bug 1670718] md-cache should be loaded at a position in graph where it sees stats in write cbk -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 03:09:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 03:09:33 +0000 Subject: [Bugs] [Bug 1670718] md-cache should be loaded at a position in graph where it sees stats in write cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670718 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672818 (glusterfs-6.0) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 03:46:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 03:46:04 +0000 Subject: [Bugs] [Bug 1672851] New: With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Bug ID: 1672851 Summary: With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" Product: GlusterFS Version: 4.1 Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: If parallel-readdir is enabled on a volume, rm -rf fails with "Directory not empty" if contains stale linkto files. Version-Release number of selected component (if applicable): How reproducible: Consistently Steps to Reproduce: 1. Create a 3 brick distribute volume 2. Enable parallel-readdir and readdir-ahead on the volume 3. Fuse mount the volume and mkdir dir0 4. Create some files inside dir0 and rename them so linkto files are created on the bricks 5. Check the bricks to see which files have linkto files. Delete the data files directly on the bricks, leaving the linkto files behind. These are now stale linkto files. 6. Remount the volume 7. rm -rf dir0 Actual results: [root at rhgs313-6 fuse1]# rm -rf dir0/ rm: cannot remove ?dir0/?: Directory not empty Expected results: dir0 should be deleted without errors Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 03:46:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 03:46:18 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Version|4.1 |mainline -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 03:46:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 03:46:30 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |nbalacha at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 04:10:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 04:10:11 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 --- Comment #1 from Nithya Balachandran --- RCA: rm -rf works by first listing and unlinking all entries in and then calling an rmdir . As DHT readdirp does not return linkto files in the listing, they are not unlinked as part of the rm -rf itself. dht_rmdir handles this by performing a readdirp internally on and deleting all stale linkto files before proceeding with the actual rmdir operation. When parallel-readdir is enabled, the rda xlator is loaded below dht in the graph and proactively lists and caches entries when an opendir is performed. Entries are returned from this cache for any subsequent readdirp calls on the directory that was opened. DHT uses the presence of the trusted.glusterfs.dht.linkto xattr to determine whether a file is a linkto file. As this call to opendir does not set trusted.glusterfs.dht.linkto in the list of requested xattrs for the opendir call, the cached entries do not contain this xattr value. As none of the entries returned will have the xattr, DHT believes they are all data files and fails the rmdir with ENOTEMPTY. Turning off parallel-readdir allows the rm -rf to succeed. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 04:37:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 04:37:57 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22160 (cluster/dht: Request linkto xattrs in dht_rmdir opendir) posted (#1) for review on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 04:38:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 04:38:46 +0000 Subject: [Bugs] [Bug 1672869] New: With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672869 Bug ID: 1672869 Summary: With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" Product: Red Hat Gluster Storage Version: 3.4 Status: NEW Component: distribute Assignee: nbalacha at redhat.com Reporter: nbalacha at redhat.com QA Contact: tdesala at redhat.com CC: bugs at gluster.org, rhs-bugs at redhat.com, sankarshan at redhat.com, storage-qa-internal at redhat.com Depends On: 1672851 Target Milestone: --- Classification: Red Hat +++ This bug was initially created as a clone of Bug #1672851 +++ Description of problem: If parallel-readdir is enabled on a volume, rm -rf fails with "Directory not empty" if contains stale linkto files. Version-Release number of selected component (if applicable): How reproducible: Consistently Steps to Reproduce: 1. Create a 3 brick distribute volume 2. Enable parallel-readdir and readdir-ahead on the volume 3. Fuse mount the volume and mkdir dir0 4. Create some files inside dir0 and rename them so linkto files are created on the bricks 5. Check the bricks to see which files have linkto files. Delete the data files directly on the bricks, leaving the linkto files behind. These are now stale linkto files. 6. Remount the volume 7. rm -rf dir0 Actual results: [root at rhgs313-6 fuse1]# rm -rf dir0/ rm: cannot remove ?dir0/?: Directory not empty Expected results: dir0 should be deleted without errors Additional info: --- Additional comment from Nithya Balachandran on 2019-02-06 04:10:11 UTC --- RCA: rm -rf works by first listing and unlinking all entries in and then calling an rmdir . As DHT readdirp does not return linkto files in the listing, they are not unlinked as part of the rm -rf itself. dht_rmdir handles this by performing a readdirp internally on and deleting all stale linkto files before proceeding with the actual rmdir operation. When parallel-readdir is enabled, the rda xlator is loaded below dht in the graph and proactively lists and caches entries when an opendir is performed. Entries are returned from this cache for any subsequent readdirp calls on the directory that was opened. DHT uses the presence of the trusted.glusterfs.dht.linkto xattr to determine whether a file is a linkto file. As this call to opendir does not set trusted.glusterfs.dht.linkto in the list of requested xattrs for the opendir call, the cached entries do not contain this xattr value. As none of the entries returned will have the xattr, DHT believes they are all data files and fails the rmdir with ENOTEMPTY. Turning off parallel-readdir allows the rm -rf to succeed. --- Additional comment from Worker Ant on 2019-02-06 04:37:57 UTC --- REVIEW: https://review.gluster.org/22160 (cluster/dht: Request linkto xattrs in dht_rmdir opendir) posted (#1) for review on master by N Balachandran Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 04:38:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 04:38:46 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672869 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672869 [Bug 1672869] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 04:38:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 04:38:47 +0000 Subject: [Bugs] [Bug 1672869] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672869 Red Hat Bugzilla Rules Engine changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |ZStream -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 04:39:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 04:39:44 +0000 Subject: [Bugs] [Bug 1672869] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672869 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 05:43:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 05:43:58 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jthottan at redhat.com Flags| |needinfo?(jthottan at redhat.c | |om) --- Comment #1 from Sahina Bose --- Jiffin, can you help with this? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 06:33:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 06:33:52 +0000 Subject: [Bugs] [Bug 1672314] thin-arbiter: Check with thin-arbiter file before marking new entry change log In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672314 Ashish Pandey changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |aspandey at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 06:36:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 06:36:58 +0000 Subject: [Bugs] [Bug 1662264] thin-arbiter: Check with thin-arbiter file before marking new entry change log In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662264 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22161 (cluster/thin-arbiter: Consider thin-arbiter before marking new entry changelog) posted (#1) for review on release-5 by Ashish Pandey -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 07:17:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 07:17:16 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #533 from Worker Ant --- COMMIT: https://review.gluster.org/22158 committed in master by "Shyamsundar Ranganathan" with a commit message- glusterd: Update op-version for release 7 Change-Id: I0f3978d7e603e6e767dc7aa2a23ef35b1f2b43f7 Updates: bz#1193929 Signed-off-by: ShyamsundarR -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 07:23:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 07:23:49 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #8 from Nithya Balachandran --- Initial analysis of one of the cores: [root at rhgs313-7 gluster-5.3]# gdb -c core.6014 /usr/sbin/glusterfs [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --direct-io-mode=disable --fuse-mountopts=noatime,context="'. Program terminated with signal 11, Segmentation fault. #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 410 if (!xl->call_cleanup && xl->cbks->forget) (gdb) bt #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 #1 0x00007fa1809e90a2 in __inode_destroy (inode=0x7fa0d0349af8) at inode.c:432 #2 inode_table_prune (table=table at entry=0x7fa15800c3c0) at inode.c:1696 #3 0x00007fa1809e9f96 in inode_forget_with_unref (inode=0x7fa0d0349af8, nlookup=128) at inode.c:1273 #4 0x00007fa177dae4e1 in do_forget (this=, unique=, nodeid=, nlookup=) at fuse-bridge.c:726 #5 0x00007fa177dae5bd in fuse_forget (this=, finh=0x7fa0a41da500, msg=, iobuf=) at fuse-bridge.c:741 #6 0x00007fa177dc5d7a in fuse_thread_proc (data=0x557a0e8ffe20) at fuse-bridge.c:5125 #7 0x00007fa17f83bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa17f103ead in msync () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? () (gdb) f 0 #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 410 if (!xl->call_cleanup && xl->cbks->forget) (gdb) l 405 for (index = 0; index < inode->table->xl->graph->xl_count; index++) { 406 if (inode->_ctx[index].value1 || inode->_ctx[index].value2) { 407 xl = (xlator_t *)(long)inode->_ctx[index].xl_key; 408 old_THIS = THIS; 409 THIS = xl; 410 if (!xl->call_cleanup && xl->cbks->forget) 411 xl->cbks->forget(xl, inode); 412 THIS = old_THIS; 413 } 414 } (gdb) p *xl Cannot access memory at address 0x0 (gdb) p index $1 = 6 (gdb) p inode->table->xl->graph->xl_count $3 = 13 (gdb) p inode->_ctx[index].value1 $4 = 0 (gdb) p inode->_ctx[index].value2 $5 = 140327960119304 (gdb) p/x inode->_ctx[index].value2 $6 = 0x7fa0a6370808 Based on the graph, the xlator with index = 6 is (gdb) p ((xlator_t*) inode->table->xl->graph->top)->next->next->next->next->next->next->next->name $31 = 0x7fa16c0122e0 "web-content-read-ahead" (gdb) p ((xlator_t*) inode->table->xl->graph->top)->next->next->next->next->next->next->next->xl_id $32 = 6 But read-ahead does not update the inode_ctx at all. There seems to be some sort of memory corruption happening here but that needs further analysis. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 08:18:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:18:36 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(archon810 at gmail.c | |om) --- Comment #18 from Nithya Balachandran --- (In reply to Artem Russakovskii from comment #17) > The fuse crash happened again yesterday, to another volume. Are there any > mount options that could help mitigate this? > > In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch > and restart the mount, which works and recovers the mount point within a > minute. Not ideal, but a temporary workaround. > > By the way, the way to reproduce this "Transport endpoint is not connected" > condition for testing purposes is to kill -9 the right "glusterfs > --process-name fuse" process. > > > monit check: > check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 > start program = "/bin/mount /mnt/glusterfs_data1" > stop program = "/bin/umount /mnt/glusterfs_data1" > if space usage > 90% for 5 times within 15 cycles > then alert else if succeeded for 10 cycles then alert > > > stack trace: > [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] > (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) > [0x7fa0249e4329] > -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) > [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) > [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] > [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] > (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) > [0x7fa0249e4329] > -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) > [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) > [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 26 times between [2019-02-01 23:21:20.857333] and > [2019-02-01 23:21:56.164427] > The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] > 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" > repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 > 23:22:03.474036] > pending frames: > frame : type(1) op(LOOKUP) > frame : type(0) op(0) > patchset: git://git.gluster.org/glusterfs.git > signal received: 6 > time of crash: > 2019-02-01 23:22:03 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 5.3 > /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] > /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] > /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] > /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] > /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] > /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] > /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] > /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] > /usr/lib64/glusterfs/5.3/xlator/cluster/replicate. > so(+0x5dc9d)[0x7fa025543c9d] > /usr/lib64/glusterfs/5.3/xlator/cluster/replicate. > so(+0x70ba1)[0x7fa025556ba1] > /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] > /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] > /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] > /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] > /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] > /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] > /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] > /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] Please mount the volume using the option lru-limit=0 and see if the crashes go away. We are currently working on analysing some coredumps and will update once we have a fix. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 08:53:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:53:01 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20755 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 08:53:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:53:25 +0000 Subject: [Bugs] [Bug 1542072] Syntactical errors in hook scripts for managing SELinux context on bricks #2 (S10selinux-label-brick.sh + S10selinux-del-fcontext.sh) In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1542072 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 19502 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 08:53:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:53:54 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20860 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 08:55:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:55:45 +0000 Subject: [Bugs] [Bug 1626543] dht/tests: Create a .t to test all possible combinations for file rename In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1626543 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21121 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 08:56:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:56:32 +0000 Subject: [Bugs] [Bug 1512691] PostgreSQL DB Restore: unexpected data beyond EOF In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1512691 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20981 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 08:56:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:56:57 +0000 Subject: [Bugs] [Bug 1512691] PostgreSQL DB Restore: unexpected data beyond EOF In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1512691 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20737 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 08:57:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:57:23 +0000 Subject: [Bugs] [Bug 1512691] PostgreSQL DB Restore: unexpected data beyond EOF In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1512691 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20980 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 08:57:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:57:50 +0000 Subject: [Bugs] [Bug 1512691] PostgreSQL DB Restore: unexpected data beyond EOF In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1512691 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21006 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 08:58:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:58:13 +0000 Subject: [Bugs] [Bug 1624701] error-out {inode, entry}lk fops with all-zero lk-owner In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1624701 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21058 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 08:58:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:58:41 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20754 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 08:59:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 08:59:46 +0000 Subject: [Bugs] [Bug 1602282] tests/bugs/bug-1371806_acl.t fails for distributed regression framework In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1602282 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20596 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 09:00:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 09:00:30 +0000 Subject: [Bugs] [Bug 1299203] resolve-gids is not needed for Linux kernels v3.8 and newer In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1299203 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20500 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 09:00:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 09:00:53 +0000 Subject: [Bugs] [Bug 1580315] gluster volume status inode getting timed out after 30 minutes with no output/error In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1580315 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20045 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 09:25:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 09:25:13 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nbalacha at redhat.com Flags| |needinfo?(nbalacha at redhat.c | |om) --- Comment #1 from abhays --- Please Update on this, Required Urgently. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 10:31:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 10:31:38 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nbalacha at redhat.c | |om) | --- Comment #2 from Nithya Balachandran --- Do you have a mixed setup where some clients are little-endian and others big-endian? If not this should not be a problem. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 10:32:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 10:32:25 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #3 from Nithya Balachandran --- (In reply to Nithya Balachandran from comment #2) > Do you have a mixed setup where some clients are little-endian and others > big-endian? If not this should not be a problem. As long as all your servers and clients have the same "endianness", gluster should work fine. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 10:52:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 10:52:48 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #4 from abhays --- Thanks for the reply. Agreed @Nithya. The setup used by the clients are having same "endianness"(Big Endian). However, certain test cases fail on our Big Endian Systems which is our major concern. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 10:59:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 10:59:37 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |abhaysingh1722 at yahoo.in Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) --- Comment #5 from Nithya Balachandran --- (In reply to abhays from comment #4) > Thanks for the reply. > > Agreed @Nithya. > The setup used by the clients are having same "endianness"(Big Endian). > However, certain test cases fail on our Big Endian Systems which is our > major concern. I think that is because the tests in question are assuming that the files will exist on a certain brick based on the results we got while running them on out little-endian systems. As the hash values are different on big-endian systems, thos assumptions no longer hold. Do you have a list of all the tests that are failing and where they fail? I can check to see if that is the case. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 11:21:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 11:21:34 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(abhaysingh1722 at ya | |hoo.in) | --- Comment #6 from abhays --- Yes @Nithya, Below are the test cases that fail, their cause of failure and possible workaround on Big-Endian:- Following are the test cases which pass after changing the bricks in the test case:- ./tests/bugs/distribute/bug-1161311.t --------passes after changing brick3 to brick1 in subtests 31 and 41. ./tests/bugs/distribute/bug-1193636.t --------passes after changing brick3 to brick1 in subtest 10. ./tests/bugs/nfs/bug-847622.t ----------------passes after giving absolute path of testfile in subtest 9. Following are the test cases that are still failing even after changing the bricks, however if little-endian hash values are hard-coded on big-endian in the file ./xlators/cluster/dht/src/dht-layout.c, then these test cases pass on Big-Endian:- ./tests/bugs/glusterfs/bug-902610.t-------------subtest 7 fails ./tests/bugs/posix/bug-1619720.t----------------subtests 13 and 14 fail Following test case is failing because of "Cannot allocate memory" issue:- ./tests/bitrot/bug-1207627-bitrot-scrub-status.t----------subtest 20 fails with the below error:- [client-rpc-fops_v2.c:961:client4_0_fgetxattr_cbk] 0-patchy-client-0: remote operation failed [Cannot allocate memory] Following test case is failing on which issue has already been raised:- ./tests/features/trash.t-------------- https://bugzilla.redhat.com/show_bug.cgi?id=1627060 So, please look into this and let us know if any workaround can be provided to make the above tests pass on Big-Endian. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 11:44:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 11:44:47 +0000 Subject: [Bugs] [Bug 1651584] [geo-rep]: validate the config checkpoint date and fail if it is not is exact format hh:mm:ss In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651584 Shwetha K Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1672979 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672979 [Bug 1672979] [geo-rep]: validate the config checkpoint date and fail if it is not is exact format hh:mm:ss -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 15:18:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 15:18:37 +0000 Subject: [Bugs] [Bug 1672076] chrome / chromium crash on gluster, sqlite issue? In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672076 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amukherj at redhat.com Component|glusterd |core -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 15:22:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 15:22:09 +0000 Subject: [Bugs] [Bug 1673058] New: Network throughput usage increased x5 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673058 Bug ID: 1673058 Summary: Network throughput usage increased x5 Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: glusterd Severity: medium Assignee: bugs at gluster.org Reporter: jsecchiero at enter.eu CC: bugs at gluster.org Target Milestone: --- Classification: Community Created attachment 1527539 --> https://bugzilla.redhat.com/attachment.cgi?id=1527539&action=edit client throughput graph Description of problem: Client network throughput in OUT direction usage increased x5 after an upgrade from 3.11, 3.12 to 5.3 of the server. Now i have ~110Mbps of traffic in OUT direction for each client and on the server i have a total of ~1450Mbps for each gluster server. Watch the attachment for graph before/after upgrade network throughput. Version-Release number of selected component (if applicable): 5.3 How reproducible: upgrade from 3.11, 3.12 to 5.3 Steps to Reproduce: 1. https://docs.gluster.org/en/v3/Upgrade-Guide/upgrade_to_3.12/ 2. https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_5/ Actual results: Network throughput usage increased x5 Expected results: Just the features and the bugfix of the 5.3 release Cluster Information: 2 nodes with 1 volume with 2 distributed brick for each node Number of Peers: 1 Hostname: 10.2.0.180 Uuid: 368055db-9e90-433f-9a56-bfc1507a25c5 State: Peer in Cluster (Connected) Volume Information: Volume Name: storage_other Type: Distributed-Replicate Volume ID: 6857bf2b-c97d-4505-896e-8fbc24bd16e8 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.2.0.181:/mnt/storage-brick1/data Brick2: 10.2.0.180:/mnt/storage-brick1/data Brick3: 10.2.0.181:/mnt/storage-brick2/data Brick4: 10.2.0.180:/mnt/storage-brick2/data Options Reconfigured: nfs.disable: on transport.address-family: inet storage.fips-mode-rchecksum: on Status of volume: storage_other Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.2.0.181:/mnt/storage-brick1/data 49152 0 Y 1165 Brick 10.2.0.180:/mnt/storage-brick1/data 49152 0 Y 1149 Brick 10.2.0.181:/mnt/storage-brick2/data 49153 0 Y 1166 Brick 10.2.0.180:/mnt/storage-brick2/data 49153 0 Y 1156 Self-heal Daemon on localhost N/A N/A Y 1183 Self-heal Daemon on 10.2.0.180 N/A N/A Y 1166 Task Status of Volume storage_other ------------------------------------------------------------------------------ There are no active volume tasks -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 18:40:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 18:40:30 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 Artem Russakovskii changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(archon810 at gmail.c | |om) | --- Comment #19 from Artem Russakovskii --- (In reply to Nithya Balachandran from comment #18) Before I make this change, what is the expected effect on performance on a busy system with lots of small and large files (millions)? Also, is it possible to set the lru-limit to 0 via gluster volume options (network.inode-lru-limit?) or will it be sufficiently different from using the fuse mount option? Do gluster docs have clear instructions on how to capture a core dump because I haven't found any other than the instructions to provide one (but not how) in case of a crash? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 19:30:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:30:30 +0000 Subject: [Bugs] [Bug 1636502] Unwanted repeated logs in self heal daemon logs In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636502 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21366 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 19:30:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:30:53 +0000 Subject: [Bugs] [Bug 1494529] Missing manpages In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1494529 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21426 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 19:31:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:31:38 +0000 Subject: [Bugs] [Bug 1614275] Fix spurious failures in tests/bugs/ec/bug-1236065.t In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1614275 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20685 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 19:32:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:32:21 +0000 Subject: [Bugs] [Bug 1635688] Keep only the valid (maintained/supported) components in the build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1635688 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21329 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 19:32:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:32:46 +0000 Subject: [Bugs] [Bug 1512691] PostgreSQL DB Restore: unexpected data beyond EOF In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1512691 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21035 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 6 19:33:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:33:55 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20690 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 19:34:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:34:19 +0000 Subject: [Bugs] [Bug 1626313] fix glfs_fini related problems In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1626313 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21218 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 19:34:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:34:42 +0000 Subject: [Bugs] [Bug 1626313] fix glfs_fini related problems In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1626313 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21217 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 19:35:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:35:48 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21328 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 19:36:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:36:59 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21298 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 19:37:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:37:23 +0000 Subject: [Bugs] [Bug 1622665] clang-scan report: glusterfs issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1622665 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21291 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 19:37:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 19:37:51 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21149 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 6 22:13:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 06 Feb 2019 22:13:59 +0000 Subject: [Bugs] [Bug 1660225] geo-rep does not replicate mv or rename of file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660225 perplexed767 changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(sunkumar at redhat.c | |om) --- Comment #7 from perplexed767 --- (In reply to Sunny Kumar from comment #1) > Hi, > > Please share- 1. gluster log 2. geo-replication log from master and 3. mount > log from slave. > > - Sunny Is there any news on this bug? It has been over a month since I filed it. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 7 03:01:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 03:01:01 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #20 from Nithya Balachandran --- (In reply to Artem Russakovskii from comment #19) > (In reply to Nithya Balachandran from comment #18) > > Before I make this change, what is the expected effect on performance on a > busy system with lots of small and large files (millions)? This turns off a feature that was introduced in 5.3 to clear unused inodes from memory for the fuse client gluster process and prevent the memory usage from rising. Turning it off would cause the client to behave the same as in the earlier releases. > > Also, is it possible to set the lru-limit to 0 via gluster volume options > (network.inode-lru-limit?) or will it be sufficiently different from using > the fuse mount option? Not at the moment. This is a mount option for the gluster fuse client. This is not the same as the network.inode-lru-limit option which controls the inodes in the brick processes. > > Do gluster docs have clear instructions on how to capture a core dump > because I haven't found any other than the instructions to provide one (but > not how) in case of a crash? To configure coredumps, you need to specify the size and location. The following link describes it well: https://stackoverflow.com/questions/17965/how-to-generate-a-core-dump-in-linux-on-a-segmentation-fault Make sure you have sufficient space in the location for a large coredump (it can run into several GBs) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 7 03:31:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 03:31:46 +0000 Subject: [Bugs] [Bug 1672727] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22065 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 03:31:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 03:31:48 +0000 Subject: [Bugs] [Bug 1672727] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-07 03:31:48 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22065 (Bump up timeout for tests on AWS) merged (#7) on master by Nigel Babu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 06:53:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 06:53:41 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #7 from Nithya Balachandran --- I will take a look and get back next week. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 07:48:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 07:48:57 +0000 Subject: [Bugs] [Bug 1673265] New: Fix timeouts so the tests pass on AWS Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673265 Bug ID: 1673265 Summary: Fix timeouts so the tests pass on AWS Product: GlusterFS Version: 4.1 Status: NEW Component: tests Assignee: bugs at gluster.org Reporter: nigelb at redhat.com CC: bugs at gluster.org Depends On: 1672727 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1672727 +++ Some test timeouts need a bump on AWS --- Additional comment from Worker Ant on 2019-02-05 17:21:47 UTC --- REVIEW: https://review.gluster.org/22065 (Bump up timeout for tests on AWS) posted (#5) for review on master by Nigel Babu --- Additional comment from Worker Ant on 2019-02-07 03:31:48 UTC --- REVIEW: https://review.gluster.org/22065 (Bump up timeout for tests on AWS) merged (#7) on master by Nigel Babu Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 [Bug 1672727] Fix timeouts so the tests pass on AWS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 07:48:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 07:48:57 +0000 Subject: [Bugs] [Bug 1672727] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1673265 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1673265 [Bug 1673265] Fix timeouts so the tests pass on AWS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 07:48:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 07:48:59 +0000 Subject: [Bugs] [Bug 1673267] New: Fix timeouts so the tests pass on AWS Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673267 Bug ID: 1673267 Summary: Fix timeouts so the tests pass on AWS Product: GlusterFS Version: 6 Status: NEW Component: tests Assignee: bugs at gluster.org Reporter: nigelb at redhat.com CC: bugs at gluster.org Depends On: 1672727 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1672727 +++ Some test timeouts need a bump on AWS --- Additional comment from Worker Ant on 2019-02-05 17:21:47 UTC --- REVIEW: https://review.gluster.org/22065 (Bump up timeout for tests on AWS) posted (#5) for review on master by Nigel Babu --- Additional comment from Worker Ant on 2019-02-07 03:31:48 UTC --- REVIEW: https://review.gluster.org/22065 (Bump up timeout for tests on AWS) merged (#7) on master by Nigel Babu Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 [Bug 1672727] Fix timeouts so the tests pass on AWS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 07:48:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 07:48:59 +0000 Subject: [Bugs] [Bug 1672727] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1673267 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1673267 [Bug 1673267] Fix timeouts so the tests pass on AWS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 07:49:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 07:49:19 +0000 Subject: [Bugs] [Bug 1673268] New: Fix timeouts so the tests pass on AWS Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673268 Bug ID: 1673268 Summary: Fix timeouts so the tests pass on AWS Product: GlusterFS Version: 5 Status: NEW Component: tests Assignee: bugs at gluster.org Reporter: nigelb at redhat.com CC: bugs at gluster.org Depends On: 1672727 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1672727 +++ Some test timeouts need a bump on AWS --- Additional comment from Worker Ant on 2019-02-05 17:21:47 UTC --- REVIEW: https://review.gluster.org/22065 (Bump up timeout for tests on AWS) posted (#5) for review on master by Nigel Babu --- Additional comment from Worker Ant on 2019-02-07 03:31:48 UTC --- REVIEW: https://review.gluster.org/22065 (Bump up timeout for tests on AWS) merged (#7) on master by Nigel Babu Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 [Bug 1672727] Fix timeouts so the tests pass on AWS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 07:49:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 07:49:19 +0000 Subject: [Bugs] [Bug 1672727] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672727 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1673268 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1673268 [Bug 1673268] Fix timeouts so the tests pass on AWS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 08:02:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 08:02:01 +0000 Subject: [Bugs] [Bug 1673267] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673267 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22168 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 08:02:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 08:02:02 +0000 Subject: [Bugs] [Bug 1673267] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673267 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22168 (Bump up timeout for tests on AWS) posted (#1) for review on release-6 by Nigel Babu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 08:04:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 08:04:46 +0000 Subject: [Bugs] [Bug 1673268] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673268 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22169 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 08:04:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 08:04:47 +0000 Subject: [Bugs] [Bug 1673268] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673268 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22169 (Bump up timeout for tests on AWS) posted (#1) for review on release-5 by Nigel Babu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 08:09:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 08:09:49 +0000 Subject: [Bugs] [Bug 1673265] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673265 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22170 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 08:09:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 08:09:49 +0000 Subject: [Bugs] [Bug 1673265] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673265 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22170 (Bump up timeout for tests on AWS) posted (#1) for review on release-4.1 by Nigel Babu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 08:24:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 08:24:27 +0000 Subject: [Bugs] [Bug 1651584] [geo-rep]: validate the config checkpoint date and fail if it is not is exact format hh:mm:ss In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651584 Bug 1651584 depends on bug 1672979, which changed state. Bug 1672979 Summary: [geo-rep]: validate the config checkpoint date and fail if it is not is exact format hh:mm:ss https://bugzilla.redhat.com/show_bug.cgi?id=1672979 What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |DUPLICATE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 09:19:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 09:19:29 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22171 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 09:19:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 09:19:29 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |POST --- Comment #534 from Worker Ant --- REVIEW: https://review.gluster.org/22171 (glusterd: improve logging) posted (#1) for review on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 10:57:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 10:57:13 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22157 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 10:57:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 10:57:14 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #535 from Worker Ant --- REVIEW: https://review.gluster.org/22157 (fuse: correctly handle setxattr values) merged (#6) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 12:22:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 12:22:27 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22159 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 12:22:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 12:22:28 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #536 from Worker Ant --- REVIEW: https://review.gluster.org/22159 (api: Update all future API versions to rel-6) merged (#3) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 12:26:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 12:26:50 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22173 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 12:26:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 12:26:51 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22173 (api: Update all future API versions to rel-6) posted (#1) for review on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 12:53:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 12:53:35 +0000 Subject: [Bugs] [Bug 1668995] DHT: Provide a virtual xattr to get the hash subvol for a file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668995 Sachin P Mali changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |smali at redhat.com --- Comment #3 from Sachin P Mali --- In release stakeholders meeting it was agreed to push this bug out of proposed list of 3.4.4, and to be considered for a future batch update. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 7 12:55:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 12:55:02 +0000 Subject: [Bugs] [Bug 1657163] Stack overflow in readdirp with parallel-readdir enabled In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657163 Sachin P Mali changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |smali at redhat.com --- Comment #7 from Sachin P Mali --- In release stakeholders meeting it was agreed to push this bug out of proposed list of 3.4.4, and to be considered for a future batch update. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 7 12:58:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 12:58:08 +0000 Subject: [Bugs] [Bug 1644681] Issuing a "heal ... full" on a disperse volume causes permanent high CPU utilization. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1644681 Sachin P Mali changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |smali at redhat.com --- Comment #7 from Sachin P Mali --- In release stakeholders meeting it was agreed to push this bug out of proposed list of 3.4 BU4, and to be considered for a future batch update. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 7 13:02:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 13:02:08 +0000 Subject: [Bugs] [Bug 1426044] read-ahead not working if open-behind is turned on In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1426044 Sachin P Mali changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |smali at redhat.com --- Comment #11 from Sachin P Mali --- In release stakeholders meeting it was agreed to push this bug out of proposed list of 3.4 BU4, and to be considered for a future batch update. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 7 14:17:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 14:17:23 +0000 Subject: [Bugs] [Bug 1673267] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673267 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-07 14:17:23 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22168 (Bump up timeout for tests on AWS) merged (#1) on release-6 by Nigel Babu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 14:17:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 14:17:44 +0000 Subject: [Bugs] [Bug 1673268] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673268 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-07 14:17:44 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22169 (Bump up timeout for tests on AWS) merged (#1) on release-5 by Nigel Babu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 7 17:41:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 17:41:17 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 David E. Smith changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(desmith at wustl.edu | |) | --- Comment #9 from David E. Smith --- As of this morning, I've added the lru-limit mount option to /etc/fstab on my servers. Was on vacation, didn't see the request until this morning. For the sake of reference, here's the full fstab lines, edited only to remove hostnames and add placeholders. (Note that I've never had a problem with the 'web-config' volume, which is very low-traffic and only a few megabytes in size; the problems always are the much more heavily-used 'web-content' volume.) gluster-server-1:/web-config /etc/httpd/conf.d glusterfs defaults,_netdev,noatime,context=unconfined_u:object_r:httpd_config_t:s0,backupvolfile-server=gluster-server-2,direct-io-mode=disable,lru-limit=0 0 0 gluster-server-1:/web-content /var/www/html glusterfs defaults,_netdev,noatime,context=unconfined_u:object_r:httpd_sys_rw_content_t:s0,backupvolfile-server=gluster-server-2,direct-io-mode=disable,lru-limit=0 0 0 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 7 17:58:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 07 Feb 2019 17:58:26 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #10 from David E. Smith --- Ran a couple of the glusterfs logs through the print-backtrace script. They all start with what you'd normally expect (clone, start_thread) and all end with (_gf_msg_backtrace_nomem) but they're all doing different things in the middle. It looks sorta like a memory leak or other memory corruption. Since it started happening on both of my servers after upgrading to 5.2 (and continued with 5.3), I really doubt it's a hardware issue -- the FUSE clients are both VMs, on hosts a few miles apart, so the odds of host RAM going wonky in both places at exactly that same time are ridiculous. Bit of a stretch, but do you think there would be value in my rebuilding the RPMs locally, to try to rule out anything on CentOS' end? /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] __restore_rt ??:0 /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] __inode_ctx_free ??:0 /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] inode_table_prune ??:0 /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] inode_forget_with_unref ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] fuse_forget ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] fuse_thread_proc ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] __clone ??:0 /lib64/libglusterfs.so.0(+0x26610)[0x7f36aff72610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f36aff7cb84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f36ae5d6280] __restore_rt ??:0 /lib64/libglusterfs.so.0(+0x36779)[0x7f36aff82779] __inode_unref ??:0 /lib64/libglusterfs.so.0(inode_unref+0x23)[0x7f36aff83203] inode_unref ??:0 /lib64/libglusterfs.so.0(gf_dirent_entry_free+0x2b)[0x7f36aff9ec4b] gf_dirent_entry_free ??:0 /lib64/libglusterfs.so.0(gf_dirent_free+0x2b)[0x7f36aff9ecab] gf_dirent_free ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x7480)[0x7f36a215b480] afr_readdir_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x60bca)[0x7f36a244dbca] client4_0_readdirp_cbk ??:0 /lib64/libgfrpc.so.0(+0xec70)[0x7f36afd3ec70] rpc_clnt_handle_reply ??:0 /lib64/libgfrpc.so.0(+0xf043)[0x7f36afd3f043] rpc_clnt_notify ??:0 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f36afd3af23] rpc_transport_notify ??:0 /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f36a492737b] socket_event_handler ??:0 /lib64/libglusterfs.so.0(+0x8aa49)[0x7f36affd6a49] event_dispatch_epoll_worker ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f36aedd5dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f36ae69dead] __clone ??:0 /lib64/libglusterfs.so.0(+0x26610)[0x7f7e13de0610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f7e13deab84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f7e12444280] __restore_rt ??:0 /lib64/libpthread.so.0(pthread_mutex_lock+0x0)[0x7f7e12c45c30] pthread_mutex_lock ??:0 /lib64/libglusterfs.so.0(__gf_free+0x12c)[0x7f7e13e0bc3c] __gf_free ??:0 /lib64/libglusterfs.so.0(+0x368ed)[0x7f7e13df08ed] __dentry_unset ??:0 /lib64/libglusterfs.so.0(+0x36b2b)[0x7f7e13df0b2b] __inode_retire ??:0 /lib64/libglusterfs.so.0(+0x36885)[0x7f7e13df0885] __inode_unref ??:0 /lib64/libglusterfs.so.0(inode_forget_with_unref+0x36)[0x7f7e13df1f86] inode_forget_with_unref ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x857a)[0x7f7e0b1b657a] fuse_batch_forget ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7f7e0b1cdd7a] fuse_thread_proc ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f7e12c43dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f7e1250bead] __clone ??:0 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 8 03:03:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 03:03:20 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #11 from Nithya Balachandran --- (In reply to David E. Smith from comment #10) > Ran a couple of the glusterfs logs through the print-backtrace script. They > all start with what you'd normally expect (clone, start_thread) and all end > with (_gf_msg_backtrace_nomem) but they're all doing different things in the > middle. It looks sorta like a memory leak or other memory corruption. Since > it started happening on both of my servers after upgrading to 5.2 (and > continued with 5.3), I really doubt it's a hardware issue -- the FUSE > clients are both VMs, on hosts a few miles apart, so the odds of host RAM > going wonky in both places at exactly that same time are ridiculous. > > Bit of a stretch, but do you think there would be value in my rebuilding the > RPMs locally, to try to rule out anything on CentOS' end? I don't think so. My guess is there is an error somewhere in the client code when handling inodes. It was never hit earlier because we never freed the inodes before 5.3. With the new inode invalidation feature, we appear to be accessing inodes that were already freed. Did you see the same crashes in 5.2? If yes, something else might be going wrong. I had a look at the coredumps you sent - most don't have any symbols (strangely). Of the ones that do, it looks like memory corruption and accessing already freed inodes. There are a few people looking at it but this going to take a while to figure out. In the meantime, let me know if you still see crashes with the lru-limit option. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 8 03:18:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 03:18:00 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #12 from Nithya Balachandran --- Another user has just reported that the lru-limit did not help with the crashes - let me know if that is your experience as well. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 8 07:19:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 07:19:55 +0000 Subject: [Bugs] [Bug 1672826] Request gerrit dashboard addition for release 6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672826 Deepshikha khandelwal changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dkhandel at redhat.com --- Comment #1 from Deepshikha khandelwal --- Dashboard for release-6 is now on Gerrit. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 09:01:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 09:01:07 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 8 10:31:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 10:31:53 +0000 Subject: [Bugs] [Bug 1672826] Request gerrit dashboard addition for release 6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672826 Deepshikha khandelwal changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-08 10:31:53 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 10:34:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 10:34:03 +0000 Subject: [Bugs] [Bug 1672828] Restrict gerrit merge permissions for branch release-6 to release owners In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672828 Deepshikha khandelwal changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED CC| |dkhandel at redhat.com Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-08 10:34:03 --- Comment #1 from Deepshikha khandelwal --- Done. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 12:00:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 12:00:31 +0000 Subject: [Bugs] [Bug 1673265] Fix timeouts so the tests pass on AWS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673265 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-08 12:00:31 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22170 (Bump up timeout for tests on AWS) merged (#1) on release-4.1 by Nigel Babu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 12:14:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 12:14:57 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22178 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 8 12:14:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 12:14:58 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #12 from Worker Ant --- REVIEW: https://review.gluster.org/22178 (mount/fuse: fix bug related to --auto-invalidation in mount script) posted (#1) for review on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 8 12:57:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 12:57:50 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(desmith at wustl.edu | |) --- Comment #13 from Nithya Balachandran --- We have found the cause of one crash but that has a different backtrace. Unfortunately we have not managed to reproduce the one you reported so we don't know if it is the same cause. Can you disable write-behind on the volume and let us know if it solves the problem? If yes, it is likely to be the same issue. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 8 15:46:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 15:46:17 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #537 from Worker Ant --- REVIEW: https://review.gluster.org/22171 (glusterd: improve logging) merged (#2) on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 15:48:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 15:48:08 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22173 (api: Update all future API versions to rel-6) merged (#2) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 15:54:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 15:54:00 +0000 Subject: [Bugs] [Bug 1673972] New: insufficient logging in glusterd_resolve_all_bricks Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673972 Bug ID: 1673972 Summary: insufficient logging in glusterd_resolve_all_bricks Product: GlusterFS Version: 6 Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: amukherj at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: In case, during glusterd restart in case host resolution fails, it's difficult to identify for which brick and the volume the resolution has failed. Until and unless glusterd is run in debug mode the sufficient logs can't be captured. This bug is to track this missing information in the log. Version-Release number of selected component (if applicable): release-6 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 15:54:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 15:54:43 +0000 Subject: [Bugs] [Bug 1673972] insufficient logging in glusterd_resolve_all_bricks In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673972 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672818 (glusterfs-6.0) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 15:54:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 15:54:43 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1673972 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1673972 [Bug 1673972] insufficient logging in glusterd_resolve_all_bricks -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 16:05:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 16:05:40 +0000 Subject: [Bugs] [Bug 1673972] insufficient logging in glusterd_resolve_all_bricks In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673972 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22181 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 8 16:05:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 08 Feb 2019 16:05:41 +0000 Subject: [Bugs] [Bug 1673972] insufficient logging in glusterd_resolve_all_bricks In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673972 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22181 (glusterd: improve logging) posted (#1) for review on release-6 by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 07:33:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 07:33:57 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22182 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 07:33:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 07:33:57 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #538 from Worker Ant --- REVIEW: https://review.gluster.org/22182 (fuse: reflect the actual default for lru-limit option) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 08:24:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 08:24:04 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22183 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 08:24:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 08:24:05 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #6 from Worker Ant --- REVIEW: https://review.gluster.org/22183 (inode: create inode outside locked region) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 08:25:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 08:25:07 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22184 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 08:25:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 08:25:08 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #7 from Worker Ant --- REVIEW: https://review.gluster.org/22184 (inode: make critical section smaller) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 11:51:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 11:51:33 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22185 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 11:51:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 11:51:34 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #8 from Worker Ant --- REVIEW: https://review.gluster.org/22185 (inode: dentry_destroy outside of dentry_unset) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 16:07:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 16:07:08 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 David E. Smith changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(desmith at wustl.edu |needinfo?(nbalacha at redhat.c |) |om) --- Comment #14 from David E. Smith --- I did have some crashes with 5.2. (I went from 3.something straight to 5.2, so I'm not going to be too helpful in terms of narrowing down exactly when this issue first appeared, sorry.) I'll see if I still have any of those cores; they all were from several weeks ago, so I may have already cleaned them up. This morning, one of my clients core dumped with the lru-limit option. It looks like it might be a different crash (in particular, this morning's crash was a SIGABRT, whereas previous crashes were SIGSEGV). I've uploaded that core to the same Box folder, in case it's useful. I'll paste its backtrace in below. For the write-behind request, do you want me to set 'performance.flush-behind off' or so you mean something else? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 9 16:07:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 16:07:49 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #15 from David E. Smith --- Backtrace for 2/9/19 crash (as promised above, put it in a separate comment for clarity): /lib64/libglusterfs.so.0(+0x26610)[0x7f3b31456610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f3b31460b84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f3b2faba280] __restore_rt ??:0 /lib64/libc.so.6(gsignal+0x37)[0x7f3b2faba207] raise ??:0 /lib64/libc.so.6(abort+0x148)[0x7f3b2fabb8f8] abort ??:0 /lib64/libc.so.6(+0x78d27)[0x7f3b2fafcd27] __libc_message ??:0 /lib64/libc.so.6(+0x81489)[0x7f3b2fb05489] _int_free ??:0 /lib64/libglusterfs.so.0(+0x1a6e9)[0x7f3b3144a6e9] dict_destroy ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x8cf9)[0x7f3b23388cf9] dht_local_wipe ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x4ab90)[0x7f3b233cab90] dht_revalidate_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x709e5)[0x7f3b236a89e5] afr_lookup_done ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71198)[0x7f3b236a9198] afr_lookup_metadata_heal_check ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71cbb)[0x7f3b236a9cbb] afr_lookup_entry_heal ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71f99)[0x7f3b236a9f99] afr_lookup_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x616d2)[0x7f3b239326d2] client4_0_lookup_cbk ??:0 /lib64/libgfrpc.so.0(+0xec70)[0x7f3b31222c70] rpc_clnt_handle_reply ??:0 /lib64/libgfrpc.so.0(+0xf043)[0x7f3b31223043] rpc_clnt_notify ??:0 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f3b3121ef23] rpc_transport_notify ??:0 /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f3b25e0b37b] socket_event_handler ??:0 /lib64/libglusterfs.so.0(+0x8aa49)[0x7f3b314baa49] event_dispatch_epoll_worker ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f3b302b9dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f3b2fb81ead] __clone ??:0 [d -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 9 17:15:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 17:15:55 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #16 from Raghavendra G --- (In reply to David E. Smith from comment #14) > I did have some crashes with 5.2. (I went from 3.something straight to 5.2, > so I'm not going to be too helpful in terms of narrowing down exactly when > this issue first appeared, sorry.) I'll see if I still have any of those > cores; they all were from several weeks ago, so I may have already cleaned > them up. > > This morning, one of my clients core dumped with the lru-limit option. It > looks like it might be a different crash (in particular, this morning's > crash was a SIGABRT, whereas previous crashes were SIGSEGV). I've uploaded > that core to the same Box folder, in case it's useful. I'll paste its > backtrace in below. > > For the write-behind request, do you want me to set > 'performance.flush-behind off' or so you mean something else? gluster volume set performance.write-behind off -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 9 18:41:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 18:41:54 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #13 from Worker Ant --- REVIEW: https://review.gluster.org/22178 (mount/fuse: fix bug related to --auto-invalidation in mount script) merged (#2) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sun Feb 10 06:04:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 10 Feb 2019 06:04:18 +0000 Subject: [Bugs] [Bug 1674225] New: flooding of "dict is NULL" logging & crash of client process Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674225 Bug ID: 1674225 Summary: flooding of "dict is NULL" logging & crash of client process Product: GlusterFS Version: mainline Status: NEW Component: core Keywords: Triaged, ZStream Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: amgad.saleh at nokia.com, archon810 at gmail.com, atumball at redhat.com, bugs at gluster.org, vpvainio at iki.fi Depends On: 1671603, 1313567 Blocks: 1667103 (glusterfs-5.4) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1671603 +++ +++ This bug was initially created as a clone of Bug #1313567 +++ Description of problem: following logs flood the log files [2016-03-01 10:45:51.688339] W [dict.c:1282:dict_foreach_match] (-->/usr/lib64/libglusterfs.so.0(dict_foreach_match+0x65) [0x7ff139e1e5d5] -->/usr/lib64/glusterfs/3.7.8/xlator/features/index.so(+0x3950) [0x7ff12de49950] -->/usr/lib64/libglusterfs.so.0(dict_foreach_match+0xe1) [0x7ff139e1e651] ) 0-dict: dict|match|action is NULL [Invalid argument] Version-Release number of selected component (if applicable): glusterfs-3.7.8 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: I have seen the older one https://bugzilla.redhat.com/show_bug.cgi?id=1289893 but since i am using the latest version (3.7.8) fix is there. Could this be related to another part of index.c ? --- Additional comment from Red Hat Bugzilla Rules Engine on 2016-03-01 23:14:38 UTC --- This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs?3.1.z' to '?'. If this bug should be proposed for a different release, please manually change the proposed release flag. --- Additional comment from Nithya Balachandran on 2016-03-09 04:16:05 UTC --- This looks like it refers to an upstream release (3.7.8). Changing the product to reflect this. --- Additional comment from evangelos on 2016-03-09 09:18:21 UTC --- is there any update on this ? thank you very much! --- Additional comment from Nithya Balachandran on 2016-03-09 10:43:09 UTC --- Moving this to Anuradha who worked on the original patch. --- Additional comment from evangelos on 2016-04-12 11:20:00 UTC --- is there any update on this ? --- Additional comment from Anuradha on 2016-06-23 10:01:53 UTC --- Hi evangelos, That fix was made in 3.7.5. You say you have been the old issue. Did you upgrade from 3.7.5 to 3.7.8 and are seeing problem or was this volume freshly created based on 3.7.8? As far as I know all the fixes for dict is NULL in index translator are sent. But there is an issue when volfiles are not updated during an upgrade. If you had upgraded the volume, could you please provided the steps that you used to upgrade? Also, could you also verify one thing for me from brick volfiles of you volume? The brick volfiles are supposed to have the following lines: volume test-index type features/index option xattrop-pending-watchlist trusted.afr.test- <--------(1) option xattrop-dirty-watchlist trusted.afr.dirty <--------(2) option index-base /export/test/brick2/.glusterfs/indices subvolumes test-barrier end-volume The two options mentioned above should exist. Otherwise you will see this problem. You can find volfiles at /var/lib/glusterd/vols/. Thanks. --- Additional comment from evangelos on 2016-07-07 19:28:59 UTC --- Hi Anuradha, in the mean time due to various issues we had we decided to downgrade to 3.6 branch so currently i do not have a 3.7 deployment up and running. But thx for the hint i will keep this in mind for the future. In the meantime you can close this bugzilla. thank you --- Additional comment from Anuradha on 2016-07-11 09:09:11 UTC --- Hi Evangelos, Thanks for the update. Closing this bug as per comment#7 Thanks, Anuradha. --- Additional comment from Emerson Gomes on 2019-01-27 15:42:59 UTC --- This error is still reproduceable in 5.3 when upgrading from a 3.x volume. I had to recreate volume from scratch in 5.3 and copy data back in order to avoid it. --- Additional comment from Artem Russakovskii on 2019-01-30 20:23:44 UTC --- I just started seeing this error after upgrading from 4.1 to 5.3. [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] And it floods like crazy with these. --- Additional comment from Emerson Gomes on 2019-01-30 20:33:12 UTC --- I "solved" the issue after upgrading to 5.3 by creating a new volume and moving all data to it. Apparently something is missing on the volumes after upgrade. --- Additional comment from Artem Russakovskii on 2019-01-30 20:37:13 UTC --- I just sent a message to the gluster mailing list about this because that's not how this problem should be resolved. I'm curious to hear what they say. --- Additional comment from Emerson Gomes on 2019-01-30 20:39:04 UTC --- Absolutely. That's the second big issue I had after upgrading. The first one is https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Still unsolved (open for more than 2 months now) --- Additional comment from Artem Russakovskii on 2019-01-30 20:40:29 UTC --- You know, I was *just* going to comment in a follow-up reply about whether the issue here is possibly related to the one you just linked. Seeing tons of those too, though at least the dupes are suppressed. ==> mnt-SITE_data1.log <== [2019-01-30 20:38:20.783713] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] ==> mnt-SITE_data3.log <== The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 413 times between [2019-01-30 20:36:23.881090] and [2019-01-30 20:38:20.015593] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0" repeated 42 times between [2019-01-30 20:36:23.290287] and [2019-01-30 20:38:20.280306] ==> mnt-SITE_data1.log <== The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0" repeated 50 times between [2019-01-30 20:36:22.247367] and [2019-01-30 20:38:19.459789] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 2654 times between [2019-01-30 20:36:22.667327] and [2019-01-30 20:38:20.546355] [2019-01-30 20:38:21.492319] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0 ==> mnt-SITE_data3.log <== [2019-01-30 20:38:22.349689] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0 ==> mnt-SITE_data1.log <== [2019-01-30 20:38:22.762941] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler --- Additional comment from Emerson Gomes on 2019-01-30 20:48:52 UTC --- Yeah, both arrised after upgrading from 3.x to 5.1, persisting in 5.2 and 5.3. The other issue is even more critical. It causes crashes, making the mount point being inacessible ("Transport endpoint is not connected" error) - Requiring a new manual umount/mount. For now I have a crontab entry doing this, but I will have to downgrade if a fix is not issued soon... --- Additional comment from Artem Russakovskii on 2019-01-31 18:00:40 UTC --- Damn, you weren't kidding, I wish I saw these bug reports before I updated from rock solid 4.1. Less than 24 hours after upgrading, I already got a crash that you referenced: [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3" repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31 09:38:03.958061] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 72 times between [2019-01-31 09:37:53.746741] and [2019-01-31 09:38:04.696993] pending frames: frame : type(1) op(READ) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-31 09:38:04 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6] /lib64/libc.so.6(+0x36160)[0x7fccd622d160] /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0] /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa] /lib64/libc.so.6(+0x2e772)[0x7fccd6225772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3] /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559] /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f] --------- --- Additional comment from Artem Russakovskii on 2019-02-02 20:15:09 UTC --- The fuse crash happened again yesterday, to another volume. Are there any mount options that could help mitigate this? In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch and restart the mount, which works and recovers the mount point within a minute. Not ideal, but a temporary workaround. By the way, the way to reproduce this "Transport endpoint is not connected" condition for testing purposes is to kill -9 the right "glusterfs --process-name fuse" process. monit check: check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 start program = "/bin/mount /mnt/glusterfs_data1" stop program = "/bin/umount /mnt/glusterfs_data1" if space usage > 90% for 5 times within 15 cycles then alert else if succeeded for 10 cycles then alert stack trace: [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 26 times between [2019-02-01 23:21:20.857333] and [2019-02-01 23:21:56.164427] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-01 23:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 [Bug 1313567] flooding of "dict is NULL" logging https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker https://bugzilla.redhat.com/show_bug.cgi?id=1671603 [Bug 1671603] flooding of "dict is NULL" logging & crash of client process -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Feb 10 06:04:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 10 Feb 2019 06:04:18 +0000 Subject: [Bugs] [Bug 1671603] flooding of "dict is NULL" logging & crash of client process In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671603 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1674225 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1674225 [Bug 1674225] flooding of "dict is NULL" logging & crash of client process -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Feb 10 06:04:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 10 Feb 2019 06:04:18 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1674225 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1674225 [Bug 1674225] flooding of "dict is NULL" logging & crash of client process -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sun Feb 10 06:04:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 10 Feb 2019 06:04:18 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1674225 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1674225 [Bug 1674225] flooding of "dict is NULL" logging & crash of client process -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 02:29:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 02:29:10 +0000 Subject: [Bugs] [Bug 1673058] Network throughput usage increased x5 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673058 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amukherj at redhat.com Component|glusterd |core -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 03:14:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 03:14:01 +0000 Subject: [Bugs] [Bug 1624724] ctime: Enable ctime feature by default and also improve usability by providing single option to enable In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1624724 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21975 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 11 04:44:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 04:44:08 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nbalacha at redhat.c | |om) | --- Comment #17 from Nithya Balachandran --- Thanks David. I'm going to hold off on looking at the coredump until we hear back from you on whether disabling performance.write-behind works. The different backtraces could be symptoms of the same underlying issue where gluster tries to access already freed memory. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 11 04:48:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 04:48:58 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22186 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 04:48:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 04:48:59 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #9 from Worker Ant --- REVIEW: https://review.gluster.org/22186 (inode: don't take lock on whole table during ref/unref) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 04:59:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 04:59:42 +0000 Subject: [Bugs] [Bug 1674348] New: Unable to access https://logs.aws.gluster.org/ Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674348 Bug ID: 1674348 Summary: Unable to access https://logs.aws.gluster.org/ Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: Unable to get the logs for the upstream regression runs from https://logs.aws.gluster.org/. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 06:38:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 06:38:49 +0000 Subject: [Bugs] [Bug 1674364] New: glusterfs-fuse client not benefiting from page cache on read after write Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 Bug ID: 1674364 Summary: glusterfs-fuse client not benefiting from page cache on read after write Product: GlusterFS Version: 6 Hardware: x86_64 OS: Linux Status: NEW Component: fuse Keywords: Performance Severity: high Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: bugs at gluster.org Depends On: 1664934 Blocks: 1670710, 1672818 (glusterfs-6.0) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1664934 +++ Description of problem: On a simple single brick distribute volume, I'm running tests to validate glusterfs-fuse client's use of page cache. The tests are indicating that a read following a write is reading from the brick, not from client cache. In contrast, a 2nd read gets data from the client cache. Version-Release number of selected component (if applicable): glusterfs-*5.2-1.el7.x86_64 kernel-3.10.0-957.el7.x86_64 (RHEL 7.6) How reproducible: Consistently Steps to Reproduce: 1. use fio to create a data set that would fit easily in the page cache. My client has 128 GB RAM; I'll create a 64 GB data set: fio --name=initialwrite --ioengine=sync --rw=write \ --direct=0 --create_on_open=1 --end_fsync=1 --bs=128k \ --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum \ --filesize=16g --size=16g --numjobs=4 2. run an fio read test that reads the data set from step 1, without invalidating the page cache: fio --name=readtest --ioengine=sync --rw=read --invalidate=0 \ --direct=0 --bs=128k --directory=/mnt/glustervol/ \ --filename_format=f.\$jobnum.\$filenum --filesize=16g \ --size=16g --numjobs=4 Read throughput is much lower than it would be if reading from page cache: READ: bw=573MiB/s (601MB/s), 143MiB/s-144MiB/s (150MB/s-150MB/s), io=64.0GiB (68.7GB), run=114171-114419msec Reads are going over the 10GbE network as shown in (edited) sar output: 05:01:04 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s 05:01:06 AM em1 755946.26 40546.26 1116287.75 3987.24 0.00 [There is some read amplification here: application is getting lower throughput than what client is reading over the n/w. More on that later] 3. Run the read test in step 2 again. This time read throughput is really high, indicating read from cache, rather than over the network: READ: bw=14.8GiB/s (15.9GB/s), 3783MiB/s-4270MiB/s (3967MB/s-4477MB/s), io=64.0GiB (68.7GB), run=3837-4331msec Expected results: The read test in step 2 should be reading from page cache, and should be giving throughput close to what we get in step 3. Additional Info: gluster volume info: Volume Name: perfvol Type: Distribute Volume ID: 7033539b-0331-44b1-96cf-46ddc6ee2255 Status: Started Snapshot Count: 0 Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 172.16.70.128:/mnt/rhs_brick1 Options Reconfigured: transport.address-family: inet nfs.disable: on --- Additional comment from Manoj Pillai on 2019-01-10 05:43:53 UTC --- (In reply to Manoj Pillai from comment #0) [...] > 1. use fio to create a data set that would fit easily in the page cache. My > client has 128 GB RAM; I'll create a 64 GB data set: > > fio --name=initialwrite --ioengine=sync --rw=write \ > --direct=0 --create_on_open=1 --end_fsync=1 --bs=128k \ > --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum \ > --filesize=16g --size=16g --numjobs=4 > Memory usage on the client while the write test is running: # sar -r 5 Linux 3.10.0-957.el7.x86_64 (c09-h08-r630.rdu.openstack.engineering.redhat.com) 01/10/2019 _x86_64_ (56 CPU) 05:35:36 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 05:35:41 AM 126671972 4937712 3.75 0 2974352 256704 0.18 1878020 1147776 36 05:35:46 AM 126671972 4937712 3.75 0 2974352 256704 0.18 1878020 1147776 36 05:35:51 AM 126666904 4942780 3.76 0 2974324 259900 0.19 1879948 1147772 16 05:35:56 AM 126665820 4943864 3.76 0 2974348 261300 0.19 1880304 1147776 24 05:36:01 AM 126663136 4946548 3.76 0 2974348 356356 0.25 1881500 1147772 20 05:36:06 AM 126663028 4946656 3.76 0 2974348 356356 0.25 1881540 1147772 20 05:36:11 AM 126664444 4945240 3.76 0 2974388 356356 0.25 1880648 1147788 32 05:36:16 AM 126174984 5434700 4.13 0 3449508 930284 0.66 1892912 1622536 32 05:36:21 AM 120539884 11069800 8.41 0 9076076 930284 0.66 1893784 7247852 32 05:36:26 AM 114979592 16630092 12.64 0 14620932 930284 0.66 1893796 12793472 32 05:36:31 AM 109392488 22217196 16.88 0 20192112 930284 0.66 1893796 18365764 32 05:36:36 AM 104113900 27495784 20.89 0 25457272 930284 0.66 1895152 23630336 32 05:36:41 AM 98713688 32895996 25.00 0 30842800 930284 0.66 1895156 29015400 32 05:36:46 AM 93355560 38254124 29.07 0 36190264 930688 0.66 1897548 34361664 32 05:36:51 AM 87640900 43968784 33.41 0 41885972 930688 0.66 1897556 40057860 32 05:36:56 AM 81903068 49706616 37.77 0 47626388 930688 0.66 1897004 45798848 0 05:37:01 AM 76209860 55399824 42.09 0 53303272 930688 0.66 1897004 51475716 0 05:37:06 AM 70540340 61069344 46.40 0 58956264 930688 0.66 1897004 57128836 0 05:37:11 AM 64872776 66736908 50.71 0 64609648 930688 0.66 1897000 62782624 0 05:37:16 AM 59376144 72233540 54.88 0 70096880 930688 0.66 1897368 68270084 0 05:37:21 AM 71333376 60276308 45.80 0 58169584 356740 0.25 1891388 56342848 0 05:37:26 AM 126653336 4956348 3.77 0 2974476 356740 0.25 1891392 1148348 0 05:37:31 AM 126654360 4955324 3.77 0 2974388 356740 0.25 1891380 1147784 0 05:37:36 AM 126654376 4955308 3.77 0 2974388 356740 0.25 1891380 1147784 0 05:37:41 AM 126654376 4955308 3.77 0 2974388 356740 0.25 1891380 1147784 0 So as the write test progresses, kbcached steadily increases. But looks like the cached data is dropped subsequently. --- Additional comment from Manoj Pillai on 2019-01-10 05:52:14 UTC --- When I run the same sequence of tests on an XFS file system on the server, I get expected results: both step 2. and step 3. of comment #0 report high read throughput (15+GiB/s) indicating data is read from the page cache. --- Additional comment from Manoj Pillai on 2019-01-10 11:01:23 UTC --- (In reply to Manoj Pillai from comment #0) [...] > > Read throughput is much lower than it would be if reading from page cache: > READ: bw=573MiB/s (601MB/s), 143MiB/s-144MiB/s (150MB/s-150MB/s), io=64.0GiB > (68.7GB), run=114171-114419msec > > Reads are going over the 10GbE network as shown in (edited) sar output: > 05:01:04 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s > 05:01:06 AM em1 755946.26 40546.26 1116287.75 3987.24 0.00 > > [There is some read amplification here: application is getting lower > throughput than what client is reading over the n/w. More on that later] > This turned out to be primarily read-ahead related. Open a new bug for it: https://bugzilla.redhat.com/show_bug.cgi?id=1665029. --- Additional comment from Raghavendra G on 2019-01-23 13:04:54 UTC --- >From preliminary tests I see two reasons for this: 1. inode-invalidations triggered by md-cache 2. Fuse auto invalidations With a hacky fix removing both of the above, I can see read after write being served from kernel page-cache. I'll update the bug with more details discussing validity/limitations with the above two approaches later. --- Additional comment from Manoj Pillai on 2019-01-24 04:43:40 UTC --- (In reply to Raghavendra G from comment #4) > From preliminary tests I see two reasons for this: > 1. inode-invalidations triggered by md-cache > 2. Fuse auto invalidations Trying with kernel NFS, another distributed fs solution. I see that cache is retained at the end of the write test, and both read-after-write and read-after-read are served from the page cache. In principle, if kNFS can do it, FUSE should be able to do it. I think :D. --- Additional comment from Worker Ant on 2019-01-29 03:15:45 UTC --- REVIEW: https://review.gluster.org/22109 (mount/fuse: expose fuse-auto-invalidation as a mount option) posted (#1) for review on master by Raghavendra G --- Additional comment from Raghavendra G on 2019-01-30 05:41:39 UTC --- (In reply to Manoj Pillai from comment #5) > (In reply to Raghavendra G from comment #4) > > From preliminary tests I see two reasons for this: > > 1. inode-invalidations triggered by md-cache > > 2. Fuse auto invalidations > > Trying with kernel NFS, another distributed fs solution. I see that cache is > retained at the end of the write test, and both read-after-write and > read-after-read are served from the page cache. > > In principle, if kNFS can do it, FUSE should be able to do it. I think :D. kNFS and FUSE have different invalidation policies. * kNFS provides close-to-open consistency. To quote from their FAQ [1] "Linux implements close-to-open cache consistency by comparing the results of a GETATTR operation done just after the file is closed to the results of a GETATTR operation done when the file is next opened. If the results are the same, the client will assume its data cache is still valid; otherwise, the cache is purged." For the workload used in this bz, file is not changed between close and open. Hence two values of stat fetched - at close and open - match and hence page-cache is retained. * FUSE auto-invalidation compares times of stats cached with the values got from the underlying filesystem implementation at all codepaths where stat is fetched. This means comparision happens in lookup, (f)stat, (f)setattr etc codepaths. Since (f)stat, lookup can happen asynchronously and concurrently wrt writes, they'll end up identifying delta between two values of stats resulting in cache purge. Please note that the consistency offered by FUSE is stronger than close-to-open consistency, which means it also provides close-to-open consistency along with consistency in codepaths like lookup, fstat etc. We have following options: * disable auto-invalidations and use glusterfs custom designed invalidation policy. The invalidation policy can be the same as NFS close-to-open consistency or something more stronger. * check whether the current form of auto-invalidation (though stricter) provides any added benefits to close-to-open consistency which are useful. If no, change FUSE auto-invalidation to close-to-open consistency. [1] http://nfs.sourceforge.net/#faq_a8 --- Additional comment from Raghavendra G on 2019-01-30 05:45:23 UTC --- Miklos, It would be helpful if you can comment on comment #7. regards, Raghavendra --- Additional comment from Raghavendra G on 2019-01-30 05:59:06 UTC --- Note that a lease based invalidation policy would be a complete solution, but it will take some time to implement that and get it working in Glusterfs. --- Additional comment from Worker Ant on 2019-02-02 03:08:22 UTC --- REVIEW: https://review.gluster.org/22109 (mount/fuse: expose auto-invalidation as a mount option) merged (#13) on master by Amar Tumballi --- Additional comment from Miklos Szeredi on 2019-02-04 09:53:18 UTC --- The underlying problem is that auto invalidate cannot differentiate local and remote modification based on mtime alone. What NFS apprently does is refresh attributes immediately after a write (not sure how often it does this, I guess not after each individual write). FUSE maybe should do this if auto invalidation is enabled, but if the filesystem can do its own invalidation, possibly based on better information than c/mtime, then that seem to be a better option. --- Additional comment from Worker Ant on 2019-02-08 12:14:58 UTC --- REVIEW: https://review.gluster.org/22178 (mount/fuse: fix bug related to --auto-invalidation in mount script) posted (#1) for review on master by Raghavendra G --- Additional comment from Worker Ant on 2019-02-09 18:41:54 UTC --- REVIEW: https://review.gluster.org/22178 (mount/fuse: fix bug related to --auto-invalidation in mount script) merged (#2) on master by Raghavendra G Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write https://bugzilla.redhat.com/show_bug.cgi?id=1670710 [Bug 1670710] glusterfs-fuse client not benefiting from page cache on read after write https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 06:38:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 06:38:49 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1674364 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 11 06:38:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 06:38:49 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1674364 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 06:45:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 06:45:04 +0000 Subject: [Bugs] [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22187 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 06:45:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 06:45:05 +0000 Subject: [Bugs] [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22187 (mount/fuse: fix bug related to --auto-invalidation in mount script) posted (#1) for review on release-6 by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 07:55:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 07:55:02 +0000 Subject: [Bugs] [Bug 1674348] Unable to access https://logs.aws.gluster.org/ In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674348 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED CC| |mscherer at redhat.com Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-11 07:55:02 --- Comment #1 from M. Scherer --- Fixed now -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:02:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:02:14 +0000 Subject: [Bugs] [Bug 1674389] New: [thin arbiter] : rpm - add thin-arbiter package Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674389 Bug ID: 1674389 Summary: [thin arbiter] : rpm - add thin-arbiter package Product: GlusterFS Version: mainline Status: NEW Component: replicate Assignee: bugs at gluster.org Reporter: aspandey at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: rpm - add thin-arbiter package Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:04:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:04:48 +0000 Subject: [Bugs] [Bug 1674389] [thin arbiter] : rpm - add thin-arbiter package In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674389 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22125 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:04:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:04:49 +0000 Subject: [Bugs] [Bug 1674389] [thin arbiter] : rpm - add thin-arbiter package In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674389 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22125 (rpm: add thin-arbiter package) posted (#5) for review on master by Ashish Pandey -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:36:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:36:12 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22188 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:36:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:36:13 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #10 from Worker Ant --- REVIEW: https://review.gluster.org/22188 (inode: do only required checks inside critical section.) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:36:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:36:55 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #8 from abhays --- (In reply to Nithya Balachandran from comment #5) > (In reply to abhays from comment #4) > > Thanks for the reply. > > > > Agreed @Nithya. > > The setup used by the clients are having same "endianness"(Big Endian). > > However, certain test cases fail on our Big Endian Systems which is our > > major concern. > > I think that is because the tests in question are assuming that the files > will exist on a certain brick based on the results we got while running them > on out little-endian systems. As the hash values are different on big-endian > systems, thos assumptions no longer hold. > > Do you have a list of all the tests that are failing and where they fail? I > can check to see if that is the case. @Nithya,The above test cases(except ./tests/bitrot/bug-1207627-bitrot-scrub-status.t) pass on big endian systems with the following change in "libglusterfs/src/hashfn.c":- diff --git a/libglusterfs/src/hashfn.c b/libglusterfs/src/hashfn.c index 62f7ab878..4e18144b8 100644 --- a/libglusterfs/src/hashfn.c +++ b/libglusterfs/src/hashfn.c @@ -10,7 +10,7 @@ #include #include - +#include #include "hashfn.h" #define get16bits(d) (*((const uint16_t *) (d))) @@ -45,7 +45,6 @@ uint32_t SuperFastHash (const char * data, int32_t len) { rem = len & 3; len >>= 2; - /* Main loop */ for (;len > 0; len--) { hash += get16bits (data); @@ -151,8 +150,9 @@ gf_dm_hashfn (const char *msg, int len) for (i = 0; i < full_quads; i++) { for (j = 0; j < 4; j++) { - word = *intmsg; - array[j] = word; + //word = *intmsg; + word = htole32(*intmsg); + array[j] = word; intmsg++; full_words--; full_bytes -= 4; @@ -162,8 +162,9 @@ gf_dm_hashfn (const char *msg, int len) for (j = 0; j < 4; j++) { if (full_words) { - word = *intmsg; - array[j] = word; + //word = *intmsg; + word = htole32(*intmsg); + array[j] = word; intmsg++; full_words--; full_bytes -= 4; This confirms that test cases were failing due to different hash value calculations on both the systems(little and big endian). Please, let us know when you have looked into the failures and got a fix. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:46:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:46:26 +0000 Subject: [Bugs] [Bug 1674406] New: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Bug ID: 1674406 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Product: GlusterFS Version: mainline Hardware: x86_64 OS: Linux Status: NEW Component: write-behind Severity: urgent Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: bugs at gluster.org Depends On: 1671556 Blocks: 1667103 (glusterfs-5.4) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1671556 +++ This is a re-post of my FUSE crash report from BZ1651246. That issue is for a crash in the FUSE client. Mine is too, but I was asked in that bug to open a new issue, so here you go. :) My servers (two, in a 'replica 2' setup) publish two volumes. One is Web site content, about 110GB; the other is Web config files, only a few megabytes. (Wasn't worth building extra servers for that second volume.) FUSE clients have been crashing on the larger volume every three or four days. I can't reproduce this on-demand, unfortunately, but I've got several cores from previous crashes that may be of value to you. I'm using Gluster 5.3 from the RPMs provided by the CentOS Storage SIG, on a Red Hat Enterprise Linux 7.x system. The client's logs show many hundreds of instances of this (I don't know if it's related): [2019-01-29 08:14:16.542674] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7384) [0x7fa171ead384] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xae3e) [0x7fa1720bee3e] -->/lib64/libglusterfs.so.0(dict_ref+0x5d) [0x7fa1809cc2ad] ) 0-dict: dict is NULL [Invalid argument] Then, when the client's glusterfs process crashes, this is logged: The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 871 times between [2019-01-29 08:12:48.390535] and [2019-01-29 08:14:17.100279] pending frames: frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2019-01-29 08:14:17 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] --------- Info on the volumes themselves, gathered from one of my servers: [davidsmith at wuit-s-10889 ~]$ sudo gluster volume info all Volume Name: web-config Type: Replicate Volume ID: 6c5dce6e-e64e-4a6d-82b3-f526744b463d Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-config Brick2: 172.23.128.27:/data/web-config Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet server.event-threads: 4 client.event-threads: 4 cluster.min-free-disk: 1 cluster.quorum-count: 2 cluster.quorum-type: fixed network.ping-timeout: 10 auth.allow: * performance.readdir-ahead: on Volume Name: web-content Type: Replicate Volume ID: fcabc15f-0cec-498f-93c4-2d75ad915730 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-content Brick2: 172.23.128.27:/data/web-content Options Reconfigured: network.ping-timeout: 10 cluster.quorum-type: fixed cluster.quorum-count: 2 performance.readdir-ahead: on auth.allow: * cluster.min-free-disk: 1 client.event-threads: 4 server.event-threads: 4 transport.address-family: inet nfs.disable: on performance.client-io-threads: off performance.cache-size: 4GB gluster> volume status all detail Status of volume: web-config ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5612 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5540 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 Status of volume: web-content ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5649 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5567 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 I'll attach a couple of the core files generated by the crashing glusterfs instances, size limits permitting (they range from 3 to 8 GB). If I can't attach them, I'll find somewhere to host them. --- Additional comment from Artem Russakovskii on 2019-01-31 22:26:25 UTC --- Also reposting my comment from https://bugzilla.redhat.com/show_bug.cgi?id=1651246. I wish I saw this bug report before I updated from rock solid 4.1 to 5.3. Less than 24 hours after upgrading, I already got a crash and had to unmount, kill gluster, and remount: [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3" repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31 09:38:03.958061] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 72 times between [2019-01-31 09:37:53.746741] and [2019-01-31 09:38:04.696993] pending frames: frame : type(1) op(READ) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-31 09:38:04 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6] /lib64/libc.so.6(+0x36160)[0x7fccd622d160] /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0] /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa] /lib64/libc.so.6(+0x2e772)[0x7fccd6225772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3] /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559] /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f] --------- Do the pending patches fix the crash or only the repeated warnings? I'm running glusterfs on OpenSUSE 15.0 installed via http://download.opensuse.org/repositories/home:/glusterfs:/Leap15-5/openSUSE_Leap_15.0/, not too sure how to make it core dump. If it's not fixed by the patches above, has anyone already opened a ticket for the crashes that I can join and monitor? This is going to create a massive problem for us since production systems are crashing. Thanks. --- Additional comment from David E. Smith on 2019-01-31 22:31:47 UTC --- Actually, I ran the cores through strings and grepped for a few things like passwords -- as you'd expect from a memory dump from a Web server, there's a log of sensitive information in there. Is there a safe/acceptable way to send the cores only to developers that can use them, or otherwise not have to make them publicly available while still letting the Gluster devs benefit from analyzing them? --- Additional comment from Ravishankar N on 2019-02-01 05:51:19 UTC --- (In reply to David E. Smith from comment #2) > Actually, I ran the cores through strings and grepped for a few things like > passwords -- as you'd expect from a memory dump from a Web server, there's a > log of sensitive information in there. Is there a safe/acceptable way to > send the cores only to developers that can use them, or otherwise not have > to make them publicly available while still letting the Gluster devs benefit > from analyzing them? Perhaps you could upload it to a shared Dropbox folder with view/download access to the red hat email IDs I've CC'ed to this email (including me) to begin with. Note: I upgraded a 1x2 replica volume with 1 fuse client from v4.1.7 to v5.3 and did some basic I/O (kernel untar and iozone) and did not observe any crashes, so maybe this this something that is hit under extreme I/O or memory pressure. :-( --- Additional comment from Artem Russakovskii on 2019-02-02 20:17:15 UTC --- The fuse crash happened again yesterday, to another volume. Are there any mount options that could help mitigate this? In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch and restart the mount, which works and recovers the mount point within a minute. Not ideal, but a temporary workaround. By the way, the way to reproduce this "Transport endpoint is not connected" condition for testing purposes is to kill -9 the right "glusterfs --process-name fuse" process. monit check: check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 start program = "/bin/mount /mnt/glusterfs_data1" stop program = "/bin/umount /mnt/glusterfs_data1" if space usage > 90% for 5 times within 15 cycles then alert else if succeeded for 10 cycles then alert stack trace: [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 26 times between [2019-02-01 23:21:20.857333] and [2019-02-01 23:21:56.164427] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-01 23:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] --- Additional comment from David E. Smith on 2019-02-05 02:59:24 UTC --- I've added the five of you to our org's Box account; all of you should have invitations to a shared folder, and I'm uploading a few of the cores now. I hope they're of value to you. The binaries are all from the CentOS Storage SIG repo at https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/ . They're all current as of a few days ago: [davidsmith at wuit-s-10882 ~]$ rpm -qa | grep gluster glusterfs-5.3-1.el7.x86_64 glusterfs-client-xlators-5.3-1.el7.x86_64 glusterfs-fuse-5.3-1.el7.x86_64 glusterfs-libs-5.3-1.el7.x86_64 --- Additional comment from Nithya Balachandran on 2019-02-05 11:00:04 UTC --- (In reply to David E. Smith from comment #5) > I've added the five of you to our org's Box account; all of you should have > invitations to a shared folder, and I'm uploading a few of the cores now. I > hope they're of value to you. > > The binaries are all from the CentOS Storage SIG repo at > https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/ . They're > all current as of a few days ago: > > [davidsmith at wuit-s-10882 ~]$ rpm -qa | grep gluster > glusterfs-5.3-1.el7.x86_64 > glusterfs-client-xlators-5.3-1.el7.x86_64 > glusterfs-fuse-5.3-1.el7.x86_64 > glusterfs-libs-5.3-1.el7.x86_64 Thanks. We will take a look and get back to you. --- Additional comment from Nithya Balachandran on 2019-02-05 16:43:45 UTC --- David, Can you try mounting the volume with the option lru-limit=0 and let us know if you still see the crashes? Regards, Nithya --- Additional comment from Nithya Balachandran on 2019-02-06 07:23:49 UTC --- Initial analysis of one of the cores: [root at rhgs313-7 gluster-5.3]# gdb -c core.6014 /usr/sbin/glusterfs [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --direct-io-mode=disable --fuse-mountopts=noatime,context="'. Program terminated with signal 11, Segmentation fault. #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 410 if (!xl->call_cleanup && xl->cbks->forget) (gdb) bt #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 #1 0x00007fa1809e90a2 in __inode_destroy (inode=0x7fa0d0349af8) at inode.c:432 #2 inode_table_prune (table=table at entry=0x7fa15800c3c0) at inode.c:1696 #3 0x00007fa1809e9f96 in inode_forget_with_unref (inode=0x7fa0d0349af8, nlookup=128) at inode.c:1273 #4 0x00007fa177dae4e1 in do_forget (this=, unique=, nodeid=, nlookup=) at fuse-bridge.c:726 #5 0x00007fa177dae5bd in fuse_forget (this=, finh=0x7fa0a41da500, msg=, iobuf=) at fuse-bridge.c:741 #6 0x00007fa177dc5d7a in fuse_thread_proc (data=0x557a0e8ffe20) at fuse-bridge.c:5125 #7 0x00007fa17f83bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa17f103ead in msync () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? () (gdb) f 0 #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 410 if (!xl->call_cleanup && xl->cbks->forget) (gdb) l 405 for (index = 0; index < inode->table->xl->graph->xl_count; index++) { 406 if (inode->_ctx[index].value1 || inode->_ctx[index].value2) { 407 xl = (xlator_t *)(long)inode->_ctx[index].xl_key; 408 old_THIS = THIS; 409 THIS = xl; 410 if (!xl->call_cleanup && xl->cbks->forget) 411 xl->cbks->forget(xl, inode); 412 THIS = old_THIS; 413 } 414 } (gdb) p *xl Cannot access memory at address 0x0 (gdb) p index $1 = 6 (gdb) p inode->table->xl->graph->xl_count $3 = 13 (gdb) p inode->_ctx[index].value1 $4 = 0 (gdb) p inode->_ctx[index].value2 $5 = 140327960119304 (gdb) p/x inode->_ctx[index].value2 $6 = 0x7fa0a6370808 Based on the graph, the xlator with index = 6 is (gdb) p ((xlator_t*) inode->table->xl->graph->top)->next->next->next->next->next->next->next->name $31 = 0x7fa16c0122e0 "web-content-read-ahead" (gdb) p ((xlator_t*) inode->table->xl->graph->top)->next->next->next->next->next->next->next->xl_id $32 = 6 But read-ahead does not update the inode_ctx at all. There seems to be some sort of memory corruption happening here but that needs further analysis. --- Additional comment from David E. Smith on 2019-02-07 17:41:17 UTC --- As of this morning, I've added the lru-limit mount option to /etc/fstab on my servers. Was on vacation, didn't see the request until this morning. For the sake of reference, here's the full fstab lines, edited only to remove hostnames and add placeholders. (Note that I've never had a problem with the 'web-config' volume, which is very low-traffic and only a few megabytes in size; the problems always are the much more heavily-used 'web-content' volume.) gluster-server-1:/web-config /etc/httpd/conf.d glusterfs defaults,_netdev,noatime,context=unconfined_u:object_r:httpd_config_t:s0,backupvolfile-server=gluster-server-2,direct-io-mode=disable,lru-limit=0 0 0 gluster-server-1:/web-content /var/www/html glusterfs defaults,_netdev,noatime,context=unconfined_u:object_r:httpd_sys_rw_content_t:s0,backupvolfile-server=gluster-server-2,direct-io-mode=disable,lru-limit=0 0 0 --- Additional comment from David E. Smith on 2019-02-07 17:58:26 UTC --- Ran a couple of the glusterfs logs through the print-backtrace script. They all start with what you'd normally expect (clone, start_thread) and all end with (_gf_msg_backtrace_nomem) but they're all doing different things in the middle. It looks sorta like a memory leak or other memory corruption. Since it started happening on both of my servers after upgrading to 5.2 (and continued with 5.3), I really doubt it's a hardware issue -- the FUSE clients are both VMs, on hosts a few miles apart, so the odds of host RAM going wonky in both places at exactly that same time are ridiculous. Bit of a stretch, but do you think there would be value in my rebuilding the RPMs locally, to try to rule out anything on CentOS' end? /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] __restore_rt ??:0 /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] __inode_ctx_free ??:0 /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] inode_table_prune ??:0 /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] inode_forget_with_unref ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] fuse_forget ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] fuse_thread_proc ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] __clone ??:0 /lib64/libglusterfs.so.0(+0x26610)[0x7f36aff72610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f36aff7cb84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f36ae5d6280] __restore_rt ??:0 /lib64/libglusterfs.so.0(+0x36779)[0x7f36aff82779] __inode_unref ??:0 /lib64/libglusterfs.so.0(inode_unref+0x23)[0x7f36aff83203] inode_unref ??:0 /lib64/libglusterfs.so.0(gf_dirent_entry_free+0x2b)[0x7f36aff9ec4b] gf_dirent_entry_free ??:0 /lib64/libglusterfs.so.0(gf_dirent_free+0x2b)[0x7f36aff9ecab] gf_dirent_free ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x7480)[0x7f36a215b480] afr_readdir_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x60bca)[0x7f36a244dbca] client4_0_readdirp_cbk ??:0 /lib64/libgfrpc.so.0(+0xec70)[0x7f36afd3ec70] rpc_clnt_handle_reply ??:0 /lib64/libgfrpc.so.0(+0xf043)[0x7f36afd3f043] rpc_clnt_notify ??:0 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f36afd3af23] rpc_transport_notify ??:0 /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f36a492737b] socket_event_handler ??:0 /lib64/libglusterfs.so.0(+0x8aa49)[0x7f36affd6a49] event_dispatch_epoll_worker ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f36aedd5dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f36ae69dead] __clone ??:0 /lib64/libglusterfs.so.0(+0x26610)[0x7f7e13de0610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f7e13deab84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f7e12444280] __restore_rt ??:0 /lib64/libpthread.so.0(pthread_mutex_lock+0x0)[0x7f7e12c45c30] pthread_mutex_lock ??:0 /lib64/libglusterfs.so.0(__gf_free+0x12c)[0x7f7e13e0bc3c] __gf_free ??:0 /lib64/libglusterfs.so.0(+0x368ed)[0x7f7e13df08ed] __dentry_unset ??:0 /lib64/libglusterfs.so.0(+0x36b2b)[0x7f7e13df0b2b] __inode_retire ??:0 /lib64/libglusterfs.so.0(+0x36885)[0x7f7e13df0885] __inode_unref ??:0 /lib64/libglusterfs.so.0(inode_forget_with_unref+0x36)[0x7f7e13df1f86] inode_forget_with_unref ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x857a)[0x7f7e0b1b657a] fuse_batch_forget ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7f7e0b1cdd7a] fuse_thread_proc ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f7e12c43dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f7e1250bead] __clone ??:0 --- Additional comment from Nithya Balachandran on 2019-02-08 03:03:20 UTC --- (In reply to David E. Smith from comment #10) > Ran a couple of the glusterfs logs through the print-backtrace script. They > all start with what you'd normally expect (clone, start_thread) and all end > with (_gf_msg_backtrace_nomem) but they're all doing different things in the > middle. It looks sorta like a memory leak or other memory corruption. Since > it started happening on both of my servers after upgrading to 5.2 (and > continued with 5.3), I really doubt it's a hardware issue -- the FUSE > clients are both VMs, on hosts a few miles apart, so the odds of host RAM > going wonky in both places at exactly that same time are ridiculous. > > Bit of a stretch, but do you think there would be value in my rebuilding the > RPMs locally, to try to rule out anything on CentOS' end? I don't think so. My guess is there is an error somewhere in the client code when handling inodes. It was never hit earlier because we never freed the inodes before 5.3. With the new inode invalidation feature, we appear to be accessing inodes that were already freed. Did you see the same crashes in 5.2? If yes, something else might be going wrong. I had a look at the coredumps you sent - most don't have any symbols (strangely). Of the ones that do, it looks like memory corruption and accessing already freed inodes. There are a few people looking at it but this going to take a while to figure out. In the meantime, let me know if you still see crashes with the lru-limit option. --- Additional comment from Nithya Balachandran on 2019-02-08 03:18:00 UTC --- Another user has just reported that the lru-limit did not help with the crashes - let me know if that is your experience as well. --- Additional comment from Nithya Balachandran on 2019-02-08 12:57:50 UTC --- We have found the cause of one crash but that has a different backtrace. Unfortunately we have not managed to reproduce the one you reported so we don't know if it is the same cause. Can you disable write-behind on the volume and let us know if it solves the problem? If yes, it is likely to be the same issue. --- Additional comment from David E. Smith on 2019-02-09 16:07:08 UTC --- I did have some crashes with 5.2. (I went from 3.something straight to 5.2, so I'm not going to be too helpful in terms of narrowing down exactly when this issue first appeared, sorry.) I'll see if I still have any of those cores; they all were from several weeks ago, so I may have already cleaned them up. This morning, one of my clients core dumped with the lru-limit option. It looks like it might be a different crash (in particular, this morning's crash was a SIGABRT, whereas previous crashes were SIGSEGV). I've uploaded that core to the same Box folder, in case it's useful. I'll paste its backtrace in below. For the write-behind request, do you want me to set 'performance.flush-behind off' or so you mean something else? --- Additional comment from David E. Smith on 2019-02-09 16:07:49 UTC --- Backtrace for 2/9/19 crash (as promised above, put it in a separate comment for clarity): /lib64/libglusterfs.so.0(+0x26610)[0x7f3b31456610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f3b31460b84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f3b2faba280] __restore_rt ??:0 /lib64/libc.so.6(gsignal+0x37)[0x7f3b2faba207] raise ??:0 /lib64/libc.so.6(abort+0x148)[0x7f3b2fabb8f8] abort ??:0 /lib64/libc.so.6(+0x78d27)[0x7f3b2fafcd27] __libc_message ??:0 /lib64/libc.so.6(+0x81489)[0x7f3b2fb05489] _int_free ??:0 /lib64/libglusterfs.so.0(+0x1a6e9)[0x7f3b3144a6e9] dict_destroy ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x8cf9)[0x7f3b23388cf9] dht_local_wipe ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x4ab90)[0x7f3b233cab90] dht_revalidate_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x709e5)[0x7f3b236a89e5] afr_lookup_done ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71198)[0x7f3b236a9198] afr_lookup_metadata_heal_check ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71cbb)[0x7f3b236a9cbb] afr_lookup_entry_heal ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71f99)[0x7f3b236a9f99] afr_lookup_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x616d2)[0x7f3b239326d2] client4_0_lookup_cbk ??:0 /lib64/libgfrpc.so.0(+0xec70)[0x7f3b31222c70] rpc_clnt_handle_reply ??:0 /lib64/libgfrpc.so.0(+0xf043)[0x7f3b31223043] rpc_clnt_notify ??:0 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f3b3121ef23] rpc_transport_notify ??:0 /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f3b25e0b37b] socket_event_handler ??:0 /lib64/libglusterfs.so.0(+0x8aa49)[0x7f3b314baa49] event_dispatch_epoll_worker ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f3b302b9dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f3b2fb81ead] __clone ??:0 [d --- Additional comment from Raghavendra G on 2019-02-09 17:15:55 UTC --- (In reply to David E. Smith from comment #14) > I did have some crashes with 5.2. (I went from 3.something straight to 5.2, > so I'm not going to be too helpful in terms of narrowing down exactly when > this issue first appeared, sorry.) I'll see if I still have any of those > cores; they all were from several weeks ago, so I may have already cleaned > them up. > > This morning, one of my clients core dumped with the lru-limit option. It > looks like it might be a different crash (in particular, this morning's > crash was a SIGABRT, whereas previous crashes were SIGSEGV). I've uploaded > that core to the same Box folder, in case it's useful. I'll paste its > backtrace in below. > > For the write-behind request, do you want me to set > 'performance.flush-behind off' or so you mean something else? gluster volume set performance.write-behind off --- Additional comment from Nithya Balachandran on 2019-02-11 04:44:08 UTC --- Thanks David. I'm going to hold off on looking at the coredump until we hear back from you on whether disabling performance.write-behind works. The different backtraces could be symptoms of the same underlying issue where gluster tries to access already freed memory. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker https://bugzilla.redhat.com/show_bug.cgi?id=1671556 [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:46:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:46:26 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1674406 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 11 09:46:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:46:26 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1674406 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:53:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:53:16 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22189 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:56:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:56:37 +0000 Subject: [Bugs] [Bug 1668259] Glusterfs 5.3 RPMs can't be build on rhel7 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668259 Niels de Vos changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED CC| |ndevos at redhat.com Resolution|--- |WORKSFORME Last Closed| |2019-02-11 09:56:37 --- Comment #1 from Niels de Vos --- The package was built successfully for CentOS 7: https://cbs.centos.org/koji/buildinfo?buildID=25022 You can consume the packages from the CentOS Storage SIG with these commands, no need to build your own (on a standard CentOS-7 system): # yum install centos-release-gluster # yum install glusterfs-server There is no intention to build el7 packages from the Fedora .spec file. Either use the .spec from the tarball, or from https://github.com/CentOS-Storage-SIG/glusterfs/tree/sig-storage6-gluster-5 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 09:59:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 09:59:32 +0000 Subject: [Bugs] [Bug 1674412] New: listing a file while writing to it causes deadlock Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674412 Bug ID: 1674412 Summary: listing a file while writing to it causes deadlock Product: GlusterFS Version: mainline Status: NEW Component: core Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Following test case was given by Nithya. Create a pure replicate volume and enable the following options: Volume Name: xvol Type: Replicate Volume ID: 095d6083-ea82-4ec9-a3a9-498fbd5f8dbe Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 192.168.122.7:/bricks/brick1/xvol-1 Brick2: 192.168.122.7:/bricks/brick1/xvol-2 Brick3: 192.168.122.7:/bricks/brick1/xvol-3 Options Reconfigured: server.event-threads: 4 client.event-threads: 4 performance.parallel-readdir: on performance.readdir-ahead: on transport.address-family: inet nfs.disable: on performance.client-io-threads: off Fuse mount using: mount -t glusterfs -o lru-limit=500 -s 192.168.122.7:/xvol /mnt/g1 mkdir /mnt/g1/dirdd >From terminal 1: cd /mnt/g1/dirdd while (true); do ls -lR dirdd; done >From terminal 2: while true; do dd if=/dev/urandom of=/mnt/g1/dirdd/1G.file bs=1M count=1; rm -f /mnt/g1/dirdd/1G.file; done On running this test, both dd and ls hang after some time. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 10:01:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 10:01:41 +0000 Subject: [Bugs] [Bug 1674412] listing a file while writing to it causes deadlock In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674412 --- Comment #1 from Raghavendra G --- (gdb) thr 8 [Switching to thread 8 (Thread 0x7f28072d1700 (LWP 26397))] #0 0x00007f2813a404cd in __lll_lock_wait () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007f2813a404cd in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f2813a3bdcb in _L_lock_812 () from /lib64/libpthread.so.0 #2 0x00007f2813a3bc98 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f2805e3122f in rda_inode_ctx_get_iatt (inode=0x7f27ec0010b8, this=0x7f2800012560, attr=0x7f28072d0700) at readdir-ahead.c:286 #4 0x00007f2805e3134d in __rda_fill_readdirp (ctx=0x7f27f800f290, request_size=, entries=0x7f28072d0890, this=0x7f2800012560) at readdir-ahead.c:326 #5 __rda_serve_readdirp (this=this at entry=0x7f2800012560, ctx=ctx at entry=0x7f27f800f290, size=size at entry=4096, entries=entries at entry=0x7f28072d0890, op_errno=op_errno at entry=0x7f28072d085c) at readdir-ahead.c:353 #6 0x00007f2805e32732 in rda_fill_fd_cbk (frame=0x7f27f801c1e8, cookie=, this=0x7f2800012560, op_ret=3, op_errno=2, entries=, xdata=0x0) at readdir-ahead.c:581 #7 0x00007f2806097447 in client4_0_readdirp_cbk (req=, iov=, count=, myframe=0x7f27f800f498) at client-rpc-fops_v2.c:2339 #8 0x00007f28149a29d1 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f2800051120, pollin=pollin at entry=0x7f280006a180) at rpc-clnt.c:755 #9 0x00007f28149a2d37 in rpc_clnt_notify (trans=0x7f28000513e0, mydata=0x7f2800051150, event=, data=0x7f280006a180) at rpc-clnt.c:922 #10 0x00007f281499f5e3 in rpc_transport_notify (this=this at entry=0x7f28000513e0, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f280006a180) at rpc-transport.c:542 #11 0x00007f2808d88f77 in socket_event_poll_in (notify_handled=true, this=0x7f28000513e0) at socket.c:2522 #12 socket_event_handler (fd=, idx=, gen=, data=0x7f28000513e0, poll_in=, poll_out=, poll_err=0, event_thread_died=0 '\000') at socket.c:2924 #13 0x00007f2814c5a926 in event_dispatch_epoll_handler (event=0x7f28072d0e80, event_pool=0x90d560) at event-epoll.c:648 #14 event_dispatch_epoll_worker (data=0x96f1e0) at event-epoll.c:762 #15 0x00007f2813a39dd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f2813302b3d in clone () from /lib64/libc.so.6 [Switching to thread 7 (Thread 0x7f2806ad0700 (LWP 26398))] #0 0x00007f2813a404cd in __lll_lock_wait () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007f2813a404cd in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f2813a3bdcb in _L_lock_812 () from /lib64/libpthread.so.0 #2 0x00007f2813a3bc98 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f2805e2cd85 in rda_mark_inode_dirty (this=this at entry=0x7f2800012560, inode=0x7f27ec009da8) at readdir-ahead.c:234 #4 0x00007f2805e2f3cc in rda_writev_cbk (frame=0x7f27f800ef48, cookie=, this=0x7f2800012560, op_ret=131072, op_errno=0, prebuf=0x7f2806acf870, postbuf=0x7f2806acf910, xdata=0x0) at readdir-ahead.c:769 #5 0x00007f2806094064 in client4_0_writev_cbk (req=, iov=, count=, myframe=0x7f27f801a7f8) at client-rpc-fops_v2.c:685 #6 0x00007f28149a29d1 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f2800051120, pollin=pollin at entry=0x7f27f8008320) at rpc-clnt.c:755 #7 0x00007f28149a2d37 in rpc_clnt_notify (trans=0x7f28000513e0, mydata=0x7f2800051150, event=, data=0x7f27f8008320) at rpc-clnt.c:922 #8 0x00007f281499f5e3 in rpc_transport_notify (this=this at entry=0x7f28000513e0, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f27f8008320) at rpc-transport.c:542 #9 0x00007f2808d88f77 in socket_event_poll_in (notify_handled=true, this=0x7f28000513e0) at socket.c:2522 #10 socket_event_handler (fd=, idx=, gen=, data=0x7f28000513e0, poll_in=, poll_out=, poll_err=0, event_thread_died=0 '\000') at socket.c:2924 #11 0x00007f2814c5a926 in event_dispatch_epoll_handler (event=0x7f2806acfe80, event_pool=0x90d560) at event-epoll.c:648 #12 event_dispatch_epoll_worker (data=0x96f4b0) at event-epoll.c:762 #13 0x00007f2813a39dd5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f2813302b3d in clone () from /lib64/libc.so.6 In writev and readdirp codepath inode and fd-ctx locks are acquired in opposite order causing a deadlock. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 10:59:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 10:59:15 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22190 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 10:59:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 10:59:16 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22190 (packagaing : check lower version for gluster-gnfs incase of upgrade) posted (#1) for review on master by jiffin tony Thottan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 11:05:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 11:05:52 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Sayalee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |saraut at redhat.com QA Contact|tdesala at redhat.com |saraut at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 11 11:07:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 11:07:13 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #11 from Worker Ant --- REVIEW: https://review.gluster.org/22183 (inode: create inode outside locked region) merged (#4) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 11:16:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 11:16:17 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22111 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 11 11:16:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 11:16:18 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #14 from Worker Ant --- REVIEW: https://review.gluster.org/22111 (performance/md-cache: introduce an option to control invalidation of inodes) merged (#18) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 11 11:17:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 11:17:57 +0000 Subject: [Bugs] [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22191 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 11:17:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 11:17:58 +0000 Subject: [Bugs] [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22191 (performance/md-cache: introduce an option to control invalidation of inodes) posted (#1) for review on release-6 by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 9 07:33:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 09 Feb 2019 07:33:59 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #539 from Worker Ant --- REVIEW: https://review.gluster.org/22182 (fuse: reflect the actual default for lru-limit option) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From Chandranana.Naik at ibm.com Mon Feb 11 10:15:52 2019 From: Chandranana.Naik at ibm.com (Chandranana Naik) Date: Mon, 11 Feb 2019 15:45:52 +0530 Subject: [Bugs] GlusterFs v4.1.5: Need help on bitrot detection Message-ID: Hi Team, We are working with Glusterfs v4.1.5 on big endian platform(Ubuntu 16.04) and encountered that the subtest 20 of test ./tests/bitrot/bug-1207627-bitrot-scrub-status.t is failing. Subtest 20 is failing as below: trusted.bit-rot.bad-file check_for_xattr trusted.bit-rot.bad-file //d/backends/patchy1/FILE1 not ok 20 Got "" instead of "trusted.bit-rot.bad-file", LINENUM:50 FAILED COMMAND: trusted.bit-rot.bad-file check_for_xattr trusted.bit-rot.bad-file //d/backends/patchy1/FILE1 The test is failing with error "remote operation failed [Cannot allocate memory]" logged in /var/log/glusterfs/scrub.log. Could you please let us know if anything is missing in making this test pass, PFA the logs for the test case (See attached file: bug-1207627-bitrot-scrub-status.7z) Note: Enough memory is available on the system. Regards, Chandranana Naik -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bug-1207627-bitrot-scrub-status.7z Type: application/octet-stream Size: 11030 bytes Desc: not available URL: From bugzilla at redhat.com Mon Feb 11 14:13:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 14:13:39 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #18 from David E. Smith --- I've updated my volumes with that option, and shortly will be scheduling a reboot of the clients (so that we can be sure we have a clean slate, so to speak). After that, I suppose there's nothing to do except wait a few days to see if it crashes. Sorry about not understanding that you wanted me to set write-behind; I did a quick scan of the docs at https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/ and didn't even see an option named that, so I went for the next-closest one. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 11 16:25:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 16:25:14 +0000 Subject: [Bugs] [Bug 1673972] insufficient logging in glusterd_resolve_all_bricks In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673972 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-11 16:25:14 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22181 (glusterd: improve logging) merged (#2) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 16:25:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 16:25:15 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Bug 1672818 depends on bug 1673972, which changed state. Bug 1673972 Summary: insufficient logging in glusterd_resolve_all_bricks https://bugzilla.redhat.com/show_bug.cgi?id=1673972 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 19:22:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 19:22:36 +0000 Subject: [Bugs] [Bug 1675076] New: [posix]: log the actual path wherever possible Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1675076 Bug ID: 1675076 Summary: [posix]: log the actual path wherever possible Product: GlusterFS Version: mainline Status: NEW Component: posix Assignee: bugs at gluster.org Reporter: rabhat at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: posix converts incoming operations on files to operations on corresponding gfid handles. While this in itself is not a problem, logging of those gfid handles in place of actual file paths can create confusions during debugging. The best way would be to print both the actual file (recieved as an argument) for path based operations and the gfid handle associated with it. Ex: [2019-01-29 08:50:24.373828] E [MSGID: 113039] [posix.c:3122:posix_open] 0-vol_f108aba8b492c40e74247293db9d544a-posix: open on /export/brick/.glusterfs/b8/b4/b8b47498-00df-4ab1-814a-b950250ef598, flags: 0 [No such file or directory] In the above log, if the actual file path was also mentioned (which posix_open gets as an argument), it would have helped in debugging and analysis better. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 11 19:30:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 11 Feb 2019 19:30:51 +0000 Subject: [Bugs] [Bug 1589705] quick-read: separate performance.cache-size tunable to affect quick-read only In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1589705 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20218 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 00:52:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 00:52:49 +0000 Subject: [Bugs] [Bug 1537457] DHT log messages: Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1537457 Artem Russakovskii changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |archon810 at gmail.com --- Comment #12 from Artem Russakovskii --- Has this actually gotten fixed? I'm still seeing tons of these in 5.3. [2019-02-12 00:50:22.373198] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in (null) (gfid = ab1a38fe-33ca-4938-a9e8-48c3b26eec47). Holes=1 overlaps=0 [2019-02-12 00:50:22.373454] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in (null) (gfid = 03dd4702-39ad-476f-be8f-8301d5c07dc1). Holes=1 overlaps=0 [2019-02-12 00:50:22.373545] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in (null) (gfid = fef9cbce-fab5-405d-b330-ee7534cc3e61). Holes=1 overlaps=0 [2019-02-12 00:50:22.373595] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in (null) (gfid = ba968ad0-a179-4cd7-8269-5de001b43725). Holes=1 overlaps=0 [2019-02-12 00:50:22.373666] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in (null) (gfid = 78733267-94a1-4493-b001-61419601d325). Holes=1 overlaps=0 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 03:22:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:22:43 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |ON_QA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 03:24:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:24:05 +0000 Subject: [Bugs] [Bug 1671014] gluster-fuse seg fault PTHREAD_MUTEX_TYPE_ELISION In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671014 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED CC| |rgowdapp at redhat.com Resolution|--- |DUPLICATE Last Closed| |2019-02-12 03:24:05 --- Comment #1 from Raghavendra G --- *** This bug has been marked as a duplicate of bug 1671556 *** -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 03:24:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:24:05 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joao.bauto at neuro.fchampalim | |aud.org --- Comment #19 from Raghavendra G --- *** Bug 1671014 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 03:25:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:25:12 +0000 Subject: [Bugs] [Bug 1676356] New: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 Bug ID: 1676356 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Product: GlusterFS Version: 6 Hardware: x86_64 OS: Linux Status: NEW Component: write-behind Severity: urgent Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: bugs at gluster.org Depends On: 1671556, 1674406 Blocks: 1667103 (glusterfs-5.4) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1674406 +++ +++ This bug was initially created as a clone of Bug #1671556 +++ This is a re-post of my FUSE crash report from BZ1651246. That issue is for a crash in the FUSE client. Mine is too, but I was asked in that bug to open a new issue, so here you go. :) My servers (two, in a 'replica 2' setup) publish two volumes. One is Web site content, about 110GB; the other is Web config files, only a few megabytes. (Wasn't worth building extra servers for that second volume.) FUSE clients have been crashing on the larger volume every three or four days. I can't reproduce this on-demand, unfortunately, but I've got several cores from previous crashes that may be of value to you. I'm using Gluster 5.3 from the RPMs provided by the CentOS Storage SIG, on a Red Hat Enterprise Linux 7.x system. The client's logs show many hundreds of instances of this (I don't know if it's related): [2019-01-29 08:14:16.542674] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7384) [0x7fa171ead384] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xae3e) [0x7fa1720bee3e] -->/lib64/libglusterfs.so.0(dict_ref+0x5d) [0x7fa1809cc2ad] ) 0-dict: dict is NULL [Invalid argument] Then, when the client's glusterfs process crashes, this is logged: The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 871 times between [2019-01-29 08:12:48.390535] and [2019-01-29 08:14:17.100279] pending frames: frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2019-01-29 08:14:17 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] --------- Info on the volumes themselves, gathered from one of my servers: [davidsmith at wuit-s-10889 ~]$ sudo gluster volume info all Volume Name: web-config Type: Replicate Volume ID: 6c5dce6e-e64e-4a6d-82b3-f526744b463d Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-config Brick2: 172.23.128.27:/data/web-config Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet server.event-threads: 4 client.event-threads: 4 cluster.min-free-disk: 1 cluster.quorum-count: 2 cluster.quorum-type: fixed network.ping-timeout: 10 auth.allow: * performance.readdir-ahead: on Volume Name: web-content Type: Replicate Volume ID: fcabc15f-0cec-498f-93c4-2d75ad915730 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-content Brick2: 172.23.128.27:/data/web-content Options Reconfigured: network.ping-timeout: 10 cluster.quorum-type: fixed cluster.quorum-count: 2 performance.readdir-ahead: on auth.allow: * cluster.min-free-disk: 1 client.event-threads: 4 server.event-threads: 4 transport.address-family: inet nfs.disable: on performance.client-io-threads: off performance.cache-size: 4GB gluster> volume status all detail Status of volume: web-config ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5612 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5540 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 Status of volume: web-content ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5649 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5567 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 I'll attach a couple of the core files generated by the crashing glusterfs instances, size limits permitting (they range from 3 to 8 GB). If I can't attach them, I'll find somewhere to host them. --- Additional comment from Artem Russakovskii on 2019-01-31 22:26:25 UTC --- Also reposting my comment from https://bugzilla.redhat.com/show_bug.cgi?id=1651246. I wish I saw this bug report before I updated from rock solid 4.1 to 5.3. Less than 24 hours after upgrading, I already got a crash and had to unmount, kill gluster, and remount: [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3" repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31 09:38:03.958061] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 72 times between [2019-01-31 09:37:53.746741] and [2019-01-31 09:38:04.696993] pending frames: frame : type(1) op(READ) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-31 09:38:04 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6] /lib64/libc.so.6(+0x36160)[0x7fccd622d160] /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0] /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa] /lib64/libc.so.6(+0x2e772)[0x7fccd6225772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3] /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559] /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f] --------- Do the pending patches fix the crash or only the repeated warnings? I'm running glusterfs on OpenSUSE 15.0 installed via http://download.opensuse.org/repositories/home:/glusterfs:/Leap15-5/openSUSE_Leap_15.0/, not too sure how to make it core dump. If it's not fixed by the patches above, has anyone already opened a ticket for the crashes that I can join and monitor? This is going to create a massive problem for us since production systems are crashing. Thanks. --- Additional comment from David E. Smith on 2019-01-31 22:31:47 UTC --- Actually, I ran the cores through strings and grepped for a few things like passwords -- as you'd expect from a memory dump from a Web server, there's a log of sensitive information in there. Is there a safe/acceptable way to send the cores only to developers that can use them, or otherwise not have to make them publicly available while still letting the Gluster devs benefit from analyzing them? --- Additional comment from Ravishankar N on 2019-02-01 05:51:19 UTC --- (In reply to David E. Smith from comment #2) > Actually, I ran the cores through strings and grepped for a few things like > passwords -- as you'd expect from a memory dump from a Web server, there's a > log of sensitive information in there. Is there a safe/acceptable way to > send the cores only to developers that can use them, or otherwise not have > to make them publicly available while still letting the Gluster devs benefit > from analyzing them? Perhaps you could upload it to a shared Dropbox folder with view/download access to the red hat email IDs I've CC'ed to this email (including me) to begin with. Note: I upgraded a 1x2 replica volume with 1 fuse client from v4.1.7 to v5.3 and did some basic I/O (kernel untar and iozone) and did not observe any crashes, so maybe this this something that is hit under extreme I/O or memory pressure. :-( --- Additional comment from Artem Russakovskii on 2019-02-02 20:17:15 UTC --- The fuse crash happened again yesterday, to another volume. Are there any mount options that could help mitigate this? In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch and restart the mount, which works and recovers the mount point within a minute. Not ideal, but a temporary workaround. By the way, the way to reproduce this "Transport endpoint is not connected" condition for testing purposes is to kill -9 the right "glusterfs --process-name fuse" process. monit check: check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 start program = "/bin/mount /mnt/glusterfs_data1" stop program = "/bin/umount /mnt/glusterfs_data1" if space usage > 90% for 5 times within 15 cycles then alert else if succeeded for 10 cycles then alert stack trace: [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 26 times between [2019-02-01 23:21:20.857333] and [2019-02-01 23:21:56.164427] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-01 23:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] --- Additional comment from David E. Smith on 2019-02-05 02:59:24 UTC --- I've added the five of you to our org's Box account; all of you should have invitations to a shared folder, and I'm uploading a few of the cores now. I hope they're of value to you. The binaries are all from the CentOS Storage SIG repo at https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/ . They're all current as of a few days ago: [davidsmith at wuit-s-10882 ~]$ rpm -qa | grep gluster glusterfs-5.3-1.el7.x86_64 glusterfs-client-xlators-5.3-1.el7.x86_64 glusterfs-fuse-5.3-1.el7.x86_64 glusterfs-libs-5.3-1.el7.x86_64 --- Additional comment from Nithya Balachandran on 2019-02-05 11:00:04 UTC --- (In reply to David E. Smith from comment #5) > I've added the five of you to our org's Box account; all of you should have > invitations to a shared folder, and I'm uploading a few of the cores now. I > hope they're of value to you. > > The binaries are all from the CentOS Storage SIG repo at > https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/ . They're > all current as of a few days ago: > > [davidsmith at wuit-s-10882 ~]$ rpm -qa | grep gluster > glusterfs-5.3-1.el7.x86_64 > glusterfs-client-xlators-5.3-1.el7.x86_64 > glusterfs-fuse-5.3-1.el7.x86_64 > glusterfs-libs-5.3-1.el7.x86_64 Thanks. We will take a look and get back to you. --- Additional comment from Nithya Balachandran on 2019-02-05 16:43:45 UTC --- David, Can you try mounting the volume with the option lru-limit=0 and let us know if you still see the crashes? Regards, Nithya --- Additional comment from Nithya Balachandran on 2019-02-06 07:23:49 UTC --- Initial analysis of one of the cores: [root at rhgs313-7 gluster-5.3]# gdb -c core.6014 /usr/sbin/glusterfs [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --direct-io-mode=disable --fuse-mountopts=noatime,context="'. Program terminated with signal 11, Segmentation fault. #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 410 if (!xl->call_cleanup && xl->cbks->forget) (gdb) bt #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 #1 0x00007fa1809e90a2 in __inode_destroy (inode=0x7fa0d0349af8) at inode.c:432 #2 inode_table_prune (table=table at entry=0x7fa15800c3c0) at inode.c:1696 #3 0x00007fa1809e9f96 in inode_forget_with_unref (inode=0x7fa0d0349af8, nlookup=128) at inode.c:1273 #4 0x00007fa177dae4e1 in do_forget (this=, unique=, nodeid=, nlookup=) at fuse-bridge.c:726 #5 0x00007fa177dae5bd in fuse_forget (this=, finh=0x7fa0a41da500, msg=, iobuf=) at fuse-bridge.c:741 #6 0x00007fa177dc5d7a in fuse_thread_proc (data=0x557a0e8ffe20) at fuse-bridge.c:5125 #7 0x00007fa17f83bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa17f103ead in msync () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? () (gdb) f 0 #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 410 if (!xl->call_cleanup && xl->cbks->forget) (gdb) l 405 for (index = 0; index < inode->table->xl->graph->xl_count; index++) { 406 if (inode->_ctx[index].value1 || inode->_ctx[index].value2) { 407 xl = (xlator_t *)(long)inode->_ctx[index].xl_key; 408 old_THIS = THIS; 409 THIS = xl; 410 if (!xl->call_cleanup && xl->cbks->forget) 411 xl->cbks->forget(xl, inode); 412 THIS = old_THIS; 413 } 414 } (gdb) p *xl Cannot access memory at address 0x0 (gdb) p index $1 = 6 (gdb) p inode->table->xl->graph->xl_count $3 = 13 (gdb) p inode->_ctx[index].value1 $4 = 0 (gdb) p inode->_ctx[index].value2 $5 = 140327960119304 (gdb) p/x inode->_ctx[index].value2 $6 = 0x7fa0a6370808 Based on the graph, the xlator with index = 6 is (gdb) p ((xlator_t*) inode->table->xl->graph->top)->next->next->next->next->next->next->next->name $31 = 0x7fa16c0122e0 "web-content-read-ahead" (gdb) p ((xlator_t*) inode->table->xl->graph->top)->next->next->next->next->next->next->next->xl_id $32 = 6 But read-ahead does not update the inode_ctx at all. There seems to be some sort of memory corruption happening here but that needs further analysis. --- Additional comment from David E. Smith on 2019-02-07 17:41:17 UTC --- As of this morning, I've added the lru-limit mount option to /etc/fstab on my servers. Was on vacation, didn't see the request until this morning. For the sake of reference, here's the full fstab lines, edited only to remove hostnames and add placeholders. (Note that I've never had a problem with the 'web-config' volume, which is very low-traffic and only a few megabytes in size; the problems always are the much more heavily-used 'web-content' volume.) gluster-server-1:/web-config /etc/httpd/conf.d glusterfs defaults,_netdev,noatime,context=unconfined_u:object_r:httpd_config_t:s0,backupvolfile-server=gluster-server-2,direct-io-mode=disable,lru-limit=0 0 0 gluster-server-1:/web-content /var/www/html glusterfs defaults,_netdev,noatime,context=unconfined_u:object_r:httpd_sys_rw_content_t:s0,backupvolfile-server=gluster-server-2,direct-io-mode=disable,lru-limit=0 0 0 --- Additional comment from David E. Smith on 2019-02-07 17:58:26 UTC --- Ran a couple of the glusterfs logs through the print-backtrace script. They all start with what you'd normally expect (clone, start_thread) and all end with (_gf_msg_backtrace_nomem) but they're all doing different things in the middle. It looks sorta like a memory leak or other memory corruption. Since it started happening on both of my servers after upgrading to 5.2 (and continued with 5.3), I really doubt it's a hardware issue -- the FUSE clients are both VMs, on hosts a few miles apart, so the odds of host RAM going wonky in both places at exactly that same time are ridiculous. Bit of a stretch, but do you think there would be value in my rebuilding the RPMs locally, to try to rule out anything on CentOS' end? /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] __restore_rt ??:0 /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] __inode_ctx_free ??:0 /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] inode_table_prune ??:0 /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] inode_forget_with_unref ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] fuse_forget ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] fuse_thread_proc ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] __clone ??:0 /lib64/libglusterfs.so.0(+0x26610)[0x7f36aff72610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f36aff7cb84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f36ae5d6280] __restore_rt ??:0 /lib64/libglusterfs.so.0(+0x36779)[0x7f36aff82779] __inode_unref ??:0 /lib64/libglusterfs.so.0(inode_unref+0x23)[0x7f36aff83203] inode_unref ??:0 /lib64/libglusterfs.so.0(gf_dirent_entry_free+0x2b)[0x7f36aff9ec4b] gf_dirent_entry_free ??:0 /lib64/libglusterfs.so.0(gf_dirent_free+0x2b)[0x7f36aff9ecab] gf_dirent_free ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x7480)[0x7f36a215b480] afr_readdir_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x60bca)[0x7f36a244dbca] client4_0_readdirp_cbk ??:0 /lib64/libgfrpc.so.0(+0xec70)[0x7f36afd3ec70] rpc_clnt_handle_reply ??:0 /lib64/libgfrpc.so.0(+0xf043)[0x7f36afd3f043] rpc_clnt_notify ??:0 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f36afd3af23] rpc_transport_notify ??:0 /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f36a492737b] socket_event_handler ??:0 /lib64/libglusterfs.so.0(+0x8aa49)[0x7f36affd6a49] event_dispatch_epoll_worker ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f36aedd5dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f36ae69dead] __clone ??:0 /lib64/libglusterfs.so.0(+0x26610)[0x7f7e13de0610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f7e13deab84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f7e12444280] __restore_rt ??:0 /lib64/libpthread.so.0(pthread_mutex_lock+0x0)[0x7f7e12c45c30] pthread_mutex_lock ??:0 /lib64/libglusterfs.so.0(__gf_free+0x12c)[0x7f7e13e0bc3c] __gf_free ??:0 /lib64/libglusterfs.so.0(+0x368ed)[0x7f7e13df08ed] __dentry_unset ??:0 /lib64/libglusterfs.so.0(+0x36b2b)[0x7f7e13df0b2b] __inode_retire ??:0 /lib64/libglusterfs.so.0(+0x36885)[0x7f7e13df0885] __inode_unref ??:0 /lib64/libglusterfs.so.0(inode_forget_with_unref+0x36)[0x7f7e13df1f86] inode_forget_with_unref ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x857a)[0x7f7e0b1b657a] fuse_batch_forget ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7f7e0b1cdd7a] fuse_thread_proc ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f7e12c43dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f7e1250bead] __clone ??:0 --- Additional comment from Nithya Balachandran on 2019-02-08 03:03:20 UTC --- (In reply to David E. Smith from comment #10) > Ran a couple of the glusterfs logs through the print-backtrace script. They > all start with what you'd normally expect (clone, start_thread) and all end > with (_gf_msg_backtrace_nomem) but they're all doing different things in the > middle. It looks sorta like a memory leak or other memory corruption. Since > it started happening on both of my servers after upgrading to 5.2 (and > continued with 5.3), I really doubt it's a hardware issue -- the FUSE > clients are both VMs, on hosts a few miles apart, so the odds of host RAM > going wonky in both places at exactly that same time are ridiculous. > > Bit of a stretch, but do you think there would be value in my rebuilding the > RPMs locally, to try to rule out anything on CentOS' end? I don't think so. My guess is there is an error somewhere in the client code when handling inodes. It was never hit earlier because we never freed the inodes before 5.3. With the new inode invalidation feature, we appear to be accessing inodes that were already freed. Did you see the same crashes in 5.2? If yes, something else might be going wrong. I had a look at the coredumps you sent - most don't have any symbols (strangely). Of the ones that do, it looks like memory corruption and accessing already freed inodes. There are a few people looking at it but this going to take a while to figure out. In the meantime, let me know if you still see crashes with the lru-limit option. --- Additional comment from Nithya Balachandran on 2019-02-08 03:18:00 UTC --- Another user has just reported that the lru-limit did not help with the crashes - let me know if that is your experience as well. --- Additional comment from Nithya Balachandran on 2019-02-08 12:57:50 UTC --- We have found the cause of one crash but that has a different backtrace. Unfortunately we have not managed to reproduce the one you reported so we don't know if it is the same cause. Can you disable write-behind on the volume and let us know if it solves the problem? If yes, it is likely to be the same issue. --- Additional comment from David E. Smith on 2019-02-09 16:07:08 UTC --- I did have some crashes with 5.2. (I went from 3.something straight to 5.2, so I'm not going to be too helpful in terms of narrowing down exactly when this issue first appeared, sorry.) I'll see if I still have any of those cores; they all were from several weeks ago, so I may have already cleaned them up. This morning, one of my clients core dumped with the lru-limit option. It looks like it might be a different crash (in particular, this morning's crash was a SIGABRT, whereas previous crashes were SIGSEGV). I've uploaded that core to the same Box folder, in case it's useful. I'll paste its backtrace in below. For the write-behind request, do you want me to set 'performance.flush-behind off' or so you mean something else? --- Additional comment from David E. Smith on 2019-02-09 16:07:49 UTC --- Backtrace for 2/9/19 crash (as promised above, put it in a separate comment for clarity): /lib64/libglusterfs.so.0(+0x26610)[0x7f3b31456610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f3b31460b84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f3b2faba280] __restore_rt ??:0 /lib64/libc.so.6(gsignal+0x37)[0x7f3b2faba207] raise ??:0 /lib64/libc.so.6(abort+0x148)[0x7f3b2fabb8f8] abort ??:0 /lib64/libc.so.6(+0x78d27)[0x7f3b2fafcd27] __libc_message ??:0 /lib64/libc.so.6(+0x81489)[0x7f3b2fb05489] _int_free ??:0 /lib64/libglusterfs.so.0(+0x1a6e9)[0x7f3b3144a6e9] dict_destroy ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x8cf9)[0x7f3b23388cf9] dht_local_wipe ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x4ab90)[0x7f3b233cab90] dht_revalidate_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x709e5)[0x7f3b236a89e5] afr_lookup_done ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71198)[0x7f3b236a9198] afr_lookup_metadata_heal_check ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71cbb)[0x7f3b236a9cbb] afr_lookup_entry_heal ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71f99)[0x7f3b236a9f99] afr_lookup_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x616d2)[0x7f3b239326d2] client4_0_lookup_cbk ??:0 /lib64/libgfrpc.so.0(+0xec70)[0x7f3b31222c70] rpc_clnt_handle_reply ??:0 /lib64/libgfrpc.so.0(+0xf043)[0x7f3b31223043] rpc_clnt_notify ??:0 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f3b3121ef23] rpc_transport_notify ??:0 /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f3b25e0b37b] socket_event_handler ??:0 /lib64/libglusterfs.so.0(+0x8aa49)[0x7f3b314baa49] event_dispatch_epoll_worker ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f3b302b9dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f3b2fb81ead] __clone ??:0 [d --- Additional comment from Raghavendra G on 2019-02-09 17:15:55 UTC --- (In reply to David E. Smith from comment #14) > I did have some crashes with 5.2. (I went from 3.something straight to 5.2, > so I'm not going to be too helpful in terms of narrowing down exactly when > this issue first appeared, sorry.) I'll see if I still have any of those > cores; they all were from several weeks ago, so I may have already cleaned > them up. > > This morning, one of my clients core dumped with the lru-limit option. It > looks like it might be a different crash (in particular, this morning's > crash was a SIGABRT, whereas previous crashes were SIGSEGV). I've uploaded > that core to the same Box folder, in case it's useful. I'll paste its > backtrace in below. > > For the write-behind request, do you want me to set > 'performance.flush-behind off' or so you mean something else? gluster volume set performance.write-behind off --- Additional comment from Nithya Balachandran on 2019-02-11 04:44:08 UTC --- Thanks David. I'm going to hold off on looking at the coredump until we hear back from you on whether disabling performance.write-behind works. The different backtraces could be symptoms of the same underlying issue where gluster tries to access already freed memory. --- Additional comment from Worker Ant on 2019-02-11 09:53:16 UTC --- REVIEW: https://review.gluster.org/22189 (performance/write-behind: fix use-after-free in readdirp) posted (#1) for review on master by Raghavendra G Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker https://bugzilla.redhat.com/show_bug.cgi?id=1671556 [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1674406 [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 03:25:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:25:12 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1676356 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 03:25:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:25:12 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1676356 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 03:25:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:25:12 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1676356 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 03:25:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:25:34 +0000 Subject: [Bugs] [Bug 1537457] DHT log messages: Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1537457 --- Comment #13 from Nithya Balachandran --- (In reply to Artem Russakovskii from comment #12) > Has this actually gotten fixed? I'm still seeing tons of these in 5.3. > > [2019-02-12 00:50:22.373198] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in > (null) (gfid = ab1a38fe-33ca-4938-a9e8-48c3b26eec47). Holes=1 overlaps=0 > [2019-02-12 00:50:22.373454] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in > (null) (gfid = 03dd4702-39ad-476f-be8f-8301d5c07dc1). Holes=1 overlaps=0 > [2019-02-12 00:50:22.373545] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in > (null) (gfid = fef9cbce-fab5-405d-b330-ee7534cc3e61). Holes=1 overlaps=0 > [2019-02-12 00:50:22.373595] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in > (null) (gfid = ba968ad0-a179-4cd7-8269-5de001b43725). Holes=1 overlaps=0 > [2019-02-12 00:50:22.373666] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 4--dht: Found anomalies in > (null) (gfid = 78733267-94a1-4493-b001-61419601d325). Holes=1 overlaps=0 The bug was for the null gfid being printed : (gfid = 00000000-0000-0000-0000-000000000000). A null gfid is an invalid gfid. The messages you see are perfectly normal if there is a problem with the layout on those directories or if a brick is down. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 03:26:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:26:41 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1676356 --- Comment #5 from Raghavendra G --- Bug 1676356 - glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 03:26:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:26:41 +0000 Subject: [Bugs] [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672818 (glusterfs-6.0) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 03:46:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:46:46 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22193 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 03:46:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 03:46:47 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #15 from Worker Ant --- REVIEW: https://review.gluster.org/22193 (performance/md-cache: change the op-version of \"global-cache-invalidation\") posted (#1) for review on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 08:06:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:06:21 +0000 Subject: [Bugs] [Bug 1676400] New: rm -rf fails with "Directory not empty" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 Bug ID: 1676400 Summary: rm -rf fails with "Directory not empty" Product: GlusterFS Version: mainline Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: When 2 clients run rm -rf concurrently, the operation fails with " Directory not empty" ls on the directory from the gluster mount point does not show any entries however there are directories on the bricks. Version-Release number of selected component (if applicable): How reproducible: Rare.This is a race condition. Steps to Reproduce: Steps: 1. Create 3x (2+1) arbiter volume and fuse mount it. Make sure lookup-optimize is enabled. 2. mkdir -p dir0/dir1/dir2. 3. Unmount and remount the volume to ensure a fresh lookup is sent. GDB into the fuse process and set a breakpoint at dht_lookup. 4. from the client mount: rm -rf mra_sources 5. When gdb breaks at dht_lookup for dir0/dir1/dir2, set a breakpoint at dht_lookup_cbk. Allow the process to continue until it hits dht_lookup_cbk. dht_lookup_cbk will return with op_ret = 0 . 6. Delete dir0/dir1/dir2 from every brick on the non-hashed subvols. 7. Set a breakpoint in dht_selfheal_dir_mkdir and allow gdb to continue. 8. When the process breaks at dht_selfheal_dir_mkdir, delete the directory from the hashed subvolume bricks. 9. In dht_selfheal_dir_mkdir_lookup_cbk, set a breakpoint at line : if (local->selfheal.hole_cnt == layout->cnt) { When gdb breaks at this point, set local->selfheal.hole_cnt to a value different from that of layout->cnt. Allow gdb to proceed. DHT will create the directories only on the non-hashed subvolumes as the layout has not been updated to indicate that the dir no longer exists on the hashed subvolume. This directory will no longer be visible on the mount point causing the rm -rf to fail. Actual results: root at server fuse1]# rm -rf mra_sources rm: cannot remove ?dir0/dir1?: Directory not empty Expected results: rm -rf should succeed. Additional info: As lookup-optimize is enabled, subsequent lookups cannot heal the directory. The same steps with lookup-optimize disabled will work as a subsequent lookup will lookup everywhere even if the entry does not exist on the hashed subvol. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 08:06:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:06:33 +0000 Subject: [Bugs] [Bug 1676400] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |nbalacha at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 08:08:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:08:31 +0000 Subject: [Bugs] [Bug 1676400] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 --- Comment #1 from Nithya Balachandran --- RCA for the invisible directory left behind with concurrent rm -rf : -------------------------------------------------------------------- dht_selfheal_dir_mkdir_lookup_cbk (...) { ... 1381 this_call_cnt = dht_frame_return (frame); 1382 1383 LOCK (&frame->lock); 1384 { 1385 if ((op_ret < 0) && 1386 (op_errno == ENOENT || op_errno == ESTALE)) { 1387 local->selfheal.hole_cnt = !local->selfheal.hole_cnt ? 1 1388 : local->selfheal.hole_cnt + 1; 1389 } 1390 1391 if (!op_ret) { 1392 dht_iatt_merge (this, &local->stbuf, stbuf, prev); 1393 } 1394 check_mds = dht_dict_get_array (xattr, conf->mds_xattr_key, 1395 mds_xattr_val, 1, &errst); 1396 if (dict_get (xattr, conf->mds_xattr_key) && check_mds && !errst) { 1397 dict_unref (local->xattr); 1398 local->xattr = dict_ref (xattr); 1399 } 1400 1401 } 1402 UNLOCK (&frame->lock); 1403 1404 if (is_last_call (this_call_cnt)) { 1405 if (local->selfheal.hole_cnt == layout->cnt) { 1406 gf_msg_debug (this->name, op_errno, 1407 "Lookup failed, an rmdir could have " 1408 "deleted this entry %s", loc->name); 1409 local->op_errno = op_errno; 1410 goto err; 1411 } else { 1412 for (i = 0; i < layout->cnt; i++) { 1413 if (layout->list[i].err == ENOENT || 1414 layout->list[i].err == ESTALE || 1415 local->selfheal.force_mkdir) 1416 missing_dirs++; 1417 } There are 2 problems here: 1. The layout is not updated with the new subvol status on error. In this case, the initial lookup found a directory on the hashed subvol so only 2 entries in the layout indicate missing directories. However, by the time the selfheal code is executed, the racing rmdir has deleted the directory from all the subvols. At this point, the directory does not exist on any subvol and dht_selfheal_dir_mkdir_lookup_cbk gets an error from all 3 subvols, but this new status is not updated in the layout which still has only 2 missing dirs marked. 2. this_call_cnt = dht_frame_return (frame); is called before processing the frame. So with a call cnt of 3, it is possible that the second response has reached 1404 before the third one has started processing the return values. At this point, local->selfheal.hole_cnt != layout->cnt so control goes to line 1412. At line 1412, since we are still using the old layout, only the directories on the non-hashed subvols are considered when incrementing missing_dirs and for the healing. The combination of these two causes the selfheal to start healing the directories on the non-hashed subvols. It succeeds in creating the dirs on the non-hashed subvols. However, to set the layout, dht takes an inodelk on the hashed subvol which fails because the directory does on exist there. We therefore end up with directories on the non-hashed subvols with no layouts set. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 08:32:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:32:26 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22194 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 08:32:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:32:27 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #540 from Worker Ant --- REVIEW: https://review.gluster.org/22194 (logging.c: do not strlen() under lock.) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 08:34:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:34:00 +0000 Subject: [Bugs] [Bug 1676400] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22195 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 08:34:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:34:01 +0000 Subject: [Bugs] [Bug 1676400] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22195 (cluster/dht: Fix lookup selfheal and rmdir race) posted (#1) for review on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 08:43:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:43:34 +0000 Subject: [Bugs] [Bug 1676400] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1661258 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 08:44:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 08:44:12 +0000 Subject: [Bugs] [Bug 1676400] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1458215 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1458215 [Bug 1458215] Slave reports ENOTEMPTY when rmdir is executed on master -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 09:08:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:08:37 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22196 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:08:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:08:38 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #541 from Worker Ant --- REVIEW: https://review.gluster.org/22196 (dht-shared.c: minor reduction of work under lock.) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:15:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:15:13 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22197 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:15:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:15:14 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #542 from Worker Ant --- REVIEW: https://review.gluster.org/22197 (fuse: reflect the actual default for lru-limit option) posted (#1) for review on release-6 by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:16:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:16:26 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22198 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:16:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:16:27 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #543 from Worker Ant --- REVIEW: https://review.gluster.org/22198 (fuse: reflect the actual default for lru-limit option) posted (#1) for review on release-5 by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:20:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:20:11 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22199 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:20:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:20:12 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #544 from Worker Ant --- REVIEW: https://review.gluster.org/22199 (auth-cache.c: minor reduction of work under lock.) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:22:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:22:01 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22200 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:22:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:22:02 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #545 from Worker Ant --- REVIEW: https://review.gluster.org/22200 (md-cache.c: minor reduction of work under lock.) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:28:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:28:05 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22201 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:28:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:28:06 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #546 from Worker Ant --- REVIEW: https://review.gluster.org/22201 (server.c: use dict_() funcs with key length.) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:31:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:31:25 +0000 Subject: [Bugs] [Bug 1676429] New: distribute: Perf regression in mkdir path Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676429 Bug ID: 1676429 Summary: distribute: Perf regression in mkdir path Product: GlusterFS Version: 6 Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: spalai at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: There seems to be perf regression of around 30% in mkdir path with patch : https://review.gluster.org/#/c/glusterfs/+/21062/. Here is the result from gbench which runs smallfile tool internally. Without patch: 3187.402238 2544.658604 2400.662029 (mkdir per second) With patch: 2439.311086 1654.222631 1634.522184 (mkdir per second) This bug is created to address the revert of the above commit. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: Run gbench -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:32:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:32:42 +0000 Subject: [Bugs] [Bug 1676430] New: distribute: Perf regression in mkdir path Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676430 Bug ID: 1676430 Summary: distribute: Perf regression in mkdir path Product: GlusterFS Version: mainline Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: spalai at redhat.com CC: bugs at gluster.org Depends On: 1676429 Target Milestone: --- Classification: Community Description of problem: There seems to be perf regression of around 30% in mkdir path with patch : https://review.gluster.org/#/c/glusterfs/+/21062/. Here is the result from gbench which runs smallfile tool internally. Without patch: 3187.402238 2544.658604 2400.662029 (mkdir per second) With patch: 2439.311086 1654.222631 1634.522184 (mkdir per second) This bug is created to address the revert of the above commit. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: Run gbench Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676429 [Bug 1676429] distribute: Perf regression in mkdir path -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:32:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:32:42 +0000 Subject: [Bugs] [Bug 1676429] distribute: Perf regression in mkdir path In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676429 Susant Kumar Palai changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1676430 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676430 [Bug 1676430] distribute: Perf regression in mkdir path -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:39:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:39:26 +0000 Subject: [Bugs] [Bug 1676430] distribute: Perf regression in mkdir path In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676430 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22202 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 09:39:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 09:39:27 +0000 Subject: [Bugs] [Bug 1676430] distribute: Perf regression in mkdir path In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676430 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22202 (Revert \"dht: Operate internal fops with negative pid\") posted (#1) for review on master by Susant Palai -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 10:58:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 10:58:01 +0000 Subject: [Bugs] [Bug 1579615] [geo-rep]: [Errno 39] Directory not empty In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1579615 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1661258 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 10:58:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 10:58:03 +0000 Subject: [Bugs] [Bug 1579615] [geo-rep]: [Errno 39] Directory not empty In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1579615 Bug 1579615 depends on bug 1575553, which changed state. Bug 1575553 Summary: [geo-rep]: [Errno 39] Directory not empty https://bugzilla.redhat.com/show_bug.cgi?id=1575553 What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |DUPLICATE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 11:26:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 11:26:40 +0000 Subject: [Bugs] [Bug 1579615] [geo-rep]: [Errno 39] Directory not empty In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1579615 Bug 1579615 depends on bug 1575553, which changed state. Bug 1575553 Summary: [geo-rep]: [Errno 39] Directory not empty https://bugzilla.redhat.com/show_bug.cgi?id=1575553 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |ASSIGNED Resolution|DUPLICATE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 11:47:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 11:47:58 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1676468 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676468 [Bug 1676468] glusterfs-fuse client not benefiting from page cache on read after write -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 11:47:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 11:47:58 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1676468 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676468 [Bug 1676468] glusterfs-fuse client not benefiting from page cache on read after write -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 11:47:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 11:47:58 +0000 Subject: [Bugs] [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1676468 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676468 [Bug 1676468] glusterfs-fuse client not benefiting from page cache on read after write -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 11:58:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 11:58:05 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22087 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 11:58:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 11:58:06 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #9 from Worker Ant --- REVIEW: https://review.gluster.org/22087 (clnt/rpc: ref leak during disconnect.) merged (#17) on master by mohammed rafi kc -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 12:11:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 12:11:51 +0000 Subject: [Bugs] [Bug 1665029] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |urgent CC| |atumball at redhat.com --- Comment #5 from Amar Tumballi --- Please go ahead and disable it by default in upstream master. Lets get a run done with these values, and if the performance is good without these 2 translators, then we can backport the patch to glusterfs-6 branch. Otherwise, it would allow us another 2 months to validate it in upstream master before glusterfs-7. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 12:13:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 12:13:38 +0000 Subject: [Bugs] [Bug 1676479] New: read-ahead and io-cache degrading performance on sequential read Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676479 Bug ID: 1676479 Summary: read-ahead and io-cache degrading performance on sequential read Product: GlusterFS Version: mainline Hardware: x86_64 OS: Linux Status: NEW Component: read-ahead Keywords: Performance Severity: high Priority: urgent Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: bugs at gluster.org Depends On: 1665029 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1665029 +++ Description of problem: A large-file sequential read test reports better throughput with read-ahead and io-cache xlators turned off. One test is obviously not enough to decide the fate of these xlators. But through this bz we can answer some of the relevant questions. Version-Release number of selected component (if applicable): glusterfs-*5.2-1.el7.x86_64 kernel-3.10.0-957.el7.x86_64 (RHEL 7.6) How reproducible: Consistently Steps to Reproduce: 1. fio write test to generate data set: fio --name=initialwrite --ioengine=sync --rw=write \ --direct=0 --create_on_open=1 --end_fsync=1 --bs=128k \ --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum \ --filesize=16g --size=16g --numjobs=4 2. unmount and re-mount volume on client 3. fio sequential read test to read back the data written in step 1: fio --name=readtest --ioengine=sync --rw=read \ --direct=0 --bs=128k --directory=/mnt/glustervol/ \ --filename_format=f.\$jobnum.\$filenum --filesize=16g \ --size=16g --numjobs=4 Actual results: With default volume settings: READ: bw=581MiB/s (610MB/s), 145MiB/s-146MiB/s (152MB/s-153MB/s), io=64.0GiB (68.7GB), run=112401-112717msec With read-ahead and io-cache turned off: READ: bw=1083MiB/s (1136MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io=64.0GiB (68.7GB), run=60487-60491msec So in this test, there is a significant performance gain with these xlators turned off. The low performance with default volume settings was first seen in another bz:https://bugzilla.redhat.com/show_bug.cgi?id=1664934#c0. Expected results: performance xlators should not degrade performance Additional info: --- Additional comment from Manoj Pillai on 2019-01-10 13:18:03 UTC --- Data showing that both read-ahead and io-cache cause performance degradation. I'm modifying the test in comment #0 here, substituting numjobs=2, instead of 4. Test sequence: fio --name=initialwrite --ioengine=sync --rw=write --direct=0 --create_on_open=1 --end_fsync=1 --bs=128k --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum --filesize=16g --size=16g --numjobs=2 [unmount and mount volume] fio --name=readtest --ioengine=sync --rw=read --direct=0 --bs=128k --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum --filesize=16g --size=16g --numjobs=2 Result with default settings: READ: bw=485MiB/s (509MB/s), 243MiB/s-243MiB/s (254MB/s-255MB/s), io=32.0GiB (34.4GB), run=67504-67522msec Result with read-ahead turned off: READ: bw=776MiB/s (813MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=32.0GiB (34.4GB), run=42220-42237msec Result with read-ahead and io-cache turned off: READ: bw=1108MiB/s (1162MB/s), 554MiB/s-554MiB/s (581MB/s-581MB/s), io=32.0GiB (34.4GB), run=29565-29573msec --- Additional comment from Raghavendra G on 2019-01-11 06:40:57 UTC --- Some observations while debugging the performance degradation with gluster read-ahead: * Kernel too does read-ahead and it sends parallel read-requests as part of this. * client-io-threads is on in this configuration. The above two points mean parallel requests sent by kernel can reach read-ahead out of order. This means read-ahead no longer sees read requests at sequential contiguous offsets and hence it things reads are random. For random reads, it resets the read sequence. But when requests reach read-ahead in order, read-ahead is turned on again. Due to this intermittent toggling, much of read-ahead data is wasted regressing the performance. With client-io-threads off and I can no longer see the regression for the test case given. If I run the test with single fio job (--numjobs=1), gluster read-ahead on outperforms gluster read-ahead off on my setup. [1] https://review.gluster.org/#/c/glusterfs/+/20981/ --- Additional comment from Raghavendra G on 2019-01-11 06:42:55 UTC --- (In reply to Raghavendra G from comment #2) > Some observations while debugging the performance degradation with gluster > read-ahead: > > * Kernel too does read-ahead and it sends parallel read-requests as part of > this. > * client-io-threads is on in this configuration. > > The above two points mean parallel requests sent by kernel can reach > read-ahead out of order. This means read-ahead no longer sees read requests > at sequential contiguous offsets and hence it things reads are random. For > random reads, it resets the read sequence. But when requests reach > read-ahead in order, read-ahead is turned on again. Due to this intermittent > toggling, much of read-ahead data is wasted regressing the performance. With > client-io-threads off and I can no longer see the regression for the test > case given. If I run the test with single fio job (--numjobs=1), gluster > read-ahead on outperforms gluster read-ahead off on my setup. ... single fio job (--numjobs=1), gluster read-ahead on with client-io-threads off outperforms gluster read-ahead off with client-io-threads off. > > [1] https://review.gluster.org/#/c/glusterfs/+/20981/ --- Additional comment from Manoj Pillai on 2019-01-11 12:40:52 UTC --- (In reply to Manoj Pillai from comment #1) > Data showing that both read-ahead and io-cache cause performance > degradation. I'm modifying the test in comment #0 here, substituting > numjobs=2, instead of 4. > > Test sequence: > fio --name=initialwrite --ioengine=sync --rw=write --direct=0 > --create_on_open=1 --end_fsync=1 --bs=128k --directory=/mnt/glustervol/ > --filename_format=f.\$jobnum.\$filenum --filesize=16g --size=16g --numjobs=2 > [unmount and mount volume] > fio --name=readtest --ioengine=sync --rw=read --direct=0 --bs=128k > --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum > --filesize=16g --size=16g --numjobs=2 > > Result with default settings: > READ: bw=485MiB/s (509MB/s), 243MiB/s-243MiB/s (254MB/s-255MB/s), io=32.0GiB > (34.4GB), run=67504-67522msec > > Result with read-ahead turned off: > READ: bw=776MiB/s (813MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=32.0GiB > (34.4GB), run=42220-42237msec > > Result with read-ahead and io-cache turned off: > READ: bw=1108MiB/s (1162MB/s), 554MiB/s-554MiB/s (581MB/s-581MB/s), > io=32.0GiB (34.4GB), run=29565-29573msec Result with ciot=off, io-cache=off, gluster ra=on, read-ahead-page-count=10: [these settings are based on comment #2 and comment #3] READ: bw=975MiB/s (1023MB/s), 488MiB/s-488MiB/s (511MB/s-512MB/s), io=32.0GiB (34.4GB) Comparing the best results seen (1108 vs 975), the gluster ra=off case is still a little bit better. Result with ciot=off, io-cache=on, gluster ra=on, read-ahead-page-count=10: READ: bw=674MiB/s (706MB/s), 337MiB/s-339MiB/s (353MB/s-355MB/s), io=32.0GiB (34.4GB) --- Additional comment from Amar Tumballi on 2019-02-12 12:11:51 UTC --- Please go ahead and disable it by default in upstream master. Lets get a run done with these values, and if the performance is good without these 2 translators, then we can backport the patch to glusterfs-6 branch. Otherwise, it would allow us another 2 months to validate it in upstream master before glusterfs-7. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 [Bug 1665029] read-ahead and io-cache degrading performance on sequential read -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 12:13:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 12:13:38 +0000 Subject: [Bugs] [Bug 1665029] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1676479 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676479 [Bug 1676479] read-ahead and io-cache degrading performance on sequential read -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 11:47:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 11:47:58 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #16 from Worker Ant --- REVIEW: https://review.gluster.org/22193 (performance/md-cache: change the op-version of \"global-cache-invalidation\") merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 12:59:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 12:59:34 +0000 Subject: [Bugs] [Bug 1676479] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676479 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |rgowdapp at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 13:07:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 13:07:57 +0000 Subject: [Bugs] [Bug 1676479] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676479 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22203 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 13:07:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 13:07:57 +0000 Subject: [Bugs] [Bug 1676479] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676479 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22203 (read-ahead/io-cache: turn off by default) posted (#1) for review on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 12 13:19:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 13:19:09 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Kaleb KEITHLEY changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kkeithle at redhat.com --- Comment #3 from Kaleb KEITHLEY --- This is because gnfs is has been deprecated (for a long time now) and the glusterfs-gnfs subpackage is no longer built for the CentOS Storage SIG packages starting with GlusterFS-5. If you manually remove the glusterfs-gnfs package first, then update, the update will work. (Are you using NFS even? If not, then you don't need -gnfs in the first place. If you are, there is NFS-Ganesha.) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 14:17:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 14:17:50 +0000 Subject: [Bugs] [Bug 1676546] New: Getting client connection error in gluster logs Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676546 Bug ID: 1676546 Summary: Getting client connection error in gluster logs Product: GlusterFS Version: 4.1 OS: Linux Status: NEW Component: glusterd Severity: urgent Assignee: bugs at gluster.org Reporter: binny_ms at outlook.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Getting client connection error after starting volume. This is not affecting in the working of Gluster MOunts. But seeing a number of error messages in glustershd.log and the .log in the client. I am running GLuster FS on RHEL 7.5 on Azure VMs Version-Release number of selected component (if applicable): glusterfs 4.1.5 How reproducible: Create a volume, start it and mount from a client Steps to Reproduce: 1.Create a volume 2. Start it 3. MOunt on a client Actual results: Everything working fine, except for the error messages in logs [2019-02-12 12:13:52.208065] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-3: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:52.208287] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-testvol2-client-1: Connected to testvol2-client-1, attached to remote volume '/data/brick1/testvol2/brk1'. [2019-02-12 12:13:52.208312] I [MSGID: 108005] [afr-common.c:5336:__afr_handle_child_up_event] 0-testvol2-replicate-0: Subvolume 'testvol2-client-1' came back up; going online. [2019-02-12 12:13:52.208411] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-3: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:52.208593] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-testvol2-client-3: changing port to 49153 (from 0) [2019-02-12 12:13:52.213669] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-3: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:52.213969] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-3: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:52.217485] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-testvol2-client-3: Connected to testvol2-client-3, attached to remote volume '/data/brick1/testvol2/brk2'. [2019-02-12 12:13:52.217527] I [MSGID: 108005] [afr-common.c:5336:__afr_handle_child_up_event] 0-testvol2-replicate-1: Subvolume 'testvol2-client-3' came back up; going online. [2019-02-12 12:13:55.516893] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-0: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:55.517086] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-0: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:55.517266] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-testvol2-client-0: changing port to 49152 (from 0) [2019-02-12 12:13:55.520686] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:55.520921] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:55.521055] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-testvol2-client-2: changing port to 49153 (from 0) [2019-02-12 12:13:55.523566] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-0: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:55.523819] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-0: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:55.525572] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-testvol2-client-0: Connected to testvol2-client-0, attached to remote volume '/data/brick1/testvol2/brk1'. [2019-02-12 12:13:55.526418] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:55.526616] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-testvol2-client-2: error returned while attempting to connect to host:(null), port:0 [2019-02-12 12:13:55.534983] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-testvol2-client-2: Connected to testvol2-client-2, attached to remote volume '/data/brick1/testvol2/brk2'. Expected results: Additional info: I am able to see these errors in brick logs also at times -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 12 15:10:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 12 Feb 2019 15:10:24 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #10 from Worker Ant --- REVIEW: https://review.gluster.org/22151 (afr/shd: Cleanup self heal daemon resources during afr fini) merged (#2) on master by Ravishankar N -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 04:41:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 04:41:33 +0000 Subject: [Bugs] [Bug 1676736] New: tests: ./tests/bugs/distribute/bug-1161311.t times out Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676736 Bug ID: 1676736 Summary: tests: ./tests/bugs/distribute/bug-1161311.t times out Product: GlusterFS Version: mainline Status: NEW Component: tests Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: The test uses loopack devices for the bricks but does not stop the volume before unmounting them. 17:44:49 ok 57, LINENUM:155 17:44:49 umount: /d/backends/patchy1: target is busy. 17:44:49 (In some cases useful info about processes that use 17:44:49 the device is found by lsof(8) or fuser(1)) 17:44:49 umount: /d/backends/patchy2: target is busy. 17:44:49 (In some cases useful info about processes that use 17:44:49 the device is found by lsof(8) or fuser(1)) 17:44:49 umount: /d/backends/patchy3: target is busy. 17:44:49 (In some cases useful info about processes that use 17:44:49 the device is found by lsof(8) or fuser(1)) 17:44:49 N Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 04:43:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 04:43:45 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 --- Comment #1 from asender at testlabs.com.au --- We need a procedure to downgrade from 5 to 4 without causing any further disruptions. I think this bug report should be a blocker. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 05:08:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 05:08:16 +0000 Subject: [Bugs] [Bug 1676736] tests: ./tests/bugs/distribute/bug-1161311.t times out In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676736 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST Assignee|bugs at gluster.org |nbalacha at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 05:08:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 05:08:29 +0000 Subject: [Bugs] [Bug 1676736] tests: ./tests/bugs/distribute/bug-1161311.t times out In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676736 --- Comment #1 from Nithya Balachandran --- https://review.gluster.org/#/c/glusterfs/+/22208/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 05:09:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 05:09:10 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |homma at allworks.co.jp, | |rgowdapp at redhat.com Flags| |needinfo?(homma at allworks.co | |.jp) --- Comment #2 from Raghavendra G --- (In reply to homma from comment #0) > Additional info: > > In readdir-ahead.c, rda_fill_fd_cbk() replaces dentries and zeroes out iatts > exept for ia_gfid and ia_type. > Then in posix-acl.c, posix_acl_readdirp_cbk() updates its inode ctx by that > zeroed permission, and permission is denied. The expectation is kernel would do a fresh lookup for getting other attributes like permissions and that's what Glusterfs indicates kernel too - that only entry information (mapping of path to inode/gfid) is valid and the attributes are not valid. How did you conclude zeroed out permissions are set on posix-acl? Did you see a call like setattr or any setxattr updating posix acls? If yes, whether these zeroed out attributes were sent from kernel? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 05:20:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 05:20:50 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 --- Comment #3 from asender at testlabs.com.au --- gluster volume info Volume Name: common Type: Replicate Volume ID: 359a079c-0c67-4a07-aa92-65d746ae6440 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: hplintnfs30063:/export/common/common Brick2: hplintnfs30065:/export/common/common Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on Volume Name: external Type: Replicate Volume ID: b76d3a71-6c0c-4df3-9411-baa30a586489 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: hplintnfs30065:/export/external/external Brick2: hplintnfs30064:/export/external/external Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on nfs.log-level: debug Volume Name: input Type: Replicate Volume ID: 399caee1-4acc-48bc-9416-5510dc056280 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: hplintnfs30063:/export/input/input Brick2: hplintnfs30065:/export/input/input Options Reconfigured: transport.address-family: inet performance.readdir-ahead: enable nfs.disable: on performance.cache-size: 1GB performance.client-io-threads: on performance.io-cache: on performance.io-thread-count: 16 performance.read-ahead: disable server.allow-insecure: on cluster.lookup-optimize: on client.event-threads: 4 server.event-threads: 4 cluster.readdir-optimize: on performance.write-behind-window-size: 1MB Volume Name: logs Type: Replicate Volume ID: a5afa578-441b-4392-887a-2e3d71a27408 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: hplintnfs30063:/export/logs/logs Brick2: hplintnfs30065:/export/logs/logs Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on Volume Name: output Type: Replicate Volume ID: bf333aa2-7260-4a8c-aa8b-cb9aeac16d36 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: hplintnfs30063:/export/output/output Brick2: hplintnfs30065:/export/output/output Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on Volume Name: report Type: Replicate Volume ID: caf38a37-9228-4d2a-b636-6a168ce89183 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: hplintnfs30065:/export/report/report Brick2: hplintnfs30064:/export/report/report Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on Volume Name: statement Type: Replicate Volume ID: 238e520d-d493-4b0e-89e2-15707847e1e7 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: hplintnfs30065:/export/statement/statement Brick2: hplintnfs30063:/export/statement/statement Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 05:33:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 05:33:15 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 --- Comment #4 from asender at testlabs.com.au --- Applications returning permission denied. 2019-02-07 15:00:35 DEBUG - stderr: gpg: can't open `/data/common/direct-entry-files/first-data-returns/sftp/512733_ADEF020701.gpg': Permission denied -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 05:38:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 05:38:26 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 --- Comment #5 from asender at testlabs.com.au --- Could someone kindly provide a "rollback" procedure with minimal impact. Can we set Gluster back to version 4 compatibility mode and downgrade.? Prefer non-impacting, but whatever is the safest. We need to go back to version 4. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 06:18:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 06:18:30 +0000 Subject: [Bugs] [Bug 1676736] tests: ./tests/bugs/distribute/bug-1161311.t times out In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676736 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22208 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 06:18:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 06:18:31 +0000 Subject: [Bugs] [Bug 1676736] tests: ./tests/bugs/distribute/bug-1161311.t times out In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676736 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22208 (tests/dht: Stop volume before unmounting bricks) posted (#1) for review on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 06:21:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 06:21:16 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22207 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 06:21:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 06:21:17 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #547 from Worker Ant --- REVIEW: https://review.gluster.org/22207 (rfc.sh: fix the missing rebase issue) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 07:26:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 07:26:28 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #9 from Nithya Balachandran --- The diffs for possible fixes for bug-1161311.t and bug-1193636.t are as follows. Please make the changes on your setup and let me know if it works. [root at rhgs313-6 tests]# git diff bugs/distribute/bug-1161311.t diff --git a/tests/bugs/distribute/bug-1161311.t b/tests/bugs/distribute/bug-1161311.t index c52c69b..3dc45a4 100755 --- a/tests/bugs/distribute/bug-1161311.t +++ b/tests/bugs/distribute/bug-1161311.t @@ -82,6 +82,8 @@ for i in {1..10}; do cat /tmp/FILE2 >> $M0/dir1/FILE2 done +brick_loc=$(get_backend_paths $M0/dir1/FILE2) + #dd if=/dev/urandom of=$M0/dir1/FILE2 bs=64k count=10240 # Rename the file to create a linkto, for rebalance to @@ -99,7 +101,7 @@ TEST $CLI volume rebalance $V0 start force # Wait for FILE to get the sticky bit on, so that file is under # active rebalance, before creating the links -TEST checksticky $B0/${V0}3/dir1/FILE1 +TEST checksticky $brick_loc # Create the links ## FILE3 FILE5 FILE7 have hashes, c8c91469 566d26ce 22ce7eba @@ -120,7 +122,7 @@ cd / # Ideally for this test to have done its job, the file should still be # under migration, so check the sticky bit again -TEST checksticky $B0/${V0}3/dir1/FILE1 +TEST checksticky $brick_loc # Wait for rebalance to complete EXPECT_WITHIN $REBALANCE_TIMEOUT "completed" rebalance_status_field $V0 [root at rhgs313-6 tests]# git diff ./bugs/distribute/bug-1193636.t diff --git a/tests/bugs/distribute/bug-1193636.t b/tests/bugs/distribute/bug-1193636.t index ccde02e..6ffa2d9 100644 --- a/tests/bugs/distribute/bug-1193636.t +++ b/tests/bugs/distribute/bug-1193636.t @@ -37,6 +37,8 @@ TEST mkdir $M0/dir1 # Create a large file (1GB), so that rebalance takes time dd if=/dev/zero of=$M0/dir1/FILE2 bs=64k count=10240 +brick_loc=$(get_backend_paths $M0/dir1/FILE2) + # Rename the file to create a linkto, for rebalance to # act on the file TEST mv $M0/dir1/FILE2 $M0/dir1/FILE1 @@ -45,7 +47,7 @@ build_tester $(dirname $0)/bug-1193636.c TEST $CLI volume rebalance $V0 start force -TEST checksticky $B0/${V0}3/dir1/FILE1 +TEST checksticky $brick_loc TEST setfattr -n "user.test1" -v "test1" $M0/dir1/FILE1 TEST setfattr -n "user.test2" -v "test1" $M0/dir1/FILE1 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 07:26:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 07:26:45 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 07:52:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 07:52:08 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 ryan at magenta.tv changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ryan at magenta.tv --- Comment #21 from ryan at magenta.tv --- Also seeing this issue in 4.1.7 with the Samba VFS client logs. Have worked around the issue by setting client log levels to error ('gluster volume set vol01 diagnostics.client-log-level ERROR'). Please could you advise on a long term solution? Many thanks, Ryan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 07:59:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 07:59:42 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #10 from Nithya Balachandran --- Possible fix for ./tests/bugs/posix/bug-1619720.t: [root at rhgs313-6 tests]# git diff bugs/posix/bug-1619720.t diff --git a/tests/bugs/posix/bug-1619720.t b/tests/bugs/posix/bug-1619720.t index bfd304d..8584476 100755 --- a/tests/bugs/posix/bug-1619720.t +++ b/tests/bugs/posix/bug-1619720.t @@ -1,6 +1,7 @@ #!/bin/bash . $(dirname $0)/../../include.rc +. $(dirname $0)/../../volume.rc . $(dirname $0)/../../dht.rc cleanup; @@ -35,7 +36,8 @@ TEST mkdir $M0/tmp # file-2 will hash to the other subvol TEST touch $M0/tmp/file-2 -pgfid_xattr_name=$(getfattr -m "trusted.pgfid.*" $B0/${V0}1/tmp/file-2 | grep "trusted.pgfid") +loc_2=$(get_backend_paths $M0/tmp/file-2) +pgfid_xattr_name=$(getfattr -m "trusted.pgfid.*" $loc_2 | grep "trusted.pgfid") echo $pgfid_xattr_name -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 08:13:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 08:13:51 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #11 from Nithya Balachandran --- (In reply to abhays from comment #6) > Yes @Nithya, Below are the test cases that fail, their cause of failure and > possible workaround on Big-Endian:- > > Following are the test cases which pass after changing the bricks in the > test case:- > ./tests/bugs/distribute/bug-1161311.t --------passes after changing brick3 > to brick1 in subtests 31 and 41. > ./tests/bugs/distribute/bug-1193636.t --------passes after changing brick3 > to brick1 in subtest 10. Provided diff of fix for these 2 tests in comment#9. Please try it out and let me know if it works. > ./tests/bugs/nfs/bug-847622.t ----------------passes after giving absolute > path of testfile in subtest 9. I don't see why this should be dependent on the hashing. Please provide output of the test and the gluster logs with debug enabled when this fails. > > Following are the test cases that are still failing even after changing the > bricks, however if little-endian hash values are hard-coded on big-endian in > the file ./xlators/cluster/dht/src/dht-layout.c, then these test cases pass > on Big-Endian:- > ./tests/bugs/glusterfs/bug-902610.t-------------subtest 7 fails I don't see why this should be dependent on the hashing. Please provide output of the test and the gluster logs with debug enabled when this fails. > ./tests/bugs/posix/bug-1619720.t----------------subtests 13 and 14 fail Diff of fix provided in comment#10. > > Following test case is failing because of "Cannot allocate memory" issue:- > ./tests/bitrot/bug-1207627-bitrot-scrub-status.t----------subtest 20 fails > with the below error:- > [client-rpc-fops_v2.c:961:client4_0_fgetxattr_cbk] 0-patchy-client-0: remote > operation failed [Cannot allocate memory] > This does not seem related to the hashing algorithm. Please check the brick log to see if there are any errors. > Following test case is failing on which issue has already been raised:- > ./tests/features/trash.t-------------- > https://bugzilla.redhat.com/show_bug.cgi?id=1627060 > I'll take a look at this and see what can be done. > So, please look into this and let us know if any workaround can be provided > to make the above tests pass on Big-Endian. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 08:25:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 08:25:41 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 homma at allworks.co.jp changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(homma at allworks.co | |.jp) | --- Comment #6 from homma at allworks.co.jp --- (In reply to Raghavendra G from comment #2) > (In reply to homma from comment #0) > > Additional info: > > > > In readdir-ahead.c, rda_fill_fd_cbk() replaces dentries and zeroes out iatts > > exept for ia_gfid and ia_type. > > Then in posix-acl.c, posix_acl_readdirp_cbk() updates its inode ctx by that > > zeroed permission, and permission is denied. > > The expectation is kernel would do a fresh lookup for getting other > attributes like permissions and that's what Glusterfs indicates kernel too - > that only entry information (mapping of path to inode/gfid) is valid and > the attributes are not valid. How did you conclude zeroed out permissions > are set on posix-acl? Did you see a call like setattr or any setxattr > updating posix acls? If yes, whether these zeroed out attributes were sent > from kernel? In the client log, 'ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:-)' indicates that owner, group, and permissions are all zero in posix-acl ctx. With gdb, the following output is obtained when executing the above commands (see 'Steps to Reproduce'). The ctx is zeroed out (uid = 0, gid = 0, perm = 32768) when updated by READDIRP, while it has correct values (uid = 1000, gid = 1000, perm = 33204) when updated by LOOKUP. (gdb) break posix-acl.c:1196 Breakpoint 1 at 0x7fbdc0fecb28: file posix-acl.c, line 1196. (gdb) commands Type commands for breakpoint(s) 1, one per line. End with a line saying just "end". >print *loc >print *(struct posix_acl_ctx *)loc.inode._ctx[13].ptr1 >continue >end (gdb) break posix-acl.c:1200 Breakpoint 2 at 0x7fbdc0fec953: file posix-acl.c, line 1200. (gdb) commands Type commands for breakpoint(s) 2, one per line. End with a line saying just "end". >print *loc >print *(struct posix_acl_ctx *)loc.inode._ctx[13].ptr1 >continue >end (gdb) set pagination off (gdb) continue Continuing. [Switching to Thread 0x7fbdbb7fe700 (LWP 7156)] Breakpoint 1, posix_acl_open (frame=frame at entry=0x7fbdac01e8b8, this=this at entry=0x7fbdbc01dc00, loc=loc at entry=0x7fbdac000f30, flags=flags at entry=32769, fd=fd at entry=0x7fbdac009d88, xdata=xdata at entry=0x0) at posix-acl.c:1196 1196 STACK_WIND(frame, posix_acl_open_cbk, FIRST_CHILD(this), $1 = {path = 0x7fbdac007f20 "/centos/test/foo", name = 0x0, inode = 0x7fbdac001e98, parent = 0x0, gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", pargfid = '\000' } $2 = {uid = 1000, gid = 1000, perm = 33204, fop = GF_FOP_LOOKUP, acl_access = 0x0, acl_default = 0x0} Breakpoint 1, posix_acl_open (frame=frame at entry=0x7fbdac014fe8, this=this at entry=0x7fbdbc01dc00, loc=loc at entry=0x7fbdac000f30, flags=flags at entry=32768, fd=fd at entry=0x7fbdac00ab28, xdata=xdata at entry=0x0) at posix-acl.c:1196 1196 STACK_WIND(frame, posix_acl_open_cbk, FIRST_CHILD(this), $3 = {path = 0x7fbdac009090 "/centos/test/foo", name = 0x0, inode = 0x7fbdac001e98, parent = 0x0, gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", pargfid = '\000' } $4 = {uid = 1000, gid = 1000, perm = 33204, fop = GF_FOP_LOOKUP, acl_access = 0x0, acl_default = 0x0} Breakpoint 2, posix_acl_open (frame=frame at entry=0x7fbdac013978, this=this at entry=0x7fbdbc01dc00, loc=loc at entry=0x7fbdac000f30, flags=flags at entry=32768, fd=fd at entry=0x7fbdac014638, xdata=xdata at entry=0x0) at posix-acl.c:1200 1200 STACK_UNWIND_STRICT(open, frame, -1, EACCES, NULL, NULL); $5 = {path = 0x7fbdac009090 "/centos/test/foo", name = 0x0, inode = 0x7fbdac001e98, parent = 0x0, gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", pargfid = '\000' } $6 = {uid = 0, gid = 0, perm = 32768, fop = GF_FOP_READDIRP, acl_access = 0x0, acl_default = 0x0} Breakpoint 2, posix_acl_open (frame=frame at entry=0x7fbdac0126a8, this=this at entry=0x7fbdbc01dc00, loc=loc at entry=0x7fbdac000f30, flags=flags at entry=32768, fd=fd at entry=0x7fbdac017b48, xdata=xdata at entry=0x0) at posix-acl.c:1200 1200 STACK_UNWIND_STRICT(open, frame, -1, EACCES, NULL, NULL); $7 = {path = 0x7fbdac007f20 "/centos/test/foo", name = 0x0, inode = 0x7fbdac001e98, parent = 0x0, gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", pargfid = '\000' } $8 = {uid = 0, gid = 0, perm = 32768, fop = GF_FOP_READDIRP, acl_access = 0x0, acl_default = 0x0} Breakpoint 2, posix_acl_open (frame=frame at entry=0x7fbdac017b48, this=this at entry=0x7fbdbc01dc00, loc=loc at entry=0x7fbdac000f30, flags=flags at entry=32768, fd=fd at entry=0x7fbdac014fe8, xdata=xdata at entry=0x0) at posix-acl.c:1200 1200 STACK_UNWIND_STRICT(open, frame, -1, EACCES, NULL, NULL); $9 = {path = 0x7fbdac009090 "/centos/test/foo", name = 0x0, inode = 0x7fbdac001e98, parent = 0x0, gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", pargfid = '\000' } $10 = {uid = 0, gid = 0, perm = 32768, fop = GF_FOP_READDIRP, acl_access = 0x0, acl_default = 0x0} Breakpoint 2, posix_acl_open (frame=frame at entry=0x7fbdac014fe8, this=this at entry=0x7fbdbc01dc00, loc=loc at entry=0x7fbdac000f30, flags=flags at entry=32768, fd=fd at entry=0x7fbdac0126a8, xdata=xdata at entry=0x0) at posix-acl.c:1200 1200 STACK_UNWIND_STRICT(open, frame, -1, EACCES, NULL, NULL); $11 = {path = 0x7fbdac007f20 "/centos/test/foo", name = 0x0, inode = 0x7fbdac001e98, parent = 0x0, gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", pargfid = '\000' } $12 = {uid = 0, gid = 0, perm = 32768, fop = GF_FOP_READDIRP, acl_access = 0x0, acl_default = 0x0} Breakpoint 2, posix_acl_open (frame=frame at entry=0x7fbdac0126a8, this=this at entry=0x7fbdbc01dc00, loc=loc at entry=0x7fbdac000f30, flags=flags at entry=32768, fd=fd at entry=0x7fbdac017b48, xdata=xdata at entry=0x0) at posix-acl.c:1200 1200 STACK_UNWIND_STRICT(open, frame, -1, EACCES, NULL, NULL); $13 = {path = 0x7fbdac009090 "/centos/test/foo", name = 0x0, inode = 0x7fbdac001e98, parent = 0x0, gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", pargfid = '\000' } $14 = {uid = 0, gid = 0, perm = 32768, fop = GF_FOP_READDIRP, acl_access = 0x0, acl_default = 0x0} Breakpoint 1, posix_acl_open (frame=frame at entry=0x7fbdac018b38, this=this at entry=0x7fbdbc01dc00, loc=loc at entry=0x7fbdac000f30, flags=flags at entry=32768, fd=fd at entry=0x7fbdac0179d8, xdata=xdata at entry=0x0) at posix-acl.c:1196 1196 STACK_WIND(frame, posix_acl_open_cbk, FIRST_CHILD(this), $15 = {path = 0x7fbdac007f20 "/centos/test/foo", name = 0x0, inode = 0x7fbdac001e98, parent = 0x0, gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", pargfid = '\000' } $16 = {uid = 1000, gid = 1000, perm = 33204, fop = GF_FOP_LOOKUP, acl_access = 0x0, acl_default = 0x0} -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 09:00:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 09:00:03 +0000 Subject: [Bugs] [Bug 1627060] ./tests/features/trash.t test case failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1627060 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) --- Comment #21 from Nithya Balachandran --- Please provide the hash of filenames rebal1 to rebal10 on big-endian. I will change the filename in the test once we figure out a name that works for both. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 09:28:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 09:28:47 +0000 Subject: [Bugs] [Bug 1676797] New: server xlator doesn't handle dict unserialization failures correctly Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676797 Bug ID: 1676797 Summary: server xlator doesn't handle dict unserialization failures correctly Product: GlusterFS Version: mainline Status: NEW Component: protocol Assignee: bugs at gluster.org Reporter: pkarampu at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: On code reading found that there exists dead-code which was handling dict unserialization failures on server-rpc-fops.c. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 09:49:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 09:49:49 +0000 Subject: [Bugs] [Bug 1676812] New: Manual Index heal throws error which is misguiding when heal is triggered to heal a brick if another brick is down Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676812 Bug ID: 1676812 Summary: Manual Index heal throws error which is misguiding when heal is triggered to heal a brick if another brick is down Product: GlusterFS Version: mainline Status: NEW Component: glusterd Keywords: ZStream Severity: high Priority: low Assignee: bugs at gluster.org Reporter: srakonde at redhat.com CC: amukherj at redhat.com, aspandey at redhat.com, bmekala at redhat.com, bugs at gluster.org, moagrawa at redhat.com, nchilaka at redhat.com, rhinduja at redhat.com, rhs-bugs at redhat.com, sankarshan at redhat.com, sheggodu at redhat.com, srakonde at redhat.com, storage-qa-internal at redhat.com, ubansal at redhat.com, vbellur at redhat.com Depends On: 1603082 Target Milestone: --- Classification: Community Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1603082 [Bug 1603082] Manual Index heal throws error which is misguiding when heal is triggered to heal a brick if another brick is down -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 09:54:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 09:54:17 +0000 Subject: [Bugs] [Bug 1627060] ./tests/features/trash.t test case failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1627060 --- Comment #22 from Nithya Balachandran --- Created attachment 1534338 --> https://bugzilla.redhat.com/attachment.cgi?id=1534338&action=edit Modified trash.t Please see if this works for you. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 09:54:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 09:54:33 +0000 Subject: [Bugs] [Bug 1627060] ./tests/features/trash.t test case failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1627060 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 09:55:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 09:55:12 +0000 Subject: [Bugs] [Bug 1676812] Manual Index heal throws error which is misguiding when heal is triggered to heal a brick if another brick is down In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676812 --- Comment #1 from Sanju --- Description of problem: Healing fails as long as a brick is down in a 4+2 EC volume Version-Release number of selected component (if applicable): ============================================================= glusterfs-server-3.12.2-13.el7rhgs.x86_64 How reproducible: Always (3/3) Steps to Reproduce: =================== 1.create a 4+2 ec vol 2.keep appending a file 3.bring down b1 4.wait for a minute or so and bring down b2 5.now again after a minute or so bring up b1 6.Healing fails to start for b1 Actual results: ============== Healing is failing for b1 Expected results: ================ Healing should start for b1 Additional info: =============== [root at dhcp35-56 ~]# gluster v heal dispersed Launching heal operation to perform index self heal on volume dispersed has been unsuccessful: Commit failed on 10.70.35.3. Please check log file for details. [root at dhcp35-56 ~]# 10.70.35.3 is the node which has b2 which is down. Logs - [2018-07-19 06:22:02.123328] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-dispersed-client-3: changing port to 49152 (from 0) [2018-07-19 06:22:02.132033] E [socket.c:2369:socket_connect_finish] 0-dispersed-client-3: connection to 10.70.35.3:49152 failed (Connection refused); disconnecting socket [2018-07-19 06:22:06.132851] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-dispersed-client-3: changing port to 49152 (from 0) [2018-07-19 06:22:06.137905] E [socket.c:2369:socket_connect_finish] 0-dispersed-client-3: connection to 10.70.35.3:49152 failed (Connection refused); disconnecting socket [2018-07-19 06:22:10.151806] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-dispersed-client-3: changing port to 49152 (from 0) [2018-07-19 06:22:10.156943] E [socket.c:2369:socket_connect_finish] 0-dispersed-client-3: connection to 10.70.35.3:49152 failed (Connection refused); disconnecting socket [2018-07-19 06:22:14.155717] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-dispersed-client-3: changing port to 49152 (from 0) [2018-07-19 06:22:14.163562] E [socket.c:2369:socket_connect_finish] 0-dispersed-client-3: connection to 10.70.35.3:49152 failed (Connection refused); disconnecting socket [2018-07-19 06:22:18.163595] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-dispersed-client-3: changing port to 49152 (from 0) [2018-07-19 06:22:18.172639] E [socket.c:2369:socket_connect_finish] 0-dispersed-client-3: connection to 10.70.35.3:49152 failed (Connection refused); disconnecting socket [2018-07-19 06:22:22.174819] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-dispersed-client-3: changing port to 49152 (from 0) [2018-07-19 06:22:22.184626] E [socket.c:2369:socket_connect_finish] 0-dispersed-client-3: connection to 10.70.35.3:49152 failed (Connection refused); disconnecting socket [2018-07-19 06:22:26 --- Additional comment from nchilaka on 2018-08-08 12:30:31 IST --- I discussed with Upasana, and based on futher analysis, below is the summary The heal was happening when i checked on my setup. However the error message is misguiding. Also the error message has a regression introduced Hence changing title. However, If incase Upasana, sees that the file is not healing(as she is unable to recollect at this point given that this bug was raised about 20days back), she will raise a new bug again, and also the reason behind calling a heal not happening. Also One very important note is that the error message is different between latest live 3.8.4-54.15 and 3.12.2-15 For the steps mentioned by Upasana, below is error message (used pkill): Also, Simple testcase, dont even have any IOs running have an ecvolume, kill brick on one node, then kill another brick on another node, and issue a heal command 3.8.4-54.15 : ----------- Launching heal operation to perform index self heal on volume ecv has been unsuccessful on bricks that are down. Please check if all brick processes are running. Note: I checked with kill -9/-15 and even with brickmux on , and saw the same error message 3.12.2-15: -------- Launching heal operation to perform index self heal on volume dispersed has been unsuccessful: Commit failed on 10.70.35.3. Please check log file for details. pkill glusterfsd//kill 15 Launching heal operation to perform index self heal on volume ecv has been unsuccessful: Glusterd Syncop Mgmt brick op 'Heal' failed. Please check glustershd log file for details. --- Additional comment from nchilaka on 2018-08-08 12:31:30 IST --- To confirm it is still a regression from CLI error part. Healing as such has no problem --- Additional comment from Atin Mukherjee on 2018-08-09 14:41:05 IST --- The only difference I see that instead of mentioning some of the bricks are down, we're highlighting "commit has failed on node x, please check log file" . The change which introduced this is as follows: Author: Mohit Agrawal Date: Tue Oct 25 19:57:02 2016 +0530 cli/afr: gluster volume heal info "healed" command output is not appropriate Problem: "gluster volume heal info [healed] [heal-failed]" command output on terminal is not appropriate in case of down any volume. Solution: To make message more appropriate change the condition in function "gd_syncop_mgmt_brick_op". Test : To verify the fix followed below procedure 1) Create 2*3 distribute replicate volume 2) set self-heal daemon off 3) kill two bricks (3, 6) 4) create some file on mount point 5) bring brick 3,6 up 6) kill other two brick (2 and 4) 7) make self heal daemon on 8) run "gluster v heal " Note: After apply the patch options (healed | heal-failed) will deprecate from command line. > BUG: 1388509 > Change-Id: I229c320c9caeb2525c76b78b44a53a64b088545a > Signed-off-by: Mohit Agrawal > (Cherry pick from commit d1f15cdeb609a1b720a04a502f7a63b2d3922f41) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 09:59:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 09:59:47 +0000 Subject: [Bugs] [Bug 1676812] Manual Index heal throws error which is misguiding when heal is triggered to heal a brick if another brick is down In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676812 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22209 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 09:59:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 09:59:53 +0000 Subject: [Bugs] [Bug 1676812] Manual Index heal throws error which is misguiding when heal is triggered to heal a brick if another brick is down In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676812 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22209 (cli/afr: modify the error message) posted (#1) for review on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 10:24:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 10:24:17 +0000 Subject: [Bugs] [Bug 1676797] server xlator doesn't handle dict unserialization failures correctly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676797 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22210 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 10:24:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 10:24:17 +0000 Subject: [Bugs] [Bug 1676797] server xlator doesn't handle dict unserialization failures correctly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676797 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22210 (Fix compilation for fops-sanity.c) posted (#1) for review on master by Pranith Kumar Karampuri -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 10:26:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 10:26:23 +0000 Subject: [Bugs] [Bug 1676797] server xlator doesn't handle dict unserialization failures correctly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676797 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22211 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 10:26:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 10:26:24 +0000 Subject: [Bugs] [Bug 1676797] server xlator doesn't handle dict unserialization failures correctly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676797 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22211 (protocol/server: Use SERVER_REQ_SET_ERROR correctly for dicts) posted (#1) for review on master by Pranith Kumar Karampuri -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 06:21:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 06:21:17 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #548 from Worker Ant --- REVIEW: https://review.gluster.org/22207 (rfc.sh: fix the missing rebase issue) merged (#1) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:06:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:06:29 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(abhaysingh1722 at ya | |hoo.in) | --- Comment #12 from abhays --- (In reply to Nithya Balachandran from comment #9) > The diffs for possible fixes for bug-1161311.t and bug-1193636.t are as > follows. Please make the changes on your setup and let me know if it works. > > > [root at rhgs313-6 tests]# git diff bugs/distribute/bug-1161311.t > diff --git a/tests/bugs/distribute/bug-1161311.t > b/tests/bugs/distribute/bug-1161311.t > index c52c69b..3dc45a4 100755 > --- a/tests/bugs/distribute/bug-1161311.t > +++ b/tests/bugs/distribute/bug-1161311.t > @@ -82,6 +82,8 @@ for i in {1..10}; do > cat /tmp/FILE2 >> $M0/dir1/FILE2 > done > > +brick_loc=$(get_backend_paths $M0/dir1/FILE2) > + > #dd if=/dev/urandom of=$M0/dir1/FILE2 bs=64k count=10240 > > # Rename the file to create a linkto, for rebalance to > @@ -99,7 +101,7 @@ TEST $CLI volume rebalance $V0 start force > > # Wait for FILE to get the sticky bit on, so that file is under > # active rebalance, before creating the links > -TEST checksticky $B0/${V0}3/dir1/FILE1 > +TEST checksticky $brick_loc > > # Create the links > ## FILE3 FILE5 FILE7 have hashes, c8c91469 566d26ce 22ce7eba > @@ -120,7 +122,7 @@ cd / > > # Ideally for this test to have done its job, the file should still be > # under migration, so check the sticky bit again > -TEST checksticky $B0/${V0}3/dir1/FILE1 > +TEST checksticky $brick_loc > > # Wait for rebalance to complete > EXPECT_WITHIN $REBALANCE_TIMEOUT "completed" rebalance_status_field $V0 > > > > [root at rhgs313-6 tests]# git diff ./bugs/distribute/bug-1193636.t > diff --git a/tests/bugs/distribute/bug-1193636.t > b/tests/bugs/distribute/bug-1193636.t > index ccde02e..6ffa2d9 100644 > --- a/tests/bugs/distribute/bug-1193636.t > +++ b/tests/bugs/distribute/bug-1193636.t > @@ -37,6 +37,8 @@ TEST mkdir $M0/dir1 > # Create a large file (1GB), so that rebalance takes time > dd if=/dev/zero of=$M0/dir1/FILE2 bs=64k count=10240 > > +brick_loc=$(get_backend_paths $M0/dir1/FILE2) > + > # Rename the file to create a linkto, for rebalance to > # act on the file > TEST mv $M0/dir1/FILE2 $M0/dir1/FILE1 > @@ -45,7 +47,7 @@ build_tester $(dirname $0)/bug-1193636.c > > TEST $CLI volume rebalance $V0 start force > > -TEST checksticky $B0/${V0}3/dir1/FILE1 > +TEST checksticky $brick_loc > > TEST setfattr -n "user.test1" -v "test1" $M0/dir1/FILE1 > TEST setfattr -n "user.test2" -v "test1" $M0/dir1/FILE1 (In reply to Nithya Balachandran from comment #10) > Possible fix for ./tests/bugs/posix/bug-1619720.t: > > > [root at rhgs313-6 tests]# git diff bugs/posix/bug-1619720.t > diff --git a/tests/bugs/posix/bug-1619720.t b/tests/bugs/posix/bug-1619720.t > index bfd304d..8584476 100755 > --- a/tests/bugs/posix/bug-1619720.t > +++ b/tests/bugs/posix/bug-1619720.t > @@ -1,6 +1,7 @@ > #!/bin/bash > > . $(dirname $0)/../../include.rc > +. $(dirname $0)/../../volume.rc > . $(dirname $0)/../../dht.rc > > cleanup; > @@ -35,7 +36,8 @@ TEST mkdir $M0/tmp > # file-2 will hash to the other subvol > > TEST touch $M0/tmp/file-2 > -pgfid_xattr_name=$(getfattr -m "trusted.pgfid.*" $B0/${V0}1/tmp/file-2 | > grep "trusted.pgfid") > +loc_2=$(get_backend_paths $M0/tmp/file-2) > +pgfid_xattr_name=$(getfattr -m "trusted.pgfid.*" $loc_2 | grep > "trusted.pgfid") > echo $pgfid_xattr_name Thanks for the reply @Nithya. Unfortunately the above changes for bugs/distribute/bug-1161311,bugs/distribute/bug-1193636.t and bugs/posix/bug-1619720.t do not work. PFA the logs for the same. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:08:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:08:36 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #13 from abhays --- Created attachment 1534374 --> https://bugzilla.redhat.com/attachment.cgi?id=1534374&action=edit bug-1161311.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:12:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:12:47 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) --- Comment #14 from Nithya Balachandran --- Please provide the complete test output and the gluster log files. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:13:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:13:52 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #15 from abhays --- Created attachment 1534375 --> https://bugzilla.redhat.com/attachment.cgi?id=1534375&action=edit bug-1161311_glusterd.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:14:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:14:56 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(abhaysingh1722 at ya | |hoo.in) | --- Comment #16 from abhays --- Created attachment 1534376 --> https://bugzilla.redhat.com/attachment.cgi?id=1534376&action=edit bug-1193636.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:15:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:15:44 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #17 from abhays --- Created attachment 1534377 --> https://bugzilla.redhat.com/attachment.cgi?id=1534377&action=edit bug-1193636_glusterd.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:16:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:16:20 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #18 from abhays --- Created attachment 1534378 --> https://bugzilla.redhat.com/attachment.cgi?id=1534378&action=edit bug-1619720.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:17:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:17:19 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #19 from abhays --- Created attachment 1534379 --> https://bugzilla.redhat.com/attachment.cgi?id=1534379&action=edit bug-1619720_glusterd.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 12:30:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 12:30:53 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) --- Comment #20 from Nithya Balachandran --- Hi, I need the client and brick logs (not glusterd which is the management daemon). Please do the following: Add the following lines to the test after the volume is started: TEST $CLI volume set $V0 client-log-level DEBUG TEST $CLI volume set $V0 brick-log-level DEBUG Run the test and send the client and brick logs. It might be that the hashe values on your system mean that files are on the same hashed subvol. Please send : the hashes of the file names in the tests the trusted.glusterfs.dht xattr values for the parent directories of these files on each brick -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 13:49:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 13:49:57 +0000 Subject: [Bugs] [Bug 1664590] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1676886 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676886 [Bug 1676886] [RFE] Enable gluster in IPv6 only scenarios -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 06:18:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 06:18:31 +0000 Subject: [Bugs] [Bug 1676736] tests: ./tests/bugs/distribute/bug-1161311.t times out In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676736 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-13 14:14:45 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22208 (tests/dht: Stop volume before unmounting bricks) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 15:11:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 15:11:29 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #21 from Nithya Balachandran --- Created attachment 1534434 --> https://bugzilla.redhat.com/attachment.cgi?id=1534434&action=edit bugs/distribute/bug-1161311.t -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 15:12:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 15:12:10 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #22 from Nithya Balachandran --- Created attachment 1534436 --> https://bugzilla.redhat.com/attachment.cgi?id=1534436&action=edit bugs/distribute/bug-1193636.t -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 17:33:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 17:33:05 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #12 from Worker Ant --- REVIEW: https://review.gluster.org/22184 (inode: make critical section smaller) merged (#10) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 18:20:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 18:20:26 +0000 Subject: [Bugs] [Bug 1676400] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-13 18:20:26 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22195 (cluster/dht: Fix lookup selfheal and rmdir race) merged (#3) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 18:22:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 18:22:37 -0000 Subject: [Bugs] [Bug 1651439] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651439 --- Comment #5 from Worker Ant --- REVIEW: https://review.gluster.org/21998 (dht: fix double extra unref of inode at heal path) merged (#4) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 18:24:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 18:24:38 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22160 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 18:24:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 18:24:39 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-13 18:24:39 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22160 (cluster/dht: Request linkto xattrs in dht_rmdir opendir) merged (#3) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 18:24:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 18:24:40 +0000 Subject: [Bugs] [Bug 1672869] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672869 Bug 1672869 depends on bug 1672851, which changed state. Bug 1672851 Summary: With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" https://bugzilla.redhat.com/show_bug.cgi?id=1672851 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 20:19:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 20:19:38 +0000 Subject: [Bugs] [Bug 1314508] tar complains: : file changed as we read it In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1314508 Andrew Robinson changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |anrobins at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 13 22:23:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 22:23:50 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 --- Comment #7 from asender at testlabs.com.au --- I also have the errors from client: data-common.log-20190210:[2019-02-07 04:00:34.845536] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 9cdaca25-8b70-4d5c-ab7c-23711af54f29, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:34.903357] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 3ddd4e64-38a2-456d-82c7-8361fd2f12a0, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:34.933803] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: ce5220ed-2ac0-436d-9c3b-6978fba22409, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:34.968269] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 01f64a01-6657-408e-80bc-2daa2fa4c3d6, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.001639] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: dd58bcb9-123c-4149-a101-87c145c8d75e, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.029941] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: afba3b44-82a4-4cc8-8412-2e6640b3ee41, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.062942] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 6e2761ce-4ae6-4a57-b77a-aca9d675d4a7, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.088658] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 95150a45-e2af-43be-97c5-3d6c68f2cc45, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.115121] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: beba071f-3107-4955-b132-6871c5a4b4a7, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.142953] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 08ce9205-f7c9-40da-bae7-0a6f313a2a4b, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.169342] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 219ae3ee-6783-4194-9952-32b859b6e9e6, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.195090] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: eec300d6-5e58-4dc4-9779-cb5872bcfde3, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.222026] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: e7ebe43c-611b-48c0-91da-bc8cff0e257d, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.248727] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 75e84ea7-ee13-499e-a9ed-9924ca795220, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.273546] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: bfb17595-25fd-48b1-b63c-5de711575e77, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.298552] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 41133cbc-0287-41e2-b38b-18e73d986b86, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 04:00:35.325860] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: c926e9f8-ec94-47de-a669-7f0b1623297e, req(uid:582601439,gid:582600513,perm:4,ngrps:1), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:(tag:1,perm:0,id:4294967295)(tag:2,perm:7,id:582601439)(tag:4,perm:5,id:4294967295)(tag:16,perm:0,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:[2019-02-07 22:42:43.312845] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 7f53991d-0db7-4f1c-8deb-35cd5bcda822, req(uid:582601182,gid:582600513,perm:1,ngrps:7), ctx(uid:0,gid:0,in-groups:0,perm:770,updated-fop:LOOKUP, acl:(tag:1,perm:7,id:4294967295)(tag:2,perm:7,id:1000)(tag:2,perm:7,id:582601439)(tag:2,perm:7,id:582601746)(tag:4,perm:0,id:4294967295)(tag:16,perm:7,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied] data-common.log-20190210:The message "I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: 7f53991d-0db7-4f1c-8deb-35cd5bcda822, req(uid:582601182,gid:582600513,perm:1,ngrps:7), ctx(uid:0,gid:0,in-groups:0,perm:770,updated-fop:LOOKUP, acl:(tag:1,perm:7,id:4294967295)(tag:2,perm:7,id:1000)(tag:2,perm:7,id:582601439)(tag:2,perm:7,id:582601746)(tag:4,perm:0,id:4294967295)(tag:16,perm:7,id:4294967295)(tag:32,perm:0,id:4294967295) [Permission denied]" repeated 3 times between [2019-02-07 22:42:43.312845] and [2019-02-07 22:42:43.314192] -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 22:39:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 22:39:27 +0000 Subject: [Bugs] [Bug 1620580] Deleted a volume and created a new volume with similar but not the same name. The kubernetes pod still keeps on running and doesn't crash. Still possible to write to gluster mount In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1620580 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bgoyal at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 13 22:39:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 13 Feb 2019 22:39:27 +0000 Subject: [Bugs] [Bug 1660404] Conditional freeing of string after returning from dict_set_dynstr function In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660404 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bgoyal at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 05:52:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 05:52:01 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) --- Comment #23 from Nithya Balachandran --- There was a mistake in the diffs sent earlier. Please try the modified .t files that I have attached and let me know if they work for you. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 06:22:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 06:22:21 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22212 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 06:22:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 06:22:22 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #549 from Worker Ant --- REVIEW: https://review.gluster.org/22212 (docs: fix typo in Developer Guide Readme) posted (#4) for review on master by Csaba Henk -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 06:23:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 06:23:48 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22213 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 06:23:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 06:23:49 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #550 from Worker Ant --- REVIEW: https://review.gluster.org/22213 (fuse lock interrupt: fix flock_interrupt.t) posted (#3) for review on master by Csaba Henk -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 06:25:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 06:25:57 +0000 Subject: [Bugs] [Bug 1654021] Gluster volume heal causes continuous info logging of "invalid argument" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654021 Rinku changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rkothiya at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 06:58:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 06:58:43 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22214 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 06:58:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 06:58:44 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #551 from Worker Ant --- REVIEW: https://review.gluster.org/22214 (upcall: some modifications to reduce work under lock) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 07:07:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 07:07:35 +0000 Subject: [Bugs] [Bug 1649054] glustereventsd does not start on Ubuntu 16.04 LTS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1649054 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |high CC| |sabose at redhat.com Assignee|bugs at gluster.org |avishwan at redhat.com -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 07:11:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 07:11:24 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1649054 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1649054 [Bug 1649054] glustereventsd does not start on Ubuntu 16.04 LTS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 07:43:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 07:43:48 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #552 from Worker Ant --- REVIEW: https://review.gluster.org/22212 (docs: fix typo in Developer Guide Readme) merged (#5) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 07:46:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 07:46:06 +0000 Subject: [Bugs] [Bug 1677160] New: Gluster 5 client can't access Gluster 3.12 servers Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Bug ID: 1677160 Summary: Gluster 5 client can't access Gluster 3.12 servers Product: GlusterFS Version: 5 Status: NEW Component: core Severity: urgent Assignee: bugs at gluster.org Reporter: sbonazzo at redhat.com CC: bugs at gluster.org Blocks: 1672318 Target Milestone: --- Classification: Community Originally reported on bug #1672318 for oVirt. In oVirt 4.2 we relied on Gluster 3.12. Being it now unmaintained we switched to Gluster 5. During the upgrade of a big datacenter it's impossible to upgrade all the hosts at the same time so at a given time there will be clients using 3.12 and 5 at the same time. In this situation 3.12 servers are still in place. Gluster 5 clients should still be able to work with Gluster 3.12 servers in order to allow clean upgrades. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672318 [Bug 1672318] "failed to fetch volume file" when trying to activate host in DC with glusterfs 3.12 domains -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 07:46:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 07:46:58 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Sandro Bonazzola changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(sabose at redhat.com | |) --- Comment #1 from Sandro Bonazzola --- Sahina, can you help here? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:22:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:22:09 +0000 Subject: [Bugs] [Bug 1676797] server xlator doesn't handle dict unserialization failures correctly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676797 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22210 (Fix compilation for fops-sanity.c) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:23:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:23:18 +0000 Subject: [Bugs] [Bug 1677174] New: Smoke test are failing for devrpm-el6 and strfmt_errors Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677174 Bug ID: 1677174 Summary: Smoke test are failing for devrpm-el6 and strfmt_errors Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: aspandey at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: Smoke tests are failing for the patch https://review.gluster.org/#/c/glusterfs/+/22125/ again and again. Failures are in https://build.gluster.org/job/strfmt_errors/16976/ : FAILURE https://build.gluster.org/job/devrpm-el6/14789/ : FAILURE Console logs are - 12:52:42 INFO: Cleaning up build root ('cleanup_on_failure=True') 12:52:42 Start: clean chroot 12:52:49 Finish: clean chroot 12:52:49 ERROR: Command failed: 12:52:49 # bash --login -c /usr/bin/rpmbuild -bb --target x86_64 --nodeps /builddir/build/SPECS/glusterfs.spec 12:52:49 12:52:49 Build step 'Execute shell' marked build as failure 12:52:49 Archiving artifacts 12:52:49 Performing Post build task... 12:52:49 Could not match :Build was aborted : False I was able to build rpm's on centos VM's. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:29:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:29:32 +0000 Subject: [Bugs] [Bug 1675076] [posix]: log the actual path wherever possible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1675076 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22192 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:29:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:29:33 +0000 Subject: [Bugs] [Bug 1675076] [posix]: log the actual path wherever possible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1675076 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-14 08:29:33 --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22192 (storage/posix: print the actual file path) merged (#3) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:50:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:50:48 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(abhaysingh1722 at ya | |hoo.in) | |needinfo?(abhaysingh1722 at ya | |hoo.in) | --- Comment #24 from abhays --- (In reply to Nithya Balachandran from comment #23) > There was a mistake in the diffs sent earlier. Please try the modified .t > files that I have attached and let me know if they work for you. Thanks for letting me know about this. The test cases bugs/distribute/bug-1193636.t and bugs/distribute/bug-1161311.t are passing successfully on big-endian. (In reply to Nithya Balachandran from comment #11) > (In reply to abhays from comment #6) > > Yes @Nithya, Below are the test cases that fail, their cause of failure and > > possible workaround on Big-Endian:- > > > > Following are the test cases which pass after changing the bricks in the > > test case:- > > ./tests/bugs/distribute/bug-1161311.t --------passes after changing brick3 > > to brick1 in subtests 31 and 41. > > ./tests/bugs/distribute/bug-1193636.t --------passes after changing brick3 > > to brick1 in subtest 10. > > > Provided diff of fix for these 2 tests in comment#9. Please try it out and > let me know if it works. > Working fine. > > > > ./tests/bugs/nfs/bug-847622.t ----------------passes after giving absolute > > path of testfile in subtest 9. > > > I don't see why this should be dependent on the hashing. Please provide > output of the test and the gluster logs with debug enabled when this fails. > > I agree @Nithya. This test might not be related to hashing. But please look into the attached logs for the same. > > > > > Following are the test cases that are still failing even after changing the > > bricks, however if little-endian hash values are hard-coded on big-endian in > > the file ./xlators/cluster/dht/src/dht-layout.c, then these test cases pass > > on Big-Endian:- > > ./tests/bugs/glusterfs/bug-902610.t-------------subtest 7 fails > > I don't see why this should be dependent on the hashing. Please provide > output of the test and the gluster logs with debug enabled when this fails. > @Nithya, I am quite certain this test fails due to differing hash values. Refer to comment #8 for the same.Providing the logs for the same. > > > ./tests/bugs/posix/bug-1619720.t----------------subtests 13 and 14 fail > > Diff of fix provided in comment#10. > > This test is failing with the changes shared. PFA the logs for the same. > > > > > Following test case is failing because of "Cannot allocate memory" issue:- > > ./tests/bitrot/bug-1207627-bitrot-scrub-status.t----------subtest 20 fails > > with the below error:- > > [client-rpc-fops_v2.c:961:client4_0_fgetxattr_cbk] 0-patchy-client-0: remote > > operation failed [Cannot allocate memory] > > > > This does not seem related to the hashing algorithm. Please check the brick > log to see if there are any errors. > > Providing logs for the same. > > > Following test case is failing on which issue has already been raised:- > > ./tests/features/trash.t-------------- > > https://bugzilla.redhat.com/show_bug.cgi?id=1627060 > > > > I'll take a look at this and see what can be done. > > > > So, please look into this and let us know if any workaround can be provided > > to make the above tests pass on Big-Endian. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:54:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:54:20 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #25 from abhays --- Created attachment 1534710 --> https://bugzilla.redhat.com/attachment.cgi?id=1534710&action=edit bug-847622_brick0.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:55:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:55:12 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment|0 |1 #1534374 is| | obsolete| | Attachment|0 |1 #1534375 is| | obsolete| | Attachment|0 |1 #1534376 is| | obsolete| | Attachment|0 |1 #1534377 is| | obsolete| | --- Comment #26 from abhays --- Created attachment 1534711 --> https://bugzilla.redhat.com/attachment.cgi?id=1534711&action=edit bug-847622_nfs.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:56:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:56:23 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #27 from abhays --- Created attachment 1534712 --> https://bugzilla.redhat.com/attachment.cgi?id=1534712&action=edit bug-847622_subtest_failure.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:57:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:57:42 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #28 from abhays --- Created attachment 1534713 --> https://bugzilla.redhat.com/attachment.cgi?id=1534713&action=edit bug-1619720_mnt_glusterfs-0.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 08:58:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 08:58:53 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment|0 |1 #1534379 is| | obsolete| | --- Comment #29 from abhays --- Created attachment 1534714 --> https://bugzilla.redhat.com/attachment.cgi?id=1534714&action=edit bug-1619720-patchy0.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 09:00:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 09:00:00 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #30 from abhays --- Created attachment 1534715 --> https://bugzilla.redhat.com/attachment.cgi?id=1534715&action=edit bug-1619720-patchy1.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 09:00:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 09:00:56 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #31 from abhays --- Created attachment 1534716 --> https://bugzilla.redhat.com/attachment.cgi?id=1534716&action=edit bug-902610_diff.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 09:04:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 09:04:34 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #32 from abhays --- Created attachment 1534720 --> https://bugzilla.redhat.com/attachment.cgi?id=1534720&action=edit bug-902610_mnt-glusterfs-0.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 09:05:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 09:05:21 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #33 from abhays --- Created attachment 1534721 --> https://bugzilla.redhat.com/attachment.cgi?id=1534721&action=edit bug-902610_patchy0.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 09:22:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 09:22:07 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment|0 |1 #1534378 is| | obsolete| | Attachment|0 |1 #1534710 is| | obsolete| | Attachment|0 |1 #1534711 is| | obsolete| | Attachment|0 |1 #1534712 is| | obsolete| | Attachment|0 |1 #1534713 is| | obsolete| | Attachment|0 |1 #1534714 is| | obsolete| | Attachment|0 |1 #1534715 is| | obsolete| | Attachment|0 |1 #1534716 is| | obsolete| | Attachment|0 |1 #1534720 is| | obsolete| | Attachment|0 |1 #1534721 is| | obsolete| | --- Comment #34 from abhays --- Created attachment 1534723 --> https://bugzilla.redhat.com/attachment.cgi?id=1534723&action=edit Zip Folder for all the logs -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 09:23:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 09:23:49 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #35 from abhays --- I have attached all the logs as per requested with the hash values too(in the Hash_Values.log file). Let me know if you need anything else. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 09:38:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 09:38:26 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Fixed In Version| |glusterfs-3.12.2-41 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 10:31:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 10:31:53 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Sayalee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nbalacha at redhat.com Flags| |needinfo?(nbalacha at redhat.c | |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 10:45:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 10:45:13 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) --- Comment #36 from Nithya Balachandran --- Does bugs/glusterfs/bug-902610.t pass if you replace kill_brick $V0 $H0 $B0/${V0}2 with kill_brick $V0 $H0 $B0/${V0}1 ? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 10:47:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 10:47:12 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #37 from Nithya Balachandran --- I have attached an updated trash.t to BZ#1627060. Please let me know if that works for you. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 10:55:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 10:55:29 +0000 Subject: [Bugs] [Bug 1654021] Gluster volume heal causes continuous info logging of "invalid argument" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654021 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22215 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 10:55:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 10:55:30 +0000 Subject: [Bugs] [Bug 1654021] Gluster volume heal causes continuous info logging of "invalid argument" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654021 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22215 (core: fix volume heal to avoid \"invalid argument\") posted (#1) for review on master by Rinku Kothiya -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 10:56:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 10:56:55 +0000 Subject: [Bugs] [Bug 1654021] Gluster volume heal causes continuous info logging of "invalid argument" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654021 Rinku changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |rkothiya at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:07:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:07:49 +0000 Subject: [Bugs] [Bug 1677174] Smoke test are failing for devrpm-el6 and strfmt_errors In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677174 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mscherer at redhat.com --- Comment #1 from M. Scherer --- The log indicate this fail because the build fail: https://build.gluster.org/job/devrpm-el6/14789/artifact/RPMS/el6/x86_64/build.log BUILDSTDERR: error: File not found: /builddir/build/BUILDROOT/glusterfs-7dev-0.19.gite6ec282.el6.x86_64/usr/lib64/glusterfs/7dev/xlator/features/thin-arbiter.so BUILDSTDERR: error: File not found: /builddir/build/BUILDROOT/glusterfs-7dev-0.19.gite6ec282.el6.x86_64/etc/init.d/gluster-ta-volume BUILDSTDERR: File not found: /builddir/build/BUILDROOT/glusterfs-7dev-0.19.gite6ec282.el6.x86_64/usr/lib64/glusterfs/7dev/xlator/features/thin-arbiter.so BUILDSTDERR: File not found: /builddir/build/BUILDROOT/glusterfs-7dev-0.19.gite6ec282.el6.x86_64/etc/init.d/gluster-ta-volume I suspect the problem is in the patch rather than the build system, unless proven otherwise -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:17:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:17:27 +0000 Subject: [Bugs] [Bug 1677174] Smoke test are failing for devrpm-el6 and strfmt_errors In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677174 --- Comment #2 from Ashish Pandey --- I will try to find out if there is anything wrong with my patch. However, in the mean time, is it possible to execute the build on any other machine if we have slightest doubt on machine. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:25:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:25:55 +0000 Subject: [Bugs] [Bug 1677174] Smoke test are failing for devrpm-el6 and strfmt_errors In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677174 --- Comment #3 from M. Scherer --- We have the same erroer on the strfmt error job, which got executed on others nodes: https://build.gluster.org/job/strfmt_errors/16974/ builder28 https://build.gluster.org/job/strfmt_errors/16976/ builder17 The job also work fine for other patchs, so I strongly think that a error message speaking of thin-arbitrer, when the patch do change something related to thin-arbitrer, that show a error on 4 differents builders who work fine for others patchs is strongly related to the patch. Does it work when you try to build in mock locally for you ? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:32:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:32:08 +0000 Subject: [Bugs] [Bug 1677174] Smoke test are failing for devrpm-el6 and strfmt_errors In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677174 --- Comment #4 from Deepshikha khandelwal --- As per your mail, I retriggered the build today afternoon and the same issue occurred on builder15 & builder28. So it's most likely the problem of this patch. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:43:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:43:44 +0000 Subject: [Bugs] [Bug 1677174] Smoke test are failing for devrpm-el6 and strfmt_errors In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677174 --- Comment #5 from M. Scherer --- So, I found the issue, cf comment on the review, so closing this one. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:44:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:44:00 +0000 Subject: [Bugs] [Bug 1677174] Smoke test are failing for devrpm-el6 and strfmt_errors In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677174 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |NOTABUG Last Closed| |2019-02-14 11:44:00 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:52:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:52:49 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(abhaysingh1722 at ya | |hoo.in) | --- Comment #38 from abhays --- (In reply to Nithya Balachandran from comment #36) > Does bugs/glusterfs/bug-902610.t pass if you replace > > kill_brick $V0 $H0 $B0/${V0}2 > with > kill_brick $V0 $H0 $B0/${V0}1 > > ? No, It doesn't. ========================= TEST 10 (line 67): 0 echo 1 not ok 10 Got "1" instead of "0", LINENUM:67 RESULT 10: 1 Failed 1/10 subtests Test Summary Report ------------------- ./tests/bugs/glusterfs/bug-902610.t (Wstat: 0 Tests: 10 Failed: 1) Failed test: 10 Files=1, Tests=10, 16 wallclock secs ( 0.04 usr 0.01 sys + 2.32 cusr 0.50 csys = 2.87 CPU) Result: FAIL End of test ./tests/bugs/glusterfs/bug-902610.t ================================================================================ -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:58:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:58:15 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #553 from Worker Ant --- REVIEW: https://review.gluster.org/22196 (dht-shared.c: minor reduction of work under lock.) merged (#2) on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:59:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:59:27 +0000 Subject: [Bugs] [Bug 1677260] New: rm -rf fails with "Directory not empty" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677260 Bug ID: 1677260 Summary: rm -rf fails with "Directory not empty" Product: GlusterFS Version: 6 Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org Depends On: 1676400 Blocks: 1458215, 1661258 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1676400 +++ Description of problem: When 2 clients run rm -rf concurrently, the operation fails with " Directory not empty" ls on the directory from the gluster mount point does not show any entries however there are directories on the bricks. Version-Release number of selected component (if applicable): How reproducible: Rare.This is a race condition. Steps to Reproduce: Steps: 1. Create 3x (2+1) arbiter volume and fuse mount it. Make sure lookup-optimize is enabled. 2. mkdir -p dir0/dir1/dir2. 3. Unmount and remount the volume to ensure a fresh lookup is sent. GDB into the fuse process and set a breakpoint at dht_lookup. 4. from the client mount: rm -rf mra_sources 5. When gdb breaks at dht_lookup for dir0/dir1/dir2, set a breakpoint at dht_lookup_cbk. Allow the process to continue until it hits dht_lookup_cbk. dht_lookup_cbk will return with op_ret = 0 . 6. Delete dir0/dir1/dir2 from every brick on the non-hashed subvols. 7. Set a breakpoint in dht_selfheal_dir_mkdir and allow gdb to continue. 8. When the process breaks at dht_selfheal_dir_mkdir, delete the directory from the hashed subvolume bricks. 9. In dht_selfheal_dir_mkdir_lookup_cbk, set a breakpoint at line : if (local->selfheal.hole_cnt == layout->cnt) { When gdb breaks at this point, set local->selfheal.hole_cnt to a value different from that of layout->cnt. Allow gdb to proceed. DHT will create the directories only on the non-hashed subvolumes as the layout has not been updated to indicate that the dir no longer exists on the hashed subvolume. This directory will no longer be visible on the mount point causing the rm -rf to fail. Actual results: root at server fuse1]# rm -rf mra_sources rm: cannot remove ?dir0/dir1?: Directory not empty Expected results: rm -rf should succeed. Additional info: As lookup-optimize is enabled, subsequent lookups cannot heal the directory. The same steps with lookup-optimize disabled will work as a subsequent lookup will lookup everywhere even if the entry does not exist on the hashed subvol. --- Additional comment from Nithya Balachandran on 2019-02-12 08:08:31 UTC --- RCA for the invisible directory left behind with concurrent rm -rf : -------------------------------------------------------------------- dht_selfheal_dir_mkdir_lookup_cbk (...) { ... 1381 this_call_cnt = dht_frame_return (frame); 1382 1383 LOCK (&frame->lock); 1384 { 1385 if ((op_ret < 0) && 1386 (op_errno == ENOENT || op_errno == ESTALE)) { 1387 local->selfheal.hole_cnt = !local->selfheal.hole_cnt ? 1 1388 : local->selfheal.hole_cnt + 1; 1389 } 1390 1391 if (!op_ret) { 1392 dht_iatt_merge (this, &local->stbuf, stbuf, prev); 1393 } 1394 check_mds = dht_dict_get_array (xattr, conf->mds_xattr_key, 1395 mds_xattr_val, 1, &errst); 1396 if (dict_get (xattr, conf->mds_xattr_key) && check_mds && !errst) { 1397 dict_unref (local->xattr); 1398 local->xattr = dict_ref (xattr); 1399 } 1400 1401 } 1402 UNLOCK (&frame->lock); 1403 1404 if (is_last_call (this_call_cnt)) { 1405 if (local->selfheal.hole_cnt == layout->cnt) { 1406 gf_msg_debug (this->name, op_errno, 1407 "Lookup failed, an rmdir could have " 1408 "deleted this entry %s", loc->name); 1409 local->op_errno = op_errno; 1410 goto err; 1411 } else { 1412 for (i = 0; i < layout->cnt; i++) { 1413 if (layout->list[i].err == ENOENT || 1414 layout->list[i].err == ESTALE || 1415 local->selfheal.force_mkdir) 1416 missing_dirs++; 1417 } There are 2 problems here: 1. The layout is not updated with the new subvol status on error. In this case, the initial lookup found a directory on the hashed subvol so only 2 entries in the layout indicate missing directories. However, by the time the selfheal code is executed, the racing rmdir has deleted the directory from all the subvols. At this point, the directory does not exist on any subvol and dht_selfheal_dir_mkdir_lookup_cbk gets an error from all 3 subvols, but this new status is not updated in the layout which still has only 2 missing dirs marked. 2. this_call_cnt = dht_frame_return (frame); is called before processing the frame. So with a call cnt of 3, it is possible that the second response has reached 1404 before the third one has started processing the return values. At this point, local->selfheal.hole_cnt != layout->cnt so control goes to line 1412. At line 1412, since we are still using the old layout, only the directories on the non-hashed subvols are considered when incrementing missing_dirs and for the healing. The combination of these two causes the selfheal to start healing the directories on the non-hashed subvols. It succeeds in creating the dirs on the non-hashed subvols. However, to set the layout, dht takes an inodelk on the hashed subvol which fails because the directory does on exist there. We therefore end up with directories on the non-hashed subvols with no layouts set. --- Additional comment from Worker Ant on 2019-02-12 08:34:01 UTC --- REVIEW: https://review.gluster.org/22195 (cluster/dht: Fix lookup selfheal and rmdir race) posted (#1) for review on master by N Balachandran --- Additional comment from Worker Ant on 2019-02-13 18:20:26 UTC --- REVIEW: https://review.gluster.org/22195 (cluster/dht: Fix lookup selfheal and rmdir race) merged (#3) on master by Raghavendra G Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1458215 [Bug 1458215] Slave reports ENOTEMPTY when rmdir is executed on master https://bugzilla.redhat.com/show_bug.cgi?id=1676400 [Bug 1676400] rm -rf fails with "Directory not empty" -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 11:59:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 11:59:27 +0000 Subject: [Bugs] [Bug 1676400] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676400 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1677260 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1677260 [Bug 1677260] rm -rf fails with "Directory not empty" -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 12:07:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 12:07:48 +0000 Subject: [Bugs] [Bug 1677260] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677260 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22216 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 12:07:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 12:07:48 +0000 Subject: [Bugs] [Bug 1677260] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677260 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22216 (cluster/dht: Fix lookup selfheal and rmdir race) posted (#1) for review on release-6 by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 12:18:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 12:18:16 +0000 Subject: [Bugs] [Bug 1627060] ./tests/features/trash.t test case failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1627060 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(abhaysingh1722 at ya | |hoo.in) | |needinfo?(abhaysingh1722 at ya | |hoo.in) | --- Comment #23 from abhays --- (In reply to Nithya Balachandran from comment #22) > Created attachment 1534338 [details] > Modified trash.t > > Please see if this works for you. No, It doesn't. ========================= TEST 41 (line 210): Y wildcard_exists /d/backends/patchy1/.trashcan/internal_op/* not ok 41 Got "N" instead of "Y", LINENUM:210 RESULT 41: 1 ========================= ========================= TEST 65 (line 264): Y wildcard_exists /d/backends/patchy1/abc/internal_op/rebal* not ok 65 Got "N" instead of "Y", LINENUM:264 RESULT 65: 1 rm: cannot remove '/mnt/glusterfs/0/abc/internal_op': Operation not permitted ========================= Failed 2/68 subtests Test Summary Report ------------------- ./tests/features/trash.t (Wstat: 0 Tests: 68 Failed: 2) Failed tests: 41, 65 Files=1, Tests=68, 86 wallclock secs ( 0.09 usr 0.01 sys + 16.29 cusr 2.30 csys = 18.69 CPU) Result: FAIL End of test ./tests/features/trash.t ================================================================================ PFA the logs for the same. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 12:19:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 12:19:12 +0000 Subject: [Bugs] [Bug 1627060] ./tests/features/trash.t test case failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1627060 --- Comment #24 from abhays --- Created attachment 1534784 --> https://bugzilla.redhat.com/attachment.cgi?id=1534784&action=edit trash_logs -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 12:22:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 12:22:18 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22217 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 12:22:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 12:22:19 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #39 from Worker Ant --- REVIEW: https://review.gluster.org/22217 (tests/dht: Remove hardcoded brick paths) posted (#1) for review on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 13:06:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 13:06:06 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |atumball at redhat.com Flags|needinfo?(sabose at redhat.com |needinfo?(atumball at redhat.c |) |om) --- Comment #2 from Sahina Bose --- Amar, do you know of any issues with 3.12 clients connecting to gluster 5 servers. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 13:08:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 13:08:29 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 --- Comment #3 from Sandro Bonazzola --- (In reply to Sahina Bose from comment #2) > Amar, do you know of any issues with 3.12 clients connecting to gluster 5 > servers. this is gluster 5 clients connecting to gluster 3 servers. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 13:37:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 13:37:31 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |urgent Assignee|bugs at gluster.org |srakonde at redhat.com Flags|needinfo?(atumball at redhat.c | |om) | --- Comment #4 from Sahina Bose --- Assigning to Sanju who's looking into it. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 13:50:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 13:50:48 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sbonazzo at redhat.com Flags| |needinfo?(sbonazzo at redhat.c | |om) --- Comment #5 from Sanju --- Sandro, Can you please provide mount logs and bricks logs? Thanks, Sanju -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 14:00:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 14:00:47 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Sandro Bonazzola changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(sbonazzo at redhat.c | |om) | --- Comment #6 from Sandro Bonazzola --- (In reply to Sanju from comment #5) > Sandro, > > Can you please provide mount logs and bricks logs? Added you to the community report of this issue so you can interact directly with original reporter. > > Thanks, > Sanju -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 14:22:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 14:22:45 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Netbulae changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |info at netbulae.com --- Comment #7 from Netbulae --- There are no brick logs at the client and nothing in the brick logs on the glusterfs servers regarding these hosts. [2019-02-04 12:47:02.979349] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-02-04 12:58:10.232517] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.3 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=*.*.*.16 --volfile-server=*.*.*.15 --volfile-server=*.*.*.14 --volfile-id=ssd5 /rhev/data-center/mnt/glusterSD/*.*.*.16:ssd5) [2019-02-04 12:58:10.242924] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-02-04 13:02:44.511106] I [glusterfsd-mgmt.c:2424:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: *.*.*.16 [2019-02-04 13:02:44.511157] I [glusterfsd-mgmt.c:2464:mgmt_rpc_notify] 0-glusterfsd-mgmt: connecting to next volfile server *.*.*.15 [2019-02-04 13:02:44.512757] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f58b4ccffbb] (--> /lib64/libgfrpc.so.0(+0xce11)[0x7f58b4a98e11] (--> /lib64/libgfrpc.so.0(+0xcf2e)[0x7f58b4a98f2e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7f58b4a9a531] (--> /lib64/libgfrpc.so.0(+0xf0d8)[0x7f58b4a9b0d8] ))))) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2019-02-04 12:47:02.979593 (xid=0x2) [2019-02-04 13:02:44.512779] E [glusterfsd-mgmt.c:2136:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:ssd5) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 14:23:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 14:23:34 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Sandro Bonazzola changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1677319 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 14:23:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 14:23:34 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Sandro Bonazzola changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1677319 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 14:31:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 14:31:08 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 James changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jayme at silverorange.com --- Comment #33 from James --- I'm also having problems with Gluster bricks going offline since upgrading to oVirt 4.3 yesterday (previously I've never had a single issue with gluster nor have had a brick ever go down). I suspect this will continue to happen daily as some other users on this group have suggested. I was able to pull some logs from engine and gluster from around the time the brick dropped. My setup is 3 node HCI and I was previously running the latest 4.2 updates (before upgrading to 4.3). My hardware is has a lot of overhead and I'm on 10Gbe gluster backend (the servers were certainly not under any significant amount of load when the brick went offline). To recover I had to place the host in maintenance mode and reboot (although I suspect I could have simply unmounted and remounted gluster mounts). grep "2019-02-14" engine.log-20190214 | grep "GLUSTER_BRICK_STATUS_CHANGED" 2019-02-14 02:41:48,018-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from UP to DOWN via cli. 2019-02-14 03:20:11,189-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/engine/engine of volume engine of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:14,819-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/prod_b/prod_b of volume prod_b of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:19,692-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/isos/isos of volume isos of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:25,022-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/prod_a/prod_a of volume prod_a of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:29,088-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:34,099-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_a/non_prod_a of volume non_prod_a of cluster Default from DOWN to UP via cli glusterd.log # grep -B20 -A20 "2019-02-14 02:41" glusterd.log [2019-02-14 02:36:49.585034] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:36:49.597788] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:36:49.597788] and [2019-02-14 02:36:49.900505] [2019-02-14 02:36:53.437539] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:36:53.452816] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:36:53.864153] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:36:53.875835] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:36:30.958649] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:36:35.322129] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b [2019-02-14 02:36:39.639645] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:36:45.301275] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:36:53.875835] and [2019-02-14 02:36:54.180780] [2019-02-14 02:37:59.193409] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:38:44.065560] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:38:44.072680] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:38:44.077841] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:38:44.082798] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:38:44.088237] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a [2019-02-14 02:38:44.093518] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:37:59.193409] and [2019-02-14 02:38:44.100494] [2019-02-14 02:41:58.649683] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 6 times between [2019-02-14 02:41:58.649683] and [2019-02-14 02:43:00.286999] [2019-02-14 02:43:46.366743] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:43:46.373587] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:43:46.378997] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:43:46.384324] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:43:46.390310] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a [2019-02-14 02:43:46.397031] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b [2019-02-14 02:43:46.404083] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:45:47.302884] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:45:47.309697] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:45:47.315149] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:45:47.320806] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:45:47.326865] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a [2019-02-14 02:45:47.332192] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b [2019-02-14 02:45:47.338991] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:46:47.789575] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:46:47.795276] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a [2019-02-14 02:46:47.800584] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b [2019-02-14 02:46:47.770601] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:46:47.778161] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:46:47.784020] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a engine.log # grep -B20 -A20 "2019-02-14 02:41:48" engine.log-20190214 2019-02-14 02:41:43,495-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 172c9ee8 2019-02-14 02:41:43,609-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 479fcb69, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 6443e68f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 2b4cf035, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 5864f06a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 6119ac8c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1a9549be, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 5614cf81, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 290c9289, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 5dd26e8, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 35355754, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 452deeb4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 8f8b442, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 647e29d3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7bee4dff, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 511c4478, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1c0bb0bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 92e325e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 260731, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 33aaacc9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 72657c59, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at aa10c89], log id: 172c9ee8 2019-02-14 02:41:43,610-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 3a0e9d63 2019-02-14 02:41:43,703-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 5ca4a20f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 57a8a76, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 7bd1b14], log id: 3a0e9d63 2019-02-14 02:41:43,704-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 49966b05 2019-02-14 02:41:44,213-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 49966b05 2019-02-14 02:41:44,214-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 30db0ce2 2019-02-14 02:41:44,311-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 61a309b5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at ea9cb2e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 749d57bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1c49f9d0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 655eb54d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 256ee273, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 3bd079dc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 6804900f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 78e0a49f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 2acfbc8a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 12e92e96, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 5ea1502c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 2398c33b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7464102e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 2f221daa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7b561852, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1eb29d18, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 4a030b80, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 75739027, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 3eac8253, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 34fc82c3], log id: 30db0ce2 2019-02-14 02:41:44,312-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 6671d0d7 2019-02-14 02:41:44,329-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:44,345-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:44,374-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:44,405-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at f6a9696, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 558e3332, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 5b449da], log id: 6671d0d7 2019-02-14 02:41:44,406-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 6d2bc6d3 2019-02-14 02:41:44,908-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6d2bc6d3 2019-02-14 02:41:44,909-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5', volumeName='non_prod_b'}), log id: 36ae23c6 2019-02-14 02:41:47,336-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:47,351-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:47,379-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:47,979-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails at 7a4a787b, log id: 36ae23c6 2019-02-14 02:41:48,018-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from UP to DOWN via cli. 2019-02-14 02:41:48,046-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_DOWN(4,151), Status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b on cluster Default is down. 2019-02-14 02:41:48,139-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:48,140-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] START, GlusterServersListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: e1fb23 2019-02-14 02:41:48,911-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterServersListVDSCommand, return: [10.12.0.220/24:CONNECTED, host1.replaced.domain.com:CONNECTED, host2.replaced.domain.com:CONNECTED], log id: e1fb23 2019-02-14 02:41:48,930-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:48,931-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] START, GlusterVolumesListVDSCommand(HostName = Host0, GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 68f1aecc 2019-02-14 02:41:49,366-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterVolumesListVDSCommand, return: {6c05dfc6-4dc0-41e3-a12f-55b4767f1d35=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 1952a85, 3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 2f6688ae, 71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 730210fb, 752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c3be510c, c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 450befac, 79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 1926e392}, log id: 68f1aecc 2019-02-14 02:41:49,489-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 38debe74 2019-02-14 02:41:49,581-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 5e5a7925, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 2cdf5c9e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 443cb62, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 49a3e880, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 443d23c0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1250bc75, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 8d27d86, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 5e6363f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 73ed78db, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 64c9d1c7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7fecbe95, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 3a551e5f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 2266926e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 88b380c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1209279e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 3c6466, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 16df63ed, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 47456262, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1c2b88c3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7f57c074, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 12fa0478], log id: 38debe74 2019-02-14 02:41:49,582-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 7ec02237 2019-02-14 02:41:49,660-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 3eedd0bc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 7f78e375, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 3d63e126], log id: 7ec02237 2019-02-14 02:41:49,661-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 42cdad27 2019-02-14 02:41:50,142-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 42cdad27 2019-02-14 02:41:50,143-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 12f5fdf2 2019-02-14 02:41:50,248-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 2aaed792, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 8e66930, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 276d599e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1aca2aec, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 46846c60, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7d103269, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 30fc25fc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7baae445, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1ea8603c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 62578afa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 33d58089, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1f71d27a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 4205e828, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1c5bbac8, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 395a002, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 12664008, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7f4faec4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 3e03d61f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1038e46d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 307e8062, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 32453127], log id: 12f5fdf2 2019-02-14 02:41:50,249-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 1256aa5e 2019-02-14 02:41:50,338-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 459a2ff5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 123cab4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 1af41fbe], log id: 1256aa5e 2019-02-14 02:41:50,339-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 3dd752e4 2019-02-14 02:41:50,847-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 3dd752e4 2019-02-14 02:41:50,848-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 29a6272c 2019-02-14 02:41:50,954-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 364f3ec6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at c7cce5e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at b3bed47, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 13bc244b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 5cca81f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 36aeba0d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 62ab384a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 1047d628, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 188a30f5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 5bb79f3b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 60e5956f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 4e3df9cd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 7796567, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 60d06cf4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 2cd2d36c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at d80a4aa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 411eaa20, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 22cac93b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 18b927bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 101465f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume at 246f927c], log id: 29a6272c 2019-02-14 02:41:50,955-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 501814db 2019-02-14 02:41:51,044-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 1cd55aa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 32c5aba2, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume at 6ae123f4], log id: 501814db 2019-02-14 02:41:51,045-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 7acf4cbf 2019-02-14 02:41:51,546-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 7acf4cbf 2019-02-14 02:41:51,547-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5', volumeName='non_prod_a'}), log id: 11c42649 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 14 14:31:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 14:31:39 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Sandro Bonazzola changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1677319 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 14 18:31:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 14 Feb 2019 18:31:22 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |urgent CC| |sabose at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 15 02:56:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 02:56:03 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #40 from Nithya Balachandran --- > > ./tests/bugs/posix/bug-1619720.t----------------subtests 13 and 14 fail > > Diff of fix provided in comment#10. > > This test is failing with the changes shared. PFA the logs for the same. As it is difficult for me to figure out what is happening without a Big Endian system, I would encourage you to understand what is expected and try to make the changes yourself. We will be happy to take your patches if they work for us as well. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 04:23:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 04:23:31 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #41 from abhays --- (In reply to Nithya Balachandran from comment #40) > > > ./tests/bugs/posix/bug-1619720.t----------------subtests 13 and 14 fail > > > > Diff of fix provided in comment#10. > > > > > > This test is failing with the changes shared. PFA the logs for the same. > > As it is difficult for me to figure out what is happening without a Big > Endian system, I would encourage you to understand what is expected and try > to make the changes yourself. We will be happy to take your patches if they > work for us as well. Yes sure. We are trying to debug further. Additionally, we'll need your timely help in resolving these failures. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 05:55:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 05:55:49 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #42 from abhays --- @Nithya, Do ya'll have a Jenkins CI Infrastructure where continuous builds are executed for glusterfs. We have come across the below links regarding the same:- https://ci.centos.org/label/gluster/ https://build.gluster.org/ Can you please confirm about these? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 07:46:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 07:46:33 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22221 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 15 07:46:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 07:46:34 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #34 from Worker Ant --- REVIEW: https://review.gluster.org/22221 (socket: socket event handlers now return void) posted (#1) for review on master by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 15 08:44:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 08:44:10 +0000 Subject: [Bugs] [Bug 1677555] New: Glusterfs brick is crashed due to segfault caused by broken gfid symlink Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677555 Bug ID: 1677555 Summary: Glusterfs brick is crashed due to segfault caused by broken gfid symlink Product: GlusterFS Version: 4.1 Hardware: x86_64 OS: Linux Status: NEW Component: core Severity: high Assignee: bugs at gluster.org Reporter: ruslanloman at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Glusterfs brick is crashed due to segfault caused by broken gfid symlink. # gdb /usr/sbin/glusterfsd core.12867 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfsd -s glusterv3-04.namecheapcloud.net --volfile-id easywp_pro'. Program terminated with signal 11, Segmentation fault. #0 __strftime_internal (s=0x7f85e44863f0 "", maxsize=256, format=0x7f861bb39dfb "%F %T", tp=0x7f85e44863b0, tzset_called=tzset_called at entry=0x7f85e4486320, loc=0xce4bbb31a32ca014) at strftime_l.c:472 472 struct __locale_data *const current = loc->__locales[LC_TIME]; Missing separate debuginfos, use: debuginfo-install glusterfs-fuse-4.1.7-1.el7.x86_64 (gdb) bt #0 __strftime_internal (s=0x7f85e44863f0 "", maxsize=256, format=0x7f861bb39dfb "%F %T", tp=0x7f85e44863b0, tzset_called=tzset_called at entry=0x7f85e4486320, loc=0xce4bbb31a32ca014) at strftime_l.c:472 #1 0x00007f861a172423 in __GI___strftime_l (s=, maxsize=, format=, tp=, loc=) at strftime_l.c:459 #2 0x00007f861ba89414 in gf_glusterlog_log_repetitions.isra.6 () from /lib64/libglusterfs.so.0 #3 0x00007f861ba89953 in gf_log_flush_message () from /lib64/libglusterfs.so.0 #4 0x00007f861ba89a39 in gf_log_flush_list () from /lib64/libglusterfs.so.0 #5 0x00007f861ba89cbd in gf_log_set_log_buf_size () from /lib64/libglusterfs.so.0 #6 0x00007f861ba89d17 in gf_log_disable_suppression_before_exit () from /lib64/libglusterfs.so.0 #7 0x00007f861ba905c5 in gf_print_trace () from /lib64/libglusterfs.so.0 #8 #9 __GI_____strtoul_l_internal (nptr=nptr at entry=0x7f85e449e18c "0cee86d5-51c9-4094-a670-091ebc518c08", endptr=endptr at entry=0x0, base=base at entry=16, group=group at entry=0, loc=0xce4bbb31a32ca014) at ../stdlib/strtol_l.c:241 #10 0x00007f861a0efe22 in __GI_strtoul (nptr=nptr at entry=0x7f85e449e18c "0cee86d5-51c9-4094-a670-091ebc518c08", endptr=endptr at entry=0x0, base=base at entry=16) at ../stdlib/strtol.c:103 #11 0x00007f861b20f4bf in uuid_parse (in=0x7f85e449e18c "0cee86d5-51c9-4094-a670-091ebc518c08", uu=0x7f85e44ab2f0 "\024\240,\243\061\273K?g\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273K?g\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273K?g\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273K?g\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273K?g\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273K?g\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273", ...) at libuuid/src/parse.c:65 #12 0x00007f860e395f4f in posix_make_ancestryfromgfid () from /usr/lib64/glusterfs/4.1.7/xlator/storage/posix.so #13 0x088c51bc1e0970a6 in ?? () If we check this gfid we'll see that it's broken symlink: # file 0cee86d5-51c9-4094-a670-091ebc518c08 0cee86d5-51c9-4094-a670-091ebc518c08: broken symbolic link to `../../14/a0/14a02ca3-31bb-4bce-9167-e560517c2a6a/..' # file 14a02ca3-31bb-4bce-9167-e560517c2a6a 14a02ca3-31bb-4bce-9167-e560517c2a6a: broken symbolic link to `../../0c/ee/0cee86d5-51c9-4094-a670-091ebc518c08/wp-admin' After removing those broken symlink and execution 'gluster volume start force', brick is back online. We have another gluster cluster version 4.1.5, it has broken symlinks as well, but brick is never crushed. Version-Release number of selected component (if applicable): # uname -r 4.18.16-1.el7.elrepo.x86_64 # rpm -qa|grep gluster glusterfs-4.1.7-1.el7.x86_64 glusterfs-fuse-4.1.7-1.el7.x86_64 glusterfs-geo-replication-4.1.7-1.el7.x86_64 centos-release-gluster41-1.0-3.el7.centos.noarch glusterfs-libs-4.1.7-1.el7.x86_64 glusterfs-client-xlators-4.1.7-1.el7.x86_64 glusterfs-extra-xlators-4.1.7-1.el7.x86_64 glusterfs-api-4.1.7-1.el7.x86_64 glusterfs-server-4.1.7-1.el7.x86_64 glusterfs-devel-4.1.7-1.el7.x86_64 python2-gluster-4.1.7-1.el7.x86_64 glusterfs-cli-4.1.7-1.el7.x86_64 rpm -qa|grep gcc libgcc-4.8.5-36.el7.x86_64 rpm -qa|grep glibc glibc-common-2.17-260.el7.x86_64 glibc-debuginfo-common-2.17-260.el7.x86_64 glibc-2.17-260.el7.x86_64 glibc-debuginfo-2.17-260.el7.x86_64 How reproducible: Steps to Reproduce: 1. Deploy gluster 4.1.7 in distributed replicated mode 2. Find symlink in gfid directory (.glusterfs) for one of brick and break it 3. Brick with broken symlink should be crashed Actual results: Crushed with segfault Expected results: Brick should ignore broken symlink -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 08:47:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 08:47:22 +0000 Subject: [Bugs] [Bug 1677557] New: gNFS crashed when processing "gluster v profile [vol] info nfs" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677557 Bug ID: 1677557 Summary: gNFS crashed when processing "gluster v profile [vol] info nfs" Product: GlusterFS Version: mainline Status: NEW Component: nfs Assignee: bugs at gluster.org Reporter: jefferymymy at 163.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: when processing "gluster v profile [vol] info nfs" after gnfs start, gnfs will crash. dump trace info: /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7fcf5cb6a872] /lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7fcf5cb743a4] /lib64/libc.so.6(+0x35670)[0x7fcf5b1d5670] /usr/sbin/glusterfs(glusterfs_handle_nfs_profile+0x114)[0x7fcf5d066474] /lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7fcf5cba1502] /lib64/libc.so.6(+0x47110)[0x7fcf5b1e7110] Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.create dht volume naming dht_vol,and start the vol 2.start volume profile 3.kill gnfs process 4.process cli "service glusterd restart;gluster volume profile dht_vol info nfs" Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 08:50:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 08:50:08 +0000 Subject: [Bugs] [Bug 1677559] New: gNFS crashed when processing "gluster v profile [vol] info nfs" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677559 Bug ID: 1677559 Summary: gNFS crashed when processing "gluster v profile [vol] info nfs" Product: GlusterFS Version: mainline Status: NEW Component: nfs Assignee: bugs at gluster.org Reporter: jefferymymy at 163.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: when processing "gluster v profile [vol] info nfs" after gnfs start, gnfs will crash. dump trace info: /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7fcf5cb6a872] /lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7fcf5cb743a4] /lib64/libc.so.6(+0x35670)[0x7fcf5b1d5670] /usr/sbin/glusterfs(glusterfs_handle_nfs_profile+0x114)[0x7fcf5d066474] /lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7fcf5cba1502] /lib64/libc.so.6(+0x47110)[0x7fcf5b1e7110] Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.create dht volume naming dht_vol,and start the vol 2.start volume profile 3.kill gnfs process 4.process cli "service glusterd restart;gluster volume profile dht_vol info nfs" Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 12:49:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 12:49:55 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Kaleb KEITHLEY changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sabose at redhat.com Flags|needinfo?(jthottan at redhat.c |needinfo?(sabose at redhat.com |om) |) --- Comment #4 from Kaleb KEITHLEY --- Does removing glusterfs-gnfs manually/explicitly, followed by update, work? AFAIK oVirt doesn't use NFS at all and never needed glusterfs-gnfs in the first place. If glusterfs-gnfs was being used on the same system for something else, it should be replaced with a nfs-ganesha deployment, also from the Storage SIG. And you can't use an old version of glusterfs-gnfs (e.g. 3.12.x) with glusterfs-5.x. I will close this as NOTABUG in a few days unless I hear back. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 12:50:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 12:50:32 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Kaleb KEITHLEY changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sbonazzo at redhat.com Flags| |needinfo?(sbonazzo at redhat.c | |om) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 13:32:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 13:32:27 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nbalacha at redhat.com --- Comment #8 from Nithya Balachandran --- (In reply to asender at testlabs.com.au from comment #5) > Could someone kindly provide a "rollback" procedure with minimal impact. Can > we set Gluster back to version 4 compatibility mode and downgrade.? Prefer > non-impacting, but whatever is the safest. > > We need to go back to version 4. >From Kaushal on IRC: sendro, To rollback do the following. 1. Kill glusterds on all the nodes. 2. Edit /var/lib/gluster/glusterd.info and manually change opversion to what you want. Do this on all the nodes. 3. Downgrade glusterfs-server to the version you want. 4. Restart glusterd. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 13:56:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 13:56:06 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(sabose at redhat.com |needinfo?(kkeithle at redhat.c |) |om) |needinfo?(sbonazzo at redhat.c | |om) | --- Comment #5 from Sahina Bose --- (In reply to Kaleb KEITHLEY from comment #4) > Does removing glusterfs-gnfs manually/explicitly, followed by update, work? That works, but is not the option that we want to provide users. Since updates happen from UI, we want this to be as seamless as possible. > > AFAIK oVirt doesn't use NFS at all and never needed glusterfs-gnfs in the > first place. OVirt users who use nfs option to mount gluster, do use glusterfs-gnfs. And we had a dependency on glusterfs-gnfs to workaround an earlier bug, so this package is now available on all hosts. > > If glusterfs-gnfs was being used on the same system for something else, it > should be replaced with a nfs-ganesha deployment, also from the Storage SIG. > > And you can't use an old version of glusterfs-gnfs (e.g. 3.12.x) with > glusterfs-5.x. > > I will close this as NOTABUG in a few days unless I hear back. We want the upgrade from 3.12 to > 4 versions to work. So adding Obsoletes in glusterfs spec as suggested by others - is this a possible option? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 17:45:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 17:45:57 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #20 from David E. Smith --- Obviously, in this case, the absence of a crash isn't necessarily proof of anything. But for what it's worth, my FUSE clients have been up for four days without crashing, which is probably the longest they've made it since I updated to 5.2 and 5.3. I'll report back in a few more days. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:46 +0000 Subject: [Bugs] [Bug 1350477] Test to check if the maintainer reviewed the patch In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1350477 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:47 +0000 Subject: [Bugs] [Bug 1357421] Fail smoke tests if cherry-picked bugs contain the old git-tags In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1357421 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:49 +0000 Subject: [Bugs] [Bug 1431199] Request to automate closing github PRs In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1431199 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:48 +0000 Subject: [Bugs] [Bug 1428047] Require a Jenkins job to validate Change-ID on commits to branches in glusterfs repository In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1428047 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:50 +0000 Subject: [Bugs] [Bug 1439706] Change default name in gerrit patch In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1439706 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:51 +0000 Subject: [Bugs] [Bug 1463273] infra: include bugzilla query in the weekly BZ email In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1463273 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:53 +0000 Subject: [Bugs] [Bug 1562670] Run libgfapi-python tests on Gerrit against glusterfs changes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1562670 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:54 +0000 Subject: [Bugs] [Bug 1564130] need option 'cherry-pick to release-x.y' in reviews In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1564130 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:55 +0000 Subject: [Bugs] [Bug 1564451] The abandon job for patches should post info in bugzilla that some patch is abandon'd. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1564451 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:56 +0000 Subject: [Bugs] [Bug 1584992] Need python pep8 and other relevant tests in smoke if a patch includes any python file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1584992 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:57 +0000 Subject: [Bugs] [Bug 1584998] Need automatic inclusion of few reviewers to a given patch In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1584998 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:58 +0000 Subject: [Bugs] [Bug 1594857] Make smoke runs detect test cases added to patch In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1594857 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:29:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:29:59 +0000 Subject: [Bugs] [Bug 1597731] need 'shellcheck' in smoke. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1597731 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:00 +0000 Subject: [Bugs] [Bug 1598326] Setup CI for gluster-block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1598326 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:01 +0000 Subject: [Bugs] [Bug 1609363] the comment on github job should post full commit message to issue. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1609363 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:02 +0000 Subject: [Bugs] [Bug 1620377] Coverity scan setup for gluster-block and related projects In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1620377 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:03 +0000 Subject: [Bugs] [Bug 1623596] Git plugin might be suffering from memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623596 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:04 +0000 Subject: [Bugs] =?utf-8?q?=5BBug_1625501=5D_gd2_smoke_tests_fail_with_cann?= =?utf-8?q?ot_create_directory_=E2=80=98/var/lib/glusterd=E2=80=99=3A_Perm?= =?utf-8?q?ission_denied?= In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1625501 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:05 +0000 Subject: [Bugs] [Bug 1627624] Run gd2-smoke only after smoke passes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1627624 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:06 +0000 Subject: [Bugs] [Bug 1631390] Run smoke and regression on a patch only after passing clang-format job In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1631390 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:07 +0000 Subject: [Bugs] [Bug 1637652] Glusterd2 is not cleaning itself In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1637652 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:08 +0000 Subject: [Bugs] [Bug 1638030] Need a regression job to test out Py3 support in Glusterfs code base In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1638030 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:09 +0000 Subject: [Bugs] [Bug 1657584] Re-enable TSAN jobs In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657584 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:10 +0000 Subject: [Bugs] [Bug 1671733] clang-format test is checking contrib files, but rfc.sh skips them In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671733 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 22:30:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 22:30:15 +0000 Subject: [Bugs] [Bug 1557127] github issue update on spec commits In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1557127 PnT Account Manager changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nigelb at redhat.com |bugs at gluster.org CC|nigelb at redhat.com | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 23:35:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 23:35:15 +0000 Subject: [Bugs] [Bug 1677804] New: POSIX ACLs are absent on FUSE-mounted volume using tmpfs bricks (posix-acl-autoload usually returns -1) Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677804 Bug ID: 1677804 Summary: POSIX ACLs are absent on FUSE-mounted volume using tmpfs bricks (posix-acl-autoload usually returns -1) Product: GlusterFS Version: 5 Status: NEW Component: posix-acl Assignee: bugs at gluster.org Reporter: chuck.wilson at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Calling `setfacl` on a volume whose bricks are on a tmpfs mount sets the ACLs on the brick dirs/files, but calling `getfacl` on the FUSE-mounted volume does not show the ACLs most of the time. However, if a file is `fallocate`d on that same tmpfs mount, and that file is mounted via loopback, that problem does not occur; `getfacl` always returns the ACLs on the FUSE-mounted volume. Version-Release number of selected component (if applicable): Tested on gluster 5.3 and 4.1. How reproducible: always Steps to Reproduce: 1. Mount a ramdisk mount -t tmpfs -o auto,nodev,nosuid,noexec,nodiratime,size=5G tmpfs /mnt/test 2. Make a test user adduser testuser 3. Create volume gluster volume create gv0 replica 3 server{1..3}:/mnt/test/brick 4. Start volume gluster volume start gv0 5. FUSE-mount the volume mkdir /mnt/gv0 && mount -t glusterfs -o acl server1,server2,server3:/gv0 /mnt/gv0 6. Create test file echo "test" > /mnt/gv0/test 7. Set ACL setfacl -m u:testuser:rwx /mnt/gv0/test 8. Check ACL Run several times to see that it is usually missing the ACL getfacl /mnt/gv0/test Actual results: ACLs are not there Expected results: ACLs should be there Additional info: When the ACLs aren't there, the gluster log has a trace that indicates `posix-acl-autoload` returns -1. When the ACLs are present, that message is not in the log. This seems to be a problem with gluster reading the ACLs from a tmpfs partition. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 15 23:41:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 15 Feb 2019 23:41:12 +0000 Subject: [Bugs] [Bug 1677804] POSIX ACLs are absent on FUSE-mounted volume using tmpfs bricks (posix-acl-autoload usually returns -1) In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677804 --- Comment #1 from Chuck Wilson --- Just to clarify, running `getfacl` on the files in the actual brick (/mnt/test/brick) always returns the ACLs, so it doesn't seem to be a problem with storing / accessing the ACLs on a tmpfs partition -- just gluster reading those ACLs. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 16 08:27:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 16 Feb 2019 08:27:59 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22222 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 16 08:28:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 16 Feb 2019 08:28:00 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #554 from Worker Ant --- REVIEW: https://review.gluster.org/22222 (leases-internal.c: minor reduction of work under lock.) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 16 09:57:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 16 Feb 2019 09:57:51 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #35 from Emerson Gomes --- Find below GDB output from crash. Id Target Id Frame 12 Thread 0x7fea4ae43700 (LWP 26597) 0x00007fea530e2361 in sigwait () from /lib64/libpthread.so.0 11 Thread 0x7fea54773780 (LWP 26595) 0x00007fea530dbf47 in pthread_join () from /lib64/libpthread.so.0 10 Thread 0x7fea47392700 (LWP 26601) 0x00007fea530e14ed in __lll_lock_wait () from /lib64/libpthread.so.0 9 Thread 0x7fea3f7fe700 (LWP 26604) 0x00007fea529a3483 in epoll_wait () from /lib64/libc.so.6 8 Thread 0x7fea3ffff700 (LWP 26603) 0x00007fea529a3483 in epoll_wait () from /lib64/libc.so.6 7 Thread 0x7fea3effd700 (LWP 26605) 0x00007fea529a3483 in epoll_wait () from /lib64/libc.so.6 6 Thread 0x7fea3dffb700 (LWP 26615) 0x00007fea530de965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 5 Thread 0x7fea49640700 (LWP 26600) 0x00007fea530ded12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x7fea4a642700 (LWP 26598) 0x00007fea52969e2d in nanosleep () from /lib64/libc.so.6 3 Thread 0x7fea49e41700 (LWP 26599) 0x00007fea530ded12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 2 Thread 0x7fea4b644700 (LWP 26596) 0x00007fea530e1e3d in nanosleep () from /lib64/libpthread.so.0 * 1 Thread 0x7fea3e7fc700 (LWP 26614) 0x00007fea45b62ff1 in ioc_inode_update () from /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so Thread 12 (Thread 0x7fea4ae43700 (LWP 26597)): #0 0x00007fea530e2361 in sigwait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000055959d410e2b in glusterfs_sigwaiter () No symbol table info available. #2 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 11 (Thread 0x7fea54773780 (LWP 26595)): #0 0x00007fea530dbf47 in pthread_join () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fea542dadb8 in event_dispatch_epoll () from /lib64/libglusterfs.so.0 No symbol table info available. #2 0x000055959d40d56b in main () No symbol table info available. Thread 10 (Thread 0x7fea47392700 (LWP 26601)): #0 0x00007fea530e14ed in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fea530dcdcb in _L_lock_883 () from /lib64/libpthread.so.0 No symbol table info available. #2 0x00007fea530dcc98 in pthread_mutex_lock () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fea45b62fb6 in ioc_inode_update () from /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so No symbol table info available. #4 0x00007fea45b6314a in ioc_lookup_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so No symbol table info available. #5 0x00007fea461a0343 in wb_lookup_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so No symbol table info available. #6 0x00007fea463f2b79 in dht_revalidate_cbk () from /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so No symbol table info available. #7 0x00007fea466d09e5 in afr_lookup_done () from /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so No symbol table info available. #8 0x00007fea466d1198 in afr_lookup_metadata_heal_check () from /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so No symbol table info available. #9 0x00007fea466d1cbb in afr_lookup_entry_heal () from /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so No symbol table info available. #10 0x00007fea466d1f99 in afr_lookup_cbk () from /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so No symbol table info available. #11 0x00007fea4695a6d2 in client4_0_lookup_cbk () from /usr/lib64/glusterfs/5.3/xlator/protocol/client.so No symbol table info available. #12 0x00007fea54043c70 in rpc_clnt_handle_reply () from /lib64/libgfrpc.so.0 No symbol table info available. #13 0x00007fea54044043 in rpc_clnt_notify () from /lib64/libgfrpc.so.0 No symbol table info available. #14 0x00007fea5403ff23 in rpc_transport_notify () from /lib64/libgfrpc.so.0 No symbol table info available. #15 0x00007fea48c2c37b in socket_event_handler () from /usr/lib64/glusterfs/5.3/rpc-transport/socket.so No symbol table info available. #16 0x00007fea542dba49 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 No symbol table info available. #17 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #18 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 9 (Thread 0x7fea3f7fe700 (LWP 26604)): #0 0x00007fea529a3483 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fea542db790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 No symbol table info available. #2 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 8 (Thread 0x7fea3ffff700 (LWP 26603)): #0 0x00007fea529a3483 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fea542db790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 No symbol table info available. #2 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 7 (Thread 0x7fea3effd700 (LWP 26605)): #0 0x00007fea529a3483 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fea542db790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 No symbol table info available. #2 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 6 (Thread 0x7fea3dffb700 (LWP 26615)): #0 0x00007fea530de965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fea4b64ddbb in notify_kernel_loop () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #2 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 5 (Thread 0x7fea49640700 (LWP 26600)): #0 0x00007fea530ded12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fea542b6cf8 in syncenv_task () from /lib64/libglusterfs.so.0 No symbol table info available. #2 0x00007fea542b7c40 in syncenv_processor () from /lib64/libglusterfs.so.0 No symbol table info available. #3 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 4 (Thread 0x7fea4a642700 (LWP 26598)): #0 0x00007fea52969e2d in nanosleep () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fea52969cc4 in sleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007fea542a2e7d in pool_sweeper () from /lib64/libglusterfs.so.0 No symbol table info available. #3 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 3 (Thread 0x7fea49e41700 (LWP 26599)): #0 0x00007fea530ded12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fea542b6cf8 in syncenv_task () from /lib64/libglusterfs.so.0 No symbol table info available. #2 0x00007fea542b7c40 in syncenv_processor () from /lib64/libglusterfs.so.0 No symbol table info available. #3 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7fea4b644700 (LWP 26596)): #0 0x00007fea530e1e3d in nanosleep () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fea54285f76 in gf_timer_proc () from /lib64/libglusterfs.so.0 No symbol table info available. #2 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (Thread 0x7fea3e7fc700 (LWP 26614)): #0 0x00007fea45b62ff1 in ioc_inode_update () from /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so No symbol table info available. #1 0x00007fea45b634cb in ioc_readdirp_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so No symbol table info available. #2 0x00007fea45d7a69f in rda_readdirp () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so No symbol table info available. #3 0x00007fea45b5eb0e in ioc_readdirp () from /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so No symbol table info available. #4 0x00007fea4594f8e7 in qr_readdirp () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so No symbol table info available. #5 0x00007fea5430bfb1 in default_readdirp () from /lib64/libglusterfs.so.0 No symbol table info available. #6 0x00007fea455333e6 in mdc_readdirp () from /usr/lib64/glusterfs/5.3/xlator/performance/md-cache.so No symbol table info available. #7 0x00007fea452f7d32 in io_stats_readdirp () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so No symbol table info available. #8 0x00007fea5430bfb1 in default_readdirp () from /lib64/libglusterfs.so.0 No symbol table info available. #9 0x00007fea450dc343 in meta_readdirp () from /usr/lib64/glusterfs/5.3/xlator/meta.so No symbol table info available. #10 0x00007fea4b659697 in fuse_readdirp_resume () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #11 0x00007fea4b64cc45 in fuse_resolve_all () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #12 0x00007fea4b64c958 in fuse_resolve () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #13 0x00007fea4b64cc8e in fuse_resolve_all () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #14 0x00007fea4b64bf23 in fuse_resolve_continue () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #15 0x00007fea4b64c8d6 in fuse_resolve () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #16 0x00007fea4b64cc6e in fuse_resolve_all () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #17 0x00007fea4b64ccb0 in fuse_resolve_and_resume () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #18 0x00007fea4b664d7a in fuse_thread_proc () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so No symbol table info available. #19 0x00007fea530dadd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #20 0x00007fea529a2ead in clone () from /lib64/libc.so.6 No symbol table info available. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 16 10:02:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 16 Feb 2019 10:02:26 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #36 from Emerson Gomes --- Core dump: https://drive.google.com/open?id=1cEehuPAdXHIR7eG_-RsbJkmu8lJz80k6 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 16 10:07:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 16 Feb 2019 10:07:09 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #22 from Emerson Gomes --- Added core dump and gdb output here: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Probably same issue as this one. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 16 10:48:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 16 Feb 2019 10:48:48 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ravishankar at redhat.com Flags| |needinfo?(ravishankar at redha | |t.com) --- Comment #37 from Milind Changire --- 1. crash listing in comment #3 points the disperse xlator /usr/lib/glusterfs/5.1/xlator/cluster/disperse.so(+0xf8a4)[0x7f0e23aec8a4] 2. crash listing in comment #15 points to inode_forget_with_unref from the fuse xlator /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] 3. crash listing in comment #17 points to the distribute xlator /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x8cf9)[0x7f300a9a7cf9] 4. crash listing in comment #27 points to the replicate xlator /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] 5. crash listing in comment #29 points to the replicate xlator /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] See comment #30 for a preliminary finding about this crash Ravi, could you please take a look at item#5 above. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 16 23:38:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 16 Feb 2019 23:38:18 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #21 from Amgad --- Hopefully a resolution would be available for R5.4. Assuming that setting "performance.write-behind off" is a workaround, what is the performance impacts for setting the option "off"? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 18 02:41:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 02:41:49 +0000 Subject: [Bugs] [Bug 1676797] server xlator doesn't handle dict unserialization failures correctly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676797 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-18 02:41:49 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22211 (protocol/server: Use SERVER_REQ_SET_ERROR correctly for dicts) merged (#2) on master by Pranith Kumar Karampuri -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 02:43:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 02:43:50 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #555 from Worker Ant --- REVIEW: https://review.gluster.org/22200 (md-cache.c: minor reduction of work under lock.) merged (#4) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 02:46:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 02:46:12 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-18 02:46:12 --- Comment #38 from Worker Ant --- REVIEW: https://review.gluster.org/22221 (socket: socket event handlers now return void) merged (#4) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 18 02:46:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 02:46:13 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1651246, which changed state. Bug 1651246 Summary: Failed to dispatch handler https://bugzilla.redhat.com/show_bug.cgi?id=1651246 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 02:47:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 02:47:08 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #556 from Worker Ant --- REVIEW: https://review.gluster.org/22201 (server.c: use dict_() funcs with key length.) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 02:47:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 02:47:08 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #557 from Worker Ant --- REVIEW: https://review.gluster.org/22199 (auth-cache.c: minor reduction of work under lock.) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 02:53:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 02:53:15 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #39 from Nithya Balachandran --- (In reply to Emerson Gomes from comment #35) > Find below GDB output from crash. > Please use BZ#1671556 to report any Fuse client crashes. These look similar to an issue in the write-behind translator that we are working to fix. Try setting performance.write-behind to off and let us know if you still see the crashes. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 18 03:47:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 03:47:14 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ravishankar at redha | |t.com) | --- Comment #40 from Ravishankar N --- Clearing the need info on me based on comment #39. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 18 04:32:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 04:32:48 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #22 from Nithya Balachandran --- (In reply to Amgad from comment #21) > Hopefully a resolution would be available for R5.4. Assuming that setting > "performance.write-behind off" is a workaround, what is the performance > impacts for setting the option "off"? If the workload is write heavy, you will most likely see a perf regression. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 18 04:45:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 04:45:03 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #43 from Worker Ant --- REVIEW: https://review.gluster.org/22217 (tests/dht: Remove hardcoded brick paths) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 08:03:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 08:03:14 +0000 Subject: [Bugs] [Bug 1622001] dht: File rename removes the .glusterfs handle for linkto file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1622001 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amanzane at redhat.com --- Comment #18 from Nithya Balachandran --- *** Bug 1672233 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 18 08:27:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 08:27:13 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #44 from Nithya Balachandran --- (In reply to abhays from comment #42) > @Nithya, Do ya'll have a Jenkins CI Infrastructure where continuous builds > are executed for glusterfs. We have come across the below links regarding > the same:- > https://ci.centos.org/label/gluster/ > https://build.gluster.org/ > > > Can you please confirm about these? https://build.gluster.org/ is the gluster project's CI. All patches that are posted on review.gluster.org will run the regression suite on this. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 08:29:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 08:29:48 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sabose at redhat.com Blocks| |1677319 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 09:25:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 09:25:15 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1678183 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1678183 [Bug 1678183] Tracker BZ : rm -rf issues -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 18 14:40:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 14:40:29 +0000 Subject: [Bugs] [Bug 1672314] thin-arbiter: Check with thin-arbiter file before marking new entry change log In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672314 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22161 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 18 14:41:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 14:41:34 +0000 Subject: [Bugs] [Bug 1672249] quorum count value not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672249 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-18 14:41:34 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22149 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) merged (#3) on release-4.1 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 14:44:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 14:44:54 +0000 Subject: [Bugs] [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22187 (mount/fuse: fix bug related to --auto-invalidation in mount script) merged (#3) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 14:45:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 14:45:16 +0000 Subject: [Bugs] [Bug 1674364] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674364 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22191 (performance/md-cache: introduce an option to control invalidation of inodes) merged (#3) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 14:46:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 14:46:01 +0000 Subject: [Bugs] [Bug 1677260] rm -rf fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677260 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-18 14:46:01 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22216 (cluster/dht: Fix lookup selfheal and rmdir race) merged (#2) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 16:25:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 16:25:27 +0000 Subject: [Bugs] [Bug 1678378] New: Add a nightly build verification job in Jenkins for release-6 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678378 Bug ID: 1678378 Summary: Add a nightly build verification job in Jenkins for release-6 Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: srangana at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Like https://build.gluster.org/job/nightly-release-5/ request addition of nightly release-6 job in Jenkins off the latest HEAD of the branch. This is to help keep an eye on stability of the branch. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 20:04:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 20:04:26 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22226 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 18 20:04:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 18 Feb 2019 20:04:27 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 --- Comment #6 from Worker Ant --- REVIEW: https://review.gluster.org/22226 (doc: Added initial release notes for release-6) posted (#1) for review on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 02:40:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 02:40:40 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22227 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 02:40:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 02:40:41 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22227 (performance/write-behind: handle call-stub leaks) posted (#1) for review on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 04:05:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 04:05:05 +0000 Subject: [Bugs] [Bug 1304350] access-control : spurious error log message on every setxattr call In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1304350 Rochelle changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(rallan at redhat.com | |) | -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 04:10:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 04:10:32 -0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 --- Comment #60 from Worker Ant --- REVIEW: https://review.gluster.org/22089 (cli : fix mem leak during cli_cmd_volume_gsync_set_cbk) merged (#4) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 04:35:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 04:35:16 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 Jiffin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jthottan at redhat.com --- Comment #9 from Jiffin --- As far as I understand, rda_fill_fd_cbk() sets iatt to zero and that info is stored in its context not passed to the other layers. I tried to reproduce, but was not able to hit till now.(turned on performance.readdir-ahead). Prior to this bug myself have seen similar issue when, the permission of acl ctx gets zeroed after readdir operations. The issue was very much spurious and there was no specific steps to hit that issue -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 05:53:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 05:53:46 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22189 (performance/write-behind: fix use-after-free in readdirp) merged (#10) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 05:54:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 05:54:08 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-19 05:54:08 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22227 (performance/write-behind: handle call-stub leaks) merged (#2) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 05:54:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 05:54:08 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1674406, which changed state. Bug 1674406 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1674406 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 05:54:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 05:54:08 +0000 Subject: [Bugs] [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 Bug 1676356 depends on bug 1674406, which changed state. Bug 1674406 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1674406 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 05:57:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 05:57:04 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22228 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 05:57:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 05:57:05 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #23 from Worker Ant --- REVIEW: https://review.gluster.org/22228 (performance/write-behind: fix use-after-free in readdirp) posted (#1) for review on release-5 by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 05:58:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 05:58:08 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22229 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 05:58:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 05:58:08 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #24 from Worker Ant --- REVIEW: https://review.gluster.org/22229 (performance/write-behind: handle call-stub leaks) posted (#1) for review on release-5 by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 06:21:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:21:22 +0000 Subject: [Bugs] [Bug 1678570] New: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 Bug ID: 1678570 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Product: GlusterFS Version: 6 Hardware: x86_64 OS: Linux Status: NEW Component: write-behind Severity: urgent Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: bugs at gluster.org, guillaume.pavese at interact-iv.com, sabose at redhat.com Depends On: 1671556, 1674406 Blocks: 1667103 (glusterfs-5.4), 1676356, 1677319 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1674406 +++ +++ This bug was initially created as a clone of Bug #1671556 +++ This is a re-post of my FUSE crash report from BZ1651246. That issue is for a crash in the FUSE client. Mine is too, but I was asked in that bug to open a new issue, so here you go. :) My servers (two, in a 'replica 2' setup) publish two volumes. One is Web site content, about 110GB; the other is Web config files, only a few megabytes. (Wasn't worth building extra servers for that second volume.) FUSE clients have been crashing on the larger volume every three or four days. I can't reproduce this on-demand, unfortunately, but I've got several cores from previous crashes that may be of value to you. I'm using Gluster 5.3 from the RPMs provided by the CentOS Storage SIG, on a Red Hat Enterprise Linux 7.x system. The client's logs show many hundreds of instances of this (I don't know if it's related): [2019-01-29 08:14:16.542674] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7384) [0x7fa171ead384] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xae3e) [0x7fa1720bee3e] -->/lib64/libglusterfs.so.0(dict_ref+0x5d) [0x7fa1809cc2ad] ) 0-dict: dict is NULL [Invalid argument] Then, when the client's glusterfs process crashes, this is logged: The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 871 times between [2019-01-29 08:12:48.390535] and [2019-01-29 08:14:17.100279] pending frames: frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2019-01-29 08:14:17 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] --------- Info on the volumes themselves, gathered from one of my servers: [davidsmith at wuit-s-10889 ~]$ sudo gluster volume info all Volume Name: web-config Type: Replicate Volume ID: 6c5dce6e-e64e-4a6d-82b3-f526744b463d Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-config Brick2: 172.23.128.27:/data/web-config Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet server.event-threads: 4 client.event-threads: 4 cluster.min-free-disk: 1 cluster.quorum-count: 2 cluster.quorum-type: fixed network.ping-timeout: 10 auth.allow: * performance.readdir-ahead: on Volume Name: web-content Type: Replicate Volume ID: fcabc15f-0cec-498f-93c4-2d75ad915730 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-content Brick2: 172.23.128.27:/data/web-content Options Reconfigured: network.ping-timeout: 10 cluster.quorum-type: fixed cluster.quorum-count: 2 performance.readdir-ahead: on auth.allow: * cluster.min-free-disk: 1 client.event-threads: 4 server.event-threads: 4 transport.address-family: inet nfs.disable: on performance.client-io-threads: off performance.cache-size: 4GB gluster> volume status all detail Status of volume: web-config ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5612 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5540 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 Status of volume: web-content ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5649 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5567 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 I'll attach a couple of the core files generated by the crashing glusterfs instances, size limits permitting (they range from 3 to 8 GB). If I can't attach them, I'll find somewhere to host them. --- Additional comment from Artem Russakovskii on 2019-01-31 22:26:25 UTC --- Also reposting my comment from https://bugzilla.redhat.com/show_bug.cgi?id=1651246. I wish I saw this bug report before I updated from rock solid 4.1 to 5.3. Less than 24 hours after upgrading, I already got a crash and had to unmount, kill gluster, and remount: [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3" repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31 09:38:03.958061] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 72 times between [2019-01-31 09:37:53.746741] and [2019-01-31 09:38:04.696993] pending frames: frame : type(1) op(READ) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-31 09:38:04 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6] /lib64/libc.so.6(+0x36160)[0x7fccd622d160] /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0] /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa] /lib64/libc.so.6(+0x2e772)[0x7fccd6225772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3] /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559] /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f] --------- Do the pending patches fix the crash or only the repeated warnings? I'm running glusterfs on OpenSUSE 15.0 installed via http://download.opensuse.org/repositories/home:/glusterfs:/Leap15-5/openSUSE_Leap_15.0/, not too sure how to make it core dump. If it's not fixed by the patches above, has anyone already opened a ticket for the crashes that I can join and monitor? This is going to create a massive problem for us since production systems are crashing. Thanks. --- Additional comment from David E. Smith on 2019-01-31 22:31:47 UTC --- Actually, I ran the cores through strings and grepped for a few things like passwords -- as you'd expect from a memory dump from a Web server, there's a log of sensitive information in there. Is there a safe/acceptable way to send the cores only to developers that can use them, or otherwise not have to make them publicly available while still letting the Gluster devs benefit from analyzing them? --- Additional comment from Ravishankar N on 2019-02-01 05:51:19 UTC --- (In reply to David E. Smith from comment #2) > Actually, I ran the cores through strings and grepped for a few things like > passwords -- as you'd expect from a memory dump from a Web server, there's a > log of sensitive information in there. Is there a safe/acceptable way to > send the cores only to developers that can use them, or otherwise not have > to make them publicly available while still letting the Gluster devs benefit > from analyzing them? Perhaps you could upload it to a shared Dropbox folder with view/download access to the red hat email IDs I've CC'ed to this email (including me) to begin with. Note: I upgraded a 1x2 replica volume with 1 fuse client from v4.1.7 to v5.3 and did some basic I/O (kernel untar and iozone) and did not observe any crashes, so maybe this this something that is hit under extreme I/O or memory pressure. :-( --- Additional comment from Artem Russakovskii on 2019-02-02 20:17:15 UTC --- The fuse crash happened again yesterday, to another volume. Are there any mount options that could help mitigate this? In the meantime, I set up a monit (https://mmonit.com/monit/) task to watch and restart the mount, which works and recovers the mount point within a minute. Not ideal, but a temporary workaround. By the way, the way to reproduce this "Transport endpoint is not connected" condition for testing purposes is to kill -9 the right "glusterfs --process-name fuse" process. monit check: check filesystem glusterfs_data1 with path /mnt/glusterfs_data1 start program = "/bin/mount /mnt/glusterfs_data1" stop program = "/bin/umount /mnt/glusterfs_data1" if space usage > 90% for 5 times within 15 cycles then alert else if succeeded for 10 cycles then alert stack trace: [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fa0249e4329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 26 times between [2019-02-01 23:21:20.857333] and [2019-02-01 23:21:56.164427] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-3" repeated 27 times between [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-01 23:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6] /lib64/libc.so.6(+0x36160)[0x7fa02c12d160] /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0] /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa] /lib64/libc.so.6(+0x2e772)[0x7fa02c125772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3] /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559] /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f] --- Additional comment from David E. Smith on 2019-02-05 02:59:24 UTC --- I've added the five of you to our org's Box account; all of you should have invitations to a shared folder, and I'm uploading a few of the cores now. I hope they're of value to you. The binaries are all from the CentOS Storage SIG repo at https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/ . They're all current as of a few days ago: [davidsmith at wuit-s-10882 ~]$ rpm -qa | grep gluster glusterfs-5.3-1.el7.x86_64 glusterfs-client-xlators-5.3-1.el7.x86_64 glusterfs-fuse-5.3-1.el7.x86_64 glusterfs-libs-5.3-1.el7.x86_64 --- Additional comment from Nithya Balachandran on 2019-02-05 11:00:04 UTC --- (In reply to David E. Smith from comment #5) > I've added the five of you to our org's Box account; all of you should have > invitations to a shared folder, and I'm uploading a few of the cores now. I > hope they're of value to you. > > The binaries are all from the CentOS Storage SIG repo at > https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/ . They're > all current as of a few days ago: > > [davidsmith at wuit-s-10882 ~]$ rpm -qa | grep gluster > glusterfs-5.3-1.el7.x86_64 > glusterfs-client-xlators-5.3-1.el7.x86_64 > glusterfs-fuse-5.3-1.el7.x86_64 > glusterfs-libs-5.3-1.el7.x86_64 Thanks. We will take a look and get back to you. --- Additional comment from Nithya Balachandran on 2019-02-05 16:43:45 UTC --- David, Can you try mounting the volume with the option lru-limit=0 and let us know if you still see the crashes? Regards, Nithya --- Additional comment from Nithya Balachandran on 2019-02-06 07:23:49 UTC --- Initial analysis of one of the cores: [root at rhgs313-7 gluster-5.3]# gdb -c core.6014 /usr/sbin/glusterfs [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --direct-io-mode=disable --fuse-mountopts=noatime,context="'. Program terminated with signal 11, Segmentation fault. #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 410 if (!xl->call_cleanup && xl->cbks->forget) (gdb) bt #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 #1 0x00007fa1809e90a2 in __inode_destroy (inode=0x7fa0d0349af8) at inode.c:432 #2 inode_table_prune (table=table at entry=0x7fa15800c3c0) at inode.c:1696 #3 0x00007fa1809e9f96 in inode_forget_with_unref (inode=0x7fa0d0349af8, nlookup=128) at inode.c:1273 #4 0x00007fa177dae4e1 in do_forget (this=, unique=, nodeid=, nlookup=) at fuse-bridge.c:726 #5 0x00007fa177dae5bd in fuse_forget (this=, finh=0x7fa0a41da500, msg=, iobuf=) at fuse-bridge.c:741 #6 0x00007fa177dc5d7a in fuse_thread_proc (data=0x557a0e8ffe20) at fuse-bridge.c:5125 #7 0x00007fa17f83bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa17f103ead in msync () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? () (gdb) f 0 #0 __inode_ctx_free (inode=inode at entry=0x7fa0d0349af8) at inode.c:410 410 if (!xl->call_cleanup && xl->cbks->forget) (gdb) l 405 for (index = 0; index < inode->table->xl->graph->xl_count; index++) { 406 if (inode->_ctx[index].value1 || inode->_ctx[index].value2) { 407 xl = (xlator_t *)(long)inode->_ctx[index].xl_key; 408 old_THIS = THIS; 409 THIS = xl; 410 if (!xl->call_cleanup && xl->cbks->forget) 411 xl->cbks->forget(xl, inode); 412 THIS = old_THIS; 413 } 414 } (gdb) p *xl Cannot access memory at address 0x0 (gdb) p index $1 = 6 (gdb) p inode->table->xl->graph->xl_count $3 = 13 (gdb) p inode->_ctx[index].value1 $4 = 0 (gdb) p inode->_ctx[index].value2 $5 = 140327960119304 (gdb) p/x inode->_ctx[index].value2 $6 = 0x7fa0a6370808 Based on the graph, the xlator with index = 6 is (gdb) p ((xlator_t*) inode->table->xl->graph->top)->next->next->next->next->next->next->next->name $31 = 0x7fa16c0122e0 "web-content-read-ahead" (gdb) p ((xlator_t*) inode->table->xl->graph->top)->next->next->next->next->next->next->next->xl_id $32 = 6 But read-ahead does not update the inode_ctx at all. There seems to be some sort of memory corruption happening here but that needs further analysis. --- Additional comment from David E. Smith on 2019-02-07 17:41:17 UTC --- As of this morning, I've added the lru-limit mount option to /etc/fstab on my servers. Was on vacation, didn't see the request until this morning. For the sake of reference, here's the full fstab lines, edited only to remove hostnames and add placeholders. (Note that I've never had a problem with the 'web-config' volume, which is very low-traffic and only a few megabytes in size; the problems always are the much more heavily-used 'web-content' volume.) gluster-server-1:/web-config /etc/httpd/conf.d glusterfs defaults,_netdev,noatime,context=unconfined_u:object_r:httpd_config_t:s0,backupvolfile-server=gluster-server-2,direct-io-mode=disable,lru-limit=0 0 0 gluster-server-1:/web-content /var/www/html glusterfs defaults,_netdev,noatime,context=unconfined_u:object_r:httpd_sys_rw_content_t:s0,backupvolfile-server=gluster-server-2,direct-io-mode=disable,lru-limit=0 0 0 --- Additional comment from David E. Smith on 2019-02-07 17:58:26 UTC --- Ran a couple of the glusterfs logs through the print-backtrace script. They all start with what you'd normally expect (clone, start_thread) and all end with (_gf_msg_backtrace_nomem) but they're all doing different things in the middle. It looks sorta like a memory leak or other memory corruption. Since it started happening on both of my servers after upgrading to 5.2 (and continued with 5.3), I really doubt it's a hardware issue -- the FUSE clients are both VMs, on hosts a few miles apart, so the odds of host RAM going wonky in both places at exactly that same time are ridiculous. Bit of a stretch, but do you think there would be value in my rebuilding the RPMs locally, to try to rule out anything on CentOS' end? /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] __restore_rt ??:0 /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] __inode_ctx_free ??:0 /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] inode_table_prune ??:0 /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] inode_forget_with_unref ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] fuse_forget ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] fuse_thread_proc ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] __clone ??:0 /lib64/libglusterfs.so.0(+0x26610)[0x7f36aff72610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f36aff7cb84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f36ae5d6280] __restore_rt ??:0 /lib64/libglusterfs.so.0(+0x36779)[0x7f36aff82779] __inode_unref ??:0 /lib64/libglusterfs.so.0(inode_unref+0x23)[0x7f36aff83203] inode_unref ??:0 /lib64/libglusterfs.so.0(gf_dirent_entry_free+0x2b)[0x7f36aff9ec4b] gf_dirent_entry_free ??:0 /lib64/libglusterfs.so.0(gf_dirent_free+0x2b)[0x7f36aff9ecab] gf_dirent_free ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x7480)[0x7f36a215b480] afr_readdir_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x60bca)[0x7f36a244dbca] client4_0_readdirp_cbk ??:0 /lib64/libgfrpc.so.0(+0xec70)[0x7f36afd3ec70] rpc_clnt_handle_reply ??:0 /lib64/libgfrpc.so.0(+0xf043)[0x7f36afd3f043] rpc_clnt_notify ??:0 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f36afd3af23] rpc_transport_notify ??:0 /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f36a492737b] socket_event_handler ??:0 /lib64/libglusterfs.so.0(+0x8aa49)[0x7f36affd6a49] event_dispatch_epoll_worker ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f36aedd5dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f36ae69dead] __clone ??:0 /lib64/libglusterfs.so.0(+0x26610)[0x7f7e13de0610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f7e13deab84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f7e12444280] __restore_rt ??:0 /lib64/libpthread.so.0(pthread_mutex_lock+0x0)[0x7f7e12c45c30] pthread_mutex_lock ??:0 /lib64/libglusterfs.so.0(__gf_free+0x12c)[0x7f7e13e0bc3c] __gf_free ??:0 /lib64/libglusterfs.so.0(+0x368ed)[0x7f7e13df08ed] __dentry_unset ??:0 /lib64/libglusterfs.so.0(+0x36b2b)[0x7f7e13df0b2b] __inode_retire ??:0 /lib64/libglusterfs.so.0(+0x36885)[0x7f7e13df0885] __inode_unref ??:0 /lib64/libglusterfs.so.0(inode_forget_with_unref+0x36)[0x7f7e13df1f86] inode_forget_with_unref ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x857a)[0x7f7e0b1b657a] fuse_batch_forget ??:0 /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7f7e0b1cdd7a] fuse_thread_proc ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f7e12c43dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f7e1250bead] __clone ??:0 --- Additional comment from Nithya Balachandran on 2019-02-08 03:03:20 UTC --- (In reply to David E. Smith from comment #10) > Ran a couple of the glusterfs logs through the print-backtrace script. They > all start with what you'd normally expect (clone, start_thread) and all end > with (_gf_msg_backtrace_nomem) but they're all doing different things in the > middle. It looks sorta like a memory leak or other memory corruption. Since > it started happening on both of my servers after upgrading to 5.2 (and > continued with 5.3), I really doubt it's a hardware issue -- the FUSE > clients are both VMs, on hosts a few miles apart, so the odds of host RAM > going wonky in both places at exactly that same time are ridiculous. > > Bit of a stretch, but do you think there would be value in my rebuilding the > RPMs locally, to try to rule out anything on CentOS' end? I don't think so. My guess is there is an error somewhere in the client code when handling inodes. It was never hit earlier because we never freed the inodes before 5.3. With the new inode invalidation feature, we appear to be accessing inodes that were already freed. Did you see the same crashes in 5.2? If yes, something else might be going wrong. I had a look at the coredumps you sent - most don't have any symbols (strangely). Of the ones that do, it looks like memory corruption and accessing already freed inodes. There are a few people looking at it but this going to take a while to figure out. In the meantime, let me know if you still see crashes with the lru-limit option. --- Additional comment from Nithya Balachandran on 2019-02-08 03:18:00 UTC --- Another user has just reported that the lru-limit did not help with the crashes - let me know if that is your experience as well. --- Additional comment from Nithya Balachandran on 2019-02-08 12:57:50 UTC --- We have found the cause of one crash but that has a different backtrace. Unfortunately we have not managed to reproduce the one you reported so we don't know if it is the same cause. Can you disable write-behind on the volume and let us know if it solves the problem? If yes, it is likely to be the same issue. --- Additional comment from David E. Smith on 2019-02-09 16:07:08 UTC --- I did have some crashes with 5.2. (I went from 3.something straight to 5.2, so I'm not going to be too helpful in terms of narrowing down exactly when this issue first appeared, sorry.) I'll see if I still have any of those cores; they all were from several weeks ago, so I may have already cleaned them up. This morning, one of my clients core dumped with the lru-limit option. It looks like it might be a different crash (in particular, this morning's crash was a SIGABRT, whereas previous crashes were SIGSEGV). I've uploaded that core to the same Box folder, in case it's useful. I'll paste its backtrace in below. For the write-behind request, do you want me to set 'performance.flush-behind off' or so you mean something else? --- Additional comment from David E. Smith on 2019-02-09 16:07:49 UTC --- Backtrace for 2/9/19 crash (as promised above, put it in a separate comment for clarity): /lib64/libglusterfs.so.0(+0x26610)[0x7f3b31456610] _gf_msg_backtrace_nomem ??:0 /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f3b31460b84] gf_print_trace ??:0 /lib64/libc.so.6(+0x36280)[0x7f3b2faba280] __restore_rt ??:0 /lib64/libc.so.6(gsignal+0x37)[0x7f3b2faba207] raise ??:0 /lib64/libc.so.6(abort+0x148)[0x7f3b2fabb8f8] abort ??:0 /lib64/libc.so.6(+0x78d27)[0x7f3b2fafcd27] __libc_message ??:0 /lib64/libc.so.6(+0x81489)[0x7f3b2fb05489] _int_free ??:0 /lib64/libglusterfs.so.0(+0x1a6e9)[0x7f3b3144a6e9] dict_destroy ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x8cf9)[0x7f3b23388cf9] dht_local_wipe ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x4ab90)[0x7f3b233cab90] dht_revalidate_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x709e5)[0x7f3b236a89e5] afr_lookup_done ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71198)[0x7f3b236a9198] afr_lookup_metadata_heal_check ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71cbb)[0x7f3b236a9cbb] afr_lookup_entry_heal ??:0 /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x71f99)[0x7f3b236a9f99] afr_lookup_cbk ??:0 /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x616d2)[0x7f3b239326d2] client4_0_lookup_cbk ??:0 /lib64/libgfrpc.so.0(+0xec70)[0x7f3b31222c70] rpc_clnt_handle_reply ??:0 /lib64/libgfrpc.so.0(+0xf043)[0x7f3b31223043] rpc_clnt_notify ??:0 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f3b3121ef23] rpc_transport_notify ??:0 /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f3b25e0b37b] socket_event_handler ??:0 /lib64/libglusterfs.so.0(+0x8aa49)[0x7f3b314baa49] event_dispatch_epoll_worker ??:0 /lib64/libpthread.so.0(+0x7dd5)[0x7f3b302b9dd5] start_thread ??:0 /lib64/libc.so.6(clone+0x6d)[0x7f3b2fb81ead] __clone ??:0 [d --- Additional comment from Raghavendra G on 2019-02-09 17:15:55 UTC --- (In reply to David E. Smith from comment #14) > I did have some crashes with 5.2. (I went from 3.something straight to 5.2, > so I'm not going to be too helpful in terms of narrowing down exactly when > this issue first appeared, sorry.) I'll see if I still have any of those > cores; they all were from several weeks ago, so I may have already cleaned > them up. > > This morning, one of my clients core dumped with the lru-limit option. It > looks like it might be a different crash (in particular, this morning's > crash was a SIGABRT, whereas previous crashes were SIGSEGV). I've uploaded > that core to the same Box folder, in case it's useful. I'll paste its > backtrace in below. > > For the write-behind request, do you want me to set > 'performance.flush-behind off' or so you mean something else? gluster volume set performance.write-behind off --- Additional comment from Nithya Balachandran on 2019-02-11 04:44:08 UTC --- Thanks David. I'm going to hold off on looking at the coredump until we hear back from you on whether disabling performance.write-behind works. The different backtraces could be symptoms of the same underlying issue where gluster tries to access already freed memory. --- Additional comment from Worker Ant on 2019-02-11 09:53:16 UTC --- REVIEW: https://review.gluster.org/22189 (performance/write-behind: fix use-after-free in readdirp) posted (#1) for review on master by Raghavendra G --- Additional comment from Worker Ant on 2019-02-19 02:40:41 UTC --- REVIEW: https://review.gluster.org/22227 (performance/write-behind: handle call-stub leaks) posted (#1) for review on master by Raghavendra G --- Additional comment from Worker Ant on 2019-02-19 05:53:46 UTC --- REVIEW: https://review.gluster.org/22189 (performance/write-behind: fix use-after-free in readdirp) merged (#10) on master by Raghavendra G --- Additional comment from Worker Ant on 2019-02-19 05:54:08 UTC --- REVIEW: https://review.gluster.org/22227 (performance/write-behind: handle call-stub leaks) merged (#2) on master by Raghavendra G Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker https://bugzilla.redhat.com/show_bug.cgi?id=1671556 [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1674406 [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1676356 [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:21:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:21:22 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1678570 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 06:21:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:21:22 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1678570 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:21:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:21:22 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1678570 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:21:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:21:22 +0000 Subject: [Bugs] [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1678570 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:24:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:24:02 +0000 Subject: [Bugs] [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22230 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:24:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:24:03 +0000 Subject: [Bugs] [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22230 (performance/write-behind: fix use-after-free in readdirp) posted (#1) for review on release-6 by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:25:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:25:09 +0000 Subject: [Bugs] [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22231 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:25:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:25:10 +0000 Subject: [Bugs] [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22231 (performance/write-behind: handle call-stub leaks) posted (#1) for review on release-6 by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:43:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:43:03 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #45 from abhays --- (In reply to Nithya Balachandran from comment #44) > (In reply to abhays from comment #42) > > @Nithya, Do ya'll have a Jenkins CI Infrastructure where continuous builds > > are executed for glusterfs. We have come across the below links regarding > > the same:- > > https://ci.centos.org/label/gluster/ > > https://build.gluster.org/ > > > > > > Can you please confirm about these? > > https://build.gluster.org/ is the gluster project's CI. All patches that are > posted on review.gluster.org will run the regression suite on this. Thanks for the information. (In reply to Nithya Balachandran from comment #40) > > > ./tests/bugs/posix/bug-1619720.t----------------subtests 13 and 14 fail > > > > Diff of fix provided in comment#10. > > > > > > This test is failing with the changes shared. PFA the logs for the same. > > As it is difficult for me to figure out what is happening without a Big > Endian system, I would encourage you to understand what is expected and try > to make the changes yourself. We will be happy to take your patches if they > work for us as well. Is it be possible for us to add our big endian(s390x) systems on gluster project's CI so that it's easier for you to debug the test failures on big endian platforms? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 06:46:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 06:46:05 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #46 from Nithya Balachandran --- (In reply to Nithya Balachandran from comment #44) > (In reply to abhays from comment #42) > > @Nithya, Do ya'll have a Jenkins CI Infrastructure where continuous builds > > are executed for glusterfs. We have come across the below links regarding > > the same:- > > https://ci.centos.org/label/gluster/ > > https://build.gluster.org/ > > > > > > Can you please confirm about these? > > https://build.gluster.org/ is the gluster project's CI. All patches that are > posted on review.gluster.org will run the regression suite on this. (In reply to abhays from comment #45) > (In reply to Nithya Balachandran from comment #44) > > (In reply to abhays from comment #42) > > > @Nithya, Do ya'll have a Jenkins CI Infrastructure where continuous builds > > > are executed for glusterfs. We have come across the below links regarding > > > the same:- > > > https://ci.centos.org/label/gluster/ > > > https://build.gluster.org/ > > > > > > > > > Can you please confirm about these? > > > > https://build.gluster.org/ is the gluster project's CI. All patches that are > > posted on review.gluster.org will run the regression suite on this. > > Thanks for the information. > > > (In reply to Nithya Balachandran from comment #40) > > > > ./tests/bugs/posix/bug-1619720.t----------------subtests 13 and 14 fail > > > > > > Diff of fix provided in comment#10. > > > > > > > > > > This test is failing with the changes shared. PFA the logs for the same. > > > > As it is difficult for me to figure out what is happening without a Big > > Endian system, I would encourage you to understand what is expected and try > > to make the changes yourself. We will be happy to take your patches if they > > work for us as well. > > Is it be possible for us to add our big endian(s390x) systems on gluster > project's CI so that it's easier for you to debug the test failures on big > endian platforms? I don't think so. I would recommend that you debug the tests on your systems and post patches which will work on both. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 08:18:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 08:18:44 +0000 Subject: [Bugs] [Bug 1677559] gNFS crashed when processing "gluster v profile [vol] info nfs" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677559 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22232 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 08:18:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 08:18:45 +0000 Subject: [Bugs] [Bug 1677559] gNFS crashed when processing "gluster v profile [vol] info nfs" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677559 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22232 (glusterfsd: Do not process PROFILE_NFS_INFO if graph is not ready) posted (#1) for review on master by Hu Jianfei -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 09:28:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 09:28:51 +0000 Subject: [Bugs] [Bug 1654270] glusterd crashed with seg fault possibly during node reboot while volume creates and deletes were happening In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654270 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22146 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 09:28:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 09:28:52 +0000 Subject: [Bugs] [Bug 1654270] glusterd crashed with seg fault possibly during node reboot while volume creates and deletes were happening In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654270 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-19 09:28:52 --- Comment #5 from Worker Ant --- REVIEW: https://review.gluster.org/22146 (glusterd: adding a comment for code readability) merged (#5) on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 09:39:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 09:39:12 +0000 Subject: [Bugs] [Bug 1678640] New: Running 'control-cpu-load.sh' prevents CTDB starting Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678640 Bug ID: 1678640 Summary: Running 'control-cpu-load.sh' prevents CTDB starting Product: GlusterFS Version: 4.1 Hardware: x86_64 OS: Linux Status: NEW Component: glusterd Severity: medium Assignee: bugs at gluster.org Reporter: ryan at magenta.tv CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: After running script found here, system will not longer start CTDB: https://github.com/gluster/glusterfs/blob/master/extras/control-cpu-load.sh Version-Release number of selected component (if applicable): How reproducible: Everytime Steps to Reproduce: 1. Run script on Gluster heal process 2. Try to start CTDB service Actual results: CTDB fails to start with following error: 2019/02/08 20:46:59.612215 ctdbd[2629]: Created PID file /var/run/ctdb/ctdbd.pid 2019/02/08 20:46:59.612267 ctdbd[2629]: Listening to ctdb socket /var/run/ctdb/ctdbd.socket 2019/02/08 20:46:59.612297 ctdbd[2629]: Unable to set scheduler to SCHED_FIFO (Operation not permitted) 2019/02/08 20:46:59.612304 ctdbd[2629]: CTDB daemon shutting down Expected results: CTDB starts Additional info: Other systems where script was not run, works fine. Have set DefaultCPUAccounting=no in /etc/systemd/system.conf. systemctl show ctdb | grep -i accounting CPUAccounting=no BlockIOAccounting=no MemoryAccounting=no TasksAccounting=no -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 11:28:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 11:28:21 +0000 Subject: [Bugs] [Bug 1564372] Setup Nagios server In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1564372 --- Comment #12 from M. Scherer --- So, NRPE seems to be confined, notification got improved (text message are better than before), and I am adding servers one by one. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 13:22:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 13:22:39 +0000 Subject: [Bugs] [Bug 1678726] New: Integer Overflow possible in md-cache.c due to data type inconsistency Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Bug ID: 1678726 Summary: Integer Overflow possible in md-cache.c due to data type inconsistency Product: GlusterFS Version: 5 Status: NEW Component: md-cache Assignee: bugs at gluster.org Reporter: david.spisla at iternity.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: The 'struct md_cache' in md-cache.c uses int data types which are not in common with the data types used in the 'struct iatt' in iatt.h . If one take a closer look to the implementations one can see that the struct in md-cache.c uses still the int data types like in the struct 'old_iatt' . This can lead to unexpected side effects and some values of iatt maybe will not mapped correctly. I would suggest to open a bug report. What do you think? Additional info: struct md_cache { ia_prot_t md_prot; uint32_t md_nlink; uint32_t md_uid; uint32_t md_gid; uint32_t md_atime; uint32_t md_atime_nsec; uint32_t md_mtime; uint32_t md_mtime_nsec; uint32_t md_ctime; uint32_t md_ctime_nsec; uint64_t md_rdev; uint64_t md_size; uint64_t md_blocks; uint64_t invalidation_time; uint64_t generation; dict_t *xattr; char *linkname; time_t ia_time; time_t xa_time; gf_boolean_t need_lookup; gf_boolean_t valid; gf_boolean_t gen_rollover; gf_boolean_t invalidation_rollover; gf_lock_t lock; }; struct iatt { uint64_t ia_flags; uint64_t ia_ino; /* inode number */ uint64_t ia_dev; /* backing device ID */ uint64_t ia_rdev; /* device ID (if special file) */ uint64_t ia_size; /* file size in bytes */ uint32_t ia_nlink; /* Link count */ uint32_t ia_uid; /* user ID of owner */ uint32_t ia_gid; /* group ID of owner */ uint32_t ia_blksize; /* blocksize for filesystem I/O */ uint64_t ia_blocks; /* number of 512B blocks allocated */ int64_t ia_atime; /* last access time */ int64_t ia_mtime; /* last modification time */ int64_t ia_ctime; /* last status change time */ int64_t ia_btime; /* creation time. Fill using statx */ uint32_t ia_atime_nsec; uint32_t ia_mtime_nsec; uint32_t ia_ctime_nsec; uint32_t ia_btime_nsec; uint64_t ia_attributes; /* chattr related:compressed, immutable, * append only, encrypted etc.*/ uint64_t ia_attributes_mask; /* Mask for the attributes */ uuid_t ia_gfid; ia_type_t ia_type; /* type of file */ ia_prot_t ia_prot; /* protection */ }; struct old_iatt { uint64_t ia_ino; /* inode number */ uuid_t ia_gfid; uint64_t ia_dev; /* backing device ID */ ia_type_t ia_type; /* type of file */ ia_prot_t ia_prot; /* protection */ uint32_t ia_nlink; /* Link count */ uint32_t ia_uid; /* user ID of owner */ uint32_t ia_gid; /* group ID of owner */ uint64_t ia_rdev; /* device ID (if special file) */ uint64_t ia_size; /* file size in bytes */ uint32_t ia_blksize; /* blocksize for filesystem I/O */ uint64_t ia_blocks; /* number of 512B blocks allocated */ uint32_t ia_atime; /* last access time */ uint32_t ia_atime_nsec; uint32_t ia_mtime; /* last modification time */ uint32_t ia_mtime_nsec; uint32_t ia_ctime; /* last status change time */ uint32_t ia_ctime_nsec; }; Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 13:47:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 13:47:29 +0000 Subject: [Bugs] [Bug 1671733] clang-format test is checking contrib files, but rfc.sh skips them In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671733 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED CC| |mscherer at redhat.com Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-19 13:47:29 --- Comment #4 from M. Scherer --- Code was merged, closing -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 13:50:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 13:50:45 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22233 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 13:50:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 13:50:46 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #558 from Worker Ant --- REVIEW: https://review.gluster.org/22233 (glusterd: fix get-state leak) posted (#2) for review on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 13:57:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 13:57:14 +0000 Subject: [Bugs] [Bug 1676546] Getting client connection error in gluster logs In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676546 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amukherj at redhat.com Component|glusterd |replicate -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 14:03:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 14:03:51 +0000 Subject: [Bugs] [Bug 1636110] http server ran out of log space In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636110 --- Comment #2 from M. Scherer --- So, I suspect this was fixed, because I can't see what domain was set in permissive anywhere, ergo closing it. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 14:04:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 14:04:03 +0000 Subject: [Bugs] [Bug 1636110] http server ran out of log space In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636110 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-19 14:04:03 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 19 14:06:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 14:06:33 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22234 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 14:06:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 14:06:34 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22234 (md-cache: Adapt integer data types to avoid integer overflow) posted (#1) for review on master by David Spisla -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 13:50:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 13:50:46 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #559 from Worker Ant --- REVIEW: https://review.gluster.org/22214 (upcall: some modifications to reduce work under lock) merged (#3) on master by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 16:33:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 16:33:05 +0000 Subject: [Bugs] [Bug 1677559] gNFS crashed when processing "gluster v profile [vol] info nfs" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677559 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-19 16:33:05 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22232 (glusterfsd: Do not process PROFILE_NFS_INFO if graph is not ready) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:10:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:10:06 +0000 Subject: [Bugs] [Bug 1677559] gNFS crashed when processing "gluster v profile [vol] info nfs" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677559 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22235 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:10:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:10:07 +0000 Subject: [Bugs] [Bug 1677559] gNFS crashed when processing "gluster v profile [vol] info nfs" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677559 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22235 (glusterfsd: Do not process PROFILE_NFS_INFO if graph is not ready) posted (#1) for review on release-6 by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:10:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:10:33 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #560 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22197 (fuse: reflect the actual default for lru-limit option) posted (#2) for review on release-6 by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:10:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:10:34 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Gluster.org Gerrit 22197 | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:10:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:10:36 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22197 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:11:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:11:03 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #561 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22198 (fuse: reflect the actual default for lru-limit option) posted (#2) for review on release-5 by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:11:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:11:05 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Gluster.org Gerrit 22198 | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:11:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:11:06 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22198 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 19 17:11:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 19 Feb 2019 17:11:07 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22198 (fuse: reflect the actual default for lru-limit option) posted (#2) for review on release-5 by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 04:46:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 04:46:39 +0000 Subject: [Bugs] [Bug 1678991] New: Granular lock to reduce lock contention at io-thread Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678991 Bug ID: 1678991 Summary: Granular lock to reduce lock contention at io-thread Product: GlusterFS Version: mainline Status: NEW Component: io-threads Assignee: bugs at gluster.org Reporter: moagrawa at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: The current code of io-thread xlator to execute fop is not granular so in case of brick multiplex performance is reduced. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 04:47:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 04:47:08 +0000 Subject: [Bugs] [Bug 1678991] Granular lock to reduce lock contention at io-thread In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678991 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |moagrawa at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 04:51:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 04:51:07 +0000 Subject: [Bugs] [Bug 1678991] Granular lock to reduce lock contention at io-thread In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678991 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21039 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 04:51:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 04:51:07 +0000 Subject: [Bugs] [Bug 1678991] Granular lock to reduce lock contention at io-thread In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678991 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21039 (iothread: Use the granular lock to reduce lock contention) posted (#33) for review on master by MOHIT AGRAWAL -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 05:02:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 05:02:43 +0000 Subject: [Bugs] [Bug 1679004] New: With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679004 Bug ID: 1679004 Summary: With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" Product: GlusterFS Version: 6 Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org Depends On: 1672851 Blocks: 1672869, 1678183 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1672851 +++ Description of problem: If parallel-readdir is enabled on a volume, rm -rf fails with "Directory not empty" if contains stale linkto files. Version-Release number of selected component (if applicable): How reproducible: Consistently Steps to Reproduce: 1. Create a 3 brick distribute volume 2. Enable parallel-readdir and readdir-ahead on the volume 3. Fuse mount the volume and mkdir dir0 4. Create some files inside dir0 and rename them so linkto files are created on the bricks 5. Check the bricks to see which files have linkto files. Delete the data files directly on the bricks, leaving the linkto files behind. These are now stale linkto files. 6. Remount the volume 7. rm -rf dir0 Actual results: [root at rhgs313-6 fuse1]# rm -rf dir0/ rm: cannot remove ?dir0/?: Directory not empty Expected results: dir0 should be deleted without errors Additional info: --- Additional comment from Nithya Balachandran on 2019-02-06 04:10:11 UTC --- RCA: rm -rf works by first listing and unlinking all entries in and then calling an rmdir . As DHT readdirp does not return linkto files in the listing, they are not unlinked as part of the rm -rf itself. dht_rmdir handles this by performing a readdirp internally on and deleting all stale linkto files before proceeding with the actual rmdir operation. When parallel-readdir is enabled, the rda xlator is loaded below dht in the graph and proactively lists and caches entries when an opendir is performed. Entries are returned from this cache for any subsequent readdirp calls on the directory that was opened. DHT uses the presence of the trusted.glusterfs.dht.linkto xattr to determine whether a file is a linkto file. As this call to opendir does not set trusted.glusterfs.dht.linkto in the list of requested xattrs for the opendir call, the cached entries do not contain this xattr value. As none of the entries returned will have the xattr, DHT believes they are all data files and fails the rmdir with ENOTEMPTY. Turning off parallel-readdir allows the rm -rf to succeed. --- Additional comment from Worker Ant on 2019-02-06 04:37:57 UTC --- REVIEW: https://review.gluster.org/22160 (cluster/dht: Request linkto xattrs in dht_rmdir opendir) posted (#1) for review on master by N Balachandran --- Additional comment from Worker Ant on 2019-02-13 18:24:39 UTC --- REVIEW: https://review.gluster.org/22160 (cluster/dht: Request linkto xattrs in dht_rmdir opendir) merged (#3) on master by Raghavendra G Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" https://bugzilla.redhat.com/show_bug.cgi?id=1672869 [Bug 1672869] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" https://bugzilla.redhat.com/show_bug.cgi?id=1678183 [Bug 1678183] Tracker BZ : rm -rf issues -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 05:02:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 05:02:43 +0000 Subject: [Bugs] [Bug 1672851] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672851 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1679004 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679004 [Bug 1679004] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 05:02:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 05:02:43 +0000 Subject: [Bugs] [Bug 1672869] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672869 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1679004 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679004 [Bug 1679004] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 06:44:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 06:44:39 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22237 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 06:44:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 06:44:39 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #41 from Worker Ant --- REVIEW: https://review.gluster.org/22237 (socket: socket event handlers now return void) posted (#1) for review on release-5 by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 06:44:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 06:44:41 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1651246, which changed state. Bug 1651246 Summary: Failed to dispatch handler https://bugzilla.redhat.com/show_bug.cgi?id=1651246 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 07:53:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 07:53:24 +0000 Subject: [Bugs] [Bug 1676429] distribute: Perf regression in mkdir path In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676429 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sheggodu at redhat.com Blocks| |1672818 (glusterfs-6.0) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 07:53:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 07:53:24 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1676429 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1676429 [Bug 1676429] distribute: Perf regression in mkdir path -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 10:15:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 10:15:39 +0000 Subject: [Bugs] [Bug 1676429] distribute: Perf regression in mkdir path In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676429 --- Comment #1 from Susant Kumar Palai --- Update: After taking statedumps, io-threads xlator showed differences in latency. And here is the responsible code path. <<<< int iot_schedule (call_frame_t *frame, xlator_t *this, call_stub_t *stub) { int ret = -1; iot_pri_t pri = IOT_PRI_MAX - 1; iot_conf_t *conf = this->private; if ((frame->root->pid < GF_CLIENT_PID_MAX) && conf->least_priority) { pri = IOT_PRI_LEAST; goto out; } >>>> It seems requests with -ve pid gets the least priority. After testing with performance.enable-least-priority to be off, the results are normalized now. Here is the summary. Numbers are in files/sec Post with performance.enable-least-priority on: 5448.965051804044, 5382.812519425897, 5358.221152245441, Post with performance.enable-least-priority off: 6589.996990998271, 6458.350431426266, 6568.009725869085 Pre: 6387.711992865287, 6412.12706152037, 6570.547263693283 Will send a patch to prioritize ops with no-root-squash pid. Susant -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 10:16:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 10:16:43 +0000 Subject: [Bugs] [Bug 1676430] distribute: Perf regression in mkdir path In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676430 --- Comment #2 from Susant Kumar Palai --- Update: After taking statedumps, io-threads xlator showed differences in latency. And here is the responsible code path. <<<< int iot_schedule (call_frame_t *frame, xlator_t *this, call_stub_t *stub) { int ret = -1; iot_pri_t pri = IOT_PRI_MAX - 1; iot_conf_t *conf = this->private; if ((frame->root->pid < GF_CLIENT_PID_MAX) && conf->least_priority) { pri = IOT_PRI_LEAST; goto out; } >>>> It seems requests with -ve pid gets the least priority. After testing with performance.enable-least-priority to be off, the results are normalized now. Here is the summary. Numbers are in files/sec Post with performance.enable-least-priority on: 5448.965051804044, 5382.812519425897, 5358.221152245441, Post with performance.enable-least-priority off: 6589.996990998271, 6458.350431426266, 6568.009725869085 Pre: 6387.711992865287, 6412.12706152037, 6570.547263693283 Will send a patch to prioritize ops with no-root-squash pid. Susant -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 11:00:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 11:00:28 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amukherj at redhat.com Flags| |needinfo?(sbonazzo at redhat.c | |om) --- Comment #8 from Atin Mukherjee --- Sahina, Sandro - could you confirm the exact gluster version (rpm -qa | grep glusterfs) running in the server side? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 11:08:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 11:08:39 +0000 Subject: [Bugs] [Bug 1676430] distribute: Perf regression in mkdir path In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676430 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22238 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 11:49:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 11:49:23 +0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22239 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 11:49:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 11:49:24 +0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 --- Comment #61 from Worker Ant --- REVIEW: https://review.gluster.org/22239 (fuse : fix memory leak) posted (#1) for review on master by Sunny Kumar -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 12:14:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 12:14:08 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-20 12:14:08 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22234 (md-cache: Adapt integer data types to avoid integer overflow) merged (#3) on master by David Spisla -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 12:34:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 12:34:10 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Netbulae changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(sbonazzo at redhat.c | |om) | --- Comment #9 from Netbulae --- Let me provide you with this info as I'm the original reporter. glusterfs-fuse-3.12.15-1.el7.x86_64 glusterfs-cli-3.12.15-1.el7.x86_64 glusterfs-events-3.12.15-1.el7.x86_64 glusterfs-libs-3.12.15-1.el7.x86_64 glusterfs-gnfs-3.12.15-1.el7.x86_64 glusterfs-server-3.12.15-1.el7.x86_64 glusterfs-client-xlators-3.12.15-1.el7.x86_64 glusterfs-api-3.12.15-1.el7.x86_64 glusterfs-geo-replication-3.12.15-1.el7.x86_64 glusterfs-3.12.15-1.el7.x86_64 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 14:01:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 14:01:15 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22240 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 14:01:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 14:01:16 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22240 (md-cache: Adapt integer data types to avoid integer overflow) posted (#1) for review on release-5 by David Spisla -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 14:02:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 14:02:40 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22241 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 14:04:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 14:04:21 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22242 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 14:04:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 14:04:22 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #13 from Worker Ant --- REVIEW: https://review.gluster.org/22242 (inode: reduce inode-path execution time) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 14:09:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 14:09:24 +0000 Subject: [Bugs] [Bug 1679169] New: Integer Overflow possible in md-cache.c due to data type inconsistency Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679169 Bug ID: 1679169 Summary: Integer Overflow possible in md-cache.c due to data type inconsistency Product: GlusterFS Version: 5 Status: NEW Component: md-cache Keywords: Reopened Assignee: bugs at gluster.org Reporter: david.spisla at iternity.com CC: bugs at gluster.org Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1678726 +++ Description of problem: The 'struct md_cache' in md-cache.c uses int data types which are not in common with the data types used in the 'struct iatt' in iatt.h . If one take a closer look to the implementations one can see that the struct in md-cache.c uses still the int data types like in the struct 'old_iatt' . This can lead to unexpected side effects and some values of iatt maybe will not mapped correctly. I would suggest to open a bug report. What do you think? Additional info: struct md_cache { ia_prot_t md_prot; uint32_t md_nlink; uint32_t md_uid; uint32_t md_gid; uint32_t md_atime; uint32_t md_atime_nsec; uint32_t md_mtime; uint32_t md_mtime_nsec; uint32_t md_ctime; uint32_t md_ctime_nsec; uint64_t md_rdev; uint64_t md_size; uint64_t md_blocks; uint64_t invalidation_time; uint64_t generation; dict_t *xattr; char *linkname; time_t ia_time; time_t xa_time; gf_boolean_t need_lookup; gf_boolean_t valid; gf_boolean_t gen_rollover; gf_boolean_t invalidation_rollover; gf_lock_t lock; }; struct iatt { uint64_t ia_flags; uint64_t ia_ino; /* inode number */ uint64_t ia_dev; /* backing device ID */ uint64_t ia_rdev; /* device ID (if special file) */ uint64_t ia_size; /* file size in bytes */ uint32_t ia_nlink; /* Link count */ uint32_t ia_uid; /* user ID of owner */ uint32_t ia_gid; /* group ID of owner */ uint32_t ia_blksize; /* blocksize for filesystem I/O */ uint64_t ia_blocks; /* number of 512B blocks allocated */ int64_t ia_atime; /* last access time */ int64_t ia_mtime; /* last modification time */ int64_t ia_ctime; /* last status change time */ int64_t ia_btime; /* creation time. Fill using statx */ uint32_t ia_atime_nsec; uint32_t ia_mtime_nsec; uint32_t ia_ctime_nsec; uint32_t ia_btime_nsec; uint64_t ia_attributes; /* chattr related:compressed, immutable, * append only, encrypted etc.*/ uint64_t ia_attributes_mask; /* Mask for the attributes */ uuid_t ia_gfid; ia_type_t ia_type; /* type of file */ ia_prot_t ia_prot; /* protection */ }; struct old_iatt { uint64_t ia_ino; /* inode number */ uuid_t ia_gfid; uint64_t ia_dev; /* backing device ID */ ia_type_t ia_type; /* type of file */ ia_prot_t ia_prot; /* protection */ uint32_t ia_nlink; /* Link count */ uint32_t ia_uid; /* user ID of owner */ uint32_t ia_gid; /* group ID of owner */ uint64_t ia_rdev; /* device ID (if special file) */ uint64_t ia_size; /* file size in bytes */ uint32_t ia_blksize; /* blocksize for filesystem I/O */ uint64_t ia_blocks; /* number of 512B blocks allocated */ uint32_t ia_atime; /* last access time */ uint32_t ia_atime_nsec; uint32_t ia_mtime; /* last modification time */ uint32_t ia_mtime_nsec; uint32_t ia_ctime; /* last status change time */ uint32_t ia_ctime_nsec; }; Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2019-02-19 14:06:34 UTC --- REVIEW: https://review.gluster.org/22234 (md-cache: Adapt integer data types to avoid integer overflow) posted (#1) for review on master by David Spisla --- Additional comment from Worker Ant on 2019-02-20 12:14:08 UTC --- REVIEW: https://review.gluster.org/22234 (md-cache: Adapt integer data types to avoid integer overflow) merged (#3) on master by David Spisla --- Additional comment from Worker Ant on 2019-02-20 14:01:16 UTC --- REVIEW: https://review.gluster.org/22240 (md-cache: Adapt integer data types to avoid integer overflow) posted (#1) for review on release-5 by David Spisla --- Additional comment from Worker Ant on 2019-02-20 14:02:40 UTC --- REVIEW: https://review.gluster.org/22241 (md-cache: Adapt integer data types to avoid integer overflow) posted (#1) for review on release-6 by David Spisla -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 14:10:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 14:10:18 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 david.spisla at iternity.com changed: What |Removed |Added ---------------------------------------------------------------------------- Version|5 |mainline -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 14:11:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 14:11:31 +0000 Subject: [Bugs] [Bug 1679170] New: Integer Overflow possible in md-cache.c due to data type inconsistency Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679170 Bug ID: 1679170 Summary: Integer Overflow possible in md-cache.c due to data type inconsistency Product: GlusterFS Version: 6 Status: NEW Component: md-cache Keywords: Reopened Assignee: bugs at gluster.org Reporter: david.spisla at iternity.com CC: bugs at gluster.org Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1678726 +++ Description of problem: The 'struct md_cache' in md-cache.c uses int data types which are not in common with the data types used in the 'struct iatt' in iatt.h . If one take a closer look to the implementations one can see that the struct in md-cache.c uses still the int data types like in the struct 'old_iatt' . This can lead to unexpected side effects and some values of iatt maybe will not mapped correctly. I would suggest to open a bug report. What do you think? Additional info: struct md_cache { ia_prot_t md_prot; uint32_t md_nlink; uint32_t md_uid; uint32_t md_gid; uint32_t md_atime; uint32_t md_atime_nsec; uint32_t md_mtime; uint32_t md_mtime_nsec; uint32_t md_ctime; uint32_t md_ctime_nsec; uint64_t md_rdev; uint64_t md_size; uint64_t md_blocks; uint64_t invalidation_time; uint64_t generation; dict_t *xattr; char *linkname; time_t ia_time; time_t xa_time; gf_boolean_t need_lookup; gf_boolean_t valid; gf_boolean_t gen_rollover; gf_boolean_t invalidation_rollover; gf_lock_t lock; }; struct iatt { uint64_t ia_flags; uint64_t ia_ino; /* inode number */ uint64_t ia_dev; /* backing device ID */ uint64_t ia_rdev; /* device ID (if special file) */ uint64_t ia_size; /* file size in bytes */ uint32_t ia_nlink; /* Link count */ uint32_t ia_uid; /* user ID of owner */ uint32_t ia_gid; /* group ID of owner */ uint32_t ia_blksize; /* blocksize for filesystem I/O */ uint64_t ia_blocks; /* number of 512B blocks allocated */ int64_t ia_atime; /* last access time */ int64_t ia_mtime; /* last modification time */ int64_t ia_ctime; /* last status change time */ int64_t ia_btime; /* creation time. Fill using statx */ uint32_t ia_atime_nsec; uint32_t ia_mtime_nsec; uint32_t ia_ctime_nsec; uint32_t ia_btime_nsec; uint64_t ia_attributes; /* chattr related:compressed, immutable, * append only, encrypted etc.*/ uint64_t ia_attributes_mask; /* Mask for the attributes */ uuid_t ia_gfid; ia_type_t ia_type; /* type of file */ ia_prot_t ia_prot; /* protection */ }; struct old_iatt { uint64_t ia_ino; /* inode number */ uuid_t ia_gfid; uint64_t ia_dev; /* backing device ID */ ia_type_t ia_type; /* type of file */ ia_prot_t ia_prot; /* protection */ uint32_t ia_nlink; /* Link count */ uint32_t ia_uid; /* user ID of owner */ uint32_t ia_gid; /* group ID of owner */ uint64_t ia_rdev; /* device ID (if special file) */ uint64_t ia_size; /* file size in bytes */ uint32_t ia_blksize; /* blocksize for filesystem I/O */ uint64_t ia_blocks; /* number of 512B blocks allocated */ uint32_t ia_atime; /* last access time */ uint32_t ia_atime_nsec; uint32_t ia_mtime; /* last modification time */ uint32_t ia_mtime_nsec; uint32_t ia_ctime; /* last status change time */ uint32_t ia_ctime_nsec; }; Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2019-02-19 14:06:34 UTC --- REVIEW: https://review.gluster.org/22234 (md-cache: Adapt integer data types to avoid integer overflow) posted (#1) for review on master by David Spisla --- Additional comment from Worker Ant on 2019-02-20 12:14:08 UTC --- REVIEW: https://review.gluster.org/22234 (md-cache: Adapt integer data types to avoid integer overflow) merged (#3) on master by David Spisla --- Additional comment from Worker Ant on 2019-02-20 14:01:16 UTC --- REVIEW: https://review.gluster.org/22240 (md-cache: Adapt integer data types to avoid integer overflow) posted (#1) for review on release-5 by David Spisla --- Additional comment from Worker Ant on 2019-02-20 14:02:40 UTC --- REVIEW: https://review.gluster.org/22241 (md-cache: Adapt integer data types to avoid integer overflow) posted (#1) for review on release-6 by David Spisla -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 15:49:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 15:49:07 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #25 from David E. Smith --- My two FUSE clients have now been up for nine days without a crash, after disabling the write-behind option. I'm pretty confident at this point that my crashes were related to the write-behind bugs being fixed here. Thank you! -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 19:02:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 19:02:17 +0000 Subject: [Bugs] [Bug 1679275] New: gluster-NFS crash while expanding volume Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679275 Bug ID: 1679275 Summary: gluster-NFS crash while expanding volume Product: GlusterFS Version: 6 Status: NEW Component: nfs Severity: medium Assignee: bugs at gluster.org Reporter: sheggodu at redhat.com CC: bugs at gluster.org, dang at redhat.com, ffilz at redhat.com, grajoria at redhat.com, jthottan at redhat.com, kkeithle at redhat.com, mbenjamin at redhat.com, msaini at redhat.com, rhs-bugs at redhat.com, sankarshan at redhat.com, skoduri at redhat.com, storage-qa-internal at redhat.com, vavuthu at redhat.com Depends On: 1651439 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1651439 +++ +++ This bug was initially created as a clone of Bug #1633177 +++ Description of problem: gluster-NFS is crashed while expanding volume Version-Release number of selected component (if applicable): glusterfs-3.12.2-18.1.el7rhgs.x86_64 How reproducible: Steps to Reproduce: While running automation runs, gluster-NFS is crashed while expanding volume 1) create distribute volume ( 1 * 4 ) 2) write IO from 2 clients 3) Add bricks while IO is in progress 4) start re-balance 5) check for IO After step 5), mount point is hung due to gluster-NFS crash. Actual results: gluster-NFS crash and IO is hung Expected results: IO should be success Additional info: volume info: [root at rhsauto023 glusterfs]# gluster vol info Volume Name: testvol_distributed Type: Distribute Volume ID: a809a120-f582-4358-8a70-5c53f71734ee Status: Started Snapshot Count: 0 Number of Bricks: 5 Transport-type: tcp Bricks: Brick1: rhsauto023.lab.eng.blr.redhat.com:/bricks/brick0/testvol_distributed_brick0 Brick2: rhsauto030.lab.eng.blr.redhat.com:/bricks/brick0/testvol_distributed_brick1 Brick3: rhsauto031.lab.eng.blr.redhat.com:/bricks/brick0/testvol_distributed_brick2 Brick4: rhsauto027.lab.eng.blr.redhat.com:/bricks/brick0/testvol_distributed_brick3 Brick5: rhsauto023.lab.eng.blr.redhat.com:/bricks/brick1/testvol_distributed_brick4 Options Reconfigured: transport.address-family: inet nfs.disable: off [root at rhsauto023 glusterfs]# > volume status [root at rhsauto023 glusterfs]# gluster vol status Status of volume: testvol_distributed Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick rhsauto023.lab.eng.blr.redhat.com:/br icks/brick0/testvol_distributed_brick0 49153 0 Y 22557 Brick rhsauto030.lab.eng.blr.redhat.com:/br icks/brick0/testvol_distributed_brick1 49153 0 Y 21814 Brick rhsauto031.lab.eng.blr.redhat.com:/br icks/brick0/testvol_distributed_brick2 49153 0 Y 20441 Brick rhsauto027.lab.eng.blr.redhat.com:/br icks/brick0/testvol_distributed_brick3 49152 0 Y 19886 Brick rhsauto023.lab.eng.blr.redhat.com:/br icks/brick1/testvol_distributed_brick4 49152 0 Y 23019 NFS Server on localhost N/A N/A N N/A NFS Server on rhsauto027.lab.eng.blr.redhat .com 2049 0 Y 20008 NFS Server on rhsauto033.lab.eng.blr.redhat .com 2049 0 Y 19752 NFS Server on rhsauto030.lab.eng.blr.redhat .com 2049 0 Y 21936 NFS Server on rhsauto031.lab.eng.blr.redhat .com 2049 0 Y 20557 NFS Server on rhsauto040.lab.eng.blr.redhat .com 2049 0 Y 20047 Task Status of Volume testvol_distributed ------------------------------------------------------------------------------ Task : Rebalance ID : 8e5b404f-5740-4d87-a0d7-3ce94178329f Status : completed [root at rhsauto023 glusterfs]# > NFS crash [2018-09-25 13:58:35.381085] I [dict.c:471:dict_get] (-->/usr/lib64/glusterfs/3.12.2/xlator/protocol/client.so(+0x22f5d) [0x7f93543fdf5d] -->/usr/lib64/glusterfs/3.12.2/xlator/cluster/distri bute.so(+0x202e7) [0x7f93541572e7] -->/lib64/libglusterfs.so.0(dict_get+0x10c) [0x7f9361aefb3c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds [Invalid argument] pending frames: frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2018-09-25 13:58:36 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.12.2 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f9361af8cc0] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f9361b02c04] /lib64/libc.so.6(+0x36280)[0x7f9360158280] /lib64/libglusterfs.so.0(+0x3b6fa)[0x7f9361b086fa] /lib64/libglusterfs.so.0(inode_parent+0x52)[0x7f9361b09822] /usr/lib64/glusterfs/3.12.2/xlator/nfs/server.so(+0xc243)[0x7f934f95c243] /usr/lib64/glusterfs/3.12.2/xlator/nfs/server.so(+0x3e1d8)[0x7f934f98e1d8] /usr/lib64/glusterfs/3.12.2/xlator/nfs/server.so(+0x3ea2b)[0x7f934f98ea2b] /usr/lib64/glusterfs/3.12.2/xlator/nfs/server.so(+0x3ead5)[0x7f934f98ead5] /usr/lib64/glusterfs/3.12.2/xlator/nfs/server.so(+0x3ecf8)[0x7f934f98ecf8] /usr/lib64/glusterfs/3.12.2/xlator/nfs/server.so(+0x29d7c)[0x7f934f979d7c] /usr/lib64/glusterfs/3.12.2/xlator/nfs/server.so(+0x2a184)[0x7f934f97a184] /lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x325)[0x7f93618ba955] /lib64/libgfrpc.so.0(rpcsvc_notify+0x10b)[0x7f93618bab3b] /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f93618bca73] /usr/lib64/glusterfs/3.12.2/rpc-transport/socket.so(+0x7566)[0x7f93566e2566] /usr/lib64/glusterfs/3.12.2/rpc-transport/socket.so(+0x9b0c)[0x7f93566e4b0c] /lib64/libglusterfs.so.0(+0x894c4)[0x7f9361b564c4] /lib64/libpthread.so.0(+0x7dd5)[0x7f9360957dd5] /lib64/libc.so.6(clone+0x6d)[0x7f9360220b3d] --------- --- Additional comment from Red Hat Bugzilla Rules Engine on 2018-09-26 07:02:14 EDT --- This bug is automatically being proposed for a Z-stream release of Red Hat Gluster Storage 3 under active development and open for bug fixes, by setting the release flag 'rhgs?3.4.z' to '?'. If this bug should be proposed for a different release, please manually change the proposed release flag. --- Additional comment from Vijay Avuthu on 2018-09-26 07:03:44 EDT --- SOS reports: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/vavuthu/nfs_crash_on_expanding_volume/ jenkin Job: http://jenkins-rhs.lab.eng.blr.redhat.com:8080/view/Auto%20RHEL%207.5/job/auto-RHGS_Downstream_BVT_RHEL_7_5_RHGS_3_4_brew/28/consoleFull Glusto Logs : http://jenkins-rhs.lab.eng.blr.redhat.com:8080/view/Auto%20RHEL%207.5/job/auto-RHGS_Downstream_BVT_RHEL_7_5_RHGS_3_4_brew/ws/glusto_28.log --- Additional comment from Jiffin on 2018-09-27 08:07:28 EDT --- 0 0x00007f9361b086fa in __inode_get_xl_index (xlator=0x7f9350018d30, inode=0x7f933c0133b0) at inode.c:455 455 if ((inode->_ctx[xlator->xl_id].xl_key != NULL) && Missing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libselinux-2.5-12.el7.x86_64 libuuid-2.23.2-52.el7_5.1.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007f9361b086fa in __inode_get_xl_index (xlator=0x7f9350018d30, inode=0x7f933c0133b0) at inode.c:455 #1 __inode_ref (inode=inode at entry=0x7f933c0133b0) at inode.c:537 #2 0x00007f9361b09822 in inode_parent (inode=inode at entry=0x7f933c01d990, pargfid=pargfid at entry=0x7f93400aa2e8 "", name=name at entry=0x0) at inode.c:1359 #3 0x00007f934f95c243 in nfs_inode_loc_fill (inode=inode at entry=0x7f933c01d990, loc=loc at entry=0x7f93400aa2b8, how=how at entry=1) at nfs-common.c:206 #4 0x00007f934f98e1d8 in nfs3_fh_resolve_inode_done (cs=cs at entry=0x7f93400a9df0, inode=inode at entry=0x7f933c01d990) at nfs3-helpers.c:3611 #5 0x00007f934f98ea2b in nfs3_fh_resolve_inode (cs=0x7f93400a9df0) at nfs3-helpers.c:3828 #6 0x00007f934f98ead5 in nfs3_fh_resolve_resume (cs=cs at entry=0x7f93400a9df0) at nfs3-helpers.c:3860 #7 0x00007f934f98ecf8 in nfs3_fh_resolve_root (cs=cs at entry=0x7f93400a9df0) at nfs3-helpers.c:3915 #8 0x00007f934f98ef41 in nfs3_fh_resolve_and_resume (cs=cs at entry=0x7f93400a9df0, fh=fh at entry=0x7f934e195ae0, entry=entry at entry=0x0, resum_fn=resum_fn at entry=0x7f934f9798b0 ) at nfs3-helpers.c:4011 #9 0x00007f934f979d7c in nfs3_access (req=req at entry=0x7f934022dcd0, fh=fh at entry=0x7f934e195ae0, accbits=31) at nfs3.c:1783 #10 0x00007f934f97a184 in nfs3svc_access (req=0x7f934022dcd0) at nfs3.c:1819 #11 0x00007f93618ba955 in rpcsvc_handle_rpc_call (svc=0x7f935002c430, trans=trans at entry=0x7f935007a960, msg=) at rpcsvc.c:695 #12 0x00007f93618bab3b in rpcsvc_notify (trans=0x7f935007a960, mydata=, event=, data=) at rpcsvc.c:789 #13 0x00007f93618bca73 in rpc_transport_notify (this=this at entry=0x7f935007a960, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f9340031290) at rpc-transport.c:538 #14 0x00007f93566e2566 in socket_event_poll_in (this=this at entry=0x7f935007a960, notify_handled=) at socket.c:2315 #15 0x00007f93566e4b0c in socket_event_handler (fd=10, idx=7, gen=46, data=0x7f935007a960, poll_in=1, poll_out=0, poll_err=0) at socket.c:2467 #16 0x00007f9361b564c4 in event_dispatch_epoll_handler (event=0x7f934e195e80, event_pool=0x55c696306210) at event-epoll.c:583 #17 event_dispatch_epoll_worker (data=0x7f9350043b00) at event-epoll.c:659 #18 0x00007f9360957dd5 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f9360220b3d in clone () from /lib64/libc.so.6 Above as part of nfs_local_filling() it was trying to find the parent inode and there is valid inode for parent as well, but context for that inode is NULL. >From code reading i was not able to find place in which ctx is NULL with valid inode p *inode -- parent $27 = {table = 0x7f935002d000, gfid = "{\033g\270K\202B\202\211\320B\"\373u", , lock = {spinlock = 0, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = -1, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\377\377\377\377", '\000' , __align = 0}}, nlookup = 0, fd_count = 0, active_fd_count = 0, ref = 1, ia_type = IA_IFDIR, fd_list = {next = 0x7f933c013408, prev = 0x7f933c013408}, dentry_list = {next = 0x7f933c013418, prev = 0x7f933c013418}, hash = { next = 0x7f933c013428, prev = 0x7f933c013428}, list = {next = 0x7f93503a5408, prev = 0x7f935002d060}, _ctx = 0x0} I tried to reproduce the issue(twice) but, it was not hitting in my setup. Requesting Vijay to recheck how frequently it can be reproduced and please try to run ith debug log level for nfs-server(diagonsis-cient log level) --- Additional comment from Worker Ant on 2018-11-20 06:00:20 UTC --- REVIEW: https://review.gluster.org/21685 (inode : prevent dentry creation if parent does not have ctx) posted (#1) for review on master by jiffin tony Thottan --- Additional comment from Worker Ant on 2018-11-29 14:03:58 UTC --- REVIEW: https://review.gluster.org/21749 (nfs : set ctx for every inode looked up nfs3_fh_resolve_inode_lookup_cbk()) posted (#1) for review on master by jiffin tony Thottan --- Additional comment from Worker Ant on 2018-12-03 05:50:44 UTC --- REVIEW: https://review.gluster.org/21749 (nfs : set ctx for every inode looked up nfs3_fh_resolve_inode_lookup_cbk()) posted (#4) for review on master by Amar Tumballi --- Additional comment from Worker Ant on 2019-01-08 08:49:15 UTC --- REVIEW: https://review.gluster.org/21998 (dht: fix inode leak when heal path) posted (#1) for review on master by Kinglong Mee --- Additional comment from Worker Ant on 2019-02-13 18:22:33 UTC --- REVIEW: https://review.gluster.org/21998 (dht: fix double extra unref of inode at heal path) merged (#4) on master by Raghavendra G Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1651439 [Bug 1651439] gluster-NFS crash while expanding volume -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 19:02:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 19:02:17 +0000 Subject: [Bugs] [Bug 1651439] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651439 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1679275 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679275 [Bug 1679275] gluster-NFS crash while expanding volume -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 19:03:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 19:03:01 +0000 Subject: [Bugs] [Bug 1679275] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679275 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |spalai at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 19:04:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 19:04:39 +0000 Subject: [Bugs] [Bug 1679275] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679275 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672818 (glusterfs-6.0) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 20 19:04:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 19:04:39 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1679275 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679275 [Bug 1679275] gluster-NFS crash while expanding volume -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 19:32:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 19:32:00 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22243 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 19:32:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 19:32:01 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #14 from Worker Ant --- REVIEW: https://review.gluster.org/22243 (inode: handle list management outside of ref/unref code) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 20:20:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 20:20:19 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Strahil Nikolov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hunter86_bg at yahoo.com --- Comment #6 from Strahil Nikolov --- It seems that glusterfs-gnfs is a depedency for vdsm-gluster without which I was not able to deploy ovirt-4.2.7 self-hosted (prep stage failed as method "gluster" was unknown): [root at ovirt2 ~]# yum --assumeno remove glusterfs-gnfs Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Repository centos-sclo-rh-release is listed more than once in the configuration Resolving Dependencies --> Running transaction check ---> Package glusterfs-gnfs.x86_64 0:3.12.15-1.el7 will be erased --> Processing Dependency: glusterfs-gnfs for package: vdsm-gluster-4.20.46-1.el7.x86_64 --> Running transaction check ---> Package vdsm-gluster.x86_64 0:4.20.46-1.el7 will be erased --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================= Removing: glusterfs-gnfs x86_64 3.12.15-1.el7 @ovirt-4.2-centos-gluster312 445 k Removing for dependencies: vdsm-gluster x86_64 4.20.46-1.el7 @ovirt-4.2 260 k Transaction Summary ======================================================================================================================================================= Remove 1 Package (+1 Dependent package) Installed size: 705 k Exiting on user command Your transaction was saved, rerun it with: yum load-transaction /tmp/yum_save_tx.2019-02-20.22-19.2e8Lcz.yumtx Uploading Enabled Repositories Report Loaded plugins: fastestmirror, product-id, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Repository centos-sclo-rh-release is listed more than once in the configuration Cannot upload enabled repos report, is this client registered? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 20:53:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 20:53:27 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 --- Comment #7 from Strahil Nikolov --- I have removed glusterfs-gnfs manually and updated. The update process finished successfully, but the web interface shows unsynced entries present , while gluster volume heal engine info - shows nothing... -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 20 20:55:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 20 Feb 2019 20:55:41 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 --- Comment #8 from Strahil Nikolov --- (In reply to Strahil Nikolov from comment #7) > I have removed glusterfs-gnfs manually and updated. > The update process finished successfully, but the web interface shows > unsynced entries present , while gluster volume heal engine info - shows > nothing... rpm -e --nodeps glusterfs-gnfs -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 00:02:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 00:02:34 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #26 from Artem Russakovskii --- I wish I could say the same thing. Unfortunately, I had another crash on the same server yesterday, with performance.write-behind still set to off. I've emailed the core file privately to the relevant people. [2019-02-19 19:50:39.511743] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7f9598991329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7f9598ba2af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7f95a137d218] ) 2-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 95 times between [2019-02-19 19:49:07.655620] and [2019-02-19 19:50:39.499284] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-_data3-replicate-0: selecting local read_child _data3-client-3" repeated 56 times between [2019-02-19 19:49:07.602370] and [2019-02-19 19:50:42.912766] pending frames: frame : type(1) op(LOOKUP) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-02-19 19:50:43 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7f95a138864c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7f95a1392cb6] /lib64/libc.so.6(+0x36160)[0x7f95a054f160] /lib64/libc.so.6(gsignal+0x110)[0x7f95a054f0e0] /lib64/libc.so.6(abort+0x151)[0x7f95a05506c1] /lib64/libc.so.6(+0x2e6fa)[0x7f95a05476fa] /lib64/libc.so.6(+0x2e772)[0x7f95a0547772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7f95a08dd0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7f95994f0c9d] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7f9599503ba1] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7f9599788f3f] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7f95a1153820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7f95a1153b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f95a1150063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7f959aea00b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7f95a13e64c3] /lib64/libpthread.so.0(+0x7559)[0x7f95a08da559] /lib64/libc.so.6(clone+0x3f)[0x7f95a061181f] --------- [2019-02-19 19:51:34.425106] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.3 (args: /usr/sbin/glusterfs --lru-limit=0 --process-name fuse --volfile-server=localhost --volfile-id=/_data3 /mnt/_data3) [2019-02-19 19:51:34.435206] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-02-19 19:51:34.450272] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2019-02-19 19:51:34.450394] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4 [2019-02-19 19:51:34.450488] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 03:57:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 03:57:27 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #27 from Nithya Balachandran --- (In reply to Artem Russakovskii from comment #26) > I wish I could say the same thing. Unfortunately, I had another crash on the > same server yesterday, with performance.write-behind still set to off. I've > emailed the core file privately to the relevant people. > Thanks Artem. I have downloaded the coredump and will take a look sometime this week. Can you share any details about the workload that might help us narrow it down? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 05:02:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:02:41 +0000 Subject: [Bugs] [Bug 1679275] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679275 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22244 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 05:02:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:02:42 +0000 Subject: [Bugs] [Bug 1679275] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679275 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22244 (dht: fix double extra unref of inode at heal path) posted (#1) for review on release-6 by Susant Palai -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 05:13:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:13:50 +0000 Subject: [Bugs] [Bug 1679004] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679004 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |nbalacha at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 05:18:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:18:20 +0000 Subject: [Bugs] [Bug 1679004] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679004 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22245 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 05:18:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:18:21 +0000 Subject: [Bugs] [Bug 1679004] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679004 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22245 (cluster/dht: Request linkto xattrs in dht_rmdir opendir) posted (#1) for review on release-6 by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 05:26:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:26:54 +0000 Subject: [Bugs] [Bug 1679401] New: Geo-rep setup creates an incorrectly formatted authorized_keys file Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679401 Bug ID: 1679401 Summary: Geo-rep setup creates an incorrectly formatted authorized_keys file Product: GlusterFS Version: mainline Status: NEW Component: geo-replication Assignee: bugs at gluster.org Reporter: sunkumar at redhat.com CC: anrobins at redhat.com, avishwan at redhat.com, bugs at gluster.org, csaba at redhat.com, rhinduja at redhat.com, rhs-bugs at redhat.com, sankarshan at redhat.com, storage-qa-internal at redhat.com, sunkumar at redhat.com Depends On: 1671862 Target Milestone: --- Classification: Community Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671862 [Bug 1671862] Geo-rep setup creates an incorrectly formatted authorized_keys file -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 05:27:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:27:21 +0000 Subject: [Bugs] [Bug 1679401] Geo-rep setup creates an incorrectly formatted authorized_keys file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679401 Sunny Kumar changed: What |Removed |Added ---------------------------------------------------------------------------- Comment #0 is|1 |0 private| | Status|NEW |ASSIGNED Assignee|bugs at gluster.org |sunkumar at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 05:30:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:30:07 +0000 Subject: [Bugs] [Bug 1679401] Geo-rep setup creates an incorrectly formatted authorized_keys file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679401 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22246 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 05:30:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:30:08 +0000 Subject: [Bugs] [Bug 1679401] Geo-rep setup creates an incorrectly formatted authorized_keys file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679401 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22246 (geo-rep : fix incorrectly formatted authorized_keys) posted (#1) for review on master by Sunny Kumar -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 06:02:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 06:02:48 +0000 Subject: [Bugs] [Bug 1679406] New: glustereventsd does not start on Ubuntu 16.04 LTS Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679406 Bug ID: 1679406 Summary: glustereventsd does not start on Ubuntu 16.04 LTS Product: GlusterFS Version: mainline Hardware: x86_64 OS: Linux Status: NEW Component: eventsapi Severity: high Priority: high Assignee: bugs at gluster.org Reporter: avishwan at redhat.com CC: avishwan at redhat.com, beat.joerg at gmail.com, bugzilla at ii.nl, captcha.is.evil at gmail.com, dchang at silverlakecorp.com, pasik at iki.fi, sabose at redhat.com Depends On: 1649054 Blocks: 1667103 (glusterfs-5.4), 1677319 (Gluster_5_Affecting_oVirt_4.3) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1649054 +++ Description of problem: glustereventsd does not start due to an import error in utils.py Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/sbin/glustereventsd", line 24, in Nov 12 20:21:04 s002 glustereventsd[28140]: import handlers Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/lib/x86_64-linux-gnu/glusterfs/events/handlers.py", line 12, in Nov 12 20:21:04 s002 glustereventsd[28140]: import utils Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in Nov 12 20:21:04 s002 glustereventsd[28140]: from .eventsapiconf import (LOG_FILE, Nov 12 20:21:04 s002 glustereventsd[28140]: SystemError: Parent module '' not loaded, cannot perform relative import Installed Pyython version is 3.5.2 --- Additional comment from Mrten on 2018-12-04 17:12:09 UTC --- same as Bug 1650017 --- Additional comment from Mrten on 2018-12-04 17:25:31 UTC --- This is a fix for the first bug: --- utils.py.old 2018-12-04 18:19:18.846463181 +0100 +++ utils.py 2018-12-04 18:21:09.830149998 +0100 @@ -26,13 +26,13 @@ from hashlib import sha256 from calendar import timegm -from .eventsapiconf import (LOG_FILE, +from eventsapiconf import (LOG_FILE, WEBHOOKS_FILE, DEFAULT_CONFIG_FILE, CUSTOM_CONFIG_FILE, UUID_FILE, CERTS_DIR) -from . import eventtypes +import eventtypes and you need this one as well: --- glustereventsd.py.old 2018-12-04 18:25:06.841481182 +0100 +++ glustereventsd.py 2018-12-04 18:22:51.529863014 +0100 @@ -28,7 +28,7 @@ from utils import logger, PidFile, PidFileLockFailed, boolify -class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): +class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): def handle(self): data = self.request[0].strip() @@ -98,7 +98,7 @@ # Start the Eventing Server, UDP Server try: - server = SocketServer.ThreadingUDPServer( + server = socketserver.ThreadingUDPServer( (SERVER_ADDRESS, port), GlusterEventsRequestHandler) except socket.error as e: --- Additional comment from Mrten on 2018-12-04 17:28:51 UTC --- but you still have "glustereventsd[20980]: Unable to get Port details from Config" in the log :( --- Additional comment from Mrten on 2018-12-04 17:36:28 UTC --- default config file does not exist: root at gluster-3:# less /etc/glusterfs/eventsconfig.json /etc/glusterfs/eventsconfig.json: No such file or directory --- Additional comment from Mrten on 2018-12-04 17:38:39 UTC --- inserting this: https://github.com/gluster/glusterfs/blob/master/events/src/eventsconfig.json into /etc/glusterfs/eventsconfig.json makes it start without problems --- Additional comment from Mrten on 2018-12-05 09:24:13 UTC --- haha, but now geo-replication is broken [2018-12-05 08:52:49.556853] E [syncdutils(monitor):338:log_raise_exception] : FAIL: Traceback (most recent call last): File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 368, in twrap tf(*aargs) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 339, in wmon slave_host, master, suuid, slavenodes) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 325, in monitor brick_path=w[0]['dir']) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 602, in gf_event from events.gf_event import gf_event as gfevent File "/usr/lib/x86_64-linux-gnu/glusterfs/events/gf_event.py", line 19, in from .utils import logger, setup_logger, get_config File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in from eventsapiconf import (LOG_FILE, ImportError: No module named 'eventsapiconf' --- Additional comment from Mrten on 2018-12-05 09:31:22 UTC --- better patch: --- utils.py.old 2018-12-04 18:19:18.846463181 +0100 +++ utils.py 2018-12-05 10:30:03.306848363 +0100 @@ -26,13 +26,25 @@ from hashlib import sha256 from calendar import timegm -from .eventsapiconf import (LOG_FILE, - WEBHOOKS_FILE, - DEFAULT_CONFIG_FILE, - CUSTOM_CONFIG_FILE, - UUID_FILE, - CERTS_DIR) -from . import eventtypes +try: + from .eventsapiconf import (LOG_FILE, + WEBHOOKS_FILE, + DEFAULT_CONFIG_FILE, + CUSTOM_CONFIG_FILE, + UUID_FILE, + CERTS_DIR) +except ImportError: + from eventsapiconf import (LOG_FILE, + WEBHOOKS_FILE, + DEFAULT_CONFIG_FILE, + CUSTOM_CONFIG_FILE, + UUID_FILE, + CERTS_DIR) + +try: + from . import eventtypes +except ImportError: + import eventtypes --- Additional comment from dchang0 on 2018-12-12 08:06:26 UTC --- Hi, Mrten-- I still have the exact same bug AFTER applying the patches you provided (the patch in Comment 2 for glustereventsd.py and the patch in Comment 7 for utils.py). I'm running Raspbian Stretch Lite (not Ubuntu 16.04 like the original bug reporter) on a Raspberry Pi Zero W and built glusterfs 5.1 from source code. The patches appear to have been applied correctly but attempting to start glustereventsd still gets this error: Dec 11 16:48:14 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'. Dec 11 16:50:53 datastore2c systemd[1]: Starting LSB: Gluster Events Server... -- Subject: Unit glustereventsd.service has begun start-up -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit glustereventsd.service has begun starting up. Dec 11 16:50:55 datastore2c glustereventsd[710]: Starting glustereventsd service: glustereventsdTraceback (most recent call last): Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/sbin/glustereventsd", line 24, in Dec 11 16:50:55 datastore2c glustereventsd[710]: import handlers Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in Dec 11 16:50:55 datastore2c glustereventsd[710]: import utils Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in Dec 11 16:50:55 datastore2c glustereventsd[710]: from .eventsapiconf import (LOG_FILE, Dec 11 16:50:55 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import Dec 11 16:50:55 datastore2c glustereventsd[710]: failed! Dec 11 16:50:56 datastore2c glustereventsd[710]: Traceback (most recent call last): Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/sbin/glustereventsd", line 24, in Dec 11 16:50:56 datastore2c glustereventsd[710]: import handlers Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in Dec 11 16:50:56 datastore2c glustereventsd[710]: import utils Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in Dec 11 16:50:56 datastore2c glustereventsd[710]: from .eventsapiconf import (LOG_FILE, Dec 11 16:50:56 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Control process exited, code=exited status=1 Dec 11 16:50:56 datastore2c systemd[1]: Failed to start LSB: Gluster Events Server. -- Subject: Unit glustereventsd.service has failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit glustereventsd.service has failed. -- -- The result is failed. Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Unit entered failed state. Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'. I can provide whatever info you need to help debug this. Thanks! --- Additional comment from Arano-kai on 2018-12-28 01:27 UTC --- Hi dchang0. According to your logs, patch is not apply. Here the compiled from Mrten's answers patch that you need apply with following cmd: `patch -p 0 -d [libdir] < glustereventsd.patch` Where [libdir]: - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/" - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs) Also, there is missing file that you can get by `wget 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/eventsconfig.json' -O /etc/glusterfs/eventsconfig.json` --- Additional comment from dchang0 on 2018-12-28 01:44:44 UTC --- (In reply to Arano-kai from comment #9) > Created attachment 1517163 [details] > glustereventsd.py and utils.py fixup > > Hi dchang0. > According to your logs, patch is not apply. > Here the compiled from Mrten's answers patch that you need apply with > following cmd: > `patch -p 0 -d [libdir] < glustereventsd.patch` > Where [libdir]: > - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/" > - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs) > Also, there is missing file that you can get by `wget > 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/ > eventsconfig.json' -O /etc/glusterfs/eventsconfig.json` Thanks. I will check the files in question manually and reapply them if needed and report back. eventsconfig.json was not missing for me (I built completely from scratch, and that file was included and installed in the proper location). --- Additional comment from Arano-kai on 2018-12-28 01:49 UTC --- Woops, forget to add 'SystemError' to catcher. Here the fixed version. --- Additional comment from dchang0 on 2018-12-28 04:14:37 UTC --- This doesn't matter now, since you've changed at least the glustereventsd.py patch, but I'm posting these anyway for diagnostic purposes. The source code was built in /root/glusterfs-5.1. The install with the patches applied is in /usr/local... ----- root at datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py 31c31 < class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): --- > class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): 101c101 < server = SocketServer.ThreadingUDPServer( --- > server = socketserver.ThreadingUDPServer( ----- root at datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py 29,35c29,42 < from .eventsapiconf import (LOG_FILE, < WEBHOOKS_FILE, < DEFAULT_CONFIG_FILE, < CUSTOM_CONFIG_FILE, < UUID_FILE, < CERTS_DIR) < from . import eventtypes --- > try: > from .eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > except ImportError: > from eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) 36a44,47 > try: > from . import eventtypes > except ImportError: > import eventtypes ----- So it looks like I did apply the patches before, but they were probably bad/broken patches. Anyway, I will replace the patched files with the original source, then apply your new patches and see what happens and report back. --- Additional comment from dchang0 on 2018-12-28 04:52:57 UTC --- Okay, I just put the original files from source back (using make install) and then applied your new patches from Attachment #1517165. I'm getting a different error (probably an easier one to solve). Strangely, the patch command does not create the .orig files. Perhaps my patch command is broken or non-standard... Here are the diffs: ----- root at datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py 31c31 < class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): --- > class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): 101c101 < server = SocketServer.ThreadingUDPServer( --- > server = socketserver.ThreadingUDPServer( ------ root at datastore2c:/usr/local/libexec# diff /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py 29,35c29,46 < from .eventsapiconf import (LOG_FILE, < WEBHOOKS_FILE, < DEFAULT_CONFIG_FILE, < CUSTOM_CONFIG_FILE, < UUID_FILE, < CERTS_DIR) < from . import eventtypes --- > try: > from .eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > except (ImportError, SystemError): > from eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > try: > from . import eventtypes > except (ImportError, SystemError): > import eventtypes ----- Here is the new error when attempting to run: service glustereventsd start It appears to be a simple path error (can't find the gluster command). root at datastore2c:/usr/local/libexec# journalctl -xe Dec 27 20:46:15 datastore2c glustereventsd[14033]: import utils Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/local/libexec/glusterfs/events/utils.py", line 37, in Dec 27 20:46:15 datastore2c glustereventsd[14033]: from eventsapiconf import (LOG_FILE, Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 33, in # systemctl edit glustereventsd.service [Service] Environment=PATH= --- Additional comment from Arano-kai on 2018-12-28 11:18:07 UTC --- As for Comment 12, patch is right and applied. That is not right is python version: # python3 -V Python 3.5.3 # /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid ... >>>SystemError<<<: Parent module '' not loaded, cannot perform relative import # python3 -V Python 3.7.2rc1 # /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid ... >>>ImportError<<<: attempted relative import with no known parent package So we need catch SystemError for older python. --- Additional comment from dchang0 on 2018-12-29 05:14:01 UTC --- (In reply to Arano-kai from comment #15) > It is probably you hit by https://stackoverflow.com/q/5658622, that is some > $PATH inconsistency from python view and shell view. > Since systemd used and units run in isolated env, you can try to explicitly > set $PATH by overriding glustereventsd.service: > # echo $PATH > > # systemctl edit glustereventsd.service > > [Service] > Environment=PATH= Thanks for the help! I ended up editing the PATH= line in the file /etc/init.d/glustereventsd It seems to work now (starts without errors). I have not tested the actual functionality yet but will do so soon enough. I would say that with the patches you provided in attachment 1517165, this bug is fixed completely for me. Thanks again! --- Additional comment from Sahina Bose on 2019-02-14 07:10:46 UTC --- Aravinda, we have multiple oVirt users hitting this issue. Can this be targeted to a 5.x release? Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1649054 [Bug 1649054] glustereventsd does not start on Ubuntu 16.04 LTS https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 06:02:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 06:02:48 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Aravinda VK changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1679406 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679406 [Bug 1679406] glustereventsd does not start on Ubuntu 16.04 LTS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 06:03:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 06:03:04 +0000 Subject: [Bugs] [Bug 1679406] glustereventsd does not start on Ubuntu 16.04 LTS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679406 Aravinda VK changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |avishwan at redhat.com -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 06:07:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 06:07:43 +0000 Subject: [Bugs] [Bug 1679409] New: [Disperse] : Improve heal info command to get the report quickly Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679409 Bug ID: 1679409 Summary: [Disperse] : Improve heal info command to get the report quickly Product: GlusterFS Version: mainline Status: NEW Component: disperse Assignee: bugs at gluster.org Reporter: aspandey at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Improve heal info command to get the report quickly . Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 06:09:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 06:09:37 +0000 Subject: [Bugs] [Bug 1679409] [Disperse] : Improve heal info command to get the report quickly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679409 Ashish Pandey changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 06:23:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 06:23:03 +0000 Subject: [Bugs] [Bug 1672656] glustereventsd: crash, ABRT report for package glusterfs has reached 100 occurrences In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672656 Aravinda VK changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |avishwan at redhat.com --- Comment #1 from Aravinda VK --- Mostly this PR fixes the issue. https://review.gluster.org/#/c/glusterfs/+/22247/ How to reproduce the issue to confirm the same. Please provide the steps to reproduce. -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 06:25:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 06:25:48 +0000 Subject: [Bugs] [Bug 1650017] glustereventsd ImportError: attempted relative import with no known parent package In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1650017 Aravinda VK changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |avishwan at redhat.com --- Comment #3 from Aravinda VK --- Sent Patch upstream: https://review.gluster.org/#/c/glusterfs/+/22247 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 07:51:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 07:51:14 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Sandro Bonazzola changed: What |Removed |Added ---------------------------------------------------------------------------- Dependent Products| |Red Hat Enterprise | |Virtualization Manager -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 07:53:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 07:53:44 +0000 Subject: [Bugs] [Bug 1673058] Network throughput usage increased x5 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673058 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jsecchiero at enter.eu, | |nbalacha at redhat.com Flags| |needinfo?(jsecchiero at enter. | |eu) --- Comment #1 from Nithya Balachandran --- Is this high throughput consistent? Please provide a tcpdump of the client process for about 30s to 1 min during the high throughput to see what packets gluster is sending: In a terminal to the client machine: tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22 Wait for 30s-1min and stop the capture. Send us the pcap file. Another user reported that turning off readdir-ahead worked for him. Please try that after capturing the statedump and see if it helps you. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 08:17:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 08:17:25 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 --- Comment #10 from Sanju --- We tried to reproduce this issue but couldn't hit it. If you happen to hit this issue, please provide us all the log files from /var/log/glusterfs (for both glusterfs-server and client from all the machines). Thanks, Sanju -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 10:24:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 10:24:18 +0000 Subject: [Bugs] [Bug 1670155] Tiered volume files disappear when a hot brick is failed/restored until the tier detached. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670155 hari gowtham changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hgowtham at redhat.com --- Comment #1 from hari gowtham --- Patch https://review.gluster.org/#/c/glusterfs/+/21331/ removes tier functionality from GlusterFS. Recommendation is to convert your tier volume to regular volume (either replicate, ec, or plain distribute) with "tier detach" command before upgrade, and use backend features like dm-cache etc to utilize the caching from backend to provide better performance and functionality. -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 11:17:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 11:17:22 +0000 Subject: [Bugs] [Bug 1673058] Network throughput usage increased x5 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673058 Alberto Bengoa changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bengoa at gmail.com --- Comment #2 from Alberto Bengoa --- (In reply to Nithya Balachandran from comment #1) > Is this high throughput consistent? > Please provide a tcpdump of the client process for about 30s to 1 min during > the high throughput to see what packets gluster is sending: > > In a terminal to the client machine: > tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22 > > Wait for 30s-1min and stop the capture. Send us the pcap file. > > Another user reported that turning off readdir-ahead worked for him. Please > try that after capturing the statedump and see if it helps you. I'm the another user and I can confirm the same behaviour here. On our tests we did: - Mounted the new cluster servers (running 5.3 version) using client 5.3 - Started a find . -type d on a directory with lots of directories. - It generated an outgoing traffic (on the client) of around 90mbps (so, inbound traffic on gluster server). We repeated the same test using 3.8 client (on 5.3 cluster) and the outgoing traffic on the client was just around 1.3 mbps. I can provide pcaps if needed. Cheers, Alberto Bengoa -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 13:37:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 13:37:25 +0000 Subject: [Bugs] [Bug 1677555] Glusterfs brick is crashed due to segfault caused by broken gfid symlink In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677555 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |medium CC| |atumball at redhat.com --- Comment #1 from Amar Tumballi --- tested by injecting broken symlink on release-6.0 branch, not happening. >From the log: --- [2019-02-21 13:33:34.841279] E [posix-handle.c:325:posix_is_malformed_link] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x11a)[0x7f2fa4d718ea] (--> /usr/local/lib/glusterfs/7dev/xlator/storage/posix.so(+0xe315)[0x7f2f9264d315] (--> /usr/local/lib/glusterfs/7dev/xlator/storage/posix.so(+0xe3fd)[0x7f2f9264d3fd] (--> /usr/local/lib/glusterfs/7dev/xlator/storage/posix.so(+0xe702)[0x7f2f9264d702] (--> /usr/local/lib/glusterfs/7dev/xlator/storage/posix.so(+0x34054)[0x7f2f92673054] ))))) 0-demo-posix: malformed internal link .. for /home/testdir/bricks/abcd.0/.glusterfs/91/e4/91e48e9c-8474-45db-9f7c-90fbeceeca6a [2019-02-21 13:33:34.841315] W [MSGID: 113077] [posix-inode-fd-ops.c:5354:posix_readdirp_fill] 0-demo-posix: Failed to create handle path, fd=0x7f2f680078a8, gfid=91e48e9c-8474-45db-9f7c-90fbeceeca6a --- So, the broken gfid issue is handled in latest release. Will get to test more possibilities in backend changes later. As we don't support touch gluster backend directly, not taking it as priority. Would be good to know how you reached to this situation. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 13:48:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 13:48:37 +0000 Subject: [Bugs] [Bug 1626085] "glusterfs --process-name fuse" crashes and leads to "Transport endpoint is not connected" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1626085 --- Comment #13 from Nithya Balachandran --- (In reply to GCth from comment #12) > One more - it's currently: > > glusterfs 5.3 > > installed from > https://download.gluster.org/pub/gluster/glusterfs/5/LATEST/Debian/stretch/ > amd64/apt We found an issue with the write-behind functionality in 5.3. Please run gluster v set performance.write-behind off and see if you still see crashes. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 21 18:12:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 18:12:24 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 --- Comment #9 from Jiffin --- # yum update glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm Loaded plugins: fastestmirror, ovl Examining glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-3.12.15-0.0.el7.x86_64 Examining glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-libs-3.12.15-0.0.el7.x86_64 Examining glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-api-3.12.15-0.0.el7.x86_64 Examining glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 Resolving Dependencies --> Running transaction check ---> Package glusterfs.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be obsoleting ---> Package glusterfs-api.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-api.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update ---> Package glusterfs-client-xlators.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-client-xlators.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update ---> Package glusterfs-gnfs.x86_64 0:3.12.15-0.0.el7 will be obsoleted ---> Package glusterfs-libs.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-libs.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: glusterfs x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 2.5 M replacing glusterfs-gnfs.x86_64 3.12.15-0.0.el7 Updating: glusterfs-api x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 197 k glusterfs-client-xlators x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 4.8 M glusterfs-libs x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1.3 M Transaction Summary =================================================================================================================================================================================================================== Install 1 Package Upgrade 3 Packages Total size: 8.9 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1/9 Installing : glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 2/9 Updating : glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 3/9 Updating : glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 4/9 Erasing : glusterfs-gnfs-3.12.15-0.0.el7.x86_64 5/9 Cleanup : glusterfs-api-3.12.15-0.0.el7.x86_64 6/9 Cleanup : glusterfs-3.12.15-0.0.el7.x86_64 7/9 Cleanup : glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 8/9 Cleanup : glusterfs-libs-3.12.15-0.0.el7.x86_64 9/9 Verifying : glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1/9 Verifying : glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 2/9 Verifying : glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 3/9 Verifying : glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 4/9 Verifying : glusterfs-gnfs-3.12.15-0.0.el7.x86_64 5/9 Verifying : glusterfs-libs-3.12.15-0.0.el7.x86_64 6/9 Verifying : glusterfs-3.12.15-0.0.el7.x86_64 7/9 Verifying : glusterfs-api-3.12.15-0.0.el7.x86_64 8/9 Verifying : glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 9/9 Installed: glusterfs.x86_64 0:4.1.7-0.3.git397ef0c.el7 Updated: glusterfs-api.x86_64 0:4.1.7-0.3.git397ef0c.el7 glusterfs-client-xlators.x86_64 0:4.1.7-0.3.git397ef0c.el7 glusterfs-libs.x86_64 0:4.1.7-0.3.git397ef0c.el7 Replaced: glusterfs-gnfs.x86_64 0:3.12.15-0.0.el7 posted the patch https://review.gluster.org/#/c/glusterfs/+/22190/ to fix this -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 18:26:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 18:26:32 +0000 Subject: [Bugs] [Bug 1679744] New: Minio gateway nas does not work with 2 + 1 dispersed volumes Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679744 Bug ID: 1679744 Summary: Minio gateway nas does not work with 2 + 1 dispersed volumes Product: GlusterFS Version: 4.1 Hardware: x86_64 OS: Linux Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: otavio.n.cipriani at gmail.com CC: bugs at gluster.org Target Milestone: --- External Bug ID: Github 7262 Classification: Community Description of problem: When starting Minio as a NAS gateway over GlusterFS, a FUTEX_WAIT_PRIVATE system call never returns. The problem does not happen when using a local filesystem, neither over NFS. It also does not occur when using 3-way replicated or 4 + 2 dispersed volumes. ## Expected Behavior Minio should work over GlusterFS. ## Current Behavior Minio never finishes starting. It gets stuck in a FUTEX_WAIT_PRIVATE system call. Here is the output of `env MINIO_ACCESS_KEY=admin MINIO_SECRET_KEY=password MINIO_REGION=sa-central-1 strace /usr/local/bin/minio gateway nas --certs-dir /usr/local/etc/minio/certs /var/local/lib/minio`: ``` execve("/usr/local/bin/minio", ["/usr/local/bin/minio", "gateway", "nas", "--certs-dir", "/usr/local/etc/minio/certs", "/var/local/lib/minio"], [/* 26 vars */]) = 0 arch_prctl(ARCH_SET_FS, 0x2612bb0) = 0 sched_getaffinity(0, 8192, [0 ...]) = 640 mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f8929000 mmap(0xc000000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000 mmap(0xc000000000, 67108864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xc000000000 mmap(NULL, 33554432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f6929000 mmap(NULL, 2162688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f6719000 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f6709000 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f66f9000 rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0 sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0 sigaltstack({ss_sp=0xc000002000, ss_flags=0, ss_size=32768}, NULL) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 gettid() = 7977 rt_sigaction(SIGHUP, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGHUP, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGINT, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGQUIT, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGILL, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGILL, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGTRAP, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGTRAP, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGABRT, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGABRT, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGBUS, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGBUS, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGFPE, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGFPE, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGUSR1, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGUSR1, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGSEGV, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGSEGV, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGUSR2, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGUSR2, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGPIPE, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGPIPE, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGALRM, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGALRM, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGTERM, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGSTKFLT, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGSTKFLT, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGCHLD, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGURG, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGURG, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGXCPU, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGXCPU, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGXFSZ, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGXFSZ, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGVTALRM, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGVTALRM, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGPROF, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGPROF, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGWINCH, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGWINCH, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGIO, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGIO, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGPWR, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGPWR, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGSYS, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGSYS, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRTMIN, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_1, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_2, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_2, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_3, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_3, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_4, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_4, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_5, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_5, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_6, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_6, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_7, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_7, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_8, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_8, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_9, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_9, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_10, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_10, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_11, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_11, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_12, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_12, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_13, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_13, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_14, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_14, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_15, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_15, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_16, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_16, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_17, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_17, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_18, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_18, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_19, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_19, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_20, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_20, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_21, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_21, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_22, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_22, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_23, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_23, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_24, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_24, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_25, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_25, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_26, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_26, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_27, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_27, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_28, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_28, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_29, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_29, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_30, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_30, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_31, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_31, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigaction(SIGRT_32, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGRT_32, {0x45df70, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x45e0a0}, NULL, 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[], [], 8) = 0 clone(child_stack=0xc00006c000, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM) = 7978 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[], [], 8) = 0 clone(child_stack=0xc00006e000, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM) = 7979 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 futex(0x2612c60, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 readlinkat(AT_FDCWD, "/proc/self/exe", "/usr/local/bin/minio", 128) = 20 fcntl(0, F_GETFL) = 0x8402 (flags O_RDWR|O_APPEND|O_LARGEFILE) mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f66b9000 fcntl(1, F_GETFL) = 0x8402 (flags O_RDWR|O_APPEND|O_LARGEFILE) fcntl(2, F_GETFL) = 0x8402 (flags O_RDWR|O_APPEND|O_LARGEFILE) ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 openat(AT_FDCWD, "/proc/sys/net/core/somaxconn", O_RDONLY|O_CLOEXEC) = 3 epoll_create1(EPOLL_CLOEXEC) = 4 epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264576, u64=140406615170816}}) = 0 fcntl(3, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fcntl(3, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0 read(3, "128\n", 65536) = 4 read(3, "", 65532) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 3, 0xc0000b1a0c) = 0 close(3) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 socket(AF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0 connect(3, {sa_family=AF_LOCAL, sun_path="/run/systemd/journal/socket"}, 30) = 0 futex(0x2612c60, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264576, u64=140406615170816}}) = 0 getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0 getpeername(3, {sa_family=AF_LOCAL, sun_path="/run/systemd/journal/socket"}, [30]) = 0 getpid() = 7977 getppid() = 7975 openat(AT_FDCWD, "/dev/tty", O_RDONLY|O_CLOEXEC) = 5 futex(0x2612c60, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264368, u64=140406615170608}}) = 0 fcntl(5, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0 getuid() = 0 getgid() = 0 newfstatat(AT_FDCWD, "/usr/local/sbin/uname", 0xc0001c6518, 0) = -1 ENOENT (No such file or directory) futex(0x2612c60, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 newfstatat(AT_FDCWD, "/sbin/uname", 0xc0001c65e8, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/bin/uname", {st_mode=S_IFREG|0755, st_size=33168, ...}, 0) = 0 openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc00018379c) = -1 EPERM (Operation not permitted) pipe2([7, 8], O_CLOEXEC) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = 0 fcntl(7, F_GETFL) = 0 (flags O_RDONLY) fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134263952, u64=140406615170192}}) = 0 fcntl(8, F_GETFL) = 0x1 (flags O_WRONLY) fcntl(8, F_SETFL, O_WRONLY|O_NONBLOCK) = 0 fcntl(8, F_GETFL) = 0x801 (flags O_WRONLY|O_NONBLOCK) fcntl(8, F_SETFL, O_WRONLY) = 0 fcntl(8, F_GETFL) = 0x1 (flags O_WRONLY) fcntl(8, F_SETFL, O_WRONLY) = 0 pipe2([9, 10], O_CLOEXEC) = 0 getpid() = 7977 rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0 clone(child_stack=0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 7982 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 close(10) = 0 read(9, "", 8) = 0 close(9) = 0 close(6) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 8, 0xc000183864) = 0 close(8) = 0 waitid(P_PID, 7982, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7982, si_uid=0, si_status=0, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0 wait4(7982, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, {ru_utime={0, 513}, ru_stime={0, 0}, ...}) = 7982 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 newfstatat(AT_FDCWD, "/usr/local/sbin/uname", 0xc0001c6858, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/sbin/uname", 0xc0001c6928, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/bin/uname", {st_mode=S_IFREG|0755, st_size=33168, ...}, 0) = 0 openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc00018379c) = -1 EPERM (Operation not permitted) pipe2([7, 8], O_CLOEXEC) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = 0 fcntl(7, F_GETFL) = 0 (flags O_RDONLY) fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134263952, u64=140406615170192}}) = 0 fcntl(8, F_GETFL) = 0x1 (flags O_WRONLY) fcntl(8, F_SETFL, O_WRONLY|O_NONBLOCK) = 0 fcntl(8, F_GETFL) = 0x801 (flags O_WRONLY|O_NONBLOCK) fcntl(8, F_SETFL, O_WRONLY) = 0 fcntl(8, F_GETFL) = 0x1 (flags O_WRONLY) fcntl(8, F_SETFL, O_WRONLY) = 0 pipe2([9, 10], O_CLOEXEC) = 0 getpid() = 7977 rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0 clone(child_stack=0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 7983 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 close(10) = 0 read(9, "", 8) = 0 close(9) = 0 close(6) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 8, 0xc000183864) = 0 close(8) = 0 waitid(P_PID, 7983, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7983, si_uid=0, si_status=0, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0 wait4(7983, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, {ru_utime={0, 473}, ru_stime={0, 0}, ...}) = 7983 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 newfstatat(AT_FDCWD, "/usr/local/sbin/uname", 0xc0001c6b98, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/sbin/uname", 0xc0001c6c68, 0) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/bin/uname", {st_mode=S_IFREG|0755, st_size=33168, ...}, 0) = 0 openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc00018379c) = -1 EPERM (Operation not permitted) pipe2([7, 8], O_CLOEXEC) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = 0 fcntl(7, F_GETFL) = 0 (flags O_RDONLY) fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134263952, u64=140406615170192}}) = 0 fcntl(8, F_GETFL) = 0x1 (flags O_WRONLY) fcntl(8, F_SETFL, O_WRONLY|O_NONBLOCK) = 0 fcntl(8, F_GETFL) = 0x801 (flags O_WRONLY|O_NONBLOCK) fcntl(8, F_SETFL, O_WRONLY) = 0 fcntl(8, F_GETFL) = 0x1 (flags O_WRONLY) fcntl(8, F_SETFL, O_WRONLY) = 0 pipe2([9, 10], O_CLOEXEC) = 0 getpid() = 7977 rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0 clone(child_stack=0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 7984 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 close(10) = 0 read(9, "", 8) = 0 close(9) = 0 close(6) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 8, 0xc000183864) = 0 close(8) = 0 waitid(P_PID, 7984, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7984, si_uid=0, si_status=0, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0 wait4(7984, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, {ru_utime={0, 467}, ru_stime={0, 0}, ...}) = 7984 getrandom("\204\234\370\334\267\237\340\345", 8, 0) = 8 getrandom("\31w\237\300\370\2329\343", 8, 0) = 8 getrandom("\0&\223\372I\\\2145\7)\211f", 12, 0) = 12 getrandom("\7\250\230\214\211\206\200\334\310\255=!", 12, 0) = 12 getpid() = 7977 newfstatat(AT_FDCWD, "/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}, 0) = 0 openat(AT_FDCWD, "/proc/stat", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = 0 fcntl(6, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fcntl(6, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0 read(6, "cpu 78425 20 74048 7128110 303 "..., 4096) = 785 read(6, "", 3311) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000183124) = 0 close(6) = 0 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 futex(0x2612190, FUTEX_WAKE_PRIVATE, 1) = 1 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 6 bind(6, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 sendto(6, "\21\0\0\0\26\0\1\3\1\0\0\0\0\0\0\0\0", 17, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 17 recvfrom(6, "L\0\0\0\24\0\2\0\1\0\0\0)\37\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 164 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 recvfrom(6, "H\0\0\0\24\0\2\0\1\0\0\0)\37\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 144 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 recvfrom(6, "\24\0\0\0\3\0\2\0\1\0\0\0)\37\0\0\0\0\0\0", 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 20 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 close(6) = 0 socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 6 bind(6, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 sendto(6, "\21\0\0\0\22\0\1\3\1\0\0\0\0\0\0\0\0", 17, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 17 recvfrom(6, "\320\4\0\0\20\0\2\0\1\0\0\0)\37\0\0\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0"..., 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 2472 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 recvfrom(6, "\24\0\0\0\3\0\2\0\1\0\0\0)\37\0\0\0\0\0\0", 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 20 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 close(6) = 0 socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 6 bind(6, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 sendto(6, "\21\0\0\0\26\0\1\3\1\0\0\0\0\0\0\0\0", 17, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 17 recvfrom(6, "L\0\0\0\24\0\2\0\1\0\0\0)\37\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 164 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 recvfrom(6, "H\0\0\0\24\0\2\0\1\0\0\0)\37\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 144 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 recvfrom(6, "\24\0\0\0\3\0\2\0\1\0\0\0)\37\0\0\0\0\0\0", 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 20 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 close(6) = 0 socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 6 bind(6, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 sendto(6, "\21\0\0\0\22\0\1\3\1\0\0\0\0\0\0\0\0", 17, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 17 recvfrom(6, "\320\4\0\0\20\0\2\0\1\0\0\0)\37\0\0\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0"..., 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 2472 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 recvfrom(6, "\24\0\0\0\3\0\2\0\1\0\0\0)\37\0\0\0\0\0\0", 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 20 getsockname(6, {sa_family=AF_NETLINK, pid=7977, groups=00000000}, [12]) = 0 close(6) = 0 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 getrandom("\345\r\205\262m.\376Q", 8, 0) = 8 getrandom("\330\236\304sX\221\"\17\270a\246\305\335&?\315", 16, 0) = 16 getrandom("\261\237\327\262\251\212\263\237", 8, 0) = 8 getrandom("`\312'Gsi\273J", 8, 0) = 8 newfstatat(AT_FDCWD, "/usr/local/bin/minio", {st_mode=S_IFREG|0755, st_size=35695328, ...}, 0) = 0 newfstatat(AT_FDCWD, "/usr/local/bin/minio", {st_mode=S_IFREG|0755, st_size=35695328, ...}, 0) = 0 newfstatat(AT_FDCWD, "/root/.minio", {st_mode=S_IFDIR|0700, st_size=6, ...}, 0) = 0 newfstatat(AT_FDCWD, "/usr/local/etc/minio/certs", {st_mode=S_IFDIR|0700, st_size=17, ...}, 0) = 0 newfstatat(AT_FDCWD, "/usr/local/etc/minio/certs/CAs", {st_mode=S_IFDIR|0700, st_size=6, ...}, 0) = 0 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_TCP) = 6 close(6) = 0 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_TCP) = 6 setsockopt(6, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 bind(6, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_TCP) = 7 setsockopt(7, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0 bind(7, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 close(7) = 0 close(6) = 0 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 6 setsockopt(6, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0 setsockopt(6, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0 setsockopt(6, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 bind(6, {sa_family=AF_INET6, sin6_port=htons(9000), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 listen(6, 128) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = 0 getsockname(6, {sa_family=AF_INET6, sin6_port=htons(9000), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc00045260c) = 0 close(6) = 0 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 6 setsockopt(6, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0 setsockopt(6, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 bind(6, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 listen(6, 128) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = 0 getsockname(6, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc00045260c) = 0 close(6) = 0 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 6 setsockopt(6, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 setsockopt(6, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0 setsockopt(6, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 bind(6, {sa_family=AF_INET6, sin6_port=htons(9000), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 listen(6, 128) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = 0 getsockname(6, {sa_family=AF_INET6, sin6_port=htons(9000), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc00045260c) = 0 close(6) = 0 newfstatat(AT_FDCWD, "/usr/local/etc/minio/certs/public.crt", 0xc00021df18, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000452374) = -1 EPERM (Operation not permitted) fstat(6, {st_mode=S_IFREG|0444, st_size=662068, ...}) = 0 mmap(NULL, 1439992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f6559000 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 epoll_pwait(4, [{EPOLLOUT, {u32=4134264368, u64=140406615170608}}], 128, 0, NULL, 4) = 1 read(6, "# AC BOA VISTA CERTIFICADORA\n---"..., 662580) = 662068 read(6, "", 512) = 0 close(6) = 0 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f6549000 epoll_pwait(4, [{EPOLLOUT, {u32=4134264368, u64=140406615170608}}], 128, 0, NULL, 73156) = 1 epoll_pwait(4, [{EPOLLOUT, {u32=4134264368, u64=140406615170608}}], 128, 0, NULL, 1) = 1 futex(0x2612190, FUTEX_WAKE_PRIVATE, 1) = 1 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb2f6539000 epoll_pwait(4, [{EPOLLOUT, {u32=4134264368, u64=140406615170608}}], 128, 0, NULL, 73156) = 1 openat(AT_FDCWD, "/etc/ssl/certs", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc00085431c) = -1 EPERM (Operation not permitted) getdents64(6, /* 7 entries */, 8192) = 240 getdents64(6, /* 0 entries */, 8192) = 0 lstat("/etc/ssl/certs/ca-bundle.crt", {st_mode=S_IFLNK|0777, st_size=49, ...}) = 0 futex(0x2612190, FUTEX_WAKE_PRIVATE, 1) = 1 lstat("/etc/ssl/certs/ca-bundle.trust.crt", {st_mode=S_IFLNK|0777, st_size=55, ...}) = 0 lstat("/etc/ssl/certs/Makefile", {st_mode=S_IFREG|0644, st_size=2516, ...}) = 0 lstat("/etc/ssl/certs/make-dummy-cert", {st_mode=S_IFREG|0755, st_size=610, ...}) = 0 lstat("/etc/ssl/certs/renew-dummy-cert", {st_mode=S_IFREG|0755, st_size=829, ...}) = 0 close(6) = 0 openat(AT_FDCWD, "/etc/ssl/certs/Makefile", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000854374) = -1 EPERM (Operation not permitted) fstat(6, {st_mode=S_IFREG|0644, st_size=2516, ...}) = 0 read(6, "UTF8 := $(shell locale -c LC_CTY"..., 3028) = 2516 read(6, "", 512) = 0 close(6) = 0 openat(AT_FDCWD, "/etc/ssl/certs/ca-bundle.crt", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000854374) = -1 EPERM (Operation not permitted) fstat(6, {st_mode=S_IFREG|0444, st_size=662068, ...}) = 0 read(6, "# AC BOA VISTA CERTIFICADORA\n---"..., 662580) = 662068 read(6, "", 512) = 0 close(6) = 0 epoll_pwait(4, [{EPOLLOUT, {u32=4134264368, u64=140406615170608}}], 128, 0, NULL, 1) = 1 epoll_pwait(4, [{EPOLLOUT, {u32=4134264368, u64=140406615170608}}], 128, 0, NULL, 73156) = 1 futex(0x2612190, FUTEX_WAKE_PRIVATE, 1) = 1 epoll_pwait(4, [{EPOLLOUT, {u32=4134264368, u64=140406615170608}}], 128, 0, NULL, 1) = 1 openat(AT_FDCWD, "/etc/ssl/certs/ca-bundle.trust.crt", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000a10374) = -1 EPERM (Operation not permitted) fstat(6, {st_mode=S_IFREG|0444, st_size=717687, ...}) = 0 read(6, "# AC BOA VISTA CERTIFICADORA\n---"..., 718199) = 717687 read(6, "", 512) = 0 close(6) = 0 openat(AT_FDCWD, "/etc/ssl/certs/make-dummy-cert", O_RDONLY|O_CLOEXEC) = 6 futex(0x2612190, FUTEX_WAKE_PRIVATE, 1) = 1 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000a10374) = -1 EPERM (Operation not permitted) fstat(6, {st_mode=S_IFREG|0755, st_size=610, ...}) = 0 read(6, "#!/bin/sh\numask 077\n\nanswers() {"..., 1122) = 610 read(6, "", 512) = 0 close(6) = 0 openat(AT_FDCWD, "/etc/ssl/certs/renew-dummy-cert", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000a10374) = -1 EPERM (Operation not permitted) fstat(6, {st_mode=S_IFREG|0755, st_size=829, ...}) = 0 read(6, "#!/bin/bash\n\nif [ $# -eq 0 ]; th"..., 1341) = 829 read(6, "", 512) = 0 close(6) = 0 openat(AT_FDCWD, "/usr/local/etc/minio/certs/CAs", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000a10474) = -1 EPERM (Operation not permitted) getdents64(6, /* 2 entries */, 262144) = 48 getdents64(6, /* 0 entries */, 262144) = 0 close(6) = 0 openat(AT_FDCWD, "/proc/sys/kernel/threads-max", O_RDONLY|O_CLOEXEC) = 6 epoll_ctl(4, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4134264160, u64=140406615170400}}) = 0 fcntl(6, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fcntl(6, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0 fstat(6, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 read(6, "15828\n", 512) = 6 read(6, "", 1530) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 6, 0xc000a105f4) = 0 close(6) = 0 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0 setrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024}) = 0 getrlimit(RLIMIT_AS, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0 setrlimit(RLIMIT_AS, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0 getrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024}) = 0 rt_sigprocmask(SIG_SETMASK, ~[], [], 8) = 0 clone(child_stack=0xc000654000, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM) = 7985 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0 futex(0xc00015d640, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x2612c60, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc0003279c0, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x2612c60, FUTEX_WAIT_PRIVATE, 0, NULL ``` ## Steps to Reproduce (for bugs) 1. Create a Gluster cluster with a 3 bricks dispersed volume; 2. Mount the Gluster volume; 3. Start `minio gateway nas` using the mounted volume as data directory. ## Your Environment * Version used (`minio version`): - Version: 2019-02-14T00:21:45Z - Release-Tag: RELEASE.2019-02-14T00-21-45Z - Commit-ID: a51781e5cf4a055546301ed3c49b68d7209e51d1 * Environment name and version: - CentOS 7. * Operating System and version (`uname -a`): - Linux myhost.localdomain 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 18:52:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 18:52:25 +0000 Subject: [Bugs] [Bug 1679744] Minio gateway nas does not work with 2 + 1 dispersed volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679744 --- Comment #1 from Otavio Cipriani --- Here is the output of `gluster --version` (latest packages from CentOS SIG, version 4.1): glusterfs 4.1.7 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. Here is the output of `gluster volume info` **after** applying the settings from the _virt_ group (defaults do not work, either): Volume Name: myvolume Type: Disperse Volume ID: 82a71fc3-2ffa-42a3-8fe1-b439b7c3211c Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: server-h01.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick2: server-h02.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick3: server-h03.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Options Reconfigured: cluster.choose-local: off user.cifs: off network.remote-dio: enable performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off features.shard: on cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.quorum-type: auto cluster.eager-lock: enable transport.address-family: inet nfs.disable: on The `cluster.shd*` settings were not applied, since they cannot be set for a non-replicated volumes. I stopped/started and umounted/mounted the volume, but the problem persists. The problem does **not** occur with a 3-way replicated volume: Volume Name: myvolume Type: Replicate Volume ID: eb0c9e63-ddb2-47ef-a6a8-26ddfd31d627 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: server-h01.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick2: server-h02.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick3: server-h03.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off Also does **not** occur when using a 4 + 2 dispersed volume: Volume Name: myvolume Type: Disperse Volume ID: 66bc6521-6e09-4f7a-a04b-79f66e424024 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (4 + 2) = 6 Transport-type: tcp Bricks: Brick1: server-p01.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick2: server-p02.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick3: server-p03.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick4: server-p04.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick5: server-p05.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Brick6: server-p06.cnj.jus.br:/var/local/lib/glusterfs/brick01/brick Options Reconfigured: nfs.disable: on transport.address-family: inet -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 21:36:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 21:36:04 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 Raghavendra Bhat changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rabhat at redhat.com Flags| |needinfo?(abhaysingh1722 at ya | |hoo.in) --- Comment #47 from Raghavendra Bhat --- I am looking at the bitrot error. The error is while doing a getxattr on the xattr "trusted.glusterfs.get-signature". But one of the attached files (bug-1207627-bitrot-scrub-status.log.txt) shows the following extended attributes. Unfortunately "trusted.bit-rot.signature" is not seen. TEST 22 (line 55): trusted.bit-rot.bad-file check_for_xattr trusted.bit-rot.bad-file //d/backends/patchy1/FILE1 not ok 22 Got "" instead of "trusted.bit-rot.bad-file", LINENUM:55 RESULT 22: 1 getfattr: Removing leading '/' from absolute path names # file: d/backends/patchy1 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000000000000ffffffff trusted.glusterfs.volume-id=0xdc6b47ebd73f46798f5a86b42678fb44 Would it be possible for you to upload the bitd.log from /var/log/glusterfs directory after running the bitrot test? Because, it is bit-rot daemon (whose log file is bitd.log that I asked) which sets sends the setxattr requests to the brick to set the extended attributes needed for bit-rot detection. If setting that attribute failed, then bit-rot daemon's log file would have got an error. Regards, Raghavendra -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 03:32:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:32:38 +0000 Subject: [Bugs] [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-22 03:32:38 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22231 (performance/write-behind: handle call-stub leaks) merged (#1) on release-6 by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 03:32:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:32:39 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1678570, which changed state. Bug 1678570 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1678570 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 03:32:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:32:39 +0000 Subject: [Bugs] [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 Bug 1676356 depends on bug 1678570, which changed state. Bug 1678570 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1678570 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 03:33:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:33:34 +0000 Subject: [Bugs] [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22230 (performance/write-behind: fix use-after-free in readdirp) merged (#2) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 03:33:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:33:34 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1678570, which changed state. Bug 1678570 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1678570 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 03:33:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:33:35 +0000 Subject: [Bugs] [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 Bug 1676356 depends on bug 1678570, which changed state. Bug 1678570 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1678570 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 03:34:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:34:33 +0000 Subject: [Bugs] [Bug 1679275] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679275 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22244 (dht: fix double extra unref of inode at heal path) merged (#2) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 03:35:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:35:04 +0000 Subject: [Bugs] [Bug 1679004] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679004 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-22 03:35:04 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22245 (cluster/dht: Request linkto xattrs in dht_rmdir opendir) merged (#2) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 03:35:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:35:04 +0000 Subject: [Bugs] [Bug 1672869] With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672869 Bug 1672869 depends on bug 1679004, which changed state. Bug 1679004 Summary: With parallel-readdir enabled, deleting a directory containing stale linkto files fails with "Directory not empty" https://bugzilla.redhat.com/show_bug.cgi?id=1679004 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 03:46:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 03:46:22 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #562 from Worker Ant --- REVIEW: https://review.gluster.org/22233 (glusterd: fix get-state leak) merged (#4) on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 04:09:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 04:09:41 +0000 Subject: [Bugs] [Bug 1673058] Network throughput usage increased x5 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673058 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |atumball at redhat.com --- Comment #3 from Nithya Balachandran --- Assigning this to Amar to be reassigned appropriately. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 04:12:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 04:12:00 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 abhays changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(abhaysingh1722 at ya | |hoo.in) | --- Comment #48 from abhays --- Created attachment 1537322 --> https://bugzilla.redhat.com/attachment.cgi?id=1537322&action=edit Bitrot.log Contains bitd.log and scrub.log for the test case bug-1207627-bitrot-scrub-status.t -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 04:13:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 04:13:17 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 --- Comment #8 from Worker Ant --- REVIEW: https://review.gluster.org/22226 (doc: Added initial release notes for release-6) merged (#4) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 04:14:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 04:14:50 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #49 from abhays --- Comment on attachment 1537322 --> https://bugzilla.redhat.com/attachment.cgi?id=1537322 Bitrot.log Contains bitd.log and scrub.log for the test case bug-1207627-bitrot-scrub-status.t We have observed error on the line below in the scrub.log:- [2019-02-14 04:16:31.164722] E [MSGID: 118008] [bit-rot.c:479:br_log_object] 0-patchy-bit-rot-0: fgetxattr() failed on object b179d471-b5a0-49c6-a22b-08fbf599e734 [Cannot allocate memory] -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 04:47:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 04:47:20 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #28 from Artem Russakovskii --- (In reply to Nithya Balachandran from comment #27) > (In reply to Artem Russakovskii from comment #26) > > I wish I could say the same thing. Unfortunately, I had another crash on the > > same server yesterday, with performance.write-behind still set to off. I've > > emailed the core file privately to the relevant people. > > > > > Thanks Artem. I have downloaded the coredump and will take a look sometime > this week. Can you share any details about the workload that might help us > narrow it down? We have a blog with images stored on gluster. Let's say 400k average views per day, split across 4 web servers, but the majority is cached by Cloudflare. Probably a hundred file writes per day, just images for blog posts. The 2nd site hosts Android apks and has quite a bit more traffic than the blog, but all apks are also cached by Cloudflare. Several hundred apks written per day. 4.1 didn't break a sweat with our setup, so I really don't know why I'm having so much trouble. And it's always the same web server of the 4 too, and they're supposed to be roughly identical. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 04:57:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 04:57:55 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22250 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 04:57:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 04:57:56 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 --- Comment #9 from Worker Ant --- REVIEW: https://review.gluster.org/22250 (doc: Update release notes for Samba integration) posted (#1) for review on release-6 by Anoop C S -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 06:51:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 06:51:32 +0000 Subject: [Bugs] [Bug 1634664] Inconsistent quorum checks during open and fd based operations In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1634664 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22251 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 06:51:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 06:51:33 +0000 Subject: [Bugs] [Bug 1634664] Inconsistent quorum checks during open and fd based operations In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1634664 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22251 (cluster/afr: Add quorum checks to open & opendir fops) posted (#1) for review on master by Karthik U S -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 07:45:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 07:45:41 +0000 Subject: [Bugs] [Bug 1679892] New: assertion failure log in glusterd.log file when a volume start is triggered Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679892 Bug ID: 1679892 Summary: assertion failure log in glusterd.log file when a volume start is triggered Product: GlusterFS Version: 4.1 Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: amukherj at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: [2019-02-22 07:38:28.772914] E [MSGID: 101191] [event-epoll.c:765:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-22 07:38:32.322872] I [glusterd-utils.c:6305:glusterd_brick_start] 0-management: starting a fresh brick process for brick /tmp/b1 [2019-02-22 07:38:32.420144] I [MSGID: 106142] [glusterd-pmap.c:290:pmap_registry_bind] 0-pmap: adding brick /tmp/b1 on port 49152 [2019-02-22 07:38:32.420635] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2019-02-22 07:38:32.491504] E [mem-pool.c:351:__gf_free] (-->/usr/local/lib/glusterfs/6.0alpha/xlator/mgmt/glusterd.so(+0x4842e) [0x7fc95a8f742e] -->/usr/local/lib/glusterfs/6.0alpha/xlator/mgmt/glusterd.so(+0x4821a) [0x7fc95a8f721a] -->/usr/local/lib/libglusterfs.so.0(__gf_free+0x22d) [0x7fc96042ccfd] ) 0-: Assertion failed: mem_acct->rec[header->type].size >= header->size [2019-02-22 07:38:32.492228] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2019-02-22 07:38:32.493431] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600 [2019-02-22 07:38:32.494848] I [rpc-clnt.c:1005:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2019-02-22 07:38:32.495530] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped [2019-02-22 07:38:32.495655] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: nfs service is stopped [2019-02-22 07:38:32.495728] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Out of a 3 node cluster setup create a replica 3 volume and start it. Actual results: assertion failure and 'failed to dispatch handler' errors are seen. Expected results: No errors in the glusterd log should be seen. Assertion failure log tends to indicate there might be a corruption too which is more severe here. Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 07:45:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 07:45:56 +0000 Subject: [Bugs] [Bug 1679892] assertion failure log in glusterd.log file when a volume start is triggered In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679892 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|unspecified |high -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 08:26:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 08:26:14 +0000 Subject: [Bugs] [Bug 1679904] New: client log flooding with intentional socket shutdown message when a brick is down Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679904 Bug ID: 1679904 Summary: client log flooding with intentional socket shutdown message when a brick is down Product: GlusterFS Version: 5 Status: NEW Component: core Assignee: mchangir at redhat.com Reporter: amukherj at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: client log flooding with intentional socket shutdown message when a brick is down [2019-02-22 08:24:42.472457] I [socket.c:811:__socket_shutdown] 0-test-vol-client-0: intentional socket shutdown(5) Version-Release number of selected component (if applicable): glusterfs-6 How reproducible: Always Steps to Reproduce: 1. 1 X 3 volume created and started over a 3 node cluster 2. mount a fuse client 3. kill a brick 4. Observe that fuse client log is flooded with the intentional socket shutdown message after every 3 seconds. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 08:31:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 08:31:49 +0000 Subject: [Bugs] [Bug 1679904] client log flooding with intentional socket shutdown message when a brick is down In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679904 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672818 (glusterfs-6.0) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 08:31:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 08:31:49 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1679904 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679904 [Bug 1679904] client log flooding with intentional socket shutdown message when a brick is down -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 08:32:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 08:32:00 +0000 Subject: [Bugs] [Bug 1679904] client log flooding with intentional socket shutdown message when a brick is down In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679904 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Version|5 |6 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 08:32:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 08:32:36 +0000 Subject: [Bugs] [Bug 1679892] assertion failure log in glusterd.log file when a volume start is triggered In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679892 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672818 (glusterfs-6.0) Assignee|bugs at gluster.org |srakonde at redhat.com Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 08:32:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 08:32:36 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1679892 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679892 [Bug 1679892] assertion failure log in glusterd.log file when a volume start is triggered -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 09:22:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 09:22:47 +0000 Subject: [Bugs] [Bug 1679409] [Disperse] : Improve heal info command to get the report quickly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679409 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22252 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 09:22:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 09:22:48 +0000 Subject: [Bugs] [Bug 1679409] [Disperse] : Improve heal info command to get the report quickly In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679409 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22252 (cluster/ec : Improve heal info command to get the report quickly) posted (#1) for review on master by Ashish Pandey -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 11:08:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 11:08:33 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-22 11:08:33 --- Comment #10 from Worker Ant --- REVIEW: https://review.gluster.org/22190 (packaging: Obsoleting glusterfs-gnfs for upgrade) merged (#4) on master by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 11:19:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 11:19:29 +0000 Subject: [Bugs] [Bug 1679965] New: Upgrade from glusterfs 3.12 to gluster 4/5 broken Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679965 Bug ID: 1679965 Summary: Upgrade from glusterfs 3.12 to gluster 4/5 broken Product: GlusterFS Version: 6 Status: NEW Component: packaging Severity: urgent Priority: urgent Assignee: bugs at gluster.org Reporter: jthottan at redhat.com CC: bugs at gluster.org, hunter86_bg at yahoo.com, jthottan at redhat.com, kkeithle at redhat.com, sabose at redhat.com, sbonazzo at redhat.com, sgirijan at redhat.com Depends On: 1672711 Blocks: 1677319 (Gluster_5_Affecting_oVirt_4.3) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1672711 +++ Description of problem: While updating glusterfs 3.12, run into below error : Error: Package: glusterfs-gnfs-3.12.15-1.el7.x86_64 (@ovirt-4.2-centos-gluster312) Requires: glusterfs(x86-64) = 3.12.15-1.el7 Removing: glusterfs-3.12.15-1.el7.x86_64 (@ovirt-4.2-centos-gluster312) glusterfs(x86-64) = 3.12.15-1.el7 Updated By: glusterfs-5.3-1.el7.x86_64 (ovirt-4.3-centos-gluster5) glusterfs(x86-64) = 5.3-1.el7 Version-Release number of selected component (if applicable): 3.12 How reproducible: Always --- Additional comment from Sahina Bose on 2019-02-06 05:43:58 UTC --- Jiffin, can you help with this? --- Additional comment from Worker Ant on 2019-02-11 10:59:16 UTC --- REVIEW: https://review.gluster.org/22190 (packagaing : check lower version for gluster-gnfs incase of upgrade) posted (#1) for review on master by jiffin tony Thottan --- Additional comment from Kaleb KEITHLEY on 2019-02-12 13:19:09 UTC --- This is because gnfs is has been deprecated (for a long time now) and the glusterfs-gnfs subpackage is no longer built for the CentOS Storage SIG packages starting with GlusterFS-5. If you manually remove the glusterfs-gnfs package first, then update, the update will work. (Are you using NFS even? If not, then you don't need -gnfs in the first place. If you are, there is NFS-Ganesha.) --- Additional comment from Kaleb KEITHLEY on 2019-02-15 12:49:55 UTC --- Does removing glusterfs-gnfs manually/explicitly, followed by update, work? AFAIK oVirt doesn't use NFS at all and never needed glusterfs-gnfs in the first place. If glusterfs-gnfs was being used on the same system for something else, it should be replaced with a nfs-ganesha deployment, also from the Storage SIG. And you can't use an old version of glusterfs-gnfs (e.g. 3.12.x) with glusterfs-5.x. I will close this as NOTABUG in a few days unless I hear back. --- Additional comment from Sahina Bose on 2019-02-15 13:56:06 UTC --- (In reply to Kaleb KEITHLEY from comment #4) > Does removing glusterfs-gnfs manually/explicitly, followed by update, work? That works, but is not the option that we want to provide users. Since updates happen from UI, we want this to be as seamless as possible. > > AFAIK oVirt doesn't use NFS at all and never needed glusterfs-gnfs in the > first place. OVirt users who use nfs option to mount gluster, do use glusterfs-gnfs. And we had a dependency on glusterfs-gnfs to workaround an earlier bug, so this package is now available on all hosts. > > If glusterfs-gnfs was being used on the same system for something else, it > should be replaced with a nfs-ganesha deployment, also from the Storage SIG. > > And you can't use an old version of glusterfs-gnfs (e.g. 3.12.x) with > glusterfs-5.x. > > I will close this as NOTABUG in a few days unless I hear back. We want the upgrade from 3.12 to > 4 versions to work. So adding Obsoletes in glusterfs spec as suggested by others - is this a possible option? --- Additional comment from Strahil Nikolov on 2019-02-20 20:20:19 UTC --- It seems that glusterfs-gnfs is a depedency for vdsm-gluster without which I was not able to deploy ovirt-4.2.7 self-hosted (prep stage failed as method "gluster" was unknown): [root at ovirt2 ~]# yum --assumeno remove glusterfs-gnfs Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Repository centos-sclo-rh-release is listed more than once in the configuration Resolving Dependencies --> Running transaction check ---> Package glusterfs-gnfs.x86_64 0:3.12.15-1.el7 will be erased --> Processing Dependency: glusterfs-gnfs for package: vdsm-gluster-4.20.46-1.el7.x86_64 --> Running transaction check ---> Package vdsm-gluster.x86_64 0:4.20.46-1.el7 will be erased --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================= Removing: glusterfs-gnfs x86_64 3.12.15-1.el7 @ovirt-4.2-centos-gluster312 445 k Removing for dependencies: vdsm-gluster x86_64 4.20.46-1.el7 @ovirt-4.2 260 k Transaction Summary ======================================================================================================================================================= Remove 1 Package (+1 Dependent package) Installed size: 705 k Exiting on user command Your transaction was saved, rerun it with: yum load-transaction /tmp/yum_save_tx.2019-02-20.22-19.2e8Lcz.yumtx Uploading Enabled Repositories Report Loaded plugins: fastestmirror, product-id, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Repository centos-sclo-rh-release is listed more than once in the configuration Cannot upload enabled repos report, is this client registered? --- Additional comment from Strahil Nikolov on 2019-02-20 20:53:27 UTC --- I have removed glusterfs-gnfs manually and updated. The update process finished successfully, but the web interface shows unsynced entries present , while gluster volume heal engine info - shows nothing... --- Additional comment from Strahil Nikolov on 2019-02-20 20:55:41 UTC --- (In reply to Strahil Nikolov from comment #7) > I have removed glusterfs-gnfs manually and updated. > The update process finished successfully, but the web interface shows > unsynced entries present , while gluster volume heal engine info - shows > nothing... rpm -e --nodeps glusterfs-gnfs --- Additional comment from Jiffin on 2019-02-21 18:12:24 UTC --- # yum update glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm Loaded plugins: fastestmirror, ovl Examining glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-3.12.15-0.0.el7.x86_64 Examining glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-libs-3.12.15-0.0.el7.x86_64 Examining glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-api-3.12.15-0.0.el7.x86_64 Examining glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 Resolving Dependencies --> Running transaction check ---> Package glusterfs.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be obsoleting ---> Package glusterfs-api.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-api.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update ---> Package glusterfs-client-xlators.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-client-xlators.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update ---> Package glusterfs-gnfs.x86_64 0:3.12.15-0.0.el7 will be obsoleted ---> Package glusterfs-libs.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-libs.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: glusterfs x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 2.5 M replacing glusterfs-gnfs.x86_64 3.12.15-0.0.el7 Updating: glusterfs-api x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 197 k glusterfs-client-xlators x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 4.8 M glusterfs-libs x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1.3 M Transaction Summary =================================================================================================================================================================================================================== Install 1 Package Upgrade 3 Packages Total size: 8.9 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1/9 Installing : glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 2/9 Updating : glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 3/9 Updating : glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 4/9 Erasing : glusterfs-gnfs-3.12.15-0.0.el7.x86_64 5/9 Cleanup : glusterfs-api-3.12.15-0.0.el7.x86_64 6/9 Cleanup : glusterfs-3.12.15-0.0.el7.x86_64 7/9 Cleanup : glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 8/9 Cleanup : glusterfs-libs-3.12.15-0.0.el7.x86_64 9/9 Verifying : glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1/9 Verifying : glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 2/9 Verifying : glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 3/9 Verifying : glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 4/9 Verifying : glusterfs-gnfs-3.12.15-0.0.el7.x86_64 5/9 Verifying : glusterfs-libs-3.12.15-0.0.el7.x86_64 6/9 Verifying : glusterfs-3.12.15-0.0.el7.x86_64 7/9 Verifying : glusterfs-api-3.12.15-0.0.el7.x86_64 8/9 Verifying : glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 9/9 Installed: glusterfs.x86_64 0:4.1.7-0.3.git397ef0c.el7 Updated: glusterfs-api.x86_64 0:4.1.7-0.3.git397ef0c.el7 glusterfs-client-xlators.x86_64 0:4.1.7-0.3.git397ef0c.el7 glusterfs-libs.x86_64 0:4.1.7-0.3.git397ef0c.el7 Replaced: glusterfs-gnfs.x86_64 0:3.12.15-0.0.el7 posted the patch https://review.gluster.org/#/c/glusterfs/+/22190/ to fix this --- Additional comment from Worker Ant on 2019-02-22 11:08:33 UTC --- REVIEW: https://review.gluster.org/22190 (packaging: Obsoleting glusterfs-gnfs for upgrade) merged (#4) on master by Niels de Vos Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 11:19:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 11:19:29 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Jiffin changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1679965 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679965 [Bug 1679965] Upgrade from glusterfs 3.12 to gluster 4/5 broken -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 11:20:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 11:20:25 +0000 Subject: [Bugs] [Bug 1679968] New: Upgrade from glusterfs 3.12 to gluster 4/5 broken Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679968 Bug ID: 1679968 Summary: Upgrade from glusterfs 3.12 to gluster 4/5 broken Product: GlusterFS Version: 5 Status: NEW Component: packaging Severity: urgent Priority: urgent Assignee: bugs at gluster.org Reporter: jthottan at redhat.com CC: bugs at gluster.org, hunter86_bg at yahoo.com, jthottan at redhat.com, kkeithle at redhat.com, sabose at redhat.com, sbonazzo at redhat.com, sgirijan at redhat.com Depends On: 1672711 Blocks: 1677319 (Gluster_5_Affecting_oVirt_4.3), 1679965 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1672711 +++ Description of problem: While updating glusterfs 3.12, run into below error : Error: Package: glusterfs-gnfs-3.12.15-1.el7.x86_64 (@ovirt-4.2-centos-gluster312) Requires: glusterfs(x86-64) = 3.12.15-1.el7 Removing: glusterfs-3.12.15-1.el7.x86_64 (@ovirt-4.2-centos-gluster312) glusterfs(x86-64) = 3.12.15-1.el7 Updated By: glusterfs-5.3-1.el7.x86_64 (ovirt-4.3-centos-gluster5) glusterfs(x86-64) = 5.3-1.el7 Version-Release number of selected component (if applicable): 3.12 How reproducible: Always --- Additional comment from Sahina Bose on 2019-02-06 05:43:58 UTC --- Jiffin, can you help with this? --- Additional comment from Worker Ant on 2019-02-11 10:59:16 UTC --- REVIEW: https://review.gluster.org/22190 (packagaing : check lower version for gluster-gnfs incase of upgrade) posted (#1) for review on master by jiffin tony Thottan --- Additional comment from Kaleb KEITHLEY on 2019-02-12 13:19:09 UTC --- This is because gnfs is has been deprecated (for a long time now) and the glusterfs-gnfs subpackage is no longer built for the CentOS Storage SIG packages starting with GlusterFS-5. If you manually remove the glusterfs-gnfs package first, then update, the update will work. (Are you using NFS even? If not, then you don't need -gnfs in the first place. If you are, there is NFS-Ganesha.) --- Additional comment from Kaleb KEITHLEY on 2019-02-15 12:49:55 UTC --- Does removing glusterfs-gnfs manually/explicitly, followed by update, work? AFAIK oVirt doesn't use NFS at all and never needed glusterfs-gnfs in the first place. If glusterfs-gnfs was being used on the same system for something else, it should be replaced with a nfs-ganesha deployment, also from the Storage SIG. And you can't use an old version of glusterfs-gnfs (e.g. 3.12.x) with glusterfs-5.x. I will close this as NOTABUG in a few days unless I hear back. --- Additional comment from Sahina Bose on 2019-02-15 13:56:06 UTC --- (In reply to Kaleb KEITHLEY from comment #4) > Does removing glusterfs-gnfs manually/explicitly, followed by update, work? That works, but is not the option that we want to provide users. Since updates happen from UI, we want this to be as seamless as possible. > > AFAIK oVirt doesn't use NFS at all and never needed glusterfs-gnfs in the > first place. OVirt users who use nfs option to mount gluster, do use glusterfs-gnfs. And we had a dependency on glusterfs-gnfs to workaround an earlier bug, so this package is now available on all hosts. > > If glusterfs-gnfs was being used on the same system for something else, it > should be replaced with a nfs-ganesha deployment, also from the Storage SIG. > > And you can't use an old version of glusterfs-gnfs (e.g. 3.12.x) with > glusterfs-5.x. > > I will close this as NOTABUG in a few days unless I hear back. We want the upgrade from 3.12 to > 4 versions to work. So adding Obsoletes in glusterfs spec as suggested by others - is this a possible option? --- Additional comment from Strahil Nikolov on 2019-02-20 20:20:19 UTC --- It seems that glusterfs-gnfs is a depedency for vdsm-gluster without which I was not able to deploy ovirt-4.2.7 self-hosted (prep stage failed as method "gluster" was unknown): [root at ovirt2 ~]# yum --assumeno remove glusterfs-gnfs Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Repository centos-sclo-rh-release is listed more than once in the configuration Resolving Dependencies --> Running transaction check ---> Package glusterfs-gnfs.x86_64 0:3.12.15-1.el7 will be erased --> Processing Dependency: glusterfs-gnfs for package: vdsm-gluster-4.20.46-1.el7.x86_64 --> Running transaction check ---> Package vdsm-gluster.x86_64 0:4.20.46-1.el7 will be erased --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================= Removing: glusterfs-gnfs x86_64 3.12.15-1.el7 @ovirt-4.2-centos-gluster312 445 k Removing for dependencies: vdsm-gluster x86_64 4.20.46-1.el7 @ovirt-4.2 260 k Transaction Summary ======================================================================================================================================================= Remove 1 Package (+1 Dependent package) Installed size: 705 k Exiting on user command Your transaction was saved, rerun it with: yum load-transaction /tmp/yum_save_tx.2019-02-20.22-19.2e8Lcz.yumtx Uploading Enabled Repositories Report Loaded plugins: fastestmirror, product-id, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Repository centos-sclo-rh-release is listed more than once in the configuration Cannot upload enabled repos report, is this client registered? --- Additional comment from Strahil Nikolov on 2019-02-20 20:53:27 UTC --- I have removed glusterfs-gnfs manually and updated. The update process finished successfully, but the web interface shows unsynced entries present , while gluster volume heal engine info - shows nothing... --- Additional comment from Strahil Nikolov on 2019-02-20 20:55:41 UTC --- (In reply to Strahil Nikolov from comment #7) > I have removed glusterfs-gnfs manually and updated. > The update process finished successfully, but the web interface shows > unsynced entries present , while gluster volume heal engine info - shows > nothing... rpm -e --nodeps glusterfs-gnfs --- Additional comment from Jiffin on 2019-02-21 18:12:24 UTC --- # yum update glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm Loaded plugins: fastestmirror, ovl Examining glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-3.12.15-0.0.el7.x86_64 Examining glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-libs-3.12.15-0.0.el7.x86_64 Examining glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-api-3.12.15-0.0.el7.x86_64 Examining glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm: glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 Marking glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64.rpm as an update to glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 Resolving Dependencies --> Running transaction check ---> Package glusterfs.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be obsoleting ---> Package glusterfs-api.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-api.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update ---> Package glusterfs-client-xlators.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-client-xlators.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update ---> Package glusterfs-gnfs.x86_64 0:3.12.15-0.0.el7 will be obsoleted ---> Package glusterfs-libs.x86_64 0:3.12.15-0.0.el7 will be updated ---> Package glusterfs-libs.x86_64 0:4.1.7-0.3.git397ef0c.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: glusterfs x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 2.5 M replacing glusterfs-gnfs.x86_64 3.12.15-0.0.el7 Updating: glusterfs-api x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 197 k glusterfs-client-xlators x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 4.8 M glusterfs-libs x86_64 4.1.7-0.3.git397ef0c.el7 /glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1.3 M Transaction Summary =================================================================================================================================================================================================================== Install 1 Package Upgrade 3 Packages Total size: 8.9 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1/9 Installing : glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 2/9 Updating : glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 3/9 Updating : glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 4/9 Erasing : glusterfs-gnfs-3.12.15-0.0.el7.x86_64 5/9 Cleanup : glusterfs-api-3.12.15-0.0.el7.x86_64 6/9 Cleanup : glusterfs-3.12.15-0.0.el7.x86_64 7/9 Cleanup : glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 8/9 Cleanup : glusterfs-libs-3.12.15-0.0.el7.x86_64 9/9 Verifying : glusterfs-libs-4.1.7-0.3.git397ef0c.el7.x86_64 1/9 Verifying : glusterfs-api-4.1.7-0.3.git397ef0c.el7.x86_64 2/9 Verifying : glusterfs-4.1.7-0.3.git397ef0c.el7.x86_64 3/9 Verifying : glusterfs-client-xlators-4.1.7-0.3.git397ef0c.el7.x86_64 4/9 Verifying : glusterfs-gnfs-3.12.15-0.0.el7.x86_64 5/9 Verifying : glusterfs-libs-3.12.15-0.0.el7.x86_64 6/9 Verifying : glusterfs-3.12.15-0.0.el7.x86_64 7/9 Verifying : glusterfs-api-3.12.15-0.0.el7.x86_64 8/9 Verifying : glusterfs-client-xlators-3.12.15-0.0.el7.x86_64 9/9 Installed: glusterfs.x86_64 0:4.1.7-0.3.git397ef0c.el7 Updated: glusterfs-api.x86_64 0:4.1.7-0.3.git397ef0c.el7 glusterfs-client-xlators.x86_64 0:4.1.7-0.3.git397ef0c.el7 glusterfs-libs.x86_64 0:4.1.7-0.3.git397ef0c.el7 Replaced: glusterfs-gnfs.x86_64 0:3.12.15-0.0.el7 posted the patch https://review.gluster.org/#/c/glusterfs/+/22190/ to fix this --- Additional comment from Worker Ant on 2019-02-22 11:08:33 UTC --- REVIEW: https://review.gluster.org/22190 (packaging: Obsoleting glusterfs-gnfs for upgrade) merged (#4) on master by Niels de Vos Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 https://bugzilla.redhat.com/show_bug.cgi?id=1679965 [Bug 1679965] Upgrade from glusterfs 3.12 to gluster 4/5 broken -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 11:20:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 11:20:25 +0000 Subject: [Bugs] [Bug 1672711] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672711 Jiffin changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1679968 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679968 [Bug 1679968] Upgrade from glusterfs 3.12 to gluster 4/5 broken -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 11:20:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 11:20:25 +0000 Subject: [Bugs] [Bug 1679965] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679965 Jiffin changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1679968 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1679968 [Bug 1679968] Upgrade from glusterfs 3.12 to gluster 4/5 broken -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 11:24:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 11:24:49 +0000 Subject: [Bugs] [Bug 1679965] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679965 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22253 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 11:24:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 11:24:50 +0000 Subject: [Bugs] [Bug 1679965] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679965 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22253 (packaging: Obsoleting glusterfs-gnfs for upgrade) posted (#1) for review on release-6 by jiffin tony Thottan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 13:15:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 13:15:43 +0000 Subject: [Bugs] [Bug 1576814] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1576814 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|CURRENTRELEASE |--- Keywords| |Reopened --- Comment #4 from Shyamsundar --- Keeping this bug open, as changes that do not need a specific bug but are improvements are backported, and 4.1 release lifecycle is not over yet. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 13:15:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 13:15:44 +0000 Subject: [Bugs] [Bug 1576815] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1576815 Bug 1576815 depends on bug 1576814, which changed state. Bug 1576814 Summary: GlusterFS can be improved https://bugzilla.redhat.com/show_bug.cgi?id=1576814 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|CURRENTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 13:15:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 13:15:44 +0000 Subject: [Bugs] [Bug 1576816] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1576816 Bug 1576816 depends on bug 1576814, which changed state. Bug 1576814 Summary: GlusterFS can be improved https://bugzilla.redhat.com/show_bug.cgi?id=1576814 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|CURRENTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 13:19:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 13:19:51 +0000 Subject: [Bugs] [Bug 1679998] New: GlusterFS can be improved Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679998 Bug ID: 1679998 Summary: GlusterFS can be improved Product: GlusterFS Version: 6 Status: ASSIGNED Component: unclassified Keywords: Improvement, Tracking, Triaged Severity: high Assignee: bugs at gluster.org Reporter: srangana at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1193929 +++ I hope this bug is never fixed. The upstream patch process requires that each patch have an associated bug ID before it can be merged. However, there is no requirement that the bug contain any information or receive any kind of signoff before the patch can proceed. As a result, many of our developers have the habit of creating such "placeholder" bugs every time they want to make a change, even if it's just a random cleanup/idea and not an actual bug fix or requested/tracked feature request. Also, any patch with the dreaded "rfc" bug ID (which would be appropriate for such changes) is unlikely to be reviewed. This bug exists to satisfy our process requirement, without the additional negatives of clogging up our triage/tracking processes and making it appear that the code has more bugs (which would be bad) when in fact a developer had more ideas (which is generally good). -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 13:21:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 13:21:19 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 --- Comment #10 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22197 (fuse: reflect the actual default for lru-limit option) posted (#3) for review on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 13:21:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 13:21:22 +0000 Subject: [Bugs] [Bug 1679998] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679998 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22197 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:20:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:20:56 +0000 Subject: [Bugs] [Bug 1642810] remove glupy from code and build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1642810 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22256 (build: include gluster/__init__.* files for python packaging) posted (#1) for review on master by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:20:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:20:55 +0000 Subject: [Bugs] [Bug 1642810] remove glupy from code and build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1642810 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22256 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:35:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:35:24 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22257 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:35:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:35:25 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #563 from Worker Ant --- REVIEW: https://review.gluster.org/22257 (glusterfind: revert shebangs to #!/usr/bin/python3) posted (#1) for review on master by Kaleb KEITHLEY -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:44:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:44:08 +0000 Subject: [Bugs] [Bug 1679965] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679965 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-22 14:44:08 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22253 (packaging: Obsoleting glusterfs-gnfs for upgrade) merged (#1) on release-6 by jiffin tony Thottan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:46:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:46:20 +0000 Subject: [Bugs] [Bug 1680020] New: Integer Overflow possible in md-cache.c due to data type inconsistency Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680020 Bug ID: 1680020 Summary: Integer Overflow possible in md-cache.c due to data type inconsistency Product: GlusterFS Version: 6 Status: NEW Component: md-cache Keywords: Reopened Assignee: bugs at gluster.org Reporter: srangana at redhat.com CC: bugs at gluster.org, david.spisla at iternity.com, pasik at iki.fi Depends On: 1678726 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1678726 +++ Description of problem: The 'struct md_cache' in md-cache.c uses int data types which are not in common with the data types used in the 'struct iatt' in iatt.h . If one take a closer look to the implementations one can see that the struct in md-cache.c uses still the int data types like in the struct 'old_iatt' . This can lead to unexpected side effects and some values of iatt maybe will not mapped correctly. I would suggest to open a bug report. What do you think? Additional info: struct md_cache { ia_prot_t md_prot; uint32_t md_nlink; uint32_t md_uid; uint32_t md_gid; uint32_t md_atime; uint32_t md_atime_nsec; uint32_t md_mtime; uint32_t md_mtime_nsec; uint32_t md_ctime; uint32_t md_ctime_nsec; uint64_t md_rdev; uint64_t md_size; uint64_t md_blocks; uint64_t invalidation_time; uint64_t generation; dict_t *xattr; char *linkname; time_t ia_time; time_t xa_time; gf_boolean_t need_lookup; gf_boolean_t valid; gf_boolean_t gen_rollover; gf_boolean_t invalidation_rollover; gf_lock_t lock; }; struct iatt { uint64_t ia_flags; uint64_t ia_ino; /* inode number */ uint64_t ia_dev; /* backing device ID */ uint64_t ia_rdev; /* device ID (if special file) */ uint64_t ia_size; /* file size in bytes */ uint32_t ia_nlink; /* Link count */ uint32_t ia_uid; /* user ID of owner */ uint32_t ia_gid; /* group ID of owner */ uint32_t ia_blksize; /* blocksize for filesystem I/O */ uint64_t ia_blocks; /* number of 512B blocks allocated */ int64_t ia_atime; /* last access time */ int64_t ia_mtime; /* last modification time */ int64_t ia_ctime; /* last status change time */ int64_t ia_btime; /* creation time. Fill using statx */ uint32_t ia_atime_nsec; uint32_t ia_mtime_nsec; uint32_t ia_ctime_nsec; uint32_t ia_btime_nsec; uint64_t ia_attributes; /* chattr related:compressed, immutable, * append only, encrypted etc.*/ uint64_t ia_attributes_mask; /* Mask for the attributes */ uuid_t ia_gfid; ia_type_t ia_type; /* type of file */ ia_prot_t ia_prot; /* protection */ }; struct old_iatt { uint64_t ia_ino; /* inode number */ uuid_t ia_gfid; uint64_t ia_dev; /* backing device ID */ ia_type_t ia_type; /* type of file */ ia_prot_t ia_prot; /* protection */ uint32_t ia_nlink; /* Link count */ uint32_t ia_uid; /* user ID of owner */ uint32_t ia_gid; /* group ID of owner */ uint64_t ia_rdev; /* device ID (if special file) */ uint64_t ia_size; /* file size in bytes */ uint32_t ia_blksize; /* blocksize for filesystem I/O */ uint64_t ia_blocks; /* number of 512B blocks allocated */ uint32_t ia_atime; /* last access time */ uint32_t ia_atime_nsec; uint32_t ia_mtime; /* last modification time */ uint32_t ia_mtime_nsec; uint32_t ia_ctime; /* last status change time */ uint32_t ia_ctime_nsec; }; Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:46:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:46:20 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1680020 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1680020 [Bug 1680020] Integer Overflow possible in md-cache.c due to data type inconsistency -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:47:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:47:39 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 --- Comment #5 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22241 (md-cache: Adapt integer data types to avoid integer overflow) posted (#2) for review on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:47:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:47:41 +0000 Subject: [Bugs] [Bug 1680020] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680020 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22241 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:48:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:48:44 +0000 Subject: [Bugs] [Bug 1678726] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678726 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed|2019-02-20 12:14:08 |2019-02-22 14:48:44 --- Comment #6 from Worker Ant --- REVIEW: https://review.gluster.org/22240 (md-cache: Adapt integer data types to avoid integer overflow) merged (#1) on release-5 by David Spisla -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:48:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:48:45 +0000 Subject: [Bugs] [Bug 1680020] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680020 Bug 1680020 depends on bug 1678726, which changed state. Bug 1678726 Summary: Integer Overflow possible in md-cache.c due to data type inconsistency https://bugzilla.redhat.com/show_bug.cgi?id=1678726 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:49:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:49:43 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-22 14:49:43 --- Comment #29 from Worker Ant --- REVIEW: https://review.gluster.org/22229 (performance/write-behind: handle call-stub leaks) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 14:49:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:49:44 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1671556, which changed state. Bug 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1671556 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:49:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:49:44 +0000 Subject: [Bugs] [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 Bug 1676356 depends on bug 1671556, which changed state. Bug 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1671556 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:49:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:49:45 +0000 Subject: [Bugs] [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 Bug 1678570 depends on bug 1671556, which changed state. Bug 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1671556 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:49:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:49:45 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Bug 1674406 depends on bug 1671556, which changed state. Bug 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1671556 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:50:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:50:30 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #30 from Worker Ant --- REVIEW: https://review.gluster.org/22228 (performance/write-behind: fix use-after-free in readdirp) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Feb 22 14:50:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:50:31 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1671556, which changed state. Bug 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1671556 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:50:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:50:31 +0000 Subject: [Bugs] [Bug 1676356] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676356 Bug 1676356 depends on bug 1671556, which changed state. Bug 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1671556 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:50:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:50:32 +0000 Subject: [Bugs] [Bug 1678570] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678570 Bug 1678570 depends on bug 1671556, which changed state. Bug 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1671556 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 14:50:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 14:50:32 +0000 Subject: [Bugs] [Bug 1674406] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1674406 Bug 1674406 depends on bug 1671556, which changed state. Bug 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' https://bugzilla.redhat.com/show_bug.cgi?id=1671556 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 15:21:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 15:21:58 +0000 Subject: [Bugs] [Bug 1679968] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679968 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22258 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Feb 22 15:21:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 22 Feb 2019 15:21:59 +0000 Subject: [Bugs] [Bug 1679968] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679968 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22258 (packaging: Obsoleting glusterfs-gnfs for upgrade) posted (#1) for review on release-5 by Kaleb KEITHLEY -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Feb 23 19:30:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 23 Feb 2019 19:30:32 +0000 Subject: [Bugs] [Bug 1589695] Provide a cli cmd to modify max-file-size In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1589695 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 20212 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Feb 23 21:31:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 23 Feb 2019 21:31:36 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Endre Karlson changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |endre.karlson at gmail.com --- Comment #42 from Endre Karlson --- Any news on a patched version for ovirt 4.3 ? We keep seeing crashes like these too.. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sun Feb 24 11:25:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 24 Feb 2019 11:25:55 +0000 Subject: [Bugs] [Bug 1677555] Glusterfs brick is crashed due to segfault caused by broken gfid symlink In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677555 --- Comment #2 from loman --- Hi Amar Tumballi, I didn't check it on release-6.0, only on stable 4.1.7. Also i found the root cause, this issue is happening if quota is enabled and reproduced all time on 4.1.7. After quota disable brick is back online without any tricks with removing broken symlinks. Please try to enable quota, also i'll try to reproduce it on release-6.0 from my side as well. Thanks! -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 04:14:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 04:14:04 +0000 Subject: [Bugs] [Bug 1670334] Some memory leaks found in GlusterFS 5.3 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670334 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nbalacha at redhat.com Component|core |glusterd Assignee|bugs at gluster.org |amukherj at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 05:18:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 05:18:35 +0000 Subject: [Bugs] [Bug 1642810] remove glupy from code and build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1642810 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22256 (build: include gluster/__init__.* files for python packaging) merged (#3) on master by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 21 05:30:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 21 Feb 2019 05:30:08 +0000 Subject: [Bugs] [Bug 1679401] Geo-rep setup creates an incorrectly formatted authorized_keys file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679401 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-25 05:21:50 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22246 (geo-rep : fix incorrectly formatted authorized_keys) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 07:06:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 07:06:47 +0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 --- Comment #62 from Worker Ant --- REVIEW: https://review.gluster.org/22239 (fuse : fix memory leak) merged (#4) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 09:33:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 09:33:11 +0000 Subject: [Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1677160 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |srakonde at redhat.com Flags| |needinfo?(srakonde at redhat.c | |om) --- Comment #11 from Sahina Bose --- (In reply to Sanju from comment #10) > We tried to reproduce this issue but couldn't hit it. If you happen to hit > this issue, please provide us all the log files from /var/log/glusterfs (for > both glusterfs-server and client from all the machines). > > Thanks, > Sanju Sanju, please look at Bug 1672318 as well. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 10:22:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 10:22:17 +0000 Subject: [Bugs] [Bug 1314508] tar complains: : file changed as we read it In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1314508 Yaniv Kaul changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |abhaumik at redhat.com Flags| |needinfo?(abhaumik at redhat.c | |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 10:25:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 10:25:13 +0000 Subject: [Bugs] [Bug 1660225] geo-rep does not replicate mv or rename of file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660225 Sunny Kumar changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(sunkumar at redhat.c | |om) | --- Comment #8 from Sunny Kumar --- Hi, Looks like https://review.gluster.org/#/c/glusterfs/+/20093/. But I am trying for reproducer to analyse more. If something is missing in step to reproduce please add. - Sunny -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 11:05:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:05:48 +0000 Subject: [Bugs] [Bug 1635688] Keep only the valid (maintained/supported) components in the build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1635688 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22259 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:05:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:05:49 +0000 Subject: [Bugs] [Bug 1635688] Keep only the valid (maintained/supported) components in the build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1635688 --- Comment #19 from Worker Ant --- REVIEW: https://review.gluster.org/22259 (build: include gluster/__init__.* files for python packaging) posted (#1) for review on release-6 by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:15:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:15:37 +0000 Subject: [Bugs] [Bug 1680585] New: remove glupy from code and build Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680585 Bug ID: 1680585 Summary: remove glupy from code and build Product: GlusterFS Version: 6 Status: NEW Component: build Assignee: bugs at gluster.org Reporter: ndevos at redhat.com CC: bugs at gluster.org Depends On: 1642810 Blocks: 1672818 (glusterfs-6.0) Target Milestone: --- Classification: Community This bug was initially created as a copy of Bug #1642810 I am copying this bug because: The glupy removal removed too much. With this bug we'll add the pieces that should not have been removed. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1642810 [Bug 1642810] remove glupy from code and build https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:15:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:15:37 +0000 Subject: [Bugs] [Bug 1642810] remove glupy from code and build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1642810 Niels de Vos changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1680585 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1680585 [Bug 1680585] remove glupy from code and build -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:15:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:15:37 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Niels de Vos changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1680585 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1680585 [Bug 1680585] remove glupy from code and build -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:17:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:17:48 +0000 Subject: [Bugs] [Bug 1635688] Keep only the valid (maintained/supported) components in the build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1635688 --- Comment #20 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22259 (build: include gluster/__init__.* files for python packaging) posted (#2) for review on release-6 by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:17:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:17:49 +0000 Subject: [Bugs] [Bug 1635688] Keep only the valid (maintained/supported) components in the build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1635688 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Gluster.org Gerrit 22259 | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:17:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:17:50 +0000 Subject: [Bugs] [Bug 1680585] remove glupy from code and build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680585 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22259 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:17:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:17:51 +0000 Subject: [Bugs] [Bug 1680585] remove glupy from code and build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680585 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22259 (build: include gluster/__init__.* files for python packaging) posted (#2) for review on release-6 by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:31:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:31:12 +0000 Subject: [Bugs] [Bug 1680586] New: Building RPM packages with _for_fedora_koji_builds enabled fails on el6 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680586 Bug ID: 1680586 Summary: Building RPM packages with _for_fedora_koji_builds enabled fails on el6 Product: GlusterFS Version: 6 Status: ASSIGNED Component: build Assignee: ndevos at redhat.com Reporter: ndevos at redhat.com CC: bugs at gluster.org Blocks: 1672818 (glusterfs-6.0) Target Milestone: --- Classification: Community Description of problem: Building RPM packages with _for_fedora_koji_builds enabled fails on el6. A mock rebuild fails with these errors: Checking for unpackaged file(s): /usr/lib/rpm/check-files /builddir/build/BUILDROOT/glusterfs-6.0-0.1.rc0.el6.x86_64 BUILDSTDERR: error: Installed (but unpackaged) file(s) found: BUILDSTDERR: /etc/init.d/glusterfsd BUILDSTDERR: /etc/sysconfig/glusterd BUILDSTDERR: /etc/sysconfig/glusterfsd BUILDSTDERR: Installed (but unpackaged) file(s) found: BUILDSTDERR: /etc/init.d/glusterfsd BUILDSTDERR: /etc/sysconfig/glusterd BUILDSTDERR: /etc/sysconfig/glusterfsd Version-Release number of selected component (if applicable): glusterfs-6.0 RC0 How reproducible: 100% Steps to Reproduce: 1. set _for_fedora_koji_builds to 1 in the glusterfs.spec.in 2. build RPMs against/on el6 Actual results: Building RPMs fails Expected results: Building RPMs should succeed. Additional info: It seems that glusterfs-6 disables building the server (./configure --without-server) and hence the installed but unpackaged files should not get installed in the first place. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 11:31:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:31:12 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Niels de Vos changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1680586 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1680586 [Bug 1680586] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 11:32:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:32:13 +0000 Subject: [Bugs] [Bug 1680587] New: Building RPM packages with _for_fedora_koji_builds enabled fails on el6 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680587 Bug ID: 1680587 Summary: Building RPM packages with _for_fedora_koji_builds enabled fails on el6 Product: GlusterFS Version: mainline Status: ASSIGNED Component: build Assignee: ndevos at redhat.com Reporter: ndevos at redhat.com CC: bugs at gluster.org Blocks: 1680586 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1680586 +++ Description of problem: Building RPM packages with _for_fedora_koji_builds enabled fails on el6. A mock rebuild fails with these errors: Checking for unpackaged file(s): /usr/lib/rpm/check-files /builddir/build/BUILDROOT/glusterfs-6.0-0.1.rc0.el6.x86_64 BUILDSTDERR: error: Installed (but unpackaged) file(s) found: BUILDSTDERR: /etc/init.d/glusterfsd BUILDSTDERR: /etc/sysconfig/glusterd BUILDSTDERR: /etc/sysconfig/glusterfsd BUILDSTDERR: Installed (but unpackaged) file(s) found: BUILDSTDERR: /etc/init.d/glusterfsd BUILDSTDERR: /etc/sysconfig/glusterd BUILDSTDERR: /etc/sysconfig/glusterfsd Version-Release number of selected component (if applicable): glusterfs-6.0 RC0 How reproducible: 100% Steps to Reproduce: 1. set _for_fedora_koji_builds to 1 in the glusterfs.spec.in 2. build RPMs against/on el6 Actual results: Building RPMs fails Expected results: Building RPMs should succeed. Additional info: It seems that glusterfs-6 disables building the server (./configure --without-server) and hence the installed but unpackaged files should not get installed in the first place. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1680586 [Bug 1680586] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 11:32:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:32:13 +0000 Subject: [Bugs] [Bug 1680586] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680586 Niels de Vos changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1680587 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1680587 [Bug 1680587] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 11:36:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:36:06 +0000 Subject: [Bugs] [Bug 1680587] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680587 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22260 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 11:36:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:36:07 +0000 Subject: [Bugs] [Bug 1680587] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680587 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22260 (build: do not install service related files when building --without-server) posted (#1) for review on master by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 11:52:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 11:52:22 +0000 Subject: [Bugs] [Bug 1664590] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Anjana Suparna Sriram changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1680596 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1680596 [Bug 1680596] [Doc RFE] Document how to set up a cluster (server) using IPv6 networking exclusively -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 13:42:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 13:42:45 +0000 Subject: [Bugs] [Bug 1673058] Network throughput usage increased x5 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1673058 Jacob changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jsecchiero at enter. | |eu) | --- Comment #4 from Jacob --- i'm not able to upload in the bugzilla portal due to the size of the pcap. You can download from here: https://mega.nz/#!FNY3CS6A!70RpciIzDgNWGwbvEwH-_b88t9e1QVOXyLoN09CG418 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 15:17:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:17:07 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22198 (fuse: reflect the actual default for lru-limit option) merged (#3) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 15:23:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:23:43 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed|2019-02-18 02:46:12 |2019-02-25 15:23:43 --- Comment #43 from Worker Ant --- REVIEW: https://review.gluster.org/22237 (socket: socket event handlers now return void) merged (#4) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Feb 25 15:23:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:23:44 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1651246, which changed state. Bug 1651246 Summary: Failed to dispatch handler https://bugzilla.redhat.com/show_bug.cgi?id=1651246 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 15:24:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:24:06 +0000 Subject: [Bugs] [Bug 1679968] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679968 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-25 15:24:06 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22258 (packaging: Obsoleting glusterfs-gnfs for upgrade) merged (#2) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 15:24:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:24:07 +0000 Subject: [Bugs] [Bug 1679965] Upgrade from glusterfs 3.12 to gluster 4/5 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679965 Bug 1679965 depends on bug 1679968, which changed state. Bug 1679968 Summary: Upgrade from glusterfs 3.12 to gluster 4/5 broken https://bugzilla.redhat.com/show_bug.cgi?id=1679968 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 15:26:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:26:13 +0000 Subject: [Bugs] [Bug 1680585] remove glupy from code and build In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680585 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22259 (build: include gluster/__init__.* files for python packaging) merged (#2) on release-6 by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 15:26:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:26:35 +0000 Subject: [Bugs] [Bug 1680020] Integer Overflow possible in md-cache.c due to data type inconsistency In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680020 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-25 15:26:35 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22241 (md-cache: Adapt integer data types to avoid integer overflow) merged (#3) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 15:26:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:26:56 +0000 Subject: [Bugs] [Bug 1679998] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679998 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22197 (fuse: reflect the actual default for lru-limit option) merged (#4) on release-6 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 15:31:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 15:31:03 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1679406, which changed state. Bug 1679406 Summary: glustereventsd does not start on Ubuntu 16.04 LTS https://bugzilla.redhat.com/show_bug.cgi?id=1679406 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 16:06:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 16:06:40 +0000 Subject: [Bugs] [Bug 1678640] Running 'control-cpu-load.sh' prevents CTDB starting In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1678640 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amukherj at redhat.com Component|glusterd |core Assignee|bugs at gluster.org |moagrawa at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 16:08:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 16:08:23 +0000 Subject: [Bugs] [Bug 1679744] Minio gateway nas does not work with 2 + 1 dispersed volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679744 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amukherj at redhat.com Component|glusterd |core -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 17:08:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 17:08:47 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #50 from Raghavendra Bhat --- Hi, Thanks for the logs. From the logs saw that the following things are happening. 1) The scrubbing is started 2) Scrubber always decides whether a file is corrupted or not by comparing the stored on-disk signature (gets by getxattr) with its own calculated signature of the file. 3) Here, while getting the on-disk signature, getxattr is failing with ENOMEM (i.e. Cannot allocate memory) because of the endianness. 4) Further testcases in the test fail because, they expect the bad-file extended attribute to be present which scrubber could not set because of the above error (i.e. had it been able to successfully get the signature of the file via getxattr, it would have been able to compare the signature with its own calculated signature and set the bad-file extended attribute to indicate the file is corrupted). Looking at the code to come up with a fix to address this. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 18:14:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 18:14:08 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #564 from Worker Ant --- REVIEW: https://review.gluster.org/22222 (leases-internal.c: minor reduction of work under lock.) merged (#3) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Feb 25 18:14:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 25 Feb 2019 18:14:08 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #565 from Worker Ant --- REVIEW: https://review.gluster.org/22257 (glusterfind: revert shebangs to #!/usr/bin/python3) merged (#4) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 04:51:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 04:51:29 +0000 Subject: [Bugs] [Bug 1676812] Manual Index heal throws error which is misguiding when heal is triggered to heal a brick if another brick is down In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1676812 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |WONTFIX Last Closed| |2019-02-26 04:51:29 --- Comment #3 from Atin Mukherjee --- Based on the discussion on the patch, we decide not to fix this. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 05:13:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 05:13:26 +0000 Subject: [Bugs] [Bug 1683008] New: glustereventsd does not start on Ubuntu 16.04 LTS Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683008 Bug ID: 1683008 Summary: glustereventsd does not start on Ubuntu 16.04 LTS Product: GlusterFS Version: 6 Hardware: x86_64 OS: Linux Status: NEW Component: eventsapi Severity: high Priority: high Assignee: bugs at gluster.org Reporter: avishwan at redhat.com CC: avishwan at redhat.com, beat.joerg at gmail.com, bugzilla at ii.nl, captcha.is.evil at gmail.com, dchang at silverlakecorp.com, pasik at iki.fi, sabose at redhat.com Depends On: 1649054, 1679406 Blocks: 1667103 (glusterfs-5.4), 1677319 (Gluster_5_Affecting_oVirt_4.3) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1679406 +++ +++ This bug was initially created as a clone of Bug #1649054 +++ Description of problem: glustereventsd does not start due to an import error in utils.py Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/sbin/glustereventsd", line 24, in Nov 12 20:21:04 s002 glustereventsd[28140]: import handlers Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/lib/x86_64-linux-gnu/glusterfs/events/handlers.py", line 12, in Nov 12 20:21:04 s002 glustereventsd[28140]: import utils Nov 12 20:21:04 s002 glustereventsd[28140]: File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in Nov 12 20:21:04 s002 glustereventsd[28140]: from .eventsapiconf import (LOG_FILE, Nov 12 20:21:04 s002 glustereventsd[28140]: SystemError: Parent module '' not loaded, cannot perform relative import Installed Pyython version is 3.5.2 --- Additional comment from Mrten on 2018-12-04 17:12:09 UTC --- same as Bug 1650017 --- Additional comment from Mrten on 2018-12-04 17:25:31 UTC --- This is a fix for the first bug: --- utils.py.old 2018-12-04 18:19:18.846463181 +0100 +++ utils.py 2018-12-04 18:21:09.830149998 +0100 @@ -26,13 +26,13 @@ from hashlib import sha256 from calendar import timegm -from .eventsapiconf import (LOG_FILE, +from eventsapiconf import (LOG_FILE, WEBHOOKS_FILE, DEFAULT_CONFIG_FILE, CUSTOM_CONFIG_FILE, UUID_FILE, CERTS_DIR) -from . import eventtypes +import eventtypes and you need this one as well: --- glustereventsd.py.old 2018-12-04 18:25:06.841481182 +0100 +++ glustereventsd.py 2018-12-04 18:22:51.529863014 +0100 @@ -28,7 +28,7 @@ from utils import logger, PidFile, PidFileLockFailed, boolify -class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): +class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): def handle(self): data = self.request[0].strip() @@ -98,7 +98,7 @@ # Start the Eventing Server, UDP Server try: - server = SocketServer.ThreadingUDPServer( + server = socketserver.ThreadingUDPServer( (SERVER_ADDRESS, port), GlusterEventsRequestHandler) except socket.error as e: --- Additional comment from Mrten on 2018-12-04 17:28:51 UTC --- but you still have "glustereventsd[20980]: Unable to get Port details from Config" in the log :( --- Additional comment from Mrten on 2018-12-04 17:36:28 UTC --- default config file does not exist: root at gluster-3:# less /etc/glusterfs/eventsconfig.json /etc/glusterfs/eventsconfig.json: No such file or directory --- Additional comment from Mrten on 2018-12-04 17:38:39 UTC --- inserting this: https://github.com/gluster/glusterfs/blob/master/events/src/eventsconfig.json into /etc/glusterfs/eventsconfig.json makes it start without problems --- Additional comment from Mrten on 2018-12-05 09:24:13 UTC --- haha, but now geo-replication is broken [2018-12-05 08:52:49.556853] E [syncdutils(monitor):338:log_raise_exception] : FAIL: Traceback (most recent call last): File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 368, in twrap tf(*aargs) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 339, in wmon slave_host, master, suuid, slavenodes) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line 325, in monitor brick_path=w[0]['dir']) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 602, in gf_event from events.gf_event import gf_event as gfevent File "/usr/lib/x86_64-linux-gnu/glusterfs/events/gf_event.py", line 19, in from .utils import logger, setup_logger, get_config File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in from eventsapiconf import (LOG_FILE, ImportError: No module named 'eventsapiconf' --- Additional comment from Mrten on 2018-12-05 09:31:22 UTC --- better patch: --- utils.py.old 2018-12-04 18:19:18.846463181 +0100 +++ utils.py 2018-12-05 10:30:03.306848363 +0100 @@ -26,13 +26,25 @@ from hashlib import sha256 from calendar import timegm -from .eventsapiconf import (LOG_FILE, - WEBHOOKS_FILE, - DEFAULT_CONFIG_FILE, - CUSTOM_CONFIG_FILE, - UUID_FILE, - CERTS_DIR) -from . import eventtypes +try: + from .eventsapiconf import (LOG_FILE, + WEBHOOKS_FILE, + DEFAULT_CONFIG_FILE, + CUSTOM_CONFIG_FILE, + UUID_FILE, + CERTS_DIR) +except ImportError: + from eventsapiconf import (LOG_FILE, + WEBHOOKS_FILE, + DEFAULT_CONFIG_FILE, + CUSTOM_CONFIG_FILE, + UUID_FILE, + CERTS_DIR) + +try: + from . import eventtypes +except ImportError: + import eventtypes --- Additional comment from dchang0 on 2018-12-12 08:06:26 UTC --- Hi, Mrten-- I still have the exact same bug AFTER applying the patches you provided (the patch in Comment 2 for glustereventsd.py and the patch in Comment 7 for utils.py). I'm running Raspbian Stretch Lite (not Ubuntu 16.04 like the original bug reporter) on a Raspberry Pi Zero W and built glusterfs 5.1 from source code. The patches appear to have been applied correctly but attempting to start glustereventsd still gets this error: Dec 11 16:48:14 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'. Dec 11 16:50:53 datastore2c systemd[1]: Starting LSB: Gluster Events Server... -- Subject: Unit glustereventsd.service has begun start-up -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit glustereventsd.service has begun starting up. Dec 11 16:50:55 datastore2c glustereventsd[710]: Starting glustereventsd service: glustereventsdTraceback (most recent call last): Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/sbin/glustereventsd", line 24, in Dec 11 16:50:55 datastore2c glustereventsd[710]: import handlers Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in Dec 11 16:50:55 datastore2c glustereventsd[710]: import utils Dec 11 16:50:55 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in Dec 11 16:50:55 datastore2c glustereventsd[710]: from .eventsapiconf import (LOG_FILE, Dec 11 16:50:55 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import Dec 11 16:50:55 datastore2c glustereventsd[710]: failed! Dec 11 16:50:56 datastore2c glustereventsd[710]: Traceback (most recent call last): Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/sbin/glustereventsd", line 24, in Dec 11 16:50:56 datastore2c glustereventsd[710]: import handlers Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/handlers.py", line 12, in Dec 11 16:50:56 datastore2c glustereventsd[710]: import utils Dec 11 16:50:56 datastore2c glustereventsd[710]: File "/usr/local/libexec/glusterfs/events/utils.py", line 30, in Dec 11 16:50:56 datastore2c glustereventsd[710]: from .eventsapiconf import (LOG_FILE, Dec 11 16:50:56 datastore2c glustereventsd[710]: SystemError: Parent module '' not loaded, cannot perform relative import Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Control process exited, code=exited status=1 Dec 11 16:50:56 datastore2c systemd[1]: Failed to start LSB: Gluster Events Server. -- Subject: Unit glustereventsd.service has failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit glustereventsd.service has failed. -- -- The result is failed. Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Unit entered failed state. Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Failed with result 'exit-code'. I can provide whatever info you need to help debug this. Thanks! --- Additional comment from Arano-kai on 2018-12-28 01:27 UTC --- Hi dchang0. According to your logs, patch is not apply. Here the compiled from Mrten's answers patch that you need apply with following cmd: `patch -p 0 -d [libdir] < glustereventsd.patch` Where [libdir]: - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/" - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs) Also, there is missing file that you can get by `wget 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/eventsconfig.json' -O /etc/glusterfs/eventsconfig.json` --- Additional comment from dchang0 on 2018-12-28 01:44:44 UTC --- (In reply to Arano-kai from comment #9) > Created attachment 1517163 [details] > glustereventsd.py and utils.py fixup > > Hi dchang0. > According to your logs, patch is not apply. > Here the compiled from Mrten's answers patch that you need apply with > following cmd: > `patch -p 0 -d [libdir] < glustereventsd.patch` > Where [libdir]: > - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/" > - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs) > Also, there is missing file that you can get by `wget > 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/ > eventsconfig.json' -O /etc/glusterfs/eventsconfig.json` Thanks. I will check the files in question manually and reapply them if needed and report back. eventsconfig.json was not missing for me (I built completely from scratch, and that file was included and installed in the proper location). --- Additional comment from Arano-kai on 2018-12-28 01:49 UTC --- Woops, forget to add 'SystemError' to catcher. Here the fixed version. --- Additional comment from dchang0 on 2018-12-28 04:14:37 UTC --- This doesn't matter now, since you've changed at least the glustereventsd.py patch, but I'm posting these anyway for diagnostic purposes. The source code was built in /root/glusterfs-5.1. The install with the patches applied is in /usr/local... ----- root at datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py 31c31 < class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): --- > class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): 101c101 < server = SocketServer.ThreadingUDPServer( --- > server = socketserver.ThreadingUDPServer( ----- root at datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py 29,35c29,42 < from .eventsapiconf import (LOG_FILE, < WEBHOOKS_FILE, < DEFAULT_CONFIG_FILE, < CUSTOM_CONFIG_FILE, < UUID_FILE, < CERTS_DIR) < from . import eventtypes --- > try: > from .eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > except ImportError: > from eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) 36a44,47 > try: > from . import eventtypes > except ImportError: > import eventtypes ----- So it looks like I did apply the patches before, but they were probably bad/broken patches. Anyway, I will replace the patched files with the original source, then apply your new patches and see what happens and report back. --- Additional comment from dchang0 on 2018-12-28 04:52:57 UTC --- Okay, I just put the original files from source back (using make install) and then applied your new patches from Attachment #1517165. I'm getting a different error (probably an easier one to solve). Strangely, the patch command does not create the .orig files. Perhaps my patch command is broken or non-standard... Here are the diffs: ----- root at datastore2c:/home/pi# diff /root/glusterfs-5.1/events/src/glustereventsd.py /usr/local/libexec/glusterfs/events/glustereventsd.py 31c31 < class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler): --- > class GlusterEventsRequestHandler(socketserver.BaseRequestHandler): 101c101 < server = SocketServer.ThreadingUDPServer( --- > server = socketserver.ThreadingUDPServer( ------ root at datastore2c:/usr/local/libexec# diff /root/glusterfs-5.1/events/src/utils.py /usr/local/libexec/glusterfs/events/utils.py 29,35c29,46 < from .eventsapiconf import (LOG_FILE, < WEBHOOKS_FILE, < DEFAULT_CONFIG_FILE, < CUSTOM_CONFIG_FILE, < UUID_FILE, < CERTS_DIR) < from . import eventtypes --- > try: > from .eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > except (ImportError, SystemError): > from eventsapiconf import (LOG_FILE, > WEBHOOKS_FILE, > DEFAULT_CONFIG_FILE, > CUSTOM_CONFIG_FILE, > UUID_FILE, > CERTS_DIR) > try: > from . import eventtypes > except (ImportError, SystemError): > import eventtypes ----- Here is the new error when attempting to run: service glustereventsd start It appears to be a simple path error (can't find the gluster command). root at datastore2c:/usr/local/libexec# journalctl -xe Dec 27 20:46:15 datastore2c glustereventsd[14033]: import utils Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/local/libexec/glusterfs/events/utils.py", line 37, in Dec 27 20:46:15 datastore2c glustereventsd[14033]: from eventsapiconf import (LOG_FILE, Dec 27 20:46:15 datastore2c glustereventsd[14033]: File "/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 33, in # systemctl edit glustereventsd.service [Service] Environment=PATH= --- Additional comment from Arano-kai on 2018-12-28 11:18:07 UTC --- As for Comment 12, patch is right and applied. That is not right is python version: # python3 -V Python 3.5.3 # /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid ... >>>SystemError<<<: Parent module '' not loaded, cannot perform relative import # python3 -V Python 3.7.2rc1 # /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid ... >>>ImportError<<<: attempted relative import with no known parent package So we need catch SystemError for older python. --- Additional comment from dchang0 on 2018-12-29 05:14:01 UTC --- (In reply to Arano-kai from comment #15) > It is probably you hit by https://stackoverflow.com/q/5658622, that is some > $PATH inconsistency from python view and shell view. > Since systemd used and units run in isolated env, you can try to explicitly > set $PATH by overriding glustereventsd.service: > # echo $PATH > > # systemctl edit glustereventsd.service > > [Service] > Environment=PATH= Thanks for the help! I ended up editing the PATH= line in the file /etc/init.d/glustereventsd It seems to work now (starts without errors). I have not tested the actual functionality yet but will do so soon enough. I would say that with the patches you provided in attachment 1517165, this bug is fixed completely for me. Thanks again! --- Additional comment from Sahina Bose on 2019-02-14 07:10:46 UTC --- Aravinda, we have multiple oVirt users hitting this issue. Can this be targeted to a 5.x release? --- Additional comment from Worker Ant on 2019-02-21 06:10:25 UTC --- REVIEW: https://review.gluster.org/22247 (eventsapi: Fix Python3 compatibility issues) posted (#1) for review on master by Aravinda VK --- Additional comment from Worker Ant on 2019-02-25 15:31:03 UTC --- REVIEW: https://review.gluster.org/22247 (eventsapi: Fix Python3 compatibility issues) merged (#5) on master by Amar Tumballi Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1649054 [Bug 1649054] glustereventsd does not start on Ubuntu 16.04 LTS https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 https://bugzilla.redhat.com/show_bug.cgi?id=1679406 [Bug 1679406] glustereventsd does not start on Ubuntu 16.04 LTS -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 05:13:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 05:13:26 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Aravinda VK changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1683008 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683008 [Bug 1683008] glustereventsd does not start on Ubuntu 16.04 LTS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 05:13:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 05:13:45 +0000 Subject: [Bugs] [Bug 1683008] glustereventsd does not start on Ubuntu 16.04 LTS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683008 Aravinda VK changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |avishwan at redhat.com -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 07:19:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 07:19:26 +0000 Subject: [Bugs] [Bug 1680587] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680587 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-26 07:19:26 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22260 (build: do not install service related files when building --without-server) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 07:19:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 07:19:27 +0000 Subject: [Bugs] [Bug 1680586] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680586 Bug 1680586 depends on bug 1680587, which changed state. Bug 1680587 Summary: Building RPM packages with _for_fedora_koji_builds enabled fails on el6 https://bugzilla.redhat.com/show_bug.cgi?id=1680587 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 08:39:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 08:39:58 +0000 Subject: [Bugs] [Bug 1680586] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680586 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22264 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 08:39:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 08:39:59 +0000 Subject: [Bugs] [Bug 1680586] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680586 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22264 (build: do not install service related files when building --without-server) posted (#1) for review on release-6 by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 11:09:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 11:09:36 +0000 Subject: [Bugs] [Bug 1679744] Minio gateway nas does not work with 2 + 1 dispersed volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679744 Otavio Cipriani changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Github | |minio/minio/issues/7262 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 11:09:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 11:09:49 +0000 Subject: [Bugs] [Bug 1679744] Minio gateway nas does not work with 2 + 1 dispersed volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1679744 Otavio Cipriani changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Github 7262 | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 11:45:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 11:45:35 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Sayalee changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ON_QA |VERIFIED --- Comment #6 from Sayalee --- I have performed the following steps (as mentioned in #Description) and did NOT hit the issue on 3.12.2-45 build: 1) Create a pure distribute volume with two bricks and start it. 2) FUSE mount the volume on a client node using: # mount -t glusterfs -s :/ /mnt/ 3) On the mount point do following operations: # cd /mnt/ # mkdir -p dir0/dir1/dir2 # cd dir0/dir1/dir2 # for i in {1..100}; do echo "Test file" > tfile-$i; done # for i in {1..100}; do mv tfile-$i ntfile-$i; done 4) Start remove-brick operation on the volume using: # gluster v remove-brick brick2 start 5) Check the remove brick status using: # gluster v remove-brick brick2 status 6) When the remove-brick status is shown as "complete" for all the nodes, stop the remove-brick operation using: # gluster v remove-brick brick2 stop 7) Go to brick2 and perform lookup on the files. (only linkto files must be present and all should point to "vol_name-client-0") 8) Now change the linkto xattr value for every file in brick2 to point to itself (in this case "vol_name-client-1", also make sure the string is null terminated) # setfattr -n trusted.glusterfs.dht.linkto -v 0x74766f6c2d636c69656e742d3100 /bricks/brick2//dir0/dir1/dir2/ntfile-* 9) Now from the mount-point perform following operations in order to delete the directory: # cd /mnt/ # rm -rf * Also covered the regression cases as mentioned in the test plan shared in #comment5 Hence moving this bug to verified. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 12:19:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 12:19:45 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nbalacha at redhat.c |needinfo?(saraut at redhat.com |om) |) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 12:40:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 12:40:15 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22266 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 12:40:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 12:40:16 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #11 from Worker Ant --- REVIEW: https://review.gluster.org/22266 (rpc/transport: Mixing a ref on dictionary while assigning to transport object) posted (#1) for review on master by mohammed rafi kc -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 13:31:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 13:31:31 +0000 Subject: [Bugs] [Bug 1668286] READDIRP incorrectly updates posix-acl inode ctx In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 --- Comment #10 from homma at allworks.co.jp --- (In reply to Jiffin from comment #9) > As far as I understand, rda_fill_fd_cbk() sets iatt to zero and that info is > stored in its context not passed to the other layers. > I tried to reproduce, but was not able to hit till now.(turned on > performance.readdir-ahead). > Prior to this bug myself have seen similar issue when, the permission of acl > ctx gets zeroed after readdir operations. The issue was > very much spurious and there was no specific steps to hit that issue I think rda_fill_fd_cbk() passes entries with zeroed iatts to other xlators. On entry of rda_fill_fd_cbk(), 'entries' holds dentries obtained by READDIRP operation. After setting iatt to zero, it calls STACK_UNWIND_STRICT with modified 'serve_entries', not the original 'entries'. Then posix_acl_readdirp_cbk() receives that modified entries information. (gdb) b rda_fill_fd_cbk Breakpoint 1 at 0x7fef2451f9d0: file readdir-ahead.c, line 424. (gdb) b readdir-ahead.c:537 b posix_acl_readdirp_cbk Breakpoint 2 at 0x7fef2451fcd9: file readdir-ahead.c, line 537. (gdb) b posix_acl_readdirp_cbk Breakpoint 3 at 0x7fef1f7990b0: file posix-acl.c, line 1654. (gdb) c Continuing. [Switching to Thread 0x7fef25b37700 (LWP 12060)] Breakpoint 1, rda_fill_fd_cbk (frame=frame at entry=0x7fef2005c628, cookie=0x7fef2006a4d8, this=0x7fef200132e0, op_ret=op_ret at entry=4, op_errno=op_errno at entry=2, entries=entries at entry=0x7fef25b36710, xdata=xdata at entry=0x0) at readdir-ahead.c:424 424 { (gdb) p *entries.next.next.next $1 = {{list = {next = 0x7fef200011a0, prev = 0x7fef20000f40}, {next = 0x7fef200011a0, prev = 0x7fef20000f40}}, d_ino = 10966013112435171471, d_off = 28, d_len = 3, d_type = 8, d_stat = {ia_flags = 6143, ia_ino = 10966013112435171471, ia_dev = 51792, ia_rdev = 0, ia_size = 5, ia_nlink = 1, ia_uid = 1000, ia_gid = 1000, ia_blksize = 4096, ia_blocks = 1, ia_atime = 1551186297, ia_mtime = 1551186488, ia_ctime = 1551186488, ia_btime = 0, ia_atime_nsec = 517274116, ia_mtime_nsec = 150035482, ia_ctime_nsec = 153035462, ia_btime_nsec = 0, ia_attributes = 0, ia_attributes_mask = 0, ia_gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", ia_type = IA_IFREG, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 1 '\001', write = 1 '\001', exec = 0 '\000'}, group = {read = 1 '\001', write = 1 '\001', exec = 0 '\000'}, other = {read = 1 '\001', write = 0 '\000', exec = 0 '\000'}}}, dict = 0x7fef200658f8, inode = 0x7fef100032c8, d_name = 0x7fef20001140 "foo"} (gdb) c Continuing. Breakpoint 2, rda_fill_fd_cbk (frame=frame at entry=0x7fef2005c628, cookie=, this=0x7fef200132e0, op_ret=op_ret at entry=4, op_errno=op_errno at entry=2, entries=entries at entry=0x7fef25b36710, xdata=xdata at entry=0x0) at readdir-ahead.c:537 537 STACK_UNWIND_STRICT(readdirp, stub->frame, ret, op_errno, (gdb) l 532 op_errno = 0; 533 534 UNLOCK(&ctx->lock); 535 536 if (serve) { 537 STACK_UNWIND_STRICT(readdirp, stub->frame, ret, op_errno, 538 &serve_entries, xdata); 539 gf_dirent_free(&serve_entries); 540 call_stub_destroy(stub); 541 } (gdb) p &serve_entries $2 = (gf_dirent_t *) 0x7fef25b364c0 (gdb) p *serve_entries.next.next.next $3 = {{list = {next = 0x7fef200011a0, prev = 0x7fef20000f40}, {next = 0x7fef200011a0, prev = 0x7fef20000f40}}, d_ino = 10966013112435171471, d_off = 28, d_len = 3, d_type = 8, d_stat = {ia_flags = 0, ia_ino = 0, ia_dev = 0, ia_rdev = 0, ia_size = 0, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_mtime = 0, ia_ctime = 0, ia_btime = 0, ia_atime_nsec = 0, ia_mtime_nsec = 0, ia_ctime_nsec = 0, ia_btime_nsec = 0, ia_attributes = 0, ia_attributes_mask = 0, ia_gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", ia_type = IA_IFREG, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}}, dict = 0x7fef200658f8, inode = 0x7fef100032c8, d_name = 0x7fef20001140 "foo"} (gdb) c Continuing. Breakpoint 3, posix_acl_readdirp_cbk (frame=0x7fef1000b8c8, cookie=0x7fef1000c9e8, this=0x7fef2001dc00, op_ret=4, op_errno=2, entries=0x7fef25b364c0, xdata=0x0) at posix-acl.c:1654 1654 { (gdb) p entries $4 = (gf_dirent_t *) 0x7fef25b364c0 (gdb) p *entries.next.next.next $5 = {{list = {next = 0x7fef200011a0, prev = 0x7fef20000f40}, {next = 0x7fef200011a0, prev = 0x7fef20000f40}}, d_ino = 10966013112435171471, d_off = 28, d_len = 3, d_type = 8, d_stat = {ia_flags = 0, ia_ino = 0, ia_dev = 0, ia_rdev = 0, ia_size = 0, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_mtime = 0, ia_ctime = 0, ia_btime = 0, ia_atime_nsec = 0, ia_mtime_nsec = 0, ia_ctime_nsec = 0, ia_btime_nsec = 0, ia_attributes = 0, ia_attributes_mask = 0, ia_gfid = "\341n\035>u\030C#\230/\032\323H\371`\217", ia_type = IA_IFREG, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}}, dict = 0x7fef200658f8, inode = 0x7fef100032c8, d_name = 0x7fef20001140 "foo"} -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 13:52:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 13:52:48 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1649054, which changed state. Bug 1649054 Summary: glustereventsd does not start on Ubuntu 16.04 LTS https://bugzilla.redhat.com/show_bug.cgi?id=1649054 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 14:11:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 14:11:50 +0000 Subject: [Bugs] [Bug 1680586] Building RPM packages with _for_fedora_koji_builds enabled fails on el6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1680586 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-26 14:11:50 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22264 (build: do not install service related files when building --without-server) merged (#1) on release-6 by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 14:11:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 14:11:50 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Bug 1672818 depends on bug 1680586, which changed state. Bug 1680586 Summary: Building RPM packages with _for_fedora_koji_builds enabled fails on el6 https://bugzilla.redhat.com/show_bug.cgi?id=1680586 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 14:12:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 14:12:13 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug 1667103 depends on bug 1683008, which changed state. Bug 1683008 Summary: glustereventsd does not start on Ubuntu 16.04 LTS https://bugzilla.redhat.com/show_bug.cgi?id=1683008 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 14:22:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 14:22:54 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22269 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 14:22:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 14:22:55 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22269 (doc: Added release notes for release 5.4) posted (#1) for review on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 14:44:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 14:44:33 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-26 14:44:33 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22269 (doc: Added release notes for release 5.4) merged (#1) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 15:10:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 15:10:04 +0000 Subject: [Bugs] [Bug 1683317] New: ./tests/bugs/glusterfs/bug-866459.t failing on s390x Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683317 Bug ID: 1683317 Summary: ./tests/bugs/glusterfs/bug-866459.t failing on s390x Product: GlusterFS Version: 5 Hardware: s390x OS: Linux Status: NEW Component: tests Assignee: bugs at gluster.org Reporter: dalefu at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: The test case ./tests/bugs/glusterfs/bug-866459.t seems to be failing on s390x due to linux-aio being enabled on the case test. If I comment out the line that enables linux-aio, the test case will pass. This line fails as it seems that we are unable to write files to /mnt/glusterfs/0/ when the linux-aio feature is on. dd of=$M0/a if=/dev/urandom bs=1024k count=1 2>&1 > /dev/null Upon further investigation, I also noticed that when I try to touch a file on /mnt/glusterfs/0/, it works. But whenever I try to edit the file on vim, it will fail with "/mnt/glusterfs/0/a" E667: Fsync failed. How important is it to have linux-aio enable in the test case? This behaviour regarding linux-aio being enabled has been seen before in previous bug reports but it seems to been fixed on x86. Version-Release number of selected component (if applicable): v5.3 How reproducible: Always Steps to Reproduce: 1. Build Glusterfs v5.3 from source 2. Run the test case with ./run-tests.sh 3. Or just enable gluster feature linux-aio and try to write files Actual results: Test fails and /mnt/glusterfs/0/a uneditable Expected results: Test passes and /mnt/glusterfs/0/a editable Additional info: (I can upload full logs if needed) Snippets of /var/log/glusterfs/mnt-glusterfs-0.log [2019-02-25 15:08:41.017171] W [MSGID: 114031] [client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-patchy-client-0: remote operation failed [Invalid argument] [2019-02-25 15:08:41.019941] W [fuse-bridge.c:1436:fuse_err_cbk] 0-glusterfs-fuse: 17: FLUSH() ERR => -1 (Invalid argument) Snippets of the strace dd command log read(0, "`)\17\3713\251\213i\25E\10^\17&\"\244t\352h\340^3\233G\371\245\260:w\377\2732"..., 1048576) = 1048576 write(1, "`)\17\3713\251\213i\25E\10^\17&\"\244t\352h\340^3\233G\371\245\260:w\377\2732"..., 1048576) = 1048576 close(0) = 0 close(1) = -1 EINVAL (Invalid argument) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 15:22:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 15:22:23 +0000 Subject: [Bugs] [Bug 1660225] geo-rep does not replicate mv or rename of file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660225 --- Comment #9 from perplexed767 --- (In reply to Sunny Kumar from comment #8) > Hi, > > Looks like https://review.gluster.org/#/c/glusterfs/+/20093/. > > But I am trying for reproducer to analyse more. > > If something is missing in step to reproduce please add. > > - Sunny sounds like it could be, which version of gluster is this released in? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 15:35:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 15:35:18 +0000 Subject: [Bugs] [Bug 1660225] geo-rep does not replicate mv or rename of file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660225 --- Comment #10 from perplexed767 --- It is quite easy to reproduce, so don't think I missed anything. default config, replica 2 on both sites, create geo session, start, create and rename a file. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Feb 26 16:16:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 16:16:16 +0000 Subject: [Bugs] [Bug 1683352] New: remove experimental xlators informations from glusterd-volume-set.c Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 Bug ID: 1683352 Summary: remove experimental xlators informations from glusterd-volume-set.c Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: srakonde at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: experimental xlators have been removed from the codebase, but we missed to remove the options from glusterd-volume-set.c Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 16:24:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 16:24:47 +0000 Subject: [Bugs] [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22271 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Feb 26 16:24:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 26 Feb 2019 16:24:48 +0000 Subject: [Bugs] [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22271 (glusterd: remove experimental xlator options from glusterd-volume-set.c) posted (#1) for review on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:24:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:24:47 +0000 Subject: [Bugs] [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-27 03:24:47 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22271 (glusterd: remove experimental xlator options from glusterd-volume-set.c) merged (#1) on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:36:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:36:55 +0000 Subject: [Bugs] [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1683506 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683506 [Bug 1683506] remove experimental xlators informations from glusterd-volume-set.c -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:36:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:36:55 +0000 Subject: [Bugs] [Bug 1683506] New: remove experimental xlators informations from glusterd-volume-set.c Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683506 Bug ID: 1683506 Summary: remove experimental xlators informations from glusterd-volume-set.c Product: GlusterFS Version: 6 Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: srakonde at redhat.com CC: bugs at gluster.org Depends On: 1683352 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1683352 +++ Description of problem: experimental xlators have been removed from the codebase, but we missed to remove the options from glusterd-volume-set.c Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2019-02-26 21:54:48 IST --- REVIEW: https://review.gluster.org/22271 (glusterd: remove experimental xlator options from glusterd-volume-set.c) posted (#1) for review on master by Sanju Rakonde --- Additional comment from Worker Ant on 2019-02-27 08:54:47 IST --- REVIEW: https://review.gluster.org/22271 (glusterd: remove experimental xlator options from glusterd-volume-set.c) merged (#1) on master by Sanju Rakonde Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:38:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:38:52 +0000 Subject: [Bugs] [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22272 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:38:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:38:53 +0000 Subject: [Bugs] [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22272 (glusterd: remove experimental xlator options from glusterd-volume-set.c) posted (#1) for review on release-6 by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:38:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:38:53 +0000 Subject: [Bugs] [Bug 1683506] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683506 Bug 1683506 depends on bug 1683352, which changed state. Bug 1683352 Summary: remove experimental xlators informations from glusterd-volume-set.c https://bugzilla.redhat.com/show_bug.cgi?id=1683352 What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:41:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:41:11 +0000 Subject: [Bugs] [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 --- Comment #4 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22272 (glusterd: remove experimental xlator options from glusterd-volume-set.c) posted (#2) for review on release-6 by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:41:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:41:12 +0000 Subject: [Bugs] [Bug 1683352] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683352 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Gluster.org Gerrit 22272 | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:41:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:41:13 +0000 Subject: [Bugs] [Bug 1683506] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683506 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22272 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 03:41:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 03:41:15 +0000 Subject: [Bugs] [Bug 1683506] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683506 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22272 (glusterd: remove experimental xlator options from glusterd-volume-set.c) posted (#2) for review on release-6 by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 06:34:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 06:34:35 +0000 Subject: [Bugs] [Bug 1683526] New: rebalance start command doesn't throw up error message if the command fails Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683526 Bug ID: 1683526 Summary: rebalance start command doesn't throw up error message if the command fails Product: GlusterFS Version: 4.1 Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: amukherj at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: When a rebalance start command fails, it doesn't throw up the error message back to CLI. Version-Release number of selected component (if applicable): release-6 How reproducible: Always Steps to Reproduce: 1. Create 1 X 1 volume, trigger rebalance start. Command fails as glusterd.log complains about following [2019-02-27 06:29:15.448303] E [MSGID: 106218] [glusterd-rebalance.c:462:glusterd_rebalance_cmd_validate] 0-glusterd: Volume test-vol5 is not a distribute type or contains only 1 brick But CLI doesn't throw up any error messages. Actual results: CLI doesn't throw up an error message. Expected results: CLI should throw up an error message. Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 09:21:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 09:21:33 +0000 Subject: [Bugs] [Bug 1683574] New: gluster-server package requires the older userspace-rcu Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683574 Bug ID: 1683574 Summary: gluster-server package requires the older userspace-rcu Product: GlusterFS Version: 6 Hardware: x86_64 OS: Linux Status: NEW Whiteboard: gluster-test-day Component: build Severity: high Priority: high Assignee: bugs at gluster.org Reporter: sasundar at redhat.com CC: bugs at gluster.org Blocks: 1672818 (glusterfs-6.0) Target Milestone: --- Classification: Community Description of problem: ------------------------ While installing glusterfs-server-6.0 package on Centos 6, observed that this glusterfs-server requires liburcu-bp.so.1, but Centos-release-gluster ships the latest version of liburcu-bp.so.6 ( updated as part of https://bugzilla.redhat.com/show_bug.cgi?id=1410302 ) Version-Release number of selected component (if applicable): ------------------------------------------------------------- Centos-7.6 glusterfs-server-6.0 How reproducible: ----------------- Always Steps to Reproduce: ------------------- 1. Install glusterfs-server-6.0 on Centos 7.6 Actual results: --------------- Installation fails with dependency on liburcu-bp.so.1, though liburcu-bp.so.6 is available Expected results: ----------------- glusterfs-server should require newer updated version of liburcu-bp.so.6 Additional info: ---------------- [root@ ~]# rpm -qpR glusterfs-server-5.3-2.el7.x86_64.rpm | grep -i rcu liburcu-bp.so.6()(64bit) liburcu-cds.so.6()(64bit) [root@ ~]# rpm -qpR glusterfs-server-6.0-0.1.rc0.el7.x86_64.rpm | grep -i rcu warning: glusterfs-server-6.0-0.1.rc0.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID c2f8238c: NOKEY liburcu-bp.so.1()(64bit) liburcu-cds.so.1()(64bit) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 09:21:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 09:21:33 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 SATHEESARAN changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1683574 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683574 [Bug 1683574] gluster-server package requires the older userspace-rcu -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 10:01:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 10:01:17 +0000 Subject: [Bugs] [Bug 1683594] New: nfs ltp ftest* fstat gets mismatch size as except after turn on md-cache Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683594 Bug ID: 1683594 Summary: nfs ltp ftest* fstat gets mismatch size as except after turn on md-cache Product: GlusterFS Version: mainline Status: NEW Component: md-cache Assignee: bugs at gluster.org Reporter: kinglongmee at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Exporting nfs exports by nfs-ganesha base on glusterfs-5.3-1.fc29.x86_64 at fedora. After md-cache turn on, ltp ftest* reports mismatch size as except; ftest01 1 TFAIL : ftest01.c:521: Test[2]: fstat() mismatch; st_size=901120, file_max=ed800. ftest01 1 TFAIL : ftest01.c:521: Test[4]: fstat() mismatch; st_size=993280, file_max=ff800. ftest01 1 TFAIL : ftest01.c:521: Test[0]: fstat() mismatch; st_size=1009664, file_max=f8000. ftest01 1 TFAIL : ftest01.c:521: Test[1]: fstat() mismatch; st_size=890880, file_max=fc800. ftest01 1 TFAIL : ftest01.c:521: Test[3]: fstat() mismatch; st_size=995328, file_max=100000. ftest01 1 TFAIL : ftest01.c:189: Test{14660} failed, expected 0 exit ftest01 2 TFAIL : ftest01.c:189: Test{14662} failed, expected 0 exit ftest01 3 TFAIL : ftest01.c:189: Test{14658} failed, expected 0 exit ftest01 4 TFAIL : ftest01.c:189: Test{14659} failed, expected 0 exit ftest01 5 TFAIL : ftest01.c:189: Test{14661} failed, expected 0 exit ftest01 6 TFAIL : ftest01.c:211: Test failed in fork and wait. ftest01 7 TFAIL : ftest01.c:113: Test failed. turn off md-cache, ftest01 1 TPASS : Test passed in fork and wait. ftest01 2 TPASS : Test passed. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 10:14:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 10:14:46 +0000 Subject: [Bugs] [Bug 1683594] nfs ltp ftest* fstat gets mismatch size as except after turn on md-cache In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683594 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22273 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 10:14:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 10:14:47 +0000 Subject: [Bugs] [Bug 1683594] nfs ltp ftest* fstat gets mismatch size as except after turn on md-cache In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683594 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22273 (md-cache: only update generation for inode at upcall and NULL stat) posted (#1) for review on master by Kinglong Mee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 10:28:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 10:28:14 +0000 Subject: [Bugs] [Bug 1637802] data-self-heal in arbiter volume results in stale locks. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1637802 Karthik U S changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |psony at redhat.com --- Comment #3 from Karthik U S --- *** Bug 1664223 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 27 12:05:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 12:05:11 +0000 Subject: [Bugs] [Bug 1230671] SELinux not supported with FUSE client In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1230671 Bug 1230671 depends on bug 1291606, which changed state. Bug 1291606 Summary: RFE: Add support for filesystem subtypes in SELinux https://bugzilla.redhat.com/show_bug.cgi?id=1291606 What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |WONTFIX -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Feb 27 12:46:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 12:46:12 +0000 Subject: [Bugs] [Bug 1683650] New: Gluster Build System failing on pip install flake8 pylint Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683650 Bug ID: 1683650 Summary: Gluster Build System failing on pip install flake8 pylint Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Severity: high Assignee: bugs at gluster.org Reporter: kiyer at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: I am unable to verify my patch as Gluster Build System is failing with the below given error: 17:05:57 Collecting isort>=4.2.5 (from pylint==1.9.2) 17:05:57 Using cached https://files.pythonhosted.org/packages/35/c5/025aaa76f0f337e3d91197973c42b4e807fa6dd509da70e87053d85ce278/isort-4.3.9.tar.gz 17:05:57 Complete output from command python setup.py egg_info: 17:05:57 /opt/rh/python27/root/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires' 17:05:57 warnings.warn(msg) 17:05:57 error in isort setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in futures; python_version < "3.2" at ; python_version < "3.2" 17:05:57 17:05:57 ---------------------------------------- 17:05:57 Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-ZxIJIL/isort 17:05:57 17:05:57 ERROR: could not install deps [flake8, pylint==1.9.2]; v = InvocationError('/home/jenkins/root/workspace/glusto-tests-lint/.tox/py27/bin/python -m pip install flake8 pylint==1.9.2 (see /home/jenkins/root/workspace/glusto-tests-lint/.tox/py27/log/py27-1.log)', 1) 17:05:57 ___________________________________ summary ____________________________________ 17:05:57 ERROR: py27: could not install deps [flake8, pylint==1.9.2]; v = InvocationError('/home/jenkins/root/workspace/glusto-tests-lint/.tox/py27/bin/python -m pip install flake8 pylint==1.9.2 (see /home/jenkins/root/workspace/glusto-tests-lint/.tox/py27/log/py27-1.log)', 1) 17:05:57 Build step 'Execute shell' marked build as failure 17:05:57 Finished: FAILURE Version-Release number of selected component (if applicable): https://build.gluster.org/job/glusto-tests-lint/3442/console How reproducible: 2/2 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 13:56:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 13:56:21 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22274 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 13:56:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 13:56:21 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #566 from Worker Ant --- REVIEW: https://review.gluster.org/22274 (mem-pool.{c|h}: minor changes) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 15:20:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 15:20:30 +0000 Subject: [Bugs] [Bug 1683506] remove experimental xlators informations from glusterd-volume-set.c In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683506 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-02-27 15:20:30 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22272 (glusterd: remove experimental xlator options from glusterd-volume-set.c) merged (#2) on release-6 by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 16:00:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 16:00:56 +0000 Subject: [Bugs] [Bug 1683716] New: glusterfind: revert shebangs to #!/usr/bin/python3 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683716 Bug ID: 1683716 Summary: glusterfind: revert shebangs to #!/usr/bin/python3 Product: GlusterFS Version: 6 Status: NEW Component: glusterfind Assignee: bugs at gluster.org Reporter: kkeithle at redhat.com QA Contact: bugs at gluster.org CC: avishwan at redhat.com, bugs at gluster.org, khiremat at redhat.com Target Milestone: --- Classification: Community Description of problem: Using #!/usr/bin/env python is illegal in Fedora and RHEL packaging. (Debian and SUSE packaging will complain about it too but don't enforce it like Fedora packaging does. Yet.) We went through a giant exercise fixing these once already. The python needs to be python2/python3 clean. There is a build scriptlet (in the glusterfs.spec file) that converts the shebangs back to /usr/bin/python on rhel7 at rpm build time. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 16:02:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 16:02:53 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22275 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 16:02:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 16:02:55 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #567 from Worker Ant --- REVIEW: https://review.gluster.org/22275 (glusterfind: revert shebangs to #!/usr/bin/python3) posted (#1) for review on release-6 by Kaleb KEITHLEY -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 16:07:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 16:07:48 +0000 Subject: [Bugs] [Bug 1683716] glusterfind: revert shebangs to #!/usr/bin/python3 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683716 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22275 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 16:07:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 16:07:49 +0000 Subject: [Bugs] [Bug 1683716] glusterfind: revert shebangs to #!/usr/bin/python3 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683716 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22275 (glusterfind: revert shebangs to #!/usr/bin/python3) posted (#3) for review on release-6 by Kaleb KEITHLEY -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 17:33:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 17:33:42 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22277 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 17:33:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 17:33:43 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #568 from Worker Ant --- REVIEW: https://review.gluster.org/22277 (logging.c: replace get_mem0() with get_mem()) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 20:08:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 20:08:53 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22278 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 20:08:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 20:08:54 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #569 from Worker Ant --- REVIEW: https://review.gluster.org/22278 (io-threads.c: Potentially skip a lock.) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 20:29:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 20:29:36 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22279 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 20:29:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 20:29:37 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #570 from Worker Ant --- REVIEW: https://review.gluster.org/22279 (fd.c: try to skip a lock in fd_ref()) posted (#1) for review on master by Yaniv Kaul -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:32:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:32:04 +0000 Subject: [Bugs] [Bug 1683815] New: Memory leak when peer detach fails Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683815 Bug ID: 1683815 Summary: Memory leak when peer detach fails Product: GlusterFS Version: 6 Status: NEW Whiteboard: gluster-test-day Component: core Severity: high Priority: high Assignee: bugs at gluster.org Reporter: vbellur at redhat.com CC: bugs at gluster.org Blocks: 1672818 (glusterfs-6.0) Target Milestone: --- Classification: Community Description of problem: When a peer detach fails, there is a memory leak. By trying to detach my local hostname, I was able to observe a slow increase in resident memory consumption of glusterd. Version-Release number of selected component (if applicable): How reproducible: # gluster peer detach Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:32:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:32:04 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Vijay Bellur changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1683815 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683815 [Bug 1683815] Memory leak when peer detach fails -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:36:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:36:35 +0000 Subject: [Bugs] [Bug 1683816] New: Memory leak when peer detach fails Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683816 Bug ID: 1683816 Summary: Memory leak when peer detach fails Product: GlusterFS Version: mainline Status: NEW Whiteboard: gluster-test-day Component: core Severity: medium Priority: high Assignee: bugs at gluster.org Reporter: vbellur at redhat.com CC: bugs at gluster.org Depends On: 1683815 Blocks: 1672818 (glusterfs-6.0) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1683815 +++ Description of problem: When a peer detach fails, there is a memory leak. By trying to detach my local hostname, I was able to observe a slow increase in resident memory consumption of glusterd. Version-Release number of selected component (if applicable): How reproducible: # gluster peer detach Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker https://bugzilla.redhat.com/show_bug.cgi?id=1683815 [Bug 1683815] Memory leak when peer detach fails -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:36:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:36:35 +0000 Subject: [Bugs] [Bug 1683815] Memory leak when peer detach fails In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683815 Vijay Bellur changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1683816 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683816 [Bug 1683816] Memory leak when peer detach fails -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:36:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:36:35 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Vijay Bellur changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1683816 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683816 [Bug 1683816] Memory leak when peer detach fails -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:36:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:36:59 +0000 Subject: [Bugs] [Bug 1683816] Memory leak when peer detach fails In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683816 Vijay Bellur changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|1672818 (glusterfs-6.0) | Whiteboard|gluster-test-day | Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:36:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:36:59 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Vijay Bellur changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On|1683816 | Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683816 [Bug 1683816] Memory leak when peer detach fails -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:43:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:43:28 +0000 Subject: [Bugs] [Bug 1683816] Memory leak when peer detach fails In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683816 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22280 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Feb 27 21:43:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 27 Feb 2019 21:43:29 +0000 Subject: [Bugs] [Bug 1683816] Memory leak when peer detach fails In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683816 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22280 (mgmt/glusterd: Fix a memory leak when peer detach fails) posted (#1) for review on master by Vijay Bellur -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 00:07:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 00:07:58 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22281 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 00:07:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 00:07:59 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #571 from Worker Ant --- REVIEW: https://review.gluster.org/22281 (Replace memdup() with gf_memdup()) posted (#1) for review on master by Vijay Bellur -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 03:02:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 03:02:31 +0000 Subject: [Bugs] [Bug 1683880] New: Multiple shd processes are running on brick_mux environmet Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683880 Bug ID: 1683880 Summary: Multiple shd processes are running on brick_mux environmet Product: GlusterFS Version: 6 Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: moagrawa at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Multiple shd processes are running while created 100 volumes in brick_mux environment Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create a 1x3 volume 2. Enable brick_mux 3.Run below command n1= n2= n3= for i in {1..10};do for h in {1..20};do gluster v create vol-$i-$h rep 3 $n1:/home/dist/brick$h/vol-$i-$h $n2:/home/dist/brick$h/vol-$i-$h $n3:/home/dist/brick$h/vol-$i-$h force gluster v start vol-$i-$h sleep 1 done done for k in $(gluster v list|grep -v heketi);do gluster v stop $k --mode=script;sleep 2;gluster v delete $k --mode=script;sleep 2;done Actual results: Multiple shd processes are running and consuming system resources Expected results: Only one shd process should be run Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 03:03:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 03:03:00 +0000 Subject: [Bugs] [Bug 1683880] Multiple shd processes are running on brick_mux environmet In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683880 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |high Hardware|Unspecified |x86_64 Assignee|bugs at gluster.org |moagrawa at redhat.com Severity|unspecified |high -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 03:03:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 03:03:19 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |moagrawa at redhat.com Depends On| |1683880 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683880 [Bug 1683880] Multiple shd processes are running on brick_mux environmet -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 03:03:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 03:03:19 +0000 Subject: [Bugs] [Bug 1683880] Multiple shd processes are running on brick_mux environmet In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683880 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1672818 (glusterfs-6.0) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 04:13:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 04:13:18 +0000 Subject: [Bugs] [Bug 1230671] SELinux not supported with FUSE client In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1230671 Bipin Kunal changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1683899 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 04:18:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 04:18:09 +0000 Subject: [Bugs] [Bug 1683900] New: Failed to dispatch handler Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683900 Bug ID: 1683900 Summary: Failed to dispatch handler Product: GlusterFS Version: 6 Hardware: x86_64 OS: Linux Status: NEW Component: core Keywords: Reopened Severity: high Priority: urgent Assignee: bugs at gluster.org Reporter: mchangir at redhat.com CC: alexander at bitspace.se, amgad.saleh at nokia.com, archon810 at gmail.com, brice at digiteyes.fr, bugs at gluster.org, desmith at wustl.edu, emerson.gomes at gmail.com, endre.karlson at gmail.com, guillaume.pavese at interact-iv.com, jayme at silverorange.com, joao.bauto at neuro.fchampalimaud.org, nbalacha at redhat.com, pasik at iki.fi, ravishankar at redhat.com, rob.dewit at coosto.com, sabose at redhat.com, tavis.paquette at eventbase.com, timo at teissler.de, vanessa.haro at nokia.com, vnosov at stonefly.com, waza123 at inbox.lv, ykaul at redhat.com Depends On: 1651246 Blocks: 1677319 (Gluster_5_Affecting_oVirt_4.3), 1667103 (glusterfs-5.4) Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1651246 +++ (description has been truncated since it was more than 64KiB) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 [Bug 1651246] Failed to dispatch handler https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker https://bugzilla.redhat.com/show_bug.cgi?id=1677319 [Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 04:18:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 04:18:09 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1683900 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683900 [Bug 1683900] Failed to dispatch handler -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 04:18:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 04:18:09 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1683900 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1683900 [Bug 1683900] Failed to dispatch handler -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 04:25:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 04:25:34 +0000 Subject: [Bugs] [Bug 1683900] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683900 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22282 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 04:25:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 04:25:35 +0000 Subject: [Bugs] [Bug 1683900] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683900 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22282 (socket: socket event handlers now return void) posted (#1) for review on release-6 by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 05:26:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 05:26:29 +0000 Subject: [Bugs] [Bug 1683574] gluster-server package currently requires the older userspace-rcu against expectation In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683574 SATHEESARAN changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|gluster-server package |gluster-server package |requires the older |currently requires the |userspace-rcu |older userspace-rcu against | |expectation -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 06:35:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 06:35:18 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Sayalee changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(saraut at redhat.com | |) | -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 07:08:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 07:08:51 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22285 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 07:08:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 07:08:52 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 --- Comment #11 from Worker Ant --- REVIEW: https://review.gluster.org/22285 (doc: update releasenote) posted (#1) for review on release-6 by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 08:38:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 08:38:22 +0000 Subject: [Bugs] [Bug 1683973] New: Adding NBD support for glusterfs and needs one separate repository in github.com/gluster/ Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683973 Bug ID: 1683973 Summary: Adding NBD support for glusterfs and needs one separate repository in github.com/gluster/ Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: xiubli at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Hi infra, For the glusterfs we are working on supporting the Network Block Device support, and need one separate repository under github.com/gluster/. The name will be: "nbd-runner", then need to get the ownership of it. Thanks. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 09:01:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:01:12 +0000 Subject: [Bugs] [Bug 1683973] Adding NBD support for glusterfs and needs one separate repository in github.com/gluster/ In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683973 Deepshikha khandelwal changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dkhandel at redhat.com --- Comment #1 from Deepshikha khandelwal --- I need GitHub handle of the owner. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 09:19:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:19:24 +0000 Subject: [Bugs] [Bug 1672480] Bugs Test Module tests failing on s390x In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672480 --- Comment #51 from Nithya Balachandran --- > > > I don't think so. I would recommend that you debug the tests on your systems > and post patches which will work on both. Please note what I am referring to is for you to look at the .t files and modify file names or remove hardcoding as required. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 09:54:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:54:59 +0000 Subject: [Bugs] [Bug 1684029] New: upgrade from 3.12, 4.1 and 5 to 6 broken Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1684029 Bug ID: 1684029 Summary: upgrade from 3.12, 4.1 and 5 to 6 broken Product: GlusterFS Version: 6 Status: NEW Whiteboard: gluster-test-day Component: core Severity: urgent Priority: high Assignee: bugs at gluster.org Reporter: hgowtham at redhat.com CC: bugs at gluster.org Blocks: 1672818 (glusterfs-6.0) Target Milestone: --- Classification: Community Description of problem: While trying to upgrade from older versions like 3.12, 4.1 and 5 to gluster 6 RC, the upgrade ends in peer rejected on one node after other. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. create a replica 3 on older versions (3, 4, or 5) 2. kill the gluster process on one node and install gluster 6 3. start glusterd Actual results: the new version gets peer rejected. and the brick processes or not started by glusterd. Expected results: peer reject should not happen. Cluster should be healthy. Additional info: Status shows the bricks on that particular node alone with N/A as status. Other nodes aren't visible. Looks like a volfile mismatch. The new volfile has "option transport.socket.ssl-enabled off" added while the old volfile misses it. The order of quick-read and open-behind are changed in the old and new versions. These changes cause the volfile mismatch and mess the cluster. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 [Bug 1672818] GlusterFS 6.0 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 09:54:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:54:59 +0000 Subject: [Bugs] [Bug 1672818] GlusterFS 6.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1672818 hari gowtham changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1684029 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1684029 [Bug 1684029] upgrade from 3.12, 4.1 and 5 to 6 broken -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 09:57:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:57:25 +0000 Subject: [Bugs] [Bug 1683973] Adding NBD support for glusterfs and needs one separate repository in github.com/gluster/ In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683973 Deepshikha khandelwal changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |NOTABUG Last Closed| |2019-02-28 09:57:25 --- Comment #3 from Deepshikha khandelwal --- Done. You're now the owner of this repo. Repo: https://github.com/gluster/nbd-runner -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 09:57:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:57:29 +0000 Subject: [Bugs] [Bug 1648768] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1648768 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22286 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 09:57:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:57:30 +0000 Subject: [Bugs] [Bug 1648768] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1648768 --- Comment #18 from Worker Ant --- REVIEW: https://review.gluster.org/22286 (afr: mark changelog_fsync as internal) posted (#1) for review on master by soumya k -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 09:58:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:58:44 +0000 Subject: [Bugs] [Bug 1648768] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1648768 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22287 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 09:58:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 09:58:45 +0000 Subject: [Bugs] [Bug 1648768] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1648768 --- Comment #19 from Worker Ant --- REVIEW: https://review.gluster.org/22287 (leases: Do not process internal fops) posted (#1) for review on master by soumya k -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 10:04:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 10:04:33 +0000 Subject: [Bugs] [Bug 1683650] Gluster Build System failing on pip install flake8 pylint In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683650 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mscherer at redhat.com --- Comment #1 from M. Scherer --- yeah, seems some package got upgraded, and this requires a newer pip to be installed. I am going to see how to fix that. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 10:21:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 10:21:12 +0000 Subject: [Bugs] [Bug 1683973] Adding NBD support for glusterfs and needs one separate repository in github.com/gluster/ In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683973 --- Comment #4 from Xiubo Li --- (In reply to Deepshikha khandelwal from comment #3) > Done. You're now the owner of this repo. > > Repo: https://github.com/gluster/nbd-runner Thanks very much. BRs -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 10:33:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 10:33:06 +0000 Subject: [Bugs] [Bug 1683650] Gluster Build System failing on pip install flake8 pylint In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1683650 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-02-28 10:33:06 --- Comment #2 from M. Scherer --- Seems to work now -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 11:20:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 11:20:17 +0000 Subject: [Bugs] [Bug 1537346] glustershd/glusterd is not using right port when connecting to glusterfsd process In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1537346 Prashant Dhange changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jthottan at redhat.com, | |pdhange at redhat.com Flags| |needinfo?(jthottan at redhat.c | |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 11:55:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 11:55:57 +0000 Subject: [Bugs] [Bug 1684029] upgrade from 3.12, 4.1 and 5 to 6 broken In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1684029 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED CC| |amukherj at redhat.com, | |srakonde at redhat.com Flags| |needinfo?(amukherj at redhat.c | |om) --- Comment #1 from Sanju --- The peers are running inro rejected state because there is a mismatch in the volfiles. Differences are: 1. Newer volfiles are having "option transport.socket.ssl-enabled off" where older volfiles are not having this option. 2. order of quick-read and open-behind are changed commit 4e0fab4 introduced this issue. previously we didn't had any default value for the option transport.socket.ssl-enabled. So this option was not captured in the volfile. with the above commit, we are adding a default value. So this is getting captured in volfile. commit 4e0fab4 has a fix for https://bugzilla.redhat.com/show_bug.cgi?id=1651059. I feel this commit has less significance, we can revert this change. If we do so, we are out of 1st problem. not sure, why the order of quick-read and open-behind are changed. Atin, do let me know your thoughts on proposal of reverting the commit 4e0fab4. Thanks, Sanju -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 14:16:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 14:16:39 +0000 Subject: [Bugs] [Bug 1684128] New: Create a github repo named "gluster-objects-store" under gluster org Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1684128 Bug ID: 1684128 Summary: Create a github repo named "gluster-objects-store" under gluster org Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: rkavunga at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: We want to create a new github repo to participate in GSSoC project. The detailed contents are captured here https://docs.google.com/document/d/1N-LhxRyQYmrh95DVuX8eClqFkAorhJlvWIjfJw3O8dg/edit?usp=sharing . Owners : https://github.com/aravindavk https://github.com/rafikc30 Project name : gluster-objects-store Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Feb 28 14:27:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 14:27:55 +0000 Subject: [Bugs] [Bug 1537346] glustershd/glusterd is not using right port when connecting to glusterfsd process In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1537346 Jiffin changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jthottan at redhat.c | |om) | -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Feb 28 15:30:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 28 Feb 2019 15:30:14 +0000 Subject: [Bugs] [Bug 1684128] Create a github repo named "gluster-objects-store" under gluster org In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1684128 Deepshikha khandelwal changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED CC| |dkhandel at redhat.com Resolution|--- |NOTABUG Last Closed| |2019-02-28 15:30:14 --- Comment #1 from Deepshikha khandelwal --- It is now done. Repo: https://github.com/gluster/gluster-objects-store -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From rabhat at redhat.com Wed Feb 13 21:25:01 2019 From: rabhat at redhat.com (FNU Raghavendra Manjunath) Date: Wed, 13 Feb 2019 21:25:01 -0000 Subject: [Bugs] GlusterFs v4.1.5: Need help on bitrot detection In-Reply-To: References: Message-ID: Hi Chandranana, Thanks for letting us know. Will go through the attachment and get back to you. Regards, Raghavendra On Mon, Feb 11, 2019 at 5:16 AM Chandranana Naik wrote: > Hi Team, > > We are working with Glusterfs v4.1.5 on big endian platform(Ubuntu 16.04) > and encountered that the subtest 20 of test > ./tests/bitrot/bug-1207627-bitrot-scrub-status.t is failing. > > Subtest 20 is failing as below: > *trusted.bit-rot.bad-file check_for_xattr trusted.bit-rot.bad-file > //d/backends/patchy1/FILE1* > *not ok 20 Got "" instead of "trusted.bit-rot.bad-file", LINENUM:50* > *FAILED COMMAND: trusted.bit-rot.bad-file check_for_xattr > trusted.bit-rot.bad-file //d/backends/patchy1/FILE1* > > The test is failing with error "*remote operation failed [Cannot allocate > memory]"* logged in /var/log/glusterfs/scrub.log. > Could you please let us know if anything is missing in making this test > pass, PFA the logs for the test case > > *(See attached file: bug-1207627-bitrot-scrub-status.7z)* > > Note: *Enough memory is available on the system*. > > Regards, > Chandranana Naik > -------------- next part -------------- An HTML attachment was scrubbed... URL: