From bugzilla at redhat.com Tue Jan 1 15:50:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 01 Jan 2019 15:50:17 +0000 Subject: [Bugs] [Bug 1138841] allow the use of the CIDR format with auth.allow In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1138841 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|UPSTREAM |--- Keywords| |Reopened External Bug ID| |Gluster.org Gerrit 21970 --- Comment #5 from Worker Ant --- REVIEW: https://review.gluster.org/21970 (Added a function to validate CIDR IP) posted (#1) for review on master by Rinku Kothiya -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 1 18:52:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 01 Jan 2019 18:52:17 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 --- Comment #33 from Znamensky Pavel --- (In reply to Nithya Balachandran from comment #31) > Then it is likely to be because the fuse client does not invalidate inodes. > Does your workload access a lot of files? The earlier statedump showed > around 3 million inodes in memory. > >... > > https://review.gluster.org/#/c/glusterfs/+/19778/ has a fix to invalidate > inodes but is not targeted for release 5 as yet. Nithya, you're right! I built glusterfs from the current master (https://github.com/gluster/glusterfs/tree/d9a8ccd354df6db94477bf9ecb09735194523665) with the new invalidate inodes mechanism that you mentioned before, and RSS memory consumption indeed became much lower. And as you supposed our apps quite often access a lot of files. Here are two tests with clients on v6dev and v4.1 (the server is still on v4.1 and read-ahead=on) The first test with default --lru-limit=0 (just did `find /in/big/dir -type f`): v4.1 - ~3GB RSS: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 633 6.6 18.5 3570216 3056136 ? Ssl 19:44 6:25 /usr/sbin/glusterfs --read-only --process-name fuse --volfile-server=srv --volfile-id=/st1 /mnt/st1 v6dev - ~1.5GB RSS: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 10851 16.5 9.2 2071036 1526456 ? Ssl 19:45 15:50 /usr/sbin/glusterfs --read-only --process-name fuse --volfile-server=srv --volfile-id=/st1 /mnt/st1 It looks good. Let's do the next test. The second test with --lru-limit=10_000 for v6dev: v4.1 - ~3GB RSS: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 3589 4.7 18.6 3570216 3060364 ? Ssl 13:11 18:40 /usr/sbin/glusterfs --process-name fuse --volfile-server=srv --volfile-id=/st1 /mnt/st1 v6dev - ~170MB RSS: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 24152 14.2 1.0 758768 173704 ? Ssl 13:58 49:06 /usr/sbin/glusterfs --lru-limit=10000 --process-name fuse --volfile-server=srv --volfile-id=/st1 /mnt/st1 170MB vs. 3GB! It's incredible! Unfortunately, the new version has a drawback - CPU time increased 2.5x times. At the moment it doesn't matter for us. Anyway, I'm sure this change solves our problem. And of course, we're looking forward to a stable version with it. Thank you a lot! -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 1 20:37:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 01 Jan 2019 20:37:11 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #510 from Worker Ant --- REVIEW: https://review.gluster.org/21971 (all: toward better string copies) posted (#1) for review on master by Kaleb KEITHLEY -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 1 20:37:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 01 Jan 2019 20:37:13 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21971 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 03:55:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 03:55:56 +0000 Subject: [Bugs] [Bug 1138841] allow the use of the CIDR format with auth.allow In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1138841 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|moagrawa at redhat.com |rkothiya at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 05:44:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 05:44:24 +0000 Subject: [Bugs] [Bug 1660732] create gerrit for github project glusterfs-containers-tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660732 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-02 05:44:24 --- Comment #4 from Nigel Babu --- Alright. Valerii is now on the committers group for the glusterfs-container-tests -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 05:51:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 05:51:33 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 Travers Carter changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tcarter at noggin.com.au --- Comment #34 from Travers Carter --- We are seeing what looks like the same issue on glusterfs 4.1.5 I'm not sure if further information is still needed, given the last few comments, but I've collected client statedumps from three systems, along the with the gluster volume info here: https://s3.amazonaws.com/public-rhbz/glusterfs-client-4.1.5-statedumps.zip This includes 2 x client statedumps from each of 3 systems taken roughly 30 to 60 minutes apart The "webserver" and "appserver-active" gluster clients were restarted after setting readdir-ahead to off as suggested earlier in the ticket (this didn't seem to help much in this case), but the "webserver" client has already reached about 15GiB VIRT in just over 48 hours. We had also historically seen somewhat slower, but still significant fuse client memory leaks on v3.x (I think 3.11 or 3.12), but not (or at least not significant) on 3.7.11 or 4.0.2 with very similar workloads. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 06:15:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 06:15:52 +0000 Subject: [Bugs] [Bug 1662830] New: [RFE] Enable parallel-readdir by default for all gluster volumes Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662830 Bug ID: 1662830 Summary: [RFE] Enable parallel-readdir by default for all gluster volumes Product: GlusterFS Version: mainline Status: NEW Component: core Keywords: FutureFeature, Performance, ZStream Severity: high Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: bugs at gluster.org Depends On: 1510724 Target Milestone: --- Classification: Community Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1510724 [Bug 1510724] [RFE] Enable parallel-readdir by default for all gluster volumes -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 06:22:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 06:22:16 +0000 Subject: [Bugs] [Bug 1662830] [RFE] Enable parallel-readdir by default for all gluster volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662830 --- Comment #1 from Raghavendra G --- For some performance data, see: 1. https://events.static.linuxfound.org/sites/events/files/slides/Gluster_DirPerf_Vault2017_0.pdf 2. https://www.spinics.net/lists/gluster-users/msg34956.html 3. https://bugzilla.redhat.com/show_bug.cgi?id=1628807#c35 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 06:22:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 06:22:44 +0000 Subject: [Bugs] [Bug 1662830] [RFE] Enable parallel-readdir by default for all gluster volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662830 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |rgowdapp at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 06:42:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 06:42:18 +0000 Subject: [Bugs] [Bug 1662830] [RFE] Enable parallel-readdir by default for all gluster volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662830 --- Comment #2 from Raghavendra G --- Also see: 1. https://lists.gluster.org/pipermail/gluster-devel/2018-September/055419.html 2. https://lists.gnu.org/archive/html/gluster-devel/2013-09/msg00034.html >From a mail to gluster-devel titled "serialized readdir(p) across subvols and effect on performance" All, As many of us are aware, readdir(p)s are serialized across DHT subvols. One of the intuitive first reactions for this algorithm is that readdir(p) is going to be slow. However this is partly true as reading the contents of a directory is normally split into multiple readdir(p) calls and most of the times (when a directory is sufficiently large to have dentries and inode data is bigger than a typical readdir(p) buffer size - 128K when readdir-ahead is enabled and 4KB on fuse when readdir-ahead is disabled - on each subvol) a single readdir(p) request is served from a single subvolume (or two subvolumes in the worst case) and hence a single readdir(p) is not serialized across all subvolumes. Having said that, there are definitely cases where a single readdir(p) request can be serialized on many subvolumes. A best example for this is a readdir(p) request on an empty directory. Other relevant examples are those directories which don't have enough dentries to fit into a single readdir(p) buffer size on each subvolume of DHT. This is where performance.parallel-readdir helps. Also, note that this is the same reason why having cache-size for each readdir-ahead (loaded as a parent for each DHT subvolume) way bigger than a single readdir(p) buffer size won't really improve the performance in proportion to cache-size when performance.parallel-readdir is enabled. Though this is not a new observation [1] (I stumbled upon [1] after realizing the above myself independently while working on performance.parallel-readdir), I felt this as a common misconception (I ran into similar argument while trying to explain DHT architecture to someone new to Glusterfs recently) and hence thought of writing out a mail to clarify the same. [1] https://lists.gnu.org/archive/html/gluster-devel/2013-09/msg00034.html regards, Raghavendra -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 06:53:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 06:53:34 +0000 Subject: [Bugs] [Bug 1662830] [RFE] Enable parallel-readdir by default for all gluster volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662830 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/21973 (performance/parallel-readdir: enable by default) posted (#1) for review on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 07:02:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 07:02:46 +0000 Subject: [Bugs] [Bug 1662838] New: FUSE mount seems to be hung and not accessible Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662838 Bug ID: 1662838 Summary: FUSE mount seems to be hung and not accessible Product: Red Hat Gluster Storage Status: NEW Component: fuse Severity: high Assignee: csaba at redhat.com Reporter: tdesala at redhat.com QA Contact: rhinduja at redhat.com CC: bugs at gluster.org, nbalacha at redhat.com, rhs-bugs at redhat.com, sankarshan at redhat.com, storage-qa-internal at redhat.com, tdesala at redhat.com Depends On: 1659334 Target Milestone: --- Classification: Red Hat Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1659334 [Bug 1659334] FUSE mount seems to be hung and not accessible -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 07:02:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 07:02:46 +0000 Subject: [Bugs] [Bug 1659334] FUSE mount seems to be hung and not accessible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659334 Prasad Desala changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1662838 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1662838 [Bug 1662838] FUSE mount seems to be hung and not accessible -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 07:02:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 07:02:48 +0000 Subject: [Bugs] [Bug 1662838] FUSE mount seems to be hung and not accessible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662838 Red Hat Bugzilla Rules Engine changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |ZStream -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 07:15:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 07:15:47 +0000 Subject: [Bugs] [Bug 1662838] FUSE mount seems to be hung and not accessible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662838 nchilaka changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nchilaka at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 07:29:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 07:29:02 +0000 Subject: [Bugs] [Bug 1654270] glusterd crashed with seg fault possibly during node reboot while volume creates and deletes were happening In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654270 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |POST --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/21974 (glusterd: kill the process without releasing the cleanup mutex lock) posted (#1) for review on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 07:29:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 07:29:03 +0000 Subject: [Bugs] [Bug 1654270] glusterd crashed with seg fault possibly during node reboot while volume creates and deletes were happening In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654270 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21974 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 07:29:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 07:29:09 +0000 Subject: [Bugs] [Bug 1362129] rename of a file can cause data loss in an replica/arbiter volume configuration In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1362129 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ravishankar at redha | |t.com) | -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 08:41:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 08:41:31 +0000 Subject: [Bugs] [Bug 1624724] ctime: Enable ctime feature by default and also improve usability by providing single option to enable In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1624724 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |ASSIGNED CC| |rgowdapp at redhat.com Resolution|NEXTRELEASE |--- Keywords| |Reopened -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 08:41:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 08:41:53 +0000 Subject: [Bugs] [Bug 1662838] FUSE mount seems to be hung and not accessible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662838 Vijay Avuthu changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Automation CC| |vavuthu at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 10:54:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 10:54:16 +0000 Subject: [Bugs] [Bug 1654103] Invalid memory read after freed in dht_rmdir_readdirp_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654103 Sayalee changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|tdesala at redhat.com |saraut at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 10:54:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 10:54:29 +0000 Subject: [Bugs] [Bug 1659439] Memory leak: dict_t leak in rda_opendir In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659439 Sayalee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |saraut at redhat.com QA Contact|tdesala at redhat.com |saraut at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 11:09:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 11:09:03 +0000 Subject: [Bugs] [Bug 1662906] New: Longevity: glusterfsd(brick process) crashed when we do volume creates and deletes Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662906 Bug ID: 1662906 Summary: Longevity: glusterfsd(brick process) crashed when we do volume creates and deletes Product: GlusterFS Version: mainline Status: NEW Component: core Keywords: ZStream Severity: urgent Priority: high Assignee: bugs at gluster.org Reporter: moagrawa at redhat.com CC: bugs at gluster.org Depends On: 1662828 Target Milestone: --- Classification: Community Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1662828 [Bug 1662828] Longevity: glusterfsd(brick process) crashed when we do volume creates and deletes -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 11:10:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 11:10:59 +0000 Subject: [Bugs] [Bug 1662906] Longevity: glusterfsd(brick process) crashed when we do volume creates and deletes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662906 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21976 (core: brick process is crashed at the time of spawn thread) posted (#1) for review on master by MOHIT AGRAWAL -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 11:11:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 11:11:00 +0000 Subject: [Bugs] [Bug 1662906] Longevity: glusterfsd(brick process) crashed when we do volume creates and deletes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662906 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21976 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 11:28:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 11:28:10 +0000 Subject: [Bugs] [Bug 1654270] glusterd crashed with seg fault possibly during node reboot while volume creates and deletes were happening In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654270 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/21974 (glusterd: kill the process without releasing the cleanup mutex lock) posted (#1) for review on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 12:29:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 12:29:50 +0000 Subject: [Bugs] [Bug 1662838] FUSE mount seems to be hung and not accessible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662838 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(tdesala at redhat.co | |m) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 12:54:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 12:54:26 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #511 from Worker Ant --- REVIEW: https://review.gluster.org/21977 (timer-wheel: run the timer function outside of locked region) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 12:54:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 12:54:27 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21977 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 12:59:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 12:59:18 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #512 from Worker Ant --- REVIEW: https://review.gluster.org/21978 (syncop: move CALLOC -> MALLOC) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 12:59:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 12:59:19 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21978 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 2 13:16:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 13:16:56 +0000 Subject: [Bugs] [Bug 1662838] FUSE mount seems to be hung and not accessible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662838 Prasad Desala changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(tdesala at redhat.co | |m) | -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 13:40:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 13:40:47 +0000 Subject: [Bugs] [Bug 1138841] allow the use of the CIDR format with auth.allow In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1138841 --- Comment #6 from Worker Ant --- REVIEW: https://review.gluster.org/21980 (Modified few functions to isolate cidr feature) posted (#1) for review on master by Rinku Kothiya -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 2 13:40:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 02 Jan 2019 13:40:48 +0000 Subject: [Bugs] [Bug 1138841] allow the use of the CIDR format with auth.allow In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1138841 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21980 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 00:35:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 00:35:03 +0000 Subject: [Bugs] [Bug 1105277] Failure to execute gverify.sh. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1105277 --- Comment #8 from vnosov --- Her is some additional info about geo-replication failure to use log file /var/log/glusterfs/cli.log. This problem is exposed on geo-replication slave system. Log file /var/log/glusterfs/cli.log is created and updated by gluster that runs on slave system. It makes log file to havenext attributes: [root at SC-10-10-63-182 log]# ls -l /var/log/glusterfs/cli.log -rw------- 1 root root 72629 Dec 31 15:24 /var/log/glusterfs/cli.log If geo-replication is based on SSH access to the slave for not a "root" user, for example, "nasgorep" from group "nasgorep", all handling of the /var/log/glusterfs/cli.log on slave including slave's gluster are successful when log file has attributes: [root at SC-10-10-63-182 log]# ls -l /var/log/glusterfs/cli.log -rw-rw---- 1 root nasgorep 41553 Jan 2 16:00 /var/log/glusterfs/cli.log Problem is that GlusterFS 5.2 does not provide these settings for the log file or lets geo-replication use it now. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 04:00:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 04:00:44 +0000 Subject: [Bugs] [Bug 1663077] New: memory leak in mgmt handshake Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663077 Bug ID: 1663077 Summary: memory leak in mgmt handshake Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: zhhuan at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Found a memory leak in mgmt handling handshake. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 04:02:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 04:02:27 +0000 Subject: [Bugs] [Bug 1663077] memory leak in mgmt handshake In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663077 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21981 (glusterd: fix memory leak in handshake) posted (#1) for review on master by Zhang Huan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 04:02:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 04:02:28 +0000 Subject: [Bugs] [Bug 1663077] memory leak in mgmt handshake In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663077 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21981 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 05:26:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 05:26:06 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #513 from Worker Ant --- REVIEW: https://review.gluster.org/21982 (extras: Add readdir-ahead to samba group command) posted (#1) for review on master by Anoop C S -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 05:26:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 05:26:07 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21982 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 05:43:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 05:43:36 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 --- Comment #35 from Travers Carter --- I believe that I'm able to artificially trigger this using fs_mark, for example: mkdir /srv/gluster/fsmark cd /srv/gluster/fsmark fs_mark -L 500 -d $PWD -v -S 0 -D 128 -n 1000 -s $[8*1024] That's 500 rounds of 128 threads each creating and deleting 1000 8KiB files each in a per-thread subdirectory, where /srv/gluster is a gluster volume mounted with the fuse client -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 05:58:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 05:58:35 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 --- Comment #36 from Nithya Balachandran --- (In reply to Znamensky Pavel from comment #33) > (In reply to Nithya Balachandran from comment #31) > > Then it is likely to be because the fuse client does not invalidate inodes. > > Does your workload access a lot of files? The earlier statedump showed > > around 3 million inodes in memory. > > > >... > > > > https://review.gluster.org/#/c/glusterfs/+/19778/ has a fix to invalidate > > inodes but is not targeted for release 5 as yet. > > > Nithya, you're right! > I built glusterfs from the current master > (https://github.com/gluster/glusterfs/tree/ > d9a8ccd354df6db94477bf9ecb09735194523665) with the new invalidate inodes > mechanism that you mentioned before, and RSS memory consumption indeed > became much lower. > And as you supposed our apps quite often access a lot of files. > Here are two tests with clients on v6dev and v4.1 (the server is still on > v4.1 and read-ahead=on) > > The first test with default --lru-limit=0 (just did `find /in/big/dir -type > f`): > > v4.1 - ~3GB RSS: > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 633 6.6 18.5 3570216 3056136 ? Ssl 19:44 6:25 > /usr/sbin/glusterfs --read-only --process-name fuse --volfile-server=srv > --volfile-id=/st1 /mnt/st1 > > v6dev - ~1.5GB RSS: > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 10851 16.5 9.2 2071036 1526456 ? Ssl 19:45 15:50 > /usr/sbin/glusterfs --read-only --process-name fuse --volfile-server=srv > --volfile-id=/st1 /mnt/st1 > > It looks good. Let's do the next test. > The second test with --lru-limit=10_000 for v6dev: > > v4.1 - ~3GB RSS: > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 3589 4.7 18.6 3570216 3060364 ? Ssl 13:11 18:40 > /usr/sbin/glusterfs --process-name fuse --volfile-server=srv > --volfile-id=/st1 /mnt/st1 > > v6dev - ~170MB RSS: > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 24152 14.2 1.0 758768 173704 ? Ssl 13:58 49:06 > /usr/sbin/glusterfs --lru-limit=10000 --process-name fuse > --volfile-server=srv --volfile-id=/st1 /mnt/st1 > > 170MB vs. 3GB! > It's incredible! > Unfortunately, the new version has a drawback - CPU time increased 2.5x > times. At the moment it doesn't matter for us. > Anyway, I'm sure this change solves our problem. And of course, we're > looking forward to a stable version with it. > Thank you a lot! Thank you for testing this. I'm glad to hear the patch is working as expected to keep the memory use down. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 06:06:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 06:06:23 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|mchangir at redhat.com |sunkumar at redhat.com --- Comment #37 from Amar Tumballi --- > Unfortunately, the new version has a drawback - CPU time increased 2.5x > times. At the moment it doesn't matter for us. > Anyway, I'm sure this change solves our problem. And of course, we're > looking forward to a stable version with it. While a release with this patch merged/tested is another 50days away, we surely would like to reduce the CPU load you see too. Whenever you get time, if you can capture CPU info with below tool "perf record -ag --call-graph=dwarf -o perf.data -p ", and then see "perf report" to see what actually caused the CPU usage, it will help us to resolve that too. Also note, lru-limit=10000 while many files are accessed may not be a good value. I recommend something like 64k at least. But well, it depends on your memory needs too. So, if you can give 512MB - 1GB RAM for glusterfs, its better at least for performance. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 06:12:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 06:12:45 +0000 Subject: [Bugs] [Bug 1663089] New: Make GD2 container nightly and push it docker hub Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663089 Bug ID: 1663089 Summary: Make GD2 container nightly and push it docker hub Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: amukherj at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: During GCS scale testing effort, we identified couple of major issues in GD2 for which the PRs were posted and merged yesterday night, but apparently they missed the window of yesterday's nightly build and hence we're sort of blocked till today evening for picking up the GD2 container image. If we can build the container from the latest GD2 head and push it to docker hub right away, it'd be great and we should get unblocked. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 06:15:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 06:15:05 +0000 Subject: [Bugs] [Bug 1663089] Make GD2 container nightly and push it docker hub In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663089 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nigelb at redhat.com --- Comment #1 from Nigel Babu --- Did it make it to the GD2 nightly RPM build? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 07:00:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 07:00:46 +0000 Subject: [Bugs] [Bug 1663089] Make GD2 container nightly and push it docker hub In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663089 --- Comment #2 from Atin Mukherjee --- As per https://ci.centos.org/view/Gluster/job/gluster_gd2-nightly-rpms/ , it seems like the last build was 6 hours 49 minutes ago which means the required PRs should be in as part of the rpms. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 07:28:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 07:28:42 +0000 Subject: [Bugs] [Bug 1663102] New: Change default value for client side heal to off for replicate volumes Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Bug ID: 1663102 Summary: Change default value for client side heal to off for replicate volumes Product: GlusterFS Version: mainline Status: NEW Component: replicate Assignee: bugs at gluster.org Reporter: sheggodu at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: client-side heal on AFR volumes are slowing down systems when top-level directories need healing. Relying on server side heal by default keeps the system in the stable state. This bug is raised to set the default value for client-side heal to "off" for AFR volumes. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 07:29:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 07:29:01 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |sheggodu at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 07:37:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 07:37:22 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21938 (cluster/afr: Disable client side heals in AFR by default.) posted (#6) for review on master by Sunil Kumar Acharya -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 07:37:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 07:37:23 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21938 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 09:11:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:11:51 +0000 Subject: [Bugs] [Bug 1651323] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651323 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21985 --- Comment #12 from Worker Ant --- REVIEW: https://review.gluster.org/21985 (gfapi: Access fs->oldvolfile under mutex lock) posted (#1) for review on release-5 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:12:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:12:44 +0000 Subject: [Bugs] [Bug 1662838] FUSE mount seems to be hung and not accessible In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662838 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |UPSTREAM Last Closed| |2019-01-03 09:12:44 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 08:50:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 08:50:23 +0000 Subject: [Bugs] [Bug 1663077] memory leak in mgmt handshake In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663077 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21981 (glusterd: fix memory leak in handshake) posted (#1) for review on master by Zhang Huan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:27:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:27:46 +0000 Subject: [Bugs] [Bug 1660577] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660577 Soumya Koduri changed: What |Removed |Added ---------------------------------------------------------------------------- Version|4.1 |mainline -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 09:28:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:28:03 +0000 Subject: [Bugs] [Bug 1660577] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660577 Soumya Koduri changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |high Hardware|Unspecified |All OS|Unspecified |All -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 09:29:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:29:00 +0000 Subject: [Bugs] [Bug 1663131] New: [Ganesha] Ganesha failed on one node while exporting volumes in loop Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663131 Bug ID: 1663131 Summary: [Ganesha] Ganesha failed on one node while exporting volumes in loop Product: GlusterFS Version: 5 Hardware: All OS: All Status: NEW Component: libgfapi Keywords: ZStream Severity: high Priority: high Assignee: bugs at gluster.org Reporter: skoduri at redhat.com QA Contact: bugs at gluster.org CC: bugs at gluster.org Depends On: 1660577 Blocks: 1658132 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1660577 +++ +++ This bug was initially created as a clone of Bug #1658132 +++ Description of problem: ----------------------- ganesha entered failed state in one node of the four node cluster while exporting volumes in loop. Tried to export 109 volumes one after the other in loop. =============================================================================== Version-Release number of selected component (if applicable): ------------------------------------------------------------- nfs-ganesha-2.5.5-10.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-10.el7rhgs.x86_64 glusterfs-ganesha-3.12.2-28.el7rhgs.x86_64 =============================================================================== How reproducible: ----------------- 1/1 =============================================================================== Steps to Reproduce: ------------------- 1. Create 4 node ganesha cluster. 2. Create and start 100 or more volumes. 3. Verify status of all volumes. 4. Export volumes one after the other in a loop. =============================================================================== Actual results: --------------- Ganesha entered failed state in one of the nodes. =============================================================================== Expected results: ----------------- No failure should be observed. ============================================================================== Additional info: ---------------- * All volumes were exported on other 3 nodes in the 4 node cluster. * The failure observed is on a different node than the one from where export operation was executed. Setup is kept in same state and can be shared if required. --- Additional comment from Red Hat Bugzilla Rules Engine on 2018-12-11 10:35:37 UTC --- This bug is automatically being proposed for a Z-stream release of Red Hat Gluster Storage 3 under active development and open for bug fixes, by setting the release flag 'rhgs?3.4.z' to '?'. If this bug should be proposed for a different release, please manually change the proposed release flag. --- Additional comment from Jilju Joy on 2018-12-11 10:37:00 UTC --- Logs and sos report will be shared shortly. --- Additional comment from Jilju Joy on 2018-12-11 11:59:20 UTC --- Logs and sosreport : http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/jj/1658132/ --- Additional comment from Soumya Koduri on 2018-12-11 16:26:13 UTC --- (gdb) bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:74 #1 0x00007f5d18130664 in glfs_mgmt_getspec_cbk (req=, iov=, count=, myframe=0x7f5b74002cb0) at glfs-mgmt.c:625 #2 0x00007f5c8e9e8960 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f5a08cc5760, pollin=pollin at entry=0x7f5b7f09acb0) at rpc-clnt.c:778 #3 0x00007f5c8e9e8d03 in rpc_clnt_notify (trans=, mydata=0x7f5a08cc5790, event=, data=0x7f5b7f09acb0) at rpc-clnt.c:971 #4 0x00007f5c8e9e4a73 in rpc_transport_notify (this=this at entry=0x7f5a08cc5930, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f5b7f09acb0) at rpc-transport.c:538 #5 0x00007f5c849e5576 in socket_event_poll_in (this=this at entry=0x7f5a08cc5930, notify_handled=) at socket.c:2322 #6 0x00007f5c849e7b1c in socket_event_handler (fd=565, idx=0, gen=1, data=0x7f5a08cc5930, poll_in=1, poll_out=0, poll_err=0) at socket.c:2474 #7 0x00007f5c8ec7e824 in event_dispatch_epoll_handler (event=0x7f59e1f44500, event_pool=0x7f5a08cb74f0) at event-epoll.c:583 #8 event_dispatch_epoll_worker (data=0x7f5b760922a0) at event-epoll.c:659 #9 0x00007f5d20e44dd5 in start_thread (arg=0x7f59e1f45700) at pthread_create.c:307 #10 0x00007f5d2050fead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 (gdb) f 1 #1 0x00007f5d18130664 in glfs_mgmt_getspec_cbk (req=, iov=, count=, myframe=0x7f5b74002cb0) at glfs-mgmt.c:625 625 (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) { (gdb) l 620 621 ret = 0; 622 size = rsp.op_ret; 623 624 if ((size == fs->oldvollen) && 625 (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) { 626 gf_msg (frame->this->name, GF_LOG_INFO, 0, 627 API_MSG_VOLFILE_INFO, 628 "No change in volfile, continuing"); 629 goto out; (gdb) p fs->olvollen There is no member named olvollen. (gdb) p fs->oldvollen $1 = 1674 (gdb) p size $2 = 1674 (gdb) p fs->oldvolfile $3 = 0x7f5b76097cd0 "volume testvol82201-client-0\n type protocol/client\n option send-gids true\n option transport.socket.keepalive-count 9\n option transport.socket.keepalive-interval 2\n option transport.sock"... (gdb) p rsp.spec $4 = 0x7f5b7f9da9d0 "volume testvol82201-client-0\n type protocol/client\n option send-gids true\n option transport.socket.keepalive-count 9\n option transport.socket.keepalive-interval 2\n option transport.sock"... (gdb) The crash happened while doing memcmp of fs->oldvolfile and the new volfile received in the response (rsp.spec). The contents of both the variables seem fine in the core. >From code reading observed that we update fs->oldvollen and fs->oldvolfile under fs->mutex lock, but that lock is not taken while reading those values here in glfs_mgmt_spec_cbk. That could have resulted in the crash while accessing un/partially intialized variable. @Jilju, Are you able to consistently reproduce this issue? --- Additional comment from Daniel Gryniewicz on 2018-12-11 16:33:41 UTC --- Are the buffers smaller than 1674? It might be going off the end of one of the buffers. --- Additional comment from Jilju Joy on 2018-12-12 04:50:00 UTC --- (In reply to Soumya Koduri from comment #4) > (gdb) bt > #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:74 > #1 0x00007f5d18130664 in glfs_mgmt_getspec_cbk (req=, > iov=, count=, myframe=0x7f5b74002cb0) at > glfs-mgmt.c:625 > #2 0x00007f5c8e9e8960 in rpc_clnt_handle_reply > (clnt=clnt at entry=0x7f5a08cc5760, pollin=pollin at entry=0x7f5b7f09acb0) at > rpc-clnt.c:778 > #3 0x00007f5c8e9e8d03 in rpc_clnt_notify (trans=, > mydata=0x7f5a08cc5790, event=, data=0x7f5b7f09acb0) at > rpc-clnt.c:971 > #4 0x00007f5c8e9e4a73 in rpc_transport_notify > (this=this at entry=0x7f5a08cc5930, > event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, > data=data at entry=0x7f5b7f09acb0) at rpc-transport.c:538 > #5 0x00007f5c849e5576 in socket_event_poll_in > (this=this at entry=0x7f5a08cc5930, notify_handled=) at > socket.c:2322 > #6 0x00007f5c849e7b1c in socket_event_handler (fd=565, idx=0, gen=1, > data=0x7f5a08cc5930, poll_in=1, poll_out=0, poll_err=0) at socket.c:2474 > #7 0x00007f5c8ec7e824 in event_dispatch_epoll_handler > (event=0x7f59e1f44500, event_pool=0x7f5a08cb74f0) at event-epoll.c:583 > #8 event_dispatch_epoll_worker (data=0x7f5b760922a0) at event-epoll.c:659 > #9 0x00007f5d20e44dd5 in start_thread (arg=0x7f59e1f45700) at > pthread_create.c:307 > #10 0x00007f5d2050fead in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > (gdb) f 1 > #1 0x00007f5d18130664 in glfs_mgmt_getspec_cbk (req=, > iov=, count=, myframe=0x7f5b74002cb0) at > glfs-mgmt.c:625 > 625 (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) { > (gdb) l > 620 > 621 ret = 0; > 622 size = rsp.op_ret; > 623 > 624 if ((size == fs->oldvollen) && > 625 (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) { > 626 gf_msg (frame->this->name, GF_LOG_INFO, 0, > 627 API_MSG_VOLFILE_INFO, > 628 "No change in volfile, continuing"); > 629 goto out; > (gdb) p fs->olvollen > There is no member named olvollen. > (gdb) p fs->oldvollen > $1 = 1674 > (gdb) p size > $2 = 1674 > (gdb) p fs->oldvolfile > $3 = 0x7f5b76097cd0 "volume testvol82201-client-0\n type > protocol/client\n option send-gids true\n option > transport.socket.keepalive-count 9\n option > transport.socket.keepalive-interval 2\n option transport.sock"... > (gdb) p rsp.spec > $4 = 0x7f5b7f9da9d0 "volume testvol82201-client-0\n type > protocol/client\n option send-gids true\n option > transport.socket.keepalive-count 9\n option > transport.socket.keepalive-interval 2\n option transport.sock"... > (gdb) > > > The crash happened while doing memcmp of fs->oldvolfile and the new volfile > received in the response (rsp.spec). The contents of both the variables seem > fine in the core. > > From code reading observed that we update fs->oldvollen and fs->oldvolfile > under fs->mutex lock, but that lock is not taken while reading those values > here in glfs_mgmt_spec_cbk. That could have resulted in the crash while > accessing un/partially intialized variable. > > @Jilju, > > Are you able to consistently reproduce this issue? Hi Soumya, The first occurrence is reported here. Kept the setup in same state for the favour of debugging. I can share the setup if required or I can try to reproduce. --- Additional comment from Worker Ant on 2018-12-18 17:05:42 UTC --- REVIEW: https://review.gluster.org/21882 (gfapi: Access fs->oldvolfile under mutex lock) posted (#1) for review on master by soumya k --- Additional comment from Worker Ant on 2018-12-26 02:17:03 UTC --- REVIEW: https://review.gluster.org/21882 (gfapi: Access fs->oldvolfile under mutex lock) posted (#2) for review on master by Amar Tumballi --- Additional comment from Worker Ant on 2018-12-26 10:33:07 UTC --- REVIEW: https://review.gluster.org/21927 (gfapi: nit cleanup related to releasing fs->mutex lock) posted (#1) for review on master by soumya k --- Additional comment from Worker Ant on 2018-12-31 16:10:41 UTC --- REVIEW: https://review.gluster.org/21927 (gfapi: nit cleanup related to releasing fs->mutex lock) posted (#2) for review on master by Kaleb KEITHLEY Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1658132 [Bug 1658132] [Ganesha] Ganesha failed on one node while exporting volumes in loop https://bugzilla.redhat.com/show_bug.cgi?id=1660577 [Bug 1660577] [Ganesha] Ganesha failed on one node while exporting volumes in loop -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:29:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:29:00 +0000 Subject: [Bugs] [Bug 1660577] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660577 Soumya Koduri changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1663131 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1663131 [Bug 1663131] [Ganesha] Ganesha failed on one node while exporting volumes in loop -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 09:29:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:29:28 +0000 Subject: [Bugs] [Bug 1663132] New: [Ganesha] Ganesha failed on one node while exporting volumes in loop Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663132 Bug ID: 1663132 Summary: [Ganesha] Ganesha failed on one node while exporting volumes in loop Product: GlusterFS Version: 4.1 Hardware: All OS: All Status: NEW Component: libgfapi Keywords: ZStream Severity: high Priority: high Assignee: bugs at gluster.org Reporter: skoduri at redhat.com QA Contact: bugs at gluster.org CC: bugs at gluster.org Depends On: 1660577 Blocks: 1658132, 1663131 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1660577 +++ +++ This bug was initially created as a clone of Bug #1658132 +++ Description of problem: ----------------------- ganesha entered failed state in one node of the four node cluster while exporting volumes in loop. Tried to export 109 volumes one after the other in loop. =============================================================================== Version-Release number of selected component (if applicable): ------------------------------------------------------------- nfs-ganesha-2.5.5-10.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-10.el7rhgs.x86_64 glusterfs-ganesha-3.12.2-28.el7rhgs.x86_64 =============================================================================== How reproducible: ----------------- 1/1 =============================================================================== Steps to Reproduce: ------------------- 1. Create 4 node ganesha cluster. 2. Create and start 100 or more volumes. 3. Verify status of all volumes. 4. Export volumes one after the other in a loop. =============================================================================== Actual results: --------------- Ganesha entered failed state in one of the nodes. =============================================================================== Expected results: ----------------- No failure should be observed. ============================================================================== Additional info: ---------------- * All volumes were exported on other 3 nodes in the 4 node cluster. * The failure observed is on a different node than the one from where export operation was executed. Setup is kept in same state and can be shared if required. --- Additional comment from Red Hat Bugzilla Rules Engine on 2018-12-11 10:35:37 UTC --- This bug is automatically being proposed for a Z-stream release of Red Hat Gluster Storage 3 under active development and open for bug fixes, by setting the release flag 'rhgs?3.4.z' to '?'. If this bug should be proposed for a different release, please manually change the proposed release flag. --- Additional comment from Jilju Joy on 2018-12-11 10:37:00 UTC --- Logs and sos report will be shared shortly. --- Additional comment from Jilju Joy on 2018-12-11 11:59:20 UTC --- Logs and sosreport : http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/jj/1658132/ --- Additional comment from Soumya Koduri on 2018-12-11 16:26:13 UTC --- (gdb) bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:74 #1 0x00007f5d18130664 in glfs_mgmt_getspec_cbk (req=, iov=, count=, myframe=0x7f5b74002cb0) at glfs-mgmt.c:625 #2 0x00007f5c8e9e8960 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f5a08cc5760, pollin=pollin at entry=0x7f5b7f09acb0) at rpc-clnt.c:778 #3 0x00007f5c8e9e8d03 in rpc_clnt_notify (trans=, mydata=0x7f5a08cc5790, event=, data=0x7f5b7f09acb0) at rpc-clnt.c:971 #4 0x00007f5c8e9e4a73 in rpc_transport_notify (this=this at entry=0x7f5a08cc5930, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f5b7f09acb0) at rpc-transport.c:538 #5 0x00007f5c849e5576 in socket_event_poll_in (this=this at entry=0x7f5a08cc5930, notify_handled=) at socket.c:2322 #6 0x00007f5c849e7b1c in socket_event_handler (fd=565, idx=0, gen=1, data=0x7f5a08cc5930, poll_in=1, poll_out=0, poll_err=0) at socket.c:2474 #7 0x00007f5c8ec7e824 in event_dispatch_epoll_handler (event=0x7f59e1f44500, event_pool=0x7f5a08cb74f0) at event-epoll.c:583 #8 event_dispatch_epoll_worker (data=0x7f5b760922a0) at event-epoll.c:659 #9 0x00007f5d20e44dd5 in start_thread (arg=0x7f59e1f45700) at pthread_create.c:307 #10 0x00007f5d2050fead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 (gdb) f 1 #1 0x00007f5d18130664 in glfs_mgmt_getspec_cbk (req=, iov=, count=, myframe=0x7f5b74002cb0) at glfs-mgmt.c:625 625 (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) { (gdb) l 620 621 ret = 0; 622 size = rsp.op_ret; 623 624 if ((size == fs->oldvollen) && 625 (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) { 626 gf_msg (frame->this->name, GF_LOG_INFO, 0, 627 API_MSG_VOLFILE_INFO, 628 "No change in volfile, continuing"); 629 goto out; (gdb) p fs->olvollen There is no member named olvollen. (gdb) p fs->oldvollen $1 = 1674 (gdb) p size $2 = 1674 (gdb) p fs->oldvolfile $3 = 0x7f5b76097cd0 "volume testvol82201-client-0\n type protocol/client\n option send-gids true\n option transport.socket.keepalive-count 9\n option transport.socket.keepalive-interval 2\n option transport.sock"... (gdb) p rsp.spec $4 = 0x7f5b7f9da9d0 "volume testvol82201-client-0\n type protocol/client\n option send-gids true\n option transport.socket.keepalive-count 9\n option transport.socket.keepalive-interval 2\n option transport.sock"... (gdb) The crash happened while doing memcmp of fs->oldvolfile and the new volfile received in the response (rsp.spec). The contents of both the variables seem fine in the core. >From code reading observed that we update fs->oldvollen and fs->oldvolfile under fs->mutex lock, but that lock is not taken while reading those values here in glfs_mgmt_spec_cbk. That could have resulted in the crash while accessing un/partially intialized variable. @Jilju, Are you able to consistently reproduce this issue? --- Additional comment from Daniel Gryniewicz on 2018-12-11 16:33:41 UTC --- Are the buffers smaller than 1674? It might be going off the end of one of the buffers. --- Additional comment from Jilju Joy on 2018-12-12 04:50:00 UTC --- (In reply to Soumya Koduri from comment #4) > (gdb) bt > #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:74 > #1 0x00007f5d18130664 in glfs_mgmt_getspec_cbk (req=, > iov=, count=, myframe=0x7f5b74002cb0) at > glfs-mgmt.c:625 > #2 0x00007f5c8e9e8960 in rpc_clnt_handle_reply > (clnt=clnt at entry=0x7f5a08cc5760, pollin=pollin at entry=0x7f5b7f09acb0) at > rpc-clnt.c:778 > #3 0x00007f5c8e9e8d03 in rpc_clnt_notify (trans=, > mydata=0x7f5a08cc5790, event=, data=0x7f5b7f09acb0) at > rpc-clnt.c:971 > #4 0x00007f5c8e9e4a73 in rpc_transport_notify > (this=this at entry=0x7f5a08cc5930, > event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, > data=data at entry=0x7f5b7f09acb0) at rpc-transport.c:538 > #5 0x00007f5c849e5576 in socket_event_poll_in > (this=this at entry=0x7f5a08cc5930, notify_handled=) at > socket.c:2322 > #6 0x00007f5c849e7b1c in socket_event_handler (fd=565, idx=0, gen=1, > data=0x7f5a08cc5930, poll_in=1, poll_out=0, poll_err=0) at socket.c:2474 > #7 0x00007f5c8ec7e824 in event_dispatch_epoll_handler > (event=0x7f59e1f44500, event_pool=0x7f5a08cb74f0) at event-epoll.c:583 > #8 event_dispatch_epoll_worker (data=0x7f5b760922a0) at event-epoll.c:659 > #9 0x00007f5d20e44dd5 in start_thread (arg=0x7f59e1f45700) at > pthread_create.c:307 > #10 0x00007f5d2050fead in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > (gdb) f 1 > #1 0x00007f5d18130664 in glfs_mgmt_getspec_cbk (req=, > iov=, count=, myframe=0x7f5b74002cb0) at > glfs-mgmt.c:625 > 625 (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) { > (gdb) l > 620 > 621 ret = 0; > 622 size = rsp.op_ret; > 623 > 624 if ((size == fs->oldvollen) && > 625 (memcmp (fs->oldvolfile, rsp.spec, size) == 0)) { > 626 gf_msg (frame->this->name, GF_LOG_INFO, 0, > 627 API_MSG_VOLFILE_INFO, > 628 "No change in volfile, continuing"); > 629 goto out; > (gdb) p fs->olvollen > There is no member named olvollen. > (gdb) p fs->oldvollen > $1 = 1674 > (gdb) p size > $2 = 1674 > (gdb) p fs->oldvolfile > $3 = 0x7f5b76097cd0 "volume testvol82201-client-0\n type > protocol/client\n option send-gids true\n option > transport.socket.keepalive-count 9\n option > transport.socket.keepalive-interval 2\n option transport.sock"... > (gdb) p rsp.spec > $4 = 0x7f5b7f9da9d0 "volume testvol82201-client-0\n type > protocol/client\n option send-gids true\n option > transport.socket.keepalive-count 9\n option > transport.socket.keepalive-interval 2\n option transport.sock"... > (gdb) > > > The crash happened while doing memcmp of fs->oldvolfile and the new volfile > received in the response (rsp.spec). The contents of both the variables seem > fine in the core. > > From code reading observed that we update fs->oldvollen and fs->oldvolfile > under fs->mutex lock, but that lock is not taken while reading those values > here in glfs_mgmt_spec_cbk. That could have resulted in the crash while > accessing un/partially intialized variable. > > @Jilju, > > Are you able to consistently reproduce this issue? Hi Soumya, The first occurrence is reported here. Kept the setup in same state for the favour of debugging. I can share the setup if required or I can try to reproduce. --- Additional comment from Worker Ant on 2018-12-18 17:05:42 UTC --- REVIEW: https://review.gluster.org/21882 (gfapi: Access fs->oldvolfile under mutex lock) posted (#1) for review on master by soumya k --- Additional comment from Worker Ant on 2018-12-26 02:17:03 UTC --- REVIEW: https://review.gluster.org/21882 (gfapi: Access fs->oldvolfile under mutex lock) posted (#2) for review on master by Amar Tumballi --- Additional comment from Worker Ant on 2018-12-26 10:33:07 UTC --- REVIEW: https://review.gluster.org/21927 (gfapi: nit cleanup related to releasing fs->mutex lock) posted (#1) for review on master by soumya k --- Additional comment from Worker Ant on 2018-12-31 16:10:41 UTC --- REVIEW: https://review.gluster.org/21927 (gfapi: nit cleanup related to releasing fs->mutex lock) posted (#2) for review on master by Kaleb KEITHLEY Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1658132 [Bug 1658132] [Ganesha] Ganesha failed on one node while exporting volumes in loop https://bugzilla.redhat.com/show_bug.cgi?id=1660577 [Bug 1660577] [Ganesha] Ganesha failed on one node while exporting volumes in loop https://bugzilla.redhat.com/show_bug.cgi?id=1663131 [Bug 1663131] [Ganesha] Ganesha failed on one node while exporting volumes in loop -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:29:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:29:28 +0000 Subject: [Bugs] [Bug 1660577] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660577 Soumya Koduri changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1663132 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1663132 [Bug 1663132] [Ganesha] Ganesha failed on one node while exporting volumes in loop -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 09:29:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:29:28 +0000 Subject: [Bugs] [Bug 1663131] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663131 Soumya Koduri changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1663132 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1663132 [Bug 1663132] [Ganesha] Ganesha failed on one node while exporting volumes in loop -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:31:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:31:12 +0000 Subject: [Bugs] [Bug 1651323] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651323 --- Comment #13 from Worker Ant --- REVISION POSTED: https://review.gluster.org/21985 (gfapi: Access fs->oldvolfile under mutex lock) posted (#3) for review on release-5 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:31:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:31:13 +0000 Subject: [Bugs] [Bug 1651323] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651323 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Gluster.org Gerrit 21985 | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:31:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:31:14 +0000 Subject: [Bugs] [Bug 1663131] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663131 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21985 (gfapi: Access fs->oldvolfile under mutex lock) posted (#3) for review on release-5 by soumya k -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:31:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:31:15 +0000 Subject: [Bugs] [Bug 1663131] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663131 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21985 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:44:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:44:17 +0000 Subject: [Bugs] [Bug 1663132] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663132 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21986 (gfapi: Access fs->oldvolfile under mutex lock) posted (#1) for review on release-4.1 by soumya k -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:44:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:44:18 +0000 Subject: [Bugs] [Bug 1663132] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663132 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21986 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:58:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:58:23 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #514 from Worker Ant --- REVIEW: https://review.gluster.org/21987 (glfs-fops.c: fix the bad string length for snprintf) posted (#1) for review on master by Kinglong Mee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 09:58:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 09:58:28 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21987 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 10:07:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 10:07:56 +0000 Subject: [Bugs] [Bug 1663089] Make GD2 container nightly and push it docker hub In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663089 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-03 10:07:56 --- Comment #3 from Nigel Babu --- ALright, Deepshika retriggerd the Jenkins job and we're good now. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 11:03:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 11:03:57 +0000 Subject: [Bugs] [Bug 1657743] Very high memory usage (25GB) on Gluster FUSE mountpoint In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657743 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nbalacha at redhat.com Assignee|bugs at gluster.org |sunkumar at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 11:12:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 11:12:44 +0000 Subject: [Bugs] [Bug 1662557] glusterfs process crashes, causing "Transport endpoint not connected". In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662557 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nbalacha at redhat.com, | |rob.dewit at coosto.com Flags| |needinfo?(rob.dewit at coosto. | |com) --- Comment #2 from Nithya Balachandran --- Can you try installing the debuginfo packages for the gluster version you are running and rerun bt on the core dump? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 11:41:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 11:41:23 +0000 Subject: [Bugs] [Bug 1662557] glusterfs process crashes, causing "Transport endpoint not connected". In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662557 robdewit changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(rob.dewit at coosto. | |com) | --- Comment #3 from robdewit --- (gdb) bt #0 0x00007fe0a5936e30 in pthread_mutex_lock () from /lib64/libpthread.so.0 #1 0x00007fe0a6b0c795 in __gf_free (free_ptr=0x7fe0843ac610) at mem-pool.c:333 #2 0x00007fe0a6ad51ee in dict_destroy (this=0x7fe0843abe78) at dict.c:701 #3 0x00007fe0a6ad5315 in dict_unref (this=) at dict.c:753 #4 0x00007fe0a0866124 in afr_local_cleanup (local=0x7fe0843ade18, this=) at afr-common.c:2091 #5 0x00007fe0a083fee1 in afr_transaction_done (frame=, this=) at afr-transaction.c:369 #6 0x00007fe0a08437f1 in afr_unlock_common_cbk (frame=frame at entry=0x7fe0843ac7b8, this=this at entry=0x7fe09c0110c0, op_ret=op_ret at entry=0, xdata=, op_errno=, cookie=) at afr-lk-common.c:243 #7 0x00007fe0a0844562 in afr_unlock_inodelk_cbk (frame=0x7fe0843ac7b8, cookie=, this=0x7fe09c0110c0, op_ret=0, op_errno=, xdata=) at afr-lk-common.c:281 #8 0x00007fe0a0b101d0 in client4_0_finodelk_cbk (req=, iov=, count=, myframe=) at client-rpc-fops_v2.c:1398 #9 0x00007fe0a68ae534 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7fe09c053bd0, pollin=pollin at entry=0x7fe09c115750) at rpc-clnt.c:755 #10 0x00007fe0a68aee77 in rpc_clnt_notify (trans=0x7fe09c053e90, mydata=0x7fe09c053c00, event=, data=0x7fe09c115750) at rpc-clnt.c:923 #11 0x00007fe0a68aaf13 in rpc_transport_notify (this=this at entry=0x7fe09c053e90, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7fe09c115750) at rpc-transport.c:525 #12 0x00007fe0a19c2a23 in socket_event_poll_in (notify_handled=true, this=0x7fe09c053e90) at socket.c:2504 #13 socket_event_handler (fd=-1676585136, idx=1, gen=4, data=0x7fe09c053e90, poll_in=, poll_out=, poll_err=0) at socket.c:2905 #14 0x00007fe0a6b43aeb in event_dispatch_epoll_handler (event=0x7fe0a1531ed0, event_pool=0x17f40b0) at event-epoll.c:591 #15 event_dispatch_epoll_worker (data=0x1830840) at event-epoll.c:668 #16 0x00007fe0a5934504 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fe0a521c19f in clone () from /lib64/libc.so.6 Somehow version in this bug report has been reset to 3.12, but this is actually version 5.2 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 11:42:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 11:42:22 +0000 Subject: [Bugs] [Bug 1654103] Invalid memory read after freed in dht_rmdir_readdirp_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654103 --- Comment #9 from Sayalee --- Ran the planned test cases in the test plan shared in Comment8 and didn't see any issues on glusterfs version 3.12.2-34 Moving this BZ to Verified. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 11:42:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 11:42:35 +0000 Subject: [Bugs] [Bug 1654103] Invalid memory read after freed in dht_rmdir_readdirp_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654103 Sayalee changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ON_QA |VERIFIED -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 12:16:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 12:16:10 +0000 Subject: [Bugs] [Bug 1662557] glusterfs process crashes, causing "Transport endpoint not connected". In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662557 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Component|fuse |replicate Version|3.12 |5 --- Comment #4 from Nithya Balachandran --- Assigning this to AFR team. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 12:20:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 12:20:11 +0000 Subject: [Bugs] [Bug 1662557] glusterfs process crashes, causing "Transport endpoint not connected". In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662557 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ravishankar at redhat.com Flags| |needinfo?(rob.dewit at coosto. | |com) --- Comment #5 from Ravishankar N --- Quick question: Is the back trace identical to what is shared in comment #3 for all crashes? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 12:22:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 12:22:53 +0000 Subject: [Bugs] [Bug 1663205] New: List dictionary is too slow Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663205 Bug ID: 1663205 Summary: List dictionary is too slow Product: GlusterFS Version: 4.1 Hardware: x86_64 OS: Linux Status: NEW Component: fuse Severity: high Assignee: bugs at gluster.org Reporter: 1490889344 at qq.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: I create a distribute volume and mount to a dictionary. Then I put 25,000 files to the dictionary. After finished, I write a program to list the dictionary. I found the program spent 20s. It's unbelievable. And I copy the dictionary to root dictionary. And running the program again. The time display just less than 1s. So I think there is some problems in glusterFS. Then I do some more test for glusterFS. I found that the spent time is normal when the dictionary contains 20,000 files, but when the number is more than 20,000, it's easy to show bad performance. Finally, I found the reason of bad performance is stat function for every file. I don't know why the stat function is spent lots of time when the dictionary contains 25,000 files. I hope someone can help me. GlusterFS vesion: glusterfs 4.1.6 Volume info: Volume Name: gv0 Type: Distribute Volume ID: 7cfccb92-5b9d-4483-8212-0f02cd1197d6 Status: Started Snapshot Count: 0 Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: dlaas-184:/data/glusterFS/gv0 Options Reconfigured: transport.address-family: inet nfs.disable: on -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 12:23:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 12:23:26 +0000 Subject: [Bugs] [Bug 1662557] glusterfs process crashes, causing "Transport endpoint not connected". In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662557 --- Comment #6 from Ravishankar N --- Also, please attach the core file to the bug. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 12:25:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 12:25:15 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 SATHEESARAN changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1663208 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1663208 [Bug 1663208] [RHV-RHGS] Fuse mount crashed while deleting a 1 TB image file from RHV -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 12:32:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 12:32:51 +0000 Subject: [Bugs] [Bug 1662557] glusterfs process crashes, causing "Transport endpoint not connected". In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662557 robdewit changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(rob.dewit at coosto. | |com) | --- Comment #7 from robdewit --- Good question! It turns out this is not always the case. I checked some other coredumps: coredump 1 - same backtrace coredump 2 - untraceable coredump 3 - A different backtrace: (gdb) bt #0 0x00007f2f2a1d32c0 in ?? () from /lib64/libuuid.so.1 #1 0x00007f2f2a1d24e0 in uuid_compare () from /lib64/libuuid.so.1 #2 0x00007f2f2aa57691 in gf_uuid_compare (u2=0x7f2f115e67f8 "\270x\274\226Z\301F\006\256\221\230\005\031\321N\342\001", u1=0x7f2eff90
) at compat-uuid.h:25 #3 __inode_find (table=table at entry=0x7f2f20063b80, gfid=gfid at entry=0x7f2f115e67f8 "\270x\274\226Z\301F\006\256\221\230\005\031\321N\342\001") at inode.c:892 #4 0x00007f2f2aa57d79 in inode_find (table=table at entry=0x7f2f20063b80, gfid=gfid at entry=0x7f2f115e67f8 "\270x\274\226Z\301F\006\256\221\230\005\031\321N\342\001") at inode.c:917 #5 0x00007f2f24a1ae72 in unserialize_rsp_direntp_v2 (this=0x7f2f2000e980, fd=, rsp=rsp at entry=0x7f2f1e164a70, entries=0x7f2f1e164aa0) at client-helpers.c:338 #6 0x00007f2f24a59005 in client_post_readdirp_v2 (this=, rsp=0x7f2f1e164a70, fd=, entries=, xdata=0x7f2f1e164a68) at client-common.c:3533 #7 0x00007f2f24a6b226 in client4_0_readdirp_cbk (req=, iov=0x7f2f0b99d508, count=, myframe=0x7f2ef4a691f8) at client-rpc-fops_v2.c:2333 #8 0x00007f2f2a814534 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f2f2004f530, pollin=pollin at entry=0x7f2f114a8290) at rpc-clnt.c:755 #9 0x00007f2f2a814e77 in rpc_clnt_notify (trans=0x7f2f2004f860, mydata=0x7f2f2004f560, event=, data=0x7f2f114a8290) at rpc-clnt.c:923 #10 0x00007f2f2a810f13 in rpc_transport_notify (this=this at entry=0x7f2f2004f860, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f2f114a8290) at rpc-transport.c:525 #11 0x00007f2f25928a23 in socket_event_poll_in (notify_handled=true, this=0x7f2f2004f860) at socket.c:2504 #12 socket_event_handler (fd=290095760, idx=2, gen=4, data=0x7f2f2004f860, poll_in=, poll_out=, poll_err=0) at socket.c:2905 #13 0x00007f2f2aaa9aeb in event_dispatch_epoll_handler (event=0x7f2f1e164ed0, event_pool=0x7510b0) at event-epoll.c:591 #14 event_dispatch_epoll_worker (data=0x7f2f2004f310) at event-epoll.c:668 #15 0x00007f2f2989a504 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f2f2918219f in clone () from /lib64/libc.so.6 coredump 4 - yet another backtrace: (gdb) bt #0 0x00007ff2249a58a4 in _int_free () from /lib64/libc.so.6 #1 0x00007ff2249aac9e in free () from /lib64/libc.so.6 #2 0x00007ff22631d6af in __gf_free (free_ptr=) at mem-pool.c:356 #3 0x00007ff223bf1410 in free_fuse_state (state=0x7ff1f4760430) at fuse-helpers.c:81 #4 0x00007ff223bf70a9 in fuse_err_cbk (frame=0x7ff1f471b1d8, cookie=, this=0x18dddb0, op_ret=0, op_errno=0, xdata=) at fuse-bridge.c:1434 #5 0x00007ff21aebc29d in io_stats_flush_cbk (frame=0x7ff206a0b088, cookie=, this=, op_ret=0, op_errno=0, xdata=0x0) at io-stats.c:2286 #6 0x00007ff226385b29 in default_flush_cbk (frame=0x7ff1f4737f58, cookie=, this=, op_ret=0, op_errno=0, xdata=0x0) at defaults.c:1159 #7 0x00007ff21b926f77 in ra_flush_cbk (frame=0x7ff1f4737238, cookie=, this=, op_ret=0, op_errno=0, xdata=0x0) at read-ahead.c:539 #8 0x00007ff21bb390dd in wb_flush_helper (frame=0x7ff2071e7488, this=, fd=, xdata=0x0) at write-behind.c:1987 #9 0x00007ff22631a055 in call_resume_keep_stub (stub=0x7ff1f4744da8) at call-stub.c:2563 #10 0x00007ff21bb3c999 in wb_do_winds (wb_inode=wb_inode at entry=0x7ff1f4742730, tasks=tasks at entry=0x7ff220d42640) at write-behind.c:1737 #11 0x00007ff21bb3ca9c in wb_process_queue (wb_inode=wb_inode at entry=0x7ff1f4742730) at write-behind.c:1778 #12 0x00007ff21bb41a07 in wb_fulfill_cbk (frame=frame at entry=0x7ff21d48e7c8, cookie=, this=, op_ret=op_ret at entry=123, op_errno=op_errno at entry=0, prebuf=prebuf at entry=0x7ff21d4ac610, postbuf=postbuf at entry=0x7ff21d4ac6a8, xdata=xdata at entry=0x7ff21d490168) at write-behind.c:1105 #13 0x00007ff21bdbde86 in dht_writev_cbk (frame=frame at entry=0x7ff21c08d7c8, cookie=, this=, op_ret=123, op_errno=0, prebuf=prebuf at entry=0x7ff21d4ac610, postbuf=postbuf at entry=0x7ff21d4ac6a8, xdata=0x7ff21d490168) at dht-inode-write.c:140 #14 0x00007ff22003e21e in afr_writev_unwind (frame=frame at entry=0x7ff21d4a3ee8, this=this at entry=0x7ff21c0110c0) at afr-inode-write.c:234 #15 0x00007ff22003e7e6 in afr_writev_wind_cbk (this=0x7ff21c0110c0, frame=0x7ff21d49ab08, cookie=, op_ret=, op_errno=, prebuf=, postbuf=, xdata=) at afr-inode-write.c:388 #16 afr_writev_wind_cbk (frame=0x7ff21d49ab08, cookie=, this=0x7ff21c0110c0, op_ret=, op_errno=, prebuf=, postbuf=0x7ff220d42980, xdata=0x7ff21d49ae58) at afr-inode-write.c:354 #17 0x00007ff220313748 in client4_0_writev_cbk (req=, iov=, count=, myframe=0x7ff21d483a58) at client-rpc-fops_v2.c:685 #18 0x00007ff2260bf534 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7ff21c04f530, pollin=pollin at entry=0x7ff21d49e650) at rpc-clnt.c:755 #19 0x00007ff2260bfe77 in rpc_clnt_notify (trans=0x7ff21c04f860, mydata=0x7ff21c04f560, event=, data=0x7ff21d49e650) at rpc-clnt.c:923 #20 0x00007ff2260bbf13 in rpc_transport_notify (this=this at entry=0x7ff21c04f860, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7ff21d49e650) at rpc-transport.c:525 #21 0x00007ff2211d3a23 in socket_event_poll_in (notify_handled=true, this=0x7ff21c04f860) at socket.c:2504 #22 socket_event_handler (fd=491382352, idx=2, gen=4, data=0x7ff21c04f860, poll_in=, poll_out=, poll_err=0) at socket.c:2905 #23 0x00007ff226354aeb in event_dispatch_epoll_handler (event=0x7ff220d42ed0, event_pool=0x18d70b0) at event-epoll.c:591 #24 event_dispatch_epoll_worker (data=0x1913840) at event-epoll.c:668 #25 0x00007ff225145504 in start_thread () from /lib64/libpthread.so.0 #26 0x00007ff224a2d19f in clone () from /lib64/libc.so.6 coredump 5 - Another one: (gdb) bt #0 0x00007fad93d3ce30 in pthread_mutex_lock () from /lib64/libpthread.so.0 #1 0x00007fad94eea73e in gf_log_set_log_buf_size (buf_size=buf_size at entry=0) at logging.c:273 #2 0x00007fad94eea8df in gf_log_disable_suppression_before_exit (ctx=0x1334010) at logging.c:444 #3 0x00007fad94ef0f94 in gf_print_trace (signum=11, ctx=0x1334010) at common-utils.c:922 #4 #5 0x00007fad94f0fd52 in fd_destroy (bound=true, fd=0x7fad64f216c8) at fd.c:478 #6 fd_unref (fd=0x7fad64f216c8) at fd.c:529 #7 0x00007fad8eeba0e8 in client_local_wipe (local=local at entry=0x7fad8a924358) at client-helpers.c:124 #8 0x00007fad8ef161e0 in client4_0_finodelk_cbk (req=, iov=, count=, myframe=) at client-rpc-fops_v2.c:1398 #9 0x00007fad94cb4534 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7fad8804f530, pollin=pollin at entry=0x7fad8a917950) at rpc-clnt.c:755 #10 0x00007fad94cb4e77 in rpc_clnt_notify (trans=0x7fad8804f860, mydata=0x7fad8804f560, event=, data=0x7fad8a917950) at rpc-clnt.c:923 #11 0x00007fad94cb0f13 in rpc_transport_notify (this=this at entry=0x7fad8804f860, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7fad8a917950) at rpc-transport.c:525 #12 0x00007fad8fdc8a23 in socket_event_poll_in (notify_handled=true, this=0x7fad8804f860) at socket.c:2504 #13 socket_event_handler (fd=-1970177712, idx=2, gen=4, data=0x7fad8804f860, poll_in=, poll_out=, poll_err=0) at socket.c:2905 #14 0x00007fad94f49aeb in event_dispatch_epoll_handler (event=0x7fad8f937ed0, event_pool=0x136b0b0) at event-epoll.c:591 #15 event_dispatch_epoll_worker (data=0x13a7840) at event-epoll.c:668 #16 0x00007fad93d3a504 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fad9362219f in clone () from /lib64/libc.so.6 coredump 6 - And another: (gdb) bt #0 0x00007f3c6caace30 in pthread_mutex_lock () from /lib64/libpthread.so.0 #1 0x00007f3c6dc82795 in __gf_free (free_ptr=0x7f3c39e43cb0) at mem-pool.c:333 #2 0x00007f3c6dc65d90 in __inode_ctx_free (inode=inode at entry=0x7f3c39e435a8) at inode.c:322 #3 0x00007f3c6dc66e12 in __inode_destroy (inode=0x7f3c39e435a8) at inode.c:338 #4 inode_table_prune (table=table at entry=0x7f3c58010950) at inode.c:1535 #5 0x00007f3c6dc671ec in inode_unref (inode=0x7f3c39e435a8) at inode.c:542 #6 0x00007f3c679dbf97 in afr_local_cleanup (local=0x7f3c39e1f3e8, this=) at afr-common.c:1995 #7 0x00007f3c679b5ee1 in afr_transaction_done (frame=, this=) at afr-transaction.c:369 #8 0x00007f3c679b97f1 in afr_unlock_common_cbk (frame=frame at entry=0x7f3c3a11d168, this=this at entry=0x7f3c600110c0, op_ret=op_ret at entry=0, xdata=0x0, op_errno=, cookie=) at afr-lk-common.c:243 #9 0x00007f3c679b98ae in afr_unlock_entrylk_cbk (frame=0x7f3c3a11d168, cookie=, this=0x7f3c600110c0, op_ret=0, op_errno=, xdata=) at afr-lk-common.c:366 #10 0x00007f3c67c857bd in client4_0_entrylk_cbk (req=, iov=, count=, myframe=) at client-rpc-fops_v2.c:1446 #11 0x00007f3c6da24534 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f3c60058e20, pollin=pollin at entry=0x7f3c5a5898e0) at rpc-clnt.c:755 #12 0x00007f3c6da24e77 in rpc_clnt_notify (trans=0x7f3c600590e0, mydata=0x7f3c60058e50, event=, data=0x7f3c5a5898e0) at rpc-clnt.c:923 #13 0x00007f3c6da20f13 in rpc_transport_notify (this=this at entry=0x7f3c600590e0, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f3c5a5898e0) at rpc-transport.c:525 #14 0x00007f3c68b38a23 in socket_event_poll_in (notify_handled=true, this=0x7f3c600590e0) at socket.c:2504 #15 socket_event_handler (fd=1515755744, idx=4, gen=1, data=0x7f3c600590e0, poll_in=, poll_out=, poll_err=0) at socket.c:2905 #16 0x00007f3c6dcb9aeb in event_dispatch_epoll_handler (event=0x7f3c65c04ed0, event_pool=0x81a0b0) at event-epoll.c:591 #17 event_dispatch_epoll_worker (data=0x7f3c60043ad0) at event-epoll.c:668 #18 0x00007f3c6caaa504 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f3c6c39219f in clone () from /lib64/libc.so.6 coredump 7 - ... (gdb) bt #0 0x00007f916b526b88 in list_add (head=0x7f91389ba228, new=0x7f91389b9d78) at ../../../../libglusterfs/src/list.h:31 #1 wb_set_invalidate (wb_inode=0x7f91389b9d10, set=) at write-behind.c:246 #2 wb_fulfill_cbk (frame=frame at entry=0x7f91617a2208, cookie=, this=, op_ret=op_ret at entry=811, op_errno=op_errno at entry=0, prebuf=prebuf at entry=0x7f91617ade00, postbuf=postbuf at entry=0x7f91617ade98, xdata=xdata at entry=0x7f9160484c38) at write-behind.c:1095 #3 0x00007f916b7a2e86 in dht_writev_cbk (frame=frame at entry=0x7f91617b8838, cookie=, this=, op_ret=811, op_errno=0, prebuf=prebuf at entry=0x7f91617ade00, postbuf=postbuf at entry=0x7f91617ade98, xdata=0x7f9160484c38) at dht-inode-write.c:140 #4 0x00007f916ba0c21e in afr_writev_unwind (frame=frame at entry=0x7f916100d918, this=this at entry=0x7f91640110c0) at afr-inode-write.c:234 #5 0x00007f916ba0c7e6 in afr_writev_wind_cbk (this=0x7f91640110c0, frame=0x7f91604865e8, cookie=, op_ret=, op_errno=, prebuf=, postbuf=, xdata=) at afr-inode-write.c:388 #6 afr_writev_wind_cbk (frame=0x7f91604865e8, cookie=, this=0x7f91640110c0, op_ret=, op_errno=, prebuf=, postbuf=0x7f916946c980, xdata=0x7f91614414b8) at afr-inode-write.c:354 #7 0x00007f916bce1748 in client4_0_writev_cbk (req=, iov=, count=, myframe=0x7f915cfa2f98) at client-rpc-fops_v2.c:685 #8 0x00007f9171a8d534 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f9164050110, pollin=pollin at entry=0x7f9160481290) at rpc-clnt.c:755 #9 0x00007f9171a8de77 in rpc_clnt_notify (trans=0x7f91640503d0, mydata=0x7f9164050140, event=, data=0x7f9160481290) at rpc-clnt.c:923 #10 0x00007f9171a89f13 in rpc_transport_notify (this=this at entry=0x7f91640503d0, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f9160481290) at rpc-transport.c:525 #11 0x00007f916cba1a23 in socket_event_poll_in (notify_handled=true, this=0x7f91640503d0) at socket.c:2504 #12 socket_event_handler (fd=1615336080, idx=2, gen=4, data=0x7f91640503d0, poll_in=, poll_out=, poll_err=0) at socket.c:2905 #13 0x00007f9171d22aeb in event_dispatch_epoll_handler (event=0x7f916946ced0, event_pool=0x24db0b0) at event-epoll.c:591 #14 event_dispatch_epoll_worker (data=0x7f9164048dc0) at event-epoll.c:668 #15 0x00007f9170b13504 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f91703fb19f in clone () from /lib64/libc.so.6 coredump 8 - (gdb) bt #0 0x00007f24560fbe30 in pthread_mutex_lock () from /lib64/libpthread.so.0 #1 0x00007f24572a973e in gf_log_set_log_buf_size (buf_size=buf_size at entry=0) at logging.c:273 #2 0x00007f24572a98df in gf_log_disable_suppression_before_exit (ctx=0x840010) at logging.c:444 #3 0x00007f24572aff94 in gf_print_trace (signum=11, ctx=0x840010) at common-utils.c:922 #4 #5 0x00007f24572ced52 in fd_destroy (bound=true, fd=0x7f24380d3f98) at fd.c:478 #6 fd_unref (fd=0x7f24380d3f98) at fd.c:529 #7 0x00007f24512790e8 in client_local_wipe (local=local at entry=0x7f243c0ad548) at client-helpers.c:124 #8 0x00007f24512d51e0 in client4_0_finodelk_cbk (req=, iov=, count=, myframe=) at client-rpc-fops_v2.c:1398 #9 0x00007f2457073534 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7f244c050110, pollin=pollin at entry=0x7f2444165d30) at rpc-clnt.c:755 #10 0x00007f2457073e77 in rpc_clnt_notify (trans=0x7f244c0503d0, mydata=0x7f244c050140, event=, data=0x7f2444165d30) at rpc-clnt.c:923 #11 0x00007f245706ff13 in rpc_transport_notify (this=this at entry=0x7f244c0503d0, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f2444165d30) at rpc-transport.c:525 #12 0x00007f2452187a23 in socket_event_poll_in (notify_handled=true, this=0x7f244c0503d0) at socket.c:2504 #13 socket_event_handler (fd=1142316336, idx=2, gen=4, data=0x7f244c0503d0, poll_in=, poll_out=, poll_err=0) at socket.c:2905 #14 0x00007f2457308aeb in event_dispatch_epoll_handler (event=0x7f244b1b7ed0, event_pool=0x8770b0) at event-epoll.c:591 #15 event_dispatch_epoll_worker (data=0x7f244c043ad0) at event-epoll.c:668 #16 0x00007f24560f9504 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f24559e119f in clone () from /lib64/libc.so.6 If you really need the info - I have some 20 more coredumps, I suspect they all have different traces... -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 12:36:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 12:36:45 +0000 Subject: [Bugs] [Bug 1644389] [GSS] Directory listings on fuse mount are very slow due to small number of getdents() entries In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1644389 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(ccalhoun at redhat.c | |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 12:41:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 12:41:21 +0000 Subject: [Bugs] [Bug 1662557] glusterfs process crashes, causing "Transport endpoint not connected". In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662557 --- Comment #8 from robdewit --- Original core file of the 1st backtrace: https://www.dropbox.com/s/a8feic6hvho413o/core?dl=0 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 13:32:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 13:32:29 +0000 Subject: [Bugs] [Bug 1661887] Add monitoring of postgrey In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1661887 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-03 13:32:29 --- Comment #1 from M. Scherer --- So, notification was added, and I think it is also managed properly now. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 13:38:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 13:38:56 +0000 Subject: [Bugs] [Bug 1663223] New: profile info command is not displaying information of bricks which are hosted on peers Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663223 Bug ID: 1663223 Summary: profile info command is not displaying information of bricks which are hosted on peers Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: srakonde at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: When we run "gluster v profile info" from node n1, it is showing information of bricks from local node only. Information of bricks which are hosted on peers is not shown in the output. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. In a cluster of more than 1 node, create and start a volume 2. start profile for the volume 3. run gluster v profile volname info Actual results: Expected results: it should display information of all the bricks of volume. Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 13:48:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 13:48:12 +0000 Subject: [Bugs] [Bug 1663223] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663223 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21988 (glusterd: aggregate rsp from peers) posted (#1) for review on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 13:48:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 13:48:13 +0000 Subject: [Bugs] [Bug 1663223] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663223 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21988 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 14:16:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:16:23 +0000 Subject: [Bugs] [Bug 1663232] New: profile info command is not displaying information of bricks which are hosted on peers Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 Bug ID: 1663232 Summary: profile info command is not displaying information of bricks which are hosted on peers Product: Red Hat Gluster Storage Version: 3.4 Status: NEW Component: glusterd Severity: urgent Assignee: amukherj at redhat.com Reporter: srakonde at redhat.com QA Contact: bmekala at redhat.com CC: bugs at gluster.org, rhs-bugs at redhat.com, sankarshan at redhat.com, storage-qa-internal at redhat.com, vbellur at redhat.com Depends On: 1663223 Target Milestone: --- Classification: Red Hat +++ This bug was initially created as a clone of Bug #1663223 +++ Description of problem: When we run "gluster v profile info" from node n1, it is showing information of bricks from local node only. Information of bricks which are hosted on peers is not shown in the output. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. In a cluster of more than 1 node, create and start a volume 2. start profile for the volume 3. run gluster v profile volname info Actual results: Expected results: it should display information of all the bricks of volume. Additional info: --- Additional comment from Worker Ant on 2019-01-03 19:18:12 IST --- REVIEW: https://review.gluster.org/21988 (glusterd: aggregate rsp from peers) posted (#1) for review on master by Sanju Rakonde Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1663223 [Bug 1663223] profile info command is not displaying information of bricks which are hosted on peers -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 14:16:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:16:23 +0000 Subject: [Bugs] [Bug 1663223] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663223 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1663232 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 14:16:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:16:27 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 Red Hat Bugzilla Rules Engine changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |ZStream -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 14:18:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:18:07 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST Assignee|amukherj at redhat.com |srakonde at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 14:36:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:36:20 +0000 Subject: [Bugs] [Bug 1663243] New: rebalance status does not display localhost statistics when op-version is not bumped up Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663243 Bug ID: 1663243 Summary: rebalance status does not display localhost statistics when op-version is not bumped up Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: srakonde at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: rebalance status command is not showing information of local host when the cluster is not running with the current max op version. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. set the cluster op-version less than the max op version 2. create and start a volume 3. start rebalance for volume and check for rebalance status Actual results: In the output of "rebalance status" information related to localhost is not displayed. Expected results: It should display te information of localhost as well. Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 14:36:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:36:56 +0000 Subject: [Bugs] [Bug 1663244] New: rebalance status does not display localhost statistics when op-version is not bumped up Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663244 Bug ID: 1663244 Summary: rebalance status does not display localhost statistics when op-version is not bumped up Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: srakonde at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: rebalance status command is not showing information of local host when the cluster is not running with the current max op version. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. set the cluster op-version less than the max op version 2. create and start a volume 3. start rebalance for volume and check for rebalance status Actual results: In the output of "rebalance status" information related to localhost is not displayed. Expected results: It should display te information of localhost as well. Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 14:41:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:41:37 +0000 Subject: [Bugs] [Bug 1663247] New: remove static memory allocations from code Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663247 Bug ID: 1663247 Summary: remove static memory allocations from code Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: srakonde at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: In the entire code base, many structures are allocating the memory statically. Instead we can allocate memory dynamically. One such structure is: struct glusterd_brickinfo { char hostname[NAME_MAX]; char path[VALID_GLUSTERD_PATHMAX]; char real_path[VALID_GLUSTERD_PATHMAX]; char device_path[VALID_GLUSTERD_PATHMAX]; char mount_dir[VALID_GLUSTERD_PATHMAX]; char brick_id[1024]; /*Client xlator name, AFR changelog name*/ char fstype[NAME_MAX]; /* Brick file-system type */ char mnt_opts[1024]; /* Brick mount options */ .. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 14:47:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:47:53 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |urgent CC| |amukherj at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 14:50:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 14:50:25 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Regression --- Comment #3 from Atin Mukherjee --- This is a regression and has to be fixed. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 15:28:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 15:28:38 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 Red Hat Bugzilla Rules Engine changed: What |Removed |Added ---------------------------------------------------------------------------- Target Release|--- |RHGS 3.4.z Batch Update 3 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 17:48:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 17:48:37 +0000 Subject: [Bugs] [Bug 1663223] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663223 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21988 (glusterd: aggregate rsp from peers for profile command) posted (#2) for review on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 3 21:44:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 21:44:11 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/21989 (afr/shd: Cleanup self heal daemon resources during afr fini) posted (#1) for review on master by mohammed rafi kc -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 21:44:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 21:44:12 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21989 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 3 22:26:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 03 Jan 2019 22:26:26 +0000 Subject: [Bugs] [Bug 1663337] New: Gluster documentation on quorum-reads option is incorrect Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663337 Bug ID: 1663337 Summary: Gluster documentation on quorum-reads option is incorrect Product: GlusterFS Version: 4.1 Status: NEW Component: doc Assignee: bugs at gluster.org Reporter: aravind.natarajan at appian.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Per glusterfs code (see https://github.com/gluster/glusterfs/commit/bd44d59741bb8c0f5d7a62c5b1094179dd0ce8a4#diff-d6c0e00b45c718e92b1a55ee7b9fe513), the quorum-reads option is no longer supported. However Gluster documentation for Client Quorum (see https://docs.gluster.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/#client-quorum) still indicates that this is a valid option. It does not indicate that the option is not supported in ver 4.0 and above. Furthermore, setting/getting the option on a gluster volume does not cause any error or "not supported" information to be displayed leading the user to believe that this is supported unless someone were to actually look at the code. Version-Release number of selected component (if applicable): 4.0 and above How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 01:49:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 01:49:03 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #515 from Worker Ant --- REVIEW: https://review.gluster.org/21982 (extras: Add readdir-ahead to samba group command) posted (#2) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 03:25:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 03:25:29 +0000 Subject: [Bugs] [Bug 1663244] rebalance status does not display localhost statistics when op-version is not bumped up In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663244 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amukherj at redhat.com, | |srakonde at redhat.com Flags| |needinfo?(srakonde at redhat.c | |om) --- Comment #1 from Atin Mukherjee --- Isn't this a duplicate of BZ 1663243 ? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 06:02:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 06:02:42 +0000 Subject: [Bugs] [Bug 1663244] rebalance status does not display localhost statistics when op-version is not bumped up In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663244 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |DUPLICATE Flags|needinfo?(srakonde at redhat.c | |om) | Last Closed| |2019-01-04 06:02:42 --- Comment #2 from Sanju --- (In reply to Atin Mukherjee from comment #1) > Isn't this a duplicate of BZ 1663243 ? Somehow, these both bugs got raised at same time. May be I would have pressed the submit button twice. Closing this bug as a duplicate of 1663243. *** This bug has been marked as a duplicate of bug 1663243 *** -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 06:02:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 06:02:42 +0000 Subject: [Bugs] [Bug 1663243] rebalance status does not display localhost statistics when op-version is not bumped up In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663243 --- Comment #1 from Sanju --- *** Bug 1663244 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 06:06:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 06:06:46 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED CC| |sheggodu at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 4 06:43:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 06:43:40 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 Rejy M Cyriac changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On|1662059 | Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1662059 [Bug 1662059] [RHV-RHGS] Fuse mount crashed while deleting a 1 TB image file from RHV -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 4 06:44:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 06:44:58 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 Rejy M Cyriac changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On|1663208 | Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1663208 [Bug 1663208] Fuse mount crashed while deleting a 1 TB image file from RHV -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 4 06:45:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 06:45:09 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 Rejy M Cyriac changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1662059 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1662059 [Bug 1662059] [RHV-RHGS] Fuse mount crashed while deleting a 1 TB image file from RHV -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 4 07:31:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 07:31:28 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 errata-xmlrpc changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |ON_QA -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 4 07:36:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 07:36:12 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Fixed In Version| |glusterfs-3.12.2-36 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 4 11:33:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 11:33:48 +0000 Subject: [Bugs] [Bug 1663232] profile info command is not displaying information of bricks which are hosted on peers In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663232 nchilaka changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ON_QA |VERIFIED CC| |nchilaka at redhat.com QA Contact|bmekala at redhat.com |nchilaka at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 4 11:49:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 11:49:11 +0000 Subject: [Bugs] [Bug 1663243] rebalance status does not display localhost statistics when op-version is not bumped up In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663243 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21992 (glusterd: rebalance status should display information of localhost) posted (#1) for review on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 11:49:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 11:49:12 +0000 Subject: [Bugs] [Bug 1663243] rebalance status does not display localhost statistics when op-version is not bumped up In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663243 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21992 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 16:49:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 16:49:02 +0000 Subject: [Bugs] [Bug 1663519] New: Memory leak when smb.conf has "store dos attributes = yes" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663519 Bug ID: 1663519 Summary: Memory leak when smb.conf has "store dos attributes = yes" Product: GlusterFS Version: 3.12 Hardware: x86_64 OS: Linux Status: NEW Component: gluster-smb Severity: urgent Assignee: bugs at gluster.org Reporter: ryan at magenta.tv CC: bugs at gluster.org Target Milestone: --- Classification: Community Created attachment 1518442 --> https://bugzilla.redhat.com/attachment.cgi?id=1518442&action=edit Python 3 script to replicate issue --------------------------------------------------------------------------- Description of problem: If glusterfs VFS is used with Samba, and the global option "store dos attributes = yes" is set, the SMBD rss memory usage balloons. If a FUSE mount is used with Samba, and the global option "store dos attributes = yes" is set, the Gluster FUSE mount process rss memory usage balloons. --------------------------------------------------------------------------- Version-Release number of selected component (if applicable): Samba 4.9.4 Gluster 4.1 How reproducible: Can reproduce every time with attached python script --------------------------------------------------------------------------- Gluster volume options: Volume Name: mcv02 Type: Distribute Volume ID: 5debe2f4-16c4-457c-8496-fcf32b298ccf Status: Started Snapshot Count: 0 Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: mcn01:/mnt/h1a/test_data Brick2: mcn02:/mnt/h1b/test_data Brick3: mcn01:/mnt/h2a/test_data Brick4: mcn02:/mnt/h2b/test_data Options Reconfigured: network.ping-timeout: 5 storage.batch-fsync-delay-usec: 0 performance.cache-size: 1000MB performance.stat-prefetch: on features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.cache-invalidation: on performance.cache-samba-metadata: on performance.md-cache-timeout: 600 performance.io-thread-count: 32 performance.parallel-readdir: on performance.nl-cache: on performance.nl-cache-timeout: 600 cluster.lookup-optimize: on performance.write-behind-window-size: 1MB performance.client-io-threads: on client.event-threads: 4 server.event-threads: 4 auth.allow: 172.30.30.* transport.address-family: inet features.quota: on features.inode-quota: on nfs.disable: on features.quota-deem-statfs: on cluster.brick-multiplex: off cluster.server-quorum-ratio: 50% --------------------------------------------------------------------------- smb.conf file: [global] security = user netbios name = NAS01 clustering = no server signing = no max log size = 10000 log file = /var/log/samba/log-%M-test.smbd logging = file at 1 log level = 1 passdb backend = tdbsam guest account = nobody map to guest = bad user force directory mode = 0777 force create mode = 0777 create mask = 0777 directory mask = 0777 store dos attributes = yes load printers = no printing = bsd printcap name = /dev/null disable spoolss = yes glusterfs:volfile_server = localhost kernel share modes = No [VFS] vfs objects = glusterfs glusterfs:volume = mcv02 path = / read only = no guest ok = yes valid users = "nobody" [FUSE] read only = no guest ok = yes path = "/mnt/mcv02" valid users = "nobody" ------------------------------------------------------------------------- Steps to Reproduce: 1. Install/compile Samba (Tested with 4.8.4,4.8.6,4.9.4). Install HTOP 2. Add 'store dos attributes = yes' to the Global section of the /etc/samba/smb.conf file 3.Restart the SMB service 4. Map the Share to a drive in windows 5. Download the attached python script, change line 41 to the mapped drive in Windows 6. Run attached Python script from a Windows OS (Tested with Win 10 & Python 3.7.1) 7. Run 'htop' or watch the RSS memory usage of the SMBD process Actual results: SMBD and FUSE memory balloons over 2-4GB on the process, and does not decrease even when IO has finished Expected results: SMBD and FUSE memory increases slightly, but then stabilises. Rarely going over 200MB Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 22:54:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 22:54:39 +0000 Subject: [Bugs] [Bug 1662178] Compilation fails for xlators/mgmt/glusterd/src with error "undefined reference to `dlclose'" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662178 vnosov changed: What |Removed |Added ---------------------------------------------------------------------------- Hardware|Unspecified |x86_64 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 23:33:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 23:33:54 +0000 Subject: [Bugs] [Bug 1663583] New: Geo-replication fails to open logfile "/var/log/glusterfs/cli.log" on slave. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663583 Bug ID: 1663583 Summary: Geo-replication fails to open logfile "/var/log/glusterfs/cli.log" on slave. Product: GlusterFS Version: 5 Hardware: All OS: Linux Status: NEW Component: geo-replication Severity: high Assignee: bugs at gluster.org Reporter: vnosov at stonefly.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: When non root account on slave system is used by SSH to setup geo-replication, the initialization fails on slave to open logfile /var/log/glusterfs/cli.log. As result geo-replication is not active and has status "Created". Version-Release number of selected component (if applicable): GlusterFS 5.2, manually installed from GlusterFS source code. How reproducible: 100% In our tests: Master system IP address: 10.10.60.182. Slave system DNS name: vn-sc-2.westus2.cloudapp.azure.com Master volume: master-volume-0007 Slave volume: slave-volume-0001 Steps to Reproduce: 1. Setup nonprivileged account on slave system. In our case account "nasgorep" was used. Test SSH from master to slave. 12592 01/04/2019 14:27:35.101560269 1546640855 command: /usr/bin/ssh nasgorep at vn-sc-2.westus2.cloudapp.azure.com /bin/pwd 12592 01/04/2019 14:27:35.839285578 1546640855 status=0 /usr/bin/ssh nasgorep at vn-sc-2.westus2.cloudapp.azure.com /bin/pwd 12640 01/04/2019 14:27:37.847050433 1546640857 command: /usr/bin/ssh -q -oConnectTimeout=5 nasgorep at vn-sc-2.westus2.cloudapp.azure.com /bin/pwd 2>&1 12640 01/04/2019 14:27:38.596877738 1546640858 status=0 /usr/bin/ssh -q -oConnectTimeout=5 nasgorep at vn-sc-2.westus2.cloudapp.azure.com /bin/pwd 2>&1 2. Setup geo-replication: 12592 01/04/2019 14:27:35.101560269 1546640855 command: /usr/bin/ssh nasgorep at vn-sc-2.westus2.cloudapp.azure.com /bin/pwd 12592 01/04/2019 14:27:35.839285578 1546640855 status=0 /usr/bin/ssh nasgorep at vn-sc-2.westus2.cloudapp.azure.com /bin/pwd 12640 01/04/2019 14:27:37.847050433 1546640857 command: /usr/bin/ssh -q -oConnectTimeout=5 nasgorep at vn-sc-2.westus2.cloudapp.azure.com /bin/pwd 2>&1 12640 01/04/2019 14:27:38.596877738 1546640858 status=0 /usr/bin/ssh -q -oConnectTimeout=5 nasgorep at vn-sc-2.westus2.cloudapp.azure.com /bin/pwd 2>&1 [2019-01-04 22:27:52.449174] : system:: execute gsec_create : SUCCESS [2019-01-04 22:28:32.288278] : volume geo-replication master-volume-0007 nasgorep at vn-sc-2.westus2.cloudapp.azure.com::slave-volume-0001 create push-pem : SUCCESS [2019-01-04 22:28:39.447194] : volume geo-replication master-volume-0007 nasgorep at vn-sc-2.westus2.cloudapp.azure.com::slave-volume-0001 start : SUCCESS Actual results: Check status of the geo-replication, it is in "Created" state: [root at SC-10-10-63-182 log]# /usr/sbin/gluster volume geo-replication master-volume-0007 nasgorep at vn-sc-2.westus2.cloudapp.azure.com::slave-olume-0001 status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.10.60.182 master-volume-0007 /exports/master-segment-0015/master-volume-0007 nasgorep nasgorep at vn-sc-2.westus2.cloudapp.azure.com::slave-volume-0001 N/A Created N/A N/A N/A N/A N/A N/A N/A N/A N/A Expected results: state of the geo-replication has to be "Active". Additional info: On master system: geo-replication logfile "gsyncd.log" has next error messages: [root at SC-10-10-63-182 log]# vi /var/log/glusterfs/geo-replication/master-volume-0005_10.10.60.183_slave-volume-0001/gsyncd.log: ???. [2019-01-04 22:28:40.276280] E [syncdutils(monitor):809:errlog] Popen: command returned error cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 nasgorep at vn-sc-2.westus2.cloudapp.azure.com /usr/sbin/gluster --xml --remote-host=localhost volume info slave-volume-0001 error=255 [2019-01-04 22:28:40.277328] E [syncdutils(monitor):813:logerr] Popen: ssh> ERROR: failed to create logfile "/var/log/glusterfs/cli.log" (Permission denied) [2019-01-04 22:28:40.277438] E [syncdutils(monitor):813:logerr] Popen: ssh> ERROR: failed to open logfile /var/log/glusterfs/cli.log ??. On slave: file "cli.log" has next attributes: [root at VN-SC-2 log]# ls -l /var/log/glusterfs/cli.log -rw------- 1 root root 28552 Jan 4 14:35 /var/log/glusterfs/cli.log It seems slave system does not let user "nasgorep" open or create logfile /var/log/glusterfs/cli.log" when "gluster" call is invoked from master system through SSH. On slave: command history log: [root at VN-SC-2 log]# cat /var/log/glusterfs/cmd_history.log [2019-01-04 22:22:49.893261] : volume status : SUCCESS [2019-01-04 22:22:59.929183] : volume geo-replication status : SUCCESS [2019-01-04 22:24:25.105256] : volume create slave-volume-0001 transport tcp VN-SC-2.cxn2n2xii2hepedpq4siv2akfe.xx.internal.cloudapp.net:/exports/nas-segment-0015/slave-volume-0001 : SUCCESS [2019-01-04 22:24:25.894525] : volume set slave-volume-0001 nfs.addr-namelookup off : SUCCESS [2019-01-04 22:24:26.418826] : volume reset slave-volume-0001 network.ping-timeout : SUCCESS [2019-01-04 22:24:27.086126] : volume set slave-volume-0001 nfs.disable on : SUCCESS [2019-01-04 22:24:28.082211] : volume set slave-volume-0001 performance.stat-prefetch off : SUCCESS [2019-01-04 22:24:28.873671] : volume set slave-volume-0001 performance.quick-read off : SUCCESS [2019-01-04 22:24:29.716433] : volume set slave-volume-0001 allow-insecure on : SUCCESS [2019-01-04 22:24:30.272508] : volume reset slave-volume-0001 nfs.rpc-auth-allow : SUCCESS [2019-01-04 22:24:31.303757] : volume start slave-volume-0001 : SUCCESS [2019-01-04 22:24:51.709793] : volume geo-replication status : SUCCESS [2019-01-04 22:24:52.575909] : volume status slave-volume-0001 : SUCCESS [2019-01-04 22:27:55.543280] : system:: uuid get : SUCCESS [2019-01-04 22:27:55.574006] : system:: execute mountbroker.py node-add slave-volume-0001 nasgorep : SUCCESS [2019-01-04 22:28:38.062065] : system:: copy file /geo-replication/master-volume-0007_slave-volume-0001_common_secret.pem.pub : SUCCESS [2019-01-04 22:28:38.296566] : system:: execute add_secret_pub nasgorep geo-replication/master-volume-0007_slave-volume-0001_common_secret.pem.pub : SUCCESS [2019-01-04 22:31:31.417565] : volume geo-replication slave-volume-0001 status : SUCCESS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Jan 6 12:18:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 06 Jan 2019 12:18:24 +0000 Subject: [Bugs] [Bug 1528641] Brick processes fail to start In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1528641 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mchangir at redhat.com, | |rob at abcxyz.nl Flags| |needinfo?(rob at abcxyz.nl) --- Comment #2 from Milind Changire --- This might be an insufficient transport.listen-backlog case. Rob, Could you set the vol file option transport.listen-backlog to 1024 in the /etc/glusterfs/glusterd.vol file on both the nodes and restart the nodes and get back on the status. In the mean time, a dump of the volume info of the all the volumes help provide an insight into the state of affairs. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 4 11:49:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 04 Jan 2019 11:49:12 +0000 Subject: [Bugs] [Bug 1663243] rebalance status does not display localhost statistics when op-version is not bumped up In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663243 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/21992 (glusterd: rebalance status should display information of localhost) posted (#3) for review on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 02:46:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 02:46:04 +0000 Subject: [Bugs] [Bug 1208099] RFE: Display File Location in Gluster CLI In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1208099 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |WONTFIX Last Closed| |2019-01-07 02:46:04 --- Comment #5 from Atin Mukherjee --- There's no plan to work on this RFE in glusterd. Closing this as won't fix. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 7 03:15:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 03:15:21 +0000 Subject: [Bugs] [Bug 1657860] Archives for ci-results mailinglist are getting wiped (with each mail?) In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657860 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED CC| |nigelb at redhat.com Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-07 03:15:21 --- Comment #2 from Nigel Babu --- The fix seems to be working. Closing bug. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 03:20:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 03:20:08 +0000 Subject: [Bugs] [Bug 1596787] glusterfs rpc-clnt.c: error returned while attempting to connect to host: (null), port 0 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1596787 --- Comment #7 from Worker Ant --- REVIEW: https://review.gluster.org/21897 (rpc-clnt: reduce transport connect log for EINPROGRESS) posted (#6) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 03:26:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 03:26:05 +0000 Subject: [Bugs] [Bug 1648768] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1648768 --- Comment #17 from Worker Ant --- REVIEW: https://review.gluster.org/21962 (leases: Reset lease_ctx->timer post deletion) posted (#3) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 7 03:37:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 03:37:47 +0000 Subject: [Bugs] [Bug 1663780] New: On docs.gluster.org, we should convert spaces in folder or file names to 301 redirects to hypens Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663780 Bug ID: 1663780 Summary: On docs.gluster.org, we should convert spaces in folder or file names to 301 redirects to hypens Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: nigelb at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community This request depends on https://github.com/gluster/glusterdocs/pull/447. Once we have the Nginx redirect code ready, we can merge in the pull request and push the change. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 03:45:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 03:45:50 +0000 Subject: [Bugs] [Bug 1493656] Storage hiccup (inaccessible a short while) when a single brick go down In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1493656 --- Comment #14 from Milind Changire --- @ko_co_ten The patch makes tunables available for tuning the Gluster system. The defaults are equal to what a normal/out-of-the-box system configuration would provide. For aggressive recovery times, you will need to tweak the tunables for smaller values. For details about the tunables, please go through the tcp(7) man page. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 7 03:51:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 03:51:01 +0000 Subject: [Bugs] [Bug 1191072] ipv6 enabled on the peer, but dns resolution fails with ipv6 and gluster does not fall back to ipv4 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1191072 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mchangir at redhat.com --- Comment #1 from Milind Changire --- Patch [1] is being worked on to provide ipv6 peer configuration and communication. However, mixed mode (ipv4 and ipv6) communication will still not be possible. 1. https://review.gluster.org/c/glusterfs/+/21948 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 04:14:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 04:14:51 +0000 Subject: [Bugs] [Bug 1658147] BZ incorrectly updated with "patch posted" message when a patch is merged In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1658147 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |DUPLICATE Last Closed| |2019-01-07 04:14:51 --- Comment #2 from Nigel Babu --- *** This bug has been marked as a duplicate of bug 1658146 *** -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 7 04:14:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 04:14:51 +0000 Subject: [Bugs] [Bug 1658146] BZ incorrectly updated with "patch posted" message when a patch is merged In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1658146 --- Comment #2 from Nigel Babu --- *** Bug 1658147 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 7 08:15:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 08:15:21 +0000 Subject: [Bugs] [Bug 1663583] Geo-replication fails to open logfile "/var/log/glusterfs/cli.log" on slave. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663583 Sunny Kumar changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sunkumar at redhat.com --- Comment #1 from Sunny Kumar --- Hi, Can you confirm that whether master and slave on same host ? - Sunny -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 08:15:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 08:15:34 +0000 Subject: [Bugs] [Bug 1663583] Geo-replication fails to open logfile "/var/log/glusterfs/cli.log" on slave. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663583 Sunny Kumar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |sunkumar at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 09:05:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 09:05:16 +0000 Subject: [Bugs] [Bug 1528641] Brick processes fail to start In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1528641 robdewit changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rob.dewit at coosto.com --- Comment #3 from robdewit --- We've expanded the cluster with another node since then and this behavior has not occurred after that as far as I recall. Could this have been caused by the number of volumes or rather by some latency in I/O (disk or network)? I'd rather not mess with the settings since the cluster have been running OK now for several months. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 09:14:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 09:14:04 +0000 Subject: [Bugs] [Bug 1528641] Brick processes fail to start In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1528641 --- Comment #4 from Milind Changire --- Glad to hear things are working for you. My hypothesis is that glusterd starting a large number of bricks causes a flood/rush of brick process to attempt to connect back to glusterd. This causes SYN Flooding and eventually drop of connection requests causing loss of service due to insufficient resources for holding connection requests until they are acknowledged. Hence the reference to tweak the glusterd vol file option: transport.listen-backlog. You could take a look at /var/log/messages and "grep -i" for "SYN Flooding" and see if that's the case. If things are working for you, you could close this BZ as WORKSFORME. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 09:17:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 09:17:21 +0000 Subject: [Bugs] [Bug 1528641] Brick processes fail to start In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1528641 robdewit changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |WORKSFORME Flags|needinfo?(rob at abcxyz.nl) | Last Closed| |2019-01-07 09:17:21 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 09:21:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 09:21:17 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #516 from Worker Ant --- REVIEW: https://review.gluster.org/21987 (glfs-fops.c: fix the bad string length for snprintf) posted (#2) for review on master by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 09:55:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 09:55:25 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 --- Comment #1595 from Worker Ant --- REVIEW: https://review.gluster.org/21995 (posix: fix coverity issue) posted (#1) for review on master by Iraj Jamali -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 09:55:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 09:55:27 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21995 -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 11:29:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 11:29:36 +0000 Subject: [Bugs] [Bug 1663519] Memory leak when smb.conf has "store dos attributes = yes" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663519 --- Comment #1 from ryan at magenta.tv --- I can confirm this issue also affects OS X clients connecting to the system. Samba 4.9 has 'store dos attributes' set to True/on by default now, so it's very likely others will encounter this issue. Please let me know if I can assist or provide more data. Many thanks, Ryan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 11:29:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 11:29:55 +0000 Subject: [Bugs] [Bug 1663519] Memory leak when smb.conf has "store dos attributes = yes" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663519 ryan at magenta.tv changed: What |Removed |Added ---------------------------------------------------------------------------- Version|3.12 |4.1 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 11:32:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 11:32:20 +0000 Subject: [Bugs] [Bug 1654642] Very high memory usage with glusterfs VFS module In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654642 ryan at magenta.tv changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |DUPLICATE Last Closed| |2019-01-07 11:32:20 --- Comment #7 from ryan at magenta.tv --- Due to lack of responses and discoveries found during testing, I'm closing this ticket and have opened another (1663519) *** This bug has been marked as a duplicate of bug 1663519 *** -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 11:32:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 11:32:20 +0000 Subject: [Bugs] [Bug 1663519] Memory leak when smb.conf has "store dos attributes = yes" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663519 --- Comment #2 from ryan at magenta.tv --- *** Bug 1654642 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 12:19:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 12:19:53 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 --- Comment #38 from ryan at magenta.tv --- Should this be fixed in Samba 4.9.4? I'm not seeing the error anymore, however I'm now seeing 'bad file descriptor' messages when trying to write files. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 7 12:27:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 12:27:47 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 --- Comment #39 from ryan at magenta.tv --- These logs may provide some insight: [2019/01/07 12:26:08.115358, 2] ../source3/smbd/open.c:1447(open_file) nobody opened file blah.sh read=No write=No (numopen=3) [2019/01/07 12:26:08.116122, 2] ../source3/smbd/close.c:802(close_normal_file) nobody closed file blah.sh (numopen=2) NT_STATUS_OK [2019/01/07 12:26:08.552909, 2] ../source3/smbd/open.c:1447(open_file) nobody opened file blah.sh read=Yes write=No (numopen=2) [2019/01/07 12:26:08.552983, 0] ../source3/modules/vfs_fruit.c:2692(fruit_check_access) fruit_check_access: fcntl get flags [blah.sh] fd [13371337] failed [Bad file descriptor] [2019/01/07 12:26:08.553071, 2] ../source3/smbd/close.c:802(close_normal_file) nobody closed file blah.sh (numopen=1) NT_STATUS_OK [2019/01/07 12:26:08.832632, 2] ../source3/smbd/open.c:1447(open_file) nobody opened file blah.sh read=No write=No (numopen=2) [2019/01/07 12:26:08.833005, 2] ../source3/smbd/open.c:1447(open_file) nobody opened file blah.sh:AFP_AfpInfo read=Yes write=Yes (numopen=3) [2019/01/07 12:26:08.833154, 0] ../source3/modules/vfs_glusterfs.c:1138(vfs_gluster_fchmod) vfs_gluster_fchmod: Failed to fetch gluster fd [2019/01/07 12:26:08.833599, 0] ../source3/modules/vfs_glusterfs.c:627(vfs_gluster_pread) vfs_gluster_pread: Failed to fetch gluster fd [2019/01/07 12:26:08.833908, 0] ../source3/modules/vfs_glusterfs.c:613(vfs_gluster_close) vfs_gluster_close: Failed to fetch gluster fd [2019/01/07 12:26:08.833926, 0] ../source3/modules/vfs_fruit.c:4512(fruit_pwrite_meta_stream) fruit_pwrite_meta_stream: Close [blah.sh:AFP_AfpInfo] failed: Success [2019/01/07 12:26:08.833948, 2] ../source3/smbd/smb2_write.c:204(smb2_write_complete_internal) smb2_write failed: fnum 1612338452, file blah.sh:AFP_AfpInfo, length=60 offset=0 nwritten=-1: NT_STATUS_UNSUCCESSFUL [2019/01/07 12:26:08.834566, 2] ../source3/smbd/close.c:802(close_normal_file) nobody closed file blah.sh:AFP_AfpInfo (numopen=2) NT_STATUS_OK [2019/01/07 12:26:08.834617, 2] ../source3/smbd/close.c:802(close_normal_file) nobody closed file blah.sh (numopen=1) NT_STATUS_OK -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 7 19:07:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 19:07:24 +0000 Subject: [Bugs] [Bug 1664122] New: do not send bit-rot virtual xattrs in lookup response Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664122 Bug ID: 1664122 Summary: do not send bit-rot virtual xattrs in lookup response Product: GlusterFS Version: mainline Status: NEW Component: bitrot Assignee: bugs at gluster.org Reporter: rabhat at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Docs Contact: bugs at gluster.org Description of problem: Though bit-rot-stub xlator removes the internal bit-rot detection specific content from the dictionary of the lookup response, it fails to do so if the file has been marked as bad. This is what is needed. ======================= 1) If the file has correct, then do not send any of the internal xattrs in the lookup callback dictionary (which is the current behavior as of now, thus no need to worry about) 2) If the file has been marked as corrupted, then only send the bad file marker in the dictionary of lookup callback. Other internal xattrs such as version, signature etc are not needed. (As of now, for bad files, all the internal xattrs are sent in the lookup response) Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. You are the Docs Contact for the bug. From bugzilla at redhat.com Mon Jan 7 19:10:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 19:10:15 +0000 Subject: [Bugs] [Bug 1664122] do not send bit-rot virtual xattrs in lookup response In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664122 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21996 (features/bit-rot: do not send version and signature keys in dict) posted (#1) for review on master by Raghavendra Bhat -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. You are the Docs Contact for the bug. From bugzilla at redhat.com Mon Jan 7 19:10:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 19:10:16 +0000 Subject: [Bugs] [Bug 1664122] do not send bit-rot virtual xattrs in lookup response In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664122 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21996 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. You are the Docs Contact for the bug. From bugzilla at redhat.com Mon Jan 7 19:15:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 19:15:34 +0000 Subject: [Bugs] [Bug 1664124] New: Improve information dumped from io-threads in statedump Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664124 Bug ID: 1664124 Summary: Improve information dumped from io-threads in statedump Product: GlusterFS Version: mainline Status: NEW Component: io-threads Severity: medium Assignee: bugs at gluster.org Reporter: vbellur at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: statedump from io-threads lacks information to understand the number of running threads & number of requests in each priority queue. Would be good to have this information captured in statedump for observability and debuggability. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 19:15:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 19:15:47 +0000 Subject: [Bugs] [Bug 1664124] Improve information dumped from io-threads in statedump In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664124 Vijay Bellur changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |vbellur at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 19:15:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 19:15:47 +0000 Subject: [Bugs] [Bug 1664124] Improve information dumped from io-threads in statedump In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664124 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21667 (performance/io-threads: Improve debuggability in statedump) posted (#3) for review on master by Vijay Bellur -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 7 19:35:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 19:35:11 +0000 Subject: [Bugs] [Bug 1664124] Improve information dumped from io-threads in statedump In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664124 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21667 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 04:27:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 04:27:41 +0000 Subject: [Bugs] [Bug 1664215] New: Toggling Read ahead translator off causes some clients to umount some of its volumes Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664215 Bug ID: 1664215 Summary: Toggling Read ahead translator off causes some clients to umount some of its volumes Product: GlusterFS Version: 3.12 Hardware: other OS: Linux Status: NEW Component: read-ahead Severity: urgent Assignee: bugs at gluster.org Reporter: amgad.saleh at nokia.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Created attachment 1519119 --> https://bugzilla.redhat.com/attachment.cgi?id=1519119&action=edit Glusterfs client logs Description of problem: After turning the Read ahead translator "off", some of the clients (fuse) got disconnected (umount) to one of the data volumes. Attached the glusterfs logs from the client that experienced the disconnect. The following is an excerpt of the messages in the glusterfs/data.log.* logfiles: --- [2019-01-07 07:40:44.625789] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: switched to graph 8 [2019-01-07 07:40:44.629594] I [MSGID: 114021] [client.c:2369:notify] 6-el_data-client-0: current graph is no longer active, destroying rpc_client [2019-01-07 07:40:44.629651] I [MSGID: 114021] [client.c:2369:notify] 6-el_data-client-1: current graph is no longer active, destroying rpc_client [2019-01-07 07:40:44.629668] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 6-el_data-client-0: disconnected from el_data-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2019-01-07 07:40:44.629724] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 6-el_data-client-1: disconnected from el_data-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2019-01-07 07:40:44.629732] E [MSGID: 108006] [afr-common.c:5118:__afr_handle_child_down_event] 6-el_data-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2019-01-07 07:40:44.869481] I [glusterfsd-mgmt.c:52:mgmt_cbk_spec] 0-mgmt: Volume file changed [2019-01-07 07:40:44.916540] I [glusterfsd-mgmt.c:52:mgmt_cbk_spec] 0-mgmt: Volume file changed ---- Version-Release number of selected component (if applicable): 3.12.13 How reproducible: Turn the Read ahead translator "off" on the server side. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 7 19:35:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 07 Jan 2019 19:35:11 +0000 Subject: [Bugs] [Bug 1664124] Improve information dumped from io-threads in statedump In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664124 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21667 (performance/io-threads: Improve debuggability in statedump) posted (#3) for review on master by Vijay Bellur -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 05:33:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 05:33:23 +0000 Subject: [Bugs] [Bug 1664122] do not send bit-rot virtual xattrs in lookup response In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664122 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21996 (features/bit-rot: do not send version and signature keys in dict) posted (#2) for review on master by Kotresh HR -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. You are the Docs Contact for the bug. From bugzilla at redhat.com Tue Jan 8 06:19:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 06:19:13 +0000 Subject: [Bugs] [Bug 1664226] New: glusterd2 PR is not triggering Tests Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664226 Bug ID: 1664226 Summary: glusterd2 PR is not triggering Tests Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: avishwan at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: PR https://github.com/gluster/glusterd2/pull/1464 is not triggering the tests. Status in https://ci.centos.org/job/gluster_glusterd2/ shows "#4?14?7 (pending?gluster-ci-slave01 is offline)" -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 06:26:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 06:26:04 +0000 Subject: [Bugs] [Bug 1664226] glusterd2 PR is not triggering Tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664226 Kaushal changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED CC| |kaushal at redhat.com Assignee|bugs at gluster.org |kaushal at redhat.com --- Comment #1 from Kaushal --- I've reported the problem to the centos-ci team. It should be resolved soon enough.or the past -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 06:52:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 06:52:07 +0000 Subject: [Bugs] [Bug 1664226] glusterd2 PR is not triggering Tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664226 Kaushal changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |WORKSFORME Last Closed| |2019-01-08 06:52:07 --- Comment #2 from Kaushal --- This has been fixed now. The slave VM is back online and jobs are being processed. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 08:15:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 08:15:09 +0000 Subject: [Bugs] [Bug 1362129] rename of a file can cause data loss in an replica/arbiter volume configuration In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1362129 Anees Patel changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ON_QA |VERIFIED --- Comment #23 from Anees Patel --- Verified the fix per the above test-plan for arbiter and replica 3, on the latest BU3 build # rpm -qa | grep gluster gluster-nagios-common-0.2.4-1.el7rhgs.noarch glusterfs-api-3.12.2-36.el7rhgs.x86_64 glusterfs-server-3.12.2-36.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-36.el7rhgs.x86_64 glusterfs-client-xlators-3.12.2-36.el7rhgs.x86_64 -Setting this to verified -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 08:49:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 08:49:15 +0000 Subject: [Bugs] [Bug 1651439] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651439 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |POST --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/21998 (dht: fix inode leak when heal path) posted (#1) for review on master by Kinglong Mee -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 08:49:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 08:49:16 +0000 Subject: [Bugs] [Bug 1651439] gluster-NFS crash while expanding volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651439 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21998 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 09:26:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 09:26:00 +0000 Subject: [Bugs] [Bug 1659439] Memory leak: dict_t leak in rda_opendir In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659439 Sayalee changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ON_QA |VERIFIED --- Comment #11 from Sayalee --- Moving this bug to verified as: * Tested the issue on 3.12.2-36 build, there was no rise in memory usage after the initial run. * Tested the issue on BU2 build (3.12.2-32), noticed that the RES in top command output increased from 3.6g to 4.2g, which means memory usage raised continuously. --> Both these observations help to conclude that the issue has been fixed on 3.12.2-36 build. --> Also other tests around the bug will be covered in regression and if any issue is hit, bug will be reported accordingly. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 10:04:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 10:04:37 +0000 Subject: [Bugs] [Bug 1467614] Gluster read/write performance improvements on NVMe backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1467614 --- Comment #75 from Worker Ant --- REVIEW: https://review.gluster.org/21999 (socket: fix issue on concurrent socket event handle) posted (#1) for review on master by Zhang Huan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 10:04:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 10:04:39 +0000 Subject: [Bugs] [Bug 1467614] Gluster read/write performance improvements on NVMe backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1467614 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21999 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 11:38:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 11:38:51 +0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 --- Comment #57 from Worker Ant --- REVIEW: https://review.gluster.org/22002 (afr : fix memory leak) posted (#1) for review on master by Sunny Kumar -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 11:38:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 11:38:52 +0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22002 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 12:09:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 12:09:39 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/21946 (features/shard: Assign fop id during background deletion to prevent excessive logging) posted (#7) for review on master by Xavi Hernandez -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 13:37:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 13:37:01 +0000 Subject: [Bugs] [Bug 1664215] Toggling Read ahead translator off causes some clients to umount some of its volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664215 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amgad.saleh at nokia.com, | |nbalacha at redhat.com Assignee|bugs at gluster.org |atumball at redhat.com Flags| |needinfo?(amgad.saleh at nokia | |.com) --- Comment #1 from Nithya Balachandran --- (In reply to Amgad from comment #0) > Created attachment 1519119 [details] > Glusterfs client logs > > Description of problem: > > After turning the Read ahead translator "off", some of the clients (fuse) > got disconnected (umount) to one of the data volumes. Attached the glusterfs > logs from the client that experienced the disconnect. > > The following is an excerpt of the messages in the glusterfs/data.log.* > logfiles: > --- > [2019-01-07 07:40:44.625789] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: > switched to graph 8 > [2019-01-07 07:40:44.629594] I [MSGID: 114021] [client.c:2369:notify] > 6-el_data-client-0: current graph is no longer active, destroying rpc_client > [2019-01-07 07:40:44.629651] I [MSGID: 114021] [client.c:2369:notify] > 6-el_data-client-1: current graph is no longer active, destroying rpc_client > [2019-01-07 07:40:44.629668] I [MSGID: 114018] > [client.c:2285:client_rpc_notify] 6-el_data-client-0: disconnected from > el_data-client-0. Client process will keep trying to connect to glusterd > until brick's port is available > [2019-01-07 07:40:44.629724] I [MSGID: 114018] > [client.c:2285:client_rpc_notify] 6-el_data-client-1: disconnected from > el_data-client-1. Client process will keep trying to connect to glusterd > until brick's port is available > [2019-01-07 07:40:44.629732] E [MSGID: 108006] > [afr-common.c:5118:__afr_handle_child_down_event] 6-el_data-replicate-0: All > subvolumes are down. Going offline until atleast one of them comes back up. > [2019-01-07 07:40:44.869481] I [glusterfsd-mgmt.c:52:mgmt_cbk_spec] 0-mgmt: > Volume file changed > [2019-01-07 07:40:44.916540] I [glusterfsd-mgmt.c:52:mgmt_cbk_spec] 0-mgmt: > Volume file changed > ---- > > Version-Release number of selected component (if applicable): > 3.12.13 > > How reproducible: > Turn the Read ahead translator "off" on the server side. > Do you mean read-ahead or readdir-ahead? They are 2 different translators and the memory leak was in readdir-ahead. Do clients lose access to the volume and do you see errors on the mount point? The graph switch messages in the logs are expected. @Amar, please assign the BZ to the appropriate person. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 13:41:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 13:41:18 +0000 Subject: [Bugs] [Bug 1664215] Toggling Read ahead translator off causes some clients to umount some of its volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664215 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |atumball at redhat.com Flags| |needinfo?(atumball at redhat.c | |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 13:46:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 13:46:30 +0000 Subject: [Bugs] [Bug 1664335] New: [geo-rep]: Transport endpoint not connected with arbiter volumes Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664335 Bug ID: 1664335 Summary: [geo-rep]: Transport endpoint not connected with arbiter volumes Product: GlusterFS Version: mainline Status: NEW Component: geo-replication Keywords: EasyFix, ZStream Severity: low Priority: low Assignee: bugs at gluster.org Reporter: hlalwani at redhat.com CC: bugs at gluster.org Depends On: 1640573 Target Milestone: --- Classification: Community Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1640573 [Bug 1640573] [geo-rep]: Transport endpoint not connected with arbiter volumes -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 13:47:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 13:47:01 +0000 Subject: [Bugs] [Bug 1664335] [geo-rep]: Transport endpoint not connected with arbiter volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664335 Harpreet Kaur Lalwani changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |hlalwani at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 13:52:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 13:52:51 +0000 Subject: [Bugs] [Bug 1664335] [geo-rep]: Transport endpoint not connected with arbiter volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664335 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22001 (geo-rep: Fix for \"Transport End Point not connected\" issue) posted (#4) for review on master by Harpreet Kaur Lalwani -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 13:52:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 13:52:52 +0000 Subject: [Bugs] [Bug 1664335] [geo-rep]: Transport endpoint not connected with arbiter volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664335 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22001 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 16:45:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 16:45:46 +0000 Subject: [Bugs] [Bug 1664398] New: ./tests/00-geo-rep/00-georep-verify-setup.t does not work with ./run-tests-in-vagrant.sh Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664398 Bug ID: 1664398 Summary: ./tests/00-geo-rep/00-georep-verify-setup.t does not work with ./run-tests-in-vagrant.sh Product: GlusterFS Version: mainline Status: NEW Component: tests Assignee: bugs at gluster.org Reporter: ndevos at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: When running local tests, ./tests/00-geo-rep/00-georep-verify-setup.t always fails. Version-Release number of selected component (if applicable): current master branch How reproducible: 100% Steps to Reproduce: 1. git clone the repository 2. run ./run-tests-in-vagrant.sh Actual results: The 1st test fails. Expected results: All tests pass. Additional info: $ ./run-tests-in-vagrant.sh Copying tests/vagrant/vagrant-template-fedora dir to tests/vagrant/bug/789278 .... Doing vagrant up.... Copying source code from host machine to VM.... Source compile and install Gluster.... Connection to 192.168.121.65 closed. Connection to 192.168.121.65 closed. Connection to 192.168.121.65 closed. Connection to 192.168.121.65 closed. Connection to 192.168.121.65 closed. ... GlusterFS Test Framework ... /home/vagrant/glusterfs /home/vagrant/glusterfs /home/vagrant/glusterfs testing 'timeout' command ================================================================================ [16:38:13] Running tests in file ./tests/00-geo-rep/00-georep-verify-setup.t Timeout set is 300, default 200 ./tests/00-geo-rep/00-georep-verify-setup.t .. 1..27 ok 1, LINENUM:12 ok 2, LINENUM:13 ok 3, LINENUM:31 ok 4, LINENUM:32 ok 5, LINENUM:35 ok 6, LINENUM:36 ok 7, LINENUM:39 ok 8, LINENUM:40 ok 9, LINENUM:41 ok 10, LINENUM:42 ok 11, LINENUM:45 ok 12, LINENUM:48 Passwordless ssh login has not been setup with 127.0.0.1 for user root. ok 13, LINENUM:55 Geo-replication session between master and 127.0.0.1::slave does not exist. not ok 14 , LINENUM:58 FAILED COMMAND: gluster --mode=script --wignore volume geo-replication master 127.0.0.1::slave config gluster-command-dir /usr/sbin Geo-replication session between master and 127.0.0.1::slave does not exist. not ok 15 , LINENUM:61 FAILED COMMAND: gluster --mode=script --wignore volume geo-replication master 127.0.0.1::slave config slave-gluster-command-dir /usr/sbin Geo-replication session between master and 127.0.0.1::slave does not exist. not ok 16 , LINENUM:64 FAILED COMMAND: gluster --mode=script --wignore volume geo-replication master 127.0.0.1::slave config use_meta_volume true stat: cannot stat ?/var/lib/glusterd/geo-replication/master_slave_common_secret.pem.pub?: No such file or directory ... and a few more errors. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 19:21:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 19:21:50 +0000 Subject: [Bugs] [Bug 1664215] Toggling Read ahead translator off causes some clients to umount some of its volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664215 Amgad changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(amgad.saleh at nokia | |.com) | |needinfo?(atumball at redhat.c | |om) | --- Comment #2 from Amgad --- Thanks Nithya: It is the "readdir-ahead" translator. Yes the client loses access to the volume and we saw the graph switch messages. For instance in the attached logs, you can see: ------- [2019-01-04 04:55:51.453074] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: switched to graph 4 [2019-01-04 04:55:51.454990] I [MSGID: 114021] [client.c:2369:notify] 2-el_data-client-0: current graph is no longer active, destroying rpc_client [2019-01-04 04:55:51.455028] I [MSGID: 114021] [client.c:2369:notify] 2-el_data-client-1: current graph is no longer active, destroying rpc_client [2019-01-04 04:55:51.455037] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-el_data-client-0: disconnected from el_data-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2019-01-04 04:55:51.455080] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-el_data-client-1: disconnected from el_data-client-1. Client process will keep trying to connect to glusterd until brick's port is available -------- Regards, Amgad -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 19:43:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 19:43:32 +0000 Subject: [Bugs] [Bug 1664215] Toggling readdir-ahead translator off causes some clients to umount some of its volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664215 Amgad changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Toggling Read ahead |Toggling readdir-ahead |translator off causes some |translator off causes some |clients to umount some of |clients to umount some of |its volumes |its volumes -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 8 21:56:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 21:56:47 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 vanessa.haro at nokia.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |vanessa.haro at nokia.com --- Comment #7 from vanessa.haro at nokia.com --- We saw this as well in V5.1.1, The stack back traces were: (gdb) t a a bt Thread 24 (LWP 20898): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=0, result=0xc) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000000 in ?? () Thread 23 (LWP 20894): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=4, result=0x8) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000004 in ?? () #2 0x0000000000000000 in ?? () Thread 22 (LWP 20897): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=4, result=0xb) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000004 in ?? () #2 0x0000000000000000 in ?? () Thread 21 (LWP 20885): #0 0x00007effe124da82 in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:347 #1 0x00007effe2e02430 in ?? () #2 0x00007effe2e06050 in ?? () #3 0x00007effd7fbde60 in ?? () #4 0x00007effe2e06098 in ?? () #5 0x00007effe24258a8 in syncenv_task () from /lib64/libglusterfs.so.0 #6 0x00007effe24267f0 in syncenv_processor () from /lib64/libglusterfs.so.0 #7 0x00007effe1249dc5 in start_thread (arg=0x7effd7fbe700) at pthread_create.c:308 #8 0x00007effe0b1776d in putspent (p=0x0, stream=0x7effd7fbe700) at putspent.c:60 #9 0x0000000000000000 in ?? () ---Type to continue, or q to quit--- Thread 20 (LWP 20880): #0 0x00007effe124aef7 in pthread_join (threadid=139637260523264, thread_return=0x0) at pthread_join.c:64 #1 0x00007effe2449968 in event_dispatch_epoll () from /lib64/libglusterfs.so.0 #2 0x00007effe28f94cb in main () Thread 19 (LWP 20888): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=0, result=0x2) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000000 in ?? () Thread 18 (LWP 20883): Python Exception Cannot access memory at address 0x100000007: Thread 17 (LWP 20890): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=3, result=0x4) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000007 in ?? () #2 0x0000000000000000 in ?? () Thread 16 (LWP 20886): #0 0x00007effe124da82 in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:347 #1 0x00007effe2e02430 in ?? () #2 0x00007effe2e06050 in ?? () #3 0x00007effd77bce60 in ?? () #4 0x00007effe2e06098 in ?? () #5 0x00007effe24258a8 in syncenv_task () from /lib64/libglusterfs.so.0 #6 0x00007effe24267f0 in syncenv_processor () from /lib64/libglusterfs.so.0 #7 0x00007effe1249dc5 in start_thread (arg=0x7effd77bd700) at pthread_create.c:308 #8 0x00007effe0b1776d in putspent (p=0x0, stream=0x7effd77bd700) at putspent.c:60 #9 0x0000000000000000 in ?? () ---Type to continue, or q to quit--- Thread 15 (LWP 20892): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=5, result=0x6) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000001 in ?? () #2 0x0000000000000000 in ?? () Thread 14 (LWP 20889): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=0, result=0x3) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000000 in ?? () Thread 13 (LWP 20896): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=4, result=0xa) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000004 in ?? () #2 0x0000000000000000 in ?? () Thread 12 (LWP 20895): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=4, result=0x9) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000004 in ?? () #2 0x0000000000000000 in ?? () Thread 11 (LWP 20900): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=4, result=0xe) at ../nss/getXXbyYY_r.c:297 ---Type to continue, or q to quit--- #1 0x0000000000000004 in ?? () #2 0x0000000000000000 in ?? () Thread 10 (LWP 20906): #0 0x00007effe124d6d5 in __pthread_cond_init (cond=0x7effe2e00ef4, cond_attr=0x80) at pthread_cond_init.c:40 #1 0x0000000000000000 in ?? () Thread 9 (LWP 20893): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=3, result=0x7) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000007 in ?? () #2 0x0000000000000000 in ?? () Thread 8 (LWP 20881): #0 0x00007effe1250bdd in __recvmsg_nocancel () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000000000 in ?? () Thread 7 (LWP 20891): #0 0x00007effe12501bd in unwind_stop (version=2013313424, actions=, exc_class=2, exc_obj=0xffffffffffffffff, context=0x7eff7800b990, stop_parameter=0x519b) at unwind.c:98 #1 0x0000000000000000 in ?? () Thread 6 (LWP 20882): #0 0x00007effe1251101 in __libc_tcdrain (fd=32511) at ../sysdeps/unix/sysv/linux/tcdrain.c:34 #1 0x0000000000000000 in ?? () Thread 5 (LWP 20905): #0 0x00007effe0b0e5c0 in tdestroy_recurse (freefct=0x7effe2df4d70, root=0x7eff7802dec0) at tsearch.c:640 #1 tdestroy_recurse (freefct=0x7effe2df4d70, root=0x7effbe7fbe60) at tsearch.c:641 #2 tdestroy_recurse (freefct=0x7effe2df4d70, root=0x7effe2df4e00) at tsearch.c:639 #3 tdestroy_recurse (freefct=0x7effe2df4d70, root=0x7effe2dee590) at tsearch.c:641 ---Type to continue, or q to quit--- #4 tdestroy_recurse (root=0x7effe2de52d8, freefct=0x7effe2df4d70) at tsearch.c:641 #5 0x00007effe2df4e00 in ?? () #6 0x00007effbe7fbe60 in ?? () #7 0x00007effd97e1b40 in fuse_thread_proc () from /usr/lib64/glusterfs/5.1/xlator/mount/fuse.so #8 0x00007effe1249dc5 in start_thread (arg=0x7effbe7fc700) at pthread_create.c:308 #9 0x00007effe0b1776d in putspent (p=0x0, stream=0x7effbe7fc700) at putspent.c:60 #10 0x0000000000000000 in ?? () Thread 4 (LWP 20901): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=0, result=0xf) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000000 in ?? () Thread 3 (LWP 20899): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=5, result=0xd) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000001 in ?? () #2 0x0000000000000000 in ?? () Thread 2 (LWP 20902): #0 0x00007effe0b17d43 in __getspnam_r (name=0x0, resbuf=0x7effe2de60b0, buffer=0x7effe244a340 "\205\300t\277\203\370\377\017\204\203\001", buflen=0, result=0x10) at ../nss/getXXbyYY_r.c:297 #1 0x0000000000000000 in ?? () Thread 1 (LWP 20887): #0 0x00007effe2411775 in __gf_free () from /lib64/libglusterfs.so.0 #1 0x00007effe23da649 in dict_destroy () from /lib64/libglusterfs.so.0 #2 0x00007effd48288b4 in afr_local_cleanup () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #3 0x00007effd4802ab4 in afr_transaction_done () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so ---Type to continue, or q to quit--- #4 0x00007effd480919a in afr_unlock () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #5 0x00007effd4800819 in afr_changelog_post_op_done () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #6 0x00007effd480362c in afr_changelog_post_op_now () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #7 0x00007effd4804f1b in afr_transaction_start () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #8 0x00007effd480537a in afr_transaction () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #9 0x00007effd47fd562 in afr_fsync () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #10 0x00007effd45971b8 in dht_fsync () from /usr/lib64/glusterfs/5.1/xlator/cluster/distribute.so #11 0x00007effd42fd093 in wb_fsync_helper () from /usr/lib64/glusterfs/5.1/xlator/performance/write-behind.so #12 0x00007effe240e1b5 in call_resume_keep_stub () from /lib64/libglusterfs.so.0 #13 0x00007effd43038b9 in wb_do_winds () from /usr/lib64/glusterfs/5.1/xlator/performance/write-behind.so #14 0x00007effd43039cb in wb_process_queue () from /usr/lib64/glusterfs/5.1/xlator/performance/write-behind.so #15 0x00007effd4303b5f in wb_fulfill_cbk () from /usr/lib64/glusterfs/5.1/xlator/performance/write-behind.so #16 0x00007effd45855f9 in dht_writev_cbk () from /usr/lib64/glusterfs/5.1/xlator/cluster/distribute.so #17 0x00007effd47f020e in afr_writev_unwind () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #18 0x00007effd47f07be in afr_writev_wind_cbk () from /usr/lib64/glusterfs/5.1/xlator/cluster/replicate.so #19 0x00007effd4abdbc5 in client4_0_writev_cbk () from /usr/lib64/glusterfs/5.1/xlator/protocol/client.so #20 0x00007effe21b2c70 in rpc_clnt_handle_reply () from /lib64/libgfrpc.so.0 #21 0x00007effe21b3043 in rpc_clnt_notify () from /lib64/libgfrpc.so.0 #22 0x00007effe21aef23 in rpc_transport_notify () from /lib64/libgfrpc.so.0 #23 0x00007effd6da937b in socket_event_handler () from /usr/lib64/glusterfs/5.1/rpc-transport/socket.so #24 0x00007effe244a5f9 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #25 0x00007effe1249dc5 in start_thread (arg=0x7effd54f9700) at pthread_create.c:308 #26 0x00007effe0b1776d in putspent (p=0x0, stream=0x7effd54f9700) at putspent.c:60 #27 0x0000000000000000 in ?? () -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 02:54:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 02:54:27 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #517 from Worker Ant --- REVIEW: https://review.gluster.org/21977 (timer-wheel: run the timer function outside of locked region) posted (#3) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 04:26:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 04:26:15 +0000 Subject: [Bugs] [Bug 1664524] New: Non-root geo-replication session goes to faulty state, when the session is started Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664524 Bug ID: 1664524 Summary: Non-root geo-replication session goes to faulty state, when the session is started Product: GlusterFS Version: 3.12 Status: NEW Component: geo-replication Assignee: bugs at gluster.org Reporter: amannath at cisco.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Non root geo-replication goes to faulty state when session is started. MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ---------------------------------------------------------------------------------------------------------------------------------------------------------------- server-114-74 gv100 /data/glusterfs/gv100 geoaccount geoaccount at server-115-215::gv100 N/A Faulty N/A N/A server-115-37 gv100 /data/glusterfs/gv100 geoaccount geoaccount at server-115-215::gv100 N/A Faulty N/A N/A Relevant Logs -------------------- [2019-01-09 01:25:10.111979] I [gsyncd(monitor):715:main_i] : Monitor Status: Created [2019-01-09 01:28:17.722640] I [gsyncd(conf):689:main_i] : Config Set: session-owner = 09b47d3f-553a-45f8-afd9-848d3f51a62f [2019-01-09 01:28:18.276072] I [gsyncdstatus(monitor):241:set_worker_status] GeorepStatus: Worker Status: Initializing... [2019-01-09 01:28:18.449407] I [monitor(monitor):275:monitor] Monitor: starting gsyncd worker(/data/glusterfs/gv100). Slave node: ssh://geoaccount at amannath-115-201:gluster://localhost:gv100 [2019-01-09 01:28:18.537137] I [changelogagent(/data/glusterfs/gv100):73:__init__] ChangelogAgent: Agent listining... [2019-01-09 01:28:19.385097] E [syncdutils(/data/glusterfs/gv100):270:log_raise_exception] : connection to peer is broken [2019-01-09 01:28:19.385895] E [resource(/data/glusterfs/gv100):234:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-u_GpUW/bd052dee486819c3bb070d3ed44cf6dc.sock geoaccount at amannath-115-201 /nonexistent/gsyncd --session-owner 09b47d3f-553a-45f8-afd9-848d3f51a62f --local-id .%2Fdata%2Fglusterfs%2Fgv100 --local-node amannath-114-74 -N --listen --timeout 120 gluster://localhost:gv100" returned with 1, saying: [2019-01-09 01:28:19.386127] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.065573] I [cli.c:759:main] 0-cli: Started running /usr/sbin/gluster with version 3.10.12 [2019-01-09 01:28:19.386301] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.065685] I [cli.c:642:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost [2019-01-09 01:28:19.386548] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.168915] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-09 01:28:19.386713] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.169141] I [socket.c:2426:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-09 01:28:19.386850] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.170204] I [cli-rpc-ops.c:7024:gf_cli_getwd_cbk] 0-cli: Received resp to getwd [2019-01-09 01:28:19.387007] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.170325] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2019-01-09 01:28:19.387537] I [syncdutils(/data/glusterfs/gv100):238:finalize] : exiting. [2019-01-09 01:28:19.389434] I [repce(/data/glusterfs/gv100):92:service_loop] RepceServer: terminating on reaching EOF. [2019-01-09 01:28:19.389765] I [syncdutils(/data/glusterfs/gv100):238:finalize] : exiting. [2019-01-09 01:28:19.389770] I [monitor(monitor):347:monitor] Monitor: worker(/data/glusterfs/gv100) died before establishing connection [2019-01-09 01:28:19.394583] I [gsyncdstatus(monitor):241:set_worker_status] GeorepStatus: Worker Status: Faulty Version-Release number of selected component (if applicable): glusterfs 3.10.12 How reproducible: Always Steps to Reproduce: 1.Follow the official documentation (https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/) to setup geo replication for non root user. 2. 3. Actual results: Geo replication session is faulty Expected results: Geo replication session should work properly and replicate the data Additional info: I had a look at the Bug 1222750, with similar symptoms, but it is marked as fixed in version 3.7. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 04:47:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 04:47:36 +0000 Subject: [Bugs] [Bug 1664215] Toggling readdir-ahead translator off causes some clients to umount some of its volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664215 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |high Assignee|atumball at redhat.com |rgowdapp at redhat.com --- Comment #3 from Amar Tumballi --- > [2019-01-04 04:55:51.453074] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: switched to graph 4 Notice the graph number is 4 here. > [2019-01-04 04:55:51.454990] I [MSGID: 114021] [client.c:2369:notify] 2-el_data-client-0: current graph is no longer active, destroying rpc_client > [2019-01-04 04:55:51.455028] I [MSGID: 114021] [client.c:2369:notify] 2-el_data-client-1: current graph is no longer active, destroying rpc_client > [2019-01-04 04:55:51.455037] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-el_data-client-0: disconnected from el_data-client-0. Client process will keep trying to connect to glusterd until brick's port is available > [2019-01-04 04:55:51.455080] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-el_data-client-1: disconnected from el_data-client-1. Client process will keep trying to connect to glusterd until brick's port is available All the above logs are from graph 2, (2-el_data-client-0 signifies, in graph 2, volume name el_data, and client-0 is protocol/client translator). So this is expected. I don't think this is the reason for the umount. These logs can turn up in any such graph change options (like write-behind on/off, read-ahead on/off, etc). Not sure if this is enough to debug this umount issues. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 04:48:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 04:48:21 +0000 Subject: [Bugs] [Bug 1664215] Toggling readdir-ahead translator off causes some clients to umount some of its volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664215 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|rgowdapp at redhat.com |varao at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 05:08:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 05:08:56 +0000 Subject: [Bugs] [Bug 1664335] [geo-rep]: Transport endpoint not connected with arbiter volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664335 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nbalacha at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 06:29:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 06:29:18 +0000 Subject: [Bugs] [Bug 1664551] New: Wrong description of localtime-logging in manpages Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664551 Bug ID: 1664551 Summary: Wrong description of localtime-logging in manpages Product: GlusterFS Version: mainline Status: NEW Component: doc Assignee: bugs at gluster.org Reporter: kinglongmee at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: The manpages of glusterd/glusterfs/glusterfsd contains wrong description of localtime-logging, -L, --localtime-logging=on|off Enable or disable localtime log timestamps. Valid options are on and off (the default is off). -L is the short option of --log-level, not --localtime-logging, and also, --localtime-logging does not contain an argument. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 07:30:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 07:30:56 +0000 Subject: [Bugs] [Bug 1664335] [geo-rep]: Transport endpoint not connected with arbiter volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664335 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22001 (geo-rep: Fix for \"Transport End Point not connected\" issue) posted (#6) for review on master by Harpreet Kaur Lalwani -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 09:08:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 09:08:02 +0000 Subject: [Bugs] [Bug 1664590] New: Gluster peer probe doesn't work for IPv6 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Bug ID: 1664590 Summary: Gluster peer probe doesn't work for IPv6 Product: Red Hat Gluster Storage Version: 3.4 OS: Linux Status: NEW Component: rpc Severity: high Assignee: mchangir at redhat.com Reporter: mchangir at redhat.com QA Contact: rhinduja at redhat.com CC: bugs at gluster.org, rhs-bugs at redhat.com, sankarshan at redhat.com, y.zhao at nokia.com Depends On: 1635863 Target Milestone: --- Classification: Red Hat +++ This bug was initially created as a clone of Bug #1635863 +++ Description of problem: # gluster peer probe gluster-1 peer probe: failed: Probe returned with Transport endpoint is not connected Version-Release number of selected component (if applicable): # gluster peer probe gluster-1 peer probe: failed: Probe returned with Transport endpoint is not connected How reproducible: Install 4.1.4 and run gluster peer probe ipv6-peer Steps to Reproduce: 1. turn on "option transport.address-family inet6" in glusterd.vol 2. Verify the remote peer is ping6-able 3. Run "gluster peer probe" Actual results: # gluster peer probe gluster-1 peer probe: failed: Probe returned with Transport endpoint is not connected # ping6 gluster-1 PING gluster-1(gluster-1 (3010::13:199:0:0:42)) 56 data bytes 64 bytes from gluster-1 (3010::13:199:0:0:42): icmp_seq=1 ttl=64 time=1.54 ms 64 bytes from gluster-1 (3010::13:199:0:0:42): icmp_seq=2 ttl=64 time=0.439 ms Expected results: Peer probe should succeeds. Additional info: 1). Log from glusterd.log [2018-10-03 19:06:25.009874] I [MSGID: 106487] [glusterd-handler.c:1244:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req gluster-1 24007 [2018-10-03 19:06:25.010729] I [MSGID: 106128] [glusterd-handler.c:3635:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: gluster-1 (24007) [2018-10-03 19:06:25.028897] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2018-10-03 19:06:25.029031] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-10-03 19:06:25.033267] E [MSGID: 101075] [common-utils.c:312:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2018-10-03 19:06:25.033366] E [name.c:267:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host gluster-1 [2018-10-03 19:06:25.033538] I [MSGID: 106498] [glusterd-handler.c:3561:glusterd_friend_add] 0-management: connect returned 0 [2018-10-03 19:06:25.033657] I [MSGID: 106004] [glusterd-handler.c:6382:__glusterd_peer_rpc_notify] 0-management: Peer (<00000000-0000-0000-0000-000000000000>), in state , has disconnected from glusterd. 2). Peer probe works for ipv4 address with the same config --- Additional comment from Yan on 2018-10-15 21:03:27 IST --- Please refer: https://bugzilla.redhat.com/show_bug.cgi?id=1609799 Could any one rebuild newer libtirpc and glusterfs with --with-ipv6-default flag to fix the issue? --- Additional comment from Worker Ant on 2018-12-28 15:35:02 IST --- REVIEW: https://review.gluster.org/21948 ([WIP] rpc: use address-family option from vol file) posted (#3) for review on master by Milind Changire Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1635863 [Bug 1635863] Gluster peer probe doesn't work for IPv6 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 09:08:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 09:08:02 +0000 Subject: [Bugs] [Bug 1635863] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1635863 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1664590 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 [Bug 1664590] Gluster peer probe doesn't work for IPv6 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 09:33:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 09:33:31 +0000 Subject: [Bugs] [Bug 1664590] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |high Blocks| |1624708 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1624708 [Bug 1624708] RHHI: add gluster storage domain over IPv6 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 09:51:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 09:51:09 +0000 Subject: [Bugs] [Bug 1664590] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1618669 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1618669 [Bug 1618669] [RFE] Enable gluster in IPv6 only scenarios -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 11:34:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 11:34:42 +0000 Subject: [Bugs] [Bug 1664647] New: dht: Add NULL check for stbuf in dht_rmdir_lookup_cbk Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664647 Bug ID: 1664647 Summary: dht: Add NULL check for stbuf in dht_rmdir_lookup_cbk Product: GlusterFS Version: mainline Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: spalai at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: In the case of lookup failure, AFR is passing NULL iatt back to dht, which on dereference can result in a crash. Affected function dht_rmdir_lookup_cbk. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 11:37:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 11:37:29 +0000 Subject: [Bugs] [Bug 1664647] dht: Add NULL check for stbuf in dht_rmdir_lookup_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664647 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22004 (dht: Add NULL check for stbuf in dht_rmdir_lookup_cbk) posted (#2) for review on master by Susant Palai -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 11:37:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 11:37:30 +0000 Subject: [Bugs] [Bug 1664647] dht: Add NULL check for stbuf in dht_rmdir_lookup_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664647 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22004 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 14:59:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 14:59:44 +0000 Subject: [Bugs] [Bug 1664215] Toggling readdir-ahead translator off causes some clients to umount some of its volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664215 --- Comment #4 from Amgad --- Thanks Amar: The full log-set on the client experienced the disconnect was attached to the ticket for your investigation. Please let us know if any additional traces can be turned-on when it happens again to help debugging on your side. Regards, Amgad -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 15:16:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:16:43 +0000 Subject: [Bugs] [Bug 1651323] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651323 --- Comment #14 from Worker Ant --- REVIEW: https://review.gluster.org/21961 (io-cache: xdata needs to be passed for readv operations) posted (#1) for review on release-5 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:17:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:17:02 +0000 Subject: [Bugs] [Bug 1663131] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663131 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21985 (gfapi: Access fs->oldvolfile under mutex lock) posted (#4) for review on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:17:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:17:46 +0000 Subject: [Bugs] [Bug 1662200] NL cache: fix typos In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662200 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21930 (core: Fixed typos in nl-cache and logging-guidelines.md) posted (#3) for review on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 15:19:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:19:59 +0000 Subject: [Bugs] [Bug 1659563] gluster-blockd segfaults because of a null-dereference in shard.so In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659563 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ndevos at redhat.com, | |srangana at redhat.com Flags| |needinfo?(ndevos at redhat.com | |) --- Comment #3 from Shyamsundar --- (In reply to Worker Ant from comment #2) > REVIEW: https://review.gluster.org/21866 (shard: prevent segfault in > shard_unlink_block_inode()) posted (#2) for review on release-5 by > Shyamsundar Ranganathan The above patch uses the "Updates" keyword, but there are no pending patches, so is the tag in the commit message correct? or are we expecting more patches around this? -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 15:21:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:21:56 +0000 Subject: [Bugs] [Bug 1662635] Fix tests/bugs/shard/zero-flag.t In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662635 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21967 (tests: Fix zero-flag.t script) posted (#2) for review on release-4.1 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:21:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:21:57 +0000 Subject: [Bugs] [Bug 1662635] Fix tests/bugs/shard/zero-flag.t In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662635 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21967 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:22:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:22:56 +0000 Subject: [Bugs] [Bug 1663132] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663132 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21986 (gfapi: Access fs->oldvolfile under mutex lock) posted (#1) for review on release-4.1 by soumya k -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:23:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:23:16 +0000 Subject: [Bugs] [Bug 1655532] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655532 --- Comment #9 from Worker Ant --- REVIEW: https://review.gluster.org/21964 (io-cache: xdata needs to be passed for readv operations) posted (#2) for review on release-4.1 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:23:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:23:37 +0000 Subject: [Bugs] [Bug 1655532] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655532 --- Comment #10 from Worker Ant --- REVIEW: https://review.gluster.org/21782 (lease: Treat unlk request as noop if lease not found) posted (#4) for review on release-4.1 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:23:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:23:57 +0000 Subject: [Bugs] [Bug 1655532] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655532 --- Comment #11 from Worker Ant --- REVIEW: https://review.gluster.org/21784 (leases: Do not conflict with internal fops) posted (#4) for review on release-4.1 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:24:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:24:16 +0000 Subject: [Bugs] [Bug 1655532] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655532 --- Comment #12 from Worker Ant --- REVIEW: https://review.gluster.org/21783 (gfapi: Offload callback notifications to synctask) posted (#4) for review on release-4.1 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:35:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:35:14 +0000 Subject: [Bugs] [Bug 1651323] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651323 --- Comment #15 from Worker Ant --- REVIEW: https://review.gluster.org/22005 (leases: Reset lease_ctx->timer post deletion) posted (#1) for review on release-5 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:35:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:35:15 +0000 Subject: [Bugs] [Bug 1651323] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651323 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22005 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 15:39:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:39:04 +0000 Subject: [Bugs] [Bug 1659563] gluster-blockd segfaults because of a null-dereference in shard.so In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659563 Niels de Vos changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ndevos at redhat.com | |) | --- Comment #4 from Niels de Vos --- (In reply to Shyamsundar from comment #3) > (In reply to Worker Ant from comment #2) > > REVIEW: https://review.gluster.org/21866 (shard: prevent segfault in > > shard_unlink_block_inode()) posted (#2) for review on release-5 by > > Shyamsundar Ranganathan > > The above patch uses the "Updates" keyword, but there are no pending > patches, so is the tag in the commit message correct? or are we expecting > more patches around this? This is the only patch that I expect is needed. If you prefer Closes: or Fixes: as a tag, feel free to change the commit message :) -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 9 15:39:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 15:39:36 +0000 Subject: [Bugs] [Bug 1655532] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655532 --- Comment #13 from Worker Ant --- REVIEW: https://review.gluster.org/22006 (leases: Reset lease_ctx->timer post deletion) posted (#1) for review on release-4.1 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 9 21:59:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 09 Jan 2019 21:59:47 +0000 Subject: [Bugs] [Bug 1651323] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651323 --- Comment #16 from Worker Ant --- REVIEW: https://review.gluster.org/22005 (leases: Reset lease_ctx->timer post deletion) posted (#1) for review on release-5 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 05:04:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 05:04:18 +0000 Subject: [Bugs] [Bug 1662679] Log connection_id in statedump for posix-locks as well for better debugging experience In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662679 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21968 (features/locks: Dump connection_id even for posix locks) posted (#5) for review on master by Krutika Dhananjay -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 10 05:17:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 05:17:17 +0000 Subject: [Bugs] [Bug 1664934] New: glusterfs-fuse not client not benefiting from page cache on read after write Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Bug ID: 1664934 Summary: glusterfs-fuse not client not benefiting from page cache on read after write Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: fuse Severity: high Assignee: bugs at gluster.org Reporter: mpillai at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: On a simple single brick distribute volume, I'm running tests to validate glusterfs-fuse client's use of page cache. The tests are indicating that a read following a write is reading from the brick, not from client cache. In contrast, a 2nd read gets data from the client cache. Version-Release number of selected component (if applicable): glusterfs-*5.2-1.el7.x86_64 kernel-3.10.0-957.el7.x86_64 (RHEL 7.6) How reproducible: Consistently Steps to Reproduce: 1. use fio to create a data set that would fit easily in the page cache. My client has 128 GB RAM; I'll create a 64 GB data set: fio --name=initialwrite --ioengine=sync --rw=write \ --direct=0 --create_on_open=1 --end_fsync=1 --bs=128k \ --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum \ --filesize=16g --size=16g --numjobs=4 2. run an fio read test that reads the data set from step 1, without invalidating the page cache: fio --name=readtest --ioengine=sync --rw=read --invalidate=0 \ --direct=0 --bs=128k --directory=/mnt/glustervol/ \ --filename_format=f.\$jobnum.\$filenum --filesize=16g \ --size=16g --numjobs=4 Read throughput is much lower than it would be if reading from page cache: READ: bw=573MiB/s (601MB/s), 143MiB/s-144MiB/s (150MB/s-150MB/s), io=64.0GiB (68.7GB), run=114171-114419msec Reads are going over the 10GbE network as shown in (edited) sar output: 05:01:04 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s 05:01:06 AM em1 755946.26 40546.26 1116287.75 3987.24 0.00 [There is some read amplification here: application is getting lower throughput than what client is reading over the n/w. More on that later] 3. Run the read test in step 2 again. This time read throughput is really high, indicating read from cache, rather than over the network: READ: bw=14.8GiB/s (15.9GB/s), 3783MiB/s-4270MiB/s (3967MB/s-4477MB/s), io=64.0GiB (68.7GB), run=3837-4331msec Expected results: The read test in step 2 should be reading from page cache, and should be giving throughput close to what we get in step 3. Additional Info: gluster volume info: Volume Name: perfvol Type: Distribute Volume ID: 7033539b-0331-44b1-96cf-46ddc6ee2255 Status: Started Snapshot Count: 0 Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 172.16.70.128:/mnt/rhs_brick1 Options Reconfigured: transport.address-family: inet nfs.disable: on -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 05:19:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 05:19:00 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse not client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Manoj Pillai changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Performance CC| |csaba at redhat.com, | |rgowdapp at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 05:24:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 05:24:59 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Manoj Pillai changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|glusterfs-fuse not client |glusterfs-fuse client not |not benefiting from page |benefiting from page cache |cache on read after write |on read after write -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 05:36:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 05:36:47 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |csaba at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 05:43:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 05:43:53 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #1 from Manoj Pillai --- (In reply to Manoj Pillai from comment #0) [...] > 1. use fio to create a data set that would fit easily in the page cache. My > client has 128 GB RAM; I'll create a 64 GB data set: > > fio --name=initialwrite --ioengine=sync --rw=write \ > --direct=0 --create_on_open=1 --end_fsync=1 --bs=128k \ > --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum \ > --filesize=16g --size=16g --numjobs=4 > Memory usage on the client while the write test is running: # sar -r 5 Linux 3.10.0-957.el7.x86_64 (c09-h08-r630.rdu.openstack.engineering.redhat.com) 01/10/2019 _x86_64_ (56 CPU) 05:35:36 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 05:35:41 AM 126671972 4937712 3.75 0 2974352 256704 0.18 1878020 1147776 36 05:35:46 AM 126671972 4937712 3.75 0 2974352 256704 0.18 1878020 1147776 36 05:35:51 AM 126666904 4942780 3.76 0 2974324 259900 0.19 1879948 1147772 16 05:35:56 AM 126665820 4943864 3.76 0 2974348 261300 0.19 1880304 1147776 24 05:36:01 AM 126663136 4946548 3.76 0 2974348 356356 0.25 1881500 1147772 20 05:36:06 AM 126663028 4946656 3.76 0 2974348 356356 0.25 1881540 1147772 20 05:36:11 AM 126664444 4945240 3.76 0 2974388 356356 0.25 1880648 1147788 32 05:36:16 AM 126174984 5434700 4.13 0 3449508 930284 0.66 1892912 1622536 32 05:36:21 AM 120539884 11069800 8.41 0 9076076 930284 0.66 1893784 7247852 32 05:36:26 AM 114979592 16630092 12.64 0 14620932 930284 0.66 1893796 12793472 32 05:36:31 AM 109392488 22217196 16.88 0 20192112 930284 0.66 1893796 18365764 32 05:36:36 AM 104113900 27495784 20.89 0 25457272 930284 0.66 1895152 23630336 32 05:36:41 AM 98713688 32895996 25.00 0 30842800 930284 0.66 1895156 29015400 32 05:36:46 AM 93355560 38254124 29.07 0 36190264 930688 0.66 1897548 34361664 32 05:36:51 AM 87640900 43968784 33.41 0 41885972 930688 0.66 1897556 40057860 32 05:36:56 AM 81903068 49706616 37.77 0 47626388 930688 0.66 1897004 45798848 0 05:37:01 AM 76209860 55399824 42.09 0 53303272 930688 0.66 1897004 51475716 0 05:37:06 AM 70540340 61069344 46.40 0 58956264 930688 0.66 1897004 57128836 0 05:37:11 AM 64872776 66736908 50.71 0 64609648 930688 0.66 1897000 62782624 0 05:37:16 AM 59376144 72233540 54.88 0 70096880 930688 0.66 1897368 68270084 0 05:37:21 AM 71333376 60276308 45.80 0 58169584 356740 0.25 1891388 56342848 0 05:37:26 AM 126653336 4956348 3.77 0 2974476 356740 0.25 1891392 1148348 0 05:37:31 AM 126654360 4955324 3.77 0 2974388 356740 0.25 1891380 1147784 0 05:37:36 AM 126654376 4955308 3.77 0 2974388 356740 0.25 1891380 1147784 0 05:37:41 AM 126654376 4955308 3.77 0 2974388 356740 0.25 1891380 1147784 0 So as the write test progresses, kbcached steadily increases. But looks like the cached data is dropped subsequently. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 10 05:52:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 05:52:14 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #2 from Manoj Pillai --- When I run the same sequence of tests on an XFS file system on the server, I get expected results: both step 2. and step 3. of comment #0 report high read throughput (15+GiB/s) indicating data is read from the page cache. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 10 09:48:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 09:48:52 +0000 Subject: [Bugs] [Bug 1664647] dht: Add NULL check for stbuf in dht_rmdir_lookup_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664647 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22004 (dht: Add NULL check for stbuf in dht_rmdir_lookup_cbk) posted (#4) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 10:35:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 10:35:13 +0000 Subject: [Bugs] [Bug 1665029] New: read-ahead and io-cache degrading performance on sequential read Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 Bug ID: 1665029 Summary: read-ahead and io-cache degrading performance on sequential read Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: read-ahead Severity: high Assignee: bugs at gluster.org Reporter: mpillai at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: A large-file sequential read test reports better throughput with read-ahead and io-cache xlators turned off. One test is obviously not enough to decide the fate of these xlators. But through this bz we can answer some of the relevant questions. Version-Release number of selected component (if applicable): glusterfs-*5.2-1.el7.x86_64 kernel-3.10.0-957.el7.x86_64 (RHEL 7.6) How reproducible: Consistently Steps to Reproduce: 1. fio write test to generate data set: fio --name=initialwrite --ioengine=sync --rw=write \ --direct=0 --create_on_open=1 --end_fsync=1 --bs=128k \ --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum \ --filesize=16g --size=16g --numjobs=4 2. unmount and re-mount volume on client 3. fio sequential read test to read back the data written in step 1: fio --name=readtest --ioengine=sync --rw=read \ --direct=0 --bs=128k --directory=/mnt/glustervol/ \ --filename_format=f.\$jobnum.\$filenum --filesize=16g \ --size=16g --numjobs=4 Actual results: With default volume settings: READ: bw=581MiB/s (610MB/s), 145MiB/s-146MiB/s (152MB/s-153MB/s), io=64.0GiB (68.7GB), run=112401-112717msec With read-ahead and io-cache turned off: READ: bw=1083MiB/s (1136MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io=64.0GiB (68.7GB), run=60487-60491msec So in this test, there is a significant performance gain with these xlators turned off. The low performance with default volume settings was first seen in another bz:https://bugzilla.redhat.com/show_bug.cgi?id=1664934#c0. Expected results: performance xlators should not degrade performance Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 10:36:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 10:36:08 +0000 Subject: [Bugs] [Bug 1665029] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 Manoj Pillai changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Performance CC| |csaba at redhat.com, | |rgowdapp at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 11:01:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 11:01:23 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #3 from Manoj Pillai --- (In reply to Manoj Pillai from comment #0) [...] > > Read throughput is much lower than it would be if reading from page cache: > READ: bw=573MiB/s (601MB/s), 143MiB/s-144MiB/s (150MB/s-150MB/s), io=64.0GiB > (68.7GB), run=114171-114419msec > > Reads are going over the 10GbE network as shown in (edited) sar output: > 05:01:04 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s > 05:01:06 AM em1 755946.26 40546.26 1116287.75 3987.24 0.00 > > [There is some read amplification here: application is getting lower > throughput than what client is reading over the n/w. More on that later] > This turned out to be primarily read-ahead related. Open a new bug for it: https://bugzilla.redhat.com/show_bug.cgi?id=1665029. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 10 11:09:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 11:09:29 +0000 Subject: [Bugs] [Bug 1665038] New: glusterd crashed while running "gluster get-state glusterd odir /get-state" Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665038 Bug ID: 1665038 Summary: glusterd crashed while running "gluster get-state glusterd odir /get-state" Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: srakonde at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: glusterd crashed while running "gluster get-state glusterd odir /get-state" Version-Release number of selected component (if applicable): master How reproducible: always Actual results: glusterd has crashed Expected results: no crashed should be seen Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 11:11:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 11:11:30 +0000 Subject: [Bugs] [Bug 1665038] glusterd crashed while running "gluster get-state glusterd odir /get-state" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665038 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22007 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 11:11:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 11:11:29 +0000 Subject: [Bugs] [Bug 1665038] glusterd crashed while running "gluster get-state glusterd odir /get-state" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665038 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22007 (glusterd: fix crash) posted (#1) for review on master by Sanju Rakonde -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 11:12:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 11:12:13 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21938 (cluster/afr: Disable client side heals in AFR by default.) posted (#8) for review on master by Ravishankar N -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 10 12:07:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 12:07:09 +0000 Subject: [Bugs] [Bug 1665055] New: kernel-writeback-cache option does not seem to be working Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665055 Bug ID: 1665055 Summary: kernel-writeback-cache option does not seem to be working Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: fuse Severity: high Assignee: bugs at gluster.org Reporter: mpillai at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: https://github.com/gluster/glusterfs/issues/435 adds support for writeback cache with fuse. However, it doesn't seem to be working as I would expect. In write tests, I don't see an dirty data in page cache increasing. Version-Release number of selected component (if applicable): glusterfs-*5.2-1.el7.x86_64 kernel-3.10.0-957.el7.x86_64 (RHEL 7.6) How reproducible: Consistently Steps to Reproduce: 1. create single brick distribute volume with default settings. 2. mount gluster volume with the kernel-writeback-cache option. output of ps showing that this has been done: /usr/sbin/glusterfs --kernel-writeback-cache=yes --process-name fuse --volfile-server=172.16.70.128 --volfile-id=/perfvol /mnt/glustervol 3. run an fio write test without fsync options: fio --name=initialwrite --ioengine=sync --rw=write \ --direct=0 --create_on_open=1 --bs=128k --directory=/mnt/glustervol/ \ --filename_format=f.\$jobnum.\$filenum --filesize=16g \ --size=16g --numjobs=4 Actual results: Not seeing any dirty data accumulating in the page cache: # sar -r 5 Linux 3.10.0-957.el7.x86_64 (c09-h08-r630.rdu.openstack.engineering.redhat.com) 01/10/2019 _x86_64_ (56 CPU) 11:50:23 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 11:50:28 AM 126637320 4972364 3.78 0 2984068 363436 0.26 1884000 1156308 32 11:50:33 AM 126637320 4972364 3.78 0 2984068 363436 0.26 1884000 1156308 32 11:50:38 AM 126637320 4972364 3.78 0 2984068 363436 0.26 1884000 1156308 32 11:50:43 AM 125801556 5808128 4.41 0 3808880 937648 0.67 1896732 1980992 0 11:50:48 AM 120168932 11440752 8.69 0 9428756 937648 0.67 1896912 7599108 4 11:50:53 AM 114769368 16840316 12.80 0 14815316 937648 0.67 1896912 12986512 4 11:50:58 AM 109458768 22150916 16.83 0 20116092 937648 0.67 1897396 18287780 4 11:51:03 AM 104207304 27402380 20.82 0 25364236 937648 0.67 1897424 23535716 0 11:51:08 AM 98995764 32613920 24.78 0 30564848 937648 0.67 1897408 28735148 0 11:51:13 AM 93582944 38026740 28.89 0 35965720 937648 0.67 1897408 34136384 0 11:51:18 AM 88071656 43538028 33.08 0 41463728 937648 0.67 1897408 39634616 0 11:51:23 AM 82411904 49197780 37.38 0 47106212 937648 0.67 1897408 45275676 0 11:51:28 AM 76742608 54867076 41.69 0 52761136 937648 0.67 1897408 50932124 0 11:51:33 AM 71736380 59873304 45.49 0 57754148 937648 0.67 1897408 55924636 0 11:51:38 AM 66740952 64868732 49.29 0 62738164 937648 0.67 1897408 60908384 0 11:51:43 AM 61620148 69989536 53.18 0 67843088 937648 0.67 1897408 66014100 0 11:51:48 AM 59375388 72234296 54.89 0 70091108 363792 0.26 1893552 68261796 0 11:51:53 AM 59375388 72234296 54.89 0 70091108 363792 0.26 1893552 68261796 0 Expected results: Evidence of dirty data building up in the page cache. Additional info: For comparison, the same test was run on an XFS file system on the server (the FS that would serve as the brick for the gluster volume). (The h/w spec of server is not same as of client; it has more RAM e.g.). In this case, we can see buildup of dirty data: # sar -r 5 Linux 3.10.0-957.el7.x86_64 (c06-h05-6048r.rdu.openstack.engineering.redhat.com) 01/10/2019 _x86_64_ (56 CPU) 11:46:33 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 11:46:38 AM 261061052 2794896 1.06 1124 244356 326432 0.12 132284 162168 40 11:46:43 AM 261061052 2794896 1.06 1124 244356 326432 0.12 132284 162168 40 11:46:48 AM 261061052 2794896 1.06 1124 244356 326432 0.12 132284 162168 40 11:46:53 AM 261061052 2794896 1.06 1124 244356 326432 0.12 132284 162168 40 11:46:58 AM 261061052 2794896 1.06 1124 244356 326432 0.12 132284 162168 40 11:47:03 AM 261061052 2794896 1.06 1124 244356 326432 0.12 132284 162168 40 11:47:08 AM 245576160 18279788 6.93 1124 15005000 896264 0.33 134232 14922152 4537836 11:47:13 AM 237023884 26832064 10.17 1124 23303832 896264 0.33 134236 23220484 4845900 11:47:18 AM 228223240 35632708 13.50 1124 31822796 896264 0.33 134236 31741232 4901984 11:47:23 AM 219775288 44080660 16.71 1124 40001140 896264 0.33 134236 39917604 4654116 11:47:28 AM 211272552 52583396 19.93 1124 48319980 896264 0.33 134236 48235832 4702104 11:47:33 AM 202607168 61248780 23.21 1124 56654988 896264 0.33 134236 56571356 4592700 11:47:38 AM 193999760 69856188 26.48 1124 65109548 896264 0.33 134236 65025612 4904092 11:47:43 AM 192078956 71776992 27.20 1124 67352876 326676 0.12 133228 67268776 4629040 11:47:48 AM 192078736 71777212 27.20 1124 67353220 326676 0.12 132644 67270272 0 11:47:53 AM 192078736 71777212 27.20 1124 67353220 326676 0.12 132644 67270272 0 11:47:58 AM 192078736 71777212 27.20 1124 67353220 326676 0.12 132644 67270272 0 11:48:03 AM 192078736 71777212 27.20 1124 67353220 326676 0.12 132644 67270272 0 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 12:07:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 12:07:52 +0000 Subject: [Bugs] [Bug 1665055] kernel-writeback-cache option does not seem to be working In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665055 Manoj Pillai changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Performance CC| |csaba at redhat.com, | |rgowdapp at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 12:16:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 12:16:23 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #518 from Worker Ant --- REVIEW: https://review.gluster.org/22008 (tests: increase the timeout for distribute bug 1117851.t) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 12:16:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 12:16:25 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22008 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 13:18:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 13:18:03 +0000 Subject: [Bugs] [Bug 1665029] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 --- Comment #1 from Manoj Pillai --- Data showing that both read-ahead and io-cache cause performance degradation. I'm modifying the test in comment #0 here, substituting numjobs=2, instead of 4. Test sequence: fio --name=initialwrite --ioengine=sync --rw=write --direct=0 --create_on_open=1 --end_fsync=1 --bs=128k --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum --filesize=16g --size=16g --numjobs=2 [unmount and mount volume] fio --name=readtest --ioengine=sync --rw=read --direct=0 --bs=128k --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum --filesize=16g --size=16g --numjobs=2 Result with default settings: READ: bw=485MiB/s (509MB/s), 243MiB/s-243MiB/s (254MB/s-255MB/s), io=32.0GiB (34.4GB), run=67504-67522msec Result with read-ahead turned off: READ: bw=776MiB/s (813MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=32.0GiB (34.4GB), run=42220-42237msec Result with read-ahead and io-cache turned off: READ: bw=1108MiB/s (1162MB/s), 554MiB/s-554MiB/s (581MB/s-581MB/s), io=32.0GiB (34.4GB), run=29565-29573msec -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 13:29:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 13:29:20 +0000 Subject: [Bugs] [Bug 1665055] kernel-writeback-cache option does not seem to be working In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665055 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |csaba at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 13:29:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 13:29:38 +0000 Subject: [Bugs] [Bug 1665029] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |rgowdapp at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 14:36:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 14:36:52 +0000 Subject: [Bugs] [Bug 1664647] dht: Add NULL check for stbuf in dht_rmdir_lookup_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664647 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1664529 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1664529 [Bug 1664529] [geo-rep]: Multiple crashes seen on the slave during automation run -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 15:03:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 15:03:46 +0000 Subject: [Bugs] [Bug 1665145] New: Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665145 Bug ID: 1665145 Summary: Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set Product: GlusterFS Version: 5 OS: Linux Status: NEW Component: core Assignee: bugs at gluster.org Reporter: public at hansmi.ch CC: bugs at gluster.org Target Milestone: --- Classification: Community After upgrading a Gluster cluster from version 3.12.15 to 5.2 writes to volumes with "cluster.consistent-metadata" enabled fail with EIO. Reproduction steps follow. ## Package information Package source: http://mirror.centos.org/centos/7/storage/x86_64/gluster-5/ # gluster --version glusterfs 5.2 # lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 7.6 (Maipo) Release: 7.6 Codename: Maipo ## Cluster information Peers: * storage1 * storage2 * storage3 # gluster peer status Number of Peers: 2 Hostname: storage2 Uuid: 0acf152f-4cba-4757-bdab-113c34333eeb State: Peer in Cluster (Connected) Hostname: storage3 Uuid: c0302a36-149a-44ca-984f-8120a1fa360a State: Peer in Cluster (Connected) ## Reproduction steps # On each storage server mkdir /data/testpv1 # Create volume gluster volume create testpv1 replica 3 storage1:/data/testpv1/brick storage2:/data/testpv1/brick storage3:/data/testpv1/brick force gluster volume start testpv1 # Mount volume on a client mount -t glusterfs -o log-level=DEBUG storage1:/testpv1 /mnt/gluster # Writes succeed $ echo foo > /mnt/gluster/a; ls -l /mnt/gluster/a -rw-r--r--. 1 root root 4 10. Jan 15:53 /mnt/gluster/a # Enable "cluster.consistent-metadata" gluster volume set testpv1 cluster.consistent-metadata on # Writes fail with EIO after truncation $ echo foo > /mnt/gluster/a; ls -l /mnt/gluster/a bash: /mnt/gluster/a: Input/output error -rw-r--r--. 1 root root 0 10. Jan 15:53 /mnt/gluster/a # Disable "cluster.consistent-metadata" again gluster volume set testpv1 cluster.consistent-metadata off # Writes work again $ echo foo > /mnt/gluster/a; ls -l /mnt/gluster/a -rw-r--r--. 1 root root 4 10. Jan 15:54 /mnt/gluster/a # Debug logs from FUSE client [2019-01-10 14:56:30.725669] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-testpv1-dht: revalidate lookup of / returned with op_ret 0 [Structure needs cleaning] [2019-01-10 14:56:30.725747] D [MSGID: 0] [dht-common.c:1648:dht_revalidate_cbk] 0-testpv1-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-01-10 14:56:30.726483] D [write-behind.c:1764:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x9a0b)[0x7fcac6b20a0b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0xd288)[0x7fcac6b24288] (--> /lib64/libglusterfs.so.0(default_lookup+0xdd)[0x7fcad4ea5c1d] (--> /lib64/libglusterfs.so.0(default_lookup+0xdd)[0x7fcad4ea5c1d] ))))) 0-testpv1-write-behind: processing queues [2019-01-10 14:56:30.726519] D [MSGID: 0] [write-behind.c:1710:__wb_pick_winds] 0-testpv1-write-behind: (unique=24, fop=LOOKUP, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): picking the request for winding [2019-01-10 14:56:30.726572] D [MSGID: 0] [dht-common.c:3374:dht_do_revalidate] 0-testpv1-dht: path = /a. Calling revalidate lookup on testpv1-replicate-0 [2019-01-10 14:56:30.726939] D [write-behind.c:419:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x26b2)[0x7fcac6b196b2] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x504c)[0x7fcac6b1c04c] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x98c1)[0x7fcac6b208c1] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x99cb)[0x7fcac6b209cb] ))))) 0-testpv1-write-behind: (unique = 24, fop=LOOKUP, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): destroying request, removing from all queues [2019-01-10 14:56:30.727875] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-testpv1-dht: revalidate lookup of /a returned with op_ret 0 [2019-01-10 14:56:30.728405] D [write-behind.c:1764:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x9a0b)[0x7fcac6b20a0b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0xc268)[0x7fcac6b23268] (--> /usr/lib64/glusterfs/5.2/xlator/performance/read-ahead.so(+0x7331)[0x7fcac690e331] (--> /usr/lib64/glusterfs/5.2/xlator/performance/readdir-ahead.so(+0x44f2)[0x7fcac66fb4f2] ))))) 0-testpv1-write-behind: processing queues [2019-01-10 14:56:30.728442] D [MSGID: 0] [write-behind.c:1710:__wb_pick_winds] 0-testpv1-write-behind: (unique=27, fop=TRUNCATE, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): picking the request for winding [2019-01-10 14:56:30.728722] D [write-behind.c:419:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x26b2)[0x7fcac6b196b2] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x504c)[0x7fcac6b1c04c] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x98c1)[0x7fcac6b208c1] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x99cb)[0x7fcac6b209cb] ))))) 0-testpv1-write-behind: (unique = 27, fop=TRUNCATE, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): destroying request, removing from all queues [2019-01-10 14:56:30.729207] D [MSGID: 0] [afr-transaction.c:2070:afr_post_nonblocking_inodelk_cbk] 0-testpv1-replicate-0: Non blocking inodelks done. Proceeding to FOP [2019-01-10 14:56:30.729245] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729271] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729286] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729297] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729345] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729368] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729380] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729396] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729435] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729450] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729461] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729472] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.730173] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-0: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.730289] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-2: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.730516] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-1: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.731446] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731483] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731499] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731513] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731582] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731609] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731624] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731637] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731680] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731710] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731726] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731740] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.732161] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-0: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.732480] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-2: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.732548] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-1: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.735749] D [write-behind.c:1764:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x9a0b)[0x7fcac6b20a0b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0xd288)[0x7fcac6b24288] (--> /lib64/libglusterfs.so.0(default_lookup+0xdd)[0x7fcad4ea5c1d] (--> /lib64/libglusterfs.so.0(default_lookup+0xdd)[0x7fcad4ea5c1d] ))))) 0-testpv1-write-behind: processing queues [2019-01-10 14:56:30.735787] D [MSGID: 0] [write-behind.c:1710:__wb_pick_winds] 0-testpv1-write-behind: (unique=29, fop=LOOKUP, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): picking the request for winding [2019-01-10 14:56:30.735827] D [MSGID: 0] [dht-common.c:3374:dht_do_revalidate] 0-testpv1-dht: path = /a. Calling revalidate lookup on testpv1-replicate-0 [2019-01-10 14:56:30.736354] D [write-behind.c:419:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x26b2)[0x7fcac6b196b2] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x504c)[0x7fcac6b1c04c] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x98c1)[0x7fcac6b208c1] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x99cb)[0x7fcac6b209cb] ))))) 0-testpv1-write-behind: (unique = 29, fop=LOOKUP, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): destroying request, removing from all queues [2019-01-10 14:56:30.738253] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-testpv1-dht: revalidate lookup of /a returned with op_ret 0 [2019-01-10 14:56:30.738311] D [fuse-bridge.c:53:fuse_invalidate] 0-fuse: Invalidate inode id 140508580080760. [2019-01-10 14:56:30.738390] E [fuse-bridge.c:219:check_and_dump_fuse_W] 0-glusterfs-fuse: writing to fuse device failed: No such file or directory -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 17:13:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 17:13:52 +0000 Subject: [Bugs] [Bug 1665216] New: Databases crashes on Gluster 5 with the option performance.write-behind enabled Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665216 Bug ID: 1665216 Summary: Databases crashes on Gluster 5 with the option performance.write-behind enabled Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: core Severity: high Assignee: bugs at gluster.org Reporter: gabisoft at freesurf.ch CC: bugs at gluster.org Target Milestone: --- Classification: Community Created attachment 1519880 --> https://bugzilla.redhat.com/attachment.cgi?id=1519880&action=edit asb-etcd-1-smjcf.log Description of problem: Running Etcd, Cassandra and PostgreSQL show a stacktrace after starting with DB files on Gluster 5.2 volumes, if the volume has enabled the volume option performance.write-behind. Using the Gluster volumes to serve normal files does not enforce the issue. Version-Release number of selected component (if applicable): 5.2 How reproducible: Steps to Reproduce: 1. Start Etcd with DB files on a gluster volume option performance.write-behind is on 2. Etcd does start and crashes after listening to clients (unexpected fault address 0x7fca0c001040) 3. Disable performance.write-behind on the gluster volume 4. Restart Etcd 5. Etcd does start normally Actual results: Output of a Etcd crashing (asb-etcd-1-smjcf.log) Expected results: Output of a Etcd running with performance.write-behind off (asb-etcd-3-dsfxf.log) Additional info: The content or size of the Etcd DB doesn't matter. It is also reproducible if the DB is created from scratch. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 17:14:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 17:14:19 +0000 Subject: [Bugs] [Bug 1602824] SMBD crashes when streams_attr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1602824 joao.bauto at neuro.fchampalimaud.org changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joao.bauto at neuro.fchampalim | |aud.org --- Comment #3 from joao.bauto at neuro.fchampalimaud.org --- I'm getting this issue with Gluster 5.2 and Samba 4.8.3 however it only kills the connection once a user begins copying data to the share. Putting streams_xattr after glusterfs prevents the issue but its painly slow when list a directory. [2019/01/10 17:04:09.313042, 0] ../lib/util/fault.c:261(log_stack_trace) BACKTRACE: 28 stack frames: #0 /lib64/libsamba-util.so.0(log_stack_trace+0x1a) [0x7f8fab1e5aaa] #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f8fa90e0930] #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7f8fab1e5b8f] #3 /lib64/libsamba-util.so.0(+0x24dc6) [0x7f8fab1e5dc6] #4 /lib64/libpthread.so.0(+0xf6d0) [0x7f8fab64e6d0] #5 /usr/lib64/samba/vfs/glusterfs.so(+0x43c3) [0x7f8f900173c3] #6 /usr/lib64/samba/libsmbd-base-samba4.so(is_posix_locked+0xd2) [0x7f8faae39642] #7 /usr/lib64/samba/libsmbd-base-samba4.so(brl_locktest+0x184) [0x7f8faae37174] #8 /usr/lib64/samba/libsmbd-base-samba4.so(strict_lock_check_default+0x7b) [0x7f8faae3255b] #9 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_smb2_request_process_read+0x51e) [0x7f8faae04e1e] #10 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_smb2_request_dispatch+0x164c) [0x7f8faadf75bc] #11 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_smb2_request_dispatch_immediate+0x5d) [0x7f8faadf8a2d] #12 /lib64/libtevent.so.0(tevent_common_loop_immediate+0xda) [0x7f8fa7adc9ea] #13 /lib64/libtevent.so.0(+0xa5fd) [0x7f8fa7ae15fd] #14 /lib64/libtevent.so.0(+0x8c07) [0x7f8fa7adfc07] #15 /lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f8fa7adbffd] #16 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f8fa7adc22b] #17 /lib64/libtevent.so.0(+0x8ba7) [0x7f8fa7adfba7] #18 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x5e1) [0x7f8faade6851] #19 /usr/sbin/smbd(+0xcdd0) [0x558d16af5dd0] #20 /lib64/libtevent.so.0(+0xa83b) [0x7f8fa7ae183b] #21 /lib64/libtevent.so.0(+0x8c07) [0x7f8fa7adfc07] #22 /lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f8fa7adbffd] #23 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f8fa7adc22b] #24 /lib64/libtevent.so.0(+0x8ba7) [0x7f8fa7adfba7] #25 /usr/sbin/smbd(main+0x16d7) [0x558d16af0d37] #26 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8fa772c445] #27 /usr/sbin/smbd(+0x83f0) [0x558d16af13f0] [2019/01/10 17:04:09.313331, 0] ../source3/lib/dumpcore.c:315(dump_core) -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 17:14:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 17:14:53 +0000 Subject: [Bugs] [Bug 1665216] Databases crashes on Gluster 5 with the option performance.write-behind enabled In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665216 --- Comment #1 from gabisoft at freesurf.ch --- Created attachment 1519881 --> https://bugzilla.redhat.com/attachment.cgi?id=1519881&action=edit asb-etcd-3-dsfxf.log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 10 19:17:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 19:17:17 +0000 Subject: [Bugs] [Bug 1632503] FUSE client segfaults when performance.md-cache-statfs is enabled for a volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1632503 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22009 (performance/md-cache: Fix a crash when statfs caching is enabled) posted (#1) for review on master by Vijay Bellur -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 10 19:17:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 19:17:18 +0000 Subject: [Bugs] [Bug 1632503] FUSE client segfaults when performance.md-cache-statfs is enabled for a volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1632503 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22009 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 01:48:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 01:48:17 +0000 Subject: [Bugs] [Bug 1665332] New: Wrong offset is used in offset for zerofill fop Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665332 Bug ID: 1665332 Summary: Wrong offset is used in offset for zerofill fop Product: GlusterFS Version: mainline Status: NEW Component: replicate Assignee: bugs at gluster.org Reporter: pkarampu at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: in afr_zerofill() { local->transaction.start = local->cont.discard.offset; //--> s/discard/zerofill/ } Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 01:50:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 01:50:19 +0000 Subject: [Bugs] [Bug 1665332] Wrong offset is used in offset for zerofill fop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665332 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21991 (cluster/afr: fix zerofill transaction.start) posted (#3) for review on master by Pranith Kumar Karampuri -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 02:57:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 02:57:14 +0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 --- Comment #58 from Worker Ant --- REVIEW: https://review.gluster.org/22002 (afr : fix memory leak) posted (#4) for review on master by Pranith Kumar Karampuri -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 03:21:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 03:21:37 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #519 from Worker Ant --- REVIEW: https://review.gluster.org/22008 (tests: increase the timeout for distribute bug 1117851.t) posted (#2) for review on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 03:24:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 03:24:11 +0000 Subject: [Bugs] [Bug 1632503] FUSE client segfaults when performance.md-cache-statfs is enabled for a volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1632503 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22009 (performance/md-cache: Fix a crash when statfs caching is enabled) posted (#3) for review on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 03:47:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 03:47:45 +0000 Subject: [Bugs] [Bug 1665216] Databases crashes on Gluster 5 with the option performance.write-behind enabled In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665216 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gabisoft at freesurf.ch, | |rgowdapp at redhat.com Flags| |needinfo?(gabisoft at freesurf | |.ch) --- Comment #2 from Raghavendra G --- Can you paste the backtrace here? If possible can you attach the core? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 03:48:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 03:48:15 +0000 Subject: [Bugs] [Bug 1665216] Databases crashes on Gluster 5 with the option performance.write-behind enabled In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665216 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Component|core |write-behind -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 03:51:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 03:51:36 +0000 Subject: [Bugs] [Bug 1665216] Databases crashes on Gluster 5 with the option performance.write-behind enabled In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665216 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |rgowdapp at redhat.com --- Comment #3 from Raghavendra G --- Sorry I interpreted the bug as glusterfs crashing. I see that etcd is having problems coming up. Can you get the following information (I don't need core of glusterfs, as there is none): * strace of etcd (strace -ff -v ...), to find out what syscalls it did. * dump of traffic between fuse kernel module and glusterfs (see --dump-fuse option of glusterfs) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 06:03:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 06:03:32 +0000 Subject: [Bugs] [Bug 1662906] Longevity: glusterfsd(brick process) crashed when we do volume creates and deletes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662906 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21976 (core: brick process is crashed at the time of spawn thread) posted (#6) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 06:40:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 06:40:57 +0000 Subject: [Bugs] [Bug 1665029] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 --- Comment #2 from Raghavendra G --- Some observations while debugging the performance degradation with gluster read-ahead: * Kernel too does read-ahead and it sends parallel read-requests as part of this. * client-io-threads is on in this configuration. The above two points mean parallel requests sent by kernel can reach read-ahead out of order. This means read-ahead no longer sees read requests at sequential contiguous offsets and hence it things reads are random. For random reads, it resets the read sequence. But when requests reach read-ahead in order, read-ahead is turned on again. Due to this intermittent toggling, much of read-ahead data is wasted regressing the performance. With client-io-threads off and I can no longer see the regression for the test case given. If I run the test with single fio job (--numjobs=1), gluster read-ahead on outperforms gluster read-ahead off on my setup. [1] https://review.gluster.org/#/c/glusterfs/+/20981/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 06:42:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 06:42:55 +0000 Subject: [Bugs] [Bug 1665029] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 --- Comment #3 from Raghavendra G --- (In reply to Raghavendra G from comment #2) > Some observations while debugging the performance degradation with gluster > read-ahead: > > * Kernel too does read-ahead and it sends parallel read-requests as part of > this. > * client-io-threads is on in this configuration. > > The above two points mean parallel requests sent by kernel can reach > read-ahead out of order. This means read-ahead no longer sees read requests > at sequential contiguous offsets and hence it things reads are random. For > random reads, it resets the read sequence. But when requests reach > read-ahead in order, read-ahead is turned on again. Due to this intermittent > toggling, much of read-ahead data is wasted regressing the performance. With > client-io-threads off and I can no longer see the regression for the test > case given. If I run the test with single fio job (--numjobs=1), gluster > read-ahead on outperforms gluster read-ahead off on my setup. ... single fio job (--numjobs=1), gluster read-ahead on with client-io-threads off outperforms gluster read-ahead off with client-io-threads off. > > [1] https://review.gluster.org/#/c/glusterfs/+/20981/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 06:47:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 06:47:55 +0000 Subject: [Bugs] [Bug 1665358] New: allow regression to not run tests with nfs, if nfs is disabled. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665358 Bug ID: 1665358 Summary: allow regression to not run tests with nfs, if nfs is disabled. Product: GlusterFS Version: mainline Status: NEW Component: nfs Severity: high Priority: high Assignee: atumball at redhat.com Reporter: atumball at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: In glusterfs, nfs component is optional. But in the regression tests, it is not. Version-Release number of selected component (if applicable): master How reproducible: 100% Expected results: Depending on the configure flag, even the tests using nfs should be considering enable/disable. Additional info: Considering in certain usecases, (like brick-mux enabled tests, we don't need nfs, we can save lot of regression time in brick-mux tests in this scenario). -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 06:57:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 06:57:04 +0000 Subject: [Bugs] [Bug 1665361] New: Alerts for offline nodes Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665361 Bug ID: 1665361 Summary: Alerts for offline nodes Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Assignee: bugs at gluster.org Reporter: nigelb at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community I want to have a report that tells us which Jenkins nodes are offline and why they're offline. This is offline in terms of Jenkins. We often have failures in a few nodes and it takes us a few weeks to get around to fixing them. This bug is for a solution as well as implementing it. Option 1: A jenkins job which makes API calls and sends us an email in case there are machines offline. Option 2: Nagios check which alerts us. This is slightly more explosive :) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 07:04:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:04:17 +0000 Subject: [Bugs] [Bug 1665363] New: Fix incorrect definition in index-mem-types.h Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665363 Bug ID: 1665363 Summary: Fix incorrect definition in index-mem-types.h Product: GlusterFS Version: mainline Status: NEW Component: index Assignee: bugs at gluster.org Reporter: vbellur at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: index-mem-types.h has the following definition: #ifndef __QUIESCE_MEM_TYPES_H__ #define __QUIESCE_MEM_TYPES_H__ Clearly, a yank paste error that needs to be fixed. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 07:05:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:05:19 +0000 Subject: [Bugs] [Bug 1665363] Fix incorrect definition in index-mem-types.h In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665363 Vijay Bellur changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |vbellur at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 07:09:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:09:07 +0000 Subject: [Bugs] [Bug 1665363] Fix incorrect definition in index-mem-types.h In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665363 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22011 (features/index: Correct incorrect definition in header) posted (#1) for review on master by Vijay Bellur -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 07:09:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:09:08 +0000 Subject: [Bugs] [Bug 1665363] Fix incorrect definition in index-mem-types.h In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665363 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22011 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 07:12:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:12:38 +0000 Subject: [Bugs] [Bug 1665358] allow regression to not run tests with nfs, if nfs is disabled. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665358 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22010 (tests: run nfs tests only if --enable-gnfs is provided) posted (#2) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 07:12:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:12:39 +0000 Subject: [Bugs] [Bug 1665358] allow regression to not run tests with nfs, if nfs is disabled. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665358 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22010 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 07:26:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:26:02 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Doc Type|If docs needed, set a value |Release Note --- Doc Text *updated* --- Performance degrades significantly on AFR volumes under certain circumstances due to client-side healing. In order to provide consistency in performance, this change disables client side heal for AFR volumes by updating default value for client-side heal to "off". Users have to explicitly set it to "on" if they need client-side heal for AFR volumes. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 07:30:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:30:46 +0000 Subject: [Bugs] [Bug 1560561] systemd service file enhancements In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1560561 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #6 from Worker Ant --- REVIEW: https://review.gluster.org/21723 (extras/systemd: Remove socket activated syslog.target and add documentation keys) posted (#2) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 07:43:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:43:49 +0000 Subject: [Bugs] [Bug 1664551] Wrong description of localtime-logging in manpages In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664551 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22003 (doc: fix wrong description of localtime-logging in manpages) posted (#3) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 07:43:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:43:50 +0000 Subject: [Bugs] [Bug 1664551] Wrong description of localtime-logging in manpages In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664551 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22003 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 07:44:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:44:12 +0000 Subject: [Bugs] [Bug 1657744] quorum count not updated in nfs-server vol file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657744 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21838 (libglusterfs/common-utils.c: Fix buffer size for checksum computation) posted (#9) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 07:46:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 07:46:58 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #520 from Worker Ant --- REVIEW: https://review.gluster.org/21956 (configure: fix the duplicate CFLAGS options) posted (#5) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 8 12:09:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 08 Jan 2019 12:09:39 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/21957 (features/shard: Fix launch of multiple synctasks for background deletion) posted (#7) for review on master by Xavi Hernandez -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 08:41:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 08:41:01 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 08:56:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 08:56:43 +0000 Subject: [Bugs] [Bug 1437780] don't send lookup in fuse_getattr() In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1437780 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|CURRENTRELEASE |--- --- Comment #10 from Worker Ant --- REVIEW: https://review.gluster.org/22012 (Revert \"fuse: make sure the send lookup on root instead of getattr()\") posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 08:56:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 08:56:44 +0000 Subject: [Bugs] [Bug 1437780] don't send lookup in fuse_getattr() In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1437780 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22012 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 09:33:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 09:33:07 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Version|4.1 |5 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 12:40:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 12:40:52 +0000 Subject: [Bugs] [Bug 1665029] read-ahead and io-cache degrading performance on sequential read In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665029 --- Comment #4 from Manoj Pillai --- (In reply to Manoj Pillai from comment #1) > Data showing that both read-ahead and io-cache cause performance > degradation. I'm modifying the test in comment #0 here, substituting > numjobs=2, instead of 4. > > Test sequence: > fio --name=initialwrite --ioengine=sync --rw=write --direct=0 > --create_on_open=1 --end_fsync=1 --bs=128k --directory=/mnt/glustervol/ > --filename_format=f.\$jobnum.\$filenum --filesize=16g --size=16g --numjobs=2 > [unmount and mount volume] > fio --name=readtest --ioengine=sync --rw=read --direct=0 --bs=128k > --directory=/mnt/glustervol/ --filename_format=f.\$jobnum.\$filenum > --filesize=16g --size=16g --numjobs=2 > > Result with default settings: > READ: bw=485MiB/s (509MB/s), 243MiB/s-243MiB/s (254MB/s-255MB/s), io=32.0GiB > (34.4GB), run=67504-67522msec > > Result with read-ahead turned off: > READ: bw=776MiB/s (813MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=32.0GiB > (34.4GB), run=42220-42237msec > > Result with read-ahead and io-cache turned off: > READ: bw=1108MiB/s (1162MB/s), 554MiB/s-554MiB/s (581MB/s-581MB/s), > io=32.0GiB (34.4GB), run=29565-29573msec Result with ciot=off, io-cache=off, gluster ra=on, read-ahead-page-count=10: [these settings are based on comment #2 and comment #3] READ: bw=975MiB/s (1023MB/s), 488MiB/s-488MiB/s (511MB/s-512MB/s), io=32.0GiB (34.4GB) Comparing the best results seen (1108 vs 975), the gluster ra=off case is still a little bit better. Result with ciot=off, io-cache=on, gluster ra=on, read-ahead-page-count=10: READ: bw=674MiB/s (706MB/s), 337MiB/s-339MiB/s (353MB/s-355MB/s), io=32.0GiB (34.4GB) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 14:05:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 14:05:38 +0000 Subject: [Bugs] [Bug 1665216] Databases crashes on Gluster 5 with the option performance.write-behind enabled In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665216 --- Comment #4 from Raghavendra G --- Also detailed steps for reproducer (even better a script or capture of the cmds you executed) would greatly speed up the debugging. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 16:50:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 16:50:07 +0000 Subject: [Bugs] [Bug 1644389] [GSS] Directory listings on fuse mount are very slow due to small number of getdents() entries In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1644389 Cal Calhoun changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ccalhoun at redhat.c |needinfo?(nbalacha at redhat.c |om) |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 11 18:27:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 18:27:43 +0000 Subject: [Bugs] [Bug 1665363] Fix incorrect definition in index-mem-types.h In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665363 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22011 (features/index: Correct incorrect definition in header) posted (#2) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Jan 12 02:23:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 12 Jan 2019 02:23:27 +0000 Subject: [Bugs] [Bug 1665055] kernel-writeback-cache option does not seem to be working In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665055 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(csaba at redhat.com) --- Comment #1 from Raghavendra G --- Csaba/Manoj, Amar made a comment doubting whether fuse write-back implementation being present in RHEL-7.6 (on which tests were run). Are we sure write-back implementation is present in RHEL 7.6? regards, Raghavendra -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Jan 12 05:41:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 12 Jan 2019 05:41:55 +0000 Subject: [Bugs] [Bug 1665656] New: testcaes glusterd/add-brick-and-validate-replicated-volume-options.t is crash while brick_mux is enable Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665656 Bug ID: 1665656 Summary: testcaes glusterd/add-brick-and-validate-replicated-volume-opti ons.t is crash while brick_mux is enable Product: GlusterFS Version: mainline Status: NEW Component: core Assignee: bugs at gluster.org Reporter: moagrawa at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: testcaes glusterd/add-brick-and-validate-replicated-volume-options.t is crash while brick_mux is enable Version-Release number of selected component (if applicable): How reproducible: Allways Steps to Reproduce: 1.Enable brick_mux in add-brick-and-validate-replicated-volume-options.t 2.Run .t in a loop after some attempt .t is crash 3. Actual results: Test case is crash. Expected results: Test case should not crash Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Jan 12 05:42:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 12 Jan 2019 05:42:14 +0000 Subject: [Bugs] [Bug 1665656] testcaes glusterd/add-brick-and-validate-replicated-volume-options.t is crash while brick_mux is enable In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665656 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |moagrawa at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sat Jan 12 05:46:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 12 Jan 2019 05:46:10 +0000 Subject: [Bugs] [Bug 1665656] testcaes glusterd/add-brick-and-validate-replicated-volume-options.t is crash while brick_mux is enable In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665656 --- Comment #1 from Mohit Agrawal --- Hi, test case is generating below crash after just call kill_brick. #0 0x0000560df20e821b in STACK_DESTROY (stack=0x3) at ../../libglusterfs/src/glusterfs/stack.h:182 182 LOCK(&stack->pool->lock); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.168-8.el7.x86_64 elfutils-libs-0.168-8.el7.x86_64 glibc-2.17-196.el7_4.2.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libacl-2.2.51-12.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7_4.2.x86_64 libselinux-2.5-11.el7.x86_64 libuuid-2.23.2-43.el7_4.2.x86_64 openssl-libs-1.0.2k-8.el7.x86_64 pcre-8.32-17.el7.x86_64 systemd-libs-219-42.el7_4.10.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x0000560df20e821b in STACK_DESTROY (stack=0x3) at ../../libglusterfs/src/glusterfs/stack.h:182 #1 mgmt_pmap_signin_cbk (req=, iov=, count=, myframe=0x7fda6802ddb8) at glusterfsd-mgmt.c:2824 #2 0x00007fda86559161 in rpc_clnt_handle_reply (clnt=clnt at entry=0x560df2d53b30, pollin=pollin at entry=0x560df2ea98f0) at rpc-clnt.c:755 #3 0x00007fda865594c7 in rpc_clnt_notify (trans=0x560df2d53e50, mydata=0x560df2d53b60, event=, data=0x560df2ea98f0) at rpc-clnt.c:922 #4 0x00007fda86555b33 in rpc_transport_notify (this=this at entry=0x560df2d53e50, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x560df2ea98f0) at rpc-transport.c:541 #5 0x00007fda7ab7f95d in socket_event_poll_in (notify_handled=true, this=0x560df2d53e50) at socket.c:2516 #6 socket_event_handler (fd=, idx=, gen=, data=0x560df2d53e50, poll_in=, poll_out=, poll_err=0, event_thread_died=0 '\000') at socket.c:2918 #7 0x00007fda86814e15 in event_dispatch_epoll_handler (event=0x7fda34ff8e70, event_pool=0x560df2d03560) at event-epoll.c:642 #8 event_dispatch_epoll_worker (data=0x7fda40054740) at event-epoll.c:756 #9 0x00007fda855eee25 in start_thread () from /usr/lib64/libpthread.so.0 #10 0x00007fda84ebb34d in clone () from /usr/lib64/libc.so.6 (gdb) f 1 #1 mgmt_pmap_signin_cbk (req=, iov=, count=, myframe=0x7fda6802ddb8) at glusterfsd-mgmt.c:2824 2824 STACK_DESTROY(frame->root); (gdb) p frame $1 = (call_frame_t *) 0x7fda6802ddb8 (gdb) p *frame $2 = {root = 0x4, parent = 0x400000001, frames = {next = 0xffffffffffffffff, prev = 0x7fda6802de18}, local = 0x7fda68059478, this = 0x0, ret = 0x0, ref_count = 0, lock = {spinlock = 0, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x7fda68059478}}, __size = '\000' , "x\224\005h\332\177\000", __align = 0}}, cookie = 0x0, complete = 232, op = 32730, begin = {tv_sec = 0, tv_nsec = 140576024633944}, end = {tv_sec = 140576024654704, tv_nsec = 1125216510}, wind_from = 0x1
, wind_to = 0x0, unwind_from = 0x0, unwind_to = 0x0} (gdb) p frame->root $3 = (call_stack_t *) 0x4 (gdb) After checked the code I have found the current glusterfs_mgmt_pmap_signin code is not perfect to send signin request. It uses same frame to send multiple requests. >>>>>>> ....... ....... if (ctx->active) { top = ctx->active->first; for (trav_p = &top->children; *trav_p; trav_p = &(*trav_p)->next) { req.brick = (*trav_p)->xlator->name; ret = mgmt_submit_request(&req, frame, ctx, &clnt_pmap_prog, GF_PMAP_SIGNIN, mgmt_pmap_signin_cbk, (xdrproc_t)xdr_pmap_signin_req); if (ret < 0) { gf_log(THIS->name, GF_LOG_WARNING, "failed to send sign in request; brick = %s", req.brick); } count++; } } else { ret = mgmt_submit_request(&req, frame, ctx, &clnt_pmap_prog, GF_PMAP_SIGNIN, mgmt_pmap_signin_cbk, (xdrproc_t)xdr_pmap_signin_req); } >>>>>>>>>>>>> Thanks, Mohit Agrawal -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Jan 12 05:52:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 12 Jan 2019 05:52:59 +0000 Subject: [Bugs] [Bug 1665656] testcaes glusterd/add-brick-and-validate-replicated-volume-options.t is crash while brick_mux is enable In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665656 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22015 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Jan 12 05:52:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 12 Jan 2019 05:52:58 +0000 Subject: [Bugs] [Bug 1665656] testcaes glusterd/add-brick-and-validate-replicated-volume-options.t is crash while brick_mux is enable In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665656 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22015 (core: glusterd/add-brick-and-validate-replicated-volume-options.t is crash) posted (#1) for review on master by MOHIT AGRAWAL -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Sat Jan 12 13:55:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sat, 12 Jan 2019 13:55:13 +0000 Subject: [Bugs] [Bug 1665677] volume create and transport change with rdma failed In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665677 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Version|unspecified |mainline Component|rdma |rdma CC| |bugs at gluster.org, | |sheggodu at redhat.com Assignee|rkavunga at redhat.com |bugs at gluster.org QA Contact|rhinduja at redhat.com | Product|Red Hat Gluster Storage |GlusterFS -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Jan 13 09:27:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 13 Jan 2019 09:27:12 +0000 Subject: [Bugs] [Bug 1664590] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Yaniv Kaul changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(sabose at redhat.com | |) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 10 11:11:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 10 Jan 2019 11:11:30 +0000 Subject: [Bugs] [Bug 1665038] glusterd crashed while running "gluster get-state glusterd odir /get-state" In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665038 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22007 (glusterd: fix crash) posted (#5) for review on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 02:28:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 02:28:09 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Amgad changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amgad.saleh at nokia.com --- Comment #8 from Amgad --- Any update on resolution? Is there any fix included in 5.3? or 5.1.?? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 11 01:50:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 11 Jan 2019 01:50:19 +0000 Subject: [Bugs] [Bug 1665332] Wrong offset is used in offset for zerofill fop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665332 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21991 (cluster/afr: fix zerofill transaction.start) posted (#5) for review on master by Pranith Kumar Karampuri -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 06:07:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:07:44 +0000 Subject: [Bugs] [Bug 1665803] New: [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 Bug ID: 1665803 Summary: [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: sharding Keywords: Triaged Severity: high Assignee: bugs at gluster.org Reporter: kdhananj at redhat.com QA Contact: bugs at gluster.org CC: bugs at gluster.org, rhs-bugs at redhat.com, sankarshan at redhat.com, sasundar at redhat.com, storage-qa-internal at redhat.com Depends On: 1662368 Blocks: 1662059 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1662368 +++ +++ This bug was initially created as a clone of Bug #1662059 +++ Description of problem: ----------------------- Attempts were made to try out the customer scenario, where the disks of bigger sizes residing on gluster volumes are deleted from ovirt. During one such attempts, its found that the fuse mount process has crashed. Version-Release number of selected component (if applicable): ------------------------------------------------------------- ovirt 4.0.5 gluster-master How reproducible: ----------------- 1/1 Hit it once. Steps to Reproduce: ------------------- 1. ovirt storage domain is configured to use gluster arbitrated replicate volume, with sharding enabled 2. Create disk of size 1TB from ovirt Manager UI 3. Delete the disk from ovirt Manager UI Actual results: --------------- Gluster fuse mount process crashed on one of the hypervisor Expected results: ----------------- No gluster process should crash --- Additional comment from SATHEESARAN on 2018-12-25 18:03:56 UTC --- 1. Gluster cluster info ------------------------ There are 3 nodes in the gluster cluster 2. Gluster volume info ---------------------- [root at rhsqa-grafton7-nic2 ~]# gluster volume info imstore Volume Name: imstore Type: Replicate Volume ID: 878eb828-0735-4ce8-a2b3-c52a757ee1b2 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: server1:/gluster_bricks/vol1/im1 Brick2: server2:/gluster_bricks/vol1/im1 Brick3: server3:/gluster_bricks/vol1/im1 (arbiter) Options Reconfigured: performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 network.ping-timeout: 30 cluster.granular-entry-heal: on server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet nfs.disable: on performance.client-io-threads: on 4. Test info ------------- File that was deleted: /rhev/data-center/mnt/glusterSD/server1\:_imstore/3d4f163a-c6e0-476e-a726-bd780e0d1b83/images/075c6ffd-318c-4108-8405-ccf8078c1e16/b62a4640-f02a-4aa1-b249-cfc4cb2f7f59 GFID of this file is: 3d231d2b-4fff-4c03-b593-70befaf77296 Before deleting the file: [root at server1 ~]# ls /gluster_bricks/vol1/im1/.shard/ |grep 3d231 | wc -l 16383 While the deleting is in progress: [root at server1 ~]# ls /gluster_bricks/vol1/im1/.shard/ |grep 3d231 | wc -l 3983 After the fuse mount crash, there were some ghost shards, but after 15 mins, there are no ghost shards [root at server1 ~]# ls /gluster_bricks/vol1/im1/.shard/.remove_me/ [root at server1 ~]# ls /gluster_bricks/vol1/im1/.shard/ |grep 3d231 | wc -l 0 --- Additional comment from SATHEESARAN on 2018-12-26 06:27:00 UTC --- (In reply to Krutika Dhananjay from comment #5) > So there is no core dump and I can't tell much from just the logs. > > From > [root at dhcp37-127 ~]# cat /proc/sys/kernel/core_pattern > |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e %i > > Seems like this should be set to a valid path for us to get the core dump. > > Would be great if you can change this value to a meaningful path and > recreate the issue. > > -Krutika I could reproduce the issue consistently outside of ovirt-gluster setup. With 3 gluster servers and 1 client. 1. Create 5 VM image files on the fuse mounted gluster volume using qemu-img command # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm1.img 10G # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm2.img 7G # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm3.img 5G # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm4.img 4G 2. Delete the files from the mount # rm -rf /mnt/testdata/* The above step hits the crash,close to consistent I will reinstall the required debug packages and will provide the setup for debugging --- Additional comment from SATHEESARAN on 2018-12-26 06:33:43 UTC --- Backtrace from the core file Core was generated by `/usr/sbin/glusterfs --volfile-server=10.70.37.152 --volfile-id=/volume1 /mnt/te'. Program terminated with signal 11, Segmentation fault. #0 0x00007fdb3d8eb53e in shard_unlink_block_inode (local=local at entry=0x7fdb2400a400, shard_block_num=) at shard.c:2945 2945 base_ictx->fsync_count--; --- Additional comment from Krutika Dhananjay on 2018-12-26 14:23:28 UTC --- (In reply to SATHEESARAN from comment #6) > (In reply to Krutika Dhananjay from comment #5) > > So there is no core dump and I can't tell much from just the logs. > > > > From > > [root at dhcp37-127 ~]# cat /proc/sys/kernel/core_pattern > > |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e %i > > > > Seems like this should be set to a valid path for us to get the core dump. > > > > Would be great if you can change this value to a meaningful path and > > recreate the issue. > > > > -Krutika > > I could reproduce the issue consistently outside of RHV-RHGS setup. > With 3 RHGS servers and 1 client. > > 1. Create 5 VM image files on the fuse mounted gluster volume using qemu-img > command > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm1.img 10G > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm2.img 7G > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm3.img 5G > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm4.img 4G > > 2. Delete the files from the mount > # rm -rf /mnt/testdata/* > > The above step hits the crash,close to consistent > > I will reinstall the required debug packages and will provide the setup for > debugging Is the mountpoint in step 1 different from the one used in 2? In step 1, files are created under /mnt/test/. But the rm -rf is done from /mnt/testdata/ -Krutika --- Additional comment from SATHEESARAN on 2018-12-26 15:38:21 UTC --- (In reply to Krutika Dhananjay from comment #8) > (In reply to SATHEESARAN from comment #6) > > (In reply to Krutika Dhananjay from comment #5) > > > So there is no core dump and I can't tell much from just the logs. > > > > > > From > > > [root at dhcp37-127 ~]# cat /proc/sys/kernel/core_pattern > > > |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e %i > > > > > > Seems like this should be set to a valid path for us to get the core dump. > > > > > > Would be great if you can change this value to a meaningful path and > > > recreate the issue. > > > > > > -Krutika > > > > I could reproduce the issue consistently outside of RHV-RHGS setup. > > With 3 RHGS servers and 1 client. > > > > 1. Create 5 VM image files on the fuse mounted gluster volume using qemu-img > > command > > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm1.img 10G > > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm2.img 7G > > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm3.img 5G > > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm4.img 4G > > > > 2. Delete the files from the mount > > # rm -rf /mnt/testdata/* > > > > The above step hits the crash,close to consistent > > > > I will reinstall the required debug packages and will provide the setup for > > debugging > > > Is the mountpoint in step 1 different from the one used in 2? In step 1, > files are created under /mnt/test/. But the rm -rf is done from > /mnt/testdata/ > > -Krutika I did it from same mount. No different mounts --- Additional comment from Worker Ant on 2018-12-28 02:10:20 UTC --- REVIEW: https://review.gluster.org/21946 (features/shard: Assign fop id during background deletion to prevent excessive logging) posted (#1) for review on master by Krutika Dhananjay --- Additional comment from Worker Ant on 2018-12-28 15:43:45 UTC --- REVIEW: https://review.gluster.org/21957 (features/shard: Fix launch of multiple synctasks for background deletion) posted (#1) for review on master by Krutika Dhananjay --- Additional comment from Worker Ant on 2019-01-08 12:09:39 UTC --- REVIEW: https://review.gluster.org/21946 (features/shard: Assign fop id during background deletion to prevent excessive logging) posted (#7) for review on master by Xavi Hernandez --- Additional comment from Worker Ant on 2019-01-11 08:36:32 UTC --- REVIEW: https://review.gluster.org/21957 (features/shard: Fix launch of multiple synctasks for background deletion) posted (#7) for review on master by Xavi Hernandez Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1662059 [Bug 1662059] [RHV-RHGS] Fuse mount crashed while deleting a 1 TB image file from RHV https://bugzilla.redhat.com/show_bug.cgi?id=1662368 [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 06:07:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:07:44 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1665803 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 06:08:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:08:03 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |kdhananj at redhat.com -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 06:39:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:39:48 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |POST External Bug ID| |Gluster.org Gerrit 22018 --- Comment #5 from Worker Ant --- REVIEW: https://review.gluster.org/22018 (features/shard: Assign fop id during background deletion to prevent excessive logging) posted (#1) for review on release-5 by Krutika Dhananjay -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 06:42:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:42:08 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 06:42:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:42:50 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 --- Comment #6 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22018 (features/shard: Assign fop id during background deletion to prevent excessive logging) posted (#2) for review on release-5 by Krutika Dhananjay -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 06:42:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:42:51 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID|Gluster.org Gerrit 22018 | -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 06:42:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:42:53 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22018 (features/shard: Assign fop id during background deletion to prevent excessive logging) posted (#2) for review on release-5 by Krutika Dhananjay -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 06:42:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 06:42:54 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22018 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 08:21:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 08:21:09 +0000 Subject: [Bugs] [Bug 1665826] New: [geo-rep]: Directory renames not synced to slave in Hybrid Crawl Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665826 Bug ID: 1665826 Summary: [geo-rep]: Directory renames not synced to slave in Hybrid Crawl Product: GlusterFS Version: mainline Status: NEW Component: geo-replication Keywords: Regression, ZStream Severity: high Assignee: bugs at gluster.org Reporter: sunkumar at redhat.com CC: bugs at gluster.org Depends On: 1664235 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1664235 +++ Description of problem: ======================= Directory renames are not synced to the slave in Hybrid crawl. Version-Release number of selected component (if applicable): ============================================================= mainline How reproducible: ================= Always Steps to Reproduce: =================== 1. Setup geo-replication 2. Set the change_detector to xsync 3. Create directory (dir1) on master and let it sync to slave 4. rename dir1 to dir2 Actual results: =============== Dir1 still exists in the slave instead of Dir2 Expected results: ================= Dir2 should be seen on the slave as well. Additional info: ================ There were no errors seen in geo-rep logs regarding the failure to rename the directory. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1664235 [Bug 1664235] [geo-rep]: Directory renames not synced to slave in Hybrid Crawl -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 08:21:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 08:21:24 +0000 Subject: [Bugs] [Bug 1665826] [geo-rep]: Directory renames not synced to slave in Hybrid Crawl In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665826 Sunny Kumar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |sunkumar at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 08:51:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 08:51:22 +0000 Subject: [Bugs] [Bug 1665826] [geo-rep]: Directory renames not synced to slave in Hybrid Crawl In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665826 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22020 (geo-rep : fix rename sync on hybrid crawl) posted (#1) for review on master by Sunny Kumar -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 08:51:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 08:51:23 +0000 Subject: [Bugs] [Bug 1665826] [geo-rep]: Directory renames not synced to slave in Hybrid Crawl In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665826 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22020 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 10:23:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 10:23:14 +0000 Subject: [Bugs] [Bug 1665361] Alerts for offline nodes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665361 M. Scherer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mscherer at redhat.com --- Comment #1 from M. Scherer --- I suspect option 2 is not what we want. But yeah, nagios do handle this quite well, doing notification, etc, etc. But would still need to do the basic script that do the API call anyway, the difference would be between "send a email", or "do a api call to nagios to trigger a alert", and I think we could switch between thel quite easily if needed. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 10:26:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 10:26:20 +0000 Subject: [Bugs] [Bug 1665055] kernel-writeback-cache option does not seem to be working In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665055 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1581904 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 10:43:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 10:43:42 +0000 Subject: [Bugs] [Bug 1665880] New: After the shard feature is enabled, the glfs_read will always return the length of the read buffer, no the actual length readed Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665880 Bug ID: 1665880 Summary: After the shard feature is enabled, the glfs_read will always return the length of the read buffer, no the actual length readed Product: GlusterFS Version: mainline Status: NEW Component: sharding Assignee: bugs at gluster.org Reporter: xiubli at redhat.com QA Contact: bugs at gluster.org CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: When I was testing the gluster-block/tcmu-runner with the glsuterfs mainline code, there will be a crash in gluster-block: # journalctl -r [...] 16878 Jan 14 17:42:16 rhel3 kernel: gluster-blockd[5209]: segfault at 0 ip 00007fc430c0ff81 sp 00007fc42bffc608 error 4 in libc-2.17.so[7fc430aa1000+1c3000] 16879 Jan 14 17:42:16 rhel3 systemd: gluster-blockd.service: main process exited, code=killed, status=11/SEGV 16880 Jan 14 17:42:16 rhel3 systemd: Unit gluster-blockd.service entered failed state. 16881 Jan 14 17:42:16 rhel3 systemd: gluster-blockd.service failed. [...] This is because when reading and parsing the meta-data file and after EOF the loop won't stop and keep reading and glfs_read returns none zero. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Create one volume and enable the features.shard 2. Try to allocate one buffer large enough and make sure the buffer size is larger than the actual file size. 3. len = glfs_read (gfd, buffer, sizeof(buffer)). Actual results: len will always equal to sizeof(buffer) Expected results: len < sizeof(buffer) Additional info: -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 10:51:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 10:51:02 +0000 Subject: [Bugs] [Bug 1665880] After the shard feature is enabled, the glfs_read will always return the length of the read buffer, no the actual length readed In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665880 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22022 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 10:51:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 10:51:04 +0000 Subject: [Bugs] [Bug 1665880] After the shard feature is enabled, the glfs_read will always return the length of the read buffer, no the actual length readed In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665880 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22022 (feature/shard: fix the return length of shard_readv) posted (#1) for review on master by Xiubo Li -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 11:07:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 11:07:27 +0000 Subject: [Bugs] [Bug 1665889] New: Too small restriction for commit topic length in review.gluster.org Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665889 Bug ID: 1665889 Summary: Too small restriction for commit topic length in review.gluster.org Product: GlusterFS Version: mainline Hardware: All OS: All Status: NEW Component: project-infrastructure Severity: high Assignee: bugs at gluster.org Reporter: vponomar at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of the problem: We, OCS QE automation team, have goal to port our downstream project [1] to the upstream [2]. And we are unable to do it, because new repo [2] has limitation to the length of a commit topic as 50 symbols. In our downstream project we followed 72 symbols length. So, need to make it be 72 symbols. [1] http://git.app.eng.bos.redhat.com/git/cns-qe/cns-automation.git/ [2] https://github.com/gluster/glusterfs-containers-tests Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Create commit 2. Push it to the gerrit -> https://review.gluster.org/#/q/project:glusterfs-containers-tests Actual results: Response from server: remote: (W) efd7f6f: commit subject >50 characters; use shorter first paragraph Expected results: Success after attempt to push code. Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 11:23:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 11:23:08 +0000 Subject: [Bugs] [Bug 1665889] Too small restriction for commit topic length in review.gluster.org In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665889 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |MODIFIED CC| |nigelb at redhat.com Assignee|bugs at gluster.org |nigelb at redhat.com --- Comment #1 from Nigel Babu --- Ack. This needs a gerrit config change and a restart. I'm going to do that now. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 11:42:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 11:42:59 +0000 Subject: [Bugs] [Bug 1665889] Too small restriction for commit topic length in review.gluster.org In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665889 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-14 11:42:59 --- Comment #2 from Nigel Babu --- This still lead to some permission troubles around pushing merge commits that did not go away despite granting merge permissions. I did the push instead and that has worked. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 12:35:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 12:35:25 +0000 Subject: [Bugs] [Bug 1665656] testcaes glusterd/add-brick-and-validate-replicated-volume-options.t is crash while brick_mux is enable In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665656 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-14 12:35:25 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22015 (core: glusterd/add-brick-and-validate-replicated-volume-options.t is crash) merged (#3) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 13:22:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 13:22:51 +0000 Subject: [Bugs] [Bug 1665055] kernel-writeback-cache option does not seem to be working In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665055 --- Comment #2 from Manoj Pillai --- So fuse writeback caching is not yet there in RHEL 7. Tried on a different kernel, 4.16.3-301.fc28.x86_64. There I do see dirty data buildup while the write test is in progress. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 13:33:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 13:33:40 +0000 Subject: [Bugs] [Bug 1535528] Gluster cli show no help message in prompt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1535528 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22036 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 13:33:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 13:33:41 +0000 Subject: [Bugs] [Bug 1535528] Gluster cli show no help message in prompt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1535528 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22036 (cli: Added a welcome and help message) posted (#1) for review on master by Rinku Kothiya -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 13:36:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 13:36:39 +0000 Subject: [Bugs] [Bug 1535528] Gluster cli show no help message in prompt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1535528 Rinku changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rkothiya at redhat.com Assignee|bugs at gluster.org |rkothiya at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 14 14:49:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 14:49:42 +0000 Subject: [Bugs] [Bug 1658146] BZ incorrectly updated with "patch posted" message when a patch is merged In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1658146 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-14 14:49:42 --- Comment #3 from Nigel Babu --- This is now fixed. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 14 17:50:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 14 Jan 2019 17:50:42 +0000 Subject: [Bugs] [Bug 1644389] [GSS] Directory listings on fuse mount are very slow due to small number of getdents() entries In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1644389 Cal Calhoun changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(nbalacha at redhat.c | |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 15 01:40:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:40:37 +0000 Subject: [Bugs] [Bug 1666143] New: Several fixes on socket pollin and pollout return value Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 Bug ID: 1666143 Summary: Several fixes on socket pollin and pollout return value Product: GlusterFS Version: mainline Status: NEW Component: rpc Assignee: bugs at gluster.org Reporter: zhhuan at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Found the return value of socket pollin and pollout for rpc message is not correctly handled. One major problem is socket EAGAIN error will be returned all the way back to dispatch handler and confuse user with error message like: [2018-12-29 07:31:41.772310] E [MSGID: 101191] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 01:55:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:55:52 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22043 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 01:55:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:55:53 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22043 (socket: fix counting of socket total_bytes_read and total_bytes_write) posted (#1) for review on master by Zhang Huan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 01:57:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:57:03 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22044 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 01:57:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:57:04 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22044 (socket: fix issue when socket write return with EAGAIN) posted (#1) for review on master by Zhang Huan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 01:58:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:58:09 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22046 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 01:58:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:58:10 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22046 (socket: don't pass return value from protocol handler to event handler) posted (#1) for review on master by Zhang Huan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 01:59:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:59:18 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22045 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 01:59:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 01:59:19 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22045 (socket: fix issue when socket read return with EAGAIN) posted (#1) for review on master by Zhang Huan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 02:04:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 02:04:12 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22018 (features/shard: Assign fop id during background deletion to prevent excessive logging) merged (#2) on release-5 by Krutika Dhananjay -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 15 04:58:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 04:58:29 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22047 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 15 04:58:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 04:58:31 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22047 (features/shard: Fix launch of multiple synctasks for background deletion) posted (#1) for review on release-5 by Krutika Dhananjay -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 15 05:28:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 05:28:42 +0000 Subject: [Bugs] [Bug 1644389] [GSS] Directory listings on fuse mount are very slow due to small number of getdents() entries In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1644389 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nbalacha at redhat.c | |om) | |needinfo?(nbalacha at redhat.c | |om) | -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 15 10:43:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 10:43:33 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #9 from Guillaume Pavese --- Similar problem on a newly provisioned ovirt 4.3 cluster (centos 7.6, gluster 5.2-1) : [2019-01-15 09:32:02.558598] I [MSGID: 100030] [glusterfsd.c:2691:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.2 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=ps-inf-int-kvm-fr-306-210.hostics.fr --volfile-server=10.199.211.7 --volfile-server=10.199.211.5 --volfile-id=/vmstore /rhev/data-center/mnt/glusterSD/ps-inf-int-kvm-fr-306-210.hostics.fr:_vmstore) [2019-01-15 09:32:02.566701] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-15 09:32:02.581138] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2019-01-15 09:32:02.581272] I [MSGID: 114020] [client.c:2354:notify] 0-vmstore-client-0: parent translators are ready, attempting connect on transport [2019-01-15 09:32:02.583283] I [MSGID: 114020] [client.c:2354:notify] 0-vmstore-client-1: parent translators are ready, attempting connect on transport [2019-01-15 09:32:02.583911] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-vmstore-client-0: changing port to 49155 (from 0) [2019-01-15 09:32:02.585505] I [MSGID: 114020] [client.c:2354:notify] 0-vmstore-client-2: parent translators are ready, attempting connect on transport [2019-01-15 09:32:02.587413] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-15 09:32:02.587441] E [MSGID: 108006] [afr-common.c:5314:__afr_handle_child_down_event] 0-vmstore-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up. [2019-01-15 09:32:02.587951] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-15 09:32:02.588685] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-vmstore-client-0: Connected to vmstore-client-0, attached to remote volume '/gluster_bricks/vmstore/vmstore'. [2019-01-15 09:32:02.588708] I [MSGID: 108005] [afr-common.c:5237:__afr_handle_child_up_event] 0-vmstore-replicate-0: Subvolume 'vmstore-client-0' came back up; going online. Final graph: +------------------------------------------------------------------------------+ 1: volume vmstore-client-0 2: type protocol/client 3: option opversion 50000 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key /vmstore 7: option client-version 5.2 8: option process-name fuse 9: option process-uuid CTX_ID:e5dad97f-5289-4464-9e2f-36e9bb115118-GRAPH_ID:0-PID:39987-HOST:ps-inf-int-kvm-fr-307-210.hostics.fr-PC_NAME:vmstore-client-0-RECON_NO:-0 10: option fops-version 1298437 11: option ping-timeout 30 12: option remote-host 10.199.211.6 13: option remote-subvolume /gluster_bricks/vmstore/vmstore 14: option transport-type socket 15: option transport.address-family inet 16: option filter-O_DIRECT off 17: option transport.tcp-user-timeout 0 18: option transport.socket.keepalive-time 20 19: option transport.socket.keepalive-interval 2 20: option transport.socket.keepalive-count 9 21: option send-gids true 22: end-volume 23: 24: volume vmstore-client-1 25: type protocol/client 26: option ping-timeout 30 27: option remote-host 10.199.211.7 28: option remote-subvolume /gluster_bricks/vmstore/vmstore 29: option transport-type socket 30: option transport.address-family inet 31: option filter-O_DIRECT off 32: option transport.tcp-user-timeout 0 33: option transport.socket.keepalive-time 20 34: option transport.socket.keepalive-interval 2 35: option transport.socket.keepalive-count 9 36: option send-gids true 37: end-volume 38: 39: volume vmstore-client-2 40: type protocol/client 41: option ping-timeout 30 42: option remote-host 10.199.211.5 43: option remote-subvolume /gluster_bricks/vmstore/vmstore 44: option transport-type socket 45: option transport.address-family inet 46: option filter-O_DIRECT off 47: option transport.tcp-user-timeout 0 48: option transport.socket.keepalive-time 20 49: option transport.socket.keepalive-interval 2 50: option transport.socket.keepalive-count 9 51: option send-gids true 52: end-volume 53: 54: volume vmstore-replicate-0 55: type cluster/replicate 56: option afr-pending-xattr vmstore-client-0,vmstore-client-1,vmstore-client-2 57: option arbiter-count 1 58: option data-self-heal-algorithm full 59: option eager-lock enable 60: option quorum-type auto 61: option choose-local off 62: option shd-max-threads 8 63: option shd-wait-qlength 10000 64: option locking-scheme granular 65: option granular-entry-heal enable 66: option use-compound-fops off 67: subvolumes vmstore-client-0 vmstore-client-1 vmstore-client-2 68: end-volume 69: 70: volume vmstore-dht 71: type cluster/distribute 72: option lock-migration off 73: option force-migration off 74: subvolumes vmstore-replicate-0 75: end-volume 76: 77: volume vmstore-shard 78: type features/shard 79: subvolumes vmstore-dht 80: end-volume 81: 82: volume vmstore-write-behind 83: type performance/write-behind 84: option strict-O_DIRECT on 85: subvolumes vmstore-shard 86: end-volume 87: 88: volume vmstore-readdir-ahead 89: type performance/readdir-ahead 90: option parallel-readdir off 91: option rda-request-size 131072 92: option rda-cache-limit 10MB 93: subvolumes vmstore-write-behind 94: end-volume 95: 96: volume vmstore-open-behind 97: type performance/open-behind 98: subvolumes vmstore-readdir-ahead 99: end-volume 100: 101: volume vmstore-md-cache 102: type performance/md-cache 103: subvolumes vmstore-open-behind 104: end-volume 105: 106: volume vmstore 107: type debug/io-stats 108: option log-level INFO 109: option latency-measurement off 110: option count-fop-hits off 111: subvolumes vmstore-md-cache 112: end-volume 113: 114: volume meta-autoload 115: type meta 116: subvolumes vmstore 117: end-volume 118: +------------------------------------------------------------------------------+ [2019-01-15 09:32:02.590376] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-vmstore-client-2: changing port to 49155 (from 0) [2019-01-15 09:32:02.592649] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-15 09:32:02.593512] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-vmstore-client-2: Connected to vmstore-client-2, attached to remote volume '/gluster_bricks/vmstore/vmstore'. [2019-01-15 09:32:02.593528] I [MSGID: 108002] [afr-common.c:5588:afr_notify] 0-vmstore-replicate-0: Client-quorum is met [2019-01-15 09:32:02.594714] I [fuse-bridge.c:4259:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2019-01-15 09:32:02.594746] I [fuse-bridge.c:4870:fuse_graph_sync] 0-fuse: switched to graph 0 [2019-01-15 09:32:06.562678] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-15 09:32:09.435695] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.2/xlator/performance/open-behind.so(+0x3d7c) [0x7f5c279cfd7c] -->/usr/lib64/glusterfs/5.2/xlator/performance/open-behind.so(+0x3bd6) [0x7f5c279cfbd6] -->/lib64/libglusterfs.so.0(dict_ref+0x5d) [0x7f5c340ae20d] ) 0-dict: dict is NULL [Invalid argument] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 7 times between [2019-01-15 09:32:06.562678] and [2019-01-15 09:32:27.578753] [2019-01-15 09:32:29.966249] W [glusterfsd.c:1481:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f5c32f1ddd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55af1f5bad45] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55af1f5babbb] ) 0-: received signum (15), shutting down [2019-01-15 09:32:29.966265] I [fuse-bridge.c:5897:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/ps-inf-int-kvm-fr-306-210.hostics.fr:_vmstore'. [2019-01-15 09:32:29.985157] I [fuse-bridge.c:5134:fuse_thread_proc] 0-fuse: initating unmount of /rhev/data-center/mnt/glusterSD/ps-inf-int-kvm-fr-306-210.hostics.fr:_vmstore [2019-01-15 09:32:29.985434] I [fuse-bridge.c:5902:fini] 0-fuse: Closing fuse connection to '/rhev/data-center/mnt/glusterSD/ps-inf-int-kvm-fr-306-210.hostics.fr:_vmstore'. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 11:53:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 11:53:01 +0000 Subject: [Bugs] [Bug 1657607] Convert nr_files to gf_atomic in posix_private structure In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657607 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21837 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 11:53:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 11:53:02 +0000 Subject: [Bugs] [Bug 1657607] Convert nr_files to gf_atomic in posix_private structure In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657607 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-15 11:53:02 --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/21837 (posix: Convert several posix_private members to gf_atomic) merged (#4) on master by Xavi Hernandez -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 14:06:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 14:06:57 +0000 Subject: [Bugs] [Bug 1666326] New: reopening bug 1405147: Failed to dispatch handler: glusterfs seems to check for "write permission" instead for "file owner" during open() when writing to a file Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666326 Bug ID: 1666326 Summary: reopening bug 1405147: Failed to dispatch handler: glusterfs seems to check for "write permission" instead for "file owner" during open() when writing to a file Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: open-behind Severity: high Assignee: bugs at gluster.org Reporter: omar at dit.upm.es CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: When using trough NFS a fuse-mounted glusterfs volume, an error is found if the owner of a file (that is not root) tries to write data in its own file when it has read-only permissions. The problem does not exist if the user is root. Version-Release number of selected component (if applicable): All versions tested (glusterfs-3.7.{4-16}, glusterfs-3.8.{4-5}), glusterfs-5.2 How reproducible: Always Steps to Reproduce: Our architecture has 3 layers: the glusterfs servers layer, an intermediate server for reexporting it via NFS, and the final nfs client at the lab. 1. Mount with nfs (at the client) a fuse mounted glusterfs volume exported via nfs (at the intermediate server) to be the home of the user. cd to home dir and check that the volume is mounted and the file that we will use for testing does not exist: user at computer:~$ ls -l dest.txt ls: cannot access dest.txt: No such file or directory user at computer:~$ 2a. Simple test with few logs: Create a readonly file and try to write data in it. You can execute the following multi-command (error comes at echo, not before). Rm is there only for repetition purposes: user at computer:~$ rm -f 444.txt; touch 444.txt; chmod 444 444.txt; echo test > 444.txt -bash: 444.txt: Permission denied This command fails but does not render information in the glusterfs client log of the NFS intermediate server. Only in the glusterfs server log: The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 14 times between [2019-01-15 13:48:17.999280] and [2019-01-15 13:49:45.530578] 2b. Test with more logs available (error shows up when executing cp): user at computer:~$ rm -f 444.txt dest.txt; echo test > 444.txt; chmod 444 444.txt; cp -p 444.txt dest.txt; cp: failed to close 'dest.txt': Permission denied Messages at the glusterfs client log of the NFS intermediate server: [2019-01-15 13:50:05.955944] I [io-stats.c:1667:io_stats_dump_fd] 0-distribuido3468: --- fd stats --- [2019-01-15 13:50:05.957628] I [io-stats.c:1670:io_stats_dump_fd] 0-distribuido3468: Filename : /home/user/444.txt [2019-01-15 13:50:05.959198] I [io-stats.c:1684:io_stats_dump_fd] 0-distribuido3468: BytesWritten : 5 bytes [2019-01-15 13:50:05.959217] I [io-stats.c:1698:io_stats_dump_fd] 0-distribuido3468: Write 000004b+ : 1 [2019-01-15 13:50:05.984823] E [MSGID: 114031] [client-rpc-fops_v2.c:281:client4_0_open_cbk] 0-distribuido3468-client-1: remote operation failed. Path: /home/user/dest.txt (25be0793-520e-4a87-9b34-f1b9f633af42) [Permission denied] [2019-01-15 13:50:05.989937] W [fuse-bridge.c:1124:fuse_fd_cbk] 0-glusterfs-fuse: 1481: OPEN() /home/user/dest.txt => -1 (Permission denied) Messages at the glusterfs server: [2019-01-15 13:50:05.931054] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-15 13:50:05.956175] I [io-stats.c:1667:io_stats_dump_fd] 0-distribuido3468-io-stats: --- fd stats --- [2019-01-15 13:50:05.958417] I [io-stats.c:1670:io_stats_dump_fd] 0-distribuido3468-io-stats: Filename : /home/user/444.txt [2019-01-15 13:50:05.961269] I [io-stats.c:1684:io_stats_dump_fd] 0-distribuido3468-io-stats: BytesWritten : 5 bytes [2019-01-15 13:50:05.961372] I [io-stats.c:1698:io_stats_dump_fd] 0-distribuido3468-io-stats: Write 000004b+ : 1 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-01-15 13:50:05.931054] and [2019-01-15 13:50:05.942111] 3. Check result: desired file is empty. user at computer:~$ ls -l 444.txt -r--r--r-- 1 user user 0 dic 15 2016 444.txt user at computer:~$ rm -f 444.txt user at computer:~$ 4. Check that the problem dissapears with root privileges: user at computer:~$ sudo su root at computer:/home/user# rm -f 444.txt; touch 444.txt; chmod 444 444.txt; echo test > 444.txt root at computer:/home/user# ls -l 444.txt -r--r--r-- 1 root root 5 dic 15 2016 444.txt root at computer:/home/user# rm -f 444.txt root at computer:/home/user# Actual results: A user who owns a file that is read-only cannot write into it under the described circumstances. Expected results: A user who owns a file that is read-only should be able to write into it even if it is read-only. Additional info: Special thanks to Bruce Fields and other components of nfs-kernel-server team because initially we thought it was their fault. The same to Soumya Koduri and Miklos Szeredi, that helped to confirm the problem. See full previous thread on nfs-kernel-server list with history about the issue: http://marc.info/?t=144801803400001 (for posterity's sake, permanent link: http://marc.info/?i=862ef8b0-61ee-8b7e-a81c-fcdc5a726e37 at dit.upm.es). This bug is reopened because the team thought that was fixed in glusterfs-5.0. See previous bug report at: https://bugzilla.redhat.com/show_bug.cgi?id=1405147. Presentation (in Spanish, 2015) about the lab config and some tests: http://www.rediris.es/jt/jt2015/ponencias/?id=jt2015-jt-ses_4b_seg_red_camp_2-a17b2c1.pdf -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 15 16:46:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 15 Jan 2019 16:46:35 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 --- Comment #38 from Znamensky Pavel --- (In reply to Amar Tumballi from comment #37) > While a release with this patch merged/tested is another 50days away, we > surely would like to reduce the CPU load you see too. Whenever you get time, > if you can capture CPU info with below tool "perf record -ag > --call-graph=dwarf -o perf.data -p ", and then see > "perf report" to see what actually caused the CPU usage, it will help us to > resolve that too. I'm sorry for the delay. I did `find` on a directory with ~ 200_000 files on v6 with lru-limit, without lru-limit and on v4.1. `ps aux` shows this: v6 with lru-limit = 10_000 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 30968 2.2 0.3 637936 54056 ? Ssl 18:15 0:11 /usr/sbin/glusterfs --read-only --lru-limit=10000 --process-name fuse --volfile-server=srv --volfile-id=/st1 /mnt/st1 v6 without lru-limit USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31193 2.0 2.0 834544 330680 ? Ssl 18:32 0:09 /usr/sbin/glusterfs --read-only --process-name fuse --volfile-server=srv --volfile-id=/st1 /mnt/st1 v4.1 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31416 1.3 2.0 817704 344996 ? Ssl 18:41 0:06 /usr/sbin/glusterfs --read-only --process-name fuse --volfile-server=srv --volfile-id=/st1 /mnt/st1 Also, I've got `perf` reports, but I'm afraid I can't correctly understand it. Unfortunately, these reports have sensitive information, so I can't attach them to this issue. Nevertheless, if it would be helpful, I could send them directly to you. (In reply to Amar Tumballi from comment #37) > Also note, lru-limit=10000 while many files are accessed may not be a good > value. I recommend something like 64k at least. But well, it depends on your > memory needs too. So, if you can give 512MB - 1GB RAM for glusterfs, its > better at least for performance. Thanks for the advice! Also, I suppose I should create a new issue for CPU problem, should I? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 04:05:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 04:05:07 +0000 Subject: [Bugs] [Bug 1656682] brick memory consumed by volume is not getting released even after delete In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1656682 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 21825 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 04:05:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 04:05:08 +0000 Subject: [Bugs] [Bug 1656682] brick memory consumed by volume is not getting released even after delete In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1656682 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21825 (core: Resolve memory leak for brick) posted (#4) for review on master by MOHIT AGRAWAL -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 07:20:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 07:20:42 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22051 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 07:20:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 07:20:43 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |POST --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22051 (cluster/afr: Change data-self-heal option to bool) posted (#1) for review on master by None -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 08:07:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 08:07:57 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22052 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 08:07:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 08:07:58 +0000 Subject: [Bugs] [Bug 1663102] Change default value for client side heal to off for replicate volumes In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663102 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22052 (cluster/afr: Change data-self-heal option to bool) posted (#1) for review on master by None -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 09:04:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 09:04:26 +0000 Subject: [Bugs] [Bug 1666634] New: nfs client cannot compile files on dispersed volume Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666634 Bug ID: 1666634 Summary: nfs client cannot compile files on dispersed volume Product: GlusterFS Version: mainline Hardware: x86_64 OS: Linux Status: NEW Component: protocol Severity: high Assignee: bugs at gluster.org Reporter: hxj_lucky at 163.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.create a dispersed volume on glusterfs 2.client machine(centos 6.9) mount the volume by nfs protocol 3.enter the mountpoint directory and compile filebench(or any other c program) Actual results: report cannot compile Expected results: compile the software successfully Additional info: glusterfs version?3.9.0 only nfs + (dispersed volume) got this problem, nfs + (distributed volume or replicated volume) works well, cifs + (dispersed volume) works well; mount the dispersed volume by fuse works well too; -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 15:00:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 15:00:00 +0000 Subject: [Bugs] [Bug 1628620] GlusterFS 5.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1628620 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22053 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 15:00:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 15:00:02 +0000 Subject: [Bugs] [Bug 1628620] GlusterFS 5.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1628620 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|CURRENTRELEASE |--- Keywords| |Reopened --- Comment #16 from Worker Ant --- REVIEW: https://review.gluster.org/22053 (core: move logs which are only developer relevant to DEBUG level) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 15:59:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 15:59:30 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22047 (features/shard: Fix launch of multiple synctasks for background deletion) merged (#1) on release-5 by Krutika Dhananjay -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 16:00:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 16:00:08 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22014 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 16:00:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 16:00:10 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-16 16:00:10 --- Comment #39 from Worker Ant --- REVIEW: https://review.gluster.org/22014 (fuse: add --lru-limit option) merged (#3) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 16:00:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 16:00:33 +0000 Subject: [Bugs] [Bug 1655532] Tracker bug for all leases related issues In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655532 --- Comment #14 from Worker Ant --- REVIEW: https://review.gluster.org/22006 (leases: Reset lease_ctx->timer post deletion) merged (#1) on release-4.1 by soumya k -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 16:03:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 16:03:37 +0000 Subject: [Bugs] [Bug 1659563] gluster-blockd segfaults because of a null-dereference in shard.so In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659563 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 17:04:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 17:04:53 +0000 Subject: [Bugs] [Bug 1666833] New: move few recurring logs to DEBUG level. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666833 Bug ID: 1666833 Summary: move few recurring logs to DEBUG level. Product: GlusterFS Version: mainline Status: NEW Component: logging Assignee: atumball at redhat.com Reporter: atumball at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Reduce the log chatter due to some repeated logs in master branch. Version-Release number of selected component (if applicable): master How reproducible: 100% Additional Info: These were already fixed in release branches, https://review.gluster.org/#/c/glusterfs/+/21173/ but kept as is, so we can fix each of these. But considering these logs are causing extra headache in understanding the proper issue, while debugging, it makes sense to move these to DEBUG log-level. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 17:09:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 17:09:06 +0000 Subject: [Bugs] [Bug 1628620] GlusterFS 5.0 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1628620 --- Comment #17 from Worker Ant --- REVISION POSTED: https://review.gluster.org/22053 (core: move logs which are only developer relevant to DEBUG level) posted (#2) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 17:09:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 17:09:09 +0000 Subject: [Bugs] [Bug 1666833] move few recurring logs to DEBUG level. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666833 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22053 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 17:09:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 17:09:10 +0000 Subject: [Bugs] [Bug 1666833] move few recurring logs to DEBUG level. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666833 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22053 (core: move logs which are only developer relevant to DEBUG level) posted (#2) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 17:32:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 17:32:44 +0000 Subject: [Bugs] [Bug 1656682] brick memory consumed by volume is not getting released even after delete In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1656682 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-16 17:32:44 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/21825 (core: Resolve memory leak for brick) merged (#7) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 17:45:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 17:45:29 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |high CC| |atumball at redhat.com Severity|unspecified |high -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 20:20:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 20:20:36 +0000 Subject: [Bugs] [Bug 1655527] Incorrect usage of local->fd in afr_open_ftruncate_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655527 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED CC| |srangana at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 20:23:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 20:23:45 +0000 Subject: [Bugs] [Bug 1649843] GlusterFS 4.1.7 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1649843 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22055 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 20:23:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 20:23:46 +0000 Subject: [Bugs] [Bug 1649843] GlusterFS 4.1.7 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1649843 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22055 (doc: Release notes for 4.1.7) posted (#1) for review on release-4.1 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 16 20:28:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 20:28:51 +0000 Subject: [Bugs] [Bug 1659085] GlusterFS 5.3 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659085 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22056 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 20:28:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 20:28:52 +0000 Subject: [Bugs] [Bug 1659085] GlusterFS 5.3 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659085 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22056 (doc: Release notes for 5.3) posted (#1) for review on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 20:42:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 20:42:57 +0000 Subject: [Bugs] [Bug 1659085] GlusterFS 5.3 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659085 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-16 20:42:57 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22056 (doc: Release notes for 5.3) merged (#1) on release-5 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 16 20:46:21 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 16 Jan 2019 20:46:21 +0000 Subject: [Bugs] [Bug 1649843] GlusterFS 4.1.7 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1649843 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-16 20:46:21 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22055 (doc: Release notes for 4.1.7) merged (#1) on release-4.1 by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 17 05:16:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 05:16:10 +0000 Subject: [Bugs] [Bug 1666954] New: gluster_glusto-patch-check job is failing with permission denied error on run tests Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666954 Bug ID: 1666954 Summary: gluster_glusto-patch-check job is failing with permission denied error on run tests Product: GlusterFS Version: mainline Status: NEW Component: project-infrastructure Severity: high Assignee: bugs at gluster.org Reporter: vavuthu at redhat.com CC: bugs at gluster.org, gluster-infra at gluster.org Target Milestone: --- Classification: Community Description of problem: gluster_glusto-patch-check ( https://ci.centos.org/job/gluster_glusto-patch-check ) job is failing with permission denied error on run tests https://ci.centos.org/job/gluster_glusto-patch-check/1070/console 05:10:00 TASK [Create an ssh keypair] *************************************************** 05:10:01 fatal: [localhost]: FAILED! => {"changed": true, "cmd": "ssh-keygen -b 2048 -t rsa -f $GLUSTO_WORKSPACE/glusto -q -N \"\"", "delta": "0:00:00.178959", "end": "2019-01-17 05:10:01.386921", "msg": "non-zero return code", "rc": 1, "start": "2019-01-17 05:10:01.207962", "stderr": "Saving key \"/home/gluster/workspace/gluster_glusto-patch-check/centosci/glusto\" failed: Permission denied", "stderr_lines": ["Saving key \"/home/gluster/workspace/gluster_glusto-patch-check/centosci/glusto\" failed: Permission denied"], "stdout": "", "stdout_lines": []} 05:10:01 to retry, use: --limit @/home/gluster/workspace/gluster_glusto-patch-check/centosci/jobs/scripts/glusto/setup-glusto.retry 05:10:01 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 17 05:37:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 05:37:31 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 17 07:21:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 07:21:14 +0000 Subject: [Bugs] [Bug 1666954] gluster_glusto-patch-check job is failing with permission denied error on run tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666954 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED CC| |nigelb at redhat.com Assignee|bugs at gluster.org |nigelb at redhat.com --- Comment #1 from Nigel Babu --- Ack. This is strange, because the user absolutely has permissions. Re-running the exact same ansible script after the job works, so I'm a bit lost as to what's failing. Will dig deeper. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 17 08:30:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 08:30:17 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 --- Comment #5 from Worker Ant --- REVIEW: https://review.gluster.org/22043 (socket: fix counting of socket total_bytes_read and total_bytes_write) merged (#2) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 17 08:30:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 08:30:40 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 --- Comment #6 from Worker Ant --- REVIEW: https://review.gluster.org/22044 (socket: fix issue when socket write return with EAGAIN) merged (#2) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 17 10:14:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 10:14:32 +0000 Subject: [Bugs] [Bug 1665826] [geo-rep]: Directory renames not synced to slave in Hybrid Crawl In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665826 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-17 10:14:32 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22020 (geo-rep : fix rename sync on hybrid crawl) merged (#4) on master by Kotresh HR -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 17 12:32:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 12:32:45 +0000 Subject: [Bugs] [Bug 1644389] [GSS] Directory listings on fuse mount are very slow due to small number of getdents() entries In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1644389 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(ccalhoun at redhat.c | |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 17 13:06:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 13:06:52 +0000 Subject: [Bugs] [Bug 1667099] New: GlusterFS 4.1.8 tracker Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667099 Bug ID: 1667099 Summary: GlusterFS 4.1.8 tracker Product: GlusterFS Version: 4.1 Status: NEW Component: core Keywords: Tracking, Triaged Assignee: bugs at gluster.org Reporter: srangana at redhat.com CC: bugs at gluster.org Target Milestone: --- Deadline: 2019-03-21 Classification: Community Tracker bug for 4.1.8 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 17 13:09:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 13:09:34 +0000 Subject: [Bugs] [Bug 1667103] New: GlusterFS 5.4 tracker Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Bug ID: 1667103 Summary: GlusterFS 5.4 tracker Product: GlusterFS Version: 5 Status: NEW Component: core Keywords: Tracking, Triaged Assignee: bugs at gluster.org Reporter: srangana at redhat.com CC: bugs at gluster.org Target Milestone: --- Deadline: 2019-03-12 Classification: Community Tracker for the release 5.4 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 17 15:54:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 15:54:13 +0000 Subject: [Bugs] [Bug 1667168] New: Thin Arbiter documentation refers commands don't exist "glustercli' Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667168 Bug ID: 1667168 Summary: Thin Arbiter documentation refers commands don't exist "glustercli' Product: GlusterFS Version: mainline Hardware: All OS: Linux Status: NEW Component: arbiter Severity: high Assignee: bugs at gluster.org Reporter: amgad.saleh at nokia.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Tried to install Thin arbiter following the documentation user: https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/ Command "glustercli" doesn't exist! glustercli volume create testvol --replica 2 server{1..2}:/bricks/brick-{1..2} --thin-arbiter server-3:/bricks/brick_ta --force volume create: testvol: success: please start the volume to access data Version-Release number of selected component (if applicable): 4.1.4 and 5.* How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 17 16:28:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 17 Jan 2019 16:28:14 +0000 Subject: [Bugs] [Bug 1644389] [GSS] Directory listings on fuse mount are very slow due to small number of getdents() entries In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1644389 Cal Calhoun changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ccalhoun at redhat.c |needinfo?(nbalacha at redhat.c |om) |om) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 03:35:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 03:35:36 +0000 Subject: [Bugs] [Bug 1656682] brick memory consumed by volume is not getting released even after delete In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1656682 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22058 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 03:35:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 03:35:37 +0000 Subject: [Bugs] [Bug 1656682] brick memory consumed by volume is not getting released even after delete In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1656682 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22058 (quotad: fix wrong memory free) posted (#1) for review on master by Kinglong Mee -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 04:14:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 04:14:34 +0000 Subject: [Bugs] [Bug 1659868] glusterd : features.selinux was missing in glusterd-volume-set file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659868 Jiffin changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-18 04:14:34 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 05:28:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 05:28:52 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #10 from Amgad --- Per 5.2 release note: NOTE: Next minor release tentative date: Week of 10th January, 2019 This issue is urgent and impacting customer deployment. Any projection on 5.3 availability and whether a fix will be available. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 18 11:13:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 11:13:36 +0000 Subject: [Bugs] [Bug 1664590] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |DUPLICATE Last Closed| |2019-01-18 11:13:36 --- Comment #2 from Milind Changire --- *** This bug has been marked as a duplicate of bug 1618669 *** -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 11:46:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 11:46:37 +0000 Subject: [Bugs] [Bug 1666954] gluster_glusto-patch-check job is failing with permission denied error on run tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666954 Nigel Babu changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-18 11:46:37 --- Comment #2 from Nigel Babu --- There seems to an issue with running ssh-keygen via the Jenkins connection. I haven't figured out a solution to that. Instead, I've just generated a key manually in .ssh and we'll be using that for all our jobs. After fixing this bug, I ran into an issue with the python-docx installation failure which is fixed as well. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 11:48:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 11:48:51 +0000 Subject: [Bugs] [Bug 1667407] New: [Ganesha] Observed ganesha crash after setting 'ganesha.enable' to 'on' on volume which is not started Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667407 Bug ID: 1667407 Summary: [Ganesha] Observed ganesha crash after setting 'ganesha.enable' to 'on' on volume which is not started Product: GlusterFS Version: mainline Status: ASSIGNED Component: core Keywords: Triaged Severity: medium Assignee: jthottan at redhat.com Reporter: jthottan at redhat.com CC: bugs at gluster.org Depends On: 1658050 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1658050 +++ Description of problem: ----------------------- Ganesha crashed after setting 'ganesha.enable' to 'on' on volume which is not started. Crash observed on all nodes in the cluster. ============================================================================= Version-Release number of selected component (if applicable): ------------------------------------------------------------- # rpm -qa | grep ganesha glusterfs-ganesha-3.12.2-29.el7rhgs.x86_64 nfs-ganesha-2.5.5-10.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-10.el7rhgs.x86_64 ============================================================================== How reproducible: ----------------- 2/2 ============================================================================= Steps to Reproduce: ------------------ 1. Create a 6 node ganesha cluster. 2. Create a volume 'testvol'. Do not start the volume. 3. Set volume option 'ganesha.enable' to 'on' in 'testvol'. 4. Observe ganesha crash after sometime. ============================================================================ Actual results: --------------- nfs-ganesha crashed on all nodes. ============================================================================= Expected results: ----------------- nfs-ganesha should not get crashed. ============================================================================= Additional info: The initialization of glusterfs client happens twice for nfs-ganesha. One via mgmt_rpc_notify() (the normal path for gfapi) and other with mgmt_cbk_spec() (callback send from glusterd at the end of volume set command) So two io threads will be created. If the volume is not started, the glfs_fini is destroy only one of the threads, leaving the context of another thread invalid and leads to crash. If the volume is in started state, post init init_export_root->mdcache_lookup_path->lookup->..->priv_glfs_active_subvol() finds out there is oldsubvol and sends notify on oldsubvol with PARENT_DOWN event so that the iot thread created first will be destroyed. If the volume is not started the init will fail, so no lookup path will be send post t Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1658050 [Bug 1658050] [Ganesha] Observed ganesha crash after setting 'ganesha.enable' to 'on' on volume which is not started -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 11:59:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 11:59:35 +0000 Subject: [Bugs] [Bug 1667407] [Ganesha] Observed ganesha crash after setting 'ganesha.enable' to 'on' on volume which is not started In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667407 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22062 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 11:59:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 11:59:36 +0000 Subject: [Bugs] [Bug 1667407] [Ganesha] Observed ganesha crash after setting 'ganesha.enable' to 'on' on volume which is not started In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667407 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22062 (graph: deactivate existing graph in glusterfs_graph_activate()) posted (#2) for review on master by jiffin tony Thottan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 17:13:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 17:13:58 +0000 Subject: [Bugs] [Bug 1138841] allow the use of the CIDR format with auth.allow In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1138841 --- Comment #7 from Worker Ant --- REVIEW: https://review.gluster.org/21970 (core: Feature added to accept CidrIp in auth.allow) merged (#10) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 18 23:01:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 18 Jan 2019 23:01:37 +0000 Subject: [Bugs] [Bug 1596746] High Memory Utilization Glusterfs In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1596746 dscott at rts-llc.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |NOTABUG Last Closed|2018-06-29 16:25:50 |2019-01-18 23:01:37 --- Comment #10 from dscott at rts-llc.com --- I updated to latest gluster and no longer having memory high memory issues. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Jan 20 08:39:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 20 Jan 2019 08:39:44 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Emerson Gomes changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emerson.gomes at gmail.com --- Comment #11 from Emerson Gomes --- Still happening in 5.3. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Jan 20 19:27:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 20 Jan 2019 19:27:09 +0000 Subject: [Bugs] [Bug 1362129] rename of a file can cause data loss in an replica/arbiter volume configuration In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1362129 Srijita Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |srmukher at redhat.com Flags| |needinfo?(ravishankar at redha | |t.com) --- Doc Text *updated* --- In replica 3 volume, renaming a file while the brick with the 'good copy' of the file is down would result in removal of the file after self-heal leading to data loss. With this release, lookup of a file will fail, if there is no good copy (as determined by xattrs) found, dismissing the rename option and limiting the data loss. --- Comment #24 from Srijita Mukherjee --- The doc text has been updated. Kindly review the technical accuracy. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 01:52:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 01:52:54 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #12 from Amgad --- (In reply to waza123 from comment #6) > I downgraded to 3.12.15 because 5.1 is not stable at all (clear install) > > Documentation for downgrade for someone who need this: > > backup your cluster data somewhere.. > > remove all instalation files > > gluster volume stop hadoop_volume > gluster volume delete hadoop_volume > killall glusterfs glusterfsd glusterd glustereventsd python > > # remove all files from bricks: > > rm -rf /hadoop/* && rm -rf /hadoop/.glusterfs > > # remove all configs > rm -rf /usr/var/lib/glusterd && rm -rf /usr/var/log/glusterfs && rm -rf > /usr/var/run/gluster && rm -rf /usr/etc/glusterfs > > # install new gluster, mount, copy all files to new cluster from backup. 3.12.13 has a memory leak in "readdir-ahead.C". I saw it fixed in 5.3, is it fixed in 3.12.15? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 01:54:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 01:54:37 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #13 from Amgad --- (In reply to Emerson Gomes from comment #11) > Still happening in 5.3. Is anybody looking at it in 5.3? This is a release waited for!!! -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 03:11:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 03:11:50 +0000 Subject: [Bugs] [Bug 1656682] brick memory consumed by volume is not getting released even after delete In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1656682 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed|2019-01-16 17:32:44 |2019-01-21 03:11:50 --- Comment #5 from Worker Ant --- REVIEW: https://review.gluster.org/22058 (quotad: fix wrong memory free) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 03:23:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 03:23:55 +0000 Subject: [Bugs] [Bug 1535528] Gluster cli show no help message in prompt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1535528 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-21 03:23:55 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22036 (cli: Added a welcome and help message) merged (#7) on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 03:29:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 03:29:27 +0000 Subject: [Bugs] [Bug 1667779] New: glusterd leaks about 1GB memory per day on single machine of storage pool Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667779 Bug ID: 1667779 Summary: glusterd leaks about 1GB memory per day on single machine of storage pool Product: GlusterFS Version: mainline Status: NEW Component: glusterd Keywords: ZStream Severity: high Assignee: bugs at gluster.org Reporter: moagrawa at redhat.com CC: bugs at gluster.org Depends On: 1667169 Target Milestone: --- Classification: Community Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667169 [Bug 1667169] glusterd leaks about 1GB memory per day on single machine of storage pool -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 03:29:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 03:29:42 +0000 Subject: [Bugs] [Bug 1667779] glusterd leaks about 1GB memory per day on single machine of storage pool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667779 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |moagrawa at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 03:56:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 03:56:02 +0000 Subject: [Bugs] [Bug 1667779] glusterd leaks about 1GB memory per day on single machine of storage pool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667779 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 04:13:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 04:13:39 +0000 Subject: [Bugs] [Bug 1362129] rename of a file can cause data loss in an replica/arbiter volume configuration In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1362129 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ravishankar at redha | |t.com) | --- Doc Text *updated* --- In replica 3 volume, renaming a file while the brick with the 'good copy' of the file is down would result in removal of the file during self-heal, leading to data loss. With this release, lookup of a file will fail if there is no good copy (as determined by AFR xattrs) found, dismissing the rename option and limiting the data loss. --- Comment #25 from Ravishankar N --- Made a minor change. Looks good to me otherwise. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 04:25:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 04:25:02 +0000 Subject: [Bugs] [Bug 1655050] automatic split resolution with size as policy should not work on a directory which is in metadata splitbrain In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655050 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-21 04:25:02 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21767 (afr: Splitbrain with size as policy must not resolve for directory) merged (#7) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 04:28:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 04:28:08 +0000 Subject: [Bugs] [Bug 789278] Issues reported by Coverity static analysis tool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=789278 --- Comment #1596 from Worker Ant --- REVIEW: https://review.gluster.org/21995 (posix: fix coverity issue) merged (#3) on master by Amar Tumballi -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 06:49:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 06:49:56 +0000 Subject: [Bugs] [Bug 1667804] New: Unable to delete directories that contain linkto files that point to itself. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Bug ID: 1667804 Summary: Unable to delete directories that contain linkto files that point to itself. Product: GlusterFS Version: 4.1 Status: NEW Component: distribute Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: A directory containing linkto files that point to itself cannot be deleted. Version-Release number of selected component (if applicable): How reproducible: Consistently Steps to Reproduce: 1. gluster v create tvol 192.168.122.7:/bricks/brick2/tvol-{1..2} 2. gluster v start tvol 3. mount -t glusterfs -s 192.168.122.7:/tvol /mnt/g1 4. cd /mnt/g1 5. mkdir -p dir0/dir1/dir2 6. cd dir0/dir1/dir2 7. for i in {1..100}; do echo "Test file" > tfile-$i; done 8. for i in {1..100}; do mv tfile-$i ntfile-$i; done 9. gluster v remove-brick tvol 192.168.122.7:/bricks/brick2/tvol-2 start Once the remove-brick status shows "completed", 10. gluster v remove-brick tvol 192.168.122.7:/bricks/brick2/tvol-2 stop You should now have only linkto files in 192.168.122.7:/bricks/brick2/tvol-2/dir0/dir1/dir2 and they should all be pointing to tvol-client-0. Manually change the linkto xattr value for every file in brick2 to point to itself, in this case "tvol-client-1"(make sure the string is null terminated). 11. setfattr -n trusted.glusterfs.dht.linkto -v 0x74766f6c2d636c69656e742d3100 /bricks/brick2/tvol-2/dir0/dir1/dir2/ntfile-* 12. Try to delete the directory from the mount point: [root at myserver g1]# rm -rf * Actual results: [root at myserver g1]# rm -rf * rm: cannot remove ?dir0/dir1/dir2?: Directory not empty Expected results: The directory should be deleted as there are no data files inside. Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 06:56:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 06:56:22 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #14 from Emerson Gomes --- (In reply to Amgad from comment #13) > (In reply to Emerson Gomes from comment #11) > > Still happening in 5.3. > > Is anybody looking at it in 5.3? This is a release waited for!!! Yes, I have updated to 5.3 yesterday, and issue is still there. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 06:56:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 06:56:40 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |nbalacha at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 06:56:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 06:56:49 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 06:57:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 06:57:03 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1667556 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 09:50:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 09:50:08 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22066 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 09:50:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 09:50:09 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22066 (cluster/dht: Delete invalid linkto files in rmdir) posted (#1) for review on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 11:07:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 11:07:42 +0000 Subject: [Bugs] [Bug 1243991] "gluster volume set group " is not in the help text In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1243991 Rinku changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rkothiya at redhat.com Assignee|bugs at gluster.org |rkothiya at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 11:11:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 11:11:33 +0000 Subject: [Bugs] [Bug 1667779] glusterd leaks about 1GB memory per day on single machine of storage pool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667779 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22064 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 11:11:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 11:11:34 +0000 Subject: [Bugs] [Bug 1667779] glusterd leaks about 1GB memory per day on single machine of storage pool In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667779 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-21 11:11:34 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22064 (glusterd: Resolve memory leak in get-state command) merged (#6) on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 11:20:16 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 11:20:16 +0000 Subject: [Bugs] [Bug 1243991] "gluster volume set group " is not in the help text In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1243991 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22067 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 11:20:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 11:20:17 +0000 Subject: [Bugs] [Bug 1243991] "gluster volume set group " is not in the help text In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1243991 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22067 (cli: Added the group option for volume set) posted (#2) for review on master by Rinku Kothiya -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 21 11:56:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 11:56:06 +0000 Subject: [Bugs] [Bug 1667905] New: dict_leak in __glusterd_handle_cli_uuid_get function Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667905 Bug ID: 1667905 Summary: dict_leak in __glusterd_handle_cli_uuid_get function Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: moagrawa at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: dict_leak in __glusterd_handle_cli_uuid_get function Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 11:56:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 11:56:28 +0000 Subject: [Bugs] [Bug 1667905] dict_leak in __glusterd_handle_cli_uuid_get function In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667905 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |medium Assignee|bugs at gluster.org |moagrawa at redhat.com Severity|unspecified |medium -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 21 11:58:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 11:58:33 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 Vivek Das changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ON_QA |VERIFIED -- You are receiving this mail because: You are on the CC list for the bug. From shaik.salam at tcs.com Mon Jan 21 12:06:07 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Mon, 21 Jan 2019 17:36:07 +0530 Subject: [Bugs] Unable to create new volume due to pending operations Message-ID: Hi, We have deployed glustrerfs as containers on openshift orgin. We are unable to create new volume for opeshift pods observed following error. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. We could observe above issue due to pending operations on heketi db. We have exported db and removed pending operations (volumes,bricks) on db, lvs from physical hosts and imported. But heketi trying to delete volumes which are removed from heketi as part of pending operations in db and still we are unable to create volumes facing server busy. Can you please let me know still heketi where it is getting volumes id which are not available on heketi deb?? why we are unable to create volumes ?? (no info on glusterd,glusterfsd logs)?? [negroni] Started POST /volumes [heketi] WARNING 2019/01/21 11:48:31 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 260.577?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/21 11:48:39 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 221.477?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 151.896?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 125.387?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 168.23?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 123.231?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 160.416?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 124.439?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 126.748?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.377?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 138.477?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 267.79?s BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaik.salam at tcs.com Mon Jan 21 13:06:02 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Mon, 21 Jan 2019 18:36:02 +0530 Subject: [Bugs] Bricks are going offline unable to recover with heal/start force commands Message-ID: Hi, Bricks are in offline when we try to recover with following commands gluster volume heal gluster volume start force But still bricks are offline. sh-4.2# gluster volume status vol_3442e86b6d994a14de73f1b8c82cf0b8 Status of volume: vol_3442e86b6d994a14de73f1b8c82cf0b8 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 192.168.3.6:/var/lib/heketi/mounts/vg _ca57f326195c243be2380ce4e42a4191/brick_952 d75fd193c7209c9a81acbc23a3747/brick 49166 0 Y 269 Brick 192.168.3.5:/var/lib/heketi/mounts/vg _d5f17487744584e3652d3ca943b0b91b/brick_e15 c12cceae12c8ab7782dd57cf5b6c1/brick N/A N/A N N/A Brick 192.168.3.15:/var/lib/heketi/mounts/v g_462ea199185376b03e4b0317363bb88c/brick_17 36459d19e8aaa1dcb5a87f48747d04/brick 49173 0 Y 225 Self-heal Daemon on localhost N/A N/A Y 45826 Self-heal Daemon on 192.168.3.6 N/A N/A Y 65196 Self-heal Daemon on 192.168.3.15 N/A N/A Y 52915 Task Status of Volume vol_3442e86b6d994a14de73f1b8c82cf0b8 ------------------------------------------------------------------------------ we can see following events from when we start forcing volumes /mgmt/glusterd.so(+0xe2b3a) [0x7fca9e139b3a] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2605) [0x7fca9e139605] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fcaa346f0e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=vol_3442e86b6d994a14de73f1b8c82cf0b8 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd [2019-01-21 08:22:34.555068] E [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2b3a) [0x7fca9e139b3a] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2563) [0x7fca9e139563] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fcaa346f0e5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_3442e86b6d994a14de73f1b8c82cf0b8 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd [2019-01-21 08:22:53.389049] I [MSGID: 106499] [glusterd-handler.c:4314:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vol_3442e86b6d994a14de73f1b8c82cf0b8 [2019-01-21 08:23:25.346839] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req we can see following events from when we heal volumes. [2019-01-21 08:20:07.576070] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:20:07.580225] I [cli-rpc-ops.c:9182:gf_cli_heal_volume_cbk] 0-cli: Received resp to heal volume [2019-01-21 08:20:07.580326] I [input.c:31:cli_batch] 0-: Exiting with: -1 [2019-01-21 08:22:30.423311] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:22:30.463648] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:22:30.463718] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:30.463859] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:22:33.427710] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:34.581555] I [cli-rpc-ops.c:1472:gf_cli_start_volume_cbk] 0-cli: Received resp to start volume [2019-01-21 08:22:34.581678] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2019-01-21 08:22:53.345351] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:22:53.387992] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:22:53.388059] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:53.388138] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:22:53.394737] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2019-01-21 08:23:25.304688] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:23:25.346319] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:23:25.346389] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:23:25.346500] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 Please let us know steps to recover bricks. BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaik.salam at tcs.com Mon Jan 21 13:47:22 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Mon, 21 Jan 2019 19:17:22 +0530 Subject: [Bugs] Unable to create new volume due to pending operations In-Reply-To: References: Message-ID: Hi, Can you please reply on my issues. I think its already known issue I feel. Can you please let me know still heketi where it is getting volumes id which are not available on heketi db?? why we are unable to create volumes ?? (no info on glusterd,glusterfsd logs)?? BR Salam From: "Shaik Salam" To: Date: 01/21/2019 05:47 PM Subject: [Bugs] Unable to create new volume due to pending operations Sent by: bugs-bounces at gluster.org "External email. Open with Caution" Hi, We have deployed glustrerfs as containers on openshift orgin. We are unable to create new volume for opeshift pods observed following error. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. We could observe above issue due to pending operations on heketi db. We have exported db and removed pending operations (volumes,bricks) on db, lvs from physical hosts and imported. But heketi trying to delete volumes which are removed from heketi as part of pending operations in db and still we are unable to create volumes facing server busy. Can you please let me know still heketi where it is getting volumes id which are not available on heketi deb?? why we are unable to create volumes ?? (no info on glusterd,glusterfsd logs)?? [negroni] Started POST /volumes [heketi] WARNING 2019/01/21 11:48:31 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 260.577?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/21 11:48:39 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 221.477?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 151.896?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 125.387?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 168.23?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 123.231?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 160.416?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 124.439?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 126.748?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.377?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 138.477?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 267.79?s BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you _______________________________________________ Bugs mailing list Bugs at gluster.org https://lists.gluster.org/mailman/listinfo/bugs -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaik.salam at tcs.com Mon Jan 21 16:27:25 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Mon, 21 Jan 2019 21:57:25 +0530 Subject: [Bugs] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Message-ID: Hi, We are facing also similar issue on openshift origin while we are creating pvc for pods. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. heketi looks fine. [negroni] Completed 200 OK in 116.41?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 124.552?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 128.632?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 134.856?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 123.378?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 134.202?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 120.114?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 141.04?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 122.628?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 150.651?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 116.978?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 110.189?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 226.655?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 129.487?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 116.809?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 118.697?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 112.947?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 134.569?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 119.018?s BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From shaik.salam at tcs.com Mon Jan 21 16:33:24 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Mon, 21 Jan 2019 22:03:24 +0530 Subject: [Bugs] Bricks are going offline unable to recover with heal/start force commands Message-ID: Hi, Bricks are in offline and unable to recover with following commands gluster volume heal gluster volume start force But still bricks are offline. sh-4.2# gluster volume status vol_3442e86b6d994a14de73f1b8c82cf0b8 Status of volume: vol_3442e86b6d994a14de73f1b8c82cf0b8 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 192.168.3.6:/var/lib/heketi/mounts/vg _ca57f326195c243be2380ce4e42a4191/brick_952 d75fd193c7209c9a81acbc23a3747/brick 49166 0 Y 269 Brick 192.168.3.5:/var/lib/heketi/mounts/vg _d5f17487744584e3652d3ca943b0b91b/brick_e15 c12cceae12c8ab7782dd57cf5b6c1/brick N/A N/A N N/A Brick 192.168.3.15:/var/lib/heketi/mounts/v g_462ea199185376b03e4b0317363bb88c/brick_17 36459d19e8aaa1dcb5a87f48747d04/brick 49173 0 Y 225 Self-heal Daemon on localhost N/A N/A Y 45826 Self-heal Daemon on 192.168.3.6 N/A N/A Y 65196 Self-heal Daemon on 192.168.3.15 N/A N/A Y 52915 Task Status of Volume vol_3442e86b6d994a14de73f1b8c82cf0b8 ------------------------------------------------------------------------------ We can see following events from when we start forcing volumes /mgmt/glusterd.so(+0xe2b3a) [0x7fca9e139b3a] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2605) [0x7fca9e139605] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fcaa346f0e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=vol_3442e86b6d994a14de73f1b8c82cf0b8 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd [2019-01-21 08:22:34.555068] E [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2b3a) [0x7fca9e139b3a] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2563) [0x7fca9e139563] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fcaa346f0e5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_3442e86b6d994a14de73f1b8c82cf0b8 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd [2019-01-21 08:22:53.389049] I [MSGID: 106499] [glusterd-handler.c:4314:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vol_3442e86b6d994a14de73f1b8c82cf0b8 [2019-01-21 08:23:25.346839] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req We can see following events from when we heal volumes. [2019-01-21 08:20:07.576070] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:20:07.580225] I [cli-rpc-ops.c:9182:gf_cli_heal_volume_cbk] 0-cli: Received resp to heal volume [2019-01-21 08:20:07.580326] I [input.c:31:cli_batch] 0-: Exiting with: -1 [2019-01-21 08:22:30.423311] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:22:30.463648] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:22:30.463718] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:30.463859] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:22:33.427710] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:34.581555] I [cli-rpc-ops.c:1472:gf_cli_start_volume_cbk] 0-cli: Received resp to start volume [2019-01-21 08:22:34.581678] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2019-01-21 08:22:53.345351] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:22:53.387992] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:22:53.388059] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:53.388138] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:22:53.394737] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2019-01-21 08:23:25.304688] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:23:25.346319] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:23:25.346389] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:23:25.346500] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 Please let us know steps to recover bricks. BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From bugzilla at redhat.com Mon Jan 21 23:21:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 21 Jan 2019 23:21:09 +0000 Subject: [Bugs] [Bug 1668118] New: Failure to start geo-replication for tiered volume. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668118 Bug ID: 1668118 Summary: Failure to start geo-replication for tiered volume. Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: geo-replication Severity: high Assignee: bugs at gluster.org Reporter: vnosov at stonefly.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Status of geo-replication workers on master nodes is "inconsistent" if master volume is tiered. Version-Release number of selected component (if applicable): GlusterFS 5.2 installation from source code TAR file How reproducible: 100% Steps to Reproduce: 1. Set up two nodes. One will host geo-replication master volume. Master volume has to be tiered. Other node will host geo-replication slave volume. [root at SC-10-10-63-182 log]# glusterfsd --version glusterfs 5.2 [root at SC-10-10-63-183 log]# glusterfsd --version glusterfs 5.2 2. On master node create tiered volume: [root at SC-10-10-63-182 log]# gluster volume info master-volume-1 Volume Name: master-volume-1 Type: Tier Volume ID: aa95df34-f181-456c-aa26-9756b68ed679 Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: 10.10.60.182:/exports/master-hot-tier/master-volume-1 Cold Tier: Cold Tier Type : Distribute Number of Bricks: 1 Brick2: 10.10.60.182:/exports/master-segment-1/master-volume-1 Options Reconfigured: features.ctr-sql-db-wal-autocheckpoint: 25000 features.ctr-sql-db-cachesize: 12500 cluster.tier-mode: cache features.ctr-enabled: on server.allow-insecure: on performance.quick-read: off performance.stat-prefetch: off nfs.addr-namelookup: off transport.address-family: inet nfs.disable: on cluster.enable-shared-storage: disable snap-activate-on-create: enable [root at SC-10-10-63-182 log]# gluster volume status master-volume-1 Status of volume: master-volume-1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.10.60.182:/exports/master-hot-tier /master-volume-1 62001 0 Y 15690 Cold Bricks: Brick 10.10.60.182:/exports/master-segment- 1/master-volume-1 62000 0 Y 9762 Tier Daemon on localhost N/A N/A Y 15713 Task Status of Volume master-volume-1 ------------------------------------------------------------------------------ There are no active volume tasks [root at SC-10-10-63-182 log]# gluster volume tier master-volume-1 status Node Promoted files Demoted files Status run time in h:m:s --------- --------- --------- --------- --------- localhost 0 0 in progress 0:3:40 Tiering Migration Functionality: master-volume-1: success 3. On slave node create slave volume: [root at SC-10-10-63-183 log]# gluster volume info slave-volume-1 Volume Name: slave-volume-1 Type: Distribute Volume ID: 569a340b-35f8-4109-8816-720982b11806 Status: Started Snapshot Count: 0 Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.10.60.183:/exports/slave-segment-1/slave-volume-1 Options Reconfigured: server.allow-insecure: on performance.quick-read: off performance.stat-prefetch: off nfs.addr-namelookup: off transport.address-family: inet nfs.disable: on cluster.enable-shared-storage: disable snap-activate-on-create: enable [root at SC-10-10-63-183 log]# gluster volume status slave-volume-1 Status of volume: slave-volume-1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.10.60.183:/exports/slave-segment-1 /slave-volume-1 62000 0 Y 2532 Task Status of Volume slave-volume-1 ------------------------------------------------------------------------------ There are no active volume tasks 4. Set up SSH access to slave node: SSH from 182 to 183: 20660 01/21/2019 13:58:54.930122501 1548107934 command: /usr/bin/ssh nasgorep at 10.10.60.183 /bin/pwd 20660 01/21/2019 13:58:55.021906148 1548107935 status=0 /usr/bin/ssh nasgorep at 10.10.60.183 /bin/pwd 20694 01/21/2019 13:58:56.169890800 1548107936 command: /usr/bin/ssh -q -oConnectTimeout=5 nasgorep at 10.10.60.183 /bin/pwd 2>&1 20694 01/21/2019 13:58:56.256032202 1548107936 status=0 /usr/bin/ssh -q -oConnectTimeout=5 nasgorep at 10.10.60.183 /bin/pwd 2>&1 5. Initialize geo-replication from master volume to slave volume: [root at SC-10-10-63-182 log]# vi /var/log/glusterfs/cmd_history.log [2019-01-21 21:59:08.942567] : system:: execute gsec_create : SUCCESS [2019-01-21 21:59:42.722194] : volume geo-replication master-volume-1 nasgorep at 10.10.60.183::slave-volume-1 create push-pem : SUCCESS [2019-01-21 21:59:49.527353] : volume geo-replication master-volume-1 nasgorep at 10.10.60.183::slave-volume-1 start : SUCCESS [2019-01-21 21:59:55.636198] : volume geo-replication master-volume-1 nasgorep at 10.10.60.183::slave-volume-1 status detail : SUCCESS 6. Check status of the geo-replication: Actual results: [root at SC-10-10-63-183 log]# /usr/sbin/gluster-mountbroker status +-----------+-------------+---------------------------+--------------+---------------------------+ | NODE | NODE STATUS | MOUNT ROOT | GROUP | USERS | +-----------+-------------+---------------------------+--------------+---------------------------+ | localhost | UP | /var/mountbroker-root(OK) | nasgorep(OK) | nasgorep(slave-volume-1) | +-----------+-------------+---------------------------+--------------+---------------------------+ [root at SC-10-10-63-182 log]# gluster volume geo-replication master-volume-1 nasgorep at 10.10.60.183::slave-volume-1 status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 10.10.60.182 master-volume-1 /exports/master-hot-tier/master-volume-1 nasgorep nasgorep at 10.10.60.183::slave-volume-1 N/A Stopped N/A N/A 10.10.60.182 master-volume-1 /exports/master-segment-1/master-volume-1 nasgorep nasgorep at 10.10.60.183::slave-volume-1 N/A Stopped N/A N/A Expected results: Status of the geo-replication workers on master node has to be "Active". Additional info: Contents of file /var/log/glusterfs/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.log on master node has explanation what is wrong: [root at SC-10-10-63-182 log]# vi /var/log/glusterfs/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.log [2019-01-21 21:59:39.347943] W [gsyncd(config-get):304:main] : Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:42.438145] I [gsyncd(monitor-status):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:42.454929] I [subcmds(monitor-status):29:subcmd_monitor_status] : Monitor Status Change status=Created [2019-01-21 21:59:48.756702] I [gsyncd(config-get):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:49.4720] I [gsyncd(config-get):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:49.239733] I [gsyncd(config-get):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:49.475193] I [gsyncd(monitor):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:49.868150] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing... [2019-01-21 21:59:49.868396] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker slave_node=10.10.60.183 brick=/exports/master-segment-1/master-volume-1 [2019-01-21 21:59:49.871593] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing... [2019-01-21 21:59:49.871963] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker slave_node=10.10.60.183 brick=/exports/master-hot-tier/master-volume-1 [2019-01-21 21:59:50.4395] I [monitor(monitor):268:monitor] Monitor: worker died before establishing connection brick=/exports/master-segment-1/master-volume-1 [2019-01-21 21:59:50.7447] I [monitor(monitor):268:monitor] Monitor: worker died before establishing connection brick=/exports/master-hot-tier/master-volume-1 [2019-01-21 21:59:50.8415] I [gsyncd(agent /exports/master-segment-1/master-volume-1):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:50.10383] I [gsyncd(agent /exports/master-hot-tier/master-volume-1):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:50.14039] I [repce(agent /exports/master-segment-1/master-volume-1):97:service_loop] RepceServer: terminating on reaching EOF. [2019-01-21 21:59:50.15556] I [changelogagent(agent /exports/master-hot-tier/master-volume-1):72:__init__] ChangelogAgent: Agent listining... [2019-01-21 21:59:50.15964] I [repce(agent /exports/master-hot-tier/master-volume-1):97:service_loop] RepceServer: terminating on reaching EOF. [2019-01-21 21:59:55.141768] I [gsyncd(config-get):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:55.380496] I [gsyncd(status):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 21:59:55.625045] I [gsyncd(status):308:main] : Using session config file path=/var/lib/glusterd/geo-replication/master-volume-1_10.10.60.183_slave-volume-1/gsyncd.conf [2019-01-21 22:00:00.66032] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=inconsistent [2019-01-21 22:00:00.66289] E [syncdutils(monitor):338:log_raise_exception] : FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 368, in twrap tf(*aargs) File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 339, in wmon slave_host, master, suuid, slavenodes) TypeError: 'int' object is not iterable Similar test on GlusterFS 3.12.14 does not show the same failure. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 05:14:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 05:14:55 +0000 Subject: [Bugs] [Bug 1667905] dict_leak in __glusterd_handle_cli_uuid_get function In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667905 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22069 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 05:14:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 05:14:56 +0000 Subject: [Bugs] [Bug 1667905] dict_leak in __glusterd_handle_cli_uuid_get function In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667905 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-22 05:14:56 --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22069 (glusterd: Avoid dict_leak in __glusterd_handle_cli_uuid_get function) merged (#3) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 05:23:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 05:23:04 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-22 05:23:04 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22066 (cluster/dht: Delete invalid linkto files in rmdir) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From shaik.salam at tcs.com Tue Jan 22 05:58:56 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Tue, 22 Jan 2019 11:28:56 +0530 Subject: [Bugs] Bricks are going offline unable to recover with heal/start force commands In-Reply-To: References: Message-ID: Can anyone respond how to recover bricks apart from heal/start force according to below events from logs. Please let me know any other logs required. Thanks in advance. BR Salam From: Shaik Salam/HYD/TCS To: bugs at gluster.org, gluster-users at gluster.org Date: 01/21/2019 10:03 PM Subject: Bricks are going offline unable to recover with heal/start force commands Hi, Bricks are in offline and unable to recover with following commands gluster volume heal gluster volume start force But still bricks are offline. sh-4.2# gluster volume status vol_3442e86b6d994a14de73f1b8c82cf0b8 Status of volume: vol_3442e86b6d994a14de73f1b8c82cf0b8 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 192.168.3.6:/var/lib/heketi/mounts/vg _ca57f326195c243be2380ce4e42a4191/brick_952 d75fd193c7209c9a81acbc23a3747/brick 49166 0 Y 269 Brick 192.168.3.5:/var/lib/heketi/mounts/vg _d5f17487744584e3652d3ca943b0b91b/brick_e15 c12cceae12c8ab7782dd57cf5b6c1/brick N/A N/A N N/A Brick 192.168.3.15:/var/lib/heketi/mounts/v g_462ea199185376b03e4b0317363bb88c/brick_17 36459d19e8aaa1dcb5a87f48747d04/brick 49173 0 Y 225 Self-heal Daemon on localhost N/A N/A Y 45826 Self-heal Daemon on 192.168.3.6 N/A N/A Y 65196 Self-heal Daemon on 192.168.3.15 N/A N/A Y 52915 Task Status of Volume vol_3442e86b6d994a14de73f1b8c82cf0b8 ------------------------------------------------------------------------------ We can see following events from when we start forcing volumes /mgmt/glusterd.so(+0xe2b3a) [0x7fca9e139b3a] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2605) [0x7fca9e139605] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fcaa346f0e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=vol_3442e86b6d994a14de73f1b8c82cf0b8 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd [2019-01-21 08:22:34.555068] E [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2b3a) [0x7fca9e139b3a] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xe2563) [0x7fca9e139563] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fcaa346f0e5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_3442e86b6d994a14de73f1b8c82cf0b8 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd [2019-01-21 08:22:53.389049] I [MSGID: 106499] [glusterd-handler.c:4314:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vol_3442e86b6d994a14de73f1b8c82cf0b8 [2019-01-21 08:23:25.346839] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req We can see following events from when we heal volumes. [2019-01-21 08:20:07.576070] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:20:07.580225] I [cli-rpc-ops.c:9182:gf_cli_heal_volume_cbk] 0-cli: Received resp to heal volume [2019-01-21 08:20:07.580326] I [input.c:31:cli_batch] 0-: Exiting with: -1 [2019-01-21 08:22:30.423311] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:22:30.463648] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:22:30.463718] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:30.463859] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:22:33.427710] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:34.581555] I [cli-rpc-ops.c:1472:gf_cli_start_volume_cbk] 0-cli: Received resp to start volume [2019-01-21 08:22:34.581678] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2019-01-21 08:22:53.345351] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:22:53.387992] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:22:53.388059] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:22:53.388138] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2019-01-21 08:22:53.394737] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2019-01-21 08:23:25.304688] I [cli.c:768:main] 0-cli: Started running gluster with version 4.1.5 [2019-01-21 08:23:25.346319] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-01-21 08:23:25.346389] I [socket.c:2632:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2019-01-21 08:23:25.346500] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 Please let us know steps to recover bricks. BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From bugzilla at redhat.com Tue Jan 22 06:53:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 06:53:12 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 --- Comment #7 from Worker Ant --- REVIEW: https://review.gluster.org/22045 (socket: fix issue when socket read return with EAGAIN) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 07:00:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 07:00:25 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 --- Comment #8 from Worker Ant --- REVIEW: https://review.gluster.org/22046 (socket: don't pass return value from protocol handler to event handler) merged (#3) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 07:31:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 07:31:01 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22072 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 07:31:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 07:31:02 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22072 (performance/readdir-ahead: Fix deadlock in readdir ahead.) posted (#3) for review on master by mohammed rafi kc -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 07:36:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 07:36:08 +0000 Subject: [Bugs] [Bug 1668190] New: Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Bug ID: 1668190 Summary: Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend Product: GlusterFS Version: mainline Status: NEW Component: core Keywords: ZStream Severity: high Assignee: bugs at gluster.org Reporter: moagrawa at redhat.com CC: bugs at gluster.org Depends On: 1654703 Target Milestone: --- Classification: Community Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1654703 [Bug 1654703] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 07:41:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 07:41:56 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |moagrawa at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 07:48:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 07:48:43 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22073 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 07:48:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 07:48:44 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22073 (core: heketi-cli is throwing error \"target is busy\") posted (#1) for review on master by MOHIT AGRAWAL -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 08:07:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 08:07:58 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22074 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 08:07:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 08:07:59 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #521 from Worker Ant --- REVIEW: https://review.gluster.org/22074 (tests/bug-brick-mux-restart: add extra information) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 09:07:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 09:07:19 +0000 Subject: [Bugs] [Bug 1668227] New: gluster(8) - Add SELinux context glusterd_brick_t to man page Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668227 Bug ID: 1668227 Summary: gluster(8) - Add SELinux context glusterd_brick_t to man page Product: GlusterFS Version: mainline Hardware: All OS: Linux Status: NEW Component: core Keywords: ZStream Severity: medium Assignee: bugs at gluster.org Reporter: psony at redhat.com CC: bugs at gluster.org Depends On: 1667961 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1667961 +++ Document URL: man 8 gluster Section Number and Name: N/A Describe the issue: glusterd_brick_t: the SELinux context is nowhere to be found inside gluster(8) or other man. Suggestions for improvement: Although I understand one should read the full product documentation before deploying Guster, glusterd_brick_t context is important information to be added to the man page, at least, as a quick reference. When using SELinux this may cause error when creating a new brick. Additional information: Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667961 [Bug 1667961] gluster(8) - Add SELinux context glusterd_brick_t to man page -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 09:20:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 09:20:44 +0000 Subject: [Bugs] [Bug 1668227] gluster(8) - Add SELinux context glusterd_brick_t to man page In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668227 Bipin Kunal changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bkunal at redhat.com Blocks| |1667961 Depends On|1667961 | Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667961 [Bug 1667961] gluster(8) - Add SELinux context glusterd_brick_t to man page -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 09:34:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 09:34:53 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22075 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 09:34:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 09:34:54 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #5 from Worker Ant --- REVIEW: https://review.gluster.org/22075 (mgmt/shd: Implement multiplexing in self heal daemon) posted (#1) for review on master by mohammed rafi kc -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 09:44:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 09:44:29 +0000 Subject: [Bugs] [Bug 1668239] New: [man page] Gluster(8) - Missing disperse-data parameter Gluster Console Manager man page Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668239 Bug ID: 1668239 Summary: [man page] Gluster(8) - Missing disperse-data parameter Gluster Console Manager man page Product: GlusterFS Version: mainline Hardware: All OS: Linux Status: NEW Component: disperse Keywords: ZStream Severity: medium Priority: medium Assignee: bugs at gluster.org Reporter: smulay at redhat.com CC: bugs at gluster.org Blocks: 1667954 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1667954 +++ Document URL: man 8 gluster Section Number and Name: COMMANDS -> Volume Commands Describe the issue: No man-page reference for `disperse-data` parameter when creating a new volume (gluster(8)): --- volume create [stripe ] [replica ] [disperse []] [redundancy ] [transport ] ... Create a new volume of the specified type using the specified bricks and transport type (the default transport type is tcp). To create a volume with both transports (tcp and rdma), give 'transport tcp,rdma' as an option. --- However "Usage:" message shows disperse-data: --- volume create [stripe ] [replica [arbiter ]] [disperse []] [disperse-data ] [redundancy ] [transport ] ?... [force] --- Suggestions for improvement: Add disperse-data to "man 8 gluster" explaining its utilization scenario. Additional information: If this bug should be proposed for a different release, please manually change the proposed release flag. Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667954 [Bug 1667954] [man page] Gluster(8) - Missing disperse-data parameter Gluster Console Manager man page -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 09:55:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 09:55:41 +0000 Subject: [Bugs] [Bug 1668245] New: gluster(8) - Man page - create gluster example session Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668245 Bug ID: 1668245 Summary: gluster(8) - Man page - create gluster example session Product: GlusterFS Version: mainline Hardware: All OS: Linux Status: NEW Component: glusterd Keywords: ZStream Severity: medium Priority: medium Assignee: bugs at gluster.org Reporter: psony at redhat.com CC: bugs at gluster.org Blocks: 1667963 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1667963 +++ Document URL: man 8 gluster Section Number and Name: N/A Describe the issue: Create a new session called EXAMPLES to gluster(8) as quick reference for usage help, or even a new man page, something like gluster(7). Suggestions for improvement: An EXAMPLE session would be a good addition to Gluster man pages. One very good example that comes to mind is LVMTHIN(7). Additional information: Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667963 [Bug 1667963] gluster(8) - Man page - create gluster example session -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 09:57:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 09:57:31 +0000 Subject: [Bugs] [Bug 1668245] gluster(8) - Man page - create gluster example session In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668245 sameer kandarkar changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |skandark at redhat.com Assignee|bugs at gluster.org |skandark at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 10:04:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 10:04:52 +0000 Subject: [Bugs] [Bug 1668259] New: Glusterfs 5.3 RPMs can't be build on rhel7 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668259 Bug ID: 1668259 Summary: Glusterfs 5.3 RPMs can't be build on rhel7 Product: GlusterFS Version: 5 Status: NEW Component: packaging Assignee: bugs at gluster.org Reporter: kompastver at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Glusterfs 5.3 RPMs can't be build on rhel7 Version-Release number of selected component (if applicable): Glusterfs 5.3 & rhel7 How reproducible: Build RPMs from this commit https://src.fedoraproject.org/rpms/glusterfs/tree/48abd2568d3b3d5c358d2de48d4f55bb8000a02e on rhel7. Actual results: Build failed because files from the spec not found on filesystem. Processing files: glusterfs-devel-5.3-1.el7.x86_64 error: Directory not found: /var/tmp/build/RPM_BUILD_CHROOT/glusterfs-5.3-1.el7.x86_64/usr/lib64/glusterfs/5.3/xlator/features/glupy error: File not found by glob: /var/tmp/build/RPM_BUILD_CHROOT/glusterfs-5.3-1.el7.x86_64/usr/lib64/glusterfs/5.3/xlator/features/glupy/debug-trace.* error: File not found by glob: /var/tmp/build/RPM_BUILD_CHROOT/glusterfs-5.3-1.el7.x86_64/usr/lib64/glusterfs/5.3/xlator/features/glupy/helloworld.* error: File not found by glob: /var/tmp/build/RPM_BUILD_CHROOT/glusterfs-5.3-1.el7.x86_64/usr/lib64/glusterfs/5.3/xlator/features/glupy/negative.* RPM build errors: Directory not found: /var/tmp/build/RPM_BUILD_CHROOT/glusterfs-5.3-1.el7.x86_64/usr/lib64/glusterfs/5.3/xlator/features/glupy File not found by glob: /var/tmp/build/RPM_BUILD_CHROOT/glusterfs-5.3-1.el7.x86_64/usr/lib64/glusterfs/5.3/xlator/features/glupy/debug-trace.* File not found by glob: /var/tmp/build/RPM_BUILD_CHROOT/glusterfs-5.3-1.el7.x86_64/usr/lib64/glusterfs/5.3/xlator/features/glupy/helloworld.* File not found by glob: /var/tmp/build/RPM_BUILD_CHROOT/glusterfs-5.3-1.el7.x86_64/usr/lib64/glusterfs/5.3/xlator/features/glupy/negative.* -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 10:23:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 10:23:19 +0000 Subject: [Bugs] [Bug 1655052] Automatic Splitbrain with size as policy must not resolve splitbrains when both the copies are of same size In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655052 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/21763 (afr: not resolve splitbrains when copies are of same size) merged (#15) on master by Ravishankar N -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 10:37:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 10:37:35 +0000 Subject: [Bugs] [Bug 1668268] New: Unable to mount gluster volume Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668268 Bug ID: 1668268 Summary: Unable to mount gluster volume Product: GlusterFS Version: mainline Status: NEW Component: rpc Assignee: bugs at gluster.org Reporter: pgurusid at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: When we try to mount gluster volume with master gluster and master gd2, it fails. The mount process crashes with the bt: (gdb) bt #0 0x00007ffff60f2207 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 #1 0x00007ffff60f38f8 in __GI_abort () at abort.c:90 #2 0x00007ffff6134d27 in __libc_message (do_abort=do_abort at entry=2, fmt=fmt at entry=0x7ffff6246678 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196 #3 0x00007ffff613d489 in malloc_printerr (ar_ptr=0x7fffe4000020, ptr=, str=0x7ffff6246738 "double free or corruption (fasttop)", action=3) at malloc.c:5004 #4 _int_free (av=0x7fffe4000020, p=, have_lock=0) at malloc.c:3843 #5 0x00007ffff7abae8e in dict_destroy (this=0x7fffe4001d48) at dict.c:700 #6 0x00007ffff7abafa0 in dict_unref (this=0x7fffe4001d48) at dict.c:739 #7 0x0000000000411151 in mgmt_getspec_cbk (req=0x7fffdc002a28, iov=0x7fffdc002a60, count=1, myframe=0x7fffdc001de8) at glusterfsd-mgmt.c:2132 #8 0x00007ffff78694c2 in rpc_clnt_handle_reply (clnt=0x697960, pollin=0x7fffe4001010) at rpc-clnt.c:755 #9 0x00007ffff78699eb in rpc_clnt_notify (trans=0x697ce0, mydata=0x697990, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fffe4001010) at rpc-clnt.c:922 #10 0x00007ffff7865a3e in rpc_transport_notify (this=0x697ce0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fffe4001010) at rpc-transport.c:541 #11 0x00007fffeba1e6eb in socket_event_poll_in (this=0x697ce0, notify_handled=true) at socket.c:2508 #12 0x00007fffeba1f703 in socket_event_handler (fd=13, idx=0, gen=1, data=0x697ce0, poll_in=1, poll_out=0, poll_err=0, event_thread_died=0 '\000') at socket.c:2908 #13 0x00007ffff7b453dc in event_dispatch_epoll_handler (event_pool=0x673e90, event=0x7fffe9f78e80) at event-epoll.c:642 #14 0x00007ffff7b458f8 in event_dispatch_epoll_worker (data=0x6d6de0) at event-epoll.c:756 #15 0x00007ffff68f1dd5 in start_thread (arg=0x7fffe9f79700) at pthread_create.c:307 #16 0x00007ffff61b9ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 (gdb) Version-Release number of selected component (if applicable): master gluster master glusterd2 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 10:48:26 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 10:48:26 +0000 Subject: [Bugs] [Bug 1668268] Unable to mount gluster volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668268 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22076 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 10:48:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 10:48:27 +0000 Subject: [Bugs] [Bug 1668268] Unable to mount gluster volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668268 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22076 (rpc: Fix double free) posted (#1) for review on master by Poornima G -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 11:11:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 11:11:54 +0000 Subject: [Bugs] [Bug 1668286] New: READDIRP incorrectly updates posix-acl inode ctx Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668286 Bug ID: 1668286 Summary: READDIRP incorrectly updates posix-acl inode ctx Product: GlusterFS Version: 5 OS: Linux Status: NEW Component: read-ahead Severity: high Assignee: bugs at gluster.org Reporter: homma at allworks.co.jp CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: On FUSE client with mount option use-readdirp=on (default) and acl, access to a file is denied for about a second after listing the directory in which the file resides. Version-Release number of selected component (if applicable): glusterfs-fuse.x86_64 5.2-1.el7 from centos-gluster5 repository How reproducible: Always, with mount option use-readdirp=on and acl Steps to Reproduce: 1. Mount GlusterFS volume with acl and use-readdirp=on 2. Chdir to the mounted directory 3. Execute the following commands: echo TEST > foo; echo -n "[`date -u --rfc-3339=ns`] "; cat foo; ls -l; while :; do echo -n "[`date -u --rfc-3339=ns`] "; cat foo && break; usleep 200000; done Actual results: Access is denied for about a second after executing ls: [2019-01-22 10:24:18.802855191+00:00] TEST total 1 -rw-rw-r-- 1 centos centos 5 Jan 22 16:30 bar -rw-rw-r-- 1 centos centos 5 Jan 22 19:24 foo [2019-01-22 10:24:18.825725474+00:00] cat: foo: Permission denied [2019-01-22 10:24:19.029015958+00:00] cat: foo: Permission denied [2019-01-22 10:24:19.232249483+00:00] cat: foo: Permission denied [2019-01-22 10:24:19.435580108+00:00] cat: foo: Permission denied [2019-01-22 10:24:19.638781941+00:00] cat: foo: Permission denied [2019-01-22 10:24:19.843016193+00:00] TEST Gluster log on the client: [2019-01-22 10:24:18.826671] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: e16e1d3e-7518-4323-982f-1ad348f9608f, req(uid:1000,gid:1000,perm:4,ngrps:4), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:-) [Permission denied] [2019-01-22 10:24:18.826711] W [fuse-bridge.c:1124:fuse_fd_cbk] 0-glusterfs-fuse: 930: OPEN() /centos/test/foo => -1 (Permission denied) [2019-01-22 10:24:19.030036] W [fuse-bridge.c:1124:fuse_fd_cbk] 0-glusterfs-fuse: 931: OPEN() /centos/test/foo => -1 (Permission denied) [2019-01-22 10:24:19.233301] W [fuse-bridge.c:1124:fuse_fd_cbk] 0-glusterfs-fuse: 932: OPEN() /centos/test/foo => -1 (Permission denied) [2019-01-22 10:24:19.436612] W [fuse-bridge.c:1124:fuse_fd_cbk] 0-glusterfs-fuse: 933: OPEN() /centos/test/foo => -1 (Permission denied) [2019-01-22 10:24:19.639804] W [fuse-bridge.c:1124:fuse_fd_cbk] 0-glusterfs-fuse: 934: OPEN() /centos/test/foo => -1 (Permission denied) The message "I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-posix-acl-autoload: client: -, gfid: e16e1d3e-7518-4323-982f-1ad348f9608f, req(uid:1000,gid:1000,perm:4,ngrps:4), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:READDIRP, acl:-) [Permission denied]" repeated 4 times between [2019-01-22 10:24:18.826671] and [2019-01-22 10:24:19.639797] Expected results: Access to the file is always granted. Additional info: In readdir-ahead.c, rda_fill_fd_cbk() replaces dentries and zeroes out iatts exept for ia_gfid and ia_type. Then in posix-acl.c, posix_acl_readdirp_cbk() updates its inode ctx by that zeroed permission, and permission is denied. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 12:36:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 12:36:57 +0000 Subject: [Bugs] [Bug 1657607] Convert nr_files to gf_atomic in posix_private structure In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657607 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22077 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 12:36:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 12:36:58 +0000 Subject: [Bugs] [Bug 1657607] Convert nr_files to gf_atomic in posix_private structure In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657607 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22077 (posix: Change data type to dump nr_files to statedump) posted (#1) for review on master by MOHIT AGRAWAL -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 13:48:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 13:48:09 +0000 Subject: [Bugs] [Bug 1635863] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1635863 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-22 13:48:09 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/21948 (rpc: use address-family option from vol file) merged (#16) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 13:48:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 13:48:10 +0000 Subject: [Bugs] [Bug 1664590] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Bug 1664590 depends on bug 1635863, which changed state. Bug 1635863 Summary: Gluster peer probe doesn't work for IPv6 https://bugzilla.redhat.com/show_bug.cgi?id=1635863 What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:49 +0000 Subject: [Bugs] [Bug 1623107] FUSE client's memory leak In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1623107 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Fixed In Version| |glusterfs-5.3 Resolution|NEXTRELEASE |CURRENTRELEASE --- Comment #40 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:49 +0000 Subject: [Bugs] [Bug 1648642] fails to sync non-ascii (utf8) file and directory names, causes permanently faulty geo-replication state In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1648642 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-5.3 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:08:49 --- Comment #4 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:50 +0000 Subject: [Bugs] [Bug 1650893] fails to sync non-ascii (utf8) file and directory names, causes permanently faulty geo-replication state In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1650893 Bug 1650893 depends on bug 1648642, which changed state. Bug 1648642 Summary: fails to sync non-ascii (utf8) file and directory names, causes permanently faulty geo-replication state https://bugzilla.redhat.com/show_bug.cgi?id=1648642 What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:49 +0000 Subject: [Bugs] [Bug 1659085] GlusterFS 5.3 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659085 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Fixed In Version| |glusterfs-5.3 Resolution|NEXTRELEASE |CURRENTRELEASE --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:49 +0000 Subject: [Bugs] [Bug 1659563] gluster-blockd segfaults because of a null-dereference in shard.so In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659563 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-5.3 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:08:49 --- Comment #5 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:51 +0000 Subject: [Bugs] [Bug 1659085] GlusterFS 5.3 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659085 Bug 1659085 depends on bug 1659563, which changed state. Bug 1659563 Summary: gluster-blockd segfaults because of a null-dereference in shard.so https://bugzilla.redhat.com/show_bug.cgi?id=1659563 What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:49 +0000 Subject: [Bugs] [Bug 1659676] Memory leak: dict_t leak in rda_opendir In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659676 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-5.3 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:08:49 --- Comment #5 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:52 +0000 Subject: [Bugs] [Bug 1659439] Memory leak: dict_t leak in rda_opendir In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659439 Bug 1659439 depends on bug 1659676, which changed state. Bug 1659676 Summary: Memory leak: dict_t leak in rda_opendir https://bugzilla.redhat.com/show_bug.cgi?id=1659676 What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:49 +0000 Subject: [Bugs] [Bug 1660736] dht_revalidate may not heal attrs on the brick root In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660736 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-5.3 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:08:49 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:49 +0000 Subject: [Bugs] [Bug 1660932] Fix tests/bugs/shard/zero-flag.t In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1660932 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-5.3 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:08:49 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:53 +0000 Subject: [Bugs] [Bug 1662635] Fix tests/bugs/shard/zero-flag.t In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662635 Bug 1662635 depends on bug 1660932, which changed state. Bug 1660932 Summary: Fix tests/bugs/shard/zero-flag.t https://bugzilla.redhat.com/show_bug.cgi?id=1660932 What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:49 +0000 Subject: [Bugs] [Bug 1662200] NL cache: fix typos In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662200 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-5.3 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:08:49 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:56 +0000 Subject: [Bugs] [Bug 1663131] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663131 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-5.3 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:08:56 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:08:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:08:56 +0000 Subject: [Bugs] [Bug 1665803] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665803 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-5.3 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:08:56 --- Comment #5 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.3, please open a new bug report. glusterfs-5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:20 +0000 Subject: [Bugs] [Bug 1649843] GlusterFS 4.1.7 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1649843 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Fixed In Version| |glusterfs-4.1.7 Resolution|NEXTRELEASE |CURRENTRELEASE --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.7, please open a new bug report. glusterfs-4.1.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:20 +0000 Subject: [Bugs] [Bug 1654118] [geo-rep]: Failover / Failback shows fault status in a non-root setup In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654118 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-4.1.7 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:09:20 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.7, please open a new bug report. glusterfs-4.1.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:22 +0000 Subject: [Bugs] [Bug 1654117] [geo-rep]: Failover / Failback shows fault status in a non-root setup In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654117 Bug 1654117 depends on bug 1654118, which changed state. Bug 1654118 Summary: [geo-rep]: Failover / Failback shows fault status in a non-root setup https://bugzilla.redhat.com/show_bug.cgi?id=1654118 What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:20 +0000 Subject: [Bugs] [Bug 1654229] Provide an option to silence glfsheal logs In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1654229 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-4.1.7 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:09:20 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.7, please open a new bug report. glusterfs-4.1.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:20 +0000 Subject: [Bugs] [Bug 1655527] Incorrect usage of local->fd in afr_open_ftruncate_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655527 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-4.1.7 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:09:20 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.7, please open a new bug report. glusterfs-4.1.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:23 +0000 Subject: [Bugs] [Bug 1651322] Incorrect usage of local->fd in afr_open_ftruncate_cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651322 Bug 1651322 depends on bug 1655527, which changed state. Bug 1655527 Summary: Incorrect usage of local->fd in afr_open_ftruncate_cbk https://bugzilla.redhat.com/show_bug.cgi?id=1655527 What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:20 +0000 Subject: [Bugs] [Bug 1655561] gfid heal does not happen when there is no source brick In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655561 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-4.1.7 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:09:20 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.7, please open a new bug report. glusterfs-4.1.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:24 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:24 +0000 Subject: [Bugs] [Bug 1655545] gfid heal does not happen when there is no source brick In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1655545 Bug 1655545 depends on bug 1655561, which changed state. Bug 1655561 Summary: gfid heal does not happen when there is no source brick https://bugzilla.redhat.com/show_bug.cgi?id=1655561 What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:20 +0000 Subject: [Bugs] [Bug 1662635] Fix tests/bugs/shard/zero-flag.t In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662635 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-4.1.7 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:09:20 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.7, please open a new bug report. glusterfs-4.1.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:20 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:20 +0000 Subject: [Bugs] [Bug 1663132] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663132 Shyamsundar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Fixed In Version| |glusterfs-4.1.7 Resolution|--- |CURRENTRELEASE Last Closed| |2019-01-22 14:09:20 --- Comment #3 from Shyamsundar --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.7, please open a new bug report. glusterfs-4.1.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-January/000118.html [2] https://www.gluster.org/pipermail/gluster-users/ -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 14:09:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 14:09:25 +0000 Subject: [Bugs] [Bug 1663131] [Ganesha] Ganesha failed on one node while exporting volumes in loop In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1663131 Bug 1663131 depends on bug 1663132, which changed state. Bug 1663132 Summary: [Ganesha] Ganesha failed on one node while exporting volumes in loop https://bugzilla.redhat.com/show_bug.cgi?id=1663132 What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |CLOSED Resolution|--- |CURRENTRELEASE -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 22 10:48:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 22 Jan 2019 10:48:27 +0000 Subject: [Bugs] [Bug 1668268] Unable to mount gluster volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668268 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-22 17:22:57 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22076 (rpc: Fix double free) merged (#3) on master by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 23 03:17:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 03:17:10 +0000 Subject: [Bugs] [Bug 1657607] Convert nr_files to gf_atomic in posix_private structure In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1657607 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed|2019-01-15 11:53:02 |2019-01-23 03:17:10 --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22077 (posix: Change data type to dump nr_files to statedump) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 23 03:58:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 03:58:14 +0000 Subject: [Bugs] [Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1662368 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |MODIFIED -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 23 10:39:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 10:39:12 +0000 Subject: [Bugs] [Bug 1318493] Introduce ctime-xlator to return correct (client-side set) ctime In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1318493 Rubin Simons changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rubin at xs4all.nl --- Comment #11 from Rubin Simons --- Hello, I seem to be hit by exactly this issue (GlusterFS 3.8.4 using OpenShift 3.9, Elasticsearch 6.5.4); I see that those patches have been adandoned. It seems such a fundamental issue that I would assume would impact a lot more than just Elasticsearch/Lucene/Solr users; has this ever been fixed in later upstream versions of GlusterFS? Are there any mount options that can help avoid the problem? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 23 12:51:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 12:51:42 +0000 Subject: [Bugs] [Bug 1318493] Introduce ctime-xlator to return correct (client-side set) ctime In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1318493 --- Comment #12 from Mohammed Rafi KC --- This is already fixed in the latest releases starting from gluster-5. The patches you mentioned got automatically abandoned because of 90 days inactivity. But later this was added with other change-id. One among them is https://review.gluster.org/#/c/glusterfs/+/19857/. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 23 12:59:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 12:59:51 +0000 Subject: [Bugs] [Bug 1318493] Introduce ctime-xlator to return correct (client-side set) ctime In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1318493 --- Comment #13 from Amar Tumballi --- @Rubin, considering you are using OpenShift, can you test if things would work fine with containers from https://github.com/gluster/gcs project? They have latest glusterfs image (nightly), and you should be able to pick a RWX PV without issues right now. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 23 13:04:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 13:04:54 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #4 from Raghavendra G --- >From preliminary tests I see two reasons for this: 1. inode-invalidations triggered by md-cache 2. Fuse auto invalidations With a hacky fix removing both of the above, I can see read after write being served from kernel page-cache. I'll update the bug with more details discussing validity/limitations with the above two approaches later. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 23 16:00:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 16:00:29 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Amgad changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amgad.saleh at nokia.com Depends On| |1651246 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 [Bug 1651246] Failed to dispatch handler -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 23 16:00:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 16:00:29 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Amgad changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1667103 (glusterfs-5.4) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 23 16:01:31 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 16:01:31 +0000 Subject: [Bugs] [Bug 1659708] Optimize by not stopping (restart) selfheal deamon (shd) when a volume is stopped unless it is the last volume In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1659708 --- Comment #6 from Worker Ant --- REVIEW: https://review.gluster.org/22072 (performance/readdir-ahead: Fix deadlock in readdir ahead.) merged (#11) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 23 16:06:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 23 Jan 2019 16:06:22 +0000 Subject: [Bugs] [Bug 1666833] move few recurring logs to DEBUG level. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666833 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22053 (core: move logs which are only developer relevant to DEBUG level) merged (#3) on master by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 04:37:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 04:37:14 +0000 Subject: [Bugs] [Bug 1668989] New: Unable to delete directories that contain linkto files that point to itself. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Bug ID: 1668989 Summary: Unable to delete directories that contain linkto files that point to itself. Product: Red Hat Gluster Storage Version: 3.4 Status: NEW Component: distribute Assignee: nbalacha at redhat.com Reporter: nbalacha at redhat.com QA Contact: tdesala at redhat.com CC: bugs at gluster.org, rhs-bugs at redhat.com, sankarshan at redhat.com, storage-qa-internal at redhat.com Depends On: 1667804 Blocks: 1667556 Target Milestone: --- Classification: Red Hat +++ This bug was initially created as a clone of Bug #1667804 +++ Description of problem: A directory containing linkto files that point to itself cannot be deleted. Version-Release number of selected component (if applicable): How reproducible: Consistently Steps to Reproduce: 1. gluster v create tvol 192.168.122.7:/bricks/brick2/tvol-{1..2} 2. gluster v start tvol 3. mount -t glusterfs -s 192.168.122.7:/tvol /mnt/g1 4. cd /mnt/g1 5. mkdir -p dir0/dir1/dir2 6. cd dir0/dir1/dir2 7. for i in {1..100}; do echo "Test file" > tfile-$i; done 8. for i in {1..100}; do mv tfile-$i ntfile-$i; done 9. gluster v remove-brick tvol 192.168.122.7:/bricks/brick2/tvol-2 start Once the remove-brick status shows "completed", 10. gluster v remove-brick tvol 192.168.122.7:/bricks/brick2/tvol-2 stop You should now have only linkto files in 192.168.122.7:/bricks/brick2/tvol-2/dir0/dir1/dir2 and they should all be pointing to tvol-client-0. Manually change the linkto xattr value for every file in brick2 to point to itself, in this case "tvol-client-1"(make sure the string is null terminated). 11. setfattr -n trusted.glusterfs.dht.linkto -v 0x74766f6c2d636c69656e742d3100 /bricks/brick2/tvol-2/dir0/dir1/dir2/ntfile-* 12. Try to delete the directory from the mount point: [root at myserver g1]# rm -rf * Actual results: [root at myserver g1]# rm -rf * rm: cannot remove ?dir0/dir1/dir2?: Directory not empty Expected results: The directory should be deleted as there are no data files inside. Additional info: --- Additional comment from Worker Ant on 2019-01-21 09:50:09 UTC --- REVIEW: https://review.gluster.org/22066 (cluster/dht: Delete invalid linkto files in rmdir) posted (#1) for review on master by N Balachandran --- Additional comment from Worker Ant on 2019-01-22 05:23:04 UTC --- REVIEW: https://review.gluster.org/22066 (cluster/dht: Delete invalid linkto files in rmdir) merged (#2) on master by Amar Tumballi Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 04:37:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 04:37:14 +0000 Subject: [Bugs] [Bug 1667804] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667804 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1668989 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 04:37:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 04:37:18 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Red Hat Bugzilla Rules Engine changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |ZStream -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 04:43:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 04:43:40 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #5 from Manoj Pillai --- (In reply to Raghavendra G from comment #4) > From preliminary tests I see two reasons for this: > 1. inode-invalidations triggered by md-cache > 2. Fuse auto invalidations Trying with kernel NFS, another distributed fs solution. I see that cache is retained at the end of the write test, and both read-after-write and read-after-read are served from the page cache. In principle, if kNFS can do it, FUSE should be able to do it. I think :D. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 04:46:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 04:46:48 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 05:14:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 05:14:40 +0000 Subject: [Bugs] [Bug 1575836] logic in S30samba-start.sh hook script needs tweaking In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1575836 Anoop C S changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|mzink at redhat.com |anoopcs at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 05:28:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 05:28:44 +0000 Subject: [Bugs] [Bug 1668989] Unable to delete directories that contain linkto files that point to itself. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668989 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |high CC| |sheggodu at redhat.com Severity|unspecified |high -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 05:38:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 05:38:40 +0000 Subject: [Bugs] [Bug 1668995] New: DHT: Provide a virtual xattr to get the hash subvol for a file Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668995 Bug ID: 1668995 Summary: DHT: Provide a virtual xattr to get the hash subvol for a file Product: Red Hat Gluster Storage Version: 3.4 Status: NEW Component: distribute Assignee: nbalacha at redhat.com Reporter: nbalacha at redhat.com QA Contact: tdesala at redhat.com CC: bugs at gluster.org, rhs-bugs at redhat.com, sankarshan at redhat.com, storage-qa-internal at redhat.com Depends On: 1624244 Target Milestone: --- Classification: Red Hat +++ This bug was initially created as a clone of Bug #1624244 +++ Description of problem: There is no easy way to figure out the hashed subvol for a particular file. Currently, we need to read the trusted.glusterfs.dht xattr for the parent directory from each brick, calculate the file hash and compare it against the dir layout to determine the correct brick. This is time consuming and tedious, especially with larger volumes and customer setups. Provide a getfattr interface to allow users to figure out the hashed subvolume for a file based on the in memory layout of the parent directory. This becomes more critical to allow easier debugging as lookup-optimize is enabled for volumes. Credit: rgowdapp at redhat.com for the idea. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2018-08-31 07:01:58 UTC --- REVIEW: https://review.gluster.org/21052 (cluster/dht: Rework the debug xattr to get hashed subvol) posted (#1) for review on master by N Balachandran --- Additional comment from Worker Ant on 2018-09-07 08:57:00 UTC --- COMMIT: https://review.gluster.org/21052 committed in master by "N Balachandran" with a commit message- cluster/dht: Rework the debug xattr to get hashed subvol The earlier implementation required the file to already exist when trying to get the hashed subvol. The reworked implementation allows a user to get the hashed subvol for any filename, whether it exists or not. Usage: getfattr -n "dht.file.hashed-subvol." Eg:To get the hashed subvol for file-1 inside dir-1 getfattr -n "dht.file.hashed-subvol.file-1" /mnt/gluster/dir1 Change-Id: Iae20bd5f56d387ef48c1c0a4ffa9f692866bf739 fixes: bz#1624244 Signed-off-by: N Balachandran --- Additional comment from Shyamsundar on 2018-10-23 15:18:02 UTC --- This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report. glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html [2] https://www.gluster.org/pipermail/gluster-users/ Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1624244 [Bug 1624244] DHT: Rework the virtual xattr to get the hash subvol -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 05:38:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 05:38:40 +0000 Subject: [Bugs] [Bug 1624244] DHT: Rework the virtual xattr to get the hash subvol In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1624244 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1668995 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1668995 [Bug 1668995] DHT: Provide a virtual xattr to get the hash subvol for a file -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 05:38:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 05:38:43 +0000 Subject: [Bugs] [Bug 1668995] DHT: Provide a virtual xattr to get the hash subvol for a file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668995 Red Hat Bugzilla Rules Engine changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |ZStream -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 05:39:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 05:39:22 +0000 Subject: [Bugs] [Bug 1668995] DHT: Provide a virtual xattr to get the hash subvol for a file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668995 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 05:47:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 05:47:45 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #522 from Worker Ant --- REVIEW: https://review.gluster.org/22074 (tests/bug-brick-mux-restart: add extra information) merged (#3) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 24 06:55:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 06:55:17 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-24 06:55:17 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22073 (core: heketi-cli is throwing error \"target is busy\") merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 07:08:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 07:08:40 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Mohit Agrawal changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1669020 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1669020 [Bug 1669020] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 07:45:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 07:45:44 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1669020 Depends On|1669020 | Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1669020 [Bug 1669020] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 07:46:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 07:46:55 +0000 Subject: [Bugs] [Bug 1668995] DHT: Provide a virtual xattr to get the hash subvol for a file In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668995 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|unspecified |medium CC| |sheggodu at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 07:47:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 07:47:55 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On|1654703 | Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1654703 [Bug 1654703] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 08:13:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 08:13:34 +0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22089 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 24 08:13:35 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 08:13:35 +0000 Subject: [Bugs] [Bug 1633930] ASan (address sanitizer) fixes - Blanket bug In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1633930 --- Comment #59 from Worker Ant --- REVIEW: https://review.gluster.org/22089 (cli : fix mem leak during cli_cmd_volume_gsync_set_cbk) posted (#1) for review on master by Sunny Kumar -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 24 09:36:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 09:36:03 +0000 Subject: [Bugs] [Bug 1669077] New: [ovirt-gluster] Fuse mount crashed while creating the preallocated image Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669077 Bug ID: 1669077 Summary: [ovirt-gluster] Fuse mount crashed while creating the preallocated image Product: GlusterFS Version: mainline Hardware: x86_64 OS: Linux Status: NEW Component: sharding Severity: high Assignee: bugs at gluster.org Reporter: kdhananj at redhat.com QA Contact: bugs at gluster.org CC: bugs at gluster.org Depends On: 1668304 Blocks: 1668309 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1668304 +++ Description of problem: ------------------------ Fuse mount crashed while creating the preallocated image of size 1TB on a arbitrated replicate volume Version-Release number of selected component (if applicable): -------------------------------------------------------------- RHGS 3.4.3 nightly ( glusterfs-3.12.2-38.el7rhgs ) RHV 4.2.8 How reproducible: ----------------- 1 out of 2 times Steps to Reproduce: ------------------- 1. Create a arbitrated replicate volume 2. Fuse mount it 3. Create preallocated image of size 1TB # qemu-img create -f qcow2 -o preallocation=falloc /mnt/test1/vm1.img 1072G Actual results: --------------- Fuse mount crashed with segfault Expected results: ----------------- Preallocated image should be created successfully --- Additional comment from SATHEESARAN on 2019-01-22 12:10:19 UTC --- 1. Cluster info ---------------- There are 3 nodes in the cluster 2. Volume info --------------- [root@ ~]# gluster volume info data Volume Name: data Type: Replicate Volume ID: 7eb49e90-e2b6-4f8f-856e-7108212dbb72 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: server1:/gluster_bricks/data/data Brick2: server2:/gluster_bricks/data/data Brick3: server3:/gluster_bricks/data/data (arbiter) Options Reconfigured: performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off cluster.choose-local: off client.event-threads: 4 server.event-threads: 4 storage.owner-uid: 36 storage.owner-gid: 36 network.ping-timeout: 30 performance.strict-o-direct: on cluster.granular-entry-heal: enable cluster.enable-shared-storage: enable --- Additional comment from SATHEESARAN on 2019-01-22 12:11:03 UTC --- Backtrace ----------- [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/data /mnt/test1'. Program terminated with signal 11, Segmentation fault. #0 0x00007fb1233e0a7a in __inode_get_xl_index (xlator=0x7fb1100143b0, inode=0x7fb104026258) at inode.c:455 455 if ((inode->_ctx[xlator->xl_id].xl_key != NULL) && (gdb) bt #0 0x00007fb1233e0a7a in __inode_get_xl_index (xlator=0x7fb1100143b0, inode=0x7fb104026258) at inode.c:455 #1 __inode_ref (inode=inode at entry=0x7fb104026258) at inode.c:537 #2 0x00007fb1233e0b81 in inode_ref (inode=inode at entry=0x7fb104026258) at inode.c:581 #3 0x00007fb1233f5d2b in __fd_create (inode=inode at entry=0x7fb104026258, pid=pid at entry=0) at fd.c:633 #4 0x00007fb1233f6f4a in __fd_anonymous (inode=inode at entry=0x7fb104026258, flags=flags at entry=2) at fd.c:779 #5 0x00007fb1233f729d in fd_anonymous (inode=0x7fb104026258) at fd.c:803 #6 0x00007fb115161534 in shard_post_lookup_fsync_handler (frame=0x7fb0c05eb178, this=0x7fb1100143b0) at shard.c:5936 #7 0x00007fb11514913c in shard_lookup_base_file (frame=frame at entry=0x7fb0c05eb178, this=this at entry=0x7fb1100143b0, loc=loc at entry=0x7fb10800a158, handler=handler at entry=0x7fb115161030 ) at shard.c:1746 #8 0x00007fb1151544c3 in shard_fsync (frame=0x7fb0c05eb178, this=0x7fb1100143b0, fd=0x7fb0f800eb78, datasync=1, xdata=0x0) at shard.c:6015 #9 0x00007fb114f30189 in wb_fsync_helper (frame=0x7fb0f80022e8, this=0x7fb1100159d0, fd=0x7fb0f800eb78, datasync=1, xdata=0x0) at write-behind.c:1974 #10 0x00007fb1233f5b15 in call_resume_keep_stub (stub=0x7fb0f80250f8) at call-stub.c:2582 #11 0x00007fb114f35a69 in wb_do_winds (wb_inode=wb_inode at entry=0x7fb0f800dd70, tasks=tasks at entry=0x7fb10dae7510) at write-behind.c:1672 #12 0x00007fb114f35b7b in wb_process_queue (wb_inode=wb_inode at entry=0x7fb0f800dd70) at write-behind.c:1709 #13 0x00007fb114f35c57 in wb_fulfill_cbk (frame=0x7fb0f8010a58, cookie=, this=, op_ret=, op_errno=, prebuf=, postbuf=0x7fb0f8003670, xdata=0x7fb0c065ee98) at write-behind.c:1054 #14 0x00007fb115156840 in shard_common_inode_write_success_unwind (fop=, frame=0x7fb0f80019b8, op_ret=65536) at shard.c:903 #15 0x00007fb115156bc0 in shard_common_inode_write_post_update_size_handler (frame=, this=) at shard.c:5214 #16 0x00007fb115147cc0 in shard_update_file_size (frame=frame at entry=0x7fb0f80019b8, this=this at entry=0x7fb1100143b0, fd=0x7fb0f800eb78, loc=loc at entry=0x0, handler=handler at entry=0x7fb115156ba0 ) at shard.c:1201 #17 0x00007fb11515e811 in shard_common_inode_write_do_cbk (frame=frame at entry=0x7fb0f80019b8, cookie=0x7fb0f800eb78, this=0x7fb1100143b0, op_ret=op_ret at entry=65536, op_errno=op_errno at entry=0, pre=pre at entry=0x7fb0f8029730, post=post at entry=0x7fb0f80297a0, xdata=xdata at entry=0x7fb0c065ee98) at shard.c:5326 #18 0x00007fb1153d467e in dht_writev_cbk (frame=0x7fb0f80021d8, cookie=, this=, op_ret=65536, op_errno=0, prebuf=0x7fb0f8029730, postbuf=0x7fb0f80297a0, xdata=0x7fb0c065ee98) at dht-inode-write.c:119 #19 0x00007fb115630b32 in afr_writev_unwind (frame=frame at entry=0x7fb0f8004888, this=this at entry=0x7fb11000fff0) at afr-inode-write.c:246 #20 0x00007fb11563105e in afr_writev_wind_cbk (frame=0x7fb0f800bd08, cookie=, this=0x7fb11000fff0, op_ret=, op_errno=, prebuf=, postbuf=0x7fb10dae7990, xdata=0x7fb0c065ee98) at afr-inode-write.c:406 #21 0x00007fb1158a7ffa in client3_3_writev_cbk (req=, iov=, count=, myframe=0x7fb0f802d148) at client-rpc-fops.c:838 #22 0x00007fb123198b30 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7fb11004a940, pollin=pollin at entry=0x7fb10bb95520) at rpc-clnt.c:778 #23 0x00007fb123198ed3 in rpc_clnt_notify (trans=, mydata=0x7fb11004a970, event=, data=0x7fb10bb95520) at rpc-clnt.c:971 #24 0x00007fb123194c33 in rpc_transport_notify (this=this at entry=0x7fb11004ac90, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7fb10bb95520) at rpc-transport.c:552 #25 0x00007fb117d89576 in socket_event_poll_in (this=this at entry=0x7fb11004ac90, notify_handled=) at socket.c:2322 #26 0x00007fb117d8bb1c in socket_event_handler (fd=11, idx=4, gen=1, data=0x7fb11004ac90, poll_in=1, poll_out=0, poll_err=0) at socket.c:2474 #27 0x00007fb12342ee84 in event_dispatch_epoll_handler (event=0x7fb10dae7e80, event_pool=0x56476beb1ec0) at event-epoll.c:583 #28 event_dispatch_epoll_worker (data=0x56476bf0b1d0) at event-epoll.c:659 #29 0x00007fb12222fdd5 in start_thread (arg=0x7fb10dae8700) at pthread_create.c:307 #30 0x00007fb121af7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 --- Additional comment from SATHEESARAN on 2019-01-22 12:12:24 UTC --- Errors as seen in the fuse mount log: [2019-01-22 11:29:05.487058] E [shard.c:5935:shard_post_lookup_fsync_handler] (-->/usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x134c3) [0x7fb1151544c3] -->/usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x813c) [0x7fb11514913c] -->/usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x2083e) [0x7fb11516183e] ) 0-: Assertion failed: fsync_count > 0 pending frames: frame : type(0) op(0) frame : type(1) op(FSYNC) frame : type(1) op(FSYNC) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2019-01-22 11:29:05 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.12.2 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x9d)[0x7fb1233d0e8d] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fb1233daf54] /lib64/libc.so.6(+0x36280)[0x7fb121a30280] /lib64/libglusterfs.so.0(+0x3ba7a)[0x7fb1233e0a7a] /lib64/libglusterfs.so.0(inode_ref+0x21)[0x7fb1233e0b81] /lib64/libglusterfs.so.0(+0x50d2b)[0x7fb1233f5d2b] /lib64/libglusterfs.so.0(+0x51f4a)[0x7fb1233f6f4a] /lib64/libglusterfs.so.0(fd_anonymous+0x2d)[0x7fb1233f729d] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x20534)[0x7fb115161534] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x813c)[0x7fb11514913c] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x134c3)[0x7fb1151544c3] /usr/lib64/glusterfs/3.12.2/xlator/performance/write-behind.so(+0x3189)[0x7fb114f30189] /lib64/libglusterfs.so.0(call_resume_keep_stub+0x75)[0x7fb1233f5b15] /usr/lib64/glusterfs/3.12.2/xlator/performance/write-behind.so(+0x8a69)[0x7fb114f35a69] /usr/lib64/glusterfs/3.12.2/xlator/performance/write-behind.so(+0x8b7b)[0x7fb114f35b7b] /usr/lib64/glusterfs/3.12.2/xlator/performance/write-behind.so(+0x8c57)[0x7fb114f35c57] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x15840)[0x7fb115156840] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x15bc0)[0x7fb115156bc0] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x6cc0)[0x7fb115147cc0] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x1d811)[0x7fb11515e811] /usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x6967e)[0x7fb1153d467e] /usr/lib64/glusterfs/3.12.2/xlator/cluster/replicate.so(+0x22b32)[0x7fb115630b32] /usr/lib64/glusterfs/3.12.2/xlator/cluster/replicate.so(+0x2305e)[0x7fb11563105e] /usr/lib64/glusterfs/3.12.2/xlator/protocol/client.so(+0x18ffa)[0x7fb1158a7ffa] /lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7fb123198b30] /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a3)[0x7fb123198ed3] /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fb123194c33] /usr/lib64/glusterfs/3.12.2/rpc-transport/socket.so(+0x7576)[0x7fb117d89576] /usr/lib64/glusterfs/3.12.2/rpc-transport/socket.so(+0x9b1c)[0x7fb117d8bb1c] /lib64/libglusterfs.so.0(+0x89e84)[0x7fb12342ee84] /lib64/libpthread.so.0(+0x7dd5)[0x7fb12222fdd5] /lib64/libc.so.6(clone+0x6d)[0x7fb121af7ead] --------- --- Additional comment from Krutika Dhananjay on 2019-01-23 16:37:55 UTC --- So this seems to happen when the vm image creation fills the lru list whose size is 16K maintained by shard translator. Anyway the bug isn't related to the shard deletion patch. Crash is seen during fsync post write. And it seems to be caused by this Dalton fix @ https://bugzilla.redhat.com/show_bug.cgi?id=1583462 at the moment. Managed to recreate the bug with a smaller data set: 1. Just set shard-lru-limit option to say 100. (In fact the very reason why I added this option in https://bugzilla.redhat.com/show_bug.cgi?id=1603118#c13 was for enabling easier testing of this code path since it is a bit complicated and prone to races). 2. And execute qemu-img command from "Description" above to create a smaller file (say 10G) and there is a crash. Made some progress in terms of identifying that this bug is hit when the shard is part of the fsync list but NOT the lru list. This defies a certain assumption made at the time of writing that patch that if a shard is part of fsync list, it is guaranteed to be part of lru list as well which is where it gets ref'd. But that is not what happened. So no lru list presence => no ref. Inode is destroyed in-memory at some point with its "ctx" object still in fsync list. When this list is traversed during fsync, the client crashes as its memory is freed by now. What's left is to identify the specific case where this might have happened. Calling it a day for now. Will resume investigation tomorrow. -Krutika --- Additional comment from Krutika Dhananjay on 2019-01-24 06:33:26 UTC --- Found the issue. When an inode write fop (in this case FALLOCATE) needs to write to more shards than the lru list can hold, some shards that are to participate in this inode write fop get evicted from lru list and later get added to fsync list (because these shards are modified and need to be fsync'd when application sends one) but without holding a ref. Because of this, at the end of the fop, the inode and its ctx get unref'd and destroyed while still being part of fsync list. When the application does send an fsync later, the fsync list is traversed and the (now) bad address accessed leading to a crash. The fix is simple - just ref the shard at the time of adding to fsync list as well and unref when fsync'd. Need to make this change and perform some tests (to be sure new ref doesn't cause mem-leaks in case it is not unref'd wherever it needs to be) after which I'll send out the patch upstream. -Krutika Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1668304 [Bug 1668304] [RHHI-V] Fuse mount crashed while creating the preallocated image https://bugzilla.redhat.com/show_bug.cgi?id=1668309 [Bug 1668309] Fuse mount crashed while creating the preallocated image with size > 1TB -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 24 09:40:56 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 09:40:56 +0000 Subject: [Bugs] [Bug 1669077] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669077 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |kdhananj at redhat.com -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 24 10:21:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 10:21:59 +0000 Subject: [Bugs] [Bug 1669077] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669077 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 12:17:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 12:17:28 +0000 Subject: [Bugs] [Bug 1653359] Self-heal:Improve heal performance In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1653359 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/21720 (afr/self-heal:Fix wrong type checking) merged (#9) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 24 12:22:58 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 12:22:58 +0000 Subject: [Bugs] [Bug 1318493] Introduce ctime-xlator to return correct (client-side set) ctime In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1318493 --- Comment #14 from Rubin Simons --- Hi Mohammed: Are you sure this is only since GlusterFS 5.x? I ask because of this: https://github.com/amarts/glusterfs/commit/0d1dbf034a4a75ff0ebd74b7218193c00b506247 Which seems to mention GD_OP_VERSION_4_1_0 for the (then) new ctime feature, of which the description is: "When this option is enabled, time attributes (ctime,mtime,atime) are stored in xattr to keep it consistent across replica and distribute set. The time attributes stored at the backend are not considered." -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 12:29:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 12:29:40 +0000 Subject: [Bugs] [Bug 1318493] Introduce ctime-xlator to return correct (client-side set) ctime In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1318493 --- Comment #15 from Amar Tumballi --- Hi Rubin, We did get the feature in 4_1_0, but it was not enabled by default. In 5_0 version, it was enabled by default. In glusterfs-5.0 version, at least we have made more testing etc (http://hrkscribbles.blogspot.com/2018/11/elastic-search-on-gluster.html) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 12:34:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 12:34:07 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1669136 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1669136 [Bug 1669136] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 12:35:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 12:35:48 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Sunil Kumar Acharya changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1669138 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1669138 [Bug 1669138] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 15:18:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 15:18:46 +0000 Subject: [Bugs] [Bug 1665358] allow regression to not run tests with nfs, if nfs is disabled. In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665358 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-24 15:18:46 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22010 (tests: run nfs tests only if --enable-gnfs is provided) merged (#7) on master by Niels de Vos -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 16:17:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 16:17:11 +0000 Subject: [Bugs] [Bug 1318493] Introduce ctime-xlator to return correct (client-side set) ctime In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1318493 --- Comment #16 from Rubin Simons --- Hi Amar and Mohammed; I'm trying to see how the glusterfs version spoken about above, i.e, version 4.1 and 5.x relates to Red Hat's GlusterFS package version. For example, the latest supported package version for the Red Hat GlusterFS product is 3.12.2-25 (that's for "Red Hat Cluster Storage version 3.4 Batch 1 Update", source: https://access.redhat.com/articles/2356261) Am I right in understanding that Red Hat's supported GlusterFS product is nowhere near version 4.1 (I hope not)? -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 24 17:42:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 17:42:45 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22094 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 24 17:42:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 17:42:46 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #523 from Worker Ant --- REVIEW: https://review.gluster.org/22094 (core: make gf_thread_create() easier to use) posted (#1) for review on master by Xavi Hernandez -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 24 18:15:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 24 Jan 2019 18:15:43 +0000 Subject: [Bugs] [Bug 1669077] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669077 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22091 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 25 04:54:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 25 Jan 2019 04:54:28 +0000 Subject: [Bugs] [Bug 1669382] New: [ovirt-gluster] Fuse mount crashed while creating the preallocated image Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669382 Bug ID: 1669382 Summary: [ovirt-gluster] Fuse mount crashed while creating the preallocated image Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: sharding Severity: high Assignee: bugs at gluster.org Reporter: kdhananj at redhat.com QA Contact: bugs at gluster.org CC: bugs at gluster.org Depends On: 1668304, 1669077 Blocks: 1668309 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1669077 +++ +++ This bug was initially created as a clone of Bug #1668304 +++ Description of problem: ------------------------ Fuse mount crashed while creating the preallocated image of size 1TB on a arbitrated replicate volume Version-Release number of selected component (if applicable): -------------------------------------------------------------- RHGS 3.4.3 nightly ( glusterfs-3.12.2-38.el7rhgs ) RHV 4.2.8 How reproducible: ----------------- 1 out of 2 times Steps to Reproduce: ------------------- 1. Create a arbitrated replicate volume 2. Fuse mount it 3. Create preallocated image of size 1TB # qemu-img create -f qcow2 -o preallocation=falloc /mnt/test1/vm1.img 1072G Actual results: --------------- Fuse mount crashed with segfault Expected results: ----------------- Preallocated image should be created successfully --- Additional comment from SATHEESARAN on 2019-01-22 12:10:19 UTC --- 1. Cluster info ---------------- There are 3 nodes in the cluster 2. Volume info --------------- [root@ ~]# gluster volume info data Volume Name: data Type: Replicate Volume ID: 7eb49e90-e2b6-4f8f-856e-7108212dbb72 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: server1:/gluster_bricks/data/data Brick2: server2:/gluster_bricks/data/data Brick3: server3:/gluster_bricks/data/data (arbiter) Options Reconfigured: performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off cluster.choose-local: off client.event-threads: 4 server.event-threads: 4 storage.owner-uid: 36 storage.owner-gid: 36 network.ping-timeout: 30 performance.strict-o-direct: on cluster.granular-entry-heal: enable cluster.enable-shared-storage: enable --- Additional comment from SATHEESARAN on 2019-01-22 12:11:03 UTC --- Backtrace ----------- [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/data /mnt/test1'. Program terminated with signal 11, Segmentation fault. #0 0x00007fb1233e0a7a in __inode_get_xl_index (xlator=0x7fb1100143b0, inode=0x7fb104026258) at inode.c:455 455 if ((inode->_ctx[xlator->xl_id].xl_key != NULL) && (gdb) bt #0 0x00007fb1233e0a7a in __inode_get_xl_index (xlator=0x7fb1100143b0, inode=0x7fb104026258) at inode.c:455 #1 __inode_ref (inode=inode at entry=0x7fb104026258) at inode.c:537 #2 0x00007fb1233e0b81 in inode_ref (inode=inode at entry=0x7fb104026258) at inode.c:581 #3 0x00007fb1233f5d2b in __fd_create (inode=inode at entry=0x7fb104026258, pid=pid at entry=0) at fd.c:633 #4 0x00007fb1233f6f4a in __fd_anonymous (inode=inode at entry=0x7fb104026258, flags=flags at entry=2) at fd.c:779 #5 0x00007fb1233f729d in fd_anonymous (inode=0x7fb104026258) at fd.c:803 #6 0x00007fb115161534 in shard_post_lookup_fsync_handler (frame=0x7fb0c05eb178, this=0x7fb1100143b0) at shard.c:5936 #7 0x00007fb11514913c in shard_lookup_base_file (frame=frame at entry=0x7fb0c05eb178, this=this at entry=0x7fb1100143b0, loc=loc at entry=0x7fb10800a158, handler=handler at entry=0x7fb115161030 ) at shard.c:1746 #8 0x00007fb1151544c3 in shard_fsync (frame=0x7fb0c05eb178, this=0x7fb1100143b0, fd=0x7fb0f800eb78, datasync=1, xdata=0x0) at shard.c:6015 #9 0x00007fb114f30189 in wb_fsync_helper (frame=0x7fb0f80022e8, this=0x7fb1100159d0, fd=0x7fb0f800eb78, datasync=1, xdata=0x0) at write-behind.c:1974 #10 0x00007fb1233f5b15 in call_resume_keep_stub (stub=0x7fb0f80250f8) at call-stub.c:2582 #11 0x00007fb114f35a69 in wb_do_winds (wb_inode=wb_inode at entry=0x7fb0f800dd70, tasks=tasks at entry=0x7fb10dae7510) at write-behind.c:1672 #12 0x00007fb114f35b7b in wb_process_queue (wb_inode=wb_inode at entry=0x7fb0f800dd70) at write-behind.c:1709 #13 0x00007fb114f35c57 in wb_fulfill_cbk (frame=0x7fb0f8010a58, cookie=, this=, op_ret=, op_errno=, prebuf=, postbuf=0x7fb0f8003670, xdata=0x7fb0c065ee98) at write-behind.c:1054 #14 0x00007fb115156840 in shard_common_inode_write_success_unwind (fop=, frame=0x7fb0f80019b8, op_ret=65536) at shard.c:903 #15 0x00007fb115156bc0 in shard_common_inode_write_post_update_size_handler (frame=, this=) at shard.c:5214 #16 0x00007fb115147cc0 in shard_update_file_size (frame=frame at entry=0x7fb0f80019b8, this=this at entry=0x7fb1100143b0, fd=0x7fb0f800eb78, loc=loc at entry=0x0, handler=handler at entry=0x7fb115156ba0 ) at shard.c:1201 #17 0x00007fb11515e811 in shard_common_inode_write_do_cbk (frame=frame at entry=0x7fb0f80019b8, cookie=0x7fb0f800eb78, this=0x7fb1100143b0, op_ret=op_ret at entry=65536, op_errno=op_errno at entry=0, pre=pre at entry=0x7fb0f8029730, post=post at entry=0x7fb0f80297a0, xdata=xdata at entry=0x7fb0c065ee98) at shard.c:5326 #18 0x00007fb1153d467e in dht_writev_cbk (frame=0x7fb0f80021d8, cookie=, this=, op_ret=65536, op_errno=0, prebuf=0x7fb0f8029730, postbuf=0x7fb0f80297a0, xdata=0x7fb0c065ee98) at dht-inode-write.c:119 #19 0x00007fb115630b32 in afr_writev_unwind (frame=frame at entry=0x7fb0f8004888, this=this at entry=0x7fb11000fff0) at afr-inode-write.c:246 #20 0x00007fb11563105e in afr_writev_wind_cbk (frame=0x7fb0f800bd08, cookie=, this=0x7fb11000fff0, op_ret=, op_errno=, prebuf=, postbuf=0x7fb10dae7990, xdata=0x7fb0c065ee98) at afr-inode-write.c:406 #21 0x00007fb1158a7ffa in client3_3_writev_cbk (req=, iov=, count=, myframe=0x7fb0f802d148) at client-rpc-fops.c:838 #22 0x00007fb123198b30 in rpc_clnt_handle_reply (clnt=clnt at entry=0x7fb11004a940, pollin=pollin at entry=0x7fb10bb95520) at rpc-clnt.c:778 #23 0x00007fb123198ed3 in rpc_clnt_notify (trans=, mydata=0x7fb11004a970, event=, data=0x7fb10bb95520) at rpc-clnt.c:971 #24 0x00007fb123194c33 in rpc_transport_notify (this=this at entry=0x7fb11004ac90, event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7fb10bb95520) at rpc-transport.c:552 #25 0x00007fb117d89576 in socket_event_poll_in (this=this at entry=0x7fb11004ac90, notify_handled=) at socket.c:2322 #26 0x00007fb117d8bb1c in socket_event_handler (fd=11, idx=4, gen=1, data=0x7fb11004ac90, poll_in=1, poll_out=0, poll_err=0) at socket.c:2474 #27 0x00007fb12342ee84 in event_dispatch_epoll_handler (event=0x7fb10dae7e80, event_pool=0x56476beb1ec0) at event-epoll.c:583 #28 event_dispatch_epoll_worker (data=0x56476bf0b1d0) at event-epoll.c:659 #29 0x00007fb12222fdd5 in start_thread (arg=0x7fb10dae8700) at pthread_create.c:307 #30 0x00007fb121af7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 --- Additional comment from SATHEESARAN on 2019-01-22 12:12:24 UTC --- Errors as seen in the fuse mount log: [2019-01-22 11:29:05.487058] E [shard.c:5935:shard_post_lookup_fsync_handler] (-->/usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x134c3) [0x7fb1151544c3] -->/usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x813c) [0x7fb11514913c] -->/usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x2083e) [0x7fb11516183e] ) 0-: Assertion failed: fsync_count > 0 pending frames: frame : type(0) op(0) frame : type(1) op(FSYNC) frame : type(1) op(FSYNC) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2019-01-22 11:29:05 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.12.2 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x9d)[0x7fb1233d0e8d] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fb1233daf54] /lib64/libc.so.6(+0x36280)[0x7fb121a30280] /lib64/libglusterfs.so.0(+0x3ba7a)[0x7fb1233e0a7a] /lib64/libglusterfs.so.0(inode_ref+0x21)[0x7fb1233e0b81] /lib64/libglusterfs.so.0(+0x50d2b)[0x7fb1233f5d2b] /lib64/libglusterfs.so.0(+0x51f4a)[0x7fb1233f6f4a] /lib64/libglusterfs.so.0(fd_anonymous+0x2d)[0x7fb1233f729d] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x20534)[0x7fb115161534] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x813c)[0x7fb11514913c] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x134c3)[0x7fb1151544c3] /usr/lib64/glusterfs/3.12.2/xlator/performance/write-behind.so(+0x3189)[0x7fb114f30189] /lib64/libglusterfs.so.0(call_resume_keep_stub+0x75)[0x7fb1233f5b15] /usr/lib64/glusterfs/3.12.2/xlator/performance/write-behind.so(+0x8a69)[0x7fb114f35a69] /usr/lib64/glusterfs/3.12.2/xlator/performance/write-behind.so(+0x8b7b)[0x7fb114f35b7b] /usr/lib64/glusterfs/3.12.2/xlator/performance/write-behind.so(+0x8c57)[0x7fb114f35c57] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x15840)[0x7fb115156840] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x15bc0)[0x7fb115156bc0] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x6cc0)[0x7fb115147cc0] /usr/lib64/glusterfs/3.12.2/xlator/features/shard.so(+0x1d811)[0x7fb11515e811] /usr/lib64/glusterfs/3.12.2/xlator/cluster/distribute.so(+0x6967e)[0x7fb1153d467e] /usr/lib64/glusterfs/3.12.2/xlator/cluster/replicate.so(+0x22b32)[0x7fb115630b32] /usr/lib64/glusterfs/3.12.2/xlator/cluster/replicate.so(+0x2305e)[0x7fb11563105e] /usr/lib64/glusterfs/3.12.2/xlator/protocol/client.so(+0x18ffa)[0x7fb1158a7ffa] /lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7fb123198b30] /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a3)[0x7fb123198ed3] /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fb123194c33] /usr/lib64/glusterfs/3.12.2/rpc-transport/socket.so(+0x7576)[0x7fb117d89576] /usr/lib64/glusterfs/3.12.2/rpc-transport/socket.so(+0x9b1c)[0x7fb117d8bb1c] /lib64/libglusterfs.so.0(+0x89e84)[0x7fb12342ee84] /lib64/libpthread.so.0(+0x7dd5)[0x7fb12222fdd5] /lib64/libc.so.6(clone+0x6d)[0x7fb121af7ead] --------- --- Additional comment from Krutika Dhananjay on 2019-01-23 16:37:55 UTC --- So this seems to happen when the vm image creation fills the lru list whose size is 16K maintained by shard translator. Anyway the bug isn't related to the shard deletion patch. Crash is seen during fsync post write. And it seems to be caused by this Dalton fix @ https://bugzilla.redhat.com/show_bug.cgi?id=1583462 at the moment. Managed to recreate the bug with a smaller data set: 1. Just set shard-lru-limit option to say 100. (In fact the very reason why I added this option in https://bugzilla.redhat.com/show_bug.cgi?id=1603118#c13 was for enabling easier testing of this code path since it is a bit complicated and prone to races). 2. And execute qemu-img command from "Description" above to create a smaller file (say 10G) and there is a crash. Made some progress in terms of identifying that this bug is hit when the shard is part of the fsync list but NOT the lru list. This defies a certain assumption made at the time of writing that patch that if a shard is part of fsync list, it is guaranteed to be part of lru list as well which is where it gets ref'd. But that is not what happened. So no lru list presence => no ref. Inode is destroyed in-memory at some point with its "ctx" object still in fsync list. When this list is traversed during fsync, the client crashes as its memory is freed by now. What's left is to identify the specific case where this might have happened. Calling it a day for now. Will resume investigation tomorrow. -Krutika --- Additional comment from Krutika Dhananjay on 2019-01-24 06:33:26 UTC --- Found the issue. When an inode write fop (in this case FALLOCATE) needs to write to more shards than the lru list can hold, some shards that are to participate in this inode write fop get evicted from lru list and later get added to fsync list (because these shards are modified and need to be fsync'd when application sends one) but without holding a ref. Because of this, at the end of the fop, the inode and its ctx get unref'd and destroyed while still being part of fsync list. When the application does send an fsync later, the fsync list is traversed and the (now) bad address accessed leading to a crash. The fix is simple - just ref the shard at the time of adding to fsync list as well and unref when fsync'd. Need to make this change and perform some tests (to be sure new ref doesn't cause mem-leaks in case it is not unref'd wherever it needs to be) after which I'll send out the patch upstream. -Krutika --- Additional comment from Worker Ant on 2019-01-24 18:15:43 UTC --- REVIEW: https://review.gluster.org/22091 (features/shard: Ref shard inode while adding to fsync list) merged (#4) on master by Xavi Hernandez Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1668304 [Bug 1668304] [RHHI-V] Fuse mount crashed while creating the preallocated image https://bugzilla.redhat.com/show_bug.cgi?id=1668309 [Bug 1668309] Fuse mount crashed while creating the preallocated image with size > 1TB https://bugzilla.redhat.com/show_bug.cgi?id=1669077 [Bug 1669077] [ovirt-gluster] Fuse mount crashed while creating the preallocated image -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 25 04:54:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 25 Jan 2019 04:54:28 +0000 Subject: [Bugs] [Bug 1669077] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669077 Krutika Dhananjay changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1669382 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1669382 [Bug 1669382] [ovirt-gluster] Fuse mount crashed while creating the preallocated image -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. From bugzilla at redhat.com Fri Jan 25 04:55:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 25 Jan 2019 04:55:49 +0000 Subject: [Bugs] [Bug 1669382] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669382 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22095 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Fri Jan 25 04:55:50 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Fri, 25 Jan 2019 04:55:50 +0000 Subject: [Bugs] [Bug 1669382] [ovirt-gluster] Fuse mount crashed while creating the preallocated image In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669382 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22095 (features/shard: Ref shard inode while adding to fsync list) posted (#1) for review on release-5 by Krutika Dhananjay -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Sun Jan 27 15:42:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Sun, 27 Jan 2019 15:42:59 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 Emerson Gomes changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emerson.gomes at gmail.com --- Comment #9 from Emerson Gomes --- This error is still reproduceable in 5.3 when upgrading from a 3.x volume. I had to recreate volume from scratch in 5.3 and copy data back in order to avoid it. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 28 03:25:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 03:25:40 +0000 Subject: [Bugs] [Bug 1668245] gluster(8) - Man page - create gluster example session In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668245 Sanju changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED CC| |srakonde at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 28 04:14:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 04:14:05 +0000 Subject: [Bugs] [Bug 1667168] Thin Arbiter documentation refers commands don't exist "glustercli' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667168 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Triaged CC| |ravishankar at redhat.com Assignee|bugs at gluster.org |ravishankar at redhat.com --- Comment #1 from Ravishankar N --- "glustercli" is the CLI for glusterd2 (gd2). Thin-arbiter is currently a gd2 based solution only. If you want to try it out, you would need to do so using the GD2 command line interface (Install from https://github.com/gluster/glusterd2/). We're still ironing out a few bugs in thin-arbiter though, for example, https://review.gluster.org/#/c/glusterfs/+/21933/ which needs to be merged. We are thinking of adding GD1 support too, so I'll keep this bug open to submit the patch when we get to it. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 04:53:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 04:53:08 +0000 Subject: [Bugs] [Bug 1467614] Gluster read/write performance improvements on NVMe backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1467614 --- Comment #76 from Worker Ant --- REVIEW: https://review.gluster.org/21999 (socket: fix issue on concurrent handle of a socket) merged (#4) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 28 05:56:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 05:56:36 +0000 Subject: [Bugs] [Bug 1669937] New: Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669937 Bug ID: 1669937 Summary: Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point Product: GlusterFS Version: 4.1 Status: NEW Whiteboard: dht-rebalance Component: distribute Keywords: ZStream Severity: medium Priority: high Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com CC: bugs at gluster.org Depends On: 994421, 1286099 Target Milestone: --- Classification: Community Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=994421 [Bug 994421] Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point https://bugzilla.redhat.com/show_bug.cgi?id=1286099 [Bug 1286099] Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 06:02:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 06:02:18 +0000 Subject: [Bugs] [Bug 1669937] Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669937 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Comment #0 is|1 |0 private| | Keywords|ZStream | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 06:52:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 06:52:30 +0000 Subject: [Bugs] [Bug 1669937] Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669937 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22103 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 06:52:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 06:52:32 +0000 Subject: [Bugs] [Bug 1669937] Rebalance : While rebalance is in progress , SGID and sticky bit which is set on the files while file migration is in progress is seen on the mount point In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1669937 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22103 (cluster/dht: Remove internal permission bits) posted (#1) for review on master by N Balachandran -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 07:37:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 07:37:55 +0000 Subject: [Bugs] [Bug 1665145] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665145 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Triaged Status|NEW |ASSIGNED CC| |ravishankar at redhat.com Assignee|bugs at gluster.org |ravishankar at redhat.com --- Comment #2 from Ravishankar N --- > $ echo foo > /mnt/gluster/a; ls -l /mnt/gluster/a > bash: /mnt/gluster/a: Input/output error > -rw-r--r--. 1 root root 0 10. Jan 15:53 /mnt/gluster/a It is the truncate that is failing. This seems to be a regression introduced by commit c9bde3021202f1d5c5a2d19ac05a510fc1f788ac (performance/readdir-ahead: keep stats of cached dentries in sync with modifications). When you set .consistent-metadata on, AFR zeroes out the ctime (commit 4c4624c9bad2edf27128cb122c64f15d7d63bbc8). When ctime is zero, readdir-ahead xlator seems to zero out the entire stat buf including the ia_type and ia_gfid. Let me attempt a fix. As a workaround, if you want to enable consistent-metadata, you can disable 'performance.readdir-ahead'. That should solve the issue interim. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 12:21:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 12:21:05 +0000 Subject: [Bugs] [Bug 1664590] Gluster peer probe doesn't work for IPv6 In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664590 Sahina Bose changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(sabose at redhat.com | |) | -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Mon Jan 28 13:12:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 13:12:32 +0000 Subject: [Bugs] [Bug 1670031] New: performance regression seen with smallfile workload tests Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Bug ID: 1670031 Summary: performance regression seen with smallfile workload tests Product: GlusterFS Version: mainline Status: ASSIGNED Component: core Keywords: Performance Severity: urgent Priority: urgent Assignee: bugs at gluster.org Reporter: atumball at redhat.com CC: bugs at gluster.org, jahernan at redhat.com, pkarampu at redhat.com, rgowdapp at redhat.com, srangana at redhat.com Target Milestone: --- Classification: Community Description of problem: When glusterfs-master performance is compared with 3.12.15 release (ie, last of 3.12 series), we are finding a lot of regression on the master branch. Version-Release number of selected component (if applicable): master How reproducible: 100% Steps to Reproduce: 1. run the gbench tests. 2. 3. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 13:14:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 13:14:32 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22107 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 13:14:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 13:14:33 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22107 (features/sdfs: disable by default) posted (#1) for review on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Mon Jan 28 18:04:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Mon, 28 Jan 2019 18:04:14 +0000 Subject: [Bugs] [Bug 1670155] New: Tiered volume files disappear when a hot brick is failed/restored until the tier detached. Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670155 Bug ID: 1670155 Summary: Tiered volume files disappear when a hot brick is failed/restored until the tier detached. Product: GlusterFS Version: 5 Status: NEW Component: tiering Severity: medium Assignee: bugs at gluster.org Reporter: jbyers at stonefly.com QA Contact: bugs at gluster.org CC: bugs at gluster.org Target Milestone: --- Classification: Community Tiered volume files disappear when a hot brick is failed/restored until the tier detached. Files residing in the hot tier of distributed hot-tiered volume disappear when a hot tier brick is failed/restored. The missing files do not reappear until the tier is detached. The hot tier resident files are expected to disappear when the brick it is on fails. When the brick is restored, they should come back, but often they do not. This occurs when they stop showing up in mount point 'ls -lsh' with results '??????????'. In some cases, doing an ls or open on the full path name of the file will bring it back, other times it will not. In those cases the hot tier needs to be detached to get them back. The problem occurs using either NFS or CIFS/Fuse mounts. The problem was first seen with the cold tier being a Disperse volume, but also occurs with a Distributed cold tier volume. The problem was first seen on GlusterFS 3.12.14, and has been reproduced on GlusterFS 5.2. Note that this first happened on a production system, and was then reproduced in a lab environment. Test plan below. # glusterd -V glusterfs 5.2 ##### Create the brick dirs and cold tier volume. # mkdir /exports/cold-brick-1/dir # mkdir /exports/cold-brick-2/dir # mkdir /exports/cold-brick-3/dir # mkdir /exports/hot-brick-1/dir # mkdir /exports/hot-brick-2/dir # mkdir /exports/hot-brick-3/dir # gluster volume create tiered-vol transport tcp 10.0.0.5:/exports/cold-brick-1/dir volume create: tiered-vol: success: please start the volume to access data # gluster volume start tiered-vol volume start: tiered-vol: success ##### Expand the cold tier volume. # gluster volume add-brick tiered-vol 10.0.0.5:/exports/cold-brick-2/dir/ volume add-brick: success # gluster volume add-brick tiered-vol 10.0.0.5:/exports/cold-brick-3/dir/ volume add-brick: success ##### Mount the volume. # gluster volume set tiered-vol nfs.disable off volume set: success # mount 127.0.0.1:tiered-vol /mnt/tiered-vol/ ##### Create volumes on the volume, not tiered yet. # xfs_mkfile 1G /mnt/tiered-vol/file-1 # xfs_mkfile 1G /mnt/tiered-vol/file-2 # xfs_mkfile 1G /mnt/tiered-vol/file-3 # ls -lsh /mnt/tiered-vol/ total 3.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 # ls -lsh /exports/*brick*/dir/* 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-2/dir/file-3 1.1G -rw------- 2 root root 1.0G Jan 28 08:49 /exports/cold-brick-3/dir/file-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-3/dir/file-2 # gluster volume info tiered-vol Volume Name: tiered-vol Type: Distribute Volume ID: 0639e4e4-249d-485c-9995-90aa8be9c94e Status: Started Snapshot Count: 0 Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 10.0.0.5:/exports/cold-brick-1/dir Brick2: 10.0.0.5:/exports/cold-brick-2/dir Brick3: 10.0.0.5:/exports/cold-brick-3/dir Options Reconfigured: transport.address-family: inet nfs.disable: off # gluster volume status tiered-vol Status of volume: tiered-vol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.0.0.5:/exports/cold-brick-1/dir 62002 0 Y 120790 Brick 10.0.0.5:/exports/cold-brick-2/dir 62003 0 Y 120929 Brick 10.0.0.5:/exports/cold-brick-3/dir 62004 0 Y 120978 NFS Server on localhost 2049 0 Y 121103 Task Status of Volume tiered-vol ------------------------------------------------------------------------------ Task : Rebalance ID : 13a856c2-f511-475c-b2ff-f6e0190ade50 Status : completed ##### Kill one of the brick processes, and note that the files on that brick disappear. This is normal, and expected. # kill 120929 # ls -lsh /mnt/tiered-vol/ total 2.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 ##### Start the brick processes, and see that all files are back. # gluster volume start tiered-vol force volume start: tiered-vol: success # ls -lsh /mnt/tiered-vol/ total 3.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 ##### Attach the hot tier, and create new files that are stored there. # gluster volume tier tiered-vol attach 10.0.0.5:/exports/hot-brick-1/dir 10.0.0.5:/exports/hot-brick-2/dir 10.0.0.5:/exports/hot-brick-3/dir volume attach-tier: success # xfs_mkfile 1G /mnt/tiered-vol/file-hot-1 # xfs_mkfile 1G /mnt/tiered-vol/file-hot-2 # xfs_mkfile 1G /mnt/tiered-vol/file-hot-3 # ls -lsh /mnt/tiered-vol/ total 6.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:59 file-hot-3 # ls -lsh /exports/*brick*/dir/* 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-2/dir/file-3 0 ---------T 2 root root 0 Jan 28 08:57 /exports/cold-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:49 /exports/cold-brick-3/dir/file-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-3/dir/file-2 0 ---------T 2 root root 0 Jan 28 08:58 /exports/cold-brick-3/dir/file-hot-2 0 ---------T 2 root root 0 Jan 28 08:58 /exports/cold-brick-3/dir/file-hot-3 1.1G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-3/dir/file-hot-2 1.1G -rw------- 2 root root 1.0G Jan 28 08:59 /exports/hot-brick-3/dir/file-hot-3 # gluster volume status tiered-vol Status of volume: tiered-vol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.0.0.5:/exports/hot-brick-3/dir 62007 0 Y 127766 Brick 10.0.0.5:/exports/hot-brick-2/dir 62006 0 Y 127744 Brick 10.0.0.5:/exports/hot-brick-1/dir 62003 0 Y 127722 Cold Bricks: Brick 10.0.0.5:/exports/cold-brick-1/dir 62002 0 Y 120790 Brick 10.0.0.5:/exports/cold-brick-2/dir 62005 0 Y 123087 Brick 10.0.0.5:/exports/cold-brick-3/dir 62004 0 Y 120978 Tier Daemon on localhost N/A N/A Y 127804 NFS Server on localhost 2049 0 Y 127795 ##### Kill a brick process for the distributed hot tier volume. See that the files stored there cannot be accessed. This is normal and expected. This is a case where things worked as expected. # kill 127744 # ls -lsh /mnt/tiered-vol/ ls: cannot access /mnt/tiered-vol/file-hot-1: No such file or directory total 5.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 ? ?????????? ? ? ? ? ? file-hot-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:59 file-hot-3 # ls -lsh /mnt/tiered-vol/ ls: cannot access /mnt/tiered-vol/file-hot-1: No such file or directory total 5.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 ? ?????????? ? ? ? ? ? file-hot-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:59 file-hot-3 ##### Start the hot tier brick process, and note that all files are back. # gluster volume start tiered-vol force volume start: tiered-vol: success # ls -lsh /mnt/tiered-vol/ total 6.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:59 file-hot-3 # ls -lsh /exports/*brick*/dir/* 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-2/dir/file-3 0 ---------T 2 root root 0 Jan 28 08:57 /exports/cold-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:49 /exports/cold-brick-3/dir/file-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-3/dir/file-2 0 ---------T 2 root root 0 Jan 28 08:58 /exports/cold-brick-3/dir/file-hot-2 0 ---------T 2 root root 0 Jan 28 08:58 /exports/cold-brick-3/dir/file-hot-3 1.1G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-3/dir/file-hot-2 1.1G -rw------- 2 root root 1.0G Jan 28 08:59 /exports/hot-brick-3/dir/file-hot-3 # gluster volume status tiered-vol Status of volume: tiered-vol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.0.0.5:/exports/hot-brick-3/dir 62007 0 Y 127766 Brick 10.0.0.5:/exports/hot-brick-2/dir 62010 0 Y 130185 Brick 10.0.0.5:/exports/hot-brick-1/dir 62003 0 Y 127722 Cold Bricks: Brick 10.0.0.5:/exports/cold-brick-1/dir 62002 0 Y 120790 Brick 10.0.0.5:/exports/cold-brick-2/dir 62005 0 Y 123087 Brick 10.0.0.5:/exports/cold-brick-3/dir 62004 0 Y 120978 Tier Daemon on localhost N/A N/A Y 127804 NFS Server on localhost 2049 0 Y 130217 ##### Kill another brick process for the distributed hot tier volume. See that the files stored there cannot be accessed. The first 'ls' shows the missing files, but the second one does not. This time the files will *not* come back when the brick is restored. This is a problem. # kill 127766 # ls -lsh /mnt/tiered-vol/ ls: cannot access /mnt/tiered-vol/file-hot-2: No such file or directory ls: cannot access /mnt/tiered-vol/file-hot-3: No such file or directory total 4.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 ? ?????????? ? ? ? ? ? file-hot-2 ? ?????????? ? ? ? ? ? file-hot-3 # ls -lsh /mnt/tiered-vol/ total 4.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 ##### Restore the failed brick, but note that the files on it are still gone. They still exist on the bricks though. # gluster volume start tiered-vol force volume start: tiered-vol: success # ls -lsh /mnt/tiered-vol/ total 4.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 # ls -lsh /exports/*brick*/dir/* 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-2/dir/file-3 0 ---------T 2 root root 0 Jan 28 08:57 /exports/cold-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:49 /exports/cold-brick-3/dir/file-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-3/dir/file-2 1.1G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-3/dir/file-hot-2 1.1G -rw------- 2 root root 1.0G Jan 28 08:59 /exports/hot-brick-3/dir/file-hot-3 ##### Accessing the missing files by their full path sometimes brings them back, but not in this case. # ls -lsh /mnt/tiered-vol/file-hot-2 ls: cannot access /mnt/tiered-vol/file-hot-2: No such file or directory # file /mnt/tiered-vol/file-hot-2 /mnt/tiered-vol/file-hot-2: cannot open `/mnt/tiered-vol/file-hot-2' (No such file or directory) # ls -lsh /mnt/tiered-vol/ total 4.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 ##### Stopping and starting the volume does not help. # gluster volume stop tiered-vol Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: tiered-vol: success # gluster volume start tiered-vol volume start: tiered-vol: success # ls -lsh /mnt/tiered-vol/ total 4.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 # ls -lsh /exports/*brick*/dir/* 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-2/dir/file-3 0 ---------T 2 root root 0 Jan 28 08:57 /exports/cold-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:49 /exports/cold-brick-3/dir/file-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-3/dir/file-2 1.1G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-3/dir/file-hot-2 1.1G -rw------- 2 root root 1.0G Jan 28 08:59 /exports/hot-brick-3/dir/file-hot-3 ##### Detaching the hot tier does usually bring the missing files back. # gluster volume tier tiered-vol detach start volume detach tier start: success ID: cec68278-f0b9-4289-81ab-6f3a60246c3e # gluster volume tier tiered-vol detach status volume detach tier status: success Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 7 0 0 in progress 0:00:21 # ls -lsh /mnt/tiered-vol/ total 6.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:59 file-hot-3 # ls -lsh /exports/*brick*/dir/* 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-2/dir/file-3 1.0G ---------T 2 root root 1.0G Jan 28 09:09 /exports/cold-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:49 /exports/cold-brick-3/dir/file-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-3/dir/file-2 1.0G ---------T 2 root root 1.0G Jan 28 09:09 /exports/cold-brick-3/dir/file-hot-2 1.0G ---------T 2 root root 1.0G Jan 28 09:09 /exports/cold-brick-3/dir/file-hot-3 1.1G -rw---S--T 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-2/dir/file-hot-1 1.1G -rw---S--T 2 root root 1.0G Jan 28 08:58 /exports/hot-brick-3/dir/file-hot-2 1.1G -rw---S--T 2 root root 1.0G Jan 28 08:59 /exports/hot-brick-3/dir/file-hot-3 # gluster volume tier tiered-vol detach status volume detach tier status: success Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 3 3.0GB 7 0 0 completed 0:01:25 # gluster volume tier tiered-vol detach commit volume detach tier commit: success # ls -lsh /mnt/tiered-vol/ total 6.0G 1.0G -rw------- 1 root root 1.0G Jan 28 08:49 file-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:50 file-3 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-1 1.0G -rw------- 1 root root 1.0G Jan 28 08:58 file-hot-2 1.0G -rw------- 1 root root 1.0G Jan 28 08:59 file-hot-3 # ls -lsh /exports/*brick*/dir/* 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-2/dir/file-3 1.0G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/cold-brick-2/dir/file-hot-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:49 /exports/cold-brick-3/dir/file-1 1.1G -rw------- 2 root root 1.0G Jan 28 08:50 /exports/cold-brick-3/dir/file-2 1.0G -rw------- 2 root root 1.0G Jan 28 08:58 /exports/cold-brick-3/dir/file-hot-2 1.0G -rw------- 2 root root 1.0G Jan 28 08:59 /exports/cold-brick-3/dir/file-hot-3 EOM -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 03:11:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 03:11:36 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 Anjana changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |akrishna at redhat.com Flags| |needinfo?(gdeschner at redhat. | |com) --- Doc Text *updated* --- Previously, when gluster was assessed using SMB from a macOS client using the recommended vfs_glusterfs module, file handles were closed incorrectly. As a consequence, the samba service on the server would hang and the user would need to reconnect. With this fix, Samba correctly closes the file handles and the right layer when using the vfs_glusterfs module. --- Comment #41 from Anjana --- Updated the doc text. Kindly review it for technical accuracy. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 03:15:43 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 03:15:43 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22109 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 03:15:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 03:15:45 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #6 from Worker Ant --- REVIEW: https://review.gluster.org/22109 (mount/fuse: expose fuse-auto-invalidation as a mount option) posted (#1) for review on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 04:02:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:02:13 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22108 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 04:02:14 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:02:14 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |POST Resolution|NEXTRELEASE |--- Keywords| |Reopened --- Comment #3 from Worker Ant --- REVIEW: https://review.gluster.org/22108 (core: heketi-cli is throwing error \"target is busy\") posted (#2) for review on master by MOHIT AGRAWAL -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 04:15:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:15:10 +0000 Subject: [Bugs] [Bug 1670253] New: Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670253 Bug ID: 1670253 Summary: Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set Product: GlusterFS Version: mainline OS: Linux Status: NEW Component: core Keywords: Triaged Assignee: bugs at gluster.org Reporter: ravishankar at redhat.com CC: bugs at gluster.org, public at hansmi.ch, ravishankar at redhat.com, suchaudh at redhat.com, tobias.brunner at vshn.ch Depends On: 1665145 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1665145 +++ After upgrading a Gluster cluster from version 3.12.15 to 5.2 writes to volumes with "cluster.consistent-metadata" enabled fail with EIO. Reproduction steps follow. ## Package information Package source: http://mirror.centos.org/centos/7/storage/x86_64/gluster-5/ # gluster --version glusterfs 5.2 # lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 7.6 (Maipo) Release: 7.6 Codename: Maipo ## Cluster information Peers: * storage1 * storage2 * storage3 # gluster peer status Number of Peers: 2 Hostname: storage2 Uuid: 0acf152f-4cba-4757-bdab-113c34333eeb State: Peer in Cluster (Connected) Hostname: storage3 Uuid: c0302a36-149a-44ca-984f-8120a1fa360a State: Peer in Cluster (Connected) ## Reproduction steps # On each storage server mkdir /data/testpv1 # Create volume gluster volume create testpv1 replica 3 storage1:/data/testpv1/brick storage2:/data/testpv1/brick storage3:/data/testpv1/brick force gluster volume start testpv1 # Mount volume on a client mount -t glusterfs -o log-level=DEBUG storage1:/testpv1 /mnt/gluster # Writes succeed $ echo foo > /mnt/gluster/a; ls -l /mnt/gluster/a -rw-r--r--. 1 root root 4 10. Jan 15:53 /mnt/gluster/a # Enable "cluster.consistent-metadata" gluster volume set testpv1 cluster.consistent-metadata on # Writes fail with EIO after truncation $ echo foo > /mnt/gluster/a; ls -l /mnt/gluster/a bash: /mnt/gluster/a: Input/output error -rw-r--r--. 1 root root 0 10. Jan 15:53 /mnt/gluster/a # Disable "cluster.consistent-metadata" again gluster volume set testpv1 cluster.consistent-metadata off # Writes work again $ echo foo > /mnt/gluster/a; ls -l /mnt/gluster/a -rw-r--r--. 1 root root 4 10. Jan 15:54 /mnt/gluster/a # Debug logs from FUSE client [2019-01-10 14:56:30.725669] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-testpv1-dht: revalidate lookup of / returned with op_ret 0 [Structure needs cleaning] [2019-01-10 14:56:30.725747] D [MSGID: 0] [dht-common.c:1648:dht_revalidate_cbk] 0-testpv1-dht: internal xattr trusted.glusterfs.dht.mds is not present on path / gfid is 00000000-0000-0000-0000-000000000001 [2019-01-10 14:56:30.726483] D [write-behind.c:1764:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x9a0b)[0x7fcac6b20a0b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0xd288)[0x7fcac6b24288] (--> /lib64/libglusterfs.so.0(default_lookup+0xdd)[0x7fcad4ea5c1d] (--> /lib64/libglusterfs.so.0(default_lookup+0xdd)[0x7fcad4ea5c1d] ))))) 0-testpv1-write-behind: processing queues [2019-01-10 14:56:30.726519] D [MSGID: 0] [write-behind.c:1710:__wb_pick_winds] 0-testpv1-write-behind: (unique=24, fop=LOOKUP, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): picking the request for winding [2019-01-10 14:56:30.726572] D [MSGID: 0] [dht-common.c:3374:dht_do_revalidate] 0-testpv1-dht: path = /a. Calling revalidate lookup on testpv1-replicate-0 [2019-01-10 14:56:30.726939] D [write-behind.c:419:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x26b2)[0x7fcac6b196b2] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x504c)[0x7fcac6b1c04c] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x98c1)[0x7fcac6b208c1] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x99cb)[0x7fcac6b209cb] ))))) 0-testpv1-write-behind: (unique = 24, fop=LOOKUP, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): destroying request, removing from all queues [2019-01-10 14:56:30.727875] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-testpv1-dht: revalidate lookup of /a returned with op_ret 0 [2019-01-10 14:56:30.728405] D [write-behind.c:1764:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x9a0b)[0x7fcac6b20a0b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0xc268)[0x7fcac6b23268] (--> /usr/lib64/glusterfs/5.2/xlator/performance/read-ahead.so(+0x7331)[0x7fcac690e331] (--> /usr/lib64/glusterfs/5.2/xlator/performance/readdir-ahead.so(+0x44f2)[0x7fcac66fb4f2] ))))) 0-testpv1-write-behind: processing queues [2019-01-10 14:56:30.728442] D [MSGID: 0] [write-behind.c:1710:__wb_pick_winds] 0-testpv1-write-behind: (unique=27, fop=TRUNCATE, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): picking the request for winding [2019-01-10 14:56:30.728722] D [write-behind.c:419:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x26b2)[0x7fcac6b196b2] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x504c)[0x7fcac6b1c04c] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x98c1)[0x7fcac6b208c1] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x99cb)[0x7fcac6b209cb] ))))) 0-testpv1-write-behind: (unique = 27, fop=TRUNCATE, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): destroying request, removing from all queues [2019-01-10 14:56:30.729207] D [MSGID: 0] [afr-transaction.c:2070:afr_post_nonblocking_inodelk_cbk] 0-testpv1-replicate-0: Non blocking inodelks done. Proceeding to FOP [2019-01-10 14:56:30.729245] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729271] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729286] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729297] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729345] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729368] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729380] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729396] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729435] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729450] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729461] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.729472] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.730173] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-0: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.730289] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-2: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.730516] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-1: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.731446] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731483] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731499] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731513] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731582] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731609] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731624] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731637] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731680] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.dirty' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731710] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-2' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731726] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-1' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.731740] D [MSGID: 101016] [glusterfs3.h:746:dict_to_xdr] 0-dict: key 'trusted.afr.testpv1-client-0' would not be sent on wire in the future [Invalid argument] [2019-01-10 14:56:30.732161] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-0: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.732480] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-2: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.732548] D [MSGID: 0] [client-rpc-fops_v2.c:1553:client4_0_xattrop_cbk] 0-testpv1-client-1: resetting op_ret to 0 from 0 [2019-01-10 14:56:30.735749] D [write-behind.c:1764:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x9a0b)[0x7fcac6b20a0b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0xd288)[0x7fcac6b24288] (--> /lib64/libglusterfs.so.0(default_lookup+0xdd)[0x7fcad4ea5c1d] (--> /lib64/libglusterfs.so.0(default_lookup+0xdd)[0x7fcad4ea5c1d] ))))) 0-testpv1-write-behind: processing queues [2019-01-10 14:56:30.735787] D [MSGID: 0] [write-behind.c:1710:__wb_pick_winds] 0-testpv1-write-behind: (unique=29, fop=LOOKUP, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): picking the request for winding [2019-01-10 14:56:30.735827] D [MSGID: 0] [dht-common.c:3374:dht_do_revalidate] 0-testpv1-dht: path = /a. Calling revalidate lookup on testpv1-replicate-0 [2019-01-10 14:56:30.736354] D [write-behind.c:419:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fcad4e13f1b] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x26b2)[0x7fcac6b196b2] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x504c)[0x7fcac6b1c04c] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x98c1)[0x7fcac6b208c1] (--> /usr/lib64/glusterfs/5.2/xlator/performance/write-behind.so(+0x99cb)[0x7fcac6b209cb] ))))) 0-testpv1-write-behind: (unique = 29, fop=LOOKUP, gfid=60fcb860-2470-45b4-9060-853979a8bc4d, gen=0): destroying request, removing from all queues [2019-01-10 14:56:30.738253] D [MSGID: 0] [dht-common.c:1559:dht_revalidate_cbk] 0-testpv1-dht: revalidate lookup of /a returned with op_ret 0 [2019-01-10 14:56:30.738311] D [fuse-bridge.c:53:fuse_invalidate] 0-fuse: Invalidate inode id 140508580080760. [2019-01-10 14:56:30.738390] E [fuse-bridge.c:219:check_and_dump_fuse_W] 0-glusterfs-fuse: writing to fuse device failed: No such file or directory Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1665145 [Bug 1665145] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 04:15:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:15:10 +0000 Subject: [Bugs] [Bug 1665145] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1665145 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1670253 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670253 [Bug 1670253] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 04:15:28 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:15:28 +0000 Subject: [Bugs] [Bug 1670253] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670253 Ravishankar N changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|bugs at gluster.org |ravishankar at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 04:20:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:20:27 +0000 Subject: [Bugs] [Bug 1670253] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670253 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22110 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 04:20:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:20:27 +0000 Subject: [Bugs] [Bug 1670253] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670253 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22110 (readdir-ahead: do not zero-out iatt in fop cbk) posted (#1) for review on master by Ravishankar N -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 04:46:27 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:46:27 +0000 Subject: [Bugs] [Bug 1670259] New: New GFID file recreated in a replica set after a GFID mismatch resolution Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670259 Bug ID: 1670259 Summary: New GFID file recreated in a replica set after a GFID mismatch resolution Product: GlusterFS Version: mainline Status: NEW Component: replicate Keywords: ZStream Severity: high Priority: high Assignee: bugs at gluster.org Reporter: nbalacha at redhat.com Depends On: 1661258 Blocks: 1662225 Target Milestone: --- Group: redhat Classification: Community -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 04:47:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:47:08 +0000 Subject: [Bugs] [Bug 1670259] New GFID file recreated in a replica set after a GFID mismatch resolution In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670259 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|ZStream | Component|replicate |distribute Blocks|1662225 | -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 04:50:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:50:47 +0000 Subject: [Bugs] [Bug 1670259] New GFID file recreated in a replica set after a GFID mismatch resolution In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670259 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 04:56:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:56:49 +0000 Subject: [Bugs] [Bug 1670259] New GFID file recreated in a replica set after a GFID mismatch resolution In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670259 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Comment #0 is|1 |0 private| | --- Comment #1 from Nithya Balachandran --- Steps to reproduce the problem: 1. Create a 3 brick distribute volume 2. Fuse mount the volume and create some directories. cd /mnt/fuse1 mkdir gfid-mismatch mkdir gfid-mismatch/dir-1 3. delete the gfid and .glusterfs handle from the hashed brick [root at rhgs313-6 brick1]# setfattr -x trusted.gfid vol1-1/gfid-mismatch/dir-1 [root at rhgs313-6 brick1]# unlink vol1-1/.glusterfs/8e/6c/8e6c686c-93e9-4bd7-ac5e-98bbf852a62b [root at rhgs313-6 brick1]# [root at rhgs313-6 brick1]# [root at rhgs313-6 brick1]# [root at rhgs313-6 brick1]# getx vol1-*/gfid-mismatch/dir-1 # file: vol1-1/gfid-mismatch/dir-1 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.glusterfs.dht=0x00000000000000000000000055555554 trusted.glusterfs.dht.mds=0x00000000 trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3 # file: vol1-2/gfid-mismatch/dir-1 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b trusted.glusterfs.dht=0x000000000000000055555555aaaaaaa9 trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3 # file: vol1-3/gfid-mismatch/dir-1 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b trusted.glusterfs.dht=0x0000000000000000aaaaaaaaffffffff trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3 4. Unmount and remount the fuse mount and list the contents of gfid-mismatch [root at rhgs313-6 ~]# umount -f /mnt/fuse1; mount -t glusterfs -s 192.168.122.6:/vol1 /mnt/fuse1 [root at rhgs313-6 ~]# cd /mnt/fuse1 [root at rhgs313-6 fuse1]# cd gfid-mismatch/ [root at rhgs313-6 gfid-mismatch]# l total 20K drwxr-xr-x. 2 root root 4.0K Jan 29 09:20 dir-1 5. Check the gfid for dir-1 on the backend bricks. [root at rhgs313-6 brick1]# getx vol1-*/gfid-mismatch/dir-1 # file: vol1-1/gfid-mismatch/dir-1 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x0c3a4860f93f416cb5261c3b2b06f52d < ---- GFID is DIFFERENT!! trusted.glusterfs.dht=0x00000000000000000000000055555554 trusted.glusterfs.dht.mds=0x00000000 trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3 # file: vol1-2/gfid-mismatch/dir-1 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b trusted.glusterfs.dht=0x000000000000000055555555aaaaaaa9 trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3 # file: vol1-3/gfid-mismatch/dir-1 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b trusted.glusterfs.dht=0x0000000000000000aaaaaaaaffffffff trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3 The GFID on brick vol1-1 is set to trusted.gfid=0x0c3a4860f93f416cb5261c3b2b06f52d The original GFID was: trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 04:58:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 04:58:11 +0000 Subject: [Bugs] [Bug 1670259] New GFID file recreated in a replica set after a GFID mismatch resolution In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670259 --- Comment #2 from Nithya Balachandran --- Upstream patch: https://review.gluster.org/#/c/glusterfs/+/22112/ -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 06:06:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 06:06:29 +0000 Subject: [Bugs] [Bug 1670259] New GFID file recreated in a replica set after a GFID mismatch resolution In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670259 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Group|redhat | -- You are receiving this mail because: You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 07:19:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 07:19:07 +0000 Subject: [Bugs] [Bug 1349620] libgfapi: Reduce memcpy in glfs write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1349620 Rinku changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |NEW CC| |rkothiya at redhat.com Resolution|CURRENTRELEASE |--- Keywords| |Reopened -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 07:21:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 07:21:18 +0000 Subject: [Bugs] [Bug 1349620] libgfapi: Reduce memcpy in glfs write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1349620 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22113 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 07:21:19 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 07:21:19 +0000 Subject: [Bugs] [Bug 1349620] libgfapi: Reduce memcpy in glfs write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1349620 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #22 from Worker Ant --- REVIEW: https://review.gluster.org/22113 (libgfapi: Introducing zero copy write for libgfapi) posted (#1) for review on master by Rinku Kothiya -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Tue Jan 29 08:19:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:19:49 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22114 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:19:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:19:51 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #524 from Worker Ant --- REVIEW: https://review.gluster.org/22114 (api: bad GFAPI_4.1.6 block) posted (#1) for review on master by Kaleb KEITHLEY -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:20:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:20:40 +0000 Subject: [Bugs] [Bug 1670303] New: api: bad GFAPI_4.1.6 block Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 Bug ID: 1670303 Summary: api: bad GFAPI_4.1.6 block Product: GlusterFS Version: 4.1 Status: NEW Component: libgfapi Assignee: bugs at gluster.org Reporter: kkeithle at redhat.com QA Contact: bugs at gluster.org CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:27:51 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:27:51 +0000 Subject: [Bugs] [Bug 1670303] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22115 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:27:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:27:53 +0000 Subject: [Bugs] [Bug 1670303] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22115 (api: bad GFAPI_4.1.6 block) posted (#1) for review on release-4.1 by Kaleb KEITHLEY -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:29:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:29:36 +0000 Subject: [Bugs] [Bug 1670303] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22116 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:29:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:29:37 +0000 Subject: [Bugs] [Bug 1670303] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22116 (api: bad GFAPI_4.1.6 block) posted (#1) for review on release-4.1 by Kaleb KEITHLEY -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:30:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:30:30 +0000 Subject: [Bugs] [Bug 1670307] New: api: bad GFAPI_4.1.6 block Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670307 Bug ID: 1670307 Summary: api: bad GFAPI_4.1.6 block Product: GlusterFS Version: 5 Status: NEW Component: libgfapi Assignee: bugs at gluster.org Reporter: kkeithle at redhat.com QA Contact: bugs at gluster.org CC: bugs at gluster.org Depends On: 1670303 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1670303 +++ Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2019-01-29 08:27:53 UTC --- REVIEW: https://review.gluster.org/22115 (api: bad GFAPI_4.1.6 block) posted (#1) for review on release-4.1 by Kaleb KEITHLEY --- Additional comment from Worker Ant on 2019-01-29 08:29:37 UTC --- REVIEW: https://review.gluster.org/22116 (api: bad GFAPI_4.1.6 block) posted (#1) for review on release-4.1 by Kaleb KEITHLEY Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 [Bug 1670303] api: bad GFAPI_4.1.6 block -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:30:30 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:30:30 +0000 Subject: [Bugs] [Bug 1670303] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 Kaleb KEITHLEY changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1670307 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670307 [Bug 1670307] api: bad GFAPI_4.1.6 block -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:35:09 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:35:09 +0000 Subject: [Bugs] [Bug 1670307] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670307 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22117 -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 08:35:10 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 08:35:10 +0000 Subject: [Bugs] [Bug 1670307] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670307 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22117 (api: bad GFAPI_4.1.6 block) posted (#1) for review on release-5 by Kaleb KEITHLEY -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 09:28:11 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 09:28:11 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #525 from Worker Ant --- REVIEW: https://review.gluster.org/21951 (Multiple files: reduce work while under lock.) merged (#14) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 10:06:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 10:06:59 +0000 Subject: [Bugs] [Bug 1670334] New: Some memory leaks found in GlusterFS 5.3 Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670334 Bug ID: 1670334 Summary: Some memory leaks found in GlusterFS 5.3 Product: GlusterFS Version: 5 Hardware: All OS: All Status: NEW Component: core Severity: medium Assignee: bugs at gluster.org Reporter: i_chips at qq.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: In xlators\mgmt\glusterd\src\glusterd-mountbroker.c, the function make_ghadoop_mountspec() has memory leak. I'm afraid that it could be modified like this: int make_ghadoop_mountspec(gf_mount_spec_t *mspec, const char *volname, char *user, char *server) { char *hadoop_mnt_desc = NULL; int ret = 0; ret = gf_asprintf(&hadoop_mnt_desc, hadoop_mnt_desc_template, server, GF_CLIENT_PID_HADOOP, volname, user); if (ret == -1) return ret; ret = parse_mount_pattern_desc(mspec, hadoop_mnt_desc); GF_FREE(hadoop_mnt_desc); return ret; } Version-Release number of selected component (if applicable): 5.3 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 13:13:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 13:13:29 +0000 Subject: [Bugs] [Bug 1670382] New: parallel-readdir prevents directories and files listing Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670382 Bug ID: 1670382 Summary: parallel-readdir prevents directories and files listing Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: gluster-smb Severity: high Assignee: bugs at gluster.org Reporter: locbus at gmail.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: It looks like the problem described at the following link still exists: https://bugzilla.redhat.com/show_bug.cgi?id=1512371 In our case, however, the main client are windows machines. However, even after directly creation of files and directories on the gluster resource, the second cluster host cannot see new files and directories... The problem does not occur, for example, in version 4.1.7 of the gluster. After disabling `performance.parallel-readdir`, the problem disappears and everything works correctly. Version-Release number of selected component (if applicable): - Ubuntu 16.4.5 LTS x64 - Gluster Versions 5.3 - Gluster Client Versions 5.3 Steps to Reproduce: 1. Enable performance.parallel-readdir on the volume. 2. Mount the volume on a client using the samba protocol. 3. Create a directory or file within the volume. Expected results: - The directory and files should show up Additional info: Volume Name: gv0 Type: Distributed-Replicate Volume ID: 8153ffd6-6da3-462d-a1c3-9a23da127a3a Status: Started Snapshot Count: 0 Number of Bricks: 3 x 2 = 6 Transport-type: tcp Bricks: Brick1: test-sn1:/storage/sda2/brick0 Brick2: test-sn2:/storage/sda2/brick0 Brick3: test-sn1:/storage/sda3-2/brick1 Brick4: test-sn2:/storage/sda3/brick1 Brick5: test-sn1:/storage/sda4/brick2 Brick6: test-sn2:/storage/sda4/brick2 Options Reconfigured: server.statedump-path: /var/log/glusterfs/ nfs.disable: on transport.address-family: inet cluster.self-heal-daemon: enable storage.build-pgfid: off server.event-threads: 4 client.event-threads: 4 cluster.lookup-optimize: on features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.nl-cache-timeout: 600 network.inode-lru-limit: 200000 performance.cache-samba-metadata: on performance.cache-size: 256MB performance.nl-cache: on performance.md-cache-timeout: 600 performance.stat-prefetch: on performance.cache-invalidation: on performance.parallel-readdir: on cluster.readdir-optimize: on performance.client-io-threads: on user.smb: enable storage.batch-fsync-delay-usec: 0 performance.readdir-ahead: on -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 13:55:07 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 13:55:07 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22107 (features/sdfs: disable by default) merged (#3) on master by Atin Mukherjee -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 13:56:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 13:56:17 +0000 Subject: [Bugs] [Bug 1670031] performance regression seen with smallfile workload tests In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670031 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |amukherj at redhat.com --- Comment #3 from Atin Mukherjee --- It might be worth to open a different bugzilla to track the perf regression for sdfs only considering this bug is used as a tracker. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Tue Jan 29 15:08:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Tue, 29 Jan 2019 15:08:37 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 David E. Smith changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |desmith at wustl.edu --- Comment #15 from David E. Smith --- I'm having what appears to be the same issue. Started when I upgraded from 3.12 to 5.2 a few weeks back, and the subsequent upgrade to 5.3 did not resolve the problem. My servers (two, in a 'replica 2' setup) publish two volumes. One is Web site content, about 110GB; the other is Web config files, only a few megabytes. (Wasn't worth building extra servers for that second volume.) FUSE clients have been crashing on the larger volume every three or four days. The client's logs show many hundreds of instances of this (I don't know if it's related): [2019-01-29 08:14:16.542674] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7384) [0x7fa171ead384] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xae3e) [0x7fa1720bee3e] -->/lib64/libglusterfs.so.0(dict_ref+0x5d) [0x7fa1809cc2ad] ) 0-dict: dict is NULL [Invalid argument] Then, when the client's glusterfs process crashes, this is logged: The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 871 times between [2019-01-29 08:12:48.390535] and [2019-01-29 08:14:17.100279] pending frames: frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2019-01-29 08:14:17 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] --------- Info on the volumes themselves, gathered from one of my servers: [davidsmith at wuit-s-10889 ~]$ sudo gluster volume info all Volume Name: web-config Type: Replicate Volume ID: 6c5dce6e-e64e-4a6d-82b3-f526744b463d Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-config Brick2: 172.23.128.27:/data/web-config Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet server.event-threads: 4 client.event-threads: 4 cluster.min-free-disk: 1 cluster.quorum-count: 2 cluster.quorum-type: fixed network.ping-timeout: 10 auth.allow: * performance.readdir-ahead: on Volume Name: web-content Type: Replicate Volume ID: fcabc15f-0cec-498f-93c4-2d75ad915730 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-content Brick2: 172.23.128.27:/data/web-content Options Reconfigured: network.ping-timeout: 10 cluster.quorum-type: fixed cluster.quorum-count: 2 performance.readdir-ahead: on auth.allow: * cluster.min-free-disk: 1 client.event-threads: 4 server.event-threads: 4 transport.address-family: inet nfs.disable: on performance.client-io-threads: off performance.cache-size: 4GB gluster> volume status all detail Status of volume: web-config ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5612 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5540 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 Status of volume: web-content ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5649 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5567 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 I have a couple of core files that appear to be from this, but I'm not much of a developer (haven't touched C in fifteen years) so I don't know what to do with them that would be of value in this case. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 05:41:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 05:41:39 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #7 from Raghavendra G --- (In reply to Manoj Pillai from comment #5) > (In reply to Raghavendra G from comment #4) > > From preliminary tests I see two reasons for this: > > 1. inode-invalidations triggered by md-cache > > 2. Fuse auto invalidations > > Trying with kernel NFS, another distributed fs solution. I see that cache is > retained at the end of the write test, and both read-after-write and > read-after-read are served from the page cache. > > In principle, if kNFS can do it, FUSE should be able to do it. I think :D. kNFS and FUSE have different invalidation policies. * kNFS provides close-to-open consistency. To quote from their FAQ [1] "Linux implements close-to-open cache consistency by comparing the results of a GETATTR operation done just after the file is closed to the results of a GETATTR operation done when the file is next opened. If the results are the same, the client will assume its data cache is still valid; otherwise, the cache is purged." For the workload used in this bz, file is not changed between close and open. Hence two values of stat fetched - at close and open - match and hence page-cache is retained. * FUSE auto-invalidation compares times of stats cached with the values got from the underlying filesystem implementation at all codepaths where stat is fetched. This means comparision happens in lookup, (f)stat, (f)setattr etc codepaths. Since (f)stat, lookup can happen asynchronously and concurrently wrt writes, they'll end up identifying delta between two values of stats resulting in cache purge. Please note that the consistency offered by FUSE is stronger than close-to-open consistency, which means it also provides close-to-open consistency along with consistency in codepaths like lookup, fstat etc. We have following options: * disable auto-invalidations and use glusterfs custom designed invalidation policy. The invalidation policy can be the same as NFS close-to-open consistency or something more stronger. * check whether the current form of auto-invalidation (though stricter) provides any added benefits to close-to-open consistency which are useful. If no, change FUSE auto-invalidation to close-to-open consistency. [1] http://nfs.sourceforge.net/#faq_a8 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 05:45:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 05:45:23 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mszeredi at redhat.com Flags| |needinfo?(mszeredi at redhat.c | |om) --- Comment #8 from Raghavendra G --- Miklos, It would be helpful if you can comment on comment #7. regards, Raghavendra -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 05:59:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 05:59:06 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 --- Comment #9 from Raghavendra G --- Note that a lease based invalidation policy would be a complete solution, but it will take some time to implement that and get it working in Glusterfs. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 07:01:53 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 07:01:53 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #526 from Worker Ant --- REVIEW: https://review.gluster.org/22114 (api: bad GFAPI_4.1.6 block) merged (#2) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 08:54:57 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 08:54:57 +0000 Subject: [Bugs] [Bug 1664934] glusterfs-fuse client not benefiting from page cache on read after write In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1664934 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1670710 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670710 [Bug 1670710] glusterfs-fuse client not benefiting from page cache on read after write -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 09:02:05 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:02:05 +0000 Subject: [Bugs] [Bug 1670718] New: md-cache should be loaded at a position in graph where it sees stats in write cbk Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670718 Bug ID: 1670718 Summary: md-cache should be loaded at a position in graph where it sees stats in write cbk Product: GlusterFS Version: mainline Status: NEW Component: glusterd Assignee: bugs at gluster.org Reporter: rgowdapp at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: The current xlator graph has write-behind as a child of md-cache. When writes are cached, write-behind returns NULL values for stats. So, a write heavy workload essentially removes stats from cache always rendering md-cache useless. If we load md-cache as a child of write-behind, write cbk will have stats from bricks and hence cache will be updated with latest stat in write workloads. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 09:02:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:02:17 +0000 Subject: [Bugs] [Bug 1670718] md-cache should be loaded at a position in graph where it sees stats in write cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670718 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs at gluster.org |rgowdapp at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 09:05:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:05:15 +0000 Subject: [Bugs] [Bug 1670718] md-cache should be loaded at a position in graph where it sees stats in write cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670718 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22124 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 09:05:17 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:05:17 +0000 Subject: [Bugs] [Bug 1670718] md-cache should be loaded at a position in graph where it sees stats in write cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670718 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22124 (performance/md-cache: load as a child of write-behind) posted (#1) for review on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 09:05:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:05:42 +0000 Subject: [Bugs] [Bug 1670719] New: md-cache should be loaded at a position in graph where it sees stats in write cbk Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670719 Bug ID: 1670719 Summary: md-cache should be loaded at a position in graph where it sees stats in write cbk Product: Red Hat Gluster Storage Version: 3.4 Status: NEW Component: glusterd Assignee: amukherj at redhat.com Reporter: rgowdapp at redhat.com QA Contact: bmekala at redhat.com CC: bugs at gluster.org, rhs-bugs at redhat.com, sankarshan at redhat.com, storage-qa-internal at redhat.com, vbellur at redhat.com Depends On: 1670718 Target Milestone: --- Classification: Red Hat +++ This bug was initially created as a clone of Bug #1670718 +++ Description of problem: The current xlator graph has write-behind as a child of md-cache. When writes are cached, write-behind returns NULL values for stats. So, a write heavy workload essentially removes stats from cache always rendering md-cache useless. If we load md-cache as a child of write-behind, write cbk will have stats from bricks and hence cache will be updated with latest stat in write workloads. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2019-01-30 09:05:17 UTC --- REVIEW: https://review.gluster.org/22124 (performance/md-cache: load as a child of write-behind) posted (#1) for review on master by Raghavendra G Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670718 [Bug 1670718] md-cache should be loaded at a position in graph where it sees stats in write cbk -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 09:05:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:05:42 +0000 Subject: [Bugs] [Bug 1670718] md-cache should be loaded at a position in graph where it sees stats in write cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670718 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1670719 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670719 [Bug 1670719] md-cache should be loaded at a position in graph where it sees stats in write cbk -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 09:05:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:05:47 +0000 Subject: [Bugs] [Bug 1670719] md-cache should be loaded at a position in graph where it sees stats in write cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670719 Red Hat Bugzilla Rules Engine changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |ZStream -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 09:06:18 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:06:18 +0000 Subject: [Bugs] [Bug 1670719] md-cache should be loaded at a position in graph where it sees stats in write cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670719 Raghavendra G changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1629589 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1629589 [Bug 1629589] Gluster-file Volume under-performing than Gluster-block Volume for postgresql workload -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 09:41:38 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 09:41:38 +0000 Subject: [Bugs] [Bug 1670719] md-cache should be loaded at a position in graph where it sees stats in write cbk In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670719 Atin Mukherjee changed: What |Removed |Added ---------------------------------------------------------------------------- Component|glusterd |md-cache Assignee|amukherj at redhat.com |rgowdapp at redhat.com QA Contact|bmekala at redhat.com |vdas at redhat.com -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 11:35:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 11:35:42 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 Kaleb KEITHLEY changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1670307 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670307 [Bug 1670307] api: bad GFAPI_4.1.6 block -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 11:35:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 11:35:42 +0000 Subject: [Bugs] [Bug 1670307] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670307 Kaleb KEITHLEY changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1667103 (glusterfs-5.4) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 11:36:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 11:36:47 +0000 Subject: [Bugs] [Bug 1667099] GlusterFS 4.1.8 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667099 Kaleb KEITHLEY changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1670303 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 [Bug 1670303] api: bad GFAPI_4.1.6 block -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 11:36:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 11:36:47 +0000 Subject: [Bugs] [Bug 1670303] api: bad GFAPI_4.1.6 block In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670303 Kaleb KEITHLEY changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1667099 (glusterfs-4.1.8) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667099 [Bug 1667099] GlusterFS 4.1.8 tracker -- You are receiving this mail because: You are the QA Contact for the bug. You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 12:30:06 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 12:30:06 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 Guenther Deschner changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(gdeschner at redhat. | |com) | --- Doc Text *updated* --- Previously, when gluster was accessed using SMB from a macOS client using the recommended vfs_glusterfs module, file handles were closed incorrectly. As a consequence, the samba service on the server would hang and the user would need to reconnect. With this fix, Samba correctly closes the file handles at the right layer when using the vfs_glusterfs module. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 14:50:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 14:50:39 +0000 Subject: [Bugs] [Bug 1671014] New: gluster-fuse seg fault Transport endpoint is not connected Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671014 Bug ID: 1671014 Summary: gluster-fuse seg fault Transport endpoint is not connected Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: fuse Severity: high Assignee: bugs at gluster.org Reporter: joao.bauto at neuro.fchampalimaud.org CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: The gluster-fuse 5.3-1 process crashes with a segmentation fault error 11 after high usage on the mount. The system is a 4 node distributed volume with 2 bricks per node. Info and backtrace below. Client has volume mounted with default options. gluster volume info ------------------------ Volume Name: tank Type: Distribute Volume ID: 9582685f-07fa-41fd-b9fc-ebab3a6989cf Status: Started Snapshot Count: 0 Number of Bricks: 8 Transport-type: tcp Bricks: Brick1: swp-gluster-01:/tank/volume1/brick Brick2: swp-gluster-02:/tank/volume1/brick Brick3: swp-gluster-03:/tank/volume1/brick Brick4: swp-gluster-04:/tank/volume1/brick Brick5: swp-gluster-01:/tank/volume2/brick Brick6: swp-gluster-02:/tank/volume2/brick Brick7: swp-gluster-03:/tank/volume2/brick Brick8: swp-gluster-04:/tank/volume2/brick Options Reconfigured: features.inode-quota: on features.quota: on nfs.disable: on features.quota-deem-statfs: on server.event-threads: 16 client.event-threads: 16 cluster.lookup-optimize: on cluster.readdir-optimize: on features.cache-invalidation: on features.cache-invalidation-timeout: 600 network.inode-lru-limit: 200000 performance.cache-samba-metadata: on performance.write-behind: on performance.client-io-threads: off performance.flush-behind: on performance.io-cache: off performance.io-thread-count: 64 performance.nl-cache: on performance.nl-cache-timeout: 600 performance.open-behind: on performance.quick-read: on performance.rda-request-size: 131072 performance.read-ahead-page-count: 16 performance.read-ahead: on performance.stat-prefetch: on performance.write-behind-window-size: 32MB storage.batch-fsync-delay-usec: 0 performance.md-cache-timeout: 600 cluster.enable-shared-storage: disable Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --process-name fuse --volfile-server=nfs.gluster.research.p'. Program terminated with signal 11, Segmentation fault. #0 __GI___pthread_mutex_lock (mutex=0x7f407ed19e78) at ../nptl/pthread_mutex_lock.c:66 66 unsigned int type = PTHREAD_MUTEX_TYPE_ELISION (mutex); Missing separate debuginfos, use: debuginfo-install glusterfs-fuse-5.3-1.el7.x86_64 (gdb) bt #0 __GI___pthread_mutex_lock (mutex=0x7f407ed19e78) at ../nptl/pthread_mutex_lock.c:66 #1 0x00007f0fce4bdc3c in __gf_free () from /lib64/libglusterfs.so.0 #2 0x00007f0fbb591c71 in ios_stat_add_to_list () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so #3 0x00007f0fbb59a6a1 in io_stats_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so #4 0x00007f0fbb7c47b1 in mdc_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/md-cache.so #5 0x00007f0fce51ddc8 in default_writev_cbk () from /lib64/libglusterfs.so.0 #6 0x00007f0fbbdf6f99 in qr_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so #7 0x00007f0fc001134a in rda_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so #8 0x00007f0fc021f751 in ra_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #9 0x00007f0fc0431571 in wb_do_unwinds () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #10 0x00007f0fc04329c0 in wb_process_queue () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #11 0x00007f0fc0433377 in wb_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #12 0x00007f0fc021fca3 in ra_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #13 0x00007f0fc000d10a in rda_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so #14 0x00007f0fbbdf5c0e in qr_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so #15 0x00007f0fce53e110 in default_writev_resume () from /lib64/libglusterfs.so.0 #16 0x00007f0fce4ba25a in call_resume_wind () from /lib64/libglusterfs.so.0 #17 0x00007f0fce4ba445 in call_resume () from /lib64/libglusterfs.so.0 #18 0x00007f0fbbbe8495 in ob_wake_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/open-behind.so #19 0x00007f0fce51c1eb in default_open_cbk () from /lib64/libglusterfs.so.0 #20 0x00007f0fc021abf6 in ra_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #21 0x00007f0fc06baa01 in dht_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so #22 0x00007f0fc0964bdd in client4_0_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/protocol/client.so #23 0x00007f0fce25ec70 in rpc_clnt_handle_reply () from /lib64/libgfrpc.so.0 #24 0x00007f0fce25f043 in rpc_clnt_notify () from /lib64/libgfrpc.so.0 #25 0x00007f0fce25af23 in rpc_transport_notify () from /lib64/libgfrpc.so.0 #26 0x00007f0fc2e4837b in socket_event_handler () from /usr/lib64/glusterfs/5.3/rpc-transport/socket.so #27 0x00007f0fce4f6a49 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #28 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fabfff700) at pthread_create.c:308 #29 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 gdb glusterd core.glfs_epoll005.1548850256.17447 Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --process-name fuse --volfile-server=nfs.gluster.research.p'. Program terminated with signal 11, Segmentation fault. #0 __GI___pthread_mutex_lock (mutex=0x7f407ed19e78) at ../nptl/pthread_mutex_lock.c:66 66 unsigned int type = PTHREAD_MUTEX_TYPE_ELISION (mutex); Missing separate debuginfos, use: debuginfo-install glusterfs-fuse-5.3-1.el7.x86_64 (gdb) t a a bt Thread 25 (Thread 0x7f0fa9ffb700 (LWP 17466)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fa9ffb700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 24 (Thread 0x7f0fc385c700 (LWP 17452)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f0fce4d1cf8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f0fce4d2c40 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc385c700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 23 (Thread 0x7f0f8dffb700 (LWP 17475)): #0 0x00007f0fccbb5660 in __libc_readv (fd=8, vector=0x7f0f8dffae70, count=2) at ../sysdeps/unix/sysv/linux/readv.c:56 #1 0x00007f0fce4bfe49 in sys_readv () from /lib64/libglusterfs.so.0 #2 0x00007f0fc5880c60 in fuse_thread_proc () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8dffb700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 22 (Thread 0x7f0fc405d700 (LWP 17451)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f0fce4d1cf8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f0fce4d2c40 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc405d700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 21 (Thread 0x7f0fa97fa700 (LWP 17467)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fa97fa700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 20 (Thread 0x7f0fb8a6a700 (LWP 17461)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fb8a6a700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 19 (Thread 0x7f0fb9a6c700 (LWP 17459)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fb9a6c700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 18 (Thread 0x7f0fc485e700 (LWP 17450)): #0 0x00007f0fccb8556d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fccb85404 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007f0fce4bde7d in pool_sweeper () from /lib64/libglusterfs.so.0 #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc485e700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 17 (Thread 0x7f0fce98b780 (LWP 17447)): #0 0x00007f0fcd2f6f97 in pthread_join (threadid=139705642981120, thread_return=0x0) at pthread_join.c:92 #1 0x00007f0fce4f5db8 in event_dispatch_epoll () from /lib64/libglusterfs.so.0 #2 0x0000561e9b26056b in main () Thread 16 (Thread 0x7f0f8e7fc700 (LWP 17472)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8e7fc700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 15 (Thread 0x7f0f8effd700 (LWP 17471)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8effd700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 14 (Thread 0x7f0f8d7fa700 (LWP 17476)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f0fc5869dbb in notify_kernel_loop () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8d7fa700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ---Type to continue, or q to quit--- Thread 13 (Thread 0x7f0fa8ff9700 (LWP 17468)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fa8ff9700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 12 (Thread 0x7f0fbaa6e700 (LWP 17457)): #0 0x00007f0fccbb5c73 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce510d3f in runner () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fbaa6e700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 11 (Thread 0x7f0fb926b700 (LWP 17460)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fb926b700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 10 (Thread 0x7f0f8ffff700 (LWP 17469)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8ffff700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 9 (Thread 0x7f0fba26d700 (LWP 17458)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fba26d700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 8 (Thread 0x7f0fc5860700 (LWP 17448)): #0 0x00007f0fcd2fcf3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4a0f76 in gf_timer_proc () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc5860700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 7 (Thread 0x7f0f8f7fe700 (LWP 17470)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8f7fe700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 6 (Thread 0x7f0fc1391700 (LWP 17456)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc1391700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 5 (Thread 0x7f0faaffd700 (LWP 17464)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0faaffd700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f0fc505f700 (LWP 17449)): #0 0x00007f0fcd2fd461 in do_sigwait (sig=0x7f0fc505ee1c, set=) at ../sysdeps/unix/sysv/linux/sigwait.c:61 #1 __sigwait (set=0x7f0fc505ee20, sig=0x7f0fc505ee1c) at ../sysdeps/unix/sysv/linux/sigwait.c:99 #2 0x0000561e9b263e2b in glusterfs_sigwaiter () #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc505f700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f0fab7fe700 (LWP 17463)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fab7fe700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 2 (Thread 0x7f0faa7fc700 (LWP 17465)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0faa7fc700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 1 (Thread 0x7f0fabfff700 (LWP 17462)): #0 __GI___pthread_mutex_lock (mutex=0x7f407ed19e78) at ../nptl/pthread_mutex_lock.c:66 #1 0x00007f0fce4bdc3c in __gf_free () from /lib64/libglusterfs.so.0 #2 0x00007f0fbb591c71 in ios_stat_add_to_list () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so #3 0x00007f0fbb59a6a1 in io_stats_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so #4 0x00007f0fbb7c47b1 in mdc_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/md-cache.so #5 0x00007f0fce51ddc8 in default_writev_cbk () from /lib64/libglusterfs.so.0 #6 0x00007f0fbbdf6f99 in qr_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so #7 0x00007f0fc001134a in rda_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so #8 0x00007f0fc021f751 in ra_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #9 0x00007f0fc0431571 in wb_do_unwinds () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #10 0x00007f0fc04329c0 in wb_process_queue () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #11 0x00007f0fc0433377 in wb_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #12 0x00007f0fc021fca3 in ra_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #13 0x00007f0fc000d10a in rda_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so #14 0x00007f0fbbdf5c0e in qr_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so #15 0x00007f0fce53e110 in default_writev_resume () from /lib64/libglusterfs.so.0 #16 0x00007f0fce4ba25a in call_resume_wind () from /lib64/libglusterfs.so.0 #17 0x00007f0fce4ba445 in call_resume () from /lib64/libglusterfs.so.0 #18 0x00007f0fbbbe8495 in ob_wake_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/open-behind.so #19 0x00007f0fce51c1eb in default_open_cbk () from /lib64/libglusterfs.so.0 #20 0x00007f0fc021abf6 in ra_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #21 0x00007f0fc06baa01 in dht_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so #22 0x00007f0fc0964bdd in client4_0_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/protocol/client.so #23 0x00007f0fce25ec70 in rpc_clnt_handle_reply () from /lib64/libgfrpc.so.0 #24 0x00007f0fce25f043 in rpc_clnt_notify () from /lib64/libgfrpc.so.0 #25 0x00007f0fce25af23 in rpc_transport_notify () from /lib64/libgfrpc.so.0 #26 0x00007f0fc2e4837b in socket_event_handler () from /usr/lib64/glusterfs/5.3/rpc-transport/socket.so #27 0x00007f0fce4f6a49 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #28 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fabfff700) at pthread_create.c:308 #29 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 16:03:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 16:03:47 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #16 from Digiteyes --- I have same issue , and my server crash 4-5 time per day , we need urgent bug fix , we cant work any more -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 16:07:45 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 16:07:45 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #17 from Digiteyes --- [2019-01-30 15:50:39.219564] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8853076771410540308.tmp (ba250583-e103-473e-92de-3e0d87afe8be) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0086.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:50:44.206312] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:50:44.350266] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-6758755102184008102.tmp (32dbb8cb-aec9-4bae-992b-fbd86cd50828) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0017.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:50:45.489090] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-6687721062137662117.tmp (62bbb010-16ff-462c-b0dd-718b0e62a8c7) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0018.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:50:45.551349] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 5 times between [2019-01-30 15:50:45.551349] and [2019-01-30 15:50:56.559333] [2019-01-30 15:51:02.317536] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8497906571178810383.tmp (15ba641a-cb3f-42d2-b9b5-b17f10e027c8) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0081.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:51:07.031853] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0086.exr (ba250583-e103-473e-92de-3e0d87afe8be) (hash=mothervolume-client-0/cache=mothervolume-client-1) => /.recycle/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Copy #2 of Seq_A_A_Sh010_comp_SH030208_v001.0086.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:51:07.109087] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8853076771410540308.tmp (d514f600-f3e6-4639-822a-05e057e1d83c) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0086.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:51:07.620516] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8568940611225156368.tmp (f7efca88-3886-4750-ad2f-4f793fd8487d) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0082.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:51:12.458961] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8711008691317848338.tmp (92ddf1a3-50a9-48b1-be22-7bfa359b5b65) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0084.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:51:15.629779] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 53 times between [2019-01-30 15:51:15.629779] and [2019-01-30 15:51:45.695496] [2019-01-30 15:51:45.700709] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8497906571178810383.tmp (f853a226-70fb-4537-a629-e1e2cefdcfe7) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0081.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:51:47.398973] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:51:47.588670] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:51:51.885883] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8568940611225156368.tmp (8f5bcab2-e3ba-478a-957c-fdf243216a4e) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0082.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:51:53.453191] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 13 times between [2019-01-30 15:51:53.453191] and [2019-01-30 15:51:56.196530] [2019-01-30 15:51:56.510824] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8711008691317848338.tmp (45aaff0e-15b5-4b6b-8b41-c4caed57f881) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0084.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:51:57.207664] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 81 times between [2019-01-30 15:51:57.207664] and [2019-01-30 15:52:19.002777] [2019-01-30 15:52:19.183448] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0094.exr (504238f8-7918-496e-835e-3246d16cf35e) (hash=mothervolume-client-1/cache=mothervolume-client-0) => /.recycle/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0094.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:52:19.257335] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/1446751043051559505.tmp (3dabe7b2-9682-4b33-842e-b144533e97d4) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0094.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:52:19.574477] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 4 times between [2019-01-30 15:52:19.574477] and [2019-01-30 15:52:24.127146] [2019-01-30 15:52:24.656623] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/nuke/Seq_A_A_Sh010_comp_scene_208.v002.nk.autosavet (9ef65443-4745-448d-a8b4-fa3f3bbf7487) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/nuke/Seq_A_A_Sh010_comp_scene_208.v002.nk.autosave ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:52:24.899131] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:52:27.431451] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8497906571178810383.tmp (749b5a12-80a7-47d5-b5d2-2e60cbea57aa) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0081.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:52:30.891799] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:52:31.047076] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:52:32.939577] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8568940611225156368.tmp (952bcc2f-a486-4d61-ade5-329e1a6165a8) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0082.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:52:37.606502] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8711008691317848338.tmp (83ebc0c0-ba5c-4fdc-808e-f343e1ae28e2) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0084.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:52:43.967857] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:52:55.087185] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/nuke/Seq_A_A_Sh010_comp_scene_208.v002.nk.autosavet (4f7b2158-c4b7-4579-8e8a-7ab3dc8d9b0d) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/nuke/Seq_A_A_Sh010_comp_scene_208.v002.nk.autosave ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:53:17.204114] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2765564699243796980.tmp (e72fb096-c4ef-490a-a39c-47531608dd63) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0080.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:53:17.396151] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:53:22.458305] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2694530659197450995.tmp (e02e5460-3557-470b-a7f3-109a9914c692) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0081.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:53:26.229226] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2623496619151105010.tmp (2fa4c400-77a0-459e-b734-8e4f28926859) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0082.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:53:32.149207] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2552462579104759025.tmp (cb9b9130-fc7e-4c04-89e9-7b925c955669) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0083.exr ((null)) (hash=mothervolume-client-1/cache=) The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 18 times between [2019-01-30 15:53:17.396151] and [2019-01-30 15:53:34.167757] [2019-01-30 15:53:37.062257] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0098.exr (f20e23cd-76ff-4371-a31b-0a9cf9022860) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /.recycle/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Copy #1 of Seq_A_A_Sh010_comp_SH030208_v001.0098.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:53:37.149778] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/1730887203236943445.tmp (f4b57fa7-4de1-4de7-bc06-00e2741c5129) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0098.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:53:37.306807] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2481428539058413040.tmp (27286f68-e368-48c1-b497-0300cc3af4c7) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0084.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:53:38.961986] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0094.exr (3dabe7b2-9682-4b33-842e-b144533e97d4) (hash=mothervolume-client-1/cache=mothervolume-client-0) => /.recycle/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Copy #1 of Seq_A_A_Sh010_comp_SH030208_v001.0094.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:53:39.053762] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/1446751043051559505.tmp (3055dc73-8b71-4437-9dab-8219f7ea6189) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0094.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:53:43.220690] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2410394499012067055.tmp (41462049-f2d4-4638-b137-c7921219820b) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0085.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:53:44.188358] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 4 times between [2019-01-30 15:53:44.188358] and [2019-01-30 15:53:45.698529] [2019-01-30 15:53:47.773401] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/8497906571178810383.tmp (def787fb-4c77-4049-9629-15db1b4acd36) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0081.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:53:48.345901] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2339360458965721070.tmp (53f63fd7-904e-412d-a789-c2170735a61f) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0086.exr ((null)) (hash=mothervolume-client-1/cache=) [2019-01-30 15:53:49.291189] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:53:49.450504] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:53:53.495085] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2268326418919375085.tmp (5a339407-94ab-456e-9670-9488c43e5a9e) (hash=mothervolume-client-1/cache=mothervolume-client-1) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0087.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:53:54.919809] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:53:56.335023] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-01-30 15:53:58.191979] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/-2197292378873029100.tmp (12a7836b-4e9d-46db-9f03-acd30edee2f1) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH010208/Seq_A_A_Sh010_comp_SH010208_v001.0088.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:53:58.920443] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 7 times between [2019-01-30 15:53:58.920443] and [2019-01-30 15:54:00.336410] [2019-01-30 15:54:00.519418] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0094.exr (3055dc73-8b71-4437-9dab-8219f7ea6189) (hash=mothervolume-client-1/cache=mothervolume-client-0) => /.recycle/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Copy #2 of Seq_A_A_Sh010_comp_SH030208_v001.0094.exr ((null)) (hash=mothervolume-client-0/cache=) [2019-01-30 15:54:00.601804] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 0-mothervolume-dht: renaming /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/1446751043051559505.tmp (1ed49b72-c1d5-4704-9ebf-00814afdcb43) (hash=mothervolume-client-0/cache=mothervolume-client-0) => /work_serveur/Peugeot_phev/sequences/Seq_A/A_Sh010/comp/work/images/EXR_Seq_A_A_Sh010_comp_v001/SH030208/Seq_A_A_Sh010_comp_SH030208_v001.0094.exr ((null)) (hash=mothervolume-client-1/cache=) pending frames: frame : type(0) op(0) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-30 15:54:00 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /lib64/libglusterfs.so.0(+0x26610)[0x7f30187de610] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f30187e8b84] /lib64/libc.so.6(+0x36280)[0x7f3016e42280] /lib64/libc.so.6(gsignal+0x37)[0x7f3016e42207] /lib64/libc.so.6(abort+0x148)[0x7f3016e438f8] /lib64/libc.so.6(+0x78d27)[0x7f3016e84d27] /lib64/libc.so.6(+0x81489)[0x7f3016e8d489] /lib64/libglusterfs.so.0(+0x1a6e9)[0x7f30187d26e9] /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x8cf9)[0x7f300a9a7cf9] /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x4ab90)[0x7f300a9e9b90] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x616d2)[0x7f300acb86d2] /lib64/libgfrpc.so.0(+0xec70)[0x7f30185aac70] /lib64/libgfrpc.so.0(+0xf043)[0x7f30185ab043] /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f30185a6f23] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7f300d19337b] /lib64/libglusterfs.so.0(+0x8aa49)[0x7f3018842a49] /lib64/libpthread.so.0(+0x7dd5)[0x7f3017641dd5] /lib64/libc.so.6(clone+0x6d)[0x7f3016f09ead] -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 16:08:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 16:08:55 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #18 from Digiteyes --- Created attachment 1525090 --> https://bugzilla.redhat.com/attachment.cgi?id=1525090&action=edit Mount Log -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 18:20:32 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 18:20:32 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 tavis.paquette at eventbase.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tavis.paquette at eventbase.co | |m --- Comment #19 from tavis.paquette at eventbase.com --- I'm also experiencing this issue, began after an upgrade to 5.1, continued to occur through upgrades to 5.3 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 447 times between [2019-01-30 18:13:29.742333] and [2019-01-30 18:15:27.890656] [2019-01-30 18:15:34.980908] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 27 times between [2019-01-30 18:15:34.980908] and [2019-01-30 18:17:23.626256] [2019-01-30 18:17:31.085125] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 31 times between [2019-01-30 18:17:31.085125] and [2019-01-30 18:19:27.231000] [2019-01-30 18:19:38.782441] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 20:23:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 20:23:44 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 Artem Russakovskii changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |archon810 at gmail.com --- Comment #10 from Artem Russakovskii --- I just started seeing this error after upgrading from 4.1 to 5.3. [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] And it floods like crazy with these. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 20:33:12 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 20:33:12 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #11 from Emerson Gomes --- I "solved" the issue after upgrading to 5.3 by creating a new volume and moving all data to it. Apparently something is missing on the volumes after upgrade. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 20:37:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 20:37:13 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #12 from Artem Russakovskii --- I just sent a message to the gluster mailing list about this because that's not how this problem should be resolved. I'm curious to hear what they say. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 20:39:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 20:39:04 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #13 from Emerson Gomes --- Absolutely. That's the second big issue I had after upgrading. The first one is https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Still unsolved (open for more than 2 months now) -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 20:40:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 20:40:29 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #14 from Artem Russakovskii --- You know, I was *just* going to comment in a follow-up reply about whether the issue here is possibly related to the one you just linked. Seeing tons of those too, though at least the dupes are suppressed. ==> mnt-SITE_data1.log <== [2019-01-30 20:38:20.783713] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] ==> mnt-SITE_data3.log <== The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 413 times between [2019-01-30 20:36:23.881090] and [2019-01-30 20:38:20.015593] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0" repeated 42 times between [2019-01-30 20:36:23.290287] and [2019-01-30 20:38:20.280306] ==> mnt-SITE_data1.log <== The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0" repeated 50 times between [2019-01-30 20:36:22.247367] and [2019-01-30 20:38:19.459789] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 2654 times between [2019-01-30 20:36:22.667327] and [2019-01-30 20:38:20.546355] [2019-01-30 20:38:21.492319] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0 ==> mnt-SITE_data3.log <== [2019-01-30 20:38:22.349689] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0 ==> mnt-SITE_data1.log <== [2019-01-30 20:38:22.762941] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 20:41:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 20:41:23 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Artem Russakovskii changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |archon810 at gmail.com --- Comment #20 from Artem Russakovskii --- Got a ton of these in my logs after upgrading from 4.1 to 5.3, in addition to a lot of repeated messages here https://bugzilla.redhat.com/show_bug.cgi?id=1313567. ==> mnt-SITE_data1.log <== [2019-01-30 20:38:20.783713] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] ==> mnt-SITE_data3.log <== The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 413 times between [2019-01-30 20:36:23.881090] and [2019-01-30 20:38:20.015593] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0" repeated 42 times between [2019-01-30 20:36:23.290287] and [2019-01-30 20:38:20.280306] ==> mnt-SITE_data1.log <== The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0" repeated 50 times between [2019-01-30 20:36:22.247367] and [2019-01-30 20:38:19.459789] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 2654 times between [2019-01-30 20:36:22.667327] and [2019-01-30 20:38:20.546355] [2019-01-30 20:38:21.492319] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0 ==> mnt-SITE_data3.log <== [2019-01-30 20:38:22.349689] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0 ==> mnt-SITE_data1.log <== [2019-01-30 20:38:22.762941] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Wed Jan 30 20:48:52 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 20:48:52 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #15 from Emerson Gomes --- Yeah, both arrised after upgrading from 3.x to 5.1, persisting in 5.2 and 5.3. The other issue is even more critical. It causes crashes, making the mount point being inacessible ("Transport endpoint is not connected" error) - Requiring a new manual umount/mount. For now I have a crontab entry doing this, but I will have to downgrade if a fix is not issued soon... -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Wed Jan 30 21:15:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Wed, 30 Jan 2019 21:15:46 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #21 from tavis.paquette at eventbase.com --- I've seen this issue in about 20 different environments (large and small, all of which were upgraded from 3.x) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:23:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:23:41 +0000 Subject: [Bugs] [Bug 1636246] [GSS] SMBD crashes when streams_xattr VFS is used with Gluster VFS In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1636246 --- Comment #43 from Anjana --- Thank you for the update. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 05:44:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:44:39 +0000 Subject: [Bugs] [Bug 1671207] New: Several fixes on socket pollin and pollout return value Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671207 Bug ID: 1671207 Summary: Several fixes on socket pollin and pollout return value Product: GlusterFS Version: 5 Status: NEW Component: rpc Severity: high Priority: high Assignee: bugs at gluster.org Reporter: mchangir at redhat.com CC: atumball at redhat.com, bugs at gluster.org, zhhuan at gmail.com Depends On: 1666143 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1666143 +++ Description of problem: Found the return value of socket pollin and pollout for rpc message is not correctly handled. One major problem is socket EAGAIN error will be returned all the way back to dispatch handler and confuse user with error message like: [2018-12-29 07:31:41.772310] E [MSGID: 101191] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2019-01-15 07:25:53 IST --- REVIEW: https://review.gluster.org/22043 (socket: fix counting of socket total_bytes_read and total_bytes_write) posted (#1) for review on master by Zhang Huan --- Additional comment from Worker Ant on 2019-01-15 07:27:04 IST --- REVIEW: https://review.gluster.org/22044 (socket: fix issue when socket write return with EAGAIN) posted (#1) for review on master by Zhang Huan --- Additional comment from Worker Ant on 2019-01-15 07:28:10 IST --- REVIEW: https://review.gluster.org/22046 (socket: don't pass return value from protocol handler to event handler) posted (#1) for review on master by Zhang Huan --- Additional comment from Worker Ant on 2019-01-15 07:29:19 IST --- REVIEW: https://review.gluster.org/22045 (socket: fix issue when socket read return with EAGAIN) posted (#1) for review on master by Zhang Huan --- Additional comment from Worker Ant on 2019-01-17 14:00:17 IST --- REVIEW: https://review.gluster.org/22043 (socket: fix counting of socket total_bytes_read and total_bytes_write) merged (#2) on master by Raghavendra G --- Additional comment from Worker Ant on 2019-01-17 14:00:40 IST --- REVIEW: https://review.gluster.org/22044 (socket: fix issue when socket write return with EAGAIN) merged (#2) on master by Raghavendra G --- Additional comment from Worker Ant on 2019-01-22 12:23:12 IST --- REVIEW: https://review.gluster.org/22045 (socket: fix issue when socket read return with EAGAIN) merged (#2) on master by Amar Tumballi --- Additional comment from Worker Ant on 2019-01-22 12:30:25 IST --- REVIEW: https://review.gluster.org/22046 (socket: don't pass return value from protocol handler to event handler) merged (#3) on master by Amar Tumballi Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 [Bug 1666143] Several fixes on socket pollin and pollout return value -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:44:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:44:39 +0000 Subject: [Bugs] [Bug 1666143] Several fixes on socket pollin and pollout return value In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1666143 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1671207 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671207 [Bug 1671207] Several fixes on socket pollin and pollout return value -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:46:54 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:46:54 +0000 Subject: [Bugs] [Bug 1671208] New: io-cache: conditionally ref dict if non-NULL Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671208 Bug ID: 1671208 Summary: io-cache: conditionally ref dict if non-NULL Product: GlusterFS Version: mainline Status: NEW Component: io-cache Assignee: bugs at gluster.org Reporter: mchangir at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: unconditional ref of dict causes "dict is NULL" logs to be printed -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:49:36 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:49:36 +0000 Subject: [Bugs] [Bug 1671208] io-cache: conditionally ref dict if non-NULL In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671208 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22126 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:49:37 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:49:37 +0000 Subject: [Bugs] [Bug 1671208] io-cache: conditionally ref dict if non-NULL In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671208 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22126 (io-cache: conditionally get ref to dict) posted (#1) for review on master by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:51:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:51:48 +0000 Subject: [Bugs] [Bug 1671210] New: io-cache: conditionally ref dict if non-NULL Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671210 Bug ID: 1671210 Summary: io-cache: conditionally ref dict if non-NULL Product: GlusterFS Version: 5 Status: NEW Component: io-cache Assignee: bugs at gluster.org Reporter: mchangir at redhat.com CC: bugs at gluster.org Depends On: 1671208 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1671208 +++ Description of problem: unconditional ref of dict causes "dict is NULL" logs to be printed --- Additional comment from Worker Ant on 2019-01-31 11:19:37 IST --- REVIEW: https://review.gluster.org/22126 (io-cache: conditionally get ref to dict) posted (#1) for review on master by Milind Changire Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671208 [Bug 1671208] io-cache: conditionally ref dict if non-NULL -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:51:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:51:48 +0000 Subject: [Bugs] [Bug 1671208] io-cache: conditionally ref dict if non-NULL In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671208 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1671210 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671210 [Bug 1671210] io-cache: conditionally ref dict if non-NULL -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:57:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:57:03 +0000 Subject: [Bugs] [Bug 1671210] io-cache: conditionally ref dict if non-NULL In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671210 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22127 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:57:04 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:57:04 +0000 Subject: [Bugs] [Bug 1671210] io-cache: conditionally ref dict if non-NULL In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671210 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22127 (io-cache: conditionally get ref to dict) posted (#1) for review on release-5 by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 05:58:39 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 05:58:39 +0000 Subject: [Bugs] [Bug 1671213] New: core: move "dict is NULL" logs to DEBUG log level Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671213 Bug ID: 1671213 Summary: core: move "dict is NULL" logs to DEBUG log level Product: GlusterFS Version: 5 Status: NEW Component: core Assignee: bugs at gluster.org Reporter: mchangir at redhat.com CC: bugs at gluster.org Target Milestone: --- Classification: Community Description of problem: too many "dict is NULL" get printed if dict_ref() and dict_unref() are passed a NULL pointer -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 06:02:33 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:02:33 +0000 Subject: [Bugs] [Bug 1671213] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671213 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22128 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 06:02:34 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:02:34 +0000 Subject: [Bugs] [Bug 1671213] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671213 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22128 (core: move \"dict is NULL\" logs to DEBUG log level) posted (#1) for review on master by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 06:04:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:04:00 +0000 Subject: [Bugs] [Bug 1671213] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671213 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Version|5 |mainline -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 06:04:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:04:55 +0000 Subject: [Bugs] [Bug 1671217] New: core: move "dict is NULL" logs to DEBUG log level Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671217 Bug ID: 1671217 Summary: core: move "dict is NULL" logs to DEBUG log level Product: GlusterFS Version: 5 Status: NEW Component: core Assignee: bugs at gluster.org Reporter: mchangir at redhat.com CC: bugs at gluster.org Depends On: 1671213 Target Milestone: --- Classification: Community +++ This bug was initially created as a clone of Bug #1671213 +++ Description of problem: too many "dict is NULL" get printed if dict_ref() and dict_unref() are passed a NULL pointer --- Additional comment from Worker Ant on 2019-01-31 11:32:34 IST --- REVIEW: https://review.gluster.org/22128 (core: move \"dict is NULL\" logs to DEBUG log level) posted (#1) for review on master by Milind Changire Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671213 [Bug 1671213] core: move "dict is NULL" logs to DEBUG log level -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 06:04:55 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:04:55 +0000 Subject: [Bugs] [Bug 1671213] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671213 Milind Changire changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1671217 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671217 [Bug 1671217] core: move "dict is NULL" logs to DEBUG log level -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 06:05:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:05:59 +0000 Subject: [Bugs] [Bug 1671217] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671217 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22129 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 06:06:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:06:00 +0000 Subject: [Bugs] [Bug 1671217] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671217 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |POST --- Comment #1 from Worker Ant --- REVIEW: https://review.gluster.org/22129 (core: move \"dict is NULL\" logs to DEBUG log level) posted (#1) for review on release-5 by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 06:18:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:18:47 +0000 Subject: [Bugs] [Bug 1670253] Writes on Gluster 5 volumes fail with EIO when "cluster.consistent-metadata" is set In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670253 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed| |2019-01-31 06:18:47 --- Comment #2 from Worker Ant --- REVIEW: https://review.gluster.org/22110 (readdir-ahead: do not zero-out iatt in fop cbk) merged (#3) on master by Raghavendra G -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 06:24:13 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 06:24:13 +0000 Subject: [Bugs] [Bug 1668190] Block hosting volume deletion via heketi-cli failed with error "target is busy" but deleted from gluster backend In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1668190 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|POST |CLOSED Resolution|--- |NEXTRELEASE Last Closed|2019-01-24 06:55:17 |2019-01-31 06:24:13 --- Comment #4 from Worker Ant --- REVIEW: https://review.gluster.org/22108 (core: heketi-cli is throwing error \"target is busy\") merged (#5) on master by Amar Tumballi -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 07:19:01 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 07:19:01 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22130 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 07:19:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 07:19:02 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #527 from Worker Ant --- REVIEW: https://review.gluster.org/22130 (syncop: remove unnecessary call to gf_backtrace_save()) posted (#1) for review on master by Xavi Hernandez -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 09:44:41 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 09:44:41 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #22 from Digiteyes --- We have not upgraded from 3.x , we have fresh install of 5.x and have same issue -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 09:52:00 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 09:52:00 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Nithya Balachandran changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED CC| |nbalacha at redhat.com Version|mainline |5 Assignee|bugs at gluster.org |mchangir at redhat.com --- Comment #23 from Nithya Balachandran --- Corrected the version and assigned this to Milind to backport the relevant patches to release-5. As per an email discussion, he confirmed that the following patches are required to fix the flood of "Failed to dispatch handler" logs. https://review.gluster.org/#/c/glusterfs/+/22044 https://review.gluster.org/#/c/glusterfs/+/22046/ -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 09:55:22 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 09:55:22 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #24 from Nithya Balachandran --- (In reply to David E. Smith from comment #15) > I'm having what appears to be the same issue. Started when I upgraded from > 3.12 to 5.2 a few weeks back, and the subsequent upgrade to 5.3 did not > resolve the problem. > > My servers (two, in a 'replica 2' setup) publish two volumes. One is Web > site content, about 110GB; the other is Web config files, only a few > megabytes. (Wasn't worth building extra servers for that second volume.) > FUSE clients have been crashing on the larger volume every three or four > days. > > The client's logs show many hundreds of instances of this (I don't know if > it's related): > [2019-01-29 08:14:16.542674] W [dict.c:761:dict_ref] > (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7384) > [0x7fa171ead384] > -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xae3e) > [0x7fa1720bee3e] -->/lib64/libglusterfs.so.0(dict_ref+0x5d) [0x7fa1809cc2ad] > ) 0-dict: dict is NULL [Invalid argument] > > Then, when the client's glusterfs process crashes, this is logged: > > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 871 times between [2019-01-29 08:12:48.390535] and > [2019-01-29 08:14:17.100279] > pending frames: > frame : type(1) op(LOOKUP) > frame : type(1) op(LOOKUP) > frame : type(0) op(0) > frame : type(0) op(0) > patchset: git://git.gluster.org/glusterfs.git > signal received: 11 > time of crash: > 2019-01-29 08:14:17 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 5.3 > /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] > /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] > /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] > /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] > /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] > /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] > /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] > /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] > /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] > /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] > --------- > > > > Info on the volumes themselves, gathered from one of my servers: > > [davidsmith at wuit-s-10889 ~]$ sudo gluster volume info all > > Volume Name: web-config > Type: Replicate > Volume ID: 6c5dce6e-e64e-4a6d-82b3-f526744b463d > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: 172.23.128.26:/data/web-config > Brick2: 172.23.128.27:/data/web-config > Options Reconfigured: > performance.client-io-threads: off > nfs.disable: on > transport.address-family: inet > server.event-threads: 4 > client.event-threads: 4 > cluster.min-free-disk: 1 > cluster.quorum-count: 2 > cluster.quorum-type: fixed > network.ping-timeout: 10 > auth.allow: * > performance.readdir-ahead: on > > Volume Name: web-content > Type: Replicate > Volume ID: fcabc15f-0cec-498f-93c4-2d75ad915730 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: 172.23.128.26:/data/web-content > Brick2: 172.23.128.27:/data/web-content > Options Reconfigured: > network.ping-timeout: 10 > cluster.quorum-type: fixed > cluster.quorum-count: 2 > performance.readdir-ahead: on > auth.allow: * > cluster.min-free-disk: 1 > client.event-threads: 4 > server.event-threads: 4 > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > performance.cache-size: 4GB > > > > gluster> volume status all detail > Status of volume: web-config > ----------------------------------------------------------------------------- > - > Brick : Brick 172.23.128.26:/data/web-config > TCP Port : 49152 > RDMA Port : 0 > Online : Y > Pid : 5612 > File System : ext3 > Device : /dev/sdb1 > Mount Options : rw,seclabel,relatime,data=ordered > Inode Size : 256 > Disk Space Free : 135.9GB > Total Disk Space : 246.0GB > Inode Count : 16384000 > Free Inodes : 14962279 > ----------------------------------------------------------------------------- > - > Brick : Brick 172.23.128.27:/data/web-config > TCP Port : 49152 > RDMA Port : 0 > Online : Y > Pid : 5540 > File System : ext3 > Device : /dev/sdb1 > Mount Options : rw,seclabel,relatime,data=ordered > Inode Size : 256 > Disk Space Free : 135.9GB > Total Disk Space : 246.0GB > Inode Count : 16384000 > Free Inodes : 14962277 > > Status of volume: web-content > ----------------------------------------------------------------------------- > - > Brick : Brick 172.23.128.26:/data/web-content > TCP Port : 49153 > RDMA Port : 0 > Online : Y > Pid : 5649 > File System : ext3 > Device : /dev/sdb1 > Mount Options : rw,seclabel,relatime,data=ordered > Inode Size : 256 > Disk Space Free : 135.9GB > Total Disk Space : 246.0GB > Inode Count : 16384000 > Free Inodes : 14962279 > ----------------------------------------------------------------------------- > - > Brick : Brick 172.23.128.27:/data/web-content > TCP Port : 49153 > RDMA Port : 0 > Online : Y > Pid : 5567 > File System : ext3 > Device : /dev/sdb1 > Mount Options : rw,seclabel,relatime,data=ordered > Inode Size : 256 > Disk Space Free : 135.9GB > Total Disk Space : 246.0GB > Inode Count : 16384000 > Free Inodes : 14962277 > > > I have a couple of core files that appear to be from this, but I'm not much > of a developer (haven't touched C in fifteen years) so I don't know what to > do with them that would be of value in this case. Please file a separate BZ for the crashes and provide the bt and corefiles. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 10:06:59 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 10:06:59 +0000 Subject: [Bugs] [Bug 1671014] gluster-fuse seg fault PTHREAD_MUTEX_TYPE_ELISION In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671014 joao.bauto at neuro.fchampalimaud.org changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|gluster-fuse seg fault |gluster-fuse seg fault |Transport endpoint is not |PTHREAD_MUTEX_TYPE_ELISION |connected | -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 10:15:48 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 10:15:48 +0000 Subject: [Bugs] [Bug 1671213] core: move "dict is NULL" logs to DEBUG log level In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671213 Amar Tumballi changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |atumball at redhat.com --- Comment #2 from Amar Tumballi --- Can you post some logs? Ideally, if dict is NULL during a 'ref()/unref()', it is a debug hint for developer during development. Surely should be a DEBUG log in release branch. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 10:35:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 10:35:47 +0000 Subject: [Bugs] [Bug 1314508] tar complains: : file changed as we read it In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1314508 nchilaka changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nchilaka at redhat.c | |om) | -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 11:29:44 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 11:29:44 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22134 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 11:29:46 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 11:29:46 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |POST --- Comment #25 from Worker Ant --- REVIEW: https://review.gluster.org/22134 (socket: fix issue when socket write return with EAGAIN) posted (#1) for review on release-5 by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 11:31:02 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 11:31:02 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 Worker Ant changed: What |Removed |Added ---------------------------------------------------------------------------- External Bug ID| |Gluster.org Gerrit 22135 -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 11:31:03 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 11:31:03 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #26 from Worker Ant --- REVIEW: https://review.gluster.org/22135 (socket: don't pass return value from protocol handler to event handler) posted (#1) for review on release-5 by Milind Changire -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 11:48:23 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 11:48:23 +0000 Subject: [Bugs] [Bug 1670382] parallel-readdir prevents directories and files listing In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1670382 --- Comment #1 from Marcin --- Maybe someone knows some way to get around this problem? (Of course, except disabling it in the gluster configuration) The parallel feature significantly improves the reading speed from our backup system. We planned to update glusterfs this month from version 3.10.3 to 5.x due to a significant number of bugs, so I would be grateful for the information. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 15:42:15 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 15:42:15 +0000 Subject: [Bugs] [Bug 1193929] GlusterFS can be improved In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1193929 --- Comment #528 from Worker Ant --- REVIEW: https://review.gluster.org/22130 (syncop: remove unnecessary call to gf_backtrace_save()) merged (#2) on master by Shyamsundar Ranganathan -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 18:00:40 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 18:00:40 +0000 Subject: [Bugs] [Bug 1313567] flooding of "dict is NULL" logging In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1313567 --- Comment #16 from Artem Russakovskii --- Damn, you weren't kidding, I wish I saw these bug reports before I updated from rock solid 4.1. Less than 24 hours after upgrading, I already got a crash that you referenced: [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3" repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31 09:38:03.958061] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 72 times between [2019-01-31 09:37:53.746741] and [2019-01-31 09:38:04.696993] pending frames: frame : type(1) op(READ) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-31 09:38:04 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6] /lib64/libc.so.6(+0x36160)[0x7fccd622d160] /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0] /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa] /lib64/libc.so.6(+0x2e772)[0x7fccd6225772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3] /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559] /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f] --------- -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 18:08:42 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 18:08:42 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #27 from Artem Russakovskii --- I wish I saw this bug report before I updated from rock solid 4.1 to 5.3. Less than 24 hours after upgrading, I already got a crash and had to unmount, kill gluster, and remount: [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3" repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31 09:38:03.958061] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 72 times between [2019-01-31 09:37:53.746741] and [2019-01-31 09:38:04.696993] pending frames: frame : type(1) op(READ) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-31 09:38:04 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6] /lib64/libc.so.6(+0x36160)[0x7fccd622d160] /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0] /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa] /lib64/libc.so.6(+0x2e772)[0x7fccd6225772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3] /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559] /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f] --------- Do the pending patches fix the crash or only the repeated warnings? I'm running glusterfs on OpenSUSE 15.0 installed via http://download.opensuse.org/repositories/home:/glusterfs:/Leap15-5/openSUSE_Leap_15.0/, not too sure how to make it core dump. If it's not fixed by the patches above, has anyone already opened a ticket for the crashes that I can join and monitor? This is going to create a massive problem for us since production systems are crashing. Thanks. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 21:59:49 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 21:59:49 +0000 Subject: [Bugs] [Bug 1671556] New: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Bug ID: 1671556 Summary: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' Product: GlusterFS Version: 5 Hardware: x86_64 OS: Linux Status: NEW Component: fuse Severity: urgent Assignee: bugs at gluster.org Reporter: desmith at wustl.edu CC: bugs at gluster.org Target Milestone: --- Classification: Community This is a re-post of my FUSE crash report from BZ1651246. That issue is for a crash in the FUSE client. Mine is too, but I was asked in that bug to open a new issue, so here you go. :) My servers (two, in a 'replica 2' setup) publish two volumes. One is Web site content, about 110GB; the other is Web config files, only a few megabytes. (Wasn't worth building extra servers for that second volume.) FUSE clients have been crashing on the larger volume every three or four days. I can't reproduce this on-demand, unfortunately, but I've got several cores from previous crashes that may be of value to you. I'm using Gluster 5.3 from the RPMs provided by the CentOS Storage SIG, on a Red Hat Enterprise Linux 7.x system. The client's logs show many hundreds of instances of this (I don't know if it's related): [2019-01-29 08:14:16.542674] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7384) [0x7fa171ead384] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xae3e) [0x7fa1720bee3e] -->/lib64/libglusterfs.so.0(dict_ref+0x5d) [0x7fa1809cc2ad] ) 0-dict: dict is NULL [Invalid argument] Then, when the client's glusterfs process crashes, this is logged: The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 871 times between [2019-01-29 08:12:48.390535] and [2019-01-29 08:14:17.100279] pending frames: frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2019-01-29 08:14:17 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /lib64/libglusterfs.so.0(+0x26610)[0x7fa1809d8610] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fa1809e2b84] /lib64/libc.so.6(+0x36280)[0x7fa17f03c280] /lib64/libglusterfs.so.0(+0x3586d)[0x7fa1809e786d] /lib64/libglusterfs.so.0(+0x370a2)[0x7fa1809e90a2] /lib64/libglusterfs.so.0(inode_forget_with_unref+0x46)[0x7fa1809e9f96] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x85bd)[0x7fa177dae5bd] /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so(+0x1fd7a)[0x7fa177dc5d7a] /lib64/libpthread.so.0(+0x7dd5)[0x7fa17f83bdd5] /lib64/libc.so.6(clone+0x6d)[0x7fa17f103ead] --------- Info on the volumes themselves, gathered from one of my servers: [davidsmith at wuit-s-10889 ~]$ sudo gluster volume info all Volume Name: web-config Type: Replicate Volume ID: 6c5dce6e-e64e-4a6d-82b3-f526744b463d Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-config Brick2: 172.23.128.27:/data/web-config Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet server.event-threads: 4 client.event-threads: 4 cluster.min-free-disk: 1 cluster.quorum-count: 2 cluster.quorum-type: fixed network.ping-timeout: 10 auth.allow: * performance.readdir-ahead: on Volume Name: web-content Type: Replicate Volume ID: fcabc15f-0cec-498f-93c4-2d75ad915730 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 172.23.128.26:/data/web-content Brick2: 172.23.128.27:/data/web-content Options Reconfigured: network.ping-timeout: 10 cluster.quorum-type: fixed cluster.quorum-count: 2 performance.readdir-ahead: on auth.allow: * cluster.min-free-disk: 1 client.event-threads: 4 server.event-threads: 4 transport.address-family: inet nfs.disable: on performance.client-io-threads: off performance.cache-size: 4GB gluster> volume status all detail Status of volume: web-config ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5612 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-config TCP Port : 49152 RDMA Port : 0 Online : Y Pid : 5540 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 Status of volume: web-content ------------------------------------------------------------------------------ Brick : Brick 172.23.128.26:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5649 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962279 ------------------------------------------------------------------------------ Brick : Brick 172.23.128.27:/data/web-content TCP Port : 49153 RDMA Port : 0 Online : Y Pid : 5567 File System : ext3 Device : /dev/sdb1 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 135.9GB Total Disk Space : 246.0GB Inode Count : 16384000 Free Inodes : 14962277 I'll attach a couple of the core files generated by the crashing glusterfs instances, size limits permitting (they range from 3 to 8 GB). If I can't attach them, I'll find somewhere to host them. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 22:02:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 22:02:08 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 David E. Smith changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |desmith at wustl.edu Mount Type|--- |fuse Blocks| |1667103 (glusterfs-5.4) Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 [Bug 1667103] GlusterFS 5.4 tracker -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 22:02:08 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 22:02:08 +0000 Subject: [Bugs] [Bug 1667103] GlusterFS 5.4 tracker In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1667103 David E. Smith changed: What |Removed |Added ---------------------------------------------------------------------------- Depends On| |1671556 Referenced Bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 22:15:29 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 22:15:29 +0000 Subject: [Bugs] [Bug 1651246] Failed to dispatch handler In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1651246 --- Comment #28 from David E. Smith --- As requested, opened a new bug report for my crashes, https://bugzilla.redhat.com/show_bug.cgi?id=1671556 . Links to cores will be added there Really Soon. -- You are receiving this mail because: You are on the CC list for the bug. From bugzilla at redhat.com Thu Jan 31 22:26:25 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 22:26:25 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 Artem Russakovskii changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |archon810 at gmail.com --- Comment #1 from Artem Russakovskii --- Also reposting my comment from https://bugzilla.redhat.com/show_bug.cgi?id=1651246. I wish I saw this bug report before I updated from rock solid 4.1 to 5.3. Less than 24 hours after upgrading, I already got a crash and had to unmount, kill gluster, and remount: [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fcccafcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument] The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3" repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31 09:38:03.958061] The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler" repeated 72 times between [2019-01-31 09:37:53.746741] and [2019-01-31 09:38:04.696993] pending frames: frame : type(1) op(READ) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 6 time of crash: 2019-01-31 09:38:04 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6] /lib64/libc.so.6(+0x36160)[0x7fccd622d160] /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0] /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1] /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa] /lib64/libc.so.6(+0x2e772)[0x7fccd6225772] /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8] /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778] /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820] /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2] /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3] /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559] /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f] --------- Do the pending patches fix the crash or only the repeated warnings? I'm running glusterfs on OpenSUSE 15.0 installed via http://download.opensuse.org/repositories/home:/glusterfs:/Leap15-5/openSUSE_Leap_15.0/, not too sure how to make it core dump. If it's not fixed by the patches above, has anyone already opened a ticket for the crashes that I can join and monitor? This is going to create a massive problem for us since production systems are crashing. Thanks. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From bugzilla at redhat.com Thu Jan 31 22:31:47 2019 From: bugzilla at redhat.com (bugzilla at redhat.com) Date: Thu, 31 Jan 2019 22:31:47 +0000 Subject: [Bugs] [Bug 1671556] glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' In-Reply-To: References: Message-ID: https://bugzilla.redhat.com/show_bug.cgi?id=1671556 --- Comment #2 from David E. Smith --- Actually, I ran the cores through strings and grepped for a few things like passwords -- as you'd expect from a memory dump from a Web server, there's a log of sensitive information in there. Is there a safe/acceptable way to send the cores only to developers that can use them, or otherwise not have to make them publicly available while still letting the Gluster devs benefit from analyzing them? -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug. From shaik.salam at tcs.com Wed Jan 23 12:42:39 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Wed, 23 Jan 2019 12:42:39 -0000 Subject: [Bugs] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume In-Reply-To: References: Message-ID: Hi, We are facing also following issue on openshift origin while we are creating pvc for pods. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. Please find heketidb dump and log [negroni] Completed 429 Too Many Requests in 250.763?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 169.08?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 148.125?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 496.624?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 101.673?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 209.681?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 103.595?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 297.594?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 96.75?s [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 477.007?s [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 165.38?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 488.253?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 171.836?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 208.59?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 125.141?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.687?s [negroni] Started POST /volumes BR Salam From: Shaik Salam/HYD/TCS To: bugs at gluster.org, gluster-users at gluster.org Date: 01/21/2019 09:57 PM Subject: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Hi, We are facing also similar issue on openshift origin while we are creating pvc for pods. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. heketi looks fine. [negroni] Completed 200 OK in 116.41?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 124.552?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 128.632?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 134.856?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 123.378?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 134.202?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 120.114?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 141.04?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 122.628?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 150.651?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 116.978?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 110.189?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 226.655?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 129.487?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 116.809?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 118.697?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 112.947?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 134.569?s [negroni] Started GET /queue/756488c7baccc2a64252b1a82b2c70b3 [negroni] Completed 200 OK in 119.018?s BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: heketi-gluster.db.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: heketi.log.txt URL: From shaik.salam at tcs.com Wed Jan 23 12:49:43 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Wed, 23 Jan 2019 12:49:43 -0000 Subject: [Bugs] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Message-ID: Hi, We are facing also following issue on openshift origin while we are creating pvc for pods. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. Please find heketidb dump and log [negroni] Completed 429 Too Many Requests in 250.763?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 169.08?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 148.125?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 496.624?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 101.673?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 209.681?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 103.595?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 297.594?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 96.75?s [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 477.007?s [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 165.38?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 488.253?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 171.836?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 208.59?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 125.141?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.687?s [negroni] Started POST /volumes BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: heketi-gluster.db.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: heketi.log.txt URL: From shaik.salam at tcs.com Thu Jan 24 05:58:14 2019 From: shaik.salam at tcs.com (Shaik Salam) Date: Thu, 24 Jan 2019 05:58:14 -0000 Subject: [Bugs] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume In-Reply-To: References: Message-ID: Hi Surya, Could you please help us to resolve below issue (at lease workaround for creating volume) Attached db dump and log. Please let me know any other things need to check. Please guide us. BR Salam From: Shaik Salam/HYD/TCS To: "Amar Tumballi Suryanarayan" , bugs at gluster.org, "gluster-users at gluster.org List" Cc: "Murali Kottakota" , "Sanju Rakonde" Date: 01/23/2019 06:19 PM Subject: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Hi, We are facing also following issue on openshift origin while we are creating pvc for pods. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. Please find heketidb dump and log [negroni] Completed 429 Too Many Requests in 250.763?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 169.08?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 148.125?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 496.624?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 101.673?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 209.681?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 103.595?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 297.594?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 96.75?s [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 477.007?s [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 165.38?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 488.253?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 171.836?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 208.59?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 125.141?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.687?s [negroni] Started POST /volumes BR Salam =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: heketi-gluster.db.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: heketi.log.txt URL: