From hgowtham at redhat.com Thu Aug 1 06:51:48 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Thu, 1 Aug 2019 12:21:48 +0530 Subject: [Gluster-users] Release 6.5: Expected tagging on 5th August Message-ID: Hi, Expected tagging date for release-6.5 is on August, 5th, 2019. Please ensure required patches are backported and also are passing regressions and are appropriately reviewed for easy merging and tagging on the date. -- Regards, Hari Gowtham. From hgowtham at redhat.com Thu Aug 1 08:35:02 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Thu, 1 Aug 2019 14:05:02 +0530 Subject: [Gluster-users] Release 5.9: Expected tagging on 5th August Message-ID: Hi, Expected tagging date for release-5.9 is on August, 5th, 2019. Please ensure required patches are backported and also are passing regressions and are appropriately reviewed for easy merging and tagging on the date. -- Regards, Hari Gowtham. From skoduri at redhat.com Thu Aug 1 17:06:09 2019 From: skoduri at redhat.com (Soumya Koduri) Date: Thu, 1 Aug 2019 22:36:09 +0530 Subject: [Gluster-users] Release 6.5: Expected tagging on 5th August In-Reply-To: References: Message-ID: <0ef5691c-4538-9bb9-76fa-e1c4eaabb54f@redhat.com> Hi Hari, [1] is a critical patch which addresses issue affecting upcall processing by applications such as NFS-Ganesha. As soon as it gets merged in master, I shall backport it to release-7/6/5 branches. Kindly consider the same. Thanks, Soumya [1] https://review.gluster.org/#/c/glusterfs/+/23108/ On 8/1/19 12:21 PM, Hari Gowtham wrote: > Hi, > > Expected tagging date for release-6.5 is on August, 5th, 2019. > > Please ensure required patches are backported and also are passing > regressions and are appropriately reviewed for easy merging and tagging > on the date. > From hgowtham at redhat.com Fri Aug 2 01:14:25 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Fri, 2 Aug 2019 06:44:25 +0530 Subject: [Gluster-users] Release 6.5: Expected tagging on 5th August In-Reply-To: <0ef5691c-4538-9bb9-76fa-e1c4eaabb54f@redhat.com> References: <0ef5691c-4538-9bb9-76fa-e1c4eaabb54f@redhat.com> Message-ID: Hi Soumya, Thanks for the update. Will keep an eye on it. Regards, Hari. On Thu, 1 Aug, 2019, 10:36 PM Soumya Koduri, wrote: > Hi Hari, > > [1] is a critical patch which addresses issue affecting upcall > processing by applications such as NFS-Ganesha. As soon as it gets > merged in master, I shall backport it to release-7/6/5 branches. Kindly > consider the same. > > Thanks, > Soumya > > [1] https://review.gluster.org/#/c/glusterfs/+/23108/ > > On 8/1/19 12:21 PM, Hari Gowtham wrote: > > Hi, > > > > Expected tagging date for release-6.5 is on August, 5th, 2019. > > > > Please ensure required patches are backported and also are passing > > regressions and are appropriately reviewed for easy merging and tagging > > on the date. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From florian at bezdeka.de Sat Aug 3 09:38:46 2019 From: florian at bezdeka.de (florian at bezdeka.de) Date: Sat, 03 Aug 2019 11:38:46 +0200 Subject: [Gluster-users] Announcing Gluster release 4.1.10 In-Reply-To: References: Message-ID: <398e32e8990d49a1ab05562054f68bdb@bezdeka.de> Hi, still waiting for the CentOS packages to be released. The cbs build available at [1] has no release tag set. Maybe someone forgot to add the right tag? Regards, Florian [1] https://cbs.centos.org/koji/buildinfo?buildID=26337 From ndevos at redhat.com Mon Aug 5 08:10:56 2019 From: ndevos at redhat.com (Niels de Vos) Date: Mon, 5 Aug 2019 10:10:56 +0200 Subject: [Gluster-users] Announcing Gluster release 4.1.10 In-Reply-To: <398e32e8990d49a1ab05562054f68bdb@bezdeka.de> References: <398e32e8990d49a1ab05562054f68bdb@bezdeka.de> Message-ID: <20190805081056.GA22831@ndevos-x270> On Sat, Aug 03, 2019 at 11:38:46AM +0200, florian at bezdeka.de wrote: > Hi, > > still waiting for the CentOS packages to be released. > > The cbs build available at [1] has no release tag set. > Maybe someone forgot to add the right tag? Somebody should run some tests and inform us that the packages are working as expected. See https://lists.gluster.org/pipermail/packaging/2019-July/000785.html for a few details. Thanks, Niels > > Regards, > Florian > > > [1] https://cbs.centos.org/koji/buildinfo?buildID=26337 > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From pgurusid at redhat.com Tue Aug 6 05:33:40 2019 From: pgurusid at redhat.com (pgurusid at redhat.com) Date: Tue, 06 Aug 2019 05:33:40 +0000 Subject: [Gluster-users] Canceled event: Gluster Community Meeting (APAC friendly hours) @ Every 2 weeks from 11:30am to 12:30pm on Tuesday 15 times (IST) (gluster-users@gluster.org) Message-ID: <000000000000ce6c09058f6c2ad7@google.com> This event has been canceled. Title: Gluster Community Meeting (APAC friendly hours) Bridge: https://bluejeans.com/836554017 Meeting minutes: https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both Previous Meeting notes: http://github.com/gluster/community When: Every 2 weeks from 11:30am to 12:30pm on Tuesday 15 times India Standard Time - Kolkata Where: https://bluejeans.com/836554017 Calendar: gluster-users at gluster.org Who: * pgurusid at redhat.com - organizer * gluster-users at gluster.org * maintainers at gluster.org * gluster-devel at gluster.org * ranaraya at redhat.com * khiremat at redhat.com * dcunningham at voisonics.com * rwareing at fb.com * kdhananj at redhat.com * pkarampu at redhat.com * mark.boulton at uwa.edu.au * sunkumar at redhat.com * gabriel.lindeborg at svenskaspel.se * m.vrgotic at activevideo.com * david.spisla at iternity.com * sthomas at rpstechnologysolutions.co.uk * javico at paradigmadigital.com * philip.ruenagel at gmail.com * pauyeung at connexity.com * Max de Graaf * sstephen at redhat.com * jpark at dexyp.com * spalai at redhat.com * rouge2507 at gmail.com * spentaparthi at idirect.net * duprel at email.sc.edu * dan at clough.xyz * m.ragusa at eurodata.de * barchu02 at unm.edu * brian.riddle at storagecraft.com * ryan_groth at wgbh.org * amnerip at fb.com * dph at fb.com Invitation from Google Calendar: https://www.google.com/calendar/ You are receiving this courtesy email at the account gluster-users at gluster.org because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn more at https://support.google.com/calendar/answer/37135#forwarding -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 5622 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: invite.ics Type: application/ics Size: 5724 bytes Desc: not available URL: From pgurusid at redhat.com Tue Aug 6 05:33:48 2019 From: pgurusid at redhat.com (pgurusid at redhat.com) Date: Tue, 06 Aug 2019 05:33:48 -0000 Subject: [Gluster-users] Cancelled: Gluster Community Meeting (APAC friendly hours) @ Monday, 13 May 2019 Message-ID: <2048941194.1339.1565069627132.JavaMail.yahoo@tardis002.cal.bf1.yahoo.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: invite.ics Type: application/ics Size: 2589 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 2589 bytes Desc: not available URL: From pgurusid at redhat.com Tue Aug 6 05:33:48 2019 From: pgurusid at redhat.com (pgurusid at redhat.com) Date: Tue, 06 Aug 2019 05:33:48 -0000 Subject: [Gluster-users] Cancelled: Gluster Community Meeting (APAC friendly hours) @ Monday, 13 May 2019 Message-ID: <968170451.1353.1565069627806.JavaMail.yahoo@tardis002.cal.bf1.yahoo.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: invite.ics Type: application/ics Size: 2587 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 2587 bytes Desc: not available URL: From pgurusid at redhat.com Tue Aug 6 05:33:58 2019 From: pgurusid at redhat.com (pgurusid at redhat.com) Date: Tue, 06 Aug 2019 05:33:58 -0000 Subject: [Gluster-users] Cancelled: Gluster Community Meeting (APAC friendly hours) @ Monday, 13 May 2019 Message-ID: <1935493240.645.1565069632789.JavaMail.yahoo@tardis97.cal.gq1.yahoo.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: invite.ics Type: application/ics Size: 2583 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 2583 bytes Desc: not available URL: From nux at li.nux.ro Wed Aug 7 16:23:10 2019 From: nux at li.nux.ro (Nux!) Date: Wed, 07 Aug 2019 17:23:10 +0100 Subject: [Gluster-users] Continue to work in "degraded mode" (missing brick) Message-ID: <0c6727a9cd614691e61a7666134a9309@li.nux.ro> Hello, I'm testing a replicated volume with 3 bricks. I've killed a brick, but the volume is still mounted and can see the files from the bricks that are still online and can do operations on them. What I cannot do is create new files in the volume, e.g.: dd: failed to open ?test1000?: Transport endpoint is not connected Is there a way to make this volume continue to work while one of the bricks is offline? There is still space available in the remaining bricks, shouldn't it try to use it? Regards -- Sent from the Delta quadrant using Borg technology! From kkeithle at redhat.com Wed Aug 7 17:38:09 2019 From: kkeithle at redhat.com (Kaleb Keithley) Date: Wed, 7 Aug 2019 13:38:09 -0400 Subject: [Gluster-users] Important: Debian and Ubuntu packages are changing Message-ID: *TL;DNR: *updates from glusterfs-5.8 to glusterfs-5.9 and from glusterfs-6.4 to glusterfs-6.5, ? using the package repos on https://download.gluster.org or the Gluster PPA on Launchpad? on buster, bullseye/sid, and some Ubuntu releases may not work, or may not work smoothly. Consider yourself warned. Plan accordingly. *Longer Answer*: updates from glusterfs-5.8 to glusterfs-5.9 and from glusterfs-6.4 to glusterfs-6.5, ? using the package repos on https://download.gluster.org or the Gluster PPA on Launchpad ? on buster, bullseye, and some Ubuntu releases may not work, or may not work smoothly. *Why*: The original packaging bits were contributed by the Debian maintainer of GlusterFS. For those that know Debian packaging, these did not follow normal Debian packaging conventions and best practices. Recently ? for some definition of recent ? the powers that be in Debian apparentl insisted that the packaging actually start to follow the conventions and best practices, and the packaging bits were rewritten for Debian. The only problem is that nobody bothered to notify the Gluster Community that this was happening. Nor did they send their new bits to GlusterFS. We were left to find out about it the hard way. *The Issue*: people who have used the packages from https://download.gluster.org are experiencing issues updating other software that depends on glusterfs. *The Change*: Gluster Community packages will now be built using packaging bits derived from the Debian packaging bits, which now follow Debian packaging conventions and best practices. *Conclusion*: This may be painful, but it's better in the long run for everyone. The volunteers who generously build packages in their copious spare time for the community appreciate your patience and understanding. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkeithle at redhat.com Wed Aug 7 19:14:26 2019 From: kkeithle at redhat.com (Kaleb Keithley) Date: Wed, 7 Aug 2019 15:14:26 -0400 Subject: [Gluster-users] Important: Debian and Ubuntu packages are changing In-Reply-To: References: Message-ID: On Wed, Aug 7, 2019 at 1:38 PM Kaleb Keithley wrote: > *... *and some Ubuntu releases > Specifically Ubuntu Disco and Eoan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ravishankar at redhat.com Thu Aug 8 05:54:38 2019 From: ravishankar at redhat.com (Ravishankar N) Date: Thu, 8 Aug 2019 11:24:38 +0530 Subject: [Gluster-users] Continue to work in "degraded mode" (missing brick) In-Reply-To: <0c6727a9cd614691e61a7666134a9309@li.nux.ro> References: <0c6727a9cd614691e61a7666134a9309@li.nux.ro> Message-ID: <5286c1cf-842a-2ed0-0561-2ee115358c12@redhat.com> On 07/08/19 9:53 PM, Nux! wrote: > Hello, > > I'm testing a replicated volume with 3 bricks. I've killed a brick, > but the volume is still mounted and can see the files from the bricks > that are still online and can do operations on them. > What I cannot do is create new files in the volume, e.g.: > > dd: failed to open ?test1000?: Transport endpoint is not connected > > > Is there a way to make this volume continue to work while one of the > bricks is offline? There is still space available in the remaining > bricks, shouldn't it try to use it? If 2 bricks are online and the clients are connected to them, writes should work.? Unless the brick that was down was the only good copy, i.e. the only one that successfully witnessed all previous writes. What version of gluster are you using? Check the mount log for more details. > > Regards > From nux at li.nux.ro Thu Aug 8 07:46:23 2019 From: nux at li.nux.ro (Nux!) Date: Thu, 08 Aug 2019 08:46:23 +0100 Subject: [Gluster-users] Continue to work in "degraded mode" (missing brick) In-Reply-To: <5286c1cf-842a-2ed0-0561-2ee115358c12@redhat.com> References: <0c6727a9cd614691e61a7666134a9309@li.nux.ro> <5286c1cf-842a-2ed0-0561-2ee115358c12@redhat.com> Message-ID: <9e9618f92f0d5d09ef37960fe3c31290@li.nux.ro> Sorry, I meant to say distributed, not replicated! I'm on 6.4 from CentOs7 SIG. I was hoping the volume might still be fully usable write-wise, with files going on the remaining bricks, but it doesn't seem to be the case. --- Sent from the Delta quadrant using Borg technology! On 2019-08-08 06:54, Ravishankar N wrote: > On 07/08/19 9:53 PM, Nux! wrote: >> Hello, >> >> I'm testing a replicated volume with 3 bricks. I've killed a brick, >> but the volume is still mounted and can see the files from the bricks >> that are still online and can do operations on them. >> What I cannot do is create new files in the volume, e.g.: >> >> dd: failed to open ?test1000?: Transport endpoint is not connected >> >> >> Is there a way to make this volume continue to work while one of the >> bricks is offline? There is still space available in the remaining >> bricks, shouldn't it try to use it? > > If 2 bricks are online and the clients are connected to them, writes > should work.? Unless the brick that was down was the only good copy, > i.e. the only one that successfully witnessed all previous writes. > What version of gluster are you using? Check the mount log for more > details. > >> >> Regards >> From nbalacha at redhat.com Thu Aug 8 08:44:04 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Thu, 8 Aug 2019 14:14:04 +0530 Subject: [Gluster-users] Continue to work in "degraded mode" (missing brick) In-Reply-To: <9e9618f92f0d5d09ef37960fe3c31290@li.nux.ro> References: <0c6727a9cd614691e61a7666134a9309@li.nux.ro> <5286c1cf-842a-2ed0-0561-2ee115358c12@redhat.com> <9e9618f92f0d5d09ef37960fe3c31290@li.nux.ro> Message-ID: Hi, This is the expected behaviour for a distribute volume. Files that hash to a brick that is down will not be created. This is to prevent issues in case the file already exists on that brick. To prevent this, please use distribute-replicate volumes. Regards, Nithya On Thu, 8 Aug 2019 at 13:17, Nux! wrote: > Sorry, I meant to say distributed, not replicated! > I'm on 6.4 from CentOs7 SIG. > > I was hoping the volume might still be fully usable write-wise, with > files going on the remaining bricks, but it doesn't seem to be the case. > > --- > Sent from the Delta quadrant using Borg technology! > > On 2019-08-08 06:54, Ravishankar N wrote: > > On 07/08/19 9:53 PM, Nux! wrote: > >> Hello, > >> > >> I'm testing a replicated volume with 3 bricks. I've killed a brick, > >> but the volume is still mounted and can see the files from the bricks > >> that are still online and can do operations on them. > >> What I cannot do is create new files in the volume, e.g.: > >> > >> dd: failed to open ?test1000?: Transport endpoint is not connected > >> > >> > >> Is there a way to make this volume continue to work while one of the > >> bricks is offline? There is still space available in the remaining > >> bricks, shouldn't it try to use it? > > > > If 2 bricks are online and the clients are connected to them, writes > > should work. Unless the brick that was down was the only good copy, > > i.e. the only one that successfully witnessed all previous writes. > > What version of gluster are you using? Check the mount log for more > > details. > > > >> > >> Regards > >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkeithle at redhat.com Fri Aug 9 13:50:59 2019 From: kkeithle at redhat.com (Kaleb Keithley) Date: Fri, 9 Aug 2019 09:50:59 -0400 Subject: [Gluster-users] Important: Debian and Ubuntu packages are changing In-Reply-To: References: Message-ID: *On Thu, Aug 8, 2019 at 4:56 PM Ingo Fischer wrote: > Hi Kaleb, > > I'm currently experiencing this issue while trying to upgrade my Proxmox > servers where gluster is installed too. > > Thank you for the official information for the community, but what > exactly do this mean? > > Will upgrades from 5.8 to 5.9 work or what exactly needs to be done in > order to get the update done? > I expect they will work as well as updating from, e.g., gluster's old style glusterfs_5.4 debs to debian's new style glusterfs_5.5 debs. IOW probably not very well. My guess is that you will probably need to uninstall 5.8 followed by installing 5.9. Here at Red Hat, as one might guess, we don't use a lot of Debian or Ubuntu. My experience with Debian and Ubuntu has been limited to building the packages. (FWIW, in a previous job I used SLES and OpenSuSE, and before that I used Slackware.) These are "community" packages and they're free. I personally do feel like the community really should shoulder some of the burden to test them and report any problems. Give them a try. Let us know what does or doesn't work. And send PRs. Debian Stretch is not affected? > TL;DNR: if it was, I would have said so. ;-) The Debian packager didn't change the packaging on stretch or bionic and xenial. The gluster community packages for those distributions are the same as they've always been. > > Thank you for additional information > > Ingo > > Am 07.08.19 um 19:38 schrieb Kaleb Keithley: > > *TL;DNR: *updates from glusterfs-5.8 to glusterfs-5.9 and from > > glusterfs-6.4 to glusterfs-6.5, ? using the package repos on > > https://download.gluster.org or the Gluster PPA on Launchpad? on > > buster, bullseye/sid, and some Ubuntu releases may not work, or may not > > work smoothly. Consider yourself warned. Plan accordingly. > > > > *Longer Answer*: updates from glusterfs-5.8 to glusterfs-5.9 and from > > glusterfs-6.4 to glusterfs-6.5, ? using the package repos on > > https://download.gluster.org or the Gluster PPA on Launchpad ? on > > buster, bullseye, and some Ubuntu releases may not work, or may not work > > smoothly. > > > > *Why*: The original packaging bits were contributed by the Debian > > maintainer of GlusterFS. For those that know Debian packaging, these did > > not follow normal Debian packaging conventions and best practices. > > Recently ? for some definition of recent ? the powers that be in Debian > > apparentl insisted that the packaging actually start to follow the > > conventions and best practices, and the packaging bits were rewritten > > for Debian. The only problem is that nobody bothered to notify the > > Gluster Community that this was happening. Nor did they send their new > > bits to GlusterFS. We were left to find out about it the hard way. > > > > *The Issue*: people who have used the packages from > > https://download.gluster.org are experiencing issues updating other > > software that depends on glusterfs. > > > > *The Change*: Gluster Community packages will now be built using > > packaging bits derived from the Debian packaging bits, which now follow > > Debian packaging conventions and best practices. > > > > *Conclusion*: This may be painful, but it's better in the long run for > > everyone. The volunteers who generously build packages in their copious > > spare time for the community appreciate your patience and understanding. > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewb at uvic.ca Fri Aug 9 21:33:04 2019 From: matthewb at uvic.ca (Matthew Benstead) Date: Fri, 9 Aug 2019 14:33:04 -0700 Subject: [Gluster-users] Brick missing trusted.glusterfs.dht xattr In-Reply-To: <803eea7e-86d0-35cd-e18a-500803bf4fc1@uvic.ca> References: <722b47be-cb8a-c036-9bd8-4913b6fd7ec6@uvic.ca> <803eea7e-86d0-35cd-e18a-500803bf4fc1@uvic.ca> Message-ID: Hi Sunny, Where would I find the changes-.log files? Is there anything else to help diagnose this? Thanks, ?-Matthew -- Matthew Benstead System Administrator Pacific Climate Impacts Consortium University of Victoria, UH1 PO Box 1800, STN CSC Victoria, BC, V8W 2Y2 Phone: +1-250-721-8432 Email: matthewb at uvic.ca On 7/29/19 9:46 AM, Matthew Benstead wrote: > Hi Sunny, > > Yes, I have attached the gsyncd.log file. I couldn't find any > changes-.log files... > > Trying to start replication goes faulty right away: > > [root at gluster01 ~]# rpm -q glusterfs > glusterfs-5.6-1.el7.x86_64 > [root at gluster01 ~]# uname -r > 3.10.0-957.21.3.el7.x86_64 > [root at gluster01 ~]# cat /etc/centos-release > CentOS Linux release 7.6.1810 (Core) > > [root at gluster01 ~]# gluster volume geo-replication storage > root at 10.0.231.81::pcic-backup start > Starting geo-replication session between storage & > 10.0.231.81::pcic-backup has been successful > [root at gluster01 ~]# gluster volume geo-replication storage > root at 10.0.231.81::pcic-backup status > ? > MASTER NODE??? MASTER VOL??? MASTER BRICK????????????????? SLAVE USER??? > SLAVE?????????????????????? SLAVE NODE??? STATUS??? CRAWL STATUS??? > LAST_SYNCED????????? > ------------------------------------------------------------------------------------------------------------------------------------------------------- > 10.0.231.50??? storage?????? /mnt/raid6-storage/storage??? root????????? > 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? > N/A????????????????? > 10.0.231.52??? storage?????? /mnt/raid6-storage/storage??? root????????? > 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? > N/A????????????????? > 10.0.231.54??? storage?????? /mnt/raid6-storage/storage??? root????????? > 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? > N/A????????????????? > 10.0.231.51??? storage?????? /mnt/raid6-storage/storage??? root????????? > 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? > N/A????????????????? > 10.0.231.53??? storage?????? /mnt/raid6-storage/storage??? root????????? > 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? > N/A????????????????? > 10.0.231.55??? storage?????? /mnt/raid6-storage/storage??? root????????? > 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? > N/A????????????????? > 10.0.231.56??? storage?????? /mnt/raid6-storage/storage??? root????????? > 10.0.231.81::pcic-backup??? N/A?????????? Faulty??? N/A???????????? > N/A????????????????? > [root at gluster01 ~]# gluster volume geo-replication storage > root at 10.0.231.81::pcic-backup stop > Stopping geo-replication session between storage & > 10.0.231.81::pcic-backup has been successful > > This is the primary cluster: > > [root at gluster01 ~]# gluster volume info storage > ? > Volume Name: storage > Type: Distribute > Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 > Status: Started > Snapshot Count: 0 > Number of Bricks: 7 > Transport-type: tcp > Bricks: > Brick1: 10.0.231.50:/mnt/raid6-storage/storage > Brick2: 10.0.231.51:/mnt/raid6-storage/storage > Brick3: 10.0.231.52:/mnt/raid6-storage/storage > Brick4: 10.0.231.53:/mnt/raid6-storage/storage > Brick5: 10.0.231.54:/mnt/raid6-storage/storage > Brick6: 10.0.231.55:/mnt/raid6-storage/storage > Brick7: 10.0.231.56:/mnt/raid6-storage/storage > Options Reconfigured: > features.read-only: off > features.inode-quota: on > features.quota: on > performance.readdir-ahead: on > nfs.disable: on > geo-replication.indexing: on > geo-replication.ignore-pid-check: on > transport.address-family: inet > features.quota-deem-statfs: on > changelog.changelog: on > diagnostics.client-log-level: INFO > > > And this is the cluster I'm trying to replicate to: > > [root at pcic-backup01 ~]# gluster volume info pcic-backup > ? > Volume Name: pcic-backup > Type: Distribute > Volume ID: 2890bcde-a023-4feb-a0e5-e8ef8f337d4c > Status: Started > Snapshot Count: 0 > Number of Bricks: 2 > Transport-type: tcp > Bricks: > Brick1: 10.0.231.81:/pcic-backup01-zpool/brick > Brick2: 10.0.231.82:/pcic-backup02-zpool/brick > Options Reconfigured: > nfs.disable: on > transport.address-family: inet > > > Thanks, > ?-Matthew > > On 7/28/19 10:56 PM, Sunny Kumar wrote: >> HI Matthew, >> >> Can you share geo-rep logs and one more log file >> (changes-.log) it will help to pinpoint actual reason >> behind failure. >> >> /sunny >> >> On Mon, Jul 29, 2019 at 9:13 AM Nithya Balachandran wrote: >>> >>> On Sat, 27 Jul 2019 at 02:31, Matthew Benstead wrote: >>>> Ok thank-you for explaining everything - that makes sense. >>>> >>>> Currently the brick file systems are pretty evenly distributed so I probably won't run the fix-layout right now. >>>> >>>> Would this state have any impact on geo-replication? I'm trying to geo-replicate this volume, but am getting a weird error: "Changelog register failed error=[Errno 21] Is a directory" >>> It should not. Sunny, can you comment on this? >>> >>> Regards, >>> Nithya >>>> I assume this is related to something else, but I wasn't sure. >>>> >>>> Thanks, >>>> -Matthew >>>> >>>> -- >>>> Matthew Benstead >>>> System Administrator >>>> Pacific Climate Impacts Consortium >>>> University of Victoria, UH1 >>>> PO Box 1800, STN CSC >>>> Victoria, BC, V8W 2Y2 >>>> Phone: +1-250-721-8432 >>>> Email: matthewb at uvic.ca >>>> >>>> On 7/26/19 12:02 AM, Nithya Balachandran wrote: >>>> >>>> >>>> >>>> On Fri, 26 Jul 2019 at 01:56, Matthew Benstead wrote: >>>>> Hi Nithya, >>>>> >>>>> Hmm... I don't remember if I did, but based on what I'm seeing it sounds like I probably didn't run rebalance or fix-layout. >>>>> >>>>> It looks like folders that haven't had any new files created have a dht of 0, while other folders have non-zero values. >>>>> >>>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ | grep dht >>>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/home | grep dht >>>>> trusted.glusterfs.dht=0x00000000000000000000000000000000 >>>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/home/matthewb | grep dht >>>>> trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 >>>>> >>>>> If I just run the fix-layout command will it re-create all of the dht values or just the missing ones? >>>> A fix-layout will recalculate the layouts entirely so files all the values will change. No files will be moved. >>>> A rebalance will recalculate the layouts like the fix-layout but will also move files to their new locations based on the new layout ranges. This could take a lot of time depending on the number of files/directories on the volume. If you do this, I would recommend that you turn off lookup-optimize until the rebalance is over. >>>> >>>>> Since the brick is already fairly size balanced could I get away with running fix-layout but not rebalance? Or would the new dht layout mean slower accesses since the files may be expected on different bricks? >>>> The first access for a file will be slower. The next one will be faster as the location will be cached in the client's in-memory structures. >>>> You may not need to run either a fix-layout or a rebalance if new file creations will be in directories created after the add-brick. Gluster will automatically include all 7 bricks for those directories. >>>> >>>> Regards, >>>> Nithya >>>> >>>>> Thanks, >>>>> -Matthew >>>>> >>>>> -- >>>>> Matthew Benstead >>>>> System Administrator >>>>> Pacific Climate Impacts Consortium >>>>> University of Victoria, UH1 >>>>> PO Box 1800, STN CSC >>>>> Victoria, BC, V8W 2Y2 >>>>> Phone: +1-250-721-8432 >>>>> Email: matthewb at uvic.ca >>>>> >>>>> On 7/24/19 9:30 PM, Nithya Balachandran wrote: >>>>> >>>>> >>>>> >>>>> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead wrote: >>>>>> So looking more closely at the trusted.glusterfs.dht attributes from the bricks it looks like they cover the entire range... and there is no range left for gluster07. >>>>>> >>>>>> The first 6 bricks range from 0x00000000 to 0xffffffff - so... is there a way to re-calculate what the dht values should be? Each of the bricks should have a gap >>>>>> >>>>>> Gluster05 00000000 -> 2aaaaaa9 >>>>>> Gluster06 2aaaaaaa -> 55555553 >>>>>> Gluster01 55555554 -> 7ffffffd >>>>>> Gluster02 7ffffffe -> aaaaaaa7 >>>>>> Gluster03 aaaaaaa8 -> d5555551 >>>>>> Gluster04 d5555552 -> ffffffff >>>>>> Gluster07 None >>>>>> >>>>>> If we split the range into 7 servers that would be a gap of about 0x24924924 for each server. >>>>>> >>>>>> Now in terms of the gluster07 brick, about 2 years ago the RAID array the brick was stored on became corrupted. I ran the remove-brick force command, then provisioned a new server, ran the add-brick command and then restored the missing files from backup by copying them back to the main gluster mount (not the brick). >>>>>> >>>>> Did you run a rebalance after performing the add-brick? Without a rebalance/fix-layout , the layout for existing directories on the volume will not be updated to use the new brick as well. >>>>> >>>>> That the layout does not include the new brick in the root dir is in itself is not a problem. Do you create a lot of files directly in the root of the volume? If yes, you might want to run a rebalance. Otherwise, if you mostly create files in newly added directories, you can probably ignore this. You can check the layout for directories on the volume and see if they incorporate the brick7. >>>>> >>>>> I would expect a lookup on the root to have set an xattr on the brick with an empty layout range . The fact that the xattr does not exist at all on the brick is what I am looking into. >>>>> >>>>> >>>>>> It looks like prior to that event this was the layout - which would make sense given the equal size of the 7 bricks: >>>>>> >>>>>> gluster02.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f >>>>>> >>>>>> gluster05.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f >>>>>> >>>>>> gluster04.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf >>>>>> >>>>>> gluster03.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f >>>>>> >>>>>> gluster01.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f >>>>>> >>>>>> gluster07.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x000000010000000000000000245fffdf >>>>>> >>>>>> gluster06.pcic.uvic.ca | SUCCESS | rc=0 >> >>>>>> # file: /mnt/raid6-storage/storage >>>>>> trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff >>>>>> >>>>>> Which yields the following: >>>>>> >>>>>> 00000000 -> 245fffdf Gluster07 >>>>>> 245fffe0 -> 48bfff1f Gluster01 >>>>>> 48bfff20 -> 6d1ffe5f Gluster02 >>>>>> 6d1ffe60 -> 917ffd9f Gluster03 >>>>>> 917ffda0 -> b5dffcdf Gluster04 >>>>>> b5dffce0 -> da3ffc1f Gluster05 >>>>>> da3ffc20 -> ffffffff Gluster06 >>>>>> >>>>>> Is there some way to get back to this? >>>>>> >>>>>> Thanks, >>>>>> -Matthew >>>>>> >>>>>> -- >>>>>> Matthew Benstead >>>>>> System Administrator >>>>>> Pacific Climate Impacts Consortium >>>>>> University of Victoria, UH1 >>>>>> PO Box 1800, STN CSC >>>>>> Victoria, BC, V8W 2Y2 >>>>>> Phone: +1-250-721-8432 >>>>>> Email: matthewb at uvic.ca >>>>>> >>>>>> On 7/18/19 7:20 AM, Matthew Benstead wrote: >>>>>> >>>>>> Hi Nithya, >>>>>> >>>>>> No - it was added about a year and a half ago. I have tried re-mounting the volume on the server, but it didn't add the attr: >>>>>> >>>>>> [root at gluster07 ~]# umount /storage/ >>>>>> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >>>>>> 10.0.231.56:/storage /storage glusterfs defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >>>>>> [root at gluster07 ~]# mount /storage/ >>>>>> [root at gluster07 ~]# df -h /storage/ >>>>>> Filesystem Size Used Avail Use% Mounted on >>>>>> 10.0.231.56:/storage 255T 194T 62T 77% /storage >>>>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/ >>>>>> # file: /mnt/raid6-storage/storage/ >>>>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>>>> trusted.gfid=0x00000000000000000000000000000001 >>>>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >>>>>> trusted.glusterfs.quota.dirty=0x3000 >>>>>> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >>>>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>>>> >>>>>> Thanks, >>>>>> -Matthew >>>>>> >>>>>> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >>>>>> >>>>>> Hi Matthew, >>>>>> >>>>>> Was this node/brick added to the volume recently? If yes, try mounting the volume on a fresh mount point - that should create the xattr on this as well. >>>>>> >>>>>> Regards, >>>>>> Nithya >>>>>> >>>>>> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I've just noticed one brick in my 7 node distribute volume is missing >>>>>>> the trusted.glusterfs.dht xattr...? How can I fix this? >>>>>>> >>>>>>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>>>>>> >>>>>>> All of the other nodes are fine, but gluster07 from the list below does >>>>>>> not have the attribute. >>>>>>> >>>>>>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . >>>>>>> --absolute-names -n trusted.glusterfs.dht -e hex >>>>>>> /mnt/raid6-storage/storage" >>>>>>> ... >>>>>>> gluster05 | SUCCESS | rc=0 >> >>>>>>> # file: /mnt/raid6-storage/storage >>>>>>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>>>>>> >>>>>>> gluster03 | SUCCESS | rc=0 >> >>>>>>> # file: /mnt/raid6-storage/storage >>>>>>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>>>>>> >>>>>>> gluster04 | SUCCESS | rc=0 >> >>>>>>> # file: /mnt/raid6-storage/storage >>>>>>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>>>>>> >>>>>>> gluster06 | SUCCESS | rc=0 >> >>>>>>> # file: /mnt/raid6-storage/storage >>>>>>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>>>>>> >>>>>>> gluster02 | SUCCESS | rc=0 >> >>>>>>> # file: /mnt/raid6-storage/storage >>>>>>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>>>>>> >>>>>>> gluster07 | FAILED | rc=1 >> >>>>>>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>>>>>> attributenon-zero return code >>>>>>> >>>>>>> gluster01 | SUCCESS | rc=0 >> >>>>>>> # file: /mnt/raid6-storage/storage >>>>>>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>>>>>> >>>>>>> Here are all of the attr's from the brick: >>>>>>> >>>>>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>>>>>> /mnt/raid6-storage/storage/ >>>>>>> # file: /mnt/raid6-storage/storage/ >>>>>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>>>>> trusted.gfid=0x00000000000000000000000000000001 >>>>>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>>>>>> trusted.glusterfs.quota.dirty=0x3000 >>>>>>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>>>>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>>>>> >>>>>>> >>>>>>> And here is the volume information: >>>>>>> >>>>>>> [root at gluster07 ~]# gluster volume info storage >>>>>>> >>>>>>> Volume Name: storage >>>>>>> Type: Distribute >>>>>>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>>>>>> Status: Started >>>>>>> Snapshot Count: 0 >>>>>>> Number of Bricks: 7 >>>>>>> Transport-type: tcp >>>>>>> Bricks: >>>>>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>>>>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>>>>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>>>>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>>>>>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>>>>>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>>>>>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>>>>>> Options Reconfigured: >>>>>>> changelog.changelog: on >>>>>>> features.quota-deem-statfs: on >>>>>>> features.read-only: off >>>>>>> features.inode-quota: on >>>>>>> features.quota: on >>>>>>> performance.readdir-ahead: on >>>>>>> nfs.disable: on >>>>>>> geo-replication.indexing: on >>>>>>> geo-replication.ignore-pid-check: on >>>>>>> transport.address-family: inet >>>>>>> >>>>>>> Thanks, >>>>>>> -Matthew >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgowtham at redhat.com Mon Aug 12 07:08:02 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Mon, 12 Aug 2019 12:38:02 +0530 Subject: [Gluster-users] Announcing Gluster release 6.5 Message-ID: Hi, The Gluster community is pleased to announce the release of Gluster 6.5 (packages available at [1]). Release notes for the release can be found at [2]. Major changes, features and limitations addressed in this release: None Thanks, Gluster community [1] Packages for 6.5: https://download.gluster.org/pub/gluster/glusterfs/6/6.5/ [2] Release notes for 6.5: https://docs.gluster.org/en/latest/release-notes/6.5/ -- Regards, Hari Gowtham. From hgowtham at redhat.com Mon Aug 12 09:38:36 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Mon, 12 Aug 2019 15:08:36 +0530 Subject: [Gluster-users] Announcing Gluster release 5.9 Message-ID: Hi, The Gluster community is pleased to announce the release of Gluster 5.9 (packages available at [1]). Release notes for the release can be found at [2]. Major changes, features and limitations addressed in this release: None Thanks, Gluster community [1] Packages for 5.9: https://download.gluster.org/pub/gluster/glusterfs/5/5.9/ [2] Release notes for 5.9: https://docs.gluster.org/en/latest/release-notes/5.9/ -- Regards, Hari Gowtham. From flyxiaoyu at gmail.com Mon Aug 12 12:37:06 2019 From: flyxiaoyu at gmail.com (Frank Yu) Date: Mon, 12 Aug 2019 20:37:06 +0800 Subject: [Gluster-users] =?utf-8?q?=E3=80=90replace-brick_failed_but_make_?= =?utf-8?q?there=E2=80=99re_two_same_client-id_of_the_gluster_clust?= =?utf-8?q?er=2C_which_lead_can=E2=80=99t_mount_the_gluster_anymore?= =?utf-8?b?44CR?= Message-ID: Hi guys, I met a terrible situations need all your helps. I have a production cluster running well at first. the version of gluster is 3.12.15-1.el7.x86_64, the cluster has 12 nodes, 12 brick(disk) per nodes, there is one distributed-replicate volume, with 144 bricks(48 * 3). then there is a node crushed(the node named nodeA), and all it?s disk can?t be used anymore, but since the os of nodes run on kvm machine, so it came back with 12 new disks. I try to replace the first brick of nodeA with cmd ?gluster volume replace-brick VOLUMENAME nodeA:/mnt/data-1/data nodeA:/mnt/data-1/data01 commit force?, after some times, it failed with error ?Error : Request timed out?. here came the problem, both ?nodeA:/mnt/data-1/data? and ?nodeA:/mnt/data-1/data01? show in the output of cmd ?gluster volume info? When I try to mount gluster to client with fuse, it report error like below: [2019-08-12 12:27:42.395440] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.15 (args: /usr/sbin/glusterfs --volfile-server=xxxxx --volfile-id=/training-data-ali /mnt/glusterfs) [2019-08-12 12:27:42.400015] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2019-08-12 12:27:42.404994] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 *[2019-08-12 12:27:42.415971] E [MSGID: 101179] [graph.y:153:new_volume] 0-parser: Line 1381: volume ?VOLUME-NAME-client-74' defined again* [2019-08-12 12:27:42.416124] E [MSGID: 100026] [glusterfsd.c:2358:glusterfs_process_volfp] 0-: failed to construct the graph [2019-08-12 12:27:42.416376] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x55898e35e092] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x150) [0x55898e357da0] -->/lib64/libglusterfs.so.0(glusterfs_graph_destroy+0x84) [0x7f95f7318754] ) 0-graph: invalid argument: graph [Invalid argument] [2019-08-12 12:27:42.416425] W [glusterfsd.c:1375:cleanup_and_exit] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x55898e35e092] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x163) [0x55898e357db3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55898e35732b] ) 0-: received signum (-1), shutting down [2019-08-12 12:27:42.416455] I [fuse-bridge.c:5852:fini] 0-fuse: Unmounting '/mnt/glusterfs'. [2019-08-12 12:27:42.429655] I [fuse-bridge.c:5857:fini] 0-fuse: Closing fuse connection to '/mnt/glusterfs-aliyun'. [2019-08-12 12:27:42.429759] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f95f6140e25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55898e3574b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55898e35732b] ) 0-: received signum (15), shutting down So, how can I solve error *?Line 1381: volume ?VOLUME-NAME-client-74' defined again? * -- Regards Frank Yu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at lucassen.org Tue Aug 13 19:23:39 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Tue, 13 Aug 2019 21:23:39 +0200 Subject: [Gluster-users] master volume Message-ID: <20190813212339.862b204c3bf711c65ac20bc2@lucassen.org> Hello list, I want to add a geo-replication server to a 2 node + arbiter environment. Reading the docs, many questions pop up, under which this one, they're talking about a "master volume" and a "slave volume": gluster volume geo-replication \ :: create push-pem [force] https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Distributed%20Geo%20Replication/ I have a few volumes running, but I fear I missed something crucial somewhere. Or should I set up a geo-replication for each and every volume I run on the two nodes? R. -- richard lucassen http://contact.xaq.nl/ From mailinglists at lucassen.org Tue Aug 13 19:17:07 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Tue, 13 Aug 2019 21:17:07 +0200 Subject: [Gluster-users] very poor performance on Debian Buster In-Reply-To: <20190630135951.66b39be769d0e4a2849bec1a@lucassen.org> References: <2h1nisnfpybdfnbr5sti7d8n.1561630008584@email.android.com> <20190627225740.b74f7768a595ba861568c065@lucassen.org> <20190630135951.66b39be769d0e4a2849bec1a@lucassen.org> Message-ID: <20190813211707.347f1451b14647fe9c2cce67@lucassen.org> On Sun, 30 Jun 2019 13:59:51 +0200 richard lucassen wrote: > > BTW, it is all hardware raid1 with 3.4GB SSD's. The server is a > > beast, a Dell R630 > > I'll come back to the issue, I'm too busy at the moment. Finally it was a piece of broken hardware. Glusterfs is running ok now. -- richard lucassen http://contact.xaq.nl/ From laurentfdumont at gmail.com Wed Aug 14 00:10:11 2019 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Tue, 13 Aug 2019 20:10:11 -0400 Subject: [Gluster-users] very poor performance on Debian Buster In-Reply-To: <20190813211707.347f1451b14647fe9c2cce67@lucassen.org> References: <2h1nisnfpybdfnbr5sti7d8n.1561630008584@email.android.com> <20190627225740.b74f7768a595ba861568c065@lucassen.org> <20190630135951.66b39be769d0e4a2849bec1a@lucassen.org> <20190813211707.347f1451b14647fe9c2cce67@lucassen.org> Message-ID: Was it a bad raid in the end? On Tue, Aug 13, 2019, 3:24 PM richard lucassen wrote: > On Sun, 30 Jun 2019 13:59:51 +0200 > richard lucassen wrote: > > > > BTW, it is all hardware raid1 with 3.4GB SSD's. The server is a > > > beast, a Dell R630 > > > > I'll come back to the issue, I'm too busy at the moment. > > Finally it was a piece of broken hardware. Glusterfs is running ok now. > > -- > richard lucassen > http://contact.xaq.nl/ > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From avishwan at redhat.com Wed Aug 14 05:54:08 2019 From: avishwan at redhat.com (Aravinda Vishwanathapura Krishna Murthy) Date: Wed, 14 Aug 2019 11:24:08 +0530 Subject: [Gluster-users] master volume In-Reply-To: <20190813212339.862b204c3bf711c65ac20bc2@lucassen.org> References: <20190813212339.862b204c3bf711c65ac20bc2@lucassen.org> Message-ID: Gluster Geo-replication is a feature to replicate data from one Gluster Volume to other. This needs to be set up for each Volume for which data needs to be replicated. If Master and Slave Volumes are in the same cluster then it is not so helpful when disaster happens. On Wed, Aug 14, 2019 at 1:02 AM richard lucassen wrote: > Hello list, > > I want to add a geo-replication server to a 2 node + arbiter > environment. Reading the docs, many questions pop up, under which this > one, they're talking about a "master volume" and a "slave volume": > > gluster volume geo-replication \ > :: create push-pem [force] > > > https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Distributed%20Geo%20Replication/ > > I have a few volumes running, but I fear I missed something crucial > somewhere. Or should I set up a geo-replication for each and every > volume I run on the two nodes? > > R. > > -- > richard lucassen > http://contact.xaq.nl/ > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -- regards Aravinda VK -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at lucassen.org Wed Aug 14 13:36:42 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Wed, 14 Aug 2019 15:36:42 +0200 Subject: [Gluster-users] master volume In-Reply-To: References: <20190813212339.862b204c3bf711c65ac20bc2@lucassen.org> Message-ID: <20190814153642.bd9c331e6f04c0e6b6533364@lucassen.org> On Wed, 14 Aug 2019 11:24:08 +0530 Aravinda Vishwanathapura Krishna Murthy wrote: > Gluster Geo-replication is a feature to replicate data from one > Gluster Volume to other. This needs to be set up for each Volume for > which data needs to be replicated. That sounds logical :) > If Master and Slave Volumes are in > the same cluster then it is not so helpful when disaster happens. So, do I need to do something like this?: cluster volume name: gvol0 replication vol name: rep-gvol0 R. -- richard lucassen http://contact.xaq.nl/ From mailinglists at lucassen.org Wed Aug 14 13:37:07 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Wed, 14 Aug 2019 15:37:07 +0200 Subject: [Gluster-users] very poor performance on Debian Buster In-Reply-To: References: <2h1nisnfpybdfnbr5sti7d8n.1561630008584@email.android.com> <20190627225740.b74f7768a595ba861568c065@lucassen.org> <20190630135951.66b39be769d0e4a2849bec1a@lucassen.org> <20190813211707.347f1451b14647fe9c2cce67@lucassen.org> Message-ID: <20190814153707.77cca5bd24e3047e1ec9b124@lucassen.org> On Tue, 13 Aug 2019 20:10:11 -0400 Laurent Dumont wrote: > Was it a bad raid in the end? No, it was due to some bitrot I suppose on a USB2 port talking USB-1 only. Therefore it was 12Mbit max. After a new kernel reboot the port functioned as expected. R. -- richard lucassen http://contact.xaq.nl/ From mailinglists at lucassen.org Wed Aug 14 13:44:14 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Wed, 14 Aug 2019 15:44:14 +0200 Subject: [Gluster-users] replace arbiter Message-ID: <20190814154414.6d7a9aec46d43496c0dfcb62@lucassen.org> Hello list, I have an arbiter, Debian Buster, standard install (two nodes + arbiter). I want to set up a new arbiter server, same version same install. Looking at the configs, the /etc/glusterfs/ is not touched by configuring glusterd, but /var/lib/glusterd/ is. Would it be feasible to simply do this: 1) stop old arbiter on old ip 2) edit DNS and /etc/hosts on the two nodes 3) rsync old:/var/lib/glusterd/ to new:/var/lib/glusterd/ 4) start new arbiter on new ip R. -- richard lucassen http://contact.xaq.nl/ From mailinglists at lucassen.org Wed Aug 14 13:47:17 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Wed, 14 Aug 2019 15:47:17 +0200 Subject: [Gluster-users] replace arbiter In-Reply-To: <20190814154414.6d7a9aec46d43496c0dfcb62@lucassen.org> References: <20190814154414.6d7a9aec46d43496c0dfcb62@lucassen.org> Message-ID: <20190814154717.fe21b25859481e57711c3ecb@lucassen.org> On Wed, 14 Aug 2019 15:44:14 +0200 richard lucassen wrote: > I want to set up a new arbiter server, same version same install. Oops: s/set up a new/replace the/ I want to replace the arbiter server, same version same install. -- richard lucassen http://contact.xaq.nl/ From avishwan at redhat.com Wed Aug 14 14:54:53 2019 From: avishwan at redhat.com (Aravinda Vishwanathapura Krishna Murthy) Date: Wed, 14 Aug 2019 20:24:53 +0530 Subject: [Gluster-users] master volume In-Reply-To: <20190814153642.bd9c331e6f04c0e6b6533364@lucassen.org> References: <20190813212339.862b204c3bf711c65ac20bc2@lucassen.org> <20190814153642.bd9c331e6f04c0e6b6533364@lucassen.org> Message-ID: On Wed, Aug 14, 2019 at 7:07 PM richard lucassen wrote: > On Wed, 14 Aug 2019 11:24:08 +0530 > Aravinda Vishwanathapura Krishna Murthy wrote: > > > Gluster Geo-replication is a feature to replicate data from one > > Gluster Volume to other. This needs to be set up for each Volume for > > which data needs to be replicated. > > That sounds logical :) > > > If Master and Slave Volumes are in > > the same cluster then it is not so helpful when disaster happens. > > So, do I need to do something like this?: > > cluster volume name: gvol0 > replication vol name: rep-gvol0 > Volume names can be kept as your convenience. > > R. > > -- > richard lucassen > http://contact.xaq.nl/ > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -- regards Aravinda VK -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Wed Aug 14 16:32:52 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Wed, 14 Aug 2019 19:32:52 +0300 Subject: [Gluster-users] very poor performance on Debian Buster Message-ID: <648rignou55nhtwoumi179la.1565800350643@email.android.com> Check for firmware updates' release notes that might fix the behaviour. Best Regards, Strahil NikolovOn Aug 14, 2019 16:37, richard lucassen wrote: > > On Tue, 13 Aug 2019 20:10:11 -0400 > Laurent Dumont wrote: > > > Was it a bad raid in the end? > > No, it was due to some bitrot I suppose on a USB2 port talking USB-1 > only. Therefore it was 12Mbit max. After a new kernel reboot the port > functioned as expected. > > R. > > -- > richard lucassen > http://contact.xaq.nl/ > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From mailinglists at lucassen.org Wed Aug 14 22:04:35 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Thu, 15 Aug 2019 00:04:35 +0200 Subject: [Gluster-users] master volume In-Reply-To: References: <20190813212339.862b204c3bf711c65ac20bc2@lucassen.org> <20190814153642.bd9c331e6f04c0e6b6533364@lucassen.org> Message-ID: <20190815000435.2763d6793ad67fea70e1f59e@lucassen.org> On Wed, 14 Aug 2019 20:24:53 +0530 Aravinda Vishwanathapura Krishna Murthy wrote: > > > If Master and Slave Volumes are in > > > the same cluster then it is not so helpful when disaster happens. > > > > So, do I need to do something like this?: > > > > cluster volume name: gvol0 > > replication vol name: rep-gvol0 > > Volume names can be kept as your convenience. The doc is quite unclear for me as a gluster newbie. It's the old Linux paradox: the moment that you will understand the docs is the moment that you don't need them anymore. Why is there a MASTER_VOL and a SLAVE_VOL? If I have a volume called "gvol0" it seems logical to me that the geo-replication slave server also uses this volume name "gvol0". The geo-replication server knows it is not part of the active cluster because it was told to do so. OTOH, I think there must be an obvious reason for this MASTER_VOL and SLAVE_VOL, but I don't understand this from the docs. R. -- richard lucassen http://contact.xaq.nl/ From sacharya at redhat.com Thu Aug 15 03:42:43 2019 From: sacharya at redhat.com (Shwetha Acharya) Date: Thu, 15 Aug 2019 09:12:43 +0530 Subject: [Gluster-users] master volume In-Reply-To: <20190815000435.2763d6793ad67fea70e1f59e@lucassen.org> References: <20190813212339.862b204c3bf711c65ac20bc2@lucassen.org> <20190814153642.bd9c331e6f04c0e6b6533364@lucassen.org> <20190815000435.2763d6793ad67fea70e1f59e@lucassen.org> Message-ID: Consider that you have a volume, named vol1. Now you want to have a replica of it, for disaster recovery. What do you do? You will create a new volume, say, repica-vol1 in a different cluster. To replicate data from vol1 to replica-vol1, you will set up a geo-rep session between vol1 and replica-vol1. In geo-replication, we call your primary volume vol1 as master. The replicated volume replica-vol1 is called slave. We call it master and slave because you will see unidirectional sync of data from master volume(vol1) to slave volume(replica-vol1). Hope this helps. On Thu, 15 Aug 2019, 3:35 am richard lucassen, wrote: > On Wed, 14 Aug 2019 20:24:53 +0530 > Aravinda Vishwanathapura Krishna Murthy wrote: > > > > > If Master and Slave Volumes are in > > > > the same cluster then it is not so helpful when disaster happens. > > > > > > So, do I need to do something like this?: > > > > > > cluster volume name: gvol0 > > > replication vol name: rep-gvol0 > > > > Volume names can be kept as your convenience. > > The doc is quite unclear for me as a gluster newbie. It's the old Linux > paradox: the moment that you will understand the docs is the moment > that you don't need them anymore. > > Why is there a MASTER_VOL and a SLAVE_VOL? If I have a volume called > "gvol0" it seems logical to me that the geo-replication slave server > also uses this volume name "gvol0". The geo-replication server knows it > is not part of the active cluster because it was told to do so. > > OTOH, I think there must be an obvious reason for this MASTER_VOL and > SLAVE_VOL, but I don't understand this from the docs. > > R. > > -- > richard lucassen > http://contact.xaq.nl/ > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lyubomir.grancharov at bottleshipvfx.com Thu Aug 15 07:42:35 2019 From: lyubomir.grancharov at bottleshipvfx.com (Lyubomir Grancharov) Date: Thu, 15 Aug 2019 10:42:35 +0300 Subject: [Gluster-users] oVirt cluster with glusterfs data domain - very slow writing speed on Windows server VM Message-ID: Hi folks, I have been experimenting with oVirt cluster based on glusterfs for the past few days. (first-timer). The cluster is up and running and it consists of 4 nodes and has 4 replicas. When I try to deploy Windows Server VM I encounter the following issue: The disk of the VM has ok reading speed ( close to bare metal) but the writing speed is very slow. ( about 10 times slower than it is supposed to be). Can anyone give me any suggestion, please? Thanks in advance! Here are the settings of the glusterfs volume: Volume Name: bottle-volume Type: Replicate Volume ID: 869b8d1e-1266-4820-8dcd-4fea92346b90 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 4 = 4 Transport-type: tcp Bricks: Brick1: cnode01.bottleship.local:/gluster_bricks/brick-cnode01/brick-cnode01 Brick2: cnode02.bottleship.local:/gluster_bricks/brick-cnode02/brick-cnode02 Brick3: cnode03.bottleship.local:/gluster_bricks/brick-cnode03/brick-cnode03 Brick4: cnode04.bottleship.local:/gluster_bricks/brick-cnode04/brick-cnode04 Options Reconfigured: network.ping-timeout: 30 cluster.granular-entry-heal: enable performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off transport.address-family: inet nfs.disable: on performance.client-io-threads: off server.allow-insecure: on Please let me know if you need any more configuration info or hardware specs. best, Lyubo -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at lucassen.org Thu Aug 15 08:25:03 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Thu, 15 Aug 2019 10:25:03 +0200 Subject: [Gluster-users] master volume In-Reply-To: References: <20190813212339.862b204c3bf711c65ac20bc2@lucassen.org> <20190814153642.bd9c331e6f04c0e6b6533364@lucassen.org> <20190815000435.2763d6793ad67fea70e1f59e@lucassen.org> Message-ID: <20190815102503.ba1a35d0da85a8851bb1a189@lucassen.org> On Thu, 15 Aug 2019 09:12:43 +0530 Shwetha Acharya wrote: > Consider that you have a volume, named vol1. Now you want to have a > replica of it, for disaster recovery. What do you do? You will create > a new volume, say, repica-vol1 in a different cluster. > > To replicate data from vol1 to replica-vol1, you will set up a geo-rep > session between vol1 and replica-vol1. In geo-replication, we call > your primary volume vol1 as master. The replicated volume > replica-vol1 is called slave. We call it master and slave because you > will see unidirectional sync of data from master volume(vol1) to > slave volume(replica-vol1). > > Hope this helps. Yep, thnx, that's much clearer now. R. -- richard lucassen http://contact.xaq.nl/ From ravishankar at redhat.com Fri Aug 16 03:51:39 2019 From: ravishankar at redhat.com (Ravishankar N) Date: Fri, 16 Aug 2019 09:21:39 +0530 Subject: [Gluster-users] replace arbiter In-Reply-To: <20190814154717.fe21b25859481e57711c3ecb@lucassen.org> References: <20190814154414.6d7a9aec46d43496c0dfcb62@lucassen.org> <20190814154717.fe21b25859481e57711c3ecb@lucassen.org> Message-ID: On 14/08/19 7:17 PM, richard lucassen wrote: > On Wed, 14 Aug 2019 15:44:14 +0200 > richard lucassen wrote: > >> I want to set up a new arbiter server, same version same install. > Oops: > > s/set up a new/replace the/ > > I want to replace the arbiter server, same version same install. The replace-brick/ reset-brick gluster CLI is what you should be looking at in the documentation/ older threads on this list. > From hunter86_bg at yahoo.com Fri Aug 16 16:56:20 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Fri, 16 Aug 2019 19:56:20 +0300 Subject: [Gluster-users] replace arbiter Message-ID: Richard, What is your gluster version ? Best Regards, Strahil NikolovOn Aug 16, 2019 06:51, Ravishankar N wrote: > > > On 14/08/19 7:17 PM, richard lucassen wrote: > > On Wed, 14 Aug 2019 15:44:14 +0200 > > richard lucassen wrote: > > > >> I want to set up a new arbiter server, same version same install. > > Oops: > > > > s/set up a new/replace the/ > > > > I want to replace the arbiter server, same version same install. > The replace-brick/ reset-brick gluster CLI is what you should be looking > at in the documentation/ older threads on this list. > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From mailinglists at lucassen.org Fri Aug 16 17:41:25 2019 From: mailinglists at lucassen.org (richard lucassen) Date: Fri, 16 Aug 2019 19:41:25 +0200 Subject: [Gluster-users] replace arbiter In-Reply-To: References: Message-ID: <20190816194125.be6b2a2e481a89b658175a81@lucassen.org> On Fri, 16 Aug 2019 19:56:20 +0300 Strahil wrote: > What is your gluster version ? The Debian Buster version: $ dpkg -l | grep gluster ii glusterfs-client 5.5-3 amd64 clustered file-system (client package) ii glusterfs-common 5.5-3 amd64 GlusterFS common libraries and translator modules ii glusterfs-server 5.5-3 amd64 clustered file-system (server package) ii libglusterfs-dev 5.5-3 amd64 Development files for GlusterFS libraries ii libglusterfs0:amd64 5.5-3 amd64 GlusterFS shared library -- richard lucassen http://contact.xaq.nl/ From amudhan83 at gmail.com Sat Aug 17 13:29:00 2019 From: amudhan83 at gmail.com (Amudhan P) Date: Sat, 17 Aug 2019 18:59:00 +0530 Subject: [Gluster-users] du output showing corrupt file system Message-ID: Hi, I am using Gluster version 3.10.1. Mounting volume through fuse mount and I have run the command du -hs "directory" which holds many subdirectories. some of the subdirectory given output with below message. du: WARNING: Circular directory structure. This almost certainly means that you have a corrupted file system. NOTIFY YOUR SYSTEM MANAGER. The following directory is part of the cycle: what could be the issue or what should be done to fix this problem? regards Amudhan P -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Sun Aug 18 17:18:03 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Sun, 18 Aug 2019 20:18:03 +0300 Subject: [Gluster-users] du output showing corrupt file system Message-ID: <8rikb8oxegrdl2hehdlqlkcy.1566148683918@email.android.com> Hm... I thought that 3.10.X is no longer on support. Can you test from a gluster client with newer version of gluster and FUSE respectively? Best Regards, Strahil NikolovOn Aug 17, 2019 16:29, Amudhan P wrote: > > Hi, > > I am using Gluster version 3.10.1. > > Mounting volume through fuse mount and I have run the command du -hs "directory" which holds many subdirectories.? > some of the subdirectory given output with below message. > > du: WARNING: Circular directory structure. > This almost certainly means that you have a corrupted file system. > NOTIFY YOUR SYSTEM MANAGER. > The following directory is part of the cycle: > > what could be the issue or what should be done to fix this problem? > > regards > Amudhan P > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amudhan83 at gmail.com Mon Aug 19 06:53:14 2019 From: amudhan83 at gmail.com (Amudhan P) Date: Mon, 19 Aug 2019 12:23:14 +0530 Subject: [Gluster-users] du output showing corrupt file system In-Reply-To: <8rikb8oxegrdl2hehdlqlkcy.1566148683918@email.android.com> References: <8rikb8oxegrdl2hehdlqlkcy.1566148683918@email.android.com> Message-ID: Thanks, Strahil. I am using 3.10.1 since it was released. I believe we should always have the same version of gluster in Server and client. I don't think upgrading client solves this issue. should be some issue with the brick layout? regards Amudhan On Sun, Aug 18, 2019 at 10:47 PM Strahil wrote: > Hm... I thought that 3.10.X is no longer on support. > > Can you test from a gluster client with newer version of gluster and FUSE > respectively? > > Best Regards, > Strahil Nikolov > On Aug 17, 2019 16:29, Amudhan P wrote: > > Hi, > > I am using Gluster version 3.10.1. > > Mounting volume through fuse mount and I have run the command du -hs > "directory" which holds many subdirectories. > some of the subdirectory given output with below message. > > du: WARNING: Circular directory structure. > This almost certainly means that you have a corrupted file system. > NOTIFY YOUR SYSTEM MANAGER. > The following directory is part of the cycle: > > what could be the issue or what should be done to fix this problem? > > regards > Amudhan P > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Mon Aug 19 10:03:18 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Mon, 19 Aug 2019 13:03:18 +0300 Subject: [Gluster-users] du output showing corrupt file system Message-ID: On Aug 19, 2019 09:53, Amudhan P wrote: > > Thanks, Strahil. > > I am using 3.10.1 since it was released.? > > I believe we should always have the same version of gluster in Server and client. That's not always necessary... Still you can bring a VM with 3.12 and check what it shows. > > I don't think upgrading client solves this issue. should be some issue with the brick layout? > > regards > Amudhan > > > On Sun, Aug 18, 2019 at 10:47 PM Strahil wrote: >> >> Hm... I thought that 3.10.X is no longer on support. >> >> Can you test from a gluster client? with newer version of gluster and FUSE respectively? >> >> Best Regards, >> Strahil Nikolov >> >> On Aug 17, 2019 16:29, Amudhan P wrote: >>> >>> Hi, >>> >>> I am using Gluster version 3.10.1. >>> >>> Mounting volume through fuse mount and I have run the command du -hs "directory" which holds many subdirectories.? >>> some of the subdirectory given output with below message. >>> >>> du: WARNING: Circular directory structure. >>> This almost certainly means that you have a corrupted file system. >>> NOTIFY YOUR SYSTEM MANAGER. >>> The following directory is part of the cycle: >>> >>> what could be the issue or what should be done to fix this problem? >>> >>> regards >>> Amudhan P >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From csirotic at evoqarchitecture.com Mon Aug 19 15:34:34 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Mon, 19 Aug 2019 11:34:34 -0400 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes References: Message-ID: Hi, we have a replicate 3 cluster. 2 other servers are clients that run VM that are stored on the Gluster volumes. I had to reboot one of the brick for maintenance. The whole VM setup went super slow and some of the client crashed. I think there is some timeout setting for KVM/Qemu vs timeout of Glusterd that could fix this. Do anyone have an idea ? The whole point of having gluster for me was to be able to shut down one of the host while the vm stay running. Carl From hunter86_bg at yahoo.com Mon Aug 19 16:03:30 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Mon, 19 Aug 2019 19:03:30 +0300 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes Message-ID: Hi Carl, Did you check for any pending heals before rebooting the gluster server? Also, it was discussed that shutting down the node, does not stop the bricks properly and thus the clients will eait for a timeout before restoring full functionality. You can stop your glusterd and actually all processes by using a script in /usr/share/gluster/scripts (the path is based on memory and could be wrong). Best Regards, Strahil NikllovOn Aug 19, 2019 18:34, Carl Sirotic wrote: > > Hi, > > we have a replicate 3 cluster. > > 2 other servers are clients that run VM that are stored on the Gluster > volumes. > > I had to reboot one of the brick for maintenance. > > The whole VM setup went super slow and some of the client crashed. > > I think there is some timeout setting for KVM/Qemu vs timeout of > Glusterd that could fix this. > > Do anyone have an idea ? > > The whole point of having gluster for me was to be able to shut down one > of the host while the vm stay running. > > > Carl > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From csirotic at evoqarchitecture.com Mon Aug 19 16:05:39 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Mon, 19 Aug 2019 12:05:39 -0400 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> Message-ID: An HTML attachment was scrubbed... URL: From budic at onholyground.com Mon Aug 19 16:08:51 2019 From: budic at onholyground.com (Darrell Budic) Date: Mon, 19 Aug 2019 11:08:51 -0500 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> Message-ID: <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume? > On Aug 19, 2019, at 11:05 AM, Carl Sirotic wrote: > > Yes, I made sure there was no heal. > This is what I am suspecting thet shutting down a host isn't the right way to go. > Hi Carl, Did you check for any pending heals before rebooting the gluster server? Also, it was discussed that shutting down the node, does not stop the bricks properly and thus the clients will eait for a timeout before restoring full functionality. You can stop your glusterd and actually all processes by using a script in /usr/share/gluster/scripts (the path is based on memory and could be wrong). Best Regards, Strahil NikllovOn Aug 19, 2019 18:34, Carl Sirotic wrote: > > Hi, > > we have a replicate 3 cluster. > > 2 other servers are clients that run VM that are stored on the Gluster > volumes. > > I had to reboot one of the brick for maintenance. > > The whole VM setup went super slow and some of the client crashed. > > I think there is some timeout setting for KVM/Qemu vs timeout of > Glusterd that could fix this. > > Do anyone have an idea ? > > The whole point of having gluster for me was to be able to shut down one > of the host while the vm stay running. > > > Carl > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users AVIS DE CONFIDENTIALIT? : Ce courriel peut contenir de l'information privil?gi?e et confidentielle. Nous vous demandons de le d?truire imm?diatement si vous n'?tes pas le destinataire. CONFIDENTIALITY NOTICE: This email may contain information that is privileged and confidential. Please delete immediately if you are not the intended recipient. _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From csirotic at evoqarchitecture.com Mon Aug 19 17:01:56 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Mon, 19 Aug 2019 13:01:56 -0400 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> Message-ID: No, I didn't. I am very interested about these settings. Also, is it possible to turn the shard feature AFTER the volume was started to be used ? Carl On 2019-08-19 12:08 p.m., Darrell Budic wrote: > You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume? > >> On Aug 19, 2019, at 11:05 AM, Carl Sirotic wrote: >> >> Yes, I made sure there was no heal. >> This is what I am suspecting thet shutting down a host isn't the right way to go. >> Hi Carl, Did you check for any pending heals before rebooting the gluster server? Also, it was discussed that shutting down the node, does not stop the bricks properly and thus the clients will eait for a timeout before restoring full functionality. You can stop your glusterd and actually all processes by using a script in /usr/share/gluster/scripts (the path is based on memory and could be wrong). Best Regards, Strahil NikllovOn Aug 19, 2019 18:34, Carl Sirotic wrote: > > Hi, > > we have a replicate 3 cluster. > > 2 other servers are clients that run VM that are stored on the Gluster > volumes. > > I had to reboot one of the brick for maintenance. > > The whole VM setup went super slow and some of the client crashed. > > I think there is some timeout setting for KVM/Qemu vs timeout of > Glusterd that could fix this. > > Do anyone have an idea ? > > The whole point of having gluster for me was to be able to shut down one > of the host while the vm stay running. > > > Carl > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users AVIS DE CONFIDENTIALIT? : Ce courriel peut contenir de l'information privil?gi?e et confidentielle. Nous vous demandons de le d?truire imm?diatement si vous n'?tes pas le destinataire. CONFIDENTIALITY NOTICE: This email may contain information that is privileged and confidential. Please delete immediately if you are not the intended recipient. _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users From guy.boisvert at ingtegration.com Mon Aug 19 17:22:25 2019 From: guy.boisvert at ingtegration.com (Guy Boisvert) Date: Mon, 19 Aug 2019 13:22:25 -0400 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> Message-ID: <7b14c43d-fdf6-7c07-4715-3d1b78831227@ingtegration.com> On 2019-08-19 12:08 p.m., Darrell Budic wrote: > You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume? That's what we did here: gluster volume set W2K16_Rhenium cluster.quorum-type auto gluster volume set W2K16_Rhenium network.ping-timeout 10 gluster volume set W2K16_Rhenium auth.allow \* gluster volume set W2K16_Rhenium group virt gluster volume set W2K16_Rhenium storage.owner-uid 36 gluster volume set W2K16_Rhenium storage.owner-gid 36 gluster volume set W2K16_Rhenium features.shard on gluster volume set W2K16_Rhenium features.shard-block-size 256MB gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm full gluster volume set W2K16_Rhenium performance.low-prio-threads 32 tuned-adm profile random-io??? ??? (a profile i added in CentOS 7) cat /usr/lib/tuned/random-io/tuned.conf =========================================== [main] summary=Optimize for Gluster virtual machine storage include=throughput-performance [sysctl] vm.dirty_ratio = 5 vm.dirty_background_ratio = 2 Any more optimization to add to this? Guy -- Guy Boisvert, ing. IngTegration inc. http://www.ingtegration.com https://www.linkedin.com/in/guy-boisvert-8990487 AVIS DE CONFIDENTIALITE : ce message peut contenir des renseignements confidentiels appartenant exclusivement a IngTegration Inc. ou a ses filiales. Si vous n'etes pas le destinataire indique ou prevu dans ce message (ou responsable de livrer ce message a la personne indiquee ou prevue) ou si vous pensez que ce message vous a ete adresse par erreur, vous ne pouvez pas utiliser ou reproduire ce message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous devez le detruire et vous etes prie d'avertir l'expediteur en repondant au courriel. CONFIDENTIALITY NOTICE : Proprietary/Confidential Information belonging to IngTegration Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply email. From budic at onholyground.com Mon Aug 19 19:58:51 2019 From: budic at onholyground.com (Darrell Budic) Date: Mon, 19 Aug 2019 14:58:51 -0500 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> Message-ID: <9965F827-56AA-4683-BBE7-31F67F1609F6@onholyground.com> You want sharding for sure, it keeps the entire disk from being locked while it heals. So you usually don?t notice it when you reboot a system, say. It?s fine to enable after the fact, but existing files won?t be sharded. You can work around this by stopping the VM and copying the file to new location, then renaming it over the old version. If you?re running something that lets you migrate live volume, you can create a new share with sharding enabled, then migrate the volume. > On Aug 19, 2019, at 12:01 PM, Carl Sirotic wrote: > > No, I didn't. > > I am very interested about these settings. > > Also, is it possible to turn the shard feature AFTER the volume was started to be used ? > > > Carl > > On 2019-08-19 12:08 p.m., Darrell Budic wrote: >> You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume? >> >>> On Aug 19, 2019, at 11:05 AM, Carl Sirotic wrote: >>> >>> Yes, I made sure there was no heal. >>> This is what I am suspecting thet shutting down a host isn't the right way to go. >>> Hi Carl, Did you check for any pending heals before rebooting the gluster server? Also, it was discussed that shutting down the node, does not stop the bricks properly and thus the clients will eait for a timeout before restoring full functionality. You can stop your glusterd and actually all processes by using a script in /usr/share/gluster/scripts (the path is based on memory and could be wrong). Best Regards, Strahil NikllovOn Aug 19, 2019 18:34, Carl Sirotic wrote: > > Hi, > > we have a replicate 3 cluster. > > 2 other servers are clients that run VM that are stored on the Gluster > volumes. > > I had to reboot one of the brick for maintenance. > > The whole VM setup went super slow and some of the client crashed. > > I think there is some timeout setting for KVM/Qemu vs timeout of > Glusterd that could fix this. > > Do anyone have an idea ? > > The whole point of having gluster for me was to be able to shut down one > of the host while the vm stay running. > > > Carl > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users AVIS DE CONFIDENTIALIT? : Ce courriel peut contenir de l'information privil?gi?e et confidentielle. Nous vous demandons de le d?truire imm?diatement si vous n'?tes pas le destinataire. CONFIDENTIALITY NOTICE: This email may contain information that is privileged and confidential. Please delete immediately if you are not the intended recipient. _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users > From budic at onholyground.com Mon Aug 19 20:15:46 2019 From: budic at onholyground.com (Darrell Budic) Date: Mon, 19 Aug 2019 15:15:46 -0500 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: <7b14c43d-fdf6-7c07-4715-3d1b78831227@ingtegration.com> References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> <7b14c43d-fdf6-7c07-4715-3d1b78831227@ingtegration.com> Message-ID: /var/lib/glusterd/groups/virt is a good start for ideas, notably some thread settings and choose-local=off to improve read performance. If you don?t have at least 10 cores on your servers, you may want to lower the recommended shd-max-threads=8 to no more than half your CPU cores to keep healing from swamping out regular work. It?s also starting to depend on what your backing store and networking setup are, so you?re going to want to test changes and find what works best for your setup. In addition to the virt group settings, I use these on most of my volumes, SSD or HDD backed, with the default 64M shard size: performance.io-thread-count: 32 # seemed good for my system, particularly a ZFS backed volume with lots of spindles client.event-threads: 8 cluster.data-self-heal-algorithm: full # 10G networking, uses more net/less cpu to heal. probably don?t use this for 1G networking? performance.stat-prefetch: on cluster.read-hash-mode: 3 # distribute reads to least loaded server (by read queue depth) and these two only on my HDD backed volume: performance.cache-size: 1G performance.write-behind-window-size: 64MB but I suspect these two need another round or six of tuning to tell if they are making a difference. I use the throughput-performance tuned profile on my servers, so you should be in good shape there. > On Aug 19, 2019, at 12:22 PM, Guy Boisvert wrote: > > On 2019-08-19 12:08 p.m., Darrell Budic wrote: >> You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume? > > That's what we did here: > > > gluster volume set W2K16_Rhenium cluster.quorum-type auto > gluster volume set W2K16_Rhenium network.ping-timeout 10 > gluster volume set W2K16_Rhenium auth.allow \* > gluster volume set W2K16_Rhenium group virt > gluster volume set W2K16_Rhenium storage.owner-uid 36 > gluster volume set W2K16_Rhenium storage.owner-gid 36 > gluster volume set W2K16_Rhenium features.shard on > gluster volume set W2K16_Rhenium features.shard-block-size 256MB > gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm full > gluster volume set W2K16_Rhenium performance.low-prio-threads 32 > > tuned-adm profile random-io (a profile i added in CentOS 7) > > > cat /usr/lib/tuned/random-io/tuned.conf > =========================================== > [main] > summary=Optimize for Gluster virtual machine storage > include=throughput-performance > > [sysctl] > > vm.dirty_ratio = 5 > vm.dirty_background_ratio = 2 > > > Any more optimization to add to this? > > > Guy > > -- > Guy Boisvert, ing. > IngTegration inc. > http://www.ingtegration.com > https://www.linkedin.com/in/guy-boisvert-8990487 > > AVIS DE CONFIDENTIALITE : ce message peut contenir des > renseignements confidentiels appartenant exclusivement a > IngTegration Inc. ou a ses filiales. Si vous n'etes pas > le destinataire indique ou prevu dans ce message (ou > responsable de livrer ce message a la personne indiquee ou > prevue) ou si vous pensez que ce message vous a ete adresse > par erreur, vous ne pouvez pas utiliser ou reproduire ce > message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous > devez le detruire et vous etes prie d'avertir l'expediteur > en repondant au courriel. > > CONFIDENTIALITY NOTICE : Proprietary/Confidential Information > belonging to IngTegration Inc. and its affiliates may be > contained in this message. If you are not a recipient > indicated or intended in this message (or responsible for > delivery of this message to such person), or you think for > any reason that this message may have been addressed to you > in error, you may not use or copy or deliver this message to > anyone else. In such case, you should destroy this message > and are asked to notify the sender by reply email. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amudhan83 at gmail.com Tue Aug 20 06:22:44 2019 From: amudhan83 at gmail.com (Amudhan P) Date: Tue, 20 Aug 2019 11:52:44 +0530 Subject: [Gluster-users] du output showing corrupt file system In-Reply-To: References: Message-ID: Hi, Can anyone suggest what could be the error and to fix this issue? regards Amudhan P On Sat, Aug 17, 2019 at 6:59 PM Amudhan P wrote: > Hi, > > I am using Gluster version 3.10.1. > > Mounting volume through fuse mount and I have run the command du -hs > "directory" which holds many subdirectories. > some of the subdirectory given output with below message. > > du: WARNING: Circular directory structure. > This almost certainly means that you have a corrupted file system. > NOTIFY YOUR SYSTEM MANAGER. > The following directory is part of the cycle: > > what could be the issue or what should be done to fix this problem? > > regards > Amudhan P > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.kinney at gmail.com Tue Aug 20 11:38:27 2019 From: jim.kinney at gmail.com (Jim Kinney) Date: Tue, 20 Aug 2019 07:38:27 -0400 Subject: [Gluster-users] du output showing corrupt file system In-Reply-To: References: Message-ID: <2AD0B4D8-A73F-4A26-B090-20B8F5786AA0@gmail.com> That's not necessarily a gluster issue. Users can create symlinks from a subdirectory up to a parent and that will create a loop. On August 20, 2019 2:22:44 AM EDT, Amudhan P wrote: >Hi, > >Can anyone suggest what could be the error and to fix this issue? > >regards >Amudhan P > >On Sat, Aug 17, 2019 at 6:59 PM Amudhan P wrote: > >> Hi, >> >> I am using Gluster version 3.10.1. >> >> Mounting volume through fuse mount and I have run the command du -hs >> "directory" which holds many subdirectories. >> some of the subdirectory given output with below message. >> >> du: WARNING: Circular directory structure. >> This almost certainly means that you have a corrupted file system. >> NOTIFY YOUR SYSTEM MANAGER. >> The following directory is part of the cycle: >> >> what could be the issue or what should be done to fix this problem? >> >> regards >> Amudhan P >> >> -- Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alpha754293 at hotmail.com Tue Aug 20 03:37:23 2019 From: alpha754293 at hotmail.com (Ewen Chan) Date: Tue, 20 Aug 2019 03:37:23 +0000 Subject: [Gluster-users] Can I create a distributed dispersed volume with RAM drives? Message-ID: To Whom It May Concern: I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. I have created a tmpfs mount point using half of the RAM. I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. Can I create a distributed dispersed volume with RAM drives? I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. Any help in regards to this matter would be greatly appreciated! Thank you. Sincerely, Ewen Chan From hunter86_bg at yahoo.com Tue Aug 20 16:21:59 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Tue, 20 Aug 2019 19:21:59 +0300 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes Message-ID: <3u4e0hxsskrj80dduro8f65g.1566318119056@email.android.com> Yes you can start it afterwards, BUT DO NOT STOP it once enabled ! Bad things happen :D Best Regards, Strahil NikolovOn Aug 19, 2019 20:01, Carl Sirotic wrote: > > No, I didn't. > > I am very interested about these settings. > > Also, is it possible to turn the shard feature AFTER the volume was > started to be used ? > > > Carl > > On 2019-08-19 12:08 p.m., Darrell Budic wrote: > > You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume? > > > >> On Aug 19, 2019, at 11:05 AM, Carl Sirotic wrote: > >> > >> Yes, I made sure there was no heal. > >> This is what I am suspecting thet shutting down a host isn't the right way to go. > >> Hi Carl, Did you check for any pending heals before rebooting the gluster server? Also, it was discussed that shutting down the node, does not stop the bricks properly and thus the clients will eait for a timeout before restoring full functionality.? You can stop your glusterd and actually all processes by using a script in /usr/share/gluster/scripts (the path is based on memory and could be wrong). Best Regards, Strahil NikllovOn Aug 19, 2019 18:34, Carl Sirotic wrote: > > Hi, > > we have a replicate 3 cluster. > > 2 other servers are clients that run VM that are stored on the Gluster > volumes. > > I had to reboot one of the brick for maintenance. > > The whole VM setup went super slow and some of the client crashed. > > I think there is some timeout setting for KVM/Qemu vs timeout of > Glusterd that could fix this. > > Do anyone have an idea ? > > The whole point of having gluster for me was to be able to shut down one > of the host while the vm stay running. > > > Carl > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users AVIS DE CONFIDENTIALIT? : Ce courriel peut contenir de l'information privil?gi?e et confidentielle. Nous vous demandons de le d?truire imm?diatement si vous n'?tes pas le destinataire. CONFIDENTIALITY NOTICE: This email may contain information that is privileged and confidential. Please delete immediately if you are not the intended recipient. _______________________________________________ > >> Gluster-users mailing list > >> Gluster-users at gluster.org > >> https://lists.gluster.org/mailman/listinfo/gluster-users > From vbellur at redhat.com Tue Aug 20 22:21:24 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Tue, 20 Aug 2019 15:21:24 -0700 Subject: [Gluster-users] Can I create a distributed dispersed volume with RAM drives? In-Reply-To: References: Message-ID: On Tue, Aug 20, 2019 at 8:13 AM Ewen Chan wrote: > To Whom It May Concern: > > I am currently using CentOS 7.6.1810 on four compute nodes, where each > node has 128 GB of RAM. > > I have created a tmpfs mount point using half of the RAM. > > I read the CentOS SIG Gluster Quickstart wiki/document ( > https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). > In there, it was creating a replicated volume. > > Can I create a distributed dispersed volume with RAM drives? > > I was able to create a distributed volume, but I am having difficulties > trying to create a distributed dispersed volume. > > > Can you provide more details about the problems encountered while creating a distributed dispersed volume? Thanks, Vijay -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbellur at redhat.com Tue Aug 20 23:50:23 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Tue, 20 Aug 2019 16:50:23 -0700 Subject: [Gluster-users] Can I create a distributed dispersed volume with RAM drives? In-Reply-To: References: Message-ID: On Tue, Aug 20, 2019 at 4:47 PM Ewen Chan wrote: > Vijay: > > Here is what it says: > > [root at node1 brick1]# gluster volume create gv0 stripe 4 > node1:/bricks/brick1/gv0 node2:/bricks/brick1/gv0 node3:/bricks/brick1/gv0 > node4:/bricks/brick1/gv0 > stripe option not supported > > Usage: > volume create [stripe ] [replica [arbiter > ]] [disperse []] [disperse-data ] [red > undancy ] [transport ] ... [force] > > It says that the stripe option is not supported, but then in the usage > right below it, it gives the correct syntax for how to use the gluster > volume create command. > Creating stripe volumes has been deprecated in recent releases. You can create a dispersed volume using the following syntax: # gluster volume create [disperse []] [redundancy ] HTH, Vijay > > Your help in this matter is greatly appreciated. > > Thank you. > > Sincerely, > Ewen > > ------------------------------ > *From:* Vijay Bellur > *Sent:* August 20, 2019 6:21 PM > *To:* Ewen Chan > *Cc:* gluster-users at gluster.org > *Subject:* Re: [Gluster-users] Can I create a distributed dispersed > volume with RAM drives? > > > > On Tue, Aug 20, 2019 at 8:13 AM Ewen Chan wrote: > > To Whom It May Concern: > > I am currently using CentOS 7.6.1810 on four compute nodes, where each > node has 128 GB of RAM. > > I have created a tmpfs mount point using half of the RAM. > > I read the CentOS SIG Gluster Quickstart wiki/document ( > https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). > In there, it was creating a replicated volume. > > Can I create a distributed dispersed volume with RAM drives? > > I was able to create a distributed volume, but I am having difficulties > trying to create a distributed dispersed volume. > > > > Can you provide more details about the problems encountered while creating > a distributed dispersed volume? > > Thanks, > Vijay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amudhan83 at gmail.com Wed Aug 21 07:49:45 2019 From: amudhan83 at gmail.com (Amudhan P) Date: Wed, 21 Aug 2019 13:19:45 +0530 Subject: [Gluster-users] du output showing corrupt file system In-Reply-To: <2AD0B4D8-A73F-4A26-B090-20B8F5786AA0@gmail.com> References: <2AD0B4D8-A73F-4A26-B090-20B8F5786AA0@gmail.com> Message-ID: it is definitely issue with gluster there is no symlink involved. On Tue, Aug 20, 2019 at 5:08 PM Jim Kinney wrote: > That's not necessarily a gluster issue. Users can create symlinks from a > subdirectory up to a parent and that will create a loop. > > > On August 20, 2019 2:22:44 AM EDT, Amudhan P wrote: >> >> Hi, >> >> Can anyone suggest what could be the error and to fix this issue? >> >> regards >> Amudhan P >> >> On Sat, Aug 17, 2019 at 6:59 PM Amudhan P wrote: >> >>> Hi, >>> >>> I am using Gluster version 3.10.1. >>> >>> Mounting volume through fuse mount and I have run the command du -hs >>> "directory" which holds many subdirectories. >>> some of the subdirectory given output with below message. >>> >>> du: WARNING: Circular directory structure. >>> This almost certainly means that you have a corrupted file system. >>> NOTIFY YOUR SYSTEM MANAGER. >>> The following directory is part of the cycle: >>> >>> what could be the issue or what should be done to fix this problem? >>> >>> regards >>> Amudhan P >>> >>> > -- > Sent from my Android device with K-9 Mail. All tyopes are thumb related > and reflect authenticity. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Wed Aug 21 10:09:37 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Wed, 21 Aug 2019 13:09:37 +0300 Subject: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? Message-ID: In such setup you definately need a remote DR volume... If you create your volumes as ramfs , you should not have issues, but use systemd.mount units - 1. For the ramfs , 2. For glusterd to require the ramfs mount point. Best Regards, Strahil NikolovOn Aug 20, 2019 06:37, Ewen Chan wrote: > > To Whom It May Concern: > > I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. > > I have created a tmpfs mount point using half of the RAM. > > I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. > > Can I create a distributed dispersed volume with RAM drives? > > I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. > > Any help in regards to this matter would be greatly appreciated! > > Thank you. > > Sincerely, > > Ewen Chan > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From alpha754293 at hotmail.com Tue Aug 20 23:47:08 2019 From: alpha754293 at hotmail.com (Ewen Chan) Date: Tue, 20 Aug 2019 23:47:08 +0000 Subject: [Gluster-users] Can I create a distributed dispersed volume with RAM drives? In-Reply-To: References: , Message-ID: Vijay: Here is what it says: [root at node1 brick1]# gluster volume create gv0 stripe 4 node1:/bricks/brick1/gv0 node2:/bricks/brick1/gv0 node3:/bricks/brick1/gv0 node4:/bricks/brick1/gv0 stripe option not supported Usage: volume create [stripe ] [replica [arbiter ]] [disperse []] [disperse-data ] [red undancy ] [transport ] ... [force] It says that the stripe option is not supported, but then in the usage right below it, it gives the correct syntax for how to use the gluster volume create command. Your help in this matter is greatly appreciated. Thank you. Sincerely, Ewen ________________________________ From: Vijay Bellur Sent: August 20, 2019 6:21 PM To: Ewen Chan Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Can I create a distributed dispersed volume with RAM drives? On Tue, Aug 20, 2019 at 8:13 AM Ewen Chan > wrote: To Whom It May Concern: I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. I have created a tmpfs mount point using half of the RAM. I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. Can I create a distributed dispersed volume with RAM drives? I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. Can you provide more details about the problems encountered while creating a distributed dispersed volume? Thanks, Vijay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.kinney at gmail.com Wed Aug 21 12:52:32 2019 From: jim.kinney at gmail.com (Jim Kinney) Date: Wed, 21 Aug 2019 08:52:32 -0400 Subject: [Gluster-users] du output showing corrupt file system In-Reply-To: References: <2AD0B4D8-A73F-4A26-B090-20B8F5786AA0@gmail.com> Message-ID: Run the du command in the source space. A symlink that uses relative pathing can turn into a problem on a new mount. That said, I've seen "too many levels of linking" errors associated with the gfids dir .glusterfs and gfids of real dirs that are chained links to other dirs. It's still a user space symlink error. It's just compounded by gluster. On August 21, 2019 3:49:45 AM EDT, Amudhan P wrote: >it is definitely issue with gluster there is no symlink involved. > > >On Tue, Aug 20, 2019 at 5:08 PM Jim Kinney >wrote: > >> That's not necessarily a gluster issue. Users can create symlinks >from a >> subdirectory up to a parent and that will create a loop. >> >> >> On August 20, 2019 2:22:44 AM EDT, Amudhan P >wrote: >>> >>> Hi, >>> >>> Can anyone suggest what could be the error and to fix this issue? >>> >>> regards >>> Amudhan P >>> >>> On Sat, Aug 17, 2019 at 6:59 PM Amudhan P >wrote: >>> >>>> Hi, >>>> >>>> I am using Gluster version 3.10.1. >>>> >>>> Mounting volume through fuse mount and I have run the command du >-hs >>>> "directory" which holds many subdirectories. >>>> some of the subdirectory given output with below message. >>>> >>>> du: WARNING: Circular directory structure. >>>> This almost certainly means that you have a corrupted file system. >>>> NOTIFY YOUR SYSTEM MANAGER. >>>> The following directory is part of the cycle: >>>> >>>> what could be the issue or what should be done to fix this problem? >>>> >>>> regards >>>> Amudhan P >>>> >>>> >> -- >> Sent from my Android device with K-9 Mail. All tyopes are thumb >related >> and reflect authenticity. >> -- Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alpha754293 at hotmail.com Wed Aug 21 02:09:42 2019 From: alpha754293 at hotmail.com (Ewen Chan) Date: Wed, 21 Aug 2019 02:09:42 +0000 Subject: [Gluster-users] Does the GlusterFS FUSE client communicate only via TCP or how does it work? Message-ID: To Whom It May Concern: My four compute nodes are all dual Xeon nodes, with 128 GB of RAM each, and a Mellanox ConnectX-4 100 Gbps 4X EDR Infiniband NIC each, connected to a Mellanox 4X EDR IB switch. The GlusterFS volume was created using this command: # gluster volume create gv0 transport=rdma node1:/bricks/brick1/gv0 node2:/bricks/brick1/gv0 node3:/bricks/brick1/gv0 node4:/bricks/brick1/gv0 volume create: gv0: success: please start the volume to access data [root at node1 ewen]# gluster volume start gv0 volume start: gv0: success [root at node1 ewen]# gluster volume info Volume Name: gv0 Type: Distribute Volume ID: 105dc537-b5bc-4561-b460-fc022f1f8033 Status: Started Snapshot Count: 0 Number of Bricks: 4 Transport-type: tcp,rdma Bricks: Brick1: node1:/bricks/brick1/gv0 Brick2: node2:/bricks/brick1/gv0 Brick3: node3:/bricks/brick1/gv0 Brick4: node4:/bricks/brick1/gv0 Options Reconfigured: nfs.disable: on [root at node1 ewen]# gluster volume status Status of volume: gv0 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick node1:/bricks/brick1/gv0 49152 49153 Y 16554 Brick node2:/bricks/brick1/gv0 49152 49153 Y 16116 Brick node3:/bricks/brick1/gv0 49152 49153 Y 16319 Brick node4:/bricks/brick1/gv0 49152 49153 Y 16403 Task Status of Volume gv0 ------------------------------------------------------------------------------ There are no active volume tasks I was trying to mount the GlusterFS volume using this command: # mount -t glusterfs -o transport=rdma node1:/gv0 /mnt/gv0 and it failed to mount. The error message in the log said (/var/log/glusterfs/gv0.log): [2019-08-21 00:27:35.796350] W [MSGID: 103071] [rdma.c:1277:gf_rdma_cm_event_handler] 0-gv0-client-0: cma event RDMA_CM_EVENT_REJE CTED, error 28 (me:10.0.1.1:49151 peer:10.0.1.1:24008) [2019-08-21 00:27:35.848728] W [MSGID: 103071] [rdma.c:1277:gf_rdma_cm_event_handler] 0-gv0-client-2: cma event RDMA_CM_EVENT_REJE CTED, error 28 (me:10.0.1.1:49149 peer:10.0.1.3:24008) [2019-08-21 00:27:35.861258] W [MSGID: 103071] [rdma.c:1277:gf_rdma_cm_event_handler] 0-gv0-client-3: cma event RDMA_CM_EVENT_REJE CTED, error 28 (me:10.0.1.1:49148 peer:10.0.1.4:24008) [2019-08-21 00:27:35.883548] W [MSGID: 103071] [rdma.c:1277:gf_rdma_cm_event_handler] 0-gv0-client-1: cma event RDMA_CM_EVENT_REJE CTED, error 28 (me:10.0.1.1:49150 peer:10.0.1.2:24008) [2019-08-21 00:27:35.887479] I [fuse-bridge.c:5142:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 ?kernel 7.22 [2019-08-21 00:27:35.887519] I [fuse-bridge.c:5753:fuse_graph_sync] 0-fuse: switched to graph 0 [2019-08-21 00:27:35.888059] I [dict.c:560:dict_get] (-->/usr/lib64/glusterfs/6.5/xlator/protocol/client.so(+0xa940) [0x7fe7acf2a9 40] -->/usr/lib64/glusterfs/6.5/xlator/cluster/distribute.so(+0x45348) [0x7fe7accac348] -->/lib64/libglusterfs.so.0(dict_get+0x94) ?[0x7fe7bbade1b4] ) 0-dict: !this || key=trusted.glusterfs.dht.mds [Invalid argument] [2019-08-21 00:27:35.888167] E [fuse-bridge.c:5211:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is n ot connected) [2019-08-21 00:27:35.888387] I [dict.c:560:dict_get] (-->/usr/lib64/glusterfs/6.5/xlator/protocol/client.so(+0xa940) [0x7fe7acf2a9 40] -->/usr/lib64/glusterfs/6.5/xlator/cluster/distribute.so(+0x45348) [0x7fe7accac348] -->/lib64/libglusterfs.so.0(dict_get+0x94) ?[0x7fe7bbade1b4] ) 0-dict: !this || key=trusted.glusterfs.dht.mds [Invalid argument] I looked up the RDMA error and disabled SELinux entirely: # cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=minimum It still wouldn't mount. So I deleted the GlusterFS volume and re-created the volume using the same bricks, but added the TCP protocol to it: # gluster volume create gv0 transport=tcp,rdma node1:/bricks/brick1/gv0 node2:/bricks/brick1/gv0 node3:/bricks/brick1/gv0 node4:/bricks/brick1/gv0 volume create: gv0: success: please start the volume to access data [root at node1 ewen]# gluster volume start gv0 volume start: gv0: success and then mounted the glusterfs volume: # mount -t glusterfs -o transport=rdma node1:/gv0 /mnt/gv0 and that worked. Does the GlusterFS FUSE client communicate only via TCP? Your assistance in helping me understand why the RDMA GlusterFS volume would mount with both TCP and RDMA protocols are enabled, but it wouldn't mount when I only specified just the RDMA protocol would be greatly appreciated. Thank you. Sincerely, Ewen From alpha754293 at hotmail.com Wed Aug 21 10:47:47 2019 From: alpha754293 at hotmail.com (Ewen Chan) Date: Wed, 21 Aug 2019 10:47:47 +0000 Subject: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? In-Reply-To: References: Message-ID: Strahil: Thank you. I was doing some research in regards to the difference between ramfs and tmpfs and what others were saying about ramfs is that you can't actually set a ramfs mount point to have a fixed size, so it will keep filling up until you run out of RAM which will either cause the application to crash or it would case it to start swapping heavily. This is why I went with tmpfs instead because with tmpfs, you can limit it to a fixed size, so it won't grow indefinitely until you run out of RAM, which can cause the host system to hang. Thanks. Sincerely, Ewen ________________________________ From: Strahil Sent: August 21, 2019 6:09 AM To: Ewen ; gluster-users Subject: Re: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? In such setup you definately need a remote DR volume... If you create your volumes as ramfs , you should not have issues, but use systemd.mount units - 1. For the ramfs , 2. For glusterd to require the ramfs mount point. Best Regards, Strahil NikolovOn Aug 20, 2019 06:37, Ewen Chan wrote: > > To Whom It May Concern: > > I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. > > I have created a tmpfs mount point using half of the RAM. > > I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. > > Can I create a distributed dispersed volume with RAM drives? > > I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. > > Any help in regards to this matter would be greatly appreciated! > > Thank you. > > Sincerely, > > Ewen Chan > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Wed Aug 21 20:32:50 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Wed, 21 Aug 2019 23:32:50 +0300 Subject: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? Message-ID: <4qwildkf6b5wfrhskvs10w5i.1566419570393@email.android.com> Yeah, you are right... I forgot that subtle difference. You still need a DR volume in case data is precious . UPS is also nice to have. You should consider sharding and also NUMA pinning. For example if your node has 2 numa nodes and 128 GB per numa node - you should create 2 tmpfs (pinned per numa) of 128GB each.If you create a single 256 GB tmpfs perforamnce will be sporadic. You can google 'SAP HANA fast reboot' for an example. Have you thought about 'replicated dispersed' type of volumes ? What about network - RDMA ? Best Regards, Strahil NikolovOn Aug 21, 2019 13:47, Ewen Chan wrote: > > Strahil: > > Thank you. > > I was doing some research in regards to the difference between ramfs and tmpfs and what others were saying about ramfs is that you can't actually set a ramfs mount point to have a fixed size, so it will keep filling up until you run out of RAM which will either cause the application to crash or it would case it to start swapping heavily. > > This is why I went with tmpfs instead because with tmpfs, you can limit it to a fixed size, so it won't grow indefinitely until you run out of RAM, which can cause the host system to hang. > > Thanks. > > Sincerely, > Ewen > > ________________________________ > From: Strahil > Sent: August 21, 2019 6:09 AM > To: Ewen ; gluster-users > Subject: Re: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? > ? > In such setup you definately need? a remote DR volume... > > If you create your volumes as ramfs , you should not have issues, but use systemd.mount units - 1. For the ramfs , 2. For glusterd to require? the ramfs mount point. > > Best Regards, > Strahil NikolovOn Aug 20, 2019 06:37, Ewen Chan wrote: > > > > To Whom It May Concern: > > > > I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. > > > > I have created a tmpfs mount point using half of the RAM. > > > > I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. > > > > Can I create a distributed dispersed volume with RAM drives? > > > > I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. > > > > Any help in regards to this matter would be greatly appreciated! > > > > Thank you. > > > > Sincerely, > > > > Ewen Chan > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From alpha754293 at hotmail.com Wed Aug 21 02:39:28 2019 From: alpha754293 at hotmail.com (Ewen Chan) Date: Wed, 21 Aug 2019 02:39:28 +0000 Subject: [Gluster-users] Can I create a distributed dispersed volume with RAM drives? In-Reply-To: References: , Message-ID: Vijay: I just tried that and the results using this command: # for i in `seq -w 1 4`; do dd if=/dev/zero of=10Gfile$1 bs=1024k count=10240; done is about 1/4th the write performance of a distributed volume. (I'm currently testing with a 64 GB RAM drive on each of my four compute nodes that are then tied together using GlusterFS). With a distributed volume, and with direct-io-mode=enable, at best, I can execute that same task at around 2 GB/s. (The host system could execute that task at around 2.8 GB/s to /dev/shm.) With a dispersed volume with 1 brick for redundancy, it drops to around 500 MB/s at best. Is there no other way to be get the kind of performance that stripping should theorectically provide? Thank you. Sincerely, Ewen ________________________________ From: Vijay Bellur Sent: August 20, 2019 7:50 PM To: Ewen Chan Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Can I create a distributed dispersed volume with RAM drives? On Tue, Aug 20, 2019 at 4:47 PM Ewen Chan > wrote: Vijay: Here is what it says: [root at node1 brick1]# gluster volume create gv0 stripe 4 node1:/bricks/brick1/gv0 node2:/bricks/brick1/gv0 node3:/bricks/brick1/gv0 node4:/bricks/brick1/gv0 stripe option not supported Usage: volume create [stripe ] [replica [arbiter ]] [disperse []] [disperse-data ] [red undancy ] [transport ] ... [force] It says that the stripe option is not supported, but then in the usage right below it, it gives the correct syntax for how to use the gluster volume create command. Creating stripe volumes has been deprecated in recent releases. You can create a dispersed volume using the following syntax: # gluster volume create [disperse []] [redundancy ] HTH, Vijay Your help in this matter is greatly appreciated. Thank you. Sincerely, Ewen ________________________________ From: Vijay Bellur > Sent: August 20, 2019 6:21 PM To: Ewen Chan > Cc: gluster-users at gluster.org > Subject: Re: [Gluster-users] Can I create a distributed dispersed volume with RAM drives? On Tue, Aug 20, 2019 at 8:13 AM Ewen Chan > wrote: To Whom It May Concern: I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. I have created a tmpfs mount point using half of the RAM. I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. Can I create a distributed dispersed volume with RAM drives? I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. Can you provide more details about the problems encountered while creating a distributed dispersed volume? Thanks, Vijay -------------- next part -------------- An HTML attachment was scrubbed... URL: From alpha754293 at hotmail.com Wed Aug 21 20:43:04 2019 From: alpha754293 at hotmail.com (Ewen Chan) Date: Wed, 21 Aug 2019 20:43:04 +0000 Subject: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? In-Reply-To: <4qwildkf6b5wfrhskvs10w5i.1566419570393@email.android.com> References: <4qwildkf6b5wfrhskvs10w5i.1566419570393@email.android.com> Message-ID: Strahil: Good ideas. No, the data isn't really all that precious. It's temporary scratch data for HPC simulations. The systems are on a UPS. A straight dispersed volume results in data transfer rates that's 1/4 of just a distributed volume (~500 MB/s vs 2 GB/s). I'm also already using 4x EDR Infiniband (100 Gbps), with RDMA and direct-io enabled. Thank you. Sincerely, Ewen ________________________________ From: Strahil Sent: August 21, 2019 4:32 PM To: Ewen Cc: gluster-users Subject: Re: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? Yeah, you are right... I forgot that subtle difference. You still need a DR volume in case data is precious . UPS is also nice to have. You should consider sharding and also NUMA pinning. For example if your node has 2 numa nodes and 128 GB per numa node - you should create 2 tmpfs (pinned per numa) of 128GB each.If you create a single 256 GB tmpfs perforamnce will be sporadic. You can google 'SAP HANA fast reboot' for an example. Have you thought about 'replicated dispersed' type of volumes ? What about network - RDMA ? Best Regards, Strahil Nikolov On Aug 21, 2019 13:47, Ewen Chan wrote: Strahil: Thank you. I was doing some research in regards to the difference between ramfs and tmpfs and what others were saying about ramfs is that you can't actually set a ramfs mount point to have a fixed size, so it will keep filling up until you run out of RAM which will either cause the application to crash or it would case it to start swapping heavily. This is why I went with tmpfs instead because with tmpfs, you can limit it to a fixed size, so it won't grow indefinitely until you run out of RAM, which can cause the host system to hang. Thanks. Sincerely, Ewen ________________________________ From: Strahil Sent: August 21, 2019 6:09 AM To: Ewen ; gluster-users Subject: Re: [Gluster-users] Can I create a distributed dispersed volume withRAM drives? In such setup you definately need a remote DR volume... If you create your volumes as ramfs , you should not have issues, but use systemd.mount units - 1. For the ramfs , 2. For glusterd to require the ramfs mount point. Best Regards, Strahil NikolovOn Aug 20, 2019 06:37, Ewen Chan wrote: > > To Whom It May Concern: > > I am currently using CentOS 7.6.1810 on four compute nodes, where each node has 128 GB of RAM. > > I have created a tmpfs mount point using half of the RAM. > > I read the CentOS SIG Gluster Quickstart wiki/document (https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart). In there, it was creating a replicated volume. > > Can I create a distributed dispersed volume with RAM drives? > > I was able to create a distributed volume, but I am having difficulties trying to create a distributed dispersed volume. > > Any help in regards to this matter would be greatly appreciated! > > Thank you. > > Sincerely, > > Ewen Chan > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From amudhan83 at gmail.com Thu Aug 22 05:57:59 2019 From: amudhan83 at gmail.com (Amudhan P) Date: Thu, 22 Aug 2019 11:27:59 +0530 Subject: [Gluster-users] du output showing corrupt file system In-Reply-To: References: <2AD0B4D8-A73F-4A26-B090-20B8F5786AA0@gmail.com> Message-ID: Hi Jim, "du" command was run from fuse mounted volume. it's a single mount point. Gluster should handle that issue right and I don't have any problem in accessing issue reported folders but only when running "du" command for the folder it throws error msg. regards Amudhan On Wed, Aug 21, 2019 at 6:22 PM Jim Kinney wrote: > Run the du command in the source space. > > A symlink that uses relative pathing can turn into a problem on a new > mount. > > That said, I've seen "too many levels of linking" errors associated with > the gfids dir .glusterfs and gfids of real dirs that are chained links to > other dirs. It's still a user space symlink error. It's just compounded by > gluster. > > On August 21, 2019 3:49:45 AM EDT, Amudhan P wrote: >> >> it is definitely issue with gluster there is no symlink involved. >> >> >> On Tue, Aug 20, 2019 at 5:08 PM Jim Kinney wrote: >> >>> That's not necessarily a gluster issue. Users can create symlinks from a >>> subdirectory up to a parent and that will create a loop. >>> >>> >>> On August 20, 2019 2:22:44 AM EDT, Amudhan P >>> wrote: >>>> >>>> Hi, >>>> >>>> Can anyone suggest what could be the error and to fix this issue? >>>> >>>> regards >>>> Amudhan P >>>> >>>> On Sat, Aug 17, 2019 at 6:59 PM Amudhan P wrote: >>>> >>>>> Hi, >>>>> >>>>> I am using Gluster version 3.10.1. >>>>> >>>>> Mounting volume through fuse mount and I have run the command du -hs >>>>> "directory" which holds many subdirectories. >>>>> some of the subdirectory given output with below message. >>>>> >>>>> du: WARNING: Circular directory structure. >>>>> This almost certainly means that you have a corrupted file system. >>>>> NOTIFY YOUR SYSTEM MANAGER. >>>>> The following directory is part of the cycle: >>>>> >>>>> what could be the issue or what should be done to fix this problem? >>>>> >>>>> regards >>>>> Amudhan P >>>>> >>>>> >>> -- >>> Sent from my Android device with K-9 Mail. All tyopes are thumb related >>> and reflect authenticity. >>> >> > -- > Sent from my Android device with K-9 Mail. All tyopes are thumb related > and reflect authenticity. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lntruong08 at yahoo.com Thu Aug 22 09:36:35 2019 From: lntruong08 at yahoo.com (Truong Le Ngoc) Date: Thu, 22 Aug 2019 09:36:35 +0000 (UTC) Subject: [Gluster-users] [server.c:499:server_rpc_notify] 0-volume name-server: disconnecting connection from CTX_ID References: <1739909572.745213.1566466595890.ref@mail.yahoo.com> Message-ID: <1739909572.745213.1566466595890@mail.yahoo.com> Dear, I am using the GlusterFS version 6.3.1.I have 4 nodes in the cluster, all volume created with the type?Distributed-Replicate. And I detected the log of the brick has a lot of messages below, please help me to fix this issue. The info my cluster:10.11.98.6 gtbkf10110.11.98.4 gtbkf10210.11.98.5 gtbkf10310.11.98.3 gtbkf104 The info log from /var/log/glusterfs/bricks/.log:[2019-08-22 09:32:23.458264] I [addr.c:54:compare_addr_and_update] 0-/data/GLUSTERFS/ldk: allowed = "10.11.98.*", received addr = "10.11.98.5"[2019-08-22 09:32:23.458295] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 4f9bb855-7511-4f89-be0f-480a73a5b9bf[2019-08-22 09:32:23.458305] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-ldk-server: accepted client from CTX_ID:bc82fb3f-1a34-4537-a564-0bcbfb790f19-GRAPH_ID:0-PID:76770-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0 (version: 6.3) with subvol /data/GLUSTERFS/ldk[2019-08-22 09:32:23.632438] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-ldk-server: disconnecting connection from CTX_ID:bc82fb3f-1a34-4537-a564-0bcbfb790f19-GRAPH_ID:0-PID:76770-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:23.632642] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-ldk-server: Shutting down connection CTX_ID:bc82fb3f-1a34-4537-a564-0bcbfb790f19-GRAPH_ID:0-PID:76770-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:37.605137] I [addr.c:54:compare_addr_and_update] 0-/data/GLUSTERFS/ldk: allowed = "10.11.98.*", received addr = "10.11.98.5"[2019-08-22 09:32:37.605180] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 4f9bb855-7511-4f89-be0f-480a73a5b9bf[2019-08-22 09:32:37.605191] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-ldk-server: accepted client from CTX_ID:95967659-08c6-41bf-bbde-bf2ee01aa566-GRAPH_ID:0-PID:78208-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0 (version: 6.3) with subvol /data/GLUSTERFS/ldk[2019-08-22 09:32:37.691807] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-ldk-server: disconnecting connection from CTX_ID:95967659-08c6-41bf-bbde-bf2ee01aa566-GRAPH_ID:0-PID:78208-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:37.691953] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-ldk-server: Shutting down connection CTX_ID:95967659-08c6-41bf-bbde-bf2ee01aa566-GRAPH_ID:0-PID:78208-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:37.719606] I [addr.c:54:compare_addr_and_update] 0-/data/GLUSTERFS/ldk: allowed = "10.11.98.*", received addr = "10.11.98.5"[2019-08-22 09:32:37.719638] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 4f9bb855-7511-4f89-be0f-480a73a5b9bf[2019-08-22 09:32:37.719675] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-ldk-server: accepted client from CTX_ID:f3e66801-121d-4644-a823-452ee9d7fd40-GRAPH_ID:0-PID:78225-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0 (version: 6.3) with subvol /data/GLUSTERFS/ldk[2019-08-22 09:32:37.805501] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-ldk-server: disconnecting connection from CTX_ID:f3e66801-121d-4644-a823-452ee9d7fd40-GRAPH_ID:0-PID:78225-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:37.805650] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-ldk-server: Shutting down connection CTX_ID:f3e66801-121d-4644-a823-452ee9d7fd40-GRAPH_ID:0-PID:78225-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:51.931701] I [addr.c:54:compare_addr_and_update] 0-/data/GLUSTERFS/ldk: allowed = "10.11.98.*", received addr = "10.11.98.5"[2019-08-22 09:32:51.931739] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 4f9bb855-7511-4f89-be0f-480a73a5b9bf[2019-08-22 09:32:51.931756] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-ldk-server: accepted client from CTX_ID:0fa81a91-2db9-4e7b-a924-8d5671957f35-GRAPH_ID:0-PID:79725-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0 (version: 6.3) with subvol /data/GLUSTERFS/ldk[2019-08-22 09:32:52.019716] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-ldk-server: disconnecting connection from CTX_ID:0fa81a91-2db9-4e7b-a924-8d5671957f35-GRAPH_ID:0-PID:79725-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:52.019872] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-ldk-server: Shutting down connection CTX_ID:0fa81a91-2db9-4e7b-a924-8d5671957f35-GRAPH_ID:0-PID:79725-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:52.049070] I [addr.c:54:compare_addr_and_update] 0-/data/GLUSTERFS/ldk: allowed = "10.11.98.*", received addr = "10.11.98.5"[2019-08-22 09:32:52.049114] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 4f9bb855-7511-4f89-be0f-480a73a5b9bf[2019-08-22 09:32:52.049132] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-ldk-server: accepted client from CTX_ID:f3313eb5-2fc0-45fa-84e7-d1088b13e3f0-GRAPH_ID:0-PID:79742-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0 (version: 6.3) with subvol /data/GLUSTERFS/ldk[2019-08-22 09:32:52.134908] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-ldk-server: disconnecting connection from CTX_ID:f3313eb5-2fc0-45fa-84e7-d1088b13e3f0-GRAPH_ID:0-PID:79742-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0[2019-08-22 09:32:52.135070] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-ldk-server: Shutting down connection CTX_ID:f3313eb5-2fc0-45fa-84e7-d1088b13e3f0-GRAPH_ID:0-PID:79742-HOST:GT-GTBackupFarm-BKF103-PC_NAME:ldk-client-0-RECON_NO:-0 Thanks,Truong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.kinney at gmail.com Thu Aug 22 11:28:40 2019 From: jim.kinney at gmail.com (Jim Kinney) Date: Thu, 22 Aug 2019 07:28:40 -0400 Subject: [Gluster-users] du output showing corrupt file system In-Reply-To: References: <2AD0B4D8-A73F-4A26-B090-20B8F5786AA0@gmail.com> Message-ID: Try running du not in the fuse mounted folder but in the folder on the server providing it. On August 22, 2019 1:57:59 AM EDT, Amudhan P wrote: >Hi Jim, > >"du" command was run from fuse mounted volume. it's a single mount >point. > >Gluster should handle that issue right and I don't have any problem in >accessing issue reported folders but only when running "du" command for >the >folder it throws error msg. > >regards >Amudhan > > >On Wed, Aug 21, 2019 at 6:22 PM Jim Kinney >wrote: > >> Run the du command in the source space. >> >> A symlink that uses relative pathing can turn into a problem on a new >> mount. >> >> That said, I've seen "too many levels of linking" errors associated >with >> the gfids dir .glusterfs and gfids of real dirs that are chained >links to >> other dirs. It's still a user space symlink error. It's just >compounded by >> gluster. >> >> On August 21, 2019 3:49:45 AM EDT, Amudhan P >wrote: >>> >>> it is definitely issue with gluster there is no symlink involved. >>> >>> >>> On Tue, Aug 20, 2019 at 5:08 PM Jim Kinney >wrote: >>> >>>> That's not necessarily a gluster issue. Users can create symlinks >from a >>>> subdirectory up to a parent and that will create a loop. >>>> >>>> >>>> On August 20, 2019 2:22:44 AM EDT, Amudhan P >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Can anyone suggest what could be the error and to fix this issue? >>>>> >>>>> regards >>>>> Amudhan P >>>>> >>>>> On Sat, Aug 17, 2019 at 6:59 PM Amudhan P >wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I am using Gluster version 3.10.1. >>>>>> >>>>>> Mounting volume through fuse mount and I have run the command du >-hs >>>>>> "directory" which holds many subdirectories. >>>>>> some of the subdirectory given output with below message. >>>>>> >>>>>> du: WARNING: Circular directory structure. >>>>>> This almost certainly means that you have a corrupted file >system. >>>>>> NOTIFY YOUR SYSTEM MANAGER. >>>>>> The following directory is part of the cycle: >>>>>> >>>>>> what could be the issue or what should be done to fix this >problem? >>>>>> >>>>>> regards >>>>>> Amudhan P >>>>>> >>>>>> >>>> -- >>>> Sent from my Android device with K-9 Mail. All tyopes are thumb >related >>>> and reflect authenticity. >>>> >>> >> -- >> Sent from my Android device with K-9 Mail. All tyopes are thumb >related >> and reflect authenticity. >> -- Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nux at li.nux.ro Thu Aug 22 16:20:28 2019 From: nux at li.nux.ro (Nux!) Date: Thu, 22 Aug 2019 17:20:28 +0100 Subject: [Gluster-users] geo-replication won't start Message-ID: <143d8bdc905173dd3743f45e67ebf8ee@li.nux.ro> Hi, I'm trying for the first time ever the geo-replication feature and I am not having much success (CentOS7, gluster 6.5). First of all, from the docs I get the impression that I can geo-replicate over ssh to a simple dir, but it doesn't seem to be the case, the "slave" must be a gluster volume, doesn't it? Second, the slave host is not in the subnet with the other gluster peers, but I reckon this would be the usual case and not a problem. I've stopped the firewall on all peers and slave host to rule it out, but I can't get the georep started. Creation is successfull, however STATUS won't change from Created. I'm looking through all the logs and I can't see anything meaningful. What steps could I take to debug this further? Cheers, Lucian -- Sent from the Delta quadrant using Borg technology! From benedikt.kaless at forumZFD.de Fri Aug 23 13:03:43 2019 From: benedikt.kaless at forumZFD.de (=?UTF-8?Q?Benedikt_Kale=c3=9f?=) Date: Fri, 23 Aug 2019 15:03:43 +0200 Subject: [Gluster-users] split brain heal - socket not connected Message-ID: <1361b0a9-941f-25df-60a2-cd40d84467ea@forumZFD.de> Hi, unfortunatelly I have a directory in split-brain state. If I do a gluster volume heal split-brain source-brick gluster:/glusterfs/brick I get a socket not connected. How can I manually heal that directory? Best regards and thanks in advance Bene -- ?forumZFD Entschieden f?r Frieden|Committed to Peace Benedikt Kale? Leiter Team IT|Head team IT Forum Ziviler Friedensdienst e.V.|Forum Civil Peace Service Am K?lner Brett 8 | 50825 K?ln | Germany Tel 0221 91273233 | Fax 0221 91273299 | http://www.forumZFD.de Vorstand nach ? 26 BGB, einzelvertretungsberechtigt|Executive Board: Oliver Knabe (Vorsitz|Chair), Sonja Wiekenberg-Mlalandle, Alexander Mauz VR 17651 Amtsgericht K?ln Spenden|Donations: IBAN DE37 3702 0500 0008 2401 01 BIC BFSWDE33XXX From csirotic at evoqarchitecture.com Fri Aug 23 13:53:53 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Fri, 23 Aug 2019 09:53:53 -0400 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> <7b14c43d-fdf6-7c07-4715-3d1b78831227@ingtegration.com> <70ee770d-c7fc-0629-1eef-b5c04703f935@evoqarchitecture.com> Message-ID: However, I must have misunderstood the whole concept of gluster. In a replica 3, for me, it's completely unacceptable, regardless of the options, that all my VMs go down when I reboot one node. The whole purpose of having a full 3 copy of my data on the fly is suposed to be this. I am in the process of sharding every file. But even if the healing time would be longer, I would still expect a non-sharded replica 3 brick with vm boot disk, to not go down if I reboot one of its copy. I am not very impressed by gluster so far. Carl On 2019-08-19 4:15 p.m., Darrell Budic wrote: > /var/lib/glusterd/groups/virt is a good start for ideas, notably some > thread settings and choose-local=off to improve read performance. If > you don?t have at least 10 cores on your servers, you may want to > lower the recommended shd-max-threads=8 to no more than half your CPU > cores to keep healing from swamping out regular work. > > It?s also starting to depend on what your backing store and networking > setup are, so you?re going to want to test changes and find what works > best for your setup. > > In addition to the virt group settings, I use these on most of my > volumes, SSD or HDD backed, with the default 64M shard size: > > performance.io -thread-count: 32# seemed good > for my system, particularly a ZFS backed volume with lots of spindles > client.event-threads: 8 > cluster.data-self-heal-algorithm: full# 10G networking, uses more > net/less cpu to heal. probably don?t use this for 1G networking? > performance.stat-prefetch: on > cluster.read-hash-mode: 3# distribute reads to least loaded server (by > read queue depth) > > and these two only on my HDD backed volume: > > performance.cache-size: 1G > performance.write-behind-window-size: 64MB > > but I suspect these two need another round or six of tuning to tell if > they are making a difference. > > I use the throughput-performance tuned profile on my servers, so you > should be in good shape there. > >> On Aug 19, 2019, at 12:22 PM, Guy Boisvert >> > > wrote: >> >> On 2019-08-19 12:08 p.m., Darrell Budic wrote: >>> You also need to make sure your volume is setup properly for best >>> performance. Did you apply the gluster virt group to your volumes, >>> or at least features.shard = on on your VM volume? >> >> That's what we did here: >> >> >> gluster volume set W2K16_Rhenium cluster.quorum-type auto >> gluster volume set W2K16_Rhenium network.ping-timeout 10 >> gluster volume set W2K16_Rhenium auth.allow \* >> gluster volume set W2K16_Rhenium group virt >> gluster volume set W2K16_Rhenium storage.owner-uid 36 >> gluster volume set W2K16_Rhenium storage.owner-gid 36 >> gluster volume set W2K16_Rhenium features.shard on >> gluster volume set W2K16_Rhenium features.shard-block-size 256MB >> gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm full >> gluster volume set W2K16_Rhenium performance.low-prio-threads 32 >> >> tuned-adm profile random-io??? ??? (a profile i added in CentOS 7) >> >> >> cat /usr/lib/tuned/random-io/tuned.conf >> =========================================== >> [main] >> summary=Optimize for Gluster virtual machine storage >> include=throughput-performance >> >> [sysctl] >> >> vm.dirty_ratio = 5 >> vm.dirty_background_ratio = 2 >> >> >> Any more optimization to add to this? >> >> >> Guy >> >> -- >> Guy Boisvert, ing. >> IngTegration inc. >> http://www.ingtegration.com >> https://www.linkedin.com/in/guy-boisvert-8990487 >> >> AVIS DE CONFIDENTIALITE : ce message peut contenir des >> renseignements confidentiels appartenant exclusivement a >> IngTegration Inc. ou a ses filiales. Si vous n'etes pas >> le destinataire indique ou prevu dans ce ?message (ou >> responsable de livrer ce message a la personne indiquee ou >> prevue) ou si vous pensez que ce message vous a ete adresse >> par erreur, vous ne pouvez pas utiliser ou reproduire ce >> message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous >> devez le detruire et vous etes prie d'avertir l'expediteur >> en repondant au courriel. >> >> CONFIDENTIALITY NOTICE : Proprietary/Confidential Information >> belonging to IngTegration Inc. and its affiliates may be >> contained in this message. If you are not a recipient >> indicated or intended in this message (or responsible for >> delivery of this message to such person), or you think for >> any reason that this message may have been addressed to you >> in error, you may not use or copy or deliver this message to >> anyone else. In such case, you should destroy this message >> and are asked to notify the sender by reply email. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From csirotic at evoqarchitecture.com Fri Aug 23 22:06:16 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Fri, 23 Aug 2019 18:06:16 -0400 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> <7b14c43d-fdf6-7c07-4715-3d1b78831227@ingtegration.com> <70ee770d-c7fc-0629-1eef-b5c04703f935@evoqarchitecture.com> <14de67b4-0b81-b88b-a526-2672e089d6cc@evoqarchitecture.com> Message-ID: Okay, so it means, at least I am not getting the expected behavior and there is hope. I put the quorum settings that I was told a couple of emails ago. After applying virt group, they are cluster.quorum-type auto cluster.quorum-count (null) cluster.server-quorum-type server cluster.server-quorum-ratio 0 cluster.quorum-reads no Also, I just put the ping timeout to 5 seconds now. Carl On 2019-08-23 5:45 p.m., Ingo Fischer wrote: > Hi Carl, > > In my understanding and experience (I have a replica 3 System running > too) this should not happen. Can you tell your client and server > quorum settings? > > Ingo > > Am 23.08.2019 um 15:53 schrieb Carl Sirotic > >: > >> However, >> >> I must have misunderstood the whole concept of gluster. >> >> In a replica 3, for me, it's completely unacceptable, regardless of >> the options, that all my VMs go down when I reboot one node. >> >> The whole purpose of having a full 3 copy of my data on the fly is >> suposed to be this. >> >> I am in the process of sharding every file. >> >> But even if the healing time would be longer, I would still expect a >> non-sharded replica 3 brick with vm boot disk, to not go down if I >> reboot one of its copy. >> >> >> I am not very impressed by gluster so far. >> >> Carl >> >> On 2019-08-19 4:15 p.m., Darrell Budic wrote: >>> /var/lib/glusterd/groups/virt is a good start for ideas, notably >>> some thread settings and choose-local=off to improve read >>> performance. If you don?t have at least 10 cores on your servers, >>> you may want to lower the recommended shd-max-threads=8 to no more >>> than half your CPU cores to keep healing from swamping out regular >>> work. >>> >>> It?s also starting to depend on what your backing store and >>> networking setup are, so you?re going to want to test changes and >>> find what works best for your setup. >>> >>> In addition to the virt group settings, I use these on most of my >>> volumes, SSD or HDD backed, with the default 64M shard size: >>> >>> performance.io -thread-count: 32# seemed good >>> for my system, particularly a ZFS backed volume with lots of spindles >>> client.event-threads: 8 >>> cluster.data-self-heal-algorithm: full# 10G networking, uses more >>> net/less cpu to heal. probably don?t use this for 1G networking? >>> performance.stat-prefetch: on >>> cluster.read-hash-mode: 3# distribute reads to least loaded server >>> (by read queue depth) >>> >>> and these two only on my HDD backed volume: >>> >>> performance.cache-size: 1G >>> performance.write-behind-window-size: 64MB >>> >>> but I suspect these two need another round or six of tuning to tell >>> if they are making a difference. >>> >>> I use the throughput-performance tuned profile on my servers, so you >>> should be in good shape there. >>> >>>> On Aug 19, 2019, at 12:22 PM, Guy Boisvert >>>> >>> > wrote: >>>> >>>> On 2019-08-19 12:08 p.m., Darrell Budic wrote: >>>>> You also need to make sure your volume is setup properly for best >>>>> performance. Did you apply the gluster virt group to your volumes, >>>>> or at least features.shard = on on your VM volume? >>>> >>>> That's what we did here: >>>> >>>> >>>> gluster volume set W2K16_Rhenium cluster.quorum-type auto >>>> gluster volume set W2K16_Rhenium network.ping-timeout 10 >>>> gluster volume set W2K16_Rhenium auth.allow \* >>>> gluster volume set W2K16_Rhenium group virt >>>> gluster volume set W2K16_Rhenium storage.owner-uid 36 >>>> gluster volume set W2K16_Rhenium storage.owner-gid 36 >>>> gluster volume set W2K16_Rhenium features.shard on >>>> gluster volume set W2K16_Rhenium features.shard-block-size 256MB >>>> gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm full >>>> gluster volume set W2K16_Rhenium performance.low-prio-threads 32 >>>> >>>> tuned-adm profile random-io??? ??? (a profile i added in CentOS 7) >>>> >>>> >>>> cat /usr/lib/tuned/random-io/tuned.conf >>>> =========================================== >>>> [main] >>>> summary=Optimize for Gluster virtual machine storage >>>> include=throughput-performance >>>> >>>> [sysctl] >>>> >>>> vm.dirty_ratio = 5 >>>> vm.dirty_background_ratio = 2 >>>> >>>> >>>> Any more optimization to add to this? >>>> >>>> >>>> Guy >>>> >>>> -- >>>> Guy Boisvert, ing. >>>> IngTegration inc. >>>> http://www.ingtegration.com >>>> https://www.linkedin.com/in/guy-boisvert-8990487 >>>> >>>> AVIS DE CONFIDENTIALITE : ce message peut contenir des >>>> renseignements confidentiels appartenant exclusivement a >>>> IngTegration Inc. ou a ses filiales. Si vous n'etes pas >>>> le destinataire indique ou prevu dans ce ?message (ou >>>> responsable de livrer ce message a la personne indiquee ou >>>> prevue) ou si vous pensez que ce message vous a ete adresse >>>> par erreur, vous ne pouvez pas utiliser ou reproduire ce >>>> message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous >>>> devez le detruire et vous etes prie d'avertir l'expediteur >>>> en repondant au courriel. >>>> >>>> CONFIDENTIALITY NOTICE : Proprietary/Confidential Information >>>> belonging to IngTegration Inc. and its affiliates may be >>>> contained in this message. If you are not a recipient >>>> indicated or intended in this message (or responsible for >>>> delivery of this message to such person), or you think for >>>> any reason that this message may have been addressed to you >>>> in error, you may not use or copy or deliver this message to >>>> anyone else. In such case, you should destroy this message >>>> and are asked to notify the sender by reply email. >>>> >>> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From csirotic at evoqarchitecture.com Fri Aug 23 23:01:32 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Fri, 23 Aug 2019 19:01:32 -0400 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes References: <51516c0e-d5ee-445a-87eb-c3f02d918f40@email.android.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded message was scrubbed... From: "Carl Sirotic" Subject: Re: [Gluster-users] Brick Reboot => VMs slowdown, client crashes Date: Fri, 23 Aug 2019 19:00:40 -0400 Size: 8433 URL: From runmatt at live.com Fri Aug 23 17:35:27 2019 From: runmatt at live.com (Matthew Evans) Date: Fri, 23 Aug 2019 17:35:27 +0000 Subject: [Gluster-users] MPIO for Single Client? Message-ID: Does Gluster support any sort of MPIO-like functionality for a single client? Maybe when using a Distributed Stripe? Or is a single client always limited to the throughput of a single node for each session? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Sun Aug 25 07:56:10 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Sun, 25 Aug 2019 10:56:10 +0300 Subject: [Gluster-users] MPIO for Single Client? Message-ID: <5kx1ii51y4kh3dfyduqpnmdg.1566719770514@email.android.com> Hi Mathew, If you use FUSE mount, the client is connecting to all servers in the volume and thus it works like MPIO. For example - 3 replica volume has the same data in all 3 gluster servers and thus the cluent has 3 'paths' to the volume. If one fails, a timeout will occur and thr client will stop trying to update that server's bricks. Of course, you can have multiple IPs defined per gluster server and thus the client will try to reach the server's bricks by any IP in the configuration. For example, Brick A has '192.168.1.2' & '192.168.2.2'. There will be no constant connection over both IPs, but as far as I know the client will try the second IP. Best Regards, Strahil NikolovOn Aug 23, 2019 20:35, Matthew Evans wrote: > > Does Gluster support any sort of MPIO-like functionality for a single client? Maybe when using a Distributed Stripe? Or is a single client always limited to the throughput of a single node for each session? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mghanawi at gmail.com Mon Aug 26 04:43:05 2019 From: mghanawi at gmail.com (Mohammed Ghanawi) Date: Mon, 26 Aug 2019 08:43:05 +0400 Subject: [Gluster-users] Gluster lvm cache vs tiering performance issue Message-ID: I am on the latest gluster version 6.x. I have an nvme lvm cache, configured on a sata based volume. I have two bricks replicated, and configured exactly the same way. I have made sure that the block size on volumes and xfs is 128k. The issue is that the write performance to the mounted gluster volume is horribly slow 39 mib/s. However if I test the lvm volume performance or the individual bricks performance I get 600-700 mib/s, which confirms the bricks are configured correctly. It is only when I attempt to mount the gluster file system remotely or even locally I get slow speeds. I had a similar config 2 nvme disks for the hot tier and 2 sata disks for the cold tier, and the mounted gluster file system v3.12 was giving near 600 mib/s. It's not the network, as mounting locally has the same issue. I tried write through on the lvm volume, but no luck. What could be the problem? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkothiya at redhat.com Mon Aug 26 05:47:10 2019 From: rkothiya at redhat.com (Rinku Kothiya) Date: Mon, 26 Aug 2019 11:17:10 +0530 Subject: [Gluster-users] [Gluster-devel] [Gluster-Maintainers] glusterfs-7.0rc0 released Message-ID: Hi, Release-7 RC0 packages are built. This is a good time to start testing the release bits, and reporting any issues on bugzilla. Do post on the lists any testing done and feedback for the same. We have about 2 weeks to GA of release-6 barring any major blockers uncovered during the test phase. Please take this time to help make the release effective, by testing the same. Packages for Fedora 29, Fedora 30, RHEL 8 are at https://download.gluster.org/pub/gluster/glusterfs/qa-releases/7.0rc0/ Packages are signed. The public key is at https://download.gluster.org/pub/gluster/glusterfs/6/rsa.pub Packages for Stretch,Bullseye and CentOS7 will be there as soon as they get built. Regards Rinku -------------- next part -------------- An HTML attachment was scrubbed... URL: From pankaj at datacabinet.systems Mon Aug 26 06:35:53 2019 From: pankaj at datacabinet.systems (pankaj kumar) Date: Sun, 25 Aug 2019 23:35:53 -0700 Subject: [Gluster-users] Access as root user when using root_squash Message-ID: I am using a gluster volume share over nfs using root_squash. I would like most of the servers in the organization use a gluster share with their specific user id/group id and not able to read other people's data. But there have to be some users who can provision the root level directory for other users and give proper ownership. Is it possible to do? Thanks, From amukherj at redhat.com Mon Aug 26 08:13:17 2019 From: amukherj at redhat.com (Atin Mukherjee) Date: Mon, 26 Aug 2019 13:43:17 +0530 Subject: [Gluster-users] [Gluster-devel] [Gluster-Maintainers] glusterfs-7.0rc0 released In-Reply-To: References: Message-ID: On Mon, Aug 26, 2019 at 11:18 AM Rinku Kothiya wrote: > Hi, > > Release-7 RC0 packages are built. This is a good time to start testing the > release bits, and reporting any issues on bugzilla. > Do post on the lists any testing done and feedback for the same. > > We have about 2 weeks to GA of release-6 barring any major blockers > uncovered during the test phase. Please take this time to help make the > release effective, by testing the same. > I believe you meant release-7 here :-) I'd like to request that just like release-6, we pay some attention on the upgrade testing (release-4/release-5/release-6 to release-7) paths and report back issues here (along with bugzilla links). > Packages for Fedora 29, Fedora 30, RHEL 8 are at > https://download.gluster.org/pub/gluster/glusterfs/qa-releases/7.0rc0/ > > Packages are signed. The public key is at > https://download.gluster.org/pub/gluster/glusterfs/6/rsa.pub > > Packages for Stretch,Bullseye and CentOS7 will be there as soon as they > get built. > > Regards > Rinku > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkothiya at redhat.com Mon Aug 26 10:07:17 2019 From: rkothiya at redhat.com (Rinku Kothiya) Date: Mon, 26 Aug 2019 15:37:17 +0530 Subject: [Gluster-users] [Gluster-devel] [Gluster-Maintainers] glusterfs-7.0rc0 released In-Reply-To: References: Message-ID: Hi, I tried upgrading from glusterfs-server-6.5-0 to glusterfs-server-7.0-0.1rc0 without any problem on fedora 30. *Note for testing upgrade : * Glusterfs-6 rpms complied for fedora30. Glusterfs-7 downloaded the rpms from the site ( https://download.gluster.org/pub/gluster/glusterfs/qa-releases/7.0rc0/Fedora/fedora-30/x86_64/). *Output : * # yum localinstall -y {glusterfs-cli,glusterfs-api,glusterfs-libs,glusterfs-client-xlators,glusterfs-fuse,glusterfs,glusterfs-server,python3-gluster,glusterfs-geo-replication} Last metadata expiration check: 0:08:52 ago on Mon 26 Aug 2019 03:10:38 PM IST. Dependencies resolved. ============================================================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================================================ Upgrading: glusterfs-cli x86_64 7.0-0.1rc0.fc30 @commandline 182 k glusterfs-api x86_64 7.0-0.1rc0.fc30 @commandline 88 k glusterfs-libs x86_64 7.0-0.1rc0.fc30 @commandline 396 k glusterfs-client-xlators x86_64 7.0-0.1rc0.fc30 @commandline 809 k glusterfs-fuse x86_64 7.0-0.1rc0.fc30 @commandline 136 k glusterfs x86_64 7.0-0.1rc0.fc30 @commandline 611 k glusterfs-server x86_64 7.0-0.1rc0.fc30 @commandline 1.3 M python3-gluster x86_64 7.0-0.1rc0.fc30 @commandline 20 k glusterfs-geo-replication x86_64 7.0-0.1rc0.fc30 @commandline 179 k Transaction Summary ============================================================================================================================================================================ Upgrade 9 Packages Total size: 3.6 M Downloading Packages: Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Running scriptlet: glusterfs-libs-7.0-0.1rc0.fc30.x86_64 Upgrading : glusterfs-libs-7.0-0.1rc0.fc30.x86_64 1/18 Running scriptlet: glusterfs-7.0-0.1rc0.fc30.x86_64 2/18 Upgrading : glusterfs-7.0-0.1rc0.fc30.x86_64 2/18 Running scriptlet: glusterfs-7.0-0.1rc0.fc30.x86_64 2/18 Upgrading : glusterfs-client-xlators-7.0-0.1rc0.fc30.x86_64 3/18 Upgrading : glusterfs-api-7.0-0.1rc0.fc30.x86_64 4/18 Upgrading : glusterfs-fuse-7.0-0.1rc0.fc30.x86_64 5/18 Upgrading : glusterfs-cli-7.0-0.1rc0.fc30.x86_64 6/18 Upgrading : glusterfs-server-7.0-0.1rc0.fc30.x86_64 7/18 Running scriptlet: glusterfs-server-7.0-0.1rc0.fc30.x86_64 7/18 Upgrading : python3-gluster-7.0-0.1rc0.fc30.x86_64 8/18 Upgrading : glusterfs-geo-replication-7.0-0.1rc0.fc30.x86_64 9/18 Running scriptlet: glusterfs-geo-replication-7.0-0.1rc0.fc30.x86_64 9/18 Cleanup : glusterfs-geo-replication-6.5-0.1.git988a3dcea.fc30.x86_64 10/18 Cleanup : python3-gluster-6.5-0.1.git988a3dcea.fc30.x86_64 11/18 Running scriptlet: glusterfs-server-6.5-0.1.git988a3dcea.fc30.x86_64 12/18 Cleanup : glusterfs-server-6.5-0.1.git988a3dcea.fc30.x86_64 12/18 Running scriptlet: glusterfs-server-6.5-0.1.git988a3dcea.fc30.x86_64 12/18 Cleanup : glusterfs-api-6.5-0.1.git988a3dcea.fc30.x86_64 . . . Verifying : python3-gluster-7.0-0.1rc0.fc30.x86_64 15/18 Verifying : python3-gluster-6.5-0.1.git988a3dcea.fc30.x86_64 16/18 Verifying : glusterfs-geo-replication-7.0-0.1rc0.fc30.x86_64 17/18 Verifying : glusterfs-geo-replication-6.5-0.1.git988a3dcea.fc30.x86_64 18/18 Upgraded: glusterfs-cli-7.0-0.1rc0.fc30.x86_64 glusterfs-api-7.0-0.1rc0.fc30.x86_64 glusterfs-libs-7.0-0.1rc0.fc30.x86_64 glusterfs-client-xlators-7.0-0.1rc0.fc30.x86_64 glusterfs-fuse-7.0-0.1rc0.fc30.x86_64 glusterfs-7.0-0.1rc0.fc30.x86_64 glusterfs-server-7.0-0.1rc0.fc30.x86_64 python3-gluster-7.0-0.1rc0.fc30.x86_64 glusterfs-geo-replication-7.0-0.1rc0.fc30.x86_64 Complete! Regards Rinku On Mon, Aug 26, 2019 at 1:43 PM Atin Mukherjee wrote: > > > On Mon, Aug 26, 2019 at 11:18 AM Rinku Kothiya > wrote: > >> Hi, >> >> Release-7 RC0 packages are built. This is a good time to start testing >> the release bits, and reporting any issues on bugzilla. >> Do post on the lists any testing done and feedback for the same. >> >> We have about 2 weeks to GA of release-6 barring any major blockers >> uncovered during the test phase. Please take this time to help make the >> release effective, by testing the same. >> > > I believe you meant release-7 here :-) > > I'd like to request that just like release-6, we pay some attention on the > upgrade testing (release-4/release-5/release-6 to release-7) paths and > report back issues here (along with bugzilla links). > > >> Packages for Fedora 29, Fedora 30, RHEL 8 are at >> https://download.gluster.org/pub/gluster/glusterfs/qa-releases/7.0rc0/ >> >> Packages are signed. The public key is at >> https://download.gluster.org/pub/gluster/glusterfs/6/rsa.pub >> >> Packages for Stretch,Bullseye and CentOS7 will be there as soon as they >> get built. >> >> Regards >> Rinku >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > On Mon, Aug 26, 2019 at 1:43 PM Atin Mukherjee wrote: > > > On Mon, Aug 26, 2019 at 11:18 AM Rinku Kothiya > wrote: > >> Hi, >> >> Release-7 RC0 packages are built. This is a good time to start testing >> the release bits, and reporting any issues on bugzilla. >> Do post on the lists any testing done and feedback for the same. >> >> We have about 2 weeks to GA of release-6 barring any major blockers >> uncovered during the test phase. Please take this time to help make the >> release effective, by testing the same. >> > > I believe you meant release-7 here :-) > > I'd like to request that just like release-6, we pay some attention on the > upgrade testing (release-4/release-5/release-6 to release-7) paths and > report back issues here (along with bugzilla links). > > >> Packages for Fedora 29, Fedora 30, RHEL 8 are at >> https://download.gluster.org/pub/gluster/glusterfs/qa-releases/7.0rc0/ >> >> Packages are signed. The public key is at >> https://download.gluster.org/pub/gluster/glusterfs/6/rsa.pub >> >> Packages for Stretch,Bullseye and CentOS7 will be there as soon as they >> get built. >> >> Regards >> Rinku >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndevos at redhat.com Mon Aug 26 10:07:53 2019 From: ndevos at redhat.com (Niels de Vos) Date: Mon, 26 Aug 2019 12:07:53 +0200 Subject: [Gluster-users] Access as root user when using root_squash In-Reply-To: References: Message-ID: <20190826100753.GB28580@ndevos-x270> On Sun, Aug 25, 2019 at 11:35:53PM -0700, pankaj kumar wrote: > I am using a gluster volume share over nfs using root_squash. > > I would like most of the servers in the organization use a gluster > share with their specific user id/group id and not able to read other > people's data. > > But there have to be some users who can provision the root level > directory for other users and give proper ownership. Is it possible to > do? You can do this by mounting the volume with fuse for provisioning (on some dedicated server, or on the Gluster servers themselves). Normal access can then be done over NFS on the servers that the normal users use. HTH, Niels From amudhan83 at gmail.com Mon Aug 26 10:12:14 2019 From: amudhan83 at gmail.com (Amudhan P) Date: Mon, 26 Aug 2019 15:42:14 +0530 Subject: [Gluster-users] same gfid for two directories and dht_layout_mismatch and failed to combine inode error msgs Message-ID: Hi, I am facing an issue in my gluster setup. glusterfs version 3.10.1 distributed disperse volume glusterfs-fuse client. One of node SAS BackPlane had some problem so my drives are ejected from service, this caused the brick process to terminate but glusterd daemon was still running. gluster peer status showing faulty node as online but bricks in the faulty node is offline. In the meantime, client upload process was going on, cpu load and memory usage on other nodes keep on increasing. so, I restarted the problem node and started glusterd service after mounting all disks. now gluster status showing all brick is online. now the problem I am facing here is folders which are uploaded or moved between one path to another path when bringing back problem node to live is not listing files originally they had and I am not able to delete the folder even though I have permission, throwing error "permission denied" and "folder not empty" but doing "ls" of the showing empty folder. error msg in fuse mount log. [2019-08-26 07:33:13.005165] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:33:13.005211] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:33:21.020529] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 07:34:59.005318] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 23 times between [2019-08-26 07:33:13.005165] and [2019-08-26 07:35:11.005147] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 25 times between [2019-08-26 07:33:13.005211] and [2019-08-26 07:35:11.005149] [2019-08-26 07:35:13.006967] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:35:13.007028] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:35:37.005121] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 2 times between [2019-08-26 07:35:37.005121] and [2019-08-26 07:36:21.012016] The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 21 times between [2019-08-26 07:35:13.006967] and [2019-08-26 07:37:11.005302] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 24 times between [2019-08-26 07:35:13.007028] and [2019-08-26 07:37:11.005305] [2019-08-26 07:37:13.006618] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 07:37:13.006721] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:37:21.024033] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:38:22.026108] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-139: Failed to combine iatt (inode: 11541463450251411113-11541463450251411113, links: 1-1, uid: 1000-1000, gid: 1000-1000, rdev: 0-0, size: 517816320-517799936, mode: 100600-100600) [2019-08-26 07:38:22.026144] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-139: Mismatching iatt in answers of 'GF_FOP_LOOKUP' The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 19 times between [2019-08-26 07:37:21.024033] and [2019-08-26 07:39:10.005761] The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 3 times between [2019-08-26 07:37:13.006618] and [2019-08-26 07:39:11.004587] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 23 times between [2019-08-26 07:37:13.006721] and [2019-08-26 07:39:11.004591] [2019-08-26 07:39:13.007257] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:39:13.007335] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:39:39.004926] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 2 times between [2019-08-26 07:39:39.004926] and [2019-08-26 07:40:11.004852] The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 23 times between [2019-08-26 07:39:13.007257] and [2019-08-26 07:41:13.006759] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 26 times between [2019-08-26 07:39:13.007335] and [2019-08-26 07:41:13.006761] [2019-08-26 07:41:21.010830] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 07:41:21.010861] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 4 times between [2019-08-26 07:18:22.003323] and [2019-08-26 07:18:42.019306] [2019-08-26 07:18:47.003349] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 07:18:47.003392] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-137: Failed to combine iatt (inode: 12704371770406674620-12704371770406674620, links: 1-1, uid: 1000-1000, gid: 1000-1000, rdev: 0-0, size: 696369152-696352768, mode: 100600-100600)" repeated 2 times between [2019-08-26 07:17:11.019084] and [2019-08-26 07:17:11.019201] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-137: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 2 times between [2019-08-26 07:17:11.019111] and [2019-08-26 07:17:11.019202] The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 19 times between [2019-08-26 07:17:08.003143] and [2019-08-26 07:18:50.003418] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 4 times between [2019-08-26 07:18:47.003392] and [2019-08-26 07:18:50.003420] [2019-08-26 07:19:08.002986] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:19:08.003012] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:19:39.011760] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 09:21:53.015705] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-qubevaultdr-disperse-139: Failed to combine iatt (inode: 11196349995399397487-11196349995399397487, links: 1-1, uid: 1000-1000, gid: 1000-1000, rdev: 0-0, size: 1177141248-1177124864, mode: 100600-100600) [2019-08-26 09:21:53.015747] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-qubevaultdr-disperse-139: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 04:49:11.580839] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-135; inode layout - 3251903722 - 3282582058 - 1; disk layout - 0 - 0 - 0 [2019-08-26 04:49:11.580850] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 [2019-08-26 04:49:11.580986] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-136; inode layout - 3282582059 - 3313260395 - 1; disk layout - 0 - 1073741822 - 1 [2019-08-26 04:49:11.580997] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 [2019-08-26 04:49:11.581026] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-137; inode layout - 3313260396 - 3343938732 - 1; disk layout - 1073741823 - 2147483645 - 1 [2019-08-26 04:49:11.581038] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 [2019-08-26 04:49:11.581232] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-138; inode layout - 3343938733 - 3374617069 - 1; disk layout - 2147483646 - 3221225468 - 1 [2019-08-26 04:49:11.581243] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 [2019-08-26 04:49:11.581333] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-139; inode layout - 3374617070 - 3405295406 - 1; disk layout - 3221225469 - 4294967295 - 1 [2019-08-26 04:49:11.581350] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 error message in brick log. [2019-08-24 13:12:54.041605] W [MSGID: 113026] [posix.c:1542:posix_mkdir] 0-glusterdr-posix: mkdir (/Incoming/TempUpload/Weekly UPload/23-.8-19/1-2-19/client_folder2): gfid (b2bf1c5b-2be4-4074-be37-8118f1279d04) is already associated with directory (/media/disk13/brick13/.glusterfs/61/9a/619a5949-9001-4457-94c1-8f02c45996ba/client_folder2). Hence, both directories will share same gfid and this can lead to inconsistencies. regards Amudhan P -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Mon Aug 26 10:32:02 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Mon, 26 Aug 2019 13:32:02 +0300 Subject: [Gluster-users] Access as root user when using root_squash Message-ID: Hi, Can you go to a gluster brick and mount via fuse the volume ? This way you can create your directories with necessary owner,group and ugo rights. Best Regards, Strahil NikolovOn Aug 26, 2019 09:35, pankaj kumar wrote: > > I am using a gluster volume share over nfs using root_squash. > > I would like most of the servers in the organization use a gluster > share with their specific user id/group id and not able to read other > people's data. > > But there have to be some users who can provision the root level > directory for other users and give proper ownership. Is it possible to > do? > > Thanks, > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From cedric.rouvrais at socgen.com Mon Aug 26 18:14:29 2019 From: cedric.rouvrais at socgen.com (ROUVRAIS Cedric ResgBscRscDef) Date: Mon, 26 Aug 2019 18:14:29 +0000 Subject: [Gluster-users] Geo Replication Failure: libgfchangelog.so: cannot open shared object file Message-ID: Hello, Having some slight difficulties with GeoReplication across 2 servers where the user is root on both sides: libgfchangelog.so: cannot open shared object file: No such file or directory I get this error on the master node. I haven't been able to get around it (I've deleted and recreated the configuration ex-nihilo a couple of time to no avail). [2019-08-26 17:43:24.213577] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing... [2019-08-26 17:43:24.213959] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker brick=/vol/gluster/fr/brick1/gv_master slave_node=srv_slave [2019-08-26 17:43:24.259780] I [gsyncd(agent /vol/gluster/fr/brick1/gv_master):309:main] : Using session config file path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf [2019-08-26 17:43:24.261590] I [changelogagent(agent /vol/gluster/fr/brick1/gv_master):72:__init__] ChangelogAgent: Agent listining... [2019-08-26 17:43:24.266072] I [gsyncd(worker /vol/gluster/fr/brick1/gv_master):309:main] : Using session config file path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf [2019-08-26 17:43:24.280029] I [resource(worker /vol/gluster/fr/brick1/gv_master):1379:connect_remote] SSH: Initializing SSH connection between master and slave... [2019-08-26 17:43:25.749739] I [resource(worker /vol/gluster/fr/brick1/gv_master):1426:connect_remote] SSH: SSH connection between master and slave established. duration=1.4696 [2019-08-26 17:43:25.749941] I [resource(worker /vol/gluster/fr/brick1/gv_master):1098:connect] GLUSTER: Mounting gluster volume locally... [2019-08-26 17:43:26.824810] I [resource(worker /vol/gluster/fr/brick1/gv_master):1121:connect] GLUSTER: Mounted gluster volume duration=1.0747 [2019-08-26 17:43:26.825088] I [subcmds(worker /vol/gluster/fr/brick1/gv_master):80:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2019-08-26 17:43:26.888922] E [repce(agent /vol/gluster/fr/brick1/gv_master):122:worker] : call failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, in init return Changes.cl_init() File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, in __getattr__ from libgfchangelog import Changes as LChanges File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 18, in class Changes(object): File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 20, in Changes use_errno=True) File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__ self._handle = _dlopen(self._name, mode) OSError: libgfchangelog.so: cannot open shared object file: No such file or directory Thank you for any insight, C?dric ========================================================= Ce message et toutes les pieces jointes (ci-apres le "message") sont confidentiels et susceptibles de contenir des informations couvertes par le secret professionnel. Ce message est etabli a l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisee interdite. Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE et ses filiales declinent toute responsabilite au titre de ce message s'il a ete altere, deforme falsifie. ========================================================= This message and any attachments (the "message") are confidential, intended solely for the addresses, and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. ========================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From sacharya at redhat.com Tue Aug 27 04:45:35 2019 From: sacharya at redhat.com (sacharya at redhat.com) Date: Tue, 27 Aug 2019 04:45:35 +0000 Subject: [Gluster-users] Invitation: Gluster Community Meeting @ Tue Aug 27, 2019 11:30am - 12:30pm (IST) (gluster-users@gluster.org) Message-ID: <00000000000076f510059111f131@google.com> You have been invited to the following event. Title: Gluster Community Meeting Bridge: https://bluejeans.com/836554017 Minutes meeting: https://hackmd.io/IGIuc8GxRv6JAUpRjWZ5sw Previous Meeting notes: https://github.com/gluster/community/meetings Flash talk: No Flash Talk When: Tue Aug 27, 2019 11:30am ? 12:30pm India Standard Time - Kolkata Where: https://bluejeans.com/836554017 Calendar: gluster-users at gluster.org Who: * sacharya at redhat.com - organizer * gluster-users at gluster.org * gluster-devel at gluster.org Event details: https://www.google.com/calendar/event?action=VIEW&eid=MDVvdTVtb3Q3MDkzOXV0aHNiYmN2NTF1cWsgZ2x1c3Rlci11c2Vyc0BnbHVzdGVyLm9yZw&tok=MTkjc2FjaGFyeWFAcmVkaGF0LmNvbTdiNzlmMzE1MjMwZTBjZGU0MmMwNGM1ZmI1ZWEwYjkyNzc4MTBiYzM&ctz=Asia%2FKolkata&hl=en&es=0 Invitation from Google Calendar: https://www.google.com/calendar/ You are receiving this courtesy email at the account gluster-users at gluster.org because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn more at https://support.google.com/calendar/answer/37135#forwarding -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 1750 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: invite.ics Type: application/ics Size: 1788 bytes Desc: not available URL: From sacharya at redhat.com Tue Aug 27 09:01:08 2019 From: sacharya at redhat.com (Shwetha Acharya) Date: Tue, 27 Aug 2019 14:31:08 +0530 Subject: [Gluster-users] Minutes of Gluster Community Meeting (APAC) 27th August 2019 Message-ID: Hi, The minutes of the meeting are as follows: # Gluster Community Meeting - 27th Aug 2019 ### Previous Meeting minutes: - http://github.com/gluster/community - Recording of this meeting- - https://bluejeans.com/s/s1Zma ### Date/Time: Check the [community calendar]( https://calendar.google.com/calendar/b/1?cid=dmViajVibDBrbnNiOWQwY205ZWg5cGJsaTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ ) ### Bridge * APAC friendly hours - Tuesday 27th August 2019, 11:30AM IST - Bridge: https://bluejeans.com/836554017 * NA/EMEA - Every 1st and 3rd Tuesday at 01:00 PM EDT - Bridge: https://bluejeans.com/486278655 ------- ### Attendance Name (#gluster-dev alias) - company * Hari Gowtham (hgowtham) - Red Hat * Ravishankar (itisravi) Red Hat * Sheetal Pamecha (spamecha) - Red Hat * David Spisla - Gluster User * Sunny Kumar (sunny) - Red hat * Rinku Kothiya (rkothiya) - Red Hat * Ashish Pandey (_apandey) Red Hat * Sunil Kumar Acharya - Red Hat * Arjun Sharma - Red Hat * Sanju Rakonde(srakonde) - Red Hat * Kotresh (kotreshhr) - Redhat * Karthik Subrahmanya (ksubrahm) - Red Hat ### User stories * Ravi - timeout of self heal crawl to be automatically updated. (Bug ID: 1743988) * Hari - user asked for project: user quota. Had to reply that we are running out of bandwidth. contribution will be helpful here. ### Community * Project metrics: Metrics | Value | |[Coverity] | 65 | |[Clang Scan] | 59 | |[Test coverage] | 70.8% | |[New Bugs in last 14 days] | 6 | [[7.x] | 2 | [[6.x] | 10 | [[ 5.x] | 1 | |[Gluster User Queries in last 14 days] | 232 | |[Total Bugs] | 345 | |[Total Github issues](https://github.com/gluster/glusterfs/issues) | 393 | * Any release updates? Rinku - We have released Release-7 rc0 on 26-August-2019, request users to report any problems seen. * Blocker issues across the project? Atin - infra issue related to a bunch of tests are failing. A fix to do retry was done. Nightly is running an old code base. * Notable thread from mailing list Amar - Moving to Github from gerrit. Atin - It will be good to move to github completely. We can discuss it with large audiance. ### Conferences / Meetups * [Developers' Conference - {Date}]({Link}) - No conferences to talk about this week. Important dates: CFP Closed Schedule Announcement: Event Open for Registration : Last Date of Registration: Event dates: Venue: Talks related to gluster: ### GlusterFS - v7.0 and beyond * Proposal - https://docs.google.com/presentation/d/1rtn38S4YBe77KK5IjczWmoAR-ZSO-i3tNHg9pAH8Wt8/edit?usp=sharing * Proposed Plan: - GlusterFS-7.0 (July 1st) - Stability, Automation - Only - GlusterFS-8.0 (Nov 1st) - - Plan for Fedora 31/RHEL8.2 - GlusterFS-9.0 (March 1st, 2020) Reflink, io_uring, and similar improvements. ### Developer focus * Any design specs to discuss? nil ### Component status * Arbiter - no new updates. * AFR - nil * DHT - nil * EC - new data corruption corner cases. Pranith has found the code path. * FUSE - nil * POSIX - nil * DOC - changes related to afr was posted by Ravi and Karthik * Geo Replication - nil * libglusterfs - nil * Glusterd - Glusto automation run is blocked because of https://bugzilla.redhat.com/show_bug.cgi?id=1744420 , team is working on it. * Snapshot - nil * NFS - nil * thin-arbiter - nil ### Flash Talk Gluster * Typical 5 min talk about Gluster with up to 5 more minutes for questions ### Recent Blog posts / Document updates * https://shwetha174.blogspot.com/search/label/Gluster * https://medium.com/@ntkumar/running-alluxio-on-hashicorp-nomad-ef78130727ef * https://medium.com/@tumballi/kadalu-ocean-of-potential-in-k8s-storage-a07be1b8b961 ### Gluster Friday Five * Every friday we release this, which basically covers highlight of week in gluster.Also you can find more videos in youtube link. https://www.youtube.com/channel/UCfilWh0JA5NfCjbqq1vsBVA ### Host Sheetal will host next meeting. * Who will host next meeting? - Host will need to send out the agenda 24hr - 12hrs in advance to mailing list, and also make sure to send the meeting minutes. - Host will need to reach out to one user at least who can talk about their usecase, their experience, and their needs. - Host needs to send meeting minutes as PR to http://github.com/gluster/community - Host has to send the meeting minutes as a mail to the gluster-devel and gluster-users list. ### Notetaker * Who will take notes from the next meeting? ### RoundTable * ### Action Items on host * Check-in Minutes of meeting for this meeting -------------- next part -------------- An HTML attachment was scrubbed... URL: From andy.coates at gmail.com Wed Aug 28 03:17:48 2019 From: andy.coates at gmail.com (Andy Coates) Date: Wed, 28 Aug 2019 13:17:48 +1000 Subject: [Gluster-users] Geo Replication Failure: libgfchangelog.so: cannot open shared object file In-Reply-To: References: Message-ID: We saw this with 4.1.x RPM on OEL (can't recall which specific version and haven't checked if its fixed in later, at least up to 4.1.6), but the issue seemed to be it just wasn't symlinked for some reason, so we symlinked libgfchangelog.so to /lib64/libgfchangelog.so.0 Not sure if the python code is meant to ask for libgfchangelog.so.0 in the first place, or if the RPM is meant to symlink it post-install. In 7.x the script seems to use a different method for finding it too. Andy. On Tue, 27 Aug 2019 at 04:21, ROUVRAIS Cedric ResgBscRscDef < cedric.rouvrais at socgen.com> wrote: > Hello, > > > > Having some slight difficulties with GeoReplication across 2 servers where > the user is root on both sides: libgfchangelog.so: cannot open shared > object file: No such file or directory > > > > I get this error on the master node. I haven?t been able to get around it > (I?ve deleted and recreated the configuration ex-nihilo a couple of time to > no avail). > > > > [2019-08-26 17:43:24.213577] I > [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status > Change status=Initializing... > > [2019-08-26 17:43:24.213959] I [monitor(monitor):157:monitor] Monitor: > starting gsyncd worker brick=/vol/gluster/fr/brick1/gv_master > slave_node=srv_slave > > [2019-08-26 17:43:24.259780] I [gsyncd(agent > /vol/gluster/fr/brick1/gv_master):309:main] : Using session config > file > path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf > > [2019-08-26 17:43:24.261590] I [changelogagent(agent > /vol/gluster/fr/brick1/gv_master):72:__init__] ChangelogAgent: Agent > listining... > > [2019-08-26 17:43:24.266072] I [gsyncd(worker > /vol/gluster/fr/brick1/gv_master):309:main] : Using session config > file > path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf > > [2019-08-26 17:43:24.280029] I [resource(worker > /vol/gluster/fr/brick1/gv_master):1379:connect_remote] SSH: Initializing > SSH connection between master and slave... > > [2019-08-26 17:43:25.749739] I [resource(worker > /vol/gluster/fr/brick1/gv_master):1426:connect_remote] SSH: SSH connection > between master and slave established. duration=1.4696 > > [2019-08-26 17:43:25.749941] I [resource(worker > /vol/gluster/fr/brick1/gv_master):1098:connect] GLUSTER: Mounting gluster > volume locally... > > [2019-08-26 17:43:26.824810] I [resource(worker > /vol/gluster/fr/brick1/gv_master):1121:connect] GLUSTER: Mounted gluster > volume duration=1.0747 > > [2019-08-26 17:43:26.825088] I [subcmds(worker > /vol/gluster/fr/brick1/gv_master):80:subcmd_worker] : Worker spawn > successful. Acknowledging back to monitor > > [2019-08-26 17:43:26.888922] E [repce(agent > /vol/gluster/fr/brick1/gv_master):122:worker] : call failed: > > Traceback (most recent call last): > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in > worker > > res = getattr(self.obj, rmeth)(*in_data[2:]) > > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line > 37, in init > > return Changes.cl_init() > > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line > 21, in __getattr__ > > from libgfchangelog import Changes as LChanges > > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line > 18, in > > class Changes(object): > > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line > 20, in Changes > > use_errno=True) > > File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__ > > self._handle = _dlopen(self._name, mode) > > OSError: libgfchangelog.so: cannot open shared object file: No such file > or directory > > > > > > Thank you for any insight, > > > > C?dric > > > > ========================================================= > > Ce message et toutes les pieces jointes (ci-apres le "message") > sont confidentiels et susceptibles de contenir des informations > couvertes par le secret professionnel. Ce message est etabli > a l'intention exclusive de ses destinataires. Toute utilisation > ou diffusion non autorisee interdite. > Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE > et ses filiales declinent toute responsabilite au titre de ce message > s'il a ete altere, deforme falsifie. > > ========================================================= > > This message and any attachments (the "message") are confidential, > intended solely for the addresses, and may contain legally privileged > information. Any unauthorized use or dissemination is prohibited. > E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any > of its subsidiaries or affiliates shall be liable for the message > if altered, changed or falsified. > > ========================================================= > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From cedric.rouvrais at socgen.com Wed Aug 28 06:43:03 2019 From: cedric.rouvrais at socgen.com (ROUVRAIS Cedric ResgBscRscDef) Date: Wed, 28 Aug 2019 06:43:03 +0000 Subject: [Gluster-users] Geo Replication Failure: libgfchangelog.so: cannot open shared object file In-Reply-To: References: Message-ID: Hi Andy, Thanks for your reply, I actually fixed the issue ? shared library wasn?t installed, so I reinstalled it. I realize that the documentation for geo replication would need more information ? I will try a clean install on a new image and document it more precisely. Thanks, C?dric De : Andy Coates Envoy? : mercredi 28 ao?t 2019 05:18 ? : ROUVRAIS Cedric ResgBscRscDef Cc : gluster-users at gluster.org Objet : Re: [Gluster-users] Geo Replication Failure: libgfchangelog.so: cannot open shared object file We saw this with 4.1.x RPM on OEL (can't recall which specific version and haven't checked if its fixed in later, at least up to 4.1.6), but the issue seemed to be it just wasn't symlinked for some reason, so we symlinked libgfchangelog.so to /lib64/libgfchangelog.so.0 Not sure if the python code is meant to ask for libgfchangelog.so.0 in the first place, or if the RPM is meant to symlink it post-install. In 7.x the script seems to use a different method for finding it too. Andy. On Tue, 27 Aug 2019 at 04:21, ROUVRAIS Cedric ResgBscRscDef > wrote: Hello, Having some slight difficulties with GeoReplication across 2 servers where the user is root on both sides: libgfchangelog.so: cannot open shared object file: No such file or directory I get this error on the master node. I haven?t been able to get around it (I?ve deleted and recreated the configuration ex-nihilo a couple of time to no avail). [2019-08-26 17:43:24.213577] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing... [2019-08-26 17:43:24.213959] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker brick=/vol/gluster/fr/brick1/gv_master slave_node=srv_slave [2019-08-26 17:43:24.259780] I [gsyncd(agent /vol/gluster/fr/brick1/gv_master):309:main] : Using session config file path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf [2019-08-26 17:43:24.261590] I [changelogagent(agent /vol/gluster/fr/brick1/gv_master):72:__init__] ChangelogAgent: Agent listining... [2019-08-26 17:43:24.266072] I [gsyncd(worker /vol/gluster/fr/brick1/gv_master):309:main] : Using session config file path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf [2019-08-26 17:43:24.280029] I [resource(worker /vol/gluster/fr/brick1/gv_master):1379:connect_remote] SSH: Initializing SSH connection between master and slave... [2019-08-26 17:43:25.749739] I [resource(worker /vol/gluster/fr/brick1/gv_master):1426:connect_remote] SSH: SSH connection between master and slave established. duration=1.4696 [2019-08-26 17:43:25.749941] I [resource(worker /vol/gluster/fr/brick1/gv_master):1098:connect] GLUSTER: Mounting gluster volume locally... [2019-08-26 17:43:26.824810] I [resource(worker /vol/gluster/fr/brick1/gv_master):1121:connect] GLUSTER: Mounted gluster volume duration=1.0747 [2019-08-26 17:43:26.825088] I [subcmds(worker /vol/gluster/fr/brick1/gv_master):80:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2019-08-26 17:43:26.888922] E [repce(agent /vol/gluster/fr/brick1/gv_master):122:worker] : call failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, in init return Changes.cl_init() File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, in __getattr__ from libgfchangelog import Changes as LChanges File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 18, in class Changes(object): File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 20, in Changes use_errno=True) File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__ self._handle = _dlopen(self._name, mode) OSError: libgfchangelog.so: cannot open shared object file: No such file or directory Thank you for any insight, C?dric ========================================================= Ce message et toutes les pieces jointes (ci-apres le "message") sont confidentiels et susceptibles de contenir des informations couvertes par le secret professionnel. Ce message est etabli a l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisee interdite. Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE et ses filiales declinent toute responsabilite au titre de ce message s'il a ete altere, deforme falsifie. ========================================================= This message and any attachments (the "message") are confidential, intended solely for the addresses, and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. ========================================================= _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From sunkumar at redhat.com Wed Aug 28 07:33:33 2019 From: sunkumar at redhat.com (Sunny Kumar) Date: Wed, 28 Aug 2019 13:03:33 +0530 Subject: [Gluster-users] Geo Replication Failure: libgfchangelog.so: cannot open shared object file In-Reply-To: References: Message-ID: Thanks Andy for detailed answer. C?dric, If possible can you share your updated document/patch which can be merged to gluster doc[1] and made available to everyone[2]. [1]. https://github.com/gluster/glusterdocs/ [2] https://docs.gluster.org/en/latest/ . /sunny On Wed, Aug 28, 2019 at 12:13 PM ROUVRAIS Cedric ResgBscRscDef wrote: > > Hi Andy, > > > > Thanks for your reply, I actually fixed the issue ? shared library wasn?t installed, so I reinstalled it. > > > > I realize that the documentation for geo replication would need more information > > > > I will try a clean install on a new image and document it more precisely. > > > > Thanks, > > > > C?dric > > > > > > De : Andy Coates > Envoy? : mercredi 28 ao?t 2019 05:18 > ? : ROUVRAIS Cedric ResgBscRscDef > Cc : gluster-users at gluster.org > Objet : Re: [Gluster-users] Geo Replication Failure: libgfchangelog.so: cannot open shared object file > > > > We saw this with 4.1.x RPM on OEL (can't recall which specific version and haven't checked if its fixed in later, at least up to 4.1.6), but the issue seemed to be it just wasn't symlinked for some reason, so we symlinked libgfchangelog.so to /lib64/libgfchangelog.so.0 > > > > Not sure if the python code is meant to ask for libgfchangelog.so.0 in the first place, or if the RPM is meant to symlink it post-install. In 7.x the script seems to use a different method for finding it too. > > > > Andy. > > > > On Tue, 27 Aug 2019 at 04:21, ROUVRAIS Cedric ResgBscRscDef wrote: > > Hello, > > > > Having some slight difficulties with GeoReplication across 2 servers where the user is root on both sides: libgfchangelog.so: cannot open shared object file: No such file or directory > > > > I get this error on the master node. I haven?t been able to get around it (I?ve deleted and recreated the configuration ex-nihilo a couple of time to no avail). > > > > [2019-08-26 17:43:24.213577] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing... > > [2019-08-26 17:43:24.213959] I [monitor(monitor):157:monitor] Monitor: starting gsyncd worker brick=/vol/gluster/fr/brick1/gv_master slave_node=srv_slave > > [2019-08-26 17:43:24.259780] I [gsyncd(agent /vol/gluster/fr/brick1/gv_master):309:main] : Using session config file path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf > > [2019-08-26 17:43:24.261590] I [changelogagent(agent /vol/gluster/fr/brick1/gv_master):72:__init__] ChangelogAgent: Agent listining... > > [2019-08-26 17:43:24.266072] I [gsyncd(worker /vol/gluster/fr/brick1/gv_master):309:main] : Using session config file path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf > > [2019-08-26 17:43:24.280029] I [resource(worker /vol/gluster/fr/brick1/gv_master):1379:connect_remote] SSH: Initializing SSH connection between master and slave... > > [2019-08-26 17:43:25.749739] I [resource(worker /vol/gluster/fr/brick1/gv_master):1426:connect_remote] SSH: SSH connection between master and slave established. duration=1.4696 > > [2019-08-26 17:43:25.749941] I [resource(worker /vol/gluster/fr/brick1/gv_master):1098:connect] GLUSTER: Mounting gluster volume locally... > > [2019-08-26 17:43:26.824810] I [resource(worker /vol/gluster/fr/brick1/gv_master):1121:connect] GLUSTER: Mounted gluster volume duration=1.0747 > > [2019-08-26 17:43:26.825088] I [subcmds(worker /vol/gluster/fr/brick1/gv_master):80:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor > > [2019-08-26 17:43:26.888922] E [repce(agent /vol/gluster/fr/brick1/gv_master):122:worker] : call failed: > > Traceback (most recent call last): > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker > > res = getattr(self.obj, rmeth)(*in_data[2:]) > > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, in init > > return Changes.cl_init() > > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, in __getattr__ > > from libgfchangelog import Changes as LChanges > > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 18, in > > class Changes(object): > > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 20, in Changes > > use_errno=True) > > File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__ > > self._handle = _dlopen(self._name, mode) > > OSError: libgfchangelog.so: cannot open shared object file: No such file or directory > > > > > > Thank you for any insight, > > > > C?dric > > > > ========================================================= > > Ce message et toutes les pieces jointes (ci-apres le "message") > sont confidentiels et susceptibles de contenir des informations > couvertes par le secret professionnel. Ce message est etabli > a l'intention exclusive de ses destinataires. Toute utilisation > ou diffusion non autorisee interdite. > Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE > et ses filiales declinent toute responsabilite au titre de ce message > s'il a ete altere, deforme falsifie. > > ========================================================= > > This message and any attachments (the "message") are confidential, > intended solely for the addresses, and may contain legally privileged > information. Any unauthorized use or dissemination is prohibited. > E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any > of its subsidiaries or affiliates shall be liable for the message > if altered, changed or falsified. > > ========================================================= > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From sunkumar at redhat.com Wed Aug 28 12:51:27 2019 From: sunkumar at redhat.com (Sunny Kumar) Date: Wed, 28 Aug 2019 18:21:27 +0530 Subject: [Gluster-users] Gluster meetup: India Message-ID: Hello folks, We are hosting Gluster meetup at our office (Redhat-BLR-IN) on 25th September 2019. Please find the agenda and location detail here [1] and plan accordingly. The highlight of this event will be Gluster -X we will keep on updating agenda with topics, so keep an eye on it. Note: * RSVP as YES if attending, this will help us to organize the facilities better. If you have any question, please reach out to me or comment on the event page [1]. Feel free to share this meetup via other channels. [1]. https://www.meetup.com/glusterfs-India/events/264366771/ /sunny From shreyansh.shah at alpha-grep.com Thu Aug 29 09:20:39 2019 From: shreyansh.shah at alpha-grep.com (Shreyansh Shah) Date: Thu, 29 Aug 2019 14:50:39 +0530 Subject: [Gluster-users] glusterfs mount crashes with "Transport endpoint is not connected" Message-ID: Hi, Running on cloud centos7.5 VM, same machine has gluster volume mounted at 2 endpoints (read/write), say A and B. Gluster version server is 5.3 and on client is 3.12.2. B is used very rarely and only for light reads. Mount A failed when our processes were running, but B was still present and could access data through B but not through A. Here is the trace from /var/log/glusterfs: The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 968 times between [2019-08-28 20:40:59.654898] and [2019-08-28 20:41:36.417335] pending frames: frame : type(1) op(FSTAT) frame : type(1) op(READ) frame : type(1) op(READ) frame : type(1) op(READ) frame : type(1) op(READ) frame : type(1) op(READ) frame : type(1) op(READ) frame : type(1) op(READ) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2019-08-28 20:41:37 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.3 /lib64/libglusterfs.so.0(+0x26610)[0x7fea89c32610] /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fea89c3cb84] /lib64/libc.so.6(+0x36340)[0x7fea88295340] /lib64/libpthread.so.0(pthread_mutex_lock+0x0)[0x7fea88a97c30] /lib64/libglusterfs.so.0(__gf_free+0x12c)[0x7fea89c5dc3c] /lib64/libglusterfs.so.0(rbthash_remove+0xd5)[0x7fea89c69d35] /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcace)[0x7fea7771dace] /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcdd7)[0x7fea7771ddd7] /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcfc5)[0x7fea7771dfc5] /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xf0ca)[0x7fea777200ca] /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xa6a1)[0x7fea77b426a1] /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xaa6f)[0x7fea77b42a6f] /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xb0ce)[0x7fea77b430ce] /lib64/libglusterfs.so.0(default_readv_cbk+0x180)[0x7fea89cbb8e0] /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x81c1a)[0x7fea77dc9c1a] /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x6d636)[0x7fea7c307636] /lib64/libgfrpc.so.0(+0xec70)[0x7fea899fec70] /lib64/libgfrpc.so.0(+0xf043)[0x7fea899ff043] /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fea899faf23] /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7fea7e5e637b] /lib64/libglusterfs.so.0(+0x8aa49)[0x7fea89c96a49] /lib64/libpthread.so.0(+0x7dd5)[0x7fea88a95dd5] /lib64/libc.so.6(clone+0x6d)[0x7fea8835d02d] -- Regards, Shreyansh Shah -------------- next part -------------- An HTML attachment was scrubbed... URL: From shreyansh.shah at alpha-grep.com Thu Aug 29 09:24:29 2019 From: shreyansh.shah at alpha-grep.com (Shreyansh Shah) Date: Thu, 29 Aug 2019 14:54:29 +0530 Subject: [Gluster-users] glusterfs mount crashes with "Transport endpoint is not connected" In-Reply-To: References: Message-ID: On Thu, Aug 29, 2019 at 2:50 PM Shreyansh Shah < shreyansh.shah at alpha-grep.com> wrote: > Hi, > Running on cloud centos7.5 VM, same machine has gluster volume mounted at > 2 endpoints (read/write), say A and B. Gluster version server is 5.3 and on > client is 3.12.2. > B is used very rarely and only for light reads. Mount A failed when our > processes were running, but B was still present and could access data > through B but not through A. > > Here is the trace from /var/log/glusterfs: > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch > handler" repeated 968 times between [2019-08-28 20:40:59.654898] and > [2019-08-28 20:41:36.417335] > pending frames: > frame : type(1) op(FSTAT) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(1) op(READ) > frame : type(0) op(0) > patchset: git://git.gluster.org/glusterfs.git > signal received: 11 > time of crash: > 2019-08-28 20:41:37 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 5.3 > /lib64/libglusterfs.so.0(+0x26610)[0x7fea89c32610] > /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fea89c3cb84] > /lib64/libc.so.6(+0x36340)[0x7fea88295340] > /lib64/libpthread.so.0(pthread_mutex_lock+0x0)[0x7fea88a97c30] > /lib64/libglusterfs.so.0(__gf_free+0x12c)[0x7fea89c5dc3c] > /lib64/libglusterfs.so.0(rbthash_remove+0xd5)[0x7fea89c69d35] > > /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcace)[0x7fea7771dace] > > /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcdd7)[0x7fea7771ddd7] > > /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcfc5)[0x7fea7771dfc5] > > /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xf0ca)[0x7fea777200ca] > > /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xa6a1)[0x7fea77b426a1] > > /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xaa6f)[0x7fea77b42a6f] > > /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xb0ce)[0x7fea77b430ce] > /lib64/libglusterfs.so.0(default_readv_cbk+0x180)[0x7fea89cbb8e0] > > /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x81c1a)[0x7fea77dc9c1a] > > /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x6d636)[0x7fea7c307636] > /lib64/libgfrpc.so.0(+0xec70)[0x7fea899fec70] > /lib64/libgfrpc.so.0(+0xf043)[0x7fea899ff043] > /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fea899faf23] > /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7fea7e5e637b] > /lib64/libglusterfs.so.0(+0x8aa49)[0x7fea89c96a49] > /lib64/libpthread.so.0(+0x7dd5)[0x7fea88a95dd5] > /lib64/libc.so.6(clone+0x6d)[0x7fea8835d02d] > > > -- > Regards, > Shreyansh Shah > -- Regards, Shreyansh Shah -------------- next part -------------- An HTML attachment was scrubbed... URL: From joao.bauto at neuro.fchampalimaud.org Thu Aug 29 10:01:41 2019 From: joao.bauto at neuro.fchampalimaud.org (=?UTF-8?B?Sm/Do28gQmHDunRv?=) Date: Thu, 29 Aug 2019 11:01:41 +0100 Subject: [Gluster-users] glusterfs mount crashes with "Transport endpoint is not connected" In-Reply-To: References: Message-ID: You're most likely hitting this bug https://bugzilla.redhat.com/show_bug.cgi ?id=1671556 Upgrading to gluster 5.5 should fix it. JB Shreyansh Shah escreveu no dia quinta, 29/08/2019 ?(s) 10:24: > > > On Thu, Aug 29, 2019 at 2:50 PM Shreyansh Shah < > shreyansh.shah at alpha-grep.com> wrote: > >> Hi, >> Running on cloud centos7.5 VM, same machine has gluster volume mounted at >> 2 endpoints (read/write), say A and B. Gluster version server is 5.3 and on >> client is 3.12.2. >> B is used very rarely and only for light reads. Mount A failed when our >> processes were running, but B was still present and could access data >> through B but not through A. >> >> Here is the trace from /var/log/glusterfs: >> The message "E [MSGID: 101191] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch >> handler" repeated 968 times between [2019-08-28 20:40:59.654898] and >> [2019-08-28 20:41:36.417335] >> pending frames: >> frame : type(1) op(FSTAT) >> frame : type(1) op(READ) >> frame : type(1) op(READ) >> frame : type(1) op(READ) >> frame : type(1) op(READ) >> frame : type(1) op(READ) >> frame : type(1) op(READ) >> frame : type(1) op(READ) >> frame : type(0) op(0) >> patchset: git://git.gluster.org/glusterfs.git >> signal received: 11 >> time of crash: >> 2019-08-28 20:41:37 >> configuration details: >> argp 1 >> backtrace 1 >> dlfcn 1 >> libpthread 1 >> llistxattr 1 >> setfsid 1 >> spinlock 1 >> epoll.h 1 >> xattr.h 1 >> st_atim.tv_nsec 1 >> package-string: glusterfs 5.3 >> /lib64/libglusterfs.so.0(+0x26610)[0x7fea89c32610] >> /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fea89c3cb84] >> /lib64/libc.so.6(+0x36340)[0x7fea88295340] >> /lib64/libpthread.so.0(pthread_mutex_lock+0x0)[0x7fea88a97c30] >> /lib64/libglusterfs.so.0(__gf_free+0x12c)[0x7fea89c5dc3c] >> /lib64/libglusterfs.so.0(rbthash_remove+0xd5)[0x7fea89c69d35] >> >> /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcace)[0x7fea7771dace] >> >> /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcdd7)[0x7fea7771ddd7] >> >> /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xcfc5)[0x7fea7771dfc5] >> >> /usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xf0ca)[0x7fea777200ca] >> >> /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xa6a1)[0x7fea77b426a1] >> >> /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xaa6f)[0x7fea77b42a6f] >> >> /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so(+0xb0ce)[0x7fea77b430ce] >> /lib64/libglusterfs.so.0(default_readv_cbk+0x180)[0x7fea89cbb8e0] >> >> /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so(+0x81c1a)[0x7fea77dc9c1a] >> >> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x6d636)[0x7fea7c307636] >> /lib64/libgfrpc.so.0(+0xec70)[0x7fea899fec70] >> /lib64/libgfrpc.so.0(+0xf043)[0x7fea899ff043] >> /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fea899faf23] >> /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa37b)[0x7fea7e5e637b] >> /lib64/libglusterfs.so.0(+0x8aa49)[0x7fea89c96a49] >> /lib64/libpthread.so.0(+0x7dd5)[0x7fea88a95dd5] >> /lib64/libc.so.6(clone+0x6d)[0x7fea8835d02d] >> >> >> -- >> Regards, >> Shreyansh Shah >> > > > -- > Regards, > Shreyansh Shah > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From dudleyperkins at gmail.com Thu Aug 29 13:19:34 2019 From: dudleyperkins at gmail.com (=?UTF-8?Q?Michael_B=C3=B6hm?=) Date: Thu, 29 Aug 2019 15:19:34 +0200 Subject: [Gluster-users] Switching from Debian Packages to Gluster Community Packages Message-ID: Hey folks, Question ist essentially in the Subject. Right now i'm running Debian oldstable/stretch with Gluster 3.8.8 from the normal repo and i'm thinking about switching to the community repo from gluster. I couldn't find much about which problems i can expect. I checked the packages, and besides that the debian packages still have init-scripts and the ones from gluster are already systemd offering only the unit file. In the end i would like to get from gluster 3 -> 6 in as few steps as possible, if i stay in with the debian packages that would mean: 3.8.8 (stretch) -> 4.1 (stretch-backports) -> 5.5 (buster) -> 6.4 (buster-backports) If i can use the gluster-repos that would be only: 3.8.8 (stretch) -> 3.12 (gluster-repo) -> 6.5 (gluster-repo) Will this be possible? And of course i want to do that online, i have only replicate and distributed-replicate volumes. Regards Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From riehecky at fnal.gov Thu Aug 29 13:12:12 2019 From: riehecky at fnal.gov (Pat Riehecky) Date: Thu, 29 Aug 2019 08:12:12 -0500 Subject: [Gluster-users] Disappearing files on gluster mount Message-ID: <1badc9e3-9625-14b2-bb26-1fc58125717a@fnal.gov> I moved my wife's photo archive to a mirrored gluster volume. Lately, she's noticed that a number of files are missing.? I'm pretty sure they are still in the .glusterfs dir as no one deleted them, but they simply don't display.... Any ideas how to get the files to reappear? Glusterfs 3.14 -- Pat Riehecky Fermi National Accelerator Laboratory www.fnal.gov www.scientificlinux.org From ml at krneki.org Thu Aug 29 13:51:14 2019 From: ml at krneki.org (Miha Verlic) Date: Thu, 29 Aug 2019 15:51:14 +0200 Subject: [Gluster-users] Several issues when using Gluster with SSL and CRL Message-ID: Hello, I've setup Glusterfs 6.3 cluster with 2 nodes + arbiter (and some additional clients), SSL and CRL: server.ssl: on client.ssl: on ssl.crl-path: /etc/ssl/crl After a month (when CRL Next Update date came) cluster collapsed with "error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired" error. I had to restart all processes on all nodes. fetch-crl is installed on all nodes and properly synces CRLs, but it seems gluster caches CRLs indefinitely and never re-reads them. When initial CRL reaches "Next Update" date Gluster starts to reject all connetions, even though CRL was updated during this time. Even -HUPing all gluster processes does not help. This can easily be reproduced by setting CRL option default_crl_days to two days and refreshing CRL every day. Cluster will crash when initial CRL will expire, even if it is updated in between. Another problem happened when one of the clients did not have up-to-dated CRL. When client was trying to connect, cluster was apparently constantly busy with client and did not come online. After client was killed, cluster came online instantly. Even debug logs were not especially helpful, as client's IP is not logged with error messages. Cheers -- Miha From amalagi at commvault.com Thu Aug 29 10:46:05 2019 From: amalagi at commvault.com (Anand Malagi) Date: Thu, 29 Aug 2019 10:46:05 +0000 Subject: [Gluster-users] Question about Healing estimated time ... In-Reply-To: References: Message-ID: Can someone please respond ?? Thanks and Regards, --Anand Extn : 6974 Mobile : 9552527199 From: Anand Malagi Sent: Wednesday, August 28, 2019 5:13 PM To: gluster-users at gluster.org; Gluster Devel Subject: Question about Healing estimated time ... Hi Gluster Team, I have Distributed-Disperse gluster volume which uses erasure coding. I basically two of the bricks within a sub volume (4+2 config), then generated some data which obviously will not be written to these two bricks which I brought down. However before bringing them up and get them healed, is there a way to know how much time it will take to heal the files or a way to measure the healing time ?? Thanks and Regards, --Anand Extn : 6974 Mobile : 9552527199 -------------- next part -------------- An HTML attachment was scrubbed... URL: From budic at onholyground.com Thu Aug 29 19:45:32 2019 From: budic at onholyground.com (Darrell Budic) Date: Thu, 29 Aug 2019 14:45:32 -0500 Subject: [Gluster-users] Question about Healing estimated time ... In-Reply-To: References: Message-ID: Depends on your disks, your network, some CPU since you?re using a dispersed volume, and the amount of data you?ve got on them. Watch this heal and see how long it takes to baseline your system. If you?ve got 10G and SSDs, it?s probably not going to take too long. If you?ve got 1G, HDDs, and your test case is a TB, it?ll be an hour or three... > On Aug 29, 2019, at 5:46 AM, Anand Malagi wrote: > > Can someone please respond ?? > > Thanks and Regards, > --Anand > Extn : 6974 > Mobile : 9552527199 > > From: Anand Malagi > Sent: Wednesday, August 28, 2019 5:13 PM > To: gluster-users at gluster.org ; Gluster Devel > > Subject: Question about Healing estimated time ... > > Hi Gluster Team, > > I have Distributed-Disperse gluster volume which uses erasure coding. I basically two of the bricks within a sub volume (4+2 config), then generated some data which obviously will not be written to these two bricks which I brought down. > > However before bringing them up and get them healed, is there a way to know how much time it will take to heal the files or a way to measure the healing time ?? > > > Thanks and Regards, > --Anand > Extn : 6974 > Mobile : 9552527199 > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From budic at onholyground.com Thu Aug 29 19:58:23 2019 From: budic at onholyground.com (Darrell Budic) Date: Thu, 29 Aug 2019 14:58:23 -0500 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> <2BF3B174-E129-4156-AEE1-137572327FE1@onholyground.com> <7b14c43d-fdf6-7c07-4715-3d1b78831227@ingtegration.com> <70ee770d-c7fc-0629-1eef-b5c04703f935@evoqarchitecture.com> <14de67b4-0b81-b88b-a526-2672e089d6cc@evoqarchitecture.com> Message-ID: <725D1DD4-DED8-4B3F-8607-2D95DF93271E@onholyground.com> You may be mis-understanding the way the gluster system works in detail here, but you?ve got the right idea overall. Since gluster is maintaining 3 copies of your data, you can lose a drive or a whole system and things will keep going without interruption (well, mostly, if a host node was using the system that just died, it may pause briefly before re-connecting to one that is still running via a backup-server setting or your dns configs). While the system is still going with one node down, that node is falling behind and new disk writes, and the remaining ones are keeping track of what?s changing. Once you repair/recover/reboot the down node, it will rejoin the cluster. Now the recovered system has to catch up, and it does this by having the other two nodes send it the changes. In the meantime, gluster is serving any reads for that data from one of the up to date nodes, even if you ask the one you just restarted. In order to do this healing, it had to lock the files to ensure no changes are made while it copies a chunk of them over the recovered node. When it locks them, your hypervisor notices they have gone read-only, and especially if it has a pending write for that file, may pause the VM because this looks like a storage issue to it. Once the file gets unlocked, it can be written again, and your hypervisor notices and will generally reactivate your VM. You may see delays too, especially if you only have 1G networking between your host nodes while everything is getting copied around. And your files could be being locked, updated, unlocked, locked again a few seconds or minutes later, etc. That?s where sharding comes into play, once you have a file broken up into shards, gluster can get away with only locking the particular shard it needs to heal, and leaving the whole disk image unlocked. You may still catch a brief pause if you try and write the specific segment of the file gluster is healing at the moment, but it?s also going to be much faster because it?s a small chuck of the file, and copies quickly. Also, check out https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/server-quorum/ , you probably want to set cluster.server-quorum-ratio to 50 for a replica-3 setup to avoid the possibility of split-brains. Your cluster will go write only if it loses two nodes though, but you can always make a change to the server-quorum-ratio later if you need to keep it running temporarily. Hope that makes sense of what?s going on for you, -Darrell > On Aug 23, 2019, at 5:06 PM, Carl Sirotic wrote: > > Okay, > > so it means, at least I am not getting the expected behavior and there is hope. > > I put the quorum settings that I was told a couple of emails ago. > > After applying virt group, they are > > cluster.quorum-type auto > cluster.quorum-count (null) > cluster.server-quorum-type server > cluster.server-quorum-ratio 0 > cluster.quorum-reads no > > > Also, > > I just put the ping timeout to 5 seconds now. > > > Carl > > On 2019-08-23 5:45 p.m., Ingo Fischer wrote: >> Hi Carl, >> >> In my understanding and experience (I have a replica 3 System running too) this should not happen. Can you tell your client and server quorum settings? >> >> Ingo >> >> Am 23.08.2019 um 15:53 schrieb Carl Sirotic >: >> >>> However, >>> >>> I must have misunderstood the whole concept of gluster. >>> >>> In a replica 3, for me, it's completely unacceptable, regardless of the options, that all my VMs go down when I reboot one node. >>> >>> The whole purpose of having a full 3 copy of my data on the fly is suposed to be this. >>> >>> I am in the process of sharding every file. >>> >>> But even if the healing time would be longer, I would still expect a non-sharded replica 3 brick with vm boot disk, to not go down if I reboot one of its copy. >>> >>> >>> >>> I am not very impressed by gluster so far. >>> >>> Carl >>> >>> On 2019-08-19 4:15 p.m., Darrell Budic wrote: >>>> /var/lib/glusterd/groups/virt is a good start for ideas, notably some thread settings and choose-local=off to improve read performance. If you don?t have at least 10 cores on your servers, you may want to lower the recommended shd-max-threads=8 to no more than half your CPU cores to keep healing from swamping out regular work. >>>> >>>> It?s also starting to depend on what your backing store and networking setup are, so you?re going to want to test changes and find what works best for your setup. >>>> >>>> In addition to the virt group settings, I use these on most of my volumes, SSD or HDD backed, with the default 64M shard size: >>>> >>>> performance.io -thread-count: 32 # seemed good for my system, particularly a ZFS backed volume with lots of spindles >>>> client.event-threads: 8 >>>> cluster.data-self-heal-algorithm: full # 10G networking, uses more net/less cpu to heal. probably don?t use this for 1G networking? >>>> performance.stat-prefetch: on >>>> cluster.read-hash-mode: 3 # distribute reads to least loaded server (by read queue depth) >>>> >>>> and these two only on my HDD backed volume: >>>> >>>> performance.cache-size: 1G >>>> performance.write-behind-window-size: 64MB >>>> >>>> but I suspect these two need another round or six of tuning to tell if they are making a difference. >>>> >>>> I use the throughput-performance tuned profile on my servers, so you should be in good shape there. >>>> >>>>> On Aug 19, 2019, at 12:22 PM, Guy Boisvert > wrote: >>>>> >>>>> On 2019-08-19 12:08 p.m., Darrell Budic wrote: >>>>>> You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume? >>>>> >>>>> That's what we did here: >>>>> >>>>> >>>>> gluster volume set W2K16_Rhenium cluster.quorum-type auto >>>>> gluster volume set W2K16_Rhenium network.ping-timeout 10 >>>>> gluster volume set W2K16_Rhenium auth.allow \* >>>>> gluster volume set W2K16_Rhenium group virt >>>>> gluster volume set W2K16_Rhenium storage.owner-uid 36 >>>>> gluster volume set W2K16_Rhenium storage.owner-gid 36 >>>>> gluster volume set W2K16_Rhenium features.shard on >>>>> gluster volume set W2K16_Rhenium features.shard-block-size 256MB >>>>> gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm full >>>>> gluster volume set W2K16_Rhenium performance.low-prio-threads 32 >>>>> >>>>> tuned-adm profile random-io (a profile i added in CentOS 7) >>>>> >>>>> >>>>> cat /usr/lib/tuned/random-io/tuned.conf >>>>> =========================================== >>>>> [main] >>>>> summary=Optimize for Gluster virtual machine storage >>>>> include=throughput-performance >>>>> >>>>> [sysctl] >>>>> >>>>> vm.dirty_ratio = 5 >>>>> vm.dirty_background_ratio = 2 >>>>> >>>>> >>>>> Any more optimization to add to this? >>>>> >>>>> >>>>> Guy >>>>> >>>>> -- >>>>> Guy Boisvert, ing. >>>>> IngTegration inc. >>>>> http://www.ingtegration.com >>>>> https://www.linkedin.com/in/guy-boisvert-8990487 >>>>> >>>>> AVIS DE CONFIDENTIALITE : ce message peut contenir des >>>>> renseignements confidentiels appartenant exclusivement a >>>>> IngTegration Inc. ou a ses filiales. Si vous n'etes pas >>>>> le destinataire indique ou prevu dans ce message (ou >>>>> responsable de livrer ce message a la personne indiquee ou >>>>> prevue) ou si vous pensez que ce message vous a ete adresse >>>>> par erreur, vous ne pouvez pas utiliser ou reproduire ce >>>>> message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous >>>>> devez le detruire et vous etes prie d'avertir l'expediteur >>>>> en repondant au courriel. >>>>> >>>>> CONFIDENTIALITY NOTICE : Proprietary/Confidential Information >>>>> belonging to IngTegration Inc. and its affiliates may be >>>>> contained in this message. If you are not a recipient >>>>> indicated or intended in this message (or responsible for >>>>> delivery of this message to such person), or you think for >>>>> any reason that this message may have been addressed to you >>>>> in error, you may not use or copy or deliver this message to >>>>> anyone else. In such case, you should destroy this message >>>>> and are asked to notify the sender by reply email. >>>>> >>>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From csirotic at evoqarchitecture.com Thu Aug 29 20:02:03 2019 From: csirotic at evoqarchitecture.com (Carl Sirotic) Date: Thu, 29 Aug 2019 16:02:03 -0400 Subject: [Gluster-users] Brick Reboot => VMs slowdown, client crashes In-Reply-To: References: <2c60c87c-14a1-437b-aca1-7a09d9bf2c47@email.android.com> <70ee770d-c7fc-0629-1eef-b5c04703f935@evoqarchitecture.com> <14de67b4-0b81-b88b-a526-2672e089d6cc@evoqarchitecture.com> <725D1DD4-DED8-4B3F-8607-2D95DF93271E@onholyground.com> Message-ID: Yes, this makes alot of sense. It's the behavior that I was experiencing that makes no sense. When one node was shut down, the whole VM cluster locked up. However, I managed to find that the culprit were the quorum settings. I put the quorum at 2 bricks for quorum now, and I am not experiencing the problem anymore. All my vm boot disks and data disks are now sharded. We are on 10gbit networks, when the node comes backs, we do not see any latency really. Carl On 2019-08-29 3:58 p.m., Darrell Budic wrote: > You may be mis-understanding the way the gluster system works in > detail here, but you?ve got the right idea overall. Since gluster is > maintaining 3 copies of your data, you can lose a drive or a whole > system and things will keep going without interruption (well, mostly, > if a host node was using the system that just died, it may pause > briefly before re-connecting to one that is still running via a > backup-server setting or your dns configs). While the system is still > going with one node down, that node is falling behind and new disk > writes, and the remaining ones are keeping track of what?s changing. > Once you repair/recover/reboot the down node, it will rejoin the > cluster. Now the recovered system has to catch up, and it does this by > having the other two nodes send it the changes. In the meantime, > gluster is serving any reads for that data from one of the up to date > nodes, even if you ask the one you just restarted. In order to do this > healing, it had to lock the files to ensure no changes are made while > it copies a chunk of them over the recovered node. When it locks them, > your hypervisor notices they have gone read-only, and especially if it > has a pending write for that file, may pause the VM because this looks > like a storage issue to it. Once the file gets unlocked, it can be > written again, and your hypervisor notices and will generally > reactivate your VM. You may see delays too, especially if you only > have 1G networking between your host nodes while everything is getting > copied around. And your files could be being locked, updated, > unlocked, locked again a few seconds or minutes later, etc. > > That?s where sharding comes into play, once you have a file broken up > into shards, gluster can get away with only locking the particular > shard it needs to heal, and leaving the whole disk image unlocked. You > may still catch a brief pause if you try and write the specific > segment of the file gluster is healing at the moment, but it?s also > going to be much faster because it?s a small chuck of the file, and > copies quickly. > > Also, check out > https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/server-quorum/, > you probably want to set cluster.server-quorum-ratio to 50 for a > replica-3 setup to avoid the possibility of split-brains. Your cluster > will go write only if it loses two nodes though, but you can always > make a change to the server-quorum-ratio later if you need to keep it > running temporarily. > > Hope that makes sense of what?s going on for you, > > ? -Darrell > >> On Aug 23, 2019, at 5:06 PM, Carl Sirotic >> > > wrote: >> >> Okay, >> >> so it means, at least I am not getting the expected behavior and >> there is hope. >> >> I put the quorum settings that I was told a couple of emails ago. >> >> After applying virt group, they are >> >> cluster.quorum-type auto >> cluster.quorum-count (null) >> cluster.server-quorum-type server >> cluster.server-quorum-ratio 0 >> cluster.quorum-reads no >> >> Also, >> >> I just put the ping timeout to 5 seconds now. >> >> >> Carl >> >> On 2019-08-23 5:45 p.m., Ingo Fischer wrote: >>> Hi Carl, >>> >>> In my understanding and experience (I have a replica 3 System >>> running too) this should not happen. Can you tell your client and >>> server quorum settings? >>> >>> Ingo >>> >>> Am 23.08.2019 um 15:53 schrieb Carl Sirotic >>> >: >>> >>>> However, >>>> >>>> I must have misunderstood the whole concept of gluster. >>>> >>>> In a replica 3, for me, it's completely unacceptable, regardless of >>>> the options, that all my VMs go down when I reboot one node. >>>> >>>> The whole purpose of having a full 3 copy of my data on the fly is >>>> suposed to be this. >>>> >>>> I am in the process of sharding every file. >>>> >>>> But even if the healing time would be longer, I would still expect >>>> a non-sharded replica 3 brick with vm boot disk, to not go down if >>>> I reboot one of its copy. >>>> >>>> >>>> I am not very impressed by gluster so far. >>>> >>>> Carl >>>> >>>> On 2019-08-19 4:15 p.m., Darrell Budic wrote: >>>>> /var/lib/glusterd/groups/virt is a good start for ideas, notably >>>>> some thread settings and choose-local=off to improve read >>>>> performance. If you don?t have at least 10 cores on your servers, >>>>> you may want to lower the recommended shd-max-threads=8 to no more >>>>> than half your CPU cores to keep healing from swamping out regular >>>>> work. >>>>> >>>>> It?s also starting to depend on what your backing store and >>>>> networking setup are, so you?re going to want to test changes and >>>>> find what works best for your setup. >>>>> >>>>> In addition to the virt group settings, I use these on most of my >>>>> volumes, SSD or HDD backed, with the default 64M shard size: >>>>> >>>>> performance.io -thread-count: 32# seemed >>>>> good for my system, particularly a ZFS backed volume with lots of >>>>> spindles >>>>> client.event-threads: 8 >>>>> cluster.data-self-heal-algorithm: full# 10G networking, uses more >>>>> net/less cpu to heal. probably don?t use this for 1G networking? >>>>> performance.stat-prefetch: on >>>>> cluster.read-hash-mode: 3# distribute reads to least loaded server >>>>> (by read queue depth) >>>>> >>>>> and these two only on my HDD backed volume: >>>>> >>>>> performance.cache-size: 1G >>>>> performance.write-behind-window-size: 64MB >>>>> >>>>> but I suspect these two need another round or six of tuning to >>>>> tell if they are making a difference. >>>>> >>>>> I use the throughput-performance tuned profile on my servers, so >>>>> you should be in good shape there. >>>>> >>>>>> On Aug 19, 2019, at 12:22 PM, Guy Boisvert >>>>>> >>>>> > wrote: >>>>>> >>>>>> On 2019-08-19 12:08 p.m., Darrell Budic wrote: >>>>>>> You also need to make sure your volume is setup properly for >>>>>>> best performance. Did you apply the gluster virt group to your >>>>>>> volumes, or at least features.shard = on on your VM volume? >>>>>> >>>>>> That's what we did here: >>>>>> >>>>>> >>>>>> gluster volume set W2K16_Rhenium cluster.quorum-type auto >>>>>> gluster volume set W2K16_Rhenium network.ping-timeout 10 >>>>>> gluster volume set W2K16_Rhenium auth.allow \* >>>>>> gluster volume set W2K16_Rhenium group virt >>>>>> gluster volume set W2K16_Rhenium storage.owner-uid 36 >>>>>> gluster volume set W2K16_Rhenium storage.owner-gid 36 >>>>>> gluster volume set W2K16_Rhenium features.shard on >>>>>> gluster volume set W2K16_Rhenium features.shard-block-size 256MB >>>>>> gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm >>>>>> full >>>>>> gluster volume set W2K16_Rhenium performance.low-prio-threads 32 >>>>>> >>>>>> tuned-adm profile random-io (a profile i added in CentOS 7) >>>>>> >>>>>> >>>>>> cat /usr/lib/tuned/random-io/tuned.conf >>>>>> =========================================== >>>>>> [main] >>>>>> summary=Optimize for Gluster virtual machine storage >>>>>> include=throughput-performance >>>>>> >>>>>> [sysctl] >>>>>> >>>>>> vm.dirty_ratio = 5 >>>>>> vm.dirty_background_ratio = 2 >>>>>> >>>>>> >>>>>> Any more optimization to add to this? >>>>>> >>>>>> >>>>>> Guy >>>>>> >>>>>> -- >>>>>> Guy Boisvert, ing. >>>>>> IngTegration inc. >>>>>> http://www.ingtegration.com >>>>>> https://www.linkedin.com/in/guy-boisvert-8990487 >>>>>> >>>>>> AVIS DE CONFIDENTIALITE : ce message peut contenir des >>>>>> renseignements confidentiels appartenant exclusivement a >>>>>> IngTegration Inc. ou a ses filiales. Si vous n'etes pas >>>>>> le destinataire indique ou prevu dans ce ?message (ou >>>>>> responsable de livrer ce message a la personne indiquee ou >>>>>> prevue) ou si vous pensez que ce message vous a ete adresse >>>>>> par erreur, vous ne pouvez pas utiliser ou reproduire ce >>>>>> message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous >>>>>> devez le detruire et vous etes prie d'avertir l'expediteur >>>>>> en repondant au courriel. >>>>>> >>>>>> CONFIDENTIALITY NOTICE : Proprietary/Confidential Information >>>>>> belonging to IngTegration Inc. and its affiliates may be >>>>>> contained in this message. If you are not a recipient >>>>>> indicated or intended in this message (or responsible for >>>>>> delivery of this message to such person), or you think for >>>>>> any reason that this message may have been addressed to you >>>>>> in error, you may not use or copy or deliver this message to >>>>>> anyone else. In such case, you should destroy this message >>>>>> and are asked to notify the sender by reply email. >>>>>> >>>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ailiev+gluster at mamul.org Thu Aug 29 21:00:23 2019 From: ailiev+gluster at mamul.org (Alexander Iliev) Date: Thu, 29 Aug 2019 23:00:23 +0200 Subject: [Gluster-users] Issues with Geo-replication (GlusterFS 6.3 on Ubuntu 18.04) Message-ID: Hello dear GlusterFS users list, I have been trying to set up geo-replication between two clusters for some time now. The desired state is (Cluster #1) being replicated to (Cluster #2). Here are some details about the setup: Cluster #1: three nodes connected via a local network (172.31.35.0/24), one replicated (3 replica) volume. Cluster #2: three nodes connected via a local network (172.31.36.0/24), one replicated (3 replica) volume. The two clusters are connected to the Internet via separate network adapters. Only SSH (port 22) is open on cluster #2 nodes' adapters connected to the Internet. All nodes are running Ubuntu 18.04 and GlusterFS 6.3 installed from [1]. The first time I followed the guide[2] everything went fine up until I reached the "Create the session" step. That was like a month ago, then I had to temporarily stop working in this and now I am coming back to it. Currently, if I try to see the mountbroker status I get the following: > # gluster-mountbroker status > Traceback (most recent call last): > File "/usr/sbin/gluster-mountbroker", line 396, in > runcli() > File "/usr/lib/python3/dist-packages/gluster/cliutils/cliutils.py", line 225, in runcli > cls.run(args) > File "/usr/sbin/gluster-mountbroker", line 275, in run > out = execute_in_peers("node-status") > File "/usr/lib/python3/dist-packages/gluster/cliutils/cliutils.py", line 127, in execute_in_peers > raise GlusterCmdException((rc, out, err, " ".join(cmd))) > gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to end. Error : Success\n', 'gluster system:: execute mountbroker.py node-status') And in /var/log/gluster/glusterd.log I have: > [2019-08-10 15:24:21.418834] E [MSGID: 106336] [glusterd-geo-rep.c:5413:glusterd_op_sys_exec] 0-management: Unable to end. Error : Success > [2019-08-10 15:24:21.418908] E [MSGID: 106122] [glusterd-syncop.c:1445:gd_commit_op_phase] 0-management: Commit of operation 'Volume Execute system commands' failed on localhost : Unable to end. Error : Success So, I have two questions right now: 1) Is there anything wrong with my setup (networking, open ports, etc.)? Is it expected to work with this setup or should I redo it in a different way? 2) How can I troubleshoot the current status of my setup? Can I find out what's missing/wrong and continue from there or should I just start from scratch? Links: [1] http://ppa.launchpad.net/gluster/glusterfs-6/ubuntu [2] https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ Thank you! Best regards, -- alexander iliev From ashayam.gupta at alpha-grep.com Fri Aug 30 05:41:12 2019 From: ashayam.gupta at alpha-grep.com (Ashayam Gupta) Date: Fri, 30 Aug 2019 11:11:12 +0530 Subject: [Gluster-users] Brick Disconnect from Gluster Volume Message-ID: Hi All, We are facing Brick Disconnect issue on our setup , please find more info about the issue below: *Gluster: 5.3-6 node distributed Cluster -No replication-2 Bricks Per Node-Ubuntu 18.04* *-Issue:* 1 of the Brick on 1 node keeps disconnecting , (after every n days) not very frequently , but observed more than 3-4 times Nothing much in glusterd or bricks logs *GlusterD Logs:* [2019-08-23 10:39:29.811950] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-08-23 10:39:29.815723] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-08-23 10:39:38.111188] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 77 times between [2019-08-23 10:39:38.111188] an d [2019-08-23 10:39:44.990156] ^^ Messages keeps pouring in the logs , this from time when we saw loss of 1 brick,nothing specail about the above logs , but this is what is there in logs from the time we saw loss of the brick *DataBrick Logs:* The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 4788 times between [2019-08-23 10:39:44.993667] ^^ Same here we see this type to logs only Would be helpful if we can get some pointers about the above issue. Thanks Ashayam -------------- next part -------------- An HTML attachment was scrubbed... URL: From sunkumar at redhat.com Fri Aug 30 07:22:42 2019 From: sunkumar at redhat.com (Sunny Kumar) Date: Fri, 30 Aug 2019 12:52:42 +0530 Subject: [Gluster-users] Issues with Geo-replication (GlusterFS 6.3 on Ubuntu 18.04) In-Reply-To: References: Message-ID: Hi Alexander, Thanks for pointing that out! But this issue is fixed now you can see below link for bz-link and patch. BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1709248 Patch - https://review.gluster.org/#/c/glusterfs/+/22716/ Hope this helps. /sunny On Fri, Aug 30, 2019 at 2:30 AM Alexander Iliev wrote: > > Hello dear GlusterFS users list, > > I have been trying to set up geo-replication between two clusters for > some time now. The desired state is (Cluster #1) being replicated to > (Cluster #2). > > Here are some details about the setup: > > Cluster #1: three nodes connected via a local network (172.31.35.0/24), > one replicated (3 replica) volume. > > Cluster #2: three nodes connected via a local network (172.31.36.0/24), > one replicated (3 replica) volume. > > The two clusters are connected to the Internet via separate network > adapters. > > Only SSH (port 22) is open on cluster #2 nodes' adapters connected to > the Internet. > > All nodes are running Ubuntu 18.04 and GlusterFS 6.3 installed from [1]. > > The first time I followed the guide[2] everything went fine up until I > reached the "Create the session" step. That was like a month ago, then I > had to temporarily stop working in this and now I am coming back to it. > > Currently, if I try to see the mountbroker status I get the following: > > > # gluster-mountbroker status > > Traceback (most recent call last): > > File "/usr/sbin/gluster-mountbroker", line 396, in > > runcli() > > File "/usr/lib/python3/dist-packages/gluster/cliutils/cliutils.py", line 225, in runcli > > cls.run(args) > > File "/usr/sbin/gluster-mountbroker", line 275, in run > > out = execute_in_peers("node-status") > > File "/usr/lib/python3/dist-packages/gluster/cliutils/cliutils.py", > line 127, in execute_in_peers > > raise GlusterCmdException((rc, out, err, " ".join(cmd))) > > gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to > end. Error : Success\n', 'gluster system:: execute mountbroker.py > node-status') > > And in /var/log/gluster/glusterd.log I have: > > > [2019-08-10 15:24:21.418834] E [MSGID: 106336] > [glusterd-geo-rep.c:5413:glusterd_op_sys_exec] 0-management: Unable to > end. Error : Success > > [2019-08-10 15:24:21.418908] E [MSGID: 106122] > [glusterd-syncop.c:1445:gd_commit_op_phase] 0-management: Commit of > operation 'Volume Execute system commands' failed on localhost : Unable > to end. Error : Success > > So, I have two questions right now: > > 1) Is there anything wrong with my setup (networking, open ports, etc.)? > Is it expected to work with this setup or should I redo it in a > different way? > 2) How can I troubleshoot the current status of my setup? Can I find out > what's missing/wrong and continue from there or should I just start from > scratch? > > Links: > [1] http://ppa.launchpad.net/gluster/glusterfs-6/ubuntu > [2] > https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ > > Thank you! > > Best regards, > -- > alexander iliev > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users From peljasz at yahoo.co.uk Fri Aug 30 09:29:41 2019 From: peljasz at yahoo.co.uk (lejeczek) Date: Fri, 30 Aug 2019 10:29:41 +0100 Subject: [Gluster-users] Ganesha, after a system reboot, showmounts nothing - why? Message-ID: <31f36af2-fbca-931b-0ba0-5947a9329597@yahoo.co.uk> hi guys, after a reboot Ganesha export are not there. Suffices to do: $ systemctl restart nfs-ganesha - and all is good again. Would you have any ideas why? I'm on Centos 7.6 with nfs-ganesha-gluster-2.7.6-1.el7.x86_64; glusterfs-server-6.4-1.el7.x86_64. many thanks, L. -------------- next part -------------- A non-text attachment was scrubbed... Name: pEpkey.asc Type: application/pgp-keys Size: 1757 bytes Desc: not available URL: From herbert.burnswell at gmail.com Fri Aug 30 17:04:08 2019 From: herbert.burnswell at gmail.com (Herb Burnswell) Date: Fri, 30 Aug 2019 10:04:08 -0700 Subject: [Gluster-users] Rebalancing newly added bricks Message-ID: All, RHEL 7.5 Gluster 3.8.15 2 Nodes: serverA & serverB I am not deeply knowledgeable about Gluster and it's administration but we have a 2 node cluster that's been running for about a year and a half. All has worked fine to date. Our main volume has consisted of two 60TB bricks on each of the cluster nodes. As we reached capacity on the volume we needed to expand. So, we've added four new 60TB bricks to each of the cluster nodes. The bricks are now seen, and the total size of the volume is as expected: # gluster vol status tank Status of volume: tank Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick serverA:/gluster_bricks/data1 49162 0 Y 20318 Brick serverB:/gluster_bricks/data1 49166 0 Y 3432 Brick serverA:/gluster_bricks/data2 49163 0 Y 20323 Brick serverB:/gluster_bricks/data2 49167 0 Y 3435 Brick serverA:/gluster_bricks/data3 49164 0 Y 4625 Brick serverA:/gluster_bricks/data4 49165 0 Y 4644 Brick serverA:/gluster_bricks/data5 49166 0 Y 5088 Brick serverA:/gluster_bricks/data6 49167 0 Y 5128 Brick serverB:/gluster_bricks/data3 49168 0 Y 22314 Brick serverB:/gluster_bricks/data4 49169 0 Y 22345 Brick serverB:/gluster_bricks/data5 49170 0 Y 22889 Brick serverB:/gluster_bricks/data6 49171 0 Y 22932 Self-heal Daemon on localhost N/A N/A Y 22981 Self-heal Daemon on serverA.example.com N/A N/A Y 6202 After adding the bricks we ran a rebalance from serverA as: # gluster volume rebalance tank start The rebalance completed: # gluster volume rebalance tank status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 0 0 0 completed 3:7:10 serverA.example.com 0 0Bytes 0 0 0 completed 0:0:0 volume rebalance: tank: success However, when I run a df, the two original bricks still show all of the consumed space (this is the same on both nodes): # df -hP Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-root 5.0G 625M 4.4G 13% / devtmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 32G 67M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/mapper/vg0-usr 20G 3.6G 17G 18% /usr /dev/md126 1014M 228M 787M 23% /boot /dev/mapper/vg0-home 5.0G 37M 5.0G 1% /home /dev/mapper/vg0-opt 5.0G 37M 5.0G 1% /opt /dev/mapper/vg0-tmp 5.0G 33M 5.0G 1% /tmp /dev/mapper/vg0-var 20G 2.6G 18G 13% /var /dev/mapper/gluster_vg-gluster_lv1_data 60T 59T 1.1T 99% /gluster_bricks/data1 /dev/mapper/gluster_vg-gluster_lv2_data 60T 58T 1.3T 98% /gluster_bricks/data2 /dev/mapper/gluster_vg-gluster_lv3_data 60T 451M 60T 1% /gluster_bricks/data3 /dev/mapper/gluster_vg-gluster_lv4_data 60T 451M 60T 1% /gluster_bricks/data4 /dev/mapper/gluster_vg-gluster_lv5_data 60T 451M 60T 1% /gluster_bricks/data5 /dev/mapper/gluster_vg-gluster_lv6_data 60T 451M 60T 1% /gluster_bricks/data6 localhost:/tank 355T 116T 239T 33% /mnt/tank We were thinking that the used space would be distributed across the now 6 bricks after rebalance. Is that not what a rebalance does? Is this expected behavior? Can anyone provide some guidance as to what the behavior here and if there is anything that we need to do at this point? Thanks in advance, HB -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Sat Aug 31 11:19:49 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Sat, 31 Aug 2019 14:19:49 +0300 Subject: [Gluster-users] Rebalancing newly added bricks Message-ID: The rebalance status show 0 Bytes. Maybe you should try with the 'gluster volume rebalance start force' ? Best Regards, Strahil Nikolov Source: https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#rebalancing-volumes On Aug 30, 2019 20:04, Herb Burnswell wrote: > > All, > > RHEL 7.5 > Gluster 3.8.15 > 2 Nodes: serverA & serverB > > I am not deeply knowledgeable about Gluster and it's administration but we have a 2 node cluster that's been running for about a year and a half.? All has worked fine to date.? Our main volume has consisted of two 60TB bricks on each of the cluster nodes.? As we reached capacity on the volume we needed to expand.? So, we've added four new 60TB bricks to each of the cluster nodes.? The bricks are now seen, and the total size of the volume is as expected: > > # gluster vol status tank > Status of volume: tank > Gluster process ? ? ? ? ? ? ? ? ? ? ? ? ? ? TCP Port ?RDMA Port ?Online ?Pid > ------------------------------------------------------------------------------ > Brick serverA:/gluster_bricks/data1 ? ? ? 49162 ? ? 0 ? ? ? ? ?Y ? ? ? 20318 > Brick serverB:/gluster_bricks/data1 ? ? ? 49166 ? ? 0 ? ? ? ? ?Y ? ? ? 3432 > Brick serverA:/gluster_bricks/data2 ? ? ? 49163 ? ? 0 ? ? ? ? ?Y ? ? ? 20323 > Brick serverB:/gluster_bricks/data2 ? ? ? 49167 ? ? 0 ? ? ? ? ?Y ? ? ? 3435 > Brick serverA:/gluster_bricks/data3 ? ? ? 49164 ? ? 0 ? ? ? ? ?Y ? ? ? 4625 > Brick serverA:/gluster_bricks/data4 ? ? ? 49165 ? ? 0 ? ? ? ? ?Y ? ? ? 4644 > Brick serverA:/gluster_bricks/data5 ? ? ? 49166 ? ? 0 ? ? ? ? ?Y ? ? ? 5088 > Brick serverA:/gluster_bricks/data6 ? ? ? 49167 ? ? 0 ? ? ? ? ?Y ? ? ? 5128 > Brick serverB:/gluster_bricks/data3 ? ? ? 49168 ? ? 0 ? ? ? ? ?Y ? ? ? 22314 > Brick serverB:/gluster_bricks/data4 ? ? ? 49169 ? ? 0 ? ? ? ? ?Y ? ? ? 22345 > Brick serverB:/gluster_bricks/data5 ? ? ? 49170 ? ? 0 ? ? ? ? ?Y ? ? ? 22889 > Brick serverB:/gluster_bricks/data6 ? ? ? 49171 ? ? 0 ? ? ? ? ?Y ? ? ? 22932 > Self-heal Daemon on localhost ? ? ? ? ? ? N/A ? ? ? N/A ? ? ? ?Y ? ? ? 22981 > Self-heal Daemon on serverA.example.com ? N/A ? ? ? N/A ? ? ? ?Y ? ? ? 6202? > > After adding the bricks we ran a rebalance from serverA as: > > #?gluster volume rebalance tank start > > The rebalance completed: > > # gluster volume rebalance tank status > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Node Rebalanced-files ? ? ? ? ?size ? ? ? scanned ? ? ?failures ? ? ? skipped ? ? ? ? ? ? ? status ?run time in h:m:s > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?--------- ? ? ?----------- ? ----------- ? ----------- ? ----------- ? ----------- ? ? ? ? ------------ ? ? -------------- > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?localhost ? ? ? ? ? ? ? ?0 ? ? ? ?0Bytes ? ? ? ? ? ? 0 ? ? ? ? ? ? 0 ? ? ? ? ? ? 0 ? ? ? ? ? ?completed ? ? ? ?3:7:10 > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?serverA.example.com ? ? ? ?0 ? ? ? ?0Bytes ? ? ? ? ? ? 0 ? ? ? ? ? ? 0 ? ? ? ? ? ? 0 ? ? ? ? ? ?completed ? ? ? ?0:0:0 > volume rebalance: tank: success > > However, when I run a df, the two original bricks still show all of the consumed space (this is the same on both nodes): > > # df -hP > Filesystem ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Size ?Used Avail Use% Mounted on > /dev/mapper/vg0-root ? ? ? ? ? ? ? ? ? ? 5.0G ?625M ?4.4G ?13% / > devtmpfs ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?32G ? ? 0 ? 32G ? 0% /dev > tmpfs ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 32G ? ? 0 ? 32G ? 0% /dev/shm > tmpfs ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 32G ? 67M ? 32G ? 1% /run > tmpfs ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 32G ? ? 0 ? 32G ? 0% /sys/fs/cgroup > /dev/mapper/vg0-usr ? ? ? ? ? ? ? ? ? ? ? 20G ?3.6G ? 17G ?18% /usr > /dev/md126 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?1014M ?228M ?787M ?23% /boot > /dev/mapper/vg0-home ? ? ? ? ? ? ? ? ? ? 5.0G ? 37M ?5.0G ? 1% /home > /dev/mapper/vg0-opt ? ? ? ? ? ? ? ? ? ? ?5.0G ? 37M ?5.0G ? 1% /opt > /dev/mapper/vg0-tmp ? ? ? ? ? ? ? ? ? ? ?5.0G ? 33M ?5.0G ? 1% /tmp > /dev/mapper/vg0-var ? ? ? ? ? ? ? ? ? ? ? 20G ?2.6G ? 18G ?13% /var > /dev/mapper/gluster_vg-gluster_lv1_data ? 60T ? 59T ?1.1T ?99% /gluster_bricks/data1 > /dev/mapper/gluster_vg-gluster_lv2_data ? 60T ? 58T ?1.3T ?98% /gluster_bricks/data2 > /dev/mapper/gluster_vg-gluster_lv3_data ? 60T ?451M ? 60T ? 1% /gluster_bricks/data3 > /dev/mapper/gluster_vg-gluster_lv4_data ? 60T ?451M ? 60T ? 1% /gluster_bricks/data4 > /dev/mapper/gluster_vg-gluster_lv5_data ? 60T ?451M ? 60T ? 1% /gluster_bricks/data5 > /dev/mapper/gluster_vg-gluster_lv6_data ? 60T ?451M ? 60T ? 1% /gluster_bricks/data6 > localhost:/tank ? ? ? ? ? ? ? ? ? ? ? ? ?355T ?116T ?239T ?33% /mnt/tank > > We were thinking that the used space would be distributed across the now 6 bricks after rebalance.? Is that not what a rebalance does?? Is this expected behavior? > > Can anyone provide some guidance as to what the behavior here and if there is anything that we need to do at this point? > > Thanks in advance, > > HB -------------- next part -------------- An HTML attachment was scrubbed... URL: From pankajk81 at gmail.com Sat Aug 31 15:21:31 2019 From: pankajk81 at gmail.com (Pankaj Kumar) Date: Sat, 31 Aug 2019 08:21:31 -0700 Subject: [Gluster-users] Mount Gluster for untrusted users Message-ID: Hi Is it possible to not let a user with root access on the network not mount glisterfs with any other UID than his/her own? Is there a way to authenticate mount requests? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From herbert.burnswell at gmail.com Sat Aug 31 17:28:46 2019 From: herbert.burnswell at gmail.com (Herb Burnswell) Date: Sat, 31 Aug 2019 10:28:46 -0700 Subject: [Gluster-users] Rebalancing newly added bricks In-Reply-To: References: Message-ID: Thank you for the reply. I started a rebalance with force on serverA as suggested. Now I see 'activity' on that node: # gluster vol rebalance tank status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 6143 6.1GB 9542 0 0 in progress 0:4:5 serverB 0 0Bytes 7 0 0 in progress 0:4:5 volume rebalance: tank: success But I am not seeing any activity on serverB. Is this expected? Does the rebalance need to run on each node even though it says both nodes are 'in progress'? Thanks, HB On Sat, Aug 31, 2019 at 4:18 AM Strahil wrote: > The rebalance status show 0 Bytes. > > Maybe you should try with the 'gluster volume rebalance start > force' ? > > Best Regards, > Strahil Nikolov > > Source: > https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#rebalancing-volumes > On Aug 30, 2019 20:04, Herb Burnswell wrote: > > All, > > RHEL 7.5 > Gluster 3.8.15 > 2 Nodes: serverA & serverB > > I am not deeply knowledgeable about Gluster and it's administration but we > have a 2 node cluster that's been running for about a year and a half. All > has worked fine to date. Our main volume has consisted of two 60TB bricks > on each of the cluster nodes. As we reached capacity on the volume we > needed to expand. So, we've added four new 60TB bricks to each of the > cluster nodes. The bricks are now seen, and the total size of the volume > is as expected: > > # gluster vol status tank > Status of volume: tank > Gluster process TCP Port RDMA Port Online > Pid > > ------------------------------------------------------------------------------ > Brick serverA:/gluster_bricks/data1 49162 0 Y > 20318 > Brick serverB:/gluster_bricks/data1 49166 0 Y > 3432 > Brick serverA:/gluster_bricks/data2 49163 0 Y > 20323 > Brick serverB:/gluster_bricks/data2 49167 0 Y > 3435 > Brick serverA:/gluster_bricks/data3 49164 0 Y > 4625 > Brick serverA:/gluster_bricks/data4 49165 0 Y > 4644 > Brick serverA:/gluster_bricks/data5 49166 0 Y > 5088 > Brick serverA:/gluster_bricks/data6 49167 0 Y > 5128 > Brick serverB:/gluster_bricks/data3 49168 0 Y > 22314 > Brick serverB:/gluster_bricks/data4 49169 0 Y > 22345 > Brick serverB:/gluster_bricks/data5 49170 0 Y > 22889 > Brick serverB:/gluster_bricks/data6 49171 0 Y > 22932 > Self-heal Daemon on localhost N/A N/A Y > 22981 > Self-heal Daemon on serverA.example.com N/A N/A Y > 6202 > > After adding the bricks we ran a rebalance from serverA as: > > # gluster volume rebalance tank start > > The rebalance completed: > > # gluster volume rebalance tank status > Node Rebalanced-files size > scanned failures skipped status run time in > h:m:s > --------- ----------- ----------- > ----------- ----------- ----------- ------------ > -------------- > localhost 0 0Bytes > 0 0 0 completed 3:7:10 > serverA.example.com 0 0Bytes > 0 0 0 completed 0:0:0 > volume rebalance: tank: success > > However, when I run a df, the two original bricks still show all of the > consumed space (this is the same on both nodes): > > # df -hP > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/vg0-root 5.0G 625M 4.4G 13% / > devtmpfs 32G 0 32G 0% /dev > tmpfs 32G 0 32G 0% /dev/shm > tmpfs 32G 67M 32G 1% /run > tmpfs 32G 0 32G 0% > /sys/fs/cgroup > /dev/mapper/vg0-usr 20G 3.6G 17G 18% /usr > /dev/md126 1014M 228M 787M 23% /boot > /dev/mapper/vg0-home 5.0G 37M 5.0G 1% /home > /dev/mapper/vg0-opt 5.0G 37M 5.0G 1% /opt > /dev/mapper/vg0-tmp 5.0G 33M 5.0G 1% /tmp > /dev/mapper/vg0-var 20G 2.6G 18G 13% /var > /dev/mapper/gluster_vg-gluster_lv1_data 60T 59T 1.1T 99% > /gluster_bricks/data1 > /dev/mapper/gluster_vg-gluster_lv2_data 60T 58T 1.3T 98% > /gluster_bricks/data2 > /dev/mapper/gluster_vg-gluster_lv3_data 60T 451M 60T 1% > /gluster_bricks/data3 > /dev/mapper/gluster_vg-gluster_lv4_data 60T 451M 60T 1% > /gluster_bricks/data4 > /dev/mapper/gluster_vg-gluster_lv5_data 60T 451M 60T 1% > /gluster_bricks/data5 > /dev/mapper/gluster_vg-gluster_lv6_data 60T 451M 60T 1% > /gluster_bricks/data6 > localhost:/tank 355T 116T 239T 33% /mnt/tank > > We were thinking that the used space would be distributed across the now 6 > bricks after rebalance. Is that not what a rebalance does? Is this > expected behavior? > > Can anyone provide some guidance as to what the behavior here and if there > is anything that we need to do at this point? > > Thanks in advance, > > HB > > -------------- next part -------------- An HTML attachment was scrubbed... URL: