From gaio at lilliput.linux.it Fri Sep 8 14:13:31 2023 From: gaio at lilliput.linux.it (Marco Gaiarin) Date: Fri, 8 Sep 2023 16:13:31 +0200 Subject: [Gluster-users] GlusterFS, move files, Samba ACL... Message-ID: <9pmrsj-kb52.ln1@hermione.lilliput.linux.it> A little strange things, but i'm hitting my head on the wall... I needed to 'enlarge' my main filesystem (XFS backed-up), that contain my main samba share and a brick for a GFS share; i've setup a new volume (for the VM), formatted XFS, move all the file taking care to umount and stop GFS (so, syncing the brick, not the GFS filesystem) using --acls and -attrs rsync options. All went well, all the share work as expected, apart the GFS one: folder appears with correct permission, files are not accessible from samba; i've tried from a unix user in console and works as expected. If (from windows) i create new files, permissions are OK. Only preexistent files (files, not folders!) are inaccessible. In samba the share is: [FVG] comment = Regionale (FVG) volume = FVG path = / browseable = yes writeable = yes map acl inherit = yes store dos attributes = yes inherit permissions = yes vfs objects = recycle full_audit glusterfs glusterfs:volume = gv0 kernel share modes = no recycle:repository = .cestino/%U recycle:keeptree = yes recycle:versions = yes recycle:exclude=*.TMP,*.tmp,*.temp,*.o,*.obj,~$* full_audit:prefix = %S|%d|%I|%M|%u full_audit:success = mkdir rmdir read pread write pwrite rename unlink full_audit:failure = none gluster version 3.8.8-1+deb9u1, samba version 4.10.18+dfsg-0.1stretch1 . Permission linux side are different but seems not incoherent: root at vdmsv1:/srv/fvg/Ufficio Paghe# getfattr -m - -d Copia\ di\ fvg.xlsx Prova\,\ DELME.txt # file: Copia di fvg.xlsx system.posix_acl_access=0sAgAAAAEABwD/////AgAHAAAAAAAEAAcA/////wgABwD5KgAACAAFAAgrAAAIAAcAHysAAAgABwA3KwAAEAAHAP////8gAAAA/////w== trusted.SGI_ACL_FILE=0sAAAACQAAAAH/////AAcAAAAAAAIAAAAAAAcAAAAAAAT/////AAcAAAAAAAgAACr5AAcAAAAAAAgAACsIAAUAAAAAAAgAACsfAAcAAAAAAAgAACs3AAcAAAAAABD/////AAcAAAAAACD/////AAAAAA== user.DOSATTRIB=0sMHgyMAAAAwADAAAAEQAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABD7z36KtcBAAAAAAAAAAA= user.SAMBA_PAI=0sAgSQCQAAAAABOCsAAAAAKTAAAAAAKTAAAAABOCsAAAABNysAAAABHysAAAABCCsAAAAB+SoAAAAC/////w== # file: Prova, DELME.txt system.posix_acl_access=0sAgAAAAEABwD/////BAAHAP////8IAAcA+SoAAAgABQAIKwAACAAHAB8rAAAIAAcANysAABAABgD/////IAAAAP////8= trusted.SGI_ACL_FILE=0sAAAACAAAAAH/////AAcAAAAAAAT/////AAcAAAAAAAgAACr5AAcAAAAAAAgAACsIAAUAAAAAAAgAACsfAAcAAAAAAAgAACs3AAcAAAAAABD/////AAYAAAAAACD/////AAAAAA== user.DOSATTRIB=0sMHgyMAAAAwADAAAAEQAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAICC3quz2dkBAAAAAAAAAAA= root at vdmsv1:/srv/fvg/Ufficio Paghe# getfacl Copia\ di\ fvg.xlsx Prova\,\ DELME.txt # file: Copia di fvg.xlsx # owner: stefano # group: pp_uffconta user::rwx user:stefano:rwx group::rw- group:sir:rw- group:sv_reamm:r-- group:sv_uffpers:rw- group:pp_uffpers:rw- group:pp_uffconta:rw- mask::rwx other::--- # file: Prova, DELME.txt # owner: gaio # group: sir user::rwx group::rwx #effective:rw- group:sir:rwx #effective:rw- group:sv_reamm:r-x #effective:r-- group:sv_uffpers:rwx #effective:rw- group:pp_uffpers:rwx #effective:rw- mask::rw- other::--- 'Copia\ di\ fvg.xlsx' is an old file, 'Prova\,\ DELME.txt' a test file created ex-novo. Thanks. -- Chi parla male, pensa male e vive male. Bisogna trovare le parole giuste: le parole sono importanti! Nanni Moretti in Palombella Rossa From jeremy at rosalind.bio Tue Sep 12 15:53:25 2023 From: jeremy at rosalind.bio (Jeremy Davis-Turak) Date: Tue, 12 Sep 2023 09:53:25 -0600 Subject: [Gluster-users] Recovering files "lost" during a rebalance on a Dispersed 3+1 Message-ID: Hello, We are running glusterfs 6.6 on Ubuntu. We have a Gluster storage system that is a few years old. There are 4 VMs running a Dispersed (NOT replicated) system - a 3 + 1 configuration. Generally performance is well tuned for our needs, but the problem arose last time we added bricks: we attempted a rebalance which is reported as failed. From the mounted POSIX view of the file system, we see many files that report to be of size 0 bytes, which they shouldn?t be. We?ve attempted all kinds of heal and other operations to no avail. I finally figured out how to find the gfid of the files , and I found where it thought the shards were located. They were indeed 0 bytes ? however, I was able to find shards *with the same gfid *located on other bricks. So, I think that when the rebalance failed, somehow the system kept thinking that the files should exist in the NEW brick location instead of the one that actually has content. For one file I did try to delete the shards of size 0, but the system still thinks that the file is of size 0, which means it didn?t point to the other shards with the same gfid. Is it possible to manually move shards from brick to another? I'm clearly tinkering with things that aren't meant to be tinkered with ... but I don't fully understand how GlusterFS functions under the hood. We?re at a loss as to how to fix this, and I haven?t had luck finding anyone who can help. We have quite a few files that we would like to recover, so it?s important that we figure out how to. Thanks, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From omri.sarig at prevas.dk Fri Sep 15 08:59:18 2023 From: omri.sarig at prevas.dk (Omri Sarig) Date: Fri, 15 Sep 2023 08:59:18 +0000 Subject: [Gluster-users] Full docementation of available volume setting Message-ID: Hello, I'm a new Gluster user, currently mostly investigating the potential use of the system for our company. I've read a lot of the official documentation, and I'm able to do most basic operations with the system. However, one thing I'm having difficulty finding is complete documentation of the available options to fine-tune volumes. I've seen that with the command "gluster volume get all"? I can see the full list of available options for the volume, but I could not find documentation for these options. I've tried looking in Gluster documentation, as well as through Google searches, but did not find information about most of these commands. What I'm looking for is a page containing a list of all the available options, with a short description for each option. Does something like this exist? Any help will be greatly appreciated. With Kind Regards, Omri Sarig, Software Engineer [e-brev2] Visit: Lysk?r 3EF, DK-2730 Herlev Mobile: +45 28 49 42 27 omri.sarig at prevas.dk www.prevas.dk -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-e-brev2.png Type: image/png Size: 8505 bytes Desc: Outlook-e-brev2.png URL: From gaio at lilliput.linux.it Wed Sep 20 13:11:42 2023 From: gaio at lilliput.linux.it (Marco Gaiarin) Date: Wed, 20 Sep 2023 15:11:42 +0200 Subject: [Gluster-users] GlusterFS, move files, Samba ACL... Message-ID: [ Received no feedback, i resend it... ] A little strange things, but i'm hitting my head on the wall... I needed to 'enlarge' my main filesystem (XFS backed-up), that contain my main samba share and a brick for a GFS share; i've setup a new volume (for the VM), formatted XFS, move all the file taking care to umount and stop GFS (so, syncing the brick, not the GFS filesystem) using --acls and -attrs rsync options. All went well, all the share work as expected, apart the GFS one: folder appears with correct permission, files are not accessible from samba; i've tried from a unix user in console and works as expected. If (from windows) i create new files, permissions are OK. Only preexistent files (files, not folders!) are inaccessible. In samba the share is: [FVG] comment = Regionale (FVG) volume = FVG path = / browseable = yes writeable = yes map acl inherit = yes store dos attributes = yes inherit permissions = yes vfs objects = recycle full_audit glusterfs glusterfs:volume = gv0 kernel share modes = no recycle:repository = .cestino/%U recycle:keeptree = yes recycle:versions = yes recycle:exclude=*.TMP,*.tmp,*.temp,*.o,*.obj,~$* full_audit:prefix = %S|%d|%I|%M|%u full_audit:success = mkdir rmdir read pread write pwrite rename unlink full_audit:failure = none gluster version 3.8.8-1+deb9u1, samba version 4.10.18+dfsg-0.1stretch1 . Permission linux side are different but seems not incoherent: root at vdmsv1:/srv/fvg/Ufficio Paghe# getfattr -m - -d Copia\ di\ fvg.xlsx Prova\,\ DELME.txt # file: Copia di fvg.xlsx system.posix_acl_access=0sAgAAAAEABwD/////AgAHAAAAAAAEAAcA/////wgABwD5KgAACAAFAAgrAAAIAAcAHysAAAgABwA3KwAAEAAHAP////8gAAAA/////w== trusted.SGI_ACL_FILE=0sAAAACQAAAAH/////AAcAAAAAAAIAAAAAAAcAAAAAAAT/////AAcAAAAAAAgAACr5AAcAAAAAAAgAACsIAAUAAAAAAAgAACsfAAcAAAAAAAgAACs3AAcAAAAAABD/////AAcAAAAAACD/////AAAAAA== user.DOSATTRIB=0sMHgyMAAAAwADAAAAEQAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABD7z36KtcBAAAAAAAAAAA= user.SAMBA_PAI=0sAgSQCQAAAAABOCsAAAAAKTAAAAAAKTAAAAABOCsAAAABNysAAAABHysAAAABCCsAAAAB+SoAAAAC/////w== # file: Prova, DELME.txt system.posix_acl_access=0sAgAAAAEABwD/////BAAHAP////8IAAcA+SoAAAgABQAIKwAACAAHAB8rAAAIAAcANysAABAABgD/////IAAAAP////8= trusted.SGI_ACL_FILE=0sAAAACAAAAAH/////AAcAAAAAAAT/////AAcAAAAAAAgAACr5AAcAAAAAAAgAACsIAAUAAAAAAAgAACsfAAcAAAAAAAgAACs3AAcAAAAAABD/////AAYAAAAAACD/////AAAAAA== user.DOSATTRIB=0sMHgyMAAAAwADAAAAEQAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAICC3quz2dkBAAAAAAAAAAA= root at vdmsv1:/srv/fvg/Ufficio Paghe# getfacl Copia\ di\ fvg.xlsx Prova\,\ DELME.txt # file: Copia di fvg.xlsx # owner: stefano # group: pp_uffconta user::rwx user:stefano:rwx group::rw- group:sir:rw- group:sv_reamm:r-- group:sv_uffpers:rw- group:pp_uffpers:rw- group:pp_uffconta:rw- mask::rwx other::--- # file: Prova, DELME.txt # owner: gaio # group: sir user::rwx group::rwx #effective:rw- group:sir:rwx #effective:rw- group:sv_reamm:r-x #effective:r-- group:sv_uffpers:rwx #effective:rw- group:pp_uffpers:rwx #effective:rw- mask::rw- other::--- 'Copia\ di\ fvg.xlsx' is an old file, 'Prova\,\ DELME.txt' a test file created ex-novo. Thanks. -- ...noi leggiamo il Vangelo come se non avessimo soldi, e usiamo i soldi come se non conoscessimo nulla del Vangelo... (un Gesuita inglese) From revirii at googlemail.com Thu Sep 21 16:34:29 2023 From: revirii at googlemail.com (Hu Bert) Date: Thu, 21 Sep 2023 18:34:29 +0200 Subject: [Gluster-users] Full docementation of available volume setting In-Reply-To: References: Message-ID: Hi, i use this: gluster volume set help > /path/to/file.txt Then you have a text file where you can search in. Best regards, Hubert Am Di., 19. Sept. 2023 um 18:55 Uhr schrieb Omri Sarig : > Hello, > > I'm a new Gluster user, currently mostly investigating the potential use > of the system for our company. > > I've read a lot of the official documentation > , and I'm able to do most basic > operations with the system. > > However, one thing I'm having difficulty finding is complete documentation > of the available options to fine-tune volumes. > I've seen that with the command "gluster volume get all"? I > can see the full list of available options for the volume, but I could not > find documentation for these options. > I've tried looking in Gluster documentation, as well as through Google > searches, but did not find information about most of these commands. > > What I'm looking for is a page containing a list of all the available > options, with a short description for each option. > Does something like this exist? > > Any help will be greatly appreciated. > > With Kind Regards, > > > *Omri Sarig, * > > Software Engineer > > *[image: e-brev2]* > > Visit: Lysk?r 3EF, DK-2730 Herlev > > Mobile: +45 28 49 42 27 > > omri.sarig at prevas.dk > > www.prevas.dk > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-e-brev2.png Type: image/png Size: 8505 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-e-brev2.png Type: image/png Size: 8505 bytes Desc: not available URL: From wkmail at bneit.com Thu Sep 21 18:22:05 2023 From: wkmail at bneit.com (W Kern) Date: Thu, 21 Sep 2023 11:22:05 -0700 Subject: [Gluster-users] mount volumes ON the arbiter? Message-ID: <43e530f7-66c5-465a-b171-5762d3ff86df@bneit.com> So, I am looking to solve an extremely minor argument among my co-workers. Since the list is quiet I figured I'd throw this out there. Its clearly not a problem, bug, etc so obviously ignore it if you better things to do We have lots of 2 data hosts +1 arbiter clusters around.? Mostly corresponding to Libvirt pools. So we have something akin to /POOL-GL1 available on the two data notes using fuse. We also mount /POOL-GL1 on the arbiter. They correspond to the actual volume GL1. Obviously we can be logged into the arbiter and manipulate the mounted files just as we would on the real data nodes and because we may use multiple arbiters on the same kit, it can be more convenient to login there when moving files among clusters. The question is: Assuming equal hardware network speed and similar hard disk i/o performance, then if we are transferring a large file (say a VM image) then is it more efficient to copy that into the mounted directory on one of the real data hosts or do you get the same efficiency just uploading it onto the arb node? Obviously if you copy a large file into the mount on the arb it is not actually being uploaded there, but is rather being copied out to the two data nodes which have the real data and only the meta data is retained on the arb. So the question is by uploading to the arb are we doing extra work and is it more efficient to upload into one of the volumes on a data host where it only has to copy data off to the other data volume and the metadata onto the arb. We ran a few tests which implied the direct to host option was a 'little' faster but they were on production equipment which has varying loads and thus we couldn't compare cpu loads or come to a firm conclusion. Its not yet an important enough issue to build up a test bed, so we were wondering if perhaps someone else already knows the answer based on an understanding of the architecture or perhaps they did do the testing? -bill From omri.sarig at prevas.dk Fri Sep 22 07:09:15 2023 From: omri.sarig at prevas.dk (Omri Sarig) Date: Fri, 22 Sep 2023 07:09:15 +0000 Subject: [Gluster-users] Full docementation of available volume setting In-Reply-To: References: Message-ID: That was what I was looking for. Thank you very much Hubert! With Kind Regards, Omri ________________________________ From: Hu Bert Sent: Thursday, September 21, 2023 6:34 PM To: Omri Sarig Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Full docementation of available volume setting Hi, i use this: gluster volume set help > /path/to/file.txt Then you have a text file where you can search in. Best regards, Hubert Am Di., 19. Sept. 2023 um 18:55 Uhr schrieb Omri Sarig >: Hello, I'm a new Gluster user, currently mostly investigating the potential use of the system for our company. I've read a lot of the official documentation, and I'm able to do most basic operations with the system. However, one thing I'm having difficulty finding is complete documentation of the available options to fine-tune volumes. I've seen that with the command "gluster volume get all"? I can see the full list of available options for the volume, but I could not find documentation for these options. I've tried looking in Gluster documentation, as well as through Google searches, but did not find information about most of these commands. What I'm looking for is a page containing a list of all the available options, with a short description for each option. Does something like this exist? Any help will be greatly appreciated. With Kind Regards, Omri Sarig, Software Engineer [e-brev2] Visit: Lysk?r 3EF, DK-2730 Herlev Mobile: +45 28 49 42 27 omri.sarig at prevas.dk www.prevas.dk ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.decker at ora-solutions.net Fri Sep 29 15:36:32 2023 From: martin.decker at ora-solutions.net (Martin Decker) Date: Fri, 29 Sep 2023 17:36:32 +0200 Subject: [Gluster-users] gluster volume status shows -> Online "N" after node reboot. Message-ID: Hi list, I am using a replica volume (3 nodes) gluster in an ovirt environment and after setting one node in maintenance mode and rebooting it, the "Online" flag in gluster volume status does not go to "Y" again. [root at node1 glusterfs]# gluster volume status Status of volume: my_volume Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.22.1.95:/gluster_bricks/my_glust er/my_gluster N/A N/A N N/A Brick 10.22.1.97:/gluster_bricks/my_glust er/my_gluster 49152 0 Y 4954 Brick 10.22.1.94:/gluster_bricks/my_glust er/my_gluster 49152 0 Y 3574 Self-heal Daemon on localhost N/A N/A Y 3585 Self-heal Daemon on node2 N/A N/A Y 3557 Self-heal Daemon on node3 N/A N/A Y 4973 Task Status of Volume my_volume ------------------------------------------------------------------------------ There are no active volume tasks Shouldn?t it go back to Online Y automatically? This is the output from gluster volume info from the same node: [root at node1 glusterfs]# gluster volume info Volume Name: my_volume Type: Replicate Volume ID: 78b9299c-1df5-4780-b108-4d3a6dee225d Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.22.1.95:/gluster_bricks/my_gluster/my_gluster Brick2: 10.22.1.97:/gluster_bricks/my_gluster/my_gluster Brick3: 10.22.1.94:/gluster_bricks/my_gluster/my_gluster Options Reconfigured: cluster.granular-entry-heal: enable storage.owner-gid: 36 storage.owner-uid: 36 cluster.lookup-optimize: off server.keepalive-count: 5 server.keepalive-interval: 2 server.keepalive-time: 10 server.tcp-user-timeout: 20 network.ping-timeout: 30 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable performance.strict-o-direct: on network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on performance.client-io-threads: on Regards, Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: