[Gluster-users] trying to avoid a penalty for renaming every file
Christian Rice
crice at pandora.com
Fri Oct 28 05:18:37 UTC 2016
argh, I note for sake of correction that I renamed my volume from REGEXED to YAY after the first set of tests. Because I wanted to test rebuilding from completely nothing…but changing the name may add a touch of confusion for someone who was trying to comprehend…
anyway, a fabulous little feature, I hope it’s stable :)
From: <gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org>> on behalf of Christian Rice <crice at pandora.com<mailto:crice at pandora.com>>
Date: Thursday, October 27, 2016 at 10:02 PM
To: Joe Julian <joe at julianfamily.org<mailto:joe at julianfamily.org>>
Cc: "gluster-users at gluster.org<mailto:gluster-users at gluster.org>" <gluster-users at gluster.org<mailto:gluster-users at gluster.org>>
Subject: Re: [Gluster-users] trying to avoid a penalty for renaming every file
That is great. I appreciate the encouragement to continue in the right direction. What a relief!
I’m including my test procedure, in case it helps someone to visualize in ways a slide cannot. Using Debian 7/Wheezy, gluster bits 3.7.15-1, two hosts: burr and van
from burr:
gluster peer probe van
vol create YAY transport tcp burr:/archive/gluster van:/archive/gluster
vol set YAY cluster.extra-hash-regex "(.*)\.COPYING"
check to make sure it looks right in /var/lib/glusterd/vols/YAY/trusted-REGEXED.tcp-fuse.vol
vol start YAY
mount -t glusterfs 127.0.0.1:REGEXED ~crice/tiempo
vol info on both hosts, it should look like so:
crice at burr:~$ sudo gluster vol info
Volume Name: REGEXED
Type: Distribute
Volume ID: e751a668-97dc-4638-81cd-40b68c2438f2
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: burr:/archive/gluster
Brick2: van:/archive/gluster
Options Reconfigured:
performance.readdir-ahead: on
cluster.extra-hash-regex: "(.*)\.COPYING"
create two dirs, testA and testB
testA will have a ten files named foo1-10, but they were created as foo1-10.COPYING then moved
testB wil have ten fies named bar1.10, but they were created as bar1-10.BAD
cd testA
for i in `seq 1 10`; do dd if=/dev/zero of=foo.${i}.COPYING bs=1M count=10; mv foo.${i}.COPYING foo.$i; done
cd testB
for i in `seq 1 10`; do dd if=/dev/zero of=foo.${i}.BAD bs=1M count=10; mv foo.${i}.BAD foo.$i; done
then check /archive/gluster/test{A,B} for link files on both burr and van. The testA dir should have no link files.
results:
crice at burr:/archive/gluster$ ls -lR testA testB
testA:
total 61440
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.10
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.3
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.4
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.5
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.6
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.7
testB:
total 61440
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.1
---------T 2 crice crice 0 Oct 27 21:17 foo.10
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.2
---------T 2 crice crice 0 Oct 27 21:17 foo.3
---------T 2 crice crice 0 Oct 27 21:17 foo.4
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.5
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.6
---------T 2 crice crice 0 Oct 27 21:17 foo.7
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.8
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.9
crice at van:/archive/gluster$ ls -lR testA testB
testA:
total 40960
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.1
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.2
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.8
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:16 foo.9
testB:
total 40960
---------T 2 crice crice 0 Oct 27 21:17 foo.1
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.10
---------T 2 crice crice 0 Oct 27 21:17 foo.2
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.3
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.4
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:17 foo.7
---------T 2 crice crice 0 Oct 27 21:17 foo.8
---------T 2 crice crice 0 Oct 27 21:17 foo.9
that totally effin worked! and validated by repeating procedure based on my notes
But having been tested using fuse, to be absolutely sure (because I don’t wanna read code) off I ran:
gluster vol set YAY nfs.disable false (glusterd immediately died, so I restarted glusterfs-server, created a testC and testD dir and ran one more set over NFS:
cd /mnt/net/burr/YAY/
mkdir testC testD
cd testC
for i in `seq 1 10`; do dd if=/dev/zero of=foo.${i}.COPYING bs=1M count=10; mv foo.${i}.COPYING foo.$i; done
cd testD
for i in `seq 1 10`; do dd if=/dev/zero of=foo.${i}.BAD bs=1M count=10; mv foo.${i}.BAD foo.$i; done
and it was also good:
crice at burr:/archive/gluster$ ls -lR testC testD
testC:
total 98304
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.10
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.3
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.4
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.5
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.6
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.7
testD:
total 65536
---------T 2 crice crice 0 Oct 27 21:42 foo.1
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.10
---------T 2 crice crice 0 Oct 27 21:42 foo.2
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.3
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.4
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.7
---------T 2 crice crice 0 Oct 27 21:42 foo.8
---------T 2 crice crice 0 Oct 27 21:42 foo.9
crice at van:/archive/gluster$ ls -lR testC testD
testC:
total 65536
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.1
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.2
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.8
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:41 foo.9
testD:
total 98304
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.1
---------T 2 crice crice 0 Oct 27 21:42 foo.10
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.2
---------T 2 crice crice 0 Oct 27 21:42 foo.3
---------T 2 crice crice 0 Oct 27 21:42 foo.4
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.5
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.6
---------T 2 crice crice 0 Oct 27 21:42 foo.7
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.8
-rw-r--r-- 2 crice crice 10485760 Oct 27 21:42 foo.9
happy happy! Thanks for sharing!
From: <gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org>> on behalf of Joe Julian <joe at julianfamily.org<mailto:joe at julianfamily.org>>
Date: Thursday, October 27, 2016 at 1:23 PM
To: "gluster-users at gluster.org<mailto:gluster-users at gluster.org>" <gluster-users at gluster.org<mailto:gluster-users at gluster.org>>
Subject: Re: [Gluster-users] trying to avoid a penalty for renaming every file
The best slide of all from the Gluster Developer Summit
https://twitter.com/JoeCyberGuru/status/784310321980674048<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_JoeCyberGuru_status_784310321980674048&d=CwMF-g&c=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw&r=NE1NbWtVhgG-K7YvLdoLZigfFx8zGPwOGk6HWpYK04I&m=oAvh1QvYKW9NOvU2J6Nv99NxcxuPLY8hEUgiKGVWIBE&s=HdVvih36RDFLYWffBJRi706ww0YDGiQQugFxvHQGtDI&e=>
[cid:part1.471D37CF.4FE9DD6D at julianfamily.org]
On 10/27/2016 11:52 AM, Christian Rice wrote:
We have a gluster pool where our developers are writing large files with the suffix .COPYING, then removing that suffix when the copy is done (renaming the file), so consumers of the data know the file is safe to read.
Back in 2014, Jeff Darcy wrote the below in a thread. I guess I am basically asking if this is still the supported/encouraged approach, and how do I implement this when I’m using gluster’s NFS?
"That said, there's also a trick you can use to avoid creation of a
linkfile. Other tools, such as rsync and our own object interface,
use the same write-then-rename idiom. To serve them, there's an
option called extra-hash-regex that can be used to place files on the
"right" brick according to their final name even though they're created
with another. Unfortunately, specifying that option via the command line
doesn't seem to work (it creates a malformed volfile) so you have to
mount a bit differently. For example:
glusterfs --volfile-server=a_server --volfile-id=a_volume \
--xlator-option a_volume-dht.extra_hash_regex='(.*+)tmp' \
/a/mountpoint
The important part is that second line. That causes any file with a
"tmp" suffix to be hashed and placed as though only the part in the
first parenthesized part of the regex (i.e. without the "tmp") was
there. Therefore, creating "xxxtmp" and then renaming it to "xxx" is
the same as just creating "xxx" in the first place as far as linkfiles
etc. are concerned. Note that the excluded part can be anything that
a regex can match, including a unique random number. If I recall,
rsync uses temp files something like this:
fubar = .fubar.NNNNNN (where NNNNNNN is a random number)
I know this probably seems a little voodoo-ish, but with a little bit
of experimentation to find the right regex you should be able to avoid
those dreaded linkfiles altogether."
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>http://www.gluster.org/mailman/listinfo/gluster-users<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gluster.org_mailman_listinfo_gluster-2Dusers&d=CwMF-g&c=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw&r=NE1NbWtVhgG-K7YvLdoLZigfFx8zGPwOGk6HWpYK04I&m=oAvh1QvYKW9NOvU2J6Nv99NxcxuPLY8hEUgiKGVWIBE&s=5cEB0fLFIY_qGLOfLvvEEFa1Fks3TxzdcDBSgzA3fDA&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161028/1d9864ab/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2016-10-27 13-22-04.png
Type: image/png
Size: 114132 bytes
Desc: Screenshot from 2016-10-27 13-22-04.png
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161028/1d9864ab/attachment-0001.png>
More information about the Gluster-users
mailing list