[Gluster-users] How to use system.affinity/distributed.migrate-data on distributed/replicated volume?

Wed Oct 31 03:57:11 UTC 2018

nufa helps you write to local brick, if replication is involved it will
still copy it to other bricks (or suppose to do so)
what might be happening is that when initial file was created other nodes
were down and it didn't replicate properly and now heal is failing
check your
gluster vol heal Volname info

maybe you will find out where second copy of the file suppose to be - and
just copy it to that brick

On Sun, Oct 28, 2018 at 6:07 PM Ingo Fischer <nl at fischer-ka.de> wrote:

> Hi All,
>
> has noone an idea on system.affinity/distributed.migrate-data ?
> Or how to correctly enable nufa?
>
> BTW: the used gluster version is 4.1.5
>
> Thank you for your help on this!
>
> Ingo
>
> Am 24.10.18 um 12:54 schrieb Ingo Fischer:
> > Hi,
> >
> > I have setup a glusterfs volume gv0 as distributed/replicated:
> >
> > root at pm1:~# gluster volume info gv0
> >
> > Volume Name: gv0
> > Type: Distributed-Replicate
> > Volume ID: 64651501-6df2-4106-b330-fdb3e1fbcdf4
> > Status: Started
> > Snapshot Count: 0
> > Number of Bricks: 3 x 2 = 6
> > Transport-type: tcp
> > Bricks:
> > Brick1: 192.168.178.50:/gluster/brick1/gv0
> > Brick2: 192.168.178.76:/gluster/brick1/gv0
> > Brick3: 192.168.178.50:/gluster/brick2/gv0
> > Brick4: 192.168.178.81:/gluster/brick1/gv0
> > Brick5: 192.168.178.50:/gluster/brick3/gv0
> > Brick6: 192.168.178.82:/gluster/brick1/gv0
> > Options Reconfigured:
> > performance.client-io-threads: off
> > nfs.disable: on
> > transport.address-family: inet
> >
> >
> > root at pm1:~# gluster volume status
> > Status of volume: gv0
> > Gluster process                             TCP Port  RDMA Port  Online
> Pid
> >
> ------------------------------------------------------------------------------
> > Brick 192.168.178.50:/gluster/brick1/gv0    49152     0          Y
> > 1665
> > Brick 192.168.178.76:/gluster/brick1/gv0    49152     0          Y
> > 26343
> > Brick 192.168.178.50:/gluster/brick2/gv0    49153     0          Y
> > 1666
> > Brick 192.168.178.81:/gluster/brick1/gv0    49152     0          Y
> > 1161
> > Brick 192.168.178.50:/gluster/brick3/gv0    49154     0          Y
> > 1679
> > Brick 192.168.178.82:/gluster/brick1/gv0    49152     0          Y
> > 1334
> > Self-heal Daemon on localhost               N/A       N/A        Y
> > 5022
> > Self-heal Daemon on 192.168.178.81          N/A       N/A        Y
> > 935
> > Self-heal Daemon on 192.168.178.82          N/A       N/A        Y
> > 1057
> > Self-heal Daemon on pm2.fritz.box           N/A       N/A        Y
> > 1651
> >
> >
> > I use the fs to store VM files, so not many, but big files.
> >
> > The distribution now put 4 big files on one brick set and only one file
> > on an other. This means that the one brick set it "overcommited" now as
> > soon as all VMs using max space. SO I would like to manually
> > redistribute the files a bit better.
> >
> > After log googling I found that the following should work:
> > setfattr -n 'system.affinity' -v $location $filepath
> > setfattr -n 'distribute.migrate-data' -v 'force' $filepath
> >
> > But I have problems with it because it gives errors or doing nothing at
> all.
> >
> > The mounting looks like:
> > 192.168.178.50:gv0 on /mnt/pve/glusterfs type fuse.glusterfs
> >
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
> >
> >
> > Here is what I tried for the first xattr:
> >
> > root at pm1:~# setfattr -n 'system.affinity' -v 'gv0-client-5'
> > /mnt/pve/glusterfs/201/imagesvm201.qcow2
> > setfattr: /mnt/pve/glusterfs/201/imagesvm201.qcow2: Operation not
> supported
> >
> > So I found on google to use trusted.affinity instead and yes this works.
> > I'm only not sure if the location "gv0-client-5" is correct to move the
> > file to "Brick 5" from "gluster volume info gv0" ... or how this
> > location is build?
> > Commit Message from http://review.gluster.org/#/c/glusterfs/+/5233/ says
> >> The value is the internal client or AFR brick name where you want the
> > file to be.
> >
> > So what do I need to set there? maybe I do need the "afr" because
> > replicated? But where to get that name from?
> > I also tried to enter other client or replicate names like
> > "gv0-replicate-0" or such which seems to be more fitting for a
> > replicated volume, but result the same.
> >
> >
> > For the second command I get:
> > root at pm1:~# setfattr -n 'distribute.migrate-data' -v 'force'
> > /mnt/pve/glusterfs/201/imagesvm201.qcow2
> > setfattr: /mnt/pve/glusterfs/images/201/vm-201-disk-0.qcow2: Operation
> > not supported
> > root at pm1:~# setfattr -n 'trusted.distribute.migrate-data' -v 'force'
> > /mnt/pve/glusterfs/201/imagesvm201.qcow2
> > setfattr: /mnt/pve/glusterfs/images/201/vm-201-disk-0.qcow2: File exists
> >
> > I also experimented with other "names" then "gv0-client-5" above but
> > always the same.
> >
> > I saw that instead of the second command I could start a rebalance with
> > force, but this also did nothing. Ended after max1 second and moved
> nothing.
> >
> > Can someone please advice how to do it right?
> >
> >
> > An other idea was to enable nufa and kind of "re-copy" the files on the
> > glusterfs, but here it seems that the documentation is wrong.
> > gluster volume set gv0 cluster.nufa enable on
> >
> > Is
> >
> > gluster volume set gv0 cluster.nufa 1
> >
> > correct?
> >
> > Thank you very much!
> >
> > Ingo
> >
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181030/b01d3f5e/attachment.html>