[Gluster-users] How to use system.affinity/distributed.migrate-data on distributed/replicated volume?

Ingo Fischer ingo at fischer-ka.de
Wed Oct 24 10:34:37 UTC 2018


I have setup a glusterfs volume gv0 as distributed/replicated:

root at pm1:~# gluster volume info gv0

Volume Name: gv0
Type: Distributed-Replicate
Volume ID: 64651501-6df2-4106-b330-fdb3e1fbcdf4
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet

root at pm1:~# gluster volume status
Status of volume: gv0
Gluster process                             TCP Port  RDMA Port  Online  Pid
Brick    49152     0          Y
Brick    49152     0          Y
Brick    49153     0          Y
Brick    49152     0          Y
Brick    49154     0          Y
Brick    49152     0          Y
Self-heal Daemon on localhost               N/A       N/A        Y
Self-heal Daemon on          N/A       N/A        Y
Self-heal Daemon on          N/A       N/A        Y
Self-heal Daemon on pm2.fritz.box           N/A       N/A        Y

I use the fs to store VM files, so not many, but big files.

The distribution now put 4 big files on one brick set and only one file
on an other. This means that the one brick set it "overcommited" now as
soon as all VMs using max space. SO I would like to manually
redistribute the files a bit better.

After log googling I found that the following should work:
setfattr -n 'system.affinity' -v $location $filepath
setfattr -n 'distribute.migrate-data' -v 'force' $filepath

But I have problems with it because it gives errors or doing nothing at all.

The mounting looks like: on /mnt/pve/glusterfs type fuse.glusterfs

Here is what I tried for the first xattr:

root at pm1:~# setfattr -n 'system.affinity' -v 'gv0-client-5'
setfattr: /mnt/pve/glusterfs/201/imagesvm201.qcow2: Operation not supported

So I found on google to use trusted.affinity instead and yes this works.
I'm only not sure if the location "gv0-client-5" is correct to move the
file to "Brick 5" from "gluster volume info gv0" ... or how this
location is build?
Commit Message from http://review.gluster.org/#/c/glusterfs/+/5233/ says
> The value is the internal client or AFR brick name where you want the
file to be.

So what do I need to set there? maybe I do need the "afr" because
replicated? But where to get that name from?
I also tried to enter other client or replicate names like
"gv0-replicate-0" or such which seems to be more fitting for a
replicated volume, but result the same.

For the second command I get:
root at pm1:~# setfattr -n 'distribute.migrate-data' -v 'force'
setfattr: /mnt/pve/glusterfs/images/201/vm-201-disk-0.qcow2: Operation
not supported
root at pm1:~# setfattr -n 'trusted.distribute.migrate-data' -v 'force'
setfattr: /mnt/pve/glusterfs/images/201/vm-201-disk-0.qcow2: File exists

I also experimented with other "names" then "gv0-client-5" above but
always the same.

I saw that instead of the second command I could start a rebalance with
force, but this also did nothing. Ended after max1 second and moved nothing.

Can someone please advice how to do it right?

An other idea was to enable nufa and kind of "re-copy" the files on the
glusterfs, but here it seems that the documentation is wrong.
gluster volume set gv0 cluster.nufa enable on


gluster volume set gv0 cluster.nufa 1


Thank you very much!


More information about the Gluster-users mailing list