[Gluster-users] FreeBSD problem adding/removing replicated bricks

[Mark Staudinger]

Hi Folks,

I'm trying to debug an issue that I've found while attempting to qualify 
GlusterFS for potential distributed storage projects on the FreeBSD-11.1 
server platform - using the existing package of GlusterFS v3.11.1_4

The main issue I've encountered is that I cannot add new bricks while 
setting/increasing the replica count.

If I create a replicated volume "poc" on two hosts, say 
s1:/gluster/1/poc and s2:/gluster/1/poc, the volume is created properly 
and shows replicated status, files are written to both volumes.

If I create a single volume: s1:/gluster/1/poc as a single / distributed 
brick, and then try to run:

gluster volume add-brick poc replica 2 s2:/gluster/1/poc

it will always fail (sometimes after a pause, sometimes not.)  The only 
error I'm seeing on the server hosting the new brick, aside from the 
generic "Unable to add bricks" message, is like so:

I [MSGID: 106578] 
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 
0-management:  replica-count is set 2
I [MSGID: 106578] 
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management: 
type is set 2, need to change it
E [MSGID: 106054] 
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 
0-management:  Failed to set extended attribute trusted.add-brick : 
Operation not   supported [Operation not supported]
E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick] 
0-glusterd: Unable to add bricks
E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 
0-management: Add-brick commit failed.

I was initially using ZFS and noted that ZFS on FreeBSD does not support 
xattr, so I reverted to using UFS as the storage type for the brick, and 
still encounter this behavior.

I also recompiled the port (again, GlusterFS v3.11.1) with the patch 
from https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals 
specifically with xattr handling in FreeBSD.

To recap - I'm able to create any type of volume (2 or 3-way replicated 
or distributed), but I'm unable to add replicated bricks to a volume.

I was, however, able to add a second distributed brick ( gluster volume 
add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to 
adding and/or changing the replica count while adding a new brick.

Please let me know if there are any other issues in addition to bug 
#1452961 I should be aware of, or additional log or debug info I can 
provide.

Best Regards,
Mark

PS apologies if this made it to the list twice - I had this prior to 
acknowledging the confirmation email so I believe the first message was 
discarded.