[Gluster-users] Problem adding replicated bricks on FreeBSD

[Mark Staudinger]

Kaushal M wrote:
> On Thu, Apr 26, 2018 at 9:06 PM Mark Staudinger <mark.staudinger at nyi.net>
> wrote:
>
>> Hi Folks,
>> I'm trying to debug an issue that I've found while attempting to qualify
>> GlusterFS for potential distributed storage projects on the FreeBSD-11.1
>> server platform - using the existing package of GlusterFS v3.11.1_4
>> The main issue I've encountered is that I cannot add new bricks while
>> setting/increasing the replica count.
>> If I create a replicated volume "poc" on two hosts, say s1:/gluster/1/poc
>> and s2:/gluster/1/poc, the volume is created properly and shows replicated
>> status, files are written to both volumes.
>> If I create a single volume: s1:/gluster/1/poc as a single / distributed
>> brick, and then try to run
>> gluster volume add-brick poc replica 2 s2:/gluster/1/poc
>> it will always fail (sometimes after a pause, sometimes not.)  The only
>> error I'm seeing on the server hosting the new brick, aside from the
>> generic "Unable to add bricks" message, is like so:
>> I [MSGID: 106578]
>> [glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 0-management:
>> replica-count is set 2
>> I [MSGID: 106578]
>> [glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management:
>> type is set 2, need to change it
>> E [MSGID: 106054]
>> [glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 0-management:
>> Failed to set extended attribute trusted.add-brick : Operation not
>> supported [Operation not supported]
> The log here seems to indicate the filesystem on the new brick being added
> doesn't seem to support setting xattrs.
> Maybe check the new brick again?
>
>> E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
>> 0-glusterd: Unable to add bricks
>> E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 0-management:
>> Add-brick commit failed.
>> I was initially using ZFS and noted that ZFS on FreeBSD does not support
>> xattr, so I reverted to using UFS as the storage type for the brick, and
>> still encounter this behavior.
>> I also recompiled the port (again, GlusterFS v3.11.1) with the patch from
>> https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
>> specifically with xattr handling in FreeBSD.
>> To recap - I'm able to create any type of volume (2 or 3-way replicated or
>> distributed), but I'm unable to add replicated bricks to a volume.
>> I was, however, able to add a second distributed brick ( gluster volume
>> add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to adding
>> and/or changing the replica count while adding a new brick.
>> Please let me know if there are any other issues in addition to bug
>> #1452961 I should be aware of, or additional log or debug info I can
>> provide.
>> Best Regards,
>> Mark
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
Hi Kaushal,

That's what the error message would indicate, but UFS2 does support 
xattr so that's a bit unexpected. Since I'm able to add a distributed 
brick, I can rule out basic setup issues.

All the same, I can see via 'truss'

extattr_set_link(0x8081ff890,0x1,0x803b27886,0x805258c40,0x400) ERR#45 
'Operation not supported’

Is anyone else using a similar setup (FreeBSD >= 11.1-RELEASE with the 
official pkg) that has tested adding bricks in this way? Are there any 
changes to the newfs or mount settings required for this to work that 
aren't obvious?

Cheers,
-=Mark