[Gluster-users] Problem adding replicated bricks on FreeBSD

[Mark Staudinger]

Mark Staudinger wrote:
> Hi Folks,
>
> I'm trying to debug an issue that I've found while attempting to 
> qualify GlusterFS for potential distributed storage projects on the 
> FreeBSD-11.1 server platform - using the existing package of GlusterFS 
> v3.11.1_4
>
> The main issue I've encountered is that I cannot add new bricks while 
> setting/increasing the replica count.
>
> If I create a replicated volume "poc" on two hosts, say 
> s1:/gluster/1/poc and s2:/gluster/1/poc, the volume is created 
> properly and shows replicated status, files are written to both volumes.
>
> If I create a single volume: s1:/gluster/1/poc as a single / 
> distributed brick, and then try to run
>
> gluster volume add-brick poc replica 2 s2:/gluster/1/poc
>
> it will always fail (sometimes after a pause, sometimes not.)  The 
> only error I'm seeing on the server hosting the new brick, aside from 
> the generic "Unable to add bricks" message, is like so:
>
> I [MSGID: 106578] 
> [glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 
> 0-management: replica-count is set 2
> I [MSGID: 106578] 
> [glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 
> 0-management: type is set 2, need to change it
> E [MSGID: 106054] 
> [glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 
> 0-management: Failed to set extended attribute trusted.add-brick : 
> Operation not supported [Operation not supported]
> E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick] 
> 0-glusterd: Unable to add bricks
> E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 
> 0-management: Add-brick commit failed.
>
> I was initially using ZFS and noted that ZFS on FreeBSD does not 
> support xattr, so I reverted to using UFS as the storage type for the 
> brick, and still encounter this behavior.
>
> I also recompiled the port (again, GlusterFS v3.11.1) with the patch 
> from https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals 
> specifically with xattr handling in FreeBSD.
>
> To recap - I'm able to create any type of volume (2 or 3-way 
> replicated or distributed), but I'm unable to add replicated bricks to 
> a volume.
>
> I was, however, able to add a second distributed brick ( gluster 
> volume add-brick poc s2:/gluster/1/poc ) - so the issue seems specific 
> to adding and/or changing the replica count while adding a new brick.
>
> Please let me know if there are any other issues in addition to bug 
> #1452961 I should be aware of, or additional log or debug info I can 
> provide.
>
>
Hi Folks,

As a follow-up to this, and it not making sense that xattrs would not be 
supported on UFS2, I am attaching the 'truss' output of the extattr 
syscalls from two operations.  In both situations, the command was 
initiated on server1, and traced on server2.

The first is a working "create-brick" issued from server1, traced from 
server2: ( gluster volume create poc transport tcp replica 2 
s1:/gluster/1/poc s2:/gluster/1/poc )

extattr_set_link(0x807a55440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x807a55440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_set_link(0x807a55440,0x1,0x803b24c17,0x807dfe250,0x10) = 16 (0x10)


And the second is from the not-working situation where the brick from s2 
is being added to the volume ( gluster volume add-brick poc replica 2 
s2:/gluster/1/poc commit )

extattr_set_link(0x80545c440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x80545c440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_set_link(0x80545c440,0x1,0x803b24c17,0x8051e9c20,0x10) = 16 (0x10)
extattr_set_link(0x80545c440,0x1,0x803b2774c,0x807bff068,0xc) = 12 (0xc)
write(6," 31:     option afr-pending-xattr 
poc-client-0,poc-client-1\n",64) = 64 (0x40)
extattr_set_link(0x807bff890,0x1,0x803b27886,0x80545d440,0x400) ERR#45 
'Operation not supported'
write(6,"[2018-04-28 15:21:51.271551] E [MSGID: 106054] 
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 
0-management: Failed to set extended attribute trusted.add-brick : 
Operation not supported [Operation not supported]\n",225) = 225 (0xe1)

Cheers,
-=Mark