<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi Sanju,</div><div><br></div><div>Here's what glusterd.log says on the new arbiter server when trying to add the node:</div><div><br></div><div>[2019-05-22 00:15:05.963059] I [run.c:242:runner_log] (-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x3b2cd) [0x7fe4ca9102cd] -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0xe6b85) [0x7fe4ca9bbb85] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fe4d5ecc955] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=gvol0 --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd<br>[2019-05-22 00:15:05.963177] I [MSGID: 106578] [glusterd-brick-ops.c:1355:glusterd_op_perform_add_bricks] 0-management: replica-count is set 3<br>[2019-05-22 00:15:05.963228] I [MSGID: 106578] [glusterd-brick-ops.c:1360:glusterd_op_perform_add_bricks] 0-management: arbiter-count is set 1<br>[2019-05-22 00:15:05.963257] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it<br>[2019-05-22 00:15:17.015268] E [MSGID: 106053] [glusterd-utils.c:13942:glusterd_handle_replicate_brick_ops] 0-management: Failed to set extended attribute trusted.add-brick : Transport endpoint is not connected [Transport endpoint is not connected]<br>[2019-05-22 00:15:17.036479] E [MSGID: 106073] [glusterd-brick-ops.c:2595:glusterd_op_add_brick] 0-glusterd: Unable to add bricks<br>[2019-05-22 00:15:17.036595] E [MSGID: 106122] [glusterd-mgmt.c:299:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit failed.<br>[2019-05-22 00:15:17.036710] E [MSGID: 106122] [glusterd-mgmt-handler.c:594:glusterd_handle_commit_fn] 0-management: commit failed on operation Add brick<br></div><div><br></div><div>As before gvol0-add-brick-mount.log said:</div><div><br></div><div>[2019-05-22 00:15:17.005695] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22<br>[2019-05-22 00:15:17.005749] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0<br>[2019-05-22 00:15:17.010101] E [fuse-bridge.c:4336:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)<br>[2019-05-22 00:15:17.014217] W [fuse-bridge.c:897:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected)<br>[2019-05-22 00:15:17.015097] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)<br>[2019-05-22 00:15:17.015158] W [fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 3: SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) resolution failed<br>[2019-05-22 00:15:17.035636] I [fuse-bridge.c:5144:fuse_thread_proc] 0-fuse: initating unmount of /tmp/mntYGNbj9<br>[2019-05-22 00:15:17.035854] W [glusterfsd.c:1500:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f7745ccedd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55c81b63de75] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55c81b63dceb] ) 0-: received signum (15), shutting down<br>[2019-05-22 00:15:17.035942] I [fuse-bridge.c:5914:fini] 0-fuse: Unmounting '/tmp/mntYGNbj9'.<br>[2019-05-22 00:15:17.035966] I [fuse-bridge.c:5919:fini] 0-fuse: Closing fuse connection to '/tmp/mntYGNbj9'.<br></div><div><br></div><div>Here are the processes running on the new arbiter server:</div><div># ps -ef | grep gluster<br>root 3466 1 0 20:13 ? 00:00:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/24c12b09f93eec8e.socket --xlator-option *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412 --process-name glustershd<br>root 6832 1 0 May16 ? 00:02:10 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO<br>root 17841 1 0 May16 ? 00:00:58 /usr/sbin/glusterfs --process-name fuse --volfile-server=gfs1 --volfile-id=/gvol0 /mnt/glusterfs<br></div><div><br></div><div>Here are the files created on the new arbiter server:</div><div># find /nodirectwritedata/gluster/gvol0 | xargs ls -ald<br>drwxr-xr-x 3 root root 4096 May 21 20:15 /nodirectwritedata/gluster/gvol0<br>drw------- 2 root root 4096 May 21 20:15 /nodirectwritedata/gluster/gvol0/.glusterfs<br></div><div><br></div><div>Thank you for your help!</div><div><br></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 21 May 2019 at 00:10, Sanju Rakonde <<a href="mailto:srakonde@redhat.com">srakonde@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">David,<div><br></div><div>can you please attach glusterd.logs? As the error message says, Commit failed on the arbitar node, we might be able to find some issue on that node.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 20, 2019 at 10:10 AM Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 17 May 2019 at 06:01, David Cunningham <<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hello,</div><div><br></div><div>We're adding an arbiter node to an existing volume and having an issue. Can anyone help? The root cause error appears to be "00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)", as below.</div><div><br></div><div>We are running glusterfs 5.6.1. Thanks in advance for any assistance!<br></div><div><br></div><div>On existing node gfs1, trying to add new arbiter node gfs3:</div><div><br></div><div># gluster volume add-brick gvol0 replica 3 arbiter 1 gfs3:/nodirectwritedata/gluster/gvol0<br>volume add-brick: failed: Commit failed on gfs3. Please check log file for details.<br></div></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>This looks like a glusterd issue. Please check the glusterd logs for more info.</div><div>Adding the glusterd dev to this thread. Sanju, can you take a look?</div><div> </div><div>Regards,</div><div>Nithya</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>On new node gfs3 in gvol0-add-brick-mount.log:</div><div><br></div><div>[2019-05-17 01:20:22.689721] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22<br>[2019-05-17 01:20:22.689778] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0<br>[2019-05-17 01:20:22.694897] E [fuse-bridge.c:4336:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)<br>[2019-05-17 01:20:22.699770] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)<br>[2019-05-17 01:20:22.699834] W [fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 2: SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) resolution failed<br>[2019-05-17 01:20:22.715656] I [fuse-bridge.c:5144:fuse_thread_proc] 0-fuse: initating unmount of /tmp/mntQAtu3f<br>[2019-05-17 01:20:22.715865] W [glusterfsd.c:1500:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fb223bf6dd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560886581e75] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560886581ceb] ) 0-: received signum (15), shutting down<br>[2019-05-17 01:20:22.715926] I [fuse-bridge.c:5914:fini] 0-fuse: Unmounting '/tmp/mntQAtu3f'.<br>[2019-05-17 01:20:22.715953] I [fuse-bridge.c:5919:fini] 0-fuse: Closing fuse connection to '/tmp/mntQAtu3f'.<br></div><div><br></div><div>Processes running on new node gfs3:</div><div><br></div><div># ps -ef | grep gluster<br>root 6832 1 0 20:17 ? 00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO<br>root 15799 1 0 20:17 ? 00:00:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/24c12b09f93eec8e.socket --xlator-option *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412 --process-name glustershd<br>root 16856 16735 0 21:21 pts/0 00:00:00 grep --color=auto gluster<br><br></div>-- <br><div dir="ltr" class="gmail-m_750124261411719583gmail-m_4549560710938974858gmail-m_-5557460900297667963gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David Cunningham, Voisonics Limited<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_750124261411719583gmail_signature"><div dir="ltr"><div>Thanks,<br></div>Sanju<br></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David Cunningham, Voisonics Limited<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div>