<div dir="ltr">Hi,<br><br><div class="gmail_quote"><div dir="ltr">On Sat, Aug 4, 2018 at 3:19 AM Fox <<a href="mailto:foxxz.net@gmail.com">foxxz.net@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Replying to the last batch of questions I've received...</div><div><br></div><div>To reiterate, I am only having problems writing files to disperse volumes when mounting it on an armhf system. Mounting the same volume on an x86-64 system works fine.</div><div>Disperse volumes running on arm can not heal.</div><div><br></div><div>Replica volumes mount and heal just fine.</div><div><br></div><div><br></div><div>All bricks are up and running. I have ensured connectivity and that MTU is correct and identical.</div><div><br></div><div>Armhf is 32bit:</div><div># uname -a<br>Linux gluster01 4.14.55-146 #1 SMP PREEMPT Wed Jul 11 22:31:01 -03 2018 armv7l armv7l armv7l GNU/Linux</div><div># file /bin/bash<br>/bin/bash: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=e0a53f804173b0cd9845bb8a76fee1a1e98a9759, stripped<br># lsb_release -a<br>No LSB modules are available.<br>Distributor ID: Ubuntu<br>Description: Ubuntu 18.04.1 LTS<br>Release: 18.04<br>Codename: bionic<br></div><div># free<br> total used free shared buff/cache available<br>Mem: 2042428 83540 1671004 6052 287884 1895684<br>Swap: 0 0 0<br></div><div><br></div><div><br></div><div>8 cores total. 4x running 2ghz and 4x running 1.4ghz<br></div><div>processor : 0<br>model name : ARMv7 Processor rev 3 (v7l)<br>BogoMIPS : 24.00<br>Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae<br>CPU implementer : 0x41<br>CPU architecture: 7<br>CPU variant : 0x0<br>CPU part : 0xc07<br>CPU revision : 3<br></div><div><br></div><div>processor : 4<br>model name : ARMv7 Processor rev 3 (v7l)<br>BogoMIPS : 72.00<br>Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae<br>CPU implementer : 0x41<br>CPU architecture: 7<br>CPU variant : 0x2<br>CPU part : 0xc0f<br>CPU revision : 3<br><br></div><div><br></div><div><br></div><div>There IS a 98MB /core file from the fuse mount so thats cool.</div><div># file /core<br>/core: ELF 32-bit LSB core file ARM, version 1 (SYSV), SVR4-style, from '/usr/sbin/glusterfs --process-name fuse --volfile-server=gluster01 --volfile-id', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/sbin/glusterfs', platform: 'v7l'<br></div></div></blockquote><div><br></div><div>On possible cause is some 64/32 bits inconsistency. If you have also installed the debug symbols and can provide a backtrace from the core dump, it would help to identify the problem.</div><div><br></div><div>Xavi</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div></div><div><br></div><div>I will try and get a bug report with logs filed over the weekend.<br></div><div></div><div><br></div><div>This is just an experimental home cluster. I don't have anything on it yet. Its possible I could grant someone SSH access to the cluster if it helps further the gluster project. But the results should be reproducible on something like a raspberry pi. I was hoping to run a dispersed volume on it eventually otherwise I would have never found this issue.</div><div></div><div><br></div><div>Thank you for the troubleshooting ideas.</div><div><br></div><div><br></div><div>-Fox<br></div><div><br></div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 3, 2018 at 3:33 AM, Milind Changire <span dir="ltr"><<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>What is the endianness of the armhf CPU ?</div><div>Are you running a 32bit or 64bit Operating System ?</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-4426449895620067531gmail-m_-9215279847284165033h5">On Fri, Aug 3, 2018 at 9:51 AM, Fox <span dir="ltr"><<a href="mailto:foxxz.net@gmail.com" target="_blank">foxxz.net@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-4426449895620067531gmail-m_-9215279847284165033h5"><div dir="ltr"><div>Just wondering if anyone else is running into the same behavior with disperse volumes described below and what I might be able to do about it.</div><div><br></div><div>I am using ubuntu 18.04LTS on Odroid HC-2 hardware (armhf) and have installed gluster 4.1.2 via PPA. I have 12 member nodes each with a single brick. I can successfully create a working volume via the command:</div><div><br></div><div>gluster volume create testvol1 disperse 12 redundancy 4 gluster01:/exports/sda/brick1/testvol1 gluster02:/exports/sda/brick1/testvol1 gluster03:/exports/sda/brick1/testvol1 gluster04:/exports/sda/brick1/testvol1 gluster05:/exports/sda/brick1/testvol1 gluster06:/exports/sda/brick1/testvol1 gluster07:/exports/sda/brick1/testvol1 gluster08:/exports/sda/brick1/testvol1 gluster09:/exports/sda/brick1/testvol1 gluster10:/exports/sda/brick1/testvol1 gluster11:/exports/sda/brick1/testvol1 gluster12:/exports/sda/brick1/testvol1</div><div><br></div><div>And start the volume:</div><div></div><div>gluster volume start testvol1<br></div><div><br></div><div>Mounting the volume on an x86-64 system it performs as expected.</div><div><br></div><div>Mounting the same volume on an armhf system (such as one of the cluster members) I can create directories but trying to create a file I get an error and the file system unmounts/crashes:</div><div>root@gluster01:~# mount -t glusterfs gluster01:/testvol1 /mnt<br>root@gluster01:~# cd /mnt<br>root@gluster01:/mnt# ls<br>root@gluster01:/mnt# mkdir test<br>root@gluster01:/mnt# cd test<br></div><div>root@gluster01:/mnt/test# cp /root/notes.txt ./<br>cp: failed to close './notes.txt': Software caused connection abort<br>root@gluster01:/mnt/test# ls<br>ls: cannot open directory '.': Transport endpoint is not connected</div><div><br></div><div>I get many of these in the glusterfsd.log:<br></div><div>The message "W [MSGID: 101088] [common-utils.c:4316:gf_backtrace_save] 0-management: Failed to save the backtrace." repeated 100 times between [2018-08-03 04:06:39.904166] and [2018-08-03 04:06:57.521895]<br></div><div><br></div><div><br></div><div>Furthermore, if a cluster member ducks out (reboots, loses connection, etc) and needs healing the self heal daemon logs messages similar to that above and can not heal - no disk activity (verified via iotop) though very high CPU usage and the volume heal info command indicates the volume needs healing.</div><div><br></div><div><br></div><div>I tested all of the above in virtual environments using x86-64 VMs and could self heal as expected.</div><div><br></div><div>Again this only happens when using disperse volumes. Should I be filing a bug report instead?<br></div></div>
<br></div></div><span>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></span></blockquote></div><span class="m_-4426449895620067531gmail-m_-9215279847284165033HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-4426449895620067531gmail-m_-9215279847284165033m_-3502651239667004525gmail_signature"><div dir="ltr"><div><div dir="ltr">Milind<br><br></div></div></div></div>
</font></span></div>
</blockquote></div><br></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div>