<div dir="ltr">Seems like a frame-loss. Could you collect statedump of the mount process? You may have to use kill -USR1 method described in the docs below when the process hangs. Please also get statedump of brick processes.<br><br><a href="https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/">https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/</a><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 18, 2017 at 12:51 AM, Walter Deignan <span dir="ltr"><<a href="mailto:WDeignan@uline.com" target="_blank">WDeignan@uline.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="3" face="Arial">I have a reproducible issue where attempting
to delete a file large enough to have been sharded hangs. I can't kill
the 'rm' command and eventually am forced to reboot the client (which in
this case is also part of the gluster cluster). After the node finishes
rebooting I can see that while the file front-end is gone, the back-end
shards are still present.</font>
<br>
<br><font size="3" face="Arial">Is this a known issue? Any way to get around
it?</font>
<br>
<br><font size="3" face="Arial">------------------------------<wbr>----------------</font>
<br>
<br><font size="2" face="Courier New">[root@dc-vihi19 ~]# gluster volume
info gv0</font>
<br>
<br><font size="2" face="Courier New">Volume Name: gv0</font>
<br><font size="2" face="Courier New">Type: Tier</font>
<br><font size="2" face="Courier New">Volume ID: d42e366f-381d-4787-bcc5-<wbr>cb6770cb7d58</font>
<br><font size="2" face="Courier New">Status: Started</font>
<br><font size="2" face="Courier New">Snapshot Count: 0</font>
<br><font size="2" face="Courier New">Number of Bricks: 24</font>
<br><font size="2" face="Courier New">Transport-type: tcp</font>
<br><font size="2" face="Courier New">Hot Tier :</font>
<br><font size="2" face="Courier New">Hot Tier Type : Distributed-Replicate</font>
<br><font size="2" face="Courier New">Number of Bricks: 4 x 2 = 8</font>
<br><font size="2" face="Courier New">Brick1: dc-vihi71:/gluster/bricks/<wbr>brick4/data</font>
<br><font size="2" face="Courier New">Brick2: dc-vihi19:/gluster/bricks/<wbr>brick4/data</font>
<br><font size="2" face="Courier New">Brick3: dc-vihi70:/gluster/bricks/<wbr>brick4/data</font>
<br><font size="2" face="Courier New">Brick4: dc-vihi19:/gluster/bricks/<wbr>brick3/data</font>
<br><font size="2" face="Courier New">Brick5: dc-vihi71:/gluster/bricks/<wbr>brick3/data</font>
<br><font size="2" face="Courier New">Brick6: dc-vihi19:/gluster/bricks/<wbr>brick2/data</font>
<br><font size="2" face="Courier New">Brick7: dc-vihi70:/gluster/bricks/<wbr>brick3/data</font>
<br><font size="2" face="Courier New">Brick8: dc-vihi19:/gluster/bricks/<wbr>brick1/data</font>
<br><font size="2" face="Courier New">Cold Tier:</font>
<br><font size="2" face="Courier New">Cold Tier Type : Distributed-Replicate</font>
<br><font size="2" face="Courier New">Number of Bricks: 8 x 2 = 16</font>
<br><font size="2" face="Courier New">Brick9: dc-vihi19:/gluster/bricks/<wbr>brick5/data</font>
<br><font size="2" face="Courier New">Brick10: dc-vihi70:/gluster/bricks/<wbr>brick1/data</font>
<br><font size="2" face="Courier New">Brick11: dc-vihi19:/gluster/bricks/<wbr>brick6/data</font>
<br><font size="2" face="Courier New">Brick12: dc-vihi71:/gluster/bricks/<wbr>brick1/data</font>
<br><font size="2" face="Courier New">Brick13: dc-vihi19:/gluster/bricks/<wbr>brick7/data</font>
<br><font size="2" face="Courier New">Brick14: dc-vihi70:/gluster/bricks/<wbr>brick2/data</font>
<br><font size="2" face="Courier New">Brick15: dc-vihi19:/gluster/bricks/<wbr>brick8/data</font>
<br><font size="2" face="Courier New">Brick16: dc-vihi71:/gluster/bricks/<wbr>brick2/data</font>
<br><font size="2" face="Courier New">Brick17: dc-vihi19:/gluster/bricks/<wbr>brick9/data</font>
<br><font size="2" face="Courier New">Brick18: dc-vihi70:/gluster/bricks/<wbr>brick5/data</font>
<br><font size="2" face="Courier New">Brick19: dc-vihi19:/gluster/bricks/<wbr>brick10/data</font>
<br><font size="2" face="Courier New">Brick20: dc-vihi71:/gluster/bricks/<wbr>brick5/data</font>
<br><font size="2" face="Courier New">Brick21: dc-vihi19:/gluster/bricks/<wbr>brick11/data</font>
<br><font size="2" face="Courier New">Brick22: dc-vihi70:/gluster/bricks/<wbr>brick6/data</font>
<br><font size="2" face="Courier New">Brick23: dc-vihi19:/gluster/bricks/<wbr>brick12/data</font>
<br><font size="2" face="Courier New">Brick24: dc-vihi71:/gluster/bricks/<wbr>brick6/data</font>
<br><font size="2" face="Courier New">Options Reconfigured:</font>
<br><font size="2" face="Courier New">nfs.disable: on</font>
<br><font size="2" face="Courier New">transport.address-family: inet</font>
<br><font size="2" face="Courier New">features.ctr-enabled: on</font>
<br><font size="2" face="Courier New">cluster.tier-mode: cache</font>
<br><font size="2" face="Courier New">features.shard: on</font>
<br><font size="2" face="Courier New">features.shard-block-size: 512MB</font>
<br><font size="2" face="Courier New">network.ping-timeout: 5</font>
<br><font size="2" face="Courier New">cluster.server-quorum-ratio: 51%</font>
<br>
<br><font size="2" face="Courier New">[root@dc-vihi19 temp]# ls -lh</font>
<br><font size="2" face="Courier New">total 26G</font>
<br><font size="2" face="Courier New">-rw-rw-rw-. 1 root root 31G May 17
10:38 win7.qcow2</font>
<br><font size="2" face="Courier New">[root@dc-vihi19 temp]# getfattr -n
glusterfs.gfid.string win7.qcow2</font>
<br><font size="2" face="Courier New"># file: win7.qcow2</font>
<br><font size="2" face="Courier New">glusterfs.gfid.string="<wbr>7f4a0fea-72c0-41e4-97a5-<wbr>6297be0a9142"</font>
<br>
<br><font size="2" face="Courier New">[root@dc-vihi19 temp]# rm win7.qcow2</font>
<br><font size="2" face="Courier New">rm: remove regular file âwin7.qcow2â?
y</font>
<br>
<br><font size="2" face="Courier New">*Process hangs and can't be killed.
A reboot later...*</font>
<br>
<br><font size="2" face="Courier New">login as: root</font>
<br><font size="2" face="Courier New">Authenticating with public key "rsa-key-20170510"</font>
<br><font size="2" face="Courier New">Last login: Wed May 17 14:04:29 2017
from ******</font>
<br><font size="2" face="Courier New">[root@dc-vihi19 ~]# find /gluster/bricks
-name "7f4a0fea-72c0-41e4-97a5-<wbr>6297be0a9142*"</font>
<br><font size="2" face="Courier New">/gluster/bricks/brick1/data/.<wbr>shard/7f4a0fea-72c0-41e4-97a5-<wbr>6297be0a9142.23</font>
<br><font size="2" face="Courier New">/gluster/bricks/brick1/data/.<wbr>shard/7f4a0fea-72c0-41e4-97a5-<wbr>6297be0a9142.35</font>
<br><font size="2" face="Courier New">/gluster/bricks/brick2/data/.<wbr>shard/7f4a0fea-72c0-41e4-97a5-<wbr>6297be0a9142.52</font>
<br><font size="2" face="Courier New">/gluster/bricks/brick2/data/.<wbr>shard/7f4a0fea-72c0-41e4-97a5-<wbr>6297be0a9142.29</font>
<br><font size="2" face="Courier New">/gluster/bricks/brick2/data/.<wbr>shard/7f4a0fea-72c0-41e4-97a5-<wbr>6297be0a9142.22</font>
<br><font size="2" face="Courier New">/gluster/bricks/brick2/data/.<wbr>shard/7f4a0fea-72c0-41e4-97a5-<wbr>6297be0a9142.24</font>
<br>
<br><font size="2" face="Courier New">and so on...</font>
<br><span class="HOEnZb"><font color="#888888">
<br>
<br><font size="3" face="Arial">-Walter Deignan<br>
-Uline IT, Systems Architect</font></font></span><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>