<font size=3 face="Arial">Sorry for the slow response. Your hunch was
right. It seems to be a problem between tiering and sharding.</font>
<br>
<br><font size=3 face="Arial">I untiered the volume and the symptom vanished.
I then deleted and recreated the volume entirely (without tiering) in order
to cleanup the orphaned shards.</font>
<br>
<br><font size=3 face="Arial">-Walter Deignan<br>
-Uline IT, Systems Architect</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Nithya Balachandran
<nbalacha@redhat.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">Walter Deignan <WDeignan@uline.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">gluster-users <gluster-users@gluster.org></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">05/17/2017 10:17 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [Gluster-users]
Deleting large files on sharded volume hangs and doesn't delete shards</font>
<br>
<hr noshade>
<br>
<br>
<br><font size=3>I don't think we have tested shards with a tiered volume.
Do you see such issues on non-tiered sharded volumes?</font>
<br>
<br><font size=3>Regards,</font>
<br><font size=3>Nithya</font>
<br>
<br><font size=3>On 18 May 2017 at 00:51, Walter Deignan <</font><a href=mailto:WDeignan@uline.com target=_blank><font size=3 color=blue><u>WDeignan@uline.com</u></font></a><font size=3>>
wrote:</font>
<br><font size=3 face="Arial">I have a reproducible issue where attempting
to delete a file large enough to have been sharded hangs. I can't kill
the 'rm' command and eventually am forced to reboot the client (which in
this case is also part of the gluster cluster). After the node finishes
rebooting I can see that while the file front-end is gone, the back-end
shards are still present.</font><font size=3> <br>
</font><font size=3 face="Arial"><br>
Is this a known issue? Any way to get around it?</font><font size=3> <br>
</font><font size=3 face="Arial"><br>
----------------------------------------------</font><font size=3> <br>
</font><font size=2 face="Courier New"><br>
[root@dc-vihi19 ~]# gluster volume info gv0</font><font size=3> <br>
</font><font size=2 face="Courier New"><br>
Volume Name: gv0</font><font size=3> </font><font size=2 face="Courier New"><br>
Type: Tier</font><font size=3> </font><font size=2 face="Courier New"><br>
Volume ID: d42e366f-381d-4787-bcc5-cb6770cb7d58</font><font size=3> </font><font size=2 face="Courier New"><br>
Status: Started</font><font size=3> </font><font size=2 face="Courier New"><br>
Snapshot Count: 0</font><font size=3> </font><font size=2 face="Courier New"><br>
Number of Bricks: 24</font><font size=3> </font><font size=2 face="Courier New"><br>
Transport-type: tcp</font><font size=3> </font><font size=2 face="Courier New"><br>
Hot Tier :</font><font size=3> </font><font size=2 face="Courier New"><br>
Hot Tier Type : Distributed-Replicate</font><font size=3> </font><font size=2 face="Courier New"><br>
Number of Bricks: 4 x 2 = 8</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick1: dc-vihi71:/gluster/bricks/brick4/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick2: dc-vihi19:/gluster/bricks/brick4/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick3: dc-vihi70:/gluster/bricks/brick4/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick4: dc-vihi19:/gluster/bricks/brick3/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick5: dc-vihi71:/gluster/bricks/brick3/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick6: dc-vihi19:/gluster/bricks/brick2/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick7: dc-vihi70:/gluster/bricks/brick3/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick8: dc-vihi19:/gluster/bricks/brick1/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Cold Tier:</font><font size=3> </font><font size=2 face="Courier New"><br>
Cold Tier Type : Distributed-Replicate</font><font size=3> </font><font size=2 face="Courier New"><br>
Number of Bricks: 8 x 2 = 16</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick9: dc-vihi19:/gluster/bricks/brick5/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick10: dc-vihi70:/gluster/bricks/brick1/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick11: dc-vihi19:/gluster/bricks/brick6/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick12: dc-vihi71:/gluster/bricks/brick1/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick13: dc-vihi19:/gluster/bricks/brick7/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick14: dc-vihi70:/gluster/bricks/brick2/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick15: dc-vihi19:/gluster/bricks/brick8/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick16: dc-vihi71:/gluster/bricks/brick2/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick17: dc-vihi19:/gluster/bricks/brick9/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick18: dc-vihi70:/gluster/bricks/brick5/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick19: dc-vihi19:/gluster/bricks/brick10/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick20: dc-vihi71:/gluster/bricks/brick5/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick21: dc-vihi19:/gluster/bricks/brick11/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick22: dc-vihi70:/gluster/bricks/brick6/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick23: dc-vihi19:/gluster/bricks/brick12/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Brick24: dc-vihi71:/gluster/bricks/brick6/data</font><font size=3> </font><font size=2 face="Courier New"><br>
Options Reconfigured:</font><font size=3> </font><font size=2 face="Courier New"><br>
nfs.disable: on</font><font size=3> </font><font size=2 face="Courier New"><br>
transport.address-family: inet</font><font size=3> </font><font size=2 face="Courier New"><br>
features.ctr-enabled: on</font><font size=3> </font><font size=2 face="Courier New"><br>
cluster.tier-mode: cache</font><font size=3> </font><font size=2 face="Courier New"><br>
features.shard: on</font><font size=3> </font><font size=2 face="Courier New"><br>
features.shard-block-size: 512MB</font><font size=3> </font><font size=2 face="Courier New"><br>
network.ping-timeout: 5</font><font size=3> </font><font size=2 face="Courier New"><br>
cluster.server-quorum-ratio: 51%</font><font size=3> <br>
</font><font size=2 face="Courier New"><br>
[root@dc-vihi19 temp]# ls -lh</font><font size=3> </font><font size=2 face="Courier New"><br>
total 26G</font><font size=3> </font><font size=2 face="Courier New"><br>
-rw-rw-rw-. 1 root root 31G May 17 10:38 win7.qcow2</font><font size=3>
</font><font size=2 face="Courier New"><br>
[root@dc-vihi19 temp]# getfattr -n glusterfs.gfid.string win7.qcow2</font><font size=3>
</font><font size=2 face="Courier New"><br>
# file: win7.qcow2</font><font size=3> </font><font size=2 face="Courier New"><br>
glusterfs.gfid.string="7f4a0fea-72c0-41e4-97a5-6297be0a9142"</font><font size=3>
<br>
</font><font size=2 face="Courier New"><br>
[root@dc-vihi19 temp]# rm win7.qcow2</font><font size=3> </font><font size=2 face="Courier New"><br>
rm: remove regular file âwin7.qcow2â? y</font><font size=3> <br>
</font><font size=2 face="Courier New"><br>
*Process hangs and can't be killed. A reboot later...*</font><font size=3>
<br>
</font><font size=2 face="Courier New"><br>
login as: root</font><font size=3> </font><font size=2 face="Courier New"><br>
Authenticating with public key "rsa-key-20170510"</font><font size=3>
</font><font size=2 face="Courier New"><br>
Last login: Wed May 17 14:04:29 2017 from ******</font><font size=3> </font><font size=2 face="Courier New"><br>
[root@dc-vihi19 ~]# find /gluster/bricks -name "7f4a0fea-72c0-41e4-97a5-6297be0a9142*"</font><font size=3>
</font><font size=2 face="Courier New"><br>
/gluster/bricks/brick1/data/.shard/7f4a0fea-72c0-41e4-97a5-6297be0a9142.23</font><font size=3>
</font><font size=2 face="Courier New"><br>
/gluster/bricks/brick1/data/.shard/7f4a0fea-72c0-41e4-97a5-6297be0a9142.35</font><font size=3>
</font><font size=2 face="Courier New"><br>
/gluster/bricks/brick2/data/.shard/7f4a0fea-72c0-41e4-97a5-6297be0a9142.52</font><font size=3>
</font><font size=2 face="Courier New"><br>
/gluster/bricks/brick2/data/.shard/7f4a0fea-72c0-41e4-97a5-6297be0a9142.29</font><font size=3>
</font><font size=2 face="Courier New"><br>
/gluster/bricks/brick2/data/.shard/7f4a0fea-72c0-41e4-97a5-6297be0a9142.22</font><font size=3>
</font><font size=2 face="Courier New"><br>
/gluster/bricks/brick2/data/.shard/7f4a0fea-72c0-41e4-97a5-6297be0a9142.24</font><font size=3>
<br>
</font><font size=2 face="Courier New"><br>
and so on...</font><font size=3> </font><font size=3 color=#8f8f8f><br>
<br>
</font><font size=3 color=#8f8f8f face="Arial"><br>
-Walter Deignan<br>
-Uline IT, Systems Architect</font><font size=3><br>
_______________________________________________<br>
Gluster-users mailing list</font><font size=3 color=blue><u><br>
</u></font><a href="mailto:Gluster-users@gluster.org"><font size=3 color=blue><u>Gluster-users@gluster.org</u></font></a><font size=3 color=blue><u><br>
</u></font><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" target=_blank><font size=3 color=blue><u>http://lists.gluster.org/mailman/listinfo/gluster-users</u></font></a>
<br>
<br>
<br>