[Gluster-devel] difficult bug in 2.5 mainline

[Harris Landgarten]

I am trying to track down a bug that is causing hangs in 2.5-patch-249 and all previous.

This happens during a full Zimbra backup of certain accounts to /mnt/glusterfs/backups

The first stage of the backup copies indexes and primary storage to /mnt/glusterfs/backups/tmp
All of this data resides in local storage and the writing to gluster is successful.

The next stage copies secondary storage to /mnt/glusterfs/backups/tmp
This fails in the following way:

Brick1 hangs with no errors
Brick2 hangs with no errors
Zimbra client hangs with no errors
second client loses connectivity

The second client bails after 2 min but cannot connect
The Zimbra client never bails

I then restart the bricks

After both bricks are restarted, the second client reconnects and a hung df -h completes

Zimbra client stays in a hung unconnected start

ls -l /mnt/glusterfs hangs

Only way is reset is

kill -9 pidof glusterfs
umount /mnt/glusterfs

glusterfs

Post mortem examination of /mnt/glusterfs/backups/tmp shows that a few files have the written from the secondary storage volume. I this can over 15,000 files should have been written.

Note: this only happen with large email boxed with some large >10M files.

Note: with patch-247 the zimbra client would seqfault. With 249 it just hangs in unrecoverable state.


Harris