[Gluster-users] Broken after 3.7.8 upgrade from 3.7.6

[Alan Millar]

I updated from 3.7.6 to 3.7.8 a few days ago, and now it looks like a number of things are broken including healing.  

This is a cluster of 3 servers.  One server is Ubuntu 14.04 using the PPA repo, and the other two are Proxmox 4 using the Debian Jessie repo.

"heal info" and "heal statistics" do not show any healing activity; everything shows as zero.  But I have broken files that are not getting healed.

Doing "heal", "heal full", and "heal enable" all say success.  But none seem to fix anything.

I have tried with entry-self-heal/metdata-self-heal/data-self-heal set both on and off; neither seems to make a difference.
I replaced a brick on a replicated volume.  Some of the files are just not being replaced/updated on the second brick.  Others have a few blocks written on the second brick but are not complete.

I don't know what to look for in the logs, but I do see a lot of messages in glustershd.log like this:

[2016-02-29 23:13:27.001474] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-vmdisk2-replicate-0: unable to get index-dir on vmdisk2-client-1
[2016-02-29 23:13:27.001524] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to get index-dir on public-client-3
[2016-02-29 23:13:27.001547] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-users-replicate-0: unable to get index-dir on users-client-6
[2016-02-29 23:13:27.001876] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-vmdisk1-replicate-0: unable to get index-dir on vmdisk1-client-2
[2016-02-29 23:13:35.001555] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-backups-local-replicate-0: unable to get index-dir on backups-local-client-2

On at least one replicated/distributed volume, I see duplicate directory entries (one with the actual file, and one zero-length placeholder)

-rw-rwSrw- 1 root 1004 255744366 Oct 18  2013 S03E05 - The One with Frank Jr.mp4
---------T 1 root 1004         0 Feb 22 08:55 S03E05 - The One with Frank Jr.mp4
-rw-rwSrw- 1 root 1004 255705796 Oct 18  2013 S03E06 - The One with the Flashback.mp4
---------T 1 root 1004         0 Feb 22 08:55 S03E06 - The One with the Flashback.mp4

This is *through the FUSE mount*, not looking directly at the bricks.

Anyone have any ideas on what I should look at?  Thanks

- Alan


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160229/d4773a94/attachment.html>