[Gluster-users] hung disk sleep process

[Computerisms Corporation]

Hi,

I have a problem, but am not really sure the question I need to ask.  So 
going to lay it all down and maybe someone can point me in the right 
direction...

I have a replicated gluster volume across two servers.  Each server has 
its OS installed on an SSD, and a RAID array is mounted on each server 
as a brick.  Both servers run a Samba AD, among other things, and an LXC 
container for a member server.  Each container's samba installation is 
configured to use CTDB, so both containers act as a clustered file 
system.  Both LXC containers at startup mount the local directory where 
the glusterfs is mounted on their respective host servers using 
lxc.mount.entry in their config files.

This works, mostly.  but after some hours, days or weeks, a problem will 
develop.  The initial symptom were reports by end users of either not 
being able to access a folder or file.  That led me to an error in the 
CTDB logs about locking a file that is in the container on the OS disk, 
so I thought it was a CTDB problem, but when I chased that with the help 
of Martin from the samba mailing list, I find under that an smbd process 
in the LXC container which has entered a disk sleep state.  This process 
causes the same thing to happen on the host.  I am unable to get any 
trace on the smbd process in the container or on the host, but under 
/proc/<pid>/stack I get something like the following each time:

[<ffffffffc047e856>] request_wait_answer+0x166/0x1f0 [fuse]
[<ffffffff8b0b8d50>] prepare_to_wait_event+0xf0/0xf0
[<ffffffffc047e958>] __fuse_request_send+0x78/0x80 [fuse]
[<ffffffffc0481bdd>] fuse_simple_request+0xbd/0x190 [fuse]
[<ffffffffc0487c37>] fuse_setlk+0x177/0x190 [fuse]
[<ffffffff8b259467>] SyS_flock+0x117/0x190
[<ffffffff8b003b1c>] do_syscall_64+0x7c/0xf0
[<ffffffff8b60632f>] entry_SYSCALL64_slow_path+0x25/0x25
[<ffffffffffffffff>] 0xffffffffffffffff

Once ps shows the D state, the only fix I have found is to reboot the 
server.  After a reboot, things are fine again, until they are not.

Given that gluster is the only thing I know of that uses fuse in this 
system, I guess this is the next thing to investigate.

fwiw, in investigating this, I found that the glusterfs could also be 
mounted in the LXC container by network, instead of mounting the local 
file system.  I set that up, and the mount works, but group permissions 
do not work when accessing the files through the samba share.  if I 
change the file owner to be that of the user I am accessing it with, it 
works, or if I chmod 007 the file that I am trying to access it works, 
but if I chmod 070 the file and access it as a user in the group owner's 
group, I will get an access denied error in the logs.  So I put it back 
to a local mount.  Not sure if this is a different problem or related.

Also not certain if this has any relevance, but this seems to happen 
consistently with certain files.  There is one directory that contains 
an xlsx file, and this file frequently shows up as the one causing the 
disk sleep process as the last file accessed by the hung process's owner 
at the time symptoms start.  Other files also show up repeatedly, though 
less frequently.  I have tried resaving these files, deleting the 
original, and replacing them, but after some time they show up again.

Also uncertain as to relevance; when I run a heal info, there are two 
GFIDs listed.  Neither of them ever seems to go away, and they are both 
listed on each node.  The GFIDs listed in the .glusterfs directory are 
not owned by same owner/group as the files that frequently show up as 
above, and so far I have not been able to find the actual path to these 
files.  I am currently searching the data storage for the related inode 
number, but it has been running for several hours and not found it yet.

I have been trawling the glusterfs logs.  lots of cryptic stuff in there 
I don't understand yet, but I have found nothing consistent with the 
times of errors in the CTDB logs that would indicate a problem.

Another problem to note, for the sake of completeness, is that I also 
have dovecot's mail store on this replicated volume.  we have noted that 
email clients will fail to see mail unless a ls is run on the directory 
from the cli.  I was informed on the IRC channel (which I cannot 
currently access due to "user limit reached") that this is because of a 
known bug that will be fixed in a later release.  I have setup a while 
loop to run this ls command, which seems to be dealing with that. 
However, I am not convinced this problem is related to my hung processes.

so, I am not sure how to chase this to the next step.  I am not sure if 
this is a gluster issue, or an LXC issue.  it seems to not be a CTDB 
issue, but it has not been definitively ruled out in my mind.  would 
anyone have any suggestions on how I might determine if this is a 
gluster issue, or perhaps point me at the right set of documentation for 
troubleshooting this kind of issue?


-- 
Bob Miller
Cell: 867-334-7117
Office: 867-633-3760
www.computerisms.ca