[Gluster-devel] Spurious termination of fuse invalidation notifier thread

[Xavier Hernandez]

Hi Raghavendra,

On 06/09/16 06:11, Raghavendra Gowdappa wrote:
>
>
> ----- Original Message -----
>> From: "Xavier Hernandez" <xhernandez at datalab.es>
>> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>, "Kaleb Keithley" <kkeithle at redhat.com>, "Pranith Kumar Karampuri"
>> <pkarampu at redhat.com>
>> Cc: "Csaba Henk" <chenk at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>
>> Sent: Monday, September 5, 2016 12:46:43 PM
>> Subject: Re: Spurious termination of fuse invalidation notifier thread
>>
>> Hi Raghavendra,
>>
>> On 03/09/16 05:42, Raghavendra Gowdappa wrote:
>>> Hi Xavi/Kaleb/Pranith,
>>>
>>> During few of our older conversations (like [1], but not only one), some of
>>> you had reported that the thread which writes invalidation notifications
>>> (of inodes, entries) to /dev/fuse terminates spuriously. Csaba tried to
>>> reproduce the issue, but without success. It would be helpful if you
>>> provide any information on reproducer and/or possible reasons for the
>>> behavior.
>>
>> I didn't found what really caused the problem. I only saw the
>> termination message on a production server after some days working but
>> hadn't had the opportunity to debug it.
>>
>> Looking at the code, the only conclusion I got is that the result from
>> the write to /dev/fuse was unexpected. The patch solves this and I
>> haven't seen the problem again.
>>
>> The old code only manages ENOENT error. It exits the thread for any
>> other error. I guess that in some situations a write to /dev/fuse can
>> return other "non fatal" errors.
>
> Thanks Xavi. Now I remember the changes. Since you have not seen spurious termination after the changes, I assume the issue is fixed.

Yes, I haven't seen the issue again since the patch was applied.

>
>>
>> As a guess, I think it may be a failure in an entry invalidation.
>> Looking at the code of fuse, it may return ENOTDIR if parent of the
>> entry is not a directory and some race happens doing rm/create while
>> sending invalidations in the background. Another possibility is
>> ENOTEMPTY if the entry references a non empty directory (again probably
>> caused by races between user mode operations and background
>> invalidations). Anyway this is only a guess, I have no more information.
>>
>> Xavi
>>
>>>
>>> [1]
>>> http://review.gluster.org/#/c/13274/1/xlators/mount/fuse/src/fuse-bridge.c
>>>
>>> regards,
>>> Raghavendra
>>>
>>