[Gluster-devel] [RFC] What if client fuse process crash?

[Changwei Ge]

Hi Niels,

On 2019/8/6 3:50 下午, Niels de Vos wrote:
> On Tue, Aug 06, 2019 at 03:14:46PM +0800, Changwei Ge wrote:
>> On 2019/8/6 2:57 下午, Ravishankar N wrote:
>>> On 06/08/19 11:44 AM, Changwei Ge wrote:
>>>> Hi Ravishankar,
>>>>
>>>>
>>>> Thanks for your share, it's very useful to me.
>>>>
>>>> I am setting up a glusterfs storage cluster recently and the
>>>> umount/mount recovering process bothered me.
>>> Hi Changwei,
>>> Why are you needing to do frequent remounts? If your gluster fuse client
>>> is crashing frequently, that should be investigated and fixed. If you
>>> have a reproducer, please raise a bug with all the details like the
>>> glusterfs version, core files and log files.
>>
>> Hi Ravi,
>>
>> Actually, glusterfs client fuse process ran well in my environment. But
>> high-availability and fault-tolerance are also my big concerns.
>>
>> So I killed the fuse process to see what would happen. AFAIK, userspace
>> processes are likely to be killed or crashed somehow, which is not under our
>> control. :-(
>>
>> Another scenario is *software upgrade*. Since we have to upgrade glusterfs
>> client version in order to enrich features and fix bugs.  It will be
>> friendly to applications if the upgrade is transparent.
> As open files have a state associated with them, and the state is lost
> when the fuse process exits. Restarting the fuse process will then need
> to restore the state of the open files (and caches, and more). This is
> not trivial and I do not think any work on this end has been done yet.


True, tons of work have to be done if we want to restore all files' 
state to make restarted fuse process continue to work as never be restarted.

I suppose two methods might be feasible:

     One is to try to fetch file state from kernel to restore files' 
state into fuse process,

     the other one is to duplicate those  state to a standby process or 
just use Linux shared memory mechanism?


>
> Some users take an alternative route. Mounted filesystems have indeed
> issues with online updating. So, maybe you do not need to mount the
> filesystem at all. Depending on the need of your applications, using
> glusterfs-coreutils instead of a FUSE (or NFS) mount might be an option
> for you. The short living processes connect to the Gluster Volume when
> needed, and do not keep a connection open. Updating userspace tools is
> much simpler than long running processes that are hooked into the
> kernel.
>
> See https://github.com/gluster/glusterfs-coreutils for details.


That's helpful, but I think then some POSIX file operations can't be 
performed anymore.


Thanks,

Changwei


>
> HTH,
> Niels
>
>
>>
>> Thanks,
>>
>> Changwei
>>
>>
>>> Regards,
>>> Ravi
>>>>
>>>> I happened to find some patches[1] from internet aiming to address
>>>> such a problem but no idea why they were not managed to merge into
>>>> glusterfs mainline.
>>>>
>>>> Do you know why?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Changwei
>>>>
>>>>
>>>> [1]:
>>>>
>>>> https://review.gluster.org/#/c/glusterfs/+/16843/
>>>>
>>>> https://github.com/gluster/glusterfs/issues/242
>>>>
>>>>
>>>> On 2019/8/6 1:12 下午, Ravishankar N wrote:
>>>>> On 05/08/19 3:31 PM, Changwei Ge wrote:
>>>>>> Hi list,
>>>>>>
>>>>>> If somehow, glusterfs client fuse process dies. All
>>>>>> subsequent file operations will be failed with error 'no
>>>>>> connection'.
>>>>>>
>>>>>> I am curious if the only way to recover is umount and mount again?
>>>>> Yes, this is pretty much the case with all fuse based file
>>>>> systems. You can use -o auto_unmount
>>>>> (https://review.gluster.org/#/c/17230/) to automatically cleanup
>>>>> and not having to manually unmount.
>>>>>> If so, that means all processes working on top of glusterfs
>>>>>> have to close files, which sometimes is hard to be
>>>>>> acceptable.
>>>>> There is
>>>>> https://research.cs.wisc.edu/wind/Publications/refuse-eurosys11.html,
>>>>> which claims to provide a framework for transparent failovers. I
>>>>> can't find any publicly available code though.
>>>>>
>>>>> Regards,
>>>>> Ravi
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Changwei
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> Community Meeting Calendar:
>>>>>>
>>>>>> APAC Schedule -
>>>>>> Every 2nd and 4th Tuesday at 11:30 AM IST
>>>>>> Bridge: https://bluejeans.com/836554017
>>>>>>
>>>>>> NA/EMEA Schedule -
>>>>>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>>>>>> Bridge: https://bluejeans.com/486278655
>>>>>>
>>>>>> Gluster-devel mailing list
>>>>>> Gluster-devel at gluster.org
>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>>>
>> _______________________________________________
>>
>> Community Meeting Calendar:
>>
>> APAC Schedule -
>> Every 2nd and 4th Tuesday at 11:30 AM IST
>> Bridge: https://bluejeans.com/836554017
>>
>> NA/EMEA Schedule -
>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>> Bridge: https://bluejeans.com/486278655
>>
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>