[Gluster-devel] optimizing gluster fuse

[riya khanna]

Hi All,

I'm trying to use the new framework to speed up lookups/attr/xattr
operations by split functionality between fast/slow execution paths. I'd
highly appreciate if you could suggest experiments to evaluate the
performance improvement.

As I mentioned in my previous email, I'm caching replies from fuse daemon
(hashed key/value blobs) in the kernel so that for the same key (e.g.,
<parent ino, child name> in case of FUSE_LOOKUP), the reply (e.g.,
fuse_entry_out) is served from the kernel itself and no call is delivered
to user-space.

While this may seem redundant due to entry_timeout/attr_timeout caching
that already exists in FUSE, this design provides more control to the
user-space daemon over when/what to invalidate. For instance, entry_timeout
caching is only valid until a timeout or until the kernel removes dentry
from its dcache.

For invalidation, fuse_lowlevel_notify_inval_entry() can also remove
entries from the hash table. Please refer to the figure attached in my last
email.

Thanks,
Riya

On Tue, Apr 3, 2018 at 1:45 PM, riya khanna <riyakhanna1983 at gmail.com>
wrote:

> I'm attaching a figure that depicts the architecture of my optimized fuse
> framework. Kindly let me know if you have any questions.
>
> On Mon, Apr 2, 2018 at 10:57 AM, riya khanna <riyakhanna1983 at gmail.com>
> wrote:
>
>> Thanks Amar! Please see my answers inline.
>>
>> On Mon, Apr 2, 2018 at 5:41 AM, Amar Tumballi <atumball at redhat.com>
>> wrote:
>>
>>> Hi Riya,
>>>
>>> Thanks for writing to us. Some questions before we start on this.
>>>
>>> * Where can we see your work of modifying the fuse module to cache the
>>> calls? Some reference would help us to provide more specific pointers. (or
>>> ask better questions).
>>>
>>> I've created a fast path framework for FUSE, where the user space daemon
>> can load a module and register handlers for file operations (lookup, open,
>> r/w, etc.) that must be handled in the kernel itself without an up call to
>> the user space. I call them fast path handlers. This design also retains
>> the regular FUSE handlers for file system operations in upser-space (slow
>> path). The fast path and slow path can communicate with each other over
>> shared memory or using  syscalls to enable/invalidate caching of data
>> structs (e.g., results of getattr, getxattr, etc.)
>>
>> There's a process I need to follow in order to make the code available
>> publicly. I've already started, but will take some time. I will try to do
>> this asap.
>>
>> * If the caching happened in fuse module, and it expects the regular
>>> arguments as the parameters, then there may not be any work required at all
>>> in glusterfs, as it works on low-level fuse api.
>>>
>>>
>> The fast handlers expect same interface and args (fuse_args) as the
>> regular user-space daemon. The fast handler code is fs-specific, therefore,
>> must come from glusterfs. Changes are also needed in glusterfs code to
>> communicate with the fast path for enabling/invalidating caching.
>>
>>
>>> * Also, how to invalidate caches from userspace program? because
>>> GlusterFS could be accessed from multiple clients, so it becomes an
>>> important piece to have.
>>>
>>>
>> Server invalidate can trigger a system call into the fast path framework
>> to invalidate caches.
>>
>>
>>> For referring at the codebase to look at integration with fuse module,
>>> please check the directory 'xlators/mount/fuse/src/' and mostly the file
>>> 'fuse-bridge.c'.
>>>
>>> Thanks for your interest in project, would be great to collaborate on
>>> this effort, as it can enhance the performance of glusterfs in many
>>> usecases.
>>>
>>
>> I'm still going through gluster developer documentation, but it'd be
>> helpful if you could mention what kind of use cases does the fast/slow
>> split FUSE framework enable? i've already applied the framework to
>> accelerate multiple FUSE-based stackable file systems, but want the
>> interface to be generic enough for all FUSE file systems to take advantage
>> of it.
>>
>>
>>> Regards,
>>> Amar
>>>
>>>
>>>
>>>
>>> On Mon, Apr 2, 2018 at 6:34 AM, riya khanna <riyakhanna1983 at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I've modified FUSE framework to take a part of user-space daemon code
>>>> and moves it into the kernel fuse driver to minimize user-kernel-user
>>>> switches during file system  operations. An example would be caching
>>>> getattr/getxattr/lookup/security checks etc. This design, therefore,
>>>> create fast (served directly from the kernel) and a slow (regular fuse)
>>>> execution paths. The fast and slow paths can also communicate with each
>>>> other using shared memory.
>>>>
>>>> I was wondering if it is possible to accelerate glusterfs using this
>>>> design. What pieces could (should) be easily moved to kernel space?
>>>> Any pointers would be highly appreciated. Thanks!
>>>>
>>>> -Riya
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>
>>>
>>>
>>> --
>>> Amar Tumballi (amarts)
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180409/d2912c28/attachment.html>