[Gluster-devel] Progress on adding support for SEEK_DATA and SEEK_HOLE

Xavier Hernandez xhernandez at datalab.es
Mon Jul 6 07:37:08 UTC 2015

On 07/06/2015 01:15 AM, Niels de Vos wrote:
> On Wed, Jul 01, 2015 at 09:41:19PM +0200, Niels de Vos wrote:
>> On Wed, Jul 01, 2015 at 07:15:12PM +0200, Xavier Hernandez wrote:
>>> On 07/01/2015 08:53 AM, Niels de Vos wrote:
>>>> On Tue, Jun 30, 2015 at 11:48:20PM +0530, Ravishankar N wrote:
>>>>> On 06/22/2015 03:22 PM, Ravishankar N wrote:
>>>>>> On 06/22/2015 01:41 PM, Miklos Szeredi wrote:
>>>>>>> On Sun, Jun 21, 2015 at 6:20 PM, Niels de Vos <ndevos at redhat.com> wrote:
>>>>>>>> Hi,
>>>>>>>> it seems that there could be a reasonable benefit for virtual machine
>>>>>>>> images on a FUSE mountpoint when SEEK_DATA and SEEK_HOLE would be
>>>>>>>> available. At the moment, FUSE does not pass lseek() on to the
>>>>>>>> userspace
>>>>>>>> process that handles the I/O.
>>>>>>>> Other filesystems that do not (need to) track the position in the
>>>>>>>> file-descriptor are starting to support SEEK_DATA/HOLE. One example is
>>>>>>>> NFS:
>>>>>>>> https://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-38#section-15.11
>>>>>>>> I would like to add this feature to Gluster, and am wondering if there
>>>>>>>> are any reasons why it should/could not be added to FUSE.
>>>>>>> I don't see any reason why it couldn't be added.  Please go ahead.
>>>>>> Thanks for bouncing the mail to me Niels, I would be happy to work on
>>>>>> this. I'll submit a patch by Monday next.
>>>>> Sent a patch @
>>>>> http://thread.gmane.org/gmane.comp.file-systems.fuse.devel/14752
>>>>> I've tested it with some skeleton code in gluster-fuse to handle lseek().
>>>> Ravi also sent his patch for glusterfs-fuse:
>>>>    http://review.gluster.org/11474
>>>> I have posted my COMPLETELY UNTESTED patches to their own Gerrit topic
>>>> so that we can easily track the progress:
>>>>    http://review.gluster.org/#/q/status:open+project:glusterfs+branch:master+topic:wip/SEEK_HOLE
>>>> My preference goes to share things early and make everyone able to
>>>> follow progress (know where to find the latest patches). Assistance in
>>>> testing, reviewing and improving is welcome! There are some outstanding
>>>> things like seek() for ec and sharding, and probably more.
>>>> This all was done as a suggestion from Christopher (kripper) Pereira,
>>>> for improving the handling of sparse files (like most VM images).
>>> I've posted the patch for ec in the same Gerrit topic:
>>>      http://review.gluster.org/11494/
>> Thanks!
>>> It has not been tested and some discussion about if it's really needed to
>>> send the request to all subvolumes will be needed.
>>> The lock and the xattrop are absolutely needed. Even if we send the request
>>> to only one subvolume, we need to know which ones are healthy (to avoid
>>> sending the request to a brick that could have invalid hole information).
>>> This could have been done in open, but since NFS does not issue open calls,
>>> we cannot rely on that.
>> Ok, yes, that makes sense. We will likely have SEEK as an operation in
>> NFS-Ganesha at one point, and that will use the handle-based gfapi
>> functions.
>>> Once we know which bricks are healthy we could opt for sending the request
>>> only to one of them. In this case we need to be aware that even healthy
>>> bricks could have different hole locations.
>> I'm not sure if I understand what you mean, but that likely has to do
>> that I dont know much about ec. I'll try to think it through later this
>> week.
> The only thing that would need to be guaranteed is that the offset of
> the hole/data is safe. The whole purpose is to improve handling of
> sparse files, this does not need to be perfect. The holes themselves are
> not important, but the non-holes are.
> When a sparse file (think VM image) is copied, the goal is to not read
> the holes which would return NUL bytes. If calculating the start of a
> hole or the end is not exact, that is not a fatal issue. Reading and
> backing up a series of NUL bytes before/after the hole should be
> acceptable.
> A drawing can probably explain things a little better.
>                          lseek(SEEK_HOLE)
>                            |       |
>                    perfect |       | acceptable
>                      match |       | match
>                            |       |
>       .....................|.......|.....................
>       :file                |       |                    :
>       : .----------------. v       v           .------. :
>       : '----------------'                 ^   '------' :
>       :                                    |   ^        :
>       .....................................|...|.........
>                                            |   |
>                                 acceptable |   | perfect
>                                      match |   | match
>                                            |   |
>                                          lseek(SEEK_DATA)
> I have no idea how ec can figure out the offset of holes/data, that
> would be interesting to know. Is it something that is available in a
> design document somewhere?

EC splits the file in chunks of 512 * #data bricks. Each brick receives 
a fragment of 512 bytes for each chunk. These fragments are the minimal 
units of data and they are a hole or they contain data but not a mix (if 
part of the fragment should be a hole, it's filled with 0's). This means 
that backend filesystems can only have data/holes aligned to offsets 
multiple of 512 bytes.

Reading some other information and your explanation, I will need to 
change the logic to detect data/holes. I'll update the patch as soon as 

> My inclination is to have the same consistency for the seek() FOP as for
> read(). The same locking and health-checks would apply. Does that help?

What provides consistency to read() is the initial check done just after 
the locking. I think this is enough to choose one healthy brick, so I'll 
also update the patch to only use a single brick for seek() instead of 
sending the request to multiple bricks.

Even if the data/hole positions can be different between healthy bricks, 
those bricks that have data where others have holes, must contain 0's 
(otherwise they shouldn't be healthy). So I think it's not so important 
to query multiple bricks to obtain more accurate information.


More information about the Gluster-devel mailing list