[Gluster-users] not sure how to troubleshoot SMB / CIFS overload when using GlusterFS

[Anand Avati]

Please fine responses inline.


> So now, knowing that GlusterFS is kicking into overdrive fretting about a
> file it can't find, I decided to eliminate the web server altogether.  I
> opened up Windows Explorer, and typed in a directory that didn't exist, and
> sure enough, I'm unable to navigate through the share in another Explorer
> window until it finally responds again a minute later.  I think the Page of
> Death was exhibiting such a massive death (e.g. only able to respond again
> upwards of five minutes later) because it was systematically trying to
> access several files that weren't found, and each one it can't find causes
> the SMB connection to hang for close to a minute.
>


Gluster does not really care if a file is not found. It just looks up the
filename on all servers and returns -ENOENT. End of story for Gluster.
What's happening here is that Samba is 'searching' through all filenames in
the directory to match some other filename with strcasecmp() to provide a
case-insensitive match to the user.



>
> I feel like this is a bit of major progress toward pinpointing the problem
> for a possible resolution.  Here are some additional details that may help:
>
> The GlusterFS directory in question, /storage, has about 80,000 subdirs in
> it.  As such, I'm using ext4 to overcome the subdir limitations of ext3.
> The non-existent image file that is able to cause everything to freeze
> exists in a directory, /storage/thisdirdoesntexist/images/blah.gif, where
> "thisdirdoesntexist" is in that storage directory along with those 80,000
> real subdirs.  I know it's a pretty laborious thing for Gluster to piece
> together a directory listing, and combined with Joseph's recognition of the
> flood of "getdents", does it seem reasonable that Gluster or Samba is
> freezing because it's for some reason generating a subdir listing of
> /storage whenever it can't find one of its subdirs?
>


Yes, it is samba searching around for the case insensitive match.


As another test, if I access a file inside a non-existent subdir of a dir
> that only has five subdirs, and nothing freezes.
>


That is because iterating over 5 names to determine non existence of a case
insensitive match is trivially fast.


>
> So the freezing seems to be a function of the number of subdirectories that
> are siblings of the first part of the path that doesn't exist, if that makes
> sense.  So in /this/is/a/long/path, if "is" doesn't exist, then Samba will
> generate a list of subdirs under "/this".  And if "/this" has 100,000
> immediate subdirs under it, then you're about to experience a world of hurt.
>
> I read some where that FUSE's implementation of readdir() is a blocking
> operation.  If true, the above explanation, plus FUSE's readdir(), are to
> blame.
>

What do you mean by that. FUSE's readdir() is as blocking or unblocking as
the rest of its open/create/getattr/setattr etc. What you probably meant was
that the fuse kernel module does not cache dirents?


> And I am therefore up a creek.  It is not feasible to enforce the system to
> only have a few subdirs at any given level to prevent the lockup.  Unless
> somebody, after reading this novel, has some ideas for me to try.  =)  Any
> magical ways to not get FUSE to block, or any trickery on Samba's side?


It is not FUSE blocking that is your problem. You need a quicker trick to
achieve case insensitivity.

Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110718/8779827f/attachment.html>