[Gluster-users] Random "No such file or directory" error in gluster client logs - FIXED

[Lee Simpson]

Now it really is fixed!   Removing the "volume statprefetch" from the client config has fixed it.

My initial tests were misleading, as umounting the glusterfs dir was not forcing the reload of the gluster client config (Stray mount -o bind i think). Once i discovered this i restarted my testing, i managed to narrow it down.

Now ive got to work out how to collate this info into something useful for the devs so i can file a bug report. Suggestions on the best way to debug this would be most welcome?

For those that have not been following this thread;

Gluster client was getting random OPEN/STAT/READ errors on files created by apache. Apache would be unable to open these files and each time you tried an error would appear in the gluster client logs. SSH'ing into the client and cat'ing the file would resolve the issue and apache would then read it fine.


Gluster Configs = https://gist.github.com/5f0e6a950712d7f24a64 

Example Errors in gluster client logs = https://gist.github.com/6e2c6dff75fd3bc19fa9

Gluster client/server version = 3.0.3

Kernel =  2.6.31-20-server #58-Ubuntu

OS = Ubuntu 9.10



On Friday 19 Mar 2010 23:26:39 Lee Simpson wrote:
> Changing KVM to use IDE mode has not helped.
> 
> Im running out of ideas... Sometimes i cant get it to do this at all, other times it will do it everytime i tell awstats to regenerate stats.
> 
> 
> 
> On Friday 19 Mar 2010 17:25:13 Lee Simpson wrote:
> > Forgot to mention that my current working theory is that is may be linked to the" Virtio Disk" module in KVM. Switched the gluster server guest over to IDE mode and waiting to see if any errors pop up in the logs.
> > 
> > On Friday 19 Mar 2010 16:13:47 Lee Simpson wrote:
> > > Hello,
> > > 
> > > Thanks for your email. Today ive noticed that this problem is cropping up again. :(  All be it at a much much reduced rate with ext3. 
> > > 
> > > I also seem to be getting some STAT() type errors with ext3 which i never saw with ext4. Again if i ssh into the effected client and cat the file in question, it works and the problem for that file disappears.
> > > 
> > > Any help you can provide would be most welcome. 
> > > 
> > > More info follows;
> > > 
> > > The setup is 2 Physical servers split into multiple KVM guests. Gluster clients and servers are all on guests. The guests are all using the "Virtio Disk" module for their disks which on the dom0 is pointed at a LVM block device.
> > > 
> > > Kernel =  2.6.31-20-server #58-Ubuntu
> > > 
> > > OS = Ubuntu 9.10
> > > 
> > > Current Configs = https://gist.github.com/5f0e6a950712d7f24a64 (Ive tried reducing those configs to have no performance volumes)
> > > 
> > > Errors in gluster client logs = https://gist.github.com/6e2c6dff75fd3bc19fa9
> > > 
> > > 
> > > Is there any further info i can provide to help your diagnosis?
> > > 
> > > 
> > > 
> > > - Lee
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > On Friday 19 Mar 2010 09:07:59 Harshavardhana wrote:
> > > > On 03/18/2010 07:18 PM, Lee Simpson wrote:
> > > > > Hello,
> > > > >
> > > > > Just thought id share the experience i had with a gluster client error and the solution i found after much searching and chatting with some IRC guys.
> > > > >
> > > > > Im running a simple 2 server with multiple clients using cluster/replicate. Randomly newly created files produced the following error in the gluster client logs when accessed;
> > > > >
> > > > > "W [fuse-bridge.c:858:fuse_fd_cbk] glusterfs-fuse: 59480: OPEN() /data/randomfile-here =>  -1 (No such file or directory)"
> > > > >
> > > > > These files are created by apache or other scripts (such as awstats on a cron).  Apache is then unable to read the file, and the above message appears in the gluster logs everytime you try. If i SSH into the apache server and cat the file it displays fine and then apache starts reading it fine.
> > > > >
> > > > > I upgraded the client and server to 3.03 and tried reducing my configs to the bare min without any performance volumes..  But the problem persisted...
> > > > >
> > > > >
> > > > > SOLUTION
> > > > >
> > > > > The solution was quite simple. It turned out that it was because the server's data drive was formatted in ext4. Switched it to ext3 and the problems went away!
> > > > >
> > > > >
> > > > > Hope that helps someone else who finds this.
> > > > >
> > > > >
> > > > > - Lee
> > > > >
> > > > >
> > > > >    
> > > > 
> > > > Hi Lee,
> > > > 
> > > >      can you provide you kernel version under use?. Your volume files?.  
> > > > Let us figure out if this is related to "ext4"
> > > > 
> > > > Regards
> > > > 
> > > > 
> > > 
> > > 
> > 
> > 
> 
> 

-- 
Lee Simpson : Bsc(hons) Software Engineering
Software Development : Server Specification : Manufacture : Configuration

9 Partridge Rd,
St Albans
Herts
AL3 6HH

Email : Lee at LeeSimpson.me.uk
www : http://www.leenix.co.uk

Tel : 01727 855 124
Mob : 07961 348 790





[------------ Disclaimer ------------]
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender by replying to this e-mail.

This email has been scanned for viruses