[Gluster-users] Invalid argument on delete with GlusterFS 3.1.1 client

[Dan Bretherton]

It is interesting to know that the problem is related to link files.  I 
have found a way to find and delete them all on back-end filesystems, 
using a command like the one below.

find /brick/path -size 0b -perm 1000 -exec /bin/rm -v {} \;

I used this technique once in the past, before I realised what link 
files were, after I rescued some data from a non-functioning GlusterFS 
volume after some of my early experiments went wrong (- definitely not 
recommended or supported). I deleted the link files from the backend 
filesystems before transferring the files brick by brick.

This might be a solution to our current problem; the link files seem to 
be regenerated again when the files are accessed, but I don't know what 
other side effects there might be.  Even if this procedure did work and 
was safe, I suspect the effect would only be temporary.  The problem 
seems to affect files randomly, including old and newly created files.  
The only work-around we have found is to to use NFS instead of GlusterFS 
to mount the data.  I am trying to stick with GlusterFS for our compute 
servers for performance reasons, and we have found that moving problem 
files to another location, using an NFS client, then moving them back 
again, allows the files to be accessed normally on GlusterFS clients again.

-Dan.

On 18/01/2011 00:05, Diego M. Vadell wrote:
> Hi
>
>     I had a similar problem: I couldn't move or read or CHMOD a file. Everything
> returned Invalid Argument. I found the file in one of the storage nodes (where
> one shouldn't be touching anything), with the full contents of the file, the
> right permissions, etc. Then I found another file in another storage node, with
> 0 bytes and with permission "---------T". I deleted the last one and now I can
> read/chmod the original file.
>
> Hope this helps,
>
>   -- Diego
>
> On Thursday 06 January 2011 07:26:01 Dan Bretherton wrote:
>> Hello All,
>> I have seen what appears to be the same problem when a user tried to
>> change the ownership of some files.  The client log file entries were
>> like this:
>>
>> [2011-01-05 17:10:02.222293] W [fuse-bridge.c:648:fuse_setattr_cbk]
>> glusterfs-fuse: 16210806: SETATTR()
>> /users/rle/INTERIM/INTERIM_2003_VOR850.nc =>  -1 (Invalid argument)
>>
>> The error reported on the command line was "permission denied" I am
>> told, not "invalid argument" as with the file deletion problem.
>> Unfortunately I didn't get a chance set the log level to TRACE because
>> the user in question went to an NFS client to change the ownership
>> before telling me there had been a problem.  However this might make it
>> easier to reproduce the problem, now that we know that it isn't
>> restricted to file deletion.
>>
>> -Dan.
>>
>> -
>>
>> On 01/06/2011 09:07 AM, Thai. Ngo Bao wrote:
>>> Dear All,
>>>
>>> I'd like to confirm that I am having the same problem. I am using
>>> glusterfs 3.1.1.
>>>
>>> I did set volume diagnostics.client-log-level to TRACE, please see the
>>> attached file for further information.
>>>
>>> Thanks for your support.
>>>
>>> ~Thai
>>> -----Original Message-----
>>> From: gluster-users-bounces at gluster.org
>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Vijay Bellur
>>> Sent: Wednesday, January 05, 2011 1:30 AM
>>> To: Dan Bretherton
>>> Cc: gluster-users
>>> Subject: Re: [Gluster-users] Invalid argument on delete with GlusterFS
>>> 3.1.1 client
>>>
>>> On Tuesday 04 January 2011 11:21 PM, Dan Bretherton wrote:
>>>> Hello Vijay,
>>>> There is nothing except routine "accepted client" messages in the
>>>> brick log files on the servers, and I can't see anything relevant in
>>>> the /var/log/messages files.  On the client the only messages relating
>>>> to this problem were included in my original mailing list message. Are
>>>> there some other log files I should be looking at?  I don't know how
>>>> to recreate this problem I'm afraid - it's just a case of waiting
>>>> until it crops up again.  When it does I will try to persuade the
>>>> users to leave the files in place while I investigate further.  When
>>>> the time comes what should I look for?
>>> When it happens again, please set the glusterfs client log level to
>>> TRACE through
>>>
>>> #gluster volume set<volname>   diagnostics.client-log-level TRACE,
>>> perform the delete operation on such files and send across the client
>>> log file to us.
>>>
>>> You can revert back the log level to INFO after you are done with this.
>>>
>>> Thanks,
>>> Vijay
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>