[Gluster-users] Gluster failure testing

[Bryan Whitehead]

are you using ext4 with redhat/centos? There is a previous thread that
shows some kind of bug with ext4 that causes similar sounding
problems.

If you are using ext4, try using xfs.

On Tue, Aug 14, 2012 at 11:12 PM, Brian Candler <B.Candler at pobox.com> wrote:
> On Tue, Aug 14, 2012 at 08:19:27PM -0700, stephen pierce wrote:
>>    I let both clients run for a while, then I stop one client. I then
>>    reset the brick/server that is not active (the other one is servicing
>>    the HTTP traffic) now.
>
> Do you mean that client1 sends HTTP traffic to brick/server1, and client2
> sends HTTP traffic to brick/server2?
>
>>    While investigating, I discover that there are a lot of phantom
>>    files that are listed with just a filename, and lots of question marks
>>    (????) when doing an ls l. rm rf * on the Gluster volume seems to
>>    complete, but leaves behind all the broken files.
>
> It would be helpful if you could show the actual ls -l output, but my guess
> is you are seeing something like this (demo on a local filesystem, not
> gluster):
>
> $ mkdir testdir
> $ touch testdir/testfile
> $ chmod -x testdir
> $ ls -l testdir
> ls: cannot access testdir/testfile: Permission denied
> total 0
> -????????? ? ? ? ?                ? testfile
>
> If so, these aren't really "phantom files", but the permissions of the
> enclosing directory are set wrongly (which might be some intermediate state
> in gluster replication, I don't know)
>
> So an "ls -ld" of the parent directory would also be a good thing. Also, are
> these filenames those you'd expect your application to create?
>
> What might be helpful is to trace your backend-application and what's making
> it return a 500 server error, which may or may not be related to these
> permissions.  If you can see what file operations the backend is trying to
> do and what filesystem error is being returned (e.g.  with strace), this may
> make it clearer what's going on.  Then you can perhaps crank up gluster logs
> at the appropriate place too.
>
> Any log messages talking about "split brain" would be especially interesting.
>
> Regards,
>
> Brian.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users