[Gluster-devel] Self Heal/Recovery Problem

[Kamil Srot]

Dear Gluster developers, fans,

as first thing, I want to say big THANK YOU for the work you do. From 
what I saw and tried out (OCFS2, GFS2, CODA, NFS), your system is the 
first one I like and in some way understand the logic behind it. The 
others seems to be too complex, hard to understand and possibly 
reconfigure in case something wents wrong.
I worked like 2 months with my test setup of OCFS (which is the simplest 
"other" solution of FS clustering) and dont have so nice feeling about 
it than after few days with GlusterFS...

Well, it wouldn't a good post into devel group w/o questions - so I'm 
composing in another window few questions regarding performance/tuning 
of my setup, but recently I run into issue.

I have quite simple setup with two servers doing mirror of data with afr 
*:2 and unify and io-threads...
The setup worked fine for several days of stress testing but recently I 
found article recommending to use some format parameter of underlaying 
XFS filesystem...
So I stopped glfs and glfsd on one of the servers and formatted the 
device... have created the exported directories and started the glfsd & 
glfs again... then I tried to kick start the self heal do remirror the 
testing data fith the find -mountpoint -type f ... ops, the glfsd 
segfaults after few seconds - in the log, I have:

The glfs is: mainline--2.5--patch-518

---------
got signal (11), printing backtrace
---------
[0xb7f7f420]
/cluster/lib/glusterfs/1.3.5/xlator/protocol/client.so[0xb7604432]
/cluster/lib/glusterfs/1.3.5/xlator/protocol/client.so[0xb7606a4b]
/cluster/lib/glusterfs/1.3.5/xlator/protocol/client.so(notify+0xe5)[0xb7607666]
/cluster/lib/libglusterfs.so.0(transport_notify+0x62)[0xb7f70a92]
/cluster/lib/libglusterfs.so.0[0xb7f712fc]
/cluster/lib/libglusterfs.so.0(sys_epoll_iteration+0x16b)[0xb7f71642]
/cluster/lib/libglusterfs.so.0(poll_iteration+0x3b)[0xb7f70dce]
[glusterfsd](main+0x4e3)[0x804991d]
/lib/tls/libc.so.6(__libc_start_main+0xc8)[0xb7e27ea8]
[glusterfsd][0x8048e51]
---------

And core file in root directory... the backtrace is:
#0  0xb75574f8 in afr_sync_ownership_permission ()
   from /cluster/lib/glusterfs/1.3.5/xlator/cluster/afr.so
#1  0xb7576432 in client_closedir_cbk ()
   from /cluster/lib/glusterfs/1.3.5/xlator/protocol/client.so
#2  0xb7578a4b in client_protocol_interpret ()
   from /cluster/lib/glusterfs/1.3.5/xlator/protocol/client.so
#3  0xb7579666 in notify () from 
/cluster/lib/glusterfs/1.3.5/xlator/protocol/client.so
#4  0xb7edfa92 in transport_notify (this=0x8053848, event=1) at 
transport.c:154
#5  0xb7ee02fc in epoll_notify (eevent=1, data=0x8053848) at epoll.c:53
#6  0xb7ee0642 in sys_epoll_iteration (ctx=0xbfb026d4) at epoll.c:155
#7  0xb7edfdce in poll_iteration (ctx=0xbfb026d4) at transport.c:300
#8  0x0804991d in main ()

It seems to be some problem with permissions?

Any hints/help is greatly appreciated!

*glusterfs-server.vol*
volume mailspool-ds
    type storage/posix
    option directory /data/mailspool-ds
end-volume

volume mailspool-ns
    type storage/posix
    option directory /data/mailspool-ns
end-volume

volume mailspool-san1-ds
    type protocol/client
    option transport-type tcp/client
    option remote-host 10.0.0.110
    option remote-subvolume mailspool-ds
end-volume

volume mailspool-san1-ns
    type protocol/client
    option transport-type tcp/client
    option remote-host 10.0.0.110
    option remote-subvolume mailspool-ns
end-volume

volume mailspool-ns-afr
    type cluster/afr
    subvolumes mailspool-ns mailspool-san1-ns
    option replicate *:2
end-volume

volume mailspool-ds-afr
    type cluster/afr
    subvolumes mailspool-ds mailspool-san1-ds
    option replicate *:2
end-volume

volume mailspool-unify
    type cluster/unify
    subvolumes mailspool-ds-afr
    option namespace mailspool-ns-afr
    option scheduler random
end-volume
volume mailspool
    type performance/io-threads
    option thread-count 8
    option cache-size 64MB
    subvolumes mailspool-unify
end-volume

volume server
    type protocol/server
    option transport-type tcp/server
    subvolumes mailspool
    option auth.ip.mailspool-ds.allow 10.0.0.*,127.0.0.1
    option auth.ip.mailspool-ns.allow 10.0.0.*,127.0.0.1
    option auth.ip.mailspool.allow *
end-volume

*glusterfs-client.vol
*volume client
    type protocol/client
    option transport-type tcp/client
    option remote-host 127.0.0.1
    option remote-subvolume mailspool
end-volume

volume writebehind
    type performance/write-behind
    option aggregate-size 131072 # aggregate block size in bytes
    subvolumes client
end-volume

volume readahead
    type performance/read-ahead
    option page-size 131072
    option page-count 2
    subvolumes writebehind
end-volume

volume iothreads    #iothreads can give performance a boost
    type performance/io-threads
    option thread-count 8
    option cache-size 64MB
    subvolumes readahead
end-volume*
*

Best Regards,
-- 
Kamil