[Gluster-devel] servers keep hanging with fuse errors

Wed Jan 30 10:03:23 UTC 2008

Jordi,
 is the fuse kernel module from 2.7.2glfs8 as well? (compiled with
./configure --enable-kernel-module and make install inside kernel/ subdir)
the kernel panic you've pasted was a well known issue with older fuse kernel
modules and it seems to me that your kernel module is still old.

avati

2008/1/30, Jordi Moles <jordi at cdmon.com>:
>
> Hi,
>
> i'm running this versions of the packages now:
>
> ii  fuse-utils
> 2.7.2-glfs8                              Filesystem in USErspace
> (utilities)
> ii  glusterfs-client
> 1.3.8-TLA643                             GlusterFS fuse client
> ii  libfuse2
> 2.7.2-glfs8                              Filesystem in USErspace library
>
> I also removed any kind of lock method from dovecot.conf and added the
> lock function to the filesystem itself, on the server side, with the
> "type features/posix-locks" option.
> The thing is that now, dovecots work grate, but postfixs hang very
> often, with the following error:
>
> ***************
>
> Unable to handle kernel paging request at 0000000000100108 RIP:
> [<ffffffff88020838>] :fuse:request_end+0x45/0x109
> PGD 1f29c067 PUD 1f327067 PMD 0
> Oops: 0002 [1] SMP
> CPU 0
> Modules linked in: ipv6 fuse dm_snapshot dm_mirror dm_mod
> Pid: 723, comm: glusterfs Not tainted 2.6.18-xen #1
> RIP: e030:[<ffffffff88020838>]  [<ffffffff88020838>]
> :fuse:request_end+0x45/0x109
> RSP: e02b:ffff88001ecb1d68  EFLAGS: 00010246
> RAX: 0000000000200200 RBX: ffff88001e82af48 RCX: ffff88001e82af58
> RDX: 0000000000100100 RSI: ffff88001e82af48 RDI: ffff88001f68d400
> RBP: ffff88001f68d400 R08: 000000001f74ab40 R09: ffff88001e82b048
> R10: 0000000000000008 R11: ffff88001ecb1cf0 R12: 0000000000000000
> R13: ffff88001e82af80 R14: ffff88001ecb1df8 R15: 0000000000000001
> FS:  00002ba3e961aae0(0063) GS:ffffffff804cd000(0000)
> knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000
> Process glusterfs (pid: 723, threadinfo ffff88001ecb0000, task
> ffff88001fe80820)
> Stack:  ffff88001e82af48 ffff88001f68d400 00000000fffffffe
> ffffffff88021056
> ffff88001ecb1ef8 000000301fcb8180 00000156fffffff4 ffff88001f72f100
> 0000000000000015 0000000000000000 ffff88001ecb1e18 0000000041000a90
> Call Trace:
> [<ffffffff88021056>] :fuse:fuse_dev_readv+0x385/0x435
> [<ffffffff802801d3>] do_readv_writev+0x271/0x294
> [<ffffffff802274c7>] default_wake_function+0x0/0xe
> [<ffffffff88021120>] :fuse:fuse_dev_read+0x1a/0x1f
> [<ffffffff802804bc>] vfs_read+0xcb/0x171
> [<ffffffff8028089b>] sys_read+0x45/0x6e
> [<ffffffff8020a436>] system_call+0x86/0x8b
> [<ffffffff8020a3b0>] system_call+0x0/0x8b
>
>
> Code: 48 89 42 08 48 89 10 48 c7 41 08 00 02 20 00 f6 46 30 08 48
> RIP  [<ffffffff88020838>] :fuse:request_end+0x45/0x109
> RSP <ffff88001ecb1d68>
> CR2: 0000000000100108
>   postfix01gluster01 kernel: Oops: 0002 [1] SMP
>                                                  postfix01gluster01
> kernel: CR2: 0000000000100108
>
> <3>BUG: soft lockup detected on CPU#0!
>
> Call Trace:
> <IRQ> [<ffffffff80257f78>] softlockup_tick+0xd8/0xea
> [<ffffffff8020f110>] timer_interrupt+0x3a9/0x405
> [<ffffffff80258264>] handle_IRQ_event+0x4e/0x96
> [<ffffffff80258350>] __do_IRQ+0xa4/0x105
> [<ffffffff8020b0e8>] call_softirq+0x1c/0x28
> [<ffffffff8020cecb>] do_IRQ+0x65/0x73
> [<ffffffff8034a8c1>] evtchn_do_upcall+0xac/0x12d
> [<ffffffff8020ac1e>] do_hypervisor_callback+0x1e/0x2c
> <EOI> [<ffffffff803f1234>] .text.lock.spinlock+0x2/0x8a
> [<ffffffff88020a4f>] :fuse:fuse_dev_writev+0xb8/0x31b
> [<ffffffff88020cb2>] :fuse:fuse_dev_write+0x0/0x1f
> [<ffffffff802800d7>] do_readv_writev+0x175/0x294
> [<ffffffff88020cb2>] :fuse:fuse_dev_write+0x0/0x1f
> [<ffffffff803efb3b>] schedule_timeout+0x1e/0xad
> [<ffffffff803f0976>] __down_read+0x12/0xec
> [<ffffffff80280695>] sys_writev+0x45/0x93
> [<ffffffff8020a436>] system_call+0x86/0x8b
> [<ffffffff8020a3b0>] system_call+0x0/0x8b
>
>
> ***************
>
> i set the log level for glusterfs on both nodes and clients to WARNING.
> However, nothing was written on any of them :( .
>
> I keep using virtual machines with xen 3.1 to test all this, but i
> already tried with non-virtual environments and got the same errors.
>
> Do you have any ideas?
>
> En/na Jordi Moles Blanco ha escrit:
> > hi,
> >
> > thanks for the details, i'll give it a try and come back to you to tell
> > whether has become stable or not.
> >
> > Thank you very much.
> >
> >
> > El dt 29 de 01 del 2008 a les 01:11 +0530, en/na Anand Avati va
> > escriure:
> >
> >> Jordi,
> >>  you should really upgrade your fuse kernel module which will fix this
> >> issue. please use the kernel module from -
> >>
> >> http://ftp.zresearch.com/pub/gluster/glusterfs/fuse/
> >>
> >> thanks,
> >> avati
> >>
> >> 2008/1/28, Jordi Moles <jordi at cdmon.com>:
> >>
> >>         Hi,
> >>
> >>         i'm sorry but i can't get any newer version with the
> >>         repositories you
> >>         gave me, after apt-update, apt-upgrade says there's nothing to
> >>         upgrade.
> >>         And i try, for example, to get the source code of every
> >>         package, i get this
> >>
> >>         fuse_2.5.3-4.4, which is even older than the one installed on
> >>         my system.
> >>
> >>
> >>
> >>
> >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>

-- 
If I traveled to the end of the rainbow
As Dame Fortune did intend,
Murphy would be there to tell me
The pot's at the other end.