[Gluster-users] Gluster (2.0.1 -> git) with fuse 2.8 crashes NFS

Justice London jlondon at lawinfo.com
Wed Jul 8 00:02:27 UTC 2009


Actually, I spoke too soon. NFS still crashes, even if the mountpoint
doesn't.

 

Justice London

jlondon at lawinfo.com



 

PLEASE NOTE: This message, including any attachments, may include 
privileged, confidential and/or inside information. Any 
distribution or use of this communication by anyone other than the 
intended recipient(s) is strictly prohibited and may be unlawful. 
If you are not the intended recipient, please notify the sender 
by replying to this message and then delete it from your system. 
Thank you.

  _____  

From: Justice London [mailto:jlondon at lawinfo.com] 
Sent: Tuesday, July 07, 2009 3:43 PM
To: 'Justice London'; 'Raghavendra G'
Cc: 'gluster-users'; 'Harshavardhana'
Subject: RE: [Gluster-users] Gluster (2.0.1 -> git) with fuse 2.8 crashes
NFS

 

The 2.0.3 release of gluster appears so far to have fixed the crash issue I
was experiencing. What was the specific patch that fixed for it I was
wondering?

 

Great job either way! It appears that with fuse 2.8 and newer kernels that
gluster absolutely flies. With a replication environment between two crummy
testbed machines it's probably about twice as fast as 2.7.4 based fuse!

 

Justice London
jlondon at lawinfo.com

  _____  

From: gluster-users-bounces at gluster.org
[mailto:gluster-users-bounces at gluster.org] On Behalf Of Justice London
Sent: Thursday, July 02, 2009 12:33 PM
To: 'Raghavendra G'
Cc: 'gluster-users'; 'Harshavardhana'
Subject: Re: [Gluster-users] Gluster (2.0.1 -> git) with fuse 2.8 crashes
NFS

 

Sure:

 

Server:

 

### Export volume "brick" with the contents of "/home/export" directory.

volume posix

  type storage/posix                   # POSIX FS translator

  option directory /home/gluster/vmglustore       # Export this directory

  option background-unlink yes

end-volume

 

volume locks

  type features/posix-locks

  subvolumes posix

end-volume

 

volume brick

  type performance/io-threads

  option thread-count 32

#  option autoscaling yes

#  option min-threads 8

#  option max-threads 200

  subvolumes locks

end-volume

 

### Add network serving capability to above brick.

volume brick-server

  type protocol/server

  option transport-type tcp

# option transport-type unix

# option transport-type ib-sdp

# option transport.socket.bind-address 192.168.1.10     # Default is to
listen on all interfaces

# option transport.socket.listen-port 6996              # Default is 6996

 

# option transport-type ib-verbs

# option transport.ib-verbs.bind-address 192.168.1.10     # Default is to
listen on all interfaces

# option transport.ib-verbs.listen-port 6996              # Default is 6996

# option transport.ib-verbs.work-request-send-size  131072

# option transport.ib-verbs.work-request-send-count 64

# option transport.ib-verbs.work-request-recv-size  131072

# option transport.ib-verbs.work-request-recv-count 64

 

  option client-volume-filename /etc/glusterfs/glusterfs.vol

  subvolumes brick

# NOTE: Access to any volume through protocol/server is denied by

# default. You need to explicitly grant access through # "auth"

# option.

  option auth.addr.brick.allow * # Allow access to "brick" volume

end-volume

 

 

Client:

 

### Add client feature and attach to remote subvolume

volume remotebrick1

  type protocol/client

  option transport-type tcp

# option transport-type unix

# option transport-type ib-sdp

  option remote-host 192.168.1.35         # IP address of the remote brick

# option transport.socket.remote-port 6996              # default server
port is 6996

 

# option transport-type ib-verbs

# option transport.ib-verbs.remote-port 6996              # default server
port is 6996

# option transport.ib-verbs.work-request-send-size  1048576

# option transport.ib-verbs.work-request-send-count 16

# option transport.ib-verbs.work-request-recv-size  1048576

# option transport.ib-verbs.work-request-recv-count 16

 

# option transport-timeout 30          # seconds to wait for a reply

                                       # from server for each request

  option remote-subvolume brick        # name of the remote volume

end-volume

 

volume remotebrick2

  type protocol/client

  option transport-type tcp

  option remote-host 192.168.1.36

  option remote-subvolume brick

end-volume

 

volume brick-replicate

  type cluster/replicate

  subvolumes remotebrick1 remotebrick2

end-volume

 

 

volume threads

  type performance/io-threads

  option thread-count 8

#  option autoscaling yes

#  option min-threads 8

#  option max-threads 200

  subvolumes brick-replicate

end-volume

 

### Add readahead feature

volume readahead

  type performance/read-ahead

  option page-count 4       # cache per file  = (page-count x page-size)

  option force-atime-update off

  subvolumes threads

end-volume

 

### Add IO-Cache feature

#volume iocache

#  type performance/io-cache

#  option page-size 1MB

#  option cache-size 64MB

#  subvolumes readahead

#end-volume

 

### Add writeback feature

volume writeback

  type performance/write-behind

  option cache-size 8MB

  option flush-behind on

  subvolumes readahead

end-volume

 

 

Justice London
jlondon at lawinfo.com

  _____  

From: Raghavendra G [mailto:raghavendra.hg at gmail.com] 
Sent: Thursday, July 02, 2009 10:17 AM
To: Justice London
Cc: Harshavardhana; gluster-users
Subject: Re: [Gluster-users] Gluster (2.0.1 -> git) with fuse 2.8 crashes
NFS

 

Hi,

Can you send across the volume specification files you are using?

regards,
Raghavendra.

2009/6/24 Justice London <jlondon at lawinfo.com>

Here you go. Let me know if you need anything else:

Core was generated by `/usr/local/sbin/glusterfsd
-p /var/run/glusterfsd.pid -f /etc/glusterfs/gluster'.
Program terminated with signal 11, Segmentation fault.
[New process 653]
[New process 656]
[New process 687]
[New process 657]
[New process 658]
[New process 659]
[New process 660]
[New process 661]
[New process 662]
[New process 663]
[New process 665]
[New process 666]
[New process 667]
[New process 668]
[New process 669]
[New process 670]
[New process 671]
[New process 672]
[New process 679]
[New process 680]
[New process 681]
[New process 682]
[New process 683]
[New process 684]
[New process 686]
[New process 676]
[New process 685]
[New process 674]
[New process 675]
[New process 677]
[New process 654]
[New process 673]
[New process 678]
[New process 664]
#0  0xb808ee9c in __glusterfs_this_location at plt ()
  from /usr/local/lib/libglusterfs.so.0
(gdb) backtrace
#0  0xb808ee9c in __glusterfs_this_location at plt ()
  from /usr/local/lib/libglusterfs.so.0
#1  0xb809b935 in default_fxattrop (frame=0x809cc68, this=0x8055a80,
   fd=0x809ca20, flags=GF_XATTROP_ADD_ARRAY, dict=0x809cac8)
   at defaults.c:1122
#2  0xb809b930 in default_fxattrop (frame=0x8063570, this=0x8055f80,
   fd=0x809ca20, flags=GF_XATTROP_ADD_ARRAY, dict=0x809cac8)
   at defaults.c:1122
#3  0xb76b3c35 in server_fxattrop (frame=0x809cc28, bound_xl=0x8055f80,
   hdr=0x8064c88, hdrlen=150, iobuf=0x0) at server-protocol.c:4596
#4  0xb76a9f1b in protocol_server_interpret (this=0x8056500,
trans=0x8064698,
   hdr_p=0x8064c88 "", hdrlen=150, iobuf=0x0) at server-protocol.c:7502
#5  0xb76aa1cc in protocol_server_pollin (this=0x8056500,
trans=0x8064698)
   at server-protocol.c:7783
#6  0xb76aa24f in notify (this=0x8056500, event=2, data=0x8064698)
   at server-protocol.c:7839
#7  0xb809737f in xlator_notify (xl=0x8056500, event=2, data=0x8064698)
   at xlator.c:912
#8  0xb4ea08dd in socket_event_poll_in (this=0x8064698) at socket.c:713
#9  0xb4ea099b in socket_event_handler (fd=8, idx=1, data=0x8064698,
   poll_in=1, poll_out=0, poll_err=0) at socket.c:813
#10 0xb80b168a in event_dispatch_epoll (event_pool=0x8050d58) at
event.c:804
#11 0xb80b0471 in event_dispatch (event_pool=0x8051338) at event.c:975
---Type <return> to continue, or q <return> to quit---
#12 0x0804b880 in main (argc=5, argv=0xbfae1044) at glusterfsd.c:1263
Current language:  auto; currently asm



Justice London
jlondon at lawinfo.com

On Mon, 2009-06-22 at 10:47 +0530, Harshavardhana wrote:
> Hi Justice,
>
>      Can you get a backtrace from the segfault through gdb? .

>
> Regards
> --
> Harshavardhana
> Z Research Inc http://www.zresearch.com/
>
>

> On Sat, Jun 20, 2009 at 10:47 PM, <jlondon at lawinfo.com> wrote:
>         Sure, the kernel version is 2.6.29 and the fuse release is the
>         just
>         released 2.8.0-pre3 (although I can use pre2 if needed).
>
>
>         Justice London
>         jlondon at lawinfo.com
>
>         > Hi Justice,
>         >
>         >      There are certain modifications required in
>         fuse-extra.c to make
>         > glusterfs work properly for fuse 2.8.0 release. glusterfs
>         2.0.1 release is
>         > not tested against 2.8.0 release fuse and certainly will not
>         work without
>         > those modifications.  May i know the kernel version you are
>         trying to use?
>         > and the version of fuse being under use? pre1 or pre2
>         release?
>         >
>         > Regards
>         > --
>         > Harshavardhana
>         > Z Research Inc http://www.zresearch.com/
>         >
>         >
>         > On Fri, Jun 19, 2009 at 11:14 PM, Justice London
>         > <jlondon at lawinfo.com>wrote:
>         >
>         >>  No matter what I do I cannot seem to get gluster to stay
>         stable when
>         >> doing any sort of writes to the mount, when using gluster
>         in combination
>         >> with fuse 2.8.0-preX and NFS. I tried both unfs3 and
>         standard kernel-nfs
>         >> and
>         >> no matter what, any sort of data transaction seems to crash
>         gluster
>         >> immediately. The error log is as such:
>         >>
>         >>
>         >>
>         >> pending frames:
>         >>
>         >>
>         >>
>         >> patchset: git://git.sv.gnu.org/gluster.git
>         >>
>         >> signal received: 11
>         >>
>         >> configuration details:argp 1
>         >>
>         >> backtrace 1
>         >>
>         >> bdb->cursor->get 1
>         >>
>         >> db.h 1
>         >>
>         >> dlfcn 1
>         >>
>         >> fdatasync 1
>         >>
>         >> libpthread 1
>         >>
>         >> llistxattr 1
>         >>
>         >> setfsid 1
>         >>
>         >> spinlock 1
>         >>
>         >> epoll.h 1
>         >>
>         >> xattr.h 1
>         >>
>         >> st_atim.tv_nsec 1
>         >>
>         >> package-string: glusterfs 2.0.0git
>         >>
>         >> [0xf57fe400]
>         >>
>         >> /usr/local/lib/libglusterfs.so.0(default_fxattrop
>         +0xc0)[0xb7f4d530]
>         >>
>         >>
>         >>
/usr/local/lib/glusterfs/2.0.0git/xlator/protocol/server.so(server_fxattrop+
0x175)[0xb7565af5]
>         >>
>         >>
>         >>
/usr/local/lib/glusterfs/2.0.0git/xlator/protocol/server.so(protocol_server_
interpret+0xbb)[0xb755beeb]
>         >>
>         >>
>         >>
/usr/local/lib/glusterfs/2.0.0git/xlator/protocol/server.so(protocol_server_
pollin+0x9c)[0xb755c19c]
>         >>
>         >>
>         >>
/usr/local/lib/glusterfs/2.0.0git/xlator/protocol/server.so(notify+0x7f)[0xb
755c21f]
>         >>
>         >> /usr/local/lib/libglusterfs.so.0(xlator_notify
>         +0x3f)[0xb7f4937f]
>         >>
>         >>
>         >>
/usr/local/lib/glusterfs/2.0.0git/transport/socket.so(socket_event_poll_in+0
x3d)[0xb4d528dd]
>         >>
>         >>
>         >>
/usr/local/lib/glusterfs/2.0.0git/transport/socket.so(socket_event_handler+0
xab)[0xb4d5299b]
>         >>
>         >> /usr/local/lib/libglusterfs.so.0[0xb7f6321a]
>         >>
>         >> /usr/local/lib/libglusterfs.so.0(event_dispatch
>         +0x21)[0xb7f62001]
>         >>
>         >> /usr/local/sbin/glusterfsd(main+0xb3b)[0x804b81b]
>         >>
>         >> /lib/libc.so.6(__libc_start_main+0xe5)[0xb7df3455]
>         >>
>         >> /usr/local/sbin/glusterfsd[0x8049db1]
>         >>
>         >>
>         >>
>         >> Any ideas on if there is a solution, or will be one
>         upcoming in either
>         >> gluster or fuse?  Other than with NFS, the git version of
>         gluster seems
>         >> to
>         >> be really, really fast with fuse 2.8
>         >>
>         >>
>         >>
>         >> Justice London
>         >> jlondon at lawinfo.com
>         >>
>         >>
>         >>
>         >> _______________________________________________
>         >> Gluster-users mailing list
>         >> Gluster-users at gluster.org
>         >> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>         >>
>         >>
>         >
>
>
>
>
>



_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users




-- 
Raghavendra G

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.375 / Virus Database: 270.13.1/2211 - Release Date: 07/02/09
05:54:00

Checked by AVG - www.avg.com
Version: 8.5.375 / Virus Database: 270.13.7/2222 - Release Date: 07/07/09
05:53:00


Checked by AVG - www.avg.com
Version: 8.5.375 / Virus Database: 270.13.7/2222 - Release Date: 07/07/09
05:53:00


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090707/e117f902/attachment.html>


More information about the Gluster-users mailing list