[Gluster-devel] Mounting with file servers is failing very frequently (in every day)
Joseph Job
joseph at spectrum.net.in
Wed Oct 31 15:00:44 UTC 2007
Hi Raghavendra,
I would to get some tips and tricks regarding in which direction I
need to explore.
Let me brief by setup.
I have two web servers as glusterfs client and both of them are
accessing the Glusterfs server for file storage.
volume afrbricks
type cluster/afr
subvolumes client_214 client_213
option replicate *:2
option self-heal on
end-volume
The glusterfs servers and clients are identical in hardware side.
The servers are using Ultra SCSI HDD and the storage capacity is around 540GB
What are the things which I need to consider for performance improvement.
See the time difference in accessing the files...
root at w3-cok /storage/rvdoc# time ls -l | wc -l (From gluster client)
96514
real 11m51.644s
user 0m1.994s
sys 0m3.858s
root at fs4-cok /storage/rvdoc# time ls -l | wc -l (From gluster server)
96514
real 0m30.812s
user 0m1.542s
sys 0m2.442s
If you want any more info from my side, please inform...
Thanks,
Joseph
At 08:38 PM 10/30/2007, Joseph Job wrote:
>Hi Raghavendra,
>
>Thanks for your help, I would like to know the default value, how
>much seconds it is ?
>
>Thanks,
>Joseph
>
>At 05:07 PM 10/30/2007, Raghavendra G wrote:
>
>>Hi Joseph,
>>transport-timeout in clients is very less in your configuration (4
>>seconds). Use a higher timeout or just comment out the option to
>>use the default one, which should be sufficient.
>>
>>regards,
>>
>>On Oct 30, 2007 3:16 PM, Joseph Job
>><<mailto:joseph at spectrum.net.in>joseph at spectrum.net.in> wrote:
>>My Setup details..
>>Operating System : Trustix Secure Linux release 3.0.5 (Mirch Masala)
>>Kernel Version : 2.6.19.7-1
>>Gluster version :
>>glusterfs-server-1.3.6-1
>>glusterfs-client-1.3.6-1
>>glusterfs-common-1.3.6-1
>>glusterfs-devel-1.3.6-1
>>Working Mode.
>>Two web servers are there and both web servers are using Gluster file
>>server in TCP network for file access.
>>see my server side configuration...
>>
>>
>>Server 1
>>## Define the stroage
>>volume fs3-storage
>> type storage/posix # POSIX FS translator
>> option directory /storage # Export this directory
>>end-volume
>>volume iothreads #iothreads can give
>>performance a boost
>> type performance/io-threads
>> option thread-count 16
>> subvolumes fs3-storage
>>end-volume
>>## Add network serving capability to above brick.
>>volume server
>> type protocol/server
>> option transport-type tcp/server # For TCP/IP transport
>> option listen-port 6996 # Default is 6996
>> option client-volume-filename /var/log/glusterfs/client.vol
>> subvolumes iothreads
>> option auth.ip.iothreads.allow * # Allow access to "brick" volume
>>end-volume
>>Server 2
>>## Define the stroage
>>volume fs4-storage
>> type storage/posix # POSIX FS translator
>> option directory /storage # Export this directory
>>end-volume
>>volume iothreads #iothreads can give
>>performance a boost
>> type performance/io-threads
>> option thread-count 16
>> subvolumes fs4-storage
>>end-volume
>>## Add network serving capability to above brick.
>>volume server
>> type protocol/server
>> option transport-type tcp/server # For TCP/IP transport
>> option listen-port 6996 # Default is 6996
>> option client-volume-filename /var/log/glusterfs/client.vol
>> subvolumes iothreads
>> option auth.ip.iothreads.allow * # Allow access to "brick" volume
>>end-volume
>>
>>Client side configuration...
>>Client 1
>>### Add client feature and attach to remote subvolume
>>volume client_214
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> option remote-host <http://10.10.0.214>10.10.0.214 # IP
>> address of the remote brick
>> option remote-port 6996 # default server port is 6996
>> option remote-subvolume iothreads # name of the remote volume
>> option transport-timeout 4
>>end-volume
>>### Add client feature and attach to remote subvolume
>>volume client_213
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> option remote-host <http://10.10.0.213>10.10.0.213 # IP
>> address of the remote brick
>> option remote-port 6996 # default server port is 6996
>> option remote-subvolume iothreads # name of the remote volume
>> option transport-timeout 4
>>end-volume
>>volume afrbricks
>> type cluster/afr
>> subvolumes client_214 client_213
>> option replicate *:2
>> option self-heal on
>>end-volume
>>volume iothreads #iothreads can give performance a boost
>> type performance/io-threads
>> option thread-count 8
>> subvolumes afrbricks
>>end-volume
>>##########################
>>Client 2
>>Client 1
>>### Add client feature and attach to remote subvolume
>>volume client_214
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> option remote-host <http://10.10.0.214>10.10.0.214 # IP
>> address of the remote brick
>> option remote-port 6996 # default server port is 6996
>> option remote-subvolume iothreads # name of the remote volume
>> option transport-timeout 4
>>end-volume
>>### Add client feature and attach to remote subvolume
>>volume client_213
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> option remote-host <http://10.10.0.213>10.10.0.213 # IP
>> address of the remote brick
>> option remote-port 6996 # default server port is 6996
>> option remote-subvolume iothreads # name of the remote volume
>> option transport-timeout 4
>>end-volume
>>volume afrbricks
>> type cluster/afr
>> subvolumes client_214 client_213
>> option replicate *:2
>> option self-heal on
>>end-volume
>>volume iothreads #iothreads can give performance a boost
>> type performance/io-threads
>> option thread-count 8
>> subvolumes afrbricks
>>end-volume
>>##########################
>>
>>I am mounting the server to client with glusterfs -f
>>/etc/glusterfs/glusterfs- client.vol /storage/
>>I could able to mount, files are getting replicated to both file
>>servers. But the problem, the mounting is braking very frequently...
>>I am getting the error in glusterfs.log
>>2007-10-30 00:46:10 C [ tcp.c:81:tcp_disconnect] client_213:
>>connection disconnected
>>2007-10-30 00:46:18 C [client-protocol.c:218:call_bail] client_214:
>>bailing transport
>>2007-10-30 00:46:18 C [client-protocol.c:218:call_bail] client_213:
>>bailing transport
>>2007-10-30 00:46:18 C [tcp.c:81:tcp_disconnect] client_214:
>>connection disconnected
>>2007-10-30 00:46:18 C [tcp.c:81:tcp_disconnect] client_213:
>>connection disconnected
>>But the physical connection is still there...I can ping from client
>>to servers...
>>The server is using gigabit networking bonding.
>>I am using kernel with Fuse supported
>>root at w3-cok ~# lsmod
>>Module Size Used by
>>fuse 39444 2
>>ipv6 221344 24
>>tg3 105860 0
>>bonding 79224 0
>>jfs 163564 2
>>usbhid 35936 0
>>ohci_hcd 18564 0
>>usbcore 112772 3 usbhid,ohci_hcd
>>parport_pc 21956 0
>>parport 20032 1 parport_pc
>>shpchp 32416 0
>>serverworks 8840 0 [permanent]
>>cciss 54020 8
>>dm_mod 49432 0
>>sd_mod 17024 0
>>piix 9604 0 [permanent]
>>ide_disk 14336 0
>>ide_generic 2048 0 [permanent]
>>ide_core 106444 4 serverworks,piix,ide_disk,ide_generic
>>
>>Also see the glusterfsd.log in server side
>>2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
>>transport_writev failed
>>2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
>>error: Transport endpoint is not connected
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 E [protocol.c:253:gf_block_unserialize_transport]
>>server: EOF from peer ( <http://10.10.0.203:1018>10.10.0.203:1018)
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
>>transport_writev failed
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
>>error: Transport endpoint is not connected
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
>>transport_writev failed
>>2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
>>error: Transport endpoint is not connected
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 E [ server-protocol.c:197:generic_reply] server:
>>transport_writev failed
>>2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
>>error: Transport endpoint is not connected
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
>>transport_writev failed
>>2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
>>error: Transport endpoint is not connected
>>2007-10-30 00:48:52 E [server-protocol.c:197:generic_reply] server:
>>transport_writev failed
>>2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
>>error: Transport endpoint is not connected
>>2007-10-30 00:48:52 C [ tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:48:52 E [server-protocol.c:197:generic_reply] server:
>>transport_writev failed
>>2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
>>error: Transport endpoint is not connected
>>2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:48:52 E [ server-protocol.c:197:generic_reply] server:
>>transport_writev failed
>>2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
>>error: Transport endpoint is not connected
>>2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection
>>disconnected
>>
>>
>>JOSEPH JOB
>>Spectrum Softtech Solutions(P)Ltd.
>>MahaKavi G Road,
>>Karikkamuri Cross Road
>>Kochi-682011
>>0484-4082000
>><mailto:joseph at spectrum.net.in>joseph at spectrum.net.in
>>Visit at <http://www.spectrum.net.in>www.spectrum.net.in
>>
>>_______________________________________________
>>Gluster-devel mailing list
>><mailto:Gluster-devel at nongnu.org>Gluster-devel at nongnu.org
>>http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>>
>>
>>
>>--
>>Raghavendra G
>>
>>A centipede was happy quite, until a toad in fun,
>>Said, "Prey, which leg comes after which?",
>>This raised his doubts to such a pitch,
>>He fell flat into the ditch,
>>Not knowing how to run.
>>-Anonymous
>
>JOSEPH JOB
>Spectrum Softtech Solutions(P)Ltd.
>MahaKavi G Road,
>Karikkamuri Cross Road
>Kochi-682011
>0484-4082000
>joseph at spectrum.net.in
>Visit at www.spectrum.net.in
JOSEPH JOB
Spectrum Softtech Solutions(P)Ltd.
MahaKavi G Road,
Karikkamuri Cross Road
Kochi-682011
0484-4082000
joseph at spectrum.net.in
Visit at www.spectrum.net.in
More information about the Gluster-devel
mailing list