[Gluster-users] servers hang occasionally

Wed Dec 19 11:34:11 UTC 2012

Hello,

Occasionally one of the servers can hang and only hardware reboot helps. Kernel logs has nothing suspicious. I suspect glusterfs or underlying fs (xfs) to be the root of problem. 

Setup is following:

Volume Name: media
Type: Replicate
Volume ID: 30193bca-77b1-4749-8ebd-eb2b5eb72954
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 192.168.1.10:/rep-storage/media
Brick2: 192.168.1.11:/rep-storage/media
Options Reconfigured:
auth.allow: 127.0.0.1

Servers are both client and storage nodes.  There are millions of small files (10-15kb) and ~50000 medium files (3mb-200mb). Brick1 was more than week out of sync. After bringing it back to cluster forced healing via script:

find <gluster-mount> -noleaf -print0 | xargs --null stat >/dev/null

Because self-heal was soooo long. Sometimes Brick1 hangs, sometimes Brick2. Brick log before latest hang:
[2012-12-18 23:35:53.468696] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518489641: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:53.468989] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518489642: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:53.469589] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518489644: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:53.469883] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518489645: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.227722] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518491782: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.228093] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518491783: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.228701] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518491785: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.229005] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518491786: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.229617] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518491788: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.229914] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518491789: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.230549] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518491791: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.230833] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518491792: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.231480] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518491794: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:54.231767] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518491795: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:55.983687] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518495863: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:55.984175] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518495864: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:55.986082] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518495866: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:55.998903] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518495867: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.013851] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518495869: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.018956] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518495870: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.022077] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518495872: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.023275] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518495873: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.042616] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518495875: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.044837] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518495876: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.047869] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518495878: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.050436] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518495879: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.831462] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518497818: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.831829] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518497819: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.832480] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518497821: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.832793] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518497822: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.833413] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518497824: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.833707] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518497825: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.834370] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518497827: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.834765] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518497828: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.835420] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518497830: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:56.835749] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518497831: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:57.565329] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518499721: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:57.565666] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518499722: OPEN (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:57.566331] I [server3_1-fops.c:252:server_inodelk_cbk] 0-media-server: 518499724: INODELK (null) (--) ==> -1 (No such file or directory)
[2012-12-18 23:35:57.566656] I [server3_1-fops.c:1538:server_open_cbk] 0-media-server: 518499725: OPEN (null) (--) ==> -1 (No s

There's no error in the end - line is not finished in log file.

fstab:
UUID=241b1558-9e6c-4068-b8d7-b61577ed7b03 /rep-storage xfs defaults,noatime 1 2
127.0.0.1:/media /home/www/storage glusterfs defaults 1 2

Using CentOS 6.3 x64 distribution.
$  uname -a
Linux example.com 2.6.32-279.14.1.el6.x86_64 #1 SMP Tue Nov 6 23:43:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

What else info can be useful?

Thanks,
Alex Vasilenko
Skype: menterr

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121219/5b938548/attachment.html>