[Gluster-users] INFO: task "####" blocked for more than 120 seconds.

Roman romeo.r at gmail.com
Tue Sep 16 06:56:18 UTC 2014 

Hi

This morning we had a VM crash and these logs (see attc)

No logs on gluster servers nor virtual host where this is glusterfs mount
is attached.
Any ideas?



-- 
Best regards,
Roman.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140916/9c95b30b/attachment.html>
-------------- next part --------------
Sep 16 06:36:58 munin kernel: [1025760.825197] jbd2/vda1-8     D ffff88003fc93780     0   179      2 0x00000000
Sep 16 06:36:58 munin kernel: [1025760.825204]  ffff88003bb30740 0000000000000046 0000000000000000 ffff88003e216180
Sep 16 06:36:58 munin kernel: [1025760.825210]  0000000000013780 ffff880036d77fd8 ffff880036d77fd8 ffff88003bb30740
Sep 16 06:36:58 munin kernel: [1025760.825214]  ffff880036d76000 0000000181066311 ffff88003e1d5c20 ffff88003fc93fd0
Sep 16 06:36:58 munin kernel: [1025760.825218] Call Trace:
Sep 16 06:36:58 munin kernel: [1025760.825262]  [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28
Sep 16 06:36:58 munin kernel: [1025760.825282]  [<ffffffff8134f399>] ? io_schedule+0x59/0x71
Sep 16 06:36:58 munin kernel: [1025760.825286]  [<ffffffff8111da0f>] ? sleep_on_buffer+0x6/0xa
Sep 16 06:36:58 munin kernel: [1025760.825289]  [<ffffffff8134f7dc>] ? __wait_on_bit+0x3e/0x71
Sep 16 06:36:58 munin kernel: [1025760.825292]  [<ffffffff8134f87e>] ? out_of_line_wait_on_bit+0x6f/0x78
Sep 16 06:36:58 munin kernel: [1025760.825295]  [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28
Sep 16 06:36:58 munin kernel: [1025760.825311]  [<ffffffff8105fd7d>] ? autoremove_wake_function+0x2a/0x2a
Sep 16 06:36:58 munin kernel: [1025760.825389]  [<ffffffffa0115a4c>] ? jbd2_journal_commit_transaction+0xbeb/0x10bf [jbd2]
Sep 16 06:36:58 munin kernel: [1025760.825403]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Sep 16 06:36:58 munin kernel: [1025760.825407]  [<ffffffff8100d69e>] ? __switch_to+0x133/0x258
Sep 16 06:36:58 munin kernel: [1025760.825413]  [<ffffffffa0119156>] ? kjournald2+0xc0/0x20a [jbd2]
Sep 16 06:36:58 munin kernel: [1025760.825417]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:36:58 munin kernel: [1025760.825422]  [<ffffffffa0119096>] ? commit_timeout+0x5/0x5 [jbd2]
Sep 16 06:36:58 munin kernel: [1025760.825426]  [<ffffffff8105f701>] ? kthread+0x76/0x7e
Sep 16 06:36:58 munin kernel: [1025760.825436]  [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10
Sep 16 06:36:58 munin kernel: [1025760.825440]  [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139
Sep 16 06:36:58 munin kernel: [1025760.825444]  [<ffffffff813575b0>] ? gs_change+0x13/0x13
Sep 16 06:36:58 munin kernel: [1025760.826398] flush-254:0     D ffff88003fc13780     0  2672      2 0x00000000
Sep 16 06:36:58 munin kernel: [1025760.826410]  ffff880037075740 0000000000000046 0000000000000000 ffffffff8160d020
Sep 16 06:36:58 munin kernel: [1025760.826421]  0000000000013780 ffff88003b955fd8 ffff88003b955fd8 ffff880037075740
Sep 16 06:36:58 munin kernel: [1025760.826424]  ffff88003b955770 000000013b955770 ffff88003d424c90 ffff88003fc13fd0
Sep 16 06:36:58 munin kernel: [1025760.826428] Call Trace:
Sep 16 06:36:58 munin kernel: [1025760.826432]  [<ffffffff8134f399>] ? io_schedule+0x59/0x71
Sep 16 06:36:58 munin kernel: [1025760.826447]  [<ffffffff8119a2fc>] ? get_request_wait+0x105/0x18f
Sep 16 06:36:58 munin kernel: [1025760.826452]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:36:58 munin kernel: [1025760.826456]  [<ffffffff8119b25b>] ? blk_queue_bio+0x17f/0x28c
Sep 16 06:36:58 munin kernel: [1025760.826459]  [<ffffffff81199d88>] ? generic_make_request+0x90/0xcf
Sep 16 06:36:58 munin kernel: [1025760.826463]  [<ffffffff81199e9a>] ? submit_bio+0xd3/0xf1
Sep 16 06:36:58 munin kernel: [1025760.826473]  [<ffffffffa012dfad>] ? ext4_io_submit+0x21/0x4a [ext4]
Sep 16 06:36:58 munin kernel: [1025760.826481]  [<ffffffffa012ae52>] ? mpage_da_submit_io+0x359/0x36f [ext4]
Sep 16 06:36:58 munin kernel: [1025760.826488]  [<ffffffffa012cac7>] ? mpage_da_map_and_submit+0x2aa/0x2f9 [ext4]
Sep 16 06:36:58 munin kernel: [1025760.826495]  [<ffffffffa012bd80>] ? ext4_mark_inode_dirty+0x1af/0x1da [ext4]
Sep 16 06:36:58 munin kernel: [1025760.826502]  [<ffffffffa012cb00>] ? mpage_da_map_and_submit+0x2e3/0x2f9 [ext4]
Sep 16 06:36:58 munin kernel: [1025760.826509]  [<ffffffffa012d0e7>] ? ext4_da_writepages+0x239/0x45d [ext4]
Sep 16 06:36:58 munin kernel: [1025760.826519]  [<ffffffffa0142724>] ? ext4_journal_start_sb+0x139/0x14f [ext4]
Sep 16 06:36:58 munin kernel: [1025760.826526]  [<ffffffffa012d172>] ? ext4_da_writepages+0x2c4/0x45d [ext4]
Sep 16 06:36:58 munin kernel: [1025760.826533]  [<ffffffff81118ad3>] ? writeback_single_inode+0x11d/0x2cc
Sep 16 06:36:58 munin kernel: [1025760.826537]  [<ffffffff81118f53>] ? writeback_sb_inodes+0x16b/0x204
Sep 16 06:36:58 munin kernel: [1025760.826541]  [<ffffffff81119059>] ? __writeback_inodes_wb+0x6d/0xab
Sep 16 06:36:58 munin kernel: [1025760.826545]  [<ffffffff811191bf>] ? wb_writeback+0x128/0x21f
Sep 16 06:36:58 munin kernel: [1025760.826553]  [<ffffffff8107116d>] ? arch_local_irq_save+0x11/0x17
Sep 16 06:36:58 munin kernel: [1025760.826558]  [<ffffffff81119676>] ? wb_do_writeback+0x146/0x1a8
Sep 16 06:36:58 munin kernel: [1025760.826562]  [<ffffffff8111975d>] ? bdi_writeback_thread+0x85/0x1e6
Sep 16 06:36:58 munin kernel: [1025760.826565]  [<ffffffff811196d8>] ? wb_do_writeback+0x1a8/0x1a8
Sep 16 06:36:58 munin kernel: [1025760.826569]  [<ffffffff8105f701>] ? kthread+0x76/0x7e
Sep 16 06:36:58 munin kernel: [1025760.826573]  [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10
Sep 16 06:36:58 munin kernel: [1025760.826576]  [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139
Sep 16 06:36:58 munin kernel: [1025760.826579]  [<ffffffff813575b0>] ? gs_change+0x13/0x13
Sep 16 06:36:58 munin kernel: [1025760.827533] munin-graph     D ffff88003fc13780     0 29913  29580 0x00000000
Sep 16 06:36:58 munin kernel: [1025760.827537]  ffff88003d3a8140 0000000000000086 0000000000000000 ffffffff8160d020
Sep 16 06:36:58 munin kernel: [1025760.827540]  0000000000013780 ffff880036d2bfd8 ffff880036d2bfd8 ffff88003d3a8140
Sep 16 06:36:58 munin kernel: [1025760.827547]  0000000000000246 0000000181350461 ffffffffa016f508 ffff88002c9a4558
Sep 16 06:36:58 munin kernel: [1025760.827550] Call Trace:
Sep 16 06:36:58 munin kernel: [1025760.827559]  [<ffffffffa012dabc>] ? ext4_ioend_wait+0x82/0x98 [ext4]
Sep 16 06:36:58 munin kernel: [1025760.827563]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:36:58 munin kernel: [1025760.827571]  [<ffffffffa012d344>] ? ext4_evict_inode+0x39/0x2af [ext4]
Sep 16 06:36:58 munin kernel: [1025760.827577]  [<ffffffffa012d30b>] ? ext4_da_writepages+0x45d/0x45d [ext4]
Sep 16 06:36:58 munin kernel: [1025760.827589]  [<ffffffff8110d748>] ? evict+0x96/0x148
Sep 16 06:36:58 munin kernel: [1025760.827593]  [<ffffffff8110ab10>] ? dentry_kill+0x112/0x12e
Sep 16 06:36:58 munin kernel: [1025760.827595]  [<ffffffff8110ad62>] ? dput+0xe2/0xee
Sep 16 06:36:58 munin kernel: [1025760.827602]  [<ffffffff81106c71>] ? sys_renameat+0x1af/0x206
Sep 16 06:36:58 munin kernel: [1025760.827608]  [<ffffffff81126b2a>] ? fsnotify+0x1e6/0x20c
Sep 16 06:36:58 munin kernel: [1025760.827619]  [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a
Sep 16 06:36:58 munin kernel: [1025760.827625]  [<ffffffff81066311>] ? timekeeping_get_ns+0xd/0x2a
Sep 16 06:36:58 munin kernel: [1025760.827628]  [<ffffffff810668f5>] ? getnstimeofday+0x4a/0x7b
Sep 16 06:36:58 munin kernel: [1025760.827631]  [<ffffffff81066955>] ? do_gettimeofday+0x10/0x30
Sep 16 06:36:58 munin kernel: [1025760.827635]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
Sep 16 06:38:58 munin kernel: [1025880.825070] jbd2/vda1-8     D ffff88003fc93780     0   179      2 0x00000000
Sep 16 06:38:58 munin kernel: [1025880.825076]  ffff88003bb30740 0000000000000046 0000000000000000 ffff88003e216180
Sep 16 06:38:58 munin kernel: [1025880.825081]  0000000000013780 ffff880036d77fd8 ffff880036d77fd8 ffff88003bb30740
Sep 16 06:38:58 munin kernel: [1025880.825084]  ffff880036d76000 0000000181066311 ffff88003e1d5c20 ffff88003fc93fd0
Sep 16 06:38:58 munin kernel: [1025880.825089] Call Trace:
Sep 16 06:38:58 munin kernel: [1025880.825102]  [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28
Sep 16 06:38:58 munin kernel: [1025880.825117]  [<ffffffff8134f399>] ? io_schedule+0x59/0x71
Sep 16 06:38:58 munin kernel: [1025880.825120]  [<ffffffff8111da0f>] ? sleep_on_buffer+0x6/0xa
Sep 16 06:38:58 munin kernel: [1025880.825123]  [<ffffffff8134f7dc>] ? __wait_on_bit+0x3e/0x71
Sep 16 06:38:58 munin kernel: [1025880.825126]  [<ffffffff8134f87e>] ? out_of_line_wait_on_bit+0x6f/0x78
Sep 16 06:38:58 munin kernel: [1025880.825130]  [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28
Sep 16 06:38:58 munin kernel: [1025880.825137]  [<ffffffff8105fd7d>] ? autoremove_wake_function+0x2a/0x2a
Sep 16 06:38:58 munin kernel: [1025880.825158]  [<ffffffffa0115a4c>] ? jbd2_journal_commit_transaction+0xbeb/0x10bf [jbd2]
Sep 16 06:38:58 munin kernel: [1025880.825165]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Sep 16 06:38:58 munin kernel: [1025880.825168]  [<ffffffff8100d69e>] ? __switch_to+0x133/0x258
Sep 16 06:38:58 munin kernel: [1025880.825174]  [<ffffffffa0119156>] ? kjournald2+0xc0/0x20a [jbd2]
Sep 16 06:38:58 munin kernel: [1025880.825178]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:38:58 munin kernel: [1025880.825183]  [<ffffffffa0119096>] ? commit_timeout+0x5/0x5 [jbd2]
Sep 16 06:38:58 munin kernel: [1025880.825187]  [<ffffffff8105f701>] ? kthread+0x76/0x7e
Sep 16 06:38:58 munin kernel: [1025880.825193]  [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10
Sep 16 06:38:58 munin kernel: [1025880.825197]  [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139
Sep 16 06:38:58 munin kernel: [1025880.825200]  [<ffffffff813575b0>] ? gs_change+0x13/0x13
Sep 16 06:38:58 munin kernel: [1025880.826146] flush-254:0     D ffff88003fc13780     0  2672      2 0x00000000
Sep 16 06:38:58 munin kernel: [1025880.826150]  ffff880037075740 0000000000000046 0000000000000000 ffffffff8160d020
Sep 16 06:38:58 munin kernel: [1025880.826154]  0000000000013780 ffff88003b955fd8 ffff88003b955fd8 ffff880037075740
Sep 16 06:38:58 munin kernel: [1025880.826157]  ffff88003b955770 000000013b955770 ffff88003d424c90 ffff88003fc13fd0
Sep 16 06:38:58 munin kernel: [1025880.826161] Call Trace:
Sep 16 06:38:58 munin kernel: [1025880.826165]  [<ffffffff8134f399>] ? io_schedule+0x59/0x71
Sep 16 06:38:58 munin kernel: [1025880.826171]  [<ffffffff8119a2fc>] ? get_request_wait+0x105/0x18f
Sep 16 06:38:58 munin kernel: [1025880.826175]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:38:58 munin kernel: [1025880.826179]  [<ffffffff8119b25b>] ? blk_queue_bio+0x17f/0x28c
Sep 16 06:38:58 munin kernel: [1025880.826182]  [<ffffffff81199d88>] ? generic_make_request+0x90/0xcf
Sep 16 06:38:58 munin kernel: [1025880.826186]  [<ffffffff81199e9a>] ? submit_bio+0xd3/0xf1
Sep 16 06:38:58 munin kernel: [1025880.826196]  [<ffffffffa012dfad>] ? ext4_io_submit+0x21/0x4a [ext4]
Sep 16 06:38:58 munin kernel: [1025880.826204]  [<ffffffffa012ae52>] ? mpage_da_submit_io+0x359/0x36f [ext4]
Sep 16 06:38:58 munin kernel: [1025880.826212]  [<ffffffffa012cac7>] ? mpage_da_map_and_submit+0x2aa/0x2f9 [ext4]
Sep 16 06:38:58 munin kernel: [1025880.826219]  [<ffffffffa012bd80>] ? ext4_mark_inode_dirty+0x1af/0x1da [ext4]
Sep 16 06:38:58 munin kernel: [1025880.826226]  [<ffffffffa012cb00>] ? mpage_da_map_and_submit+0x2e3/0x2f9 [ext4]
Sep 16 06:38:58 munin kernel: [1025880.826232]  [<ffffffffa012d0e7>] ? ext4_da_writepages+0x239/0x45d [ext4]
Sep 16 06:38:58 munin kernel: [1025880.826242]  [<ffffffffa0142724>] ? ext4_journal_start_sb+0x139/0x14f [ext4]
Sep 16 06:38:58 munin kernel: [1025880.826249]  [<ffffffffa012d172>] ? ext4_da_writepages+0x2c4/0x45d [ext4]
Sep 16 06:38:58 munin kernel: [1025880.826266]  [<ffffffff81118ad3>] ? writeback_single_inode+0x11d/0x2cc
Sep 16 06:38:58 munin kernel: [1025880.826270]  [<ffffffff81118f53>] ? writeback_sb_inodes+0x16b/0x204
Sep 16 06:38:58 munin kernel: [1025880.826274]  [<ffffffff81119059>] ? __writeback_inodes_wb+0x6d/0xab
Sep 16 06:38:58 munin kernel: [1025880.826278]  [<ffffffff811191bf>] ? wb_writeback+0x128/0x21f
Sep 16 06:38:58 munin kernel: [1025880.826283]  [<ffffffff8107116d>] ? arch_local_irq_save+0x11/0x17
Sep 16 06:38:58 munin kernel: [1025880.826287]  [<ffffffff81119676>] ? wb_do_writeback+0x146/0x1a8
Sep 16 06:38:58 munin kernel: [1025880.826291]  [<ffffffff8111975d>] ? bdi_writeback_thread+0x85/0x1e6
Sep 16 06:38:58 munin kernel: [1025880.826295]  [<ffffffff811196d8>] ? wb_do_writeback+0x1a8/0x1a8
Sep 16 06:38:58 munin kernel: [1025880.826298]  [<ffffffff8105f701>] ? kthread+0x76/0x7e
Sep 16 06:38:58 munin kernel: [1025880.826302]  [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10
Sep 16 06:38:58 munin kernel: [1025880.826305]  [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139
Sep 16 06:38:58 munin kernel: [1025880.826308]  [<ffffffff813575b0>] ? gs_change+0x13/0x13
Sep 16 06:38:58 munin kernel: [1025880.827272] munin-graph     D ffff88003fc13780     0 29913  29580 0x00000000
Sep 16 06:38:58 munin kernel: [1025880.827276]  ffff88003d3a8140 0000000000000086 0000000000000000 ffffffff8160d020
Sep 16 06:38:58 munin kernel: [1025880.827279]  0000000000013780 ffff880036d2bfd8 ffff880036d2bfd8 ffff88003d3a8140
Sep 16 06:38:58 munin kernel: [1025880.827283]  0000000000000246 0000000181350461 ffffffffa016f508 ffff88002c9a4558
Sep 16 06:38:58 munin kernel: [1025880.827287] Call Trace:
Sep 16 06:38:58 munin kernel: [1025880.827295]  [<ffffffffa012dabc>] ? ext4_ioend_wait+0x82/0x98 [ext4]
Sep 16 06:38:58 munin kernel: [1025880.827299]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:38:58 munin kernel: [1025880.827307]  [<ffffffffa012d344>] ? ext4_evict_inode+0x39/0x2af [ext4]
Sep 16 06:38:58 munin kernel: [1025880.827313]  [<ffffffffa012d30b>] ? ext4_da_writepages+0x45d/0x45d [ext4]
Sep 16 06:38:58 munin kernel: [1025880.827319]  [<ffffffff8110d748>] ? evict+0x96/0x148
Sep 16 06:38:58 munin kernel: [1025880.827322]  [<ffffffff8110ab10>] ? dentry_kill+0x112/0x12e
Sep 16 06:38:58 munin kernel: [1025880.827325]  [<ffffffff8110ad62>] ? dput+0xe2/0xee
Sep 16 06:38:58 munin kernel: [1025880.827330]  [<ffffffff81106c71>] ? sys_renameat+0x1af/0x206
Sep 16 06:38:58 munin kernel: [1025880.827335]  [<ffffffff81126b2a>] ? fsnotify+0x1e6/0x20c
Sep 16 06:38:58 munin kernel: [1025880.827340]  [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a
Sep 16 06:38:58 munin kernel: [1025880.827344]  [<ffffffff81066311>] ? timekeeping_get_ns+0xd/0x2a
Sep 16 06:38:58 munin kernel: [1025880.827347]  [<ffffffff810668f5>] ? getnstimeofday+0x4a/0x7b
Sep 16 06:38:58 munin kernel: [1025880.827350]  [<ffffffff81066955>] ? do_gettimeofday+0x10/0x30
Sep 16 06:38:58 munin kernel: [1025880.827354]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
Sep 16 06:38:58 munin kernel: [1025880.829214] munin-update    D ffff88003fc93780     0 29920  29919 0x00000000
Sep 16 06:38:58 munin kernel: [1025880.829220]  ffff8800370cf800 0000000000000082 0000000000000000 ffff88003e216180
Sep 16 06:38:58 munin kernel: [1025880.829227]  0000000000013780 ffff88003712ffd8 ffff88003712ffd8 ffff8800370cf800
Sep 16 06:38:58 munin kernel: [1025880.829233]  0000000000000005 000000013712fe68 ffff88003712fd68 ffff88003dd068e0
Sep 16 06:38:58 munin kernel: [1025880.829239] Call Trace:
Sep 16 06:38:58 munin kernel: [1025880.829245]  [<ffffffff8134fa44>] ? __mutex_lock_common.isra.5+0xff/0x164
Sep 16 06:38:58 munin kernel: [1025880.829250]  [<ffffffff8134f932>] ? mutex_lock+0x1a/0x2d
Sep 16 06:38:58 munin kernel: [1025880.829256]  [<ffffffff8110576b>] ? do_last+0x1b1/0x58d
Sep 16 06:38:58 munin kernel: [1025880.829261]  [<ffffffff8110613f>] ? path_openat+0xce/0x33a
Sep 16 06:38:58 munin kernel: [1025880.829277]  [<ffffffff810fa734>] ? do_sync_write+0xb4/0xec
Sep 16 06:38:58 munin kernel: [1025880.829282]  [<ffffffff8110646d>] ? do_filp_open+0x2a/0x6e
Sep 16 06:38:58 munin kernel: [1025880.829287]  [<ffffffff8134f144>] ? _cond_resched+0x7/0x1c
Sep 16 06:38:58 munin kernel: [1025880.829307]  [<ffffffff811b4f49>] ? __strncpy_from_user+0x18/0x48
Sep 16 06:38:58 munin kernel: [1025880.829314]  [<ffffffff8110f1f3>] ? alloc_fd+0x64/0x109
Sep 16 06:38:58 munin kernel: [1025880.829318]  [<ffffffff810fa439>] ? do_sys_open+0x5e/0xe5
Sep 16 06:38:58 munin kernel: [1025880.829323]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
Sep 16 06:40:58 munin kernel: [1026000.830447] jbd2/vda1-8     D ffff88003fc93780     0   179      2 0x00000000
Sep 16 06:40:58 munin kernel: [1026000.830454]  ffff88003bb30740 0000000000000046 0000000000000000 ffff88003e216180
Sep 16 06:40:58 munin kernel: [1026000.830467]  0000000000013780 ffff880036d77fd8 ffff880036d77fd8 ffff88003bb30740
Sep 16 06:40:58 munin kernel: [1026000.830471]  ffff880036d76000 0000000181066311 ffff88003e1d5c20 ffff88003fc93fd0
Sep 16 06:40:58 munin kernel: [1026000.830476] Call Trace:
Sep 16 06:40:58 munin kernel: [1026000.830491]  [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28
Sep 16 06:40:58 munin kernel: [1026000.830498]  [<ffffffff8134f399>] ? io_schedule+0x59/0x71
Sep 16 06:40:58 munin kernel: [1026000.830501]  [<ffffffff8111da0f>] ? sleep_on_buffer+0x6/0xa
Sep 16 06:40:58 munin kernel: [1026000.830504]  [<ffffffff8134f7dc>] ? __wait_on_bit+0x3e/0x71
Sep 16 06:40:58 munin kernel: [1026000.830508]  [<ffffffff8134f87e>] ? out_of_line_wait_on_bit+0x6f/0x78
Sep 16 06:40:58 munin kernel: [1026000.830511]  [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28
Sep 16 06:40:58 munin kernel: [1026000.830518]  [<ffffffff8105fd7d>] ? autoremove_wake_function+0x2a/0x2a
Sep 16 06:40:58 munin kernel: [1026000.830539]  [<ffffffffa0115a4c>] ? jbd2_journal_commit_transaction+0xbeb/0x10bf [jbd2]
Sep 16 06:40:58 munin kernel: [1026000.830546]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Sep 16 06:40:58 munin kernel: [1026000.830550]  [<ffffffff8100d69e>] ? __switch_to+0x133/0x258
Sep 16 06:40:58 munin kernel: [1026000.830556]  [<ffffffffa0119156>] ? kjournald2+0xc0/0x20a [jbd2]
Sep 16 06:40:58 munin kernel: [1026000.830560]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:40:58 munin kernel: [1026000.830565]  [<ffffffffa0119096>] ? commit_timeout+0x5/0x5 [jbd2]
Sep 16 06:40:58 munin kernel: [1026000.830569]  [<ffffffff8105f701>] ? kthread+0x76/0x7e
Sep 16 06:40:58 munin kernel: [1026000.830575]  [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10
Sep 16 06:40:58 munin kernel: [1026000.830579]  [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139
Sep 16 06:40:58 munin kernel: [1026000.830582]  [<ffffffff813575b0>] ? gs_change+0x13/0x13
Sep 16 06:40:58 munin kernel: [1026000.832915] flush-254:0     D ffff88003fc13780     0  2672      2 0x00000000
Sep 16 06:40:58 munin kernel: [1026000.832915]  ffff880037075740 0000000000000046 0000000000000000 ffffffff8160d020
Sep 16 06:40:58 munin kernel: [1026000.832919]  0000000000013780 ffff88003b955fd8 ffff88003b955fd8 ffff880037075740
Sep 16 06:40:58 munin kernel: [1026000.832925]  ffff88003b955770 000000013b955770 ffff88003d424c90 ffff88003fc13fd0
Sep 16 06:40:58 munin kernel: [1026000.832930] Call Trace:
Sep 16 06:40:58 munin kernel: [1026000.832936]  [<ffffffff8134f399>] ? io_schedule+0x59/0x71
Sep 16 06:40:58 munin kernel: [1026000.832944]  [<ffffffff8119a2fc>] ? get_request_wait+0x105/0x18f
Sep 16 06:40:58 munin kernel: [1026000.832950]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:40:58 munin kernel: [1026000.832956]  [<ffffffff8119b25b>] ? blk_queue_bio+0x17f/0x28c
Sep 16 06:40:58 munin kernel: [1026000.832961]  [<ffffffff81199d88>] ? generic_make_request+0x90/0xcf
Sep 16 06:40:58 munin kernel: [1026000.832966]  [<ffffffff81199e9a>] ? submit_bio+0xd3/0xf1
Sep 16 06:40:58 munin kernel: [1026000.832981]  [<ffffffffa012dfad>] ? ext4_io_submit+0x21/0x4a [ext4]
Sep 16 06:40:58 munin kernel: [1026000.832991]  [<ffffffffa012ae52>] ? mpage_da_submit_io+0x359/0x36f [ext4]
Sep 16 06:40:58 munin kernel: [1026000.833002]  [<ffffffffa012cac7>] ? mpage_da_map_and_submit+0x2aa/0x2f9 [ext4]
Sep 16 06:40:58 munin kernel: [1026000.833012]  [<ffffffffa012bd80>] ? ext4_mark_inode_dirty+0x1af/0x1da [ext4]
Sep 16 06:40:58 munin kernel: [1026000.833022]  [<ffffffffa012cb00>] ? mpage_da_map_and_submit+0x2e3/0x2f9 [ext4]
Sep 16 06:40:58 munin kernel: [1026000.833031]  [<ffffffffa012d0e7>] ? ext4_da_writepages+0x239/0x45d [ext4]
Sep 16 06:40:58 munin kernel: [1026000.833045]  [<ffffffffa0142724>] ? ext4_journal_start_sb+0x139/0x14f [ext4]
Sep 16 06:40:58 munin kernel: [1026000.833055]  [<ffffffffa012d172>] ? ext4_da_writepages+0x2c4/0x45d [ext4]
Sep 16 06:40:58 munin kernel: [1026000.833064]  [<ffffffff81118ad3>] ? writeback_single_inode+0x11d/0x2cc
Sep 16 06:40:58 munin kernel: [1026000.833070]  [<ffffffff81118f53>] ? writeback_sb_inodes+0x16b/0x204
Sep 16 06:40:58 munin kernel: [1026000.833076]  [<ffffffff81119059>] ? __writeback_inodes_wb+0x6d/0xab
Sep 16 06:40:58 munin kernel: [1026000.833081]  [<ffffffff811191bf>] ? wb_writeback+0x128/0x21f
Sep 16 06:40:58 munin kernel: [1026000.833088]  [<ffffffff8107116d>] ? arch_local_irq_save+0x11/0x17
Sep 16 06:40:58 munin kernel: [1026000.833093]  [<ffffffff81119676>] ? wb_do_writeback+0x146/0x1a8
Sep 16 06:40:58 munin kernel: [1026000.833098]  [<ffffffff8111975d>] ? bdi_writeback_thread+0x85/0x1e6
Sep 16 06:40:58 munin kernel: [1026000.833104]  [<ffffffff811196d8>] ? wb_do_writeback+0x1a8/0x1a8
Sep 16 06:40:58 munin kernel: [1026000.833109]  [<ffffffff8105f701>] ? kthread+0x76/0x7e
Sep 16 06:40:58 munin kernel: [1026000.833114]  [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10
Sep 16 06:40:58 munin kernel: [1026000.833119]  [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139
Sep 16 06:40:58 munin kernel: [1026000.833124]  [<ffffffff813575b0>] ? gs_change+0x13/0x13
Sep 16 06:40:58 munin kernel: [1026000.835590] munin-graph     D ffff88003fc13780     0 29913  29580 0x00000000
Sep 16 06:40:58 munin kernel: [1026000.835597]  ffff88003d3a8140 0000000000000086 0000000000000000 ffffffff8160d020
Sep 16 06:40:58 munin kernel: [1026000.835601]  0000000000013780 ffff880036d2bfd8 ffff880036d2bfd8 ffff88003d3a8140
Sep 16 06:40:58 munin kernel: [1026000.835604]  0000000000000246 0000000181350461 ffffffffa016f508 ffff88002c9a4558
Sep 16 06:40:58 munin kernel: [1026000.835608] Call Trace:
Sep 16 06:40:58 munin kernel: [1026000.835618]  [<ffffffffa012dabc>] ? ext4_ioend_wait+0x82/0x98 [ext4]
Sep 16 06:40:58 munin kernel: [1026000.835628]  [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c
Sep 16 06:40:58 munin kernel: [1026000.835636]  [<ffffffffa012d344>] ? ext4_evict_inode+0x39/0x2af [ext4]
Sep 16 06:40:58 munin kernel: [1026000.835643]  [<ffffffffa012d30b>] ? ext4_da_writepages+0x45d/0x45d [ext4]
Sep 16 06:40:58 munin kernel: [1026000.835649]  [<ffffffff8110d748>] ? evict+0x96/0x148
Sep 16 06:40:58 munin kernel: [1026000.835652]  [<ffffffff8110ab10>] ? dentry_kill+0x112/0x12e
Sep 16 06:40:58 munin kernel: [1026000.835655]  [<ffffffff8110ad62>] ? dput+0xe2/0xee
Sep 16 06:40:58 munin kernel: [1026000.835660]  [<ffffffff81106c71>] ? sys_renameat+0x1af/0x206
Sep 16 06:40:58 munin kernel: [1026000.835665]  [<ffffffff81126b2a>] ? fsnotify+0x1e6/0x20c
Sep 16 06:40:58 munin kernel: [1026000.835670]  [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a
Sep 16 06:40:58 munin kernel: [1026000.835674]  [<ffffffff81066311>] ? timekeeping_get_ns+0xd/0x2a
Sep 16 06:40:58 munin kernel: [1026000.835677]  [<ffffffff810668f5>] ? getnstimeofday+0x4a/0x7b
Sep 16 06:40:58 munin kernel: [1026000.835680]  [<ffffffff81066955>] ? do_gettimeofday+0x10/0x30
Sep 16 06:40:58 munin kernel: [1026000.835684]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
-------------- next part --------------
Sep 16 06:36:58 munin kernel: [1025760.824197] INFO: task jbd2/vda1-8:179 blocked for more than 120 seconds.
Sep 16 06:36:58 munin kernel: [1025760.824699] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 16 06:36:58 munin kernel: [1025760.825461] INFO: task flush-254:0:2672 blocked for more than 120 seconds.
Sep 16 06:36:58 munin kernel: [1025760.825909] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 16 06:36:58 munin kernel: [1025760.826597] INFO: task munin-graph:29913 blocked for more than 120 seconds.
Sep 16 06:36:58 munin kernel: [1025760.827045] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 16 06:38:58 munin kernel: [1025880.824119] INFO: task jbd2/vda1-8:179 blocked for more than 120 seconds.
Sep 16 06:38:58 munin kernel: [1025880.824580] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

More information about the Gluster-users mailing list