[Gluster-users] [ovirt-users] Re: Gluster problems, cluster performance issues

Ravishankar N ravishankar at redhat.com
Wed May 30 04:43:55 UTC 2018


@Jim Kusznir

For the heal issue, can you provide the getfattr output of one of the 8 
files in question from all 3 bricks?
Example: `getfattr -d -m . -e hex 
/gluster/brick3/data-hdd/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/48d7ecb8-7ac5-4725-bca5-b3519681cf2f/0d6080b0-7018-4fa3-bb82-1dd9ef07d9b9`
Also provide the stat output of the same file from all 3 bricks.

Thanks,
Ravi



On 05/30/2018 09:47 AM, Krutika Dhananjay wrote:
> Adding Ravi to look into the heal issue.
>
> As for the fsync hang and subsequent IO errors, it seems a lot like 
> https://bugzilla.redhat.com/show_bug.cgi?id=1497156 and Paolo Bonzini 
> from qemu had pointed out that this would be fixed by the following 
> commit:
>
>    commit e72c9a2a67a6400c8ef3d01d4c461dbbbfa0e1f0
>      Author: Paolo Bonzini <pbonzini at redhat.com <mailto:pbonzini at redhat.com>>
>      Date:   Wed Jun 21 16:35:46 2017 +0200
>
>      scsi: virtio_scsi: let host do exception handling
>      
>      virtio_scsi tries to do exception handling after the default 30 seconds
>      timeout expires.  However, it's better to let the host control the
>      timeout, otherwise with a heavy I/O load it is likely that an abort will
>      also timeout.  This leads to fatal errors like filesystems going
>      offline.
>      
>      Disable the 'sd' timeout and allow the host to do exception handling,
>      following the precedent of the storvsc driver.
>      
>      Hannes has a proposal to introduce timeouts in virtio, but this provides
>      an immediate solution for stable kernels too.
>      
>      [mkp: fixed typo]
>      
>      Reported-by: Douglas Miller <dougmill at linux.vnet.ibm.com <mailto:dougmill at linux.vnet.ibm.com>>
>      Cc: "James E.J. Bottomley" <jejb at linux.vnet.ibm.com <mailto:jejb at linux.vnet.ibm.com>>
>      Cc: "Martin K. Petersen" <martin.petersen at oracle.com <mailto:martin.petersen at oracle.com>>
>      Cc: Hannes Reinecke <hare at suse.de <mailto:hare at suse.de>>
>      Cc:linux-scsi at vger.kernel.org <mailto:linux-scsi at vger.kernel.org>
>      Cc:stable at vger.kernel.org <mailto:stable at vger.kernel.org>
>      Signed-off-by: Paolo Bonzini <pbonzini at redhat.com <mailto:pbonzini at redhat.com>>
>      Signed-off-by: Martin K. Petersen <martin.petersen at oracle.com <mailto:martin.petersen at oracle.com>>
>
> Adding Paolo/Kevin to comment.
>
> As for the poor gluster performance, could you disable 
> cluster.eager-lock and see if that makes any difference:
>
> # gluster volume set <VOL> cluster.eager-lock off
>
> Do also capture the volume profile again if you still see performance 
> issues after disabling eager-lock.
>
> -Krutika
>
>
> On Wed, May 30, 2018 at 6:55 AM, Jim Kusznir <jim at palousetech.com 
> <mailto:jim at palousetech.com>> wrote:
>
>     I also finally found the following in my system log on one server:
>
>     [10679.524491] INFO: task glusterclogro:14933 blocked for more
>     than 120 seconds.
>     [10679.525826] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [10679.527144] glusterclogro   D ffff97209832bf40    0 14933     
>     1 0x00000080
>     [10679.527150] Call Trace:
>     [10679.527161]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [10679.527218]  [<ffffffffc060e388>]
>     _xfs_log_force_lsn+0x2e8/0x340 [xfs]
>     [10679.527225]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [10679.527254]  [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>     [10679.527260]  [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>     [10679.527268]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [10679.527271]  [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20
>     [10679.527275]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [10679.527279]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [10679.527283] INFO: task glusterposixfsy:14941 blocked for more
>     than 120 seconds.
>     [10679.528608] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [10679.529956] glusterposixfsy D ffff972495f84f10    0 14941     
>     1 0x00000080
>     [10679.529961] Call Trace:
>     [10679.529966]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [10679.530003]  [<ffffffffc060e388>]
>     _xfs_log_force_lsn+0x2e8/0x340 [xfs]
>     [10679.530008]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [10679.530038]  [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>     [10679.530042]  [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>     [10679.530046]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [10679.530050]  [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20
>     [10679.530054]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [10679.530058]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [10679.530062] INFO: task glusteriotwr13:15486 blocked for more
>     than 120 seconds.
>     [10679.531805] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [10679.533732] glusteriotwr13  D ffff9720a83f0000    0 15486     
>     1 0x00000080
>     [10679.533738] Call Trace:
>     [10679.533747]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [10679.533799]  [<ffffffffc060e388>]
>     _xfs_log_force_lsn+0x2e8/0x340 [xfs]
>     [10679.533806]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [10679.533846]  [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>     [10679.533852]  [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>     [10679.533858]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [10679.533863]  [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20
>     [10679.533868]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [10679.533873]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [10919.512757] INFO: task glusterclogro:14933 blocked for more
>     than 120 seconds.
>     [10919.514714] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [10919.516663] glusterclogro   D ffff97209832bf40    0 14933     
>     1 0x00000080
>     [10919.516677] Call Trace:
>     [10919.516690]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [10919.516696]  [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0
>     [10919.516703]  [<ffffffffb951cc04>] ? blk_finish_plug+0x14/0x40
>     [10919.516768]  [<ffffffffc05e9224>] ?
>     _xfs_buf_ioapply+0x334/0x460 [xfs]
>     [10919.516774]  [<ffffffffb991432d>] wait_for_completion+0xfd/0x140
>     [10919.516782]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [10919.516821]  [<ffffffffc05eb0a3>] ? _xfs_buf_read+0x23/0x40 [xfs]
>     [10919.516859]  [<ffffffffc05eafa9>]
>     xfs_buf_submit_wait+0xf9/0x1d0 [xfs]
>     [10919.516902]  [<ffffffffc061b279>] ?
>     xfs_trans_read_buf_map+0x199/0x400 [xfs]
>     [10919.516940]  [<ffffffffc05eb0a3>] _xfs_buf_read+0x23/0x40 [xfs]
>     [10919.516977]  [<ffffffffc05eb1b9>] xfs_buf_read_map+0xf9/0x160 [xfs]
>     [10919.517022]  [<ffffffffc061b279>]
>     xfs_trans_read_buf_map+0x199/0x400 [xfs]
>     [10919.517057]  [<ffffffffc05c8d04>] xfs_da_read_buf+0xd4/0x100 [xfs]
>     [10919.517091]  [<ffffffffc05c8d53>] xfs_da3_node_read+0x23/0xd0 [xfs]
>     [10919.517126]  [<ffffffffc05c9fee>]
>     xfs_da3_node_lookup_int+0x6e/0x2f0 [xfs]
>     [10919.517160]  [<ffffffffc05d5a1d>]
>     xfs_dir2_node_lookup+0x4d/0x170 [xfs]
>     [10919.517194]  [<ffffffffc05ccf5d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs]
>     [10919.517233]  [<ffffffffc05fd8d9>] xfs_lookup+0x69/0x140 [xfs]
>     [10919.517271]  [<ffffffffc05fa018>] xfs_vn_lookup+0x78/0xc0 [xfs]
>     [10919.517278]  [<ffffffffb9425cf3>] lookup_real+0x23/0x60
>     [10919.517283]  [<ffffffffb9426702>] __lookup_hash+0x42/0x60
>     [10919.517288]  [<ffffffffb942d519>] SYSC_renameat2+0x3a9/0x5a0
>     [10919.517296]  [<ffffffffb94d3753>] ?
>     selinux_file_free_security+0x23/0x30
>     [10919.517304]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [10919.517309]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [10919.517313]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [10919.517318]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [10919.517323]  [<ffffffffb942e58e>] SyS_renameat2+0xe/0x10
>     [10919.517328]  [<ffffffffb942e5ce>] SyS_rename+0x1e/0x20
>     [10919.517333]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [10919.517339]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [11159.496095] INFO: task glusteriotwr9:15482 blocked for more
>     than 120 seconds.
>     [11159.497546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [11159.498978] glusteriotwr9   D ffff971fa0fa1fa0    0 15482     
>     1 0x00000080
>     [11159.498984] Call Trace:
>     [11159.498995]  [<ffffffffb9911f00>] ? bit_wait+0x50/0x50
>     [11159.498999]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [11159.499003]  [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0
>     [11159.499056]  [<ffffffffc05dd9b7>] ?
>     xfs_iext_bno_to_ext+0xa7/0x1a0 [xfs]
>     [11159.499082]  [<ffffffffc05dd43e>] ?
>     xfs_iext_bno_to_irec+0x8e/0xd0 [xfs]
>     [11159.499090]  [<ffffffffb92f7a12>] ? ktime_get_ts64+0x52/0xf0
>     [11159.499093]  [<ffffffffb9911f00>] ? bit_wait+0x50/0x50
>     [11159.499097]  [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130
>     [11159.499101]  [<ffffffffb9913528>] io_schedule+0x18/0x20
>     [11159.499104]  [<ffffffffb9911f11>] bit_wait_io+0x11/0x50
>     [11159.499107]  [<ffffffffb9911ac1>] __wait_on_bit_lock+0x61/0xc0
>     [11159.499113]  [<ffffffffb9393634>] __lock_page+0x74/0x90
>     [11159.499118]  [<ffffffffb92bc210>] ? wake_bit_function+0x40/0x40
>     [11159.499121]  [<ffffffffb9394154>] __find_lock_page+0x54/0x70
>     [11159.499125]  [<ffffffffb9394e85>]
>     grab_cache_page_write_begin+0x55/0xc0
>     [11159.499130]  [<ffffffffb9484b76>] iomap_write_begin+0x66/0x100
>     [11159.499135]  [<ffffffffb9484edf>] iomap_write_actor+0xcf/0x1d0
>     [11159.499140]  [<ffffffffb9484e10>] ? iomap_write_end+0x80/0x80
>     [11159.499144]  [<ffffffffb94854e7>] iomap_apply+0xb7/0x150
>     [11159.499149]  [<ffffffffb9485621>]
>     iomap_file_buffered_write+0xa1/0xe0
>     [11159.499153]  [<ffffffffb9484e10>] ? iomap_write_end+0x80/0x80
>     [11159.499182]  [<ffffffffc05f025d>]
>     xfs_file_buffered_aio_write+0x12d/0x2c0 [xfs]
>     [11159.499213]  [<ffffffffc05f057d>]
>     xfs_file_aio_write+0x18d/0x1b0 [xfs]
>     [11159.499217]  [<ffffffffb941a533>] do_sync_write+0x93/0xe0
>     [11159.499222]  [<ffffffffb941b010>] vfs_write+0xc0/0x1f0
>     [11159.499225]  [<ffffffffb941c002>] SyS_pwrite64+0x92/0xc0
>     [11159.499230]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [11159.499234]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [11159.499238]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [11279.488720] INFO: task xfsaild/dm-10:1134 blocked for more than
>     120 seconds.
>     [11279.490197] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [11279.491665] xfsaild/dm-10   D ffff9720a8660fd0    0  1134     
>     2 0x00000000
>     [11279.491671] Call Trace:
>     [11279.491682]  [<ffffffffb92a3a2e>] ? try_to_del_timer_sync+0x5e/0x90
>     [11279.491688]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [11279.491744]  [<ffffffffc060de36>] _xfs_log_force+0x1c6/0x2c0 [xfs]
>     [11279.491750]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [11279.491783]  [<ffffffffc0619fec>] ? xfsaild+0x16c/0x6f0 [xfs]
>     [11279.491817]  [<ffffffffc060df5c>] xfs_log_force+0x2c/0x70 [xfs]
>     [11279.491849]  [<ffffffffc0619e80>] ?
>     xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>     [11279.491880]  [<ffffffffc0619fec>] xfsaild+0x16c/0x6f0 [xfs]
>     [11279.491913]  [<ffffffffc0619e80>] ?
>     xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>     [11279.491919]  [<ffffffffb92bb161>] kthread+0xd1/0xe0
>     [11279.491926]  [<ffffffffb92bb090>] ? insert_kthread_work+0x40/0x40
>     [11279.491932]  [<ffffffffb9920677>]
>     ret_from_fork_nospec_begin+0x21/0x21
>     [11279.491936]  [<ffffffffb92bb090>] ? insert_kthread_work+0x40/0x40
>     [11279.491976] INFO: task glusterclogfsyn:14934 blocked for more
>     than 120 seconds.
>     [11279.493466] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [11279.494952] glusterclogfsyn D ffff97209832af70    0 14934     
>     1 0x00000080
>     [11279.494957] Call Trace:
>     [11279.494979]  [<ffffffffc0309839>] ?
>     __split_and_process_bio+0x2e9/0x520 [dm_mod]
>     [11279.494983]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [11279.494987]  [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0
>     [11279.494997]  [<ffffffffc0309d98>] ? dm_make_request+0x128/0x1a0
>     [dm_mod]
>     [11279.495001]  [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130
>     [11279.495005]  [<ffffffffb99145ad>] wait_for_completion_io+0xfd/0x140
>     [11279.495010]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [11279.495016]  [<ffffffffb951e574>] blkdev_issue_flush+0xb4/0x110
>     [11279.495049]  [<ffffffffc06064b9>]
>     xfs_blkdev_issue_flush+0x19/0x20 [xfs]
>     [11279.495079]  [<ffffffffc05eec40>] xfs_file_fsync+0x1b0/0x1e0 [xfs]
>     [11279.495086]  [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>     [11279.495090]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [11279.495094]  [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20
>     [11279.495098]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [11279.495102]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [11279.495105] INFO: task glusterposixfsy:14941 blocked for more
>     than 120 seconds.
>     [11279.496606] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [11279.498114] glusterposixfsy D ffff972495f84f10    0 14941     
>     1 0x00000080
>     [11279.498118] Call Trace:
>     [11279.498134]  [<ffffffffc0309839>] ?
>     __split_and_process_bio+0x2e9/0x520 [dm_mod]
>     [11279.498138]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [11279.498142]  [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0
>     [11279.498152]  [<ffffffffc0309d98>] ? dm_make_request+0x128/0x1a0
>     [dm_mod]
>     [11279.498156]  [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130
>     [11279.498160]  [<ffffffffb99145ad>] wait_for_completion_io+0xfd/0x140
>     [11279.498165]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [11279.498169]  [<ffffffffb951e574>] blkdev_issue_flush+0xb4/0x110
>     [11279.498202]  [<ffffffffc06064b9>]
>     xfs_blkdev_issue_flush+0x19/0x20 [xfs]
>     [11279.498231]  [<ffffffffc05eec40>] xfs_file_fsync+0x1b0/0x1e0 [xfs]
>     [11279.498238]  [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>     [11279.498242]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [11279.498246]  [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20
>     [11279.498250]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [11279.498254]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [11279.498257] INFO: task glusteriotwr1:14950 blocked for more
>     than 120 seconds.
>     [11279.499789] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [11279.501343] glusteriotwr1   D ffff97208b6daf70    0 14950     
>     1 0x00000080
>     [11279.501348] Call Trace:
>     [11279.501353]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [11279.501390]  [<ffffffffc060e388>]
>     _xfs_log_force_lsn+0x2e8/0x340 [xfs]
>     [11279.501396]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [11279.501428]  [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>     [11279.501432]  [<ffffffffb944ef3f>] generic_write_sync+0x4f/0x70
>     [11279.501461]  [<ffffffffc05f0545>]
>     xfs_file_aio_write+0x155/0x1b0 [xfs]
>     [11279.501466]  [<ffffffffb941a533>] do_sync_write+0x93/0xe0
>     [11279.501471]  [<ffffffffb941b010>] vfs_write+0xc0/0x1f0
>     [11279.501475]  [<ffffffffb941c002>] SyS_pwrite64+0x92/0xc0
>     [11279.501479]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [11279.501483]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [11279.501489]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [11279.501493] INFO: task glusteriotwr4:14953 blocked for more
>     than 120 seconds.
>     [11279.503047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [11279.504630] glusteriotwr4   D ffff972499f2bf40    0 14953     
>     1 0x00000080
>     [11279.504635] Call Trace:
>     [11279.504640]  [<ffffffffb9913f79>] schedule+0x29/0x70
>     [11279.504676]  [<ffffffffc060e388>]
>     _xfs_log_force_lsn+0x2e8/0x340 [xfs]
>     [11279.504681]  [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>     [11279.504710]  [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>     [11279.504714]  [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>     [11279.504718]  [<ffffffffb992076f>] ?
>     system_call_after_swapgs+0xbc/0x160
>     [11279.504722]  [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20
>     [11279.504725]  [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>     [11279.504730]  [<ffffffffb992077b>] ?
>     system_call_after_swapgs+0xc8/0x160
>     [12127.466494] perf: interrupt took too long (8263 > 8150),
>     lowering kernel.perf_event_max_sample_rate to 24000
>
>     --------------------
>     I think this is the cause of the massive ovirt performance issues
>     irrespective of gluster volume.  At the time this happened, I was
>     also ssh'ed into the host, and was doing some rpm querry
>     commands.  I had just run rpm -qa |grep glusterfs (to verify what
>     version was actually installed), and that command took almost 2
>     minutes to return!  Normally it takes less than 2 seconds.  That
>     is all pure local SSD IO, too....
>
>     I'm no expert, but its my understanding that anytime a software
>     causes these kinds of issues, its a serious bug in the software,
>     even if its mis-handled exceptions.  Is this correct?
>
>     --Jim
>
>     On Tue, May 29, 2018 at 3:01 PM, Jim Kusznir <jim at palousetech.com
>     <mailto:jim at palousetech.com>> wrote:
>
>         I think this is the profile information for one of the volumes
>         that lives on the SSDs and is fully operational with no
>         down/problem disks:
>
>         [root at ovirt2 yum.repos.d]# gluster volume profile data info
>         Brick: ovirt2.nwfiber.com:/gluster/brick2/data
>         ----------------------------------------------
>         Cumulative Stats:
>            Block Size:                256b+            512b+         
>               1024b+
>          No. of Reads:                  983             2696         
>                 1059
>         No. of Writes:                    0             1113         
>                  302
>            Block Size:               2048b+           4096b+         
>               8192b+
>          No. of Reads:                  852            88608         
>                53526
>         No. of Writes:                  522           812340         
>                76257
>            Block Size:              16384b+          32768b+         
>              65536b+
>          No. of Reads:                54351           241901         
>                15024
>         No. of Writes:                21636             8656         
>                 8976
>            Block Size:             131072b+
>          No. of Reads:               524156
>         No. of Writes:               296071
>          %-latency   Avg-latency   Min-Latency  Max-Latency   No. of
>         calls         Fop
>          ---------   -----------   -----------  ----------- 
>          ------------        ----
>               0.00       0.00 us       0.00 us    0.00 us         
>          4189     RELEASE
>               0.00       0.00 us       0.00 us    0.00 us         
>          1257  RELEASEDIR
>               0.00      46.19 us      12.00 us  187.00 us           
>          69       FLUSH
>               0.00     147.00 us      78.00 us  367.00 us           
>          86 REMOVEXATTR
>               0.00     223.46 us      24.00 us 1166.00 us           
>         149     READDIR
>               0.00     565.34 us      76.00 us 3639.00 us           
>          88   FTRUNCATE
>               0.00     263.28 us      20.00 us  28385.00 us           
>         228          LK
>               0.00      98.84 us       2.00 us  880.00 us         
>          1198     OPENDIR
>               0.00      91.59 us      26.00 us  10371.00 us         
>          3853      STATFS
>               0.00     494.14 us      17.00 us 193439.00 us         
>          1171    GETXATTR
>               0.00     299.42 us      35.00 us 9799.00 us         
>          2044    READDIRP
>               0.00    1965.31 us     110.00 us 382258.00 us           
>         321     XATTROP
>               0.01     113.40 us      24.00 us  61061.00 us         
>          8134        STAT
>               0.01     755.38 us      57.00 us 607603.00 us         
>          3196     DISCARD
>               0.05    2690.09 us      58.00 us 2704761.00 us         
>          3206        OPEN
>               0.10  119978.25 us      97.00 us 9406684.00 us         
>           154     SETATTR
>               0.18     101.73 us      28.00 us 700477.00 us       
>          313379       FSTAT
>               0.23    1059.84 us      25.00 us 2716124.00 us         
>         38255      LOOKUP
>               0.47    1024.11 us      54.00 us 6197164.00 us         
>         81455    FXATTROP
>               1.72    2984.00 us      15.00 us 37098954.00 us       
>          103020    FINODELK
>               5.92   44315.32 us      51.00 us 24731536.00 us         
>         23957       FSYNC
>              13.27    2399.78 us      25.00 us 22089540.00 us       
>          991005        READ
>              37.00    5980.43 us      52.00 us 22099889.00 us       
>         1108976       WRITE
>              41.04    5452.75 us      13.00 us 22102452.00 us       
>         1349053     INODELK
>             Duration: 10026 seconds
>            Data Read: 80046027759 bytes
>         Data Written: 44496632320 bytes
>         Interval 1 Stats:
>            Block Size:                256b+            512b+         
>               1024b+
>          No. of Reads:                  983             2696         
>                 1059
>         No. of Writes:                    0              838         
>                  185
>            Block Size:               2048b+           4096b+         
>               8192b+
>          No. of Reads:                  852            85856         
>                51575
>         No. of Writes:                  382           705802         
>                57812
>            Block Size:              16384b+          32768b+         
>              65536b+
>          No. of Reads:                52673           232093         
>                14984
>         No. of Writes:                13499             4908         
>                 4242
>            Block Size:             131072b+
>          No. of Reads:               460040
>         No. of Writes:                 6411
>          %-latency   Avg-latency   Min-Latency  Max-Latency   No. of
>         calls         Fop
>          ---------   -----------   -----------  ----------- 
>          ------------        ----
>               0.00       0.00 us       0.00 us    0.00 us         
>          2093     RELEASE
>               0.00       0.00 us       0.00 us    0.00 us         
>          1093  RELEASEDIR
>               0.00      53.38 us      26.00 us  111.00 us           
>          16       FLUSH
>               0.00     145.14 us      78.00 us  367.00 us           
>          71 REMOVEXATTR
>               0.00     190.96 us     114.00 us  298.00 us           
>          71     SETATTR
>               0.00     213.38 us      24.00 us 1145.00 us           
>          90     READDIR
>               0.00     263.28 us      20.00 us  28385.00 us           
>         228          LK
>               0.00     101.76 us       2.00 us  880.00 us         
>          1093     OPENDIR
>               0.01      93.60 us      27.00 us  10371.00 us         
>          3090      STATFS
>               0.02     537.47 us      17.00 us 193439.00 us         
>          1038    GETXATTR
>               0.03     297.44 us      35.00 us 9799.00 us         
>          1990    READDIRP
>               0.03    2357.28 us     110.00 us 382258.00 us           
>         253     XATTROP
>               0.04     385.93 us      58.00 us  47593.00 us         
>          2091        OPEN
>               0.04     114.86 us      24.00 us  61061.00 us         
>          7715        STAT
>               0.06     444.59 us      57.00 us 333240.00 us         
>          3053     DISCARD
>               0.42     316.24 us      25.00 us 290728.00 us         
>         29823      LOOKUP
>               0.73     257.92 us      54.00 us 344812.00 us         
>         63296    FXATTROP
>               1.37      98.30 us      28.00 us  67621.00 us       
>          313172       FSTAT
>               1.58    2124.69 us      51.00 us 849200.00 us         
>         16717       FSYNC
>               5.73     162.46 us      52.00 us 748492.00 us       
>          794079       WRITE
>               7.19    2065.17 us      16.00 us 37098954.00 us         
>         78381    FINODELK
>              36.44     886.32 us      25.00 us 2216436.00 us       
>          925421        READ
>              46.30    1178.04 us      13.00 us 1700704.00 us       
>          884635     INODELK
>             Duration: 7485 seconds
>            Data Read: 71250527215 bytes
>         Data Written: 5119903744 bytes
>         Brick: ovirt3.nwfiber.com:/gluster/brick2/data
>         ----------------------------------------------
>         Cumulative Stats:
>            Block Size:                  1b+
>          No. of Reads:                    0
>         No. of Writes:              3264419
>          %-latency   Avg-latency   Min-Latency  Max-Latency   No. of
>         calls         Fop
>          ---------   -----------   -----------  ----------- 
>          ------------        ----
>               0.00       0.00 us       0.00 us    0.00 us           
>          90      FORGET
>               0.00       0.00 us       0.00 us    0.00 us         
>          9462     RELEASE
>               0.00       0.00 us       0.00 us    0.00 us         
>          4254  RELEASEDIR
>               0.00      50.52 us      13.00 us  190.00 us           
>          71       FLUSH
>               0.00     186.97 us      87.00 us  713.00 us           
>          86 REMOVEXATTR
>               0.00      79.32 us      33.00 us  189.00 us           
>         228          LK
>               0.00     220.98 us     129.00 us  513.00 us           
>          86     SETATTR
>               0.01     259.30 us      26.00 us 2632.00 us           
>         137     READDIR
>               0.02     322.76 us     145.00 us 2125.00 us           
>         321     XATTROP
>               0.03     109.55 us       2.00 us 1258.00 us         
>          1193     OPENDIR
>               0.05      70.21 us      21.00 us  431.00 us         
>          3196     DISCARD
>               0.05     169.26 us      21.00 us 2315.00 us         
>          1545    GETXATTR
>               0.12     176.85 us      63.00 us 2844.00 us         
>          3206        OPEN
>               0.61     303.49 us      90.00 us 3085.00 us         
>          9633       FSTAT
>               2.44     305.66 us      28.00 us 3716.00 us         
>         38230      LOOKUP
>               4.52     266.22 us      55.00 us  53424.00 us         
>         81455    FXATTROP
>               6.96    1397.99 us      51.00 us  64822.00 us         
>         23889       FSYNC
>              16.48      84.74 us      25.00 us 6917.00 us       
>          932592       WRITE
>              30.16     106.90 us      13.00 us 3920189.00 us       
>         1353046     INODELK
>              38.55    1794.52 us      14.00 us 16210553.00 us       
>          103039    FINODELK
>             Duration: 66562 seconds
>            Data Read: 0 bytes
>         Data Written: 3264419 bytes
>         Interval 1 Stats:
>            Block Size:                  1b+
>          No. of Reads:                    0
>         No. of Writes:               794080
>          %-latency   Avg-latency   Min-Latency  Max-Latency   No. of
>         calls         Fop
>          ---------   -----------   -----------  ----------- 
>          ------------        ----
>               0.00       0.00 us       0.00 us    0.00 us         
>          2093     RELEASE
>               0.00       0.00 us       0.00 us    0.00 us         
>          1093  RELEASEDIR
>               0.00      70.31 us      26.00 us  125.00 us           
>          16       FLUSH
>               0.00     193.10 us     103.00 us  713.00 us           
>          71 REMOVEXATTR
>               0.01     227.32 us     133.00 us  513.00 us           
>          71     SETATTR
>               0.01      79.32 us      33.00 us  189.00 us           
>         228          LK
>               0.01     259.83 us      35.00 us 1138.00 us           
>          89     READDIR
>               0.03     318.26 us     145.00 us 2047.00 us           
>         253     XATTROP
>               0.04     112.67 us       3.00 us 1258.00 us         
>          1093     OPENDIR
>               0.06     167.98 us      23.00 us 1951.00 us         
>          1014    GETXATTR
>               0.08      70.97 us      22.00 us  431.00 us         
>          3053     DISCARD
>               0.13     183.78 us      66.00 us 2844.00 us         
>          2091        OPEN
>               1.01     303.82 us      90.00 us 3085.00 us         
>          9610       FSTAT
>               3.27     316.59 us      30.00 us 3716.00 us         
>         29820      LOOKUP
>               5.83     265.79 us      59.00 us  53424.00 us         
>         63296    FXATTROP
>               7.95    1373.89 us      51.00 us  64822.00 us         
>         16717       FSYNC
>              23.17     851.99 us      14.00 us 16210553.00 us         
>         78555    FINODELK
>              24.04      87.44 us      27.00 us 6917.00 us       
>          794081       WRITE
>              34.36     111.91 us      14.00 us 984871.00 us       
>          886790     INODELK
>             Duration: 7485 seconds
>            Data Read: 0 bytes
>         Data Written: 794080 bytes
>
>
>         -----------------------
>         Here is the data from the volume that is backed by the SHDDs
>         and has one failed disk:
>         [root at ovirt2 yum.repos.d]# gluster volume profile data-hdd info
>         Brick: 172.172.1.12:/gluster/brick3/data-hdd
>         --------------------------------------------
>         Cumulative Stats:
>            Block Size:                256b+            512b+         
>               1024b+
>          No. of Reads:                 1702               86         
>                   16
>         No. of Writes:                    0              767         
>                   71
>            Block Size:               2048b+           4096b+         
>               8192b+
>          No. of Reads:                   19            51841         
>                 2049
>         No. of Writes:                   76            60668         
>                35727
>            Block Size:              16384b+          32768b+         
>              65536b+
>          No. of Reads:                 1744              639         
>                 1088
>         No. of Writes:                 8524             2410         
>                 1285
>            Block Size:             131072b+
>          No. of Reads:               771999
>         No. of Writes:                29584
>          %-latency   Avg-latency   Min-Latency  Max-Latency   No. of
>         calls         Fop
>          ---------   -----------   -----------  ----------- 
>          ------------        ----
>               0.00       0.00 us       0.00 us    0.00 us         
>          2902     RELEASE
>               0.00       0.00 us       0.00 us    0.00 us         
>          1517  RELEASEDIR
>               0.00     197.00 us     197.00 us  197.00 us             
>         1   FTRUNCATE
>               0.00      70.24 us      16.00 us  758.00 us           
>          51       FLUSH
>               0.00     143.93 us      82.00 us  305.00 us           
>          57 REMOVEXATTR
>               0.00     178.63 us     105.00 us  712.00 us           
>          60     SETATTR
>               0.00      67.30 us      19.00 us  572.00 us           
>         555          LK
>               0.00     322.80 us      23.00 us 4673.00 us           
>         138     READDIR
>               0.00     336.56 us     106.00 us  11994.00 us           
>         237     XATTROP
>               0.00      84.70 us      28.00 us 1071.00 us         
>          3469      STATFS
>               0.01     387.75 us       2.00 us 146017.00 us         
>          1467     OPENDIR
>               0.01     148.59 us      21.00 us  64374.00 us         
>          4454        STAT
>               0.02     783.02 us      16.00 us  93502.00 us         
>          1902    GETXATTR
>               0.03    1516.10 us      17.00 us 210690.00 us         
>          1364     ENTRYLK
>               0.03    2555.47 us     300.00 us 674454.00 us         
>          1064    READDIRP
>               0.07      85.74 us      19.00 us  68340.00 us         
>         62849       FSTAT
>               0.07    1978.12 us      59.00 us 202596.00 us         
>          2729        OPEN
>               0.22     708.57 us      15.00 us 394799.00 us         
>         25447      LOOKUP
>               5.94    2331.74 us      15.00 us 1099530.00 us       
>          207534    FINODELK
>               7.31    8311.75 us      58.00 us 1800216.00 us         
>         71668    FXATTROP
>              12.49    7735.19 us      51.00 us 3595513.00 us       
>          131642       WRITE
>              17.70     957.08 us      16.00 us 13700466.00 us       
>         1508160     INODELK
>              24.55    2546.43 us      26.00 us 5077347.00 us       
>          786060        READ
>              31.56   49699.15 us      47.00 us 3746331.00 us         
>         51777       FSYNC
>             Duration: 10101 seconds
>            Data Read: 101562897361 bytes
>         Data Written: 4834450432 bytes
>         Interval 0 Stats:
>            Block Size:                256b+            512b+         
>               1024b+
>          No. of Reads:                 1702               86         
>                   16
>         No. of Writes:                    0              767         
>                   71
>            Block Size:               2048b+           4096b+         
>               8192b+
>          No. of Reads:                   19            51841         
>                 2049
>         No. of Writes:                   76            60668         
>                35727
>            Block Size:              16384b+          32768b+         
>              65536b+
>          No. of Reads:                 1744              639         
>                 1088
>         No. of Writes:                 8524             2410         
>                 1285
>            Block Size:             131072b+
>          No. of Reads:               771999
>         No. of Writes:                29584
>          %-latency   Avg-latency   Min-Latency  Max-Latency   No. of
>         calls         Fop
>          ---------   -----------   -----------  ----------- 
>          ------------        ----
>               0.00       0.00 us       0.00 us    0.00 us         
>          2902     RELEASE
>               0.00       0.00 us       0.00 us    0.00 us         
>          1517  RELEASEDIR
>               0.00     197.00 us     197.00 us  197.00 us             
>         1   FTRUNCATE
>               0.00      70.24 us      16.00 us  758.00 us           
>          51       FLUSH
>               0.00     143.93 us      82.00 us  305.00 us           
>          57 REMOVEXATTR
>               0.00     178.63 us     105.00 us  712.00 us           
>          60     SETATTR
>               0.00      67.30 us      19.00 us  572.00 us           
>         555          LK
>               0.00     322.80 us      23.00 us 4673.00 us           
>         138     READDIR
>               0.00     336.56 us     106.00 us  11994.00 us           
>         237     XATTROP
>               0.00      84.70 us      28.00 us 1071.00 us         
>          3469      STATFS
>               0.01     387.75 us       2.00 us 146017.00 us         
>          1467     OPENDIR
>               0.01     148.59 us      21.00 us  64374.00 us         
>          4454        STAT
>               0.02     783.02 us      16.00 us  93502.00 us         
>          1902    GETXATTR
>               0.03    1516.10 us      17.00 us 210690.00 us         
>          1364     ENTRYLK
>               0.03    2555.47 us     300.00 us 674454.00 us         
>          1064    READDIRP
>               0.07      85.73 us      19.00 us  68340.00 us         
>         62849       FSTAT
>               0.07    1978.12 us      59.00 us 202596.00 us         
>          2729        OPEN
>               0.22     708.57 us      15.00 us 394799.00 us         
>         25447      LOOKUP
>               5.94    2334.57 us      15.00 us 1099530.00 us       
>          207534    FINODELK
>               7.31    8311.49 us      58.00 us 1800216.00 us         
>         71668    FXATTROP
>              12.49    7735.32 us      51.00 us 3595513.00 us       
>          131642       WRITE
>              17.71     957.08 us      16.00 us 13700466.00 us       
>         1508160     INODELK
>              24.56    2546.42 us      26.00 us 5077347.00 us       
>          786060        READ
>              31.54   49651.63 us      47.00 us 3746331.00 us         
>         51777       FSYNC
>             Duration: 10101 seconds
>            Data Read: 101562897361 bytes
>         Data Written: 4834450432 bytes
>
>
>         On Tue, May 29, 2018 at 2:55 PM, Jim Kusznir
>         <jim at palousetech.com <mailto:jim at palousetech.com>> wrote:
>
>             Thank you for your response.
>
>             I have 4 gluster volumes.  3 are replica 2 + arbitrator. 
>             replica bricks are on ovirt1 and ovirt2, arbitrator on
>             ovirt3.  The 4th volume is replica 3, with a brick on all
>             three ovirt machines.
>
>             The first 3 volumes are on an SSD disk; the 4th is on a
>             Seagate SSHD (same in all three machines).  On ovirt3, the
>             SSHD has reported hard IO failures, and that brick is
>             offline.  However, the other two replicas are fully
>             operational (although they still show contents in the heal
>             info command that won't go away, but that may be the case
>             until I replace the failed disk).
>
>             What is bothering me is that ALL 4 gluster volumes are
>             showing horrible performance issues.  At this point, as
>             the bad disk has been completely offlined, I would expect
>             gluster to perform at normal speed, but that is definitely
>             not the case.
>
>             I've also noticed that the performance hits seem to come
>             in waves: things seem to work acceptably (but slow) for a
>             while, then suddenly, its as if all disk IO on all volumes
>             (including non-gluster local OS disk volumes for the
>             hosts) pause for about 30 seconds, then IO resumes again.
>             During those times, I start getting VM not responding and
>             host not responding notices as well as the applications
>             having major issues.
>
>             I've shut down most of my VMs and am down to just my
>             essential core VMs (shedded about 75% of my VMs). I still
>             am experiencing the same issues.
>
>             Am I correct in believing that once the failed disk was
>             brought offline that performance should return to normal?
>
>             On Tue, May 29, 2018 at 1:27 PM, Alex K
>             <rightkicktech at gmail.com <mailto:rightkicktech at gmail.com>>
>             wrote:
>
>                 I would check disks status and accessibility of mount
>                 points where your gluster volumes reside.
>
>                 On Tue, May 29, 2018, 22:28 Jim Kusznir
>                 <jim at palousetech.com <mailto:jim at palousetech.com>> wrote:
>
>                     On one ovirt server, I'm now seeing these messages:
>                     [56474.239725] blk_update_request: 63 callbacks
>                     suppressed
>                     [56474.239732] blk_update_request: I/O error, dev
>                     dm-2, sector 0
>                     [56474.240602] blk_update_request: I/O error, dev
>                     dm-2, sector 3905945472
>                     [56474.241346] blk_update_request: I/O error, dev
>                     dm-2, sector 3905945584
>                     [56474.242236] blk_update_request: I/O error, dev
>                     dm-2, sector 2048
>                     [56474.243072] blk_update_request: I/O error, dev
>                     dm-2, sector 3905943424
>                     [56474.243997] blk_update_request: I/O error, dev
>                     dm-2, sector 3905943536
>                     [56474.247347] blk_update_request: I/O error, dev
>                     dm-2, sector 0
>                     [56474.248315] blk_update_request: I/O error, dev
>                     dm-2, sector 3905945472
>                     [56474.249231] blk_update_request: I/O error, dev
>                     dm-2, sector 3905945584
>                     [56474.250221] blk_update_request: I/O error, dev
>                     dm-2, sector 2048
>
>
>
>
>                     On Tue, May 29, 2018 at 11:59 AM, Jim Kusznir
>                     <jim at palousetech.com <mailto:jim at palousetech.com>>
>                     wrote:
>
>                         I see in messages on ovirt3 (my 3rd machine,
>                         the one upgraded to 4.2):
>
>                         May 29 11:54:41 ovirt3 ovs-vsctl:
>                         ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock:
>                         database connection failed (No such file or
>                         directory)
>                         May 29 11:54:51 ovirt3 ovs-vsctl:
>                         ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock:
>                         database connection failed (No such file or
>                         directory)
>                         May 29 11:55:01 ovirt3 ovs-vsctl:
>                         ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock:
>                         database connection failed (No such file or
>                         directory)
>                         (appears a lot).
>
>                         I also found on the ssh session of that, some
>                         sysv warnings about the backing disk for one
>                         of the gluster volumes (straight replica 3).
>                         The glusterfs process for that disk on that
>                         machine went offline. Its my understanding
>                         that it should continue to work with the other
>                         two machines while I attempt to replace that
>                         disk, right? Attempted writes (touching an
>                         empty file) can take 15 seconds, repeating it
>                         later will be much faster.
>
>                         Gluster generates a bunch of different log
>                         files, I don't know what ones you want, or
>                         from which machine(s).
>
>                         How do I do "volume profiling"?
>
>                         Thanks!
>
>                         On Tue, May 29, 2018 at 11:53 AM, Sahina Bose
>                         <sabose at redhat.com <mailto:sabose at redhat.com>>
>                         wrote:
>
>                             Do you see errors reported in the mount
>                             logs for the volume? If so, could you
>                             attach the logs?
>                             Any issues with your underlying disks. Can
>                             you also attach output of volume profiling?
>
>                             On Wed, May 30, 2018 at 12:13 AM, Jim
>                             Kusznir <jim at palousetech.com
>                             <mailto:jim at palousetech.com>> wrote:
>
>                                 Ok, things have gotten MUCH worse this
>                                 morning.  I'm getting random errors
>                                 from VMs, right now, about a third of
>                                 my VMs have been paused due to storage
>                                 issues, and most of the remaining VMs
>                                 are not performing well.
>
>                                 At this point, I am in full EMERGENCY
>                                 mode, as my production services are
>                                 now impacted, and I'm getting calls
>                                 coming in with problems...
>
>                                 I'd greatly appreciate help...VMs are
>                                 running VERY slowly (when they run),
>                                 and they are steadily getting worse. 
>                                 I don't know why.  I was seeing CPU
>                                 peaks (to 100%) on several VMs, in
>                                 perfect sync, for a few minutes at a
>                                 time (while the VM became unresponsive
>                                 and any VMs I was logged into that
>                                 were linux were giving me the CPU
>                                 stuck messages in my origional post). 
>                                 Is all this storage related?
>
>                                 I also have two different gluster
>                                 volumes for VM storage, and only one
>                                 had the issues, but now VMs in both
>                                 are being affected at the same time
>                                 and same way.
>
>                                 --Jim
>
>                                 On Mon, May 28, 2018 at 10:50 PM,
>                                 Sahina Bose <sabose at redhat.com
>                                 <mailto:sabose at redhat.com>> wrote:
>
>                                     [Adding gluster-users to look at
>                                     the heal issue]
>
>                                     On Tue, May 29, 2018 at 9:17 AM,
>                                     Jim Kusznir <jim at palousetech.com
>                                     <mailto:jim at palousetech.com>> wrote:
>
>                                         Hello:
>
>                                         I've been having some cluster
>                                         and gluster performance issues
>                                         lately.  I also found that my
>                                         cluster was out of date, and
>                                         was trying to apply updates
>                                         (hoping to fix some of these),
>                                         and discovered the ovirt 4.1
>                                         repos were taken completely
>                                         offline.  So, I was forced to
>                                         begin an upgrade to 4.2.
>                                         According to docs I
>                                         found/read, I needed only add
>                                         the new repo, do a yum update,
>                                         reboot, and be good on my
>                                         hosts (did the yum update, the
>                                         engine-setup on my hosted
>                                         engine). Things seemed to work
>                                         relatively well, except for a
>                                         gluster sync issue that showed up.
>
>                                         My cluster is a 3 node
>                                         hyperconverged cluster.  I
>                                         upgraded the hosted engine
>                                         first, then engine 3. When
>                                         engine 3 came back up, for
>                                         some reason one of my gluster
>                                         volumes would not sync. Here's
>                                         sample output:
>
>                                         [root at ovirt3 ~]# gluster
>                                         volume heal data-hdd info
>                                         Brick
>                                         172.172.1.11:/gluster/brick3/data-hdd
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/48d7ecb8-7ac5-4725-bca5-b3519681cf2f/0d6080b0-7018-4fa3-bb82-1dd9ef07d9b9
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/6da854d1-b6be-446b-9bf0-90a0dbbea830/3c93bd1f-b7fa-4aa2-b445-6904e31839ba
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/7f647567-d18c-44f1-a58e-9b8865833acb/f9364470-9770-4bb1-a6b9-a54861849625
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/f3c8e7aa-6ef2-42a7-93d4-e0a4df6dd2fa/2eb0b1ad-2606-44ef-9cd3-ae59610a504b
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/b1ea3f62-0f05-4ded-8c82-9c91c90e0b61/d5d6bf5a-499f-431d-9013-5453db93ed32
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/8c8b5147-e9d6-4810-b45b-185e3ed65727/16f08231-93b0-489d-a2fd-687b6bf88eaa
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/12924435-b9c2-4aab-ba19-1c1bc31310ef/07b3db69-440e-491e-854c-bbfa18a7cff2
>
>                                         Status: Connected
>                                         Number of entries: 8
>
>                                         Brick
>                                         172.172.1.12:/gluster/brick3/data-hdd
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/48d7ecb8-7ac5-4725-bca5-b3519681cf2f/0d6080b0-7018-4fa3-bb82-1dd9ef07d9b9
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/b1ea3f62-0f05-4ded-8c82-9c91c90e0b61/d5d6bf5a-499f-431d-9013-5453db93ed32
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/6da854d1-b6be-446b-9bf0-90a0dbbea830/3c93bd1f-b7fa-4aa2-b445-6904e31839ba
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/7f647567-d18c-44f1-a58e-9b8865833acb/f9364470-9770-4bb1-a6b9-a54861849625
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/8c8b5147-e9d6-4810-b45b-185e3ed65727/16f08231-93b0-489d-a2fd-687b6bf88eaa
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/12924435-b9c2-4aab-ba19-1c1bc31310ef/07b3db69-440e-491e-854c-bbfa18a7cff2
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/f3c8e7aa-6ef2-42a7-93d4-e0a4df6dd2fa/2eb0b1ad-2606-44ef-9cd3-ae59610a504b
>
>                                         Status: Connected
>                                         Number of entries: 8
>
>                                         Brick
>                                         172.172.1.13:/gluster/brick3/data-hdd
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/b1ea3f62-0f05-4ded-8c82-9c91c90e0b61/d5d6bf5a-499f-431d-9013-5453db93ed32
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/8c8b5147-e9d6-4810-b45b-185e3ed65727/16f08231-93b0-489d-a2fd-687b6bf88eaa
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/12924435-b9c2-4aab-ba19-1c1bc31310ef/07b3db69-440e-491e-854c-bbfa18a7cff2
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/f3c8e7aa-6ef2-42a7-93d4-e0a4df6dd2fa/2eb0b1ad-2606-44ef-9cd3-ae59610a504b
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/48d7ecb8-7ac5-4725-bca5-b3519681cf2f/0d6080b0-7018-4fa3-bb82-1dd9ef07d9b9
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/6da854d1-b6be-446b-9bf0-90a0dbbea830/3c93bd1f-b7fa-4aa2-b445-6904e31839ba
>
>                                         /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/7f647567-d18c-44f1-a58e-9b8865833acb/f9364470-9770-4bb1-a6b9-a54861849625
>
>                                         Status: Connected
>                                         Number of entries: 8
>
>                                         ---------
>                                         Its been in this state for a
>                                         couple days now, and bandwidth
>                                         monitoring shows no
>                                         appreciable data moving. I've
>                                         tried repeatedly commanding a
>                                         full heal from all three
>                                         clusters in the node.  Its
>                                         always the same files that
>                                         need healing.
>
>                                         When running gluster volume
>                                         heal data-hdd statistics, I
>                                         see sometimes different
>                                         information, but always some
>                                         number of "heal failed"
>                                         entries.  It shows 0 for split
>                                         brain.
>
>                                         I'm not quite sure what to
>                                         do.  I suspect it may be due
>                                         to nodes 1 and 2 still being
>                                         on the older ovirt/gluster
>                                         release, but I'm afraid to
>                                         upgrade and reboot them until
>                                         I have a good gluster sync
>                                         (don't need to create a split
>                                         brain issue).  How do I
>                                         proceed with this?
>
>                                         Second issue: I've been
>                                         experiencing VERY POOR
>                                         performance on most of my
>                                         VMs.  To the tune that logging
>                                         into a windows 10 vm via
>                                         remote desktop can take 5
>                                         minutes, launching quickbooks
>                                         inside said vm can easily take
>                                         10 minutes.  On some linux
>                                         VMs, I get random messages
>                                         like this:
>                                         Message from syslogd at unifi at
>                                         May 28 20:39:23 ...
>                                          kernel:[6171996.308904] NMI
>                                         watchdog: BUG: soft lockup -
>                                         CPU#0 stuck for 22s!
>                                         [mongod:14766]
>
>                                         (the process and PID are often
>                                         different)
>
>                                         I'm not quite sure what to do
>                                         about this either.  My initial
>                                         thought was upgrad everything
>                                         to current and see if its
>                                         still there, but I cannot move
>                                         forward with that until my
>                                         gluster is healed...
>
>                                         Thanks!
>                                         --Jim
>
>                                         _______________________________________________
>                                         Users mailing list --
>                                         users at ovirt.org
>                                         <mailto:users at ovirt.org>
>                                         To unsubscribe send an email
>                                         to users-leave at ovirt.org
>                                         <mailto:users-leave at ovirt.org>
>                                         Privacy Statement:
>                                         https://www.ovirt.org/site/privacy-policy/
>                                         <https://www.ovirt.org/site/privacy-policy/>
>                                         oVirt Code of Conduct:
>                                         https://www.ovirt.org/community/about/community-guidelines/
>                                         <https://www.ovirt.org/community/about/community-guidelines/>
>                                         List Archives:
>                                         https://lists.ovirt.org/archives/list/users@ovirt.org/message/3LEV6ZQ3JV2XLAL7NYBTXOYMYUOTIRQF/
>                                         <https://lists.ovirt.org/archives/list/users@ovirt.org/message/3LEV6ZQ3JV2XLAL7NYBTXOYMYUOTIRQF/>
>
>
>
>
>
>
>                     _______________________________________________
>                     Users mailing list -- users at ovirt.org
>                     <mailto:users at ovirt.org>
>                     To unsubscribe send an email to
>                     users-leave at ovirt.org <mailto:users-leave at ovirt.org>
>                     Privacy Statement:
>                     https://www.ovirt.org/site/privacy-policy/
>                     <https://www.ovirt.org/site/privacy-policy/>
>                     oVirt Code of Conduct:
>                     https://www.ovirt.org/community/about/community-guidelines/
>                     <https://www.ovirt.org/community/about/community-guidelines/>
>                     List Archives:
>                     https://lists.ovirt.org/archives/list/users@ovirt.org/message/ACO7RFSLBSRBAIONIC2HQ6Z24ZDES5MF/
>                     <https://lists.ovirt.org/archives/list/users@ovirt.org/message/ACO7RFSLBSRBAIONIC2HQ6Z24ZDES5MF/>
>
>
>
>
>
>     _______________________________________________
>     Users mailing list -- users at ovirt.org <mailto:users at ovirt.org>
>     To unsubscribe send an email to users-leave at ovirt.org
>     <mailto:users-leave at ovirt.org>
>     Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>     <https://www.ovirt.org/site/privacy-policy/>
>     oVirt Code of Conduct:
>     https://www.ovirt.org/community/about/community-guidelines/
>     <https://www.ovirt.org/community/about/community-guidelines/>
>     List Archives:
>     https://lists.ovirt.org/archives/list/users@ovirt.org/message/3DEQQLJM3WHQNZJ7KEMRZVFZ52MTIL74/
>     <https://lists.ovirt.org/archives/list/users@ovirt.org/message/3DEQQLJM3WHQNZJ7KEMRZVFZ52MTIL74/>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180530/0704061f/attachment.html>


More information about the Gluster-users mailing list