[Gluster-users] [ovirt-users] Re: Gluster problems, cluster performance issues
Sahina Bose
sabose at redhat.com
Wed May 30 07:03:54 UTC 2018
[Adding gluster-users back]
Nothing amiss with volume info and status.
Can you check the agent.log and broker.log - will be under
/var/log/ovirt-hosted-engine-ha/
Also the gluster client logs - under
/var/log/glusterfs/rhev-data-center-mnt-glusterSD<volume>.log
On Wed, May 30, 2018 at 12:08 PM, Jim Kusznir <jim at palousetech.com> wrote:
> I believe the gluster data store for the engine is up and working
> correctly. The rest are not mounted, as the engine hasn't managed to start
> correctly yet. I did perform the copy-a-junk-file into the data store,
> then check to ensure its there on another host, then delete that and see
> that it disappeared on the first host; it passed that test. Here's the
> info and status. (I have NOT performed the steps that Krutika and
> Ravishankar suggested yet, as I don't have my data volumes working again
> yet.
>
> [root at ovirt2 images]# gluster volume info
>
> Volume Name: data
> Type: Replicate
> Volume ID: e670c488-ac16-4dd1-8bd3-e43b2e42cc59
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: ovirt1.nwfiber.com:/gluster/brick2/data
> Brick2: ovirt2.nwfiber.com:/gluster/brick2/data
> Brick3: ovirt3.nwfiber.com:/gluster/brick2/data (arbiter)
> Options Reconfigured:
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> changelog.changelog: on
> geo-replication.ignore-pid-check: on
> geo-replication.indexing: on
> server.allow-insecure: on
> performance.readdir-ahead: on
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> storage.owner-uid: 36
> storage.owner-gid: 36
> features.shard: on
> features.shard-block-size: 512MB
> performance.low-prio-threads: 32
> cluster.data-self-heal-algorithm: full
> cluster.locking-scheme: granular
> cluster.shd-wait-qlength: 10000
> cluster.shd-max-threads: 8
> network.ping-timeout: 30
> user.cifs: off
> nfs.disable: on
> performance.strict-o-direct: on
>
> Volume Name: data-hdd
> Type: Replicate
> Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 172.172.1.11:/gluster/brick3/data-hdd
> Brick2: 172.172.1.12:/gluster/brick3/data-hdd
> Brick3: 172.172.1.13:/gluster/brick3/data-hdd
> Options Reconfigured:
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> network.ping-timeout: 30
> server.allow-insecure: on
> storage.owner-gid: 36
> storage.owner-uid: 36
> user.cifs: off
> features.shard: on
> cluster.shd-wait-qlength: 10000
> cluster.shd-max-threads: 8
> cluster.locking-scheme: granular
> cluster.data-self-heal-algorithm: full
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> cluster.eager-lock: enable
> network.remote-dio: enable
> performance.low-prio-threads: 32
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> changelog.changelog: on
> geo-replication.ignore-pid-check: on
> geo-replication.indexing: on
> transport.address-family: inet
> performance.readdir-ahead: on
>
> Volume Name: engine
> Type: Replicate
> Volume ID: 87ad86b9-d88b-457e-ba21-5d3173c612de
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: ovirt1.nwfiber.com:/gluster/brick1/engine
> Brick2: ovirt2.nwfiber.com:/gluster/brick1/engine
> Brick3: ovirt3.nwfiber.com:/gluster/brick1/engine (arbiter)
> Options Reconfigured:
> changelog.changelog: on
> geo-replication.ignore-pid-check: on
> geo-replication.indexing: on
> performance.readdir-ahead: on
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: off
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> storage.owner-uid: 36
> storage.owner-gid: 36
> features.shard: on
> features.shard-block-size: 512MB
> performance.low-prio-threads: 32
> cluster.data-self-heal-algorithm: full
> cluster.locking-scheme: granular
> cluster.shd-wait-qlength: 10000
> cluster.shd-max-threads: 6
> network.ping-timeout: 30
> user.cifs: off
> nfs.disable: on
> performance.strict-o-direct: on
>
> Volume Name: iso
> Type: Replicate
> Volume ID: b1ba15f5-0f0f-4411-89d0-595179f02b92
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: ovirt1.nwfiber.com:/gluster/brick4/iso
> Brick2: ovirt2.nwfiber.com:/gluster/brick4/iso
> Brick3: ovirt3.nwfiber.com:/gluster/brick4/iso (arbiter)
> Options Reconfigured:
> performance.strict-o-direct: on
> nfs.disable: on
> user.cifs: off
> network.ping-timeout: 30
> cluster.shd-max-threads: 6
> cluster.shd-wait-qlength: 10000
> cluster.locking-scheme: granular
> cluster.data-self-heal-algorithm: full
> performance.low-prio-threads: 32
> features.shard-block-size: 512MB
> features.shard: on
> storage.owner-gid: 36
> storage.owner-uid: 36
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> network.remote-dio: off
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.readdir-ahead: on
>
>
> [root at ovirt2 images]# gluster volume status
> Status of volume: data
> Gluster process TCP Port RDMA Port Online
> Pid
> ------------------------------------------------------------
> ------------------
> Brick ovirt1.nwfiber.com:/gluster/brick2/da
> ta 49152 0 Y
> 3226
> Brick ovirt2.nwfiber.com:/gluster/brick2/da
> ta 49152 0 Y
> 2967
> Brick ovirt3.nwfiber.com:/gluster/brick2/da
> ta 49152 0 Y
> 2554
> Self-heal Daemon on localhost N/A N/A Y
> 4818
> Self-heal Daemon on ovirt3.nwfiber.com N/A N/A Y
> 17853
> Self-heal Daemon on ovirt1.nwfiber.com N/A N/A Y
> 4771
>
> Task Status of Volume data
> ------------------------------------------------------------
> ------------------
> There are no active volume tasks
>
> Status of volume: data-hdd
> Gluster process TCP Port RDMA Port Online
> Pid
> ------------------------------------------------------------
> ------------------
> Brick 172.172.1.11:/gluster/brick3/data-hdd 49153 0 Y
> 3232
> Brick 172.172.1.12:/gluster/brick3/data-hdd 49153 0 Y
> 2976
> Brick 172.172.1.13:/gluster/brick3/data-hdd N/A N/A N
> N/A
> NFS Server on localhost N/A N/A N
> N/A
> Self-heal Daemon on localhost N/A N/A Y
> 4818
> NFS Server on ovirt3.nwfiber.com N/A N/A N
> N/A
> Self-heal Daemon on ovirt3.nwfiber.com N/A N/A Y
> 17853
> NFS Server on ovirt1.nwfiber.com N/A N/A N
> N/A
> Self-heal Daemon on ovirt1.nwfiber.com N/A N/A Y
> 4771
>
> Task Status of Volume data-hdd
> ------------------------------------------------------------
> ------------------
> There are no active volume tasks
>
> Status of volume: engine
> Gluster process TCP Port RDMA Port Online
> Pid
> ------------------------------------------------------------
> ------------------
> Brick ovirt1.nwfiber.com:/gluster/brick1/en
> gine 49154 0 Y
> 3239
> Brick ovirt2.nwfiber.com:/gluster/brick1/en
> gine 49154 0 Y
> 2982
> Brick ovirt3.nwfiber.com:/gluster/brick1/en
> gine 49154 0 Y
> 2578
> Self-heal Daemon on localhost N/A N/A Y
> 4818
> Self-heal Daemon on ovirt3.nwfiber.com N/A N/A Y
> 17853
> Self-heal Daemon on ovirt1.nwfiber.com N/A N/A Y
> 4771
>
> Task Status of Volume engine
> ------------------------------------------------------------
> ------------------
> There are no active volume tasks
>
> Status of volume: iso
> Gluster process TCP Port RDMA Port Online
> Pid
> ------------------------------------------------------------
> ------------------
> Brick ovirt1.nwfiber.com:/gluster/brick4/is
> o 49155 0 Y
> 3247
> Brick ovirt2.nwfiber.com:/gluster/brick4/is
> o 49155 0 Y
> 2990
> Brick ovirt3.nwfiber.com:/gluster/brick4/is
> o 49155 0 Y
> 2580
> Self-heal Daemon on localhost N/A N/A Y
> 4818
> Self-heal Daemon on ovirt3.nwfiber.com N/A N/A Y
> 17853
> Self-heal Daemon on ovirt1.nwfiber.com N/A N/A Y
> 4771
>
> Task Status of Volume iso
> ------------------------------------------------------------
> ------------------
> There are no active volume tasks
>
>
> On Tue, May 29, 2018 at 11:30 PM, Sahina Bose <sabose at redhat.com> wrote:
>
>>
>>
>> On Wed, May 30, 2018 at 10:42 AM, Jim Kusznir <jim at palousetech.com>
>> wrote:
>>
>>> hosted-engine --deploy failed (would not come up on my existing gluster
>>> storage). However, I realized no changes were written to my existing
>>> storage. So, I went back to trying to get my old engine running.
>>>
>>> hosted-engine --vm-status is now taking a very long time (5+minutes) to
>>> return, and it returns stail information everywhere. I thought perhaps the
>>> lockspace is corrupt, so tried to clean that and metadata, but both are
>>> failing (--cleam-metadata has hung and I can't even ctrl-c out of it).
>>>
>>> How can I reinitialize all the lockspace/metadata safely? There is no
>>> engine or VMs running currently....
>>>
>>
>> I think the first thing to make sure is that your storage is up and
>> running. So can you mount the gluster volumes and able to access the
>> contents there?
>> Please provide the gluster volume info and gluster volume status of the
>> volumes that you're using.
>>
>>
>>
>>> --Jim
>>>
>>> On Tue, May 29, 2018 at 9:33 PM, Jim Kusznir <jim at palousetech.com>
>>> wrote:
>>>
>>>> Well, things went from bad to very, very bad....
>>>>
>>>> It appears that during one of the 2 minute lockups, the fencing agents
>>>> decided that another node in the cluster was down. As a result, 2 of the 3
>>>> nodes were simultaneously reset with fencing agent reboot. After the nodes
>>>> came back up, the engine would not start. All running VMs (including VMs
>>>> on the 3rd node that was not rebooted) crashed.
>>>>
>>>> I've now been working for about 3 hours trying to get the engine to
>>>> come up. I don't know why it won't start. hosted-engine --vm-start says
>>>> its starting, but it doesn't start (virsh doesn't show any VMs running).
>>>> I'm currently running --deploy, as I had run out of options for anything
>>>> else I can come up with. I hope this will allow me to re-import all my
>>>> existing VMs and allow me to start them back up after everything comes back
>>>> up.
>>>>
>>>> I do have an unverified geo-rep backup; I don't know if it is a good
>>>> backup (there were several prior messages to this list, but I didn't get
>>>> replies to my questions. It was running in what I believe to be "strange",
>>>> and the data directories are larger than their source).
>>>>
>>>> I'll see if my --deploy works, and if not, I'll be back with another
>>>> message/help request.
>>>>
>>>> When the dust settles and I'm at least minimally functional again, I
>>>> really want to understand why all these technologies designed to offer
>>>> redundancy conspired to reduce uptime and create failures where there
>>>> weren't any otherwise. I thought with hosted engine, 3 ovirt servers and
>>>> glusterfs with minimum replica 2+arb or replica 3 should have offered
>>>> strong resilience against server failure or disk failure, and should have
>>>> prevented / recovered from data corruption. Instead, all of the above
>>>> happened (once I get my cluster back up, I still have to try and recover my
>>>> webserver VM, which won't boot due to XFS corrupt journal issues created
>>>> during the gluster crashes). I think a lot of these issues were rooted
>>>> from the upgrade from 4.1 to 4.2.
>>>>
>>>> --Jim
>>>>
>>>> On Tue, May 29, 2018 at 6:25 PM, Jim Kusznir <jim at palousetech.com>
>>>> wrote:
>>>>
>>>>> I also finally found the following in my system log on one server:
>>>>>
>>>>> [10679.524491] INFO: task glusterclogro:14933 blocked for more than
>>>>> 120 seconds.
>>>>> [10679.525826] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [10679.527144] glusterclogro D ffff97209832bf40 0 14933 1
>>>>> 0x00000080
>>>>> [10679.527150] Call Trace:
>>>>> [10679.527161] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [10679.527218] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340
>>>>> [xfs]
>>>>> [10679.527225] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [10679.527254] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>>>>> [10679.527260] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>>>>> [10679.527268] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [10679.527271] [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20
>>>>> [10679.527275] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [10679.527279] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [10679.527283] INFO: task glusterposixfsy:14941 blocked for more than
>>>>> 120 seconds.
>>>>> [10679.528608] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [10679.529956] glusterposixfsy D ffff972495f84f10 0 14941 1
>>>>> 0x00000080
>>>>> [10679.529961] Call Trace:
>>>>> [10679.529966] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [10679.530003] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340
>>>>> [xfs]
>>>>> [10679.530008] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [10679.530038] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>>>>> [10679.530042] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>>>>> [10679.530046] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [10679.530050] [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20
>>>>> [10679.530054] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [10679.530058] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [10679.530062] INFO: task glusteriotwr13:15486 blocked for more than
>>>>> 120 seconds.
>>>>> [10679.531805] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [10679.533732] glusteriotwr13 D ffff9720a83f0000 0 15486 1
>>>>> 0x00000080
>>>>> [10679.533738] Call Trace:
>>>>> [10679.533747] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [10679.533799] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340
>>>>> [xfs]
>>>>> [10679.533806] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [10679.533846] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>>>>> [10679.533852] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>>>>> [10679.533858] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [10679.533863] [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20
>>>>> [10679.533868] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [10679.533873] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [10919.512757] INFO: task glusterclogro:14933 blocked for more than
>>>>> 120 seconds.
>>>>> [10919.514714] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [10919.516663] glusterclogro D ffff97209832bf40 0 14933 1
>>>>> 0x00000080
>>>>> [10919.516677] Call Trace:
>>>>> [10919.516690] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [10919.516696] [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0
>>>>> [10919.516703] [<ffffffffb951cc04>] ? blk_finish_plug+0x14/0x40
>>>>> [10919.516768] [<ffffffffc05e9224>] ? _xfs_buf_ioapply+0x334/0x460
>>>>> [xfs]
>>>>> [10919.516774] [<ffffffffb991432d>] wait_for_completion+0xfd/0x140
>>>>> [10919.516782] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [10919.516821] [<ffffffffc05eb0a3>] ? _xfs_buf_read+0x23/0x40 [xfs]
>>>>> [10919.516859] [<ffffffffc05eafa9>] xfs_buf_submit_wait+0xf9/0x1d0
>>>>> [xfs]
>>>>> [10919.516902] [<ffffffffc061b279>] ? xfs_trans_read_buf_map+0x199/0x400
>>>>> [xfs]
>>>>> [10919.516940] [<ffffffffc05eb0a3>] _xfs_buf_read+0x23/0x40 [xfs]
>>>>> [10919.516977] [<ffffffffc05eb1b9>] xfs_buf_read_map+0xf9/0x160 [xfs]
>>>>> [10919.517022] [<ffffffffc061b279>] xfs_trans_read_buf_map+0x199/0x400
>>>>> [xfs]
>>>>> [10919.517057] [<ffffffffc05c8d04>] xfs_da_read_buf+0xd4/0x100 [xfs]
>>>>> [10919.517091] [<ffffffffc05c8d53>] xfs_da3_node_read+0x23/0xd0 [xfs]
>>>>> [10919.517126] [<ffffffffc05c9fee>] xfs_da3_node_lookup_int+0x6e/0x2f0
>>>>> [xfs]
>>>>> [10919.517160] [<ffffffffc05d5a1d>] xfs_dir2_node_lookup+0x4d/0x170
>>>>> [xfs]
>>>>> [10919.517194] [<ffffffffc05ccf5d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs]
>>>>> [10919.517233] [<ffffffffc05fd8d9>] xfs_lookup+0x69/0x140 [xfs]
>>>>> [10919.517271] [<ffffffffc05fa018>] xfs_vn_lookup+0x78/0xc0 [xfs]
>>>>> [10919.517278] [<ffffffffb9425cf3>] lookup_real+0x23/0x60
>>>>> [10919.517283] [<ffffffffb9426702>] __lookup_hash+0x42/0x60
>>>>> [10919.517288] [<ffffffffb942d519>] SYSC_renameat2+0x3a9/0x5a0
>>>>> [10919.517296] [<ffffffffb94d3753>] ? selinux_file_free_security+0x2
>>>>> 3/0x30
>>>>> [10919.517304] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [10919.517309] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [10919.517313] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [10919.517318] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [10919.517323] [<ffffffffb942e58e>] SyS_renameat2+0xe/0x10
>>>>> [10919.517328] [<ffffffffb942e5ce>] SyS_rename+0x1e/0x20
>>>>> [10919.517333] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [10919.517339] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [11159.496095] INFO: task glusteriotwr9:15482 blocked for more than
>>>>> 120 seconds.
>>>>> [11159.497546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [11159.498978] glusteriotwr9 D ffff971fa0fa1fa0 0 15482 1
>>>>> 0x00000080
>>>>> [11159.498984] Call Trace:
>>>>> [11159.498995] [<ffffffffb9911f00>] ? bit_wait+0x50/0x50
>>>>> [11159.498999] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [11159.499003] [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0
>>>>> [11159.499056] [<ffffffffc05dd9b7>] ? xfs_iext_bno_to_ext+0xa7/0x1a0
>>>>> [xfs]
>>>>> [11159.499082] [<ffffffffc05dd43e>] ? xfs_iext_bno_to_irec+0x8e/0xd0
>>>>> [xfs]
>>>>> [11159.499090] [<ffffffffb92f7a12>] ? ktime_get_ts64+0x52/0xf0
>>>>> [11159.499093] [<ffffffffb9911f00>] ? bit_wait+0x50/0x50
>>>>> [11159.499097] [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130
>>>>> [11159.499101] [<ffffffffb9913528>] io_schedule+0x18/0x20
>>>>> [11159.499104] [<ffffffffb9911f11>] bit_wait_io+0x11/0x50
>>>>> [11159.499107] [<ffffffffb9911ac1>] __wait_on_bit_lock+0x61/0xc0
>>>>> [11159.499113] [<ffffffffb9393634>] __lock_page+0x74/0x90
>>>>> [11159.499118] [<ffffffffb92bc210>] ? wake_bit_function+0x40/0x40
>>>>> [11159.499121] [<ffffffffb9394154>] __find_lock_page+0x54/0x70
>>>>> [11159.499125] [<ffffffffb9394e85>] grab_cache_page_write_begin+0x
>>>>> 55/0xc0
>>>>> [11159.499130] [<ffffffffb9484b76>] iomap_write_begin+0x66/0x100
>>>>> [11159.499135] [<ffffffffb9484edf>] iomap_write_actor+0xcf/0x1d0
>>>>> [11159.499140] [<ffffffffb9484e10>] ? iomap_write_end+0x80/0x80
>>>>> [11159.499144] [<ffffffffb94854e7>] iomap_apply+0xb7/0x150
>>>>> [11159.499149] [<ffffffffb9485621>] iomap_file_buffered_write+0xa1
>>>>> /0xe0
>>>>> [11159.499153] [<ffffffffb9484e10>] ? iomap_write_end+0x80/0x80
>>>>> [11159.499182] [<ffffffffc05f025d>] xfs_file_buffered_aio_write+0x12d/0x2c0
>>>>> [xfs]
>>>>> [11159.499213] [<ffffffffc05f057d>] xfs_file_aio_write+0x18d/0x1b0
>>>>> [xfs]
>>>>> [11159.499217] [<ffffffffb941a533>] do_sync_write+0x93/0xe0
>>>>> [11159.499222] [<ffffffffb941b010>] vfs_write+0xc0/0x1f0
>>>>> [11159.499225] [<ffffffffb941c002>] SyS_pwrite64+0x92/0xc0
>>>>> [11159.499230] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [11159.499234] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [11159.499238] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [11279.488720] INFO: task xfsaild/dm-10:1134 blocked for more than 120
>>>>> seconds.
>>>>> [11279.490197] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [11279.491665] xfsaild/dm-10 D ffff9720a8660fd0 0 1134 2
>>>>> 0x00000000
>>>>> [11279.491671] Call Trace:
>>>>> [11279.491682] [<ffffffffb92a3a2e>] ? try_to_del_timer_sync+0x5e/0x90
>>>>> [11279.491688] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [11279.491744] [<ffffffffc060de36>] _xfs_log_force+0x1c6/0x2c0 [xfs]
>>>>> [11279.491750] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [11279.491783] [<ffffffffc0619fec>] ? xfsaild+0x16c/0x6f0 [xfs]
>>>>> [11279.491817] [<ffffffffc060df5c>] xfs_log_force+0x2c/0x70 [xfs]
>>>>> [11279.491849] [<ffffffffc0619e80>] ? xfs_trans_ail_cursor_first+0x90/0x90
>>>>> [xfs]
>>>>> [11279.491880] [<ffffffffc0619fec>] xfsaild+0x16c/0x6f0 [xfs]
>>>>> [11279.491913] [<ffffffffc0619e80>] ? xfs_trans_ail_cursor_first+0x90/0x90
>>>>> [xfs]
>>>>> [11279.491919] [<ffffffffb92bb161>] kthread+0xd1/0xe0
>>>>> [11279.491926] [<ffffffffb92bb090>] ? insert_kthread_work+0x40/0x40
>>>>> [11279.491932] [<ffffffffb9920677>] ret_from_fork_nospec_begin+0x2
>>>>> 1/0x21
>>>>> [11279.491936] [<ffffffffb92bb090>] ? insert_kthread_work+0x40/0x40
>>>>> [11279.491976] INFO: task glusterclogfsyn:14934 blocked for more than
>>>>> 120 seconds.
>>>>> [11279.493466] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [11279.494952] glusterclogfsyn D ffff97209832af70 0 14934 1
>>>>> 0x00000080
>>>>> [11279.494957] Call Trace:
>>>>> [11279.494979] [<ffffffffc0309839>] ? __split_and_process_bio+0x2e9/0x520
>>>>> [dm_mod]
>>>>> [11279.494983] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [11279.494987] [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0
>>>>> [11279.494997] [<ffffffffc0309d98>] ? dm_make_request+0x128/0x1a0
>>>>> [dm_mod]
>>>>> [11279.495001] [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130
>>>>> [11279.495005] [<ffffffffb99145ad>] wait_for_completion_io+0xfd/0x140
>>>>> [11279.495010] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [11279.495016] [<ffffffffb951e574>] blkdev_issue_flush+0xb4/0x110
>>>>> [11279.495049] [<ffffffffc06064b9>] xfs_blkdev_issue_flush+0x19/0x20
>>>>> [xfs]
>>>>> [11279.495079] [<ffffffffc05eec40>] xfs_file_fsync+0x1b0/0x1e0 [xfs]
>>>>> [11279.495086] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>>>>> [11279.495090] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [11279.495094] [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20
>>>>> [11279.495098] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [11279.495102] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [11279.495105] INFO: task glusterposixfsy:14941 blocked for more than
>>>>> 120 seconds.
>>>>> [11279.496606] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [11279.498114] glusterposixfsy D ffff972495f84f10 0 14941 1
>>>>> 0x00000080
>>>>> [11279.498118] Call Trace:
>>>>> [11279.498134] [<ffffffffc0309839>] ? __split_and_process_bio+0x2e9/0x520
>>>>> [dm_mod]
>>>>> [11279.498138] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [11279.498142] [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0
>>>>> [11279.498152] [<ffffffffc0309d98>] ? dm_make_request+0x128/0x1a0
>>>>> [dm_mod]
>>>>> [11279.498156] [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130
>>>>> [11279.498160] [<ffffffffb99145ad>] wait_for_completion_io+0xfd/0x140
>>>>> [11279.498165] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [11279.498169] [<ffffffffb951e574>] blkdev_issue_flush+0xb4/0x110
>>>>> [11279.498202] [<ffffffffc06064b9>] xfs_blkdev_issue_flush+0x19/0x20
>>>>> [xfs]
>>>>> [11279.498231] [<ffffffffc05eec40>] xfs_file_fsync+0x1b0/0x1e0 [xfs]
>>>>> [11279.498238] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>>>>> [11279.498242] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [11279.498246] [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20
>>>>> [11279.498250] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [11279.498254] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [11279.498257] INFO: task glusteriotwr1:14950 blocked for more than
>>>>> 120 seconds.
>>>>> [11279.499789] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [11279.501343] glusteriotwr1 D ffff97208b6daf70 0 14950 1
>>>>> 0x00000080
>>>>> [11279.501348] Call Trace:
>>>>> [11279.501353] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [11279.501390] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340
>>>>> [xfs]
>>>>> [11279.501396] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [11279.501428] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>>>>> [11279.501432] [<ffffffffb944ef3f>] generic_write_sync+0x4f/0x70
>>>>> [11279.501461] [<ffffffffc05f0545>] xfs_file_aio_write+0x155/0x1b0
>>>>> [xfs]
>>>>> [11279.501466] [<ffffffffb941a533>] do_sync_write+0x93/0xe0
>>>>> [11279.501471] [<ffffffffb941b010>] vfs_write+0xc0/0x1f0
>>>>> [11279.501475] [<ffffffffb941c002>] SyS_pwrite64+0x92/0xc0
>>>>> [11279.501479] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [11279.501483] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [11279.501489] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [11279.501493] INFO: task glusteriotwr4:14953 blocked for more than
>>>>> 120 seconds.
>>>>> [11279.503047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [11279.504630] glusteriotwr4 D ffff972499f2bf40 0 14953 1
>>>>> 0x00000080
>>>>> [11279.504635] Call Trace:
>>>>> [11279.504640] [<ffffffffb9913f79>] schedule+0x29/0x70
>>>>> [11279.504676] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340
>>>>> [xfs]
>>>>> [11279.504681] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20
>>>>> [11279.504710] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs]
>>>>> [11279.504714] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0
>>>>> [11279.504718] [<ffffffffb992076f>] ? system_call_after_swapgs+0xbc/
>>>>> 0x160
>>>>> [11279.504722] [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20
>>>>> [11279.504725] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21
>>>>> [11279.504730] [<ffffffffb992077b>] ? system_call_after_swapgs+0xc8/
>>>>> 0x160
>>>>> [12127.466494] perf: interrupt took too long (8263 > 8150), lowering
>>>>> kernel.perf_event_max_sample_rate to 24000
>>>>>
>>>>> --------------------
>>>>> I think this is the cause of the massive ovirt performance issues
>>>>> irrespective of gluster volume. At the time this happened, I was also
>>>>> ssh'ed into the host, and was doing some rpm querry commands. I had just
>>>>> run rpm -qa |grep glusterfs (to verify what version was actually
>>>>> installed), and that command took almost 2 minutes to return! Normally it
>>>>> takes less than 2 seconds. That is all pure local SSD IO, too....
>>>>>
>>>>> I'm no expert, but its my understanding that anytime a software causes
>>>>> these kinds of issues, its a serious bug in the software, even if its
>>>>> mis-handled exceptions. Is this correct?
>>>>>
>>>>> --Jim
>>>>>
>>>>> On Tue, May 29, 2018 at 3:01 PM, Jim Kusznir <jim at palousetech.com>
>>>>> wrote:
>>>>>
>>>>>> I think this is the profile information for one of the volumes that
>>>>>> lives on the SSDs and is fully operational with no down/problem disks:
>>>>>>
>>>>>> [root at ovirt2 yum.repos.d]# gluster volume profile data info
>>>>>> Brick: ovirt2.nwfiber.com:/gluster/brick2/data
>>>>>> ----------------------------------------------
>>>>>> Cumulative Stats:
>>>>>> Block Size: 256b+ 512b+
>>>>>> 1024b+
>>>>>> No. of Reads: 983 2696
>>>>>> 1059
>>>>>> No. of Writes: 0 1113
>>>>>> 302
>>>>>>
>>>>>> Block Size: 2048b+ 4096b+
>>>>>> 8192b+
>>>>>> No. of Reads: 852 88608
>>>>>> 53526
>>>>>> No. of Writes: 522 812340
>>>>>> 76257
>>>>>>
>>>>>> Block Size: 16384b+ 32768b+
>>>>>> 65536b+
>>>>>> No. of Reads: 54351 241901
>>>>>> 15024
>>>>>> No. of Writes: 21636 8656
>>>>>> 8976
>>>>>>
>>>>>> Block Size: 131072b+
>>>>>> No. of Reads: 524156
>>>>>> No. of Writes: 296071
>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>>>>>> Fop
>>>>>> --------- ----------- ----------- ----------- ------------
>>>>>> ----
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 4189
>>>>>> RELEASE
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 1257
>>>>>> RELEASEDIR
>>>>>> 0.00 46.19 us 12.00 us 187.00 us 69
>>>>>> FLUSH
>>>>>> 0.00 147.00 us 78.00 us 367.00 us 86
>>>>>> REMOVEXATTR
>>>>>> 0.00 223.46 us 24.00 us 1166.00 us 149
>>>>>> READDIR
>>>>>> 0.00 565.34 us 76.00 us 3639.00 us 88
>>>>>> FTRUNCATE
>>>>>> 0.00 263.28 us 20.00 us 28385.00 us 228
>>>>>> LK
>>>>>> 0.00 98.84 us 2.00 us 880.00 us 1198
>>>>>> OPENDIR
>>>>>> 0.00 91.59 us 26.00 us 10371.00 us 3853
>>>>>> STATFS
>>>>>> 0.00 494.14 us 17.00 us 193439.00 us 1171
>>>>>> GETXATTR
>>>>>> 0.00 299.42 us 35.00 us 9799.00 us 2044
>>>>>> READDIRP
>>>>>> 0.00 1965.31 us 110.00 us 382258.00 us 321
>>>>>> XATTROP
>>>>>> 0.01 113.40 us 24.00 us 61061.00 us 8134
>>>>>> STAT
>>>>>> 0.01 755.38 us 57.00 us 607603.00 us 3196
>>>>>> DISCARD
>>>>>> 0.05 2690.09 us 58.00 us 2704761.00 us 3206
>>>>>> OPEN
>>>>>> 0.10 119978.25 us 97.00 us 9406684.00 us 154
>>>>>> SETATTR
>>>>>> 0.18 101.73 us 28.00 us 700477.00 us 313379
>>>>>> FSTAT
>>>>>> 0.23 1059.84 us 25.00 us 2716124.00 us 38255
>>>>>> LOOKUP
>>>>>> 0.47 1024.11 us 54.00 us 6197164.00 us 81455
>>>>>> FXATTROP
>>>>>> 1.72 2984.00 us 15.00 us 37098954.00 us 103020
>>>>>> FINODELK
>>>>>> 5.92 44315.32 us 51.00 us 24731536.00 us 23957
>>>>>> FSYNC
>>>>>> 13.27 2399.78 us 25.00 us 22089540.00 us 991005
>>>>>> READ
>>>>>> 37.00 5980.43 us 52.00 us 22099889.00 us 1108976
>>>>>> WRITE
>>>>>> 41.04 5452.75 us 13.00 us 22102452.00 us 1349053
>>>>>> INODELK
>>>>>>
>>>>>> Duration: 10026 seconds
>>>>>> Data Read: 80046027759 bytes
>>>>>> Data Written: 44496632320 bytes
>>>>>>
>>>>>> Interval 1 Stats:
>>>>>> Block Size: 256b+ 512b+
>>>>>> 1024b+
>>>>>> No. of Reads: 983 2696
>>>>>> 1059
>>>>>> No. of Writes: 0 838
>>>>>> 185
>>>>>>
>>>>>> Block Size: 2048b+ 4096b+
>>>>>> 8192b+
>>>>>> No. of Reads: 852 85856
>>>>>> 51575
>>>>>> No. of Writes: 382 705802
>>>>>> 57812
>>>>>>
>>>>>> Block Size: 16384b+ 32768b+
>>>>>> 65536b+
>>>>>> No. of Reads: 52673 232093
>>>>>> 14984
>>>>>> No. of Writes: 13499 4908
>>>>>> 4242
>>>>>>
>>>>>> Block Size: 131072b+
>>>>>> No. of Reads: 460040
>>>>>> No. of Writes: 6411
>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>>>>>> Fop
>>>>>> --------- ----------- ----------- ----------- ------------
>>>>>> ----
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 2093
>>>>>> RELEASE
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 1093
>>>>>> RELEASEDIR
>>>>>> 0.00 53.38 us 26.00 us 111.00 us 16
>>>>>> FLUSH
>>>>>> 0.00 145.14 us 78.00 us 367.00 us 71
>>>>>> REMOVEXATTR
>>>>>> 0.00 190.96 us 114.00 us 298.00 us 71
>>>>>> SETATTR
>>>>>> 0.00 213.38 us 24.00 us 1145.00 us 90
>>>>>> READDIR
>>>>>> 0.00 263.28 us 20.00 us 28385.00 us 228
>>>>>> LK
>>>>>> 0.00 101.76 us 2.00 us 880.00 us 1093
>>>>>> OPENDIR
>>>>>> 0.01 93.60 us 27.00 us 10371.00 us 3090
>>>>>> STATFS
>>>>>> 0.02 537.47 us 17.00 us 193439.00 us 1038
>>>>>> GETXATTR
>>>>>> 0.03 297.44 us 35.00 us 9799.00 us 1990
>>>>>> READDIRP
>>>>>> 0.03 2357.28 us 110.00 us 382258.00 us 253
>>>>>> XATTROP
>>>>>> 0.04 385.93 us 58.00 us 47593.00 us 2091
>>>>>> OPEN
>>>>>> 0.04 114.86 us 24.00 us 61061.00 us 7715
>>>>>> STAT
>>>>>> 0.06 444.59 us 57.00 us 333240.00 us 3053
>>>>>> DISCARD
>>>>>> 0.42 316.24 us 25.00 us 290728.00 us 29823
>>>>>> LOOKUP
>>>>>> 0.73 257.92 us 54.00 us 344812.00 us 63296
>>>>>> FXATTROP
>>>>>> 1.37 98.30 us 28.00 us 67621.00 us 313172
>>>>>> FSTAT
>>>>>> 1.58 2124.69 us 51.00 us 849200.00 us 16717
>>>>>> FSYNC
>>>>>> 5.73 162.46 us 52.00 us 748492.00 us 794079
>>>>>> WRITE
>>>>>> 7.19 2065.17 us 16.00 us 37098954.00 us 78381
>>>>>> FINODELK
>>>>>> 36.44 886.32 us 25.00 us 2216436.00 us 925421
>>>>>> READ
>>>>>> 46.30 1178.04 us 13.00 us 1700704.00 us 884635
>>>>>> INODELK
>>>>>>
>>>>>> Duration: 7485 seconds
>>>>>> Data Read: 71250527215 bytes
>>>>>> Data Written: 5119903744 bytes
>>>>>>
>>>>>> Brick: ovirt3.nwfiber.com:/gluster/brick2/data
>>>>>> ----------------------------------------------
>>>>>> Cumulative Stats:
>>>>>> Block Size: 1b+
>>>>>> No. of Reads: 0
>>>>>> No. of Writes: 3264419
>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>>>>>> Fop
>>>>>> --------- ----------- ----------- ----------- ------------
>>>>>> ----
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 90
>>>>>> FORGET
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 9462
>>>>>> RELEASE
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 4254
>>>>>> RELEASEDIR
>>>>>> 0.00 50.52 us 13.00 us 190.00 us 71
>>>>>> FLUSH
>>>>>> 0.00 186.97 us 87.00 us 713.00 us 86
>>>>>> REMOVEXATTR
>>>>>> 0.00 79.32 us 33.00 us 189.00 us 228
>>>>>> LK
>>>>>> 0.00 220.98 us 129.00 us 513.00 us 86
>>>>>> SETATTR
>>>>>> 0.01 259.30 us 26.00 us 2632.00 us 137
>>>>>> READDIR
>>>>>> 0.02 322.76 us 145.00 us 2125.00 us 321
>>>>>> XATTROP
>>>>>> 0.03 109.55 us 2.00 us 1258.00 us 1193
>>>>>> OPENDIR
>>>>>> 0.05 70.21 us 21.00 us 431.00 us 3196
>>>>>> DISCARD
>>>>>> 0.05 169.26 us 21.00 us 2315.00 us 1545
>>>>>> GETXATTR
>>>>>> 0.12 176.85 us 63.00 us 2844.00 us 3206
>>>>>> OPEN
>>>>>> 0.61 303.49 us 90.00 us 3085.00 us 9633
>>>>>> FSTAT
>>>>>> 2.44 305.66 us 28.00 us 3716.00 us 38230
>>>>>> LOOKUP
>>>>>> 4.52 266.22 us 55.00 us 53424.00 us 81455
>>>>>> FXATTROP
>>>>>> 6.96 1397.99 us 51.00 us 64822.00 us 23889
>>>>>> FSYNC
>>>>>> 16.48 84.74 us 25.00 us 6917.00 us 932592
>>>>>> WRITE
>>>>>> 30.16 106.90 us 13.00 us 3920189.00 us 1353046
>>>>>> INODELK
>>>>>> 38.55 1794.52 us 14.00 us 16210553.00 us 103039
>>>>>> FINODELK
>>>>>>
>>>>>> Duration: 66562 seconds
>>>>>> Data Read: 0 bytes
>>>>>> Data Written: 3264419 bytes
>>>>>>
>>>>>> Interval 1 Stats:
>>>>>> Block Size: 1b+
>>>>>> No. of Reads: 0
>>>>>> No. of Writes: 794080
>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>>>>>> Fop
>>>>>> --------- ----------- ----------- ----------- ------------
>>>>>> ----
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 2093
>>>>>> RELEASE
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 1093
>>>>>> RELEASEDIR
>>>>>> 0.00 70.31 us 26.00 us 125.00 us 16
>>>>>> FLUSH
>>>>>> 0.00 193.10 us 103.00 us 713.00 us 71
>>>>>> REMOVEXATTR
>>>>>> 0.01 227.32 us 133.00 us 513.00 us 71
>>>>>> SETATTR
>>>>>> 0.01 79.32 us 33.00 us 189.00 us 228
>>>>>> LK
>>>>>> 0.01 259.83 us 35.00 us 1138.00 us 89
>>>>>> READDIR
>>>>>> 0.03 318.26 us 145.00 us 2047.00 us 253
>>>>>> XATTROP
>>>>>> 0.04 112.67 us 3.00 us 1258.00 us 1093
>>>>>> OPENDIR
>>>>>> 0.06 167.98 us 23.00 us 1951.00 us 1014
>>>>>> GETXATTR
>>>>>> 0.08 70.97 us 22.00 us 431.00 us 3053
>>>>>> DISCARD
>>>>>> 0.13 183.78 us 66.00 us 2844.00 us 2091
>>>>>> OPEN
>>>>>> 1.01 303.82 us 90.00 us 3085.00 us 9610
>>>>>> FSTAT
>>>>>> 3.27 316.59 us 30.00 us 3716.00 us 29820
>>>>>> LOOKUP
>>>>>> 5.83 265.79 us 59.00 us 53424.00 us 63296
>>>>>> FXATTROP
>>>>>> 7.95 1373.89 us 51.00 us 64822.00 us 16717
>>>>>> FSYNC
>>>>>> 23.17 851.99 us 14.00 us 16210553.00 us 78555
>>>>>> FINODELK
>>>>>> 24.04 87.44 us 27.00 us 6917.00 us 794081
>>>>>> WRITE
>>>>>> 34.36 111.91 us 14.00 us 984871.00 us 886790
>>>>>> INODELK
>>>>>>
>>>>>> Duration: 7485 seconds
>>>>>> Data Read: 0 bytes
>>>>>> Data Written: 794080 bytes
>>>>>>
>>>>>>
>>>>>> -----------------------
>>>>>> Here is the data from the volume that is backed by the SHDDs and has
>>>>>> one failed disk:
>>>>>> [root at ovirt2 yum.repos.d]# gluster volume profile data-hdd info
>>>>>> Brick: 172.172.1.12:/gluster/brick3/data-hdd
>>>>>> --------------------------------------------
>>>>>> Cumulative Stats:
>>>>>> Block Size: 256b+ 512b+
>>>>>> 1024b+
>>>>>> No. of Reads: 1702 86
>>>>>> 16
>>>>>> No. of Writes: 0 767
>>>>>> 71
>>>>>>
>>>>>> Block Size: 2048b+ 4096b+
>>>>>> 8192b+
>>>>>> No. of Reads: 19 51841
>>>>>> 2049
>>>>>> No. of Writes: 76 60668
>>>>>> 35727
>>>>>>
>>>>>> Block Size: 16384b+ 32768b+
>>>>>> 65536b+
>>>>>> No. of Reads: 1744 639
>>>>>> 1088
>>>>>> No. of Writes: 8524 2410
>>>>>> 1285
>>>>>>
>>>>>> Block Size: 131072b+
>>>>>> No. of Reads: 771999
>>>>>> No. of Writes: 29584
>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>>>>>> Fop
>>>>>> --------- ----------- ----------- ----------- ------------
>>>>>> ----
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 2902
>>>>>> RELEASE
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 1517
>>>>>> RELEASEDIR
>>>>>> 0.00 197.00 us 197.00 us 197.00 us 1
>>>>>> FTRUNCATE
>>>>>> 0.00 70.24 us 16.00 us 758.00 us 51
>>>>>> FLUSH
>>>>>> 0.00 143.93 us 82.00 us 305.00 us 57
>>>>>> REMOVEXATTR
>>>>>> 0.00 178.63 us 105.00 us 712.00 us 60
>>>>>> SETATTR
>>>>>> 0.00 67.30 us 19.00 us 572.00 us 555
>>>>>> LK
>>>>>> 0.00 322.80 us 23.00 us 4673.00 us 138
>>>>>> READDIR
>>>>>> 0.00 336.56 us 106.00 us 11994.00 us 237
>>>>>> XATTROP
>>>>>> 0.00 84.70 us 28.00 us 1071.00 us 3469
>>>>>> STATFS
>>>>>> 0.01 387.75 us 2.00 us 146017.00 us 1467
>>>>>> OPENDIR
>>>>>> 0.01 148.59 us 21.00 us 64374.00 us 4454
>>>>>> STAT
>>>>>> 0.02 783.02 us 16.00 us 93502.00 us 1902
>>>>>> GETXATTR
>>>>>> 0.03 1516.10 us 17.00 us 210690.00 us 1364
>>>>>> ENTRYLK
>>>>>> 0.03 2555.47 us 300.00 us 674454.00 us 1064
>>>>>> READDIRP
>>>>>> 0.07 85.74 us 19.00 us 68340.00 us 62849
>>>>>> FSTAT
>>>>>> 0.07 1978.12 us 59.00 us 202596.00 us 2729
>>>>>> OPEN
>>>>>> 0.22 708.57 us 15.00 us 394799.00 us 25447
>>>>>> LOOKUP
>>>>>> 5.94 2331.74 us 15.00 us 1099530.00 us 207534
>>>>>> FINODELK
>>>>>> 7.31 8311.75 us 58.00 us 1800216.00 us 71668
>>>>>> FXATTROP
>>>>>> 12.49 7735.19 us 51.00 us 3595513.00 us 131642
>>>>>> WRITE
>>>>>> 17.70 957.08 us 16.00 us 13700466.00 us 1508160
>>>>>> INODELK
>>>>>> 24.55 2546.43 us 26.00 us 5077347.00 us 786060
>>>>>> READ
>>>>>> 31.56 49699.15 us 47.00 us 3746331.00 us 51777
>>>>>> FSYNC
>>>>>>
>>>>>> Duration: 10101 seconds
>>>>>> Data Read: 101562897361 bytes
>>>>>> Data Written: 4834450432 bytes
>>>>>>
>>>>>> Interval 0 Stats:
>>>>>> Block Size: 256b+ 512b+
>>>>>> 1024b+
>>>>>> No. of Reads: 1702 86
>>>>>> 16
>>>>>> No. of Writes: 0 767
>>>>>> 71
>>>>>>
>>>>>> Block Size: 2048b+ 4096b+
>>>>>> 8192b+
>>>>>> No. of Reads: 19 51841
>>>>>> 2049
>>>>>> No. of Writes: 76 60668
>>>>>> 35727
>>>>>>
>>>>>> Block Size: 16384b+ 32768b+
>>>>>> 65536b+
>>>>>> No. of Reads: 1744 639
>>>>>> 1088
>>>>>> No. of Writes: 8524 2410
>>>>>> 1285
>>>>>>
>>>>>> Block Size: 131072b+
>>>>>> No. of Reads: 771999
>>>>>> No. of Writes: 29584
>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>>>>>> Fop
>>>>>> --------- ----------- ----------- ----------- ------------
>>>>>> ----
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 2902
>>>>>> RELEASE
>>>>>> 0.00 0.00 us 0.00 us 0.00 us 1517
>>>>>> RELEASEDIR
>>>>>> 0.00 197.00 us 197.00 us 197.00 us 1
>>>>>> FTRUNCATE
>>>>>> 0.00 70.24 us 16.00 us 758.00 us 51
>>>>>> FLUSH
>>>>>> 0.00 143.93 us 82.00 us 305.00 us 57
>>>>>> REMOVEXATTR
>>>>>> 0.00 178.63 us 105.00 us 712.00 us 60
>>>>>> SETATTR
>>>>>> 0.00 67.30 us 19.00 us 572.00 us 555
>>>>>> LK
>>>>>> 0.00 322.80 us 23.00 us 4673.00 us 138
>>>>>> READDIR
>>>>>> 0.00 336.56 us 106.00 us 11994.00 us 237
>>>>>> XATTROP
>>>>>> 0.00 84.70 us 28.00 us 1071.00 us 3469
>>>>>> STATFS
>>>>>> 0.01 387.75 us 2.00 us 146017.00 us 1467
>>>>>> OPENDIR
>>>>>> 0.01 148.59 us 21.00 us 64374.00 us 4454
>>>>>> STAT
>>>>>> 0.02 783.02 us 16.00 us 93502.00 us 1902
>>>>>> GETXATTR
>>>>>> 0.03 1516.10 us 17.00 us 210690.00 us 1364
>>>>>> ENTRYLK
>>>>>> 0.03 2555.47 us 300.00 us 674454.00 us 1064
>>>>>> READDIRP
>>>>>> 0.07 85.73 us 19.00 us 68340.00 us 62849
>>>>>> FSTAT
>>>>>> 0.07 1978.12 us 59.00 us 202596.00 us 2729
>>>>>> OPEN
>>>>>> 0.22 708.57 us 15.00 us 394799.00 us 25447
>>>>>> LOOKUP
>>>>>> 5.94 2334.57 us 15.00 us 1099530.00 us 207534
>>>>>> FINODELK
>>>>>> 7.31 8311.49 us 58.00 us 1800216.00 us 71668
>>>>>> FXATTROP
>>>>>> 12.49 7735.32 us 51.00 us 3595513.00 us 131642
>>>>>> WRITE
>>>>>> 17.71 957.08 us 16.00 us 13700466.00 us 1508160
>>>>>> INODELK
>>>>>> 24.56 2546.42 us 26.00 us 5077347.00 us 786060
>>>>>> READ
>>>>>> 31.54 49651.63 us 47.00 us 3746331.00 us 51777
>>>>>> FSYNC
>>>>>>
>>>>>> Duration: 10101 seconds
>>>>>> Data Read: 101562897361 bytes
>>>>>> Data Written: 4834450432 bytes
>>>>>>
>>>>>>
>>>>>> On Tue, May 29, 2018 at 2:55 PM, Jim Kusznir <jim at palousetech.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you for your response.
>>>>>>>
>>>>>>> I have 4 gluster volumes. 3 are replica 2 + arbitrator. replica
>>>>>>> bricks are on ovirt1 and ovirt2, arbitrator on ovirt3. The 4th volume is
>>>>>>> replica 3, with a brick on all three ovirt machines.
>>>>>>>
>>>>>>> The first 3 volumes are on an SSD disk; the 4th is on a Seagate SSHD
>>>>>>> (same in all three machines). On ovirt3, the SSHD has reported hard IO
>>>>>>> failures, and that brick is offline. However, the other two replicas are
>>>>>>> fully operational (although they still show contents in the heal info
>>>>>>> command that won't go away, but that may be the case until I replace the
>>>>>>> failed disk).
>>>>>>>
>>>>>>> What is bothering me is that ALL 4 gluster volumes are showing
>>>>>>> horrible performance issues. At this point, as the bad disk has been
>>>>>>> completely offlined, I would expect gluster to perform at normal speed, but
>>>>>>> that is definitely not the case.
>>>>>>>
>>>>>>> I've also noticed that the performance hits seem to come in waves:
>>>>>>> things seem to work acceptably (but slow) for a while, then suddenly, its
>>>>>>> as if all disk IO on all volumes (including non-gluster local OS disk
>>>>>>> volumes for the hosts) pause for about 30 seconds, then IO resumes again.
>>>>>>> During those times, I start getting VM not responding and host not
>>>>>>> responding notices as well as the applications having major issues.
>>>>>>>
>>>>>>> I've shut down most of my VMs and am down to just my essential core
>>>>>>> VMs (shedded about 75% of my VMs). I still am experiencing the same issues.
>>>>>>>
>>>>>>> Am I correct in believing that once the failed disk was brought
>>>>>>> offline that performance should return to normal?
>>>>>>>
>>>>>>> On Tue, May 29, 2018 at 1:27 PM, Alex K <rightkicktech at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I would check disks status and accessibility of mount points where
>>>>>>>> your gluster volumes reside.
>>>>>>>>
>>>>>>>> On Tue, May 29, 2018, 22:28 Jim Kusznir <jim at palousetech.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On one ovirt server, I'm now seeing these messages:
>>>>>>>>> [56474.239725] blk_update_request: 63 callbacks suppressed
>>>>>>>>> [56474.239732] blk_update_request: I/O error, dev dm-2, sector 0
>>>>>>>>> [56474.240602] blk_update_request: I/O error, dev dm-2, sector
>>>>>>>>> 3905945472
>>>>>>>>> [56474.241346] blk_update_request: I/O error, dev dm-2, sector
>>>>>>>>> 3905945584
>>>>>>>>> [56474.242236] blk_update_request: I/O error, dev dm-2, sector 2048
>>>>>>>>> [56474.243072] blk_update_request: I/O error, dev dm-2, sector
>>>>>>>>> 3905943424
>>>>>>>>> [56474.243997] blk_update_request: I/O error, dev dm-2, sector
>>>>>>>>> 3905943536
>>>>>>>>> [56474.247347] blk_update_request: I/O error, dev dm-2, sector 0
>>>>>>>>> [56474.248315] blk_update_request: I/O error, dev dm-2, sector
>>>>>>>>> 3905945472
>>>>>>>>> [56474.249231] blk_update_request: I/O error, dev dm-2, sector
>>>>>>>>> 3905945584
>>>>>>>>> [56474.250221] blk_update_request: I/O error, dev dm-2, sector 2048
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 29, 2018 at 11:59 AM, Jim Kusznir <jim at palousetech.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> I see in messages on ovirt3 (my 3rd machine, the one upgraded to
>>>>>>>>>> 4.2):
>>>>>>>>>>
>>>>>>>>>> May 29 11:54:41 ovirt3 ovs-vsctl: ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock:
>>>>>>>>>> database connection failed (No such file or directory)
>>>>>>>>>> May 29 11:54:51 ovirt3 ovs-vsctl: ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock:
>>>>>>>>>> database connection failed (No such file or directory)
>>>>>>>>>> May 29 11:55:01 ovirt3 ovs-vsctl: ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock:
>>>>>>>>>> database connection failed (No such file or directory)
>>>>>>>>>> (appears a lot).
>>>>>>>>>>
>>>>>>>>>> I also found on the ssh session of that, some sysv warnings about
>>>>>>>>>> the backing disk for one of the gluster volumes (straight replica 3). The
>>>>>>>>>> glusterfs process for that disk on that machine went offline. Its my
>>>>>>>>>> understanding that it should continue to work with the other two machines
>>>>>>>>>> while I attempt to replace that disk, right? Attempted writes (touching an
>>>>>>>>>> empty file) can take 15 seconds, repeating it later will be much faster.
>>>>>>>>>>
>>>>>>>>>> Gluster generates a bunch of different log files, I don't know
>>>>>>>>>> what ones you want, or from which machine(s).
>>>>>>>>>>
>>>>>>>>>> How do I do "volume profiling"?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> On Tue, May 29, 2018 at 11:53 AM, Sahina Bose <sabose at redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Do you see errors reported in the mount logs for the volume? If
>>>>>>>>>>> so, could you attach the logs?
>>>>>>>>>>> Any issues with your underlying disks. Can you also attach
>>>>>>>>>>> output of volume profiling?
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 30, 2018 at 12:13 AM, Jim Kusznir <
>>>>>>>>>>> jim at palousetech.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Ok, things have gotten MUCH worse this morning. I'm getting
>>>>>>>>>>>> random errors from VMs, right now, about a third of my VMs have been paused
>>>>>>>>>>>> due to storage issues, and most of the remaining VMs are not performing
>>>>>>>>>>>> well.
>>>>>>>>>>>>
>>>>>>>>>>>> At this point, I am in full EMERGENCY mode, as my production
>>>>>>>>>>>> services are now impacted, and I'm getting calls coming in with problems...
>>>>>>>>>>>>
>>>>>>>>>>>> I'd greatly appreciate help...VMs are running VERY slowly (when
>>>>>>>>>>>> they run), and they are steadily getting worse. I don't know why. I was
>>>>>>>>>>>> seeing CPU peaks (to 100%) on several VMs, in perfect sync, for a few
>>>>>>>>>>>> minutes at a time (while the VM became unresponsive and any VMs I was
>>>>>>>>>>>> logged into that were linux were giving me the CPU stuck messages in my
>>>>>>>>>>>> origional post). Is all this storage related?
>>>>>>>>>>>>
>>>>>>>>>>>> I also have two different gluster volumes for VM storage, and
>>>>>>>>>>>> only one had the issues, but now VMs in both are being affected at the same
>>>>>>>>>>>> time and same way.
>>>>>>>>>>>>
>>>>>>>>>>>> --Jim
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, May 28, 2018 at 10:50 PM, Sahina Bose <
>>>>>>>>>>>> sabose at redhat.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> [Adding gluster-users to look at the heal issue]
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, May 29, 2018 at 9:17 AM, Jim Kusznir <
>>>>>>>>>>>>> jim at palousetech.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've been having some cluster and gluster performance issues
>>>>>>>>>>>>>> lately. I also found that my cluster was out of date, and was trying to
>>>>>>>>>>>>>> apply updates (hoping to fix some of these), and discovered the ovirt 4.1
>>>>>>>>>>>>>> repos were taken completely offline. So, I was forced to begin an upgrade
>>>>>>>>>>>>>> to 4.2. According to docs I found/read, I needed only add the new repo, do
>>>>>>>>>>>>>> a yum update, reboot, and be good on my hosts (did the yum update, the
>>>>>>>>>>>>>> engine-setup on my hosted engine). Things seemed to work relatively well,
>>>>>>>>>>>>>> except for a gluster sync issue that showed up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My cluster is a 3 node hyperconverged cluster. I upgraded
>>>>>>>>>>>>>> the hosted engine first, then engine 3. When engine 3 came back up, for
>>>>>>>>>>>>>> some reason one of my gluster volumes would not sync. Here's sample output:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root at ovirt3 ~]# gluster volume heal data-hdd info
>>>>>>>>>>>>>> Brick 172.172.1.11:/gluster/brick3/data-hdd
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/48d7ecb8-7ac5-4
>>>>>>>>>>>>>> 725-bca5-b3519681cf2f/0d6080b0-7018-4fa3-bb82-1dd9ef07d9b9
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4
>>>>>>>>>>>>>> cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/6da854d1-b6be-4
>>>>>>>>>>>>>> 46b-9bf0-90a0dbbea830/3c93bd1f-b7fa-4aa2-b445-6904e31839ba
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/7f647567-d18c-4
>>>>>>>>>>>>>> 4f1-a58e-9b8865833acb/f9364470-9770-4bb1-a6b9-a54861849625
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/f3c8e7aa-6ef2-4
>>>>>>>>>>>>>> 2a7-93d4-e0a4df6dd2fa/2eb0b1ad-2606-44ef-9cd3-ae59610a504b
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/b1ea3f62-0f05-4
>>>>>>>>>>>>>> ded-8c82-9c91c90e0b61/d5d6bf5a-499f-431d-9013-5453db93ed32
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/8c8b5147-e9d6-4
>>>>>>>>>>>>>> 810-b45b-185e3ed65727/16f08231-93b0-489d-a2fd-687b6bf88eaa
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/12924435-b9c2-4
>>>>>>>>>>>>>> aab-ba19-1c1bc31310ef/07b3db69-440e-491e-854c-bbfa18a7cff2
>>>>>>>>>>>>>> Status: Connected
>>>>>>>>>>>>>> Number of entries: 8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Brick 172.172.1.12:/gluster/brick3/data-hdd
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/48d7ecb8-7ac5-4
>>>>>>>>>>>>>> 725-bca5-b3519681cf2f/0d6080b0-7018-4fa3-bb82-1dd9ef07d9b9
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4
>>>>>>>>>>>>>> cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/b1ea3f62-0f05-4
>>>>>>>>>>>>>> ded-8c82-9c91c90e0b61/d5d6bf5a-499f-431d-9013-5453db93ed32
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/6da854d1-b6be-4
>>>>>>>>>>>>>> 46b-9bf0-90a0dbbea830/3c93bd1f-b7fa-4aa2-b445-6904e31839ba
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/7f647567-d18c-4
>>>>>>>>>>>>>> 4f1-a58e-9b8865833acb/f9364470-9770-4bb1-a6b9-a54861849625
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/8c8b5147-e9d6-4
>>>>>>>>>>>>>> 810-b45b-185e3ed65727/16f08231-93b0-489d-a2fd-687b6bf88eaa
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/12924435-b9c2-4
>>>>>>>>>>>>>> aab-ba19-1c1bc31310ef/07b3db69-440e-491e-854c-bbfa18a7cff2
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/f3c8e7aa-6ef2-4
>>>>>>>>>>>>>> 2a7-93d4-e0a4df6dd2fa/2eb0b1ad-2606-44ef-9cd3-ae59610a504b
>>>>>>>>>>>>>> Status: Connected
>>>>>>>>>>>>>> Number of entries: 8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Brick 172.172.1.13:/gluster/brick3/data-hdd
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/b1ea3f62-0f05-4
>>>>>>>>>>>>>> ded-8c82-9c91c90e0b61/d5d6bf5a-499f-431d-9013-5453db93ed32
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/8c8b5147-e9d6-4
>>>>>>>>>>>>>> 810-b45b-185e3ed65727/16f08231-93b0-489d-a2fd-687b6bf88eaa
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/12924435-b9c2-4
>>>>>>>>>>>>>> aab-ba19-1c1bc31310ef/07b3db69-440e-491e-854c-bbfa18a7cff2
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/f3c8e7aa-6ef2-4
>>>>>>>>>>>>>> 2a7-93d4-e0a4df6dd2fa/2eb0b1ad-2606-44ef-9cd3-ae59610a504b
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4
>>>>>>>>>>>>>> cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/48d7ecb8-7ac5-4
>>>>>>>>>>>>>> 725-bca5-b3519681cf2f/0d6080b0-7018-4fa3-bb82-1dd9ef07d9b9
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/6da854d1-b6be-4
>>>>>>>>>>>>>> 46b-9bf0-90a0dbbea830/3c93bd1f-b7fa-4aa2-b445-6904e31839ba
>>>>>>>>>>>>>> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/7f647567-d18c-4
>>>>>>>>>>>>>> 4f1-a58e-9b8865833acb/f9364470-9770-4bb1-a6b9-a54861849625
>>>>>>>>>>>>>> Status: Connected
>>>>>>>>>>>>>> Number of entries: 8
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------
>>>>>>>>>>>>>> Its been in this state for a couple days now, and bandwidth
>>>>>>>>>>>>>> monitoring shows no appreciable data moving. I've tried repeatedly
>>>>>>>>>>>>>> commanding a full heal from all three clusters in the node. Its always the
>>>>>>>>>>>>>> same files that need healing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When running gluster volume heal data-hdd statistics, I see
>>>>>>>>>>>>>> sometimes different information, but always some number of "heal failed"
>>>>>>>>>>>>>> entries. It shows 0 for split brain.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not quite sure what to do. I suspect it may be due to
>>>>>>>>>>>>>> nodes 1 and 2 still being on the older ovirt/gluster release, but I'm
>>>>>>>>>>>>>> afraid to upgrade and reboot them until I have a good gluster sync (don't
>>>>>>>>>>>>>> need to create a split brain issue). How do I proceed with this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Second issue: I've been experiencing VERY POOR performance on
>>>>>>>>>>>>>> most of my VMs. To the tune that logging into a windows 10 vm via remote
>>>>>>>>>>>>>> desktop can take 5 minutes, launching quickbooks inside said vm can easily
>>>>>>>>>>>>>> take 10 minutes. On some linux VMs, I get random messages like this:
>>>>>>>>>>>>>> Message from syslogd at unifi at May 28 20:39:23 ...
>>>>>>>>>>>>>> kernel:[6171996.308904] NMI watchdog: BUG: soft lockup -
>>>>>>>>>>>>>> CPU#0 stuck for 22s! [mongod:14766]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (the process and PID are often different)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not quite sure what to do about this either. My initial
>>>>>>>>>>>>>> thought was upgrad everything to current and see if its still there, but I
>>>>>>>>>>>>>> cannot move forward with that until my gluster is healed...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>> --Jim
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Users mailing list -- users at ovirt.org
>>>>>>>>>>>>>> To unsubscribe send an email to users-leave at ovirt.org
>>>>>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>>>>>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/communit
>>>>>>>>>>>>>> y/about/community-guidelines/
>>>>>>>>>>>>>> List Archives: https://lists.ovirt.org/archiv
>>>>>>>>>>>>>> es/list/users at ovirt.org/message/3LEV6ZQ3JV2XLAL7NYBTXOYMYUOT
>>>>>>>>>>>>>> IRQF/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list -- users at ovirt.org
>>>>>>>>> To unsubscribe send an email to users-leave at ovirt.org
>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/communit
>>>>>>>>> y/about/community-guidelines/
>>>>>>>>> List Archives: https://lists.ovirt.org/archiv
>>>>>>>>> es/list/users at ovirt.org/message/ACO7RFSLBSRBAIONIC2HQ6Z24ZDES5MF/
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Users mailing list -- users at ovirt.org
>>> To unsubscribe send an email to users-leave at ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct: https://www.ovirt.org/communit
>>> y/about/community-guidelines/
>>> List Archives: https://lists.ovirt.org/archiv
>>> es/list/users at ovirt.org/message/JKBSLSZFRMFGILXGZB4YAMJIJKEKNGMA/
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180530/803a67b4/attachment-0001.html>
More information about the Gluster-users
mailing list