[Gluster-users] High load on glusterfs client

Krutika Dhananjay kdhananj at redhat.com
Tue Mar 15 03:28:28 UTC 2016


Looks like a case of gfid split-brain.

What does the output of `gluster volume heal <VOL> info split-brain` say?

-Krutika

On Mon, Mar 14, 2016 at 1:05 PM, <Sebastian.Gumprich at t-systems.com> wrote:

> Today, after setting the self-heal options to off, the problem happened
> again.
>
>
>
> Here’s what happens when I try to heal the problems:
>
>
>
> [root at nfs02 gluster volume heal opt info
>
> Brick nfs01:/opt/bkk
>
> Number of entries: 0
>
>
>
> Brick nfs02:/opt/bkk
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Standalone_template_source_948126c20bcb68e1839bd1cb11b2326776a5a89a.php
>
> <gfid:869bab4e-f83f-4876-b619-a7c008361bd2>
>
> <gfid:e5cd468a-c3cc-453f-bd10-67d4474bd57c>
>
> <gfid:e792eef8-a58f-4808-985b-dc906a707593>
>
> <gfid:7ab28ace-94ae-468e-9a11-0b5a3a42a9d7>
>
> <gfid:ae890cd6-97b8-493a-a6ea-bedf45707f17>
>
> <gfid:7bd13cdb-4c6c-4aa5-a7df-1f3cf78bfca3>
>
> <gfid:d7f63c6b-0b69-4271-827e-16bb1bfa689e>
>
> <gfid:a128f009-1edc-47fb-8e53-9490887f9937>
>
> <gfid:d0a0d264-ebc2-4370-8114-4732fb894e66>
>
> <gfid:317f6a2e-f243-4ead-a4f4-8508b70d76d2>
>
> <gfid:082a3550-15ad-4719-a550-c60c7f8c8791>
>
> <gfid:e5d7c0fb-42e5-4f81-aebd-9f55fb12d8cf>
>
> <gfid:4ec4830b-c65e-451b-a754-d48bd4c385a6>
>
> <gfid:f0453756-50b4-423f-84b5-82bf840ae465>
>
> <gfid:264d0bba-7208-4866-9820-e561439afbb8>
>
> <gfid:ebe43f9f-6b55-40bb-8a59-750ab3dc7b1f>
>
> <gfid:6f235178-5981-42d8-83fe-ce222c22600a>
>
> <gfid:c2ee0feb-8edc-4fe3-bda6-55845c6ab351>
>
> /releases/1.0.1/typo3temp/locks/b3e7327e62654b234caf95033dd5f5db
>
> <gfid:f450c66b-d6ae-4c65-a95a-f13ef5285900>
>
> <gfid:98aabb13-dafe-474c-87e0-2741645a1e6b>
>
> <gfid:1bebfba1-13c1-4c22-8b1b-3d34e5d7b968>
>
> <gfid:97e5a8cc-7c66-47b0-8dbd-9f9098b419d9>
>
> <gfid:438a2995-32d2-4a21-ae5d-45593a74addf>
>
> <gfid:008f5f63-9dda-408b-abf7-2201b029462e>
>
> <gfid:f89a5507-f9c7-45f1-8760-ea9a897c4659>
>
> <gfid:3da89478-9c08-4c57-a44f-0df2dc59efb4>
>
> <gfid:f0969ff6-f49a-4497-a24f-648d4e130775>
>
> <gfid:6c7f9d1e-64e5-43f7-a454-9f823641f0ae>
>
> <gfid:1d7c6cc0-fd07-43bb-8caf-30e80544e516>
>
> /releases/1.0.1/typo3temp/locks/dabccbb64a6426cf9b1005b79f5765a9
>
> Number of entries: 32
>
>
>
> [root at nfs02 ~]# gluster volume heal opt full
>
> Launching heal operation to perform full self heal on volume opt has been
> unsuccessful
>
>
>
> [root at nfs02 ~]# gluster volume heal opt
>
> Launching heal operation to perform index self heal on volume opt has been
> successful
>
> Use heal info commands to check status
>
>
>
> [root at nfs02 ~]# gluster volume heal opt info
>
> Brick nfs01:/opt/bkk
>
> Number of entries: 0
>
>
>
> Brick nfs02:/opt/bkk
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Standalone_template_source_948126c20bcb68e1839bd1cb11b2326776a5a89a.php
>
> <gfid:869bab4e-f83f-4876-b619-a7c008361bd2>
>
> <gfid:e5cd468a-c3cc-453f-bd10-67d4474bd57c>
>
> <gfid:e792eef8-a58f-4808-985b-dc906a707593>
>
> <gfid:7ab28ace-94ae-468e-9a11-0b5a3a42a9d7>
>
> <gfid:ae890cd6-97b8-493a-a6ea-bedf45707f17>
>
> <gfid:7bd13cdb-4c6c-4aa5-a7df-1f3cf78bfca3>
>
> <gfid:d7f63c6b-0b69-4271-827e-16bb1bfa689e>
>
> <gfid:a128f009-1edc-47fb-8e53-9490887f9937>
>
> <gfid:d0a0d264-ebc2-4370-8114-4732fb894e66>
>
> <gfid:317f6a2e-f243-4ead-a4f4-8508b70d76d2>
>
> <gfid:082a3550-15ad-4719-a550-c60c7f8c8791>
>
> <gfid:e5d7c0fb-42e5-4f81-aebd-9f55fb12d8cf>
>
> <gfid:4ec4830b-c65e-451b-a754-d48bd4c385a6>
>
> <gfid:f0453756-50b4-423f-84b5-82bf840ae465>
>
> <gfid:264d0bba-7208-4866-9820-e561439afbb8>
>
> <gfid:ebe43f9f-6b55-40bb-8a59-750ab3dc7b1f>
>
> <gfid:6f235178-5981-42d8-83fe-ce222c22600a>
>
> <gfid:c2ee0feb-8edc-4fe3-bda6-55845c6ab351>
>
> /releases/1.0.1/typo3temp/locks/b3e7327e62654b234caf95033dd5f5db
>
> <gfid:f450c66b-d6ae-4c65-a95a-f13ef5285900>
>
> <gfid:98aabb13-dafe-474c-87e0-2741645a1e6b>
>
> <gfid:1bebfba1-13c1-4c22-8b1b-3d34e5d7b968>
>
> <gfid:97e5a8cc-7c66-47b0-8dbd-9f9098b419d9>
>
> <gfid:438a2995-32d2-4a21-ae5d-45593a74addf>
>
> <gfid:008f5f63-9dda-408b-abf7-2201b029462e>
>
> <gfid:f89a5507-f9c7-45f1-8760-ea9a897c4659>
>
> <gfid:3da89478-9c08-4c57-a44f-0df2dc59efb4>
>
> <gfid:f0969ff6-f49a-4497-a24f-648d4e130775>
>
> <gfid:6c7f9d1e-64e5-43f7-a454-9f823641f0ae>
>
> <gfid:1d7c6cc0-fd07-43bb-8caf-30e80544e516>
>
> Number of entries: 31
>
>
>
>
>
> Here’s the log to it:
>
> [2016-03-14 07:25:34.593287] W [MSGID: 108008]
> [afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check]
> 0-opt-replicate-0: GFID mismatch for
> <gfid:7bfa065f-50c7-43d6-b513-df6cd76fa144>/Standalone_layout_Mail_fa22189c871e1e7c1a36b59209ef0812d5e709e9.php
> 082a3550-15ad-4719-a550-c60c7f8c8791 on opt-client-1 and
> c2f289ca-6058-46ae-af8a-18491975eb7d on opt-client-0
>
>
>
>
>
> [root at -nfs02 ~]# gluster volume info
>
>
>
> Volume Name: opt
>
> Type: Replicate
>
> Volume ID: 5b77070f-5378-45ec-9eda-5f7dd007ff8a
>
> Status: Started
>
> Number of Bricks: 1 x 2 = 2
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: nfs01:/opt/bkk
>
> Brick2: nfs02:/opt/bkk
>
> Options Reconfigured:
>
> cluster.metadata-self-heal: off
>
> cluster.data-self-heal: off
>
> cluster.entry-self-heal: off
>
> performance.md-cache-timeout: 1
>
> performance.cache-max-file-size: 2MB
>
> performance.io-thread-count: 16
>
> network.ping-timeout: 42
>
> performance.write-behind-window-size: 4MB
>
> performance.read-ahead: off
>
> performance.cache-refresh-timeout: 10
>
> performance.cache-size: 512MB
>
> performance.quick-read: off
>
> performance.readdir-ahead: on
>
>
>
>
>
> Any other ideas?
>
>
>
> Thanks!
>
>
>
> *Von:* Krutika Dhananjay [mailto:kdhananj at redhat.com]
> *Gesendet:* Dienstag, 8. März 2016 04:04
> *An:* Gumprich, Sebastian
> *Cc:* gluster-users at gluster.org
> *Betreff:* Re: [Gluster-users] High load on glusterfs client
>
>
>
> Could you try disabling client-side heals and see if it works for you?
>
> Here's what you'd need to do:
>
> #gluster volume set <VOL> entry-self-heal off
>
> #gluster volume set <VOL> data-self-heal off
>
> #gluster volume set <VOL> metadata-self-heal off
>
> -Krutika
>
>
>
> On Wed, Mar 2, 2016 at 12:37 AM, <Sebastian.Gumprich at t-systems.com> wrote:
>
> Hello everyone,
>
>
>
> I’m experiencing high load on our glusterfs clients.
>
>
>
> Here’s the setup:
>
>
>
> There are to glusterfs server:
>
>
>
> Nfs01 and nfs02 with the following configuration:
>
>
>
> [root nfs01 ~]# gluster volume info opt
>
>
>
> Volume Name: opt
>
> Type: Replicate
>
> Volume ID: 5b77070f-5378-45ec-9eda-5f7dd007ff8a
>
> Status: Started
>
> Number of Bricks: 1 x 2 = 2
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1:  nfs01:/opt/bkk
>
> Brick2: nfs02:/opt/bkk
>
> Options Reconfigured:
>
> performance.readdir-ahead: on
>
> performance.quick-read: off
>
> performance.cache-size: 512MB
>
> performance.cache-refresh-timeout: 10
>
> performance.read-ahead: off
>
> performance.write-behind-window-size: 4MB
>
> network.ping-timeout: 2
>
> performance.io-thread-count: 16
>
> performance.cache-max-file-size: 2MB
>
> performance.md-cache-timeout: 1
>
>
>
> Then there are two clients (web01 and web02) that mount the brick via a
> virtual ip-address (nfs-VIP):
>
> nfs-VIP:/opt on /opt/bkk type fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
>
>
>
> operating system on all server is CentOS Linux release 7.2.1511 (Core).
>
> Glusterfs version is glusterfs 3.7.6 built on Nov  9 2015 15:20:26
>
>
>
> On the brick lies the PHP dynamic webcontent from a typo3 CMS.
>
>
>
> On the client (web01) the following is logged in the gluster.log:
>
>
>
> iner_08850598886fb5f39c9cf1d269d7e20677f97ede.php>,
> e09948dd-1e9b-4430-8f55-3df64cda2385 on opt-client-1 and
> ba80a475-7b83-4c83-bd0c-798a108bfb63 on opt-client-0. Skipping conservative
> merge on the file.
>
> [2016-03-01 18:40:50.570040] E [MSGID: 108008]
> [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-opt-replicate-0: Gfid mismatch detected for
> <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_layout_Detail_html_bd113d9c433c8f88376e47547db3b94e698a5ecd.php>,
> 739ee14c-2d5d-458b-bffd-83595bfcbe6a on opt-client-1 and
> 5a311733-731e-4478-ad3c-a70fbf66ba30 on opt-client-0. Skipping conservative
> merge on the file.
>
> [2016-03-01 18:40:50.572992] E [MSGID: 108008]
> [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-opt-replicate-0: Gfid mismatch detected for
> <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_partial_Detail_FalMediaContainer_9c1b3fd40fca9019726b3f6b8bc04618ffadab7b.php>,
> bb6907a1-ce80-4e03-92df-6fbc69d24a4d on opt-client-1 and
> 6f88aa67-cb81-4c26-94b2-e3aaa8704e8d on opt-client-0. Skipping conservative
> merge on the file.
>
> [2016-03-01 18:40:50.791704] E [MSGID: 108008]
> [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-opt-replicate-0: Gfid mismatch detected for
> <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_action_create_f40464a6a7f73d86cda514065167d59a7ddece73.php>,
> 5e5b224b-ea20-4d38-8504-61b24f5d6a3b on opt-client-1 and
> fab07af5-2aa5-4873-a6e2-6265ec78e304 on opt-client-0. Skipping conservative
> merge on the file.
>
> [2016-03-01 18:40:54.085964] E [MSGID: 108008]
> [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-opt-replicate-0: Gfid mismatch detected for
> <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_action_detail_8d30b654cd8343fe40616b8a2f8a5343b1ed776e.php>,
> 4d75a687-b9ab-4f97-b698-38668d1981ae on opt-client-1 and
> 110b315e-2e28-4859-a8b9-e0f1629faa3c on opt-client-0. Skipping conservative
> merge on the file.
>
> [2016-03-01 18:40:56.153651] E [MSGID: 108008]
> [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-opt-replicate-0: Gfid mismatch detected for
> <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_layout_Default_aae217b167ad82f4b1258bb01fa73f305844dbd8.php>,
> 6f7e2709-8c14-486a-85a2-a3cb48af4ca5 on opt-client-1 and
> 6ab62408-0406-4834-96b9-a51e18441d4c on opt-client-0. Skipping conservative
> merge on the file.
>
> [2016-03-01 18:41:05.476126] I [MSGID: 108026]
> [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-opt-replicate-0:
> performing entry selfheal on 7a922c37-48d0-4dfb-8abb-18a435c948af
>
> [2016-03-01 18:41:05.597093] I [MSGID: 108026]
> [afr-self-heal-common.c:651:afr_log_selfheal] 0-opt-replicate-0: Completed
> entry selfheal on 7a922c37-48d0-4dfb-8abb-18a435c948af. source=1 sinks=0
>
> [2016-03-01 18:41:05.790944] E [MSGID: 108008]
> [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-opt-replicate-0: Gfid mismatch detected for
> <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php>,
> 118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and
> a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0. Skipping conservative
> merge on the file.
>
> [2016-03-01 18:41:06.649695] W [MSGID: 108008]
> [afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check]
> 0-opt-replicate-0: GFID mismatch for
> <gfid:4c6dda77-6a2b-4996-bca4-9ace4cee45cc>/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
> 118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and
> a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0
>
> [2016-03-01 18:41:06.661277] W [fuse-bridge.c:462:fuse_entry_cbk]
> 0-glusterfs-fuse: 184415191: LOOKUP()
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
> => -1 (Input/output error)
>
> [2016-03-01 18:41:06.680968] W [fuse-bridge.c:462:fuse_entry_cbk]
> 0-glusterfs-fuse: 184422672: LOOKUP()
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
> => -1 (Input/output error)
>
> [2016-03-01 18:41:06.680222] W [MSGID: 108008]
> [afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check]
> 0-opt-replicate-0: GFID mismatch for
> <gfid:4c6dda77-6a2b-4996-bca4-9ace4cee45cc>/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
> 118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and
> a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0
>
>
>
> There are many more of these entries, this is just a really small excerpt.
> The files that have a mismatch are tempary php-cache files.
>
> When I delete these files, the load goes down and the files in the volume
> heal info become less (see below).
>
>
>
> Here’s the output of gluster volume heal opt info. Note that this output
> is **after** deleting most of the cache files, before that there were
> many more entries.
>
>
>
> [root at nfs01 fluid_template]# gluster volume heal opt info
>
> Brick nfs01:/opt/bkk
>
> <gfid:23fc1027-0aec-4b84-9ffb-c164a9d43d20>
>
> <gfid:92cb9dde-2721-4c11-93a6-2582ed9edd5d>
>
> <gfid:a0dbcf8a-67f8-4870-ab57-3d5d1218601c>
>
> <gfid:947cbcc4-1978-4b9e-b726-2acd0a4fda5a>
>
> <gfid:440b9b36-bad5-4cb6-b935-8a004132340a>
>
> Number of entries: 5
>
>
>
> Brick nfs02:/opt/bkk
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template - Possibly undergoing
> heal
>
>
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Select_7bf809152d985037de761d8d375d286e44b4f13a.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_GoogleAdwordsConversion_f7254aeb252ea43cd89f9051b5a43109d47938f1.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_File_4b3a3f667c475577847aa77118f3af5666ecb2c6.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Input_b3e08744b23680f0a14e60e716f9994d9580e3f4.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_action_detail_8d30b654cd8343fe40616b8a2f8a5343b1ed776e.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_layout_Detail_html_bd113d9c433c8f88376e47547db3b94e698a5ecd.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_Opengraph_b98680f3686dccf00e22181e66d11ca9de7a44bd.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_FalMediaContainer_9c1b3fd40fca9019726b3f6b8bc04618ffadab7b.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_MediaContainer_08850598886fb5f39c9cf1d269d7e20677f97ede.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Text_c10766db8d335d5cd9555878aef5d886dcb6926e.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_create_f40464a6a7f73d86cda514065167d59a7ddece73.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_layout_Default_aae217b167ad82f4b1258bb01fa73f305844dbd8.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_HoneyPod_fc83c414f744612c3cb44c8827372a30f17791d0.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Textarea_39d24d8e3e2813636dfff2a89b7cefb8e9117c97.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Submit_86e69c50ccebf20584db2e3c74859373c53d320f.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Check_a2a11c64ac58dab16eab29e4cda88518c15a4d25.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_FormError_7cade8e8fc1d23c761360c0efbe8cb145eed2e39.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Radio_d038a263b5ea81f0f7795e1e47516e5e2937cbd9.php
>
>
> /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Hidden_a7651f5498e0d36b4e2eae5fca015ef0e9365067.php
>
> Number of entries: 21
>
>
>
> Here are some heal-infos during the high load:
>
> Starting time of crawl: Tue Mar  1 19:09:45 2016
>
>
>
> Ending time of crawl: Tue Mar  1 19:09:52 2016
>
>
>
> Type of crawl: INDEX
>
> No. of entries healed: 2
>
> No. of entries in split-brain: 0
>
> No. of heal failed entries: 168
>
>
>
>
>
> And here’s the performance monitoring info during 60 seconds of high load:
>
>
>
> Brick: nfs01:/opt/bkk
>
> ------------------------------
>
> Cumulative Stats:
>
>    Block Size:                  1b+                   2b+
>                 4b+
>
> No. of Reads:                   18                    33
> 308
>
> No. of Writes:                   64                    88
> 2994
>
>
>
>    Block Size:                  8b+                  16b+
> 32b+
>
> No. of Reads:                  117                   154
> 15612
>
> No. of Writes:                  370                   369
> 1432
>
>
>
>    Block Size:                 64b+                 128b+
> 256b+
>
> No. of Reads:                 3721                 12884
> 19917
>
> No. of Writes:                 7585                900221
> 135011
>
>
>
>    Block Size:                512b+                1024b+
> 2048b+
>
> No. of Reads:                23929                 12251
> 19835
>
> No. of Writes:                63067                 30950
> 23540
>
>
>
>    Block Size:               4096b+                8192b+
> 16384b+
>
> No. of Reads:                 9096                  9449
> 5566
>
> No. of Writes:                40455                 36397
> 13926
>
>
>
>    Block Size:              32768b+               65536b+
> 131072b+
>
> No. of Reads:                 5159                  6055
> 34001
>
> No. of Writes:                20722                  6600
> 12762
>
>
>
> %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
> Fop
>
> ---------   -----------   -----------   -----------   ------------
> ----
>
>       0.00       0.00 us       0.00 us       0.00 us         212065
> FORGET
>
>       0.00       0.00 us       0.00 us       0.00 us        4118713
> RELEASE
>
>       0.00       0.00 us       0.00 us       0.00 us        9931097
> RELEASEDIR
>
>       0.00      47.00 us      47.00 us      47.00 us              1
> GETXATTR
>
>       0.00     124.00 us     124.00 us     124.00 us              1
> XATTROP
>
>       0.00     144.00 us     144.00 us     144.00 us              1
> UNLINK
>
>       0.00      37.17 us      35.00 us      41.00 us              6
> STATFS
>
>       0.01      36.84 us      32.00 us      61.00 us             19
> FSTAT
>
>       0.01      48.80 us      44.00 us      62.00 us             20
> STAT
>
>       0.05      75.52 us      54.00 us     156.00 us             48
> REMOVEXATTR
>
>       0.05      81.69 us      69.00 us     146.00 us             48
> SETATTR
>
>       0.05      37.16 us      14.00 us     436.00 us            109
> FLUSH
>
>       0.05      38.31 us      18.00 us     115.00 us            108
> FINODELK
>
>       0.06      73.16 us      46.00 us     170.00 us             63
> OPEN
>
>       0.10      38.70 us      20.00 us     167.00 us            195
> INODELK
>
>       0.11     335.37 us      40.00 us     563.00 us             27
> READDIR
>
>       0.15      50.58 us      27.00 us     392.00 us            232
> OPENDIR
>
>       0.15      81.74 us      35.00 us     215.00 us            144
> FXATTROP
>
>       0.16     130.53 us      68.00 us     697.00 us            100
> WRITE
>
>       0.93    1532.04 us     171.00 us   15356.00 us             48
> CREATE
>
>       1.40     284.90 us      25.00 us    1146.00 us            390
> READDIRP
>
>      10.20      52.29 us      28.00 us    2323.00 us          15482
> READLINK
>
>      18.94      33.25 us      11.00 us   27242.00 us          45219
> ENTRYLK
>
>      67.58      93.74 us      32.00 us   27521.00 us          57223
> LOOKUP
>
>
>
>     Duration: 6492593 seconds
>
>    Data Read: 5679496942 bytes
>
> Data Written: 4510536316 bytes
>
>
>
> Interval 1 Stats:
>
>    Block Size:               4096b+                8192b+
> 16384b+
>
> No. of Reads:                    0
> 0                     0
>
> No. of Writes:                    1
> 50                     2
>
>
>
>    Block Size:              32768b+
>
> No. of Reads:                    0
>
> No. of Writes:                   47
>
> %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
> Fop
>
> ---------   -----------   -----------   -----------   ------------
> ----
>
>       0.00       0.00 us       0.00 us       0.00 us             48
> FORGET
>
>       0.00       0.00 us       0.00 us       0.00 us            104
> RELEASE
>
>       0.00       0.00 us       0.00 us       0.00 us            231
> RELEASEDIR
>
>       0.00     124.00 us     124.00 us     124.00 us              1
> XATTROP
>
>       0.00     144.00 us     144.00 us     144.00 us              1
> UNLINK
>
>       0.00      36.40 us      35.00 us      37.00 us              5
> STATFS
>
>       0.01      36.84 us      32.00 us      61.00 us             19
> FSTAT
>
>       0.01      48.80 us      44.00 us      62.00 us             20
> STAT
>
>       0.05      75.52 us      54.00 us     156.00 us             48
> REMOVEXATTR
>
>       0.05      37.69 us      14.00 us     436.00 us            101
> FLUSH
>
>       0.06      81.69 us      69.00 us     146.00 us             48
> SETATTR
>
>       0.06      74.26 us      46.00 us     170.00 us             53
> OPEN
>
>       0.06      38.31 us      18.00 us     115.00 us            108
> FINODELK
>
>       0.10     311.33 us      40.00 us     563.00 us             24
> READDIR
>
>       0.11      38.70 us      20.00 us     167.00 us            195
> INODELK
>
>       0.16      50.58 us      27.00 us     392.00 us            231
> OPENDIR
>
>       0.17      81.74 us      35.00 us     215.00 us            144
> FXATTROP
>
>       0.18     130.53 us      68.00 us     697.00 us            100
> WRITE
>
>       1.03    1532.04 us     171.00 us   15356.00 us             48
> CREATE
>
>       1.56     284.90 us      25.00 us    1146.00 us            390
> READDIRP
>
>      11.31      52.27 us      28.00 us    2323.00 us          15395
> READLINK
>
>      18.24      33.67 us      11.00 us   27242.00 us          38571
> ENTRYLK
>
>      66.83      94.49 us      32.00 us   27521.00 us          50338
> LOOKUP
>
>
>
>     Duration: 68 seconds
>
>    Data Read: 0 bytes
>
> Data Written: 3347998 bytes
>
>
>
> Brick: nfs02:/opt/bkk
>
> ------------------------------
>
> Cumulative Stats:
>
>    Block Size:                  1b+
> 2b+                   4b+
>
> No. of Reads:                   26                    49
> 541
>
> No. of Writes:                   64                    94
> 3848
>
>
>
>    Block Size:                  8b+                  16b+
> 32b+
>
> No. of Reads:                  218                   205
> 1267
>
> No. of Writes:                  452                   417
> 1448
>
>
>
>    Block Size:                 64b+                 128b+
> 256b+
>
> No. of Reads:                 6097                 39042
> 11111
>
> No. of Writes:                 8617                924503
> 136768
>
>
>
>    Block Size:                512b+                1024b+
> 2048b+
>
> No. of Reads:               120819                 37802
> 16506
>
> No. of Writes:                64399                 35996
> 24999
>
>
>
>    Block Size:               4096b+                8192b+
> 16384b+
>
> No. of Reads:                76162                 20449
> 10948
>
> No. of Writes:                41302                 37488
>               14034
>
>
>
>    Block Size:              32768b+               65536b+
> 131072b+
>
> No. of Reads:                 7733                  7306
> 31648
>
> No. of Writes:                20849                  6750
> 12886
>
>
>
> %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
> Fop
>
> ---------   -----------   -----------   -----------   ------------
> ----
>
>       0.00       0.00 us       0.00 us       0.00 us         231622
> FORGET
>
>       0.00       0.00 us       0.00 us       0.00 us        6123626
> RELEASE
>
>       0.00       0.00 us       0.00 us       0.00 us       10869781
> RELEASEDIR
>
>       0.00     116.00 us     116.00 us     116.00 us              1
> XATTROP
>
>       0.00      40.00 us      38.00 us      43.00 us              6
> STATFS
>
>       0.01      40.85 us      33.00 us      97.00 us             13
> FSTAT
>
>       0.01      46.26 us      29.00 us     100.00 us             23
> STAT
>
>       0.04      73.96 us      53.00 us     150.00 us             48
> REMOVEXATTR
>
>       0.04      76.85 us      61.00 us      99.00 us             48
> SETATTR
>
>       0.04      35.83 us      28.00 us     155.00 us            103
> UNLINK
>
>       0.04      36.28 us      15.00 us     142.00 us            112
> FINODELK
>
>       0.05      38.68 us      13.00 us     220.00 us            133
> FLUSH
>
>       0.09     324.11 us      28.00 us     589.00 us             28
> READDIR
>
>       0.12      85.88 us      36.00 us     215.00 us            144
> FXATTROP
>
>       0.12     124.55 us      76.00 us     192.00 us            100
> WRITE
>
>       0.13      78.76 us      19.00 us     529.00 us            161
> GETXATTR
>
>       0.20      78.65 us      43.00 us     384.00 us            261
> OPEN
>
>       0.23      54.09 us       2.00 us     260.00 us            426
> OPENDIR
>
>       0.60    1261.23 us     174.00 us   10655.00 us             48
> CREATE
>
>       0.66      81.15 us      17.00 us    9254.00 us            819
> INODELK
>
>       1.21     279.97 us      23.00 us    1587.00 us            434
> READDIRP
>
>       8.10      52.61 us      27.00 us    1283.00 us          15496
> READLINK
>
>      15.48      34.54 us      10.00 us   13810.00 us          45133
> ENTRYLK
>
>      72.84     104.30 us      14.00 us   14613.00 us          70322
> LOOKUP
>
>
>
>     Duration: 6492593 seconds
>
>    Data Read: 6308054987 bytes
>
> Data Written: 4579768980 bytes
>
>
>
> Interval 1 Stats:
>
>    Block Size:               4096b+                8192b+
>         16384b+
>
> No. of Reads:                    0
> 0                     0
>
> No. of Writes:                    1
> 50                     2
>
>
>
>    Block Size:              32768b+
>
> No. of Reads:                    0
>
> No. of Writes:                   47
>
> %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
> Fop
>
> ---------   -----------   -----------   -----------   ------------
> ----
>
>       0.00       0.00 us       0.00 us       0.00 us             48
> FORGET
>
>       0.00       0.00 us       0.00 us       0.00 us            286
> RELEASE
>
>       0.00       0.00 us       0.00 us       0.00 us            400
> RELEASEDIR
>
>       0.00     116.00 us     116.00 us     116.00 us              1
> XATTROP
>
>       0.00      40.40 us      38.00 us      43.00 us              5
> STATFS
>
>       0.01      40.85 us      33.00 us      97.00 us             13
> FSTAT
>
>       0.01      46.14 us      29.00 us     100.00 us             22
> STAT
>
>       0.04      73.96 us      53.00 us     150.00 us             48
> REMOVEXATTR
>
>       0.04      76.85 us      61.00 us      99.00 us             48
> SETATTR
>
>       0.04      35.83 us      28.00 us     155.00 us            103
> UNLINK
>
>       0.05      36.28 us      15.00 us     142.00 us            112
> FINODELK
>
>       0.05      39.64 us      13.00 us     220.00 us            117
> FLUSH
>
>       0.10     330.28 us      28.00 us     589.00 us             25
> READDIR
>
>       0.14      85.88 us      36.00 us     215.00 us            144
> FXATTROP
>
>       0.14     124.55 us      76.00 us     192.00 us            100
> WRITE
>
>       0.14      79.13 us      19.00 us     529.00 us            159
> GETXATTR
>
>       0.22      79.51 us      43.00 us     384.00 us            235
> OPEN
>
>       0.25      54.27 us       2.00 us     260.00 us            400
> OPENDIR
>
>       0.70    1261.23 us     174.00 us   10655.00 us             48
> CREATE
>
>       0.77      81.15 us      17.00 us    9254.00 us            819
> INODELK
>
>       1.26     283.31 us      23.00 us    1587.00 us            386
> READDIRP
>
>       7.67      52.88 us      27.00 us    1283.00 us          12595
> READLINK
>
>      15.47      34.83 us      10.00 us   13810.00 us          38576
> ENTRYLK
>
>      72.91     105.05 us      14.00 us   14613.00 us          60295
> LOOKUP
>
>
>
>     Duration: 68 seconds
>
>    Data Read: 0 bytes
>
> Data Written: 3347998 bytes
>
>
>
>
>
>
>
> Can anybody tell me how to fix the problem with the high load and these
> cache files?
>
>
>
> Thanks in advance!
>
>
>
> Regards
>
> Sebastian
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160315/a983ea89/attachment.html>


More information about the Gluster-users mailing list