<div dir="ltr">
<div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">I met a big problem,the cluster rebalance takes a long time after adding a new node</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif"></div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">gluster volume rebalance web status</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> localhost 900 43.5MB 2232 0 69 in progress 0:36:49</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> gluster2 1052 39.3MB 4393 0 1052 in progress 0:36:49</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Estimated time left for rebalance to complete : 9919:44:34</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">volume rebalance: web: success</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif"></div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">the rebalance log</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">[glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.8 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/web --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=d47ad89d-7979-4ede-9aba-e04f020bb4f0 --xlator-option *dht.commit-hash=3610561770 --socket-file /var/run/gluster/gluster-rebalance-bdef10eb-1c83-410c-8ad3-fe286450004b.sock --pid-file /var/lib/glusterd/vols/web/rebalance/d47ad89d-7979-4ede-9aba-e04f020bb4f0.pid -l /var/log/glusterfs/web-rebalance.log)</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">[2018-04-30 04:20:45.100902] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">[2018-04-30 04:20:45.103927] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">[2018-04-30 04:20:55.191261] E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk] 0-web-dht: getxattr err for dir [No data available]</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">[2018-04-30 04:21:19.783469] E [MSGID: 109023] [dht-rebalance.c:2669:gf_defrag_migrate_single_file] 0-web-dht: Migrate file failed: /2018/02/x187f6596-36ac-45e6-bd7a-019804dfe427.jpg, lookup failed [Stale file handle]</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">The message "E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk] 0-web-dht: getxattr err for dir [No data available]" repeated 2 times between [2018-04-30 04:20:55.191261] and [2018-04-30 04:20:55.193615]</div></div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif"></div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">the gluster info</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif;color:rgb(49,53,59);font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:22.4px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Volume Name: web</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Type: Distribute</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Volume ID: bdef10eb-1c83-410c-8ad3-fe286450004b</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Status: Started</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Snapshot Count: 0</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Number of Bricks: 3</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Transport-type: tcp</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Bricks:</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Brick1: gluster1:/home/export/md3/brick</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Brick2: gluster1:/export/md2/brick</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Brick3: gluster2:/home/export/md3/brick</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">Options Reconfigured:</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">nfs.trusted-sync: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">nfs.trusted-write: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.rebal-throttle: aggressive</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">features.inode-quota: off</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">features.quota: off</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.shd-wait-qlength: 1024</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">transport.address-family: inet</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.lookup-unhashed: auto</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.cache-size: 1GB</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.client-io-threads: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.write-behind-window-size: 4MB</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.io-thread-count: 8</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.force-readdirp: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.readdir-ahead: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.readdir-optimize: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.high-prio-threads: 8</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.flush-behind: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.write-behind: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.quick-read: off</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.io-cache: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.read-ahead: off</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">server.event-threads: 8</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.lookup-optimize: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">features.cache-invalidation: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">features.cache-invalidation-timeout: 600</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.stat-prefetch: off</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.md-cache-timeout: 60</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">network.inode-lru-limit: 90000</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">diagnostics.brick-log-level: ERROR</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">diagnostics.brick-sys-log-level: ERROR</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">diagnostics.client-log-level: ERROR</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">diagnostics.client-sys-log-level: ERROR</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.min-free-disk: 20%</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.self-heal-window-size: 16</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.self-heal-readdir-size: 1024</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.background-self-heal-count: 4</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.heal-wait-queue-length: 128</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">client.event-threads: 8</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">performance.cache-invalidation: on</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">nfs.disable: off</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">nfs.acl: off</div><div style="font-family:Verdana,"Microsoft Yahei",SimSun,sans-serif">cluster.brick-multiplex: disable</div></div>
<br></div>