<div dir="ltr"><div>Hi,</div><div><br></div>There is not much information in this log file. Which server's file is this? I will need to see the rebalance logs from both nodes.<div><br></div><div>It sounds like there are a lot of files on the volume which is why the rebalance will take time. What is the current rebalance status for the volume?</div><div><br></div><div>Rebalance should not affect volume operations so is there a particular reason why the estimated time is a cause of concern?</div><div><br></div><div><br></div><div>Regards,</div><div>Nithya</div><div><br><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 30 April 2018 at 13:10, shadowsocks飞飞 <span dir="ltr"><<a href="mailto:kiwizhang618@gmail.com" target="_blank">kiwizhang618@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div dir="ltr">
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">I cannot calculate the number of files normally</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div>Through df -i I got the approximate number of files is <span> </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">63694442<span> </span></span></div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span><br></span></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div>[root@CentOS-73-64-minimal ~]# df -i</div><div>Filesystem Inodes IUsed IFree IUse% Mounted on</div><div>/dev/md2 131981312 30901030 101080282 24% /</div><div>devtmpfs 8192893 435 8192458 1% /dev</div><div>tmpfs 8199799 8029 8191770 1% /dev/shm</div><div>tmpfs 8199799 1415 8198384 1% /run</div><div>tmpfs 8199799 16 8199783 1% /sys/fs/cgroup</div><div>/dev/md3 110067712 29199861 80867851 27% /home</div><div>/dev/md1 131072 363 130709 1% /boot</div><div>gluster1:/web 2559860992 63694442 2496166550 3% /web</div><div>tmpfs 8199799 1 8199798 1% /run/user/0</div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">The rebalance log is in the attachment</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">the cluster information</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div>gluster volume status web detail</div><div>Status of volume: web</div><div>------------------------------<wbr>------------------------------<wbr>------------------</div><div>Brick : Brick gluster1:/home/export/md3/bric<wbr>k</div><div>TCP Port : 49154 </div><div>RDMA Port : 0 </div><div>Online : Y </div><div>Pid : 16730 </div><div>File System : ext4 </div><div>Device : /dev/md3 </div><div>Mount Options : rw,noatime,nodiratime,nobarrie<wbr>r,data=ordered</div><div>Inode Size : 256 </div><div>Disk Space Free : 239.4GB </div><div>Total Disk Space : 1.6TB </div><div>Inode Count : 110067712 </div><div>Free Inodes : 80867992 </div><div>------------------------------<wbr>------------------------------<wbr>------------------</div><div>Brick : Brick gluster1:/export/md2/brick</div><div>TCP Port : 49155 </div><div>RDMA Port : 0 </div><div>Online : Y </div><div>Pid : 16758 </div><div>File System : ext4 </div><div>Device : /dev/md2 </div><div>Mount Options : rw,noatime,nodiratime,nobarrie<wbr>r,data=ordered</div><div>Inode Size : 256 </div><div>Disk Space Free : 589.4GB </div><div>Total Disk Space : 1.9TB </div><div>Inode Count : 131981312 </div><div>Free Inodes : 101080484 </div><div>------------------------------<wbr>------------------------------<wbr>------------------</div><div>Brick : Brick gluster2:/home/export/md3/bric<wbr>k</div><div>TCP Port : 49152 </div><div>RDMA Port : 0 </div><div>Online : Y </div><div>Pid : 12556 </div><div>File System : xfs </div><div>Device : /dev/md3 </div><div>Mount Options : rw,noatime,nodiratime,attr2,in<wbr>ode64,sunit=1024,swidth=3072,<wbr>noquota</div><div>Inode Size : 256 </div><div>Disk Space Free : 10.7TB </div><div>Total Disk Space : 10.8TB </div><div>Inode Count : 2317811968 </div><div>Free Inodes : 2314218207 </div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Most of the files in the cluster are pictures smaller than 1M</div>
<br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">2018-04-30 15:16 GMT+08:00 Nithya Balachandran <span dir="ltr"><<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>></span>:<br></div></div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div><br></div><div>This value is an ongoing rough estimate based on the amount of data rebalance has migrated since it started. The values will cange as the rebalance progresses.</div><div>A few questions:</div><div><ol><li>How many files/dirs do you have on this volume? <br></li><li>What is the average size of the files?<br></li><li>What is the total size of the data on the volume?<br></li></ol></div><div><br></div><div>Can you send us the rebalance log?</div><div><br></div><div><br></div><div>Thanks,</div><div>Nithya</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 30 April 2018 at 10:33, kiwizhang618 <span dir="ltr"><<a href="mailto:kiwizhang618@gmail.com" target="_blank">kiwizhang618@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="font-family:"\005fae\008f6f\0096c5\009ed1";line-height:1.6">
<div> I met a big problem,the cluster rebalance takes a long time after adding a new node</div><div><br></div><div>gluster volume rebalance web status</div><div> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</div><div> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div><div> localhost 900 43.5MB 2232 0 69 in progress 0:36:49</div><div> gluster2 1052 39.3MB 4393 0 1052 in progress 0:36:49</div><div>Estimated time left for rebalance to complete : 9919:44:34</div><div>volume rebalance: web: success</div><div><br></div><div>the rebalance log</div><div><div>[glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.8 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/web --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal<wbr>=off --xlator-option *replicate*.entry-self-heal=of<wbr>f --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=d47ad89d-7979-4<wbr>ede-9aba-e04f020bb4f0 --xlator-option *dht.commit-hash=3610561770 --socket-file /var/run/gluster/gluster-rebal<wbr>ance-bdef10eb-1c83-410c-8ad3-f<wbr>e286450004b.sock --pid-file /var/lib/glusterd/vols/web/reb<wbr>alance/d47ad89d-7979-4ede-9aba<wbr>-e04f020bb4f0.pid -l /var/log/glusterfs/web-rebalan<wbr>ce.log)</div><div>[2018-04-30 04:20:45.100902] W [MSGID: 101002] [options.c:995:xl_opt_validate<wbr>] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction</div><div>[2018-04-30 04:20:45.103927] I [MSGID: 101190] [event-epoll.c:613:event_dispa<wbr>tch_epoll_worker] 0-epoll: Started thread with index 1</div><div>[2018-04-30 04:20:55.191261] E [MSGID: 109039] [dht-common.c:3113:dht_find_lo<wbr>cal_subvol_cbk] 0-web-dht: getxattr err for dir [No data available]</div><div>[2018-04-30 04:21:19.783469] E [MSGID: 109023] [dht-rebalance.c:2669:gf_defra<wbr>g_migrate_single_file] 0-web-dht: Migrate file failed: /2018/02/x187f6596-36ac-45e6-b<wbr>d7a-019804dfe427.jpg, lookup failed [Stale file handle]</div><div>The message "E [MSGID: 109039] [dht-common.c:3113:dht_find_lo<wbr>cal_subvol_cbk] 0-web-dht: getxattr err for dir [No data available]" repeated 2 times between [2018-04-30 04:20:55.191261] and [2018-04-30 04:20:55.193615]</div></div><div><br></div><div>the gluster info</div><div><div>Volume Name: web</div><div>Type: Distribute</div><div>Volume ID: bdef10eb-1c83-410c-8ad3-fe2864<wbr>50004b</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: gluster1:/home/export/md3/bric<wbr>k</div><div>Brick2: gluster1:/export/md2/brick</div><div>Brick3: gluster2:/home/export/md3/bric<wbr>k</div><div>Options Reconfigured:</div><div>nfs.trusted-sync: on</div><div>nfs.trusted-write: on</div><div>cluster.rebal-throttle: aggressive</div><div>features.inode-quota: off</div><div>features.quota: off</div><div>cluster.shd-wait-qlength: 1024</div><div>transport.address-family: inet</div><div>cluster.lookup-unhashed: auto</div><div>performance.cache-size: 1GB</div><div>performance.client-io-threads: on</div><div>performance.write-behind-windo<wbr>w-size: 4MB</div><div>performance.io-thread-count: 8</div><div>performance.force-readdirp: on</div><div>performance.readdir-ahead: on</div><div>cluster.readdir-optimize: on</div><div>performance.high-prio-threads: 8</div><div>performance.flush-behind: on</div><div>performance.write-behind: on</div><div>performance.quick-read: off</div><div>performance.io-cache: on</div><div>performance.read-ahead: off</div><div>server.event-threads: 8</div><div>cluster.lookup-optimize: on</div><div>features.cache-invalidation: on</div><div>features.cache-invalidation-ti<wbr>meout: 600</div><div>performance.stat-prefetch: off</div><div>performance.md-cache-timeout: 60</div><div>network.inode-lru-limit: 90000</div><div>diagnostics.brick-log-level: ERROR</div><div>diagnostics.brick-sys-log-leve<wbr>l: ERROR</div><div>diagnostics.client-log-level: ERROR</div><div>diagnostics.client-sys-log-lev<wbr>el: ERROR</div><div>cluster.min-free-disk: 20%</div><div>cluster.self-heal-window-size: 16</div><div>cluster.self-heal-readdir-size<wbr>: 1024</div><div>cluster.background-self-heal-c<wbr>ount: 4</div><div>cluster.heal-wait-queue-length<wbr>: 128</div><div>client.event-threads: 8</div><div>performance.cache-invalidation<wbr>: on</div><div>nfs.disable: off</div><div>nfs.acl: off</div><div>cluster.brick-multiplex: disable</div></div><div><br></div>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>