<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hi Mauro,<br></div><div><br></div><div>Yes, I can provide you step by step procedure to correct it. <br></div><div>Is it fine If i provide you the steps tomorrow as it is quite late over here and I don't want to miss anything in hurry?<br></div><div><br></div><div>---<br></div><div>Ashish<br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Mauro Tridici" <mauro.tridici@cmcc.it><br><b>To: </b>"Ashish Pandey" <aspandey@redhat.com><br><b>Cc: </b>"gluster-users" <gluster-users@gluster.org><br><b>Sent: </b>Wednesday, September 26, 2018 6:54:19 PM<br><b>Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br><div><br></div><div class=""><br class=""></div><div class="">Hi Ashish,</div><div class=""><br class=""></div><div class="">in attachment you can find the rebalance log file and the last updated brick log file (the other files in /var/log/glusterfs/bricks directory seem to be too old).</div><div class="">I just stopped the running rebalance (as you can see at the bottom of the rebalance log file).</div><div class="">So, if exists a safe procedure to correct the problem I would like execute it.</div><div class=""><br class=""></div><div class="">I don’t know if I can ask you it, but, if it is possible, could you please describe me step by step the right procedure to remove the newly added bricks without losing the data that have been already rebalanced?</div><div class=""><br class=""></div><div class="">The following outputs show the result of “df -h” command executed on one of the first 3 nodes (s01, s02, s03) already existing and on one of the last 3 nodes (s04, s05, s06) added recently.</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s06 bricks]# df -h</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">File system Dim. Usati Dispon. Uso% Montato su</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s06-root 100G 2,1G 98G 3% /</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">devtmpfs 32G 0 32G 0% /dev</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 4,0K 32G 1% /dev/shm</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 26M 32G 1% /run</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 0 32G 0% /sys/fs/cgroup</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s06-var 100G 2,0G 99G 2% /var</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s06-gluster 100G 33M 100G 1% /gluster</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/sda1 1014M 152M 863M 15% /boot</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgd-gluster_lvd 9,0T 807G 8,3T 9% /gluster/mnt3</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgg-gluster_lvg 9,0T 807G 8,3T 9% /gluster/mnt6</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgc-gluster_lvc 9,0T 807G 8,3T 9% /gluster/mnt2</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vge-gluster_lve 9,0T 807G 8,3T 9% /gluster/mnt4</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgj-gluster_lvj 9,0T 887G 8,2T 10% /gluster/mnt9</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgb-gluster_lvb 9,0T 807G 8,3T 9% /gluster/mnt1</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgh-gluster_lvh 9,0T 887G 8,2T 10% /gluster/mnt7</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgf-gluster_lvf 9,0T 807G 8,3T 9% /gluster/mnt5</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgi-gluster_lvi 9,0T 887G 8,2T 10% /gluster/mnt8</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgl-gluster_lvl 9,0T 887G 8,2T 10% /gluster/mnt11</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgk-gluster_lvk 9,0T 887G 8,2T 10% /gluster/mnt10</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgm-gluster_lvm 9,0T 887G 8,2T 10% /gluster/mnt12</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 6,3G 0 6,3G 0% /run/user/0</span></div></div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s01 ~]# df -h</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">File system Dim. Usati Dispon. Uso% Montato su</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s01-root 100G 5,3G 95G 6% /</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">devtmpfs 32G 0 32G 0% /dev</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 39M 32G 1% /dev/shm</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 26M 32G 1% /run</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 0 32G 0% /sys/fs/cgroup</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s01-var 100G 11G 90G 11% /var</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/md127 1015M 151M 865M 15% /boot</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s01-gluster 100G 33M 100G 1% /gluster</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgi-gluster_lvi 9,0T 5,5T 3,6T 61% /gluster/mnt7</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgm-gluster_lvm 9,0T 5,4T 3,6T 61% /gluster/mnt11</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgf-gluster_lvf 9,0T 5,7T 3,4T 63% /gluster/mnt4</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgl-gluster_lvl 9,0T 5,8T 3,3T 64% /gluster/mnt10</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgj-gluster_lvj 9,0T 5,5T 3,6T 61% /gluster/mnt8</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgn-gluster_lvn 9,0T 5,4T 3,6T 61% /gluster/mnt12</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgk-gluster_lvk 9,0T 5,8T 3,3T 64% /gluster/mnt9</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgh-gluster_lvh 9,0T 5,6T 3,5T 63% /gluster/mnt6</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgg-gluster_lvg 9,0T 5,6T 3,5T 63% /gluster/mnt5</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vge-gluster_lve 9,0T 5,7T 3,4T 63% /gluster/mnt3</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgc-gluster_lvc 9,0T 5,6T 3,5T 62% /gluster/mnt1</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgd-gluster_lvd 9,0T 5,6T 3,5T 62% /gluster/mnt2</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 6,3G 0 6,3G 0% /run/user/0</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s01-stg:tier2 420T 159T 262T 38% /tier2</span></div></div><div class=""><br class=""></div><div class="">As you can see, used space value of each brick of the last servers is about 800GB.</div><div class=""><br class=""></div><div class="">Thank you,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br></div><br><div><br></div><br><div class=""><br></div><br class=""><div><blockquote class=""><div class="">Il giorno 26 set 2018, alle ore 14:51, Ashish Pandey <<a href="mailto:aspandey@redhat.com" class="" target="_blank" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">Hi Mauro,<br class=""></div><div class=""><br class=""></div><div class="">rebalance and brick logs should be the first thing we should go through.<br class=""></div><div class=""><br class=""></div><div class="">There is a procedure to correct the configuration/setup but the situation you are in is difficult to follow that procedure.<br class=""></div><div class="">You should have added the bricks hosted on <span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg</span>, <span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s05-stg and <span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s06-stg the same way you had the previous configuration.</span></span><br class=""></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">That means 2 bricks on each node for one subvolume.<br class=""></span></span></div><div class="">The procedure will require a lot of replace bricks which will again need healing and all. In addition to that we have to wait for re-balance to complete.<br class=""></div><div class=""><br class=""></div><div class="">I would suggest that if whole data has not been rebalanced and if you can stop the rebalance and remove these newly added bricks properly then you should remove these newly added bricks.<br class=""></div><div class="">After that, add these bricks so that you have 2 bricks of each volume on 3 newly added nodes.<br class=""></div><div class=""><br class=""></div><div class="">Yes, it is like undoing whole effort but it is better to do it now then facing issues in future when it will be almost impossible to correct these things if you have lots of data.</div><div class=""><br class=""></div><div class="">---<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><div style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" class="" target="_blank" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" class="" target="_blank" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" class="" target="_blank" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Wednesday, September 26, 2018 5:55:02 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Dear Ashish,</div><div class=""><br class=""></div><div class="">thank you for you answer.</div><div class="">I could provide you the entire log file related to glusterd, glusterfsd and rebalance.</div><div class="">Please, could you indicate which one you need first?</div><div class=""><br class=""></div><div class="">Yes, we added the last 36 bricks after creating vol. Is there a procedure to correct this error? Is it still possible to do it?</div><div class=""><br class=""></div><div class="">Many thanks,</div><div class="">Mauro</div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 26 set 2018, alle ore 14:13, Ashish Pandey <<a href="mailto:aspandey@redhat.com" class="" target="_blank" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class=""><br class=""></div><div class="">I think we don't have enough logs to debug this so I would suggest you to provide more logs/info.<br class=""></div><div class="">I have also observed that the configuration and setup of your volume is not very efficient.<br class=""></div><div class=""><br class=""></div><div class="">For example: <br class=""></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick37: s04-stg:/gluster/mnt1/brick</span><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick38: s04-stg:/gluster/mnt2/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick39: s04-stg:/gluster/mnt3/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick40: s04-stg:/gluster/mnt4/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick41: s04-stg:/gluster/mnt5/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick42: s04-stg:/gluster/mnt6/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick43: s04-stg:/gluster/mnt7/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick44: s04-stg:/gluster/mnt8/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick45: s04-stg:/gluster/mnt9/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick46: s04-stg:/gluster/mnt10/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick47: s04-stg:/gluster/mnt11/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick48: s04-stg:/gluster/mnt12/brick</span></div></div><div class=""><br class=""></div><div class="">These 12 bricks are on same node and the sub volume made up of these bricks will be of same subvolume, which is not good. Same is true for the bricks hosted on <span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s05-stg and <span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s06-stg</span></span><br class=""></div><div class="">I think you have added these bricks after creating vol. The probability of disruption in connection of these bricks will be higher in this case.<br class=""></div><div class=""><br class=""></div><div class="">---<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><div style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" class="" target="_blank" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" class="" target="_blank" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Wednesday, September 26, 2018 3:38:35 PM<br class=""><b class="">Subject: </b>[Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div>Dear All, Dear Nithya,<div class=""><br class=""></div><div class="">after upgrading from 3.10.5 version to 3.12.14, I tried to start a rebalance process to distribute data across the bricks, but something goes wrong.</div><div class="">Rebalance failed on different nodes and the time value needed to complete the procedure seems to be very high.</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s01 ~]# gluster volume rebalance tier2 status</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> localhost 19 161.6GB 537 2 2 in progress 0:32:23</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s02-stg 25 212.7GB 526 5 2 in progress 0:32:25</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s03-stg 4 69.1GB 511 0 0 in progress 0:32:25</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s04-stg 4 484Bytes 12283 0 3 in progress 0:32:25</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s05-stg 23 484Bytes 11049 0 10 in progress 0:32:25</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s06-stg 3 1.2GB 8032 11 3 failed 0:17:57</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Estimated time left for rebalance to complete : 3601:05:41</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures;" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">volume rebalance: tier2: success</span></div></div><div class=""><br class=""></div><div class="">When rebalance processes fail, I can see the following kind of errors in /var/log/glusterfs/tier2-rebalance.log</div><div class=""><br class=""></div><div class="">Error type 1)</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:50:19.872575] W [MSGID: 122053] [ec-common.c:269:ec_check_status] 0-tier2-disperse-10: Operation failed on 2 of 6 subvolumes.(up=111111, mask=100111, remaining=</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">000000, good=100111, bad=011000)</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:50:19.901792] W [MSGID: 122053] [ec-common.c:269:ec_check_status] 0-tier2-disperse-11: Operation failed on 1 of 6 subvolumes.(up=111111, mask=111101, remaining=</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">000000, good=111101, bad=000010)</span></div></div><div class=""><br class=""></div><div class="">Error type 2)</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:53:31.566836] W [socket.c:600:__socket_rwv] 0-tier2-client-53: readv on 192.168.0.55:49153 failed (Connection reset by peer)</span></div></div><div class=""><br class=""></div><div class="">Error type 3)</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:57:37.852590] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10)</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:57:39.282306] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10)</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 09:02:04.928408] W [MSGID: 109023] [dht-rebalance.c:1013:__dht_check_free_space] 0-tier2-dht: data movement of file {blocks:0 name:(/OPA/archive/historical/dts/MRE</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">A/Observations/Observations/MREA14/Cs-1/CMCC/raw/CS013.ext)} would result in dst node (tier2-disperse-5:2440190848) having lower disk space than the source node (tier2-dispers</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">e-11:71373083776).Skipping file.</span></div></div><div class=""><br class=""></div><div class="">Error type 4)</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-tier2-client-7: socket disconnected</span></div></div><div class=""><br class=""></div><div class="">Error type 5)</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 09:07:42.333720] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f0417e0ee25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">90086004b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55900860032b] ) 0-: received signum (15), shutting down</span></div></div><div class=""><br class=""></div><div class="">Error type 6)</div><div class=""><br class=""></div><div class="">[2018-09-25 08:09:18.340658] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-4: server 192.168.0.52:49153 has not responded in the last 42 seconds, disconnecting.</div><div class=""><br class=""></div><div class="">It seems that there are some network or timeout problems, but the network usage/traffic values are not so high.</div><div class="">Do you think that, in my volume configuration, I have to modify some volume options related to thread and/or network parameters?</div><div class="">Could you, please, help me to understand the cause of the problems above?</div><div class=""><br class=""></div><div class="">You can find below our volume info:</div><div class="">(volume is implemented on 6 servers; each server configuration: 2 cpu 10-cores, 64GB RAM, 1 SSD dedicated to the OS, 12 x 10TB HD)</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s04 ~]# gluster vol info</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0); min-height: 18px;" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000; min-height: 18px;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span><br class="webkit-block-placeholder"></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Volume Name: tier2</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Type: Distributed-Disperse</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Status: Started</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Snapshot Count: 0</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Number of Bricks: 12 x (4 + 2) = 72</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Transport-type: tcp</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Bricks:</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick1: s01-stg:/gluster/mnt1/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick2: s02-stg:/gluster/mnt1/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick3: s03-stg:/gluster/mnt1/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick4: s01-stg:/gluster/mnt2/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick5: s02-stg:/gluster/mnt2/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick6: s03-stg:/gluster/mnt2/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick7: s01-stg:/gluster/mnt3/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick8: s02-stg:/gluster/mnt3/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick9: s03-stg:/gluster/mnt3/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick10: s01-stg:/gluster/mnt4/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick11: s02-stg:/gluster/mnt4/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick12: s03-stg:/gluster/mnt4/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick13: s01-stg:/gluster/mnt5/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick14: s02-stg:/gluster/mnt5/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick15: s03-stg:/gluster/mnt5/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick16: s01-stg:/gluster/mnt6/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick17: s02-stg:/gluster/mnt6/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick18: s03-stg:/gluster/mnt6/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick19: s01-stg:/gluster/mnt7/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick20: s02-stg:/gluster/mnt7/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick21: s03-stg:/gluster/mnt7/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick22: s01-stg:/gluster/mnt8/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick23: s02-stg:/gluster/mnt8/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick24: s03-stg:/gluster/mnt8/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick25: s01-stg:/gluster/mnt9/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick26: s02-stg:/gluster/mnt9/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick27: s03-stg:/gluster/mnt9/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick28: s01-stg:/gluster/mnt10/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick29: s02-stg:/gluster/mnt10/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick30: s03-stg:/gluster/mnt10/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick31: s01-stg:/gluster/mnt11/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick32: s02-stg:/gluster/mnt11/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick33: s03-stg:/gluster/mnt11/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick34: s01-stg:/gluster/mnt12/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick35: s02-stg:/gluster/mnt12/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick36: s03-stg:/gluster/mnt12/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick37: s04-stg:/gluster/mnt1/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick38: s04-stg:/gluster/mnt2/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick39: s04-stg:/gluster/mnt3/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick40: s04-stg:/gluster/mnt4/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick41: s04-stg:/gluster/mnt5/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick42: s04-stg:/gluster/mnt6/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick43: s04-stg:/gluster/mnt7/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick44: s04-stg:/gluster/mnt8/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick45: s04-stg:/gluster/mnt9/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick46: s04-stg:/gluster/mnt10/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick47: s04-stg:/gluster/mnt11/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick48: s04-stg:/gluster/mnt12/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick49: s05-stg:/gluster/mnt1/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick50: s05-stg:/gluster/mnt2/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick51: s05-stg:/gluster/mnt3/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick52: s05-stg:/gluster/mnt4/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick53: s05-stg:/gluster/mnt5/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick54: s05-stg:/gluster/mnt6/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick55: s05-stg:/gluster/mnt7/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick56: s05-stg:/gluster/mnt8/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick57: s05-stg:/gluster/mnt9/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick58: s05-stg:/gluster/mnt10/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick59: s05-stg:/gluster/mnt11/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick60: s05-stg:/gluster/mnt12/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick61: s06-stg:/gluster/mnt1/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick62: s06-stg:/gluster/mnt2/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick63: s06-stg:/gluster/mnt3/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick64: s06-stg:/gluster/mnt4/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick65: s06-stg:/gluster/mnt5/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick66: s06-stg:/gluster/mnt6/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick67: s06-stg:/gluster/mnt7/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick68: s06-stg:/gluster/mnt8/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick69: s06-stg:/gluster/mnt9/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick70: s06-stg:/gluster/mnt10/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick71: s06-stg:/gluster/mnt11/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick72: s06-stg:/gluster/mnt12/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Options Reconfigured:</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">network.ping-timeout: 60</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">diagnostics.count-fop-hits: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">diagnostics.latency-measurement: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">cluster.server-quorum-type: server</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">features.default-soft-limit: 90</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">features.quota-deem-statfs: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><a href="http://performance.io/" class="" target="_blank" data-mce-href="http://performance.io/">performance.io</a>-thread-count: 16</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">disperse.cpu-extensions: auto</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><a href="http://performance.io/" class="" target="_blank" data-mce-href="http://performance.io/">performance.io</a>-cache: off</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">network.inode-lru-limit: 50000</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">performance.md-cache-timeout: 600</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">performance.cache-invalidation: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">performance.stat-prefetch: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">features.cache-invalidation-timeout: 600</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">features.cache-invalidation: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">cluster.readdir-optimize: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">performance.parallel-readdir: off</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">performance.readdir-ahead: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">cluster.lookup-optimize: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">client.event-threads: 4</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">server.event-threads: 4</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">nfs.disable: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">transport.address-family: inet</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">cluster.quorum-type: auto</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">cluster.min-free-disk: 10</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">performance.client-io-threads: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">features.quota: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">features.inode-quota: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">features.bitrot: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">features.scrub: Active</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">cluster.brick-multiplex: on</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">cluster.server-quorum-ratio: 51%</span></div></div><div class=""><br class=""></div><div class="">If it can help, I paste here the output of “free -m” command executed on all the cluster nodes:</div><div class=""><br class=""></div><div class="">The result is almost the same on every nodes. In your opinion, the available RAM is enough to support data movement?</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s06 ~]# free -m</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> total used free shared buff/cache available</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Mem: 64309 10409 464 15 53434 52998</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Swap: 65535 103 65432</span></div></div><div class=""><br class=""></div><div class="">Thank you in advance.</div><div class="">Sorry for my long message, but I’m trying to notify you all available information.</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="" target="_blank" data-mce-href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br class=""><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" class="" target="_blank" data-mce-href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br data-mce-bogus="1"></div><div class=""><br class=""></div></div></div></div></blockquote></div><br class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;" class="" data-mce-style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><br class=""></div></div></div><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="" target="_blank" data-mce-href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br class="">https://lists.gluster.org/mailman/listinfo/gluster-users</div><div class=""><br class=""></div></div></div></div></blockquote></div><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="" data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;" class="" data-mce-style="color: #000000; font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><br class="Apple-interchange-newline"></div></div></div><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>https://lists.gluster.org/mailman/listinfo/gluster-users</div><div><br></div></div></body></html>