<div dir="ltr">Hi Mauro,<div><br></div><div>Yes, a rebalance consists of 2 operations for every directory:</div><div><ol><li>Fix the layout for the new volume config (newly added or removed bricks)<br></li><li>Migrate files to their new hashed subvols based on the new layout<br></li></ol><div><br></div></div><div>Are you running a rebalance because you added new bricks to the volume ? As per an earlier email you have already run a fix-layout.</div><div><br></div><div>On s04, please check the rebalance log file to see why the rebalance failed.</div><div><br></div><div>Regards,</div><div>Nithya</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 8 October 2018 at 13:22, Mauro Tridici <span dir="ltr">&lt;<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Hi All,</div><div><br></div><div>for your information, this is the current rebalance status:</div><div><br></div><div><span class=""><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s01 ~]# gluster volume rebalance tier2 status</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------</span></div></span><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                               localhost           551922        20.3TB       2349397             0         61849          in progress       55:25:38</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s02-stg           287631        13.2TB        959954             0         30262          in progress       55:25:39</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s03-stg           288523        12.7TB        973111             0         30220          in progress       55:25:39</span></div><span class=""><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s04-stg                0        0Bytes             0             0             0               failed        0:00:37</span></div></span><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s05-stg                0        0Bytes             0             0             0            completed       48:33:03</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s06-stg                0        0Bytes             0             0             0            completed       48:33:02</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Estimated time left for rebalance to complete :     1023:49:56</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">volume rebalance: tier2: success</span></div></div><div><br></div><div>Rebalance is migrating files on s05, s06 servers and on s04 too (although it is marked as failed).</div><div>s05 and s06 tasks are completed.</div><div><br></div><div>Questions:</div><div><br></div><div>1) it seems that rebalance is moving files, but it is fixing the layout also, is it normal?</div><div>2) when rebalance will be completed, what we need to do before return the gluster storage to the users? We have to launch rebalance again in order to involve s04 server too or a fix-layout to eventually fix some error on s04?</div><span class=""><div><br></div><div>Thank you very much,</div><div>Mauro</div><div><br></div><br></span><div><blockquote type="cite"><div><div class="h5"><div>Il giorno 07 ott 2018, alle ore 10:29, Mauro Tridici &lt;<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>&gt; ha scritto:</div><br class="m_-5191715533486204971Apple-interchange-newline"></div></div><div><div><div class="h5"><div style="word-wrap:break-word"><div><br></div><div>Hi All,</div><div><br></div><div>some important updates about the issue mentioned below.</div><div>After rebalance failed on all the servers, I decided to:</div><div><br></div><div>- stop gluster volume</div><div>- reboot the servers</div><div>- start gluster volume</div><div>- change some gluster volume options</div><div>- start the rebalance again</div><div><br></div><div>The options that I changed are listed below after reading some threads on gluster users mailing list:</div><div><br></div><div>BEFORE CHANGE:</div><div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set tier2 network.ping-timeout 02</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set all cluster.brick-multiplex on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set tier2 cluster.server-quorum-ratio 51%</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set tier2 cluster.server-quorum-type server</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)">gluster volume set tier2 cluster.quorum-type auto</div></div><div><br></div><div>AFTER CHANGE:</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set tier2 network.ping-timeout 42</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set all cluster.brick-multiplex off</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set tier2 cluster.server-quorum-ratio none</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set tier2 cluster.server-quorum-type none</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">gluster volume set tier2 cluster.quorum-type none</span></div></div><div><br></div><div>The result was that rebalance starts moving data from s01, s02 ed s03 servers to s05 and s06 servers (the new added ones), but it failed on s04 server after 37 seconds.</div><div>The rebalance is still running and moving data as you can see from the output:</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s01 ~]# gluster volume rebalance tier2 status</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                               localhost           286680        12.6TB       1217960             0         43343          in progress       32:10:24</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s02-stg           126291        12.4TB        413077             0         21932          in progress       32:10:25</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s03-stg           126516        11.9TB        433014             0         21870          in progress       32:10:25</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s04-stg                0        0Bytes             0             0             0               failed        0:00:37</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s05-stg                0        0Bytes             0             0             0          in progress       32:10:25</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                                 s06-stg                0        0Bytes             0             0             0          in progress       32:10:25</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Estimated time left for rebalance to complete :      624:47:48</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">volume rebalance: tier2: success</span></div></div><div><br></div><div>When rebalance will be completed, we are planning to re-launch it to try to involve s04 server also.</div><div>Do you have some idea about what happened in my previous message and why, now, rebalance it’s running although it’s not involve s04 server?</div><div><br></div><div>In attachment the complete tier2-rebalance.log  file related to s04 server.</div><div><br></div><div><div>Thank you very much for your help,</div><div>Mauro</div></div><div><br></div><div><br></div><div></div></div></div></div><span id="m_-5191715533486204971cid:1BCD329C-5AAD-4D01-9A37-E87A1CE408BF@openvpn">&lt;tier2-rebalance.log.gz&gt;</span><div style="word-wrap:break-word"><div></div><br><div><blockquote type="cite"><span class=""><div>Il giorno 06 ott 2018, alle ore 02:01, Mauro Tridici &lt;<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>&gt; ha scritto:</div><br class="m_-5191715533486204971Apple-interchange-newline"></span><div><div style="word-wrap:break-word"><span class=""><div><br></div><div>Hi All,</div><div><br></div><div>since we need to restore gluster storage as soon as possible, we decided to ignore the few files that could be lost and to go ahead.</div><div>So we cleaned all bricks content of servers s04, s05 and s06.</div><div><br></div><div>As planned some days ago, we executed the following commands:</div><div><br></div><div><div><i>gluster peer detach s04</i></div><div><i>gluster peer detach s05</i></div><div><i>gluster peer detach s06</i></div><div><i><br></i></div><div><i>gluster peer probe s04</i></div><div><i>gluster peer probe s05</i></div><div><i>gluster peer probe s06</i></div><div><i><br></i></div><div><i>gluster volume add-brick tier2 s04-stg:/gluster/mnt1/brick s05-stg:/gluster/mnt1/brick s06-stg:/gluster/mnt1/brick s04-stg:/gluster/mnt2/brick s05-stg:/gluster/mnt2/brick s06-stg:/gluster/mnt2/brick s04-stg:/gluster/mnt3/brick s05-stg:/gluster/mnt3/brick s06-stg:/gluster/mnt3/brick s04-stg:/gluster/mnt4/brick s05-stg:/gluster/mnt4/brick s06-stg:/gluster/mnt4/brick s04-stg:/gluster/mnt5/brick s05-stg:/gluster/mnt5/brick s06-stg:/gluster/mnt5/brick s04-stg:/gluster/mnt6/brick s05-stg:/gluster/mnt6/brick s06-stg:/gluster/mnt6/brick s04-stg:/gluster/mnt7/brick s05-stg:/gluster/mnt7/brick s06-stg:/gluster/mnt7/brick s04-stg:/gluster/mnt8/brick s05-stg:/gluster/mnt8/brick s06-stg:/gluster/mnt8/brick s04-stg:/gluster/mnt9/brick s05-stg:/gluster/mnt9/brick s06-stg:/gluster/mnt9/brick s04-stg:/gluster/mnt10/brick s05-stg:/gluster/mnt10/brick s06-stg:/gluster/mnt10/brick s04-stg:/gluster/mnt11/brick s05-stg:/gluster/mnt11/brick s06-stg:/gluster/mnt11/brick s04-stg:/gluster/mnt12/brick s05-stg:/gluster/mnt12/brick s06-stg:/gluster/mnt12/brick force</i></div><div><i><br></i></div><div><i>gluster volume rebalance tier2 fix-layout start</i></div></div><div><i><br></i></div><div><span>Everything seem to be fine and fix-layout ended.</span></div><div><span><br></span></div></span><div><span><span class=""><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s01 ~]# gluster volume rebalance tier2 status </span></div></span><div><div class="h5"><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                               Node                                    status           run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                          ---------                               -----------                ------------</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                          localhost                               fix-layout completed       12:11:6</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                            s02-stg                               fix-layout completed       12:11:18</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                            s03-stg                               fix-layout completed       12:11:12</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                            s04-stg                               fix-layout completed       12:11:20</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                            s05-stg                               fix-layout completed       12:11:14</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">                            s06-stg                               fix-layout completed       12:10:47</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">volume rebalance: tier2: success</span></div></div></div></span></div><div><div class="h5"><div><br></div><div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s01 ~]# gluster volume info</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0);min-height:18px"><span style="font-variant-ligatures:no-common-ligatures"> </span><br class="m_-5191715533486204971webkit-block-placeholder"></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Volume Name: tier2</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Type: Distributed-Disperse</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Volume ID: a28d88c5-3295-4e35-98d4-<wbr>210b3af9358c</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Status: Started</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Snapshot Count: 0</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Number of Bricks: 12 x (4 + 2) = 72</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Transport-type: tcp</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Bricks:</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick1: s01-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick2: s02-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick3: s03-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick4: s01-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick5: s02-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick6: s03-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick7: s01-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick8: s02-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick9: s03-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick10: s01-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick11: s02-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick12: s03-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick13: s01-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick14: s02-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick15: s03-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick16: s01-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick17: s02-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick18: s03-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick19: s01-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick20: s02-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick21: s03-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick22: s01-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick23: s02-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick24: s03-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick25: s01-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick26: s02-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick27: s03-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick28: s01-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick29: s02-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick30: s03-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick31: s01-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick32: s02-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick33: s03-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick34: s01-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick35: s02-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick36: s03-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick37: s04-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick38: s05-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick39: s06-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick40: s04-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick41: s05-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick42: s06-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick43: s04-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick44: s05-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick45: s06-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick46: s04-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick47: s05-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick48: s06-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick49: s04-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick50: s05-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick51: s06-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick52: s04-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick53: s05-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick54: s06-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick55: s04-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick56: s05-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick57: s06-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick58: s04-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick59: s05-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick60: s06-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick61: s04-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick62: s05-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick63: s06-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick64: s04-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick65: s05-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick66: s06-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick67: s04-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick68: s05-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick69: s06-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick70: s04-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick71: s05-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick72: s06-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Options Reconfigured:</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">network.ping-timeout: 42</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.scrub: Active</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.bitrot: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.inode-quota: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.quota: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.client-io-threads: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.min-free-disk: 10</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.quorum-type: none</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">transport.address-family: inet</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">nfs.disable: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">server.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">client.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.lookup-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.readdir-ahead: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.parallel-readdir: off</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.readdir-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.cache-invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.cache-invalidation-<wbr>timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.stat-prefetch: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.cache-<wbr>invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.md-cache-timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">network.inode-lru-limit: 50000</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><a href="http://performance.io/" target="_blank">performance.io</a>-cache: off</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">disperse.cpu-extensions: auto</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><a href="http://performance.io/" target="_blank">performance.io</a>-thread-count: 16</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.quota-deem-statfs: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.default-soft-limit: 90</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.server-quorum-type: none</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">diagnostics.latency-<wbr>measurement: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">diagnostics.count-fop-hits: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.brick-multiplex: off</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.server-quorum-ratio: 51%</span></div></div><div><br></div><div>The last step should be the data rebalance between the servers, but rebalance failed soon with a lot of errors like the following ones:</div><div><br></div><div><div>[2018-10-05 23:48:38.644978] I [MSGID: 114035] [client-handshake.c:202:<wbr>client_set_lk_version_cbk] 0-tier2-client-70: Server lk version = 1</div><div>[2018-10-05 23:48:44.735323] I [dht-rebalance.c:4512:gf_<wbr>defrag_start_crawl] 0-tier2-dht: gf_defrag_start_crawl using commit hash 3720331860</div><div>[2018-10-05 23:48:44.736205] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr>update_cbk] 0-tier2-disperse-7: Failed to get size and version [Input/output error]</div><div>[2018-10-05 23:48:44.736266] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr>select] 0-tier2-disperse-7: Insufficient available children for this request (have 0, need 4)</div><div>[2018-10-05 23:48:44.736282] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr>size_version_done] 0-tier2-disperse-7: Failed to update version and size [Input/output error]</div><div>[2018-10-05 23:48:44.736377] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr>update_cbk] 0-tier2-disperse-8: Failed to get size and version [Input/output error]</div><div>[2018-10-05 23:48:44.736436] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr>select] 0-tier2-disperse-8: Insufficient available children for this request (have 0, need 4)</div><div>[2018-10-05 23:48:44.736459] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr>size_version_done] 0-tier2-disperse-8: Failed to update version and size [Input/output error]</div><div>[2018-10-05 23:48:44.736460] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr>update_cbk] 0-tier2-disperse-10: Failed to get size and version [Input/output error]</div><div>[2018-10-05 23:48:44.736537] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr>update_cbk] 0-tier2-disperse-9: Failed to get size and version [Input/output error]</div><div>[2018-10-05 23:48:44.736571] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr>select] 0-tier2-disperse-10: Insufficient available children for this request (have 0, need 4)</div><div>[2018-10-05 23:48:44.736574] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr>select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</div><div>[2018-10-05 23:48:44.736604] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr>size_version_done] 0-tier2-disperse-9: Failed to update version and size [Input/output error]</div><div>[2018-10-05 23:48:44.736604] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr>size_version_done] 0-tier2-disperse-10: Failed to update version and size [Input/output error]</div><div>[2018-10-05 23:48:44.736827] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr>update_cbk] 0-tier2-disperse-11: Failed to get size and version [Input/output error]</div><div>[2018-10-05 23:48:44.736887] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr>select] 0-tier2-disperse-11: Insufficient available children for this request (have 0, need 4)</div><div>[2018-10-05 23:48:44.736904] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr>size_version_done] 0-tier2-disperse-11: Failed to update version and size [Input/output error]</div><div>[2018-10-05 23:48:44.740337] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr>update_cbk] 0-tier2-disperse-6: Failed to get size and version [Input/output error]</div><div>[2018-10-05 23:48:44.740381] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr>select] 0-tier2-disperse-6: Insufficient available children for this request (have 0, need 4)</div><div>[2018-10-05 23:48:44.740394] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr>size_version_done] 0-tier2-disperse-6: Failed to update version and size [Input/output error]</div><div>[2018-10-05 23:48:50.066103] I [MSGID: 109081] [dht-common.c:4379:dht_<wbr>setxattr] 0-tier2-dht: fixing the layout of /</div></div><div><br></div><div>In attachment you can find the first logs captured during the rebalance execution.</div><div>In your opinion, is there a way to restore the gluster storage or all the data have been lost?</div><div><br></div><div>Thank you in advance,</div><div>Mauro</div><div><br></div><div></div></div></div></div><span id="m_-5191715533486204971cid:1C722A07-5FB4-4759-BB6C-AE4D17E25230@openvpn">&lt;rebalance_log.txt&gt;</span><div style="word-wrap:break-word"><div></div><div><br></div><div><br></div><br><div><blockquote type="cite"><div>Il giorno 04 ott 2018, alle ore 15:31, Mauro Tridici &lt;<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>&gt; ha scritto:</div><br class="m_-5191715533486204971Apple-interchange-newline"><div><div style="word-wrap:break-word"><div><br></div><div>Hi Nithya,</div><div><br></div><div>thank you very much.</div><div>This is the current “gluster volume info” output after removing bricks (and after peer detach command).</div><div><br></div><div><span class=""><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s01 ~]# gluster volume info</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0);min-height:18px"><span style="font-variant-ligatures:no-common-ligatures"> </span><br class="m_-5191715533486204971webkit-block-placeholder"></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Volume Name: tier2</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Type: Distributed-Disperse</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Volume ID: a28d88c5-3295-4e35-98d4-<wbr>210b3af9358c</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Status: Started</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Snapshot Count: 0</span></div></span><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Number of Bricks: 6 x (4 + 2) = 36</span></div><div><div class="h5"><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Transport-type: tcp</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Bricks:</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick1: s01-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick2: s02-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick3: s03-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick4: s01-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick5: s02-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick6: s03-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick7: s01-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick8: s02-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick9: s03-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick10: s01-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick11: s02-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick12: s03-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick13: s01-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick14: s02-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick15: s03-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick16: s01-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick17: s02-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick18: s03-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick19: s01-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick20: s02-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick21: s03-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick22: s01-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick23: s02-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick24: s03-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick25: s01-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick26: s02-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick27: s03-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick28: s01-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick29: s02-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick30: s03-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick31: s01-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick32: s02-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick33: s03-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick34: s01-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick35: s02-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick36: s03-stg:/gluster/mnt12/brick</span></div></div></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Options Reconfigured:</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">network.ping-timeout: 0</span></div><span class=""><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.scrub: Active</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.bitrot: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.inode-quota: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.quota: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.client-io-threads: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.min-free-disk: 10</span></div></span><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.quorum-type: auto</span></div><span class=""><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">transport.address-family: inet</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">nfs.disable: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">server.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">client.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.lookup-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.readdir-ahead: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.parallel-readdir: off</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.readdir-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.cache-invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.cache-invalidation-<wbr>timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.stat-prefetch: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.cache-<wbr>invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.md-cache-timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">network.inode-lru-limit: 50000</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><a href="http://performance.io/" target="_blank">performance.io</a>-cache: off</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">disperse.cpu-extensions: auto</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><a href="http://performance.io/" target="_blank">performance.io</a>-thread-count: 16</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.quota-deem-statfs: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.default-soft-limit: 90</span></div></span><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.server-quorum-type: server</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">diagnostics.latency-<wbr>measurement: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">diagnostics.count-fop-hits: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.brick-multiplex: on</span></div><div style="margin:0px;line-height:normal;font-family:&#39;Courier New&#39;;color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.server-quorum-ratio: 51%</span></div></div><div><br></div><div>Regards,</div><div>Mauro</div><br><div><blockquote type="cite"><div>Il giorno 04 ott 2018, alle ore 15:22, Nithya Balachandran &lt;<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>&gt; ha scritto:</div><br class="m_-5191715533486204971Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr">Hi Mauro,<div><br></div><div><br></div><div>The files on s04 and s05 can be deleted safely as long as those bricks have been removed from the volume and their brick processes are not running.</div><div><br></div><div><br><pre class="m_-5191715533486204971gmail-aLF-aPX-K0-aPE" style="display:block;margin:0px;word-wrap:break-word;background-color:rgb(255,255,255);text-align:left;text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif"><span style="font-size:14px;white-space:pre-wrap">.glusterfs/indices/xattrop/<wbr>xattrop-* are links to files that need to be healed.
.glusterfs/quarantine/stub-<wbr>00000000-0000-0000-0000-<wbr>000000000008 links to files that bitrot (if enabled)says are corrupted. (none in this case)</span></font><font><span style="font-size:14px;white-space:pre-wrap"><br></span></font></pre><pre class="m_-5191715533486204971gmail-aLF-aPX-K0-aPE" style="display:block;margin:0px;word-wrap:break-word;background-color:rgb(255,255,255);text-align:left;text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></pre><pre class="m_-5191715533486204971gmail-aLF-aPX-K0-aPE" style="display:block;margin:0px;word-wrap:break-word;background-color:rgb(255,255,255);text-align:left;text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></pre><pre class="m_-5191715533486204971gmail-aLF-aPX-K0-aPE" style="display:block;margin:0px;word-wrap:break-word;background-color:rgb(255,255,255);text-align:left;text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif"><span style="font-size:14px;white-space:pre-wrap">I will get back to you on s06. Can you please provide the output of gluster volume info again?</span></font></pre></div><div><br></div><div>Regards,</div><div>Nithya<br><div><br></div><div><br></div></div></div></div><div><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 4 October 2018 at 13:47, Mauro Tridici <span dir="ltr">&lt;<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div>
<span class="m_-5191715533486204971m_-7415967095465071885Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word"><span class="m_-5191715533486204971m_-7415967095465071885Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"></div>Dear Ashish, Dear Nithya,</span></div><div style="word-wrap:break-word"><span class="m_-5191715533486204971m_-7415967095465071885Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><br></span></div><div style="word-wrap:break-word"><span class="m_-5191715533486204971m_-7415967095465071885Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px">I’m writing this message only to summarize and simplify the information about the &quot;not migrated” files left on removed bricks on server s04, s05 and s06.</span></div><div style="word-wrap:break-word"><span style="text-align:-webkit-auto">In attachment, you can find 3 files (1 file for each server) containing the “not migrated” files lists and related brick number.</span></div><div style="word-wrap:break-word"><span style="text-align:-webkit-auto"><br></span></div><div style="word-wrap:break-word"><span style="text-align:-webkit-auto">In particular:</span></div><div style="word-wrap:break-word"><ul><li><span style="text-align:-webkit-auto">s04 and s05 bricks contain only not migrated files in hidden directories “/gluster/mnt#/brick/.glusterf<wbr>s&quot; (I could delete them, doesn’t it?)</span></li><li><span style="text-align:-webkit-auto">s06 bricks contain </span></li><ul><li><span style="text-align:-webkit-auto">not migrated files in hidden directories </span>“/gluster/mnt#/bri<wbr>ck/.glusterfs”;</li><li>not migrated files with size equal to 0;</li><li>not migrated files with size greater than 0.</li></ul></ul><div><br></div><div>I think it was necessary to collect and summarize information to simplify your analysis.</div><div>Thank you very much,</div><div>Mauro</div><div><br></div><div><br></div><div></div></div></span></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div><br></div></div></blockquote></div><br><div>
<span class="m_-5191715533486204971Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;border-spacing:0px"><div style="word-wrap:break-word"><span class="m_-5191715533486204971Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div></span></div></span></div></div></blockquote></div><br></div>