<div dir="ltr"><div dir="ltr">Hi Mauro,<div><br></div><div><br></div><div>Please send the rebalance logs from s04-stg. I will take a look and get back.</div><div><br></div><div><br></div><div>Regards,</div><div>Nithya</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 28 September 2018 at 16:21, Mauro Tridici <span dir="ltr"><<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div><div>Hi Ashish,</div><div><br></div><div>as I said in my previous message, we adopted the first approach you suggested (setting network.ping-timeout option to 0).</div><div>This choice was due to the absence of empty brick to be used as indicated in the second approach.</div><div><br></div><div>So, we launched remove-brick command on the first subvolume (V1, bricks 1,2,3,4,5,6 on server s04).</div><div>Rebalance started moving the data across the other bricks, but, after about 3TB of moved data, rebalance speed slowed down and some transfer errors appeared in the rebalance.log of server s04.</div><div>At this point, since remaining 1,8TB need to be moved in order to complete the step, we decided to stop the remove-brick execution and start it again (I hope it doesn’t stop again before complete the rebalance)</div><div><br></div><div>Now rebalance is not moving data, it’s only scanning files (please, take a look to the following output)</div></div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s01 ~]# gluster volume remove-brick tier2 s04-stg:/gluster/mnt1/brick s04-stg:/gluster/mnt2/brick s04-stg:/gluster/mnt3/brick s04-stg:/gluster/mnt4/brick s04-stg:/gluster/mnt5/brick s04-stg:/gluster/mnt6/brick status</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> s04-stg 0 0Bytes 182008 0 0 in progress 3:08:09</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Estimated time left for rebalance to complete : 442:45:06</span></div></div><div><br></div><div><div>If I’m not wrong, remove-brick rebalances entire cluster each time it start.</div><div>Is there a way to speed up this procedure? Do you have some other suggestion that, in this particular case, could be useful to reduce errors (I know that they are related to the current volume configuration) and improve rebalance performance avoiding to rebalance the entire cluster?</div></div><div><br></div><div>Thank you in advance,</div><div>Mauro</div><br><div><blockquote type="cite"><div>Il giorno 27 set 2018, alle ore 13:14, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>> ha scritto:</div><br class="m_-2508199895865596987Apple-interchange-newline"><div><div><div style="font-family:'times new roman','new york',times,serif;font-size:12pt"><div><br></div><div>Yes, you can. </div><div>If not me others may also reply.<br></div><div><br></div><div>---<br></div><div>Ashish<br></div><div><br></div><hr id="m_-2508199895865596987zwchr"><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>><br><b>To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Sent: </b>Thursday, September 27, 2018 4:24:12 PM<br><b>Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br><div><br></div><div><br></div><div>Dear Ashish,</div><div><br></div><div>I can not thank you enough!</div><div>Your procedure and description is very detailed.</div><div>I think to follow the first approach after setting network.ping-timeout option to 0 (If I’m not wrong “0" means “infinite”...I noticed that this value reduced rebalance errors).</div><div>After the fix I will set network.ping-timeout option to default value.</div><div><br></div><div>Could I contact you again if I need some kind of suggestion?</div><div><br></div><div>Thank you very much again.</div><div>Have a good day,</div><div>Mauro</div><div><br></div><br><div><blockquote><div>Il giorno 27 set 2018, alle ore 12:38, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>> ha scritto:</div><br class="m_-2508199895865596987Apple-interchange-newline"><div><div><div style="font-family:'times new roman','new york',times,serif;font-size:12pt"><div><br></div><div><div>Hi Mauro,</div><div><br></div><div>We can divide the 36 newly added bricks into 6 set of 6 bricks each starting from brick37.</div><div>That means, there are 6 ec subvolumes and we have to deal with one sub volume at a time.</div><div>I have named it V1 to V6.</div><div><br></div><div>Problem:</div><div>Take the case of V1.</div><div>The best configuration/setup would be to have all the 6 bricks of V1 on 6 different nodes.</div><div>However, in your case you have added 3 new nodes. So, at least we should have 2 bricks on 3 different newly added nodes.</div><div>This way, in 4+2 EC configuration, even if one node goes down you will have 4 other bricks of that volume and the data on that volume would be accessible.</div><div>In current setup if <span style="font-variant-ligatures:no-common-ligatures">s04-stg</span> goes down, you will loose all the data on V1 and V2 as all the bricks will be down. We want to avoid and correct it.</div><div><br></div><div>Now, we can have two approach to correct/modify this setup.</div><div><br></div><div><strong>Approach 1</strong></div><div>We have to remove all the newly added bricks in a set of 6 bricks. This will trigger re- balance and move whole data to other sub volumes.</div><div>Repeat the above step and then once all the bricks are removed, add those bricks again in a set of 6 bricks, this time have 2 bricks from each of the 3 newly added Nodes.</div><div><br></div><div>While this is a valid and working approach, I personally think that this will take long time and also require lot of movement of data.</div><div><br></div><div><div><strong>Approach 2</strong></div><div><br></div><div>In this approach we can use the heal process. We have to deal with all the volumes (V1 to V6) one by one. Following are the steps for V1-</div><div><br></div><div><strong>Step 1 - </strong></div><div>Use replace-brick command to move following bricks on <strong>s05-stg</strong> node <strong>one by one (heal should be completed after every replace brick command)</strong></div><div><br><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><strong><span style="font-variant-ligatures:no-common-ligatures">Brick39: s04-stg:/gluster/mnt3/brick to s05-stg/<brick which is free><br></span></strong></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><strong><span style="font-variant-ligatures:no-common-ligatures">Brick40: s04-stg:/gluster/mnt4/brick <span style="font-variant-ligatures:no-common-ligatures">to s05-stg/<other brick which is free></span><br></span></strong></div></div><div><br></div><div>Command :</div><div>gluster v replace-brick <volname> <strong><span style="font-variant-ligatures:no-common-ligatures">s04-stg:/gluster/mnt3/brick</span></strong> <wbr> <strong><span style="font-variant-ligatures:no-common-ligatures">s05-stg:/<brick which is free></span></strong> commit force</div><div>Try to give names to the bricks so that you can identify which 6 bricks belongs to same ec subvolume</div><div><br></div><div><br></div><div><div>Use replace-brick command to move following bricks on <strong>s06-stg</strong> node one by one</div><br><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick41: s04-stg:/gluster/mnt5/brick to <strong><span style="font-variant-ligatures:no-common-ligatures">s06-stg/<brick which is free></span></strong><br></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick42: s04-stg:/gluster/mnt6/brick <span style="font-variant-ligatures:no-common-ligatures">to <strong><span style="font-variant-ligatures:no-common-ligatures">s06-stg/<other brick which is free></span></strong></span><br></span></div></div><div><br></div><div><div><br></div><div><strong>Step 2</strong> - After, every replace-brick command, you have to wait for heal to be completed.</div><div>check <strong>"gluster v heal <volname> info "</strong> if it shows any entry you have to wait for it to be completed.</div><div><br></div></div><div>After successful step 1 and step 2, setup for sub volume V1 will be fixed. The same steps you have to perform for other volumes. Only thing is that</div><div>the nodes would be different on which you have to move the bricks.</div><div><br></div><div><br></div></div><div><br></div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">V1<br></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick37: s04-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick38: s04-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick39: s04-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick40: s04-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick41: s04-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick42: s04-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"><span style="font-variant-ligatures:no-common-ligatures">V2</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick43: s04-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick44: s04-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick45: s04-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick46: s04-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick47: s04-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick48: s04-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"><span style="font-variant-ligatures:no-common-ligatures">V3</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick49: s05-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick50: s05-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick51: s05-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick52: s05-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick53: s05-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick54: s05-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"><span style="font-variant-ligatures:no-common-ligatures">V4</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick55: s05-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick56: s05-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick57: s05-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick58: s05-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick59: s05-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick60: s05-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"><span style="font-variant-ligatures:no-common-ligatures">V5</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick61: s06-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick62: s06-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick63: s06-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick64: s06-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick65: s06-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick66: s06-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"><span style="font-variant-ligatures:no-common-ligatures">V6</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick67: s06-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick68: s06-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick69: s06-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick70: s06-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick71: s06-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick72: s06-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures"> </span></div></div><div><br></div><div>Just a note that these steps need movement of data.</div><div>Be careful while performing these steps and do one replace brick at a time and only after heal completion go to next.</div><div>Let me know if you have any issues.</div><div><br></div><div>---</div><div>Ashish</div><div><br></div><div><br></div><div><br></div></div><hr id="m_-2508199895865596987zwchr"><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>><br><b>To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Sent: </b>Thursday, September 27, 2018 4:03:04 PM<br><b>Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br><div><br></div><div><br></div><div>Dear Ashish,</div><div><br></div><div>I hope I don’t disturb you so much, but I would like to ask you if you had some time to dedicate to our problem.</div><div>Please, forgive my insistence.</div><div><br></div><div>Thank you in advance,</div><div>Mauro</div><br><div><blockquote><div>Il giorno 26 set 2018, alle ore 19:56, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>> ha scritto:</div><br class="m_-2508199895865596987Apple-interchange-newline"><div><div style="word-wrap:break-word"><div>Hi Ashish,</div><div><br></div><div>sure, no problem! We are a little bit worried, but we can wait :-)</div><div>Thank you very much for your support and your availability.</div><div><br></div><div>Regards,</div><div>Mauro</div><div><br></div><br><div><blockquote><div>Il giorno 26 set 2018, alle ore 19:33, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>> ha scritto:</div><br class="m_-2508199895865596987Apple-interchange-newline"><div><div><div style="font-family:'times new roman','new york',times,serif;font-size:12pt"><div>Hi Mauro,<br></div><div><br></div><div>Yes, I can provide you step by step procedure to correct it. <br></div><div>Is it fine If i provide you the steps tomorrow as it is quite late over here and I don't want to miss anything in hurry?<br></div><div><br></div><div>---<br></div><div>Ashish<br></div><div><br></div><hr id="m_-2508199895865596987zwchr"><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>><br><b>To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Sent: </b>Wednesday, September 26, 2018 6:54:19 PM<br><b>Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br><div><br></div><div><br></div><div>Hi Ashish,</div><div><br></div><div>in attachment you can find the rebalance log file and the last updated brick log file (the other files in /var/log/glusterfs/bricks directory seem to be too old).</div><div>I just stopped the running rebalance (as you can see at the bottom of the rebalance log file).</div><div>So, if exists a safe procedure to correct the problem I would like execute it.</div><div><br></div><div>I don’t know if I can ask you it, but, if it is possible, could you please describe me step by step the right procedure to remove the newly added bricks without losing the data that have been already rebalanced?</div><div><br></div><div>The following outputs show the result of “df -h” command executed on one of the first 3 nodes (s01, s02, s03) already existing and on one of the last 3 nodes (s04, s05, s06) added recently.</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s06 bricks]# df -h</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">File system Dim. Usati Dispon. Uso% Montato su</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/cl_s06-root 100G 2,1G 98G 3% /</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">devtmpfs 32G 0 32G 0% /dev</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">tmpfs 32G 4,0K 32G 1% /dev/shm</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">tmpfs 32G 26M 32G 1% /run</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">tmpfs 32G 0 32G 0% /sys/fs/cgroup</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/cl_s06-var 100G 2,0G 99G 2% /var</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/cl_s06-gluster 100G 33M 100G 1% /gluster</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/sda1 1014M 152M 863M 15% /boot</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgd-<wbr>gluster_lvd 9,0T 807G 8,3T 9% /gluster/mnt3</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgg-<wbr>gluster_lvg 9,0T 807G 8,3T 9% /gluster/mnt6</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgc-<wbr>gluster_lvc 9,0T 807G 8,3T 9% /gluster/mnt2</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vge-<wbr>gluster_lve 9,0T 807G 8,3T 9% /gluster/mnt4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgj-<wbr>gluster_lvj 9,0T 887G 8,2T 10% /gluster/mnt9</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgb-<wbr>gluster_lvb 9,0T 807G 8,3T 9% /gluster/mnt1</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgh-<wbr>gluster_lvh 9,0T 887G 8,2T 10% /gluster/mnt7</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgf-<wbr>gluster_lvf 9,0T 807G 8,3T 9% /gluster/mnt5</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgi-<wbr>gluster_lvi 9,0T 887G 8,2T 10% /gluster/mnt8</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgl-<wbr>gluster_lvl 9,0T 887G 8,2T 10% /gluster/mnt11</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgk-<wbr>gluster_lvk 9,0T 887G 8,2T 10% /gluster/mnt10</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgm-<wbr>gluster_lvm 9,0T 887G 8,2T 10% /gluster/mnt12</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">tmpfs 6,3G 0 6,3G 0% /run/user/0</span></div></div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s01 ~]# df -h</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">File system Dim. Usati Dispon. Uso% Montato su</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/cl_s01-root 100G 5,3G 95G 6% /</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">devtmpfs 32G 0 32G 0% /dev</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">tmpfs 32G 39M 32G 1% /dev/shm</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">tmpfs 32G 26M 32G 1% /run</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">tmpfs 32G 0 32G 0% /sys/fs/cgroup</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/cl_s01-var 100G 11G 90G 11% /var</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/md127 1015M 151M 865M 15% /boot</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/cl_s01-gluster 100G 33M 100G 1% /gluster</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgi-<wbr>gluster_lvi 9,0T 5,5T 3,6T 61% /gluster/mnt7</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgm-<wbr>gluster_lvm 9,0T 5,4T 3,6T 61% /gluster/mnt11</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgf-<wbr>gluster_lvf 9,0T 5,7T 3,4T 63% /gluster/mnt4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgl-<wbr>gluster_lvl 9,0T 5,8T 3,3T 64% /gluster/mnt10</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgj-<wbr>gluster_lvj 9,0T 5,5T 3,6T 61% /gluster/mnt8</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgn-<wbr>gluster_lvn 9,0T 5,4T 3,6T 61% /gluster/mnt12</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgk-<wbr>gluster_lvk 9,0T 5,8T 3,3T 64% /gluster/mnt9</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgh-<wbr>gluster_lvh 9,0T 5,6T 3,5T 63% /gluster/mnt6</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgg-<wbr>gluster_lvg 9,0T 5,6T 3,5T 63% /gluster/mnt5</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vge-<wbr>gluster_lve 9,0T 5,7T 3,4T 63% /gluster/mnt3</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgc-<wbr>gluster_lvc 9,0T 5,6T 3,5T 62% /gluster/mnt1</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/dev/mapper/gluster_vgd-<wbr>gluster_lvd 9,0T 5,6T 3,5T 62% /gluster/mnt2</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">tmpfs 6,3G 0 6,3G 0% /run/user/0</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">s01-stg:tier2 420T 159T 262T 38% /tier2</span></div></div><div><br></div><div>As you can see, used space value of each brick of the last servers is about 800GB.</div><div><br></div><div>Thank you,</div><div>Mauro</div><div><br></div><div><br></div><div><br></div><br><div><br></div><br><div><br></div><br><div><blockquote><div>Il giorno 26 set 2018, alle ore 14:51, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>> ha scritto:</div><br class="m_-2508199895865596987Apple-interchange-newline"><div><div><div style="font-family:'times new roman','new york',times,serif;font-size:12pt"><div>Hi Mauro,<br></div><div><br></div><div>rebalance and brick logs should be the first thing we should go through.<br></div><div><br></div><div>There is a procedure to correct the configuration/setup but the situation you are in is difficult to follow that procedure.<br></div><div>You should have added the bricks hosted on <span style="font-variant-ligatures:no-common-ligatures">s04-stg</span>, <span style="font-variant-ligatures:no-common-ligatures">s05-stg and <span style="font-variant-ligatures:no-common-ligatures">s06-stg the same way you had the previous configuration.</span></span><br></div><div><span style="font-variant-ligatures:no-common-ligatures"><span style="font-variant-ligatures:no-common-ligatures">That means 2 bricks on each node for one subvolume.<br></span></span></div><div>The procedure will require a lot of replace bricks which will again need healing and all. In addition to that we have to wait for re-balance to complete.<br></div><div><br></div><div>I would suggest that if whole data has not been rebalanced and if you can stop the rebalance and remove these newly added bricks properly then you should remove these newly added bricks.<br></div><div>After that, add these bricks so that you have 2 bricks of each volume on 3 newly added nodes.<br></div><div><br></div><div>Yes, it is like undoing whole effort but it is better to do it now then facing issues in future when it will be almost impossible to correct these things if you have lots of data.</div><div><br></div><div>---<br></div><div>Ashish<br></div><div><br></div><div><br></div><div><br></div><hr id="m_-2508199895865596987zwchr"><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>><br><b>To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Sent: </b>Wednesday, September 26, 2018 5:55:02 PM<br><b>Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br><div><br></div><div><br></div><div>Dear Ashish,</div><div><br></div><div>thank you for you answer.</div><div>I could provide you the entire log file related to glusterd, glusterfsd and rebalance.</div><div>Please, could you indicate which one you need first?</div><div><br></div><div>Yes, we added the last 36 bricks after creating vol. Is there a procedure to correct this error? Is it still possible to do it?</div><div><br></div><div>Many thanks,</div><div>Mauro</div><br><div><blockquote><div>Il giorno 26 set 2018, alle ore 14:13, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>> ha scritto:</div><br class="m_-2508199895865596987Apple-interchange-newline"><div><div><div style="font-family:'times new roman','new york',times,serif;font-size:12pt"><div><br></div><div>I think we don't have enough logs to debug this so I would suggest you to provide more logs/info.<br></div><div>I have also observed that the configuration and setup of your volume is not very efficient.<br></div><div><br></div><div>For example: <br></div><div><span style="font-variant-ligatures:no-common-ligatures">Brick37: s04-stg:/gluster/mnt1/brick</span><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick38: s04-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick39: s04-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick40: s04-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick41: s04-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick42: s04-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick43: s04-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick44: s04-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick45: s04-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick46: s04-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick47: s04-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000"><span style="font-variant-ligatures:no-common-ligatures">Brick48: s04-stg:/gluster/mnt12/brick</span></div></div><div><br></div><div>These 12 bricks are on same node and the sub volume made up of these bricks will be of same subvolume, which is not good. Same is true for the bricks hosted on <span style="font-variant-ligatures:no-common-ligatures">s05-stg and <span style="font-variant-ligatures:no-common-ligatures">s06-stg</span></span><br></div><div>I think you have added these bricks after creating vol. The probability of disruption in connection of these bricks will be higher in this case.<br></div><div><br></div><div>---<br></div><div>Ashish<br></div><div><br></div><hr id="m_-2508199895865596987zwchr"><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>><br><b>To: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Sent: </b>Wednesday, September 26, 2018 3:38:35 PM<br><b>Subject: </b>[Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br><div><br></div>Dear All, Dear Nithya,<div><br></div><div>after upgrading from 3.10.5 version to 3.12.14, I tried to start a rebalance process to distribute data across the bricks, but something goes wrong.</div><div>Rebalance failed on different nodes and the time value needed to complete the procedure seems to be very high.</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s01 ~]# gluster volume rebalance tier2 status</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> --------- ----------- ----------- <wbr>----------- ----------- --<wbr>--------- ------------ --------------</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> localhost 19 161.6GB 537 2 2 in progress 0:32:23</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> s02-stg 25 212.7GB 526 5 2 in progress 0:32:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> s03-stg 4 69.1GB 511 0 0 in progress 0:32:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> s04-stg 4 484Bytes 12283 0 3 in progress 0:32:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> s05-stg 23 484Bytes 11049 0 10 in progress 0:32:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> s06-stg 3 1.2GB 8032 11 3 failed 0:17:57</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Estimated time left for rebalance to complete : 3601:05:41</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">volume rebalance: tier2: success</span></div></div><div><br></div><div>When rebalance processes fail, I can see the following kind of errors in /var/log/glusterfs/tier2-<wbr>rebalance.log</div><div><br></div><div>Error type 1)</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[2018-09-26 08:50:19.872575] W [MSGID: 122053] [ec-common.c:269:ec_check_<wbr>status] 0-tier2-disperse-10: Operation failed on 2 of 6 subvolumes.(up=111111, mask=100111, remaining=</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">000000, good=100111, bad=011000)</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[2018-09-26 08:50:19.901792] W [MSGID: 122053] [ec-common.c:269:ec_check_<wbr>status] 0-tier2-disperse-11: Operation failed on 1 of 6 subvolumes.(up=111111, mask=111101, remaining=</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">000000, good=111101, bad=000010)</span></div></div><div><br></div><div>Error type 2)</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[2018-09-26 08:53:31.566836] W [socket.c:600:__socket_rwv] 0-tier2-client-53: readv on <a href="http://192.168.0.55:49153" target="_blank">192.168.0.55:49153</a> failed (Connection reset by peer)</span></div></div><div><br></div><div>Error type 3)</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[2018-09-26 08:57:37.852590] W [MSGID: 122035] [ec-common.c:571:ec_child_<wbr>select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10)</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[2018-09-26 08:57:39.282306] W [MSGID: 122035] [ec-common.c:571:ec_child_<wbr>select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10)</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[2018-09-26 09:02:04.928408] W [MSGID: 109023] [dht-rebalance.c:1013:__dht_<wbr>check_free_space] 0-tier2-dht: data movement of file {blocks:0 name:(/OPA/archive/historical/<wbr>dts/MRE</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">A/Observations/Observations/<wbr>MREA14/Cs-1/CMCC/raw/CS013.<wbr>ext)} would result in dst node (tier2-disperse-5:2440190848) having lower disk space than the source node (tier2-dispers</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">e-11:71373083776).Skipping file.</span></div></div><div><br></div><div>Error type 4)</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">W [rpc-clnt-ping.c:223:rpc_clnt_<wbr>ping_cbk] 0-tier2-client-7: socket disconnected</span></div></div><div><br></div><div>Error type 5)</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[2018-09-26 09:07:42.333720] W [glusterfsd.c:1375:cleanup_<wbr>and_exit] (-->/lib64/libpthread.so.0(+<wbr>0x7e25) [0x7f0417e0ee25] -->/usr/sbin/glusterfs(<wbr>glusterfs_sigwaiter+0xe5) [0x55</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">90086004b5] -->/usr/sbin/glusterfs(<wbr>cleanup_and_exit+0x6b) [0x55900860032b] ) 0-: received signum (15), shutting down</span></div></div><div><br></div><div>Error type 6)</div><div><br></div><div>[2018-09-25 08:09:18.340658] C [rpc-clnt-ping.c:166:rpc_clnt_<wbr>ping_timer_expired] 0-tier2-client-4: server <a href="http://192.168.0.52:49153" target="_blank">192.168.0.52:49153</a> has not responded in the last 42 seconds, disconnecting.</div><div><br></div><div>It seems that there are some network or timeout problems, but the network usage/traffic values are not so high.</div><div>Do you think that, in my volume configuration, I have to modify some volume options related to thread and/or network parameters?</div><div>Could you, please, help me to understand the cause of the problems above?</div><div><br></div><div>You can find below our volume info:</div><div>(volume is implemented on 6 servers; each server configuration: 2 cpu 10-cores, 64GB RAM, 1 SSD dedicated to the OS, 12 x 10TB HD)</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s04 ~]# gluster vol info</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0);min-height:18px"><span style="font-variant-ligatures:no-common-ligatures"> </span><br class="m_-2508199895865596987webkit-block-placeholder"></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Volume Name: tier2</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Type: Distributed-Disperse</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Volume ID: a28d88c5-3295-4e35-98d4-<wbr>210b3af9358c</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Status: Started</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Snapshot Count: 0</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Number of Bricks: 12 x (4 + 2) = 72</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Transport-type: tcp</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Bricks:</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick1: s01-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick2: s02-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick3: s03-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick4: s01-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick5: s02-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick6: s03-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick7: s01-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick8: s02-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick9: s03-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick10: s01-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick11: s02-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick12: s03-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick13: s01-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick14: s02-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick15: s03-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick16: s01-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick17: s02-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick18: s03-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick19: s01-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick20: s02-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick21: s03-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick22: s01-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick23: s02-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick24: s03-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick25: s01-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick26: s02-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick27: s03-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick28: s01-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick29: s02-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick30: s03-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick31: s01-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick32: s02-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick33: s03-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick34: s01-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick35: s02-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick36: s03-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick37: s04-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick38: s04-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick39: s04-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick40: s04-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick41: s04-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick42: s04-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick43: s04-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick44: s04-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick45: s04-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick46: s04-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick47: s04-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick48: s04-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick49: s05-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick50: s05-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick51: s05-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick52: s05-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick53: s05-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick54: s05-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick55: s05-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick56: s05-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick57: s05-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick58: s05-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick59: s05-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick60: s05-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick61: s06-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick62: s06-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick63: s06-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick64: s06-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick65: s06-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick66: s06-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick67: s06-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick68: s06-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick69: s06-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick70: s06-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick71: s06-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Brick72: s06-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Options Reconfigured:</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">network.ping-timeout: 60</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">diagnostics.count-fop-hits: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">diagnostics.latency-<wbr>measurement: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.server-quorum-type: server</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.default-soft-limit: 90</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.quota-deem-statfs: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><a href="http://performance.io/" target="_blank">performance.io</a>-thread-count: 16</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">disperse.cpu-extensions: auto</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><a href="http://performance.io/" target="_blank">performance.io</a>-cache: off</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">network.inode-lru-limit: 50000</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.md-cache-timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.cache-<wbr>invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.stat-prefetch: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.cache-invalidation-<wbr>timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.cache-invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.readdir-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.parallel-readdir: off</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.readdir-ahead: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.lookup-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">client.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">server.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">nfs.disable: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">transport.address-family: inet</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.quorum-type: auto</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.min-free-disk: 10</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">performance.client-io-threads: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.quota: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.inode-quota: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.bitrot: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">features.scrub: Active</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.brick-multiplex: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">cluster.server-quorum-ratio: 51%</span></div></div><div><br></div><div>If it can help, I paste here the output of “free -m” command executed on all the cluster nodes:</div><div><br></div><div>The result is almost the same on every nodes. In your opinion, the available RAM is enough to support data movement?</div><div><br></div><div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">[root@s06 ~]# free -m</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> total used free shared buff/cache available</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Mem: 64309 10409 464 15 53434 52998</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Swap: 65535 103 65432</span></div></div><div><br></div><div>Thank you in advance.</div><div>Sorry for my long message, but I’m trying to notify you all available information.</div><div><br></div><div>Regards,</div><div>Mauro</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br>______________________________<wbr>_________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></div><div><br></div></div></div></div></blockquote></div><br><div><div style="word-wrap:break-word"><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div></div></div><br>______________________________<wbr>_________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></div><div><br></div></div></div></div></blockquote></div><div><div style="word-wrap:break-word"><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br class="m_-2508199895865596987Apple-interchange-newline"></div></div></div><br>______________________________<wbr>_________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></div><div><br></div></div></div></div></blockquote></div><br><div>
<span class="m_-2508199895865596987Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;border-spacing:0px"><div style="word-wrap:break-word"><span class="m_-2508199895865596987Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br class="m_-2508199895865596987Apple-interchange-newline">-------------------------</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Mauro Tridici</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Fondazione CMCC</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">CMCC Supercomputing Center</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">presso Complesso Ecotekne - Università del Salento -</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Strada Prov.le Lecce - Monteroni sn</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">73100 Lecce IT</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.cmcc.it/" target="_blank">http://www.cmcc.it</a><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">mobile: (+39) 327 5630841</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">email:<span class="m_-2508199895865596987Apple-converted-space"> </span><a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a><br></div><div><a href="https://it.linkedin.com/in/mauro-tridici-5977238b" target="_blank">https://it.linkedin.com/in/<wbr>mauro-tridici-5977238b</a><br></div></span></div></span>
</div>
<br></div></div></blockquote></div><br><div>
<span class="m_-2508199895865596987Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word"><span class="m_-2508199895865596987Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br class="m_-2508199895865596987Apple-interchange-newline">-------------------------</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Mauro Tridici</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Fondazione CMCC</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">CMCC Supercomputing Center</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">presso Complesso Ecotekne - Università del Salento -</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Strada Prov.le Lecce - Monteroni sn</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">73100 Lecce IT</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.cmcc.it/" target="_blank">http://www.cmcc.it</a><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">mobile: (+39) 327 5630841</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">email:<span class="m_-2508199895865596987Apple-converted-space"> </span><a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a><br></div><div><a href="https://it.linkedin.com/in/mauro-tridici-5977238b" target="_blank">https://it.linkedin.com/in/<wbr>mauro-tridici-5977238b</a><br></div></span></div></span>
</div>
<br><br>______________________________<wbr>_________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></div><div><br></div></div></div></div></blockquote></div><br><div>
<span class="m_-2508199895865596987Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word"><span class="m_-2508199895865596987Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br class="m_-2508199895865596987Apple-interchange-newline">-------------------------</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Mauro Tridici</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Fondazione CMCC</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">CMCC Supercomputing Center</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">presso Complesso Ecotekne - Università del Salento -</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Strada Prov.le Lecce - Monteroni sn</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">73100 Lecce IT</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.cmcc.it/" target="_blank">http://www.cmcc.it</a><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">mobile: (+39) 327 5630841</div><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">email:<span class="m_-2508199895865596987Apple-converted-space"> </span><a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a><br></div><div><a href="https://it.linkedin.com/in/mauro-tridici-5977238b" target="_blank">https://it.linkedin.com/in/<wbr>mauro-tridici-5977238b</a><br></div></span></div></span>
</div>
<br><br>______________________________<wbr>_________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></div><div><br></div></div></div></div></blockquote></div><br><div>
<span class="m_-2508199895865596987Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word"><span class="m_-2508199895865596987Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br class="m_-2508199895865596987Apple-interchange-newline">-------------------------</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Mauro Tridici</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Fondazione CMCC</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">CMCC Supercomputing Center</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">presso Complesso Ecotekne - Università del Salento -</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Strada Prov.le Lecce - Monteroni sn</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">73100 Lecce IT</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.cmcc.it" target="_blank">http://www.cmcc.it</a></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">mobile: (+39) 327 5630841</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">email:<span class="m_-2508199895865596987Apple-converted-space"> </span><a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a></div><div><a href="https://it.linkedin.com/in/mauro-tridici-5977238b" target="_blank">https://it.linkedin.com/in/<wbr>mauro-tridici-5977238b</a></div></span></div></span>
</div>
<br></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>