<div dir="ltr"><div dir="ltr">Hi Nithya<div><br></div><div>I tried attching the logs but it was tool big. So I have put it on one drive accessible by everyone</div><div><br></div><div><a href="https://drive.google.com/drive/folders/1744WcOfrqe_e3lRPxLpQ-CBuXHp_o44T?usp=sharing">https://drive.google.com/drive/folders/1744WcOfrqe_e3lRPxLpQ-CBuXHp_o44T?usp=sharing</a> <br></div><div><div><br></div><div><br></div><div>I am attaching rebalance-logs which is for the period when I ran fix-layout after adding new disk and then started remove-disk option.</div><div><br></div><div> All of the nodes have atleast 8 TB disk available</div><div><br></div><div><div>/dev/sdb 73T 65T 8.0T 90% /glusteratlas/brick001</div><div>/dev/sdb 73T 65T 8.0T 90% /glusteratlas/brick002</div><div>/dev/sdb 73T 65T 8.0T 90% /glusteratlas/brick003</div><div>/dev/sdb 73T 65T 8.0T 90% /glusteratlas/brick004</div><div>/dev/sdb 73T 65T 8.0T 90% /glusteratlas/brick005</div><div>/dev/sdb 80T 67T 14T 83% /glusteratlas/brick006</div><div>/dev/sdb 37T 1.6T 35T 5% /glusteratlas/brick007</div><div>/dev/sdb 89T 15T 75T 17% /glusteratlas/brick008</div><div>/dev/sdb 89T 14T 76T 16% /glusteratlas/brick009</div></div><div><br></div><div>brick007 is the one I am removing</div><div><br></div><div><br></div><div><br></div><div><div>gluster volume info</div><div><br></div><div>Volume Name: atlasglust</div><div>Type: Distribute</div><div>Volume ID: fbf0ebb8-deab-4388-9d8a-f722618a624b</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 9</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: pplxgluster01**:/glusteratlas/brick001/gv0</div><div>Brick2: pplxgluster02.**:/glusteratlas/brick002/gv0</div><div>Brick3: pplxgluster03.**:/glusteratlas/brick003/gv0</div><div>Brick4: pplxgluster04.**:/glusteratlas/brick004/gv0</div><div>Brick5: pplxgluster05.**:/glusteratlas/brick005/gv0</div><div>Brick6: pplxgluster06.**:/glusteratlas/brick006/gv0</div><div>Brick7: pplxgluster07.**:/glusteratlas/brick007/gv0</div><div>Brick8: pplxgluster08.**:/glusteratlas/brick008/gv0</div><div>Brick9: pplxgluster09.**:/glusteratlas/brick009/gv0</div><div>Options Reconfigured:</div><div>nfs.disable: on</div><div>performance.readdir-ahead: on</div><div>transport.address-family: inet</div><div>auth.allow: ***</div><div>features.cache-invalidation: on</div><div>features.cache-invalidation-timeout: 600</div><div>performance.stat-prefetch: on</div><div>performance.md-cache-timeout: 600</div><div>performance.parallel-readdir: off</div><div>performance.cache-size: 1GB</div><div>performance.client-io-threads: on</div><div>cluster.lookup-optimize: on</div><div>client.event-threads: 4</div><div>server.event-threads: 4</div><div>performance.cache-invalidation: on</div><div>diagnostics.brick-log-level: WARNING</div><div>diagnostics.client-log-level: WARNING</div></div><div><br></div><div><br></div><div>Thanks</div></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 4, 2019 at 11:37 AM Nithya Balachandran <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 4 Feb 2019 at 16:39, mohammad kashif <<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Nithya<div><br></div><div>Thanks for replying so quickly. It is very much appreciated.</div><div><br></div><div>There are lots if " [No space left on device] " errors which I can not understand as there are much space on all of the nodes.</div></div></div></blockquote><div> </div><div>This means that Gluster could not find sufficient space for the file. Would you be willing to share your rebalance log file? </div><div>Please provide the following information:</div><div><ul><li>The gluster version <br></li><li>The gluster volume info for the volume<br></li><li>How full are the individual bricks for the volume?<br></li></ul></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div></div><div>A little bit of background will be useful in this case. I had cluster of seven nodes of varying capacity(73, 73, 73, 46, 46, 46,46 TB) . The cluster was almost 90% full so every node has almost 8 to 15 TB free space. I added two new nodes with 100TB each and ran fix-layout which completed successfully.</div><div><br></div><div>After that I started remove-brick operation. I don't think that any point , any of the nodes were 100% full. Looking at my ganglia graph, there is minimum 5TB always available at every node.</div><div><br></div><div>I was keeping an eye on remove-brick status and for very long time there was no failures and then at some point these 17000 failures appeared and it stayed like that.</div><div><br></div><div> Thanks</div><div><br></div><div>Kashif</div><div> </div><div><br></div><div><br></div><div> </div><div><br></div><div>Let me explain a little bit of background. </div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 4, 2019 at 5:09 AM Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>The status shows quite a few failures. Please check the rebalance logs to see why that happened. We can decide what to do based on the errors.</div><div>Once you run a commit, the brick will no longer be part of the volume and you will not be able to access those files via the client.</div><div>Do you have sufficient space on the remaining bricks for the files on the removed brick?</div><div><br></div><div>Regards,</div><div>Nithya</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 4 Feb 2019 at 03:50, mohammad kashif <<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi<div><br></div><div>I have a pure distributed gluster volume with nine nodes and trying to remove one node, I ran </div><div>gluster volume remove-brick atlasglust nodename:/glusteratlas/brick007/gv0 start<br></div><div><br></div><div>It completed but with around 17000 failures</div><div><br></div><div><div> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</div><div> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div><div> nodename 4185858 27.5TB 6746030 17488 0 completed 405:15:34</div></div><div><br></div><div>I can see that there is still 1.5 TB of data on the node which I was trying to remove.</div><div><br></div><div>I am not sure what to do now? Should I run remove-brick command again so the files which has been failed can be tried again?</div><div> </div><div>or should I run commit first and then try to remove node again?</div><div><br></div><div>Please advise as I don't want to remove files.</div><div><br></div><div>Thanks</div><div><br></div><div>Kashif</div><div><br></div><div><br></div><div><br></div></div></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>