<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr">On Tue, 18 Dec 2018 at 14:56, Stephen Remde <<a href="mailto:stephen.remde@gaist.co.uk">stephen.remde@gaist.co.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Nithya,<div><br></div><div>I've realised, I will not have enough space on the other bricks in my cluster to migrate data off the server so I can remove the single brick - is there a work around?</div><div><br></div><div>As you can see below, the new brick was created with the wrong raid configuration, so I want to remove it recreate the raid, and re add it.</div><div><pre id="gmail-m_6873517035830012855gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)">xxxxxx Filesystem Size Used Avail Use% Mounted on
dc4-01 /dev/md0 95T 87T 8.0T 92% /export/md0
dc4-01 /dev/md1 95T 87T 8.4T 92% /export/md1
dc4-01 /dev/md2 95T 86T 9.3T 91% /export/md2
dc4-01 /dev/md3 95T 86T 8.9T 91% /export/md3
dc4-02 /dev/md0 95T 89T 6.5T 94% /export/md0
dc4-02 /dev/md1 95T 87T 8.4T 92% /export/md1
dc4-02 /dev/md2 95T 87T 8.6T 91% /export/md2
dc4-02 /dev/md3 95T 86T 8.8T 91% /export/md3
dc4-03 /dev/md0 95T 74T 21T 78% /export/md0
dc4-03 /dev/md1 102T 519G 102T 1% /export/md1</pre></div></div></blockquote><div><br></div><div>I believe this is the brick being removed - the one that has about 519G of data? If I have understood the scenario properly, there seems to be plenty of free space on the other bricks (most seem to have terabytes free) . Is there something I am missing ?</div><div><br></div><div>Regards,</div><div>Nithya</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>This is the backup storage, so if I HAVE to lose the 519GB and resync, that's an acceptable worst-case.</div><div><br></div><div><pre id="gmail-m_6873517035830012855gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)">gluster> v info video-backup
Volume Name: video-backup
Type: Distribute
Volume ID: 887bdc2a-ca5e-4ca2-b30d-86831839ed04
Status: Started
Snapshot Count: 0
Number of Bricks: 10
Transport-type: tcp
Bricks:
Brick1: 10.0.0.41:/export/md0/brick
Brick2: 10.0.0.42:/export/md0/brick
Brick3: 10.0.0.43:/export/md0/brick
Brick4: 10.0.0.41:/export/md1/brick
Brick5: 10.0.0.42:/export/md1/brick
Brick6: 10.0.0.41:/export/md2/brick
Brick7: 10.0.0.42:/export/md2/brick
Brick8: 10.0.0.41:/export/md3/brick
Brick9: 10.0.0.42:/export/md3/brick
Brick10: 10.0.0.43:/export/md1/brick
Options Reconfigured:
cluster.rebal-throttle: aggressive
cluster.min-free-disk: 1%
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on</pre><pre id="gmail-m_6873517035830012855gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)"><br></pre><pre id="gmail-m_6873517035830012855gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)">Best,</pre><pre id="gmail-m_6873517035830012855gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)">Steve<br></pre></div><br><div class="gmail_quote"><div dir="ltr">On Wed, 12 Dec 2018 at 03:07, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div>This is the current behaviour of rebalance and nothing to be concerned about - it will migrate data on all bricks on the nodes which host the bricks being removed. The data on the removed bricks will be moved to other bricks, some of the data on the other bricks on the node will just be moved to other bricks based on the new directory layouts. </div><div>I will fix this in the near future but you don't need to to stop the remove-brick operation.<br></div><div><br></div><div>Regards,</div><div>Nithya</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, 12 Dec 2018 at 06:36, Stephen Remde <<a href="mailto:stephen.remde@gaist.co.uk" target="_blank">stephen.remde@gaist.co.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div><pre id="gmail-m_6873517035830012855gmail-m_7615781136042444765gmail-m_6876883359560866333gmail-hterm:copy-to-clipboard-source"><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal">I requested a brick be removed from a distribute only volume and it seems to be migrating data from the wrong brick... unless I am reading this wrong which I doubt because the disk usage is definitely decreasing on the wrong brick.</span></font></div><div><br></div><div><span style="white-space:normal"><font face="monospace, monospace">gluster> volume status</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Status of volume: video-backup</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Gluster process TCP Port RDMA Port Online Pid</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">------------------------------------------------------------------------------</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.41:/export/md0/brick 49172 0 Y 5306 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.42:/export/md0/brick 49172 0 Y 3651 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.43:/export/md0/brick 49155 0 Y 2826 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.41:/export/md1/brick 49173 0 Y 5311 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.42:/export/md1/brick 49173 0 Y 3656 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.41:/export/md2/brick 49174 0 Y 5316 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.42:/export/md2/brick 49174 0 Y 3662 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.41:/export/md3/brick 49175 0 Y 5322 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.42:/export/md3/brick 49175 0 Y 3667 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.43:/export/md1/brick 49156 0 Y 4836 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Task Status of Volume video-backup</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">------------------------------------------------------------------------------</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Task : Rebalance </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">ID : 7895be7c-4ab9-440d-a301-c11dae0dd9e1</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Status : completed </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">gluster> volume remove-brick video-backup 10.0.0.43:/export/md1/brick start</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">volume remove-brick start: success</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">ID: f666a196-03c2-4940-bd38-45d8383345a4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"><br></font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">gluster> volume status </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Status of volume: video-backup</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Gluster process TCP Port RDMA Port Online Pid</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">------------------------------------------------------------------------------</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.41:/export/md0/brick 49172 0 Y 5306 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.42:/export/md0/brick 49172 0 Y 3651 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.43:/export/md0/brick 49155 0 Y 2826 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.41:/export/md1/brick 49173 0 Y 5311 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.42:/export/md1/brick 49173 0 Y 3656 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.41:/export/md2/brick 49174 0 Y 5316 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.42:/export/md2/brick 49174 0 Y 3662 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.41:/export/md3/brick 49175 0 Y 5322 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.42:/export/md3/brick 49175 0 Y 3667 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Brick 10.0.0.43:/export/md1/brick 49156 0 Y 4836 </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Task Status of Volume video-backup</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">------------------------------------------------------------------------------</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Task : Remove brick </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">ID : f666a196-03c2-4940-bd38-45d8383345a4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Removed bricks: </font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">10.0.0.43:/export/md1/brick</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">Status : in progress </font></span></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal"><br></span></font></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal"><br></span></font></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal">But when I check the rebalance log on the host with the brick being removed, it is actually migrating data from the other brick on the same host 10.0.0.43:/export/md0/brick </span></font></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal"><br></span></font></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal"><br></span></font></div><div><span style="white-space:normal"><font face="monospace, monospace">.....</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:52.572657] I [MSGID: 109086] [dht-shared.c:297:dht_parse_decommissioned_bricks] 0-video-backup-dht: <b>decommissioning subvolume video-backup-client-9</b></font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">....</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 29: volume video-backup-client-2</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 30: type protocol/client</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 31: option clnt-lk-version 1</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 32: option volfile-checksum 0</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 33: option volfile-key rebalance/video-backup</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 34: option client-version 3.8.15</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 35: option process-uuid node-dc4-03-25536-2018/12/11-11:59:47:551328-video-backup-client-2-0-0</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 36: option fops-version 1298437</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 37: option ping-timeout 42</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 38: option remote-host 10.0.0.43</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 39: option remote-subvolume /export/md0/brick</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 40: option transport-type socket</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 41: option transport.address-family inet</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 42: option username 9e7fe743-ecd7-40aa-b3db-e112086b2fc7</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 43: option password dab178d6-ecb4-4293-8c1d-6281ec2cafc2</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"> 44: end-volume</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">...</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">112: volume video-backup-client-9</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">113: type protocol/client</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">114: option ping-timeout 42</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">115: option remote-host 10.0.0.43</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">116: option remote-subvolume /export/md1/brick</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">117: option transport-type socket</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">118: option transport.address-family inet</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">119: option username 9e7fe743-ecd7-40aa-b3db-e112086b2fc7</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">120: option password dab178d6-ecb4-4293-8c1d-6281ec2cafc2</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">121: end-volume</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">...</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:52.608698] I [dht-rebalance.c:3668:gf_defrag_start_crawl] 0-video-backup-dht: gf_defrag_start_crawl using commit hash 3766302106</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:52.609478] I [MSGID: 109081] [dht-common.c:4198:dht_setxattr] 0-video-backup-dht: fixing the layout of /</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:52.615348] I [MSGID: 0] [dht-rebalance.c:3746:gf_defrag_start_crawl] 0-video-backup-dht: local subvols are video-backup-client-2</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:52.615378] I [MSGID: 0] [dht-rebalance.c:3746:gf_defrag_start_crawl] 0-video-backup-dht: local subvols are video-backup-client-9</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">...</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:52.616554] I [dht-rebalance.c:2652:gf_defrag_process_dir] 0-video-backup-dht: migrate data called on /</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:54.000363] I [dht-rebalance.c:1230:dht_migrate_file] 0-video-backup-dht: /symlinks.txt: attempting to move from video-backup-client-2 to video-backup-client-4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:55.110549] I [MSGID: 109022] [dht-rebalance.c:1703:dht_migrate_file] 0-video-backup-dht: completed migration of /symlinks.txt from subvolume video-backup-client-2 to video-backup-client-4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.100931] I [MSGID: 109081] [dht-common.c:4198:dht_setxattr] 0-video-backup-dht: fixing the layout of /A6</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.107389] I [dht-rebalance.c:2652:gf_defrag_process_dir] 0-video-backup-dht: migrate data called on /A6</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.132138] I [dht-rebalance.c:2866:gf_defrag_process_dir] 0-video-backup-dht: Migration operation on dir /A6 took 0.02 secs</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.330393] I [MSGID: 109081] [dht-common.c:4198:dht_setxattr] 0-video-backup-dht: fixing the layout of /A6/2017</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.337601] I [dht-rebalance.c:2652:gf_defrag_process_dir] 0-video-backup-dht: migrate data called on /A6/2017</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.493906] I [dht-rebalance.c:1230:dht_migrate_file] 0-video-backup-dht: /A6/2017/57c81ed09f31cd6c1c8990ae20160908101048: attempting to move from video-backup-client-2 to video-backup-client-4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.706068] I [dht-rebalance.c:1230:dht_migrate_file] 0-video-backup-dht: /A6/2017/57c81ed09f31cd6c1c8990ae20160908120734132317: attempting to move from video-backup-client-2 to video-backup-client-4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.783952] I [dht-rebalance.c:1230:dht_migrate_file] 0-video-backup-dht: /A6/2017/584a8bcdaca0515f595dff8820161124091841: attempting to move from video-backup-client-2 to video-backup-client-4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.843315] I [dht-rebalance.c:1230:dht_migrate_file] 0-video-backup-dht: /A6/2017/584a8bcdaca0515f595dff8820161124135453: attempting to move from video-backup-client-2 to video-backup-client-4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:58.951637] I [dht-rebalance.c:1230:dht_migrate_file] 0-video-backup-dht: /A6/2017/584a8bcdaca0515f595dff8820161122111252: attempting to move from video-backup-client-2 to video-backup-client-4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:59.005324] I [dht-rebalance.c:2866:gf_defrag_process_dir] 0-video-backup-dht: Migration operation on dir /A6/2017 took 0.67 secs</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">[2018-12-11 11:59:59.005362] I [dht-rebalance.c:1230:dht_migrate_file] 0-video-backup-dht: /A6/2017/58906aaaaca0515f5994104d20170213154555: attempting to move from video-backup-client-2 to video-backup-client-4</font></span></div><div><span style="white-space:normal"><font face="monospace, monospace"><br></font></span></div><div><span style="white-space:normal"><font face="monospace, monospace">etc...</font></span></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal"><br></span></font></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal">Can I stop/cancel it without data loss? How can I make gluster remove the correct brick? </span></font></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal"><br></span></font></div><div><font face="Arial, Helvetica, sans-serif"><span style="white-space:normal">Thanks</span></font></div></pre></div><div dir="ltr" class="gmail-m_6873517035830012855gmail-m_7615781136042444765gmail-m_6876883359560866333gmail_signature"><div dir="ltr"><div><div dir="ltr"></div></div></div></div></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>
</blockquote></div><br clear="all"><div><br></div><div dir="ltr" class="gmail-m_6873517035830012855gmail_signature"><div dir="ltr"><div><div dir="ltr"></div></div></div></div></div>
</blockquote></div></div>