[Gluster-users] A bunch of comments about brick management
James
purpleidea at gmail.com
Wed Oct 30 10:40:08 UTC 2013
Hey there,
I've been madly hacking on cool new puppet-gluster features... In my
lack of sleep, I've put together some comments about gluster add/remove
brick features. Hopefully they are useful, and make sense. These are
sort of "bugs". Have a look, and let me know if I should formally report
any of these...
Cheers...
James
PS: this is also mirrored here:
http://paste.fedoraproject.org/50402/12956713
because email has destroyed formatting :P
All tests are done on gluster 3.4.1, using CentOS 6.4 on vm's.
Firewall has been disabled for testing purposes.
gluster --version
glusterfs 3.4.1 built on Sep 27 2013 13:13:58
### 1) simple operations shouldn't fail
# running the following commands in succession without files:
# gluster volume add-brick examplevol vmx1.example.com:/tmp/foo9
vmx2.example.com:/tmp/foo9
# gluster volume remove-brick examplevol vmx1.example.com:/tmp/foo9
vmx2.example.com:/tmp/foo9 start ... status
shows a failure:
[root at vmx1 ~]# gluster volume add-brick examplevol
vmx1.example.com:/tmp/foo9 vmx2.example.com:/tmp/foo9
volume add-brick: success
[root at vmx1 ~]# gluster volume remove-brick examplevol
vmx1.example.com:/tmp/foo9 vmx2.example.com:/tmp/foo9 status
Node Rebalanced-files size
scanned failures skipped status run-time in secs
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes
0 0 not started 0.00
vmx2.example.com 0 0Bytes
0 0 not started 0.00
[root at vmx1 ~]# gluster volume remove-brick examplevol
vmx1.example.com:/tmp/foo9 vmx2.example.com:/tmp/foo9 start
volume remove-brick start: success
ID: ecbcc2b6-4351-468a-8f53-3a09159e4059
[root at vmx1 ~]# gluster volume remove-brick examplevol
vmx1.example.com:/tmp/foo9 vmx2.example.com:/tmp/foo9 status
Node Rebalanced-files size
scanned failures skipped status run-time in secs
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes
8 0 completed 0.00
vmx2.example.com 0 0Bytes
0 1 failed 0.00
[root at vmx1 ~]# gluster volume remove-brick examplevol
vmx1.example.com:/tmp/foo9 vmx2.example.com:/tmp/foo9 commit
Removing brick(s) can result in data loss. Do you want to Continue?
(y/n) y
volume remove-brick commit: success
[root at vmx1 ~]#
### 1b) on the other node, the output shows an extra row (also including
the failure)
[root at vmx2 ~]# gluster volume remove-brick examplevol
vmx1.example.com:/tmp/foo9 vmx2.example.com:/tmp/foo9 status
Node Rebalanced-files size
scanned failures skipped status run-time in secs
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes
0 0 completed 0.00
localhost 0 0Bytes
0 0 completed 0.00
vmx1.example.com 0 0Bytes
0 1 failed 0.00
### 2) formatting:
# the "skipped" column doesn't seem to have any data, as a result
formatting is broken...
# this problem is obviously not seen in the more useful --xml output
below. neither is the 'skipped' column.
[root at vmx1 examplevol]# gluster volume remove-brick examplevol
vmx1.example.com:/tmp/foo3 vmx2.example.com:/tmp/foo3 status
Node Rebalanced-files size
scanned failures skipped status run-time in secs
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes
8 0 completed 0.00
vmx2.example.com 0 0Bytes
8 0 completed 0.00
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
<opRet>0</opRet>
<opErrno>115</opErrno>
<opErrstr/>
<volRemoveBrick>
<task-id>d99cab76-cd7d-4579-80ae-c1e6faff3d1d</task-id>
<nodeCount>2</nodeCount>
<node>
<nodeName>localhost</nodeName>
<files>0</files>
<size>0</size>
<lookups>8</lookups>
<failures>0</failures>
<status>3</status>
<statusStr>completed</statusStr>
</node>
<node>
<nodeName>vmx2.example.com</nodeName>
<files>0</files>
<size>0</size>
<lookups>8</lookups>
<failures>0</failures>
<status>3</status>
<statusStr>completed</statusStr>
</node>
<aggregate>
<files>0</files>
<size>0</size>
<lookups>16</lookups>
<failures>0</failures>
<status>3</status>
<statusStr>completed</statusStr>
</aggregate>
</volRemoveBrick>
</cliOutput>
### 3)
[root at vmx1 examplevol]# gluster volume remove-brick examplevol
vmx1.example.com:/tmp/foo3 vmx2.example.com:/tmp/foo3 status
Node Rebalanced-files size
scanned failures skipped status run-time in secs
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes
8 0 completed 0.00
vmx2.example.com 0 0Bytes
8 0 completed 0.00
[root at vmx1 examplevol]# gluster volume remove-brick examplevol
vmx1.example.com:/tmp/foo3 vmx2.example.com:/tmp/foo3 commit
Removing brick(s) can result in data loss. Do you want to Continue?
(y/n) y
volume remove-brick commit: success
This shouldn't warn you that you might experience data loss. If the
rebalance has successfully worked, and bricks shouldn't be accepting new
files, then gluster should know this, and just let you commit safely.
I guess you can consider this a UI bug, as long as it checks before hand
that it's safe.
### 4)
Aggregate "totals", aka the <aggregate> </aggregate> data isn't shown in
the normal command line output.
### 5) the volume shouldn't have to be "started" for a rebalance to
work... we might want to do a rebalance, but keep it "stopped" so that
clients can't mount.
This is probably due to gluster needing it "online" to rebalance, but
nonetheless, it doesn't work with what users/sysadmins expect.
### 6) in the command: gluster volume rebalance myvolume status ;
gluster volume rebalance myvolume status --xml && echo t
No where does it mention the volume, or the specific bricks which are
being [re-]balanced. In particular, a volume name would be especially
useful in the --xml output.
This would be useful if multiple rebalances are going on... I realize
this is because the rebalance command only allows you to specify one
volume at a time, but to be consistent with other commands, a volume
rebalance status command should let you get info on many volumes.
Also, still missing per brick information.
Node Rebalanced-files size
scanned failures skipped status run time in secs
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes
2 0 0 in progress 1.00
vmx2.example.com 0 0Bytes
7 0 0 in progress 1.00
volume rebalance: examplevol: success:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
<opRet>0</opRet>
<opErrno>115</opErrno>
<opErrstr/>
<volRebalance>
<task-id>c5e9970b-f96a-4a28-af14-5477cf90d638</task-id>
<op>3</op>
<nodeCount>2</nodeCount>
<node>
<nodeName>localhost</nodeName>
<files>0</files>
<size>0</size>
<lookups>2</lookups>
<failures>0</failures>
<status>1</status>
<statusStr>in progress</statusStr>
</node>
<node>
<nodeName>vmx2.example.com</nodeName>
<files>0</files>
<size>0</size>
<lookups>7</lookups>
<failures>0</failures>
<status>1</status>
<statusStr>in progress</statusStr>
</node>
<aggregate>
<files>0</files>
<size>0</size>
<lookups>9</lookups>
<failures>0</failures>
<status>1</status>
<statusStr>in progress</statusStr>
</aggregate>
</volRebalance>
</cliOutput>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131030/94d04e44/attachment.sig>
More information about the Gluster-users
mailing list