[Gluster-users] Creating cluster replica on 2 nodes 2 bricks each.
Nithya Balachandran
nbalacha at redhat.com
Fri Jan 12 15:46:15 UTC 2018
---------- Forwarded message ----------
From: Jose Sanchez <josesanc at carc.unm.edu>
Date: 11 January 2018 at 22:05
Subject: Re: [Gluster-users] Creating cluster replica on 2 nodes 2 bricks
each.
To: Nithya Balachandran <nbalacha at redhat.com>
Cc: gluster-users <gluster-users at gluster.org>
Hi Nithya
Thanks for helping me with this, I understand now , but I have few
questions.
When i had it setup in replica (just 2 nodes with 2 bricks) and tried to
added , it failed.
[root at gluster01 ~]# gluster volume add-brick scratch replica 2
> gluster01ib:/gdata/brick2/scratch gluster02ib:/gdata/brick2/scratch
> volume add-brick: failed: /gdata/brick2/scratch is already part of a volume
>
Did you try the add brick operation several times with the same bricks? If
yes, that could be the cause as Gluster sets xattrs on the brick root
directory.
and after that, I ran the status and info in it and on the status i get
just the two brikcs
Brick gluster01ib:/gdata/brick1/scratch 49152 49153 Y
> 3140
> Brick gluster02ib:/gdata/brick1/scratch 49153 49154 Y
> 2634
>
and on the info i get all 4 ( 2 x2) is this normal?? behavior?
So the brick count does not match for the same volume in the gluster volume
status and gluster volume info commands? No, that is not normal.
Bricks:
> Brick1: gluster01ib:/gdata/brick1/scratch
> Brick2: gluster02ib:/gdata/brick1/scratch
> Brick3: gluster01ib:/gdata/brick2/scratch
> Brick4: gluster02ib:/gdata/brick2/scratch
>
Now when i try to mount it , i still get only 14 tb and not 28? Am i doing
something wrong? also when I start/stop services, cluster goes back to
replicated mode from distributed-replicate
If the fuse mount sees only 2 bricks , that would explain the 14TB.
gluster01ib:/scratch 14T 34M 14T 1% /mnt/gluster_test
—— Gluster mount log file ——
[2018-01-11 16:06:44.963043] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-scratch-client-1:
Connected to scratch-client-1, attached to remote volume
'/gdata/brick1/scratch'.
[2018-01-11 16:06:44.963065] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-scratch-client-1: Server
and Client lk-version numbers are not same, reopening the fds
[2018-01-11 16:06:44.968291] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-scratch-client-1:
Server lk version = 1
[2018-01-11 16:06:44.968404] I [fuse-bridge.c:4147:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
7.22
[2018-01-11 16:06:44.968438] I [fuse-bridge.c:4832:fuse_graph_sync] 0-fuse:
switched to graph 0
[2018-01-11 16:06:44.969544] I [MSGID: 108031]
[afr-common.c:2166:afr_local_discovery_cbk] 0-scratch-replicate-0:
selecting local read_child scratch-client-0
—— CLI Log File ——
[root at gluster01 glusterfs]# tail cli.log
[2018-01-11 15:54:14.468122] I [socket.c:2403:socket_event_handler]
0-transport: disconnecting now
[2018-01-11 15:54:14.468737] I [cli-rpc-ops.c:817:gf_cli_get_volume_cbk]
0-cli: Received resp to get vol: 0
[2018-01-11 15:54:14.469462] I [cli-rpc-ops.c:817:gf_cli_get_volume_cbk]
0-cli: Received resp to get vol: 0
[2018-01-11 15:54:14.469530] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2018-01-11 16:03:40.422568] I [cli.c:728:main] 0-cli: Started running
gluster with version 3.8.15
[2018-01-11 16:03:40.430195] I [cli-cmd-volume.c:1828:cli_check_gsync_present]
0-: geo-replication not installed
[2018-01-11 16:03:40.430492] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2018-01-11 16:03:40.430568] I [socket.c:2403:socket_event_handler]
0-transport: disconnecting now
[2018-01-11 16:03:40.485256] I [cli-rpc-ops.c:2244:gf_cli_set_volume_cbk]
0-cli: Received resp to set
[2018-01-11 16:03:40.485497] I [input.c:31:cli_batch] 0-: Exiting with: 0
—— etc-glusterfs-glusterd.vol.log —
[2018-01-10 14:59:23.676814] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume scratch
[2018-01-10 15:00:29.516071] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
[2018-01-10 15:01:09.872082] I [MSGID: 106482]
[glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management:
Received add brick req
[2018-01-10 15:01:09.872128] I [MSGID: 106578]
[glusterd-brick-ops.c:499:__glusterd_handle_add_brick] 0-management:
replica-count is 2
[2018-01-10 15:01:09.876763] E [MSGID: 106451]
[glusterd-utils.c:6207:glusterd_is_path_in_use] 0-management:
/gdata/brick2/scratch is already part of a volume [File exists]
[2018-01-10 15:01:09.876807] W [MSGID: 106122]
[glusterd-mgmt.c:188:gd_mgmt_v3_pre_validate_fn] 0-management: ADD-brick
prevalidation failed.
[2018-01-10 15:01:09.876822] E [MSGID: 106122]
[glusterd-mgmt.c:884:glusterd_mgmt_v3_pre_validate] 0-management: Pre
Validation failed for operation Add brick on local node
[2018-01-10 15:01:09.876834] E [MSGID: 106122]
[glusterd-mgmt.c:2009:glusterd_mgmt_v3_initiate_all_phases] 0-management:
Pre Validation Failed
[2018-01-10 15:01:16.005881] I [run.c:191:runner_log]
(-->/usr/lib64/glusterfs/3.8.15/xlator/mgmt/glusterd.so(+0x33045)
[0x7f1066d15045] -->/usr/lib64/glusterfs/3.8.15
/xlator/mgmt/glusterd.so(+0xcbd85) [0x7f1066dadd85]
-->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f10726491e5] )
0-management: Ran script: /var/lib/glusterd/hooks/1/add-
brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=scratch --version=1
--volume-op=add-brick --gd-workdir=/var/lib/glusterd
[2018-01-10 15:01:15.982929] E [MSGID: 106451]
[glusterd-utils.c:6207:glusterd_is_path_in_use] 0-management:
/gdata/brick2/scratch is already part of a volume [File exists]
[2018-01-10 15:01:16.005959] I [MSGID: 106578]
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 0-management:
replica-count is set 0
Atin, is this correct? It looks like it tries to add the bricks even though
the prevalidation failed
[2018-01-10 15:01:16.006018] I [MSGID: 106578]
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management:
type is set 0, need to change it
[2018-01-10 15:01:16.062001] I [MSGID: 106143]
[glusterd-pmap.c:250:pmap_registry_bind] 0-pmap: adding brick
/gdata/brick2/scratch on port 49154
[2018-01-10 15:01:16.062137] I [MSGID: 106143]
[glusterd-pmap.c:250:pmap_registry_bind] 0-pmap: adding brick
/gdata/brick2/scratch.rdma on port 49155
[2018-01-10 15:01:16.062673] E [MSGID: 106005]
[glusterd-utils.c:4877:glusterd_brick_start] 0-management: Unable to start
brick gluster01ib:/gdata/brick2/scratch
[2018-01-10 15:01:16.062715] E [MSGID: 106074]
[glusterd-brick-ops.c:2493:glusterd_op_add_brick] 0-glusterd: Unable to add
bricks
[2018-01-10 15:01:16.062729] E [MSGID: 106123]
[glusterd-mgmt.c:294:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit
failed.
[2018-01-10 15:01:16.062741] E [MSGID: 106123]
[glusterd-mgmt.c:1427:glusterd_mgmt_v3_commit] 0-management: Commit failed
for operation Add brick on local node
[2018-01-10 15:01:16.062754] E [MSGID: 106123]
[glusterd-mgmt.c:2018:glusterd_mgmt_v3_initiate_all_phases] 0-management:
Commit Op Failed
[2018-01-10 15:01:35.914090] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume scratch
[2018-01-10 15:01:15.979236] I [MSGID: 106482]
[glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management:
Received add brick req
[2018-01-10 15:01:15.979250] I [MSGID: 106578]
[glusterd-brick-ops.c:499:__glusterd_handle_add_brick] 0-management:
replica-count is 2
The message "I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
0-management: Received get vol req" repeated 3 times between [2018-01-10
15:00:29.516071] and [2018-01-10 15:01:39.652014]
[2018-01-10 16:16:42.776653] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
[2018-01-10 16:16:42.777614] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
[2018-01-11 15:45:09.023393] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
[2018-01-11 15:45:19.916301] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume scratch
[2018-01-11 15:45:09.024217] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
[2018-01-11 15:54:10.172137] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume scratch
[2018-01-11 15:54:14.468529] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
[2018-01-11 15:54:14.469408] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
Thanks
Jose
---------------------------------
Jose Sanchez
Center of Advanced Research Computing
Albuquerque, NM 87131
On Jan 10, 2018, at 9:02 PM, Nithya Balachandran <nbalacha at redhat.com>
wrote:
Hi Jose,
Gluster is working as expected. The Distribute-replicated type just means
that there are now 2 replica sets and files will be distributed across
them.
A volume of type Replicate (1xn where n is the number of bricks in the
replica set) indicates there is no distribution (all files on the
volume will be present on all the bricks in the volume).
A volume of type Distributed-Replicate indicates the volume is both
distributed (as in files will only be created on one of the replicated
sets) and replicated. So in the above example, a file will exist on either
Brick1 and Brick2 or Brick3 and Brick4.
After the add brick, the volume will have a total capacity of 28TB and
store 2 copies of every file. Let me know if that is not what you are
looking for.
Regards,
Nithya
On 10 January 2018 at 20:40, Jose Sanchez <josesanc at carc.unm.edu> wrote:
>
>
> Hi Nithya
>
> This is what i have so far, I have peer both cluster nodes together as
> replica, from node 1A and 1B , now when i tried to add it , i get the error
> that it is already part of a volume. when i run the cluster volume info , i
> see that has switch to distributed-replica.
>
> Thanks
>
> Jose
>
>
>
>
>
> [root at gluster01 ~]# gluster volume status
> Status of volume: scratch
> Gluster process TCP Port RDMA Port Online
> Pid
> ------------------------------------------------------------
> ------------------
> Brick gluster01ib:/gdata/brick1/scratch 49152 49153 Y
> 3140
> Brick gluster02ib:/gdata/brick1/scratch 49153 49154 Y
> 2634
> Self-heal Daemon on localhost N/A N/A Y
> 3132
> Self-heal Daemon on gluster02ib N/A N/A Y
> 2626
>
> Task Status of Volume scratch
> ------------------------------------------------------------
> ------------------
> There are no active volume tasks
>
> [root at gluster01 ~]#
>
> [root at gluster01 ~]# gluster volume info
>
> Volume Name: scratch
> Type: *Replicate*
> Volume ID: a6e20f7d-13ed-4293-ab8b-d783d1748246
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp,rdma
> Bricks:
> Brick1: gluster01ib:/gdata/brick1/scratch
> Brick2: gluster02ib:/gdata/brick1/scratch
> Options Reconfigured:
> performance.readdir-ahead: on
> nfs.disable: on
> [root at gluster01 ~]#
>
>
> -------------------------------------
>
> [root at gluster01 ~]# gluster volume add-brick scratch replica 2
> gluster01ib:/gdata/brick2/scratch gluster02ib:/gdata/brick2/scratch
> volume add-brick: failed: /gdata/brick2/scratch is already part of a volume
>
>
> [root at gluster01 ~]# gluster volume status
> Status of volume: scratch
> Gluster process TCP Port RDMA Port Online
> Pid
> ------------------------------------------------------------
> ------------------
> Brick gluster01ib:/gdata/brick1/scratch 49152 49153 Y
> 3140
> Brick gluster02ib:/gdata/brick1/scratch 49153 49154 Y
> 2634
> Self-heal Daemon on gluster02ib N/A N/A Y
> 2626
> Self-heal Daemon on localhost N/A N/A Y
> 3132
>
> Task Status of Volume scratch
> ------------------------------------------------------------
> ------------------
> There are no active volume tasks
>
> [root at gluster01 ~]# gluster volume info
>
> Volume Name: scratch
> Type: *Distributed-Replicate*
> Volume ID: a6e20f7d-13ed-4293-ab8b-d783d1748246
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp,rdma
> Bricks:
> Brick1: gluster01ib:/gdata/brick1/scratch
> Brick2: gluster02ib:/gdata/brick1/scratch
> Brick3: gluster01ib:/gdata/brick2/scratch
> Brick4: gluster02ib:/gdata/brick2/scratch
> Options Reconfigured:
> performance.readdir-ahead: on
> nfs.disable: on
> [root at gluster01 ~]#
>
>
>
> --------------------------------
> Jose Sanchez
> Center of Advanced Research Computing
> Albuquerque, NM 87131-0001
> carc.unm.edu
>
>
> On Jan 9, 2018, at 9:04 PM, Nithya Balachandran <nbalacha at redhat.com>
> wrote:
>
> Hi,
>
> Please let us know what commands you ran so far and the output of the *gluster
> volume info* command.
>
> Thanks,
> Nithya
>
> On 9 January 2018 at 23:06, Jose Sanchez <josesanc at carc.unm.edu> wrote:
>
>> Hello
>>
>> We are trying to setup Gluster for our project/scratch storage HPC
>> machine using a replicated mode with 2 nodes, 2 bricks each (14tb each).
>>
>> Our goal is to be able to have a replicated system between node 1 and 2
>> (A bricks) and add an additional 2 bricks (B bricks) from the 2 nodes. so
>> we can have a total of 28tb replicated mode.
>>
>> Node 1 [ (Brick A) (Brick B) ]
>> Node 2 [ (Brick A) (Brick B) ]
>> --------------------------------------------
>> 14Tb + 14Tb = 28Tb
>>
>> At this I was able to create the replica nodes between node 1 and 2
>> (brick A) but I’ve not been able to add to the replica together, Gluster
>> switches to distributed replica when i add it with only 14Tb.
>>
>> Any help will be appreciated.
>>
>> Thanks
>>
>> Jose
>>
>> ---------------------------------
>> Jose Sanchez
>> Center of Advanced Research Computing
>> Albuquerque, NM 87131
>> carc.unm.edu
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180112/48ae0783/attachment.html>
More information about the Gluster-users
mailing list