[Gluster-users] (3.1.6-1) How should I add extra peers to existing file systems ?

Mohit Anchlia mohitanchlia at gmail.com
Tue Aug 16 00:35:23 UTC 2011


I should have also asked you to stop and delete volume before getting
rid of gluster config files. Can you get rid of directories also
inside vols and try to restart? It's trying to look for volume files
that we just removed.

Also, just disable iptables for now explicitly.

On Mon, Aug 15, 2011 at 5:22 PM, Tomoaki Sato <tsato at valinux.co.jp> wrote:
>
>> 1) run peer detach for all the servers
>
> done.
>
>> 2) from server 1 ->3 and 3->1 make sure ports are open and iptables
>> are turned off.
>
> done.
> by the way, the same test on 3.1.5-1 works fine with same environment.
>
>> 3) remove config files under /etc/glusterd
>
> please review following logs.
>
>> 4) run your tests again.
>
> I don't know why but glusterd service failed to start on all 3 hosts.
>
> [root at vhead-010 ~]# date
> Tue Aug 16 09:12:53 JST 2011
> [root at vhead-010 ~]# cat a.sh
> #!/bin/bash
> for i in foo-{1..3}-private
> do
>        ssh ${i} service glusterd stop
>        ssh ${i} 'find /etc/glusterd -type f|xargs rm -f'
>        ssh ${i} service iptables restart
>        ssh ${i} iptables -vL
>        ssh ${i} service glusterd start
>        ssh ${i} find /etc/glusterd
>        ssh ${i} service glusterd status
> done
> [root at vhead-010 ~]# bash a.sh
> Stopping glusterd:[  OK  ]
> Flushing firewall rules: [  OK  ]
> Setting chains to policy ACCEPT: filter [  OK  ]
> Unloading iptables modules: [  OK  ]
> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>
> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
> Starting glusterd:[  OK  ]
> /etc/glusterd
> /etc/glusterd/glusterd.info
> /etc/glusterd/nfs
> /etc/glusterd/nfs/run
> /etc/glusterd/peers
> /etc/glusterd/vols
> /etc/glusterd/vols/foo
> /etc/glusterd/vols/foo/bricks
> /etc/glusterd/vols/foo/run
> glusterd is stopped
> Stopping glusterd:[  OK  ]
> Flushing firewall rules: [  OK  ]
> Setting chains to policy ACCEPT: filter [  OK  ]
> Unloading iptables modules: [  OK  ]
> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>
> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
> Starting glusterd:[  OK  ]
> /etc/glusterd
> /etc/glusterd/glusterd.info
> /etc/glusterd/nfs
> /etc/glusterd/nfs/run
> /etc/glusterd/peers
> /etc/glusterd/vols
> /etc/glusterd/vols/foo
> /etc/glusterd/vols/foo/bricks
> /etc/glusterd/vols/foo/run
> glusterd is stopped
> Stopping glusterd:[  OK  ]
> Flushing firewall rules: [  OK  ]
> Setting chains to policy ACCEPT: filter [  OK  ]
> Unloading iptables modules: [  OK  ]
> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
>
> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source
> destination
> Starting glusterd:[  OK  ]
> /etc/glusterd
> /etc/glusterd/glusterd.info
> /etc/glusterd/nfs
> /etc/glusterd/nfs/run
> /etc/glusterd/peers
> /etc/glusterd/vols
> /etc/glusterd/vols/foo
> /etc/glusterd/vols/foo/bricks
> /etc/glusterd/vols/foo/run
> glusterd is stopped
> [root at vhead-010 ~]# date
> Tue Aug 16 09:13:20 JST 2011
> [root at vhead-010 ~]# ssh foo-1-private
> Last login: Tue Aug 16 09:06:57 2011 from dlp.local.valinux.co.jp
> [root at localhost ~]# tail -20
> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
> ...
> [2011-08-16 09:13:01.85858] I [glusterd.c:304:init] 0-management: Using
> /etc/glusterd as working directory
> [2011-08-16 09:13:01.87294] E [rpc-transport.c:799:rpc_transport_load]
> 0-rpc-transport:
> /opt/glusterfs/3.1.6/lib64/glusterfs/3.1.6/rpc-transport/rdma.so: cannot
> open shared object file: No such file or directory
> [2011-08-16 09:13:01.87340] E [rpc-transport.c:803:rpc_transport_load]
> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not
> valid or not found on this machine
> [2011-08-16 09:13:01.87402] E
> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd: Unable to
> retrieve store handle for /etc/glusterd/glusterd.info, error: No such file
> or directory
> [2011-08-16 09:13:01.87422] E [glusterd-store.c:761:glusterd_retrieve_uuid]
> 0-: Unable to get store handle!
> [2011-08-16 09:13:01.87514] I [glusterd.c:95:glusterd_uuid_init] 0-glusterd:
> generated UUID: c0cef9f9-a79e-4189-8955-d83927db9cee
> [2011-08-16 09:13:01.87681] E
> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd: Unable to
> retrieve store handle for /etc/glusterd/vols/foo/info, error: No such file
> or directory
> [2011-08-16 09:13:01.87704] E
> [glusterd-store.c:1328:glusterd_store_retrieve_volumes] 0-: Unable to
> restore volume: foo

> [2011-08-16 09:13:01.87732] E [xlator.c:843:xlator_init] 0-management:
> Initialization of volume 'management' failed, review your volfile again
> [2011-08-16 09:13:01.87751] E [graph.c:331:glusterfs_graph_init]
> 0-management: initializing translator failed
> [2011-08-16 09:13:01.87818] I [glusterfsd.c:712:cleanup_and_exit]
> 0-glusterfsd: shutting down
> [root at localhost ~]# exit
>
> Best,
>
> (2011/08/16 8:52), Mohit Anchlia wrote:
>>
>> Logs are generally in /var/log/gluster
>>
>> Since you are playing with it. I would suggest this:
>>
>> 1) run peer detach for all the servers
>> 2) from server 1 ->3 and 3->1 make sure ports are open and iptables
>> are turned off.
>> 3) remove config files under /etc/glusterd
>> 4) run your tests again.
>>
>> On Mon, Aug 15, 2011 at 4:28 PM, Tomoaki Sato<tsato at valinux.co.jp>  wrote:
>>>
>>> Thanks, Mohit
>>>
>>> (2011/08/16 8:05), Mohit Anchlia wrote:
>>>>
>>>> What's in your logs?
>>>
>>> I can obtain logs needed. could you tell me the instruction to take the
>>> logs?
>>>
>>>>
>>>> Did you have  foo-3-private before in your gluster cluster ever or
>>>> adding this host for the first time?
>>>
>>> It was first time.
>>> All foo-X-private has no entries in /etc/glusterd/peers/ and
>>> /etc/glusterd/vols/.
>>>
>>>>
>>>> Try gluster peer detach and then remove any left over configuration in
>>>> /etc/glusterd config directory. After that try again and see if that
>>>> works.
>>>
>>> [root at vhead-010 ~]# date
>>> Tue Aug 16 08:17:49 JST 2011
>>> [root at vhead-010 ~]# cat a.sh
>>> #!/bin/bash
>>> for i in foo-{1..3}-private
>>> do
>>>        ssh ${i} service glusterd stop
>>>        ssh ${i} rm -rf /etc/glusterd/peers/*
>>>        ssh ${i} rm -rf /etc/glusterd/vols/*
>>>        ssh ${i} service glusterd start
>>>        ssh ${i} find /etc/glusterd
>>> done
>>> [root at vhead-010 ~]# bash a.sh
>>> Stopping glusterd:[  OK  ]
>>> Starting glusterd:[  OK  ]
>>> /etc/glusterd
>>> /etc/glusterd/glusterd.info
>>> /etc/glusterd/nfs
>>> /etc/glusterd/nfs/nfs-server.vol
>>> /etc/glusterd/nfs/run
>>> /etc/glusterd/peers
>>> /etc/glusterd/vols
>>> Stopping glusterd:[  OK  ]
>>> Starting glusterd:[  OK  ]
>>> /etc/glusterd
>>> /etc/glusterd/glusterd.info
>>> /etc/glusterd/nfs
>>> /etc/glusterd/nfs/nfs-server.vol
>>> /etc/glusterd/nfs/run
>>> /etc/glusterd/peers
>>> /etc/glusterd/vols
>>> Stopping glusterd:[  OK  ]
>>> Starting glusterd:[  OK  ]
>>> /etc/glusterd
>>> /etc/glusterd/glusterd.info
>>> /etc/glusterd/nfs
>>> /etc/glusterd/nfs/nfs-server.vol
>>> /etc/glusterd/nfs/run
>>> /etc/glusterd/peers
>>> /etc/glusterd/vols
>>> [root at vhead-010 ~]# ssh foo-1-private
>>> [root at localhost ~]# gluster peer probe foo-2-private
>>> Probe successful
>>> [root at localhost ~]# gluster peer status
>>> Number of Peers: 1
>>>
>>> Hostname: foo-2-private
>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>> State: Peer in Cluster (Connected)
>>> [root at localhost ~]# gluster volume create foo foo-1-private:/mnt/brick
>>> Creation of volume foo has been successful. Please start the volume to
>>> access da
>>> ta.
>>> [root at localhost ~]# gluster volume start foo
>>> Starting volume foo has been successful
>>> [root at localhost ~]# gluster volume add-brick foo foo-2-private:/mnt/brick
>>> Add Brick successful
>>> [root at localhost ~]# gluster peer probe foo-3-private
>>> Probe successful
>>> [root at localhost ~]# gluster peer status
>>> Number of Peers: 2
>>>
>>> Hostname: foo-2-private
>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: foo-3-private
>>> Uuid: 7fb98dac-fef7-4b33-837c-6483a767ec3e
>>> State: Peer Rejected (Connected)
>>> [root at localhost ~]# cat /var/log/glusterfs/.cmd_log_history
>>> ...
>>> [2011-08-16 08:20:28.862619] peer probe :  on host foo-2-private:24007
>>> [2011-08-16 08:20:28.912419] peer probe : on host foo-2-private:24007
>>> FAILED
>>> [2011-08-16 08:20:58.382350] Volume create : on volname: foo attempted
>>> [2011-08-16 08:20:58.382461] Volume create : on volname: foo type:DEFAULT
>>> count:
>>> 1 bricks: foo-1-private:/mnt/brick
>>> [2011-08-16 08:20:58.384674] Volume create : on volname: foo SUCCESS
>>> [2011-08-16 08:21:04.831772] volume start : on volname: foo SUCCESS
>>> [2011-08-16 08:21:22.682292] Volume add-brick : on volname: foo attempted
>>> [2011-08-16 08:21:22.682385] Volume add-brick : volname: foo type DEFAULT
>>> count:
>>> 1 bricks: foo-2-private:/mnt/brick
>>> [2011-08-16 08:21:22.682499] Volume add-brick : on volname: foo SUCCESS
>>> [2011-08-16 08:21:39.124574] peer probe :  on host foo-3-private:24007
>>> [2011-08-16 08:21:39.135609] peer probe : on host foo-3-private:24007
>>> FAILED
>>>
>>> Tomo
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 15, 2011 at 3:37 PM, Tomoaki Sato<tsato at valinux.co.jp>
>>>>  wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> following instructions work fine with 3.1.5-1 but with 3.1.6-1.
>>>>>
>>>>> 1. make a new file system without peers. [OK]
>>>>>
>>>>> foo-1-private# gluster volume create foo foo-1-private:/mnt/brick
>>>>> foo-1-private# gluster volume start foo
>>>>> foo-1-private# gluster peer status
>>>>> No peers present
>>>>> foo-1-private#
>>>>>
>>>>> 2. add a peer to the file system. [NG]
>>>>>
>>>>> foo-1-private# gluster peer probe foo-2-private
>>>>> Probe successful
>>>>> foo-1-private# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: foo-2-private
>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>> State: Peer Rejected (Connected)
>>>>> foo-1-private# gluster volume add-brick foo foo-2-private:/mnt/brick
>>>>> Host foo-2-private not connected
>>>>> foo-1-private#
>>>>>
>>>>>
>>>>> following instructions work fine even with 3.1.6-1.
>>>>>
>>>>> 1. make a new file system with single peer. [OK]
>>>>>
>>>>> foo-1-private# gluster peer status
>>>>> No peer presents
>>>>> foo-1-private# gluster peer probe foo-2-private
>>>>> Probe successful
>>>>> foo-1-private# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: foo-2-private
>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>> State: Peer in Cluster (Connected)
>>>>> foo-1-private# gluster volume create foo foo-1-private:/mnt/brick
>>>>> Creation of volume foo has been successful. Please start the volume to
>>>>> access data.
>>>>> foo-1-private# gluster volume start foo
>>>>> Starting volume foo has been successful
>>>>> foo-1-private# gluster volume add-brick foo foo-2-private:/mnt/brick
>>>>> Add Brick successful
>>>>> foo-1-private#
>>>>>
>>>>> But ...
>>>>>
>>>>> 2. add a peer to the file system. [NG]
>>>>>
>>>>> foo-1-private# gluster peer probe foo-3-private
>>>>> Probe successful
>>>>> foo-1-private# gluster peer status
>>>>> Number of Peers: 2
>>>>>
>>>>> Hostname: foo-2-private
>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> Hostname: foo-3-private
>>>>> Uuid: 7fb98dac-fef704b33-837c-6483a767ec3e
>>>>> State: Peer Rejected (Connected)
>>>>> foo-1-private# gluster volume add-brick foo foo-3-private:/mnt/brick
>>>>> Host foo-3-private not connected
>>>>> foo-1-private#
>>>>>
>>>>> How should I add extra peers to existing file systems ?
>>>>>
>>>>> Best,
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>
>>>
>>>
>
>



More information about the Gluster-users mailing list