[Gluster-users] gsyncd_template.conf: Permission denied during geo-replication

Stefan Förster cite+gluster at incertum.net
Thu Nov 17 11:29:44 UTC 2016


Hello world,

I'm currently setting up a few Vagrant boxes to test out
geo-replication. I have three boxes in two locations, fra-node[1-3] and
chi-node[1-3]. I set up a single replicated volume per cluster
(fra-volume and chi-volume) and then initiate a geo-replication from fra
to chi. After setting up the mountbroker on one of the slave nodes, when
restaring glusterd, it re-creates the file gsyncd_template.conf with
root:root ownership:

#v+
[root at chi-node2 ~]# chgrp geogroup /var/lib/glusterd/geo-replication/gsyncd_template.conf 
[root at chi-node2 ~]# chmod g+rw !$
chmod g+rw /var/lib/glusterd/geo-replication/gsyncd_template.conf
[root at chi-node2 ~]# ls -l !$
ls -l /var/lib/glusterd/geo-replication/gsyncd_template.conf
-rwxrwxr-x 1 root geogroup 1858 Nov 17 10:28 /var/lib/glusterd/geo-replication/gsyncd_template.conf
[root at chi-node2 ~]# service glusterd restart
Stopping glusterd:                                         [  OK  ]
Starting glusterd:                                         [  OK  ]
[root at chi-node2 ~]# ls -l /var/lib/glusterd/geo-replication/gsyncd_template.conf
-rwxr-xr-x 1 root root 1858 Nov 17 12:26 /var/lib/glusterd/geo-replication/gsyncd_template.conf
#v-

And geo-replication fails then fails:

#v+
Popen: ssh> [2016-11-17 09:33:56.824232] I [cli.c:730:main] 0-cli: Started running /usr/sbin/gluster with version 3.8.5
Popen: ssh> [2016-11-17 09:33:56.824321] I [cli.c:613:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost
Popen: ssh> [2016-11-17 09:33:56.907702] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
Popen: ssh> [2016-11-17 09:33:56.908634] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
Popen: ssh> [2016-11-17 09:33:56.908747] I [cli-rpc-ops.c:6655:gf_cli_getwd_cbk] 0-cli: Received resp to getwd
Popen: ssh> [2016-11-17 09:33:56.908833] I [input.c:31:cli_batch] 0-: Exiting with: 0
Popen: ssh> [2016-11-17 09:33:56.985793] E [syncdutils:279:log_raise_exception] <top>: FAIL: 
Popen: ssh> Traceback (most recent call last):
Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 203, in main
Popen: ssh>     main_i()
Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 542, in main_i
Popen: ssh>     upgrade_config_file(rconf['config_file'], confdata)
Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/configinterface.py", line 158, in upgrade_config_file
Popen: ssh>     shutil.move(tempConfigFile.name, path)
Popen: ssh>   File "/usr/lib64/python2.6/shutil.py", line 260, in move
Popen: ssh>     copy2(src, real_dst)
Popen: ssh>   File "/usr/lib64/python2.6/shutil.py", line 95, in copy2
Popen: ssh>     copyfile(src, dst)
Popen: ssh>   File "/usr/lib64/python2.6/shutil.py", line 51, in copyfile
Popen: ssh>     with open(dst, 'wb') as fdst:
Popen: ssh> IOError: [Errno 13] Permission denied: '/var/lib/glusterd/geo-replication/gsyncd_template.conf'
Popen: ssh> failed with IOError.
#v-

This seems related to:

https://bugzilla.redhat.com/show_bug.cgi?id=1339683

Involved software versions:

glusterfs-3.8.5-1.el6.x86_64
glusterfs-api-3.8.5-1.el6.x86_64
glusterfs-cli-3.8.5-1.el6.x86_64
glusterfs-client-xlators-3.8.5-1.el6.x86_64
glusterfs-fuse-3.8.5-1.el6.x86_64
glusterfs-geo-replication-3.8.5-1.el6.x86_64
glusterfs-libs-3.8.5-1.el6.x86_64
glusterfs-server-3.8.5-1.el6.x86_64

Steps leading up to this:

1. On all nodes (I probably don't need to setup all this on all nodes,
but it shouldn't hurt, and make it easier if I ever  (I probably don't
need to setup all this on all nodes, but it shouldn't hurt, and make it
easier if I ever want to switch geoip replication direction):

# install packages
yum -q -y install glusterfs glusterfs-fuse glusterfs-server \
  glusterfs-geo-replication
# add geoaccount:geogroup
groupadd -r geogroup
useradd -r -g geogroup -s /bin/bash -m geoaccount
mkdir /home/geoaccount/.ssh
# setup passwordless SSH
cp /vagrant/geoaccount_key.pub /home/geoaccount/.ssh/authorized_keys
chmod -R go-rwx /home/geoaccount/.ssh
chown -R geoaccount. /home/geoaccount/.ssh
mkdir /root/.ssh
cp /vagrant/geoaccount_key /root/.ssh/id_rsa
chmod -R go-rwx /root/.ssh
# add mountbroker directory
mkdir /var/mountbroker-root
chmod 0711 /var/mountbroker-root
# geo-replication directories and permissions
mkdir -p /var/log/glusterfs/geo-replication-slaves /var/lib/glusterd/geo-replication
chgrp -R geogroup /var/log/glusterfs/geo-replication-slaves
chgrp -R geogroup /var/lib/glusterd/geo-replication
chmod -R 770 /var/lib/glusterd/geo-replication
chmod -R 770 /var/log/glusterfs/geo-replication-slaves

2. On the fra-node3 and chi-node3 servers ($area is either "fra" or
"chi"):

gluster peer probe ${area}-node1
gluster peer probe ${area}-node2
gluster volume create ${area}-volume replica $count transport tcp $nodes
gluster volume start ${area}-volume
gluster volume set all cluster.enable-shared-storage enable


3. On the chi-node3 server:

gluster system:: execute mountbroker opt  mountbroker-root /var/mountbroker-root
gluster system:: execute mountbroker user geoaccount ${area}-volume
gluster system:: execute mountbroker opt  geo-replication-log-group geogroup
gluster system:: execute mountbroker opt  rpc-auth-allow-insecure on


4. On all chi-node* servers (since those are replication targets), after
settting up the mount broker:

service glusterd restart


5. On fra-node3:

gluster system:: execute gsec_create
gluster volume geo-replication fra-volume geoaccount at chi-node3::chi-volume create push-pem
gluster volume geo-replication fra-volume geoaccount at chi-node3::chi-volume config use_meta_volume true
gluster volume geo-replication fra-volume geoaccount at chi-node3::chi-volume start


6. On chi-node3:

/usr/libexec/glusterfs/set_geo_rep_pem_keys.sh geoaccount fra-volume chi-volume


Volume an geo-replication status:

$ vagrant ssh fra-node1 -c 'sudo gluster volume status fra-volume' ; \
    vagrant ssh chi-node1 -c 'sudo gluster volume status chi-volume'; \
    vagrant ssh fra-node1 -c 'sudo gluster volume geo-replication status'
Status of volume: fra-volume
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick fra-node1:/data1/glusterfs            49152     0          Y       4499 
Brick fra-node2:/data1/glusterfs            49152     0          Y       4434 
Brick fra-node3:/data1/glusterfs            49152     0          Y       3753 
Self-heal Daemon on localhost               N/A       N/A        Y       5091 
Self-heal Daemon on fra-node3.vagrant.finan
cial.com                                    N/A       N/A        Y       4623 
Self-heal Daemon on fra-node2               N/A       N/A        Y       5024 
 
Task Status of Volume fra-volume
------------------------------------------------------------------------------
There are no active volume tasks
 
Connection to 127.0.0.1 closed.
Status of volume: chi-volume
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick chi-node1:/data1/glusterfs            49152     0          Y       4498 
Brick chi-node2:/data1/glusterfs            49152     0          Y       4435 
Brick chi-node3:/data1/glusterfs            49152     0          Y       3750 
Self-heal Daemon on localhost               N/A       N/A        Y       5756 
Self-heal Daemon on chi-node3.vagrant.finan
cial.com                                    N/A       N/A        Y       5900 
Self-heal Daemon on chi-node2               N/A       N/A        Y       5667 
 
Task Status of Volume chi-volume
------------------------------------------------------------------------------
There are no active volume tasks
 
Connection to 127.0.0.1 closed.
 
MASTER NODE    MASTER VOL    MASTER BRICK        SLAVE USER    SLAVE                                     SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED          
-----------------------------------------------------------------------------------------------------------------------------------------------------------
fra-node1      fra-volume    /data1/glusterfs    geoaccount    ssh://geoaccount@chi-node3::chi-volume    N/A           Faulty    N/A             N/A                  
fra-node3      fra-volume    /data1/glusterfs    geoaccount    ssh://geoaccount@chi-node3::chi-volume    N/A           Faulty    N/A             N/A                  
fra-node2      fra-volume    /data1/glusterfs    geoaccount    ssh://geoaccount@chi-node3::chi-volume    N/A           Faulty    N/A             N/A                  

How can I convince glusterd to create that template with correct permissions?


Cheers,
Stefan


More information about the Gluster-users mailing list