[Gluster-users] WG: Strange issu concerning glusterfs 3.5.1 on centos 6.5

Daniel Müller mueller at tropenklinik.de
Thu Jul 31 07:10:58 UTC 2014


Hello and thank you so far,
What I have recognized is, having more than one nic running this is confusing glusterfs 3.5. I never
saw this on my glusterfs 3.4 and 3.2 systems still working.
So I set up just clean erased gluster with yum glusterfs* erase  and did:
Logged in to my both nodes in the 135 subnet,ex:
Ssh 192.168.135.36 (centclust1)  (172.17.2.30 is the 2nd nic)
Ssh 192.168.135.46 (centclust2)  (172.17.2.31 is the 2nd nic)
Started gluster on both nodes , service glusterd start.
Did the peer probe on 192.168.135.36/centclust1:
Gluster peer probe 192.168.135.46 //Former I did gluster peer probe centclust2 
This result in:
[root at centclust1 ~]# gluster peer status
Number of Peers: 1

Hostname: 192.168.135.46
Uuid: c395c15d-5187-4e5b-b680-57afcb88b881
State: Peer in Cluster (Connected)

[root at centclust2 backup]# gluster peer status
Number of Peers: 1

Hostname: 192.168.135.36
Uuid: 94d5903b-ebe9-40d6-93bf-c2f2e92909a0
State: Peer in Cluster (Connected)
The signifent difference gluster now shows the ip of both nodes

Now I did the create the replicating vol:
gluster volume create smbcluster replica 2 transport tcp  192.168.135.36:/sbu/glusterfs/export  192.168.135.46:/sbu/glusterfs/export
started the volume
gluster volume status
Status of volume: smbcluster
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 192.168.135.36:/sbu/glusterfs/export              49152   Y       27421
Brick 192.168.135.46:/sbu/glusterfs/export              49152   Y       12186
NFS Server on localhost                                 2049    Y       27435
Self-heal Daemon on localhost                           N/A     Y       27439
NFS Server on 192.168.135.46                            2049    Y       12200
Self-heal Daemon on 192.168.135.46                      N/A     Y       12204

Task Status of Volume smbcluster
------------------------------------------------------------------------------
There are no active volume tasks

Mounted the volumes:

Centclust1:mount -t glusterfs 192.168.135.36:/smbcluster /mntgluster -o acl
Centclust2:mount -t glusterfs 192.168.135.46:/smbcluster /mntgluster -o acl

And BINGO up and running!!!!!!!


EDV Daniel Müller

Leitung EDV
Tropenklinik Paul-Lechler-Krankenhaus
Paul-Lechler-Str. 24
72076 Tübingen 
Tel.: 07071/206-463, Fax: 07071/206-499
eMail: mueller at tropenklinik.de
Internet: www.tropenklinik.de




-----Ursprüngliche Nachricht-----
Von: Krishnan Parthasarathi [mailto:kparthas at redhat.com] 
Gesendet: Mittwoch, 30. Juli 2014 16:52
An: mueller at tropenklinik.de
Cc: gluster-devel-bounces at gluster.org; gluster-users at gluster.org
Betreff: Re: [Gluster-users] WG: Strange issu concerning glusterfs 3.5.1 on centos 6.5

Daniel,

I didn't get a chance to follow up with
debugging this issue. I will look into this and get back to you. I suspect that there is something different about the network layer behaviour in your setup.

~KP

----- Original Message -----
> Just another other test:
> [root at centclust1 sicherung]# getfattr -d -e hex -m . /sicherung/bu
> getfattr: Entferne führenden '/' von absoluten Pfadnamen # file: 
> sicherung/bu 
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c6
> 55f743a733000 
> trusted.afr.smbbackup-client-0=0x000000000000000000000000
> trusted.afr.smbbackup-client-1=0x000000000000000200000001
> trusted.gfid=0x00000000000000000000000000000001
> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
> trusted.glusterfs.volume-id=0x6f51d002e634437db58d9b952693f1df
> 
> [root at centclust2 glusterfs]# getfattr -d -e hex -m . /sicherung/bu
> getfattr: Entferne führenden '/' von absoluten Pfadnamen # file: 
> sicherung/bu 
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c6
> 55f743a733000
> trusted.afr.smbbackup-client-0=0x000000000000000200000001
> trusted.afr.smbbackup-client-1=0x000000000000000000000000
> trusted.gfid=0x00000000000000000000000000000001
> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
> trusted.glusterfs.volume-id=0x6f51d002e634437db58d9b952693f1df
> 
> Is this ok?
> 
> After long testing and doing a /etc/init.d/network restart the 
> replication started once/a short time then ended up!?
> Any idea???????
> 
> 
> EDV Daniel Müller
> 
> Leitung EDV
> Tropenklinik Paul-Lechler-Krankenhaus
> Paul-Lechler-Str. 24
> 72076 Tübingen
> Tel.: 07071/206-463, Fax: 07071/206-499
> eMail: mueller at tropenklinik.de
> Internet: www.tropenklinik.de
> 
> "Der Mensch ist die Medizin des Menschen"
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Krishnan Parthasarathi [mailto:kparthas at redhat.com]
> Gesendet: Mittwoch, 30. Juli 2014 11:09
> An: mueller at tropenklinik.de
> Cc: gluster-devel-bounces at gluster.org; gluster-users at gluster.org
> Betreff: Re: [Gluster-users] WG: Strange issu concerning glusterfs 
> 3.5.1 on centos 6.5
> 
> Could you provide the output of the following command?
> 
> netstat -ntap | grep gluster
> 
> This should tell us if glusterfsd processes (bricks) are listening on 
> all interfaces.
> 
> ~KP
> 
> ----- Original Message -----
> > Just one idea
> > I add a second NIC with a 172.2.17... adress on both machines.
> > Could this cause the trouble!?
> > 
> > EDV Daniel Müller
> > 
> > Leitung EDV
> > Tropenklinik Paul-Lechler-Krankenhaus Paul-Lechler-Str. 24
> > 72076 Tübingen
> > Tel.: 07071/206-463, Fax: 07071/206-499
> > eMail: mueller at tropenklinik.de
> > Internet: www.tropenklinik.de
> > 
> > 
> > 
> > 
> > -----Ursprüngliche Nachricht-----
> > Von: Krishnan Parthasarathi [mailto:kparthas at redhat.com]
> > Gesendet: Mittwoch, 30. Juli 2014 09:29
> > An: mueller at tropenklinik.de
> > Cc: gluster-devel-bounces at gluster.org; gluster-users at gluster.org
> > Betreff: Re: [Gluster-users] WG: Strange issu concerning glusterfs
> > 3.5.1 on centos 6.5
> > 
> > Daniel,
> > 
> > From a quick look, I see that glustershd and the nfs client is 
> > unable to connect to one of the bricks. This is resulting in data 
> > from mounts being written to local bricks only.
> > I should have asked this before, could you provide the bricks logs as well?
> > 
> > Could you also try to connect to the bricks using telnet?
> > For eg, from centclust1, telnet centclust2 <brick-port>.
> > 
> > ~KP
> > 
> > ----- Original Message -----
> > > So my logs. I disable ssl meanwhile but it is the same situation. 
> > > No replication!?
> > > 
> > > 
> > > 
> > > EDV Daniel Müller
> > > 
> > > Leitung EDV
> > > Tropenklinik Paul-Lechler-Krankenhaus Paul-Lechler-Str. 24
> > > 72076 Tübingen
> > > Tel.: 07071/206-463, Fax: 07071/206-499
> > > eMail: mueller at tropenklinik.de
> > > Internet: www.tropenklinik.de
> > > 
> > > 
> > > 
> > > 
> > > 
> > > -----Ursprüngliche Nachricht-----
> > > Von: Krishnan Parthasarathi [mailto:kparthas at redhat.com]
> > > Gesendet: Mittwoch, 30. Juli 2014 08:56
> > > An: mueller at tropenklinik.de
> > > Cc: gluster-users at gluster.org; gluster-devel-bounces at gluster.org
> > > Betreff: Re: [Gluster-users] WG: Strange issu concerning glusterfs
> > > 3.5.1 on centos 6.5
> > > 
> > > Could you attach the entire mount and glustershd log files to this 
> > > thread?
> > > 
> > > ~KP
> > > 
> > > ----- Original Message -----
> > > > NO ONE!??
> > > > This is an entry of my glustershd.log:
> > > > [2014-07-30 06:40:59.294334] W
> > > > [client-handshake.c:1846:client_dump_version_cbk] 0-smbbackup-client-1:
> > > > received RPC status error
> > > > [2014-07-30 06:40:59.294352] I [client.c:2229:client_rpc_notify]
> > > > 0-smbbackup-client-1: disconnected from 172.17.2.31:49152. 
> > > > Client process will keep trying to connect to glusterd until 
> > > > brick's port is available
> > > > 
> > > > 
> > > > This is from mnt-sicherung.log:
> > > > [2014-07-30 06:40:38.259850] E [socket.c:2820:socket_connect]
> > > > 1-smbbackup-client-0: connection attempt on 172.17.2.30:24007 
> > > > failed, (Connection timed out) [2014-07-30 06:40:41.275120] I 
> > > > [rpc-clnt.c:1729:rpc_clnt_reconfig]
> > > > 1-smbbackup-client-0: changing port to 49152 (from 0)
> > > > 
> > > > [root at centclust1 sicherung]# gluster --remote-host=centclust1 
> > > > peer status Number of Peers: 1
> > > > 
> > > > Hostname: centclust2
> > > > Uuid: 4f15e9bd-9b5a-435b-83d2-4ed202c66b11
> > > > State: Peer in Cluster (Connected)
> > > > 
> > > > [root at centclust1 sicherung]# gluster --remote-host=centclust2 
> > > > peer status Number of Peers: 1
> > > > 
> > > > Hostname: 172.17.2.30
> > > > Uuid: 99fe6a2c-df7e-4475-a7bc-a35abba620fb
> > > > State: Peer in Cluster (Connected)
> > > > 
> > > > [root at centclust1 ssl]# ps aux | grep gluster
> > > > root     13655  0.0  0.0 413848 16872 ?        Ssl  08:10   0:00
> > > > /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid
> > > > root     13958  0.0  0.0 12139920 44812 ?      Ssl  08:11   0:00
> > > > /usr/sbin/glusterfsd -s centclust1.tplk.loc --volfile-id 
> > > > smbbackup.centclust1.tplk.loc.sicherung-bu -p 
> > > > /var/lib/glusterd/vols/smbbackup/run/centclust1.tplk.loc-sicherung-bu.
> > > > pid -S /var/run/4c65260e12e2d3a9a5549446f491f383.socket
> > > > --brick-name /sicherung/bu -l
> > > > /var/log/glusterfs/bricks/sicherung-bu.log
> > > > --xlator-option
> > > > *-posix.glusterd-uuid=99fe6a2c-df7e-4475-a7bc-a35abba620fb
> > > > --brick-port
> > > > 49152 --xlator-option smbbackup-server.listen-port=49152
> > > > root     13972  0.0  0.0 815748 58252 ?        Ssl  08:11   0:00
> > > > /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p 
> > > > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log 
> > > > -S /var/run/ee6f37fc79b9cb1968eca387930b39fb.socket
> > > > root     13976  0.0  0.0 831160 29492 ?        Ssl  08:11   0:00
> > > > /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd 
> > > > -p /var/lib/glusterd/glustershd/run/glustershd.pid -l 
> > > > /var/log/glusterfs/glustershd.log -S 
> > > > /var/run/aa970d146eb23ba7124e6c4511879850.socket --xlator-option 
> > > > *replicate*.node-uuid=99fe6a2c-df7e-4475-a7bc-a35abba620fb
> > > > root     15781  0.0  0.0 105308   932 pts/1    S+   08:47   0:00 grep
> > > > gluster
> > > > root     29283  0.0  0.0 451116 56812 ?        Ssl  Jul29   0:21
> > > > /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p 
> > > > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log 
> > > > -S /var/run/a7fcb1d1d3a769d28df80b85ae5d13c4.socket
> > > > root     29287  0.0  0.0 335432 25848 ?        Ssl  Jul29   0:21
> > > > /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd 
> > > > -p /var/lib/glusterd/glustershd/run/glustershd.pid -l 
> > > > /var/log/glusterfs/glustershd.log -S 
> > > > /var/run/833e60f976365c2a307f92fb233942a2.socket --xlator-option 
> > > > *replicate*.node-uuid=64b1a7eb-2df3-47bd-9379-39c29e5a001a
> > > > root     31698  0.0  0.0 1438392 57952 ?       Ssl  Jul29   0:12
> > > > /usr/sbin/glusterfs --acl --volfile-server=centclust1.tplk.loc
> > > > --volfile-id=/smbbackup /mnt/sicherung
> > > > 
> > > > [root at centclust2 glusterfs]#  ps aux | grep gluster
> > > > root      1561  0.0  0.0 1481492 60152 ?       Ssl  Jul29   0:12
> > > > /usr/sbin/glusterfs --acl --volfile-server=centclust2.tplk.loc
> > > > --volfile-id=/smbbackup /mnt/sicherung
> > > > root     15656  0.0  0.0 413848 16832 ?        Ssl  08:11   0:01
> > > > /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid
> > > > root     15942  0.0  0.0 12508704 43860 ?      Ssl  08:11   0:00
> > > > /usr/sbin/glusterfsd -s centclust2.tplk.loc --volfile-id 
> > > > smbbackup.centclust2.tplk.loc.sicherung-bu -p 
> > > > /var/lib/glusterd/vols/smbbackup/run/centclust2.tplk.loc-sicherung-bu.
> > > > pid -S /var/run/40a554af3860eddd5794b524576d0520.socket
> > > > --brick-name /sicherung/bu -l
> > > > /var/log/glusterfs/bricks/sicherung-bu.log
> > > > --xlator-option
> > > > *-posix.glusterd-uuid=4f15e9bd-9b5a-435b-83d2-4ed202c66b11
> > > > --brick-port
> > > > 49152 --xlator-option smbbackup-server.listen-port=49152
> > > > root     15956  0.0  0.0 825992 57496 ?        Ssl  08:11   0:00
> > > > /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p 
> > > > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log 
> > > > -S /var/run/602d1d8ba7b80ded2b70305ed7417cf5.socket
> > > > root     15960  0.0  0.0 841404 26760 ?        Ssl  08:11   0:00
> > > > /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd 
> > > > -p /var/lib/glusterd/glustershd/run/glustershd.pid -l 
> > > > /var/log/glusterfs/glustershd.log -S 
> > > > /var/run/504d01c7f7df8b8306951cc2aaeaf52c.socket --xlator-option
> > > > *replicate*.node-uuid=4f15e9bd-9b5a-435b-83d2-4ed202c66b11
> > > > root     17728  0.0  0.0 105312   936 pts/0    S+   08:48   0:00 grep
> > > > gluster
> > > > root     32363  0.0  0.0 451100 55584 ?        Ssl  Jul29   0:21
> > > > /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p 
> > > > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log 
> > > > -S /var/run/73054288d1cadfb87b4b9827bd205c7b.socket
> > > > root     32370  0.0  0.0 335432 26220 ?        Ssl  Jul29   0:21
> > > > /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd 
> > > > -p /var/lib/glusterd/glustershd/run/glustershd.pid -l 
> > > > /var/log/glusterfs/glustershd.log -S 
> > > > /var/run/de1427ce373c792c76c38b12c106f029.socket --xlator-option
> > > > *replicate*.node-uuid=83e6d78c-0119-4537-8922-b3e731718864
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Leitung EDV
> > > > Tropenklinik Paul-Lechler-Krankenhaus Paul-Lechler-Str. 24
> > > > 72076 Tübingen
> > > > Tel.: 07071/206-463, Fax: 07071/206-499
> > > > eMail: mueller at tropenklinik.de
> > > > Internet: www.tropenklinik.de
> > > > 
> > > > 
> > > > 
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Daniel Müller [mailto:mueller at tropenklinik.de]
> > > > Gesendet: Dienstag, 29. Juli 2014 16:02
> > > > An: 'gluster-users at gluster.org'
> > > > Betreff: Strange issu concerning glusterfs 3.5.1 on centos 6.5
> > > > 
> > > > Dear all,
> > > > 
> > > > there is a strange issue centos6.5 and glusterfs 3.5.1:
> > > > 
> > > >  glusterd -V
> > > > glusterfs 3.5.1 built on Jun 24 2014 15:09:41 Repository revision:
> > > > git://git.gluster.com/glusterfs.git
> > > > Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> 
> > > > GlusterFS comes with ABSOLUTELY NO WARRANTY.
> > > > It is licensed to you under your choice of the GNU Lesser 
> > > > General Public License, version 3 or any later version (LGPLv3 
> > > > or later), or the GNU General Public License, version 2 (GPLv2), 
> > > > in all cases as published by the Free Software Foundation
> > > > 
> > > > I try to set up a replicated 2 brick vol on two centos 6.5 server.
> > > > I can probe well and my nodes are reporting no errors:
> > > >  
> > > > [root at centclust1 mnt]# gluster peer status Number of Peers: 1
> > > > 
> > > > Hostname: centclust2
> > > > Uuid: 4f15e9bd-9b5a-435b-83d2-4ed202c66b11
> > > > State: Peer in Cluster (Connected)
> > > > 
> > > > [root at centclust2 sicherung]# gluster peer status Number of Peers:
> > > > 1
> > > > 
> > > > Hostname: 172.17.2.30
> > > > Uuid: 99fe6a2c-df7e-4475-a7bc-a35abba620fb
> > > > State: Peer in Cluster (Connected)
> > > > 
> > > > Now I set up a replicating VOl on an XFS-Disk: /dev/sdb1 on 
> > > > /sicherung type xfs (rw)
> > > > 
> > > > gluster volume create smbbackup replica 2 transport tcp 
> > > > centclust1.tplk.loc:/sicherung/bu 
> > > > centclust2.tplk.loc:/sicherung/bu
> > > > 
> > > > gluster volume smbbackup status reports ok:
> > > > 
> > > > [root at centclust1 mnt]# gluster volume status smbbackup Status of
> > > > volume: smbbackup
> > > > Gluster process                                         Port    Online
> > > > Pid
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > --
> > > > ------
> > > > --
> > > > Brick centclust1.tplk.loc:/sicherung/bu                 49152   Y
> > > > 31969
> > > > Brick centclust2.tplk.loc:/sicherung/bu                 49152   Y
> > > > 2124
> > > > NFS Server on localhost                                 2049    Y
> > > > 31983
> > > > Self-heal Daemon on localhost                           N/A     Y
> > > > 31987
> > > > NFS Server on centclust2                                2049    Y
> > > > 2138
> > > > Self-heal Daemon on centclust2                          N/A     Y
> > > > 2142
> > > > 
> > > > Task Status of Volume smbbackup
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > --
> > > > ------
> > > > --
> > > > There are no active volume tasks
> > > > 
> > > > [root at centclust2 sicherung]# gluster volume status smbbackup 
> > > > Status of
> > > > volume: smbbackup
> > > > Gluster process                                         Port    Online
> > > > Pid
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > --
> > > > ------
> > > > --
> > > > Brick centclust1.tplk.loc:/sicherung/bu                 49152   Y
> > > > 31969
> > > > Brick centclust2.tplk.loc:/sicherung/bu                 49152   Y
> > > > 2124
> > > > NFS Server on localhost                                 2049    Y
> > > > 2138
> > > > Self-heal Daemon on localhost                           N/A     Y
> > > > 2142
> > > > NFS Server on 172.17.2.30                               2049    Y
> > > > 31983
> > > > Self-heal Daemon on 172.17.2.30                         N/A     Y
> > > > 31987
> > > > 
> > > > Task Status of Volume smbbackup
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > --
> > > > ------
> > > > --
> > > > There are no active volume tasks
> > > > 
> > > > I mounted the vol on both servers with:
> > > > 
> > > > mount -t glusterfs centclust1.tplk.loc:/smbbackup  
> > > > /mnt/sicherung -o acl mount -t glusterfs 
> > > > centclust2.tplk.loc:/smbbackup /mnt/sicherung -o acl
> > > > 
> > > > But when I write in /mnt/sicherung the files are not replicated 
> > > > to the other node in anyway!??
> > > > 
> > > > They rest on the local server in /mnt/sicherung and 
> > > > /sicherung/bu On each node separate:#
> > > > [root at centclust1 sicherung]# pwd /mnt/sicherung
> > > > 
> > > > [root at centclust1 sicherung]# touch test.txt
> > > > [root at centclust1 sicherung]# ls
> > > > test.txt
> > > > [root at centclust2 sicherung]# pwd /mnt/sicherung
> > > > [root at centclust2 sicherung]# ls
> > > > more.txt
> > > > [root at centclust1 sicherung]# ls -la /sicherung/bu insgesamt 0 
> > > > drwxr-xr-x.  3 root root  38 29. Jul 15:56 .
> > > > drwxr-xr-x.  3 root root  15 29. Jul 14:31 ..
> > > > drw-------. 15 root root 142 29. Jul 15:56 .glusterfs
> > > > -rw-r--r--.  2 root root   0 29. Jul 15:56 test.txt
> > > > [root at centclust2 sicherung]# ls -la /sicherung/bu insgesamt 0 
> > > > drwxr-xr-x. 3 root root 38 29. Jul 15:32 .
> > > > drwxr-xr-x. 3 root root 15 29. Jul 14:31 ..
> > > > drw-------. 7 root root 70 29. Jul 15:32 .glusterfs -rw-r--r--. 
> > > > 2 root root  0 29. Jul 15:32 more.txt
> > > > 
> > > > 
> > > > 
> > > > Greetings
> > > > Daniel
> > > > 
> > > > 
> > > > 
> > > > EDV Daniel Müller
> > > > 
> > > > Leitung EDV
> > > > Tropenklinik Paul-Lechler-Krankenhaus Paul-Lechler-Str. 24
> > > > 72076 Tübingen
> > > > Tel.: 07071/206-463, Fax: 07071/206-499
> > > > eMail: mueller at tropenklinik.de
> > > > Internet: www.tropenklinik.de
> > > > 
> > > > 
> > > > 
> > > > 
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > > > 
> > > 
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > 
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users




More information about the Gluster-users mailing list