[Gluster-users] After Centos 6 yum update client fails to mount glusterfs volume

[Atin Mukherjee]

Thanks Joe for the correction. However as of now we do not have a
mechanism to let the clients know how should they behave based on the
op-version just like what glusterd does. This is the only implication
why we always encourage to upgrade the servers first followed by the
clients.

~Atin

On 08/14/2015 01:53 AM, Joe Julian wrote:
> "... a client which is running higher version than the server [...]
> glusterd will reject the mount request." is incorrect.  You are correct
> in your understanding that the rpc is supposed to be cross-version
> compatible with newer features not being allowed as long as older
> versions are in the cluster as either servers or clients.
> 
> The problem seems to have come from a policy shift between 3.7.2 and
> 3.7.3 where 3.7.3 uses unprivileged ports by default (there was supposed
> to be an email set to gluster-announce but I can't find that nor hardly
> any other announcements this year).
> 
> To solve this, add "option rpc-auth-allow-insecure on" to
> "/etc/glusterfs/glusterd.vol" and set "server.allow-insecure on" for
> your volumes.
> 
> On 08/13/2015 11:47 AM, Taylor Lewick wrote:
>>
>> Ok, Thank you, I had read online where gluster supported backwards
>> compatibility, with respect to newer client versions working with
>> older server versions.   Thanks for the clarification…
>>
>> *From:*Atin Mukherjee [mailto:atin.mukherjee83 at gmail.com]
>> *Sent:* Thursday, August 13, 2015 10:46 AM
>> *To:* Taylor Lewick
>> *Cc:* gluster-users at gluster.org
>> *Subject:* Re: [Gluster-users] After Centos 6 yum update client fails
>> to mount glusterfs volume
>>
>> -Atin
>> Sent from one plus one
>> On Aug 13, 2015 9:11 PM, "Taylor Lewick" <tlewick at adknowledge.com
>> <mailto:tlewick at adknowledge.com>> wrote:
>> >
>> > As a follow up, I created a test gluster cluster, installed 3.6, and
>> upgraded it to 3.7.  I verified client machines running glusterfs 3.6
>> and 3.7 could mount the volume via glusterfs…
>> >
>> >
>> >
>> > So currently, clients running 3.7 can’t mount a gluster volume via
>> glusterfs if the gluster servers are running 3.6.x.  I didn’t test
>> mounting the volume via NFS.
>> I think I didn't understood your problem earlier. If you are trying to
>> connect a client which is running higher version than the server then
>> its expected as glusterd will reject the mount request. You should
>> first upgrade your servers followed by the clients.
>> >
>> > When it fails it still returns an authentication error.
>> >
>> >
>> >
>> > Not sure if this is a bug or just requires specifically setting some
>> authentication parameters on the volume/cluster.
>> >
>> >
>> >
>> > From: gluster-users-bounces at gluster.org
>> <mailto:gluster-users-bounces at gluster.org>
>> [mailto:gluster-users-bounces at gluster.org
>> <mailto:gluster-users-bounces at gluster.org>] On Behalf Of Taylor Lewick
>> > Sent: Wednesday, August 12, 2015 10:52 AM
>> > To: gluster-users at gluster.org <mailto:gluster-users at gluster.org>
>> > Subject: [Gluster-users] After Centos 6 yum update client fails to
>> mount glusterfs volume
>> >
>> >
>> >
>> > I have an 8 node gluster cluster running 3.6.2. Yesterday one of our
>> admin ran a yum update on his Centos 6.6 machine, and it updated the
>> gluster client to 3.7.3.  (Client update only on his machine, not the
>> gluster cluster servers).
>> >
>> > After the update he was not able to mount the gluster volume.
>> >
>> > He then removed and reinstalled the 3.6 gluster client, and he was
>> able to mount the volume again.
>> >
>> >
>> >
>> > So to test/confirm, I did the same thing.  Prior to the running the
>> upgrade I was running 3.6.4 glusterfs and glusterfs-fuse, and I had
>> the mountpoint mounted on my test server.
>> >
>> > I ran the yum update glusterfs, it completed successfully, and I
>> could access the mount.  I then unmounted the gluster mount, and when
>> I tried to mount it, and I received the following error:
>> >
>> > Mount failed. Please check the log file for more details.
>> >
>> > Since I could access the mountpoint, it appears something has
>> happened in this update to change default behavior from a client mount
>> request.  Any ideas what changes I need to make to be able to mount
>> the volume?
>> >
>> > With 3.7 version of client, do you have to specifically set
>> auth.allow options for volumes?
>> >
>> > Below is the logfile from the client
>> >
>> > [2015-08-12 15:47:10.982346] I [MSGID: 100030]
>> [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running
>> /usr/sbin/glusterfs version 3.7.3 (args: /usr/sbin/glusterfs
>> --volfile-server=infarchive01 --volfile-id=/archives /media)
>> >
>> > [2015-08-12 15:47:11.029757] I [MSGID: 101190]
>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
>> thread with index 1
>> >
>> > [2015-08-12 15:47:11.058367] I [MSGID: 114020]
>> [client.c:2118:notify] 0-archives-client-0: parent translators are
>> ready, attempting connect on transport
>> >
>> > [2015-08-12 15:47:11.058529] I [MSGID: 101190]
>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
>> thread with index 2
>> >
>> > [2015-08-12 15:47:11.065361] I [MSGID: 114020]
>> [client.c:2118:notify] 0-archives-client-1: parent translators are
>> ready, attempting connect on transport
>> >
>> > [2015-08-12 15:47:11.066220] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
>> 0-archives-client-0: changing port to 49152 (from 0)
>> >
>> > [2015-08-12 15:47:11.073949] I [MSGID: 114057]
>> [client-handshake.c:1437:select_server_supported_programs]
>> 0-archives-client-0: Using Program GlusterFS 3.3, Num (1298437),
>> Version (330)
>> >
>> > [2015-08-12 15:47:11.074260] I [MSGID: 114020]
>> [client.c:2118:notify] 0-archives-client-2: parent translators are
>> ready, attempting connect on transport
>> >
>> > [2015-08-12 15:47:11.075504] W [MSGID: 114043]
>> [client-handshake.c:1114:client_setvolume_cbk] 0-archives-client-0:
>> failed to set the volume [Permission denied]
>> >
>> > [2015-08-12 15:47:11.075671] W [MSGID: 114007]
>> [client-handshake.c:1143:client_setvolume_cbk] 0-archives-client-0:
>> failed to get 'process-uuid' from reply dict [Invalid argument]
>> >
>> > [2015-08-12 15:47:11.075761] E [MSGID: 114044]
>> [client-handshake.c:1149:client_setvolume_cbk] 0-archives-client-0:
>> SETVOLUME on remote-host failed [Permission denied]
>> >
>> > [2015-08-12 15:47:11.075801] I [MSGID: 114049]
>> [client-handshake.c:1240:client_setvolume_cbk] 0-archives-client-0:
>> sending AUTH_FAILED event
>> >
>> > [2015-08-12 15:47:11.075840] E [fuse-bridge.c:5130:notify] 0-fuse:
>> Server authenication failed. Shutting down.
>> >
>> > [2015-08-12 15:47:11.075892] I [fuse-bridge.c:5595:fini] 0-fuse:
>> Unmounting '/media'.
>> >
>> > [2015-08-12 15:47:11.082675] I [MSGID: 114020]
>> [client.c:2118:notify] 0-archives-client-3: parent translators are
>> ready, attempting connect on transport
>> >
>> > [2015-08-12 15:47:11.084889] W [glusterfsd.c:1219:cleanup_and_exit]
>> (-->/lib64/libpthread.so.0() [0x3fe5a07a51]
>> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-:
>> received signum (15), shutting down
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
~Atin