[Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set
Steve Dainard
sdainard at spd1.com
Mon Mar 7 18:43:27 UTC 2016
Yep, unmount was successful. I didn't see any useful logging to say
otherwise, but I have reduced logging on my clients because the logs grow
way too quickly otherwise.
I had a maintenance window this morning, and worked around this by
unmounting and rebooting the two debian hosts. I was then able to set the
lazy rebalance without restarting any gluster services so I'm ticking along
now.
Maybe the client state info is cached for some period of time after the
client disconnects.
On Fri, Mar 4, 2016 at 11:20 PM, Atin Mukherjee <atin.mukherjee83 at gmail.com>
wrote:
> This is really weird, if you unmount the clients then statedump shouldn't
> be showing the same. Was unmount successful? Do you see any related error
> log entry in mount & glusterd log?
>
> -Atin
> Sent from one plus one
> On 04-Mar-2016 10:23 pm, "Steve Dainard" <sdainard at spd1.com> wrote:
>
>> Except that even when I unmount gluster from both Debian clients, I'm
>> still getting the same error, and a dump of those clients after unmounting
>> still shows:
>>
>> glusterd.client1.identifier=10.0.231.10:1023
>> glusterd.client1.volname=storage
>> glusterd.client1.max-op-version=30603
>> glusterd.client1.min-op-version=1
>>
>> glusterd.client4.identifier=10.0.231.11:1022
>> glusterd.client4.volname=storage
>> glusterd.client4.max-op-version=30603
>> glusterd.client4.min-op-version=1
>>
>>
>>
>>
>>
>> On Thu, Mar 3, 2016 at 5:28 PM, Atin Mukherjee <
>> atin.mukherjee83 at gmail.com> wrote:
>>
>>> -Atin
>>> Sent from one plus one
>>> On 04-Mar-2016 3:35 am, "Steve Dainard" <sdainard at spd1.com> wrote:
>>> >
>>> > FYI Gluster storage node hostnames are gluster0[1-6].
>>> >
>>> > Full dump attached. I see a few clients not on 30706. Most notably the
>>> two debian 7 servers (using packages from gluster.org) seem to be
>>> running lower op versions than the centos7 machines (every other client in
>>> 10.0.231.0/24 subnet).
>>> >
>>> > glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs
>>> 3.7.6 built on Feb 4 2016 06:25:19
>>> > glusterd.client1.volname=storage
>>> > glusterd.client1.max-op-version=30603
>>> This is running with 3.6.3 and that's why volume set fails. Unmount this
>>> client and upgrade and remount it back.
>>>
>>> > glusterd.client1.min-op-version=1
>>> >
>>> > glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02
>>> (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
>>> > glusterd.client2.volname=
>>> > glusterd.client2.max-op-version=0
>>> > glusterd.client2.min-op-version=0
>>> >
>>> > glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no
>>> actual mounts)
>>> > glusterd.client3.volname=
>>> > glusterd.client3.max-op-version=0
>>> > glusterd.client3.min-op-version=0
>>> >
>>> > glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs
>>> 3.7.6 built on Feb 4 2016 06:25:19
>>> > glusterd.client4.volname=storage
>>> > glusterd.client4.max-op-version=30603
>>> > glusterd.client4.min-op-version=1
>>> >
>>> > glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no
>>> actual mounts)
>>> > glusterd.client5.volname=
>>> > glusterd.client5.max-op-version=0
>>> > glusterd.client5.min-op-version=0
>>> >
>>> > glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04
>>> (mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
>>> > glusterd.client6.volname=
>>> > glusterd.client6.max-op-version=0
>>> > glusterd.client6.min-op-version=0
>>> >
>>> > glusterd.client7.identifier=10.0.231.50:65529
>>> >
>>> glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage
>>> > glusterd.client7.max-op-version=30706
>>> > glusterd.client7.min-op-version=1
>>> >
>>> > ...
>>> >
>>> > Debian package info:
>>> > apt-cache policy glusterfs-client
>>> > glusterfs-client:
>>> > Installed: 3.7.6-2
>>> > Candidate: 3.7.6-2
>>> > Version table:
>>> > *** 3.7.6-2 0
>>> > 500
>>> http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/
>>> wheezy/main amd64 Packages
>>> >
>>> >
>>> > Thanks
>>> >
>>> > On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg <ggarg at redhat.com> wrote:
>>> >>
>>> >> Hi Steve,
>>> >>
>>> >> As atin pointed out to take statedump by running #kill -SIGUSR1
>>> $(pidof glusterd) command. it will create .dump file in /var/run/gluster/
>>> directory. client-op-version information will be present in dump file.
>>> >>
>>> >> Thanks,
>>> >> ~Gaurav
>>> >>
>>> >> ----- Original Message -----
>>> >> From: "Steve Dainard" <sdainard at spd1.com>
>>> >> To: "Gaurav Garg" <ggarg at redhat.com>
>>> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>
>>> >> Sent: Thursday, March 3, 2016 12:07:25 AM
>>> >> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or
>>> more connected clients cannot support the feature being set
>>> >>
>>> >> From the the client side logs I can see version info on mount:
>>> >>
>>> >> Final graph:
>>> >>
>>> +------------------------------------------------------------------------------+
>>> >> 1: volume storage-client-0
>>> >> 2: type protocol/client
>>> >> 3: option clnt-lk-version 1
>>> >> 4: option volfile-checksum 0
>>> >> 5: option volfile-key /storage
>>> >> 6: option client-version 3.7.6
>>> >> 7: option process-uuid
>>> >>
>>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
>>> >> 8: option fops-version 1298437
>>> >> 9: option ping-timeout 42
>>> >> 10: option remote-host 10.0.231.50
>>> >> 11: option remote-subvolume /mnt/raid6-storage/storage
>>> >> 12: option transport-type socket
>>> >> 13: option send-gids true
>>> >> 14: end-volume
>>> >> 15:
>>> >> 16: volume storage-client-1
>>> >> 17: type protocol/client
>>> >> 18: option clnt-lk-version 1
>>> >> 19: option volfile-checksum 0
>>> >> 20: option volfile-key /storage
>>> >> 21: option client-version 3.7.6
>>> >> 22: option process-uuid
>>> >>
>>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
>>> >> 23: option fops-version 1298437
>>> >> 24: option ping-timeout 42
>>> >> 25: option remote-host 10.0.231.51
>>> >> 26: option remote-subvolume /mnt/raid6-storage/storage
>>> >> 27: option transport-type socket
>>> >> 28: option send-gids true
>>> >> 29: end-volume
>>> >> 30:
>>> >> 31: volume storage-client-2
>>> >> 32: type protocol/client
>>> >> 33: option clnt-lk-version 1
>>> >> 34: option volfile-checksum 0
>>> >> 35: option volfile-key /storage
>>> >> 36: option client-version 3.7.6
>>> >> 37: option process-uuid
>>> >>
>>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
>>> >> 38: option fops-version 1298437
>>> >> 39: option ping-timeout 42
>>> >> 40: option remote-host 10.0.231.52
>>> >> 41: option remote-subvolume /mnt/raid6-storage/storage
>>> >> 42: option transport-type socket
>>> >> 43: option send-gids true
>>> >> 44: end-volume
>>> >> 45:
>>> >> 46: volume storage-client-3
>>> >> 47: type protocol/client
>>> >> 48: option clnt-lk-version 1
>>> >> 49: option volfile-checksum 0
>>> >> 50: option volfile-key /storage
>>> >> 51: option client-version 3.7.6
>>> >> 52: option process-uuid
>>> >>
>>> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-3-0-0
>>> >> 53: option fops-version 1298437
>>> >> 54: option ping-timeout 42
>>> >> 55: option remote-host 10.0.231.53
>>> >> 56: option remote-subvolume /mnt/raid6-storage/storage
>>> >> 57: option transport-type socket
>>> >> 58: option send-gids true
>>> >> 59: end-volume
>>> >> 60:
>>> >> 61: volume storage-client-4
>>> >> 62: type protocol/client
>>> >> 63: option ping-timeout 42
>>> >> 64: option remote-host 10.0.231.54
>>> >> 65: option remote-subvolume /mnt/raid6-storage/storage
>>> >> 66: option transport-type socket
>>> >> 67: option send-gids true
>>> >> 68: end-volume
>>> >> 69:
>>> >> 70: volume storage-client-5
>>> >> 71: type protocol/client
>>> >> 72: option ping-timeout 42
>>> >> 73: option remote-host 10.0.231.55
>>> >> 74: option remote-subvolume /mnt/raid6-storage/storage
>>> >> 75: option transport-type socket
>>> >> 76: option send-gids true
>>> >> 77: end-volume
>>> >> 78:
>>> >> 79: volume storage-dht
>>> >> 80: type cluster/distribute
>>> >> 81: subvolumes storage-client-0 storage-client-1 storage-client-2
>>> >> storage-client-3 storage-client-4 storage-client-5
>>> >> 82: end-volume
>>> >>
>>> >>
>>> >> But not the client op-version, how can I retrieve this info?
>>> >>
>>> >> Thanks
>>> >>
>>> >> On Tue, Mar 1, 2016 at 10:19 PM, Gaurav Garg <ggarg at redhat.com>
>>> wrote:
>>> >>
>>> >> > Hi Steve,
>>> >> >
>>> >> > Which version you have upgraded client, could you tell us client
>>> >> > op-version after upgrade ?
>>> >> >
>>> >> >
>>> >> > have you upgraded all of your clients ?
>>> >> >
>>> >> >
>>> >> > Thanks,
>>> >> > Gaurav
>>> >> >
>>> >> >
>>> >> > ----- Original Message -----
>>> >> > From: "Steve Dainard" <sdainard at spd1.com>
>>> >> > To: "gluster-users at gluster.org List" <gluster-users at gluster.org>
>>> >> > Sent: Wednesday, March 2, 2016 1:10:27 AM
>>> >> > Subject: [Gluster-users] gluster 3.7.6 volume set: failed: One or
>>> more
>>> >> > connected clients cannot support the feature being set
>>> >> >
>>> >> > Gluster 3.7.6
>>> >> > 'storage' is a distributed volume
>>> >> >
>>> >> > # gluster volume set storage rebal-throttle lazy
>>> >> > volume set: failed: One or more connected clients cannot support the
>>> >> > feature being set. These clients need to be upgraded or
>>> disconnected before
>>> >> > running this command again
>>> >> >
>>> >> > I found a client connected using version 3.6.7 so I upgraded &
>>> >> > umount/mount the gluster volume on the client but I'm still getting
>>> this
>>> >> > error.
>>> >> >
>>> >> > I've run grep "accepted client from" /var/log/glusterfs/bricks/* |
>>> grep -v
>>> >> > 3.7.6 and I get a few returns from the client above, all dated last
>>> week.
>>> >> >
>>> >> > I've run 'gluster volume status storage clients' and checked the
>>> connected
>>> >> > clients manually, they're all running 3.7.6.
>>> >> >
>>> >> > /var/log/gluster/etc-glusterfs-glusterd.vol.log:
>>> >> > [2016-03-01 19:23:20.180821] E [MSGID: 106022]
>>> >> > [glusterd-utils.c:10154:glusterd_check_client_op_version_support]
>>> >> > 0-management: One or more c
>>> >> > lients don't support the required op-version
>>> >> > [2016-03-01 19:23:20.180853] E [MSGID: 106301]
>>> >> > [glusterd-syncop.c:1274:gd_stage_op_phase] 0-management: Staging of
>>> >> > operation 'Volume Set' fa
>>> >> > iled on localhost : One or more connected clients cannot support the
>>> >> > feature being set. These clients need to be upgraded or
>>> disconnected be
>>> >> > fore running this command again
>>> >> >
>>> >> > Also tried setting the diagnostics.brick-log-level logging level
>>> and got
>>> >> > the same error.
>>> >> >
>>> >> > /var/lib/glusterd/vols/storage/info:
>>> >> > type=0
>>> >> > count=6
>>> >> > status=1
>>> >> > sub_count=0
>>> >> > stripe_count=1
>>> >> > replica_count=1
>>> >> > disperse_count=0
>>> >> > redundancy_count=0
>>> >> > version=26
>>> >> > transport-type=0
>>> >> > volume-id=26d355cb-c486-481f-ac16-e25390e73775
>>> >> > username=eb9e2063-6ba8-4d16-a54f-2c7cf7740c4c
>>> >> > password=
>>> >> > op-version=3
>>> >> > client-op-version=3
>>> >> > quota-version=1
>>> >> > parent_volname=N/A
>>> >> > restored_from_snap=00000000-0000-0000-0000-000000000000
>>> >> > snap-max-hard-limit=256
>>> >> > features.quota-deem-statfs=on
>>> >> > features.inode-quota=on
>>> >> > diagnostics.brick-log-level=WARNING
>>> >> > features.quota=on
>>> >> > performance.readdir-ahead=on
>>> >> > performance.cache-size=1GB
>>> >> > performance.stat-prefetch=on
>>> >> > brick-0=10.0.231.50:-mnt-raid6-storage-storage
>>> >> > brick-1=10.0.231.51:-mnt-raid6-storage-storage
>>> >> > brick-2=10.0.231.52:-mnt-raid6-storage-storage
>>> >> > brick-3=10.0.231.53:-mnt-raid6-storage-storage
>>> >> > brick-4=10.0.231.54:-mnt-raid6-storage-storage
>>> >> > brick-5=10.0.231.55:-mnt-raid6-storage-storage
>>> >> >
>>> >> >
>>> >> > _______________________________________________
>>> >> > Gluster-users mailing list
>>> >> > Gluster-users at gluster.org
>>> >> > http://www.gluster.org/mailman/listinfo/gluster-users
>>> >> >
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Gluster-users mailing list
>>> > Gluster-users at gluster.org
>>> > http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160307/0a180115/attachment.html>
More information about the Gluster-users
mailing list