[Bugs] [Bug 1219048] New: Data Tiering:Enabling quota command fails with "quota command failed : Commit failed on localhost"
bugzilla at redhat.com
bugzilla at redhat.com
Wed May 6 13:31:26 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1219048
Bug ID: 1219048
Summary: Data Tiering:Enabling quota command fails with "quota
command failed : Commit failed on localhost"
Product: GlusterFS
Version: 3.7.0
Component: tiering
Severity: urgent
Priority: urgent
Assignee: bugs at gluster.org
Reporter: josferna at redhat.com
QA Contact: bugs at gluster.org
CC: bugs at gluster.org, dlambrig at redhat.com,
nchilaka at redhat.com
Depends On: 1214219
Blocks: 1186580 (qe_tracker_everglades), 1199352
(glusterfs-3.7.0), 1214666
+++ This bug was initially created as a clone of Bug #1214219 +++
Description of problem:
======================
On a fresh 3 node setup, when i tried to enable quota, it failed with following
error:
quota command failed : Commit failed on localhost. Please check the log file
for more details.
Also, the vol info shows quota as enabled on the local node where the command
was executed, but doesnt show the same on other nodes
Note, I was using RHEL 7 with upstream gluster installed.
Logs showed as failed to construct graph
Version-Release number of selected component (if applicable):
=============================================================
[root at zod glusterfs]# gluster --version
glusterfs 3.7dev built on Apr 17 2015 14:27:16
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
[root at zod glusterfs]# rpm -qa|grep gluster
glusterfs-api-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-cli-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-server-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-fuse-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-libs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
[root at zod glusterfs]#
Steps to Reproduce:
==================
1.setup a multinode cluster
2.now create a tier volume
3.try to turn on quota using vol quota <vname> enable
Expected results:
=================
quotas should get enabled for all nodes
Additional info:
==============
[root at zod ~]# gluster v create vol1 replica 2 10.70.35.144:/brick_100G_1/vol1
yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
10.70.35.144:/brick_100G_2/vol1
yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1 force
volume create: vol1: success: please start the volume to access data
[root at zod ~]# gluster v info vol1
Volume Name: vol1
Type: Distributed-Replicate
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.35.144:/brick_100G_1/vol1
Brick2: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick3: 10.70.35.144:/brick_100G_2/vol1
Brick4: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root at zod ~]# gluster v start vol1
volume start: vol1: success
[root at zod ~]# gluster v attach-tier vol1 replica 2
moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
volume add-brick: success
[root at zod ~]# gluster v info vol1
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root at zod ~]# gluster v quota vol1 enable
quota command failed : Commit failed on localhost. Please check the log file
for more details.
[root at zod ~]# gluster v info vol1
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
Options Reconfigured:
features.quota: on
[root at zod ~]# gluster v status vol1
Status of volume: vol1
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_2/vol1 49155 0 Y 18270
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_2/vol1 49153 0 Y 5007
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_1/vol1 49154 0 Y 18249
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_1/vol1 49152 0 Y 4988
Brick 10.70.35.144:/brick_100G_1/vol1 49152 0 Y 32581
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_1/vol1 49152 0 Y 18004
Brick 10.70.35.144:/brick_100G_2/vol1 49153 0 Y 32599
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_2/vol1 49153 0 Y 18023
NFS Server on localhost N/A N/A N N/A
Quota Daemon on localhost N/A N/A N N/A
NFS Server on moonshine.lab.eng.blr.redhat.
com N/A N/A N N/A
NFS Server on yarrow.lab.eng.blr.redhat.com N/A N/A N N/A
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks
[root at zod ~]#
[root at zod ~]# gluster v status vol1
Locking failed on moonshine.lab.eng.blr.redhat.com. Please check log file for
details.
Locking failed on yarrow.lab.eng.blr.redhat.com. Please check log file for
details.
[root at zod ~]# gluster v status vol1
Status of volume: vol1
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_2/vol1 49155 0 Y 18270
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_2/vol1 49153 0 Y 5007
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_1/vol1 49154 0 Y 18249
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_1/vol1 49152 0 Y 4988
Brick 10.70.35.144:/brick_100G_1/vol1 49152 0 Y 32581
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_1/vol1 49152 0 Y 18004
Brick 10.70.35.144:/brick_100G_2/vol1 49153 0 Y 32599
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_2/vol1 49153 0 Y 18023
NFS Server on localhost N/A N/A N N/A
Quota Daemon on localhost N/A N/A N N/A
NFS Server on yarrow.lab.eng.blr.redhat.com N/A N/A N N/A
NFS Server on moonshine.lab.eng.blr.redhat.
com N/A N/A N N/A
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks
=======================
[root at yarrow ~]# gluster peer status
Number of Peers: 2
Hostname: 10.70.35.144
Uuid: d2e5c2f7-1391-4d0b-80aa-35243c5ad286
State: Peer in Cluster (Connected)
Hostname: moonshine.lab.eng.blr.redhat.com
Uuid: a4f5c421-1636-44ed-a7bf-3772deef9346
State: Peer in Cluster (Connected)
[root at yarrow ~]# gluster v info vol1
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root at yarrow ~]#
[root at yarrow ~]# rpm -qa|grep gluster
glusterfs-api-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-cli-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-server-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-fuse-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-libs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
[root at yarrow ~]# gluster v info vol1
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root at yarrow ~]# gluster v status vol1
Locking failed on 10.70.35.144. Please check log file for details.
Locking failed on moonshine.lab.eng.blr.redhat.com. Please check log file for
details.
[root at yarrow ~]# gluster v status vol1
Status of volume: vol1
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_2/vol1 49155 0 Y 18270
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_2/vol1 49153 0 Y 5007
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_1/vol1 49154 0 Y 18249
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_1/vol1 49152 0 Y 4988
Brick 10.70.35.144:/brick_100G_1/vol1 49152 0 Y 32581
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_1/vol1 49152 0 Y 18004
Brick 10.70.35.144:/brick_100G_2/vol1 49153 0 Y 32599
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_2/vol1 49153 0 Y 18023
NFS Server on localhost N/A N/A N N/A
NFS Server on 10.70.35.144 N/A N/A N N/A
Quota Daemon on 10.70.35.144 N/A N/A N N/A
NFS Server on moonshine.lab.eng.blr.redhat.
com N/A N/A N N/A
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks
[root at yarrow ~]# gluster v quota vol1 enable
quota command failed : Commit failed on localhost. Please check the log file
for more details.
[root at yarrow ~]# cd /var/log/glusterfs/
[root at yarrow glusterfs]# ls
bricks etc-glusterfs-glusterd.vol.log quotad.log
cli.log glustershd.log quota-mount-vol1.log
cmd_history.log nfs.log snaps
[root at yarrow glusterfs]# less quota
quotad.log quota-mount-vol1.log
[root at yarrow glusterfs]# less quotad.log
[root at yarrow glusterfs]#
==============================
[root at moonshine ~]# gluster v info vol1
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root at moonshine ~]# gluster v status vol1
Locking failed on 10.70.35.144. Please check log file for details.
Locking failed on yarrow.lab.eng.blr.redhat.com. Please check log file for
details.
[root at moonshine ~]# gluster v status vol1
Status of volume: vol1
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_2/vol1 49155 0 Y 18270
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_2/vol1 49153 0 Y 5007
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_1/vol1 49154 0 Y 18249
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_1/vol1 49152 0 Y 4988
Brick 10.70.35.144:/brick_100G_1/vol1 49152 0 Y 32581
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_1/vol1 49152 0 Y 18004
Brick 10.70.35.144:/brick_100G_2/vol1 49153 0 Y 32599
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_2/vol1 49153 0 Y 18023
NFS Server on localhost N/A N/A N N/A
NFS Server on 10.70.35.144 N/A N/A N N/A
Quota Daemon on 10.70.35.144 N/A N/A N N/A
NFS Server on yarrow.lab.eng.blr.redhat.com N/A N/A N N/A
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks
--- Additional comment from nchilaka on 2015-04-22 05:40:43 EDT ---
logs @ rhsqe-repo.lab.eng.blr.redhat.com:/home/repo/sosreports/1214219/
--- Additional comment from Anand Avati on 2015-04-30 07:39:51 EDT ---
REVIEW: http://review.gluster.org/10474 (glusterd/quota/tiering: Fixing volgen
of quotad) posted (#1) for review on master by Joseph Fernandes
(josferna at redhat.com)
--- Additional comment from Anand Avati on 2015-04-30 08:07:36 EDT ---
REVIEW: http://review.gluster.org/10474 (glusterd/quota/tiering: Fixing volgen
of quotad) posted (#2) for review on master by Joseph Fernandes
(josferna at redhat.com)
--- Additional comment from Anand Avati on 2015-05-01 10:46:20 EDT ---
COMMIT: http://review.gluster.org/10474 committed in master by Kaleb KEITHLEY
(kkeithle at redhat.com)
------
commit cfb9ea4dc68440a18b7f07422901a715b00776f0
Author: Joseph Fernandes <josferna at redhat.com>
Date: Thu Apr 30 16:54:16 2015 +0530
glusterd/quota/tiering: Fixing volgen of quotad
The quotad's graph generation was happening wrongly for
tiered volume. The check is been inserted.
Change-Id: I5554bc5280b0fbaec750e9008fdd930ad53a774f
BUG: 1214219
Signed-off-by: Joseph Fernandes <josferna at redhat.com>
Reviewed-on: http://review.gluster.org/10474
Tested-by: Gluster Build System <jenkins at build.gluster.com>
Reviewed-by: Atin Mukherjee <amukherj at redhat.com>
Reviewed-by: Dan Lambright <dlambrig at redhat.com>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1186580
[Bug 1186580] QE tracker bug for Everglades
https://bugzilla.redhat.com/show_bug.cgi?id=1199352
[Bug 1199352] GlusterFS 3.7.0 tracker
https://bugzilla.redhat.com/show_bug.cgi?id=1214219
[Bug 1214219] Data Tiering:Enabling quota command fails with "quota command
failed : Commit failed on localhost"
https://bugzilla.redhat.com/show_bug.cgi?id=1214666
[Bug 1214666] Data Tiering:command prompt hangs when fetching quota list of
a tiered volume
--
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list