[Bugs] [Bug 1224081] New: Detaching tier start failed on dist-rep volume

bugzilla at redhat.com bugzilla at redhat.com
Fri May 22 07:48:47 UTC 2015 

https://bugzilla.redhat.com/show_bug.cgi?id=1224081

            Bug ID: 1224081
           Summary: Detaching tier start failed on dist-rep volume
           Product: Red Hat Gluster Storage
           Version: 3.1
         Component: glusterfs
     Sub Component: tiering
          Keywords: TestBlocker
          Severity: urgent
          Assignee: rhs-bugs at redhat.com
          Reporter: trao at redhat.com
        QA Contact: nchilaka at redhat.com
                CC: bugs at gluster.org, dlambrig at redhat.com,
                    rkavunga at redhat.com, trao at redhat.com
        Depends On: 1221061



+++ This bug was initially created as a clone of Bug #1221061 +++

Description of problem:

Detaching tier start failed on dist-rep volume

Version-Release number of selected component (if applicable):

root at rhsqa14-vm3 ~]# glusterfs --version
glusterfs 3.7.0beta2 built on May 11 2015 01:27:45
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
You have new mail in /var/spool/mail/root
[root at rhsqa14-vm3 ~]# 
[root at rhsqa14-vm3 ~]# 
[root at rhsqa14-vm3 ~]# rpm -qa | grep gluster
glusterfs-libs-3.7.0beta2-0.0.el6.x86_64
glusterfs-fuse-3.7.0beta2-0.0.el6.x86_64
glusterfs-rdma-3.7.0beta2-0.0.el6.x86_64
glusterfs-3.7.0beta2-0.0.el6.x86_64
glusterfs-api-3.7.0beta2-0.0.el6.x86_64
glusterfs-cli-3.7.0beta2-0.0.el6.x86_64
glusterfs-geo-replication-3.7.0beta2-0.0.el6.x86_64
glusterfs-extra-xlators-3.7.0beta2-0.0.el6.x86_64
glusterfs-client-xlators-3.7.0beta2-0.0.el6.x86_64
glusterfs-server-3.7.0beta2-0.0.el6.x86_64
[root at rhsqa14-vm3 ~]#

How reproducible:

easily

Steps to Reproduce:
1. create dist-rep volume, attach tier.
2. detach tier from the volume
3. detaching tier start failed

Actual results:

detach tier start failed

Expected results:

It should not fail.

Additional info:

[root at rhsqa14-vm3 ~]# gluster v info V1

Volume Name: V1
Type: Tier
Volume ID: 99f99d6d-b24d-4cc8-96e0-25444dbf10fd
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 2 = 2
Brick1: 10.70.46.240:/rhs/brick3/t2
Brick2: 10.70.46.243:/rhs/brick3/t2
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.46.243:/rhs/brick1/t2
Brick4: 10.70.46.240:/rhs/brick1/t2
Brick5: 10.70.46.243:/rhs/brick2/t2
Brick6: 10.70.46.240:/rhs/brick2/t2
Options Reconfigured:
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
cluster.min-free-disk: 10
performance.readdir-ahead: on
[root at rhsqa14-vm3 ~]# 


root at rhsqa14-vm3 ~]# gluster v detach-tier V1 start
volume detach-tier start: failed: Commit failed on localhost. Please check the
log file for more details.
[root at rhsqa14-vm3 ~]# 



Log messages:

[2015-05-13 08:46:13.905887] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid
argument)
[2015-05-13 08:46:15.906141] W [socket.c:3059:socket_connect] 0-snapd: Ignore
failed connection attempt on
/var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or
directory)
[2015-05-13 08:46:15.906165] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid
argument)
[2015-05-13 08:46:15.906248] W [socket.c:642:__socket_rwv] 0-snapd: readv on
/var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket failed (Invalid
argument)
[2015-05-13 08:46:16.910658] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid
argument)
[2015-05-13 08:46:17.454556] I
[glusterd-brick-ops.c:770:__glusterd_handle_remove_brick] 0-management:
Received rem brick req
[2015-05-13 08:46:17.489518] I
[glusterd-utils.c:8599:glusterd_generate_and_set_task_id] 0-management:
Generated task-id 955bc0fd-5253-4537-ae0c-8fe1ca9d02ee for key remove-brick-id
[2015-05-13 08:46:18.300221] E [graph.y:153:new_volume] 0-parser: Line 1295:
volume 'tier-dht' defined again
[2015-05-13 08:46:18.315339] W
[glusterd-brick-ops.c:2253:glusterd_op_remove_brick] 0-management: Unable to
reconfigure NFS-Server
[2015-05-13 08:46:18.315381] E [glusterd-syncop.c:1372:gd_commit_op_phase]
0-management: Commit of operation 'Volume Remove brick' failed on localhost
[2015-05-13 08:46:18.911020] W [socket.c:3059:socket_connect] 0-snapd: Ignore
failed connection attempt on
/var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or
directory)
[2015-05-13 08:46:18.913711] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid
argument)
[2015-05-13 08:46:18.914225] W [socket.c:642:__socket_rwv] 0-snapd: readv on
/var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket failed (Invalid
argument)
[2015-05-13 08:46:19.917878] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid
argument)
[2015-05-13 08:46:21.918057] W [socket.c:3059:socket_connect] 0-snapd: Ignore
failed connection attempt on
/var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or
directory)
[2015-05-13 08:46:21.918160] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid
argument)

--- Additional comment from Mohammed Rafi KC on 2015-05-14 02:18:39 EDT ---

I couldn't reproduce the issue when follow the steps given. Can you make sure
this is reproducible ?

--- Additional comment from Triveni Rao on 2015-05-15 06:50:31 EDT ---

i could reproduce this issue on new downstream build:


root at rhsqa14-vm1 ~]# rpm -qa | grep gluster
glusterfs-3.7.0-2.el6rhs.x86_64
glusterfs-cli-3.7.0-2.el6rhs.x86_64
glusterfs-libs-3.7.0-2.el6rhs.x86_64
glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64
glusterfs-api-3.7.0-2.el6rhs.x86_64
glusterfs-server-3.7.0-2.el6rhs.x86_64
glusterfs-fuse-3.7.0-2.el6rhs.x86_64
[root at rhsqa14-vm1 ~]# 
[root at rhsqa14-vm1 ~]# glusterfs --version
glusterfs 3.7.0 built on May 15 2015 01:31:10
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root at rhsqa14-vm1 ~]# 





root at rhsqa14-vm1 ~]# gluster v info vol2

Volume Name: vol2
Type: Tier
Volume ID: 46c79842-2d5d-4f0a-9776-10504fbc93e4
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: 10.70.46.236:/rhs/brick5/v2
Brick2: 10.70.46.233:/rhs/brick5/v2
Brick3: 10.70.46.236:/rhs/brick3/v2
Brick4: 10.70.46.233:/rhs/brick3/v2
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick5: 10.70.46.233:/rhs/brick1/v2
Brick6: 10.70.46.236:/rhs/brick1/v2
Brick7: 10.70.46.233:/rhs/brick2/v2
Brick8: 10.70.46.236:/rhs/brick2/v2
Options Reconfigured:
features.uss: enable
features.inode-quota: on
features.quota: on
cluster.min-free-disk: 10
performance.readdir-ahead: on
[root at rhsqa14-vm1 ~]# 
[root at rhsqa14-vm1 ~]# 
[root at rhsqa14-vm1 ~]# gluster v detach-tier vol2 start
volume detach-tier start: failed: Commit failed on localhost. Please check the
log file for more details.
You have new mail in /var/spool/mail/root
[root at rhsqa14-vm1 ~]# 



[root at rhsqa14-vm1 ~]# 
[root at rhsqa14-vm1 ~]# less /var/log/glusterfs/etc-glusterfs-glusterd.vol.log 
[2015-05-14 06:34:47.596813] I [MSGID: 100030] [glusterfsd.c:2294:main]
0-glusterd: Started running glusterd version 3.7.0beta2 (args: glusterd
--xlator-option *.upgrade=on -N)
[2015-05-14 06:34:47.605211] I [graph.c:269:gf_add_cmdline_options]
0-management: adding option 'upgrade' for volume 'management' with value 'on'
[2015-05-14 06:34:47.605328] I [glusterd.c:1282:init] 0-management: Maximum
allowed open file descriptors set to 65536
[2015-05-14 06:34:47.605370] I [glusterd.c:1327:init] 0-management: Using
/var/lib/glusterd as working directory
[2015-05-14 06:34:47.630137] E [rpc-transport.c:291:rpc_transport_load]
0-rpc-transport: /usr/lib64/glusterfs/3.7.0beta2/rpc-transport/rdma.so: cannot
open shared object file: No such file or directory
[2015-05-14 06:34:47.630198] W [rpc-transport.c:295:rpc_transport_load]
0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid
or not found on this machine
[2015-05-14 06:34:47.630218] W [rpcsvc.c:1595:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2015-05-14 06:34:47.630235] E [glusterd.c:1515:init] 0-management: creation of
1 listeners failed, continuing with succeeded transport
[2015-05-14 06:34:47.649973] I [glusterd.c:413:glusterd_check_gsync_present]
0-glusterd: geo-replication module not installed in the system
[2015-05-14 06:34:47.650135] E [store.c:432:gf_store_handle_retrieve] 0-: Path
corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file
or directory)
[2015-05-14 06:34:47.650161] E [store.c:432:gf_store_handle_retrieve] 0-: Path
corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file
or directory)
[2015-05-14 06:34:47.650173] I
[glusterd-store.c:2005:glusterd_restore_op_version] 0-management: Detected new
install. Setting op-version to maximum : 30700
[2015-05-14 06:34:47.650462] E [store.c:432:gf_store_handle_retrieve] 0-: Path
corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file
or directory)
[2015-05-14 06:34:47.650781] I [glusterd.c:184:glusterd_uuid_generate_save]
0-management: generated UUID: 87acbf29-e821-48bf-9aa8-bbda9321e609
[2015-05-14 06:34:47.817571] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-glustershd: setting frame-timeout to 600
[2015-05-14 06:34:47.818755] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs:
setting frame-timeout to 600
[2015-05-14 06:34:47.819295] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-quotad: setting frame-timeout to 600
[2015-05-14 06:34:47.819873] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-bitd: setting frame-timeout to 600
[2015-05-14 06:34:47.820373] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-scrub: setting frame-timeout to 600
[2015-05-14 06:34:47.820867] I
[glusterd-store.c:3371:glusterd_store_retrieve_missed_snaps_list] 0-management:
No missed snaps list.
[2015-05-14 06:34:47.821075] E [store.c:432:gf_store_handle_retrieve] 0-: Path
corresponding to /var/lib/glusterd/options, returned error: (No such file or
directory)
Final graph:
+------------------------------------------------------------------------------+
...skipping...
gument)
[2015-05-15 10:49:42.993889] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid
argument)
[2015-05-15 10:49:45.994646] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid
argument)
[2015-05-15 10:49:45.994734] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid
argument)
[2015-05-15 10:49:48.995525] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid
argument)
[2015-05-15 10:49:48.995645] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid
argument)
[2015-05-15 10:49:51.996236] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid
argument)
[2015-05-15 10:49:51.996339] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid
argument)
[2015-05-15 10:49:54.997046] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid
argument)
[2015-05-15 10:49:54.997175] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid
argument)
[2015-05-15 10:49:57.997799] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid
argument)
[2015-05-15 10:49:57.997881] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid
argument)
[2015-05-15 10:50:00.998300] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid
argument)
[2015-05-15 10:50:00.998384] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid
argument)
[2015-05-15 10:50:03.999585] W [socket.c:642:__socket_rwv] 0-nfs: readv on
/var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid
argument)
[2015-05-15 10:50:03.999684] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid
argument)

--- Additional comment from Mohammed Rafi KC on 2015-05-19 05:41:17 EDT ---

upstream patch : http://review.gluster.org/10820

--- Additional comment from Mohammed Rafi KC on 2015-05-19 05:47:10 EDT ---

http://review.gluster.org/10820


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1221061
[Bug 1221061] Detaching tier start failed on dist-rep volume
-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=QpTmx47qSz&a=cc_unsubscribe

More information about the Bugs mailing list