[Bugs] [Bug 1224075] New: Directories are missing on the mount point after attaching tier to distribute replicate volume.

bugzilla at redhat.com bugzilla at redhat.com
Fri May 22 07:42:01 UTC 2015 

https://bugzilla.redhat.com/show_bug.cgi?id=1224075

            Bug ID: 1224075
           Summary: Directories are missing on the mount point after
                    attaching tier to distribute replicate volume.
           Product: Red Hat Gluster Storage
           Version: 3.1
         Component: glusterfs
     Sub Component: tiering
          Keywords: TestBlocker, Triaged
          Severity: urgent
          Assignee: rhs-bugs at redhat.com
          Reporter: trao at redhat.com
        QA Contact: nchilaka at redhat.com
                CC: annair at redhat.com, bugs at gluster.org,
                    dlambrig at redhat.com, josferna at redhat.com,
                    nchilaka at redhat.com, rkavunga at redhat.com
        Depends On: 1214222
            Blocks: 1186580 (qe_tracker_everglades), 1214666, 1219848



+++ This bug was initially created as a clone of Bug #1214222 +++

Description of problem:
Directories are missing on the mount point after attaching tier to distribute
replicate volume.

Version-Release number of selected component (if applicable):

[root at rhsqa14-vm1 ~]# rpm -qa | grep gluster
glusterfs-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-devel-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-geo-replication-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-resource-agents-3.7dev-0.952.gita7f1d08.el6.noarch
glusterfs-debuginfo-3.7dev-0.952.gita7f1d08.el6.x86_64
glusterfs-libs-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-api-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-fuse-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-extra-xlators-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-regression-tests-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-rdma-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-cli-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-server-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-api-devel-3.7dev-0.994.gitf522001.el6.x86_64
[root at rhsqa14-vm1 ~]# 

[root at rhsqa14-vm1 ~]# glusterfs --version
glusterfs 3.7dev built on Apr 13 2015 07:14:26
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root at rhsqa14-vm1 ~]# 


How reproducible:

easy

Steps to Reproduce:
1. create a distrep volume
2. fuse mount the volume and create few directories with files.
3. ls -la on mount and keep the record of output.
4. Attach a tier to the volume and execute ls -la on mount point.
5. directories will be missing.

Actual results:


Volume Name: testing
Type: Distributed-Replicate
Volume ID: 42ac4aff-461e-4001-b1c0-f4d42e04452f
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.46.233:/rhs/brick1/T4
Brick2: 10.70.46.236:/rhs/brick1/T4
Brick3: 10.70.46.233:/rhs/brick2/T4
Brick4: 10.70.46.236:/rhs/brick2/T4


Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhsqa14vm5-lv_root
                       18G  3.2G   14G  20% /
tmpfs                 3.8G     0  3.8G   0% /dev/shm
/dev/vda1             477M   33M  419M   8% /boot
10.70.46.233:/testing
                      100G  244M  100G   1% /mnt
10.70.46.233:/mix     199G  330M  199G   1% /mnt1
10.70.46.233:/everglades
                       20G  5.2M   20G   1% /mnt2
10.70.46.233:/Tim      20G  3.3M   20G   1% /mnt3
[root at rhsqa14-vm5 mnt2]# 
[root at rhsqa14-vm5 mnt2]# 
[root at rhsqa14-vm5 mnt2]# 
[root at rhsqa14-vm5 mnt2]# cd /mnt
[root at rhsqa14-vm5 mnt]# ls -la
total 4
drwxr-xr-x.  5 root root  110 Apr 17 02:54 .
dr-xr-xr-x. 28 root root 4096 Apr 22 01:50 ..
drwxr-xr-x.  6 root root  140 Apr 16 06:14 linux-4.0
drwxr-xr-x.  3 root root   48 Apr 16 02:52 .trashcan
[root at rhsqa14-vm5 mnt]# 


[root at rhsqa14-vm1 ~]# gluster v attach-tier testing replica 2
10.70.46.233:/rhs/brick3/mko 10.70.46.236:/rhs/brick3/mko
volume add-brick: success
[root at rhsqa14-vm1 ~]# gluster v info testing

Volume Name: testing
Type: Tier
Volume ID: 42ac4aff-461e-4001-b1c0-f4d42e04452f
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.46.236:/rhs/brick3/mko
Brick2: 10.70.46.233:/rhs/brick3/mko
Brick3: 10.70.46.233:/rhs/brick1/T4
Brick4: 10.70.46.236:/rhs/brick1/T4
Brick5: 10.70.46.233:/rhs/brick2/T4
Brick6: 10.70.46.236:/rhs/brick2/T4
[root at rhsqa14-vm1 ~]# 


[root at rhsqa14-vm5 mnt]# 
[root at rhsqa14-vm5 mnt]# ls -la
total 4
drwxr-xr-x.  5 root root  149 Apr 22  2015 .
dr-xr-xr-x. 28 root root 4096 Apr 22 01:50 ..
drwxr-xr-x.  3 root root   48 Apr 16 02:52 .trashcan
[root at rhsqa14-vm5 mnt]# 

Expected results:
Irrespective of the tiers data must be presented to user.

Additional info:

--- Additional comment from Dan Lambright on 2015-04-22 16:06:20 EDT ---

The problem here, is you did not start the migration daemon.

gluster v rebalance t tier start

This does the "fix layout" to make all directories on all bricks.

You should not have to worry about that. It should be done automatically when
you attach a tier.

We will write a fix for that.

--- Additional comment from Anand Avati on 2015-04-24 06:43:07 EDT ---

REVIEW: http://review.gluster.org/10363 (tiering: Send both attach-tier and
tier-start together) posted (#1) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Anand Avati on 2015-04-24 06:44:54 EDT ---

REVIEW: http://review.gluster.org/10363 (tiering: Send both attach-tier and
tier-start together) posted (#2) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Anand Avati on 2015-04-28 02:44:17 EDT ---

REVIEW: http://review.gluster.org/10363 (tiering: Send both attach-tier and
tier-start together) posted (#3) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Joseph Elwin Fernandes on 2015-05-01 02:57:28 EDT
---



--- Additional comment from Anand Avati on 2015-05-04 07:49:28 EDT ---

REVIEW: http://review.gluster.org/10363 (tiering: Send both attach-tier and
tier-start together) posted (#4) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Anand Avati on 2015-05-04 11:30:28 EDT ---

REVIEW: http://review.gluster.org/10363 (tiering: Send both attach-tier and
tier-start together) posted (#5) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Anand Avati on 2015-05-05 01:17:28 EDT ---

REVIEW: http://review.gluster.org/10363 (tiering: Send both attach-tier and
tier-start together) posted (#6) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Anoop on 2015-05-13 08:40:36 EDT ---

Reproduced this ont the BETA2 build too, hence moving it to ASSIGNED.

--- Additional comment from Mohammed Rafi KC on 2015-05-14 02:21:18 EDT ---



--- Additional comment from Mohammed Rafi KC on 2015-05-14 02:25:23 EDT ---

I couldn't reproduce this using glusterfs-3.7-bet2. Can you paste the output of
attach-tier in your set up.

--- Additional comment from Triveni Rao on 2015-05-15 06:44:46 EDT ---



I could reproduce the same problem with new downstream build.

root at rhsqa14-vm1 ~]# rpm -qa | grep gluster
glusterfs-3.7.0-2.el6rhs.x86_64
glusterfs-cli-3.7.0-2.el6rhs.x86_64
glusterfs-libs-3.7.0-2.el6rhs.x86_64
glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64
glusterfs-api-3.7.0-2.el6rhs.x86_64
glusterfs-server-3.7.0-2.el6rhs.x86_64
glusterfs-fuse-3.7.0-2.el6rhs.x86_64
[root at rhsqa14-vm1 ~]# 
[root at rhsqa14-vm1 ~]# glusterfs --version
glusterfs 3.7.0 built on May 15 2015 01:31:10
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root at rhsqa14-vm1 ~]# 




[root at rhsqa14-vm1 ~]# gluster v create vol2 replica 2 
10.70.46.233:/rhs/brick1/v2 10.70.46.236:/rhs/brick1/v2
10.70.46.233:/rhs/brick2/v2  10.70.46.236:/rhs/brick2/v2
volume create: vol2: success: please start the volume to access data
You have new mail in /var/spool/mail/root
[root at rhsqa14-vm1 ~]# gluster v start vol2
volume start: vol2: success
[root at rhsqa14-vm1 ~]# gluster v info vol2

Volume Name: vol2
Type: Distributed-Replicate
Volume ID: 46c79842-2d5d-4f0a-9776-10504fbc93e4
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.46.233:/rhs/brick1/v2
Brick2: 10.70.46.236:/rhs/brick1/v2
Brick3: 10.70.46.233:/rhs/brick2/v2
Brick4: 10.70.46.236:/rhs/brick2/v2
Options Reconfigured:
performance.readdir-ahead: on
[root at rhsqa14-vm1 ~]# 


oot at rhsqa14-vm1 ~]# gluster v attach-tier vol2 replica 2
10.70.46.233:/rhs/brick3/v2 10.70.46.236:/rhs/brick3/v2
10.70.46.233:/rhs/brick5/v2 10.70.46.236:/rhs/brick5/v2
Attach tier is recommended only for testing purposes in this release. Do you
want to continue? (y/n) y
volume attach-tier: success
volume rebalance: vol2: success: Rebalance on vol2 has been started
successfully. Use rebalance status command to check status of the rebalance
process.
ID: 72408f67-06c1-4b2a-b4e3-01ffcb0d8b17

You have new mail in /var/spool/mail/root
[root at rhsqa14-vm1 ~]# 




root at rhsqa14-vm5 ~]# mount -t glusterfs 10.70.46.233:vol2 /mnt2
[root at rhsqa14-vm5 ~]# cd /vol2
-bash: cd: /vol2: No such file or directory
[root at rhsqa14-vm5 ~]# cd /mnt2
[root at rhsqa14-vm5 mnt2]# ls -la
total 4
drwxr-xr-x.  4 root root   78 May 15 06:30 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
drwxr-xr-x.  3 root root   48 May 15 06:30 .trashcan
[root at rhsqa14-vm5 mnt2]# 
[root at rhsqa14-vm5 mnt2]# mkdir triveni
[root at rhsqa14-vm5 mnt2]# ls -la
total 4
drwxr-xr-x.  5 root root  106 May 15  2015 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
drwxr-xr-x.  3 root root   48 May 15 06:30 .trashcan
drwxr-xr-x.  2 root root   12 May 15  2015 triveni
[root at rhsqa14-vm5 mnt2]# cp -r /root/linux-4.0 .
^C
[root at rhsqa14-vm5 mnt2]# ls -la
total 4
drwxr-xr-x.  6 root root  138 May 15  2015 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
drwxr-xr-x.  6 root root  140 May 15  2015 linux-4.0
drwxr-xr-x.  3 root root   48 May 15 06:30 .trashcan
drwxr-xr-x.  2 root root   12 May 15  2015 triveni
[root at rhsqa14-vm5 mnt2]# cd linux-4.0/
[root at rhsqa14-vm5 linux-4.0]# ls -la
total 35
drwxr-xr-x.  6 root root   140 May 15  2015 .
drwxr-xr-x.  6 root root   138 May 15  2015 ..
drwxr-xr-x.  4 root root    78 May 15  2015 arch
-rw-r--r--.  1 root root 18693 May 15  2015 COPYING
-rw-r--r--.  1 root root   252 May 15  2015 Kconfig
drwxr-xr-x.  9 root root   350 May 15  2015 security
drwxr-xr-x. 22 root root   557 May 15  2015 sound
drwxr-xr-x. 19 root root 16384 May 15  2015 tools
[root at rhsqa14-vm5 linux-4.0]# cd ..
[root at rhsqa14-vm5 mnt2]# 
[root at rhsqa14-vm5 mnt2]# 
[root at rhsqa14-vm5 mnt2]# ls -la
total 4
drwxr-xr-x.  4 root root  216 May 15  2015 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
drwxr-xr-x.  3 root root   96 May 15  2015 .trashcan
[root at rhsqa14-vm5 mnt2]# touch f1
[root at rhsqa14-vm5 mnt2]#

root at rhsqa14-vm5 mnt2]# touch f2
[root at rhsqa14-vm5 mnt2]# ls -la
total 4
drwxr-xr-x.  4 root root  234 May 15  2015 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
-rw-r--r--.  1 root root    0 May 15 06:36 f1
-rw-r--r--.  1 root root    0 May 15 06:36 f2
drwxr-xr-x.  3 root root   96 May 15  2015 .trashcan
[root at rhsqa14-vm5 mnt2]# 


root at rhsqa14-vm1 ~]# gluster v info vol2

Volume Name: vol2
Type: Tier
Volume ID: 46c79842-2d5d-4f0a-9776-10504fbc93e4
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: 10.70.46.236:/rhs/brick5/v2
Brick2: 10.70.46.233:/rhs/brick5/v2
Brick3: 10.70.46.236:/rhs/brick3/v2
Brick4: 10.70.46.233:/rhs/brick3/v2
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick5: 10.70.46.233:/rhs/brick1/v2
Brick6: 10.70.46.236:/rhs/brick1/v2
Brick7: 10.70.46.233:/rhs/brick2/v2
Brick8: 10.70.46.236:/rhs/brick2/v2
Options Reconfigured:
features.uss: enable
features.inode-quota: on
features.quota: on
cluster.min-free-disk: 10
performance.readdir-ahead: on


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1186580
[Bug 1186580] QE tracker bug for Everglades
https://bugzilla.redhat.com/show_bug.cgi?id=1214222
[Bug 1214222] Directories are missing on the mount point after attaching
tier to distribute replicate volume.
https://bugzilla.redhat.com/show_bug.cgi?id=1214666
[Bug 1214666] Data Tiering:command prompt hangs when fetching quota list of
a tiered volume
https://bugzilla.redhat.com/show_bug.cgi?id=1219848
[Bug 1219848] Directories are missing on the mount point after attaching
tier to distribute replicate volume.
-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=Yr6cxnCRSk&a=cc_unsubscribe

More information about the Bugs mailing list