[Bugs] [Bug 1214289] I/O failure on attaching tier

bugzilla at redhat.com bugzilla at redhat.com
Tue Jun 2 15:56:53 UTC 2015


https://bugzilla.redhat.com/show_bug.cgi?id=1214289



--- Comment #10 from nchilaka <nchilaka at redhat.com> ---
Seeing the following issue on latest downstream build 
Following are the steps to reproduce:
1)create a dist-rep volume 
  gluster v create tiervol2 replica 2 10.70.46.233:/rhs/brick1/tiervol2
10.70.46.236:/rhs/brick1/tiervol2 10.70.46.240:/rhs/brick1/tiervol2
10.70.46.243:/rhs/brick1  /tiervol2
2)start and issue commands like info and status
3)Now mount using NFS
4) Trigger some IOs on this volume
5)While IOs are happening attach a tier

It can be seen that the tier gets attached successfully, but the IOs fail to
write anymore

Some Observations worth noting:
1)This happens only when we mount using NFS. With glusterfs mount works
well(Anoop, comment if you see issue even on glusterfs mount)
2)Seems to be some problem with tiering and NFS interaction as I see that NFS
ports are all down when I run above scenario
3)This issue is hit only when IOs were in progress while attaching
tier(although this will be the most valid case in customer site)


[root at rhsqa14-vm1 ~]# gluster v status tiervol2
Status of volume: tiervol2
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.46.233:/rhs/brick1/tiervol2     49153     0          Y       1973 
Brick 10.70.46.236:/rhs/brick1/tiervol2     49154     0          Y       24453
Brick 10.70.46.240:/rhs/brick1/tiervol2     49154     0          Y       32272
Brick 10.70.46.243:/rhs/brick1/tiervol2     49153     0          Y       31759
NFS Server on localhost                     2049      0          Y       1992 
Self-heal Daemon on localhost               N/A       N/A        Y       2017 
NFS Server on 10.70.46.243                  2049      0          Y       31778
Self-heal Daemon on 10.70.46.243            N/A       N/A        Y       31790
NFS Server on 10.70.46.236                  2049      0          Y       24472
Self-heal Daemon on 10.70.46.236            N/A       N/A        Y       24482
NFS Server on 10.70.46.240                  2049      0          Y       32292
Self-heal Daemon on 10.70.46.240            N/A       N/A        Y       32312

Task Status of Volume tiervol2
------------------------------------------------------------------------------
There are no active volume tasks

[root at rhsqa14-vm1 ~]# gluster v info tiervol2

Volume Name: tiervol2
Type: Distributed-Replicate
Volume ID: a98f39c2-03ed-4ec7-909f-573b89a2a3e8
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.46.233:/rhs/brick1/tiervol2
Brick2: 10.70.46.236:/rhs/brick1/tiervol2
Brick3: 10.70.46.240:/rhs/brick1/tiervol2
Brick4: 10.70.46.243:/rhs/brick1/tiervol2
Options Reconfigured:
performance.readdir-ahead: on
[root at rhsqa14-vm1 ~]# #################Now i have mounted the regular dist-rep
vol  https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.0.4.tar.xz
You have new mail in /var/spool/mail/root
[root at rhsqa14-vm1 ~]# #################Now i have mounted the regular dist-rep
vol  tiervol2##########
[root at rhsqa14-vm1 ~]# ls /rhs/brick1/tiervol2
linux-4.0.4.tar.xz
[root at rhsqa14-vm1 ~]#  #################Next I will attach a tier while
untaring the image, and will check status of vol, it will show nfs
down###########
[root at rhsqa14-vm1 ~]# ls /rhs/brick1/tiervol2 ;gluster v attach-tier tiervol2
10.70.46.236:/rhs/brick2/tiervol2 10.70.46.240:/rhs/brick2/tiervol2
linux-4.0.4  linux-4.0.4.tar.xz
Attach tier is recommended only for testing purposes in this release. Do you
want to continue? (y/n) y
volume attach-tier: success
volume rebalance: tiervol2: success: Rebalance on tiervol2 has been started
successfully. Use rebalance status command to check status of the rebalance
process.
ID: 1e59a5cc-2ff0-48ce-a34e-0521cbe65d73

You have mail in /var/spool/mail/root
[root at rhsqa14-vm1 ~]# ls /rhs/brick1/tiervol2
linux-4.0.4  linux-4.0.4.tar.xz
[root at rhsqa14-vm1 ~]# gluster v info tiervol2

Volume Name: tiervol2
Type: Tier
Volume ID: a98f39c2-03ed-4ec7-909f-573b89a2a3e8
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: 10.70.46.240:/rhs/brick2/tiervol2
Brick2: 10.70.46.236:/rhs/brick2/tiervol2
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.46.233:/rhs/brick1/tiervol2
Brick4: 10.70.46.236:/rhs/brick1/tiervol2
Brick5: 10.70.46.240:/rhs/brick1/tiervol2
Brick6: 10.70.46.243:/rhs/brick1/tiervol2
Options Reconfigured:
performance.readdir-ahead: on
[root at rhsqa14-vm1 ~]# gluster v status tiervol2
Status of volume: tiervol2
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.46.240:/rhs/brick2/tiervol2     49155     0          Y       32411
Brick 10.70.46.236:/rhs/brick2/tiervol2     49155     0          Y       24590
Brick 10.70.46.233:/rhs/brick1/tiervol2     49153     0          Y       1973 
Brick 10.70.46.236:/rhs/brick1/tiervol2     49154     0          Y       24453
Brick 10.70.46.240:/rhs/brick1/tiervol2     49154     0          Y       32272
Brick 10.70.46.243:/rhs/brick1/tiervol2     49153     0          Y       31759
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on 10.70.46.236                  N/A       N/A        N       N/A  
NFS Server on 10.70.46.243                  N/A       N/A        N       N/A  
NFS Server on 10.70.46.240                  N/A       N/A        N       N/A  

Task Status of Volume tiervol2
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : 1e59a5cc-2ff0-48ce-a34e-0521cbe65d73
Status               : in progress         




sosreport Logs attached

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.


More information about the Bugs mailing list