[Gluster-users] Pretty much any operation related to Gluster mounted fs hangs for a while

[A Ghoshal]

 I am plagued with something of this sort, too!

What I mostly see when I explore these things is that

A) it's a split-brain. 
B) the split-brain is because the gfid's on the two replicas are at odds. 

You could check that out by 
1. On each server, first 'cd' to where your brick is mounted.
2. getfattr -m . -d -e hex templates/assets/prod/temporary/13/user_1339200.png

You will see a trusted.gfid kind of extended attribute. If it's not the same on both servers, there's a problem.

Thanks, 
Anirban

 -----Tiago Santos <tiago at musthavemenus.com> wrote: -----

 =======================
 To: gluster-users at gluster.org
 From: Tiago Santos <tiago at musthavemenus.com>
 Date: 01/26/2015 09:38PM 
 Subject: [Gluster-users] Pretty much any operation related to Gluster	mounted fs hangs for a while
 =======================
   Hey guys,

I'm experiencing this weird case for pretty much any command (ls, df, find,
etc) I try to run against a Gluster client filesystem.


Just for you guys to understand what I'm talking about, follows this easy
and simple test I just ran:


root at web3:~# date; time ls -ltrh
/var/www/site-images/templates/assets/prod/temporary/13/user_1339200.png
Mon Jan 26 07:00:27 PST 2015
-rwx---r-- 1 mhmadmin mhmadmin 61K Jan 22 14:37
/var/www/site-images/templates/assets/prod/temporary/13/user_1339200.png

real 0m*33.651s*
user 0m0.001s
sys 0m0.004s
root at web3:~# date; time ls -ltrh
/var/www/site-images/templates/assets/prod/temporary/13/user_1339200.png
Mon Jan 26 07:01:03 PST 2015
ls: cannot access
/var/www/site-images/templates/assets/prod/temporary/13/user_1339200.png:
*Input/output
error*

real *1m40.241s*
user 0m0.000s
sys 0m0.003s
root at web3:~# date; time ls -ltrh
/var/www/site-images/templates/assets/prod/temporary/13/user_1339200.png
Mon Jan 26 07:02:51 PST 2015
ls: cannot access
/var/www/site-images/templates/assets/prod/temporary/13/user_1339200.png:
*Input/output
error*

real *0m12.834s*
user 0m0.000s
sys 0m0.003s
root at web3:~# date; time ls -ltrh
/var/www/site-images/templates/assets/prod/temporary/13/user_1339200.png
Mon Jan 26 07:03:10 PST 2015
-rwx---r-- 1 mhmadmin mhmadmin 61K Jan 22 14:37
/var/www/site-images/templates/assets/prod/temporary/13/user_1339200.png

real *2m10.150s*
user 0m0.000s
sys 0m0.005s


Sometimes it passes, but takes a really long time to run a simple command
(this is a 61K file), sometimes I see the Input/output error. The important
thing to mention is that this behavior happens almost all the time. I can
quickly reproduce it if asked.


This is a 2-node gluster setup. Both VMs act as Client and Server (sorry if
I'm not using the correct gluster naming.. I'm getting to know it for weeks
now).


More info:

# gluster --version
glusterfs 3.5.3 built on Nov 18 2014 03:53:25
Repository revision: git://git.gluster.com/glusterfs.git

# df -Th
Filesystem                      Type            Size  Used Avail Use%
Mounted on
/dev/mapper/data_vg-data_lv     ext4           1007G  506G  451G  53%
/export/images1-1
images1.mydomain.com:/site-images fuse.glusterfs 1007G  506G  451G  53%
/var/www/site-images

# uname -a
Linux web3 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux

# gluster volume info

Volume Name: site-images
Type: Replicate
Volume ID: 68bca3c9-210c-45a9-b2bc-6a0e2ee630bb
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: images1.mydomain.com:/export/images1-1/brick
Brick2: images2.mydomain.com:/export/images2-1/brick



Would anyone help me identify what is going on here?


Thanks in advance!

-- 
*Tiago Santos*
MustHaveMenus.com
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users    
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you