[Gluster-users] Linux (ls -l) command pauses/slow on GlusterFS mounts

Fri Aug 12 05:34:46 UTC 2016

Ok I tried kernel.nfs(ie no & its hangs as well. So as you said Vijay, the issue might be with my setup on virtual on the network side.

--
Deepak

-----Original Message-----
From: Deepak Naidu 
Sent: Thursday, August 11, 2016 6:54 PM
To: 'Vijay Bellur'; 'gluster-users at gluster.org'
Subject: RE: [Gluster-users] Linux (ls -l) command pauses/slow on GlusterFS mounts

I tried more things to figure out the issue. Like upgrading NFS-ganesha to the latest version(as the earlier version had some bug regarding crashing), that helped a bit.

But still again the ls -ls or rm -rf files were hanging but not much as earlier. So upgrade of NFS ganesha to stable version did help help a bit.

I did strace again, looks like its pausing/hanging at "lstat" I had to [crtl+c] to get the exact hang/pausing line.

lgetxattr("/mnt/gluster/rand.26.0", "security.selinux", 0x1990a00, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.25.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.25.0", "security.selinux", 0x1990a20, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.24.0", ^C

NOTE: I am running fio to generate some write operation & hangs are seen when issuing ls during write operation.

Next thing, I might try is to use NFS mount rather than Glustefs fuse to see if its related to fuse client.

====strace of ls -l /mnt/gluster/======

munmap(0x7efebec71000, 4096)            = 0
openat(AT_FDCWD, "/mnt/gluster/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, /* 14 entries */, 32768)    = 464
lstat("/mnt/gluster/9e50d562-5846-4a60-ad75-e95dcbe0e38a.vhd", {st_mode=S_IFREG|0644, st_size=19474461184, ...}) = 0 lgetxattr("/mnt/gluster/9e50d562-5846-4a60-ad75-e95dcbe0e38a.vhd", "security.selinux", 0x1990900, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/file1", {st_mode=S_IFREG|0644, st_size=19474461184, ...}) = 0 lgetxattr("/mnt/gluster/file1", "security.selinux", 0x1990940, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.0.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.0.0", "security.selinux", 0x1990940, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.31.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.31.0", "security.selinux", 0x1990960, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.30.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.30.0", "security.selinux", 0x1990980, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.29.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.29.0", "security.selinux", 0x19909a0, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.28.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.28.0", "security.selinux", 0x19909c0, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.27.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.27.0", "security.selinux", 0x19909e0, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.26.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.26.0", "security.selinux", 0x1990a00, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.25.0", {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0 lgetxattr("/mnt/gluster/rand.25.0", "security.selinux", 0x1990a20, 255) = -1 ENODATA (No data available) lstat("/mnt/gluster/rand.24.0", ^C

====strace of end -  ls -l /mnt/gluster/======

-----Original Message-----
From: Deepak Naidu
Sent: Wednesday, August 10, 2016 2:25 PM
To: Vijay Bellur
Cc: gluster-users at gluster.org
Subject: RE: [Gluster-users] Linux (ls -l) command pauses/slow on GlusterFS mounts

To be more precious the hang is clearly seen when there is some IO(write) to the mount point. Even rm -rf takes time to clear the files.

Below, time command showing the delay. Typically it should take less then a second, but glusterfs take more than 5seconds just to list 32x 2GB files.

[root at client-host ~]# time ls -l /mnt/gluster/ total 34575680 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.0.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.1.0 -rw-r--r--. 1 root root 2147454976 Aug 10 12:23 rand.10.0 -rw-r--r--. 1 root root 2147463168 Aug 10 12:23 rand.11.0 -rw-r--r--. 1 root root 2147467264 Aug 10 12:23 rand.12.0 -rw-r--r--. 1 root root 2147475456 Aug 10 12:23 rand.13.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.14.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.15.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.16.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.17.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.18.0 -rw-r--r--. 1 root root 2147467264 Aug 10 12:23 rand.19.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.2.0 -rw-r--r--. 1 root root 2147475456 Aug 10 12:23 rand.20.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.21.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.22.0 -rw-r--r--. 1 root root 2147459072 Aug 10 12:23 rand.23.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.24.0 -rw-r--r--. 1 root root 2147471360 Aug 10 12:23 rand.25.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.26.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.27.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.28.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.29.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.3.0 -rw-r--r--. 1 root root 2147442688 Aug 10 12:23 rand.30.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.31.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.4.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.5.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.6.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.7.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.8.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.9.0

real    0m7.478s
user    0m0.001s
sys     0m0.005s
 [root at client-host ~]#

--
Deepak

-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Deepak Naidu
Sent: Wednesday, August 10, 2016 2:18 PM
To: Vijay Bellur
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Linux (ls -l) command pauses/slow on GlusterFS mounts

* PGP Signed: 08/10/2016 at 02:18:22 PM, Decrypted

I did strace & its waiting on IO.

--
Deepak

-----Original Message-----
From: Vijay Bellur [mailto:vbellur at redhat.com]
Sent: Wednesday, August 10, 2016 2:17 PM
To: Deepak Naidu
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Linux (ls -l) command pauses/slow on GlusterFS mounts

On 08/10/2016 05:12 PM, Deepak Naidu wrote:
> Before we can try physical we wanted POC on VM.
>
> Just a note the VMs are decently powerful 18cpus, 10gig NIC, 45GB Ram 1TB SSD drives. This is per node spec.
>
> I don't see the ls -l command hanging when I try to list the files from the gluster-node VMs itself So the question.

The reason I alluded to a physical setup was to remove the variables that can affect performance in a virtual setup. The behavior is not usual for the scale of deployment that you mention. You could use strace in conjunction with gluster volume profile to determine where the latency is stemming from.

Regards,
Vijay

>
> --
> Deepak
>
>> On Aug 10, 2016, at 2:01 PM, Vijay Bellur <vbellur at redhat.com> wrote:
>>
>>> On 08/10/2016 04:54 PM, Deepak Naidu wrote:
>>> Anyone who has seen the issue in their env ?
>>
>>
>>> --
>>> Deepak
>>>
>>> -----Original Message-----
>>> From: gluster-users-bounces at gluster.org 
>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Deepak Naidu
>>> Sent: Tuesday, August 09, 2016 9:14 PM
>>> To: gluster-users at gluster.org
>>> Subject: [Gluster-users] Linux (ls -l) command pauses/slow on 
>>> GlusterFS mounts
>>>
>>> Greetings,
>>>
>>> I have 3node GlusterFS on VM for POC each node has 2x bricks of 200GB. Regardless of what type of volume I create when listing files under directory using ls command the GlusterFS mount hangs pauses for few seconds. This is same if there're 2-5 19gb file each or 2gb file each. There are less than  10 files under the GlusterFS mount.
>>>
>>> I am using NFS-Ganesha for NFS server with GlusterFS and the Linux client is mounted using GlusterFS fuse mount with direct-io enabled.
>>>
>>> GlusterFS version 3.8(latest)
>>>
>>>
>>> Any insight is appreciated.
>>
>> This does not seem usual for the deployment that you describe. Can you try on a physical setup to see if the same behavior is observed?
>>
>> -Vijay
>>
>>
> ----------------------------------------------------------------------
> ------------- This email message is for the sole use of the intended
> recipient(s) and may contain confidential information.  Any 
> unauthorized review, use, disclosure or distribution is prohibited.
> If you are not the intended recipient, please contact the sender by 
> reply email and destroy all copies of the original message.
> ----------------------------------------------------------------------
> -------------
>

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

* Deepak Naidu <dnaidu at nvidia.com>
* 0x15098040