[Gluster-devel] LOOKUP conflict => OPEN fails_

Wed Apr 14 08:51:17 UTC 2010

Hello,

I think I'm getting similar problem at 3.0.3. My configuration is 2 distributions and replication for each distribution. (2x2)
And I turned off all the performance options on both client and server to check if caching cause this issue.

My test were creating and deleting a same files through TCP client.
Creating and deleting itself worked fine, as the file was created and deleted on a distributed volume and its replica correctly.
But even when the file existed and readdir said the file was there, I was not able to access the file(got NOENT) once every two trials.

[2010-04-14 03:28:12] D [libglusterfsclient.c:2932:libgf_client_opendir] libglusterfsclient: opendir: path /t/test1201/m, status 0, errno 117
[2010-04-14 03:28:12] D [libglusterfsclient.c:4394:glusterfs_readdir] libglusterfsclient: offset 0
[2010-04-14 03:28:12] D [libglusterfsclient.c:4412:glusterfs_readdir] libglusterfsclient: new offset 8, entry .
[2010-04-14 03:28:12] D [libglusterfsclient.c:4394:glusterfs_readdir] libglusterfsclient: offset 8
[2010-04-14 03:28:12] D [libglusterfsclient.c:4412:glusterfs_readdir] libglusterfsclient: new offset 12, entry ..
[2010-04-14 03:28:12] D [libglusterfsclient.c:4394:glusterfs_readdir] libglusterfsclient: offset 12
[2010-04-14 03:28:12] D [libglusterfsclient.c:4412:glusterfs_readdir] libglusterfsclient: new offset 1024, entry myentunnel-unicode.zip
[2010-04-14 03:28:12] D [libglusterfsclient.c:4394:glusterfs_readdir] libglusterfsclient: offset 1024
[2010-04-14 03:28:12] D [libglusterfsclient.c:4400:glusterfs_readdir] libglusterfsclient: readdir failed: Success
[2010-04-14 03:28:12] D [libglusterfsclient.c:4971:glusterfs_stat] libglusterfsclient: path /hs2/t/test1201/m/myentunnel-unicode.zip
[2010-04-14 03:28:12] D [libglusterfsclient.c:1643:_libgf_vmp_search_entry] libglusterfsclient: VMP Search: path /hs2/t/test1201/m/myentunnel-unicode.zip, type: LongestPrefix
[2010-04-14 03:28:12] D [libglusterfsclient.c:1730:libgf_vmp_search_entry] libglusterfsclient: VMP Entry found: path :/hs2/t/test1201/m/m entunnel-unicode.zip vmp: /hs2/
[2010-04-14 03:28:12] D [libglusterfsclient.c:898:libgf_resolve_path_light] libglusterfsclient: Path: /t/test1201/m/myentunnel-unicode.zip, Resolved Path: /t/test1201/m/myentunnel-unicode.zip
[2010-04-14 03:28:12] D [libglusterfsclient.c:4892:__glusterfs_stat] libglusterfsclient: path /t/test1201/m/myentunnel-unicode.zip, op: 2
[2010-04-14 03:28:12] D [libglusterfsclient-dentry.c:390:libgf_client_path_lookup] libglusterfsclient: resolved path(/t/test1201/m/myentunnel-unicode.zip) to 1466437/1462232
[2010-04-14 03:28:12] D [dht-common.c:976:dht_attr_cbk] distribute: subvolume replicate01 returned -1 (No such file or directory)
[2010-04-14 03:28:12] D [libglusterfsclient.c:4822:libgf_client_stat] libglusterfsclient: path /t/test1201/m/myentunnel-unicode.zip, status -1, errno 2

Thanks

DongMin Yu

HOSTWAY IDC Corp. / R&D Principal Researcher
TEL. +822 2105 6037
FAX. +822 2105 6019
CELL. +8216 2086 1357
EMAIL: min.yu at hostwaycorp.com<mailto:min.yu at hostwaycorp.com>
Website: http://www.hostway.com

NOTICE:  This email and any file transmitted are confidential and/or legally privileged and intended only for the person(s) directly addressed.  If you are not the intended recipient, any use, copying, transmission, distribution, or other forms of dissemination is strictly prohibited.  If you have received this email in error, please notify the sender immediately and permanently delete the email and files, if any.
From: gluster-devel-bounces+min=hostway.co.kr at nongnu.org [mailto:gluster-devel-bounces+min=hostway.co.kr at nongnu.org] On Behalf Of Raghavendra G
Sent: Thursday, February 11, 2010 2:49 AM
To: =SMTP:fredrik.widlund at qbrick.com
Cc: =SMTP:gluster-devel at nongnu.org
Subject: Re: [Gluster-devel] LOOKUP conflict => OPEN fails_

Sorry, No need of logfiles. I saw this mail first and then other mails, hence the confusion :).
On Wed, Feb 10, 2010 at 8:56 PM, Raghavendra G <raghavendra at gluster.com<mailto:raghavendra at gluster.com>> wrote:
Hi Fredrik,

Can you please send complete log files of client and server?

regards,
On Mon, Feb 8, 2010 at 6:02 PM, Fredrik Widlund <fredrik.widlund at qbrick.com<mailto:fredrik.widlund at qbrick.com>> wrote:

Hi,

I'm running a simple AFR setup, thouch currently with only one backend, and 2 tcp clients. Version is 3.0.0 from jan 20.

Basically one client is writing a large number of files, continuously, and the other client is reading.

I have a growing problem with lookup "conflicts", resulting in files being listed in directories but where reads are returning "-1 (No such file...".

Restarting the client does not solve the conflict, but restarting the server does and the files becomes available again.

The filesystem is a 5TB XFS hw raid-5 with around 150k files.

Debug trace of client:
[2010-02-08 13:39:29] N [trace.c:148:trace_open_cbk] replicated: 3073: (op_ret=0, op_errno=117, *fd=0x129a430)
[2010-02-08 13:39:37] N [trace.c:1837:trace_open] replicated: 3094: (loc {path=/download/90910/live/webb1/webb1/Layer3/prog_index.m3u8, ino=5042185}, flags=32768, fd=0x1296fc0, wbflags=0)
[2010-02-08 13:39:37] N [trace.c:148:trace_open_cbk] replicated: 3094: (op_ret=-1, op_errno=2, *fd=0x1296fc0)
[2010-02-08 13:39:37] W [fuse-bridge.c:858:fuse_fd_cbk] glusterfs-fuse: 3094: OPEN() /download/90910/live/webb1/webb1/Layer3/prog_index.m3u8 => -1 (No such file or directory)
[2010-02-08 13:39:38] N [trace.c:1837:trace_open] replicated: 3100: (loc {path=/download/90910/live/webb1/webb1/Layer4/prog_index.m3u8, ino=5013773}, flags=32768, fd=0x1296fc0, wbflags=0)
[2010-02-08 13:39:38] N [trace.c:148:trace_open_cbk] replicated: 3100: (op_ret=0, op_errno=117, *fd=0x1296fc0)
[2010-02-08 13:39:38] N [trace.c:1837:trace_open] replicated: 3106: (loc {path=/download/90910/live/webb1/webb1/Layer4/Period1/segment277.ts, ino=5050371}, flags=32768, fd=0x129a430, wbflags=0)
[...]

And server:
[2010-02-08 13:39:09] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e43f3
[2010-02-08 13:39:09] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e440b
[2010-02-08 13:39:17] D [server-protocol.c:2037:server_open_cbk] server: 1719: OPEN (null) (0) ==> -1 (No such file or directory)
[2010-02-08 13:39:18] D [server-protocol.c:2037:server_open_cbk] server: 1724: OPEN (null) (0) ==> -1 (No such file or directory)
[2010-02-08 13:39:28] D [server-resolve.c:238:resolve_path_deep] store0: RESOLVE OPEN() seeking deep resolution of /download/90910/live/webb1/webb1/Layer3/prog_index.m3u8
[2010-02-08 13:39:28] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e43db
[2010-02-08 13:39:28] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e43f3
[2010-02-08 13:39:28] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e440b
[2010-02-08 13:39:28] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7fedee4e43db
[...]

Kind regards,
Fredrik Widlund

_______________________________________________
Gluster-devel mailing list
Gluster-devel at nongnu.org<mailto:Gluster-devel at nongnu.org>
http://lists.nongnu.org/mailman/listinfo/gluster-devel

--
Raghavendra G

--
Raghavendra G
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20100414/b4cf3f4e/attachment-0003.html>