[Gluster-users] open() issues on 3.0.0

Sun Jan 17 18:31:45 UTC 2010

Aaron,

Nice piece of information !!  ...

Regards,
Tejas.

----- Original Message -----
From: "Aaron Knister" <aaron.knister at gmail.com>
To: brs at usf.edu
Cc: gluster-users at gluster.org
Sent: Sunday, January 17, 2010 11:42:14 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
Subject: Re: [Gluster-users] open() issues on 3.0.0

I ran into a similar problem when compiling code on a gluster mount with the intel compiler. The problem was intermittent and after digging into it further it appeared to only happy when compiling from source files that had very high inode numbers. There's a command run by icc that's a 32 bit executable and my guess is that somewhere internally the extremely large inode number wasn't able to be handled and caused the compilation to fail. The "fix" was to upgrade the compiler. I can verify that version 11.0.064 works without a hitch (at least in this regard).

More Detail:

I can reproduce this error with version 10.0.026 of the intel compiler.

I have file test.c on a glusterfs with inode num 21479794854 (greater than 32 bits)
21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c

Running icc fails:
...
$ icc test.c 
Catastrophic error: could not open source file "test.c"

compilation aborted for test.c (code 4)
...

However if I copy the same file to a directory on the same fs that has a 32 bit inode number (le 4294967295) (hoping the new file will also have a 32 bit inode number):
11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c

Compilation is a success:
...
$ !icc
icc test.c 
$ echo $?
0
...

If you run icc with the -v option you'll see that a program called mcpcom gets run and produces the error. At least with version 10.0.026 it was a 32 bit program:
$ file /opt/intel/cce/10.0.026/bin/mcpcom
/opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped

Anyway I hope that helps.

-Aaron

On Jan 17, 2010, at 11:13 AM, Brian Smith wrote:

> Hi,
> 
> I'm new to Gluster, so please forgive any ignorance on my part.  I've
> tried reading over everything I could find to resolve this myself.
> 
> I'm having an issue building executables on our gluster volume (there
> are some other similar issues as well, but we'll deal with this one
> first).  I have the presta infiniband utilities extracted into a
> directory in my glusterfs volume and I'm attempting to build them.  One
> of the source files, util.c (coincidentally, the first one referenced in
> the make file) fails to build.  I get an error message:
> 
> Catastrophic error: could not open source file "util.c"
> 
> Some further investigating reveals the command that causes the issue:
> 
> icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
> -L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib -lmpi
> -lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath -lnuma
> -ldl -Wl,--export-dynamic -lnsl -lutil -c util.c 
> Catastrophic error: could not open source file "util.c"
> 
> compilation aborted for util.c (code 4)
> 
> And further, we see an strace of the command:
> ...
> 2717 6601  brk(0x957b000)                    = 0x957b000
> 2718 6601  open("util.c", O_RDONLY)          = 3
> 2719 6601  stat64("util.c", {st_mode=S_IFREG|0600, st_size=14370, ...})
> = 0
> 2720 6601  close(3)                          = 0
> 2721 6601  fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
> 1), ...}) = 0
> 2722 6601  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
> MAP_ANONYMOUS, -1, 0x1000) = 0xfffffffff0bce000
> 2723 6601  write(2, "Catastrophic error: could not open source file
> \"util.c\"\n", 56) = 56
> 2724 6601  write(2, "\n", 1)                 = 1
> 2725 6601  exit_group(4)
> ...
> 
> We can also see that the file is indeed present:
> 
> $ ls -l util.c
> -rw------- 1 brs users 14370 Apr  8  2002 util.c
> $ stat util.c
>  File: `util.c'
>  Size: 14370     	Blocks: 32         IO Block: 4096   regular file
> Device: 18h/24d	Inode: 4295774314  Links: 1
> Access: (0600/-rw-------)  Uid: (1229806/     brs)   Gid: (10001/
> users)
> Access: 2010-01-17 00:27:43.000000000 -0500
> Modify: 2002-04-08 21:13:57.000000000 -0400
> Change: 2010-01-17 00:27:43.000000000 -0500
> 
> If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
> above works correctly.
> 
> diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
> 
> also appears clean.  Any ideas what is happening here?  Is there an
> issue with mmap2() and glusterfs?
> 
> Many thanks in advance,
> 
> -Brian
> 
> 
> 
> -- 
> Brian Smith
> Senior Systems Administrator
> IT Research Computing, University of South Florida
> 4202 E. Fowler Ave. ENB308
> Office Phone: +1 813 974-1467
> Organization URL: http://rc.usf.edu
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users