[Gluster-devel] Application segfaults when executed on glusterfs mounted volume

[Erik Osterman]

Unfortunately, our mapserv isn't compiled with debugging symbols. If it 
comes down to it, I can try and arrange that.


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1209813296 (LWP 14678)]
0x080c54e3 in __libc_csu_init ()
(gdb) bt
#0  0x080c54e3 in __libc_csu_init ()
#1  0x0804caa1 in _start ()
(gdb)


... And the tail end of the strace.

open("/usr/lib/libk5crypto.so.3", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220W\245"..., 
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=157196, ...}) = 0
mmap2(0x42a52000, 155040, PROT_READ|PROT_EXEC, 
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x42a52000
mmap2(0x42a77000, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25) = 0x42a77000
close(3)                                = 0
open("/lib/libresolv.so.2", O_RDONLY)   = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\241"..., 
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=80696, ...}) = 0
mmap2(0x42808000, 80072, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 
3, 0) = 0x42808000
mmap2(0x42818000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf) = 0x42818000
mmap2(0x4281a000, 6344, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4281a000
close(3)                                = 0
open("/usr/lib/libkrb5support.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 \266\233"..., 
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=31236, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7e76000
mmap2(0x429ba000, 32488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 
3, 0) = 0x429ba000
mmap2(0x429c1000, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6) = 0x429c1000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7e75000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7e74000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7e746d0, 
limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, 
limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x42818000, 4096, PROT_READ)   = 0
mprotect(0x427ba000, 4096, PROT_READ)   = 0
mprotect(0xb7e7d000, 4096, PROT_READ)   = 0
mprotect(0x42762000, 4096, PROT_READ)   = 0
mprotect(0x42c2b000, 12288, PROT_READ)  = 0
mprotect(0xb7ecf000, 4096, PROT_READ)   = 0
mprotect(0x42757000, 8192, PROT_READ)   = 0
mprotect(0x42605000, 4096, PROT_READ)   = 0
munmap(0xb7f5e000, 36223)               = 0
set_tid_address(0xb7e74718)             = 14686
set_robust_list(0xb7e74720, 0xc)        = -1 ENOSYS (Function not 
implemented)
rt_sigaction(SIGRTMIN, {0x427aa2c0, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x427aa340, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sys="Linux", node="domU-12-31-36-00-35-03", ...}) = 0
futex(0x42c314fc, FUTEX_WAKE, 2147483647) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 14686 detached




Anand Avati wrote:
> Erik,
>  please give a 'bt' command output from gdb, also if possible an
> strace log of running mapsrv over glusterfs (to see the last FS
> operation or more clues)
>
>
> regards,
> avati
>
> On Tue, May 01, 2007 at 05:32:12PM -0700, Erik Osterman wrote:
>   
>> If I try to run our mapserv application from a glusterfs mounted volume 
>> I consistently get segfaults. Yet if I copy that binary from the 
>> glusterfs volume to /tmp, it works fine.
>>
>> I only have the postfix translator enabled on the glusterfsd processes.  
>> The clients are are running afr (*:2) between 3 bricks, with unify, 
>> writebehind, readhead enabled. stat-prefetch and io-threads are both 
>> disabled.
>>
>> Where would be a good place to start diagnosing this problem?
>>
>>
>> Thanks,
>>
>> Erik Osterman
>>
>> -- Transcript --
>>
>> # ./mapserv
>> Segmentation fault
>>
>> GDB reports,
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread -1209043248 (LWP 11563)]
>> 0x080c54e3 in __libc_csu_init ()
>>
>> # cp mapserv /tmp
>> # /tmp/mapserv
>> This script can only be used to decode form results and
>> should be initiated as a CGI process via a httpd server.
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>>     
>
>