[Gluster-infra] slave26 was full, slave23 disappeared

[Justin Clift]

On 18 Mar 2015, at 07:15, Vijay Bellur <vbellur at redhat.com> wrote:
> On 03/18/2015 10:29 AM, Vijay Bellur wrote:
>> On 03/18/2015 03:10 AM, Justin Clift wrote:
>>> On 17 Mar 2015, at 18:03, Michael Scherer <mscherer at redhat.com> wrote:
>>>> Hi,
>>>> 
>>>> so slave 26 was full ( some 35G tar in /archiveds build ). I fixed, so
>>>> that one is good. it was just a lot of core files.
>>> 
>>> Yeah, seems to happen occasionally.  I've just been nuking the oversized
>>> tar file which fills the partition, and restarting the box.
>>> 
>>> We should probably get one of these and look into why the core files
>>> are doing this occasionally.  We don't want to let a bug through into
>>> production builds that might do this... :/
>>> 
>> 
>> slave27 is now full :-/. Can anybody please help?
>> 
>> {standard input}: Assembler messages:
>> {standard input}:84778: Fatal error: can't write .libs/nfs3.o: No space
>> left on device
>> {standard input}:84778: Fatal error:make[5]: *** [nfs3.lo] Error 1
> 
> and now slave20 is full. can somebody please provide a backtrace from one of the cores?

Yep:

************************************

# gdb /build/install/sbin/glusterfs core.25500
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /d/build/install/sbin/glusterfs...done.
[New Thread 25530]
[New Thread 25535]
[New Thread 25509]
[New Thread 25557]
[New Thread 25558]
[New Thread 25670]
[New Thread 25741]
[New Thread 25502]
[New Thread 25504]
[New Thread 25516]
[New Thread 25518]
[New Thread 25523]
[New Thread 25525]
[New Thread 25527]
[New Thread 25537]
[New Thread 25534]
[New Thread 25536]
[New Thread 25500]
[New Thread 25501]
[New Thread 25503]
[New Thread 25517]
[New Thread 25520]
[New Thread 25521]
[New Thread 25522]
[New Thread 25524]
[New Thread 25526]
[New Thread 25528]
[New Thread 25529]
Missing separate debuginfo for /build/install/lib/libglusterfs.so.0
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/41/f4b53e62cb1131673928612a8be1965a343d02
Missing separate debuginfo for /build/install/lib/libgfrpc.so.0
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/20/0c23bb55f24fde1b5c06fd88b116c5aa294d71
Missing separate debuginfo for /build/install/lib/libgfxdr.so.0
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/06/1b9436338d254f416763c906fd1def6c034fd6
Missing separate debuginfo for /build/install/lib/glusterfs/3.7dev/rpc-transport/socket.so
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/54/fdcbdd45d0ba35fe96f07dbbd3f2ddf3d609b5
Missing separate debuginfo for /build/install/lib/glusterfs/3.7dev/xlator/protocol/client.so
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7d/7a0dedde5a4eb7bac6b42a250d98764e7edecb
Missing separate debuginfo for /build/install/lib/glusterfs/3.7dev/xlator/features/bit-rot.so
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c6/71e679afc7f690ebeec32b5c0b1e1fda0ebc0e
Missing separate debuginfo for /build/install/lib/libgfchangelog.so.0
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/82/a7f441722883eac1c2c6515bc8bca38b7d4002
Missing separate debuginfo for 
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4b/8e1260ccd7c3d8cc131e84001f7220651617ec
Reading symbols from /build/install/lib/libglusterfs.so.0...done.
Loaded symbols for /build/install/lib/libglusterfs.so.0
Reading symbols from /build/install/lib/libgfrpc.so.0...done.
Loaded symbols for /build/install/lib/libgfrpc.so.0
Reading symbols from /build/install/lib/libgfxdr.so.0...done.
Loaded symbols for /build/install/lib/libgfxdr.so.0
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /build/install/lib/glusterfs/3.7dev/rpc-transport/socket.so...done.
Loaded symbols for /build/install/lib/glusterfs/3.7dev/rpc-transport/socket.so
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /build/install/lib/glusterfs/3.7dev/xlator/protocol/client.so...done.
Loaded symbols for /build/install/lib/glusterfs/3.7dev/xlator/protocol/client.so
Reading symbols from /build/install/lib/glusterfs/3.7dev/xlator/features/bit-rot.so...done.
Loaded symbols for /build/install/lib/glusterfs/3.7dev/xlator/features/bit-rot.so
Reading symbols from /build/install/lib/libgfchangelog.so.0...done.
Loaded symbols for /build/install/lib/libgfchangelog.so.0
Core was generated by `/build/install/sbin/glusterfs -s localhost --volfile-id gluster/bitd -p /var/li'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000040e3ea in mgmt_rpc_notify (rpc=0x158c780, mydata=0x7f2cc1a6e4a0, event=RPC_CLNT_DISCONNECT, data=0x0)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd-mgmt.c:1813
1813	                        if (server->list.next == &ctx->cmd_args.volfile_servers) {
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 openssl-1.0.1e-30.el6_6.5.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) 

************************************

Hopefully that's helpful.

+ Justin

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift