[Bugs] [Bug 1314834] New: [GSS] - Samba crash
bugzilla at redhat.com
bugzilla at redhat.com
Fri Mar 4 16:10:27 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1314834
Bug ID: 1314834
Summary: [GSS] - Samba crash
Product: Red Hat Gluster Storage
Version: 3.1
Component: gluster-smb
Keywords: Triaged, ZStream
Severity: urgent
Priority: medium
Assignee: rhs-smb at redhat.com
Reporter: mmalhotr at redhat.com
QA Contact: storage-qa-internal at redhat.com
CC: anoopcs at redhat.com, bugs at gluster.org,
bugzilla.redhat at flying-snail.net,
denis.lambolez at gmail.com, dijuremo at gatech.edu,
gluster-bugs at redhat.com, monotek23 at gmail.com,
ndevos at redhat.com, nlevinki at redhat.com,
rhs-smb at redhat.com, rjoseph at redhat.com,
sdharane at redhat.com, srangana at redhat.com
Depends On: 1234877
External Bug ID: Samba Project 11115
Keywords: ZStream
+++ This bug was initially created as a clone of Bug #1234877 +++
Description of problem:
After upgrading to "3.7.2-ubuntu1~vivid1" for GlusterFS and
"2:4.1.13+dfsg-4ubuntu3glusterfs3.7.2vivid1" for the Samba VFS Module, I'm
experiencing crash of the smbd deamon, at a high frequency rate. The system was
stable with the 3.6 version.
Actual results:
Here is the dump of the samba panic action script. It seems that there is a
problem in glfs_chdir () from /usr/lib/x86_64-linux-gnu/libgfapi.so.0.
============================================================================
The Samba 'panic action' script, /usr/share/samba/panic-action,
was called for PID 3286 (/usr/sbin/smbd).
This means there was a problem with the program, such as a segfault.
Below is a backtrace for this process generated with gdb, which shows
the state of the program at the time the error occurred. The Samba log
files may contain additional information about the problem.
If the problem persists, you are encouraged to first install the
samba-dbg package, which contains the debugging symbols for the Samba
binaries. Then submit the provided information as a bug report to
Ubuntu by visiting this link:
https://launchpad.net/ubuntu/+source/samba/+filebug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fc5fcf4389b in __GI___waitpid (pid=12767,
stat_loc=stat_loc at entry=0x7ffcf6243f90, options=options at entry=0) at
../sysdeps/unix/sysv/linux/waitpid.c:40
#0 0x00007fc5fcf4389b in __GI___waitpid (pid=12767,
stat_loc=stat_loc at entry=0x7ffcf6243f90, options=options at entry=0) at
../sysdeps/unix/sysv/linux/waitpid.c:40
#1 0x00007fc5fcebcffb in do_system (line=line at entry=0x7fc601f34d10
"/usr/share/samba/panic-action 3286") at ../sysdeps/posix/system.c:148
#2 0x00007fc5fcebd3da in __libc_system (line=line at entry=0x7fc601f34d10
"/usr/share/samba/panic-action 3286") at ../sysdeps/posix/system.c:184
#3 0x00007fc6001d7c05 in system (line=line at entry=0x7fc601f34d10
"/usr/share/samba/panic-action 3286") at pt-system.c:28
#4 0x00007fc5fe8612d1 in smb_panic_s3 (why=<optimized out>) at
../source3/lib/util.c:798
#5 0x00007fc5fffaedcf in smb_panic (why=why at entry=0x7fc5fffbb77c "internal
error") at ../lib/util/fault.c:159
#6 0x00007fc5fffaefef in fault_report (sig=<optimized out>) at
../lib/util/fault.c:77
#7 sig_fault (sig=<optimized out>) at ../lib/util/fault.c:88
#8 <signal handler called>
#9 0x00007fc5eeb722dd in glfs_chdir () from
/usr/lib/x86_64-linux-gnu/libgfapi.so.0
#10 0x00007fc5ffb862a3 in vfs_ChDir (conn=0x7fc601fb1830, path=0x7fc5ffc9c3f6
"/") at ../source3/smbd/vfs.c:840
#11 0x00007fc5ffb9b8fb in close_cnum (conn=0x7fc601fb1830, vuid=3164412539) at
../source3/smbd/service.c:1136
#12 0x00007fc5ffbc4b6c in smbXsrv_tcon_disconnect (tcon=0x7fc601fb06c0,
vuid=3164412539) at ../source3/smbd/smbXsrv_tcon.c:977
#13 0x00007fc5ffbc4f52 in smbXsrv_tcon_disconnect_all_callback
(local_rec=0x7ffcf6244a00, private_data=0x7ffcf6244ad0) at
../source3/smbd/smbXsrv_tcon.c:1058
#14 0x00007fc5fb539a03 in db_rbt_traverse_internal (db=db at entry=0x7fc601f32da0,
n=0x0, f=f at entry=0x7fc5ffbc4ee0 <smbXsrv_tcon_disconnect_all_callback>,
private_data=private_data at entry=0x7ffcf6244ad0,
count=count at entry=0x7ffcf6244a8c, rw=rw at entry=true) at
../lib/dbwrap/dbwrap_rbt.c:401
#15 0x00007fc5fb539b3f in db_rbt_traverse (db=0x7fc601f32da0, f=0x7fc5ffbc4ee0
<smbXsrv_tcon_disconnect_all_callback>, private_data=0x7ffcf6244ad0) at
../lib/dbwrap/dbwrap_rbt.c:427
#16 0x00007fc5fb53860a in dbwrap_traverse (db=<optimized out>,
f=f at entry=0x7fc5ffbc4ee0 <smbXsrv_tcon_disconnect_all_callback>,
private_data=private_data at entry=0x7ffcf6244ad0,
count=count at entry=0x7ffcf6244acc) at ../lib/dbwrap/dbwrap.c:353
#17 0x00007fc5ffbc3a39 in smbXsrv_tcon_disconnect_all (table=<optimized out>,
vuid=<optimized out>) at ../source3/smbd/smbXsrv_tcon.c:1007
#18 0x00007fc5ffbc50ca in smb2srv_tcon_disconnect_all
(session=session at entry=0x7fc601f33a20) at ../source3/smbd/smbXsrv_tcon.c:1165
#19 0x00007fc5ffbc26f7 in smbXsrv_session_logoff (session=0x7fc601f33a20) at
../source3/smbd/smbXsrv_session.c:1387
#20 0x00007fc5ffbc2aa2 in smbXsrv_session_logoff_all_callback
(local_rec=0x7ffcf6244ba0, private_data=0x7ffcf6244c70) at
../source3/smbd/smbXsrv_session.c:1473
#21 0x00007fc5fb539a03 in db_rbt_traverse_internal (db=db at entry=0x7fc601f30f90,
n=0x0, f=f at entry=0x7fc5ffbc2a50 <smbXsrv_session_logoff_all_callback>,
private_data=private_data at entry=0x7ffcf6244c70,
count=count at entry=0x7ffcf6244c2c, rw=rw at entry=true) at
../lib/dbwrap/dbwrap_rbt.c:401
#22 0x00007fc5fb539b3f in db_rbt_traverse (db=0x7fc601f30f90, f=0x7fc5ffbc2a50
<smbXsrv_session_logoff_all_callback>, private_data=0x7ffcf6244c70) at
../lib/dbwrap/dbwrap_rbt.c:427
#23 0x00007fc5fb53860a in dbwrap_traverse (db=<optimized out>,
f=f at entry=0x7fc5ffbc2a50 <smbXsrv_session_logoff_all_callback>,
private_data=private_data at entry=0x7ffcf6244c70,
count=count at entry=0x7ffcf6244c6c) at ../lib/dbwrap/dbwrap.c:353
#24 0x00007fc5ffbc2aeb in smbXsrv_session_logoff_all
(conn=conn at entry=0x7fc601f22050) at ../source3/smbd/smbXsrv_session.c:1428
#25 0x00007fc5ffbc7a26 in exit_server_common (how=how at entry=SERVER_EXIT_NORMAL,
reason=0x7fc5ffc8c26f "termination signal") at
../source3/smbd/server_exit.c:138
#26 0x00007fc5ffbc7e9e in smbd_exit_server_cleanly (explanation=<optimized
out>) at ../source3/smbd/server_exit.c:238
#27 0x00007fc5fe1b8b32 in exit_server_cleanly
(reason=reason at entry=0x7fc5ffc8c26f "termination signal") at
../source3/lib/smbd_shim.c:113
#28 0x00007fc5ffb92ce0 in smbd_sig_term_handler (ev=<optimized out>,
se=<optimized out>, signum=<optimized out>, count=<optimized out>,
siginfo=<optimized out>, private_data=<optimized out>) at
../source3/smbd/process.c:903
#29 0x00007fc5fd249fcf in tevent_common_check_signal () from
/usr/lib/x86_64-linux-gnu/libtevent.so.0
#30 0x00007fc5fe877674 in run_events_poll (ev=0x7fc601f1cfc0, pollrtn=-1,
pfds=0x7fc601f23e60, num_pfds=3) at ../source3/lib/events.c:187
#31 0x00007fc5fe877a37 in s3_event_loop_once (ev=0x7fc601f1cfc0,
location=<optimized out>) at ../source3/lib/events.c:326
#32 0x00007fc5fd2469ad in _tevent_loop_once () from
/usr/lib/x86_64-linux-gnu/libtevent.so.0
#33 0x00007fc5ffb990cc in smbd_process (ev_ctx=0x7fc601f1cfc0,
msg_ctx=0x7fc6001c67c0 <DEBUGLEVEL_CLASS>, sock_fd=32721024, interactive=64) at
../source3/smbd/process.c:3695
#34 0x00007fc600615100 in smbd_accept_connection (ev=0x7fc601f1cfc0,
fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at
../source3/smbd/server.c:610
#35 0x00007fc5fe8777c1 in run_events_poll (ev=0x7fc601f1cfc0,
pollrtn=<optimized out>, pfds=0x7fc601f23e60, num_pfds=5) at
../source3/lib/events.c:257
#36 0x00007fc5fe877a37 in s3_event_loop_once (ev=0x7fc601f1cfc0,
location=<optimized out>) at ../source3/lib/events.c:326
#37 0x00007fc5fd2469ad in _tevent_loop_once () from
/usr/lib/x86_64-linux-gnu/libtevent.so.0
#38 0x00007fc600611e43 in smbd_parent_loop (parent=<optimized out>,
ev_ctx=<optimized out>) at ../source3/smbd/server.c:934
#39 main (argc=32624576, argv=0x7fc601f1d7b0) at ../source3/smbd/server.c:1566
A debugging session is active.
Inferior 1 [process 3286] will be detached.
Quit anyway? (y or n) [answered Y; input not from terminal]
============================================================================
Additional info:
The Samba server runs in cluster mode (CTDB). Shares are stored on a Gluster
replicated volume and are exposed through Samba VFS module.
--- Additional comment from André Bauer on 2015-06-23 08:36:00 EDT ---
The samba packages can be found here:
https://launchpad.net/~monotek/+archive/ubuntu/samba-vfs-glusterfs-3.7
--- Additional comment from André Bauer on 2015-07-29 14:01:20 EDT ---
Just uploaded new packages which are build against Gluster 3.7.3.
https://launchpad.net/~monotek/+archive/ubuntu/samba-vfs-glusterfs-3.7
--- Additional comment from Denis Lambolez on 2015-08-09 13:58:02 EDT ---
Just tested with GlusterFS 3.7.3 and the new package from André. Same
behaviour. No change. Here is the dump of the samba panic action script.
Problem is still within libgfapi.so.0. This time it seems that there is a
problem in "glfs_resolve_at".
==========================================================================
The Samba 'panic action' script, /usr/share/samba/panic-action,
was called for PID 26859 (/usr/sbin/smbd).
This means there was a problem with the program, such as a segfault.
Below is a backtrace for this process generated with gdb, which shows
the state of the program at the time the error occurred. The Samba log
files may contain additional information about the problem.
If the problem persists, you are encouraged to first install the
samba-dbg package, which contains the debugging symbols for the Samba
binaries. Then submit the provided information as a bug report to
Ubuntu by visiting this link:
https://launchpad.net/ubuntu/+source/samba/+filebug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f26a19d489b in __GI___waitpid (pid=26868,
stat_loc=stat_loc at entry=0x7ffdb5695550, options=options at entry=0) at
../sysdeps/unix/sysv/linux/waitpid.c:40
#0 0x00007f26a19d489b in __GI___waitpid (pid=26868,
stat_loc=stat_loc at entry=0x7ffdb5695550, options=options at entry=0) at
../sysdeps/unix/sysv/linux/waitpid.c:40
#1 0x00007f26a194dffb in do_system (line=line at entry=0x7f26a54cf1b0
"/usr/share/samba/panic-action 26859") at ../sysdeps/posix/system.c:148
#2 0x00007f26a194e3da in __libc_system (line=line at entry=0x7f26a54cf1b0
"/usr/share/samba/panic-action 26859") at ../sysdeps/posix/system.c:184
#3 0x00007f26a4c68c05 in system (line=line at entry=0x7f26a54cf1b0
"/usr/share/samba/panic-action 26859") at pt-system.c:28
#4 0x00007f26a32f22d1 in smb_panic_s3 (why=<optimized out>) at
../source3/lib/util.c:798
#5 0x00007f26a4a3fdcf in smb_panic (why=why at entry=0x7f26a4a4c77c "internal
error") at ../lib/util/fault.c:159
#6 0x00007f26a4a3ffef in fault_report (sig=<optimized out>) at
../lib/util/fault.c:77
#7 sig_fault (sig=<optimized out>) at ../lib/util/fault.c:88
#8 <signal handler called>
#9 0x00007f26931fe2b1 in glfs_resolve_at () from
/usr/lib/x86_64-linux-gnu/libgfapi.so.0
#10 0x00007f26931ff71f in ?? () from /usr/lib/x86_64-linux-gnu/libgfapi.so.0
#11 0x00007f26931ff7a1 in glfs_resolve () from
/usr/lib/x86_64-linux-gnu/libgfapi.so.0
#12 0x00007f26931fc36f in glfs_chdir () from
/usr/lib/x86_64-linux-gnu/libgfapi.so.0
#13 0x00007f26a46172a3 in vfs_ChDir (conn=0x7f26a54d17a0, path=0x7f26a472d3f6
"/") at ../source3/smbd/vfs.c:840
#14 0x00007f26a462c8fb in close_cnum (conn=0x7f26a54d17a0, vuid=52003) at
../source3/smbd/service.c:1136
#15 0x00007f26a4655b6c in smbXsrv_tcon_disconnect (tcon=0x7f26a54cad40,
vuid=52003) at ../source3/smbd/smbXsrv_tcon.c:977
#16 0x00007f26a4655f52 in smbXsrv_tcon_disconnect_all_callback
(local_rec=0x7ffdb56960c0, private_data=0x7ffdb5696190) at
../source3/smbd/smbXsrv_tcon.c:1058
#17 0x00007f269ffcaa03 in db_rbt_traverse_internal (db=db at entry=0x7f26a54c1cf0,
n=0x0, f=f at entry=0x7f26a4655ee0 <smbXsrv_tcon_disconnect_all_callback>,
private_data=private_data at entry=0x7ffdb5696190,
count=count at entry=0x7ffdb569614c, rw=rw at entry=true) at
../lib/dbwrap/dbwrap_rbt.c:401
#18 0x00007f269ffcab3f in db_rbt_traverse (db=0x7f26a54c1cf0, f=0x7f26a4655ee0
<smbXsrv_tcon_disconnect_all_callback>, private_data=0x7ffdb5696190) at
../lib/dbwrap/dbwrap_rbt.c:427
#19 0x00007f269ffc960a in dbwrap_traverse (db=<optimized out>,
f=f at entry=0x7f26a4655ee0 <smbXsrv_tcon_disconnect_all_callback>,
private_data=private_data at entry=0x7ffdb5696190,
count=count at entry=0x7ffdb569618c) at ../lib/dbwrap/dbwrap.c:353
#20 0x00007f26a4654a39 in smbXsrv_tcon_disconnect_all (table=<optimized out>,
vuid=vuid at entry=0) at ../source3/smbd/smbXsrv_tcon.c:1007
#21 0x00007f26a4656012 in smb1srv_tcon_disconnect_all
(conn=conn at entry=0x7f26a54b1d60) at ../source3/smbd/smbXsrv_tcon.c:1119
#22 0x00007f26a4658a14 in exit_server_common (how=how at entry=SERVER_EXIT_NORMAL,
reason=0x7f26a4753d9f "failed to receive smb request") at
../source3/smbd/server_exit.c:127
#23 0x00007f26a4658e9e in smbd_exit_server_cleanly (explanation=<optimized
out>) at ../source3/smbd/server_exit.c:238
#24 0x00007f26a2c49b32 in exit_server_cleanly (reason=<optimized out>) at
../source3/lib/smbd_shim.c:113
#25 0x00007f26a4628ccc in smbd_server_connection_read_handler
(sconn=0x7f26a54bef30, fd=33) at ../source3/smbd/process.c:2433
#26 0x00007f26a33087c1 in run_events_poll (ev=0x7f26a54acfc0,
pollrtn=<optimized out>, pfds=0x7f26a54b2990, num_pfds=3) at
../source3/lib/events.c:257
#27 0x00007f26a3308a37 in s3_event_loop_once (ev=0x7f26a54acfc0,
location=<optimized out>) at ../source3/lib/events.c:326
#28 0x00007f26a1cd79ad in _tevent_loop_once () from
/usr/lib/x86_64-linux-gnu/libtevent.so.0
#29 0x00007f26a462a0cc in smbd_process (ev_ctx=0x7f26a54acfc0,
msg_ctx=0x7f26a4c577c0 <DEBUGLEVEL_CLASS>, sock_fd=-1521750224, interactive=4)
at ../source3/smbd/process.c:3695
#30 0x00007f26a50a6100 in smbd_accept_connection (ev=0x7f26a54acfc0,
fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at
../source3/smbd/server.c:610
#31 0x00007f26a33087c1 in run_events_poll (ev=0x7f26a54acfc0,
pollrtn=<optimized out>, pfds=0x7f26a54b2990, num_pfds=5) at
../source3/lib/events.c:257
#32 0x00007f26a3308a37 in s3_event_loop_once (ev=0x7f26a54acfc0,
location=<optimized out>) at ../source3/lib/events.c:326
#33 0x00007f26a1cd79ad in _tevent_loop_once () from
/usr/lib/x86_64-linux-gnu/libtevent.so.0
#34 0x00007f26a50a2e43 in smbd_parent_loop (parent=<optimized out>,
ev_ctx=<optimized out>) at ../source3/smbd/server.c:934
#35 main (argc=-1521823808, argv=0x7f26a54ad7b0) at
../source3/smbd/server.c:1566
A debugging session is active.
Inferior 1 [process 26859] will be detached.
Quit anyway? (y or n) [answered Y; input not from terminal]
--- Additional comment from André Bauer on 2015-09-02 08:03:42 EDT ---
Just uploaded new packages which are build against Gluster 3.7.4.
https://launchpad.net/~monotek/+archive/ubuntu/samba-vfs-glusterfs-3.7
--- Additional comment from Denis Lambolez on 2015-09-09 17:23:23 EDT ---
(In reply to André Bauer from comment #4)
> Just uploaded new packages which are build against Gluster 3.7.4.
>
> https://launchpad.net/~monotek/+archive/ubuntu/samba-vfs-glusterfs-3.7
Just tested with GlusterFS 3.7.4 and the new package from André. Same
behaviour. No change. The crash is still in the same area. Same dump.
--- Additional comment from dijuremo on 2015-09-09 18:29:35 EDT ---
I am having the same problem, upgraded glusterfs from 3.6.x to 3.7.3 and now
samba is producing core dumps.
# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
# rpm -qa | grep glusterfs
samba-vfs-glusterfs-4.1.12-23.el7_1.x86_64
glusterfs-libs-3.7.3-1.el7.x86_64
glusterfs-cli-3.7.3-1.el7.x86_64
glusterfs-client-xlators-3.7.3-1.el7.x86_64
glusterfs-server-3.7.3-1.el7.x86_64
glusterfs-api-3.7.3-1.el7.x86_64
glusterfs-rdma-3.7.3-1.el7.x86_64
glusterfs-fuse-3.7.3-1.el7.x86_64
glusterfs-3.7.3-1.el7.x86_64
Do you need me to upload any of the core dumps?
--- Additional comment from Anoop C S on 2015-09-24 04:28:58 EDT ---
Can you explain the GlusterFS(volume configuration), Samba-CTDB setup and
procedure followed which resulted in dumping core by smbd?
--- Additional comment from dijuremo on 2015-09-24 08:45:41 EDT ---
Gluster volume:
[root at ysmha01 core]# gluster volume info export
Volume Name: export
Type: Replicate
Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.0.1.7:/bricks/hdds/brick
Brick2: 10.0.1.6:/bricks/hdds/brick
Options Reconfigured:
performance.io-cache: on
performance.io-thread-count: 64
nfs.disable: on
cluster.server-quorum-type: server
performance.cache-size: 1024MB
server.allow-insecure: on
cluster.server-quorum-ratio: 51%
Samba configuration for each share has the entries:
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-homes.log
glusterfs:volume = export
I am currently not even using ctdb, just manually starting samba in one server.
[root at ysmha01 core]# rpm -qa | grep samba
samba-vfs-glusterfs-4.1.12-23.el7_1.x86_64
samba-common-4.1.12-23.el7_1.x86_64
samba-4.1.12-23.el7_1.x86_64
samba-libs-4.1.12-23.el7_1.x86_64
Everything was working just fine on glusterfs 3.6.5. After upgrading to 3.7.3,
samba started core dumping.
What specific information do you want?
--- Additional comment from Anoop C S on 2015-09-29 07:33:21 EDT ---
Need some more info:
[1] On which platform are you working on (CentOS, RHEL)? For CentOS 7, I see
that the latest samba version is 4.1.12-21.el7_1
(http://mirror.centos.org/centos/7/os/x86_64/Packages/).
[2] Have you installed glusterfs packages from download.gluster.org ?
[3] Can you please share the general workload/procedure followed which causes
the core dump so as to try reproducing the situation for better debugging of
the issue?
--- Additional comment from dijuremo on 2015-09-29 09:07:50 EDT ---
[1]* Answer
[root at ysmha01 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root at ysmha01 ~]# yum info samba-4.1.12-23.el7_1.x86_64
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.greenmountainaccess.net
* extras: centos.den.host-engine.com
* ovirt-3.5-epel: mirror.us.leaseweb.net
* updates: centos.mirrors.tds.net
Installed Packages
Name : samba
Arch : x86_64
Version : 4.1.12
Release : 23.el7_1
Size : 1.6 M
Repo : installed
>From repo : updates
Summary : Server and Client software to interoperate with Windows machines
URL : http://www.samba.org/
License : GPLv3+ and LGPLv3+
Description : Samba is the standard Windows interoperability suite of programs
for Linux and Unix.
The updates repo section shows:
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
You are looking at CentOS 7.0 instead of 7.1 which has samba-
4.1.12-23.el7_1.x86_64:
http://mirror.centos.org/centos/7.1.1503/os/x86_64/Packages/samba-4.1.12-21.el7_1.x86_64.rpm
[2]* Answer
I have the gluster packages from:
[root at ysmha01 ~]# yum info glusterfs
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.greenmountainaccess.net
* extras: centos.den.host-engine.com
* ovirt-3.5-epel: mirror.us.leaseweb.net
* updates: centos.mirrors.tds.net
Installed Packages
Name : glusterfs
Arch : x86_64
Version : 3.7.3
Release : 1.el7
Size : 1.6 M
Repo : installed
>From repo : ovirt-3.5-glusterfs-epel
Summary : Cluster File System
URL : http://www.gluster.org/docs/index.php/GlusterFS
License : GPLv2 or LGPLv3+
Description : GlusterFS is a distributed file-system capable of scaling to
several
: petabytes. It aggregates various storage bricks over Infiniband
RDMA
: or TCP/IP interconnect into one large parallel network file
: system. GlusterFS is one of the most sophisticated file systems
in
: terms of features and extensibility. It borrows a powerful
concept
: called Translators from GNU Hurd kernel. Much of the code in
GlusterFS
: is in user space and easily manageable.
:
: This package includes the glusterfs binary, the glusterfsd daemon
and the
: libglusterfs and glusterfs translator modules common to both
GlusterFS server
: and client framework.
[3]* Answer
I have no specific procedure to produce the core dumps, they are just produced
by users accessing the server via the samba shares. All shares are currently
using the samba vfs gluster feature. The core dumps started just after the
upgrade of gluster 3.6.x to 3.7.x. I have not had a chance to downgrade to 3.6
nor upgrade to 3.7.4 to see if that helps at all.
Here is a list of core dumps by time since 00:00AM today, I will be happy to
upload one for you if you want to run it through the debugger:
[root at ysmha01 core]# file core.6502.1443531854.dump
core.6502.1443531854.dump: ELF 64-bit LSB core file x86-64, version 1 (SYSV),
SVR4-style, from '/usr/sbin/smbd'
[root at ysmha01 core]# ls -l --sort=time
total 1040848
-rw-------. 1 root root 67354624 Sep 29 09:04 core.6502.1443531854.dump
-rw-------. 1 root root 62304256 Sep 29 08:36 core.23166.1443530215.dump
-rw-------. 1 root root 59539456 Sep 29 08:08 core.29407.1443528538.dump
-rw-------. 1 root root 57675776 Sep 29 08:01 core.26438.1443528119.dump
-rw-------. 1 root root 63483904 Sep 29 07:16 core.4331.1443525364.dump
-rw-------. 1 root root 61505536 Sep 29 06:45 core.24178.1443523500.dump
-rw-------. 1 root root 67715072 Sep 29 06:28 core.25758.1443522504.dump
-rw-------. 1 root root 67682304 Sep 29 06:22 core.23897.1443522148.dump
-rw-------. 1 root root 63135744 Sep 29 05:35 core.22984.1443519340.dump
-rw-------. 1 root root 58912768 Sep 29 05:35 core.21996.1443519312.dump
-rw-------. 1 root root 58888192 Sep 29 05:34 core.21670.1443519263.dump
-rw-------. 1 root root 58945536 Sep 29 05:33 core.21142.1443519233.dump
-rw-------. 1 root root 58986496 Sep 29 05:33 core.20819.1443519210.dump
-rw-------. 1 root root 58892288 Sep 29 05:33 core.20286.1443519190.dump
-rw-------. 1 root root 57675776 Sep 29 05:32 core.19898.1443519151.dump
-rw-------. 1 root root 58904576 Sep 29 05:32 core.19343.1443519130.dump
-rw-------. 1 root root 58966016 Sep 29 05:31 core.17579.1443519060.dump
-rw-------. 1 root root 58826752 Sep 29 05:24 core.12325.1443518662.dump
-rw-------. 1 root root 57675776 Sep 29 05:23 core.12095.1443518628.dump
-rw-------. 1 root root 58945536 Sep 29 05:23 core.11661.1443518608.dump
-rw-------. 1 root root 57671680 Sep 29 05:17 core.6417.1443518252.dump
-rw-------. 1 root root 58871808 Sep 29 05:17 core.6052.1443518228.dump
-rw-------. 1 root root 60137472 Sep 29 05:07 core.30876.1443517655.dump
-rw-------. 1 root root 57532416 Sep 29 05:00 core.16957.1443517236.dump
-rw-------. 1 root root 59256832 Sep 29 04:47 core.11112.1443516445.dump
-rw-------. 1 root root 59236352 Sep 29 04:45 core.9295.1443516325.dump
-rw-------. 1 root root 60268544 Sep 29 04:45 core.5285.1443516301.dump
-rw-------. 1 root root 59240448 Sep 29 04:44 core.7476.1443516254.dump
-rw-------. 1 root root 57696256 Sep 29 04:37 core.1972.1443515822.dump
-rw-------. 1 root root 64811008 Sep 29 04:36 core.1531.1443515807.dump
-rw-------. 1 root root 59228160 Sep 29 04:30 core.28017.1443515431.dump
-rw-------. 1 root root 59674624 Sep 29 03:26 core.11734.1443511619.dump
-rw-------. 1 root root 57556992 Sep 29 03:22 core.31346.1443511341.dump
-rw-------. 1 root root 59392000 Sep 29 03:22 core.30577.1443511324.dump
-rw-------. 1 root root 57552896 Sep 29 03:21 core.30112.1443511267.dump
-rw-------. 1 root root 59346944 Sep 29 03:20 core.29235.1443511237.dump
-rw-------. 1 root root 57556992 Sep 29 03:19 core.28504.1443511161.dump
-rw-------. 1 root root 59322368 Sep 29 03:18 core.27591.1443511134.dump
-rw-------. 1 root root 57667584 Sep 29 03:18 core.27311.1443511080.dump
-rw-------. 1 root root 59318272 Sep 29 03:17 core.26885.1443511067.dump
-rw-------. 1 root root 64516096 Sep 29 03:03 core.13713.1443510199.dump
-rw-------. 1 root root 58908672 Sep 29 03:01 core.12424.1443510111.dump
-rw-------. 1 root root 57671680 Sep 29 03:00 core.11462.1443510040.dump
-rw-------. 1 root root 58859520 Sep 29 03:00 core.10719.1443510019.dump
-rw-------. 1 root root 57671680 Sep 29 02:55 core.6832.1443509729.dump
-rw-------. 1 root root 58888192 Sep 29 02:55 core.6362.1443509702.dump
-rw-------. 1 root root 57671680 Sep 29 02:45 core.30494.1443509159.dump
-rw-------. 1 root root 58925056 Sep 29 02:45 core.29922.1443509137.dump
-rw-------. 1 root root 58941440 Sep 29 02:45 core.28703.1443509107.dump
-rw-------. 1 root root 58904576 Sep 29 02:43 core.27644.1443509005.dump
-rw-------. 1 root root 64491520 Sep 29 02:42 core.26854.1443508949.dump
-rw-------. 1 root root 67346432 Sep 29 01:02 core.14361.1443502958.dump
-rw-------. 1 root root 65433600 Sep 29 00:45 core.10603.1443501907.dump
Let me know what other information I can provide.
--- Additional comment from dijuremo on 2015-09-29 09:11:31 EDT ---
**Correction, the latest samba package is coming from the updates section:
http://mirror.centos.org/centos/7.1.1503/updates/x86_64/Packages/samba-test-4.1.12-23.el7_1.x86_64.rpm
--- Additional comment from Anoop C S on 2015-09-29 09:36:36 EDT ---
Of course, it would be good if you can upload one among the above listed core
dumps and many thanks for the detailed reply.
--- Additional comment from dijuremo on 2015-09-29 09:44 EDT ---
Attached compressed tar file for core.6502.1443531854.dump
--- Additional comment from Denis Lambolez on 2015-10-04 14:28:21 EDT ---
(In reply to Anoop C S from comment #7)
> Can you explain the GlusterFS(volume configuration), Samba-CTDB setup and
> procedure followed which resulted in dumping core by smbd?
Sorry, I was out for while, on business trips.
Here are the requested information. I've one glusterFS volume (replicate) that
I use to store samba shares. I use Samba-CTDB to balance access load between
the two servers. The shares are published by Samba through André's VFS object.
smb.conf is shared between the two servers (the glusterFS volume is also
mounted on the two servers).
There is no specific action to crash samba. As soon as you start samba, it
starts to crash and restart.
This configuration was running perfectly well with the 3.6.
GlusterFS volume configuration:
-------------------------------
matou at catsserver-1:~$ sudo gluster vol info smbshare
Volume Name: smbshare
Type: Replicate
Volume ID: 40bfc10d-6f7a-45cf-81ba-0e4d531da890
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: catsserver-node1:/srv/gluster/bricks/smbshare
Brick2: catsserver-node2:/srv/gluster/bricks/smbshare
Options Reconfigured:
nfs.disable: on
server.allow-insecure: on
CTDB status:
------------
matou at catsserver-1:~$ sudo ctdb status
Number of nodes:2
pnn:0 192.168.0.21 OK (THIS NODE)
pnn:1 192.168.0.22 OK
Generation:1125798900
Size:2
hash:0 lmaster:0
hash:1 lmaster:1
Recovery mode:NORMAL (0)
Recovery master:0
Shares configuration (same configuration for each share):
---------------------------------------------------------
[Share]
path = /share
wide links = no
writeable = yes
kernel share modes = no
vfs objects = glusterfs
glusterfs:volfile_server = localhost
glusterfs:volume = smbshare
glusterfs:logfile = /var/log/samba/glusterfs-share.log
glusterfs:loglevel = 7
CTDB & Samba & Gluster paquets:
-------------------------------
ctdb/vivid,now 2.5.4+debian0-4 amd64 [installé]
ctdb-dbg/vivid 2.5.4+debian0-4 amd64
ctdb-pcp-pmda/vivid 2.5.4+debian0-4 amd64
samba/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64 [installé]
samba-common/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 all
[installé]
samba-common-bin/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installé]
samba-dbg/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installé]
samba-dev/vivid 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
samba-doc/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 all [installé]
samba-dsdb-modules/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installé]
samba-libs/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installé]
samba-testsuite/vivid 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
samba-vfs-modules/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installé]
smbclient/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installé]
glusterfs-client/vivid,now 3.7.4-ubuntu1~vivid1 amd64 [installé, automatique]
glusterfs-common/vivid,now 3.7.4-ubuntu1~vivid1 amd64 [installé, automatique]
glusterfs-dbg/vivid 3.7.4-ubuntu1~vivid1 amd64
glusterfs-server/vivid,now 3.7.4-ubuntu1~vivid1 amd64 [installé]
--- Additional comment from Anoop C S on 2015-10-06 08:56:13 EDT ---
(In reply to Denis Lambolez from comment #14)
> (In reply to Anoop C S from comment #7)
> > Can you explain the GlusterFS(volume configuration), Samba-CTDB setup and
> > procedure followed which resulted in dumping core by smbd?
>
> Sorry, I was out for while, on business trips.
> Here are the requested information. I've one glusterFS volume (replicate)
> that I use to store samba shares. I use Samba-CTDB to balance access load
> between the two servers. The shares are published by Samba through André's
> VFS object. smb.conf is shared between the two servers (the glusterFS volume
> is also mounted on the two servers).
> There is no specific action to crash samba. As soon as you start samba, it
> starts to crash and restart.
Thanks for the info. I tried to reproduce with the following packages
recreating your Samba-CTDB-GlusterFS setup on two Ubuntu 15.04 VMs with
https://launchpad.net/~monotek/+archive/ubuntu/samba-vfs-glusterfs-3.7 enabled
and I couldn't see any coredumps from smbd. If possible, can you please upload
one of the coredump file?
samba/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64 [installed]
samba-common/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 all
[installed]
samba-common-bin/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installed]
samba-dbg/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installed]
samba-dev/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installed]
samba-doc/vivid,now 2:4.1.13+dfsg-4ubuntu3 all [installed,upgradable to:
2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1]
samba-dsdb-modules/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installed,automatic]
samba-libs/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installed]
samba-testsuite/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installed]
samba-vfs-modules/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installed]
smbclient/vivid,now 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.4vivid1 amd64
[installed]
ctdb/vivid,now 2.5.4+debian0-4 amd64 [installed]
ctdb-dbg/vivid,now 2.5.4+debian0-4 amd64 [installed]
ctdb-pcp-pmda/vivid,now 2.5.4+debian0-4 amd64 [installed]
glusterfs-client/now 3.7.4-ubuntu1~vivid2 amd64 [installed,local]
glusterfs-common/now 3.7.4-ubuntu1~vivid2 amd64 [installed,local]
glusterfs-dbg/now 3.7.4-ubuntu1~vivid2 amd64 [installed,local]
glusterfs-server/now 3.7.4-ubuntu1~vivid2 amd64 [installed,local]
--- Additional comment from Denis Lambolez on 2015-10-11 16:10 EDT ---
--- Additional comment from Denis Lambolez on 2015-10-11 16:41 EDT ---
--- Additional comment from Denis Lambolez on 2015-10-11 16:48:04 EDT ---
Hi,
I've uploaded the core dumps from both servers.
I have also uploaded the the samba log files for one of the client (catspc) on
both servers. I see a lot of 'Conversion error: Incomplete multibyte sequence'
in those log files. We may have a problem with multibyte strings and
accentuated characters in file name?
--- Additional comment from W. Andrew Denton on 2015-10-19 15:23:41 EDT ---
I'm also seeing the same Samba crash on CentOS7 when using VFS:
glusterfs-client-xlators-3.7.5-1.el7.x86_64
samba-4.1.12-23.el7_1.x86_64
samba-winbind-4.1.12-23.el7_1.x86_64
glusterfs-3.7.5-1.el7.x86_64
glusterfs-api-3.7.5-1.el7.x86_64
samba-libs-4.1.12-23.el7_1.x86_64
samba-common-4.1.12-23.el7_1.x86_64
samba-winbind-modules-4.1.12-23.el7_1.x86_64
samba-winbind-krb5-locator-4.1.12-23.el7_1.x86_64
samba-vfs-glusterfs-4.1.12-23.el7_1.x86_64
glusterfs-libs-3.7.5-1.el7.x86_64
glusterfs-fuse-3.7.5-1.el7.x86_64
[data]
path = /data
comment = Data
browsable = yes
writable = yes
kernel share modes = no
vfs objects = glusterfs
glusterfs:volfile_server = gluster
glusterfs:volume = customer0
glusterfs:logfile = /var/log/samba/gluster-customer0-data.log
glusterfs:loglevel = 7
Volume Name: customer0
Type: Distribute
Volume ID: 041f57b4-2c88-4dc9-8e89-afe333747e5a
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 192.168.254.28:/mnt/brick0/customer0
Brick2: 192.168.254.17:/mnt/brick0/customer0
Options Reconfigured:
features.inode-quota: off
features.quota: off
performance.readdir-ahead: on
--- Additional comment from Denis Lambolez on 2015-10-24 06:44:58 EDT ---
Hi,
Just tested with GlusterFS 3.7.5 and the new package from André:
- samba: 2:4.1.13+dfsg-4ubuntu3glusterfs3.7.5vivid1
- glusterfs: 3.7.5-ubuntu1~vivid1
Same behaviour. No change. Here is the dump of the samba panic action script.
Problem is still within libgfapi.so.0.
===============================================================
[2015/10/24 12:36:10.166977, 0] ../source3/lib/util.c:785(smb_panic_s3)
PANIC (pid 25385): internal error
[2015/10/24 12:36:10.167893, 0] ../source3/lib/util.c:896(log_stack_trace)
BACKTRACE: 33 stack frames:
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a)
[0x7f7cba9d319a]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20)
[0x7f7cba9d3280]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x2f)
[0x7f7cbc120dcf]
#3 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(+0x1afef) [0x7f7cbc120fef]
#4 /lib/x86_64-linux-gnu/libpthread.so.0(+0x10d10) [0x7f7cbc349d10]
#5 /usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_resolve_at+0x231)
[0x7f7caa8df7e1]
#6 /usr/lib/x86_64-linux-gnu/libgfapi.so.0(+0x14c4f) [0x7f7caa8e0c4f]
#7 /usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_resolve+0x11)
[0x7f7caa8e0cd1]
#8 /usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_chdir+0xa4) [0x7f7caa8dd804]
#9 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(vfs_ChDir+0x63)
[0x7f7cbbcf82a3]
#10 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(close_cnum+0x6b)
[0x7f7cbbd0d8fb]
#11
/usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(smbXsrv_tcon_disconnect+0x12c)
[0x7f7cbbd36b6c]
#12 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(+0x141f52)
[0x7f7cbbd36f52]
#13 /usr/lib/x86_64-linux-gnu/samba/libdbwrap.so.0(+0x4a03) [0x7f7cb76aba03]
#14 /usr/lib/x86_64-linux-gnu/samba/libdbwrap.so.0(+0x4b3f) [0x7f7cb76abb3f]
#15 /usr/lib/x86_64-linux-gnu/samba/libdbwrap.so.0(dbwrap_traverse+0xa)
[0x7f7cb76aa60a]
#16 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(+0x140a39)
[0x7f7cbbd35a39]
#17
/usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(smb1srv_tcon_disconnect_all+0x12)
[0x7f7cbbd37012]
#18 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(+0x144a14)
[0x7f7cbbd39a14]
#19 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(+0x144e9e)
[0x7f7cbbd39e9e]
#20
/usr/lib/x86_64-linux-gnu/samba/libsmbd_shim.so.0(exit_server_cleanly+0x12)
[0x7f7cba32ab32]
#21 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(+0x114ccc)
[0x7f7cbbd09ccc]
#22 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(run_events_poll+0x171)
[0x7f7cba9e97c1]
#23 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(+0x37a37) [0x7f7cba9e9a37]
#24 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d)
[0x7f7cb93b89ad]
#25 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0(smbd_process+0xc7c)
[0x7f7cbbd0b0cc]
#26 /usr/sbin/smbd(+0xa100) [0x7f7cbc787100]
#27 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(run_events_poll+0x171)
[0x7f7cba9e97c1]
#28 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(+0x37a37) [0x7f7cba9e9a37]
#29 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d)
[0x7f7cb93b89ad]
#30 /usr/sbin/smbd(main+0x1573) [0x7f7cbc783e43]
#31 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f7cb900ba40]
#32 /usr/sbin/smbd(_start+0x29) [0x7f7cbc784069]
--- Additional comment from Anoop C S on 2015-10-28 05:09:59 EDT ---
(In reply to Denis Lambolez from comment #18)
> Hi,
>
> I've uploaded the core dumps from both servers.
> I have also uploaded the the samba log files for one of the client (catspc)
> on both servers. I see a lot of 'Conversion error: Incomplete multibyte
> sequence' in those log files. We may have a problem with multibyte strings
> and accentuated characters in file name?
'Conversion error: Incomplete multibyte sequence'
Issue regarding logs (with DEBUG level 0) being flooded with above message got
fixed sometime back. Here's the commit
https://github.com/samba-team/samba/commit/1c60dc5c. AFAIK, this fix has not
yet back ported to Samba-4.1 stable branch, but is present in 4.2 stable
branch.
--- Additional comment from Anoop C S on 2015-11-18 06:11:52 EST ---
I came across this samba upstream bug which seems to be the cause for the crash
mentioned in the bug.
https://bugzilla.samba.org/show_bug.cgi?id=11115
This has been fixed with samba 4.1.18.
https://www.samba.org/samba/history/samba-4.1.18.html
--- Additional comment from André Bauer on 2015-11-18 06:48:39 EST ---
Thanks for the info.
I will try to backport this fix for my ubuntu packages next week.
--- Additional comment from André Bauer on 2015-11-23 14:32:54 EST ---
Uploaded package for Ubuntu Wily with backported 11115 fix.
https://launchpad.net/~monotek/+archive/ubuntu/samba-vfs-glusterfs-3.7
Please test :-)
--- Additional comment from Denis Lambolez on 2015-11-28 18:06:45 EST ---
OK. Tested during few days. Everything is ok. No more crash and no core dump
for smbd.
Current config is:
Package: glusterfs-server
Version: 3.7.6-ubuntu1~wily1
Package: samba-vfs-modules
Version: 2:4.1.17+dfsg-4ubuntu3glusterfs3.7.6wily2 -> from André
Thanks for solving this bug. Good job!
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1234877
[Bug 1234877] Samba crashes with 3.7.4 and VFS module
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=WlR15PzpeV&a=cc_unsubscribe
More information about the Bugs
mailing list