[Gluster-devel] Glusterfs core report.

Fri Apr 22 05:47:42 UTC 2016

----- Original Message -----
> From: "yang bin18" <yang.bin18 at zte.com.cn>
> To: "Vijay Bellur" <vbellur at redhat.com>
> Cc: "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Friday, April 22, 2016 6:38:07 AM
> Subject: Re: [Gluster-devel] Glusterfs core report.
> 
> There is some clue I think will be useful,thanks.

Did you check about the glibc corruption issue I mentioned in my previous mail? This seems like the same issue.

> 
> (gdb) up
> #1 0x00007f308ca06098 in abort () from /lib64/libc.so.6
> (gdb) up
> #2 0x00007f308ca45197 in __libc_message () from /lib64/libc.so.6
> (gdb) up
> #3 0x00007f308ca4c56d in _int_free () from /lib64/libc.so.6
> (gdb) up
> #4 0x00007f308096ebc1 in dht_local_wipe (this=0x7f308f306460,
> local=0x7f3080042880) at dht-helper.c:475
> 475 iobref_unref (local->rebalance.iobref);
> (gdb) p local->rebalance.iobref
> $1 = (struct iobref *) 0x7f30740022c0
> (gdb) p *local->rebalance.iobref
> $2 = {lock = 1, ref = 2, iobrefs = 0x7f307402f240, alloced = 16, used = 1}
> (gdb)
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks for your response.
> 
> We use glusterfs 3.6.7.
> 
> Sure ,We use Centos7.0.
> 
> Related log show bellow:
> 
> 143 [2016-04-13 06:33:54.236013] W [glusterfsd.c:1211:cleanup_and_exit] (-->
> 0-: received signum (15), shutting down
> 144 [2016-04-13 06:33:54.236081] I [fuse-bridge.c:5607:fini] 0-fuse:
> Unmounting '/var/lib/nova/mnt/a77401a594b06b2b56cc52ee61bb4 def'.
> 145 [2016-04-13 06:55:11.365558] I [MSGID: 100030] [glusterfsd.c:2035:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glu sterfs version 3.6.7
> (args: /usr/sbin/glusterfs --volfile-server=10.74.125.254
> --volfile-id=/xitos-volume /var/lib/nova/mnt/
> a77401a594b06b2b56cc52ee61bb4def)
> 146 [2016-04-13 06:55:11.402762] I [dht-shared.c:337:dht_init_regex]
> 0-xitos-volume-dht: using regex rsync-hash-regex = ^\.(.+)\ .[^.]+$
> 147 [2016-04-13 06:55:11.404087] I [client.c:2268:notify]
> 0-xitos-volume-client-0: parent translators are ready, attempting conn ect
> on transport
> 148 Final graph:
> 149
> +------------------------------------------------------------------------------+
> 150 1: volume xitos-volume-client-0
> 151 2: type protocol/client
> 152 3: option ping-timeout 42
> 153 4: option remote-host 10.74.125.247
> 154 5: option remote-subvolume /mnt/dht/volume
> 155 6: option transport-type socket
> 156 7: option send-gids true
> 157 8: end-volume
> 158 9:
> 159 10: volume xitos-volume-dht
> 160 11: type cluster/distribute
> 161 12: subvolumes xitos-volume-client-0
> 162 13: end-volume
> 163 14:
> 164 15: volume xitos-volume-write-behind
> 165 16: type performance/write-behind
> 166 17: subvolumes xitos-volume-dht
> 167 18: end-volume
> 168 19:
> 169 20: volume xitos-volume-read-ahead
> 170 21: type performance/read-ahead
> 171 22: subvolumes xitos-volume-write-behind
> 172 23: end-volume
> 173 24:
> 174 25: volume xitos-volume-io-cache
> 175 26: type performance/io-cache
> 176 27: subvolumes xitos-volume-read-ahead
> 177 28: end-volume
> 178 29:
> 179 30: volume xitos-volume-quick-read
> 180 31: type performance/quick-read
> 181 32: subvolumes xitos-volume-io-cache
> 182 33: end-volume
> 183 34:
> 184 35: volume xitos-volume-open-behind
> 185 36: type performance/open-behind
> 186 37: subvolumes xitos-volume-quick-read
> 187 38: end-volume
> 188 39:
> 189 40: volume xitos-volume-md-cache
> 190 41: type performance/md-cache
> 191 42: subvolumes xitos-volume-open-behind
> 192 43: end-volume
> 193 44:
> 194 45: volume xitos-volume
> 195 46: type debug/io-stats
> 196 47: option latency-measurement off
> 197 48: option count-fop-hits off
> 198 49: subvolumes xitos-volume-md-cache
> 199 50: end-volume
> 200 51:
> 201 52: volume meta-autoload
> 202 53: type meta
> 203 54: subvolumes xitos-volume
> 204 55: end-volume
> 205 56:
> 206
> +------------------------------------------------------------------------------+
> 207 [2016-04-13 06:55:11.408891] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-xitos-volume-client-0: changing port to 49152 (from 0)
> 208 [2016-04-13 06:55:11.413254] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-xitos-volume-client-0: Using Pro gram GlusterFS 3.3, Num (1298437),
> Version (330)
> 209 [2016-04-13 06:55:11.413766] I
> [client-handshake.c:1200:client_setvolume_cbk] 0-xitos-volume-client-0:
> Connected to xitos-vo lume-client-0, attached to remote volume
> '/mnt/dht/volume'.
> 210 [2016-04-13 06:55:11.413795] I
> [client-handshake.c:1210:client_setvolume_cbk] 0-xitos-volume-client-0:
> Server and Client lk- version numbers are not same, reopening the fds
> 211 [2016-04-13 06:55:11.420494] I [fuse-bridge.c:5086:fuse_graph_setup]
> 0-fuse: switched to graph 0
> 212 [2016-04-13 06:55:11.420691] I
> [client-handshake.c:188:client_set_lk_version_cbk] 0-xitos-volume-client-0:
> Server lk version = 1
> 213 [2016-04-13 06:55:11.420921] I [fuse-bridge.c:4015:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterf s 7.22 kernel
> 7.22
> 214 pending frames:
> 215 frame : type(0) op(0)
> 216 frame : type(0) op(0)
> 217 frame : type(0) op(0)
> 218 frame : type(0) op(0)
> 219 frame : type(0) op(0)
> 220 frame : type(0) op(0)
> 221 frame : type(0) op(0)
> 222 patchset: git://git.gluster.com/glusterfs.git
> 223 signal received: 6
> 224 time of crash:
> 225 2016-04-13 08:15:37
> 226 configuration details:
> 227 argp 1
> 228 backtrace 1
> 229 dlfcn 1
> 230 libpthread 1
> 231 llistxattr 1
> 232 setfsid 1
> 233 spinlock 1
> 234 epoll.h 1
> 235 xattr.h 1
> 236 st_atim.tv_nsec 1
> 237 package-string: glusterfs 3.6.7
> 238 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f308d9e93d2]
> 239 /lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f308da008bd]
> 240 /lib64/libc.so.6(+0x35a00)[0x7f308ca04a00]
> 241 /lib64/libc.so.6(gsignal+0x39)[0x7f308ca04989]
> 242 /lib64/libc.so.6(abort+0x148)[0x7f308ca06098]
> 243 /lib64/libc.so.6(+0x76197)[0x7f308ca45197]
> 244 /lib64/libc.so.6(+0x7d56d)[0x7f308ca4c56d]
> 245
> /usr/lib64/glusterfs/3.6.7/xlator/cluster/distribute.so(dht_local_wipe+0x151)[0x7f308096ebc1]
> 246
> /usr/lib64/glusterfs/3.6.7/xlator/cluster/distribute.so(dht_writev_cbk+0x19d)[0x7f308099b9fd]
> 247
> /usr/lib64/glusterfs/3.6.7/xlator/protocol/client.so(client3_3_writev_cbk+0x672)[0x7f3080be4512]
> 248 /lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7f308d7bd100]
> 249 /lib64/libgfrpc.so.0(rpc_clnt_notify+0x174)[0x7f308d7bd374]
> 250 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f308d7b92c3]
> 251
> /usr/lib64/glusterfs/3.6.7/rpc-transport/socket.so(+0x87a0)[0x7f3082ac17a0]
> 252
> /usr/lib64/glusterfs/3.6.7/rpc-transport/socket.so(+0xaf94)[0x7f3082ac3f94]
> 253 /lib64/libglusterfs.so.0(+0x768c2)[0x7f308da3e8c2]
> 254 /usr/sbin/glusterfs(main+0x502)[0x7f308de92fe2]
> 255 /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f308c9f0af5]
> 256 /usr/sbin/glusterfs(+0x6381)[0x7f308de93381]
> 257 ---------
> 258 [2016-04-13 08:36:22.385866] I [MSGID: 100030] [glusterfsd.c:2035:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glu sterfs version 3.6.7
> (args: /usr/sbin/glusterfs --volfile-server=10.74.125.254
> --volfile-id=/xitos-volume /var/lib/nova/mnt/
> a77401a594b06b2b56cc52ee61bb4def)
> 259 [2016-04-13 08:36:22.409856] I [dht-shared.c:337:dht_init_regex]
> 0-xitos-volume-dht: using regex rsync-hash-regex = ^\.(.+)\ .[^.]+$
> 260 [2016-04-13 08:36:22.411477] I [client.c:2268:notify]
> 0-xitos-volume-client-0: parent translators are ready, attempting conn ect
> on transport
> 261 Final graph:
> 262
> +------------------------------------------------------------------------------+
> 263 1: volume xitos-volume-client-0
> 264 2: type protocol/client
> 265 3: option ping-timeout 42
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Fri, Apr 15, 2016 at 2:43 AM, yang.bin18 at zte.com.cn <
> yang.bin18 at zte.com.cn > wrote:
> Glusterfs core when mounting. here is the backtree.
> 
> 
> Program terminated with signal 6, Aborted.
> #0 0x00007f308ca04989 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install glibc-2.17-55.el7.x86_64
> keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.11.3-49.el7.x86_64
> libcom_err-1.42.9-4.el7.x86_64 libgcc-4.8.2-16.el7.x86_64
> libselinux-2.2.2-6.el7.x86_64 openssl-libs-1.0.1e-34.el7.7.x86_64
> pcre-8.32-12.el7.x86_64 sssd-client-1.11.2-65.el7.x86_64
> xz-libs-5.1.2-8alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
> (gdb) bt
> #0 0x00007f308ca04989 in raise () from /lib64/libc.so.6
> #1 0x00007f308ca06098 in abort () from /lib64/libc.so.6
> #2 0x00007f308ca45197 in __libc_message () from /lib64/libc.so.6
> #3 0x00007f308ca4c56d in _int_free () from /lib64/libc.so.6
> #4 0x00007f308096ebc1 in dht_local_wipe (this=0x7f308f306460,
> local=0x7f3080042880) at dht-helper.c:475
> #5 0x00007f308099b9fd in dht_writev_cbk (frame=0x7f308bc11e5c,
> cookie=<optimized out>, this=<optimized out>, op_ret=131072,
> op_errno=0, prebuf=<optimized out>, postbuf=0x7fff2020c870, xdata=0x0) at
> dht-inode-write.c:84
> #6 0x00007f3080be4512 in client3_3_writev_cbk (req=<optimized out>,
> iov=<optimized out>, count=<optimized out>,
> myframe=0x7f308bc116f8) at client-rpc-fops.c:856
> #7 0x00007f308d7bd100 in rpc_clnt_handle_reply
> (clnt=clnt at entry=0x7f308f3292d0, pollin=pollin at entry=0x7f308f39bb10)
> at rpc-clnt.c:763
> #8 0x00007f308d7bd374 in rpc_clnt_notify (trans=<optimized out>,
> mydata=0x7f308f329300, event=<optimized out>,
> data=0x7f308f39bb10) at rpc-clnt.c:891
> #9 0x00007f308d7b92c3 in rpc_transport_notify
> (this=this at entry=0x7f308f35f3f0,
> event=event at entry=RPC_TRANSPORT_MSG_RECEIVED,
> data=data at entry=0x7f308f39bb10) at rpc-transport.c:516
> #10 0x00007f3082ac17a0 in socket_event_poll_in
> (this=this at entry=0x7f308f35f3f0) at socket.c:2234
> #11 0x00007f3082ac3f94 in socket_event_handler (fd=<optimized out>, idx=1,
> data=data at entry=0x7f308f35f3f0, poll_in=1, poll_out=0,
> poll_err=0) at socket.c:2347
> #12 0x00007f308da3e8c2 in event_dispatch_epoll_handler (i=<optimized out>,
> events=0x7f308f2febd0, event_pool=0x7f308f2b76c0)
> at event-epoll.c:384
> #13 event_dispatch_epoll (event_pool=0x7f308f2b76c0) at event-epoll.c:445
> #14 0x00007f308de92fe2 in main (argc=4, argv=0x7fff2020de78) at
> glusterfsd.c:2060
> 
> 
> Thanks for the report. What version of gluster is being used here? I assume
> this is on Centos 7.
> 
> Would it be possible to share the client log file?
> 
> Regards,
> Vijay
> 
> --------------------------------------------------------
> ZTE Information Security Notice: The information contained in this mail (and
> any attachment transmitted herewith) is privileged and confidential and is
> intended for the exclusive use of the addressee(s).  If you are not an
> intended recipient, any disclosure, reproduction, distribution or other
> dissemination or use of the information contained is strictly prohibited.
> If you have received this mail in error, please delete it and notify us
> immediately.
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel