<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
A took the easy way and changed KillUserProcesse=1, for now.
<div class="">We are considering this a bug so we have reported this to systemd developers, which closed the report pointing to our old redhat version of systemd and to RedHat support where we now have support case.</div>
<div class=""><br class="">
</div>
<div class="">Cheers</div>
<div class="">Gabbe</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">1 juni 2017 kl. 19:31 skrev Niels de Vos <<a href="mailto:ndevos@redhat.com" class="">ndevos@redhat.com</a>>:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On
Thu, Jun 01, 2017 at 01:52:23PM +0000, Gabriel Lindeborg wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
This has been solved, as far as we can tell.<br class="">
<br class="">
Problem was with KillUserProcesses=1 in logind.conf. This has shown to<br class="">
kill mounts made using mount -a booth by root and by any user with<br class="">
sudo at session logout.<br class="">
</blockquote>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Ah,
yes, that could well be the cause of the problem.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
Hope this will anybody else who run into this.<br class="">
</blockquote>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Care
to share how you solved it? Just disabling the option might not be</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">the
most suitable approach. Did you convert it to systemd.mount units,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">or
maybe setup automounting with x-systemd.automount or autofs? Were</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">there
considerations that made you choose one solution over an other?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Thanks!</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Niels</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
Thanks 4 all your help and<br class="">
cheers<br class="">
Gabbe<br class="">
<br class="">
1 juni 2017 kl. 09:24 skrev Gabriel Lindeborg <<a href="mailto:gabriel.lindeborg@svenskaspel.se" class="">gabriel.lindeborg@svenskaspel.se</a><<a href="mailto:gabriel.lindeborg@svenskaspel.se" class="">mailto:gabriel.lindeborg@svenskaspel.se</a>>>:<br class="">
<br class="">
All four clients did run 3.10.2 as well<br class="">
<br class="">
The volumes has been running fine until we upgraded to 3.10, when we hit some issues with port mismatches. We restarted all the volumes, the servers and the clients and now hit this issue.<br class="">
We’ve since backed up the files, remove the volumes, removed the bricks, removed gluster, installed glusterfs 3.7.20, created new volumes on new bricks, restored the files and still hit the same issue at clients on the nodes that also runs the servers. We’ve
got to clients on connected to one of the volumes that has been working fine all the time.<br class="">
<br class="">
This is the debug logs from one of the mount as the client gets disconnected:<br class="">
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]" repeated 26 times between [2017-05-31 13:48:51.680757] and [2017-05-31 13:50:46.325368]<br class="">
/DAEMON/DEBUG [2017-05-31T15:50:50.589272+02:00] [] [] [logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present<br class="">
/DAEMON/DEBUG [2017-05-31T15:50:50.589520+02:00] [] [] [logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5<br class="">
[2017-05-31 13:50:51.908797] D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]<br class="">
/DAEMON/DEBUG [2017-05-31T15:51:24.592190+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:51:24.592469+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib<br class="">
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c] ))))) 0-: 10.3.48.179:49155: ping timer event
already remove<br class="">
d<br class="">
/DAEMON/DEBUG [2017-05-31T15:51:26.325230+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib<br class="">
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c] ))))) 0-: 10.3.48.180:49155: ping timer event
already remove<br class="">
d<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:08.595536+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:08.595735+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:12.059895+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib<br class="">
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c] ))))) 0-: 10.3.48.179:49155: ping timer event
already remove<br class="">
d<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:12.060170+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib<br class="">
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c] ))))) 0-: 10.3.48.180:49155: ping timer event
already remove<br class="">
d<br class="">
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]" repeated 26 times between [2017-05-31 13:50:51.908797] and [2017-05-31 13:52:46.326381]<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:50.598987+02:00] [] [] [logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:50.599226+02:00] [] [] [logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5<br class="">
[2017-05-31 13:52:52.138032] D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:54.599435+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:54.599718+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:56.325482+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib<br class="">
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c] ))))) 0-: 10.3.48.179:49155: ping timer event
already remove<br class="">
d<br class="">
/DAEMON/DEBUG [2017-05-31T15:52:56.325731+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib<br class="">
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c] ))))) 0-: 10.3.48.180:49155: ping timer event
already remove<br class="">
d<br class="">
/DAEMON/DEBUG [2017-05-31T15:53:38.603305+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:53:38.603533+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:53:42.226123+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib<br class="">
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c] ))))) 0-: 10.3.48.179:49155: ping timer event
already remove<br class="">
d<br class="">
/DAEMON/DEBUG [2017-05-31T15:53:42.226345+02:00] [] [] [rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib<br class="">
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (--> /usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c] ))))) 0-: 10.3.48.180:49155: ping timer event
already remove<br class="">
d<br class="">
/DAEMON/DEBUG [2017-05-31T15:54:24.607225+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
/DAEMON/DEBUG [2017-05-31T15:54:24.607479+02:00] [] [] [rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)<br class="">
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]" repeated 20 times between [2017-05-31 13:52:52.138032] and [2017-05-31 13:54:22.302116]<br class="">
[2017-05-31 13:54:25.284676] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f36b20c6dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f36b3740915] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f36b374078b] ) 0-: received
signum (15), shutting down<br class="">
/DAEMON/DEBUG [2017-05-31T15:54:25.285804+02:00] [] [] [logging.c:1766:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 1 extra log messages<br class="">
/DAEMON/DEBUG [2017-05-31T15:54:25.286570+02:00] [] [] [logging.c:1769:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 1 extra log messages<br class="">
/DAEMON/DEBUG [2017-05-31T15:54:25.287431+02:00] [] [] [glusterfsd-mgmt.c:2361:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given<br class="">
/DAEMON/INFO [2017-05-31T15:54:25.287785+02:00] [] [] [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/mnt/gluster/mule’.<br class="">
<br class="">
Cheers<br class="">
Gabbe<br class="">
31 maj 2017 kl. 17:03 skrev Sunil Kumar Heggodu Gopala Acharya <<a href="mailto:sheggodu@redhat.com" class="">sheggodu@redhat.com</a><<a href="mailto:sheggodu@redhat.com" class="">mailto:sheggodu@redhat.com</a>>>:<br class="">
<br class="">
Hi Gabriel,<br class="">
<br class="">
What is the version of gluster you are running on client? Also, please share the steps you followed to hit the issue.<br class="">
<br class="">
Regards,<br class="">
SUNIL KUMAR ACHARYA<br class="">
SENIOR SOFTWARE ENGINEER<br class="">
Red Hat<br class="">
<br class="">
T: +91-8067935170<br class="">
<br class="">
[<a href="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" class="">https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png</a>]<br class="">
TRIED. TESTED. TRUSTED.<br class="">
<br class="">
<br class="">
On Wed, May 31, 2017 at 12:10 PM, Gabriel Lindeborg <<a href="mailto:gabriel.lindeborg@svenskaspel.se" class="">gabriel.lindeborg@svenskaspel.se</a>> wrote:<br class="">
Hello again<br class="">
<br class="">
The volumes are old, since version 3.6 if I remember correctly…<br class="">
These är det 40 last rows of the mnt-loggs go all the mounts<br class="">
<br class="">
==> /var/log/glusterfs/mnt-gluster-alfresco.log <==<br class="">
/DAEMON/INFO [2017-05-31T07:55:12.787035+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.153779+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.160995+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.167075+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.171872+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.427057+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.432239+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.439572+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.444286+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.035710+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.042366+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.049401+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.054483+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.524103+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.540314+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.543748+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.558459+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.623109+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.643979+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.648717+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.662941+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.268446+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.284765+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.287503+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.302461+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.408757+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.415831+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.421736+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.421887+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.431820+02:00] [] []<br class="">
[2017-05-31 06:09:54.514841] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting local read_child alfresco-client-2<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.179031+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.186811+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.194886+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.195062+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.205582+02:00] [] []<br class="">
[2017-05-31 06:11:14.513620] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting local read_child alfresco-client-2<br class="">
[2017-05-31 06:21:21.579748] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa1d2e4cdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa1d44e4fd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa1d44e4dfb] ) 0-: received
signum (15), shutting down<br class="">
/DAEMON/INFO [2017-05-31T08:21:21.580626+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:21:21.581440+02:00] [] []<br class="">
<br class="">
==> /var/log/glusterfs/mnt-gluster-c1_2.log <==<br class="">
82: type meta<br class="">
83: subvolumes c1<br class="">
84: end-volume<br class="">
85:<br class="">
+------------------------------------------------------------------------------+<br class="">
/DAEMON/ERR [2017-05-24T10:39:11.130038+02:00] [] []<br class="">
[2017-05-24 08:39:11.130638] E [MSGID: 114058] [client-handshake.c:1538:client_query_portmap_cbk] 2-c1-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.<br class="">
[2017-05-24 08:39:11.130693] I [MSGID: 114018] [client.c:2276:client_rpc_notify] 2-c1-client-1: disconnected from c1-client-1. Client process will keep trying to connect to glusterd until brick's port is available<br class="">
[2017-05-24 08:39:11.130711] E [MSGID: 108006] [afr-common.c:4781:afr_notify] 2-c1-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.<br class="">
/DAEMON/INFO [2017-05-24T10:39:11.471603+02:00] [] []<br class="">
/DAEMON/ERR [2017-05-24T10:39:11.473783+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-24T10:39:14.479153+02:00] [] []<br class="">
/DAEMON/ERR [2017-05-24T10:39:14.481430+02:00] [] []<br class="">
[2017-05-24 08:39:14.513688] I [MSGID: 108006] [afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up<br class="">
/DAEMON/INFO [2017-05-24T10:39:14.513712+02:00] [] []<br class="">
[2017-05-24 08:39:14.513858] I [MSGID: 114021] [client.c:2361:notify] 0-c1-client-2: current graph is no longer active, destroying rpc_client<br class="">
[2017-05-24 08:39:14.513879] I [MSGID: 114021] [client.c:2361:notify] 0-c1-client-3: current graph is no longer active, destroying rpc_client<br class="">
[2017-05-24 08:39:14.513933] I [MSGID: 108006] [afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up<br class="">
/DAEMON/WARNING [2017-05-24T10:39:14.513997+02:00] [] []<br class="">
/DAEMON/WARNING [2017-05-24T10:39:14.514161+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-24T10:39:18.485909+02:00] [] []<br class="">
/DAEMON/ERR [2017-05-24T10:39:18.488664+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-24T10:39:23.737962+02:00] [] []<br class="">
[2017-05-24 08:39:23.740893] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 2-c1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br class="">
[2017-05-24 08:39:23.742339] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-0: Connected to c1-client-0, attached to remote volume '/gluster/c1/brick'.<br class="">
[2017-05-24 08:39:23.742359] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-0: Server and Client lk-version numbers are not same, reopening the fds<br class="">
[2017-05-24 08:39:23.742437] I [MSGID: 108005] [afr-common.c:4756:afr_notify] 2-c1-replicate-0: Subvolume 'c1-client-0' came back up; going online.<br class="">
[2017-05-24 08:39:23.743083] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-0: Server lk version = 1<br class="">
[2017-05-24 08:39:24.524310] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local read_child c1-client-0<br class="">
[2017-05-24 08:39:24.524414] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-c1-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0<br class="">
/DAEMON/INFO [2017-05-24T10:39:26.883379+02:00] [] []<br class="">
[2017-05-24 08:39:26.885753] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 2-c1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br class="">
[2017-05-24 08:39:26.887336] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-1: Connected to c1-client-1, attached to remote volume '/gluster/c1/brick'.<br class="">
[2017-05-24 08:39:26.887353] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-1: Server and Client lk-version numbers are not same, reopening the fds<br class="">
[2017-05-24 08:39:26.887479] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-1: Server lk version = 1<br class="">
[2017-05-24 08:39:34.515063] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local read_child c1-client-0<br class="">
The message "I [MSGID: 108006] [afr-common.c:4923:afr_local_init] 0-c1-replicate-0: no subvolumes up" repeated 4 times between [2017-05-24 08:38:24.514047] and [2017-05-24 08:39:04.514186]<br class="">
/DAEMON/INFO [2017-05-24T10:40:56.356298+02:00] [] []<br class="">
[2017-05-24 08:40:56.356694] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7feba7f55dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7feba95edf85] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7feba95eddfb] ) 0-: received
signum (15), shutting down<br class="">
/DAEMON/INFO [2017-05-24T10:40:56.356773+02:00] [] []<br class="">
<br class="">
==> /var/log/glusterfs/mnt-gluster-c1.log <==<br class="">
/DAEMON/INFO [2017-05-31T07:55:12.781267+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:12.786589+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.153964+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.161317+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.167226+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.172425+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.425676+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.433114+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.439753+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.444685+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.034185+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.042732+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.049094+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.053587+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.525624+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.540864+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.546015+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.546167+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.549146+02:00] [] []<br class="">
[2017-05-31 05:56:14.515062] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-c1-replicate-0: selecting local read_child c1-client-0<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.558847+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.622583+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.644252+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.648540+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.662541+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.269583+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.284254+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.291514+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.303324+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.408574+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.415968+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.421254+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.431030+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.179194+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.187103+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.194300+02:00] [] []<br class="">
[2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f7dad36fdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f7daea07fd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f7daea07dfb] ) 0-: received
signum (15), shutting down<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.205913+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:21:21.581037+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:21:21.581789+02:00] [] []<br class="">
<br class="">
==> /var/log/glusterfs/mnt-gluster-c2.log <==<br class="">
66: end-volume<br class="">
67:<br class="">
68: volume c2-md-cache<br class="">
69: type performance/md-cache<br class="">
70: subvolumes c2-open-behind<br class="">
71: end-volume<br class="">
72:<br class="">
73: volume c2<br class="">
74: type debug/io-stats<br class="">
75: option log-level INFO<br class="">
76: option latency-measurement off<br class="">
77: option count-fop-hits off<br class="">
78: subvolumes c2-md-cache<br class="">
79: end-volume<br class="">
80:<br class="">
81: volume meta-autoload<br class="">
82: type meta<br class="">
83: subvolumes c2<br class="">
84: end-volume<br class="">
85:<br class="">
+------------------------------------------------------------------------------+<br class="">
/DAEMON/INFO [2017-05-24T09:20:01.247296+02:00] [] []<br class="">
[2017-05-24 07:20:01.249248] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-c2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br class="">
[2017-05-24 07:20:01.249950] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-0: Connected to c2-client-0, attached to remote volume '/gluster/c1/newbrick'.<br class="">
[2017-05-24 07:20:01.249980] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-0: Server and Client lk-version numbers are not same, reopening the fds<br class="">
[2017-05-24 07:20:01.250050] I [MSGID: 108005] [afr-common.c:4756:afr_notify] 0-c2-replicate-0: Subvolume 'c2-client-0' came back up; going online.<br class="">
[2017-05-24 07:20:01.250557] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-0: Server lk version = 1<br class="">
[2017-05-24 07:20:01.251421] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-c2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br class="">
[2017-05-24 07:20:01.252016] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-1: Connected to c2-client-1, attached to remote volume '/gluster/c1/newbrick'.<br class="">
[2017-05-24 07:20:01.252055] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-1: Server and Client lk-version numbers are not same, reopening the fds<br class="">
[2017-05-24 07:20:01.253072] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-1: Server lk version = 1<br class="">
/DAEMON/INFO [2017-05-24T09:20:01.253273+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-24T09:20:01.253547+02:00] [] []<br class="">
[2017-05-24 07:20:01.254056] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-c2-replicate-0: selecting local read_child c2-client-0<br class="">
[2017-05-24 07:20:01.254268] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-c2-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0<br class="">
/DAEMON/INFO [2017-05-24T10:39:11.116695+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-24T10:39:11.118327+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-24T10:40:47.577696+02:00] [] []<br class="">
[2017-05-24 08:40:47.578001] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f3eaabefdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f3eac287f85] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f3eac287dfb] ) 0-: received
signum (15), shutting down<br class="">
/DAEMON/INFO [2017-05-24T10:40:47.578100+02:00] [] []<br class="">
<br class="">
==> /var/log/glusterfs/mnt-gluster-cdn.log <==<br class="">
/DAEMON/INFO [2017-05-31T07:55:12.785775+02:00] [] []<br class="">
[2017-05-31 05:55:14.515355] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local read_child cdn-client-2<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.152981+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.161155+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.166602+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.171520+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.426866+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.432536+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.439418+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.443795+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.035303+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.042561+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.048928+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.053195+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.525267+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.540464+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.545703+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.559227+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.622792+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.644113+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.649031+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.649173+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.653281+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.663382+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.269801+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.283977+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.288305+02:00] [] []<br class="">
[2017-05-31 05:56:24.514977] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local read_child cdn-client-2<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.302883+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.408914+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.415563+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.421558+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.432374+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.178859+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.186967+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.193439+02:00] [] []<br class="">
[2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa420a29dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa4220c1fd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa4220c1dfb] ) 0-: received
signum (15), shutting down<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.204701+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:21:21.581209+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:21:21.581945+02:00] [] []<br class="">
<br class="">
==> /var/log/glusterfs/mnt-gluster-mule.log <==<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.154147+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.161457+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.168560+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.168765+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:18.172803+02:00] [] []<br class="">
[2017-05-31 05:55:24.517077] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local read_child mule-client-2<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.426664+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.432816+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.439270+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:46.443344+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.035528+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.042893+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.049247+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:55:52.054006+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.525469+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.540732+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.545856+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:13.559699+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.622955+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.644386+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.648877+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:18.663766+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.269958+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.284895+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.291677+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.291845+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.294037+02:00] [] []<br class="">
[2017-05-31 05:56:34.516246] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local read_child mule-client-2<br class="">
/DAEMON/INFO [2017-05-31T07:56:26.303715+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.408417+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.416098+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.421403+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:09:49.431421+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.178644+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.187364+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.193590+02:00] [] []<br class="">
[2017-05-31 06:21:21.579746] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fbd1beafdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fbd1d547fd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbd1d547dfb] ) 0-: received
signum (15), shutting down<br class="">
/DAEMON/INFO [2017-05-31T08:11:10.205040+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:21:21.580846+02:00] [] []<br class="">
/DAEMON/INFO [2017-05-31T08:21:21.581623+02:00] [] []<br class="">
<br class="">
And these are 40 last rows of the brick<br class="">
==> /var/log/glusterfs/bricks/gluster-alfresco-brick.log <==<br class="">
[2017-05-31 05:52:19.117394] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0<br class="">
[2017-05-31 05:52:19.117469] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0<br class="">
[2017-05-31 05:52:43.020403] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:52:43.027816] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0<br class="">
[2017-05-31 05:52:43.027931] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0<br class="">
[2017-05-31 05:56:48.604780] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:57:50.817223] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:57:50.825987] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0<br class="">
[2017-05-31 05:57:50.826138] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0<br class="">
[2017-05-31 05:58:25.746161] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:58:25.755174] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0<br class="">
[2017-05-31 05:58:25.755279] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0<br class="">
[2017-05-31 06:02:47.958784] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:02:47.976543] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0<br class="">
[2017-05-31 06:02:47.976683] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0<br class="">
[2017-05-31 06:07:53.126343] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:07:53.133904] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0<br class="">
[2017-05-31 06:07:53.133998] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0<br class="">
[2017-05-31 06:12:56.138792] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:12:56.146253] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0<br class="">
[2017-05-31 06:12:56.146351] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0<br class="">
[2017-05-31 06:16:22.552274] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0<br class="">
[2017-05-31 06:16:22.552428] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0<br class="">
[2017-05-31 06:17:54.775497] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:17:54.792370] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0<br class="">
[2017-05-31 06:17:54.792446] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0<br class="">
[2017-05-31 06:21:21.582947] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0<br class="">
[2017-05-31 06:21:21.583181] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0<br class="">
[2017-05-31 06:21:42.714049] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:21:42.722227] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0<br class="">
[2017-05-31 06:21:42.722320] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0<br class="">
[2017-05-31 06:22:35.445697] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:22:35.453208] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0<br class="">
[2017-05-31 06:22:35.453334] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0<br class="">
[2017-05-31 06:22:58.071308] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:22:58.078524] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0<br class="">
[2017-05-31 06:22:58.078608] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0<br class="">
[2017-05-31 06:28:06.645148] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:28:06.652828] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0<br class="">
[2017-05-31 06:28:06.652942] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0<br class="">
<br class="">
==> /var/log/glusterfs/bricks/gluster-c1-brick.log <==<br class="">
[2017-05-31 05:49:00.793792] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-39066-2017/05/31-05:29:48:927855-c1-client-0-0-0<br class="">
[2017-05-31 05:52:36.512431] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:52:36.522623] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0<br class="">
[2017-05-31 05:52:36.522723] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0<br class="">
[2017-05-31 05:56:13.544120] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br class="">
[2017-05-31 05:56:48.796772] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:58:39.772391] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:58:39.782681] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0<br class="">
[2017-05-31 05:58:39.782820] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0<br class="">
[2017-05-31 06:03:45.195471] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:03:45.205497] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0<br class="">
[2017-05-31 06:03:45.205610] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0<br class="">
[2017-05-31 06:08:45.229186] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:08:45.239068] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0<br class="">
[2017-05-31 06:08:45.239173] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0<br class="">
[2017-05-31 06:13:51.433086] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:13:51.443161] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0<br class="">
[2017-05-31 06:13:51.443257] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0<br class="">
[2017-05-31 06:16:22.552265] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0<br class="">
[2017-05-31 06:16:22.552418] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0<br class="">
[2017-05-31 06:18:55.232358] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:18:55.242110] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0<br class="">
[2017-05-31 06:18:55.242188] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0<br class="">
[2017-05-31 06:21:21.586024] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0<br class="">
[2017-05-31 06:21:21.586153] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0<br class="">
[2017-05-31 06:21:34.443201] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:21:34.453810] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0<br class="">
[2017-05-31 06:21:34.453897] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0<br class="">
[2017-05-31 06:23:55.706224] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:23:55.716126] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0<br class="">
[2017-05-31 06:23:55.716221] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0<br class="">
[2017-05-31 06:28:58.213831] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:28:58.223474] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0<br class="">
[2017-05-31 06:28:58.223557] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0<br class="">
[2017-05-31 06:33:07.622832] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:33:07.642391] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0<br class="">
[2017-05-31 06:33:07.642520] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0<br class="">
[2017-05-31 06:34:07.776928] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:34:07.787116] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0<br class="">
[2017-05-31 06:34:07.787200] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0<br class="">
<br class="">
==> /var/log/glusterfs/bricks/gluster-cdn-brick.log <==<br class="">
[2017-05-31 05:50:18.876202] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-40008-2017/05/31-05:50:18:867052-cdn-client-2-0-0<br class="">
[2017-05-31 05:55:09.869166] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:55:09.877790] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0<br class="">
[2017-05-31 05:55:09.877898] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0<br class="">
[2017-05-31 05:55:20.944369] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 05:55:20.952275] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0<br class="">
[2017-05-31 05:55:20.952379] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0<br class="">
[2017-05-31 05:56:48.701158] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:00:20.651639] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:00:20.659659] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0<br class="">
[2017-05-31 06:00:20.659782] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0<br class="">
[2017-05-31 06:01:17.578233] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:01:17.587267] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0<br class="">
[2017-05-31 06:01:17.587370] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0<br class="">
[2017-05-31 06:05:25.961313] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:05:25.976281] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0<br class="">
[2017-05-31 06:05:25.976378] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0<br class="">
[2017-05-31 06:10:24.626906] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:10:24.634914] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0<br class="">
[2017-05-31 06:10:24.635016] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0<br class="">
[2017-05-31 06:15:21.944215] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:15:21.952554] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0<br class="">
[2017-05-31 06:15:21.952655] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0<br class="">
[2017-05-31 06:16:22.552086] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0<br class="">
[2017-05-31 06:16:22.552203] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cdn-server: fd cleanup on /test2/nyx/nyxclient/assets.json<br class="">
[2017-05-31 06:16:22.552416] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0<br class="">
[2017-05-31 06:20:25.255523] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:20:25.263860] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0<br class="">
[2017-05-31 06:20:25.263963] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0<br class="">
[2017-05-31 06:21:21.584357] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0<br class="">
[2017-05-31 06:21:21.584449] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0<br class="">
[2017-05-31 06:21:37.944798] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:21:37.952580] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0<br class="">
[2017-05-31 06:21:37.952712] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0<br class="">
[2017-05-31 06:25:27.632381] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:25:27.640591] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0<br class="">
[2017-05-31 06:25:27.640721] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0<br class="">
[2017-05-31 06:35:42.166035] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:35:42.173662] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0<br class="">
[2017-05-31 06:35:42.173752] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0<br class="">
<br class="">
==> /var/log/glusterfs/bricks/gluster-mule-brick.log <==<br class="">
[2017-05-31 06:26:30.580063] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-38988-2017/05/31-06:26:30:551135-mule-client-2-0-0<br class="">
[2017-05-31 06:27:31.269224] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:27:31.276755] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0<br class="">
[2017-05-31 06:27:31.276859] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0<br class="">
[2017-05-31 06:28:33.119250] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:28:33.126230] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0<br class="">
[2017-05-31 06:28:33.126328] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0<br class="">
[2017-05-31 06:29:35.271637] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:29:35.279799] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0<br class="">
[2017-05-31 06:29:35.279901] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0<br class="">
[2017-05-31 06:30:34.816144] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:30:34.824228] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0<br class="">
[2017-05-31 06:30:34.824319] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0<br class="">
[2017-05-31 06:30:36.304718] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:30:36.311898] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0<br class="">
[2017-05-31 06:30:36.311998] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0<br class="">
[2017-05-31 06:31:34.929107] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:31:34.936764] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0<br class="">
[2017-05-31 06:31:34.936843] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0<br class="">
[2017-05-31 06:31:41.145828] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:31:41.153317] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0<br class="">
[2017-05-31 06:31:41.153427] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0<br class="">
[2017-05-31 06:32:36.230320] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:32:36.239295] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0<br class="">
[2017-05-31 06:32:36.239420] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0<br class="">
[2017-05-31 06:32:36.481688] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:32:36.489207] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0<br class="">
[2017-05-31 06:32:36.489327] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0<br class="">
[2017-05-31 06:33:38.413891] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:33:38.422092] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0<br class="">
[2017-05-31 06:33:38.422188] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0<br class="">
[2017-05-31 06:33:41.618786] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:33:41.626214] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0<br class="">
[2017-05-31 06:33:41.626310] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0<br class="">
[2017-05-31 06:34:37.654479] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:34:37.661939] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0<br class="">
[2017-05-31 06:34:37.662043] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms01.horisont.svenskaspel.se/" class="">tms01.horisont.svenskaspel.se</a>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0<br class="">
[2017-05-31 06:36:12.557364] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0
(version: 3.10.2)<br class="">
[2017-05-31 06:36:12.564751] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0<br class="">
[2017-05-31 06:36:12.564861] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection<span class="Apple-converted-space"> </span><a href="http://tms02.horisont.svenskaspel.se/" class="">tms02.horisont.svenskaspel.se</a>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0<br class="">
<br class="">
<br class="">
Cheers<br class="">
Gabbe<br class="">
<br class="">
30 maj 2017 kl. 14:45 skrev Sunil Kumar Heggodu Gopala Acharya <<a href="mailto:sheggodu@redhat.com" class="">sheggodu@redhat.com</a>>:<br class="">
<br class="">
Hi Gabriel,<br class="">
<br class="">
I am not able to hit the issue mentioned on my setup.<br class="">
<br class="">
Please share the log files(both brick and client log files) from your setup. It would be great if you can share the details about steps you followed to hit the issue.<br class="">
<br class="">
<br class="">
Regards,<br class="">
SUNIL KUMAR ACHARYA<br class="">
SENIOR SOFTWARE ENGINEER<br class="">
Red Hat<br class="">
<br class="">
T: +91-8067935170<br class="">
<br class="">
[<a href="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" class="">https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png</a>]<br class="">
TRIED. TESTED. TRUSTED.<br class="">
<br class="">
<br class="">
On Tue, May 30, 2017 at 3:30 PM, Gabriel Lindeborg <<a href="mailto:gabriel.lindeborg@svenskaspel.se" class="">gabriel.lindeborg@svenskaspel.se</a>> wrote:<br class="">
Hello<br class="">
<br class="">
A manual mount failed the same way<br class="">
<br class="">
Cheers<br class="">
Gabbe<br class="">
<br class="">
30 maj 2017 kl. 10:24 skrev Sunil Kumar Heggodu Gopala Acharya <<a href="mailto:sheggodu@redhat.com" class="">sheggodu@redhat.com</a>>:<br class="">
<br class="">
Hi Gabriel,<br class="">
<br class="">
Which gluster version are your running? Are you able to fuse mount the volume?<br class="">
<br class="">
Please share the failure logs.<br class="">
<br class="">
Regards,<br class="">
SUNIL KUMAR ACHARYA<br class="">
SENIOR SOFTWARE ENGINEER<br class="">
Red Hat<br class="">
<br class="">
T: +91-8067935170<br class="">
<br class="">
[<a href="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" class="">https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png</a>]<br class="">
TRIED. TESTED. TRUSTED.<br class="">
<br class="">
<br class="">
On Tue, May 30, 2017 at 1:04 PM, Gabriel Lindeborg <<a href="mailto:gabriel.lindeborg@svenskaspel.se" class="">gabriel.lindeborg@svenskaspel.se</a>> wrote:<br class="">
Hello All<br class="">
<br class="">
We’ve have a problem with cluster client mounts fail in mid run with this in the log<br class="">
glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f640c8b3dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f640df4bfd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f640df4bdfb] ) 0-: received signum (15), shutting down.<br class="">
<br class="">
We’ve tried running debug but have not found anything suspicious happening at the time of the failures<br class="">
We’ve searched the inter web but can not find anyone else having the same problem in mid flight<br class="">
<br class="">
The clients have four mounts of volumes from the same server, all mounts fail simultaneously<br class="">
Peer status looks ok<br class="">
Volume status looks ok<br class="">
Volume info looks like this:<br class="">
Volume Name: GLUSTERVOLUME<br class="">
Type: Replicate<br class="">
Volume ID: ca7af017-4f0f-44cc-baf6-43168eed0748<br class="">
Status: Started<br class="">
Snapshot Count: 0<br class="">
Number of Bricks: 1 x 2 = 2<br class="">
Transport-type: tcp<br class="">
Bricks:<br class="">
Brick1: GLUSTERSERVER1:/gluster/GLUSTERVOLUME/brick<br class="">
Brick2: GLUSTERSERVER2:/gluster/GLUSTERVOLUME/brick<br class="">
Options Reconfigured:<br class="">
transport.address-family: inet<br class="">
cluster.self-heal-daemon: enable<br class="">
nfs.disable: on<br class="">
server.allow-insecure: on<br class="">
client.bind-insecure: on<br class="">
network.ping-timeout: 5<br class="">
features.bitrot: on<br class="">
features.scrub: Active<br class="">
features.scrub-freq: weekly<br class="">
<br class="">
Any ideas?<br class="">
<br class="">
Cheers<br class="">
Gabbe<br class="">
<br class="">
<br class="">
<br class="">
AB SVENSKA SPEL<br class="">
621 80 Visby<br class="">
Norra Hansegatan 17, Visby<br class="">
Växel: +4610-120 00 00<br class="">
<a href="https://svenskaspel.se" class="">https://svenskaspel.se</a><br class="">
<br class="">
Please consider the environment before printing this email<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
Gluster-users@gluster.org<br class="">
http://lists.gluster.org/mailman/listinfo/gluster-users<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</blockquote>
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" class="">http://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>