<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi,<br>
Yes, of cause...should have included it from start.<br>
Yes, I know an old version, but I will rebuild a new cluster later on,<br>
that is another story.<br>
<br>
Client side:<br>
Archlinux<br>
glusterfs 1:3.10.1-1<br>
<br>
Sever side:<br>
Replicated cluster on two physical machines.<br>
Both running:<br>
Centos 7 3.10.0-514.16.1.el7.x86_64<br>
Gluster glusterfs 3.8.11 from centos-gluster38<br>
<br>
Typical user case(the one we have problem with now; typical):<br>
Our users handle genomic evaluations, where loads of calculations<br>
are done, intermediate results are saved to files (MB-GB size and<br>
up to a hundred files),<br>
and used for next calculation step where it is read from file,<br>
calculated, written to file aso. a couple of times.<br>
The lenght of these processes are about 8-12 hours and up to<br>
processes running for up til about 72-96 hours.<br>
For this run we had 12 clients (all connected to gluster and all<br>
file read/writes done to gluster). On each client we had assign<br>
3 cores to be used to run the processes, and most of the time all<br>
3 cores were beeing used on all 12 clients.<br>
<br>
Regards<br>
Marcus</p>
<p><br>
</p>
<p><br>
</p>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>Frċn:</b> Milind Changire <mchangir@redhat.com><br>
<b>Skickat:</b> den 23 januari 2018 15:46<br>
<b>Till:</b> Marcus Pedersén<br>
<b>Kopia:</b> Gluster Users<br>
<b>Ämne:</b> Re: [Gluster-users] Understanding client logs</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>Marcus,<br>
</div>
Please paste the name-version-release of the primary glusterfs package on your system.<br>
<br>
</div>
If possible, also describe the typical workload that happens at the mount via the user application.<br>
<br>
<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jan 23, 2018 at 7:43 PM, Marcus Pedersén <span dir="ltr">
<<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
Hi all,<br>
I have problem pin pointing an error, that users of<br>
my system experience processes that crash.<br>
The thing that have changed since the craches started<br>
is that I added a gluster cluster.<br>
Of cause the users start to attack my gluster cluster.<br>
<br>
I started looking at logs, starting from the client side.<br>
I just need help to understand how to read it in the right way.<br>
I can see that every ten minutes the client changes port and<br>
attach to the remote volume. About five minutes later<br>
the client unmounts the volume.<br>
I guess that this is the "old" mount and that the "new" mount<br>
is already responding to user interaction?<br>
<br>
As this repeates every ten minutes I see this as normal behavior<br>
and just want to get a better understanding on how the client<br>
interacts with the cluster.<br>
<br>
Have you experienced that this switch malfunctions and the<br>
mount becomes unreachable for a while?<br>
<br>
Many thanks in advance!<br>
<br>
Best regards<br>
Marcus Pederén<br>
<br>
An example of the output:<br>
[2017-11-09 10:10:39.776403] I [rpc-clnt.c:2000:rpc_clnt_<wbr>reconfig] 0-interbull-interbull-client-<wbr>1: changing port to 49160 (from 0)<br>
[2017-11-09 10:10:39.776830] I [MSGID: 114057] [client-handshake.c:1451:<wbr>select_server_supported_<wbr>programs] 0-interbull-interbull-client-<wbr>0: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>
[2017-11-09 10:10:39.777642] I [MSGID: 114046] [client-handshake.c:1216:<wbr>client_setvolume_cbk] 0-interbull-interbull-client-<wbr>0: Connected to interbull-interbull-client-0, attached to remote volume '/interbullfs/i\<br>
nterbull'.<br>
[2017-11-09 10:10:39.777663] I [MSGID: 114047] [client-handshake.c:1227:<wbr>client_setvolume_cbk] 0-interbull-interbull-client-<wbr>0: Server and Client lk-version numbers are not same, reopening the fds<br>
[2017-11-09 10:10:39.777724] I [MSGID: 108005] [afr-common.c:4756:afr_notify] 0-interbull-interbull-<wbr>replicate-0: Subvolume 'interbull-interbull-client-0' came back up; going online.<br>
[2017-11-09 10:10:39.777954] I [MSGID: 114035] [client-handshake.c:202:<wbr>client_set_lk_version_cbk] 0-interbull-interbull-client-<wbr>0: Server lk version = 1<br>
[2017-11-09 10:10:39.779909] I [MSGID: 114057] [client-handshake.c:1451:<wbr>select_server_supported_<wbr>programs] 0-interbull-interbull-client-<wbr>1: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>
[2017-11-09 10:10:39.780481] I [MSGID: 114046] [client-handshake.c:1216:<wbr>client_setvolume_cbk] 0-interbull-interbull-client-<wbr>1: Connected to interbull-interbull-client-1, attached to remote volume '/interbullfs/i\<br>
nterbull'.<br>
[2017-11-09 10:10:39.780509] I [MSGID: 114047] [client-handshake.c:1227:<wbr>client_setvolume_cbk] 0-interbull-interbull-client-<wbr>1: Server and Client lk-version numbers are not same, reopening the fds<br>
[2017-11-09 10:10:39.781544] I [MSGID: 114035] [client-handshake.c:202:<wbr>client_set_lk_version_cbk] 0-interbull-interbull-client-<wbr>1: Server lk version = 1<br>
[2017-11-09 10:10:39.781608] I [fuse-bridge.c:4146:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.26<br>
[2017-11-09 10:10:39.781632] I [fuse-bridge.c:4831:fuse_<wbr>graph_sync] 0-fuse: switched to graph 0<br>
[2017-11-09 10:16:10.609922] I [fuse-bridge.c:5089:fuse_<wbr>thread_proc] 0-fuse: unmounting /interbull<br>
[2017-11-09 10:16:10.610258] W [glusterfsd.c:1329:cleanup_<wbr>and_exit] (-->/usr/lib/libpthread.so.0(+<wbr>0x72e7) [0x7f98c02282e7] -->/usr/bin/glusterfs(<wbr>glusterfs_sigwaiter+0xdd) [0x40890d] -->/usr/bin/glusterfs(cleanu\<br>
p_and_exit+0x4b) [0x40878b] ) 0-: received signum (15), shutting down<br>
[2017-11-09 10:16:10.610290] I [fuse-bridge.c:5802:fini] 0-fuse: Unmounting '/interbull'.<br>
[2017-11-09 10:20:39.752079] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/bin/glusterfs: Started running /usr/bin/glusterfs version 3.10.1 (args: /usr/bin/glusterfs --negative-timeout=60 --volfile-server=1\<br>
92.168.67.31 --volfile-id=/interbull-<wbr>interbull /interbull)<br>
[2017-11-09 10:20:39.763902] I [MSGID: 101190] [event-epoll.c:629:event_<wbr>dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>
[2017-11-09 10:20:39.768738] I [afr.c:94:fix_quorum_options] 0-interbull-interbull-<wbr>replicate-0: reindeer: incoming qtype = none<br>
[2017-11-09 10:20:39.768756] I [afr.c:116:fix_quorum_options] 0-interbull-interbull-<wbr>replicate-0: reindeer: quorum_count = 0<br>
[2017-11-09 10:20:39.768856] W [MSGID: 108040] [afr.c:315:afr_pending_xattrs_<wbr>init] 0-interbull-interbull-<wbr>replicate-0: Unable to fetch afr-pending-xattr option from volfile. Falling back to using client translat\<br>
or names.<br>
[2017-11-09 10:20:39.769832] I [MSGID: 101190] [event-epoll.c:629:event_<wbr>dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>
[2017-11-09 10:20:39.770193] I [MSGID: 114020] [client.c:2352:notify] 0-interbull-interbull-client-<wbr>0: parent translators are ready, attempting connect on transport<br>
[2017-11-09 10:20:39.773109] I [MSGID: 114020] [client.c:2352:notify] 0-interbull-interbull-client-<wbr>1: parent translators are ready, attempting connect on transport<br>
[2017-11-09 10:20:39.773712] I [rpc-clnt.c:2000:rpc_clnt_<wbr>reconfig] 0-interbull-interbull-client-<wbr>0: changing port to 49177 (from 0)<br>
<br>
<br>
--<br>
******************************<wbr>********************<br>
* Marcus Pedersén *<br>
* System administrator *<br>
******************************<wbr>********************<br>
* Interbull Centre *<br>
* ================ *<br>
* Department of Animal Breeding & Genetics — SLU *<br>
* Box 7023, SE-750 07 *<br>
* Uppsala, Sweden *<br>
******************************<wbr>********************<br>
* Visiting address: *<br>
* Room 55614, Ulls väg 26, Ultuna *<br>
* Uppsala *<br>
* Sweden *<br>
* *<br>
* Tel: +46-(0)18-67 1962 *<br>
* *<br>
******************************<wbr>********************<br>
* ISO 9001 Bureau Veritas No SE004561-1 *<br>
******************************<wbr>********************<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">Milind<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>