<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>We have a Gluster volume hosting VMs for ESXi exported via Ganesha.</p>
<p>Im getting the following messages in g<span>anesha-gfapi.log and <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">ganesha.log</span></span></p>
<p><span><br>
</span></p>
<p><span><br>
</span></p>
<p>=====</p>
<p></p>
<div>[2017-02-28 07:44:55.194621] E [MSGID: 109040] [dht-helper.c:1198:dht_migration_complete_check_task] 0-vmware2-dht: <gfid:ec846aeb-50f9-4b39-b0c9-24a8b833afe6>: failed to lookup the file on vmware2-dht [Stale file handle]</div>
<div>[2017-02-28 07:44:55.194660] E [MSGID: 133014] [shard.c:1129:shard_common_stat_cbk] 0-vmware2-shard: stat failed: ec846aeb-50f9-4b39-b0c9-24a8b833afe6 [Stale file handle]</div>
<div>[2017-02-28 07:44:55.207154] W [MSGID: 108008] [afr-read-txn.c:228:afr_read_txn] 0-vmware2-replicate-5: Unreadable subvolume -1 found with event generation 8 for gfid 4a50127e-4403-49a5-9886-80541a76299c. (Possible split-brain)</div>
<div>[2017-02-28 07:44:55.209205] E [MSGID: 109040] [dht-helper.c:1198:dht_migration_complete_check_task] 0-vmware2-dht: <gfid:4a50127e-4403-49a5-9886-80541a76299c>: failed to lookup the file on vmware2-dht [Stale file handle]</div>
<div>[2017-02-28 07:44:55.209265] E [MSGID: 133014] [shard.c:1129:shard_common_stat_cbk] 0-vmware2-shard: stat failed: 4a50127e-4403-49a5-9886-80541a76299c [Stale file handle]</div>
<div>[2017-02-28 07:44:55.212556] W [MSGID: 108008] [afr-read-txn.c:228:afr_read_txn] 0-vmware2-replicate-4: Unreadable subvolume -1 found with event generation 2 for gfid cec80035-1f51-434a-9dbf-8bcdd5f4a8f7. (Possible split-brain)</div>
<div>[2017-02-28 07:44:55.214702] E [MSGID: 109040] [dht-helper.c:1198:dht_migration_complete_check_task] 0-vmware2-dht: <gfid:cec80035-1f51-434a-9dbf-8bcdd5f4a8f7>: failed to lookup the file on vmware2-dht [Stale file handle]</div>
<div>[2017-02-28 07:44:55.214741] E [MSGID: 133014] [shard.c:1129:shard_common_stat_cbk] 0-vmware2-shard: stat failed: cec80035-1f51-434a-9dbf-8bcdd5f4a8f7 [Stale file handle]</div>
<div>[2017-02-28 07:44:55.259729] I [MSGID: 108031] [afr-common.c:2154:afr_local_discovery_cbk] 0-vmware2-replicate-0: selecting local read_child vmware2-client-0</div>
<div>[2017-02-28 07:44:55.259937] I [MSGID: 108031] [afr-common.c:2154:afr_local_discovery_cbk] 0-vmware2-replicate-4: selecting local read_child vmware2-client-8</div>
<p>=====</p>
<p></p>
<div>28/02/2017 06:27:54 : epoch 58b05af4 : gluster01 : ganesha.nfsd-2015[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 06:33:36 : epoch 58b05af4 : gluster01 : ganesha.nfsd-2015[work-9] cache_inode_avl_qp_insert :INODE :CRIT :cache_inode_avl_qp_insert_s: name conflict (access, access)</div>
=====
<p></p>
<p><br>
</p>
<p>The volume is hosting a few VMs without any noticeable workload, and all bricks are SSDs.</p>
<p>Im censored about the logs messages because i have another cluster and ganesha keeps on crashing every few days with the following message spamming the log:</p>
<p><br>
</p>
<p></p>
<div>28/02/2017 08:02:45 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 08:41:08 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 08:48:38 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 08:48:52 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 09:16:27 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 09:46:54 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 09:50:02 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 09:57:03 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 09:57:14 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<div>28/02/2017 10:48:41 : epoch 58b1e2f5 : gfs01 : ganesha.nfsd-31929[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health status is unhealthy. Not sending heartbeat</div>
<br>
<p></p>
<p><br>
</p>
<p>SSDs volumes are running Gluster 3.8.9 and Ganesha <span>V2.3.3 and the other cluster is running Gluster <span>3.7.19 and Ganesha <span>V2.3.0.</span></span></span></p>
<p><span><span><span><br>
</span></span></span></p>
<p><span><span><span>also, how can i get IO statics from Ganesha ? </span></span></span></p>
<p><span><span><span>I appreciate your help.</span></span></span></p>
<p><br>
</p>
<p></p>
<div id="Signature"><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</div>
</body>
</html>