<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Thank you both for your reply,</p>
<p><br>
</p>
<p>The "<span style="font-size: 12pt;">DBUS :WARN :Health </span><span style="font-size: 12pt;">status is unhealthy" is weird because the volume is not having any workload, it's just mounted by ESXi servers and the vms are shutdown, also all bricks are SSDs.</span></p>
<p>You mentioned that it might be related to <span>requests queue being full, where can i check the current queue status ?</span></p>
<p><br>
</p>
<div id="Signature"><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Nithya Balachandran <nbalacha@redhat.com><br>
<b>Sent:</b> Thursday, March 2, 2017 8:32:52 AM<br>
<b>To:</b> Soumya Koduri<br>
<b>Cc:</b> Mahdi Adnan; gluster-users@gluster.org; Krutika Dhananjay; Frank Filz<br>
<b>Subject:</b> Re: [Gluster-users] nfs-ganesha logs</font>
<div> </div>
</div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 1 March 2017 at 18:25, Soumya Koduri <span dir="ltr">
<<a href="mailto:skoduri@redhat.com" target="_blank">skoduri@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I am not sure if there are any outstanding issues with exposing shard volume via gfapi. CCin Krutika.<span><br>
<br>
On 02/28/2017 01:29 PM, Mahdi Adnan wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi,<br>
<br>
<br>
We have a Gluster volume hosting VMs for ESXi exported via Ganesha.<br>
<br>
Im getting the following messages in ganesha-gfapi.log and ganesha.log<br>
<br>
<br>
<br>
=====<br>
<br>
[2017-02-28 07:44:55.194621] E [MSGID: 109040]<br>
[dht-helper.c:1198:dht_migrati<wbr>on_complete_check_task] 0-vmware2-dht:<br>
<gfid:ec846aeb-50f9-4b39-b0c9-<wbr>24a8b833afe6>: failed to lookup the file<br>
on vmware2-dht [Stale file handle]<br>
</blockquote>
<br>
</span>This "Stale file handle" error suggests that the file may have just got removed at the back-end. Probably someone more familiar with dht (cc'ed Nithya) can confirm if there are other possibilities.</blockquote>
<div><br>
</div>
<div>That is one possibility. In case a FOP returns an ENOENT/ESTALE because the file was deleted before it could go through, DHT checks to see if the file was migrated to another brick. However, as the file is no longer present on the volume, you will see
the dht_migrati<wbr>on_complete_check_task message above. </div>
<div><br>
</div>
<div>You might want to check if the file in question still exists. There should also be messages in the log indicating which fop has failed.</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div class="gmail-m_3695413420648026393h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
[2017-02-28 07:44:55.194660] E [MSGID: 133014]<br>
[shard.c:1129:shard_common_sta<wbr>t_cbk] 0-vmware2-shard: stat failed:<br>
ec846aeb-50f9-4b39-b0c9-24a8b8<wbr>33afe6 [Stale file handle]<br>
[2017-02-28 07:44:55.207154] W [MSGID: 108008]<br>
[afr-read-txn.c:228:afr_read_t<wbr>xn] 0-vmware2-replicate-5: Unreadable<br>
subvolume -1 found with event generation 8 for gfid<br>
4a50127e-4403-49a5-9886-80541a<wbr>76299c. (Possible split-brain)<br>
[2017-02-28 07:44:55.209205] E [MSGID: 109040]<br>
[dht-helper.c:1198:dht_migrati<wbr>on_complete_check_task] 0-vmware2-dht:<br>
<gfid:4a50127e-4403-49a5-9886-<wbr>80541a76299c>: failed to lookup the file<br>
on vmware2-dht [Stale file handle]<br>
[2017-02-28 07:44:55.209265] E [MSGID: 133014]<br>
[shard.c:1129:shard_common_sta<wbr>t_cbk] 0-vmware2-shard: stat failed:<br>
4a50127e-4403-49a5-9886-80541a<wbr>76299c [Stale file handle]<br>
[2017-02-28 07:44:55.212556] W [MSGID: 108008]<br>
[afr-read-txn.c:228:afr_read_t<wbr>xn] 0-vmware2-replicate-4: Unreadable<br>
subvolume -1 found with event generation 2 for gfid<br>
cec80035-1f51-434a-9dbf-8bcdd5<wbr>f4a8f7. (Possible split-brain)<br>
[2017-02-28 07:44:55.214702] E [MSGID: 109040]<br>
[dht-helper.c:1198:dht_migrati<wbr>on_complete_check_task] 0-vmware2-dht:<br>
<gfid:cec80035-1f51-434a-9dbf-<wbr>8bcdd5f4a8f7>: failed to lookup the file<br>
on vmware2-dht [Stale file handle]<br>
[2017-02-28 07:44:55.214741] E [MSGID: 133014]<br>
[shard.c:1129:shard_common_sta<wbr>t_cbk] 0-vmware2-shard: stat failed:<br>
cec80035-1f51-434a-9dbf-8bcdd5<wbr>f4a8f7 [Stale file handle]<br>
[2017-02-28 07:44:55.259729] I [MSGID: 108031]<br>
[afr-common.c:2154:afr_local_d<wbr>iscovery_cbk] 0-vmware2-replicate-0:<br>
selecting local read_child vmware2-client-0<br>
[2017-02-28 07:44:55.259937] I [MSGID: 108031]<br>
[afr-common.c:2154:afr_local_d<wbr>iscovery_cbk] 0-vmware2-replicate-4:<br>
selecting local read_child vmware2-client-8<br>
<br>
=====<br>
<br>
28/02/2017 06:27:54 : epoch 58b05af4 : gluster01 :<br>
ganesha.nfsd-2015[dbus_heartbe<wbr>at] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 06:33:36 : epoch 58b05af4 : gluster01 :<br>
ganesha.nfsd-2015[work-9] cache_inode_avl_qp_insert :INODE :CRIT<br>
:cache_inode_avl_qp_insert_s: name conflict (access, access)<br>
=====<br>
<br>
<br>
The volume is hosting a few VMs without any noticeable workload, and all<br>
bricks are SSDs.<br>
<br>
Im censored about the logs messages because i have another cluster and<br>
ganesha keeps on crashing every few days with the following message<br>
spamming the log:<br>
<br>
</blockquote>
<br>
</div>
</div>
Do you happen to have core? If yes, could you please check the bt. Below messages are just heartbeat warnings typically thrown when the outstanding request queue is above certain bench mark and nfs-ganesha server is taking a while to process them. Also you
seem to be using nfs-ganesha 2.3.x version. Its not being actively maintained. There are many improvements and fixes done in nfs-ganesha 2.4.x. I suggest to try out that version if possible.
<div>
<div class="gmail-m_3695413420648026393h5"><br>
<br>
><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
28/02/2017 08:02:45 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 08:41:08 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 08:48:38 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 08:48:52 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 09:16:27 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 09:46:54 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 09:50:02 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 09:57:03 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 09:57:14 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
28/02/2017 10:48:41 : epoch 58b1e2f5 : gfs01 :<br>
ganesha.nfsd-31929[dbus_heartb<wbr>eat] dbus_heartbeat_cb :DBUS :WARN :Health<br>
status is unhealthy. Not sending heartbeat<br>
<br>
<br>
SSDs volumes are running Gluster 3.8.9 and Ganesha V2.3.3 and the other<br>
cluster is running Gluster 3.7.19 and Ganesha V2.3.0.<br>
<br>
<br>
also, how can i get IO statics from Ganesha ?<br>
</blockquote>
<br>
</div>
</div>
AFAIK, there are no tools integrated with nfs-ganesha which monitors and displays IO statistics. Request Frank & others to comment.<br>
<br>
Thanks,<br>
Soumya<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span><br>
I appreciate your help.<br>
<br>
<br>
<br>
<br>
<br>
--<br>
<br>
</span>Respectfully*<br>
**Mahdi A. Mahdi*<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<br>
</blockquote>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</body>
</html>