<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><font size="2" class=""><br class=""></font><div><blockquote type="cite" class=""><div class=""><font size="2" class="">On 8 Nov 2017, at 9:03 pm, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" class="">nbalacha@redhat.com</a>> wrote:</font></div><font size="2" class=""><br class="Apple-interchange-newline"></font><div class=""><font size="2" class=""><br class=""></font><div class="gmail_quote" style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><font size="2" class="">That is not the log for the mount. Please check /var/log/glusterfs/var-lib-mo<wbr class="">untedgluster.log on the system on which you are running the mount process.</font></div><div class=""><font size="2" class=""><br class=""></font></div><div class=""><font size="2" class="">Please provide the volume config details as well (gluster volume info) from one of the server nodes.</font></div><div class=""><font size="2" class=""><br class=""></font></div></div></div></blockquote><font size="2" class=""><br class=""></font></div><div><font size="2" class="">Oh I'm sorry, I totally misread that - didn't realise it was on the client.</font></div><div><font size="2" class=""><br class=""></font></div><div><font size="2" class="">Clarification for below logs:</font></div><div><font size="2" class=""><br class=""></font></div><div><font size="2" class="">- 'dev_static' is the gluster volume.</font></div><div><font size="2" class="">- 'int-kube-01' is the gluster client.</font></div><div><font size="2" class="">- '10.51.70.151' is the first node in a three node (2 replica, 1 arbiter) gluster cluster.</font></div><div><font size="2" class="">- '/var/lib/kubelet/...../iss3dev-static' is a directory on the client that should be mounting '10.51.70.151:/dev_static/iss3dev-static', where 'iss3dev-static' is a directory inside the gluster mount 'dev_static'.</font></div><div><font size="2" class="">- FYI only, we are essentially mounting directories inside gluster volumes as per Kubernetes example: <a href="https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md" class="">https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md</a></font></div><div><font size="2" class=""><br class=""></font></div><div><div><font face="Monaco" size="2" class="">root@int-kube-01:/var/log/glusterfs # tail -20 mnt.log</font></div><div><font face="Monaco" size="2" class=""><br class=""></font></div><div><font face="Monaco" size="2" class="">[2017-11-07 21:15:03.561470] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.2 (args: /usr/sbin/glusterfs --volfile-server=10.51.70.151 --volfile-id=/dev_static /mnt)</font></div><div><font face="Monaco" size="2" class="">[2017-11-07 21:15:03.571205] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction</font></div><div><font face="Monaco" size="2" class="">[2017-11-07 21:15:03.584098] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1</font></div><div><font size="2" class=""><br class=""></font></div><div><font size="2" class=""><br class=""></font></div><div><font size="2" class="">No new log entries appear after executing the following to demonstrate the problem:</font></div><div><font size="2" class=""><br class=""></font></div><div><font size="2" class=""><br class=""></font></div><div><div><font size="2" class=""><font face="Monaco" class="">root@int-kube-01:</font><span style="font-family: Monaco;" class="">/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/<a href="http://kubernetes.io" class="">kubernetes.io</a>~glusterfs/iss3dev-static</span><font face="Monaco" class=""> # touch test</font></font></div><div class=""><font size="2" class=""><font face="Monaco" class="">root@int-kube-01:</font><span style="font-family: Monaco;" class="">/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/<a href="http://kubernetes.io" class="">kubernetes.io</a>~glusterfs/iss3dev-static</span><font face="Monaco" class=""> # mkdir testdir2</font></font></div></div><div><div><font face="Monaco" size="2" class=""><br class=""></font></div><div><font size="2" class=""><font face="Monaco" class="">root@int-kube-01:</font><span style="font-family: Monaco;" class="">/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/<a href="http://kubernetes.io" class="">kubernetes.io</a>~glusterfs/iss3dev-static</span><font face="Monaco" class=""> # ls -ltar</font></font></div><div><font face="Monaco" size="2" class="">total 0</font></div><div><font face="Monaco" size="2" class="">-rw-r--r--. 1 root root 0 Nov 6 10:10 test</font></div><div><font face="Monaco" size="2" class=""><br class=""></font></div><div><font size="2" class=""><font face="Monaco" class="">root@int-kube-01:</font><span style="font-family: Monaco;" class="">/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/<a href="http://kubernetes.io" class="">kubernetes.io</a>~glusterfs/iss3dev-static</span><font face="Monaco" class=""> # cd testdir2</font></font></div><div><font size="2" class=""><font face="Monaco" class="">root@int-kube-01:</font><span style="font-family: Monaco;" class="">/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/<a href="http://kubernetes.io" class="">kubernetes.io</a>~glusterfs/iss3dev-static/testdir2</span><font face="Monaco" class=""> # ls -la</font></font></div><div><font face="Monaco" size="2" class="">total 0</font></div><div><font face="Monaco" size="2" class=""><br class=""></font></div><div><font size="2" class=""><font face="Monaco" class="">root@int-kube-01:</font><span style="font-family: Monaco;" class="">/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/<a href="http://kubernetes.io" class="">kubernetes.io</a>~glusterfs/iss3dev-static</span><font face="Monaco" class="">/testdir2 # cd ..</font></font></div><div><font size="2" class=""><font face="Monaco" class="">root@int-kube-01:</font><span style="font-family: Monaco;" class="">/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/<a href="http://kubernetes.io" class="">kubernetes.io</a>~glusterfs/iss3dev-static</span><font face="Monaco" class=""># ls -ltar</font></font></div><div><font face="Monaco" size="2" class="">total 0</font></div><div><font face="Monaco" size="2" class="">-rw-r--r--. 1 root root 0 Nov 6 10:10 test</font></div><div><font face="Monaco" size="2" class=""><br class=""></font></div><div><font face="Monaco" size="2" class=""><br class=""></font></div><div><font size="2" class="">The mount entry looks like:</font></div><div><font face="Monaco" size="2" class=""><br class=""></font></div><div><font face="Monaco" size="2" class=""><div>root@int-kube-01:~ # mount | grep iss3dev-static</div><div>10.51.70.151:/dev_static on /var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/<a href="http://kubernetes.io" class="">kubernetes.io</a>~glusterfs/iss3dev-static type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)</div></font></div></div></div></body></html>