[Gluster-users] ESXI cannot access stripped gluster volume

[Bryan Whitehead]

I don't have ESXi experience but the first thing that jumps out at me is
you probably need to mount NFS/tcp. NFS/udp doesn't work on glusterfs
(unless this has changed and I've not been paying close enough attention
lately).




On Sat, Mar 1, 2014 at 8:35 AM, Carlos Capriotti <capriotti.carlos at gmail.com
> wrote:

> ESXI cannot access stripped gluster volume
>
> Hello all.
>
> Unfortunately this is going to be a long post, so I will not spend too
> many words with compliments; Gluster is a great solution and I should be
> writing odes about it, so, bravo all of you.
>
> A bit about me: I've been working with FreeBSD and Linux for over a decade
> now. I use CentOS nowadays because of some convenient features, regarding
> my applications.
>
> Now a bit more about my problem: After adding my stripped gluster volume
> to my ESXI via NFS, I try to browse it with vSphere's browser,  and it...
>
> a) cannot see any of the folders already there. The operation never times
> out, and I have a line of dots telling me the system (esxi) is trying to do
> something. Other systems CAN access and see content on the same volume.
>
> b) gluster or ESXI return/log NO ERROR when I (try to) create folders
> there, but does not show them either. Folder IS created.
>
> c) When trying to create a virtual machine it DOES return an error. Folder
> and first file related to the VM ARE created but I get an error about the
> an "Invalid virtual machine configuration". I am under the impression esxi
> returns this when it tries to create the file for the virtual disk.
>
> d) When trying to remove the volume, it DOES return an error, stating the
> resource is busy. I am forced to reboot the ESXI host in order to
> successfully delete the nfs volume.
>
> Now, a bit about my environment:
>
> I am one of those cursed with an Isilon, but with NO service contract and
> NO license. So, basically I have a big, fast and resilient NAS. Cool stuff,
> but with great inconveniences as well. Goes without saying that me, as a
> free software guy, would love to build something that can retire the
> isilon, or at least, move it to a secondary role.
>
> Anyway, trying to add an alternative to it, I searched for days and
> decided Gluster was the way to go. And I am not going back. I will make it
> work.
>
> All of my VM servers, (about 15) are spread across 3 metal boxes, and -
> please, don't blame me, I inherited this situation - there is no backup
> solution whatsoever. Gluster will, in its final configuration, run on 4
> boxes, providing HA and backup.
>
> So, on esxi,  Isilon's NFS volume/share works like a charm; pure NFS
> sharing on CentOS works like a charm; gluster stripe - using two of my four
> servers -  does not like me.
>
> The very same gluster NSF volume is mounted and works happily on a ContOS
> client. Actually, more than one.
>
> I have been reading literally dozens of docs, guides, manuals, VMware,
> Gluster and Red Hat for more than a week, and while that, I've even created
> ESXI VIRTUAL SERVERS *INSIDE* my EXSI physical servers, because I can no
> longer afford rebooting a production server whenever I need to test yet
> another change on gluster.
>
>
> My software versions:
>
> CentOS 6.5
> Gluster 3.4.2
> ESXI 5.1 all patches applied
> ESXI 5.5
>
> My hardware for the nodes: 2 x Dell PE2950 with RAID5, bricks are single
> volume of about 1.5 TB on each node.
>
> One stand-alone PE2900 with a single volume, on RAID5, and about 2.4 TB,
> which will be added to the stripe, eventually. One PE2950, brick on RAID5
> 800GB, which will also be added eventually.
>
> All of them with one NIC for regular networking and a bonded nic made out
> of 2 physical NICs, for gluster/nfs.
>
> ESXIs are running on R710s, lots of ram, and, at least one NIC dedicated
> to NFS. I have one test server running with all four NICs on the NFS
> netowrk.
>
> NFS network is 9000 MTU, tuned for iSCSI (in the future).
>
> Now, trying to make it all work, these are the steps I took:
>
> Regarding ESXI tweaks, I've changed GLUSTER'S ping limit, lowering it to
> 20, to stop the volume from being intermittently inaccessible. On ESXI
> itself I've set the NFS max queue length to 64.
>
> I've chmoded gluster's share with 777, and you can find my gluster tweaks
> for the volume below.
>
> Gluster's NFS and regular NFS both are forcing uid and gid to
> nfsnobody:nfsnobody.
>
> iptables has been disabled, along with SElinux.
>
> Of course regular NFS is disabled.
>
> My gluster settings:
>
>
> Volume Name: glvol0
> Type: Stripe
> Volume ID: f76af2ac-6a42-42ea-9887-941bf1600ced
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.1.21:/export/glusterroot
> Brick2: 10.0.1.22:/export/glusterroot
> Options Reconfigured:
> nfs.ports-insecure: on
> nfs.addr-namelookup: off
> auth.reject: NONE
> nfs.volume-access: read-write
> nfs.nlm: off
> network.ping-timeout: 20
> server.root-squash: on
> performance.nfs.write-behind: on
> performance.nfs.read-ahead: on
> performance.nfs.io-cache: on
> performance.nfs.quick-read: on
> performance.nfs.stat-prefetch: on
> performance.nfs.io-threads: on
> storage.owner-uid: 65534
> storage.owner-gid: 65534
> nfs.disable: off
>
>
> My regular NFS settings, that work just the way I need:
>
> /export/share
> *(rw,all_squash,anonuid=65534,anongid=65534,no_subtree_check)
>
> Once I get this all to work, I intend to create a nice page with
> instructions/info on gluster for ESXi. This field needs some more work out
> there.
>
> Now the question: is there anything I forgot, overlooked and don't know ?
>
> Could you help me ? Except for a comment from someone, saying "stripe does
> not work with esxi", nothing REALLY rings a bell. I've used all the
> pertinent info I had, and ran out of moves.
>
> My only option now would be testing a volume that is not a stripe. I'll do
> that for now, but I don't think it will work.
>
> BTW, can I be added to the list ?
>
> Cheers,
>
> Carlos.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140302/0030a839/attachment.html>