[Gluster-users] Gluster 3.3.0 and VMware ESXi 5

[Atha Kouroussis]

Hi everybody,
we are testing Gluster 3.3 as an alternative to our current Nexenta based storage. With the introduction of granular based locking gluster seems like a viable alternative for VM storage.

Regrettably we cannot get it to work even for the most rudimentary tests. We have a two brick setup with two ESXi 5 servers. We created both distributed and replicated volumes. We can mount the volumes via NFS on the ESXi servers without any issues but that is as far as we can go.

When we try to migrate a VM to the gluster backed datastore there is no activity on the bricks and eventually the operation times out on the ESXi side. The nfs.log shows messages like these (distributed volume):

[2012-06-07 00:00:16.992649] E [nfs3.c:3551:nfs3_rmdir_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.11.11:646) vmvol : 7d25cb9a-b9c8-440d-bbd8-973694ccad17
[2012-06-07 00:00:17.027559] W [nfs3.c:3525:nfs3svc_rmdir_cbk] 0-nfs: 3bb48d69: /TEST => -1 (Directory not empty)
[2012-06-07 00:00:17.066276] W [nfs3.c:3525:nfs3svc_rmdir_cbk] 0-nfs: 3bb48d90: /TEST => -1 (Directory not empty)
[2012-06-07 00:00:17.097118] E [nfs3.c:3551:nfs3_rmdir_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.11.11:646) vmvol : 00000000-0000-0000-0000-000000000001


When the volume is mounted on the ESXi servers, we get messages like these in nfs.log:

[2012-06-06 23:57:34.697460] W [socket.c:195:__socket_rwv] 0-socket.nfs-server: readv failed (Connection reset by peer)


The same volumes mounted via NFS on a linux box work fine and we did a couple of benchmarks with bonnie++ with very promising results.
Curiously, if we ssh into the ESXi boxes and go to the mount point of the volume, we can see it contents and write.

Any clues of what might be going on? Thanks in advance.

Cheers,
Atha