[Gluster-users] Openstack Havana Gluster /var/lib/instances issues

Wed Mar 19 19:10:47 UTC 2014

So a little further down the road I have destroyed/rebuilt the gluster volume and restarted glusterd on each of the nodes. Shortly after the instances directory is created and the console.log file is placed, I receive the following fuse errors on the client:

 [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-4: remote operation failed: Invalid argument
[2014-03-19 19:06:35.326487] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-5: remote operation failed: Invalid argument
[2014-03-19 19:06:35.328750] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106504: WRITE => -1 (Invalid argument)
[2014-03-19 19:06:35.328855] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106505: FTRUNCATE() ERR => -1 (Bad file descriptor)
[2014-03-19 19:06:35.329506] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106507: WRITE => -1 (Bad file descriptor)
[2014-03-19 19:06:35.329775] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106508: FTRUNCATE() ERR => -1 (Bad file descriptor)
[2014-03-19 19:06:35.330495] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106510: WRITE => -1 (Bad file descriptor)
[2014-03-19 19:06:35.330758] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106511: FTRUNCATE() ERR => -1 (Bad file descriptor)
[2014-03-19 19:06:35.331538] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106513: WRITE => -1 (Bad file descriptor)
[2014-03-19 19:06:35.331792] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106514: FTRUNCATE() ERR => -1 (Bad file descriptor)

One one of the servers the normal messages apply:

[2014-03-13 21:47:33.735042] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 (version: 3.4.2)
[2014-03-18 21:17:42.763044] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
[2014-03-18 21:17:42.763076] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
[2014-03-18 21:17:42.763101] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
[2014-03-18 21:50:02.206635] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 (version: 3.4.2)
[2014-03-18 21:50:23.037886] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 (version: 3.4.2)
[2014-03-18 21:50:23.051419] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
[2014-03-18 21:50:23.051446] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
[2014-03-18 21:50:23.051466] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
[2014-03-18 21:56:22.215866] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
[2014-03-18 21:56:22.215898] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
[2014-03-18 21:56:22.215923] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
[2014-03-18 21:57:04.821397] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 (version: 3.4.2)
[2014-03-19 16:48:32.135593] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
[2014-03-19 16:48:32.135629] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
[2014-03-19 16:48:32.135652] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
[2014-03-19 16:48:47.712973] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 (version: 3.4.2)
[2014-03-19 18:07:44.332240] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
[2014-03-19 18:07:44.332272] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
[2014-03-19 18:07:44.332296] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
[2014-03-19 19:00:51.369205] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-21755-2014/03/19-19:00:51:286018-novashared-client-0-0 (version: 3.4.2)

These are all on 3.4.2 with Ubuntu 13.10. 

-ryan

On Mar 19, 2014, at 11:15 AM, Carlos Capriotti <capriotti.carlos at gmail.com> wrote:

> This rings a bell...
> 
> I mapped a remote nfs share to one of my nodes and mapped the local nfs
> share, to the "local" gluster volume as well and started to copy (with cp)
> files from one mount to the other.
> 
> Somewhere down the line all stalled, the server was irresponsive, even
> restarting glusterd solved nothing. Had to reboot the nodes.
> 
> CentOS 6.5 and Gluster 3.4.2.
> 
> Since then,  I haven't tried those copies, but, I also, I am now using the
> native glusterfs client just in case.
> 
> 
> 
> 
> On Wed, Mar 19, 2014 at 3:51 PM, Ray Powell <rpowell1 at uchicago.edu> wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>> 
>> I have not seen that with the fuse client.  I had it happen repeatedly
>> when we were using gluster/NFS for /var/lib/nova/instances and/or
>> /var/lib/nova/instances/_base.  The access to instances-XXXXXX/disk
>> (or its qcow2 backing image) would hang completely or be very slow
>> under heavy load.  The VM interprets the delay as a problem on
>> /dev/vda and remounts RO.
>> 
>> 
>> On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote:
>>> I was wondering if anyone had seen similar issues before, when
>>> booting a vm with a root volume on local disk things work fine.
>>> 
>>> When I try to do the same with /var/lib/nova/instances mounted via
>>> gluster/fuse I get the following: http://paste.ubuntu.com/7116594/
>>> 
>>> I have of course insured perms were correct with chown -R nova:nova
>>> /var/lib/nova/instances, additionally app armor is completely
>>> unloaded/disabled.
>>> 
>>> Has anyone encountered this before? Gluster version is 3.4.2
>>> 
>>> 
>>> -ryan
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________ Gluster-users
>>> mailing list Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>> 
>> 
>> - --
>> Raymond Powell
>> Systems Administrator
>> Laboratory for Advanced Computing
>> University of Chicago, KCBD 10160
>> 900 East 57th Street
>> Chicago, IL 60637
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>> 
>> iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9
>> lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy
>> pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B
>> YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm
>> eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg
>> huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/
>> HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD
>> lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa
>> sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA
>> o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF
>> JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe
>> +UhAq6+74ie1MokNssB8
>> =un7j
>> -----END PGP SIGNATURE-----
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140319/54effb7c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140319/54effb7c/attachment.sig>