[Gluster-users] Openstack Havana Gluster /var/lib/instances issues

SATHEESARAN sasundar at redhat.com
Tue Mar 25 14:33:45 UTC 2014 

Resending it as I suppose it was not delievered :(
> Hi Ryan/Narayan,
>
> Apologies for top posting.
> After today's call, we had a chat with Vijay and he added a hint.
>
> You need to enable/*remote-dio*/ for that volume that serves for 
> virt-store.
> > gluster volume set <vol-name> remote-dio enable
>
> Hope this could solve your issue most probably.
> Could you test it with this option 'on' and let us know the results ?
>
> -- sas
>
> On 03/20/2014 12:40 AM, Aydelott, Ryan M. wrote:
>> So a little further down the road I have destroyed/rebuilt the 
>> gluster volume and restarted glusterd on each of the nodes. Shortly 
>> after the instances directory is created and the console.log file is 
>> placed, I receive the following fuse errors on the client:
>>
>>  [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-4: 
>> remote operation failed: Invalid argument
>> [2014-03-19 19:06:35.326487] W 
>> [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-5: 
>> remote operation failed: Invalid argument
>> [2014-03-19 19:06:35.328750] W [fuse-bridge.c:2167:fuse_writev_cbk] 
>> 0-glusterfs-fuse: 106504: WRITE => -1 (Invalid argument)
>> [2014-03-19 19:06:35.328855] W [fuse-bridge.c:648:fuse_truncate_cbk] 
>> 0-glusterfs-fuse: 106505: FTRUNCATE() ERR => -1 (Bad file descriptor)
>> [2014-03-19 19:06:35.329506] W [fuse-bridge.c:2167:fuse_writev_cbk] 
>> 0-glusterfs-fuse: 106507: WRITE => -1 (Bad file descriptor)
>> [2014-03-19 19:06:35.329775] W [fuse-bridge.c:648:fuse_truncate_cbk] 
>> 0-glusterfs-fuse: 106508: FTRUNCATE() ERR => -1 (Bad file descriptor)
>> [2014-03-19 19:06:35.330495] W [fuse-bridge.c:2167:fuse_writev_cbk] 
>> 0-glusterfs-fuse: 106510: WRITE => -1 (Bad file descriptor)
>> [2014-03-19 19:06:35.330758] W [fuse-bridge.c:648:fuse_truncate_cbk] 
>> 0-glusterfs-fuse: 106511: FTRUNCATE() ERR => -1 (Bad file descriptor)
>> [2014-03-19 19:06:35.331538] W [fuse-bridge.c:2167:fuse_writev_cbk] 
>> 0-glusterfs-fuse: 106513: WRITE => -1 (Bad file descriptor)
>> [2014-03-19 19:06:35.331792] W [fuse-bridge.c:648:fuse_truncate_cbk] 
>> 0-glusterfs-fuse: 106514: FTRUNCATE() ERR => -1 (Bad file descriptor)
>>
>>
>> One one of the servers the normal messages apply:
>>
>> [2014-03-13 21:47:33.735042] I 
>> [server-handshake.c:567:server_setvolume] 0-novashared-server: 
>> accepted client from 
>> cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 (version: 
>> 3.4.2)
>> [2014-03-18 21:17:42.763044] I [server.c:762:server_rpc_notify] 
>> 0-novashared-server: disconnecting connectionfrom 
>> cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
>> [2014-03-18 21:17:42.763076] I 
>> [server-helpers.c:729:server_connection_put] 0-novashared-server: 
>> Shutting down connection 
>> cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
>> [2014-03-18 21:17:42.763101] I 
>> [server-helpers.c:617:server_connection_destroy] 0-novashared-server: 
>> destroyed connection of 
>> cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
>> [2014-03-18 21:50:02.206635] I 
>> [server-handshake.c:567:server_setvolume] 0-novashared-server: 
>> accepted client from 
>> cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 (version: 
>> 3.4.2)
>> [2014-03-18 21:50:23.037886] I 
>> [server-handshake.c:567:server_setvolume] 0-novashared-server: 
>> accepted client from 
>> cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 (version: 
>> 3.4.2)
>> [2014-03-18 21:50:23.051419] I [server.c:762:server_rpc_notify] 
>> 0-novashared-server: disconnecting connectionfrom 
>> cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
>> [2014-03-18 21:50:23.051446] I 
>> [server-helpers.c:729:server_connection_put] 0-novashared-server: 
>> Shutting down connection 
>> cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
>> [2014-03-18 21:50:23.051466] I 
>> [server-helpers.c:617:server_connection_destroy] 0-novashared-server: 
>> destroyed connection of 
>> cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
>> [2014-03-18 21:56:22.215866] I [server.c:762:server_rpc_notify] 
>> 0-novashared-server: disconnecting connectionfrom 
>> cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
>> [2014-03-18 21:56:22.215898] I 
>> [server-helpers.c:729:server_connection_put] 0-novashared-server: 
>> Shutting down connection 
>> cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
>> [2014-03-18 21:56:22.215923] I 
>> [server-helpers.c:617:server_connection_destroy] 0-novashared-server: 
>> destroyed connection of 
>> cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
>> [2014-03-18 21:57:04.821397] I 
>> [server-handshake.c:567:server_setvolume] 0-novashared-server: 
>> accepted client from 
>> cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 (version: 
>> 3.4.2)
>> [2014-03-19 16:48:32.135593] I [server.c:762:server_rpc_notify] 
>> 0-novashared-server: disconnecting connectionfrom 
>> cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
>> [2014-03-19 16:48:32.135629] I 
>> [server-helpers.c:729:server_connection_put] 0-novashared-server: 
>> Shutting down connection 
>> cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
>> [2014-03-19 16:48:32.135652] I 
>> [server-helpers.c:617:server_connection_destroy] 0-novashared-server: 
>> destroyed connection of 
>> cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
>> [2014-03-19 16:48:47.712973] I 
>> [server-handshake.c:567:server_setvolume] 0-novashared-server: 
>> accepted client from 
>> cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 
>> (version: 3.4.2)
>> [2014-03-19 18:07:44.332240] I [server.c:762:server_rpc_notify] 
>> 0-novashared-server: disconnecting connectionfrom 
>> cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
>> [2014-03-19 18:07:44.332272] I 
>> [server-helpers.c:729:server_connection_put] 0-novashared-server: 
>> Shutting down connection 
>> cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
>> [2014-03-19 18:07:44.332296] I 
>> [server-helpers.c:617:server_connection_destroy] 0-novashared-server: 
>> destroyed connection of 
>> cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
>> [2014-03-19 19:00:51.369205] I 
>> [server-handshake.c:567:server_setvolume] 0-novashared-server: 
>> accepted client from 
>> cm6-p-21755-2014/03/19-19:00:51:286018-novashared-client-0-0 
>> (version: 3.4.2)
>>
>>
>> These are all on 3.4.2 with Ubuntu 13.10.
>>
>>
>> -ryan
>>
>>
>> On Mar 19, 2014, at 11:15 AM, Carlos Capriotti 
>> <capriotti.carlos at gmail.com <mailto:capriotti.carlos at gmail.com>> wrote:
>>
>>> This rings a bell...
>>>
>>> I mapped a remote nfs share to one of my nodes and mapped the local nfs
>>> share, to the "local" gluster volume as well and started to copy 
>>> (with cp)
>>> files from one mount to the other.
>>>
>>> Somewhere down the line all stalled, the server was irresponsive, even
>>> restarting glusterd solved nothing. Had to reboot the nodes.
>>>
>>> CentOS 6.5 and Gluster 3.4.2.
>>>
>>> Since then,  I haven't tried those copies, but, I also, I am now 
>>> using the
>>> native glusterfs client just in case.
>>>
>>>
>>>
>>>
>>> On Wed, Mar 19, 2014 at 3:51 PM, Ray Powell <rpowell1 at uchicago.edu 
>>> <mailto:rpowell1 at uchicago.edu>> wrote:
>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA512
>>>>
>>>> I have not seen that with the fuse client.  I had it happen repeatedly
>>>> when we were using gluster/NFS for /var/lib/nova/instances and/or
>>>> /var/lib/nova/instances/_base.  The access to instances-XXXXXX/disk
>>>> (or its qcow2 backing image) would hang completely or be very slow
>>>> under heavy load.  The VM interprets the delay as a problem on
>>>> /dev/vda and remounts RO.
>>>>
>>>>
>>>> On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote:
>>>>> I was wondering if anyone had seen similar issues before, when
>>>>> booting a vm with a root volume on local disk things work fine.
>>>>>
>>>>> When I try to do the same with /var/lib/nova/instances mounted via
>>>>> gluster/fuse I get the following: http://paste.ubuntu.com/7116594/
>>>>>
>>>>> I have of course insured perms were correct with chown -R nova:nova
>>>>> /var/lib/nova/instances, additionally app armor is completely
>>>>> unloaded/disabled.
>>>>>
>>>>> Has anyone encountered this before? Gluster version is 3.4.2
>>>>>
>>>>>
>>>>> -ryan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________ Gluster-users
>>>>> mailing list Gluster-users at gluster.org 
>>>>> <mailto:Gluster-users at gluster.org>
>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>> - --
>>>> Raymond Powell
>>>> Systems Administrator
>>>> Laboratory for Advanced Computing
>>>> University of Chicago, KCBD 10160
>>>> 900 East 57th Street
>>>> Chicago, IL 60637
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>>>> Comment: GPGTools -http://gpgtools.org <http://gpgtools.org/>
>>>> Comment: Using GnuPG with Thunderbird -http://www.enigmail.net/
>>>>
>>>> iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9
>>>> lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy
>>>> pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B
>>>> YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm
>>>> eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg
>>>> huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/
>>>> HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD
>>>> lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa
>>>> sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA
>>>> o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF
>>>> JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe
>>>> +UhAq6+74ie1MokNssB8
>>>> =un7j
>>>> -----END PGP SIGNATURE-----
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140325/645f22f0/attachment.html>

More information about the Gluster-users mailing list