[Gluster-users] Disastrous performance with rsync to mounted Gluster volume.

[Joe Julian]

On 04/24/2015 11:03 AM, Ben Turner wrote:
> ----- Original Message -----
>> From: "Ernie Dunbar" <maillist at lightspeed.ca>
>> To: "Gluster Users" <gluster-users at gluster.org>
>> Sent: Friday, April 24, 2015 1:15:32 PM
>> Subject: Re: [Gluster-users] Disastrous performance with rsync to mounted Gluster volume.
>>
>> On 2015-04-23 18:10, Joe Julian wrote:
>>> On 04/23/2015 04:41 PM, Ernie Dunbar wrote:
>>>> On 2015-04-23 12:58, Ben Turner wrote:
>>>>
>>>>> +1, lets nuke everything and start from a known good.  Those error
>>>>> messages make me think something is really wrong with how we are
>>>>> copying the data.  Gluster does NFS by default so you shouldn't have
>>>>> have to reconfigure anything after you recreate the volume.
>>>>>
>>>> Okay... this is a silly question. How do I do that? Deleting the
>>>> volume doesn't affect the files in the underlying filesystem, and I
>>>> get the impression that trying to delete the files in the underlying
>>>> filesystem without shutting down or deleting the volume would result
>>>> in Gluster trying to write the files back where they "belong".
>>>>
>>>> Should I stop the volume, delete it, then delete the files and start
>>>> from scratch, re-creating the volume?
>>> That's what I would do.
>>>
>> Well, apparently removing the .glusterfs directory from the brick is an
>> exceptionally bad thing, and breaks gluster completely, rendering it
>> inoperable. I'm going to have to post another thread about how to fix
>> this mess now.
> You are correct and I would just start from scratch Ernie.  Creating a gluster cluster is only about 3-4 commands and should only take a minute or two.  Also with all the problems you are having I am not confident in your data integrity.  All you need to do to clear EVERYTHING out is:
>
> service glusterd stop
> killall glusterfsd
> killall glusterfs
> sleep 1
> for file in /var/lib/glusterd/*; do if ! echo $file | grep 'hooks' >/dev/null 2>&1;then rm -rf $file; fi; done
>
>  From there restart the gluster service and recreate everything:
>
> service glusterd restart
> <make a new filesystem on your bricks, mount>
> gluster peer probe <my peer>
> gluster v create <my vol>
> gluster v start <my vol>
> gluster v info
>
>  From there mount the new volume on your system with the data  you want to migrate:
>
> mount -t nfs -o vers=3 <my vol> <my mount>
> rsync <your rsync command>

And your rsync command should include, "--inplace".
>
> This should get you where you need to be.  Before you start to migrate the data maybe do a couple DDs and send me the output so we can get an idea of how your cluster performs:
>
> time `dd if=/dev/zero of=<gluster-mount>/myfile bs=1024k count=1000; sync`
> echo 3 > /proc/sys/vm/drop_caches
> dd if=<gluster mount> of=/dev/null bs=1024k count=1000
>
> If you are using gigabit and glusterfs mounts with replica 2 you should get ~55 MB / sec writes and ~110 MB / sec reads.  With NFS you will take a bit of a hit since NFS doesnt know where files live like glusterfs does.
>
> -b
>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users