[Gluster-users] Several questions from replicas to performance

[Dean Bruhn]

Elias, 
	I would suggest testing your performance of your bricks directly and see if you have an issue there, verify all of your building blocks and them worry about gluster. XFS is the suggested starting point, a lot of people have avoided EXT4 because of some upstream issues that happened with the kernel development a while back, my understanding is that it is stable and working, but most people still shy away from it because of the previous issue. 

 There is a writeup on ZFS on the wiki, I have not personally used it, but I have seen successful reports, and reports of better performance over XFS. Again unverified by me. 

- Dean 



On Jan 23, 2014, at 9:17 AM, Elías David <elias.moreno.tec at gmail.com> wrote:

> Thanks Dean, I've tested that this morning and sure enough is better but still I think I'm missing something.
> 
> Having replica 2 I tested the same dd if=/dev/zero of=/mnt/data bs=1m count=50000 and it took 60 minutes to complete at 14 MB/s
> 
> Running iperf against the server I get something around 936 Mbits/sec so I'm wondering if theres something wrong with my setup (be it replica 2 or not), could it be xfs? I've read of people who were more lucky with ext4 for instance. Could it be lack of tuning of the gluster vol file?
> 
> Thanks again!
> 
> On Jan 23, 2014 10:04 AM, "Dean Bruhn" <deanmichael at artisnotcrime.com> wrote:
> Elias,
>         Looks like you’ve got your replication setup turned around. The replica count is the number of times you want the data replicated across the volume. So right now, for every file you write, it is being written to 5 bricks, I would suspect you want more of a replica 2. You would modify your volume create command to look more like this
> 
> gluster vol create replica 2 server1:/gv0/vol1/data server2:/gv0/vol1/data server3:/gv0/vol1/data server4:/gv0/vol1/data server5:/gv0/vol1/data server1:/gv0/vol2/data server2:/gv0/vol2/data server3:/gv0/vol2/data server4:/gv0/vol2/data server5:/gv0/vol2/data
> 
> - Dean
> 
> 
> 
> 
> On Jan 22, 2014, at 11:33 PM, Elías David <elias.moreno.tec at gmail.com> wrote:
> 
> > Hello everyone,
> >
> > The place I work for is starting to look at gluster to replace a current windows share we have. The amount of files and sizes varies a lot, from very small (~5kb) to somewhat large (~25GB).
> >
> > Since we're checking if gluster is a viable option for us, and we're still learning about the filesystem, we're pretty sure that a problem we're seeing right now is coming from our ignorance.
> >
> > Right now our biggest concern during our tests is a ridiculously low performance, I'm talking about a 'cp -R /home/user/* /mnt/data' where /mnt/data is: mount -t glusterfs 192.168.0.10:/VolName /mnt/data that took something ridiculous like 12 "hours" or more to transfer mere 226GB of data (from ISOs to documents to flat files....).
> >
> > Right now our setup is this:
> > -We have 5 servers (peers) with each having two 2TB disks WD black formatted with xfs -i size=512, each disk is a brick so we have:
> >
> > 192.168.0.10 disk0 (2TB) on /gv0/vol1
> > 192.168.0.10 disk1 (2TB) on /gv0/vol2
> > 192.168.0.11 disk0 (2TB) on /gv0/vol1
> > 192.168.0.11 disk1 (2TB) on /gv0/vol2
> > and so on...
> >
> > Total: 2TB disk x 10 = 20TB
> >
> > Now, not being really sure yet how replica counts really work, we created a volume with "replica 5", as in:
> >
> > gluster vol create replica 5 server1:/gv0/vol1/data server2:/gv0/vol1/data server3:/gv0/vol1/data server4:/gv0/vol1/data server5:/gv0/vol1/data server1:/gv0/vol2/data server2:/gv0/vol2/data server3:/gv0/vol2/data server4:/gv0/vol2/data server5:/gv0/vol2/data
> >
> > Vol info goes like this:
> >
> > Volume Name: Data
> > Type: Distributed-Replicate
> > Volume ID: 2c938585-d2bd-43cf-98d8-caab70033750
> > Status: Started
> > Number of Bricks: 2 x 5 = 10
> > Transport-type: tcp
> > Bricks:
> > Brick1: 192.168.0.10:/gv0/vol1/data
> > Brick2: 192.168.0.11:/gv0/vol1/data
> > Brick3: 192.168.0.12:/gv0/vol1/data
> > Brick4: 192.168.0.13:/gv0/vol1/data
> > Brick5: 192.168.0.14:/gv0/vol1/data
> > Brick6: 192.168.0.10:/gv0/vol2/data
> > Brick7: 192.168.0.11:/gv0/vol2/data
> > Brick8: 192.168.0.12:/gv0/vol2/data
> > Brick9: 192.168.0.13:/gv0/vol2/data
> > Brick10: 192.168.0.14:/gv0/vol2/data
> >
> > The servers are not bad really, Intel(R) Xeon(R) CPU X5450  @ 3.00GHz 8 cores, 32 GB of ram and 1GB link for gluster
> >
> > As I said earlier I mounted this vol on another machine in the lan using 'mount -t glusterfs 192.168.0.10:/Data /mnt/data' and I use a simple cp -R to put data on it, I also tested with 'dd if=/dev/zero of=/mnt/data/zerofile bs=1M count=50000' and this dd process is running four about 5 hours now and I'm about to reach 24GB of 50GB file size...
> >
> > I'm pretty sure that this problem is solely caused from our ignorance of the filesystem and that's why I ask you guys
> >
> > The servers are all running CentOS 6.5, glusterfs-* packages from EPEL repo, glusterfs version 3.4.2
> >
> > Another question I would like to add if I may is that, I run df -h after mounting the volume and I'm seeing as total volume capacity of 3.7 terabytes when I expected something like 10 terabytes given 10 2TB disks in a replica 5 setup, is this normal or I'm misunderstanding the replica count thing?
> >
> > That's it, sorry for the long message just wanted to be clear, any input or info about this would be greatly appreciated.
> >
> > Thanks!
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140123/5ac80d97/attachment.html>