[Gluster-users] Not giving up [ was: Re: read-subvolume]

Thu Jul 11 05:45:06 UTC 2013

Hi all - especially Jeff, Marcus and HL

I couldn't resist a quick test after compiling 3.4 beta. Looks
good. Same (very quick) times to do md5sums on both servers so it must
be doing local reads. So gluster is still in the running.

I repeat - you guys are doing a great job. Software like gluster is not
trivial. You are making great progress. My comments are not a rant and
are intended to be helpful.

All the best and many thanks again

Allan

On 10/07/13 17:55, HL wrote:
> Been there ...
> here is my 10cent advise
> a) Prepare for tomorrow
> b) Rest
> c) Think
> d) Plan
> e) act
> 
> I am sure it will work for you when calmed
> 
> Tech hints.
> ifconfig iface mtu 9000 or whatever your nic can afford
> Having a 100Mbit is not a good idea.
> I 've recently located a dual port 1Gbit nic on ebay for $15 USD
> Get them.
> 
> And last but not least
> In case you happen to have a switch between the nodes make sure that you
> enable jumbo frames on it.
> Otherwise you are in DEEP Trouble.
> 
> stat' ing a 120G file in my case takes miliseconds not even seconds.
> 
> GL
> Harry.
> 
> 
> On 10/07/2013 02:01 μμ, Allan Latham wrote:
>> Hi all
>>
>> Thanks to all those volunteers who are working to get gluster into a
>> state where it can be used for live work.
>>
>> I understand that you are giving your free time and I very much
>> appreciate it on this project and the many others we use for live
>> production work.
>>
>> There seems to be a problem with the way gluster is going.
>> For me it would be an ideal solution if it actually worked.
>>
>> I have a simple scenario and it just simply doesn't work. Reading over
>> the network when the file is available locally is plainly wrong. Our
>> application cannot take the performance hit nor the extra network
>> traffic.
>>
>> I would suggest:
>>
>> 1. get a simple minimalist configuration working - 2 hosts and
>> replication only.
>> 2. make it bomb-proof.
>> 2a. it must cope with network failures, random reboots etc.
>> 2b. if it stops it has to auto-recover quickly.
>> 2c. if it can't it needs thorough documentation and adequate logs so a
>> reasonable sysop can rescue it.
>> 2d. it needs a fast validation scanner which verifies that data is where
>> it should be and is identical everywhere (md5sum).
>> 3. make it efficient (read local whenever possible - use rsync
>> techniques - remove scalability obstacles so it doesn't get
>> exponentially slower as more files are replicated)
>> 4. when that works expand to multiple hosts and clever distribution
>> techniques.
>> (repeat items 2 and 3 in the more complex environment)
>>
>> If it doesn't work rock solid in a simple scenario it will never work in
>> a large scale cluster.
>>
>> Until point 3 is reached I cannot use it - which is a great
>> disappointment for me as well as the good guys doing the development.
>>
>> Good luck and thanks again
>>
>> Allan
>>
>>
>> On 04/07/13 13:10, Allan Latham wrote:
>>> Hi all
>>>
>>> Does anyone use read-subvolume?
>>>
>>> Has anyone tested read-subvolume?
>>>
>>> Does read-subvolume work in such a way that if the file is present on
>>> the local node the local copy is used rather than a remote one?
>>>
>>> Alternatively is there any way to configure (or patch) gluster to always
>>> prefer the local file?
>>>
>>> I have read everything available and have found no answer.
>>>
>>> Unison works very well in our environment but is not real time and needs
>>> to be run every few minutes and/or be kicked off with inotify.
>>>
>>> If I could get gluster to always read the local copy it would be a much
>>> better drop in replacement.
>>>
>>> This is a small scale deployment not a massive cluster but I can imagine
>>> there are many potential users of gluster in this mode. It should beat
>>> unison and similar solutions in every way - but it doesn't because it is
>>> reading from the network even when it has a local up-to-date copy. This
>>> can't be intended behaviour.
>>>
>>> So what have I configured wrong?
>>>
>>> Thanks in advance
>>>
>>> Allan
>>>
>>>
>>> On 02/07/13 13:38, Allan Latham wrote:
>>>> Hi everyone
>>>>
>>>> I have installed 3.3.1-1 from the Debian repository you provide.
>>>>
>>>> I am using a simple 2 node cluster and running in replication mode. The
>>>> connection between the nodes is limited to 100MB/sec (that's bits not
>>>> bytes!). Usage will be mainly for read access and since there is always
>>>> a local copy available [ exactly 2 replicas on exactly 2 machines ] I
>>>> expect very fast read performance. Writes are low volume and very
>>>> infrequent - performance is not an issue.
>>>>
>>>> Almost everything works as I would expect.
>>>>
>>>> Write speed is limited to 10Mb (bytes) per second which is what I would
>>>> expect and is adequate for the application.
>>>>
>>>> But read speed is either super fast or 10Mb/sec. i.e. read operations
>>>> take place on the local copy or the remote seemingly at random.
>>>>
>>>> This not the 'small files problem'. I am aware that Gluster must use
>>>> network access for stat() etc. This is all about where the data comes
>>>> from on a read(). If I do an m5dum on a 200Mb file it takes either half
>>>> a second or 18 seconds.
>>>>
>>>> There is an option read-subvolume.
>>>>
>>>> I have tried to understand how this works from the documentation
>>>> available and from the few examples on the web.
>>>>
>>>> I have added the option using:
>>>>
>>>> gluster volume set X read-subvolume Y
>>>>
>>>> It has no effect even after stopping and starting the volume,
>>>> remounting, restarting gluster servers etc.
>>>>
>>>> What's more I fail to see how this option could ever work at all. The
>>>> configuration changes caused by the above command are rolled out to
>>>> both
>>>> nodes - but what is right for one node is exactly the wrong
>>>> configuration for the other node.
>>>>
>>>> Configs attached are in /var/lib/glusterd/vols/shared except
>>>> glusterd.vol which is in /etc/glusterfs.
>>>>
>>>> Here is the output of the mount command filtered to just the glusterfs
>>>> mount:
>>>>
>>>> 10.255.255.1:/shared on /gluster/rw/shared type fuse.glusterfs
>>>> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
>>>>
>>>>
>>>> 10.255.255.1 is local to this host.
>>>>
>>>> I would be very thankful if someone can enlighten me. I am obviously
>>>> configuring this wrong. I may have missed something important.
>>>>
>>>> Best regards to all
>>>>
>>>> Allan
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 