[Gluster-devel] Re: OS X client wrong disk size in combination with DHT

Amar Tumballi amar at gluster.com
Tue Jul 7 07:44:23 UTC 2009


Hi Ate,
 Thanks for this info. Let me look into this again.

Regards,

On Tue, Jul 7, 2009 at 12:39 AM, Ate Poorthuis <atepoorthuis at gmail.com>wrote:

> Hi Amar,
>
> Unfortunately, it does not seem to be a problem in the OS itself. If I use
> MacFuse and sshfs, I am getting the right disk size, independent whether or
> not I mount with -o local.
>
> ./sshfs-static-leopard ate at 10.0.0.11:/mnt/gluster /mnt/ssh/
> root at 10.0.0.11:/mnt/gluster   21Ti  4.9Ti   15Ti    25%    /mnt/ssh
>
> Ate
>
>
> On Tue, Jul 7, 2009 at 7:40 AM, Amar Tumballi <amar at gluster.com> wrote:
>
>> Hi Ate,
>>  The problem here is that, in OSX  the OS itself restricts the 'df' output
>> to 16TB, after which it is overflowed (~22 - 16 = ~5.4). This behavior is a
>> limitation from the OSX itself (I traced it through fuse stack and asked the
>> same question in macfuse mailing list too). I suspect we can't solve the
>> issue within the glusterfs + fuse space.
>>
>> Regards,
>>
>> On Mon, Jul 6, 2009 at 2:21 AM, Ate Poorthuis <atepoorthuis at gmail.com>wrote:
>>
>>> A little extra information after experimenting with the distribute
>>> sequence. The outputted disk size is NOT that of the first brick. In the
>>> below mentioned setup, I have 4 bricks (2 are 4.5TB and 2 are 6.3TB). If I
>>> only add 2 or 3 bricks to the dht translator disk size is outputted
>>> correctly, regardless of which bricks I add - the same errors remain in the
>>> logs though. Upon adding a fourth, disk size is set to 5.4TB, which happens
>>> to be the size of the largest and the smallest brick divided by two.
>>>
>>>
>>> On Fri, Jul 3, 2009 at 2:45 PM, Ate Poorthuis <atepoorthuis at gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> Using gluster 2.0.2 client os OS X 10.5. Disk size is not outputted
>>>> correctly, only the size of the first brick in dht is counted. Used disk
>>>> space is calculated correctly.
>>>>
>>>> OS X client: /Users/ate/glusterfs_ip.vol  5.4Ti  1.1Ti  3.2Ti    26%
>>>> /mnt/gluster
>>>> Debian client: /etc/glusterfs/client_st.vol  22T  1.1T   20T   6%
>>>> /mnt/gluster
>>>>
>>>> There are also some errors from dht-diskusage in the log about full
>>>> disks. The % differs each time I mount (ranging from 96-100%).
>>>>
>>>>
>>>> ================================================================================
>>>> Version      : glusterfs 2.0.2 built on Jun 30 2009 14:14:33
>>>> TLA Revision : 07019da2e16534d527215a91904298ede09bb798
>>>> Starting Time: 2009-07-03 14:40:34
>>>> Command line : /usr/local/gluster/sbin/glusterfs --debug -f
>>>> /Users/ate/glusterfs_ip.vol /mnt/gluster/
>>>> PID          : 97785
>>>> System name  : Darwin
>>>> Nodename     : ate-poorthuiss-macbook-2.local
>>>> Kernel Release : 9.7.0
>>>> Hardware Identifier: i386
>>>>
>>>> Given volfile:
>>>>
>>>> +------------------------------------------------------------------------------+
>>>>   1: ### file: client-volume.vol
>>>>   2:
>>>>   3: #####################################
>>>>   4: ###  GlusterFS Client Volume File  ##
>>>>   5: #####################################
>>>>   6:
>>>>   7: #### CONFIG FILE RULES:
>>>>   8: ### "#" is comment character.
>>>>   9: ### - Config file is case sensitive
>>>>  10: ### - Options within a volume block can be in any order.
>>>>  11: ### - Spaces or tabs are used as delimitter within a line.
>>>>  12: ### - Each option should end within a line.
>>>>  13: ### - Missing or commented fields will assume default values.
>>>>  14: ### - Blank/commented lines are allowed.
>>>>  15: ### - Sub-volumes should already be defined above before referring.
>>>>  16:
>>>>  17: ### Add client feature and attach to remote subvolume
>>>>  18: volume gfs-001-afr1
>>>>  19:   type protocol/client
>>>>  20:   option transport-type tcp
>>>>  21:   option remote-host 10.0.0.30         # IP address of the remote
>>>> brick
>>>>  22:   option remote-subvolume afr1        # name of the remote volume
>>>>  23: #  option ping-timeout 5
>>>>  24: end-volume
>>>>  25: volume gfs-001-afr2
>>>>  26:   type protocol/client
>>>>  27:   option transport-type tcp
>>>>  28:   option remote-host 10.0.0.30         # IP address of the remote
>>>> brick
>>>>  29:   option remote-subvolume afr2        # name of the remote volume
>>>>  30: #  option ping-timeout 5
>>>>  31: end-volume
>>>>  32: volume gfs-002-afr1
>>>>  33:   type protocol/client
>>>>  34:   option transport-type tcp
>>>>  35:   option remote-host 10.0.0.32         # IP address of the remote
>>>> brick
>>>>  36:   option remote-subvolume afr1        # name of the remote volume
>>>>  37: #  option ping-timeout 5
>>>>  38: end-volume
>>>>  39: volume gfs-002-afr2
>>>>  40:   type protocol/client
>>>>  41:   option transport-type tcp
>>>>  42:   option remote-host 10.0.0.32         # IP address of the remote
>>>> brick
>>>>  43:   option remote-subvolume afr2        # name of the remote volume
>>>>  44: #  option ping-timeout 5
>>>>  45: end-volume
>>>>  46:
>>>>  47: volume bricks
>>>>  48:   type cluster/distribute
>>>>  49: #  option lookup-unhashed yes
>>>>  50:   option min-free-disk 5%
>>>>  51:   subvolumes gfs-001-afr1 gfs-001-afr2 gfs-002-afr1 gfs-002-afr2
>>>>  52: end-volume
>>>>  53:
>>>>  54: ### Add readahead feature
>>>>  55: volume readahead
>>>>  56:   type performance/read-ahead
>>>>  57:   option page-count 8       # cache per file  = (page-count x
>>>> page-size)
>>>>  58:   subvolumes bricks
>>>>  59: end-volume
>>>>  60:
>>>>  61: ### Add IO-Cache feature
>>>>  62: volume iocache
>>>>  63:   type performance/io-cache
>>>>  64:   option cache-size 256MB
>>>>  65:   subvolumes readahead
>>>>  66: end-volume
>>>>  67: volume iothreads
>>>>  68:   type performance/io-threads
>>>>  69:   option thread-count 32  # default is 1
>>>>  70:   subvolumes iocache
>>>>  71: end-volume
>>>>  72: ### Add writeback feature
>>>>  73: volume writeback
>>>>  74:   type performance/write-behind
>>>>  75:   option cache-size 2MB
>>>>  76:   option flush-behind off
>>>>  77:   subvolumes iothreads
>>>>  78: end-volume
>>>>  79:
>>>>
>>>>
>>>> +------------------------------------------------------------------------------+
>>>> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
>>>> dlsym(notify) on dlsym(0x101730, notify): symbol not found -- neglecting
>>>> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
>>>> dlsym(notify) on dlsym(0x101a40, notify): symbol not found -- neglecting
>>>> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
>>>> dlsym(notify) on dlsym(0x101d50, notify): symbol not found -- neglecting
>>>> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
>>>> dlsym(notify) on dlsym(0x102060, notify): symbol not found -- neglecting
>>>> [2009-07-03 14:40:34] D [glusterfsd.c:256:_add_fuse_mount] glusterfs:
>>>> 'direct-io-mode' in fuse causes data corruption if O_APPEND is used.
>>>> disabling 'direct-io-mode'
>>>> [2009-07-03 14:40:34] D [glusterfsd.c:1179:main] glusterfs: running in
>>>> pid 97785
>>>> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-001-afr1:
>>>> defaulting frame-timeout to 30mins
>>>> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-001-afr1:
>>>> defaulting ping-timeout to 10
>>>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>>>> attempt to load file
>>>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>>>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>>>> attempt to load file
>>>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>>>> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-001-afr2:
>>>> defaulting frame-timeout to 30mins
>>>> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-001-afr2:
>>>> defaulting ping-timeout to 10
>>>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>>>> attempt to load file
>>>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>>>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>>>> attempt to load file
>>>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>>>> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-002-afr1:
>>>> defaulting frame-timeout to 30mins
>>>> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-002-afr1:
>>>> defaulting ping-timeout to 10
>>>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>>>> attempt to load file
>>>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>>>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>>>> attempt to load file
>>>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>>>> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-002-afr2:
>>>> defaulting frame-timeout to 30mins
>>>> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-002-afr2:
>>>> defaulting ping-timeout to 10
>>>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>>>> attempt to load file
>>>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>>>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>>>> attempt to load file
>>>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>>>> [2009-07-03 14:40:34] D [read-ahead.c:786:init] readahead: Using
>>>> conf->page_count = 8
>>>> [2009-07-03 14:40:34] D [io-threads.c:2280:init] iothreads: io-threads:
>>>> Autoscaling: off, min_threads: 32, max_threads: 32
>>>> [2009-07-03 14:40:34] D [write-behind.c:1859:init] writeback: disabling
>>>> write-behind for first 1 bytes
>>>> [2009-07-03 14:40:34] D [dict.c:297:dict_get] dict: @this=0x0
>>>> @key=0x22dd0b
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2:
>>>> got GF_EVENT_PARENT_UP, attempting connect on transport
>>>> [2009-07-03 14:40:34] N [glusterfsd.c:1198:main] glusterfs: Successfully
>>>> started
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr1:
>>>> got GF_EVENT_CHILD_UP
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr1:
>>>> got GF_EVENT_CHILD_UP
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr2:
>>>> got GF_EVENT_CHILD_UP
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr2:
>>>> got GF_EVENT_CHILD_UP
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr1:
>>>> got GF_EVENT_CHILD_UP
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr1:
>>>> got GF_EVENT_CHILD_UP
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr2:
>>>> got GF_EVENT_CHILD_UP
>>>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr2:
>>>> got GF_EVENT_CHILD_UP
>>>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>>>> gfs-001-afr1: Connected to 10.0.0.30:6996, attached to remote volume
>>>> 'afr1'.
>>>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>>>> gfs-001-afr1: Connected to 10.0.0.30:6996, attached to remote volume
>>>> 'afr1'.
>>>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>>>> gfs-001-afr2: Connected to 10.0.0.30:6996, attached to remote volume
>>>> 'afr2'.
>>>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>>>> gfs-002-afr1: Connected to 10.0.0.32:6996, attached to remote volume
>>>> 'afr1'.
>>>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>>>> gfs-002-afr1: Connected to 10.0.0.32:6996, attached to remote volume
>>>> 'afr1'.
>>>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>>>> gfs-002-afr2: Connected to 10.0.0.32:6996, attached to remote volume
>>>> 'afr2'.
>>>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>>>> gfs-002-afr2: Connected to 10.0.0.32:6996, attached to remote volume
>>>> 'afr2'.
>>>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>>>> gfs-001-afr2: Connected to 10.0.0.30:6996, attached to remote volume
>>>> 'afr2'.
>>>> [2009-07-03 14:40:35] D [dht-common.c:1405:dht_err_cbk] bricks:
>>>> subvolume gfs-001-afr1 returned -1 (Invalid argument)
>>>> [2009-07-03 14:40:51] C [dht-diskusage.c:197:dht_is_subvol_filled]
>>>> bricks: disk space on subvolume 'gfs-001-afr1' is getting full (97.00 %),
>>>> consider adding more nodes
>>>> [2009-07-03 14:40:51] W
>>>> [dht-diskusage.c:231:dht_free_disk_available_subvol] bricks: No subvolume
>>>> has enough free space to create
>>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at nongnu.org
>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>>
>>>
>>
>>
>> --
>> Regards,
>> Amar Tumballi
>>
>>
>


-- 
Regards,
Amar Tumballi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090707/a728522e/attachment-0003.html>


More information about the Gluster-devel mailing list