[Gluster-devel] Re: OS X client wrong disk size in combination with DHT
Amar Tumballi
amar at gluster.com
Tue Jul 7 05:40:03 UTC 2009
Hi Ate,
The problem here is that, in OSX the OS itself restricts the 'df' output
to 16TB, after which it is overflowed (~22 - 16 = ~5.4). This behavior is a
limitation from the OSX itself (I traced it through fuse stack and asked the
same question in macfuse mailing list too). I suspect we can't solve the
issue within the glusterfs + fuse space.
Regards,
On Mon, Jul 6, 2009 at 2:21 AM, Ate Poorthuis <atepoorthuis at gmail.com>wrote:
> A little extra information after experimenting with the distribute
> sequence. The outputted disk size is NOT that of the first brick. In the
> below mentioned setup, I have 4 bricks (2 are 4.5TB and 2 are 6.3TB). If I
> only add 2 or 3 bricks to the dht translator disk size is outputted
> correctly, regardless of which bricks I add - the same errors remain in the
> logs though. Upon adding a fourth, disk size is set to 5.4TB, which happens
> to be the size of the largest and the smallest brick divided by two.
>
>
> On Fri, Jul 3, 2009 at 2:45 PM, Ate Poorthuis <atepoorthuis at gmail.com>wrote:
>
>> Hi,
>>
>> Using gluster 2.0.2 client os OS X 10.5. Disk size is not outputted
>> correctly, only the size of the first brick in dht is counted. Used disk
>> space is calculated correctly.
>>
>> OS X client: /Users/ate/glusterfs_ip.vol 5.4Ti 1.1Ti 3.2Ti 26%
>> /mnt/gluster
>> Debian client: /etc/glusterfs/client_st.vol 22T 1.1T 20T 6%
>> /mnt/gluster
>>
>> There are also some errors from dht-diskusage in the log about full disks.
>> The % differs each time I mount (ranging from 96-100%).
>>
>>
>> ================================================================================
>> Version : glusterfs 2.0.2 built on Jun 30 2009 14:14:33
>> TLA Revision : 07019da2e16534d527215a91904298ede09bb798
>> Starting Time: 2009-07-03 14:40:34
>> Command line : /usr/local/gluster/sbin/glusterfs --debug -f
>> /Users/ate/glusterfs_ip.vol /mnt/gluster/
>> PID : 97785
>> System name : Darwin
>> Nodename : ate-poorthuiss-macbook-2.local
>> Kernel Release : 9.7.0
>> Hardware Identifier: i386
>>
>> Given volfile:
>>
>> +------------------------------------------------------------------------------+
>> 1: ### file: client-volume.vol
>> 2:
>> 3: #####################################
>> 4: ### GlusterFS Client Volume File ##
>> 5: #####################################
>> 6:
>> 7: #### CONFIG FILE RULES:
>> 8: ### "#" is comment character.
>> 9: ### - Config file is case sensitive
>> 10: ### - Options within a volume block can be in any order.
>> 11: ### - Spaces or tabs are used as delimitter within a line.
>> 12: ### - Each option should end within a line.
>> 13: ### - Missing or commented fields will assume default values.
>> 14: ### - Blank/commented lines are allowed.
>> 15: ### - Sub-volumes should already be defined above before referring.
>> 16:
>> 17: ### Add client feature and attach to remote subvolume
>> 18: volume gfs-001-afr1
>> 19: type protocol/client
>> 20: option transport-type tcp
>> 21: option remote-host 10.0.0.30 # IP address of the remote
>> brick
>> 22: option remote-subvolume afr1 # name of the remote volume
>> 23: # option ping-timeout 5
>> 24: end-volume
>> 25: volume gfs-001-afr2
>> 26: type protocol/client
>> 27: option transport-type tcp
>> 28: option remote-host 10.0.0.30 # IP address of the remote
>> brick
>> 29: option remote-subvolume afr2 # name of the remote volume
>> 30: # option ping-timeout 5
>> 31: end-volume
>> 32: volume gfs-002-afr1
>> 33: type protocol/client
>> 34: option transport-type tcp
>> 35: option remote-host 10.0.0.32 # IP address of the remote
>> brick
>> 36: option remote-subvolume afr1 # name of the remote volume
>> 37: # option ping-timeout 5
>> 38: end-volume
>> 39: volume gfs-002-afr2
>> 40: type protocol/client
>> 41: option transport-type tcp
>> 42: option remote-host 10.0.0.32 # IP address of the remote
>> brick
>> 43: option remote-subvolume afr2 # name of the remote volume
>> 44: # option ping-timeout 5
>> 45: end-volume
>> 46:
>> 47: volume bricks
>> 48: type cluster/distribute
>> 49: # option lookup-unhashed yes
>> 50: option min-free-disk 5%
>> 51: subvolumes gfs-001-afr1 gfs-001-afr2 gfs-002-afr1 gfs-002-afr2
>> 52: end-volume
>> 53:
>> 54: ### Add readahead feature
>> 55: volume readahead
>> 56: type performance/read-ahead
>> 57: option page-count 8 # cache per file = (page-count x
>> page-size)
>> 58: subvolumes bricks
>> 59: end-volume
>> 60:
>> 61: ### Add IO-Cache feature
>> 62: volume iocache
>> 63: type performance/io-cache
>> 64: option cache-size 256MB
>> 65: subvolumes readahead
>> 66: end-volume
>> 67: volume iothreads
>> 68: type performance/io-threads
>> 69: option thread-count 32 # default is 1
>> 70: subvolumes iocache
>> 71: end-volume
>> 72: ### Add writeback feature
>> 73: volume writeback
>> 74: type performance/write-behind
>> 75: option cache-size 2MB
>> 76: option flush-behind off
>> 77: subvolumes iothreads
>> 78: end-volume
>> 79:
>>
>>
>> +------------------------------------------------------------------------------+
>> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
>> dlsym(notify) on dlsym(0x101730, notify): symbol not found -- neglecting
>> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
>> dlsym(notify) on dlsym(0x101a40, notify): symbol not found -- neglecting
>> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
>> dlsym(notify) on dlsym(0x101d50, notify): symbol not found -- neglecting
>> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
>> dlsym(notify) on dlsym(0x102060, notify): symbol not found -- neglecting
>> [2009-07-03 14:40:34] D [glusterfsd.c:256:_add_fuse_mount] glusterfs:
>> 'direct-io-mode' in fuse causes data corruption if O_APPEND is used.
>> disabling 'direct-io-mode'
>> [2009-07-03 14:40:34] D [glusterfsd.c:1179:main] glusterfs: running in pid
>> 97785
>> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-001-afr1:
>> defaulting frame-timeout to 30mins
>> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-001-afr1:
>> defaulting ping-timeout to 10
>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>> attempt to load file
>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>> attempt to load file
>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-001-afr2:
>> defaulting frame-timeout to 30mins
>> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-001-afr2:
>> defaulting ping-timeout to 10
>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>> attempt to load file
>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>> attempt to load file
>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-002-afr1:
>> defaulting frame-timeout to 30mins
>> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-002-afr1:
>> defaulting ping-timeout to 10
>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>> attempt to load file
>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>> attempt to load file
>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-002-afr2:
>> defaulting frame-timeout to 30mins
>> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-002-afr2:
>> defaulting ping-timeout to 10
>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>> attempt to load file
>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport:
>> attempt to load file
>> /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
>> [2009-07-03 14:40:34] D [read-ahead.c:786:init] readahead: Using
>> conf->page_count = 8
>> [2009-07-03 14:40:34] D [io-threads.c:2280:init] iothreads: io-threads:
>> Autoscaling: off, min_threads: 32, max_threads: 32
>> [2009-07-03 14:40:34] D [write-behind.c:1859:init] writeback: disabling
>> write-behind for first 1 bytes
>> [2009-07-03 14:40:34] D [dict.c:297:dict_get] dict: @this=0x0
>> @key=0x22dd0b
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2: got
>> GF_EVENT_PARENT_UP, attempting connect on transport
>> [2009-07-03 14:40:34] N [glusterfsd.c:1198:main] glusterfs: Successfully
>> started
>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr1: got
>> GF_EVENT_CHILD_UP
>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr1: got
>> GF_EVENT_CHILD_UP
>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr2: got
>> GF_EVENT_CHILD_UP
>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr2: got
>> GF_EVENT_CHILD_UP
>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr1: got
>> GF_EVENT_CHILD_UP
>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr1: got
>> GF_EVENT_CHILD_UP
>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr2: got
>> GF_EVENT_CHILD_UP
>> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr2: got
>> GF_EVENT_CHILD_UP
>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>> gfs-001-afr1: Connected to 10.0.0.30:6996, attached to remote volume
>> 'afr1'.
>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>> gfs-001-afr1: Connected to 10.0.0.30:6996, attached to remote volume
>> 'afr1'.
>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>> gfs-001-afr2: Connected to 10.0.0.30:6996, attached to remote volume
>> 'afr2'.
>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>> gfs-002-afr1: Connected to 10.0.0.32:6996, attached to remote volume
>> 'afr1'.
>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>> gfs-002-afr1: Connected to 10.0.0.32:6996, attached to remote volume
>> 'afr1'.
>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>> gfs-002-afr2: Connected to 10.0.0.32:6996, attached to remote volume
>> 'afr2'.
>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>> gfs-002-afr2: Connected to 10.0.0.32:6996, attached to remote volume
>> 'afr2'.
>> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
>> gfs-001-afr2: Connected to 10.0.0.30:6996, attached to remote volume
>> 'afr2'.
>> [2009-07-03 14:40:35] D [dht-common.c:1405:dht_err_cbk] bricks: subvolume
>> gfs-001-afr1 returned -1 (Invalid argument)
>> [2009-07-03 14:40:51] C [dht-diskusage.c:197:dht_is_subvol_filled] bricks:
>> disk space on subvolume 'gfs-001-afr1' is getting full (97.00 %), consider
>> adding more nodes
>> [2009-07-03 14:40:51] W
>> [dht-diskusage.c:231:dht_free_disk_available_subvol] bricks: No subvolume
>> has enough free space to create
>>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
--
Regards,
Amar Tumballi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090706/9fab6942/attachment-0003.html>
More information about the Gluster-devel
mailing list