[Gluster-devel] Re: OS X client wrong disk size in combination with DHT

Ate Poorthuis atepoorthuis at gmail.com
Mon Jul 6 09:21:22 UTC 2009


A little extra information after experimenting with the distribute sequence.
The outputted disk size is NOT that of the first brick. In the below
mentioned setup, I have 4 bricks (2 are 4.5TB and 2 are 6.3TB). If I only
add 2 or 3 bricks to the dht translator disk size is outputted correctly,
regardless of which bricks I add - the same errors remain in the logs
though. Upon adding a fourth, disk size is set to 5.4TB, which happens to be
the size of the largest and the smallest brick divided by two.

On Fri, Jul 3, 2009 at 2:45 PM, Ate Poorthuis <atepoorthuis at gmail.com>wrote:

> Hi,
>
> Using gluster 2.0.2 client os OS X 10.5. Disk size is not outputted
> correctly, only the size of the first brick in dht is counted. Used disk
> space is calculated correctly.
>
> OS X client: /Users/ate/glusterfs_ip.vol  5.4Ti  1.1Ti  3.2Ti    26%
> /mnt/gluster
> Debian client: /etc/glusterfs/client_st.vol  22T  1.1T   20T   6%
> /mnt/gluster
>
> There are also some errors from dht-diskusage in the log about full disks.
> The % differs each time I mount (ranging from 96-100%).
>
>
> ================================================================================
> Version      : glusterfs 2.0.2 built on Jun 30 2009 14:14:33
> TLA Revision : 07019da2e16534d527215a91904298ede09bb798
> Starting Time: 2009-07-03 14:40:34
> Command line : /usr/local/gluster/sbin/glusterfs --debug -f
> /Users/ate/glusterfs_ip.vol /mnt/gluster/
> PID          : 97785
> System name  : Darwin
> Nodename     : ate-poorthuiss-macbook-2.local
> Kernel Release : 9.7.0
> Hardware Identifier: i386
>
> Given volfile:
>
> +------------------------------------------------------------------------------+
>   1: ### file: client-volume.vol
>   2:
>   3: #####################################
>   4: ###  GlusterFS Client Volume File  ##
>   5: #####################################
>   6:
>   7: #### CONFIG FILE RULES:
>   8: ### "#" is comment character.
>   9: ### - Config file is case sensitive
>  10: ### - Options within a volume block can be in any order.
>  11: ### - Spaces or tabs are used as delimitter within a line.
>  12: ### - Each option should end within a line.
>  13: ### - Missing or commented fields will assume default values.
>  14: ### - Blank/commented lines are allowed.
>  15: ### - Sub-volumes should already be defined above before referring.
>  16:
>  17: ### Add client feature and attach to remote subvolume
>  18: volume gfs-001-afr1
>  19:   type protocol/client
>  20:   option transport-type tcp
>  21:   option remote-host 10.0.0.30         # IP address of the remote
> brick
>  22:   option remote-subvolume afr1        # name of the remote volume
>  23: #  option ping-timeout 5
>  24: end-volume
>  25: volume gfs-001-afr2
>  26:   type protocol/client
>  27:   option transport-type tcp
>  28:   option remote-host 10.0.0.30         # IP address of the remote
> brick
>  29:   option remote-subvolume afr2        # name of the remote volume
>  30: #  option ping-timeout 5
>  31: end-volume
>  32: volume gfs-002-afr1
>  33:   type protocol/client
>  34:   option transport-type tcp
>  35:   option remote-host 10.0.0.32         # IP address of the remote
> brick
>  36:   option remote-subvolume afr1        # name of the remote volume
>  37: #  option ping-timeout 5
>  38: end-volume
>  39: volume gfs-002-afr2
>  40:   type protocol/client
>  41:   option transport-type tcp
>  42:   option remote-host 10.0.0.32         # IP address of the remote
> brick
>  43:   option remote-subvolume afr2        # name of the remote volume
>  44: #  option ping-timeout 5
>  45: end-volume
>  46:
>  47: volume bricks
>  48:   type cluster/distribute
>  49: #  option lookup-unhashed yes
>  50:   option min-free-disk 5%
>  51:   subvolumes gfs-001-afr1 gfs-001-afr2 gfs-002-afr1 gfs-002-afr2
>  52: end-volume
>  53:
>  54: ### Add readahead feature
>  55: volume readahead
>  56:   type performance/read-ahead
>  57:   option page-count 8       # cache per file  = (page-count x
> page-size)
>  58:   subvolumes bricks
>  59: end-volume
>  60:
>  61: ### Add IO-Cache feature
>  62: volume iocache
>  63:   type performance/io-cache
>  64:   option cache-size 256MB
>  65:   subvolumes readahead
>  66: end-volume
>  67: volume iothreads
>  68:   type performance/io-threads
>  69:   option thread-count 32  # default is 1
>  70:   subvolumes iocache
>  71: end-volume
>  72: ### Add writeback feature
>  73: volume writeback
>  74:   type performance/write-behind
>  75:   option cache-size 2MB
>  76:   option flush-behind off
>  77:   subvolumes iothreads
>  78: end-volume
>  79:
>
>
> +------------------------------------------------------------------------------+
> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
> dlsym(notify) on dlsym(0x101730, notify): symbol not found -- neglecting
> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
> dlsym(notify) on dlsym(0x101a40, notify): symbol not found -- neglecting
> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
> dlsym(notify) on dlsym(0x101d50, notify): symbol not found -- neglecting
> [2009-07-03 14:40:34] D [xlator.c:634:xlator_set_type] xlator:
> dlsym(notify) on dlsym(0x102060, notify): symbol not found -- neglecting
> [2009-07-03 14:40:34] D [glusterfsd.c:256:_add_fuse_mount] glusterfs:
> 'direct-io-mode' in fuse causes data corruption if O_APPEND is used.
> disabling 'direct-io-mode'
> [2009-07-03 14:40:34] D [glusterfsd.c:1179:main] glusterfs: running in pid
> 97785
> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-001-afr1:
> defaulting frame-timeout to 30mins
> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-001-afr1:
> defaulting ping-timeout to 10
> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-001-afr2:
> defaulting frame-timeout to 30mins
> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-001-afr2:
> defaulting ping-timeout to 10
> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-002-afr1:
> defaulting frame-timeout to 30mins
> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-002-afr1:
> defaulting ping-timeout to 10
> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
> [2009-07-03 14:40:34] D [client-protocol.c:5948:init] gfs-002-afr2:
> defaulting frame-timeout to 30mins
> [2009-07-03 14:40:34] D [client-protocol.c:5959:init] gfs-002-afr2:
> defaulting ping-timeout to 10
> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
> [2009-07-03 14:40:34] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/gluster/lib/glusterfs/2.0.2/transport/socket.so
> [2009-07-03 14:40:34] D [read-ahead.c:786:init] readahead: Using
> conf->page_count = 8
> [2009-07-03 14:40:34] D [io-threads.c:2280:init] iothreads: io-threads:
> Autoscaling: off, min_threads: 32, max_threads: 32
> [2009-07-03 14:40:34] D [write-behind.c:1859:init] writeback: disabling
> write-behind for first 1 bytes
> [2009-07-03 14:40:34] D [dict.c:297:dict_get] dict: @this=0x0 @key=0x22dd0b
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-001-afr2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] D [client-protocol.c:6276:notify] gfs-002-afr2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-07-03 14:40:34] N [glusterfsd.c:1198:main] glusterfs: Successfully
> started
> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr1: got
> GF_EVENT_CHILD_UP
> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr1: got
> GF_EVENT_CHILD_UP
> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr2: got
> GF_EVENT_CHILD_UP
> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-001-afr2: got
> GF_EVENT_CHILD_UP
> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr1: got
> GF_EVENT_CHILD_UP
> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr1: got
> GF_EVENT_CHILD_UP
> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr2: got
> GF_EVENT_CHILD_UP
> [2009-07-03 14:40:34] D [client-protocol.c:6290:notify] gfs-002-afr2: got
> GF_EVENT_CHILD_UP
> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
> gfs-001-afr1: Connected to 10.0.0.30:6996, attached to remote volume
> 'afr1'.
> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
> gfs-001-afr1: Connected to 10.0.0.30:6996, attached to remote volume
> 'afr1'.
> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
> gfs-001-afr2: Connected to 10.0.0.30:6996, attached to remote volume
> 'afr2'.
> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
> gfs-002-afr1: Connected to 10.0.0.32:6996, attached to remote volume
> 'afr1'.
> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
> gfs-002-afr1: Connected to 10.0.0.32:6996, attached to remote volume
> 'afr1'.
> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
> gfs-002-afr2: Connected to 10.0.0.32:6996, attached to remote volume
> 'afr2'.
> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
> gfs-002-afr2: Connected to 10.0.0.32:6996, attached to remote volume
> 'afr2'.
> [2009-07-03 14:40:34] N [client-protocol.c:5551:client_setvolume_cbk]
> gfs-001-afr2: Connected to 10.0.0.30:6996, attached to remote volume
> 'afr2'.
> [2009-07-03 14:40:35] D [dht-common.c:1405:dht_err_cbk] bricks: subvolume
> gfs-001-afr1 returned -1 (Invalid argument)
> [2009-07-03 14:40:51] C [dht-diskusage.c:197:dht_is_subvol_filled] bricks:
> disk space on subvolume 'gfs-001-afr1' is getting full (97.00 %), consider
> adding more nodes
> [2009-07-03 14:40:51] W
> [dht-diskusage.c:231:dht_free_disk_available_subvol] bricks: No subvolume
> has enough free space to create
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090706/972351dd/attachment-0003.html>


More information about the Gluster-devel mailing list