[Gluster-users] Data is Copying when a new brick is added.

[Andrew Hallman]

== Data Copying when a new brick is added. ==

Hi.  I'm a first time glusterfs user, and I'm trying to simulate what will
happen when I need to add more bricks for storage capacity.  My config files
are below but I'll try and explain what is going on.  I have 2 machines with
2 hard drives in each.  I created a replicated storage system where machine
a replicates to machine b.   Everything worked fine when only 1 harddrive
was set up on each machine.   I filled up the harddrives to the point where
there is only 1.5 gigs free on each one.  They match because it was all
replicated.

After this, I added in the configuration for the second harddrive in each
machine.  The machines are still replicating, so the harddrives should be
distributed, which is what I tried.  All is well with the new configuration
if I add new files to the system.  It correctly sees that the old harddrives
are full and puts all the new files onto the second empty drive.

However, if I go into the mount point on the client machine and do a
directory listing (or the shell autocomplete attempts to read it) something
strange happens.  Machine 1 is fine, but machine 2 starts to copy over files
from the full harddrive to the empty one.  It's still copying this time
after I retried the configuration, but the first time it copied about half
the data over to the second harddrive.  The first harddrive remained full
though.

Has anyone experienced this, or can see something wrong with my
configuration files?  The server files are exactly the same on both machines
and the machines have the same hardware components and OS (Ubuntu) as well.
Any help would be appreciated.


===========================================================
Server Configuration
===========================================================

  volume posix1
    type storage/posix
    option directory /data/export
  end-volume
  volume posix2
    type storage/posix
    option directory /media/disk2
  end-volume

  volume locks
    type features/locks
    subvolumes posix1
  end-volume

  volume locks2
    type features/locks
    subvolumes posix2
  end-volume

  volume brick
    type performance/io-threads
    option thread-count 8
    subvolumes locks
  end-volume

  volume brick2
    type performance/io-threads
    option thread-count 8
    subvolumes locks2
  end-volume

  volume server
    type protocol/server
    option transport-type tcp
    option auth.addr.brick.allow 192.168.0.191,192.168.0.192
    option auth.addr.brick2.allow 192.168.0.191,192.168.0.192
    subvolumes brick brick2
  end-volume


===========================================================
Client Configuration
===========================================================
  volume remote1a
    type protocol/client
    option transport-type tcp
    option remote-host 192.168.0.151
    option remote-subvolume brick
  end-volume
  volume remote1b
    type protocol/client
    option transport-type tcp
    option remote-host 192.168.0.151
    option remote-subvolume brick2
  end-volume

  volume remote2a
    type protocol/client
    option transport-type tcp
    option remote-host 192.168.0.152
    option remote-subvolume brick
  end-volume

  volume remote2b
    type protocol/client
    option transport-type tcp
    option remote-host 192.168.0.152
    option remote-subvolume brick2
  end-volume


  volume unify1
    type cluster/distribute
    subvolumes remote1a remote1b
    option scheduler alu   # use the ALU scheduler
    option alu.limits.min-free-disk  5%      # Don't create files one a
volume with less than 5% free diskspace
    option alu.limits.max-open-files 10000   # Don't create files on a
volume with more than 10000 files open
    option alu.order
disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
    option alu.disk-usage.entry-threshold 3GB   # Kick in if the discrepancy
in disk-usage between volumes is more than 3GB
    option alu.disk-usage.exit-threshold  60MB   # Don't stop writing to the
least-used volume until the discrepancy is 1988MB
    option alu.open-files-usage.entry-threshold 1024   # Kick in if the
discrepancy in open files is 1024
    option alu.open-files-usage.exit-threshold 32   # Don't stop until 992
files have been written the least-used volume
    option alu.stat-refresh.interval 60sec   # Refresh the statistics used
for decision-making every 60 seconds
  end-volume

  volume unify2
    type cluster/distribute
    subvolumes remote2a remote2b
    option scheduler alu   # use the ALU scheduler
    option alu.limits.min-free-disk  5%      # Don't create files one a
volume with less than 5% free diskspace
    option alu.limits.max-open-files 10000   # Don't create files on a
volume with more than 10000 files open
    option alu.order
disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
    option alu.disk-usage.entry-threshold 3GB   # Kick in if the discrepancy
in disk-usage between volumes is more than 3GB
    option alu.disk-usage.exit-threshold  60MB   # Don't stop writing to the
least-used volume until the discrepancy is 1988MB
    option alu.open-files-usage.entry-threshold 1024   # Kick in if the
discrepancy in open files is 1024
    option alu.open-files-usage.exit-threshold 32   # Don't stop until 992
files have been written the least-used volume
    option alu.stat-refresh.interval 60sec   # Refresh the statistics used
for decision-making every 60 seconds
  end-volume

  volume replicate
    type cluster/afr
    subvolumes unify1 unify2
  end-volume

  volume writebehind
    type performance/write-behind
    option window-size 1MB
    subvolumes replicate
  end-volume

  volume cache
    type performance/io-cache
    option cache-size 512MB
    subvolumes writebehind
  end-volume
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110514/3d4f1b75/attachment.html>