[Gluster-users] Problem with duplicate files

[Jon Tegner]

On 13/02/14 12:39, Vijay Bellur wrote:
> On 02/11/2014 08:48 PM, tegner at renget.se wrote:
>> Hi,
>>
>> have a system consisting of 4 bricks, distributed, 3.4.1, and I have
>> noticed that some of the files are stored on three of the bricks.
>> Typically a listing can look something like this:
>>
>> brick1: ---------T 2 root root 0 Feb 11 15:47 /mnt/raid6/file
>> brick2: -rw------- 2 2686 2022 10545 Mar  6  2012 /mnt/raid6/file
>> brick3: -rw------- 2 2686 2022 10545 Mar  6  2012 /mnt/raid6/file
>> brick4: no such file or directory
>
> Can you please post your volume configuration?
gluster volume info gives:

**************************************************************
Volume Name: glusterKumiko
Type: Distribute
Volume ID: f54efdc7-0664-409d-b3bc-c39f31040f5e
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: kumiko01IB:/mnt/raid6
Brick2: kumiko02IB:/mnt/raid6
Brick3: kumiko03IB:/mnt/raid6
Brick4: kumiko04IB:/mnt/raid6
**********************************************************************

Also, the file /etc/glusterfs/glusterd.vol have the following content:

**********************************************************************
volume management
     type mgmt/glusterd
     option working-directory /var/lib/glusterd
     option transport-type socket,rdma
     option transport.socket.keepalive-time 10
     option transport.socket.keepalive-interval 2
     option transport.socket.read-fail-log off
#   option base-port 49152
end-volume
***********************************************************************

Should I be worried that rdma is listed as transport-type (and the lack 
of tcp)?


>
>>
>> There are quite a few of these files, and it would be tedious to clean
>> them up manually. Is there a way to have gluster fix these?
>>
>
> This is not a normal occurrence. Do you happen to know the sequence of 
> steps which resulted in this?
Probably messed up:

1. When the volume was created (with RDMA) two of the bricks were 
populated with files I wanted to "merge into" gluster. It seemed as if 
they became "incorporated" after doing a "rsync -vaun" from our primary 
file system (the system in question is used as a mirror).

2. At a later time I wanted to go away from RDMA (using IPoIB instead), 
and at that time I removed the old volume, followed hints from

http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/

and created the new volume (with tcp instead of rdma).

3. Just recently I upgraded to 3.4.2-1, and started a

gluster volume rebalance glusterKumiko fix-layout start (since I have a 
lot of
"disk layout missing" and "mismatching layouts" in the logs).

Regards, and thanks!

/jon