[Gluster-users] duplicate files on client side

[Gerrit Giehl]

Hi @all,

since a few hours we have a problem with duplicate files in one directory (and it’s subdirectories) of our volume.
It looks like that the error occurs only on client side (on the server side the folder structure looks fine) and that only files (not directories) are affected.

-------------------------------------------------------
$ ls -l /glusterfs/gv0/affected-directory/
total 132
[...]
-rw-r--r--  1 user    group    19878 May 16  2017 Error_404.html
-rw-r--r--  1 user    group    19878 May 16  2017 Error_404.html
[…]
-rw-rw-r--  1 user    group       91 Mar 10  2017 robots.txt
-rw-rw-r--  1 user    group       91 Mar 10  2017 robots.txt
[…]
-------------------------------------------------------

an remount didn’t fix the problem but after replacing the mount directive everything works fine, but I don’t no why (and if there are hidden side effects) and `$ gluster volume heal gv0 info` shows also still an error.
So I also started a full heal request via `$ gluster volume heal gv0 full` (hopefully it will fix the whole thing).

-------------------------------------------------------
old
-------------------------------------------------------
host1:/gv0	/glusterfs/gv0 glusterfs	defaults,_netdev,log-level=WARNING 0 0
-------------------------------------------------------

-------------------------------------------------------
new
-------------------------------------------------------
/etc/glusterfs/datastore-vg0.vol	/glusterfs/gv0 glusterfs	defaults,_netdev,log-level=WARNING 0 0
-------------------------------------------------------

Does anyone have an idea why this happened? Thanks in advance.

Best regards,
Gerrit




-------------------------------------------------------
$ cat /etc/glusterfs/datastore-vg0.vol

volume remote1
  type protocol/client
  option transport-type tcp
  option remote-host host1
  option remote-subvolume /export/xvda3/brick
end-volume

volume remote2
  type protocol/client
  option transport-type tcp
  option remote-host host2
  option remote-subvolume /export/xvda3/brick
end-volume

volume remote3
  type protocol/client
  option transport-type tcp
  option remote-host host3
  option remote-subvolume /export/xvda3/brick
end-volume

volume remote4
  type protocol/client
  option transport-type tcp
  option remote-host host4
  option remote-subvolume /export/xvda3/brick
end-volume

volume replicate
  type cluster/replicate
  subvolumes remote1 remote2 remote3 remote4
end-volume

volume writebehind
  type performance/write-behind
  #option window-size 1MB
  subvolumes replicate
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume
-------------------------------------------------------


-------------------------------------------------------
Server - glusterfs 3.8.7
-------------------------------------------------------
$ sudo gluster volume info

Volume Name: gv0
Type: Distributed-Replicate
Volume ID: xxxxxxxxxxxxxxxxxxx
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: host1:/export/xvda3/brick
Brick2: host2:/export/xvda3/brick
Brick3: host3:/export/xvda3/brick
Brick4: host4:/export/xvda3/brick
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: off
performance.cache-size: 512MB
performance.client-io-threads: on
performance.io-thread-count: 64
performance.read-ahead: off
performance.cache-refresh-timeout: 1
-------------------------------------------------------


-------------------------------------------------------
$ gluster volume heal gv0 info
Brick host1:/export/xvda3/brick
Status: Connected
Number of entries: 0

Brick host2:/export/xvda3/brick
Status: Connected
Number of entries: 0

Brick host3:/export/xvda3/brick
Status: Connected
Number of entries: 0

Brick host4:/export/xvda3/brick
<gfid:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx>
Status: Connected
Number of entries: 1
-------------------------------------------------------



-------------------------------------------------------
$ gluster volume heal gv0 statistics

[...]

Ending time of crawl: Thu Oct 10 17:30:54 2019

Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 2

Starting time of crawl: Thu Oct 10 17:40:54 2019

Ending time of crawl: Thu Oct 10 17:40:55 2019

Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 1

[...]

Starting time of crawl: Thu Oct 10 17:28:01 2019

Crawl is in progress
Type of crawl: FULL
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 0
-------------------------------------------------------