[Gluster-users] GFID is null after adding large amounts of data

Thu Jul 27 09:05:18 UTC 2017

Hi Cluster Community,

we are seeing some problems when adding multiple terrabytes of data to a 2 node replicated GlusterFS installation.

The version is 3.8.11 on CentOS 7.
The machines are connected via 10Gbit LAN and are running 24/7. The OS is virtualized on VMWare.

After a restart of node-1 we see that the log files are growing to multiple Gigabytes a day.

Also there seem to be problems with the replication.
The setup worked fine until sometime after we added the additional data (around 3 TB in size) to node-1. We added the data to a mountpoint via the client, not directly to the brick.
What we did is add tar files via a client-mount and then untar them while in the client-mount folder.
The brick (/mnt/brick1/gv0) is using the XFS filesystem.

When checking the file attributes of one of the files mentioned in the brick logs, i can see that the gfid attribute is missing on node-1. On node-2 the file does not even exist.

getfattr -m . -d -e hex mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet

# file: mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000

We repeated this scenario a second time with a fresh setup and got the same results. 

This is a real problem for us, so I would be very thankful if anyone can help us with this issue or point us in the right direction.

Log excerpts:

glustershd.log

[2017-07-26 15:31:36.290908] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06
[2017-07-26 15:31:36.294289] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-26 15:31:36.298287] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141
[2017-07-26 15:31:36.300695] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-26 15:31:36.303626] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on 2cc9dafe-64d3-454a-a647-20deddfaebfe
[2017-07-26 15:31:36.305763] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-26 15:31:36.308639] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on cbabf9ed-41be-4d08-9cdb-5734557ddbea
[2017-07-26 15:31:36.310819] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-26 15:31:36.315057] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on 8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69
[2017-07-26 15:31:36.317196] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]

bricks/mnt-brick1-gv0.log

2017-07-26 15:31:36.287831] E [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153546: LOOKUP <gfid:d99930df-6b47-4b55-9af3-c767afd6584c>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet (d99930df-6b47-4b55-9af3-c767afd6584c/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) ==> (No data available) [No data available]
[2017-07-26 15:31:36.294202] E [MSGID: 113002] [posix.c:266:posix_lookup] 0-gv0-posix: buf->ia_gfid is null for /mnt/brick1/gv0/.glusterfs/e7/2d/e72d9005-b958-432b-b4a9-37aaadd9d2df/type=type1/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet [No data available]
[2017-07-26 15:31:36.294235] E [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153564: LOOKUP <gfid:fe5c42ac-5fda-47d4-8221-484c8d826c06>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet (fe5c42ac-5fda-47d4-8221-484c8d826c06/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) ==> (No data available) [No data available]
[2017-07-26 15:31:36.300611] E [MSGID: 113002] [posix.c:266:posix_lookup] 0-gv0-posix: buf->ia_gfid is null for /mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet [No data available]
[2017-07-26 15:31:36.300645] E [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153582: LOOKUP <gfid:e31ae2ca-a3d2-4a27-a6ce-9aae24608141>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet (e31ae2ca-a3d2-4a27-a6ce-9aae24608141/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) ==> (No data available) [No data available]
[2017-07-26 15:31:36.305671] E [MSGID: 113002] [posix.c:266:posix_lookup] 0-gv0-posix: buf->ia_gfid is null for /mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type1/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet [No data available]
[2017-07-26 15:31:36.305711] E [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153600: LOOKUP <gfid:2cc9dafe-64d3-454a-a647-20deddfaebfe>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet (2cc9dafe-64d3-454a-a647-20deddfaebfe/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) ==> (No data available) [No data available]
[2017-07-26 15:31:36.310735] E [MSGID: 113002] [posix.c:266:posix_lookup] 0-gv0-posix: buf->ia_gfid is null for /mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet [No data available]
[2017-07-26 15:31:36.310767] E [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153618: LOOKUP <gfid:cbabf9ed-41be-4d08-9cdb-5734557ddbea>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet (cbabf9ed-41be-4d08-9cdb-5734557ddbea/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) ==> (No data available) [No data available]
[2017-07-26 15:31:36.317113] E [MSGID: 113002] [posix.c:266:posix_lookup] 0-gv0-posix: buf->ia_gfid is null for /mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type3/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet [No data available]
[2017-07-26 15:31:36.317146] E [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153636: LOOKUP <gfid:8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet (8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) ==> (No data available) [No data available]

Regards,
Christoph