[Gluster-users] GFID attir is missing after adding large amounts of data

Tue Aug 29 04:35:05 UTC 2017

This is strange, a couple of questions:

1.  What volume type is this?  What tuning have you done?  gluster v info output would be helpful here.

2.  How big are your bricks?

3.  Can you write me a quick reproducer so I can try this in the lab?  Is it just a single multi TB file you are untarring or many?  If you give me the steps to repro, and I hit it, we can get a bug open.  

4.  Other than this are you seeing any other problems?  What if you untar a smaller file(s)?  Can you read and write to the volume with say DD without any problems?

It sounds like you have some other issues affecting things here, there is no reason why you shouldn't be able to untar and write multiple TBs of data to gluster.  Go ahead and answer those questions and I'll see what I can do to help you out.

-b

----- Original Message -----
> From: "Christoph Schäbel" <christoph.schaebel at dc-square.de>
> To: gluster-users at gluster.org
> Sent: Monday, August 28, 2017 3:55:31 AM
> Subject: [Gluster-users] GFID attir is missing after adding large amounts of	data
> 
> Hi Cluster Community,
> 
> we are seeing some problems when adding multiple terrabytes of data to a 2
> node replicated GlusterFS installation.
> 
> The version is 3.8.11 on CentOS 7.
> The machines are connected via 10Gbit LAN and are running 24/7. The OS is
> virtualized on VMWare.
> 
> After a restart of node-1 we see that the log files are growing to multiple
> Gigabytes a day.
> 
> Also there seem to be problems with the replication.
> The setup worked fine until sometime after we added the additional data
> (around 3 TB in size) to node-1. We added the data to a mountpoint via the
> client, not directly to the brick.
> What we did is add tar files via a client-mount and then untar them while in
> the client-mount folder.
> The brick (/mnt/brick1/gv0) is using the XFS filesystem.
> 
> When checking the file attributes of one of the files mentioned in the brick
> logs, i can see that the gfid attribute is missing on node-1. On node-2 the
> file does not even exist.
> 
> getfattr -m . -d -e hex
> mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
> 
> # file:
> mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> 
> We repeated this scenario a second time with a fresh setup and got the same
> results.
> 
> Does anyone know what we are doing wrong ?
> 
> Is there maybe a problem with glusterfs and tar ?
> 
> 
> Log excerpts:
> 
> 
> glustershd.log
> 
> [2017-07-26 15:31:36.290908] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06
> [2017-07-26 15:31:36.294289] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> [2017-07-26 15:31:36.298287] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141
> [2017-07-26 15:31:36.300695] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> [2017-07-26 15:31:36.303626] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on 2cc9dafe-64d3-454a-a647-20deddfaebfe
> [2017-07-26 15:31:36.305763] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> [2017-07-26 15:31:36.308639] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on cbabf9ed-41be-4d08-9cdb-5734557ddbea
> [2017-07-26 15:31:36.310819] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> [2017-07-26 15:31:36.315057] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on 8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69
> [2017-07-26 15:31:36.317196] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> 
> 
> 
> bricks/mnt-brick1-gv0.log
> 
> 2017-07-26 15:31:36.287831] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153546: LOOKUP
> <gfid:d99930df-6b47-4b55-9af3-c767afd6584c>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> (d99930df-6b47-4b55-9af3-c767afd6584c/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.294202] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
> /mnt/brick1/gv0/.glusterfs/e7/2d/e72d9005-b958-432b-b4a9-37aaadd9d2df/type=type1/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.294235] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153564: LOOKUP
> <gfid:fe5c42ac-5fda-47d4-8221-484c8d826c06>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> (fe5c42ac-5fda-47d4-8221-484c8d826c06/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.300611] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
> /mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.300645] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153582: LOOKUP
> <gfid:e31ae2ca-a3d2-4a27-a6ce-9aae24608141>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> (e31ae2ca-a3d2-4a27-a6ce-9aae24608141/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.305671] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
> /mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type1/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.305711] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153600: LOOKUP
> <gfid:2cc9dafe-64d3-454a-a647-20deddfaebfe>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> (2cc9dafe-64d3-454a-a647-20deddfaebfe/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.310735] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
> /mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.310767] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153618: LOOKUP
> <gfid:cbabf9ed-41be-4d08-9cdb-5734557ddbea>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> (cbabf9ed-41be-4d08-9cdb-5734557ddbea/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.317113] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
> /mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type3/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.317146] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153636: LOOKUP
> <gfid:8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> (8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> 
> 
> Regards,
> Christoph
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
> 