[Gluster-users] GFID attir is missing after adding large amounts of data

Tue Aug 29 04:36:45 UTC 2017

Also include gluster v status, I want to check the status of your bricks and SHD processes.

-b

----- Original Message -----
> From: "Ben Turner" <bturner at redhat.com>
> To: "Christoph Schäbel" <christoph.schaebel at dc-square.de>
> Cc: gluster-users at gluster.org
> Sent: Tuesday, August 29, 2017 12:35:05 AM
> Subject: Re: [Gluster-users] GFID attir is missing after adding large amounts of	data
> 
> This is strange, a couple of questions:
> 
> 1.  What volume type is this?  What tuning have you done?  gluster v info
> output would be helpful here.
> 
> 2.  How big are your bricks?
> 
> 3.  Can you write me a quick reproducer so I can try this in the lab?  Is it
> just a single multi TB file you are untarring or many?  If you give me the
> steps to repro, and I hit it, we can get a bug open.
> 
> 4.  Other than this are you seeing any other problems?  What if you untar a
> smaller file(s)?  Can you read and write to the volume with say DD without
> any problems?
> 
> It sounds like you have some other issues affecting things here, there is no
> reason why you shouldn't be able to untar and write multiple TBs of data to
> gluster.  Go ahead and answer those questions and I'll see what I can do to
> help you out.
> 
> -b
> 
> ----- Original Message -----
> > From: "Christoph Schäbel" <christoph.schaebel at dc-square.de>
> > To: gluster-users at gluster.org
> > Sent: Monday, August 28, 2017 3:55:31 AM
> > Subject: [Gluster-users] GFID attir is missing after adding large amounts
> > of	data
> > 
> > Hi Cluster Community,
> > 
> > we are seeing some problems when adding multiple terrabytes of data to a 2
> > node replicated GlusterFS installation.
> > 
> > The version is 3.8.11 on CentOS 7.
> > The machines are connected via 10Gbit LAN and are running 24/7. The OS is
> > virtualized on VMWare.
> > 
> > After a restart of node-1 we see that the log files are growing to multiple
> > Gigabytes a day.
> > 
> > Also there seem to be problems with the replication.
> > The setup worked fine until sometime after we added the additional data
> > (around 3 TB in size) to node-1. We added the data to a mountpoint via the
> > client, not directly to the brick.
> > What we did is add tar files via a client-mount and then untar them while
> > in
> > the client-mount folder.
> > The brick (/mnt/brick1/gv0) is using the XFS filesystem.
> > 
> > When checking the file attributes of one of the files mentioned in the
> > brick
> > logs, i can see that the gfid attribute is missing on node-1. On node-2 the
> > file does not even exist.
> > 
> > getfattr -m . -d -e hex
> > mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
> > 
> > # file:
> > mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
> > security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> > 
> > We repeated this scenario a second time with a fresh setup and got the same
> > results.
> > 
> > Does anyone know what we are doing wrong ?
> > 
> > Is there maybe a problem with glusterfs and tar ?
> > 
> > 
> > Log excerpts:
> > 
> > 
> > glustershd.log
> > 
> > [2017-07-26 15:31:36.290908] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06
> > [2017-07-26 15:31:36.294289] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> > data available]
> > [2017-07-26 15:31:36.298287] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141
> > [2017-07-26 15:31:36.300695] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> > data available]
> > [2017-07-26 15:31:36.303626] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on 2cc9dafe-64d3-454a-a647-20deddfaebfe
> > [2017-07-26 15:31:36.305763] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> > data available]
> > [2017-07-26 15:31:36.308639] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on cbabf9ed-41be-4d08-9cdb-5734557ddbea
> > [2017-07-26 15:31:36.310819] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> > data available]
> > [2017-07-26 15:31:36.315057] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on 8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69
> > [2017-07-26 15:31:36.317196] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> > data available]
> > 
> > 
> > 
> > bricks/mnt-brick1-gv0.log
> > 
> > 2017-07-26 15:31:36.287831] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153546: LOOKUP
> > <gfid:d99930df-6b47-4b55-9af3-c767afd6584c>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > (d99930df-6b47-4b55-9af3-c767afd6584c/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.294202] E [MSGID: 113002] [posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> > /mnt/brick1/gv0/.glusterfs/e7/2d/e72d9005-b958-432b-b4a9-37aaadd9d2df/type=type1/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.294235] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153564: LOOKUP
> > <gfid:fe5c42ac-5fda-47d4-8221-484c8d826c06>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > (fe5c42ac-5fda-47d4-8221-484c8d826c06/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.300611] E [MSGID: 113002] [posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> > /mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.300645] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153582: LOOKUP
> > <gfid:e31ae2ca-a3d2-4a27-a6ce-9aae24608141>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > (e31ae2ca-a3d2-4a27-a6ce-9aae24608141/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.305671] E [MSGID: 113002] [posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> > /mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type1/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.305711] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153600: LOOKUP
> > <gfid:2cc9dafe-64d3-454a-a647-20deddfaebfe>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > (2cc9dafe-64d3-454a-a647-20deddfaebfe/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.310735] E [MSGID: 113002] [posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> > /mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.310767] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153618: LOOKUP
> > <gfid:cbabf9ed-41be-4d08-9cdb-5734557ddbea>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > (cbabf9ed-41be-4d08-9cdb-5734557ddbea/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.317113] E [MSGID: 113002] [posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> > /mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type3/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.317146] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153636: LOOKUP
> > <gfid:8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > (8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > 
> > 
> > Regards,
> > Christoph
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> > 
> 