[Gluster-users] Gluster log files being flooded with messages.

[Mr. John J. Hoffman]

Bradley,

You're not the only one that's happened to ;)

It looks like you have some inconsistencies between the 2 peered nodes.

The following will give you some more insight
 > gluster volume heal help

Try to see if its attempting to heal files
 > gluster volume heal NewRHS info

also if any heals have failed
 > gluster volume heal NewRHS info heal-failed

or you're in a split-brain situation (files exist on both peers, but 
differ).
 > gluster volume heal NewRHS split-brain



On 08/12/14 09:53, Koby, Bradley wrote:
> Hello,
>
> I am new to Gluster, having recently been made admin of this after a
> sudden departure of a fellow co-worker.   There does not seem to be any
> visible degradation of Gluster, and replication seems to be happening as
> well.  However, there are two log files that are filling up so fast,
> that the previous administrator had added some cron jobs to purge the
> files every 20min.
>
> The running version is 3.3.0.
>
>
> Any help or ideas would be greatly appreciated!
>
> Thanks.
>
>
> millions of these type of entries every 20 min in glustershd.log
>
> [2014-08-12 09:28:23.309811] E
> [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0:
> fd1abea4-472d-4db2-8ffd-13e4cb4325fe: Failed to remove index on
> NewRHS-client-0 - No such file or directory
> [2014-08-12 09:28:23.310745] E
> [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0:
> d8e4419f-1ba5-4a42-9c4d-1058e4bcbaf3: Failed to remove index on
> NewRHS-client-0 - No such file or directory
> [2014-08-12 09:28:23.311651] E
> [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0:
> 7de3320a-eff7-43f6-ab88-67ce765df6a4: Failed to remove index on
> NewRHS-client-0 - No such file or directory
>
> and
>
> about 4 million of these type of entries every 20min in
> mnt-gluster-NewRHS-start.log
>
> [2014-08-12 09:34:54.492495] E [posix.c:223:posix_stat] 0-NewRHS-posix:
> lstat on
> /mnt/gluster/NewRHS/start/.glusterfs/a0/79/a079b4de-2998-48ea-9dca-835858ae2ff1
> failed: No such file or directory
> [2014-08-12 09:34:54.492528] I [server3_1-fops.c:1085:server_unlink_cbk]
> 0-NewRHS-server: 715465019: UNLINK
> <gfid:7c0d26c5-07b2-4ff4-8b8c-e2dc2e6eba47>/34c985ca-2273-4c00-ae42-3c2df7ecd40a
> (a079b4de-2998-48ea-9dca-835858ae2ff1) ==> -1 (No such file or directory)
> [2014-08-12 09:34:54.493213] W
> [marker-quota.c:2047:mq_inspect_directory_xattr] 0-NewRHS-marker: cannot
> add a new contribution node
> [2014-08-12 09:34:54.493322] E [posix.c:223:posix_stat] 0-NewRHS-posix:
> lstat on
> /mnt/gluster/NewRHS/start/.glusterfs/11/74/117433f7-ea58-486f-9305-fdebc98e0f0a
> failed: No such file or directory
> [2014-08-12 09:34:54.493355] I [server3_1-fops.c:1085:server_unlink_cbk]
> 0-NewRHS-server: 715465021: UNLINK
> <gfid:7c0d26c5-07b2-4ff4-8b8c-e2dc2e6eba47>/79493dd2-a267-4dcb-b813-a9de10833de7
> (117433f7-ea58-486f-9305-fdebc98e0f0a) ==> -1 (No such file or directory)
>
>
> Some basic info.......
>
> gluster> volume info NewRHS
>
> Volume Name: NewRHS
> Type: Replicate
> Volume ID: 6a63d285-d14c-4ac8-8405-3db771d25e34
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: mualglup01.mcs.muohio.edu:/mnt/gluster/NewRHS/start
> Brick2: mualglup02.mcs.muohio.edu:/mnt/gluster/NewRHS/start
> Options Reconfigured:
> auth.allow:
> 10.0.72.135,10.0.93.*,192.168.251.*,127.0.0.1,10.0.72.132,10.0.72.133
> nfs.disable: off
> features.quota: on
> nfs.register-with-portmap: 1
> diagnostics.client-log-level: ERROR
>
>
> gluster> peer status
> Number of Peers: 1
>
> Hostname: mualglup02
> Uuid: 33ab9b77-b49b-43af-8edd-f7c39b19841b
> State: Peer in Cluster (Connected)
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>

-- 
John J. Hoffman, CISSP
Computer Engineer, AFRL/RCMI
AFRL DoD Supercomputing Resource Center (DSRC)
Air Force Research Laboratory
2435 Fifth Street
Area B, Bldg 676, Rm 205
Wright-Patterson AFB, OH 45433-7802
john.hoffman at wpafb.af.mil
jhoffman at afrl.hpc.mil
(937) 904-5129

Caution: This message may contain competitive, sensitive or other 
non-public information not intended for disclosure outside official 
government channels. Do not disseminate this message without the 
approval of the Office of the Assistant Secretary of the Air Force for 
Acquisition. If you received this message in error, please notify the 
sender by reply e-mail and delete all copies of this message.