[Bugs] [Bug 1217938] New: Dist-geo-rep: Too many "remote operation failed: No such file or directory" warning messages in auxilary mount log on slave while executing "rm -rf"

Sun May 3 07:17:38 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1217938

            Bug ID: 1217938
           Summary: Dist-geo-rep: Too many "remote operation failed: No
                    such file or directory" warning messages in auxilary
                    mount log on slave while executing "rm -rf"
           Product: GlusterFS
           Version: 3.7.0
         Component: geo-replication
          Keywords: ZStream
          Severity: medium
          Priority: medium
          Assignee: bugs at gluster.org
          Reporter: khiremat at redhat.com
                CC: aavati at redhat.com, avishwan at redhat.com,
                    bugs at gluster.org, csaba at redhat.com,
                    gluster-bugs at redhat.com, rhs-bugs at redhat.com,
                    storage-qa-internal at redhat.com, vagarwal at redhat.com,
                    vbhat at redhat.com
        Depends On: 990558, 1210562

+++ This bug was initially created as a clone of Bug #1210562 +++

Description of problem:
I was running rm -rf * on the master mount point and saw lots of Warning
messages in the auxiliary mount logs in the slave.

The messages were 

[2013-07-31 11:11:59.688460] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.688548] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.690008] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691900] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691959] I [afr-lk-common.c:1085:afr_lock_blocking]
0-hosa_slave-replicate-1: unable to lock on even one child
[2013-07-31 11:11:59.691993] I
[afr-transaction.c:1063:afr_post_blocking_inodelk_cbk]
0-hosa_slave-replicate-1: Blocking inodelks failed.
[2013-07-31 11:11:59.693990] W [client-rpc-fops.c:2523:client3_3_opendir_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory.
Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.694328] W [client-rpc-fops.c:2523:client3_3_opendir_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory.
Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)

Version-Release number of selected component (if applicable):
mailine

How reproducible:
Hit once. Not sure about reproducibility.

Steps to Reproduce:
1. Create and start a geo-rep session between master and slave.
2. Untar the linux kernel twice in 2 different locations on the master
mountpoint.
3. Now run from master mount point run, rm -rf *

Actual results:
 hosa_slave-client-2
[2013-07-31 11:11:58.042628] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:58.042946] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:58.087397] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-0: remote operation failed: No such file or directory
[2013-07-31 11:11:58.088347] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-1: remote operation failed: No such file or directory
[2013-07-31 11:11:58.088422] W [fuse-bridge.c:1624:fuse_unlink_cbk]
0-glusterfs-fuse: 1528415: RMDIR()
<gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/vhost_test => -1 (No such file or
directory)
[2013-07-31 11:11:58.291267] E
[afr-self-heal-entry.c:2325:afr_sh_post_nonblocking_entry_cbk]
0-hosa_slave-replicate-1: Non Blocking entrylks failed for
<gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace.
[2013-07-31 11:11:58.291409] E
[afr-self-heal-common.c:2722:afr_log_self_heal_completion_status]
0-hosa_slave-replicate-1: foreground gfid or missing entry self heal  is not
attempted, medatadata self heal  is not attempted, data self heal  is not
attempted, entry self heal  failed on 
<gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace
[2013-07-31 11:11:58.296013] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:58.296128] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:58.324498] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-0: remote operation failed: No such file or directory
[2013-07-31 11:11:58.325157] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-1: remote operation failed: No such file or directory
[2013-07-31 11:11:58.325237] W [fuse-bridge.c:1624:fuse_unlink_cbk]
0-glusterfs-fuse: 1581366: RMDIR()
<gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace => -1 (No such file or
directory)
[2013-07-31 11:11:59.613599] I [dht-layout.c:636:dht_layout_normalize]
0-hosa_slave-dht: found anomalies in
<gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools. holes=1 overlaps=1 missing=1
down=0 misc=0
[2013-07-31 11:11:59.674199] W [client-rpc-fops.c:1983:client3_3_setattr_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.674292] W [client-rpc-fops.c:1983:client3_3_setattr_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.675293] W [client-rpc-fops.c:1744:client3_3_xattrop_cbk]
0-hosa_slave-client-2: remote operation failed: Success. Path:
<gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.675997] W [client-rpc-fops.c:1744:client3_3_xattrop_cbk]
0-hosa_slave-client-3: remote operation failed: Success. Path:
<gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.688460] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.688548] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.690008] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691900] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691959] I [afr-lk-common.c:1085:afr_lock_blocking]
0-hosa_slave-replicate-1: unable to lock on even one child
[2013-07-31 11:11:59.691993] I
[afr-transaction.c:1063:afr_post_blocking_inodelk_cbk]
0-hosa_slave-replicate-1: Blocking inodelks failed.
[2013-07-31 11:11:59.693990] W [client-rpc-fops.c:2523:client3_3_opendir_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory.
Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.694328] W [client-rpc-fops.c:2523:client3_3_opendir_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory.
Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)

Expected results:
Not so much of warning and error logs.

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=990558
[Bug 990558] Dist-geo-rep: Too many "remote operation failed: No such file
or directory" warning messages in auxilary mount log on slave while
executing "rm -rf"
https://bugzilla.redhat.com/show_bug.cgi?id=1210562
[Bug 1210562] Dist-geo-rep: Too many "remote operation failed: No such file
or directory" warning messages in auxilary mount log on slave while
executing "rm -rf"
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.