[Bugs] [Bug 1283473] New: Dist-geo-rep: Too many "remote operation failed: No such file or directory" warning messages in auxilary mount log on slave while executing "rm -rf"

bugzilla at redhat.com bugzilla at redhat.com
Thu Nov 19 04:43:15 UTC 2015 

https://bugzilla.redhat.com/show_bug.cgi?id=1283473

            Bug ID: 1283473
           Summary: Dist-geo-rep: Too many "remote operation failed: No
                    such file or directory" warning messages in auxilary
                    mount log on slave while executing "rm -rf"
           Product: GlusterFS
           Version: 3.7.6
         Component: geo-replication
          Assignee: bugs at gluster.org
          Reporter: avishwan at redhat.com
                CC: avishwan at redhat.com, bugs at gluster.org,
                    gluster-bugs at redhat.com, khiremat at redhat.com
        Depends On: 1250009



+++ This bug was initially created as a clone of Bug #1250009 +++

Description of problem:

NOTE: BZ 1210562 is not completely fixed, hence opening a new bug

I was running rm -rf * on the master mount point and saw lots of Warning
messages in the auxiliary mount logs in the slave.

The messages were 

[2013-07-31 11:11:59.688460] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.688548] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.690008] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691900] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691959] I [afr-lk-common.c:1085:afr_lock_blocking]
0-hosa_slave-replicate-1: unable to lock on even one child
[2013-07-31 11:11:59.691993] I
[afr-transaction.c:1063:afr_post_blocking_inodelk_cbk]
0-hosa_slave-replicate-1: Blocking inodelks failed.
[2013-07-31 11:11:59.693990] W [client-rpc-fops.c:2523:client3_3_opendir_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory.
Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.694328] W [client-rpc-fops.c:2523:client3_3_opendir_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory.
Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)


Version-Release number of selected component (if applicable):
mailine

How reproducible:
Hit once. Not sure about reproducibility.

Steps to Reproduce:
1. Create and start a geo-rep session between master and slave.
2. Untar the linux kernel twice in 2 different locations on the master
mountpoint.
3. Now run from master mount point run, rm -rf *

Actual results:
 hosa_slave-client-2
[2013-07-31 11:11:58.042628] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:58.042946] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:58.087397] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-0: remote operation failed: No such file or directory
[2013-07-31 11:11:58.088347] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-1: remote operation failed: No such file or directory
[2013-07-31 11:11:58.088422] W [fuse-bridge.c:1624:fuse_unlink_cbk]
0-glusterfs-fuse: 1528415: RMDIR()
<gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/vhost_test => -1 (No such file or
directory)
[2013-07-31 11:11:58.291267] E
[afr-self-heal-entry.c:2325:afr_sh_post_nonblocking_entry_cbk]
0-hosa_slave-replicate-1: Non Blocking entrylks failed for
<gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace.
[2013-07-31 11:11:58.291409] E
[afr-self-heal-common.c:2722:afr_log_self_heal_completion_status]
0-hosa_slave-replicate-1: foreground gfid or missing entry self heal  is not
attempted, medatadata self heal  is not attempted, data self heal  is not
attempted, entry self heal  failed on 
<gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace
[2013-07-31 11:11:58.296013] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:58.296128] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:58.324498] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-0: remote operation failed: No such file or directory
[2013-07-31 11:11:58.325157] W [client-rpc-fops.c:695:client3_3_rmdir_cbk]
0-hosa_slave-client-1: remote operation failed: No such file or directory
[2013-07-31 11:11:58.325237] W [fuse-bridge.c:1624:fuse_unlink_cbk]
0-glusterfs-fuse: 1581366: RMDIR()
<gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace => -1 (No such file or
directory)
[2013-07-31 11:11:59.613599] I [dht-layout.c:636:dht_layout_normalize]
0-hosa_slave-dht: found anomalies in
<gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools. holes=1 overlaps=1 missing=1
down=0 misc=0
[2013-07-31 11:11:59.674199] W [client-rpc-fops.c:1983:client3_3_setattr_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.674292] W [client-rpc-fops.c:1983:client3_3_setattr_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.675293] W [client-rpc-fops.c:1744:client3_3_xattrop_cbk]
0-hosa_slave-client-2: remote operation failed: Success. Path:
<gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.675997] W [client-rpc-fops.c:1744:client3_3_xattrop_cbk]
0-hosa_slave-client-3: remote operation failed: Success. Path:
<gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.688460] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.688548] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.690008] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691900] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691959] I [afr-lk-common.c:1085:afr_lock_blocking]
0-hosa_slave-replicate-1: unable to lock on even one child
[2013-07-31 11:11:59.691993] I
[afr-transaction.c:1063:afr_post_blocking_inodelk_cbk]
0-hosa_slave-replicate-1: Blocking inodelks failed.
[2013-07-31 11:11:59.693990] W [client-rpc-fops.c:2523:client3_3_opendir_cbk]
0-hosa_slave-client-2: remote operation failed: No such file or directory.
Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.694328] W [client-rpc-fops.c:2523:client3_3_opendir_cbk]
0-hosa_slave-client-3: remote operation failed: No such file or directory.
Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools
(dce9238f-e5c7-49c1-9741-73388f92f050)


Expected results:
Not so much of warning and error logs.

--- Additional comment from Anand Avati on 2015-08-05 06:46:00 EDT ---

REVIEW: http://review.gluster.org/11833 (geo-rep: Don't log geo-rep safe errors
in mount logs) posted (#1) for review on master by Kotresh HR
(khiremat at redhat.com)

--- Additional comment from Vijay Bellur on 2015-11-18 23:01:41 EST ---

COMMIT: http://review.gluster.org/11833 committed in master by Venky Shankar
(vshankar at redhat.com) 
------
commit a52fd2cb7fa3aaff74461f58f32f4ff0b8e0904d
Author: Kotresh HR <khiremat at redhat.com>
Date:   Tue Aug 4 16:20:13 2015 +0530

    geo-rep: Don't log geo-rep safe errors in mount logs

    ENOENT is a safe error for geo-replication in case of
    rm -rf. RMDIR is recorded in changelog of each brick,
    geo-rep processes all changelogs among which one will
    succeed and rest will get ENOENT which can be ignored.
    Similarly ENOENT can also be ignored in case of all
    unlink operation during changelog replay that can
    happen when worker goes down and comes back.

    Change-Id: I6756f8f4c3fce7a159751a2bfce891ff16ad31a4
    BUG: 1250009
    Signed-off-by: Kotresh HR <khiremat at redhat.com>
    Reviewed-on: http://review.gluster.org/11833
    Tested-by: NetBSD Build System <jenkins at build.gluster.org>
    Tested-by: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: Aravinda VK <avishwan at redhat.com>
    Reviewed-by: Milind Changire <mchangir at redhat.com>
    Reviewed-by: Saravanakumar Arumugam <sarumuga at redhat.com>
    Reviewed-by: Venky Shankar <vshankar at redhat.com>


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1250009
[Bug 1250009] Dist-geo-rep: Too many "remote operation failed: No such file
or directory" warning messages in auxilary mount log on slave while
executing "rm -rf"
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.

More information about the Bugs mailing list