[Bugs] [Bug 1374596] New: [geo-rep]: AttributeError: 'Popen' object has no attribute 'elines'

Fri Sep 9 07:50:04 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1374596

            Bug ID: 1374596
           Summary: [geo-rep]: AttributeError: 'Popen' object has no
                    attribute 'elines'
           Product: GlusterFS
           Version: 3.8.3
         Component: geo-replication
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: avishwan at redhat.com
                CC: bugs at gluster.org, csaba at redhat.com,
                    jeff at socialstrata.com, rhinduja at redhat.com,
                    rhs-bugs at redhat.com, storage-qa-internal at redhat.com
        Depends On: 1340756, 1372193

+++ This bug was initially created as a clone of Bug #1372193 +++

+++ This bug was initially created as a clone of Bug #1340756 +++

Description of problem:
=======================

During tarssh syncup and rmdir from Master nfs client, following traceback is
seen on one of the master nodes:

[2016-05-29 22:40:09.701273] I
[master(/bricks/brick0/master_brick1):1192:crawl] _GMaster: slave's time:
(1464560999, 0)
[2016-05-29 22:40:10.335872] I
[syncdutils(/bricks/brick1/master_brick6):220:finalize] <top>: exiting.
[2016-05-29 22:40:10.336153] E
[syncdutils(/bricks/brick0/master_brick1):276:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 306, in
twrap
    tf(*aa)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1575, in
syncjob
    po.errfail()
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 242, in
errfail
    self.errlog()
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 223, in
errlog
    if self.elines:
AttributeError: 'Popen' object has no attribute 'elines'
[2016-05-29 22:40:10.336755] E
[syncdutils(/bricks/brick0/master_brick1):252:log_raise_exception] <top>:
connection to peer is broken
[2016-05-29 22:40:10.340313] E
[resource(/bricks/brick0/master_brick1):226:errlog] Popen: command "ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-jsxe5z/55c207f77f58e7b6352df3f6f7e6779b.sock
root at 10.70.37.88 /nonexistent/gsyncd --session-owner
4f616379-ac10-4dde-a8c8-1a8c5dfb71f8 -N --listen --timeout 120
gluster://localhost:slave" returned with 255, saying:
[2016-05-29 22:40:10.341049] E
[resource(/bricks/brick0/master_brick1):230:logerr] Popen: ssh> [2016-05-29
22:30:00.731420] I [cli.c:721:main] 0-cli: Started running /usr/sbin/gluster
with version 3.7.9

How reproducible:
=================

Seen only once, not sure about the occurrence. Will update BZ if observe again. 

Steps to Reproduce:
===================
1. Geo-Rep automated cases which does create,rmdir and other fops. Client: NFS
protocol and Sync: tarssh

--- Additional comment from Jeff on 2016-07-13 17:46:03 EDT ---

I am receiving the same error

[2016-07-13 19:25:52.16657] I [master(/srv/gluster):1192:crawl] _GMaster:
slave's time: (1468363119, 0)
[2016-07-13 19:25:52.583666] E
[syncdutils(/srv/gluster):276:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py",
line 306, in twrap
    tf(*aa)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", line
1575, in syncjob
    po.errfail()
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
line 242, in errfail
    self.errlog()
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
line 223, in errlog
    if self.elines:
AttributeError: 'Popen' object has no attribute 'elines'
[2016-07-13 19:25:52.585411] I [syncdutils(/srv/gluster):220:finalize] <top>:
exiting.
[2016-07-13 19:25:52.588901] I [repce(agent):92:service_loop] RepceServer:
terminating on reaching EOF.
[2016-07-13 19:25:52.589368] I [syncdutils(agent):220:finalize] <top>: exiting.
[2016-07-13 19:25:52.928951] I [monitor(monitor):343:monitor] Monitor:
worker(/srv/gluster) died in startup phase

geo-replication status is Faulty and it appears as if the gsyncd.py process is
unable to start on the slave server.

gluster-server 3.7.11-1~bpo8+1 on Debian 8.

--- Additional comment from Jeff on 2016-07-14 19:59:55 EDT ---

The error I was receiving was due to rsync not being installed on the slave
server.

--- Additional comment from Rahul Hinduja on 2016-08-31 12:29:00 EDT ---

While trying one of the rm case on cascading cases(private build based on
3.1.0+patches), I could reproduce this issue by following steps:

1. Create geo-rep cascaded setup with (vol0,vol1,vol2). Such that vol0=>vol1,
vol1=>vol2
2. Mount the vol0 volume and perform:
[root at fan data]# for i in {1..100}; do cp -rf /root/data/new_data/* . ; sleep 5
; rm -rf * ; sleep 2 ; done
[root at fan data]#

Following traceback on one of the master is seen:

[2016-08-31 16:20:55.410460] E
[syncdutils(/rhs/brick2/b3):276:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 306, in
twrap
    tf(*aa)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1530, in
syncjob
    po.errfail()
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 242, in
errfail
    self.errlog()
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 223, in
errlog
    if self.elines:
AttributeError: 'Popen' object has no attribute 'elines'
[2016-08-31 16:20:55.415150] I [syncdutils(/rhs/brick2/b3):220:finalize] <top>:
exiting.

--- Additional comment from Worker Ant on 2016-09-01 03:11:15 EDT ---

REVIEW: http://review.gluster.org/15379 (geo-rep: Fix logging sync failures)
posted (#1) for review on master by Aravinda VK (avishwan at redhat.com)

--- Additional comment from Worker Ant on 2016-09-05 01:43:27 EDT ---

REVIEW: http://review.gluster.org/15379 (geo-rep: Fix logging sync failures)
posted (#2) for review on master by Aravinda VK (avishwan at redhat.com)

--- Additional comment from Worker Ant on 2016-09-08 12:14:53 EDT ---

COMMIT: http://review.gluster.org/15379 committed in master by Aravinda VK
(avishwan at redhat.com) 
------
commit c0f877c0374d97e0bee17aac4850d7655a35e61b
Author: Aravinda VK <avishwan at redhat.com>
Date:   Thu Sep 1 12:35:46 2016 +0530

    geo-rep: Fix logging sync failures

    If Rsync/Tar subprocess dies, while logging error Geo-rep fails
    with EBADF while accessing error file. Also worker dies while
    accessing elines before it is set.

    BUG: 1372193
    Change-Id: I9cfce116e8aafa4a98654f5190d40a455af8ec95
    Signed-off-by: Aravinda VK <avishwan at redhat.com>
    Reviewed-on: http://review.gluster.org/15379
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Kotresh HR <khiremat at redhat.com>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1340756
[Bug 1340756] [geo-rep]: AttributeError: 'Popen' object has no attribute
'elines'
https://bugzilla.redhat.com/show_bug.cgi?id=1372193
[Bug 1372193] [geo-rep]: AttributeError: 'Popen' object has no attribute
'elines'
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.