[Bugs] [Bug 1679406] New: glustereventsd does not start on Ubuntu 16.04 LTS

bugzilla at redhat.com bugzilla at redhat.com
Thu Feb 21 06:02:48 UTC 2019


https://bugzilla.redhat.com/show_bug.cgi?id=1679406

            Bug ID: 1679406
           Summary: glustereventsd does not start on Ubuntu 16.04 LTS
           Product: GlusterFS
           Version: mainline
          Hardware: x86_64
                OS: Linux
            Status: NEW
         Component: eventsapi
          Severity: high
          Priority: high
          Assignee: bugs at gluster.org
          Reporter: avishwan at redhat.com
                CC: avishwan at redhat.com, beat.joerg at gmail.com,
                    bugzilla at ii.nl, captcha.is.evil at gmail.com,
                    dchang at silverlakecorp.com, pasik at iki.fi,
                    sabose at redhat.com
        Depends On: 1649054
            Blocks: 1667103 (glusterfs-5.4), 1677319
                    (Gluster_5_Affecting_oVirt_4.3)
  Target Milestone: ---
    Classification: Community



+++ This bug was initially created as a clone of Bug #1649054 +++

Description of problem:
glustereventsd does not start due to an import error in utils.py

Nov 12 20:21:04 s002 glustereventsd[28140]:   File "/usr/sbin/glustereventsd",
line 24, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     import handlers
Nov 12 20:21:04 s002 glustereventsd[28140]:   File
"/usr/lib/x86_64-linux-gnu/glusterfs/events/handlers.py", line 12, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     import utils
Nov 12 20:21:04 s002 glustereventsd[28140]:   File
"/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in <module>
Nov 12 20:21:04 s002 glustereventsd[28140]:     from .eventsapiconf import
(LOG_FILE,
Nov 12 20:21:04 s002 glustereventsd[28140]: SystemError: Parent module '' not
loaded, cannot perform relative import

Installed Pyython version is 3.5.2

--- Additional comment from Mrten on 2018-12-04 17:12:09 UTC ---

same as Bug 1650017

--- Additional comment from Mrten on 2018-12-04 17:25:31 UTC ---

This is a fix for the first bug:

--- utils.py.old        2018-12-04 18:19:18.846463181 +0100
+++ utils.py    2018-12-04 18:21:09.830149998 +0100
@@ -26,13 +26,13 @@
 from hashlib import sha256
 from calendar import timegm

-from .eventsapiconf import (LOG_FILE,
+from eventsapiconf import (LOG_FILE,
                            WEBHOOKS_FILE,
                            DEFAULT_CONFIG_FILE,
                            CUSTOM_CONFIG_FILE,
                            UUID_FILE,
                            CERTS_DIR)
-from . import eventtypes
+import eventtypes


and you need this one as well:

--- glustereventsd.py.old       2018-12-04 18:25:06.841481182 +0100
+++ glustereventsd.py   2018-12-04 18:22:51.529863014 +0100
@@ -28,7 +28,7 @@
 from utils import logger, PidFile, PidFileLockFailed, boolify


-class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
+class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):

     def handle(self):
         data = self.request[0].strip()
@@ -98,7 +98,7 @@

     # Start the Eventing Server, UDP Server
     try:
-        server = SocketServer.ThreadingUDPServer(
+        server = socketserver.ThreadingUDPServer(
             (SERVER_ADDRESS, port),
             GlusterEventsRequestHandler)
     except socket.error as e:

--- Additional comment from Mrten on 2018-12-04 17:28:51 UTC ---

but you still have "glustereventsd[20980]: Unable to get Port details from
Config"

in the log :(

--- Additional comment from Mrten on 2018-12-04 17:36:28 UTC ---

default config file does not exist:

root at gluster-3:# less /etc/glusterfs/eventsconfig.json
/etc/glusterfs/eventsconfig.json: No such file or directory

--- Additional comment from Mrten on 2018-12-04 17:38:39 UTC ---

inserting this:

https://github.com/gluster/glusterfs/blob/master/events/src/eventsconfig.json

into /etc/glusterfs/eventsconfig.json makes it start without problems

--- Additional comment from Mrten on 2018-12-05 09:24:13 UTC ---

haha, but now geo-replication is broken

[2018-12-05 08:52:49.556853] E [syncdutils(monitor):338:log_raise_exception]
<top>: FAIL:
Traceback (most recent call last):
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py",
line 368, in twrap
    tf(*aargs)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line
339, in wmon
    slave_host, master, suuid, slavenodes)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/monitor.py", line
325, in monitor
    brick_path=w[0]['dir'])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py",
line 602, in gf_event
    from events.gf_event import gf_event as gfevent
  File "/usr/lib/x86_64-linux-gnu/glusterfs/events/gf_event.py", line 19, in
<module>
    from .utils import logger, setup_logger, get_config
  File "/usr/lib/x86_64-linux-gnu/glusterfs/events/utils.py", line 29, in
<module>
    from eventsapiconf import (LOG_FILE,
ImportError: No module named 'eventsapiconf'

--- Additional comment from Mrten on 2018-12-05 09:31:22 UTC ---

better patch:

--- utils.py.old        2018-12-04 18:19:18.846463181 +0100
+++ utils.py    2018-12-05 10:30:03.306848363 +0100
@@ -26,13 +26,25 @@
 from hashlib import sha256
 from calendar import timegm

-from .eventsapiconf import (LOG_FILE,
-                           WEBHOOKS_FILE,
-                           DEFAULT_CONFIG_FILE,
-                           CUSTOM_CONFIG_FILE,
-                           UUID_FILE,
-                           CERTS_DIR)
-from . import eventtypes
+try:
+    from .eventsapiconf import (LOG_FILE,
+                                WEBHOOKS_FILE,
+                                DEFAULT_CONFIG_FILE,
+                                CUSTOM_CONFIG_FILE,
+                                UUID_FILE,
+                                CERTS_DIR)
+except ImportError:
+    from eventsapiconf import (LOG_FILE,
+                               WEBHOOKS_FILE,
+                               DEFAULT_CONFIG_FILE,
+                               CUSTOM_CONFIG_FILE,
+                               UUID_FILE,
+                               CERTS_DIR)
+
+try:
+  from . import eventtypes
+except ImportError:
+  import eventtypes

--- Additional comment from dchang0 on 2018-12-12 08:06:26 UTC ---

Hi, Mrten--

I still have the exact same bug AFTER applying the patches you provided (the
patch in Comment 2 for glustereventsd.py and the patch in Comment 7 for
utils.py).

I'm running Raspbian Stretch Lite (not Ubuntu 16.04 like the original bug
reporter) on a Raspberry Pi Zero W and built glusterfs 5.1 from source code.
The patches appear to have been applied correctly but attempting to start
glustereventsd still gets this error:

Dec 11 16:48:14 datastore2c systemd[1]: glustereventsd.service: Failed with
result 'exit-code'.
Dec 11 16:50:53 datastore2c systemd[1]: Starting LSB: Gluster Events Server...
-- Subject: Unit glustereventsd.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has begun starting up.
Dec 11 16:50:55 datastore2c glustereventsd[710]: Starting glustereventsd
service: glustereventsdTraceback (most recent call last):
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File
"/usr/local/sbin/glustereventsd", line 24, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     import handlers
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File
"/usr/local/libexec/glusterfs/events/handlers.py", line 12, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     import utils
Dec 11 16:50:55 datastore2c glustereventsd[710]:   File
"/usr/local/libexec/glusterfs/events/utils.py", line 30, in <module>
Dec 11 16:50:55 datastore2c glustereventsd[710]:     from .eventsapiconf import
(LOG_FILE,
Dec 11 16:50:55 datastore2c glustereventsd[710]: SystemError: Parent module ''
not loaded, cannot perform relative import
Dec 11 16:50:55 datastore2c glustereventsd[710]:  failed!
Dec 11 16:50:56 datastore2c glustereventsd[710]: Traceback (most recent call
last):
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File
"/usr/local/sbin/glustereventsd", line 24, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     import handlers
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File
"/usr/local/libexec/glusterfs/events/handlers.py", line 12, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     import utils
Dec 11 16:50:56 datastore2c glustereventsd[710]:   File
"/usr/local/libexec/glusterfs/events/utils.py", line 30, in <module>
Dec 11 16:50:56 datastore2c glustereventsd[710]:     from .eventsapiconf import
(LOG_FILE,
Dec 11 16:50:56 datastore2c glustereventsd[710]: SystemError: Parent module ''
not loaded, cannot perform relative import
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Control process
exited, code=exited status=1
Dec 11 16:50:56 datastore2c systemd[1]: Failed to start LSB: Gluster Events
Server.
-- Subject: Unit glustereventsd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has failed.
-- 
-- The result is failed.
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Unit entered
failed state.
Dec 11 16:50:56 datastore2c systemd[1]: glustereventsd.service: Failed with
result 'exit-code'.


I can provide whatever info you need to help debug this. Thanks!

--- Additional comment from Arano-kai on 2018-12-28 01:27 UTC ---

Hi dchang0.
According to your logs, patch is not apply.
Here the compiled from Mrten's answers patch that you need apply with following
cmd:
`patch -p 0 -d [libdir] < glustereventsd.patch`
Where [libdir]:
- Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/"
- Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs)
Also, there is missing file that you can get by `wget
'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/eventsconfig.json'
-O /etc/glusterfs/eventsconfig.json`

--- Additional comment from dchang0 on 2018-12-28 01:44:44 UTC ---

(In reply to Arano-kai from comment #9)
> Created attachment 1517163 [details]
> glustereventsd.py and utils.py fixup
> 
> Hi dchang0.
> According to your logs, patch is not apply.
> Here the compiled from Mrten's answers patch that you need apply with
> following cmd:
> `patch -p 0 -d [libdir] < glustereventsd.patch`
> Where [libdir]:
> - Debian stretch amd64: "/usr/lib/x86_64-linux-gnu/"
> - Raspbian Stretch Lite: "/usr/local/libexec/" (according to logs)
> Also, there is missing file that you can get by `wget
> 'https://raw.githubusercontent.com/gluster/glusterfs/master/events/src/
> eventsconfig.json' -O /etc/glusterfs/eventsconfig.json`

Thanks. I will check the files in question manually and reapply them if needed
and report back.

eventsconfig.json was not missing for me (I built completely from scratch, and
that file was included and installed in the proper location).

--- Additional comment from Arano-kai on 2018-12-28 01:49 UTC ---

Woops, forget to add 'SystemError' to catcher.
Here the fixed version.

--- Additional comment from dchang0 on 2018-12-28 04:14:37 UTC ---

This doesn't matter now, since you've changed at least the glustereventsd.py
patch, but I'm posting these anyway for diagnostic purposes.

The source code was built in /root/glusterfs-5.1. The install with the patches
applied is in /usr/local...

-----


root at datastore2c:/home/pi# diff
/root/glusterfs-5.1/events/src/glustereventsd.py
/usr/local/libexec/glusterfs/events/glustereventsd.py
31c31
< class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
---
> class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):
101c101
<         server = SocketServer.ThreadingUDPServer(
---
>         server = socketserver.ThreadingUDPServer(


-----


root at datastore2c:/home/pi# diff  /root/glusterfs-5.1/events/src/utils.py
/usr/local/libexec/glusterfs/events/utils.py
29,35c29,42
< from .eventsapiconf import (LOG_FILE,
<                            WEBHOOKS_FILE,
<                            DEFAULT_CONFIG_FILE,
<                            CUSTOM_CONFIG_FILE,
<                            UUID_FILE,
<                            CERTS_DIR)
< from . import eventtypes
---
> try:
>     from .eventsapiconf import (LOG_FILE,
>                                 WEBHOOKS_FILE,
>                                 DEFAULT_CONFIG_FILE,
>                                 CUSTOM_CONFIG_FILE,
>                                 UUID_FILE,
>                                 CERTS_DIR)
> except ImportError:
>     from eventsapiconf import (LOG_FILE,
>                                WEBHOOKS_FILE,
>                                DEFAULT_CONFIG_FILE,
>                                CUSTOM_CONFIG_FILE,
>                                UUID_FILE,
>                                CERTS_DIR)
36a44,47
> try:
>     from . import eventtypes
> except ImportError:
>     import eventtypes


-----

So it looks like I did apply the patches before, but they were probably
bad/broken patches.

Anyway, I will replace the patched files with the original source, then apply
your new patches and see what happens and report back.

--- Additional comment from dchang0 on 2018-12-28 04:52:57 UTC ---

Okay, I just put the original files from source back (using make install) and
then applied your new patches from Attachment #1517165. I'm getting a different
error (probably an easier one to solve).

Strangely, the patch command does not create the .orig files. Perhaps my patch
command is broken or non-standard...


Here are the diffs:

-----

root at datastore2c:/home/pi# diff
/root/glusterfs-5.1/events/src/glustereventsd.py
/usr/local/libexec/glusterfs/events/glustereventsd.py
31c31
< class GlusterEventsRequestHandler(SocketServer.BaseRequestHandler):
---
> class GlusterEventsRequestHandler(socketserver.BaseRequestHandler):
101c101
<         server = SocketServer.ThreadingUDPServer(
---
>         server = socketserver.ThreadingUDPServer(

------

root at datastore2c:/usr/local/libexec# diff 
/root/glusterfs-5.1/events/src/utils.py
/usr/local/libexec/glusterfs/events/utils.py
29,35c29,46
< from .eventsapiconf import (LOG_FILE,
<                            WEBHOOKS_FILE,
<                            DEFAULT_CONFIG_FILE,
<                            CUSTOM_CONFIG_FILE,
<                            UUID_FILE,
<                            CERTS_DIR)
< from . import eventtypes
---
> try:
>     from .eventsapiconf import (LOG_FILE,
>                                 WEBHOOKS_FILE,
>                                 DEFAULT_CONFIG_FILE,
>                                 CUSTOM_CONFIG_FILE,
>                                 UUID_FILE,
>                                 CERTS_DIR)
> except (ImportError, SystemError):
>     from eventsapiconf import (LOG_FILE,
>                                WEBHOOKS_FILE,
>                                DEFAULT_CONFIG_FILE,
>                                CUSTOM_CONFIG_FILE,
>                                UUID_FILE,
>                                CERTS_DIR)
> try:
>     from . import eventtypes
> except (ImportError, SystemError):
>     import eventtypes


-----

Here is the new error when attempting to run:

service glustereventsd start

It appears to be a simple path error (can't find the gluster command).



root at datastore2c:/usr/local/libexec# journalctl -xe
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     import utils
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File
"/usr/local/libexec/glusterfs/events/utils.py", line 37, in <module>
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     from eventsapiconf
import (LOG_FILE,
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File
"/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 33, in <modul
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     CUSTOM_CONFIG_FILE =
get_glusterd_workdir() + CUSTOM_CONFIG_FILE_TO_SYNC
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File
"/usr/local/libexec/glusterfs/events/eventsapiconf.py", line 22, in get_gl
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     universal_newlines =
True)
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File
"/usr/lib/python3.5/subprocess.py", line 676, in __init__
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     restore_signals,
start_new_session)
Dec 27 20:46:15 datastore2c glustereventsd[14033]:   File
"/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
Dec 27 20:46:15 datastore2c glustereventsd[14033]:     raise
child_exception_type(errno_num, err_msg)
Dec 27 20:46:15 datastore2c glustereventsd[14033]: FileNotFoundError: [Errno 2]
No such file or directory: 'gluster'
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Control process
exited, code=exited status=1
Dec 27 20:46:15 datastore2c systemd[1]: Failed to start LSB: Gluster Events
Server.
-- Subject: Unit glustereventsd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit glustereventsd.service has failed.
-- 
-- The result is failed.
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Unit entered
failed state.
Dec 27 20:46:15 datastore2c systemd[1]: glustereventsd.service: Failed with
result 'exit-code'.


-----

The gluster command is at this path:

root at datastore2c:/usr/local/libexec# which gluster
/usr/local/sbin/gluster


I confirm that it runs insofar as it can put me into the interactive command
line mode and respond to simple commands like help.

Thanks for your help so far. Let me know what I can do to help.

--- Additional comment from dchang0 on 2018-12-28 08:20:18 UTC ---

BTW, if what I'm reporting in Comment 13 is now a new bug, please let me know
and I will start a different bug report for it. Unless it is actually better to
treat these two errors as one bug.  Thanks!

--- Additional comment from Arano-kai on 2018-12-28 10:28:31 UTC ---

It is probably you hit by https://stackoverflow.com/q/5658622, that is some
$PATH inconsistency from python view and shell view.
Since systemd used and units run in isolated env, you can try to explicitly set
$PATH by overriding glustereventsd.service:
# echo $PATH
<current shell PATH>
# systemctl edit glustereventsd.service

[Service]
Environment=PATH=<current shell PATH>

--- Additional comment from Arano-kai on 2018-12-28 11:18:07 UTC ---

As for Comment 12, patch is right and applied. That is not right is python
version:
# python3 -V
Python 3.5.3
# /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid
...
>>>SystemError<<<: Parent module '' not loaded, cannot perform relative import

# python3 -V
Python 3.7.2rc1
# /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid
...
>>>ImportError<<<: attempted relative import with no known parent package

So we need catch SystemError for older python.

--- Additional comment from dchang0 on 2018-12-29 05:14:01 UTC ---

(In reply to Arano-kai from comment #15)
> It is probably you hit by https://stackoverflow.com/q/5658622, that is some
> $PATH inconsistency from python view and shell view.
> Since systemd used and units run in isolated env, you can try to explicitly
> set $PATH by overriding glustereventsd.service:
> # echo $PATH
> <current shell PATH>
> # systemctl edit glustereventsd.service
> 
> [Service]
> Environment=PATH=<current shell PATH>

Thanks for the help! I ended up editing the PATH= line in the file
/etc/init.d/glustereventsd

It seems to work now (starts without errors). I have not tested the actual
functionality yet but will do so soon enough.

I would say that with the patches you provided in attachment 1517165, this bug
is fixed completely for me. Thanks again!

--- Additional comment from Sahina Bose on 2019-02-14 07:10:46 UTC ---

Aravinda, we have multiple oVirt users hitting this issue. Can this be targeted
to a 5.x release?


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1649054
[Bug 1649054] glustereventsd does not start on Ubuntu 16.04 LTS
https://bugzilla.redhat.com/show_bug.cgi?id=1667103
[Bug 1667103] GlusterFS 5.4 tracker
https://bugzilla.redhat.com/show_bug.cgi?id=1677319
[Bug 1677319] [Tracker] Gluster 5 issues affecting oVirt 4.3
-- 
You are receiving this mail because:
You are the assignee for the bug.


More information about the Bugs mailing list