[Bugs] [Bug 1809326] Volumes keeps going offline with messages -- posix-helpers.c:2150:posix_health_check_thread_proc

bugzilla at redhat.com bugzilla at redhat.com
Thu Mar 5 19:29:36 UTC 2020 

https://bugzilla.redhat.com/show_bug.cgi?id=1809326



--- Comment #6 from Amgad <amgad.saleh at nokia.com> ---
I tried to restart glusterd now on that node, but it doesn't start. with
messages -- I'm also attaching glusterd.log and glusterd.vol files as well

[root at telco-control-02 glusterfs]# systemctl start glusterd
Job for glusterd.service failed because the control process exited with error
code. See "systemctl status glusterd.service" and "journalctl -xe" for details.
[root at telco-control-02 glusterfs]# systemctl status glusterd.service
â glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/etc/systemd/system/glusterd.service; enabled; vendor
preset: disabled)
   Active: activating (start) since Thu 2020-03-05 19:23:58 UTC; 6s ago
  Control: 31068 (glusterd)
    Tasks: 190
   Memory: 493.9M (limit: 23.5G)
   CGroup: /system.slice/glusterd.service
           ââ 3873 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
ER...
           ââ19874 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
bcmt-glus...
           ââ19905 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
bcmt-helm...
           ââ19938 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
cbur-glus...
           ââ19977 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
cbur-glus...
           ââ20078 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
cbur-glus...
           ââ20090 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
vol_04c09...
           ââ24041 /usr/sbin/glusterfs -s 172.16.1.17 --volfile-id
gluster/gl...
           ââ31068 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
ER...
           ââ31071 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
ER...

Mar 05 19:23:58 telco-control-02 systemd[1]: Stopped GlusterFS, a clustered
....
Mar 05 19:23:58 telco-control-02 systemd[1]: Starting GlusterFS, a
clustered....
Hint: Some lines were ellipsized, use -l to show in full.
[root at telco-control-02 glusterfs]# systemctl status -l glusterd.service
â glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/etc/systemd/system/glusterd.service; enabled; vendor
preset: disabled)
   Active: activating (start) since Thu 2020-03-05 19:24:24 UTC; 4ms ago
  Control: 31746 (glusterd)
    Tasks: 183
   Memory: 491.9M (limit: 23.5G)
   CGroup: /system.slice/glusterd.service
           ââ 3873 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
ERROR
           ââ19874 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
bcmt-glusterfs.172.16.1.17.data0-glusterfs -p
/var/run/gluster/vols/bcmt-glusterfs/172.16.1.17-data0-glusterfs.pid -S
/var/run/gluster/1ee29a143ced3772.socket --brick-name /data0/glusterfs -l
/var/log/glusterfs/bricks/data0-glusterfs.log --xlator-option
*-posix.glusterd-uuid=6e013921-73b5-46cd-b8d7-5e7d93cd6bf0 --process-name brick
--brick-port 49152 --xlator-option bcmt-glusterfs-server.listen-port=49152
--xlator-option transport.socket.bind-address=172.16.1.17
           ââ19905 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
bcmt-helm-home.172.16.1.17.data0-glusterfs-helm-home -p
/var/run/gluster/vols/bcmt-helm-home/172.16.1.17-data0-glusterfs-helm-home.pid
-S /var/run/gluster/32e46f40f6326f49.socket --brick-name
/data0/glusterfs-helm-home -l
/var/log/glusterfs/bricks/data0-glusterfs-helm-home.log --xlator-option
*-posix.glusterd-uuid=6e013921-73b5-46cd-b8d7-5e7d93cd6bf0 --process-name brick
--brick-port 49153 --xlator-option bcmt-helm-home-server.listen-port=49153
--xlator-option transport.socket.bind-address=172.16.1.17
           ââ19938 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
cbur-glusterfs-backup.172.16.1.17.data0-glusterfs-cbur-backup -p
/var/run/gluster/vols/cbur-glusterfs-backup/172.16.1.17-data0-glusterfs-cbur-backup.pid
-S /var/run/gluster/9faf700427a7b186.socket --brick-name
/data0/glusterfs-cbur-backup -l
/var/log/glusterfs/bricks/data0-glusterfs-cbur-backup.log --xlator-option
*-posix.glusterd-uuid=6e013921-73b5-46cd-b8d7-5e7d93cd6bf0 --process-name brick
--brick-port 49154 --xlator-option
cbur-glusterfs-backup-server.listen-port=49154 --xlator-option
transport.socket.bind-address=172.16.1.17
           ââ19977 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
cbur-glusterfs-backup-cluster.172.16.1.17.data0-glusterfs-backup-cluster -p
/var/run/gluster/vols/cbur-glusterfs-backup-cluster/172.16.1.17-data0-glusterfs-backup-cluster.pid
-S /var/run/gluster/f9bad66bcadcc1cf.socket --brick-name
/data0/glusterfs-backup-cluster -l
/var/log/glusterfs/bricks/data0-glusterfs-backup-cluster.log --xlator-option
*-posix.glusterd-uuid=6e013921-73b5-46cd-b8d7-5e7d93cd6bf0 --process-name brick
--brick-port 49155 --xlator-option
cbur-glusterfs-backup-cluster-server.listen-port=49155 --xlator-option
transport.socket.bind-address=172.16.1.17
           ââ20078 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
cbur-glusterfs-repo.172.16.1.17.data0-glusterfs-cbur-repo -p
/var/run/gluster/vols/cbur-glusterfs-repo/172.16.1.17-data0-glusterfs-cbur-repo.pid
-S /var/run/gluster/0fd1d96c1f9e0319.socket --brick-name
/data0/glusterfs-cbur-repo -l
/var/log/glusterfs/bricks/data0-glusterfs-cbur-repo.log --xlator-option
*-posix.glusterd-uuid=6e013921-73b5-46cd-b8d7-5e7d93cd6bf0 --process-name brick
--brick-port 49156 --xlator-option cbur-glusterfs-repo-server.listen-port=49156
--xlator-option transport.socket.bind-address=172.16.1.17
           ââ20090 /usr/sbin/glusterfsd -s 172.16.1.17 --volfile-id
vol_04c09b2af8fdf1c596fe8cef4d03f966.172.16.1.17.var-lib-heketi-mounts-vg_260e40bf44e9568fe20ed05545208104-brick_2a3320145759ee25cd0a814959ec98e0-brick
-p
/var/run/gluster/vols/vol_04c09b2af8fdf1c596fe8cef4d03f966/172.16.1.17-var-lib-heketi-mounts-vg_260e40bf44e9568fe20ed05545208104-brick_2a3320145759ee25cd0a814959ec98e0-brick.pid
-S /var/run/gluster/117adbbe3cbaa25c.socket --brick-name
/var/lib/heketi/mounts/vg_260e40bf44e9568fe20ed05545208104/brick_2a3320145759ee25cd0a814959ec98e0/brick
-l
/var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_260e40bf44e9568fe20ed05545208104-brick_2a3320145759ee25cd0a814959ec98e0-brick.log
--xlator-option *-posix.glusterd-uuid=6e013921-73b5-46cd-b8d7-5e7d93cd6bf0
--process-name brick --brick-port 49157 --xlator-option
vol_04c09b2af8fdf1c596fe8cef4d03f966-server.listen-port=49157 --xlator-option
transport.socket.bind-address=172.16.1.17
           ââ24041 /usr/sbin/glusterfs -s 172.16.1.17 --volfile-id
gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l
/var/log/glusterfs/glustershd.log -S /var/run/gluster/67898f7af0cbf69c.socket
--xlator-option *replicate*.node-uuid=6e013921-73b5-46cd-b8d7-5e7d93cd6bf0
--process-name glustershd --client-pid=-6
           ââ31746 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
ERROR

Mar 05 19:24:24 telco-control-02 systemd[1]: Starting GlusterFS, a clustered
file-system server...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

More information about the Bugs mailing list