[Gluster-users] Healing : No space left on device

[Nithya Balachandran]

Hi,

We need some more information in order to debug this.
The version of Gluster you were running before the upgrade
The output of gluster volume info <volname>
The brick logs for the volume when the operation is performed.

Regards,
Nithya



On 2 May 2018 at 15:19, Hoggins! <fuckspam at wheres5.com> wrote:

> Hello list,
>
> I have an issue on my Gluster cluster. It is composed of two data nodes
> and an arbiter for all my volumes.
>
> After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is
> what I get :
>
>     - on node 1, volumes won't start, and glusterd.log shows a lot of :
>         [2018-05-02 09:46:06.267817] W
> [glusterd-locks.c:843:glusterd_mgmt_v3_unlock]
> (-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549)
> [0x7f0047ae2549]
> -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0)
> [0x7f0047aebdf0]
> -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371)
> [0x7f0047b98371] ) 0-management: Lock for vol thedude not held
>         The message "W [MSGID: 106118]
> [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
> not released for rom" repeated 3 times between [2018-05-02
> 09:45:57.262321] and [2018-05-02 09:46:06.267804]
>         [2018-05-02 09:46:06.267826] W [MSGID: 106118]
> [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
> not released for thedude
>
>
>     - on node 2, volume are up but don't seem to be willing to correctly
> heal. The logs show a lot of :
>         [2018-05-02 09:23:01.054196] I [MSGID: 108026]
> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-thedude-replicate-0:
> performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3
>         [2018-05-02 09:23:01.222596] E [MSGID: 114031]
> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote
> operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No
> space left on device]
>
>
>     - on arbiter, glustershd.log shows a lot of :
>         [2018-05-02 09:44:54.619476] I [MSGID: 108026]
> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0:
> performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667
>         [2018-05-02 09:44:54.640276] E [MSGID: 114031]
> [client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote
> operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No
> space left on device]
>         [2018-05-02 09:44:54.657045] I [MSGID: 108026]
> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0:
> performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389
>         [2018-05-02 09:47:09.121060] W [MSGID: 101088]
> [common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to
> save the backtrace.
>
>
> The clients connecting to the cluster experience problems, such as
> Gluster refusing to create files, etc.
>
> I'm lost here, where should I start ?
>
>     Thanks for your help !
>
>         Hoggins!
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180503/1abb0433/attachment.html>