[Gluster-users] Healing : No space left on device

[Hoggins!]

There are also free inodes on the disks of all the machines... don't
where to look to solve this. Any idea ?

Le 02/05/2018 à 12:39, Hoggins! a écrit :
> Oh, and *there is* space on the device where the brick's data is located.
>
>     /dev/mapper/fedora-home   942G    868G   74G  93% /export
>
> Le 02/05/2018 à 11:49, Hoggins! a écrit :
>> Hello list,
>>
>> I have an issue on my Gluster cluster. It is composed of two data nodes
>> and an arbiter for all my volumes.
>>
>> After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is
>> what I get :
>>
>>     - on node 1, volumes won't start, and glusterd.log shows a lot of :
>>         [2018-05-02 09:46:06.267817] W
>> [glusterd-locks.c:843:glusterd_mgmt_v3_unlock]
>> (-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549)
>> [0x7f0047ae2549]
>> -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0)
>> [0x7f0047aebdf0]
>> -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371)
>> [0x7f0047b98371] ) 0-management: Lock for vol thedude not held
>>         The message "W [MSGID: 106118]
>> [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
>> not released for rom" repeated 3 times between [2018-05-02
>> 09:45:57.262321] and [2018-05-02 09:46:06.267804]
>>         [2018-05-02 09:46:06.267826] W [MSGID: 106118]
>> [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
>> not released for thedude
>>
>>
>>     - on node 2, volume are up but don't seem to be willing to correctly
>> heal. The logs show a lot of :
>>         [2018-05-02 09:23:01.054196] I [MSGID: 108026]
>> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-thedude-replicate-0:
>> performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3
>>         [2018-05-02 09:23:01.222596] E [MSGID: 114031]
>> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote
>> operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No
>> space left on device]
>>
>>
>>     - on arbiter, glustershd.log shows a lot of :
>>         [2018-05-02 09:44:54.619476] I [MSGID: 108026]
>> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0:
>> performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667
>>         [2018-05-02 09:44:54.640276] E [MSGID: 114031]
>> [client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote
>> operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No
>> space left on device]
>>         [2018-05-02 09:44:54.657045] I [MSGID: 108026]
>> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0:
>> performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389
>>         [2018-05-02 09:47:09.121060] W [MSGID: 101088]
>> [common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to
>> save the backtrace.
>>
>>
>> The clients connecting to the cluster experience problems, such as
>> Gluster refusing to create files, etc.
>>
>> I'm lost here, where should I start ?
>>
>>     Thanks for your help !
>>
>>         Hoggins!
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180503/22f83307/attachment.sig>