[Gluster-users] glusterfs 4.1.7 + nfs-ganesha 2.7.1 freeze during write

[Soumya Koduri]

On 2/8/19 11:53 AM, Soumya Koduri wrote:
> 
> 
> On 2/8/19 3:20 AM, Maurits Lamers wrote:
>> Hi,
>>
>>>
>>>> [2019-02-07 10:11:24.812606] E [MSGID: 104055] 
>>>> [glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
>>>> event_type(1) and gfid(yøêÙ
>>>>          Mz„–îSL4_@) failed
>>>> [2019-02-07 10:11:24.819376] E [MSGID: 104055] 
>>>> [glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
>>>> event_type(1) and gfid(eTnôEU«H.<Y4failed
>>>> [2019-02-07 10:11:24.833299] E [MSGID: 104055] 
>>>> [glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
>>>> event_type(1) and gfid(gÇLÁèFà»0bЯk) failed
>>>> [2019-02-07 10:25:01.642509] C 
>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-2: 
>>>> server [node1]:49152 has not responded in the last 42 seconds, 
>>>> disconnecting.
>>>> [2019-02-07 10:25:01.642805] C 
>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-1: 
>>>> server [node2]:49152 has not responded in the last 42 seconds, 
>>>> disconnecting.
>>>> [2019-02-07 10:25:01.642946] C 
>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-4: 
>>>> server [node3]:49152 has not responded in the last 42 seconds, 
>>>> disconnecting.
>>>> [2019-02-07 10:25:02.643120] C 
>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-3: 
>>>> server 127.0.1.1:49152 has not responded in the last 42 seconds, 
>>>> disconnecting.
>>>> [2019-02-07 10:25:02.643314] C 
>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-0: 
>>>> server [node4]:49152 has not responded in the last 42 seconds, 
>>>> disconnecting.
>>>
>>> Strange that synctask failed. Could you please turn off 
>>> features.cache-invalidation volume option and check if the issue 
>>> still persists.
>>>>
>>
>> Turning the cache invalidation option off seems to have solved the 
>> freeze. Still testing, but it looks promising.
>>
> 
> If thats the case, please turn on cache invalidation option back and 
> collect couple of stack traces (using gstack) when the system freezes 
> again.

FYI - Have got a chance to reproduce and RCA the issue [1]. Posted fix 
for review in the upstream [2]

Thanks,
Soumya

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1693575
[2] https://review.gluster.org/22436

> 
> Thanks,
> Soumya
>> cheers
>>
>> Maurits
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users