[Gluster-users] RE : Frequent connect and disconnect messages flooded in logs
Mohammed Rafi K C
rkavunga at redhat.com
Mon Dec 19 06:28:18 UTC 2016
Hi Micha,
Sorry for the late reply. I was busy with some other things.
If you have still the setup available Can you enable TRACE log level
[1],[2] and see if you could find any log entries when the network start
disconnecting. Basically I'm trying to find out any disconnection had
occurred other than ping timer expire issue.
[1] : gluster volume <volname> diagnostics.brick-log-level TRACE
[2] : gluster volume <volname> diagnostics.client-log-level TRACE
Regards
Rafi KC
On 12/08/2016 07:59 PM, Atin Mukherjee wrote:
>
>
> On Thu, Dec 8, 2016 at 4:37 PM, Micha Ober <micha2k at gmail.com
> <mailto:micha2k at gmail.com>> wrote:
>
> Hi Rafi,
>
> thank you for your support. It is greatly appreciated.
>
> Just some more thoughts from my side:
>
> There have been no reports from other users in *this* thread
> until now, but I have found at least one user with a very simiar
> problem in an older thread:
>
> https://www.gluster.org/pipermail/gluster-users/2014-November/019637.html
> <https://www.gluster.org/pipermail/gluster-users/2014-November/019637.html>
>
> He is also reporting disconnects with no apparent reasons,
> althogh his setup is a bit more complicated, also involving a
> firewall. In our setup, all servers/clients are connected via 1
> GbE with no firewall or anything that might block/throttle
> traffic. Also, we are using exactly the same software versions on
> all nodes.
>
>
> I can also find some reports in the bugtracker when searching for
> "rpc_client_ping_timer_expired" and "rpc_clnt_ping_timer_expired"
> (looks like spelling changed during versions).
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1096729
> <https://bugzilla.redhat.com/show_bug.cgi?id=1096729>
>
>
> Just FYI, this is a different issue, here GlusterD fails to handle the
> volume of incoming requests on time since MT-epoll is not enabled here.
>
>
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1370683
> <https://bugzilla.redhat.com/show_bug.cgi?id=1370683>
>
> But both reports involve large traffic/load on the bricks/disks,
> which is not the case for out setup.
> To give a ballpark figure: Over three days, 30 GiB were written.
> And the data was not written at once, but continuously over the
> whole time.
>
>
> Just to be sure, I have checked the logfiles of one of the other
> clusters right now, which are sitting in the same building, in the
> same rack, even on the same switch, running the same jobs, but
> with glusterfs 3.4.2 and I can see no disconnects in the logfiles.
> So I can definitely rule out our infrastructure as problem.
>
> Regards,
> Micha
>
>
>
> Am 07.12.2016 um 18:08 schrieb Mohammed Rafi K C:
>>
>> Hi Micha,
>>
>> This is great. I will provide you one debug build which has two
>> fixes which I possible suspect for a frequent disconnect issue,
>> though I don't have much data to validate my theory. So I will
>> take one more day to dig in to that.
>>
>> Thanks for your support, and opensource++
>>
>> Regards
>>
>> Rafi KC
>>
>> On 12/07/2016 05:02 AM, Micha Ober wrote:
>>> Hi,
>>>
>>> thank you for your answer and even more for the question!
>>> Until now, I was using FUSE. Today I changed all mounts to NFS
>>> using the same 3.7.17 version.
>>>
>>> But: The problem is still the same. Now, the NFS logfile
>>> contains lines like these:
>>>
>>> [2016-12-06 15:12:29.006325] C
>>> [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired]
>>> 0-gv0-client-7: server X.X.18.62:49153 has not responded in the
>>> last 42 seconds, disconnecting.
>>>
>>> Interestingly enough, the IP address X.X.18.62 is the same
>>> machine! As I wrote earlier, each node serves both as a server
>>> and a client, as each node contributes bricks to the volume.
>>> Every server is connecting to itself via its hostname. For
>>> example, the fstab on the node "giant2" looks like:
>>>
>>> #giant2:/gv0 /shared_data glusterfs defaults,noauto
>>> 0 0
>>> #giant2:/gv2 /shared_slurm glusterfs defaults,noauto
>>> 0 0
>>>
>>> giant2:/gv0 /shared_data nfs
>>> defaults,_netdev,vers=3 0 0
>>> giant2:/gv2 /shared_slurm nfs
>>> defaults,_netdev,vers=3 0 0
>>>
>>> So I understand the disconnects even less.
>>>
>>> I don't know if it's possible to create a dummy cluster which
>>> exposes the same behaviour, because the disconnects only happen
>>> when there are compute jobs running on those nodes - and they
>>> are GPU compute jobs, so that's something which cannot be easily
>>> emulated in a VM.
>>>
>>> As we have more clusters (which are running fine with an ancient
>>> 3.4 version :-)) and we are currently not dependent on this
>>> particular cluster (which may stay like this for this month, I
>>> think) I should be able to deploy the debug build on the "real"
>>> cluster, if you can provide a debug build.
>>>
>>> Regards and thanks,
>>> Micha
>>>
>>>
>>>
>>> Am 06.12.2016 um 08:15 schrieb Mohammed Rafi K C:
>>>>
>>>>
>>>>
>>>> On 12/03/2016 12:56 AM, Micha Ober wrote:
>>>>> ** Update: ** I have downgraded from 3.8.6 to 3.7.17 now, but
>>>>> the problem still exists.
>>>>>
>>>>> Client log: http://paste.ubuntu.com/23569065/
>>>>> <http://paste.ubuntu.com/23569065/>
>>>>> Brick log: http://paste.ubuntu.com/23569067/
>>>>> <http://paste.ubuntu.com/23569067/>
>>>>>
>>>>> Please note that each server has two bricks.
>>>>> Whereas, according to the logs, one brick loses the connection
>>>>> to all other hosts:
>>>>> [2016-12-02 18:38:53.703301] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.219:49121 failed (Broken pipe)
>>>>> [2016-12-02 18:38:53.703381] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.62:49118 failed (Broken pipe)
>>>>> [2016-12-02 18:38:53.703380] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.107:49121 failed (Broken pipe)
>>>>> [2016-12-02 18:38:53.703424] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.206:49120 failed (Broken pipe)
>>>>> [2016-12-02 18:38:53.703359] W [socket.c:596:__socket_rwv] 0-tcp.gv0-server: writev on X.X.X.58:49121 failed (Broken pipe)
>>>>>
>>>>> The SECOND brick on the SAME host is NOT affected, i.e. no disconnects!
>>>>> As I said, the network connection is fine and the disks are idle.
>>>>> The CPU always has 2 free cores.
>>>>>
>>>>> It looks like I have to downgrade to 3.4 now in order for the disconnects to stop.
>>>>
>>>> Hi Micha,
>>>>
>>>> Thanks for the update and sorry for what happened with gluster
>>>> higher versions. I can understand the need for downgrade as it
>>>> is a production setup.
>>>>
>>>> Can you tell me the clients used here ? whether it is a
>>>> fuse,nfs,nfs-ganesha, smb or libgfapi ?
>>>>
>>>> Since I'm not able to reproduce the issue (I have been trying
>>>> from last 3days) and the logs are not much helpful here (we
>>>> don't have much logs in socket layer), Could you please create
>>>> a dummy cluster and try to reproduce the issue? If then we can
>>>> play with that volume and I could provide some debug build
>>>> which we can use for further debugging?
>>>>
>>>> If you don't have bandwidth for this, please leave it ;).
>>>>
>>>> Regards
>>>> Rafi KC
>>>>
>>>>> - Micha
>>>>>
>>>>> Am 30.11.2016 um 06:57 schrieb Mohammed Rafi K C:
>>>>>>
>>>>>> Hi Micha,
>>>>>>
>>>>>> I have changed the thread and subject so that your original
>>>>>> thread remain same for your query. Let's try to fix the
>>>>>> problem what you observed with 3.8.4, So I have started a new
>>>>>> thread to discuss the frequent disconnect problem.
>>>>>>
>>>>>> *If any one else has experienced the same problem, please
>>>>>> respond to the mail.*
>>>>>>
>>>>>> It would be very helpful if you could give us some more logs
>>>>>> from clients and bricks. Also any reproducible steps will
>>>>>> surely help to chase the problem further.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Rafi KC
>>>>>>
>>>>>> On 11/30/2016 04:44 AM, Micha Ober wrote:
>>>>>>> I had opened another thread on this mailing list (Subject:
>>>>>>> "After upgrade from 3.4.2 to 3.8.5 - High CPU usage
>>>>>>> resulting in disconnects and split-brain").
>>>>>>>
>>>>>>> The title may be a bit misleading now, as I am no longer
>>>>>>> observing high CPU usage after upgrading to 3.8.6, but the
>>>>>>> disconnects are still happening and the number of files in
>>>>>>> split-brain is growing.
>>>>>>>
>>>>>>> Setup: 6 compute nodes, each serving as a glusterfs server
>>>>>>> and client, Ubuntu 14.04, two bricks per node,
>>>>>>> distribute-replicate
>>>>>>>
>>>>>>> I have two gluster volumes set up (one for scratch data, one
>>>>>>> for the slurm scheduler). Only the scratch data volume shows
>>>>>>> critical errors "[...] has not responded in the last 42
>>>>>>> seconds, disconnecting.". So I can rule out network
>>>>>>> problems, the gigabit link between the nodes is not
>>>>>>> saturated at all. The disks are almost idle (<10%).
>>>>>>>
>>>>>>> I have glusterfs 3.4.2 on Ubuntu 12.04 on a another compute
>>>>>>> cluster, running fine since it was deployed.
>>>>>>> I had glusterfs 3.4.2 on Ubuntu 14.04 on this cluster,
>>>>>>> running fine for almost a year.
>>>>>>>
>>>>>>> After upgrading to 3.8.5, the problems (as described)
>>>>>>> started. I would like to use some of the new features of the
>>>>>>> newer versions (like bitrot), but the users can't run their
>>>>>>> compute jobs right now because the result files are garbled.
>>>>>>>
>>>>>>> There also seems to be a bug report with a smiliar problem:
>>>>>>> (but no progress)
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1370683
>>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1370683>
>>>>>>>
>>>>>>> For me, ALL servers are affected (not isolated to one or two
>>>>>>> servers)
>>>>>>>
>>>>>>> I also see messages like "INFO: task gpu_graphene_bv:4476
>>>>>>> blocked for more than 120 seconds." in the syslog.
>>>>>>>
>>>>>>> For completeness (gv0 is the scratch volume, gv2 the slurm
>>>>>>> volume):
>>>>>>>
>>>>>>> [root at giant2: ~]# gluster v info
>>>>>>>
>>>>>>> Volume Name: gv0
>>>>>>> Type: Distributed-Replicate
>>>>>>> Volume ID: 993ec7c9-e4bc-44d0-b7c4-2d977e622e86
>>>>>>> Status: Started
>>>>>>> Snapshot Count: 0
>>>>>>> Number of Bricks: 6 x 2 = 12
>>>>>>> Transport-type: tcp
>>>>>>> Bricks:
>>>>>>> Brick1: giant1:/gluster/sdc/gv0
>>>>>>> Brick2: giant2:/gluster/sdc/gv0
>>>>>>> Brick3: giant3:/gluster/sdc/gv0
>>>>>>> Brick4: giant4:/gluster/sdc/gv0
>>>>>>> Brick5: giant5:/gluster/sdc/gv0
>>>>>>> Brick6: giant6:/gluster/sdc/gv0
>>>>>>> Brick7: giant1:/gluster/sdd/gv0
>>>>>>> Brick8: giant2:/gluster/sdd/gv0
>>>>>>> Brick9: giant3:/gluster/sdd/gv0
>>>>>>> Brick10: giant4:/gluster/sdd/gv0
>>>>>>> Brick11: giant5:/gluster/sdd/gv0
>>>>>>> Brick12: giant6:/gluster/sdd/gv0
>>>>>>> Options Reconfigured:
>>>>>>> auth.allow: X.X.X.*,127.0.0.1
>>>>>>> nfs.disable: on
>>>>>>>
>>>>>>> Volume Name: gv2
>>>>>>> Type: Replicate
>>>>>>> Volume ID: 30c78928-5f2c-4671-becc-8deaee1a7a8d
>>>>>>> Status: Started
>>>>>>> Snapshot Count: 0
>>>>>>> Number of Bricks: 1 x 2 = 2
>>>>>>> Transport-type: tcp
>>>>>>> Bricks:
>>>>>>> Brick1: giant1:/gluster/sdd/gv2
>>>>>>> Brick2: giant2:/gluster/sdd/gv2
>>>>>>> Options Reconfigured:
>>>>>>> auth.allow: X.X.X.*,127.0.0.1
>>>>>>> cluster.granular-entry-heal: on
>>>>>>> cluster.locking-scheme: granular
>>>>>>> nfs.disable: on
>>>>>>>
>>>>>>>
>>>>>>> 2016-11-30 0:10 GMT+01:00 Micha Ober <micha2k at gmail.com
>>>>>>> <mailto:micha2k at gmail.com>>:
>>>>>>>
>>>>>>> There also seems to be a bug report with a smiliar
>>>>>>> problem: (but no progress)
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1370683
>>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1370683>
>>>>>>>
>>>>>>> For me, ALL servers are affected (not isolated to one or
>>>>>>> two servers)
>>>>>>>
>>>>>>> I also see messages like "INFO: task
>>>>>>> gpu_graphene_bv:4476 blocked for more than 120 seconds."
>>>>>>> in the syslog.
>>>>>>>
>>>>>>> For completeness (gv0 is the scratch volume, gv2 the
>>>>>>> slurm volume):
>>>>>>>
>>>>>>> [root at giant2: ~]# gluster v info
>>>>>>>
>>>>>>> Volume Name: gv0
>>>>>>> Type: Distributed-Replicate
>>>>>>> Volume ID: 993ec7c9-e4bc-44d0-b7c4-2d977e622e86
>>>>>>> Status: Started
>>>>>>> Snapshot Count: 0
>>>>>>> Number of Bricks: 6 x 2 = 12
>>>>>>> Transport-type: tcp
>>>>>>> Bricks:
>>>>>>> Brick1: giant1:/gluster/sdc/gv0
>>>>>>> Brick2: giant2:/gluster/sdc/gv0
>>>>>>> Brick3: giant3:/gluster/sdc/gv0
>>>>>>> Brick4: giant4:/gluster/sdc/gv0
>>>>>>> Brick5: giant5:/gluster/sdc/gv0
>>>>>>> Brick6: giant6:/gluster/sdc/gv0
>>>>>>> Brick7: giant1:/gluster/sdd/gv0
>>>>>>> Brick8: giant2:/gluster/sdd/gv0
>>>>>>> Brick9: giant3:/gluster/sdd/gv0
>>>>>>> Brick10: giant4:/gluster/sdd/gv0
>>>>>>> Brick11: giant5:/gluster/sdd/gv0
>>>>>>> Brick12: giant6:/gluster/sdd/gv0
>>>>>>> Options Reconfigured:
>>>>>>> auth.allow: X.X.X.*,127.0.0.1
>>>>>>> nfs.disable: on
>>>>>>>
>>>>>>> Volume Name: gv2
>>>>>>> Type: Replicate
>>>>>>> Volume ID: 30c78928-5f2c-4671-becc-8deaee1a7a8d
>>>>>>> Status: Started
>>>>>>> Snapshot Count: 0
>>>>>>> Number of Bricks: 1 x 2 = 2
>>>>>>> Transport-type: tcp
>>>>>>> Bricks:
>>>>>>> Brick1: giant1:/gluster/sdd/gv2
>>>>>>> Brick2: giant2:/gluster/sdd/gv2
>>>>>>> Options Reconfigured:
>>>>>>> auth.allow: X.X.X.*,127.0.0.1
>>>>>>> cluster.granular-entry-heal: on
>>>>>>> cluster.locking-scheme: granular
>>>>>>> nfs.disable: on
>>>>>>>
>>>>>>>
>>>>>>> 2016-11-29 19:21 GMT+01:00 Micha Ober <micha2k at gmail.com
>>>>>>> <mailto:micha2k at gmail.com>>:
>>>>>>>
>>>>>>> I had opened another thread on this mailing list
>>>>>>> (Subject: "After upgrade from 3.4.2 to 3.8.5 - High
>>>>>>> CPU usage resulting in disconnects and split-brain").
>>>>>>>
>>>>>>> The title may be a bit misleading now, as I am no
>>>>>>> longer observing high CPU usage after upgrading to
>>>>>>> 3.8.6, but the disconnects are still happening and
>>>>>>> the number of files in split-brain is growing.
>>>>>>>
>>>>>>> Setup: 6 compute nodes, each serving as a glusterfs
>>>>>>> server and client, Ubuntu 14.04, two bricks per
>>>>>>> node, distribute-replicate
>>>>>>>
>>>>>>> I have two gluster volumes set up (one for scratch
>>>>>>> data, one for the slurm scheduler). Only the scratch
>>>>>>> data volume shows critical errors "[...] has not
>>>>>>> responded in the last 42 seconds, disconnecting.".
>>>>>>> So I can rule out network problems, the gigabit link
>>>>>>> between the nodes is not saturated at all. The disks
>>>>>>> are almost idle (<10%).
>>>>>>>
>>>>>>> I have glusterfs 3.4.2 on Ubuntu 12.04 on a another
>>>>>>> compute cluster, running fine since it was deployed.
>>>>>>> I had glusterfs 3.4.2 on Ubuntu 14.04 on this
>>>>>>> cluster, running fine for almost a year.
>>>>>>>
>>>>>>> After upgrading to 3.8.5, the problems (as
>>>>>>> described) started. I would like to use some of the
>>>>>>> new features of the newer versions (like bitrot),
>>>>>>> but the users can't run their compute jobs right now
>>>>>>> because the result files are garbled.
>>>>>>>
>>>>>>> 2016-11-29 18:53 GMT+01:00 Atin Mukherjee
>>>>>>> <amukherj at redhat.com <mailto:amukherj at redhat.com>>:
>>>>>>>
>>>>>>> Would you be able to share what is not working
>>>>>>> for you in 3.8.x (mention the exact version).
>>>>>>> 3.4 is quite old and falling back to an
>>>>>>> unsupported version doesn't look a feasible option.
>>>>>>>
>>>>>>> On Tue, 29 Nov 2016 at 17:01, Micha Ober
>>>>>>> <micha2k at gmail.com <mailto:micha2k at gmail.com>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I was using gluster 3.4 and upgraded to 3.8,
>>>>>>> but that version showed to be unusable for
>>>>>>> me. I now need to downgrade.
>>>>>>>
>>>>>>> I'm running Ubuntu 14.04. As upgrades of the
>>>>>>> op version are irreversible, I guess I have
>>>>>>> to delete all gluster volumes and re-create
>>>>>>> them with the downgraded version.
>>>>>>>
>>>>>>> 0. Backup data
>>>>>>> 1. Unmount all gluster volumes
>>>>>>> 2. apt-get purge glusterfs-server
>>>>>>> glusterfs-client
>>>>>>> 3. Remove PPA for 3.8
>>>>>>> 4. Add PPA for older version
>>>>>>> 5. apt-get install glusterfs-server
>>>>>>> glusterfs-client
>>>>>>> 6. Create volumes
>>>>>>>
>>>>>>> Is "purge" enough to delete all
>>>>>>> configuration files of the currently
>>>>>>> installed version or do I need to manually
>>>>>>> clear some residues before installing an
>>>>>>> older version?
>>>>>>>
>>>>>>> Thanks.
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> <mailto:Gluster-users at gluster.org>
>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>> <http://www.gluster.org/mailman/listinfo/gluster-users>
>>>>>>>
>>>>>>> --
>>>>>>> - Atin (atinm)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>> <http://www.gluster.org/mailman/listinfo/gluster-users>
>>>>>
> --
> ~ Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161219/178402e3/attachment.html>
More information about the Gluster-users
mailing list