[Gluster-users] File operation failure on simple distributed volume

Tue Feb 14 08:19:46 UTC 2017

Hi Yonex,

Are you still hitting this issue ?

Regards

Rafi KC

On 01/16/2017 10:36 AM, yonex wrote:
> Hi
>
> I noticed that there is a high throughput degradation while attaching
> the gdb script to a glusterfs client process. Write speed becomes 2%
> or less. It is not be able to keep thrown in production.
>
> Could you provide the custom build that you mentioned before? I am
> going to keep trying to reproduce the problem outside of the
> production environment.
>
> Regards
>
> 2017年1月8日 21:54、Mohammed Rafi K C <rkavunga at redhat.com>:
>
>> Is there any update on this ?
>>
>>
>> Regards
>>
>> Rafi KC
>>
>> On 12/24/2016 03:53 PM, yonex wrote:
>>> Rafi,
>>>
>>> Thanks again. I will try that and get back to you.
>>>
>>> Regards.
>>>
>>>
>>> 2016-12-23 18:03 GMT+09:00 Mohammed Rafi K C <rkavunga at redhat.com>:
>>>> Hi Yonex,
>>>>
>>>> As we discussed in irc #gluster-devel , I have attached the gdb script
>>>> along with this mail.
>>>>
>>>> Procedure to run the gdb script.
>>>>
>>>> 1) Install gdb,
>>>>
>>>> 2) Download and install gluster debuginfo for your machine . packages
>>>> location --- > https://cbs.centos.org/koji/buildinfo?buildID=12757
>>>>
>>>> 3) find the process id and attach gdb to the process using the command
>>>> gdb attach <pid> -x <path_to_script>
>>>>
>>>> 4) Continue running the script till you hit the problem
>>>>
>>>> 5) Stop the gdb
>>>>
>>>> 6) You will see a file called mylog.txt in the location where you ran
>>>> the gdb
>>>>
>>>>
>>>> Please keep an eye on the attached process. If you have any doubt
>>>> please
>>>> feel free to revert me.
>>>>
>>>> Regards
>>>>
>>>> Rafi KC
>>>>
>>>>
>>>> On 12/19/2016 05:33 PM, Mohammed Rafi K C wrote:
>>>>> On 12/19/2016 05:32 PM, Mohammed Rafi K C wrote:
>>>>>> Client 0-glusterfs01-client-2 has disconnected from bricks around
>>>>>> 2016-12-15 11:21:17.854249 . Can you look and/or paste the brick logs
>>>>>> around the time.
>>>>> You can find the brick name and hostname for
>>>>> 0-glusterfs01-client-2 from
>>>>> client graph.
>>>>>
>>>>> Rafi
>>>>>
>>>>>> Are you there in any of gluster irc channel, if so Have you got a
>>>>>> nickname that I can search.
>>>>>>
>>>>>> Regards
>>>>>> Rafi KC
>>>>>>
>>>>>> On 12/19/2016 04:28 PM, yonex wrote:
>>>>>>> Rafi,
>>>>>>>
>>>>>>> OK. Thanks for your guide. I found the debug log and pasted
>>>>>>> lines around that.
>>>>>>> http://pastebin.com/vhHR6PQN
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>>
>>>>>>> 2016-12-19 14:58 GMT+09:00 Mohammed Rafi K C <rkavunga at redhat.com>:
>>>>>>>> On 12/16/2016 09:10 PM, yonex wrote:
>>>>>>>>> Rafi,
>>>>>>>>>
>>>>>>>>> Thanks, the .meta feature I didn't know is very nice. I
>>>>>>>>> finally have
>>>>>>>>> captured debug logs from a client and bricks.
>>>>>>>>>
>>>>>>>>> A mount log:
>>>>>>>>> - http://pastebin.com/Tjy7wGGj
>>>>>>>>>
>>>>>>>>> FYI rickdom126 is my client's hostname.
>>>>>>>>>
>>>>>>>>> Brick logs around that time:
>>>>>>>>> - Brick1: http://pastebin.com/qzbVRSF3
>>>>>>>>> - Brick2: http://pastebin.com/j3yMNhP3
>>>>>>>>> - Brick3: http://pastebin.com/m81mVj6L
>>>>>>>>> - Brick4: http://pastebin.com/JDAbChf6
>>>>>>>>> - Brick5: http://pastebin.com/7saP6rsm
>>>>>>>>>
>>>>>>>>> However I could not find any message like "EOF on socket". I hope
>>>>>>>>> there is any helpful information in the logs above.
>>>>>>>> Indeed. I understand that the connections are in disconnected
>>>>>>>> state. But
>>>>>>>> what particularly I'm looking for is the cause of the
>>>>>>>> disconnect, Can
>>>>>>>> you paste the debug logs when it start disconnects, and around
>>>>>>>> that. You
>>>>>>>> may see a debug logs that says "disconnecting now".
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Rafi KC
>>>>>>>>
>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2016-12-14 15:20 GMT+09:00 Mohammed Rafi K C
>>>>>>>>> <rkavunga at redhat.com>:
>>>>>>>>>> On 12/13/2016 09:56 PM, yonex wrote:
>>>>>>>>>>> Hi Rafi,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your response. OK, I think it is possible to
>>>>>>>>>>> capture debug
>>>>>>>>>>> logs, since the error seems to be reproduced a few times per
>>>>>>>>>>> day. I
>>>>>>>>>>> will try that. However, so I want to avoid redundant debug
>>>>>>>>>>> outputs if
>>>>>>>>>>> possible, is there a way to enable debug log only on
>>>>>>>>>>> specific client
>>>>>>>>>>> nodes?
>>>>>>>>>> if you are using fuse mount, there is proc kind of feature
>>>>>>>>>> called .meta
>>>>>>>>>> . You can set log level through that for a particular client
>>>>>>>>>> [1] . But I
>>>>>>>>>> also want log from bricks because I suspect bricks process for
>>>>>>>>>> initiating the disconnects.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] eg : echo 8 > /mnt/glusterfs/.meta/logging/loglevel
>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>>
>>>>>>>>>>> Yonex
>>>>>>>>>>>
>>>>>>>>>>> 2016-12-13 23:33 GMT+09:00 Mohammed Rafi K C
>>>>>>>>>>> <rkavunga at redhat.com>:
>>>>>>>>>>>> Hi Yonex,
>>>>>>>>>>>>
>>>>>>>>>>>> Is this consistently reproducible ? if so, Can you enable
>>>>>>>>>>>> debug log [1]
>>>>>>>>>>>> and check for any message similar to [2]. Basically you can
>>>>>>>>>>>> even search
>>>>>>>>>>>> for "EOF on socket".
>>>>>>>>>>>>
>>>>>>>>>>>> You can set your log level back to default (INFO) after
>>>>>>>>>>>> capturing for
>>>>>>>>>>>> some time.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1] : gluster volume set <volname>
>>>>>>>>>>>> diagnostics.brick-log-level DEBUG and
>>>>>>>>>>>> gluster volume set <volname> diagnostics.client-log-level DEBUG
>>>>>>>>>>>>
>>>>>>>>>>>> [2] : http://pastebin.com/xn8QHXWa
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>>
>>>>>>>>>>>> Rafi KC
>>>>>>>>>>>>
>>>>>>>>>>>> On 12/12/2016 09:35 PM, yonex wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> When my application moves a file from it's local disk to
>>>>>>>>>>>>> FUSE-mounted
>>>>>>>>>>>>> GlusterFS volume, the client outputs many warnings and
>>>>>>>>>>>>> errors not
>>>>>>>>>>>>> always but occasionally. The volume is a simple
>>>>>>>>>>>>> distributed volume.
>>>>>>>>>>>>>
>>>>>>>>>>>>> A sample of logs pasted: http://pastebin.com/axkTCRJX
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems to come from something like a network disconnection
>>>>>>>>>>>>> ("Transport endpoint is not connected") at a glance, but other
>>>>>>>>>>>>> networking applications on the same machine don't observe
>>>>>>>>>>>>> such a
>>>>>>>>>>>>> thing. So I guess there may be a problem somewhere in
>>>>>>>>>>>>> GlusterFS stack.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It ended in failing to rename a file, logging PHP Warning
>>>>>>>>>>>>> like below:
>>>>>>>>>>>>>
>>>>>>>>>>>>> PHP Warning: rename(/glusterfs01/db1/stack/f0/13a9a2f0):
>>>>>>>>>>>>> failed
>>>>>>>>>>>>> to open stream: Input/output error in [snipped].php on
>>>>>>>>>>>>> line 278
>>>>>>>>>>>>> PHP Warning:
>>>>>>>>>>>>> rename(/var/stack/13a9a2f0,/glusterfs01/db1/stack/f0/13a9a2f0):
>>>>>>>>>>>>> Input/output error in [snipped].php on line 278
>>>>>>>>>>>>>
>>>>>>>>>>>>> Conditions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - GlusterFS 3.8.5 installed via yum CentOS-Gluster-3.8.repo
>>>>>>>>>>>>> - Volume info and status pasted: http://pastebin.com/JPt2KeD8
>>>>>>>>>>>>> - Client machines' OS: Scientific Linux 6 or CentOS 6.
>>>>>>>>>>>>> - Server machines' OS: CentOS 6.
>>>>>>>>>>>>> - Kernel version is 2.6.32-642.6.2.el6.x86_64 on all machines.
>>>>>>>>>>>>> - The number of connected FUSE clients is 260.
>>>>>>>>>>>>> - No firewall between connected machines.
>>>>>>>>>>>>> - Neither remounting volumes nor rebooting client machines
>>>>>>>>>>>>> take effect.
>>>>>>>>>>>>> - It is caused by not only rename() but also copy() and
>>>>>>>>>>>>> filesize() operation.
>>>>>>>>>>>>> - No outputs in brick logs when it happens.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any ideas? I'd appreciate any help.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170214/c5a890a4/attachment.html>