[Gluster-devel] [Gluster-users] heal hanging

Pranith Kumar Karampuri pkarampu at redhat.com
Mon Jan 25 03:22:04 UTC 2016


You guys use more than 128 clients don't you? We recently found a memory 
corruption in client-table which is used in locking. I wonder if it has 
some role to play here.
http://review.gluster.org/13241 is the fix. Could you see if you are 
seeing this issue even after this fix?

Pranith
On 01/22/2016 08:36 AM, Glomski, Patrick wrote:
> Pranith, attached are stack traces collected every second for 20 
> seconds from the high-%cpu glusterfsd process.
>
> Patrick
>
> On Thu, Jan 21, 2016 at 9:46 PM, Glomski, Patrick 
> <patrick.glomski at corvidtec.com <mailto:patrick.glomski at corvidtec.com>> 
> wrote:
>
>     Last entry for get_real_filename on any of the bricks was when we
>     turned off the samba gfapi vfs plugin earlier today:
>
>     /var/log/glusterfs/bricks/data-brick01a-homegfs.log:[2016-01-21
>     15:13:00.008239] E [server-rpc-fops.c:768:server_getxattr_cbk]
>     0-homegfs-server: 105: GETXATTR /wks_backup
>     (40e582d6-b0c7-4099-ba88-9168a3c32ca6)
>     (glusterfs.get_real_filename:desktop.ini) ==> (Permission denied)
>
>     We'll get back to you with those traces when %cpu spikes again. As
>     with most sporadic problems, as soon as you want something out of
>     it, the issue becomes harder to reproduce.
>
>
>     On Thu, Jan 21, 2016 at 9:21 PM, Pranith Kumar Karampuri
>     <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>
>
>
>         On 01/22/2016 07:25 AM, Glomski, Patrick wrote:
>>         Unfortunately, all samba mounts to the gluster volume through
>>         the gfapi vfs plugin have been disabled for the last 6 hours
>>         or so and frequency of %cpu spikes is increased. We had
>>         switched to sharing a fuse mount through samba, but I just
>>         disabled that as well. There are no samba shares of this
>>         volume now. The spikes now happen every thirty minutes or so.
>>         We've resorted to just rebooting the machine with high load
>>         for the present.
>
>         Could you see if the logs of following type are not at all coming?
>         [2016-01-21 15:13:00.005736] E
>         [server-rpc-fops.c:768:server_getxattr_cbk] 0-homegfs-server:
>         110: GETXATTR /wks_backup (40e582d6-b0c7-4099-ba88-9168a3c
>         32ca6) (glusterfs.get_real_filename:desktop.ini) ==>
>         (Permission denied)
>
>         These are operations that failed. Operations that succeed are
>         the ones that will scan the directory. But I don't have a way
>         to find them other than using tcpdumps.
>
>         At the moment I have 2 theories:
>         1) these get_real_filename calls
>         2) [2016-01-21 16:10:38.017828] E
>         [server-helpers.c:46:gid_resolve] 0-gid-cache: getpwuid_r(494)
>         failed
>         "
>
>         Yessir they are.  Normally, sssd would look to the local cache
>         file in /var/lib/sss/db/ first, to get any group or userid
>         information, then go out to the domain controller.  I put the
>         options that we are using on our GFS volumes below…  Thanks
>         for your help.
>
>         We had been running sssd with sssd_nss and sssd_be
>         sub-processes on these systems for a long time, under the GFS
>         3.5.2 code, and not run into the problem that David described
>         with the high cpu usage on sssd_nss.
>
>         *"
>         *That was Tom Young's email 1.5 years back when we debugged
>         it. But the process which was consuming lot of cpu is
>         sssd_nss. So I am not sure if it is same issue. Let us debug
>         to see '1)' doesn't happen. The gstack traces I asked for
>         should also help.
>
>
>         Pranith
>>
>>         On Thu, Jan 21, 2016 at 8:49 PM, Pranith Kumar Karampuri
>>         <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>
>>
>>
>>             On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
>>>             We use the samba glusterfs virtual filesystem (the
>>>             current version provided on download.gluster.org
>>>             <http://download.gluster.org>), but no windows clients
>>>             connecting directly.
>>
>>             Hmm.. Is there a way to disable using this and check if
>>             the CPU% still increases? What getxattr of
>>             "glusterfs.get_real_filename <filanme>" does is to scan
>>             the entire directory looking for strcasecmp(<filname>,
>>             <scanned-filename>). If anything matches then it will
>>             return the <scanned-filename>. But the problem is the
>>             scan is costly. So I wonder if this is the reason for the
>>             CPU spikes.
>>
>>             Pranith
>>
>>>
>>>             On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri
>>>             <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>>
>>>                 Do you have any windows clients? I see a lot of
>>>                 getxattr calls for "glusterfs.get_real_filename"
>>>                 which lead to full readdirs of the directories on
>>>                 the brick.
>>>
>>>                 Pranith
>>>
>>>                 On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
>>>>                 Pranith, could this kind of behavior be
>>>>                 self-inflicted by us deleting files directly from
>>>>                 the bricks? We have done that in the past to clean
>>>>                 up an issues where gluster wouldn't allow us to
>>>>                 delete from the mount.
>>>>
>>>>                 If so, is it feasible to clean them up by running a
>>>>                 search on the .glusterfs directories directly and
>>>>                 removing files with a reference count of 1 that are
>>>>                 non-zero size (or directly checking the xattrs to
>>>>                 be sure that it's not a DHT link).
>>>>
>>>>                 find /data/brick01a/homegfs/.glusterfs -type f -not
>>>>                 -empty -links -2 -exec rm -f "{}" \;
>>>>
>>>>                 Is there anything I'm inherently missing with that
>>>>                 approach that will further corrupt the system?
>>>>
>>>>
>>>>                 On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick
>>>>                 <patrick.glomski at corvidtec.com
>>>>                 <mailto:patrick.glomski at corvidtec.com>> wrote:
>>>>
>>>>                     Load spiked again: ~1200%cpu on gfs02a for
>>>>                     glusterfsd. Crawl has been running on one of
>>>>                     the bricks on gfs02b for 25 min or so and users
>>>>                     cannot access the volume.
>>>>
>>>>                     I re-listed the xattrop directories as well as
>>>>                     a 'top' entry and heal statistics. Then I
>>>>                     restarted the gluster services on gfs02a.
>>>>
>>>>                     =================== top ===================
>>>>                     PID USER PR  NI  VIRT RES  SHR S %CPU %MEM
>>>>                     TIME+ COMMAND
>>>>                      8969 root      20 0 2815m 204m 3588 S 1181.0
>>>>                     0.6 591:06.93 glusterfsd
>>>>
>>>>                     =================== xattrop ===================
>>>>                     /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>>>>                     xattrop-41f19453-91e4-437c-afa9-3b25614de210
>>>>                     xattrop-9b815879-2f4d-402b-867c-a6d65087788c
>>>>
>>>>                     /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>>>>                     xattrop-70131855-3cfb-49af-abce-9d23f57fb393
>>>>                     xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
>>>>
>>>>                     /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>>>>                     e6e47ed9-309b-42a7-8c44-28c29b9a20f8
>>>>                     xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
>>>>                     xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
>>>>                     xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0
>>>>
>>>>                     /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>>>>                     xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
>>>>                     xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413
>>>>
>>>>                     /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>>>>                     xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531
>>>>
>>>>                     /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>>>>                     xattrop-7e20fdb1-5224-4b9a-be06-568708526d70
>>>>
>>>>                     /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>>>>                     8034bc06-92cd-4fa5-8aaf-09039e79d2c8
>>>>                     c9ce22ed-6d8b-471b-a111-b39e57f0b512
>>>>                     94fa1d60-45ad-4341-b69c-315936b51e8d
>>>>                     xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7
>>>>
>>>>                     /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>>>>                     xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d
>>>>
>>>>
>>>>                     =================== heal stats ===================
>>>>
>>>>                     homegfs [b0-gfsib01a] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:36:45 2016
>>>>                     homegfs [b0-gfsib01a] : Ending time of crawl :
>>>>                     Thu Jan 21 12:36:45 2016
>>>>                     homegfs [b0-gfsib01a] : Type of crawl: INDEX
>>>>                     homegfs [b0-gfsib01a] : No. of entries healed : 0
>>>>                     homegfs [b0-gfsib01a] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b0-gfsib01a] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b1-gfsib01b] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:36:19 2016
>>>>                     homegfs [b1-gfsib01b] : Ending time of crawl :
>>>>                     Thu Jan 21 12:36:19 2016
>>>>                     homegfs [b1-gfsib01b] : Type of crawl: INDEX
>>>>                     homegfs [b1-gfsib01b] : No. of entries healed : 0
>>>>                     homegfs [b1-gfsib01b] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b1-gfsib01b] : No. of heal failed
>>>>                     entries   : 1
>>>>
>>>>                     homegfs [b2-gfsib01a] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:36:48 2016
>>>>                     homegfs [b2-gfsib01a] : Ending time of crawl :
>>>>                     Thu Jan 21 12:36:48 2016
>>>>                     homegfs [b2-gfsib01a] : Type of crawl: INDEX
>>>>                     homegfs [b2-gfsib01a] : No. of entries healed : 0
>>>>                     homegfs [b2-gfsib01a] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b2-gfsib01a] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b3-gfsib01b] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:36:47 2016
>>>>                     homegfs [b3-gfsib01b] : Ending time of crawl :
>>>>                     Thu Jan 21 12:36:47 2016
>>>>                     homegfs [b3-gfsib01b] : Type of crawl: INDEX
>>>>                     homegfs [b3-gfsib01b] : No. of entries healed : 0
>>>>                     homegfs [b3-gfsib01b] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b3-gfsib01b] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b4-gfsib02a] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:36:06 2016
>>>>                     homegfs [b4-gfsib02a] : Ending time of crawl :
>>>>                     Thu Jan 21 12:36:06 2016
>>>>                     homegfs [b4-gfsib02a] : Type of crawl: INDEX
>>>>                     homegfs [b4-gfsib02a] : No. of entries healed : 0
>>>>                     homegfs [b4-gfsib02a] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b4-gfsib02a] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b5-gfsib02b] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:13:40 2016
>>>>                     homegfs [b5-gfsib02b] : *** Crawl is in
>>>>                     progress ***
>>>>                     homegfs [b5-gfsib02b] : Type of crawl: INDEX
>>>>                     homegfs [b5-gfsib02b] : No. of entries healed : 0
>>>>                     homegfs [b5-gfsib02b] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b5-gfsib02b] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b6-gfsib02a] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:36:58 2016
>>>>                     homegfs [b6-gfsib02a] : Ending time of crawl :
>>>>                     Thu Jan 21 12:36:58 2016
>>>>                     homegfs [b6-gfsib02a] : Type of crawl: INDEX
>>>>                     homegfs [b6-gfsib02a] : No. of entries healed : 0
>>>>                     homegfs [b6-gfsib02a] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b6-gfsib02a] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b7-gfsib02b] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:36:50 2016
>>>>                     homegfs [b7-gfsib02b] : Ending time of crawl :
>>>>                     Thu Jan 21 12:36:50 2016
>>>>                     homegfs [b7-gfsib02b] : Type of crawl: INDEX
>>>>                     homegfs [b7-gfsib02b] : No. of entries healed : 0
>>>>                     homegfs [b7-gfsib02b] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b7-gfsib02b] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>
>>>>                     ========================================================================================
>>>>                     I waited a few minutes for the heals to finish
>>>>                     and ran the heal statistics and info again. one
>>>>                     file is in split-brain. Aside from the
>>>>                     split-brain, the load on all systems is down
>>>>                     now and they are behaving normally.
>>>>                     glustershd.log is attached. What is going on???
>>>>
>>>>                     Thu Jan 21 12:53:50 EST 2016
>>>>
>>>>                     =================== homegfs ===================
>>>>
>>>>                     homegfs [b0-gfsib01a] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:53:02 2016
>>>>                     homegfs [b0-gfsib01a] : Ending time of crawl :
>>>>                     Thu Jan 21 12:53:02 2016
>>>>                     homegfs [b0-gfsib01a] : Type of crawl: INDEX
>>>>                     homegfs [b0-gfsib01a] : No. of entries healed : 0
>>>>                     homegfs [b0-gfsib01a] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b0-gfsib01a] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b1-gfsib01b] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:53:38 2016
>>>>                     homegfs [b1-gfsib01b] : Ending time of crawl :
>>>>                     Thu Jan 21 12:53:38 2016
>>>>                     homegfs [b1-gfsib01b] : Type of crawl: INDEX
>>>>                     homegfs [b1-gfsib01b] : No. of entries healed : 0
>>>>                     homegfs [b1-gfsib01b] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b1-gfsib01b] : No. of heal failed
>>>>                     entries   : 1
>>>>
>>>>                     homegfs [b2-gfsib01a] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:53:04 2016
>>>>                     homegfs [b2-gfsib01a] : Ending time of crawl :
>>>>                     Thu Jan 21 12:53:04 2016
>>>>                     homegfs [b2-gfsib01a] : Type of crawl: INDEX
>>>>                     homegfs [b2-gfsib01a] : No. of entries healed : 0
>>>>                     homegfs [b2-gfsib01a] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b2-gfsib01a] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b3-gfsib01b] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:53:04 2016
>>>>                     homegfs [b3-gfsib01b] : Ending time of crawl :
>>>>                     Thu Jan 21 12:53:04 2016
>>>>                     homegfs [b3-gfsib01b] : Type of crawl: INDEX
>>>>                     homegfs [b3-gfsib01b] : No. of entries healed : 0
>>>>                     homegfs [b3-gfsib01b] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b3-gfsib01b] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b4-gfsib02a] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:53:33 2016
>>>>                     homegfs [b4-gfsib02a] : Ending time of crawl :
>>>>                     Thu Jan 21 12:53:33 2016
>>>>                     homegfs [b4-gfsib02a] : Type of crawl: INDEX
>>>>                     homegfs [b4-gfsib02a] : No. of entries healed : 0
>>>>                     homegfs [b4-gfsib02a] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b4-gfsib02a] : No. of heal failed
>>>>                     entries   : 1
>>>>
>>>>                     homegfs [b5-gfsib02b] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:53:14 2016
>>>>                     homegfs [b5-gfsib02b] : Ending time of crawl :
>>>>                     Thu Jan 21 12:53:15 2016
>>>>                     homegfs [b5-gfsib02b] : Type of crawl: INDEX
>>>>                     homegfs [b5-gfsib02b] : No. of entries healed : 0
>>>>                     homegfs [b5-gfsib02b] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b5-gfsib02b] : No. of heal failed
>>>>                     entries   : 3
>>>>
>>>>                     homegfs [b6-gfsib02a] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:53:04 2016
>>>>                     homegfs [b6-gfsib02a] : Ending time of crawl :
>>>>                     Thu Jan 21 12:53:04 2016
>>>>                     homegfs [b6-gfsib02a] : Type of crawl: INDEX
>>>>                     homegfs [b6-gfsib02a] : No. of entries healed : 0
>>>>                     homegfs [b6-gfsib02a] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b6-gfsib02a] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     homegfs [b7-gfsib02b] : Starting time of
>>>>                     crawl       : Thu Jan 21 12:53:09 2016
>>>>                     homegfs [b7-gfsib02b] : Ending time of crawl :
>>>>                     Thu Jan 21 12:53:09 2016
>>>>                     homegfs [b7-gfsib02b] : Type of crawl: INDEX
>>>>                     homegfs [b7-gfsib02b] : No. of entries healed : 0
>>>>                     homegfs [b7-gfsib02b] : No. of entries in
>>>>                     split-brain: 0
>>>>                     homegfs [b7-gfsib02b] : No. of heal failed
>>>>                     entries   : 0
>>>>
>>>>                     *** gluster bug in 'gluster volume heal homegfs
>>>>                     statistics' ***
>>>>                     *** Use 'gluster volume heal homegfs info'
>>>>                     until bug is fixed ***
>>>>
>>>>                     Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/
>>>>                     Number of entries: 0
>>>>
>>>>                     Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/
>>>>                     Number of entries: 0
>>>>
>>>>                     Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/
>>>>                     Number of entries: 0
>>>>
>>>>                     Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/
>>>>                     Number of entries: 0
>>>>
>>>>                     Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/
>>>>                     /users/bangell/.gconfd - Is in split-brain
>>>>
>>>>                     Number of entries: 1
>>>>
>>>>                     Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/
>>>>                     /users/bangell/.gconfd - Is in split-brain
>>>>
>>>>                     /users/bangell/.gconfd/saved_state
>>>>                     Number of entries: 2
>>>>
>>>>                     Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/
>>>>                     Number of entries: 0
>>>>
>>>>                     Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/
>>>>                     Number of entries: 0
>>>>
>>>>
>>>>
>>>>
>>>>                     On Thu, Jan 21, 2016 at 11:10 AM, Pranith Kumar
>>>>                     Karampuri <pkarampu at redhat.com
>>>>                     <mailto:pkarampu at redhat.com>> wrote:
>>>>
>>>>
>>>>
>>>>                         On 01/21/2016 09:26 PM, Glomski, Patrick wrote:
>>>>>                         I should mention that the problem is not
>>>>>                         currently occurring and there are no heals
>>>>>                         (output appended). By restarting the
>>>>>                         gluster services, we can stop the crawl,
>>>>>                         which lowers the load for a while.
>>>>>                         Subsequent crawls seem to finish properly.
>>>>>                         For what it's worth, files/folders that
>>>>>                         show up in the 'volume info' output during
>>>>>                         a hung crawl don't seem to be anything out
>>>>>                         of the ordinary.
>>>>>
>>>>>                         Over the past four days, the typical time
>>>>>                         before the problem recurs after
>>>>>                         suppressing it in this manner is an hour.
>>>>>                         Last night when we reached out to you was
>>>>>                         the last time it happened and the load has
>>>>>                         been low since (a relief). David believes
>>>>>                         that recursively listing the files (ls
>>>>>                         -alR or similar) from a client mount can
>>>>>                         force the issue to happen, but obviously
>>>>>                         I'd rather not unless we have some precise
>>>>>                         thing we're looking for. Let me know if
>>>>>                         you'd like me to attempt to drive the
>>>>>                         system unstable like that and what I
>>>>>                         should look for. As it's a production
>>>>>                         system, I'd rather not leave it in this
>>>>>                         state for long.
>>>>
>>>>                         Will it be possible to send glustershd,
>>>>                         mount logs of the past 4 days? I would like
>>>>                         to see if this is because of directory
>>>>                         self-heal going wild (Ravi is working on
>>>>                         throttling feature for 3.8, which will
>>>>                         allow to put breaks on self-heal traffic)
>>>>
>>>>                         Pranith
>>>>
>>>>>
>>>>>                         [root at gfs01a xattrop]# gluster volume heal
>>>>>                         homegfs info
>>>>>                         Brick
>>>>>                         gfs01a.corvidtec.com:/data/brick01a/homegfs/
>>>>>                         Number of entries: 0
>>>>>
>>>>>                         Brick
>>>>>                         gfs01b.corvidtec.com:/data/brick01b/homegfs/
>>>>>                         Number of entries: 0
>>>>>
>>>>>                         Brick
>>>>>                         gfs01a.corvidtec.com:/data/brick02a/homegfs/
>>>>>                         Number of entries: 0
>>>>>
>>>>>                         Brick
>>>>>                         gfs01b.corvidtec.com:/data/brick02b/homegfs/
>>>>>                         Number of entries: 0
>>>>>
>>>>>                         Brick
>>>>>                         gfs02a.corvidtec.com:/data/brick01a/homegfs/
>>>>>                         Number of entries: 0
>>>>>
>>>>>                         Brick
>>>>>                         gfs02b.corvidtec.com:/data/brick01b/homegfs/
>>>>>                         Number of entries: 0
>>>>>
>>>>>                         Brick
>>>>>                         gfs02a.corvidtec.com:/data/brick02a/homegfs/
>>>>>                         Number of entries: 0
>>>>>
>>>>>                         Brick
>>>>>                         gfs02b.corvidtec.com:/data/brick02b/homegfs/
>>>>>                         Number of entries: 0
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                         On Thu, Jan 21, 2016 at 10:40 AM, Pranith
>>>>>                         Kumar Karampuri <pkarampu at redhat.com
>>>>>                         <mailto:pkarampu at redhat.com>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>                             On 01/21/2016 08:25 PM, Glomski,
>>>>>                             Patrick wrote:
>>>>>>                             Hello, Pranith. The typical behavior
>>>>>>                             is that the %cpu on a glusterfsd
>>>>>>                             process jumps to number of processor
>>>>>>                             cores available (800% or 1200%,
>>>>>>                             depending on the pair of nodes
>>>>>>                             involved) and the load average on the
>>>>>>                             machine goes very high (~20). The
>>>>>>                             volume's heal statistics output shows
>>>>>>                             that it is crawling one of the bricks
>>>>>>                             and trying to heal, but this crawl
>>>>>>                             hangs and never seems to finish.
>>>>>>
>>>>>>                             The number of files in the xattrop
>>>>>>                             directory varies over time, so I ran
>>>>>>                             a wc -l as you requested periodically
>>>>>>                             for some time and then started
>>>>>>                             including a datestamped list of the
>>>>>>                             files that were in the xattrops
>>>>>>                             directory on each brick to see which
>>>>>>                             were persistent. All bricks had files
>>>>>>                             in the xattrop folder, so all results
>>>>>>                             are attached.
>>>>>                             Thanks this info is helpful. I don't
>>>>>                             see a lot of files. Could you give
>>>>>                             output of "gluster volume heal
>>>>>                             <volname> info"? Is there any
>>>>>                             directory in there which is LARGE?
>>>>>
>>>>>                             Pranith
>>>>>
>>>>>>
>>>>>>                             Please let me know if there is
>>>>>>                             anything else I can provide.
>>>>>>
>>>>>>                             Patrick
>>>>>>
>>>>>>
>>>>>>                             On Thu, Jan 21, 2016 at 12:01 AM,
>>>>>>                             Pranith Kumar Karampuri
>>>>>>                             <pkarampu at redhat.com
>>>>>>                             <mailto:pkarampu at redhat.com>> wrote:
>>>>>>
>>>>>>                                 hey,
>>>>>>                                        Which process is consuming
>>>>>>                                 so much cpu? I went through the
>>>>>>                                 logs you gave me. I see that the
>>>>>>                                 following files are in gfid
>>>>>>                                 mismatch state:
>>>>>>
>>>>>>                                 <066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,
>>>>>>                                 <1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,
>>>>>>                                 <ddc92637-303a-4059-9c56-ab23b1bb6ae9/patch0008.cnvrg>,
>>>>>>
>>>>>>                                 Could you give me the output of
>>>>>>                                 "ls <brick-path>/indices/xattrop
>>>>>>                                 | wc -l" output on all the bricks
>>>>>>                                 which are acting this way? This
>>>>>>                                 will tell us the number of
>>>>>>                                 pending self-heals on the system.
>>>>>>
>>>>>>                                 Pranith
>>>>>>
>>>>>>
>>>>>>                                 On 01/20/2016 09:26 PM, David
>>>>>>                                 Robinson wrote:
>>>>>>>                                 resending with parsed logs...
>>>>>>>>>                                 I am having issues with 3.6.6
>>>>>>>>>                                 where the load will spike up
>>>>>>>>>                                 to 800% for one of the
>>>>>>>>>                                 glusterfsd processes and the
>>>>>>>>>                                 users can no longer access the
>>>>>>>>>                                 system.  If I reboot the node,
>>>>>>>>>                                 the heal will finish normally
>>>>>>>>>                                 after a few minutes and the
>>>>>>>>>                                 system will be responsive,
>>>>>>>>>                                 but a few hours later the
>>>>>>>>>                                 issue will start again. It
>>>>>>>>>                                 look like it is hanging in a
>>>>>>>>>                                 heal and spinning up the load
>>>>>>>>>                                 on one of the bricks.  The
>>>>>>>>>                                 heal gets stuck and says it is
>>>>>>>>>                                 crawling and never returns.
>>>>>>>>>                                 After a few minutes of the
>>>>>>>>>                                 heal saying it is crawling,
>>>>>>>>>                                 the load spikes up and the
>>>>>>>>>                                 mounts become unresponsive.
>>>>>>>>>                                 Any suggestions on how to fix
>>>>>>>>>                                 this?  It has us stopped cold
>>>>>>>>>                                 as the user can no longer
>>>>>>>>>                                 access the systems when the
>>>>>>>>>                                 load spikes... Logs attached.
>>>>>>>>>                                 System setup info is:
>>>>>>>>>                                 [root at gfs01a ~]# gluster
>>>>>>>>>                                 volume info homegfs
>>>>>>>>>
>>>>>>>>>                                 Volume Name: homegfs
>>>>>>>>>                                 Type: Distributed-Replicate
>>>>>>>>>                                 Volume ID:
>>>>>>>>>                                 1e32672a-f1b7-4b58-ba94-58c085e59071
>>>>>>>>>                                 Status: Started
>>>>>>>>>                                 Number of Bricks: 4 x 2 = 8
>>>>>>>>>                                 Transport-type: tcp
>>>>>>>>>                                 Bricks:
>>>>>>>>>                                 Brick1:
>>>>>>>>>                                 gfsib01a.corvidtec.com:/data/brick01a/homegfs
>>>>>>>>>                                 Brick2:
>>>>>>>>>                                 gfsib01b.corvidtec.com:/data/brick01b/homegfs
>>>>>>>>>                                 Brick3:
>>>>>>>>>                                 gfsib01a.corvidtec.com:/data/brick02a/homegfs
>>>>>>>>>                                 Brick4:
>>>>>>>>>                                 gfsib01b.corvidtec.com:/data/brick02b/homegfs
>>>>>>>>>                                 Brick5:
>>>>>>>>>                                 gfsib02a.corvidtec.com:/data/brick01a/homegfs
>>>>>>>>>                                 Brick6:
>>>>>>>>>                                 gfsib02b.corvidtec.com:/data/brick01b/homegfs
>>>>>>>>>                                 Brick7:
>>>>>>>>>                                 gfsib02a.corvidtec.com:/data/brick02a/homegfs
>>>>>>>>>                                 Brick8:
>>>>>>>>>                                 gfsib02b.corvidtec.com:/data/brick02b/homegfs
>>>>>>>>>                                 Options Reconfigured:
>>>>>>>>>                                 performance.io-thread-count: 32
>>>>>>>>>                                 performance.cache-size: 128MB
>>>>>>>>>                                 performance.write-behind-window-size:
>>>>>>>>>                                 128MB
>>>>>>>>>                                 server.allow-insecure: on
>>>>>>>>>                                 network.ping-timeout: 42
>>>>>>>>>                                 storage.owner-gid: 100
>>>>>>>>>                                 geo-replication.indexing: off
>>>>>>>>>                                 geo-replication.ignore-pid-check:
>>>>>>>>>                                 on
>>>>>>>>>                                 changelog.changelog: off
>>>>>>>>>                                 changelog.fsync-interval: 3
>>>>>>>>>                                 changelog.rollover-time: 15
>>>>>>>>>                                 server.manage-gids: on
>>>>>>>>>                                 diagnostics.client-log-level:
>>>>>>>>>                                 WARNING
>>>>>>>>>                                 [root at gfs01a ~]# rpm -qa |
>>>>>>>>>                                 grep gluster
>>>>>>>>>                                 gluster-nagios-common-0.1.1-0.el6.noarch
>>>>>>>>>                                 glusterfs-fuse-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-debuginfo-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-libs-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-geo-replication-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-api-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-devel-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-api-devel-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-cli-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-rdma-3.6.6-1.el6.x86_64
>>>>>>>>>                                 samba-vfs-glusterfs-4.1.11-2.el6.x86_64
>>>>>>>>>                                 glusterfs-server-3.6.6-1.el6.x86_64
>>>>>>>>>                                 glusterfs-extra-xlators-3.6.6-1.el6.x86_64
>>>>>>>
>>>>>>>
>>>>>>>                                 _______________________________________________
>>>>>>>                                 Gluster-devel mailing list
>>>>>>>                                 Gluster-devel at gluster.org  <mailto:Gluster-devel at gluster.org>
>>>>>>>                                 http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>>
>>>>>>
>>>>>>                                 _______________________________________________
>>>>>>                                 Gluster-users mailing list
>>>>>>                                 Gluster-users at gluster.org
>>>>>>                                 <mailto:Gluster-users at gluster.org>
>>>>>>                                 http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160125/ef1363ab/attachment-0001.html>


More information about the Gluster-devel mailing list