[Gluster-devel] [Gluster-users] heal hanging

Pranith Kumar Karampuri pkarampu at redhat.com
Fri Jan 22 01:49:26 UTC 2016



On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
> We use the samba glusterfs virtual filesystem (the current version 
> provided on download.gluster.org <http://download.gluster.org>), but 
> no windows clients connecting directly.

Hmm.. Is there a way to disable using this and check if the CPU% still 
increases? What getxattr of "glusterfs.get_real_filename <filanme>" does 
is to scan the entire directory looking for strcasecmp(<filname>, 
<scanned-filename>). If anything matches then it will return the 
<scanned-filename>. But the problem is the scan is costly. So I wonder 
if this is the reason for the CPU spikes.

Pranith
>
> On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri 
> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>
>     Do you have any windows clients? I see a lot of getxattr calls for
>     "glusterfs.get_real_filename" which lead to full readdirs of the
>     directories on the brick.
>
>     Pranith
>
>     On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
>>     Pranith, could this kind of behavior be self-inflicted by us
>>     deleting files directly from the bricks? We have done that in the
>>     past to clean up an issues where gluster wouldn't allow us to
>>     delete from the mount.
>>
>>     If so, is it feasible to clean them up by running a search on the
>>     .glusterfs directories directly and removing files with a
>>     reference count of 1 that are non-zero size (or directly checking
>>     the xattrs to be sure that it's not a DHT link).
>>
>>     find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links
>>     -2 -exec rm -f "{}" \;
>>
>>     Is there anything I'm inherently missing with that approach that
>>     will further corrupt the system?
>>
>>
>>     On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick
>>     <patrick.glomski at corvidtec.com
>>     <mailto:patrick.glomski at corvidtec.com>> wrote:
>>
>>         Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl
>>         has been running on one of the bricks on gfs02b for 25 min or
>>         so and users cannot access the volume.
>>
>>         I re-listed the xattrop directories as well as a 'top' entry
>>         and heal statistics. Then I restarted the gluster services on
>>         gfs02a.
>>
>>         =================== top ===================
>>         PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
>>         COMMAND
>>          8969 root      20   0 2815m 204m 3588 S 1181.0  0.6
>>         591:06.93 glusterfsd
>>
>>         =================== xattrop ===================
>>         /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>>         xattrop-41f19453-91e4-437c-afa9-3b25614de210
>>         xattrop-9b815879-2f4d-402b-867c-a6d65087788c
>>
>>         /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>>         xattrop-70131855-3cfb-49af-abce-9d23f57fb393
>>         xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
>>
>>         /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>>         e6e47ed9-309b-42a7-8c44-28c29b9a20f8
>>         xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
>>         xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
>>         xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0
>>
>>         /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>>         xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
>>         xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413
>>
>>         /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>>         xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531
>>
>>         /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>>         xattrop-7e20fdb1-5224-4b9a-be06-568708526d70
>>
>>         /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>>         8034bc06-92cd-4fa5-8aaf-09039e79d2c8
>>         c9ce22ed-6d8b-471b-a111-b39e57f0b512
>>         94fa1d60-45ad-4341-b69c-315936b51e8d
>>         xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7
>>
>>         /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>>         xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d
>>
>>
>>         =================== heal stats ===================
>>
>>         homegfs [b0-gfsib01a] : Starting time of crawl       : Thu
>>         Jan 21 12:36:45 2016
>>         homegfs [b0-gfsib01a] : Ending time of crawl         : Thu
>>         Jan 21 12:36:45 2016
>>         homegfs [b0-gfsib01a] : Type of crawl: INDEX
>>         homegfs [b0-gfsib01a] : No. of entries healed        : 0
>>         homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
>>         homegfs [b0-gfsib01a] : No. of heal failed entries   : 0
>>
>>         homegfs [b1-gfsib01b] : Starting time of crawl       : Thu
>>         Jan 21 12:36:19 2016
>>         homegfs [b1-gfsib01b] : Ending time of crawl         : Thu
>>         Jan 21 12:36:19 2016
>>         homegfs [b1-gfsib01b] : Type of crawl: INDEX
>>         homegfs [b1-gfsib01b] : No. of entries healed        : 0
>>         homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
>>         homegfs [b1-gfsib01b] : No. of heal failed entries   : 1
>>
>>         homegfs [b2-gfsib01a] : Starting time of crawl       : Thu
>>         Jan 21 12:36:48 2016
>>         homegfs [b2-gfsib01a] : Ending time of crawl         : Thu
>>         Jan 21 12:36:48 2016
>>         homegfs [b2-gfsib01a] : Type of crawl: INDEX
>>         homegfs [b2-gfsib01a] : No. of entries healed        : 0
>>         homegfs [b2-gfsib01a] : No. of entries in split-brain: 0
>>         homegfs [b2-gfsib01a] : No. of heal failed entries   : 0
>>
>>         homegfs [b3-gfsib01b] : Starting time of crawl       : Thu
>>         Jan 21 12:36:47 2016
>>         homegfs [b3-gfsib01b] : Ending time of crawl         : Thu
>>         Jan 21 12:36:47 2016
>>         homegfs [b3-gfsib01b] : Type of crawl: INDEX
>>         homegfs [b3-gfsib01b] : No. of entries healed        : 0
>>         homegfs [b3-gfsib01b] : No. of entries in split-brain: 0
>>         homegfs [b3-gfsib01b] : No. of heal failed entries   : 0
>>
>>         homegfs [b4-gfsib02a] : Starting time of crawl       : Thu
>>         Jan 21 12:36:06 2016
>>         homegfs [b4-gfsib02a] : Ending time of crawl         : Thu
>>         Jan 21 12:36:06 2016
>>         homegfs [b4-gfsib02a] : Type of crawl: INDEX
>>         homegfs [b4-gfsib02a] : No. of entries healed        : 0
>>         homegfs [b4-gfsib02a] : No. of entries in split-brain: 0
>>         homegfs [b4-gfsib02a] : No. of heal failed entries   : 0
>>
>>         homegfs [b5-gfsib02b] : Starting time of crawl       : Thu
>>         Jan 21 12:13:40 2016
>>         homegfs [b5-gfsib02b] :                                ***
>>         Crawl is in progress ***
>>         homegfs [b5-gfsib02b] : Type of crawl: INDEX
>>         homegfs [b5-gfsib02b] : No. of entries healed        : 0
>>         homegfs [b5-gfsib02b] : No. of entries in split-brain: 0
>>         homegfs [b5-gfsib02b] : No. of heal failed entries   : 0
>>
>>         homegfs [b6-gfsib02a] : Starting time of crawl       : Thu
>>         Jan 21 12:36:58 2016
>>         homegfs [b6-gfsib02a] : Ending time of crawl         : Thu
>>         Jan 21 12:36:58 2016
>>         homegfs [b6-gfsib02a] : Type of crawl: INDEX
>>         homegfs [b6-gfsib02a] : No. of entries healed        : 0
>>         homegfs [b6-gfsib02a] : No. of entries in split-brain: 0
>>         homegfs [b6-gfsib02a] : No. of heal failed entries   : 0
>>
>>         homegfs [b7-gfsib02b] : Starting time of crawl       : Thu
>>         Jan 21 12:36:50 2016
>>         homegfs [b7-gfsib02b] : Ending time of crawl         : Thu
>>         Jan 21 12:36:50 2016
>>         homegfs [b7-gfsib02b] : Type of crawl: INDEX
>>         homegfs [b7-gfsib02b] : No. of entries healed        : 0
>>         homegfs [b7-gfsib02b] : No. of entries in split-brain: 0
>>         homegfs [b7-gfsib02b] : No. of heal failed entries   : 0
>>
>>
>>         ========================================================================================
>>         I waited a few minutes for the heals to finish and ran the
>>         heal statistics and info again. one file is in split-brain.
>>         Aside from the split-brain, the load on all systems is down
>>         now and they are behaving normally. glustershd.log is
>>         attached. What is going on???
>>
>>         Thu Jan 21 12:53:50 EST 2016
>>
>>         =================== homegfs ===================
>>
>>         homegfs [b0-gfsib01a] : Starting time of crawl       : Thu
>>         Jan 21 12:53:02 2016
>>         homegfs [b0-gfsib01a] : Ending time of crawl         : Thu
>>         Jan 21 12:53:02 2016
>>         homegfs [b0-gfsib01a] : Type of crawl: INDEX
>>         homegfs [b0-gfsib01a] : No. of entries healed        : 0
>>         homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
>>         homegfs [b0-gfsib01a] : No. of heal failed entries   : 0
>>
>>         homegfs [b1-gfsib01b] : Starting time of crawl       : Thu
>>         Jan 21 12:53:38 2016
>>         homegfs [b1-gfsib01b] : Ending time of crawl         : Thu
>>         Jan 21 12:53:38 2016
>>         homegfs [b1-gfsib01b] : Type of crawl: INDEX
>>         homegfs [b1-gfsib01b] : No. of entries healed        : 0
>>         homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
>>         homegfs [b1-gfsib01b] : No. of heal failed entries   : 1
>>
>>         homegfs [b2-gfsib01a] : Starting time of crawl       : Thu
>>         Jan 21 12:53:04 2016
>>         homegfs [b2-gfsib01a] : Ending time of crawl         : Thu
>>         Jan 21 12:53:04 2016
>>         homegfs [b2-gfsib01a] : Type of crawl: INDEX
>>         homegfs [b2-gfsib01a] : No. of entries healed        : 0
>>         homegfs [b2-gfsib01a] : No. of entries in split-brain: 0
>>         homegfs [b2-gfsib01a] : No. of heal failed entries   : 0
>>
>>         homegfs [b3-gfsib01b] : Starting time of crawl       : Thu
>>         Jan 21 12:53:04 2016
>>         homegfs [b3-gfsib01b] : Ending time of crawl         : Thu
>>         Jan 21 12:53:04 2016
>>         homegfs [b3-gfsib01b] : Type of crawl: INDEX
>>         homegfs [b3-gfsib01b] : No. of entries healed        : 0
>>         homegfs [b3-gfsib01b] : No. of entries in split-brain: 0
>>         homegfs [b3-gfsib01b] : No. of heal failed entries   : 0
>>
>>         homegfs [b4-gfsib02a] : Starting time of crawl       : Thu
>>         Jan 21 12:53:33 2016
>>         homegfs [b4-gfsib02a] : Ending time of crawl         : Thu
>>         Jan 21 12:53:33 2016
>>         homegfs [b4-gfsib02a] : Type of crawl: INDEX
>>         homegfs [b4-gfsib02a] : No. of entries healed        : 0
>>         homegfs [b4-gfsib02a] : No. of entries in split-brain: 0
>>         homegfs [b4-gfsib02a] : No. of heal failed entries   : 1
>>
>>         homegfs [b5-gfsib02b] : Starting time of crawl       : Thu
>>         Jan 21 12:53:14 2016
>>         homegfs [b5-gfsib02b] : Ending time of crawl         : Thu
>>         Jan 21 12:53:15 2016
>>         homegfs [b5-gfsib02b] : Type of crawl: INDEX
>>         homegfs [b5-gfsib02b] : No. of entries healed        : 0
>>         homegfs [b5-gfsib02b] : No. of entries in split-brain: 0
>>         homegfs [b5-gfsib02b] : No. of heal failed entries   : 3
>>
>>         homegfs [b6-gfsib02a] : Starting time of crawl       : Thu
>>         Jan 21 12:53:04 2016
>>         homegfs [b6-gfsib02a] : Ending time of crawl         : Thu
>>         Jan 21 12:53:04 2016
>>         homegfs [b6-gfsib02a] : Type of crawl: INDEX
>>         homegfs [b6-gfsib02a] : No. of entries healed        : 0
>>         homegfs [b6-gfsib02a] : No. of entries in split-brain: 0
>>         homegfs [b6-gfsib02a] : No. of heal failed entries   : 0
>>
>>         homegfs [b7-gfsib02b] : Starting time of crawl       : Thu
>>         Jan 21 12:53:09 2016
>>         homegfs [b7-gfsib02b] : Ending time of crawl         : Thu
>>         Jan 21 12:53:09 2016
>>         homegfs [b7-gfsib02b] : Type of crawl: INDEX
>>         homegfs [b7-gfsib02b] : No. of entries healed        : 0
>>         homegfs [b7-gfsib02b] : No. of entries in split-brain: 0
>>         homegfs [b7-gfsib02b] : No. of heal failed entries   : 0
>>
>>         *** gluster bug in 'gluster volume heal homegfs statistics'   ***
>>         *** Use 'gluster volume heal homegfs info' until bug is fixed ***
>>
>>         Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/
>>         Number of entries: 0
>>
>>         Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/
>>         Number of entries: 0
>>
>>         Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/
>>         Number of entries: 0
>>
>>         Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/
>>         Number of entries: 0
>>
>>         Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/
>>         /users/bangell/.gconfd - Is in split-brain
>>
>>         Number of entries: 1
>>
>>         Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/
>>         /users/bangell/.gconfd - Is in split-brain
>>
>>         /users/bangell/.gconfd/saved_state
>>         Number of entries: 2
>>
>>         Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/
>>         Number of entries: 0
>>
>>         Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/
>>         Number of entries: 0
>>
>>
>>
>>
>>         On Thu, Jan 21, 2016 at 11:10 AM, Pranith Kumar Karampuri
>>         <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>
>>
>>
>>             On 01/21/2016 09:26 PM, Glomski, Patrick wrote:
>>>             I should mention that the problem is not currently
>>>             occurring and there are no heals (output appended). By
>>>             restarting the gluster services, we can stop the crawl,
>>>             which lowers the load for a while. Subsequent crawls
>>>             seem to finish properly. For what it's worth,
>>>             files/folders that show up in the 'volume info' output
>>>             during a hung crawl don't seem to be anything out of the
>>>             ordinary.
>>>
>>>             Over the past four days, the typical time before the
>>>             problem recurs after suppressing it in this manner is an
>>>             hour. Last night when we reached out to you was the last
>>>             time it happened and the load has been low since (a
>>>             relief).  David believes that recursively listing the
>>>             files (ls -alR or similar) from a client mount can force
>>>             the issue to happen, but obviously I'd rather not unless
>>>             we have some precise thing we're looking for. Let me
>>>             know if you'd like me to attempt to drive the system
>>>             unstable like that and what I should look for. As it's a
>>>             production system, I'd rather not leave it in this state
>>>             for long.
>>
>>             Will it be possible to send glustershd, mount logs of the
>>             past 4 days? I would like to see if this is because of
>>             directory self-heal going wild (Ravi is working on
>>             throttling feature for 3.8, which will allow to put
>>             breaks on self-heal traffic)
>>
>>             Pranith
>>
>>>
>>>             [root at gfs01a xattrop]# gluster volume heal homegfs info
>>>             Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/
>>>             Number of entries: 0
>>>
>>>             Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/
>>>             Number of entries: 0
>>>
>>>             Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/
>>>             Number of entries: 0
>>>
>>>             Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/
>>>             Number of entries: 0
>>>
>>>             Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/
>>>             Number of entries: 0
>>>
>>>             Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/
>>>             Number of entries: 0
>>>
>>>             Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/
>>>             Number of entries: 0
>>>
>>>             Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/
>>>             Number of entries: 0
>>>
>>>
>>>
>>>
>>>             On Thu, Jan 21, 2016 at 10:40 AM, Pranith Kumar
>>>             Karampuri <pkarampu at redhat.com
>>>             <mailto:pkarampu at redhat.com>> wrote:
>>>
>>>
>>>
>>>                 On 01/21/2016 08:25 PM, Glomski, Patrick wrote:
>>>>                 Hello, Pranith. The typical behavior is that the
>>>>                 %cpu on a glusterfsd process jumps to number of
>>>>                 processor cores available (800% or 1200%, depending
>>>>                 on the pair of nodes involved) and the load average
>>>>                 on the machine goes very high (~20). The volume's
>>>>                 heal statistics output shows that it is crawling
>>>>                 one of the bricks and trying to heal, but this
>>>>                 crawl hangs and never seems to finish.
>>>>
>>>>                 The number of files in the xattrop directory varies
>>>>                 over time, so I ran a wc -l as you requested
>>>>                 periodically for some time and then started
>>>>                 including a datestamped list of the files that were
>>>>                 in the xattrops directory on each brick to see
>>>>                 which were persistent. All bricks had files in the
>>>>                 xattrop folder, so all results are attached.
>>>                 Thanks this info is helpful. I don't see a lot of
>>>                 files. Could you give output of "gluster volume heal
>>>                 <volname> info"? Is there any directory in there
>>>                 which is LARGE?
>>>
>>>                 Pranith
>>>
>>>>
>>>>                 Please let me know if there is anything else I can
>>>>                 provide.
>>>>
>>>>                 Patrick
>>>>
>>>>
>>>>                 On Thu, Jan 21, 2016 at 12:01 AM, Pranith Kumar
>>>>                 Karampuri <pkarampu at redhat.com
>>>>                 <mailto:pkarampu at redhat.com>> wrote:
>>>>
>>>>                     hey,
>>>>                            Which process is consuming so much cpu?
>>>>                     I went through the logs you gave me. I see that
>>>>                     the following files are in gfid mismatch state:
>>>>
>>>>                     <066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,
>>>>                     <1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,
>>>>                     <ddc92637-303a-4059-9c56-ab23b1bb6ae9/patch0008.cnvrg>,
>>>>
>>>>                     Could you give me the output of "ls
>>>>                     <brick-path>/indices/xattrop | wc -l" output on
>>>>                     all the bricks which are acting this way? This
>>>>                     will tell us the number of pending self-heals
>>>>                     on the system.
>>>>
>>>>                     Pranith
>>>>
>>>>
>>>>                     On 01/20/2016 09:26 PM, David Robinson wrote:
>>>>>                     resending with parsed logs...
>>>>>>>                     I am having issues with 3.6.6 where the load
>>>>>>>                     will spike up to 800% for one of the
>>>>>>>                     glusterfsd processes and the users can no
>>>>>>>                     longer access the system.  If I reboot the
>>>>>>>                     node, the heal will finish normally after a
>>>>>>>                     few minutes and the system will be
>>>>>>>                     responsive, but a few hours later the issue
>>>>>>>                     will start again. It look like it is hanging
>>>>>>>                     in a heal and spinning up the load on one of
>>>>>>>                     the bricks.  The heal gets stuck and says it
>>>>>>>                     is crawling and never returns. After a few
>>>>>>>                     minutes of the heal saying it is crawling,
>>>>>>>                     the load spikes up and the mounts become
>>>>>>>                     unresponsive.
>>>>>>>                     Any suggestions on how to fix this?  It has
>>>>>>>                     us stopped cold as the user can no longer
>>>>>>>                     access the systems when the load spikes...
>>>>>>>                     Logs attached.
>>>>>>>                     System setup info is:
>>>>>>>                     [root at gfs01a ~]# gluster volume info homegfs
>>>>>>>
>>>>>>>                     Volume Name: homegfs
>>>>>>>                     Type: Distributed-Replicate
>>>>>>>                     Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071
>>>>>>>                     Status: Started
>>>>>>>                     Number of Bricks: 4 x 2 = 8
>>>>>>>                     Transport-type: tcp
>>>>>>>                     Bricks:
>>>>>>>                     Brick1:
>>>>>>>                     gfsib01a.corvidtec.com:/data/brick01a/homegfs
>>>>>>>                     Brick2:
>>>>>>>                     gfsib01b.corvidtec.com:/data/brick01b/homegfs
>>>>>>>                     Brick3:
>>>>>>>                     gfsib01a.corvidtec.com:/data/brick02a/homegfs
>>>>>>>                     Brick4:
>>>>>>>                     gfsib01b.corvidtec.com:/data/brick02b/homegfs
>>>>>>>                     Brick5:
>>>>>>>                     gfsib02a.corvidtec.com:/data/brick01a/homegfs
>>>>>>>                     Brick6:
>>>>>>>                     gfsib02b.corvidtec.com:/data/brick01b/homegfs
>>>>>>>                     Brick7:
>>>>>>>                     gfsib02a.corvidtec.com:/data/brick02a/homegfs
>>>>>>>                     Brick8:
>>>>>>>                     gfsib02b.corvidtec.com:/data/brick02b/homegfs
>>>>>>>                     Options Reconfigured:
>>>>>>>                     performance.io-thread-count: 32
>>>>>>>                     performance.cache-size: 128MB
>>>>>>>                     performance.write-behind-window-size: 128MB
>>>>>>>                     server.allow-insecure: on
>>>>>>>                     network.ping-timeout: 42
>>>>>>>                     storage.owner-gid: 100
>>>>>>>                     geo-replication.indexing: off
>>>>>>>                     geo-replication.ignore-pid-check: on
>>>>>>>                     changelog.changelog: off
>>>>>>>                     changelog.fsync-interval: 3
>>>>>>>                     changelog.rollover-time: 15
>>>>>>>                     server.manage-gids: on
>>>>>>>                     diagnostics.client-log-level: WARNING
>>>>>>>                     [root at gfs01a ~]# rpm -qa | grep gluster
>>>>>>>                     gluster-nagios-common-0.1.1-0.el6.noarch
>>>>>>>                     glusterfs-fuse-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-debuginfo-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-libs-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-geo-replication-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-api-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-devel-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-api-devel-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-cli-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-rdma-3.6.6-1.el6.x86_64
>>>>>>>                     samba-vfs-glusterfs-4.1.11-2.el6.x86_64
>>>>>>>                     glusterfs-server-3.6.6-1.el6.x86_64
>>>>>>>                     glusterfs-extra-xlators-3.6.6-1.el6.x86_64
>>>>>
>>>>>
>>>>>                     _______________________________________________
>>>>>                     Gluster-devel mailing list
>>>>>                     Gluster-devel at gluster.org  <mailto:Gluster-devel at gluster.org>
>>>>>                     http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>>
>>>>                     _______________________________________________
>>>>                     Gluster-users mailing list
>>>>                     Gluster-users at gluster.org
>>>>                     <mailto:Gluster-users at gluster.org>
>>>>                     http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160122/f3d43fa9/attachment-0001.html>


More information about the Gluster-devel mailing list