[Gluster-devel] [Gluster-users] heal hanging
Pranith Kumar Karampuri
pkarampu at redhat.com
Fri Jan 22 02:21:51 UTC 2016
On 01/22/2016 07:25 AM, Glomski, Patrick wrote:
> Unfortunately, all samba mounts to the gluster volume through the
> gfapi vfs plugin have been disabled for the last 6 hours or so and
> frequency of %cpu spikes is increased. We had switched to sharing a
> fuse mount through samba, but I just disabled that as well. There are
> no samba shares of this volume now. The spikes now happen every thirty
> minutes or so. We've resorted to just rebooting the machine with high
> load for the present.
Could you see if the logs of following type are not at all coming?
[2016-01-21 15:13:00.005736] E
[server-rpc-fops.c:768:server_getxattr_cbk] 0-homegfs-server: 110:
GETXATTR /wks_backup (40e582d6-b0c7-4099-ba88-9168a3c
32ca6) (glusterfs.get_real_filename:desktop.ini) ==> (Permission denied)
These are operations that failed. Operations that succeed are the ones
that will scan the directory. But I don't have a way to find them other
than using tcpdumps.
At the moment I have 2 theories:
1) these get_real_filename calls
2) [2016-01-21 16:10:38.017828] E [server-helpers.c:46:gid_resolve]
0-gid-cache: getpwuid_r(494) failed
"
Yessir they are. Normally, sssd would look to the local cache file in
/var/lib/sss/db/ first, to get any group or userid information, then go
out to the domain controller. I put the options that we are using on
our GFS volumes below… Thanks for your help.
We had been running sssd with sssd_nss and sssd_be sub-processes on
these systems for a long time, under the GFS 3.5.2 code, and not run
into the problem that David described with the high cpu usage on sssd_nss.
*"
*That was Tom Young's email 1.5 years back when we debugged it. But the
process which was consuming lot of cpu is sssd_nss. So I am not sure if
it is same issue. Let us debug to see '1)' doesn't happen. The gstack
traces I asked for should also help.
Pranith
>
> On Thu, Jan 21, 2016 at 8:49 PM, Pranith Kumar Karampuri
> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>
>
>
> On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
>> We use the samba glusterfs virtual filesystem (the current
>> version provided on download.gluster.org
>> <http://download.gluster.org>), but no windows clients connecting
>> directly.
>
> Hmm.. Is there a way to disable using this and check if the CPU%
> still increases? What getxattr of "glusterfs.get_real_filename
> <filanme>" does is to scan the entire directory looking for
> strcasecmp(<filname>, <scanned-filename>). If anything matches
> then it will return the <scanned-filename>. But the problem is the
> scan is costly. So I wonder if this is the reason for the CPU spikes.
>
> Pranith
>
>>
>> On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri
>> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>
>> Do you have any windows clients? I see a lot of getxattr
>> calls for "glusterfs.get_real_filename" which lead to full
>> readdirs of the directories on the brick.
>>
>> Pranith
>>
>> On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
>>> Pranith, could this kind of behavior be self-inflicted by us
>>> deleting files directly from the bricks? We have done that
>>> in the past to clean up an issues where gluster wouldn't
>>> allow us to delete from the mount.
>>>
>>> If so, is it feasible to clean them up by running a search
>>> on the .glusterfs directories directly and removing files
>>> with a reference count of 1 that are non-zero size (or
>>> directly checking the xattrs to be sure that it's not a DHT
>>> link).
>>>
>>> find /data/brick01a/homegfs/.glusterfs -type f -not -empty
>>> -links -2 -exec rm -f "{}" \;
>>>
>>> Is there anything I'm inherently missing with that approach
>>> that will further corrupt the system?
>>>
>>>
>>> On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick
>>> <patrick.glomski at corvidtec.com
>>> <mailto:patrick.glomski at corvidtec.com>> wrote:
>>>
>>> Load spiked again: ~1200%cpu on gfs02a for glusterfsd.
>>> Crawl has been running on one of the bricks on gfs02b
>>> for 25 min or so and users cannot access the volume.
>>>
>>> I re-listed the xattrop directories as well as a 'top'
>>> entry and heal statistics. Then I restarted the gluster
>>> services on gfs02a.
>>>
>>> =================== top ===================
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>> COMMAND
>>> 8969 root 20 0 2815m 204m 3588 S 1181.0 0.6
>>> 591:06.93 glusterfsd
>>>
>>> =================== xattrop ===================
>>> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-41f19453-91e4-437c-afa9-3b25614de210
>>> xattrop-9b815879-2f4d-402b-867c-a6d65087788c
>>>
>>> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-70131855-3cfb-49af-abce-9d23f57fb393
>>> xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
>>>
>>> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>>> e6e47ed9-309b-42a7-8c44-28c29b9a20f8
>>> xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
>>> xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
>>> xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0
>>>
>>> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
>>> xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413
>>>
>>> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531
>>>
>>> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-7e20fdb1-5224-4b9a-be06-568708526d70
>>>
>>> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>>> 8034bc06-92cd-4fa5-8aaf-09039e79d2c8
>>> c9ce22ed-6d8b-471b-a111-b39e57f0b512
>>> 94fa1d60-45ad-4341-b69c-315936b51e8d
>>> xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7
>>>
>>> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d
>>>
>>>
>>> =================== heal stats ===================
>>>
>>> homegfs [b0-gfsib01a] : Starting time of crawl :
>>> Thu Jan 21 12:36:45 2016
>>> homegfs [b0-gfsib01a] : Ending time of crawl :
>>> Thu Jan 21 12:36:45 2016
>>> homegfs [b0-gfsib01a] : Type of crawl: INDEX
>>> homegfs [b0-gfsib01a] : No. of entries healed : 0
>>> homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
>>> homegfs [b0-gfsib01a] : No. of heal failed entries : 0
>>>
>>> homegfs [b1-gfsib01b] : Starting time of crawl :
>>> Thu Jan 21 12:36:19 2016
>>> homegfs [b1-gfsib01b] : Ending time of crawl :
>>> Thu Jan 21 12:36:19 2016
>>> homegfs [b1-gfsib01b] : Type of crawl: INDEX
>>> homegfs [b1-gfsib01b] : No. of entries healed : 0
>>> homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
>>> homegfs [b1-gfsib01b] : No. of heal failed entries : 1
>>>
>>> homegfs [b2-gfsib01a] : Starting time of crawl :
>>> Thu Jan 21 12:36:48 2016
>>> homegfs [b2-gfsib01a] : Ending time of crawl :
>>> Thu Jan 21 12:36:48 2016
>>> homegfs [b2-gfsib01a] : Type of crawl: INDEX
>>> homegfs [b2-gfsib01a] : No. of entries healed : 0
>>> homegfs [b2-gfsib01a] : No. of entries in split-brain: 0
>>> homegfs [b2-gfsib01a] : No. of heal failed entries : 0
>>>
>>> homegfs [b3-gfsib01b] : Starting time of crawl :
>>> Thu Jan 21 12:36:47 2016
>>> homegfs [b3-gfsib01b] : Ending time of crawl :
>>> Thu Jan 21 12:36:47 2016
>>> homegfs [b3-gfsib01b] : Type of crawl: INDEX
>>> homegfs [b3-gfsib01b] : No. of entries healed : 0
>>> homegfs [b3-gfsib01b] : No. of entries in split-brain: 0
>>> homegfs [b3-gfsib01b] : No. of heal failed entries : 0
>>>
>>> homegfs [b4-gfsib02a] : Starting time of crawl :
>>> Thu Jan 21 12:36:06 2016
>>> homegfs [b4-gfsib02a] : Ending time of crawl :
>>> Thu Jan 21 12:36:06 2016
>>> homegfs [b4-gfsib02a] : Type of crawl: INDEX
>>> homegfs [b4-gfsib02a] : No. of entries healed : 0
>>> homegfs [b4-gfsib02a] : No. of entries in split-brain: 0
>>> homegfs [b4-gfsib02a] : No. of heal failed entries : 0
>>>
>>> homegfs [b5-gfsib02b] : Starting time of crawl :
>>> Thu Jan 21 12:13:40 2016
>>> homegfs [b5-gfsib02b] : *** Crawl is in progress ***
>>> homegfs [b5-gfsib02b] : Type of crawl: INDEX
>>> homegfs [b5-gfsib02b] : No. of entries healed : 0
>>> homegfs [b5-gfsib02b] : No. of entries in split-brain: 0
>>> homegfs [b5-gfsib02b] : No. of heal failed entries : 0
>>>
>>> homegfs [b6-gfsib02a] : Starting time of crawl :
>>> Thu Jan 21 12:36:58 2016
>>> homegfs [b6-gfsib02a] : Ending time of crawl :
>>> Thu Jan 21 12:36:58 2016
>>> homegfs [b6-gfsib02a] : Type of crawl: INDEX
>>> homegfs [b6-gfsib02a] : No. of entries healed : 0
>>> homegfs [b6-gfsib02a] : No. of entries in split-brain: 0
>>> homegfs [b6-gfsib02a] : No. of heal failed entries : 0
>>>
>>> homegfs [b7-gfsib02b] : Starting time of crawl :
>>> Thu Jan 21 12:36:50 2016
>>> homegfs [b7-gfsib02b] : Ending time of crawl :
>>> Thu Jan 21 12:36:50 2016
>>> homegfs [b7-gfsib02b] : Type of crawl: INDEX
>>> homegfs [b7-gfsib02b] : No. of entries healed : 0
>>> homegfs [b7-gfsib02b] : No. of entries in split-brain: 0
>>> homegfs [b7-gfsib02b] : No. of heal failed entries : 0
>>>
>>>
>>> ========================================================================================
>>> I waited a few minutes for the heals to finish and ran
>>> the heal statistics and info again. one file is in
>>> split-brain. Aside from the split-brain, the load on all
>>> systems is down now and they are behaving normally.
>>> glustershd.log is attached. What is going on???
>>>
>>> Thu Jan 21 12:53:50 EST 2016
>>>
>>> =================== homegfs ===================
>>>
>>> homegfs [b0-gfsib01a] : Starting time of crawl : Thu Jan
>>> 21 12:53:02 2016
>>> homegfs [b0-gfsib01a] : Ending time of crawl : Thu Jan
>>> 21 12:53:02 2016
>>> homegfs [b0-gfsib01a] : Type of crawl: INDEX
>>> homegfs [b0-gfsib01a] : No. of entries healed : 0
>>> homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
>>> homegfs [b0-gfsib01a] : No. of heal failed entries : 0
>>>
>>> homegfs [b1-gfsib01b] : Starting time of crawl : Thu Jan
>>> 21 12:53:38 2016
>>> homegfs [b1-gfsib01b] : Ending time of crawl : Thu Jan
>>> 21 12:53:38 2016
>>> homegfs [b1-gfsib01b] : Type of crawl: INDEX
>>> homegfs [b1-gfsib01b] : No. of entries healed : 0
>>> homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
>>> homegfs [b1-gfsib01b] : No. of heal failed entries : 1
>>>
>>> homegfs [b2-gfsib01a] : Starting time of crawl : Thu Jan
>>> 21 12:53:04 2016
>>> homegfs [b2-gfsib01a] : Ending time of crawl : Thu Jan
>>> 21 12:53:04 2016
>>> homegfs [b2-gfsib01a] : Type of crawl: INDEX
>>> homegfs [b2-gfsib01a] : No. of entries healed : 0
>>> homegfs [b2-gfsib01a] : No. of entries in split-brain: 0
>>> homegfs [b2-gfsib01a] : No. of heal failed entries : 0
>>>
>>> homegfs [b3-gfsib01b] : Starting time of crawl : Thu Jan
>>> 21 12:53:04 2016
>>> homegfs [b3-gfsib01b] : Ending time of crawl : Thu Jan
>>> 21 12:53:04 2016
>>> homegfs [b3-gfsib01b] : Type of crawl: INDEX
>>> homegfs [b3-gfsib01b] : No. of entries healed : 0
>>> homegfs [b3-gfsib01b] : No. of entries in split-brain: 0
>>> homegfs [b3-gfsib01b] : No. of heal failed entries : 0
>>>
>>> homegfs [b4-gfsib02a] : Starting time of crawl : Thu Jan
>>> 21 12:53:33 2016
>>> homegfs [b4-gfsib02a] : Ending time of crawl : Thu Jan
>>> 21 12:53:33 2016
>>> homegfs [b4-gfsib02a] : Type of crawl: INDEX
>>> homegfs [b4-gfsib02a] : No. of entries healed : 0
>>> homegfs [b4-gfsib02a] : No. of entries in split-brain: 0
>>> homegfs [b4-gfsib02a] : No. of heal failed entries : 1
>>>
>>> homegfs [b5-gfsib02b] : Starting time of crawl : Thu Jan
>>> 21 12:53:14 2016
>>> homegfs [b5-gfsib02b] : Ending time of crawl : Thu Jan
>>> 21 12:53:15 2016
>>> homegfs [b5-gfsib02b] : Type of crawl: INDEX
>>> homegfs [b5-gfsib02b] : No. of entries healed : 0
>>> homegfs [b5-gfsib02b] : No. of entries in split-brain: 0
>>> homegfs [b5-gfsib02b] : No. of heal failed entries : 3
>>>
>>> homegfs [b6-gfsib02a] : Starting time of crawl : Thu Jan
>>> 21 12:53:04 2016
>>> homegfs [b6-gfsib02a] : Ending time of crawl : Thu Jan
>>> 21 12:53:04 2016
>>> homegfs [b6-gfsib02a] : Type of crawl: INDEX
>>> homegfs [b6-gfsib02a] : No. of entries healed : 0
>>> homegfs [b6-gfsib02a] : No. of entries in split-brain: 0
>>> homegfs [b6-gfsib02a] : No. of heal failed entries : 0
>>>
>>> homegfs [b7-gfsib02b] : Starting time of crawl : Thu Jan
>>> 21 12:53:09 2016
>>> homegfs [b7-gfsib02b] : Ending time of crawl : Thu Jan
>>> 21 12:53:09 2016
>>> homegfs [b7-gfsib02b] : Type of crawl: INDEX
>>> homegfs [b7-gfsib02b] : No. of entries healed : 0
>>> homegfs [b7-gfsib02b] : No. of entries in split-brain: 0
>>> homegfs [b7-gfsib02b] : No. of heal failed entries : 0
>>>
>>> *** gluster bug in 'gluster volume heal homegfs
>>> statistics' ***
>>> *** Use 'gluster volume heal homegfs info' until bug is
>>> fixed ***
>>>
>>> Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/
>>> Number of entries: 0
>>>
>>> Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/
>>> Number of entries: 0
>>>
>>> Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/
>>> Number of entries: 0
>>>
>>> Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/
>>> Number of entries: 0
>>>
>>> Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/
>>> /users/bangell/.gconfd - Is in split-brain
>>>
>>> Number of entries: 1
>>>
>>> Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/
>>> /users/bangell/.gconfd - Is in split-brain
>>>
>>> /users/bangell/.gconfd/saved_state
>>> Number of entries: 2
>>>
>>> Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/
>>> Number of entries: 0
>>>
>>> Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/
>>> Number of entries: 0
>>>
>>>
>>>
>>>
>>> On Thu, Jan 21, 2016 at 11:10 AM, Pranith Kumar
>>> Karampuri <pkarampu at redhat.com
>>> <mailto:pkarampu at redhat.com>> wrote:
>>>
>>>
>>>
>>> On 01/21/2016 09:26 PM, Glomski, Patrick wrote:
>>>> I should mention that the problem is not currently
>>>> occurring and there are no heals (output appended).
>>>> By restarting the gluster services, we can stop the
>>>> crawl, which lowers the load for a while.
>>>> Subsequent crawls seem to finish properly. For what
>>>> it's worth, files/folders that show up in the
>>>> 'volume info' output during a hung crawl don't seem
>>>> to be anything out of the ordinary.
>>>>
>>>> Over the past four days, the typical time before
>>>> the problem recurs after suppressing it in this
>>>> manner is an hour. Last night when we reached out
>>>> to you was the last time it happened and the load
>>>> has been low since (a relief). David believes that
>>>> recursively listing the files (ls -alR or similar)
>>>> from a client mount can force the issue to happen,
>>>> but obviously I'd rather not unless we have some
>>>> precise thing we're looking for. Let me know if
>>>> you'd like me to attempt to drive the system
>>>> unstable like that and what I should look for. As
>>>> it's a production system, I'd rather not leave it
>>>> in this state for long.
>>>
>>> Will it be possible to send glustershd, mount logs
>>> of the past 4 days? I would like to see if this is
>>> because of directory self-heal going wild (Ravi is
>>> working on throttling feature for 3.8, which will
>>> allow to put breaks on self-heal traffic)
>>>
>>> Pranith
>>>
>>>>
>>>> [root at gfs01a xattrop]# gluster volume heal homegfs info
>>>> Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/
>>>> Number of entries: 0
>>>>
>>>> Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/
>>>> Number of entries: 0
>>>>
>>>> Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/
>>>> Number of entries: 0
>>>>
>>>> Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/
>>>> Number of entries: 0
>>>>
>>>> Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/
>>>> Number of entries: 0
>>>>
>>>> Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/
>>>> Number of entries: 0
>>>>
>>>> Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/
>>>> Number of entries: 0
>>>>
>>>> Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/
>>>> Number of entries: 0
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jan 21, 2016 at 10:40 AM, Pranith Kumar
>>>> Karampuri <pkarampu at redhat.com
>>>> <mailto:pkarampu at redhat.com>> wrote:
>>>>
>>>>
>>>>
>>>> On 01/21/2016 08:25 PM, Glomski, Patrick wrote:
>>>>> Hello, Pranith. The typical behavior is that
>>>>> the %cpu on a glusterfsd process jumps to
>>>>> number of processor cores available (800% or
>>>>> 1200%, depending on the pair of nodes
>>>>> involved) and the load average on the machine
>>>>> goes very high (~20). The volume's heal
>>>>> statistics output shows that it is crawling
>>>>> one of the bricks and trying to heal, but this
>>>>> crawl hangs and never seems to finish.
>>>>>
>>>>> The number of files in the xattrop directory
>>>>> varies over time, so I ran a wc -l as you
>>>>> requested periodically for some time and then
>>>>> started including a datestamped list of the
>>>>> files that were in the xattrops directory on
>>>>> each brick to see which were persistent. All
>>>>> bricks had files in the xattrop folder, so all
>>>>> results are attached.
>>>> Thanks this info is helpful. I don't see a lot
>>>> of files. Could you give output of "gluster
>>>> volume heal <volname> info"? Is there any
>>>> directory in there which is LARGE?
>>>>
>>>> Pranith
>>>>
>>>>>
>>>>> Please let me know if there is anything else I
>>>>> can provide.
>>>>>
>>>>> Patrick
>>>>>
>>>>>
>>>>> On Thu, Jan 21, 2016 at 12:01 AM, Pranith
>>>>> Kumar Karampuri <pkarampu at redhat.com
>>>>> <mailto:pkarampu at redhat.com>> wrote:
>>>>>
>>>>> hey,
>>>>> Which process is consuming so much
>>>>> cpu? I went through the logs you gave me.
>>>>> I see that the following files are in gfid
>>>>> mismatch state:
>>>>>
>>>>> <066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,
>>>>> <1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,
>>>>> <ddc92637-303a-4059-9c56-ab23b1bb6ae9/patch0008.cnvrg>,
>>>>>
>>>>> Could you give me the output of "ls
>>>>> <brick-path>/indices/xattrop | wc -l"
>>>>> output on all the bricks which are acting
>>>>> this way? This will tell us the number of
>>>>> pending self-heals on the system.
>>>>>
>>>>> Pranith
>>>>>
>>>>>
>>>>> On 01/20/2016 09:26 PM, David Robinson wrote:
>>>>>> resending with parsed logs...
>>>>>>>> I am having issues with 3.6.6 where the
>>>>>>>> load will spike up to 800% for one of
>>>>>>>> the glusterfsd processes and the users
>>>>>>>> can no longer access the system. If I
>>>>>>>> reboot the node, the heal will finish
>>>>>>>> normally after a few minutes and the
>>>>>>>> system will be responsive, but a few
>>>>>>>> hours later the issue will start again.
>>>>>>>> It look like it is hanging in a heal
>>>>>>>> and spinning up the load on one of the
>>>>>>>> bricks. The heal gets stuck and says
>>>>>>>> it is crawling and never returns. After
>>>>>>>> a few minutes of the heal saying it is
>>>>>>>> crawling, the load spikes up and the
>>>>>>>> mounts become unresponsive.
>>>>>>>> Any suggestions on how to fix this? It
>>>>>>>> has us stopped cold as the user can no
>>>>>>>> longer access the systems when the load
>>>>>>>> spikes... Logs attached.
>>>>>>>> System setup info is:
>>>>>>>> [root at gfs01a ~]# gluster volume info
>>>>>>>> homegfs
>>>>>>>>
>>>>>>>> Volume Name: homegfs
>>>>>>>> Type: Distributed-Replicate
>>>>>>>> Volume ID:
>>>>>>>> 1e32672a-f1b7-4b58-ba94-58c085e59071
>>>>>>>> Status: Started
>>>>>>>> Number of Bricks: 4 x 2 = 8
>>>>>>>> Transport-type: tcp
>>>>>>>> Bricks:
>>>>>>>> Brick1:
>>>>>>>> gfsib01a.corvidtec.com:/data/brick01a/homegfs
>>>>>>>> Brick2:
>>>>>>>> gfsib01b.corvidtec.com:/data/brick01b/homegfs
>>>>>>>> Brick3:
>>>>>>>> gfsib01a.corvidtec.com:/data/brick02a/homegfs
>>>>>>>> Brick4:
>>>>>>>> gfsib01b.corvidtec.com:/data/brick02b/homegfs
>>>>>>>> Brick5:
>>>>>>>> gfsib02a.corvidtec.com:/data/brick01a/homegfs
>>>>>>>> Brick6:
>>>>>>>> gfsib02b.corvidtec.com:/data/brick01b/homegfs
>>>>>>>> Brick7:
>>>>>>>> gfsib02a.corvidtec.com:/data/brick02a/homegfs
>>>>>>>> Brick8:
>>>>>>>> gfsib02b.corvidtec.com:/data/brick02b/homegfs
>>>>>>>> Options Reconfigured:
>>>>>>>> performance.io-thread-count: 32
>>>>>>>> performance.cache-size: 128MB
>>>>>>>> performance.write-behind-window-size: 128MB
>>>>>>>> server.allow-insecure: on
>>>>>>>> network.ping-timeout: 42
>>>>>>>> storage.owner-gid: 100
>>>>>>>> geo-replication.indexing: off
>>>>>>>> geo-replication.ignore-pid-check: on
>>>>>>>> changelog.changelog: off
>>>>>>>> changelog.fsync-interval: 3
>>>>>>>> changelog.rollover-time: 15
>>>>>>>> server.manage-gids: on
>>>>>>>> diagnostics.client-log-level: WARNING
>>>>>>>> [root at gfs01a ~]# rpm -qa | grep gluster
>>>>>>>> gluster-nagios-common-0.1.1-0.el6.noarch
>>>>>>>> glusterfs-fuse-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-debuginfo-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-libs-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-geo-replication-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-api-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-devel-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-api-devel-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-cli-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-rdma-3.6.6-1.el6.x86_64
>>>>>>>> samba-vfs-glusterfs-4.1.11-2.el6.x86_64
>>>>>>>> glusterfs-server-3.6.6-1.el6.x86_64
>>>>>>>> glusterfs-extra-xlators-3.6.6-1.el6.x86_64
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-devel mailing list
>>>>>> Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> <mailto:Gluster-users at gluster.org>
>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160122/42006861/attachment-0001.html>
More information about the Gluster-devel
mailing list