[Gluster-devel] nodes don't use swap
Jordi Moles
jordi at cdmon.com
Thu Mar 13 12:36:51 UTC 2008
Hi,
i've just read through the whole log files from the last few days, when
i started these massive tests.
just to let you know in case i haven't done so before.... i'm running
two types of tests.
1. sending to the mail system several emails per second, postfixs just
keep delivering mails every second. A part from that, a script checks
several mailboxes via imap and pop3 every minute.
2. running some disk tests, like bonnie.
anyway... a part from that issue with only 1 cpu used and also running
out of ram memory... i was wondering if anything would be logged. And
actually it is.
The thing is that nodes haven't reported any error or warning at all in
days, nor have dovecots.
But postfixs keep logging this from time to time. It doesn't happen all
the time, but when it does, it keeps logging this for several minutes,
every second.
*************
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785301:
/dummymassiu/new/1205166062.V12I514f6M451321.postfix01gluster01.cdmon.com,S=1322
=> 333046 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785307:
/dummymassiu/new/1205169576.V12I528b6M828341.postfix01gluster01.cdmon.com,S=2113
=> 338102 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785308:
/dummymassiu/new/1205165435.V12I5115eM368089.postfix01gluster01.cdmon.com,S=1322
=> 332126 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785310:
/dummymassiu/new/1205163678.V12I50696M779037.postfix01gluster01.cdmon.com,S=2110
=> 329366 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785311:
/dummymassiu/new/1205166503.V12I517e8M525608.postfix02gluster01.cdmon.com,S=1321
=> 333800 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785312:
/dummymassiu/new/1205168060.V12I521beM376767.postfix01gluster01.cdmon.com,S=1323
=> 336318 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785313:
/dummymassiu/new/1205172223.V12I52adaM551011.postfix02gluster01.cdmon.com,S=1321
=> 338650 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785316:
/dummymassiu/new/1205163863.V12I507c2M877288.postfix01gluster01.cdmon.com,S=2113
=> 329666 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785320:
/dummymassiu/new/1205163951.V12I50862M599096.postfix01gluster01.cdmon.com,S=1323
=> 329826 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785322:
/dummymassiu/new/1205169551.V12I52898M543504.postfix03gluster01.cdmon.com,S=2110
=> 338072 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785324:
/dummymassiu/new/1205167646.V12I51f74M976792.postfix03gluster01.cdmon.com,S=2110
=> 335732 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785326:
/dummymassiu/new/1205167981.V12I52149M809051.postfix04gluster01.cdmon.com,S=1323
=> 336201 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785330:
/dummymassiu/new/1205166685.V12I51916M157626.postfix01gluster01.cdmon.com,S=2112
=> 334102 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785332:
/dummymassiu/new/1205164950.V12I50e5eM93799.postfix01gluster01.cdmon.com,S=2112
=> 331358 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785333:
/dummymassiu/new/1205164539.V12I50c5aM252840.postfix01gluster01.cdmon.com,S=2112
=> 330842 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785336:
/dummymassiu/new/1205166068.V12I51505M733823.postfix04gluster01.cdmon.com,S=1323
=> 333061 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785339:
/dummymassiu/new/1205164143.V12I5098dM237821.postfix04gluster01.cdmon.com,S=2113
=> 330125 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785340:
/dummymassiu/new/1205169554.V12I5289bM892195.postfix02gluster01.cdmon.com,S=2110
=> 338075 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785341:
/dummymassiu/new/1205169041.V12I52639M355241.postfix04gluster01.cdmon.com,S=1322
=> 337465 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785344:
/dummymassiu/new/1205168969.V12I525f9M212765.postfix01gluster01.cdmon.com,S=2112
=> 337401 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785345:
/dummymassiu/new/1205165036.V12I50f0bM828482.postfix03gluster01.cdmon.com,S=1321
=> 331531 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785347:
/dummymassiu/new/1205167618.V12I51f40M129843.postfix03gluster01.cdmon.com,S=2110
=> 335680 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785348:
/dummymassiu/new/1205165356.V12I510f0M733837.postfix02gluster01.cdmon.com,S=2111
=> 332016 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785349:
/dummymassiu/new/1205166260.V12I51656M140344.postfix01gluster01.cdmon.com,S=1323
=> 333398 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785350:
/dummymassiu/new/1205166740.V12I5197bM561623.postfix01gluster01.cdmon.com,S=1322
=> 334203 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785353:
/dummymassiu/new/1205165612.V12I51257M218424.postfix02gluster01.cdmon.com,S=1321
=> 332375 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785359:
/dummymassiu/new/1205166335.V12I516d7M906967.postfix03gluster01.cdmon.com,S=2110
=> 333527 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785360:
/dummymassiu/new/1205166706.V12I5193bM376041.postfix02gluster01.cdmon.com,S=2111
=> 334139 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785362:
/dummymassiu/new/1205167774.V12I52045M925044.postfix04gluster01.cdmon.com,S=1322
=> 335941 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785363:
/dummymassiu/new/1205163717.V12I506d3M585524.postfix01gluster01.cdmon.com,S=2111
=> 329427 Rehashing because st_nlink less than dentry maps
2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse:
2785366:
/dummymassiu/new/1205166212.V12I5160dM479271.postfix04gluster01.cdmon.com,S=1322
=> 333325 Rehashing because st_nlink less than dentry maps
*************
En/na Jordi Moles ha escrit:
> Hi,
>
> yes, sorry, that is a typo, i mean:
>
> dom0 is NOT running any server or client daemon.
>
> En/na Basavanagowda Kanur ha escrit:
>> Jordi,
>>
>> I have one more question, inline with your the message below.
>>
>> On Wed, Mar 12, 2008 at 10:25 PM, Jordi Moles <jordi at cdmon.com> wrote:
>>
>>
>>> Hi,
>>>
>>> I'm using Xen 3.2.0 on debian lenny machines, xen installed from
>>> apt-get
>>> repositories.
>>>
>>> All machines, nodes, postfixs and dovecots are virtuals.
>>>
>>> nodes have 2GB of ram and 4GB of swap, dovecots and postfixs 512M of
>>> ram
>>> and also 4GB of swap.
>>>
>>> dom0 is running any server or client gluster daemon.
>>>
>>
>>
>> Did you mean 'dom0 is not running any server or client glusterfs
>> daemon'?
>>
>>
>>
>>> I've attached a picture from cacti, which i use to monitor all parts of
>>> gluster. There are two points at which the node runs out of ram
>>> memory.
>>> The first one is when i perform a postmark test, and the second one
>>> is a
>>> bonnie test. The other readings which are close to 1GB refers to
>>> "normal" use of my mail system, receiving mails, people checking their
>>> mailboxes and so on.
>>>
>>> no errors or warnings are reported in the gluster log files.
>>>
>>> here's the xen config files from any kind of machine:
>>>
>>> NODES
>>>
>>> *************
>>>
>>> #
>>> # Configuration file for the Xen instance node01gluster01, created
>>> # by xen-tools 3.9 on Thu Feb 28 16:39:27 2008.
>>> #
>>>
>>> #
>>> # Kernel + memory size
>>> #
>>> kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64'
>>> ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64'
>>> memory = '2048'
>>> vcpus = '2'
>>>
>>> #
>>> # Disk device(s).
>>> #
>>> root = '/dev/sda2 ro'
>>> disk = [
>>>
>>> 'file:/home/xen/domains/node01gluster01/swap.img,sda1,w',
>>>
>>> 'file:/home/xen/domains/node01gluster01/disk.img,sda2,w',
>>> ]
>>>
>>>
>>> #
>>> # Hostname
>>> #
>>> name = 'node01gluster01'
>>>
>>> #
>>> # Networking
>>> #
>>> vif = [
>>> 'bridge=xenbr0,ip=192.168.1.204,mac=00:16:3E:B3:2F:18','bridge=xenbr1,ip=
>>>
>>> 192.168.2.204,mac=00:16:3E:B3:2F:19'
>>> ]
>>>
>>> #
>>> # Behaviour
>>> #
>>> on_poweroff = 'destroy'
>>> on_reboot = 'restart'
>>> on_crash = 'restart'
>>>
>>>
>>> *************
>>>
>>> DOVECOTS
>>>
>>> *************
>>>
>>>
>>> # Configuration file for the Xen instance dovecot02gluster01.cdmon.com,
>>> created
>>> # by xen-tools 3.9 on Thu Feb 28 16:51:32 2008.
>>> #
>>>
>>> #
>>> # Kernel + memory size
>>> #
>>> kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64'
>>> ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64'
>>> memory = '512'
>>>
>>> #
>>> # Disk device(s).
>>> #
>>> root = '/dev/sda2 ro'
>>> disk = [
>>>
>>> 'file:/home/xen/domains/dovecot02gluster01.cdmon.com/swap.img,sda1,w',
>>>
>>> 'file:/home/xen/domains/dovecot02gluster01.cdmon.com/disk.img,sda2,w',
>>> ]
>>>
>>>
>>> #
>>> # Hostname
>>> #
>>> name = 'dovecot02gluster01.cdmon.com'
>>>
>>> #
>>> # Networking
>>> #
>>> vif = [
>>> 'bridge=xenbr0,ip=212.x.x.x,mac=00:16:3E:B2:E7:69','bridge=xenbr1,ip=
>>> 192.168.1.36,mac=00:16:3E:B2:E7:70'
>>> ]
>>>
>>> #
>>> # Behaviour
>>> #
>>> on_poweroff = 'destroy'
>>> on_reboot = 'restart'
>>> on_crash = 'restart'
>>>
>>>
>>> **************
>>>
>>> POSTFIXS
>>>
>>> **************
>>>
>>> #
>>> # Configuration file for the Xen instance postfix02gluster01.cdmon.com,
>>> created
>>> # by xen-tools 3.9 on Thu Feb 28 16:40:20 2008.
>>> #
>>>
>>> #
>>> # Kernel + memory size
>>> #
>>> kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64'
>>> ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64'
>>> memory = '512'
>>>
>>> #
>>> # Disk device(s).
>>> #
>>> root = '/dev/sda2 ro'
>>> disk = [
>>>
>>> 'file:/home/xen/domains/postfix02gluster01.cdmon.com/swap.img,sda1,w',
>>>
>>> 'file:/home/xen/domains/postfix02gluster01.cdmon.com/disk.img,sda2,w',
>>> ]
>>>
>>>
>>> #
>>> # Hostname
>>> #
>>> name = 'postfix02gluster01.cdmon.com'
>>>
>>> #
>>> # Networking
>>> #
>>> vif = [
>>> 'bridge=xenbr0,ip=212.x.x.x,mac=00:16:3E:F9:47:0D','bridge=xenbr1,ip=
>>> 192.168.1.212,mac=00:16:3E:F9:47:0E'
>>> ]
>>>
>>> #
>>> # Behaviour
>>> #
>>> on_poweroff = 'destroy'
>>> on_reboot = 'restart'
>>> on_crash = 'restart'
>>>
>>>
>>>
>>> **************
>>>
>>> En/na Basavanagowda Kanur ha escrit:
>>>
>>>> Jordi,
>>>> We will be trying to reproduce your problem at our setup. To go
>>>>
>>> further,
>>>
>>>> we have few questions for you:
>>>>
>>>> 1. Where are glusterfs servers running? On virtual machines or real
>>>> machines?
>>>> 2. Where are glusterfs clients running? On virtual machines or real
>>>> machines?
>>>> 3. Is the Xen dom-0 also using glusterfs client? If yes, what all
>>>>
>>> processes
>>>
>>>> are accessing the glusterfs mount.
>>>> 4. If Xen dom-0 hosts the block device images for Xen guest OSes on
>>>> glusterfs mount, then what is the access method for guest OSes?
>>>> loopback
>>>> (disk=['file:']) or blktp (disk=['tap:aio:']) or nfs rexport?
>>>>
>>>> Detailed description of your Xen setup in addition to answers to above
>>>> questions will help us debug the problem.
>>>>
>>>> --
>>>> Gowda
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
More information about the Gluster-devel
mailing list