[Gluster-devel] nodes don't use swap
Jordi Moles
jordi at cdmon.com
Mon Mar 31 07:06:09 UTC 2008
hi,
did you guys manage to reproduce the issues i'm facing?
do you need any extra information?
thanks.
En/na Jordi Moles ha escrit:
> Hi,
>
> i've just read through the whole log files from the last few days,
> when i started these massive tests.
>
> just to let you know in case i haven't done so before.... i'm running
> two types of tests.
>
> 1. sending to the mail system several emails per second, postfixs just
> keep delivering mails every second. A part from that, a script checks
> several mailboxes via imap and pop3 every minute.
> 2. running some disk tests, like bonnie.
>
> anyway... a part from that issue with only 1 cpu used and also running
> out of ram memory... i was wondering if anything would be logged. And
> actually it is.
> The thing is that nodes haven't reported any error or warning at all
> in days, nor have dovecots.
>
> But postfixs keep logging this from time to time. It doesn't happen
> all the time, but when it does, it keeps logging this for several
> minutes, every second.
>
> *************
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785301:
> /dummymassiu/new/1205166062.V12I514f6M451321.postfix01gluster01.cdmon.com,S=1322
> => 333046 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785307:
> /dummymassiu/new/1205169576.V12I528b6M828341.postfix01gluster01.cdmon.com,S=2113
> => 338102 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785308:
> /dummymassiu/new/1205165435.V12I5115eM368089.postfix01gluster01.cdmon.com,S=1322
> => 332126 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785310:
> /dummymassiu/new/1205163678.V12I50696M779037.postfix01gluster01.cdmon.com,S=2110
> => 329366 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785311:
> /dummymassiu/new/1205166503.V12I517e8M525608.postfix02gluster01.cdmon.com,S=1321
> => 333800 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785312:
> /dummymassiu/new/1205168060.V12I521beM376767.postfix01gluster01.cdmon.com,S=1323
> => 336318 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785313:
> /dummymassiu/new/1205172223.V12I52adaM551011.postfix02gluster01.cdmon.com,S=1321
> => 338650 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785316:
> /dummymassiu/new/1205163863.V12I507c2M877288.postfix01gluster01.cdmon.com,S=2113
> => 329666 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785320:
> /dummymassiu/new/1205163951.V12I50862M599096.postfix01gluster01.cdmon.com,S=1323
> => 329826 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785322:
> /dummymassiu/new/1205169551.V12I52898M543504.postfix03gluster01.cdmon.com,S=2110
> => 338072 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785324:
> /dummymassiu/new/1205167646.V12I51f74M976792.postfix03gluster01.cdmon.com,S=2110
> => 335732 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785326:
> /dummymassiu/new/1205167981.V12I52149M809051.postfix04gluster01.cdmon.com,S=1323
> => 336201 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785330:
> /dummymassiu/new/1205166685.V12I51916M157626.postfix01gluster01.cdmon.com,S=2112
> => 334102 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785332:
> /dummymassiu/new/1205164950.V12I50e5eM93799.postfix01gluster01.cdmon.com,S=2112
> => 331358 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785333:
> /dummymassiu/new/1205164539.V12I50c5aM252840.postfix01gluster01.cdmon.com,S=2112
> => 330842 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785336:
> /dummymassiu/new/1205166068.V12I51505M733823.postfix04gluster01.cdmon.com,S=1323
> => 333061 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785339:
> /dummymassiu/new/1205164143.V12I5098dM237821.postfix04gluster01.cdmon.com,S=2113
> => 330125 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785340:
> /dummymassiu/new/1205169554.V12I5289bM892195.postfix02gluster01.cdmon.com,S=2110
> => 338075 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785341:
> /dummymassiu/new/1205169041.V12I52639M355241.postfix04gluster01.cdmon.com,S=1322
> => 337465 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785344:
> /dummymassiu/new/1205168969.V12I525f9M212765.postfix01gluster01.cdmon.com,S=2112
> => 337401 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785345:
> /dummymassiu/new/1205165036.V12I50f0bM828482.postfix03gluster01.cdmon.com,S=1321
> => 331531 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785347:
> /dummymassiu/new/1205167618.V12I51f40M129843.postfix03gluster01.cdmon.com,S=2110
> => 335680 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785348:
> /dummymassiu/new/1205165356.V12I510f0M733837.postfix02gluster01.cdmon.com,S=2111
> => 332016 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785349:
> /dummymassiu/new/1205166260.V12I51656M140344.postfix01gluster01.cdmon.com,S=1323
> => 333398 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785350:
> /dummymassiu/new/1205166740.V12I5197bM561623.postfix01gluster01.cdmon.com,S=1322
> => 334203 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785353:
> /dummymassiu/new/1205165612.V12I51257M218424.postfix02gluster01.cdmon.com,S=1321
> => 332375 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785359:
> /dummymassiu/new/1205166335.V12I516d7M906967.postfix03gluster01.cdmon.com,S=2110
> => 333527 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785360:
> /dummymassiu/new/1205166706.V12I5193bM376041.postfix02gluster01.cdmon.com,S=2111
> => 334139 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785362:
> /dummymassiu/new/1205167774.V12I52045M925044.postfix04gluster01.cdmon.com,S=1322
> => 335941 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785363:
> /dummymassiu/new/1205163717.V12I506d3M585524.postfix01gluster01.cdmon.com,S=2111
> => 329427 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk]
> glusterfs-fuse: 2785366:
> /dummymassiu/new/1205166212.V12I5160dM479271.postfix04gluster01.cdmon.com,S=1322
> => 333325 Rehashing because st_nlink less than dentry maps
> *************
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> En/na Jordi Moles ha escrit:
>> Hi,
>>
>> yes, sorry, that is a typo, i mean:
>>
>> dom0 is NOT running any server or client daemon.
>>
>> En/na Basavanagowda Kanur ha escrit:
>>> Jordi,
>>>
>>> I have one more question, inline with your the message below.
>>>
>>> On Wed, Mar 12, 2008 at 10:25 PM, Jordi Moles <jordi at cdmon.com> wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> I'm using Xen 3.2.0 on debian lenny machines, xen installed from
>>>> apt-get
>>>> repositories.
>>>>
>>>> All machines, nodes, postfixs and dovecots are virtuals.
>>>>
>>>> nodes have 2GB of ram and 4GB of swap, dovecots and postfixs 512M
>>>> of ram
>>>> and also 4GB of swap.
>>>>
>>>> dom0 is running any server or client gluster daemon.
>>>>
>>>
>>>
>>> Did you mean 'dom0 is not running any server or client glusterfs
>>> daemon'?
>>>
>>>
>>>
>>>> I've attached a picture from cacti, which i use to monitor all
>>>> parts of
>>>> gluster. There are two points at which the node runs out of ram
>>>> memory.
>>>> The first one is when i perform a postmark test, and the second one
>>>> is a
>>>> bonnie test. The other readings which are close to 1GB refers to
>>>> "normal" use of my mail system, receiving mails, people checking their
>>>> mailboxes and so on.
>>>>
>>>> no errors or warnings are reported in the gluster log files.
>>>>
>>>> here's the xen config files from any kind of machine:
>>>>
>>>> NODES
>>>>
>>>> *************
>>>>
>>>> #
>>>> # Configuration file for the Xen instance node01gluster01, created
>>>> # by xen-tools 3.9 on Thu Feb 28 16:39:27 2008.
>>>> #
>>>>
>>>> #
>>>> # Kernel + memory size
>>>> #
>>>> kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64'
>>>> ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64'
>>>> memory = '2048'
>>>> vcpus = '2'
>>>>
>>>> #
>>>> # Disk device(s).
>>>> #
>>>> root = '/dev/sda2 ro'
>>>> disk = [
>>>>
>>>> 'file:/home/xen/domains/node01gluster01/swap.img,sda1,w',
>>>>
>>>> 'file:/home/xen/domains/node01gluster01/disk.img,sda2,w',
>>>> ]
>>>>
>>>>
>>>> #
>>>> # Hostname
>>>> #
>>>> name = 'node01gluster01'
>>>>
>>>> #
>>>> # Networking
>>>> #
>>>> vif = [
>>>> 'bridge=xenbr0,ip=192.168.1.204,mac=00:16:3E:B3:2F:18','bridge=xenbr1,ip=
>>>>
>>>> 192.168.2.204,mac=00:16:3E:B3:2F:19'
>>>> ]
>>>>
>>>> #
>>>> # Behaviour
>>>> #
>>>> on_poweroff = 'destroy'
>>>> on_reboot = 'restart'
>>>> on_crash = 'restart'
>>>>
>>>>
>>>> *************
>>>>
>>>> DOVECOTS
>>>>
>>>> *************
>>>>
>>>>
>>>> # Configuration file for the Xen instance
>>>> dovecot02gluster01.cdmon.com,
>>>> created
>>>> # by xen-tools 3.9 on Thu Feb 28 16:51:32 2008.
>>>> #
>>>>
>>>> #
>>>> # Kernel + memory size
>>>> #
>>>> kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64'
>>>> ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64'
>>>> memory = '512'
>>>>
>>>> #
>>>> # Disk device(s).
>>>> #
>>>> root = '/dev/sda2 ro'
>>>> disk = [
>>>>
>>>> 'file:/home/xen/domains/dovecot02gluster01.cdmon.com/swap.img,sda1,w',
>>>>
>>>> 'file:/home/xen/domains/dovecot02gluster01.cdmon.com/disk.img,sda2,w',
>>>> ]
>>>>
>>>>
>>>> #
>>>> # Hostname
>>>> #
>>>> name = 'dovecot02gluster01.cdmon.com'
>>>>
>>>> #
>>>> # Networking
>>>> #
>>>> vif = [
>>>> 'bridge=xenbr0,ip=212.x.x.x,mac=00:16:3E:B2:E7:69','bridge=xenbr1,ip=
>>>> 192.168.1.36,mac=00:16:3E:B2:E7:70'
>>>> ]
>>>>
>>>> #
>>>> # Behaviour
>>>> #
>>>> on_poweroff = 'destroy'
>>>> on_reboot = 'restart'
>>>> on_crash = 'restart'
>>>>
>>>>
>>>> **************
>>>>
>>>> POSTFIXS
>>>>
>>>> **************
>>>>
>>>> #
>>>> # Configuration file for the Xen instance
>>>> postfix02gluster01.cdmon.com,
>>>> created
>>>> # by xen-tools 3.9 on Thu Feb 28 16:40:20 2008.
>>>> #
>>>>
>>>> #
>>>> # Kernel + memory size
>>>> #
>>>> kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64'
>>>> ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64'
>>>> memory = '512'
>>>>
>>>> #
>>>> # Disk device(s).
>>>> #
>>>> root = '/dev/sda2 ro'
>>>> disk = [
>>>>
>>>> 'file:/home/xen/domains/postfix02gluster01.cdmon.com/swap.img,sda1,w',
>>>>
>>>> 'file:/home/xen/domains/postfix02gluster01.cdmon.com/disk.img,sda2,w',
>>>> ]
>>>>
>>>>
>>>> #
>>>> # Hostname
>>>> #
>>>> name = 'postfix02gluster01.cdmon.com'
>>>>
>>>> #
>>>> # Networking
>>>> #
>>>> vif = [
>>>> 'bridge=xenbr0,ip=212.x.x.x,mac=00:16:3E:F9:47:0D','bridge=xenbr1,ip=
>>>> 192.168.1.212,mac=00:16:3E:F9:47:0E'
>>>> ]
>>>>
>>>> #
>>>> # Behaviour
>>>> #
>>>> on_poweroff = 'destroy'
>>>> on_reboot = 'restart'
>>>> on_crash = 'restart'
>>>>
>>>>
>>>>
>>>> **************
>>>>
>>>> En/na Basavanagowda Kanur ha escrit:
>>>>
>>>>> Jordi,
>>>>> We will be trying to reproduce your problem at our setup. To go
>>>>>
>>>> further,
>>>>
>>>>> we have few questions for you:
>>>>>
>>>>> 1. Where are glusterfs servers running? On virtual machines or real
>>>>> machines?
>>>>> 2. Where are glusterfs clients running? On virtual machines or real
>>>>> machines?
>>>>> 3. Is the Xen dom-0 also using glusterfs client? If yes, what all
>>>>>
>>>> processes
>>>>
>>>>> are accessing the glusterfs mount.
>>>>> 4. If Xen dom-0 hosts the block device images for Xen guest OSes on
>>>>> glusterfs mount, then what is the access method for guest OSes?
>>>>> loopback
>>>>> (disk=['file:']) or blktp (disk=['tap:aio:']) or nfs rexport?
>>>>>
>>>>> Detailed description of your Xen setup in addition to answers to
>>>>> above
>>>>> questions will help us debug the problem.
>>>>>
>>>>> --
>>>>> Gowda
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
More information about the Gluster-devel
mailing list