[Gluster-users] hanging "df" (3.1, infiniband)
Craig Carl
craig at gluster.com
Tue Oct 19 23:15:05 UTC 2010
Lana -
Looks like you have the IPoIB stack installed, but not support for ibverbs. Let's try this -
# yum install libibverbs
# service glusterd restart
Thanks,
Craig
--
Craig Carl
Senior Systems Engineer; Gluster, Inc.
Cell - ( 408) 829-9953 (California, USA)
Office - ( 408) 770-1884
Gtalk - craig.carl at gmail.com
Twitter - @gluster
Installing Gluster Storage Platform, the movie!
http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/
From: "Lana Deere" <lana.deere at gmail.com>
To: "Craig Carl" <craig at gluster.com>
Cc: gluster-users at gluster.org, landman at scalableinformatics.com
Sent: Tuesday, October 19, 2010 4:02:11 PM
Subject: Re: [Gluster-users] hanging "df" (3.1, infiniband)
They show up in ibhosts and I can ping or ssh via IPoIB to them, but
perhaps they are not completely configured properly. Or perhaps I
have mixed some references to the regular Ethernet into the
configuration for rdma? Anyway, here are the outputs you requested:
[root at storage0 ~]# lsmod
Module Size Used by
iptable_filter 36161 0
ip_tables 55201 1 iptable_filter
x_tables 50505 1 ip_tables
fuse 83057 1
autofs4 63049 3
hidp 83521 2
rfcomm 104937 0
l2cap 89409 10 hidp,rfcomm
bluetooth 118853 5 hidp,rfcomm,l2cap
lockd 101553 0
sunrpc 199945 2 lockd
cpufreq_ondemand 42449 8
acpi_cpufreq 47937 0
freq_table 38977 2 cpufreq_ondemand,acpi_cpufreq
ib_iser 69569 0
libiscsi2 77765 1 ib_iser
scsi_transport_iscsi2 74073 2 ib_iser,libiscsi2
scsi_transport_iscsi 35017 1 scsi_transport_iscsi2
ib_srp 67465 0
rds 401393 0
ib_sdp 144285 0
ib_ipoib 113057 0
ipoib_helper 35537 2 ib_ipoib
ipv6 435489 77 ib_ipoib
xfrm_nalgo 43333 1 ipv6
crypto_api 42945 1 xfrm_nalgo
rdma_ucm 47681 0
rdma_cm 68437 4 ib_iser,rds,ib_sdp,rdma_ucm
ib_ucm 50121 0
ib_uverbs 68720 2 rdma_ucm,ib_ucm
ib_umad 50153 0
ib_cm 72809 4 ib_srp,ib_ipoib,rdma_cm,ib_ucm
iw_cm 43465 1 rdma_cm
ib_addr 41929 1 rdma_cm
ib_sa 74953 4 ib_srp,ib_ipoib,rdma_cm,ib_cm
mlx4_ib 94461 0
ib_mad 70629 4 ib_umad,ib_cm,ib_sa,mlx4_ib
ib_core 104901 15
ib_iser,ib_srp,rds,ib_sdp,ib_ipoib,rdma_ucm,rdma_cm,ib_ucm,ib_uverbs,ib_umad,ib_cm,iw_cm,ib_sa,mlx4_ib,ib_mad
xfs 508625 1
loop 48721 0
dm_mirror 54737 0
dm_multipath 56921 0
scsi_dh 42177 1 dm_multipath
raid456 152417 1
xor 38865 1 raid456
video 53197 0
backlight 39873 1 video
sbs 49921 0
power_meter 47053 0
hwmon 36553 1 power_meter
i2c_ec 38593 1 sbs
dell_wmi 37601 0
wmi 41985 1 dell_wmi
button 40545 0
battery 43849 0
asus_acpi 50917 0
acpi_memhotplug 40517 0
ac 38729 0
parport_pc 62313 0
lp 47121 0
parport 73165 2 parport_pc,lp
mlx4_en 107985 0
joydev 43969 0
i2c_i801 41813 0
igb 122709 0
i2c_core 56641 2 i2c_ec,i2c_i801
8021q 57425 1 igb
shpchp 70893 0
mlx4_core 152773 2 mlx4_ib,mlx4_en
serio_raw 40517 0
dca 41221 1 igb
sg 70377 0
pcspkr 36289 0
dm_raid45 99657 0
dm_message 36289 1 dm_raid45
dm_region_hash 46145 1 dm_raid45
dm_log 44993 3 dm_mirror,dm_raid45,dm_region_hash
dm_mod 101649 4 dm_mirror,dm_multipath,dm_raid45,dm_log
dm_mem_cache 38977 1 dm_raid45
mpt2sas 159337 12
scsi_transport_sas 66753 1 mpt2sas
ahci 69705 6
libata 209489 1 ahci
sd_mod 56513 32
scsi_mod 196953 10
ib_iser,libiscsi2,scsi_transport_iscsi2,ib_srp,scsi_dh,sg,mpt2sas,scsi_transport_sas,libata,sd_mod
raid1 56001 3
ext3 168913 2
jbd 94769 1 ext3
uhci_hcd 57433 0
ohci_hcd 56309 0
ehci_hcd 66125 0
[root at storage0 ~]# ibv_devinfo
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
No IB devices found
[root at storage0 ~]# lspci
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
Express Root Port 3 (rev 22)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express
Root Port 5 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
Express Root Port 7 (rev 22)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI
Express Root Port 9 (rev 22)
00:13.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC
Interrupt Controller (rev 22)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management
Registers (rev 22)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch
Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status
and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset
QuickData Technology Device (rev 22)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset
QuickData Technology Device (rev 22)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset
QuickData Technology Device (rev 22)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset
QuickData Technology Device (rev 22)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset
QuickData Technology Device (rev 22)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset
QuickData Technology Device (rev 22)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset
QuickData Technology Device (rev 22)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset
QuickData Technology Device (rev 22)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2
EHCI Controller #2
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB
UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2
EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA
AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network
Connection (rev 01)
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
05:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe
2.0 5GT/s - IB QDR / 10GigE] (rev b0)
06:01.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW
WPCM450 (rev 0a)
[root at storage0 ~]# /etc/init.d/openibd status
Low level hardware support loaded:
mlx4_ib
Upper layer protocol modules:
ib_iser ib_srp rds ib_sdp ib_ipoib
User space access modules:
rdma_ucm ib_ucm ib_uverbs ib_umad
Connection management modules:
rdma_cm ib_cm iw_cm
Configured IPoIB interfaces: ib0
Currently active IPoIB interfaces: ib0
[root at storage0 ~]#
.. Lana (lana.deere at gmail.com)
On Tue, Oct 19, 2010 at 6:48 PM, Craig Carl <craig at gluster.com> wrote:
> Lana -
> The first couple of lines of the log identify our problem -
>
> [2010-10-19 07:47:49.315416] C [rdma.c:3817:rdma_init] rpc-transport/rdma:
> No IB devices found
> [2010-10-19 07:47:49.315438] E [rdma.c:4744:init] rdma.management: Failed to
> initialize IB Device
> [2010-10-19 07:47:49.315452] E [rpc-transport.c:965:rpc_transport_load]
> rpc-transport: 'rdma' initialization failed
>
> Are you sure your IB cards are working? Can you send the output of -
>
> # lsmod
> # ibv_devinfo
> # lspci
> # /etc/init.d/openibd status
>
>
>
> Thanks,
>
> Craig
>
> --
> Craig Carl
> Senior Systems Engineer; Gluster, Inc.
> Cell - (408) 829-9953 (California, USA)
> Office - (408) 770-1884
> Gtalk - craig.carl at gmail.com
> Twitter - @gluster
> Installing Gluster Storage Platform, the movie!
> http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/
>
>
> ________________________________
> From: "Lana Deere" <lana.deere at gmail.com>
> To: "Craig Carl" <craig at gluster.com>
> Cc: gluster-users at gluster.org, landman at scalableinformatics.com
> Sent: Tuesday, October 19, 2010 3:29:41 PM
> Subject: Re: [Gluster-users] hanging "df" (3.1, infiniband)
>
> For the last little while I've been using storage0 as both client and
> server, so those files are both client and server files at the same
> time. If it would be helpful, I could go back to using a different
> host as client (but then 'df' will hang instead of reporting the
> Transport message).
>
> [root at storage0 ~]# cat /etc/glusterd/.cmd_log_history
> [2010-10-19 07:54:36.244333] peer probe : on host storage1:24007
> [2010-10-19 07:54:36.249891] peer probe : on host storage1:24007 FAILED
> [2010-10-19 07:54:43.745558] peer probe : on host storage2:24007
> [2010-10-19 07:54:43.750752] peer probe : on host storage2:24007 FAILED
> [2010-10-19 07:54:48.915378] peer probe : on host storage3:24007
> [2010-10-19 07:54:48.920595] peer probe : on host storage3:24007 FAILED
> [2010-10-19 07:59:49.737251] Volume create : on volname: RaidData attempted
> [2010-10-19 07:59:49.737314] Volume create : on volname: RaidData
> type:DEFAULT count:4 bricks: storage0:/data storage1:/data
> storage2:/data storage3:/data
> [2010-10-19 07:59:49.737631] Volume create : on volname: RaidData SUCCESS
> [2010-10-19 08:01:36.909963] volume start : on volname: RaidData SUCCESS
>
> The /var/log file was pretty big, so I put it on pastebin:
> http://pastebin.com/m6WbHPUp
>
>
> .. Lana (lana.deere at gmail.com)
>
>
>
>
>
>
> On Tue, Oct 19, 2010 at 6:10 PM, Craig Carl <craig at gluster.com> wrote:
>> Lana -
>> Can you also post the contents of
>>
>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
>> and
>> /etc/glusterd/.cmd_log_history
>>
>> on both the client and server to the list?
>
More information about the Gluster-users
mailing list