[Gluster-users] hanging "df" (3.1, infiniband)

[Craig Carl]

Lana - 
The first couple of lines of the log identify our problem - 


    1. 
[2010-10-19 07:47:49.315416] C [rdma.c:3817:rdma_init] rpc-transport/rdma: No IB devices found 
    2. 
[2010-10-19 07:47:49.315438] E [rdma.c:4744:init] rdma.management: Failed to initialize IB Device 
    3. 
[2010-10-19 07:47:49.315452] E [rpc-transport.c:965:rpc_transport_load] rpc-transport: 'rdma' initialization failed 

Are you sure your IB cards are working? Can you send the output of - 

# lsmod 
# ibv_devinfo 
# lspci 
# /etc/init.d/openibd status 





Thanks, 

Craig 

-- 
Craig Carl 













Senior Systems Engineer; Gluster, Inc. 
Cell - ( 408) 829-9953 (California, USA) 
Office - ( 408) 770-1884 
Gtalk - craig.carl at gmail.com 
Twitter - @gluster 
Installing Gluster Storage Platform, the movie! 
http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ 



From: "Lana Deere" <lana.deere at gmail.com> 
To: "Craig Carl" <craig at gluster.com> 
Cc: gluster-users at gluster.org, landman at scalableinformatics.com 
Sent: Tuesday, October 19, 2010 3:29:41 PM 
Subject: Re: [Gluster-users] hanging "df" (3.1, infiniband) 

For the last little while I've been using storage0 as both client and 
server, so those files are both client and server files at the same 
time. If it would be helpful, I could go back to using a different 
host as client (but then 'df' will hang instead of reporting the 
Transport message). 

[root at storage0 ~]# cat /etc/glusterd/.cmd_log_history 
[2010-10-19 07:54:36.244333] peer probe : on host storage1:24007 
[2010-10-19 07:54:36.249891] peer probe : on host storage1:24007 FAILED 
[2010-10-19 07:54:43.745558] peer probe : on host storage2:24007 
[2010-10-19 07:54:43.750752] peer probe : on host storage2:24007 FAILED 
[2010-10-19 07:54:48.915378] peer probe : on host storage3:24007 
[2010-10-19 07:54:48.920595] peer probe : on host storage3:24007 FAILED 
[2010-10-19 07:59:49.737251] Volume create : on volname: RaidData attempted 
[2010-10-19 07:59:49.737314] Volume create : on volname: RaidData 
type:DEFAULT count:4 bricks: storage0:/data storage1:/data 
storage2:/data storage3:/data 
[2010-10-19 07:59:49.737631] Volume create : on volname: RaidData SUCCESS 
[2010-10-19 08:01:36.909963] volume start : on volname: RaidData SUCCESS 

The /var/log file was pretty big, so I put it on pastebin: 
http://pastebin.com/m6WbHPUp 


.. Lana (lana.deere at gmail.com) 






On Tue, Oct 19, 2010 at 6:10 PM, Craig Carl <craig at gluster.com> wrote: 
> Lana - 
> Can you also post the contents of 
> 
> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log 
> and 
> /etc/glusterd/.cmd_log_history 
> 
> on both the client and server to the list?