[Gluster-users] hanging "df" (3.1, infiniband)

[Lana Deere]

OK, I tried on one of the servers to mount onto itself.  It gives me a
"Transport endpoint is not connected" message when I try to run "df".

[root at storage0 ~]# mount -t glusterfs -o transport=rdma
storage0:/RaidData /mnt/RaidData
[root at storage0 ~]# mount
/dev/md1 on / type ext3 (rw)
[...]
glusterfs#storage0:/RaidData on /mnt/RaidData type fuse
(rw,allow_other,default_permissions,max_read=131072)
[root at storage0 ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md1             149308588   5671116 135930696   5% /
/dev/md0                101018     18010     77792  19% /boot
tmpfs                 12338356         0  12338356   0% /dev/shm
/dev/md3             2930204288      5536 2930198752   1% /data
df: `/mnt/RaidData': Transport endpoint is not connected

The same thing happens if I do the same commands on storage1.

If I do service iptables status, it tells me
  # service iptables status
  Firewall is stopped.

The "Transport" message sounds like the Infiniband might not be
configured the way GlusterFS expects.  Do I need to turn on the IPoIB
and mount via the IB address instead?


.. Lana (lana.deere at gmail.com)






On Tue, Oct 19, 2010 at 4:00 PM, Craig Carl <craig at gluster.com> wrote:
> Lana -
>    Can you try to mount locally on one of the servers, if that works please
> make sure that iptables is turned off on the client or that you have these
> chains in your firewall config -
>
> http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Installing_GlusterFS_on_Red_Hat_Package_Manager_%28RPM%29_Distributions
>
> Thanks,
>
> Craig
>
> --
> Craig Carl
> Senior Systems Engineer; Gluster, Inc.
> Cell - (408) 829-9953 (California, USA)
> Office - (408) 770-1884
> Gtalk - craig.carl at gmail.com
> Twitter - @gluster
> Installing Gluster Storage Platform, the movie!
> http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/
>
>
> ________________________________
> From: "Lana Deere" <lana.deere at gmail.com>
> To: "Craig Carl" <craig at gluster.com>
> Cc: gluster-users at gluster.org
> Sent: Tuesday, October 19, 2010 12:45:23 PM
> Subject: Re: [Gluster-users] hanging "df" (3.1, infiniband)
>
> It's CentOS 5.4 (the Rocks variant, technically).  I have only tried
> installing the client software on one node so far, figuring that once
> I had it working and knew how I got to that state I could then install
> onto the rest.
>
> I did install all three of glusterfs-core, -rdma, -fuse RPMs on that node.
>
> .. Lana (lana.deere at gmail.com)
>
>
>
>
>
>
> On Tue, Oct 19, 2010 at 3:37 PM, Craig Carl <craig at gluster.com> wrote:
>> Lana -
>>    I wanted to make sure you had installed the glusterfs-core, rdma, and
>> fuse rpms if you are on CentOS, Fedora or RHEL on every client?
>>
>> Thanks.
>>
>> --
>> Craig Carl
>> Gluster, Inc.
>> Cell - (408) 829-9953 (California, USA)
>> Gtalk - craig.carl at gmail.com
>>
>>
>> ________________________________
>> From: "Lana Deere" <lana.deere at gmail.com>
>> To: gluster-users at gluster.org
>> Sent: Tuesday, October 19, 2010 8:59:32 AM
>> Subject: [Gluster-users] hanging "df" (3.1, infiniband)
>>
>> I tried to install 3.1 on some machines but I have run into trouble.
>>
>> I have 4 nodes I'm using for storage in a distributed configuration.
>>     # gluster volume info
>>
>>     Volume Name: RaidData
>>     Type: Distribute
>>     Status: Started
>>     Number of Bricks: 4
>>     Transport-type: rdma
>>     Bricks:
>>     Brick1: storage0:/data
>>     Brick2: storage1:/data
>>     Brick3: storage2:/data
>>     Brick4: storage3:/data
>>
>> When I try to mount the volume on another machine,
>>   A. If I forget to modprobe fuse first, the mount command reports no
>> errors
>>      but no volume is listed in the mount table.  (I.e., "mount -t..."
>> seems
>>      to work, but "mount" doesn't show any volumes.)
>>   B. If I remember to modprobe fuse first, again the mount command reports
>>      no errors and this time I can see the volume in the mount table.
>>         # mount
>>         /dev/sda1 on / type ext3 (rw)
>>         [...]
>>         glusterfs#storage0:/RaidData on /mnt/RaidData type fuse
>> (rw,allow_other,default_permissions,max_read=131072)
>>     However, if I do a "df" or try to "cd" to that volume, the command
>> hangs without
>>     printing any messages.
>>
>>     I wasn't sure if I needed to specify "-o transport=rdma" on the
>> mount command
>>     so I tried it both ways, but with no apparent difference.
>>
>> Anyone have suggestions for things to try?  Do I need to have IPoIB
>> running on all the nodes?
>>
>> Thanks!
>>
>>
>> .. Lana (lana.deere at gmail.com)
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>