[Gluster-users] help me understand this matrix log.
Khoi Mai
KHOIMAI at UP.COM
Wed Feb 19 12:22:09 UTC 2014
Can someone please help me interpret this from my client: I'm not fully
understanding the matrix. I have a 4 server 2x2 distri.-rep brick.
[2014-02-19 12:15:15.921993] E
[afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
0-devstatic-replicate-0: Unable to self-heal contents of
'/employees/htdocs/software/contact_mstr/secure/app/views/.svn/props'
(possible split-brain). Please delete the file from all but the preferred
subvolume.- Pending matrix: [ [ 0 2 ] [ 3 0 ] ]
Khoi
From: gluster-users-request at gluster.org
To: gluster-users at gluster.org
Date: 02/19/2014 05:59 AM
Subject: Gluster-users Digest, Vol 70, Issue 18
Sent by: gluster-users-bounces at gluster.org
Send Gluster-users mailing list submissions to
gluster-users at gluster.org
To subscribe or unsubscribe via the World Wide Web, visit
http://supercolony.gluster.org/mailman/listinfo/gluster-users
or, via email, send a message with subject or body 'help' to
gluster-users-request at gluster.org
You can reach the person managing the list at
gluster-users-owner at gluster.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Gluster-users digest..."
Today's Topics:
1. Complete machine lockup, v3.4.2 (Laurent Chouinard)
2. Want to Ask (Prian Dwiatma)
3. Re: [Bug 1057645] ownership of diskimage changes during
livemigration, livemigration with kvm/libvirt fails (Adam Huffman)
4. Re: [Bug 1057645] ownership of diskimage changes during
livemigration, livemigration with kvm/libvirt fails (Paul Boven)
5. Re: Complete machine lockup, v3.4.2 (James)
6. Re: Problem with duplicate files (Justin Dossey)
7. upgrading from gluster-3.2.6 to gluster-3.4.2 (Dmitry Kopelevich)
8. Problems to work with mounted directory in Gluster 3.2.7
(Targino Silveira)
9. Re: Node down and volumes unreachable (Marco Zanger)
10. Re: <host> not in 'Peer in Cluster' state (Jon Cope)
11. Re: [Bug 1057645] ownership of diskimage changes during
livemigration, livemigration with kvm/libvirt fails (bernhard glomm)
12. Re: [Bug 1057645] ownership of diskimage changes during
livemigration, livemigration with kvm/libvirt fails (Josh Boon)
13. 3.4 gluster volume heal-failed recovery (Steve Dainard)
14. Re: add-brick and fix-layout takes some VMs offline
(Nicholas Majeran)
15. Re: Problems to work with mounted directory in Gluster 3.2.7
(Franco Broi)
16. Re: Problems to work with mounted directory in Gluster 3.2.7
(Targino Silveira)
17. Agenda for Community meeting today (Vijay Bellur)
----------------------------------------------------------------------
Message: 1
Date: Tue, 18 Feb 2014 08:12:33 -0500
From: Laurent Chouinard <laurent.chouinard at ubisoft.com>
To: "gluster-users at gluster.org" <gluster-users at gluster.org>
Subject: [Gluster-users] Complete machine lockup, v3.4.2
Message-ID:
<8CFDF989A34FE54FA678C09C6D9F6D39B560BC342B at MDC-MAIL-CMS01.ubisoft.org>
Content-Type: text/plain; charset="us-ascii"
Hi everyone,
We've been using 3.3.2 for a while, and recently started to migrate to
3.4.2. We run on platform CentOS 6.5 for 3.4.2 (while 3.3.2 were installed
on CentOS 6.4)
Recently, we've have a very scary condition happen and we do not know
exactly the cause of it.
We have a 3 nodes cluster with a replication factor of 3. Each node has
one brick, which is made out of one RAID0 volume, comprised of multiple
SSDs.
Following some read/write errors, nodes 2 and 3 have completely locked.
Nothing could be done physically (nothing on the screen, nothing by SSH),
physical power cycle had to be done. Node 1 was still accessible, but its
fuse client rejected most if not all reads and writes.
Has anyone experienced something similar?
Before the system freeze, the last thing the kernel seemed to be doing is
killing HTTPD threads (INFO: task httpd:7910 blocked for more than 120
seconds.) End-users talk to Apache in order to read/write from the
Gluster volume, so it seems a simple case of "something wrong" with
gluster which locks read/writes, and eventually the kernel kills them.
At this point, we're unsure where to look. Nothing very specific can be
found in the logs, but perhaps if someone has pointers of what to look
for, that could give us a new search track.
Thanks
Laurent Chouinard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/c03f9e09/attachment-0001.html
>
------------------------------
Message: 2
Date: Tue, 18 Feb 2014 10:24:48 +0700
From: Prian Dwiatma <priandwiatma at gmail.com>
To: gluster-users at gluster.org
Subject: [Gluster-users] Want to Ask
Message-ID:
<CAAN9Jt2v2a2ftNZNUhBztZpLgqhs=drGiYoxYUhXxEbi9YwB2A at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
hello....i'm Prian
I want to ask about glusterfs, is there any file types that affect Gluster
for data transmission?
thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/d9454deb/attachment-0001.html
>
------------------------------
Message: 3
Date: Tue, 18 Feb 2014 13:59:10 +0000
From: Adam Huffman <adam.huffman at gmail.com>
To: gluster-users at gluster.org
Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage
changes during livemigration, livemigration with
kvm/libvirt fails
Message-ID:
<CAP5prOjzhC1fBqYtpjSnFsfT1xQkDWCGzkX2e_JErq+oW1R1pQ at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Hi Paul,
Could you keep the list updated? That bug has been marked private, so
I can't see it.
Best Wishes,
Adam
On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven at jive.nl> wrote:
> Hi Bernhard, everyone,
>
> The same problem has now been reproduced on RedHat, please see:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1058032
>
> With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it
broke
> when the packages were upgraded to 3.4.1.
>
> I've set AppArmor to 'complain' as part of the debugging, so that's not
the
> issue.
>
> I'm still not convinced that the file ownership itself is the root cause
of
> this issue, it could well be just a symptom. Libvirt/qemu is perfectly
happy
> to start a VM when its image file is owned root:root, and change
ownership
> to libvirt-qemu:kvm. So I see no reason why it couldn't do the same
during a
> live migration.
>
> In my opinion the real issue is the failure at the fuse level, that
makes
> file access to the image on the destination impossible, even for root.
>
> Regards, Paul Boven.
>
>
> On 01/27/2014 07:51 PM, BGM wrote:
>>
>> Hi Paul & all
>> I'm really keen on getting this solved,
>> right now it's a nasty show stopper.
>> I could try different gluster versions,
>> as long as I can get the .debs for it,
>> wouldn't want to start compiling
>> (although.... does a config option have changed on package build?)
>> you reported that 3.4.0 on ubuntu 13.04 was working, right?
>> code diff, config options for package build.
>> Another approach: can anyone verify or falsify
>> https://bugzilla.redhat.com/show_bug.cgi?id=1057645
>> on another distro than ubuntu/debian?
>> thinking of it... could it be an apparmor interference?
>> I had fun with apparmor and mysql on ubuntu 12.04 once...
>> will have a look at that tomorrow.
>> As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed
>> and the pain of maintenance) so AFAIK I have to blame the ownership
change
>> on gluster, not on an issue with my general setup....
>> best regards
>> Bernhard
>
>
>
> --
> Paul Boven <boven at jive.nl> +31 (0)521-596547
> Unix/Linux/Networking specialist
> Joint Institute for VLBI in Europe - www.jive.nl
> VLBI - It's a fringe science
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
------------------------------
Message: 4
Date: Tue, 18 Feb 2014 15:11:05 +0100
From: Paul Boven <boven at jive.nl>
To: gluster-users at gluster.org
Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage
changes during livemigration, livemigration with
kvm/libvirt fails
Message-ID: <530369F9.4060404 at jive.nl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi Adam,
This is rather odd - the Redhat specific clone of the bug we filed is
now marked private, even if I log in with my RH bugtracker account.
However, the original bug is still accessible:
https://bugzilla.redhat.com/show_bug.cgi?id=1057645
There's not been any progress as far as I know. We are using the
workaround (which stops libvirt/qemu from doing the chown) in
production. With the release of Ubuntu 14.04 LTS, I hope to be able to
use libgfapi on our setup.
Perhaps the fact that the RedHat specific bug is now private might mean
that they're actually doing something with it, but I wouldn't know.
Regards, Paul Boven.
On 02/18/2014 02:59 PM, Adam Huffman wrote:
> Hi Paul,
>
> Could you keep the list updated? That bug has been marked private, so
> I can't see it.
>
> Best Wishes,
> Adam
>
> On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven at jive.nl> wrote:
>> Hi Bernhard, everyone,
>>
>> The same problem has now been reproduced on RedHat, please see:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1058032
>>
>> With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it
broke
>> when the packages were upgraded to 3.4.1.
>>
>> I've set AppArmor to 'complain' as part of the debugging, so that's not
the
>> issue.
>>
>> I'm still not convinced that the file ownership itself is the root
cause of
>> this issue, it could well be just a symptom. Libvirt/qemu is perfectly
happy
>> to start a VM when its image file is owned root:root, and change
ownership
>> to libvirt-qemu:kvm. So I see no reason why it couldn't do the same
during a
>> live migration.
>>
>> In my opinion the real issue is the failure at the fuse level, that
makes
>> file access to the image on the destination impossible, even for root.
>>
>> Regards, Paul Boven.
>>
>>
>> On 01/27/2014 07:51 PM, BGM wrote:
>>>
>>> Hi Paul & all
>>> I'm really keen on getting this solved,
>>> right now it's a nasty show stopper.
>>> I could try different gluster versions,
>>> as long as I can get the .debs for it,
>>> wouldn't want to start compiling
>>> (although.... does a config option have changed on package build?)
>>> you reported that 3.4.0 on ubuntu 13.04 was working, right?
>>> code diff, config options for package build.
>>> Another approach: can anyone verify or falsify
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1057645
>>> on another distro than ubuntu/debian?
>>> thinking of it... could it be an apparmor interference?
>>> I had fun with apparmor and mysql on ubuntu 12.04 once...
>>> will have a look at that tomorrow.
>>> As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed
>>> and the pain of maintenance) so AFAIK I have to blame the ownership
change
>>> on gluster, not on an issue with my general setup....
>>> best regards
>>> Bernhard
>>
>>
>>
>> --
>> Paul Boven <boven at jive.nl> +31 (0)521-596547
>> Unix/Linux/Networking specialist
>> Joint Institute for VLBI in Europe - www.jive.nl
>> VLBI - It's a fringe science
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
--
Paul Boven <boven at jive.nl> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
------------------------------
Message: 5
Date: Tue, 18 Feb 2014 11:17:46 -0500
From: James <purpleidea at gmail.com>
To: Laurent Chouinard <laurent.chouinard at ubisoft.com>
Cc: "gluster-users at gluster.org" <gluster-users at gluster.org>
Subject: Re: [Gluster-users] Complete machine lockup, v3.4.2
Message-ID:
<CADCaTgrkYpU00E6Gk4t2S0oZaERq6mQAq8nPTscdiRf2XnWFWg at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On Tue, Feb 18, 2014 at 8:12 AM, Laurent Chouinard
<laurent.chouinard at ubisoft.com> wrote:
> Before the system freeze, the last thing the kernel seemed to be doing
is
> killing HTTPD threads (INFO: task httpd:7910 blocked for more than 120
> seconds.) End-users talk to Apache in order to read/write from the
Gluster
> volume, so it seems a simple case of ?something wrong? with gluster
which
> locks read/writes, and eventually the kernel kills them.
If the kernel was killing things, check that it wasn't the OOM killer.
If so, you might want to ensure you've got swap, enough memory, check
if anything is leaking, and finally if you have memory management
issues between services, cgroups might be the thing to use to control
this.
HTH,
James
------------------------------
Message: 6
Date: Tue, 18 Feb 2014 08:30:20 -0800
From: Justin Dossey <jbd at podomatic.com>
To: tegner at renget.se
Cc: gluster-users <gluster-users at gluster.org>
Subject: Re: [Gluster-users] Problem with duplicate files
Message-ID:
<CAPMPShzN6nprhm17DQzFbM4oBz09-VLntnK2XsVa3XPLB0sgrw at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
I've seen issues with GlusterFS and rsync. They were mostly ameliorated
by
adding --inplace to my rsync arguments.
On Mon, Feb 17, 2014 at 12:57 AM, <tegner at renget.se> wrote:
> As a follow up. Lots of the following from log:
>
> [dht_lookup_everywhere_cbk] 0-glusterKumiko-dht: deleting stale linkfile
> 0.33702/T on glusterKumiko-client-2
>
> [client-rpc-fops.c:5724:client3_3_setattr]
>
(-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x262)
> [0x7fd78746a8f2]
>
(-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_linkfile_attr_heal+0x3ce)
> [0x7fd7874541fe]
>
(-->/usr/lib64/glusterfs/3.4.2/xlator/protocol/client.so(client_setattr+0x89)
> [0x7fd78769a649]))) 0-: Assertion failed: 0
>
> [2014-02-17 08:21:48.200685] E
> [dht-linkfile.c:213:dht_linkfile_setattr_cbk] 0-glusterKumiko-dht:
setattr
> of uid/gid on 0.33702/T :<gfid:00000000-0000-0000-0000-000000000000>
failed
> (Invalid argument)
>
> There were also quite a few cases were there were directories that were
> empty except for a bunch of "mode 1000" files.
>
> The system in question is used as a mirror of a file system which is
used
> quite heavily, new files are created, old ones are deleted (and quite
often
> recreated). I use rsync to update the system, like
>
> rsync -vau --delete live:/gluster_1/ mirror:gluster_2/
>
> Could the issues I'm experiencing be connected to the way I use rsync?
>
> Thanks,
>
> /jon
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
--
Justin Dossey
CTO, PodOmatic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/71f28aab/attachment-0001.html
>
------------------------------
Message: 7
Date: Tue, 18 Feb 2014 14:51:39 -0500
From: Dmitry Kopelevich <dkopelevich at che.ufl.edu>
To: <gluster-users at gluster.org>
Subject: [Gluster-users] upgrading from gluster-3.2.6 to gluster-3.4.2
Message-ID: <5303B9CB.7000800 at che.ufl.edu>
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
I am attempting to upgrade my GlusterFS from 3.2.6 to 3.4.2 using the
instructions posted at
http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3.
These guidelines are for an upgrade to 3.3 but it is stated at
http://vbellur.wordpress.com/2013/07/15/upgrading-to-glusterfs-3-4 that
they can also be used to upgrade to 3.4.0. So I was hoping that they
would also work with an upgrade to 3.4.2.
I'm running CentOS 5 and installed the following rpms on the gluster
servers:
glusterfs-libs-3.4.2-1.el5.x86_64.rpm
glusterfs-3.4.2-1.el5.x86_64.rpm
glusterfs-fuse-3.4.2-1.el5.x86_64.rpm
glusterfs-cli-3.4.2-1.el5.x86_64.rpm
glusterfs-server-3.4.2-1.el5.x86_64.rpm
glusterfs-rdma-3.4.2-1.el5.x86_64.rpm
glusterfs-geo-replication-3.4.2-1.el5.x86_64.rpm
According to the installation guidelines, installation from rpms should
automatically copy the files from /etc/glusterd to /var/lib/glusterd.
This didn't happen for me -- the directory /var/lib/glusterd contained
only empty subdirectories. But the content of /etc/glusterd directory
has moved to /etc/glusterd/glusterd.
So, I decided to manually copy files from /etc/glusterd/glusterd to
/var/lib/glusterd and follow step 5 of the installation guidelines
(which was supposed to be skipped when installing from rpms):
glusterd --xlator-option *.upgrade=on -N
This didn't work (error message: glusterd: No match)
Then I triedspecifying explicitly the name of my volume:
glusterd --xlator-option <volume>.upgrade=on -N
This lead to the following messages in file
etc-glusterfs-glusterd.vol.log:
[2014-02-18 17:22:27.146449] I [glusterd.c:961:init] 0-management: Using
/var/lib/glusterd as working directory
[2014-02-18 17:22:27.149097] I [socket.c:3480:socket_init]
0-socket.management: SSL support is NOT enabled
[2014-02-18 17:22:27.149126] I [socket.c:3495:socket_init]
0-socket.management: using system polling thread
[2014-02-18 17:22:29.282665] I
[glusterd-store.c:1339:glusterd_restore_op_version] 0-glusterd:
retrieved op-version: 1
[2014-02-18 17:22:29.283478] E
[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-0
[2014-02-18 17:22:29.283513] E
[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-1
[2014-02-18 17:22:29.283534] E
[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-2
...
and so on for all other bricks.
After that, files nfs.log, glustershd.log, and
etc-glusterfs-glusterd.vol.log get filled with a large number of warning
messages and nothing else seems to happen. The following messages appear
to be relevant:
- Files nfs.log, glustershd.log:
2014-02-18 15:58:01.889847] W [rdma.c:1079:gf_rdma_cm_event_handler]
0-data-volume-client-2: cma event RDMA_CM_EVENT_ADDR_ERROR, error -2
(me: peer:)
(the name of my volume is data-volume and its transport type is RDMA)
- File etc-glusterfs-glusterd.vol.log
[2014-02-18 17:22:33.322565] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
Also, for some reason the time stamps in the log files are incorrect.
Any suggestions for fixing this would be greatly appreciated.
Thanks,
Dmitry
--
Dmitry Kopelevich
Associate Professor
Chemical Engineering Department
University of Florida
Gainesville, FL 32611
Phone: (352)-392-4422
Fax: (352)-392-9513
E-mail:dkopelevich at che.ufl.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/a7d9f296/attachment-0001.html
>
------------------------------
Message: 8
Date: Tue, 18 Feb 2014 17:30:35 -0300
From: Targino Silveira <targinosilveira at gmail.com>
To: gluster-users at gluster.org
Subject: [Gluster-users] Problems to work with mounted directory in
Gluster 3.2.7
Message-ID:
<CA+PDx1i_6=baTLc=s_F2sA6vwT=YV3vT_pLPNh+kYUM_TSDcwA at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi all,
I am getting a problem and can't understand why, I have created a cluster
with gluster following the most simple way as explaind in QuickStart on
the
glustger.org.
I have created 3 VM in KVM.
2 To host gluster server with one disk image with 1tb.
1 To host gluster client to mounting my volume.
I'm using Debian 7 and used apt-get to install Gluster 3.2.7, Server and
Client.
After all finished I could to mount as glusterfs with no problem "mount -t
glusterfs /mnt/cloudbackup_data/ vm-gluster-cloudbackup1:/vol1" but I
can't
to work on the mounted directory, if I perform a "ls -lh" it's running
forver, and I can't do any other operation and VM is blocked.
If I try to mount as NFS "mount -t nfs vm-gluster-cloudbackup2:/vol1
/mnt/cloudbackup_data/ "I get a "Time out" I'm not so much expert in
gluster, but I can't see any reason for this problem, someone know
something like that?
Regards,
Targino Silveira
+55-85-8626-7297
www.twitter.com/targinosilveira
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/e27144fe/attachment-0001.html
>
------------------------------
Message: 9
Date: Tue, 18 Feb 2014 20:44:18 +0000
From: Marco Zanger <mzanger at hexacta.com>
To: Vijay Bellur <vbellur at redhat.com>, "gluster-users at gluster.org"
<gluster-users at gluster.org>
Subject: Re: [Gluster-users] Node down and volumes unreachable
Message-ID:
<A2DF3CE8B1F5C8418E1721A57D20B527DB792827 at hxmx01.hexacta.com>
Content-Type: text/plain; charset="iso-8859-1"
The Log of that particular volume says:
[2014-02-18 09:43:17.136182] W [socket.c:410:__socket_keepalive] 0-socket:
failed to set keep idle on socket 8
[2014-02-18 09:43:17.136285] W [socket.c:1876:socket_server_event_handler]
0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
[2014-02-18 09:43:18.343409] I [server-handshake.c:571:server_setvolume]
0-teoswitch_default_storage-server: accepted client from
xxxxx55.domain.com-2075-2014/02/18-09:43:14:302234-teoswitch_default_storage-client-1-0
(version: 3.3.0)
[2014-02-18 09:43:21.356302] I [server-handshake.c:571:server_setvolume]
0-teoswitch_default_storage-server: accepted client from xxxxx54.
domain.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0
(version: 3.3.0)
[2014-02-18 10:38:26.488333] W [socket.c:195:__socket_rwv]
0-tcp.teoswitch_default_storage-server: readv failed (Connection timed
out)
[2014-02-18 10:38:26.488431] I [server.c:685:server_rpc_notify]
0-teoswitch_default_storage-server: disconnecting connectionfrom
xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0
[2014-02-18 10:38:26.488494] I
[server-helpers.c:741:server_connection_put]
0-teoswitch_default_storage-server: Shutting down connection
xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0
[2014-02-18 10:38:26.488541] I
[server-helpers.c:629:server_connection_destroy]
0-teoswitch_default_storage-server: destroyed connection of
xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0
When I try to access the folder I get.
[root at hxteo55 ~]# ll /<path> /1001/voicemail/
ls: /<path>/1001/voicemail/: Input/output error?
This is the volume info:
Volume Name: teoswitch_default_storage
Type: Distribute
Volume ID: 83c9d6f3-0288-4358-9fdc-b1d062cc8fca
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 12.12.123.54:/<path>/gluster/36779974/teoswitch_default_storage
Brick2: 12.12.123.55:/<path>/gluster/36779974/teoswitch_default_storage
Any ideas?
Marco Zanger
Phone 54 11 5299-5400 (int. 5501)
Clay 2954, C1426DLD, Buenos Aires, Argentina
Think Green - Please do not print this email unless you really need to
-----Original Message-----
From: Vijay Bellur [mailto:vbellur at redhat.com]
Sent: martes, 18 de febrero de 2014 03:56 a.m.
To: Marco Zanger; gluster-users at gluster.org
Subject: Re: [Gluster-users] Node down and volumes unreachable
On 02/17/2014 11:19 PM, Marco Zanger wrote:
> Read/write operations hang for long period of time (too long). I've
> seen it in that state (waiting) for something like 5 minutes, which
> makes every application fail trying to read or write. These are the
> Errors I found in the logs in the server A which is still accessible
> (B was down)
>
> etc-glusterfs-glusterd.vol.log
>
> ...
> [2014-01-31 07:56:49.780247] W
> [socket.c:1512:__socket_proto_state_machine] 0-management: reading
> from socket failed. Error (Connection timed out), peer
> (<SERVER_B_IP>:24007)
> [2014-01-31 07:58:25.965783] E [socket.c:1715:socket_connect_finish]
> 0-management: connection to <SERVER_B_IP>:24007 failed (No route to
> host)
> [2014-01-31 08:59:33.923250] I
> [glusterd-handshake.c:397:glusterd_set_clnt_mgmt_program] 0-: Using
> Program glusterd mgmt, Num (1238433), Version (2)
> [2014-01-31 08:59:33.923289] I
> [glusterd-handshake.c:403:glusterd_set_clnt_mgmt_program] 0-: Using
Program Peer mgmt, Num (1238437), Version (2) ...
>
>
> glustershd.log
>
> [2014-01-27 12:07:03.644849] W
> [socket.c:1512:__socket_proto_state_machine]
> 0-teoswitch_custom_music-client-1: reading from socket failed. Error
> (Connection timed out), peer (<SERVER_B_IP>:24010)
> [2014-01-27 12:07:03.644888] I [client.c:2090:client_rpc_notify]
> 0-teoswitch_custom_music-client-1: disconnected
> [2014-01-27 12:09:35.553628] E [socket.c:1715:socket_connect_finish]
> 0-teoswitch_greetings-client-1: connection to <SERVER_B_IP>:24011
> failed (Connection timed out)
> [2014-01-27 12:10:13.588148] E [socket.c:1715:socket_connect_finish]
> 0-license_path-client-1: connection to <SERVER_B_IP>:24013 failed
> (Connection timed out)
> [2014-01-27 12:10:15.593699] E [socket.c:1715:socket_connect_finish]
> 0-upload_path-client-1: connection to <SERVER_B_IP>:24009 failed
> (Connection timed out)
> [2014-01-27 12:10:21.601670] E [socket.c:1715:socket_connect_finish]
> 0-teoswitch_ivr_greetings-client-1: connection to <SERVER_B_IP>:24012
> failed (Connection timed out)
> [2014-01-27 12:10:23.607312] E [socket.c:1715:socket_connect_finish]
> 0-teoswitch_custom_music-client-1: connection to <SERVER_B_IP>:24010
> failed (Connection timed out)
> [2014-01-27 12:11:21.866604] E [afr-self-heald.c:418:_crawl_proceed]
> 0-teoswitch_ivr_greetings-replicate-0: Stopping crawl as < 2 children
> are up
> [2014-01-27 12:11:21.867874] E [afr-self-heald.c:418:_crawl_proceed]
> 0-teoswitch_greetings-replicate-0: Stopping crawl as < 2 children are
> up
> [2014-01-27 12:11:21.868134] E [afr-self-heald.c:418:_crawl_proceed]
> 0-teoswitch_custom_music-replicate-0: Stopping crawl as < 2 children
> are up
> [2014-01-27 12:11:21.869417] E [afr-self-heald.c:418:_crawl_proceed]
> 0-license_path-replicate-0: Stopping crawl as < 2 children are up
> [2014-01-27 12:11:21.869659] E [afr-self-heald.c:418:_crawl_proceed]
> 0-upload_path-replicate-0: Stopping crawl as < 2 children are up
> [2014-01-27 12:12:53.948154] I
> [client-handshake.c:1636:select_server_supported_programs]
> 0-teoswitch_greetings-client-1: Using Program GlusterFS 3.3.0, Num
> (1298437), Version (330)
> [2014-01-27 12:12:53.952894] I
> [client-handshake.c:1433:client_setvolume_cbk]
> 0-teoswitch_greetings-client-1: Connected to <SERVER_B_IP>:24011,
> attached to remote volume
>
> nfs.log there are lots of errors but the one that insist most Is this:
>
> [2014-01-27 12:12:27.136033] E [socket.c:1715:socket_connect_finish]
> 0-teoswitch_custom_music-client-1: connection to <SERVER_B_IP>:24010
> failed (Connection timed out)
>
> Any ideas? From the logs I see nothing but confirm the fact that A
cannot reach B which makes sense since B is down. But A is not, and it's
volume should still be accesible. Right?
Nothing very obvious from these logs.
Can you share relevant portions of the client log file? Usually the name
of the mount point would be a part of the client log file.
-Vijay
------------------------------
Message: 10
Date: Tue, 18 Feb 2014 16:23:19 -0500 (EST)
From: Jon Cope <jcope at redhat.com>
To: Kaushal M <kshlmster at gmail.com>
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] <host> not in 'Peer in Cluster' state
Message-ID:
<515830110.1228610.1392758599748.JavaMail.zimbra at redhat.com>
Content-Type: text/plain; charset=utf-8
Hi Kaushal,
Thanks for the input. I gave it a go and it produced identical results. I
was, however, pointed towards an article that's led me to a solution,
linked below. As I understand it, by assigning each node an elastic IP,
you cement the public DNS (now containing the elasticIP), preventing AWS
from changing it during reboot. Querying the public DNS from inside EC2
returns the private IP addresses, while a query from outside EC2 returns
the elastic IP. Gluster seems happy with this, so I am too.
Regards,
Jon
http://alestic.com/2009/06/ec2-elastic-ip-internal
----- Original Message -----
From: "Kaushal M" <kshlmster at gmail.com>
To: "Jon Cope" <jcope at redhat.com>
Cc: gluster-users at gluster.org
Sent: Saturday, February 15, 2014 5:40:32 AM
Subject: Re: [Gluster-users] <host> not in 'Peer in Cluster' state
Peer status having node1's elastic ip suggests that you probed the
other peers from node1. This would mean that the other peers don't
know of node1's hostname. Even though you've edited the hosts file on
the peers, a reverse resolution on node1s ip wouldn't return the
hostnames you've set. Gluster uses reverse resolution to match
hostnames when it doesn't have a straight match in the peer list.
To recover from this. just probe node1 from another peer. Do '#
gluster peer probe node1.ec2' from another peer. This will update
gluster's peerlist to contain the name node1.ec2. After this other
operations will continue successfully.
~kaushal
On Sat, Feb 15, 2014 at 5:23 AM, Jon Cope <jcope at redhat.com> wrote:
> Hi all,
>
> I'm attempting to create a 4 nodes cluster over EC2. I'm fairly new to
this and so may not be seeing something obvious.
>
> - Established passworldless SSH between nodes.
> - edited /etc/sysconfig/network HOSTNAME=node#.ec2 to satisfy FQDN
> - mounted xfs /dev/xvdh /mnt/brick1
> - stopped iptables
>
>
> The error I'm getting occurs when invoking the following, where <volume>
is the volume name:
>
> # gluster volume create <volume> replica 2 node1.ec2:/mnt/brick1
node2.ec2:/mnt/brick1 node3.ec2:/mnt/brick1 node4.ec2:/mnt/brick1
> # volume create: <volume>: failed: Host node1.ec2 is not in 'Peer in
Cluster' state
>
> Checking peer status of node1.ec2 from node{2..4}.ec2 produces the
following. Note that node1.ec2's elastic IP appears instead of the FQDN;
not sure if that's relevant or not.
>
> [root at node2 ~]# gluster peer status
> Number of Peers: 3
>
> Hostname: node4.ec2
> Uuid: ab2bcdd8-2c0b-439d-b685-3be457988abc
> State: Peer in Cluster (Connected)
>
> Hostname: node3.ec2
> Uuid: 4f128213-3549-494a-af04-822b5e2f2b96
> State: Peer in Cluster (Connected)
>
> Hostname: ###.##.##.### #node1.ec2 elastic IP
> Uuid: 09d81803-e5e1-43b1-9faf-e94f730acc3e
> State: Peer in Cluster (Connected)
>
> The error as it appears in vim etc-glusterfs-glusterd.vol.log:
>
> [2014-02-14 23:28:44.634663] E
[glusterd-utils.c:5351:glusterd_new_brick_validate] 0-management: Host
node1.ec2 is not in 'Peer in Cluster' state
> [2014-02-14 23:28:44.634699] E
[glusterd-volume-ops.c:795:glusterd_op_stage_create_volume] 0-management:
Host node1.ec2 is not in 'Peer in Cluster' state
> [2014-02-14 23:28:44.634718] E [glusterd-syncop.c:890:gd_stage_op_phase]
0-management: Staging of operation 'Volume Create' failed on localhost :
Host node1.ec2 is not in 'Peer in Cluster' state
>
> Can someone suggest possible cause of this error or point me in a viable
direction?
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
------------------------------
Message: 11
Date: Tue, 18 Feb 2014 22:23:53 +0100
From: bernhard glomm <bernhard.glomm at ecologic.eu>
To: Paul Boven <boven at jive.nl>, gluster-devel at nongnu.org
Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>
Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage
changes during livemigration,
livemigration with kvm/libvirt fails
Message-ID: <7396719A-ADEE-4F49-93BA-D924B469A0EE at ecologic.eu>
Content-Type: text/plain; charset="windows-1252"
Hi Paul, and all
> With the release of Ubuntu 14.04 LTS, I hope to be able to use libgfapi
on our setup.
Well I hope that too, but I'm not sure if that will happen (soon).
I , recompiled qemu as described here:
http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems
with little luck since finally the ipxe-qemu component (which I need/want)
and I didn't had the time to dig deeper into that.
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517
still says "won't fix it" :-(
Josh Boon (hi Josh) suggested to create a PPA for an ubuntu qemu with
gfapi support
which might be a temporary solution?
But from my "happy end - user" perspective (as a "non-dev-but-op")
that looks like a lot of parallel extra work
on maintaining that "fork"in the long run.
I hope the gluster devels either can get it into debian(and/or ubuntu)
or even qemu (as a default option?) directly
> Perhaps the fact that the RedHat specific bug is now private might mean
that they're actually doing something with it, but I wouldn't know.
we'll stay on that topic
Regards,
Bernahrd
> Regards, Paul Boven.
> On 02/18/2014 02:59 PM, Adam Huffman wrote:
>> Hi Paul,
>>
>> Could you keep the list updated? That bug has been marked private, so
>> I can't see it.
>>
>> Best Wishes,
>> Adam
>>
>> On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven at jive.nl> wrote:
>>> Hi Bernhard, everyone,
>>>
>>> The same problem has now been reproduced on RedHat, please see:
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1058032
>>>
>>> With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it
broke
>>> when the packages were upgraded to 3.4.1.
>>>
>>> I've set AppArmor to 'complain' as part of the debugging, so that's
not the
>>> issue.
>>>
>>> I'm still not convinced that the file ownership itself is the root
cause of
>>> this issue, it could well be just a symptom. Libvirt/qemu is perfectly
happy
>>> to start a VM when its image file is owned root:root, and change
ownership
>>> to libvirt-qemu:kvm. So I see no reason why it couldn't do the same
during a
>>> live migration.
>>>
>>> In my opinion the real issue is the failure at the fuse level, that
makes
>>> file access to the image on the destination impossible, even for root.
>>>
>>> Regards, Paul Boven.
>>>
>>>
>>> On 01/27/2014 07:51 PM, BGM wrote:
>>>>
>>>> Hi Paul & all
>>>> I'm really keen on getting this solved,
>>>> right now it's a nasty show stopper.
>>>> I could try different gluster versions,
>>>> as long as I can get the .debs for it,
>>>> wouldn't want to start compiling
>>>> (although.... does a config option have changed on package build?)
>>>> you reported that 3.4.0 on ubuntu 13.04 was working, right?
>>>> code diff, config options for package build.
>>>> Another approach: can anyone verify or falsify
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1057645
>>>> on another distro than ubuntu/debian?
>>>> thinking of it... could it be an apparmor interference?
>>>> I had fun with apparmor and mysql on ubuntu 12.04 once...
>>>> will have a look at that tomorrow.
>>>> As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed
>>>> and the pain of maintenance) so AFAIK I have to blame the ownership
change
>>>> on gluster, not on an issue with my general setup....
>>>> best regards
>>>> Bernhard
>>>
>>>
>>>
>>> --
>>> Paul Boven <boven at jive.nl> +31 (0)521-596547
>>> Unix/Linux/Networking specialist
>>> Joint Institute for VLBI in Europe - www.jive.nl
>>> VLBI - It's a fringe science
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> --
> Paul Boven <boven at jive.nl> +31 (0)521-596547
> Unix/Linux/Networking specialist
> Joint Institute for VLBI in Europe - www.jive.nl
> VLBI - It's a fringe science
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/ffabe77c/attachment-0001.html
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/ffabe77c/attachment-0001.sig
>
------------------------------
Message: 12
Date: Tue, 18 Feb 2014 15:31:24 -0600 (CST)
From: Josh Boon <gluster at joshboon.com>
To: bernhard glomm <bernhard.glomm at ecologic.eu>
Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>,
gluster-devel at nongnu.org
Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage
changes during livemigration, livemigration with
kvm/libvirt fails
Message-ID: <1078179789.1672169.1392759084410.JavaMail.zimbra at 80ok.be>
Content-Type: text/plain; charset="utf-8"
I've yet to crack into 14.04 and see what that beast looks like as we
don't have it in use here yet. I'll add that to my todo list. I can be
bribed if someone needs it sooner :)
As a separate task, I'll look into the ipxe problem. I remember it was
file conflict between the packages so I may have to do some hacks in the
debian build rules for one of the packages.
The PPA is an option but yes maintenance would be ongoing and best effort
as that's all I can afford currently.
----- Original Message -----
From: "bernhard glomm" <bernhard.glomm at ecologic.eu>
To: "Paul Boven" <boven at jive.nl>, gluster-devel at nongnu.org
Cc: "Josh Boon" <gluster at joshboon.com>, "gluster-users at gluster.org List"
<gluster-users at gluster.org>
Sent: Tuesday, February 18, 2014 4:23:53 PM
Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage changes
during livemigration, livemigration with kvm/libvirt fails
Hi Paul, and all
With the release of Ubuntu 14.04 LTS, I hope to be able to use libgfapi on
our setup.
Well I hope that too, but I'm not sure if that will happen (soon).
I , recompiled qemu as described here:
http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems
with little luck since finally the ipxe-qemu component (which I need/want)
and I didn't had the time to dig deeper into that.
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517
still says "won't fix it" :-(
Josh Boon (hi Josh) suggested to create a PPA for an ubuntu qemu with
gfapi support
which might be a temporary solution ?
But from my "happy end - user" perspective (as a "non-dev-but-op")
that looks like a lot of parallel extra work
on maintaining that "fork"in the long run.
I hope the gluster devels either can get it into debian(and/or ubuntu)
or even qemu (as a default option?) directly
<blockquote>
Perhaps the fact that the RedHat specific bug is now private might mean
that they're actually doing something with it, but I wouldn't know.
</blockquote>
we'll stay on that topic
Regards,
Bernahrd
<blockquote>
Regards, Paul Boven.
</blockquote>
<blockquote>
On 02/18/2014 02:59 PM, Adam Huffman wrote:
<blockquote>
Hi Paul,
Could you keep the list updated? That bug has been marked private, so
I can't see it.
Best Wishes,
Adam
On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven < boven at jive.nl > wrote:
<blockquote>
Hi Bernhard, everyone,
The same problem has now been reproduced on RedHat, please see:
https://bugzilla.redhat.com/show_bug.cgi?id=1058032
With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it broke
when the packages were upgraded to 3.4.1.
I've set AppArmor to 'complain' as part of the debugging, so that's not
the
issue.
I'm still not convinced that the file ownership itself is the root cause
of
this issue, it could well be just a symptom. Libvirt/qemu is perfectly
happy
to start a VM when its image file is owned root:root, and change ownership
to libvirt-qemu:kvm. So I see no reason why it couldn't do the same during
a
live migration.
In my opinion the real issue is the failure at the fuse level, that makes
file access to the image on the destination impossible, even for root.
Regards, Paul Boven.
On 01/27/2014 07:51 PM, BGM wrote:
<blockquote>
Hi Paul & all
I'm really keen on getting this solved,
right now it's a nasty show stopper.
I could try different gluster versions,
as long as I can get the .debs for it,
wouldn't want to start compiling
(although.... does a config option have changed on package build?)
you reported that 3.4.0 on ubuntu 13.04 was working, right?
code diff, config options for package build.
Another approach: can anyone verify or falsify
https://bugzilla.redhat.com/show_bug.cgi?id=1057645
on another distro than ubuntu/debian?
thinking of it... could it be an apparmor interference?
I had fun with apparmor and mysql on ubuntu 12.04 once...
will have a look at that tomorrow.
As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed
and the pain of maintenance) so AFAIK I have to blame the ownership change
on gluster, not on an issue with my general setup....
best regards
Bernhard
</blockquote>
--
Paul Boven <boven at jive.nl> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
</blockquote>
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
</blockquote>
--
Paul Boven < boven at jive.nl > +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
</blockquote>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/2839804c/attachment-0001.html
>
------------------------------
Message: 13
Date: Tue, 18 Feb 2014 16:57:14 -0500
From: Steve Dainard <sdainard at miovision.com>
To: gluster-users at gluster.org
Subject: [Gluster-users] 3.4 gluster volume heal-failed recovery
Message-ID:
<CAHnsdUtwRTccoAy-ghiqUHx-KbL5jVmOuGFxmC0cvtwktJnR=g at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Is there a proper method to recover from heal-failed entries?
rep1 is a 2 brick, quorum enabled volume used for virt storage for Ovirt,
accessed via NFS:
gluster> volume info rep1
Volume Name: rep1
Type: Replicate
Volume ID: 5876746d-5977-4fa9-a829-69f44b3364ec
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.0.10.2:/mnt/storage/lv-storage-domain/rep1
Brick2: 10.0.10.3:/mnt/storage/lv-storage-domain/rep1
Options Reconfigured:
cluster.server-quorum-type: server
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
nfs.trusted-write: on
nfs.trusted-sync: off
storage.owner-gid: 36
storage.owner-uid: 36
server.allow-insecure: on
cluster.quorum-type: auto
gluster> volume heal rep1 info heal-failed
Gathering Heal info on volume rep1 has been successful
Brick 10.0.10.2:/mnt/storage/lv-storage-domain/rep1
Number of entries: 6
at path on brick
-----------------------------------
2014-02-14 17:25:50 <gfid:2d619e4d-6823-4fb3-ad9d-f33459833f85>
2014-02-14 17:25:50 <gfid:0708b030-3445-4015-a6dd-c28446e81e67>
2014-02-14 17:25:50 <gfid:ce29e947-bebd-4b6a-a619-010931b5f9a8>
2014-02-14 17:25:50 <gfid:2d619e4d-6823-4fb3-ad9d-f33459833f85>
2014-02-14 17:25:50 <gfid:0708b030-3445-4015-a6dd-c28446e81e67>
2014-02-14 17:25:50 <gfid:ce29e947-bebd-4b6a-a619-010931b5f9a8>
Brick 10.0.10.3:/mnt/storage/lv-storage-domain/rep1
Number of entries: 0
Seeing as the hosts are both in quorum I was surprised to see this, but
these id's have been repeating in my logs.
Thanks,
*Steve Dainard *
IT Infrastructure Manager
Miovision <http://miovision.com/> | *Rethink Traffic*
*Blog <http://miovision.com/blog> | **LinkedIn
<https://www.linkedin.com/company/miovision-technologies> | Twitter
<https://twitter.com/miovision> | Facebook
<https://www.facebook.com/miovision>*
------------------------------
Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, Kitchener,
ON,
Canada | N2C 1L3
This e-mail may contain information that is privileged or confidential. If
you are not the intended recipient, please delete the e-mail and any
attachments and notify us immediately.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/4385a702/attachment-0001.html
>
------------------------------
Message: 14
Date: Tue, 18 Feb 2014 16:19:42 -0600
From: Nicholas Majeran <nmajeran at suntradingllc.com>
To: gluster-users at gluster.org
Subject: Re: [Gluster-users] add-brick and fix-layout takes some VMs
offline
Message-ID: <5303DC7E.7040902 at suntradingllc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 02/13/2014 09:22 AM, Nicholas Majeran wrote:
> Hi there,
>
> We have a distributed-replicated volume hosting KVM guests running
> Gluster 3.4.1.
> We've grown from 1 x 2 -> 2 x 2 -> 3 x 2,but each time we've added
> nodes or run a fix layout,
> some of our guests go offline (or worse with error=continue they
> silently error).
> After the last addition we didn't even run fix-layout as the guests
> are becoming increasingly important.
>
> Those guests are currently are using a combination of FUSE and libgfapi.
> Is there a setting or group of settings we should use to ameliorate
> this problem?
> Is FUSE or libgfapi more forgiving when add-brick or fix-layout is run?
>
> Any info is greatly appreciated.
> Thanks.
>
>
Hi,
Does anyone have any ideas on the above?
Thanks.
------------------------------
Message: 15
Date: Wed, 19 Feb 2014 08:30:25 +0800
From: Franco Broi <franco.broi at iongeo.com>
To: Targino Silveira <targinosilveira at gmail.com>
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Problems to work with mounted directory
in Gluster 3.2.7
Message-ID: <1392769825.2437.83.camel at tc1>
Content-Type: text/plain; charset="UTF-8"
You need to check the log files for obvious problems. On the servers
these should be in /var/log/glusterfs/ and you can turn on logging for
the fuse client like this:
# mount -olog-level=debug,log-file=/var/log/glusterfs/glusterfs.log -t
glusterfs ....
On Tue, 2014-02-18 at 17:30 -0300, Targino Silveira wrote:
> Hi all,
>
>
> I am getting a problem and can't understand why, I have created a
> cluster with gluster following the most simple way as explaind in
> QuickStart on the glustger.org.
>
>
> I have created 3 VM in KVM.
>
>
> 2 To host gluster server with one disk image with 1tb.
>
>
> 1 To host gluster client to mounting my volume.
>
>
> I'm using Debian 7 and used apt-get to install Gluster 3.2.7, Server
> and Client.
>
>
> After all finished I could to mount as glusterfs with no problem
> "mount -t glusterfs /mnt/cloudbackup_data/
> vm-gluster-cloudbackup1:/vol1" but I can't to work on the mounted
> directory, if I perform a "ls -lh" it's running forver, and I can't do
> any other operation and VM is blocked.
>
>
> If I try to mount as NFS "mount -t nfs
> vm-gluster-cloudbackup2:/vol1 /mnt/cloudbackup_data/ "I get a "Time
> out" I'm not so much expert in gluster, but I can't see any reason for
> this problem, someone know something like that?
>
>
> Regards,
>
>
>
> Targino Silveira
> +55-85-8626-7297
> www.twitter.com/targinosilveira
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
------------------------------
Message: 16
Date: Wed, 19 Feb 2014 00:08:50 -0300
From: Targino Silveira <targinosilveira at gmail.com>
To: bernhard glomm <bernhard.glomm at ecologic.eu>,
gluster-users at gluster.org
Subject: Re: [Gluster-users] Problems to work with mounted directory
in Gluster 3.2.7
Message-ID:
<CA+PDx1iwT5z22b7ytiwXvDxnc=H6gRZS7YgzyfoH=sxkR3deyw at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi Bernhard,
2014-02-18 19:45 GMT-03:00 bernhard glomm <bernhard.glomm at ecologic.eu>:
> hi,
> not very clear to me,
> you run VMs as gluster servers?
>
Right, I am using VM to run glusters servers.
> so 2 of your VMs using each a 1tb brick for what a ...
>
I am making a server for store some old data via FTP, so I have a two
bricks with 1tb each one, in a volume replica 2.
> could you give a
> node1:~ # gluster volume info <volname>
> ...
> node1:~ # gluster volume status <volname>
> ...
>
#First Node
root at vm-gluster-cloudbackup1:~# gluster volume info
Volume Name: vol1
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: vm-gluster-cloudbackup1:/mnt/data1/vol1
Brick2: vm-gluster-cloudbackup2:/mnt/data1/vol1
#Second node
root at vm-gluster-cloudbackup2:~# gluster volume info
Volume Name: vol1
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: vm-gluster-cloudbackup1:/mnt/data1/vol1
Brick2: vm-gluster-cloudbackup2:/mnt/data1/vol1
The volumes are started normaly with no problem, I am trying to get some
thing on the log files, I know that speed may be slow, but it's not a
problem to me at this moment.
Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140219/35a04de2/attachment-0001.html
>
------------------------------
Message: 17
Date: Wed, 19 Feb 2014 16:12:44 +0530
From: Vijay Bellur <vbellur at redhat.com>
To: "'gluster-devel at nongnu.org'" <gluster-devel at nongnu.org>,
gluster-users Discussion List <Gluster-users at gluster.org>
Subject: [Gluster-users] Agenda for Community meeting today
Message-ID: <53048AA4.2070005 at redhat.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Agenda for the weekly community meeting has been updated at:
http://titanpad.com/gluster-community-meetings
Please update the agenda if you have items for discussion.
Cheers,
Vijay
------------------------------
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
End of Gluster-users Digest, Vol 70, Issue 18
*********************************************
**
This email and any attachments may contain information that is confidential and/or privileged for the sole use of the intended recipient. Any use, review, disclosure, copying, distribution or reliance by others, and any forwarding of this email or its contents, without the express permission of the sender is strictly prohibited by law. If you are not the intended recipient, please contact the sender immediately, delete the e-mail and destroy all copies.
**
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140219/be057e1c/attachment.html>
More information about the Gluster-users
mailing list