[Gluster-users] Gluster-users Digest, Vol 41, Issue 16
Pranith Kumar K
pranithk at gluster.com
Wed Sep 7 11:41:24 UTC 2011
hi,
Could you please elaborate on replication causing data loss.
Please let us know the test case which lead you to this.
Pranith.
On 09/07/2011 04:01 PM, Jürgen Winkler wrote:
> Hi Phil,
>
> we´d the same Problem, try to compile with debug options.
> Yes this sounds strange but it help´s when u are using SLES, the
> glusterd works ok and u can start to work with it.
>
> just put
>
> exportCFLAGS='-g3 -O0'
>
> between %build and %configure in the glusterfs spec file.
>
>
>
> But be warned don´t use it with important data especially when u are
> planing to use the replication feature, this will cause in data loss
> sooner or later.
>
> Cheers !
>
>
>
>
>
> Am 07.09.2011 11:21, schrieb gluster-users-request at gluster.org:
>> Send Gluster-users mailing list submissions to
>> gluster-users at gluster.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> or, via email, send a message with subject or body 'help' to
>> gluster-users-request at gluster.org
>>
>> You can reach the person managing the list at
>> gluster-users-owner at gluster.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Gluster-users digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: Reading directly from brick (Reinis Rozitis)
>> 2. Re: NFS secondary groups not working. (Di Pe)
>> 3. Inconsistent md5sum of replicated file (Anthony Delviscio)
>> 4. Re: Inconsistent md5sum of replicated file (Pranith Kumar K)
>> 5. Problems with SLES 11 (Phil Bayfield)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 6 Sep 2011 23:24:24 +0300
>> From: "Reinis Rozitis"<r at roze.lv>
>> Subject: Re: [Gluster-users] Reading directly from brick
>> To:<gluster-users at gluster.org>
>> Message-ID:<F7DAC991835C44889BCDB281F977B692 at NeiRoze>
>> Content-Type: text/plain; format=flowed; charset="utf-8";
>> reply-type=original
>>
>>> Simple answer - no, it's not ever safe to do writes to an active
>>> Gluster
>>> backend.
>> Question was about reads though and then the answer is it is
>> perfectly fine
>> (and faster) to do reads directly from the filesystem (in replicated
>> setups)
>> if you keep in mind that by doing so you lose the Glusters autoheal
>> eature - eg if one of the gluster nodes goes down and there is a file
>> written meanwhile when the server comes up if you access the file
>> directly
>> it won't show up while it would when accessing it via the gluster mount
>> point (you can work arround it by manually triggering the self heal).
>>
>>
>>> I've heard that reads from glusterfs are around 20 times slower than
>>> from
>>> ext3:
>> "20 times" might be fetched out of thin air but of course there is a
>> significant overhead of serving a file from a gluster which basically
>> involves network operations and additional meta data checks versus
>> fetching
>> the file directly from iron.
>>
>>
>> rr
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Tue, 6 Sep 2011 14:46:28 -0700
>> From: Di Pe<dipeit at gmail.com>
>> Subject: Re: [Gluster-users] NFS secondary groups not working.
>> To: gluster-users at gluster.org
>> Message-ID:
>> <CAB9T+o+fAb+YasVxMsUsVmMw0Scp3BLSqc0Y_grusRmV11qejg at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> Anand, has this issue been confirmed by gluster and is it in the pipe
>> to get fixed or do you need .additional information? We are no gluster
>> experts but are happy to help if we know who to provide additional
>> debugging info.
>>
>> On Mon, Aug 29, 2011 at 9:44 AM, Mike Hanby<mhanby at uab.edu> wrote:
>>> I just noticed the problem happening on one client in our
>>> environment (clients and servers running 3.2.2), other clients work
>>> fine.
>>>
>>> The clients and servers are all CentOS 5.6 x86_64
>>>
>>> I get the same permission denied using Gluster FUSE and Gluster NFS
>>> mounts on this client.
>>>
>>> I'm not mounting it with ACL.
>>>
>>> The volume is a simple distributed volume with two servers.
>>>
>>>> -----Original Message-----
>>>> From: gluster-users-bounces at gluster.org [mailto:gluster-users-
>>>> bounces at gluster.org] On Behalf Of Hubert-Jan Schaminee
>>>> Sent: Saturday, August 27, 2011 10:10 AM
>>>> To: Anand Avati
>>>> Cc: gluster-users at gluster.org
>>>> Subject: Re: [Gluster-users] NFS secondary groups not working.
>>>>
>>>> Op zaterdag 13-08-2011 om 20:22 uur [tijdzone +0530], schreef Anand
>>>> Avati:
>>>>>
>>>>> On Sat, Aug 13, 2011 at 5:29 PM, Dipeit<dipeit at gmail.com> wrote:
>>>>> ? ? ? ? We noticed this bug too using the gluster client. I'm
>>>>> ? ? ? ? surprised that not more people noticed this lack of posix
>>>>> ? ? ? ? compliance. This makes gluster really unusable in multiuser
>>>>> ? ? ? ? environments. Is that because gluster is mostly used in large
>>>>> ? ? ? ? web farms like pandora?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> GlusterFS is POSIX compliant w.r.t user groups. We have not seen this
>>>>> issue in our testing. Can you give more info about your setup? Have
>>>>> you mounted with -o acl or without? Anything unusual in the logs?
>>>>>
>>>>>
>>>>> Avati
>>>> I'm having the same problem here.
>>>>
>>>> I use the latest version (3.2.3 build on Aug 23 2011 19:54:51 of the
>>>> download site) on a Centos 5.6 as a gluster servers, Debian squeeze
>>>> (same version) as client.
>>>> I'm refused access to files and directories despite having correct
>>>> group permissions.
>>>>
>>>> So I installed a clean Centos client (also latest version) for a test
>>>> and everything is working perfectly .... ?
>>>>
>>>> The used Debian (squeeze) and Centos are 64 bits (repository from
>>>> gluster.com).
>>>> Using Debian testing (64 and 32 bits) and gluster from the Debian
>>>> repository also denies me access in 64 and 32 bits version.
>>>>
>>>> I assume the mixed environment explains why this bug is rare.
>>>>
>>>> The used gluster installation is a basic replicated setup one with two
>>>> servers like described in the de Gluster docs.
>>>>
>>>>
>>>> Hubert-Jan Schamin?e
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Tue, 6 Sep 2011 18:52:52 -0400
>> From: Anthony Delviscio<adelviscio at gmail.com>
>> Subject: [Gluster-users] Inconsistent md5sum of replicated file
>> To: gluster-users at gluster.org
>> Message-ID:
>> <CAKE0inQy3Tjf3TB11kc+F_F-P7kN2CJ+eG+2FaRUxOe4tnzgwQ at mail.gmail.com>
>> Content-Type: text/plain; charset="windows-1252"
>>
>> I was wondering if anyone would be able to shed some light on how a file
>> could end up with inconsistent md5sums on Gluster backend storage.
>>
>>
>>
>> Our configuration is running on Gluster v3.1.5 in a distribute-replicate
>> setup consisting of 8 bricks.
>>
>> Our OS is Red Hat 5.6 x86_64. Backend storage is an ext3 RAID 5.
>>
>>
>>
>> The 8 bricks are in RR DNS and are mounted for reading/writing via NFS
>> automounts.
>>
>>
>>
>> When comparing md5sums of the file from two different NFS clients,
>> they were
>> different.
>>
>>
>>
>> The extended attributes of the files on backend storage are
>> identical. The
>> file size and permissions are identical. The stat data (excluding
>> inode on
>> backend storage file system) is identical.
>>
>> However, running md5sum on the two files, results in two different
>> md5sums.
>>
>>
>>
>> Copying both files to another location/server and running the md5sum
>> also
>> results in no change ? they?re still different.
>>
>>
>>
>> Gluster logs do not show anything related to the filename in question.
>> Triggering
>> a self-healing operation didn?t seem to do anything and it may have
>> to do
>> with the fact that the extended attributes are identical.
>>
>>
>>
>> If more information is required, let me know and I will try to
>> accommodate.
>>
>>
>> Thank you
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:<http://gluster.org/pipermail/gluster-users/attachments/20110906/4628faa2/attachment-0001.htm>
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Wed, 7 Sep 2011 14:13:56 +0530
>> From: Pranith Kumar K<pranithk at gluster.com>
>> Subject: Re: [Gluster-users] Inconsistent md5sum of replicated file
>> To: Anthony Delviscio<adelviscio at gmail.com>
>> Cc: gluster-users at gluster.org
>> Message-ID:<4E672ECC.7050703 at gluster.com>
>> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
>>
>> hi Anthony,
>> Could you send the output of the getfattr -d -m . -e hex
>> <filepath> on both the bricks and also the stat output on the both the
>> backends. Give the outputs for its parent directory also.
>>
>> Pranith.
>>
>> On 09/07/2011 04:22 AM, Anthony Delviscio wrote:
>>> I was wondering if anyone would be able to shed some light on how a
>>> file could end up with inconsistent md5sums on Gluster backend storage.
>>>
>>> Our configuration is running on Gluster v3.1.5 in a
>>> distribute-replicate setup consisting of 8 bricks.
>>>
>>> Our OS is Red Hat 5.6 x86_64.Backend storage is an ext3 RAID 5.
>>>
>>> The 8 bricks are in RR DNS and are mounted for reading/writing via NFS
>>> automounts.
>>>
>>> When comparing md5sums of the file from two different NFS clients,
>>> they were different.
>>>
>>> The extended attributes of the files on backend storage are
>>> identical.The file size and permissions are identical.The stat data
>>> (excluding inode on backend storage file system) is identical.
>>>
>>> However, running md5sum on the two files, results in two different
>>> md5sums.
>>>
>>> Copying both files to another location/server and running the md5sum
>>> also results in no change ? they?re still different.
>>>
>>> Gluster logs do not show anything related to the filename in
>>> question.Triggering a self-healing operation didn?t seem to do
>>> anything and it may have to do with the fact that the extended
>>> attributes are identical.
>>>
>>> If more information is required, let me know and I will try to
>>> accommodate.
>>>
>>> Thank you
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:<http://gluster.org/pipermail/gluster-users/attachments/20110907/86d14cab/attachment-0001.htm>
>>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Wed, 7 Sep 2011 10:15:43 +0100
>> From: Phil Bayfield<phil at techlightenment.com>
>> Subject: [Gluster-users] Problems with SLES 11
>> To: gluster-users at gluster.org
>> Message-ID:
>> <CAFXH-fW0DBE9YomJzAtvdFAWaf5Zpq-TfbfTPb+K7gBu-R+06Q at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Hi there,
>>
>> I compiled and installed the latest version of Gluster on a couple of
>> SLES
>> 11 SP1 boxes, everything up to this point seemed ok.
>>
>> I start the daemon on both boxes, and both are listening on 24007.
>>
>> I issue a "gluster peer probe" command on one of the boxes and the
>> daemon
>> instantly dies, I restart it and it shows:
>>
>> # gluster peer status
>> Number of Peers: 1
>>
>> Hostname: mckalcpap02
>> Uuid: 00000000-0000-0000-0000-000000000000
>> State: Establishing Connection (Connected)
>>
>> I attempted to run the probe on the other box, the daemon crashes,
>> now as I
>> start the daemon on each box the daemon just crashes on the other box.
>>
>> The log output immediately prior to the crash is as follows:
>>
>> [2011-06-07 08:05:10.700710] I
>> [glusterd-handler.c:623:glusterd_handle_cli_probe] 0-glusterd:
>> Received CLI
>> probe req mckalcpap02 24007
>> [2011-06-07 08:05:10.701058] I
>> [glusterd-handler.c:391:glusterd_friend_find]
>> 0-glusterd: Unable to find hostname: mckalcpap02
>> [2011-06-07 08:05:10.701086] I
>> [glusterd-handler.c:3422:glusterd_probe_begin] 0-glusterd: Unable to
>> find
>> peerinfo for host: mckalcpap02 (24007)
>> [2011-06-07 08:05:10.702832] I
>> [glusterd-handler.c:3404:glusterd_friend_add]
>> 0-glusterd: connect returned 0
>> [2011-06-07 08:05:10.703110] I
>> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using
>> Program
>> glusterd clnt mgmt, Num (1238433), Version (1)
>>
>> If I use the IP address the same thing happens:
>>
>> [2011-06-07 08:07:12.873075] I
>> [glusterd-handler.c:623:glusterd_handle_cli_probe] 0-glusterd:
>> Received CLI
>> probe req 10.9.54.2 24007
>> [2011-06-07 08:07:12.873410] I
>> [glusterd-handler.c:391:glusterd_friend_find]
>> 0-glusterd: Unable to find hostname: 10.9.54.2
>> [2011-06-07 08:07:12.873438] I
>> [glusterd-handler.c:3422:glusterd_probe_begin] 0-glusterd: Unable to
>> find
>> peerinfo for host: 10.9.54.2 (24007)
>> [2011-06-07 08:07:12.875046] I
>> [glusterd-handler.c:3404:glusterd_friend_add]
>> 0-glusterd: connect returned 0
>> [2011-06-07 08:07:12.875280] I
>> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using
>> Program
>> glusterd clnt mgmt, Num (1238433), Version (1)
>>
>> There is no firewall issue:
>>
>> # telnet mckalcpap02 24007
>> Trying 10.9.54.2...
>> Connected to mckalcpap02.
>> Escape character is '^]'.
>>
>> Following restart (which crashes the other node) the log output is as
>> follows:
>>
>> [2011-06-07 08:10:09.616486] I [glusterd.c:564:init] 0-management: Using
>> /etc/glusterd as working directory
>> [2011-06-07 08:10:09.617619] C [rdma.c:3933:rdma_init]
>> 0-rpc-transport/rdma:
>> Failed to get IB devices
>> [2011-06-07 08:10:09.617676] E [rdma.c:4812:init] 0-rdma.management:
>> Failed
>> to initialize IB Device
>> [2011-06-07 08:10:09.617700] E [rpc-transport.c:741:rpc_transport_load]
>> 0-rpc-transport: 'rdma' initialization failed
>> [2011-06-07 08:10:09.617724] W [rpcsvc.c:1288:rpcsvc_transport_create]
>> 0-rpc-service: cannot create listener, initing the transport failed
>> [2011-06-07 08:10:09.617830] I [glusterd.c:88:glusterd_uuid_init]
>> 0-glusterd: retrieved UUID: 1e344f5d-6904-4d14-9be2-8f0f44b97dd7
>> [2011-06-07 08:10:11.258098] I
>> [glusterd-handler.c:3404:glusterd_friend_add]
>> 0-glusterd: connect returned 0
>> Given volfile:
>> +------------------------------------------------------------------------------+
>>
>> 1: volume management
>> 2: type mgmt/glusterd
>> 3: option working-directory /etc/glusterd
>> 4: option transport-type socket,rdma
>> 5: option transport.socket.keepalive-time 10
>> 6: option transport.socket.keepalive-interval 2
>> 7: end-volume
>> 8:
>>
>> +------------------------------------------------------------------------------+
>>
>> [2011-06-07 08:10:11.258431] I
>> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using
>> Program
>> glusterd clnt mgmt, Num (1238433), Version (1)
>> [2011-06-07 08:10:11.280533] W
>> [socket.c:1494:__socket_proto_state_machine]
>> 0-socket.management: reading from socket failed. Error (Transport
>> endpoint
>> is not connected), peer (10.9.54.2:1023)
>> [2011-06-07 08:10:11.280595] W
>> [socket.c:1494:__socket_proto_state_machine]
>> 0-management: reading from socket failed. Error (Transport endpoint
>> is not
>> connected), peer (10.9.54.2:24007)
>> [2011-06-07 08:10:17.256235] E [socket.c:1685:socket_connect_finish]
>> 0-management: connection to 10.9.54.2:24007 failed (Connection refused)
>>
>> There are no logs on the node which crashes.
>>
>> I've tried various possibly solutions from searching the net but got
>> getting
>> anywhere, can anyone advise how to proceed?
>>
>> Thanks,
>> Phil.
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list