[Gluster-users] add bricks on distributed replicated volume failed
Laurent DOUCHY
Laurent.Douchy at unige.ch
Thu Sep 1 18:01:06 UTC 2011
see below.
Cheers,
Laurent DOUCHY.
System Administrator
ISDC Data Centre for Astrophysics
16, ch. d'Ecogia
CH-1290 VERSOIX
Tel.: +41 (0)22 379 21 31
On 9/1/11 7:11 PM, Mohit Anchlia wrote:
> Can you paste the error logs from node05 and the node you are using to
> perform add.
the one I'm using to perform the add:
[2011-09-01 19:44:13.903186] I
[glusterd-handler.c:1305:glusterd_handle_add_brick] 0-glusterd: Received
add brick req
[2011-09-01 19:44:13.903274] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 19:44:13.903284] I
[glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired
local lock
[2011-09-01 19:44:13.903795] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd:
Received ACC from uuid: dd114546-5b94-4a62-9301-260703bf5707
[2011-09-01 19:44:13.903834] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd:
Received ACC from uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
[2011-09-01 19:44:13.904069] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd:
Received ACC from uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
[2011-09-01 19:44:13.904167] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd:
Received ACC from uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
[2011-09-01 19:44:13.904188] I
[glusterd-rpc-ops.c:742:glusterd3_1_cluster_lock_cbk] 0-glusterd:
Received ACC from uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
[2011-09-01 19:44:13.904214] I
[glusterd-utils.c:776:glusterd_volume_brickinfo_get_by_brick] 0-: brick:
node05:/gluster2
[2011-09-01 19:44:13.904231] I
[glusterd-utils.c:776:glusterd_volume_brickinfo_get_by_brick] 0-: brick:
node06:/gluster2
[2011-09-01 19:44:13.904305] I
[glusterd-op-sm.c:6453:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op
req to 5 peers
[2011-09-01 19:44:13.904670] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received
ACC from uuid: dd114546-5b94-4a62-9301-260703bf5707
[2011-09-01 19:44:13.904703] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received
ACC from uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
[2011-09-01 19:44:13.904725] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received
ACC from uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
[2011-09-01 19:44:13.905542] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received
ACC from uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
[2011-09-01 19:44:13.905655] I
[glusterd-rpc-ops.c:1040:glusterd3_1_stage_op_cbk] 0-glusterd: Received
RJT from uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
[2011-09-01 19:44:13.905929] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd:
Received ACC from uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
[2011-09-01 19:44:13.905988] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd:
Received ACC from uuid: dd114546-5b94-4a62-9301-260703bf5707
[2011-09-01 19:44:13.906019] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd:
Received ACC from uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
[2011-09-01 19:44:13.906039] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd:
Received ACC from uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
[2011-09-01 19:44:13.906058] I
[glusterd-rpc-ops.c:801:glusterd3_1_cluster_unlock_cbk] 0-glusterd:
Received ACC from uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
[2011-09-01 19:44:13.906070] I
[glusterd-op-sm.c:6987:glusterd_op_txn_complete] 0-glusterd: Cleared
local lock
[2011-09-01 19:44:13.906677] W
[socket.c:1494:__socket_proto_state_machine] 0-socket.management:
reading from socket failed. Error (Transport endpoint is not connected),
peer (127.0.0.1:1016)
node05:
[2011-09-01 20:57:03.419202] I
[glusterd-handler.c:448:glusterd_handle_cluster_lock] 0-glusterd:
Received LOCK from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 20:57:03.419292] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 20:57:03.419320] I
[glusterd-handler.c:2641:glusterd_op_lock_send_resp] 0-glusterd:
Responded, ret: 0
[2011-09-01 20:57:03.419818] I
[glusterd-handler.c:488:glusterd_req_ctx_create] 0-glusterd: Received op
from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 20:57:03.419896] I
[glusterd-utils.c:776:glusterd_volume_brickinfo_get_by_brick] 0-: brick:
node05:/gluster2
[2011-09-01 20:57:03.420141] E
[glusterd-op-sm.c:715:glusterd_op_stage_add_brick] 0-glusterd: resolve
brick failed
[2011-09-01 20:57:03.420157] E
[glusterd-op-sm.c:7107:glusterd_op_ac_stage_op] 0-: Validate failed: 1
[2011-09-01 20:57:03.420183] I
[glusterd-handler.c:2733:glusterd_op_stage_send_resp] 0-glusterd:
Responded to stage, ret: 0
[2011-09-01 20:57:03.420508] I
[glusterd-handler.c:2683:glusterd_handle_cluster_unlock] 0-glusterd:
Received UNLOCK from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 20:57:03.420553] I
[glusterd-handler.c:2661:glusterd_op_unlock_send_resp] 0-glusterd:
Responded to unlock, ret: 0
[2011-09-01 21:43:43.584850] I
[glusterd-handler.c:448:glusterd_handle_cluster_lock] 0-glusterd:
Received LOCK from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 21:43:43.585045] I [glusterd-utils.c:243:glusterd_lock]
0-glusterd: Cluster lock held by a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 21:43:43.585077] I
[glusterd-handler.c:2641:glusterd_op_lock_send_resp] 0-glusterd:
Responded, ret: 0
[2011-09-01 21:43:43.585341] I
[glusterd-handler.c:488:glusterd_req_ctx_create] 0-glusterd: Received op
from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 21:43:43.585424] I
[glusterd-utils.c:776:glusterd_volume_brickinfo_get_by_brick] 0-: brick:
node05:/gluster2
[2011-09-01 21:43:43.586524] E
[glusterd-op-sm.c:715:glusterd_op_stage_add_brick] 0-glusterd: resolve
brick failed
[2011-09-01 21:43:43.586540] E
[glusterd-op-sm.c:7107:glusterd_op_ac_stage_op] 0-: Validate failed: 1
[2011-09-01 21:43:43.586567] I
[glusterd-handler.c:2733:glusterd_op_stage_send_resp] 0-glusterd:
Responded to stage, ret: 0
[2011-09-01 21:43:43.586873] I
[glusterd-handler.c:2683:glusterd_handle_cluster_unlock] 0-glusterd:
Received UNLOCK from uuid: a35fb0a1-af35-4a04-b38a-434f68369508
[2011-09-01 21:43:43.586910] I
[glusterd-handler.c:2661:glusterd_op_unlock_send_resp] 0-glusterd:
Responded to unlock, ret: 0
I have to fix the time issues.
I fix the time issues but steel the same message
[root at node07 ~]# gluster volume add-brick cluster node05:/gluster2
node06:/gluster2
Operation failed on node05
> Last thing you could try is gluster peer detach node 5 and 6 and then
> add them back and try again.
I've done the detach, restart glusterd, probe, check status, restart
glusterd, check status. everything ok ... but:
[root at node07 ~]# gluster volume add-brick cluster node05:/gluster2
node06:/gluster2
Operation failed on node05
>
> On Thu, Sep 1, 2011 at 9:58 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch> wrote:
>> ping is ok
>>
>> restart of gluster done :
>> [root at node00 ~]# for i in `seq -w 1 10` ; do echo ; echo node$i ; echo ; ssh
>> node$i "service glusterd restart";done
>>
>> node01
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node02
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node03
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node04
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node05
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node06
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node07
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node08
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node09
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>> node10
>>
>> Stopping glusterd:[ OK ]
>> Starting glusterd:[ OK ]
>>
>>
>> but same error message ...
>>
>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster2
>> node06:/gluster2
>> Operation failed on node05
>>
>> Cheers,
>> Laurent DOUCHY.
>>
>>
>> On 9/1/11 6:54 PM, Mohit Anchlia wrote:
>>> Can you ping node05 from node07 from where you are trying to do the
>>> add? Also, try restarting gluster process on every node and try again.
>>>
>>> On Thu, Sep 1, 2011 at 9:39 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch>
>>> wrote:
>>>> see below
>>>>
>>>> Cheers,
>>>> Laurent DOUCHY.
>>>>
>>>>
>>>> On 9/1/11 6:01 PM, Mohit Anchlia wrote:
>>>>> You can check few things on 5 and 6:
>>>>>
>>>>> 1) gluster processes are running on node5 and 6
>>>> yes:
>>>>
>>>> node05
>>>>
>>>> root 4902 1 0 Aug31 ? 00:00:00
>>>> /opt/glusterfs/3.2.2/sbin/glusterd
>>>> root 9626 1 0 19:55 ? 00:00:00
>>>> /opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol
>>>> -p
>>>> /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
>>>> root 9690 9686 0 20:04 ? 00:00:00 bash -c ps -edf | grep
>>>> gluster
>>>> root 9704 9690 0 20:04 ? 00:00:00 grep gluster
>>>>
>>>> node06
>>>>
>>>> root 4441 1 0 Aug31 ? 00:00:00
>>>> /opt/glusterfs/3.2.2/sbin/glusterd
>>>> root 9178 1 0 19:55 ? 00:00:00
>>>> /opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol
>>>> -p
>>>> /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
>>>> root 9242 9238 0 20:04 ? 00:00:00 bash -c ps -edf | grep
>>>> gluster
>>>> root 9256 9242 0 20:04 ? 00:00:00 grep gluster
>>>>
>>>>> 2) both nodes are able to see each other
>>>> yes:
>>>>
>>>> [root at node05 ~]# ping node06
>>>> PING node06.isdc.unige.ch (129.194.168.70) 56(84) bytes of data.
>>>> 64 bytes from node06.isdc.unige.ch (129.194.168.70): icmp_seq=1 ttl=64
>>>> time=0.376 ms
>>>>
>>>> [root at node06 ~]# ping node05
>>>> PING node05.isdc.unige.ch (129.194.168.69) 56(84) bytes of data.
>>>> 64 bytes from node05.isdc.unige.ch (129.194.168.69): icmp_seq=1 ttl=64
>>>> time=0.337 ms
>>>>> 3) do gluster peer status on both the nodes and see what you see
>>>> node 5 trust node 6 and node 6 trust node 5
>>>>
>>>> [root at node05 ~]# gluster peer status
>>>> Number of Peers: 5
>>>>
>>>> Hostname: node08
>>>> Uuid: dd114546-5b94-4a62-9301-260703bf5707
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: node06
>>>> Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: node10
>>>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: 129.194.168.71
>>>> Uuid: a35fb0a1-af35-4a04-b38a-434f68369508
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: node09
>>>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
>>>> State: Peer in Cluster (Connected)
>>>>
>>>>
>>>>
>>>>
>>>> [root at node06 ~]# gluster peer status
>>>> Number of Peers: 5
>>>>
>>>> Hostname: node08
>>>> Uuid: dd114546-5b94-4a62-9301-260703bf5707
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: node09
>>>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: node05
>>>> Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: node10
>>>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: 129.194.168.71
>>>> Uuid: a35fb0a1-af35-4a04-b38a-434f68369508
>>>> State: Peer in Cluster (Connected)
>>>>
>>>>
>>>>> 4) check iptables
>>>> same file on each node (the installation is manage by puppet)
>>>>> On Thu, Sep 1, 2011 at 8:57 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch>
>>>>> wrote:
>>>>>> It works ...
>>>>>>
>>>>>> [root at node07 ~]# gluster volume add-brick cluster node09:/gluster3
>>>>>> node10:/gluster3
>>>>>> Add Brick successful
>>>>>>
>>>>>>
>>>>>> On 9/1/11 5:39 PM, Mohit Anchlia wrote:
>>>>>>> Can you try with node09:/gluster3 and node10:gluster3 instead?
>>>>>>>
>>>>>>> On Thu, Sep 1, 2011 at 2:49 AM, Laurent
>>>>>>> DOUCHY<Laurent.Douchy at unige.ch>
>>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I work on the node7 so it doesn't appear on the list.
>>>>>>>>
>>>>>>>> I create a folder /gluster3 on node5 and node6 and try to add them to
>>>>>>>> my
>>>>>>>> volume but it failed with the same message :(
>>>>>>>>
>>>>>>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster3
>>>>>>>> node06:/gluster3
>>>>>>>> Operation failed on node05
>>>>>>>>
>>>>>>>> next step is to reinstall from scratch the node I hope I can avoid
>>>>>>>> this.
>>>>>>>>
>>>>>>>> On 8/31/11 9:08 PM, Mohit Anchlia wrote:
>>>>>>>>> I don't see node07 in above output of gluster peer status.
>>>>>>>>>
>>>>>>>>> Can you try to add bricks on the hosts that gluster1, gluster2? So
>>>>>>>>> add
>>>>>>>>> gluster3 and see if that works.
>>>>>>>>>
>>>>>>>>> On Wed, Aug 31, 2011 at 11:56 AM, Laurent DOUCHY
>>>>>>>>> <Laurent.Douchy at unige.ch> wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I try to add 2 bricks or 4 bricks for the same effect.
>>>>>>>>>>
>>>>>>>>>> I try to reinstall gluster without success.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Laurent DOUCHY.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 8/31/11 8:07 PM, Burnash, James wrote:
>>>>>>>>>>> Hi Laurent.
>>>>>>>>>>>
>>>>>>>>>>> Since your configuration specifies replication, you must add
>>>>>>>>>>> bricks
>>>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>> same number as your number of replicas.
>>>>>>>>>>>
>>>>>>>>>>> For instance - if you have 2 replicas (most normal case), you
>>>>>>>>>>> would
>>>>>>>>>>> need
>>>>>>>>>>> to do something like this:
>>>>>>>>>>>
>>>>>>>>>>> gluster volume add-brick cluster node05:/gluster1 node06:/gluster1
>>>>>>>>>>>
>>>>>>>>>>> James Burnash
>>>>>>>>>>> Unix Engineer
>>>>>>>>>>> Knight Capital Group
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: gluster-users-bounces at gluster.org
>>>>>>>>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Laurent
>>>>>>>>>>> DOUCHY
>>>>>>>>>>> Sent: Wednesday, August 31, 2011 12:49 PM
>>>>>>>>>>> To: gluster-users at gluster.org
>>>>>>>>>>> Subject: [Gluster-users] add bricks on distributed replicated
>>>>>>>>>>> volume
>>>>>>>>>>> failed
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I'm using gluster 3.2.2 on 10 nodes. Each node have 2x2 TB disk
>>>>>>>>>>> for
>>>>>>>>>>> gluster.
>>>>>>>>>>>
>>>>>>>>>>> I manage to configure a distributed and replicated volume on 4
>>>>>>>>>>> nodes
>>>>>>>>>>> :
>>>>>>>>>>>
>>>>>>>>>>> [root at node07 ~]# gluster volume info cluster
>>>>>>>>>>>
>>>>>>>>>>> Volume Name: cluster
>>>>>>>>>>> Type: Distributed-Replicate
>>>>>>>>>>> Status: Started
>>>>>>>>>>> Number of Bricks: 4 x 2 = 8
>>>>>>>>>>> Transport-type: tcp
>>>>>>>>>>> Bricks:
>>>>>>>>>>> Brick1: node09:/gluster1
>>>>>>>>>>> Brick2: node10:/gluster1
>>>>>>>>>>> Brick3: node09:/gluster2
>>>>>>>>>>> Brick4: node10:/gluster2
>>>>>>>>>>> Brick5: node07:/gluster1
>>>>>>>>>>> Brick6: node08:/gluster1
>>>>>>>>>>> Brick7: node07:/gluster2
>>>>>>>>>>> Brick8: node08:/gluster2
>>>>>>>>>>>
>>>>>>>>>>> But I can't add new nodes to this volume
>>>>>>>>>>>
>>>>>>>>>>> [root at node07 ~]# gluster peer status
>>>>>>>>>>> Number of Peers: 5
>>>>>>>>>>>
>>>>>>>>>>> Hostname: node10
>>>>>>>>>>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>>
>>>>>>>>>>> Hostname: node08
>>>>>>>>>>> Uuid: dd114546-5b94-4a62-9301-260703bf5707
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>>
>>>>>>>>>>> Hostname: node09
>>>>>>>>>>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>>
>>>>>>>>>>> Hostname: node06
>>>>>>>>>>> Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>>
>>>>>>>>>>> Hostname: node05
>>>>>>>>>>> Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster1
>>>>>>>>>>> node06:/gluster1 node05:/gluster2 node06:/gluster2 Operation
>>>>>>>>>>> failed
>>>>>>>>>>> on
>>>>>>>>>>> node05
>>>>>>>>>>>
>>>>>>>>>>> I try to detach nodes 5 and 6, restart glusterd do the probe and
>>>>>>>>>>> the
>>>>>>>>>>> add-brick but still nothing ...
>>>>>>>>>>>
>>>>>>>>>>> Did some one have any idea to fix this ?
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>> Laurent.
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> DISCLAIMER:
>>>>>>>>>>> This e-mail, and any attachments thereto, is intended only for use
>>>>>>>>>>> by
>>>>>>>>>>> the
>>>>>>>>>>> addressee(s) named herein and may contain legally privileged
>>>>>>>>>>> and/or
>>>>>>>>>>> confidential information. If you are not the intended recipient of
>>>>>>>>>>> this
>>>>>>>>>>> e-mail, you are hereby notified that any dissemination,
>>>>>>>>>>> distribution
>>>>>>>>>>> or
>>>>>>>>>>> copying of this e-mail, and any attachments thereto, is strictly
>>>>>>>>>>> prohibited.
>>>>>>>>>>> If you have received this in error, please immediately notify me
>>>>>>>>>>> and
>>>>>>>>>>> permanently delete the original and any copy of any e-mail and any
>>>>>>>>>>> printout
>>>>>>>>>>> thereof. E-mail transmission cannot be guaranteed to be secure or
>>>>>>>>>>> error-free. The sender therefore does not accept liability for any
>>>>>>>>>>> errors or
>>>>>>>>>>> omissions in the contents of this message which arise as a result
>>>>>>>>>>> of
>>>>>>>>>>> e-mail
>>>>>>>>>>> transmission.
>>>>>>>>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group
>>>>>>>>>>> may,
>>>>>>>>>>> at
>>>>>>>>>>> its discretion, monitor and review the content of all e-mail
>>>>>>>>>>> communications.
>>>>>>>>>>> http://www.knight.com
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list