[Gluster-users] Unexpected behaviour adding a third server

[Pranith Kumar Karampuri]

On 01/23/2016 02:17 PM, Steve Spence wrote:
> We've a simple two-server one volume arrangement, replicating ~340k 
> files (15GB) between our web servers.
>
> The servers are in AWS, sat in different availability zones. One of 
> the operations for this weekend is to add another pair of machines, 
> one in each AZ.
>
> I've deployed the same OS image of the gluster server (3.6) and was 
> under the impression I could add a brick to the existing replica 
> simply by issuing the below:
>
> gluster volume add-brick volume1 replica 3 pd-wfe3:/gluster-store
>
> And then presumably would add the fourth server by repeating the above 
> with "replica 4" and the fourth server name.
>
> The operation appeared to succeed, the brick appears alongside the others:
>
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: pd-wfe1:/gluster-store
> Brick2: pd-wfe2:/gluster-store
> Brick3: pd-wfe3:/gluster-store
>
> but almost immediately pd-wfe1 crept up to 100% CPU with the gluster 
> processes, and nginx began timing out serving content from the volume.

Could you disable client-side healing?
"gluster volume set <volname> cluster.entry-self-heal off
"gluster volume set <volname> cluster.data-self-heal off
"gluster volume set <volname> cluster.metadata-self-heal off

We are in the process of making this experience smooth for 3.8. by 
introducing throttling of self-heal traffic, automatic healing.
+Anuradha,
        Could you give him the steps he need to perform after doing 
add-brick until the patch you sent is merged?

Pranith
>
> The glusterfs-glusterd-vol log is filled with this error at pd-wfe1:
>
> [2016-01-23 08:43:28.459215] W [socket.c:620:__socket_rwv] 
> 0-management: readv on 
> /var/run/c8bc2f99e7584cb9cf077c4f98d1db2e.socket failed (Invalid argument)
>
> while I see this error for the log named by the mount point:
>
> [2016-01-23 08:43:28.986379] W 
> [client-rpc-fops.c:306:client3_3_mkdir_cbk] 2-volume1-client-2: remote 
> operation failed: Permission denied. Path: (null)
>
> Does anyone have any suggestions how to proceed? I would appreciate 
> any input on this one.
>
> Steve
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160125/a1b305b3/attachment.html>