[Gluster-users] 42 node gluster volume create fails silently

[Prasad, Nirmal]

3.4.2 - tracing out the problem on other nodes. I think they had something left out partially. The probe and addition process could definitely use some speed - plumbing should be quick - fun is in the data ....

Cleared out with - 

setfattr -x trusted.glusterfs.volume-id <mount>
setfattr -x trusted.gfid <mount>


-----Original Message-----
From: Joe Julian [mailto:joe at julianfamily.org] 
Sent: Monday, March 31, 2014 7:06 PM
To: Prasad, Nirmal
Subject: Re: [Gluster-users] 42 node gluster volume create fails silently

What version?

On 03/31/2014 03:59 PM, Prasad, Nirmal wrote:
> Ok - for some reason it did not like 6 of my nodes - but able to add 34 nodes two at a time - may be client can do the similar split internally based on replica count. The failure from add-brick is simple "volume add-brick: failed: "
>
> gluster volume info
>
> Volume Name: gl_disk
> Type: Distributed-Replicate
> Volume ID: c70d525e-a255-41e2-af03-718d6dec0319
> Status: Created
> Number of Bricks: 17 x 2 = 34
> Transport-type: tcp
>
> -----Original Message-----
> From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Prasad, Nirmal
> Sent: Monday, March 31, 2014 6:20 PM
> To: Dan Lambright
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] 42 node gluster volume create fails silently
>
> Hi Dan,
>
> Thanks for the quick response. I'm trying to create the volume so have not reached this stage - the client exited out when I gave it :
>
> gluster volume create <vol-name> replica 2 server1:.. server2:.. .... server41:.. server42:..
>
> If I do :
>
> gluster volume create <vol-name> replica 2 server1:.. server2:..
> gluster volume add-brick <vol-name> replica 2 server3:.. server4:..
>
> it gets me farther ... looks like there is some timeout for the gluster command - not sure - just an observation.
>
> Thanks
> Regards
> Nirmal
> -----Original Message-----
> From: Dan Lambright [mailto:dlambrig at redhat.com]
> Sent: Monday, March 31, 2014 6:16 PM
> To: Prasad, Nirmal
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] 42 node gluster volume create fails silently
>
> Hello,
>
> the cli logs do not contain much. If you remount your gluster volume, and create the problem again, there may be more to see.
>
> On the client side:
>
> mount -t glusterfs  -o log-level=DEBUG,log-file=/tmp/my_client.log 10.16.159.219:/myvol /mnt
>
> On the server side:
>
> gluster volume set myvol diagnostics.brick-sys-log-level WARNING gluster volume set myvol diagnostics.brick-log-level WARNING
>
> You could then attach the most recent log files to your email, or the parts that seem relevant so the email is not too large.
>   
> /tmp/my_client.log
> /var/log/glusterfs/etc*.log
> /var/log/glusterfs/bricks/*.log
>
> ----- Original Message -----
> From: "Nirmal Prasad" <nprasad at idirect.net>
> To: gluster-users at gluster.org
> Sent: Monday, March 31, 2014 6:04:31 PM
> Subject: Re: [Gluster-users] 42 node gluster volume create fails silently
>
>
>
> .. and glusterd died. I had success adding individually up to 21 nodes – will go down that path. Anyone interested in log files or core files?
>
>
>
> service glusterd status
>
> glusterd dead but pid file exists
>
>
>
>
> From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Prasad, Nirmal
> Sent: Monday, March 31, 2014 5:57 PM
> To: gluster-users at gluster.org
> Subject: Re: [Gluster-users] 42 node gluster volume create fails silently
>
>
>
>
> Looks symptomatic of some timeout – subsequent status command gave:
>
>
>
> gluster volume status
>
> Another transaction is in progress. Please try again after sometime.
>
>
>
>
> From: gluster-users-bounces at gluster.org [ mailto:gluster-users-bounces at gluster.org ] On Behalf Of Prasad, Nirmal
> Sent: Monday, March 31, 2014 5:53 PM
> To: gluster-users at gluster.org
> Subject: [Gluster-users] 42 node gluster volume create fails silently
>
>
>
>
> Not much of output – not sure where to see. This is the output in the cli.log – There are 42 servers and (21 brick pairs) – timeout perhaps ??
>
>
>
> [2014-03-31 13:44:34.228467] I [cli-cmd-volume.c:1336:cli_check_gsync_present] 0-: geo-replication not installed
>
> [2014-03-31 13:44:34.229619] I [cli-cmd-volume.c:392:cli_cmd_volume_create_cbk] 0-cli: Replicate cluster type found. Checking brick order.
>
> [2014-03-31 13:44:34.230821] I [cli-cmd-volume.c:304:cli_cmd_check_brick_order] 0-cli: Brick order okay
>
> [2014-03-31 13:44:47.758977] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
>
> [2014-03-31 13:44:47.763286] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled
>
> [2014-03-31 13:44:47.763326] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread
>
> [2014-03-31 13:44:47.777000] I [cli-cmd-volume.c:1336:cli_check_gsync_present] 0-: geo-replication not installed
>
> [2014-03-31 13:44:47.780574] I [cli-rpc-ops.c:332:gf_cli_list_friends_cbk] 0-cli: Received resp to list: 0
>
> [2014-03-31 13:44:47.782086] I [input.c:36:cli_batch] 0-: Exiting with: 0
>
> [2014-03-31 13:46:34.231761] I [input.c:36:cli_batch] 0-: Exiting with: 110
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users