[Gluster-users] glusterfsd not starting at boot with Debian 7

Thu Feb 5 21:01:22 UTC 2015

As promised here the log file of the brick itself:

[2015-02-05 20:38:41.614491] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.5.3 (/usr/sbin/glusterfsd -s gfs1a --volfile-id myvol.gfs1a.data-myvol-brick1 -p /var/lib/glusterd/vols/myvol/run/gfs1a-data-myvol-brick1.pid -S /var/run/23669fa92fc09b0dd09f4f997f6a3adf.socket --brick-name /data/myvol/brick1 -l /var/log/glusterfs/bricks/data-myvol-brick1.log --xlator-option *-posix.glusterd-uuid=459ca0c5-8400-4392-9846-fcef98772f44 --brick-port 49153 --xlator-option myvol-server.listen-port=49153)
[2015-02-05 20:38:41.616083] I [socket.c:3645:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
[2015-02-05 20:38:41.616111] I [socket.c:3660:socket_init] 0-socket.glusterfsd: using system polling thread
[2015-02-05 20:38:41.616184] I [socket.c:3645:socket_init] 0-glusterfs: SSL support is NOT enabled
[2015-02-05 20:38:41.616193] I [socket.c:3660:socket_init] 0-glusterfs: using system polling thread
[2015-02-05 20:38:41.616234] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2015-02-05 20:38:41.616242] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1a
[2015-02-05 20:38:41.616273] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1a (Success)
[2015-02-05 20:38:41.616283] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2015-02-05 20:38:41.616382] W [glusterfsd.c:1095:cleanup_and_exit] (-->/usr/sbin/glusterfsd(glusterfs_mgmt_init+0x16d) [0x7f0d393ac31d] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_start+0x12) [0x7f0d3815c492] (-->/usr/sbin/glusterfsd(+0xd588) [0x7f0d393ac588]))) 0-: received signum (1), shutting down
[2015-02-05 20:38:41.616435] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2015-02-05 20:38:41.616449] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1a
[2015-02-05 20:38:41.616455] I [socket.c:3143:socket_submit_request] 0-glusterfs: not connected (priv->connected = -1)
[2015-02-05 20:38:41.616460] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (XID: 0x1 Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)

Looks like a DNS issue to me. I have to mention here that I use gfs1a as hostname in my brick but do not have it into my DNS server, I just added it to my /etc/hosts file which in theory should be enough.

Also note here that I am using ZFS and looking at the startup scripts of rc2.d ZFS gets started before glusterfs so the device should be available when gluster starts.

On Thursday, February 5, 2015 5:03 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:

On 02/05/2015 06:46 PM, ML mail wrote:
> Dear Pranith
>
> As asked you will find below the glusterd logfile including the log entries at the boot of the server. On Debian the logfile is located in the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file.
>
>
> [2015-02-05 13:13:03.566059] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.3 (/usr/sbin/glusterd -p /var/run/glusterd.pid)
> [2015-02-05 13:13:03.580131] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory
> [2015-02-05 13:13:03.583040] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled
> [2015-02-05 13:13:03.583060] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread
> [2015-02-05 13:13:03.585326] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
> [2015-02-05 13:13:03.585343] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device
> [2015-02-05 13:13:03.585350] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
> [2015-02-05 13:13:03.585392] W [rpcsvc.c:1535:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
> [2015-02-05 13:13:03.585446] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled
> [2015-02-05 13:13:03.585454] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread
> [2015-02-05 13:13:06.111967] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30501
> [2015-02-05 13:13:06.114871] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
> [2015-02-05 13:13:06.277826] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 459ca0c5-8400-4392-9846-fcef98772f44
> Final graph:
> +------------------------------------------------------------------------------+
> 1: volume management
> 2:     type mgmt/glusterd
> 3:     option rpc-auth.auth-glusterfs on
> 4:     option rpc-auth.auth-unix on
> 5:     option rpc-auth.auth-null on
> 6:     option transport.socket.listen-backlog 128
> 7:     option rpc-auth-allow-insecure on
> 8:     option transport.socket.read-fail-log off
> 9:     option transport.socket.keepalive-interval 2
> 10:     option transport.socket.keepalive-time 10
> 11:     option transport-type rdma
> 12:     option working-directory /var/lib/glusterd
> 13: end-volume
> 14:
> +------------------------------------------------------------------------------+
> [2015-02-05 13:13:06.281802] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
> [2015-02-05 13:13:06.281889] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled
> [2015-02-05 13:13:06.281902] I [socket.c:3660:socket_init] 0-management: using system polling thread
> [2015-02-05 13:13:06.282359] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now
> [2015-02-05 13:13:06.285676] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
> [2015-02-05 13:13:06.285770] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled
> [2015-02-05 13:13:06.285786] I [socket.c:3660:socket_init] 0-management: using system polling thread
> [2015-02-05 13:13:06.285982] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1a:/data/myvol/brick
> [2015-02-05 13:13:06.286042] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now
Log above suggests that glusterd launched the brick but it failed to 
come up. Let us see the logfile of the brick as well. It should be 
present in

/var/log/glusterfs/bricks

Pranith

>
>
>
>
>
> On Thursday, February 5, 2015 12:02 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:
>
> On 02/05/2015 04:08 AM, ML mail wrote:
>> Hi,
>>
>> I have installed Gluster 3.5.3 on Debian 7 and have one single test volume right now. Unfortunately after a reboot this volume does not get get started automatically: the glusterfsd process for that volume is inexistant although the glusterd process is running.
>>
>>
>> After a boot running "gluster volume status" shows:
>> Status of volume: myvol
>> Gluster process                        Port    Online    Pid
>> ------------------------------------------------------------------------------
>> Brick gfs1a:/data/myvol/brick                N/A    N    N/A
>>
>> Task Status of Volume myvol
>> ------------------------------------------------------------------------------
>>
>> There are no active volume tasks
>>
>> as you can see all N/A there. In order to make it work I have to run "gluster volume stop myvol" and "gluster volume start myvol". Then it is ready and working. What could the problem be? Is the init.d startup srcript on Debian 7 buggy?
> If glusterd is starting then there shouldn't be any bug in the init
> scripts. Could you attach the log file of glusterd? I am not sure where
> it is stored in debian may be /var/log/glusterfs or
> /usr/local/var/log/glusterfs it will have glusterd in the name of the file.
>
> Pranith
>
>> Best regards
>> ML
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users