[Gluster-devel] Issues with encrypted management on 3.8.0

Michael Wyraz michael at wyraz.de
Fri Jun 17 14:09:24 UTC 2016


Hi,

I have set up a 3-node-cluster (4 bricks each node, 3 replicas). Without
TLS on management connection, everything is fine. With TLS I get several
errors with different symptomps. Basically the errors causes some
daemons not to start. So e.g. Quota is not working correctly since quota
daemon cannot be started.

My environment is:

    Debian Jessie, "LATEST" glusterfs repository

    On each node in /etc/hosts:
       xxx.xxx.xxx.xxx full.qualified.node.name
    for each node including the local one.

    Certs:
    - Created a self signed CA Cert with XCA /etc/ssl/glusterfs.ca

        Certificate:
            Data:
                Version: 3 (0x2)
                Serial Number: 1 (0x1)
            Signature Algorithm: sha224WithRSAEncryption
                Issuer: CN=cluster1.backups....
                Validity
                    Not Before: Jun 17 00:00:00 2016 GMT
                    Not After : Jun 16 23:59:59 2041 GMT
                Subject: CN=cluster1.backups....
                Subject Public Key Info:
                    Public Key Algorithm: rsaEncryption
                        Public-Key: (4096 bit)
                        Modulus:
                            ...
                X509v3 extensions:
                    X509v3 Basic Constraints: critical
                        CA:TRUE
                    X509v3 Subject Key Identifier:
                       
        FE:BD:92:1D:8D:B5:DB:42:32:7E:BC:A3:EC:15:0D:D3:9F:64:34:69
                    X509v3 Key Usage:
                        Certificate Sign, CRL Sign
                    Netscape Cert Type:
                        SSL CA, S/MIME CA, Object Signing CA
                    Netscape Comment:
                        xca certificate
            Signature Algorithm: sha224WithRSAEncryption
                     ....

    - Created a Cert for each node /etc/ssl/glusterfs.pem

        Certificate:
            Data:
                Version: 3 (0x2)
                Serial Number: 4 (0x4)
            Signature Algorithm: sha256WithRSAEncryption
                Issuer: CN=cluster1.backups....
                Validity
                    Not Before: Jun 17 00:00:00 2016 GMT
                    Not After : Jun 16 23:59:59 2041 GMT
                Subject: CN=c1-m3.cluster1.backups....
                Subject Public Key Info:
                    Public Key Algorithm: rsaEncryption
                        Public-Key: (4096 bit)
                        Modulus:
                            ...
                X509v3 extensions:
                    X509v3 Basic Constraints: critical
                        CA:FALSE
                    X509v3 Subject Key Identifier:
                       
        35:36:9D:37:BC:AA:59:34:94:3D:D9:20:73:17:74:08:CA:AA:9F:FA
                    X509v3 Key Usage:
                        Digital Signature, Non Repudiation, Key Encipherment
                    Netscape Cert Type:
                        SSL Server
                    Netscape Comment:
                        xca certificate
            Signature Algorithm: sha256WithRSAEncryption
                 ...

    - Put the Cert private key to /etc/ssl/glusterfs.key
    - Created 4096 bit dh params to /etc/ssl/dhparam.pem

Here are the corresponding error logs when I start the volume with TLS
enabled from this node (other nodes are similar):


==> /var/log/glusterfs/cli.log <==
[2016-06-17 13:44:22.577033] I [cli.c:730:main] 0-cli: Started running
gluster with version 3.8.0
[2016-06-17 13:44:22.580865] I [socket.c:4047:socket_init] 0-glusterfs:
SSL support for glusterd is ENABLED
[2016-06-17 13:44:22.581855] I [socket.c:4047:socket_init] 0-glusterfs:
SSL support for glusterd is ENABLED
[2016-06-17 13:44:22.654191] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2016-06-17 13:44:22.654277] W [socket.c:696:__socket_rwv] 0-glusterfs:
readv on /var/run/gluster/quotad.socket failed (Invalid argument)

==> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <==
[2016-06-17 13:44:22.668903] W [common-utils.c:1805:gf_string2boolean]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.0/xlator/mgmt/glusterd.so(+0xe94e0)
[0x7fc6ef8ab4e0]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.0/xlator/mgmt/glusterd.so(+0xb14f0)
[0x7fc6ef8734f0]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_string2boolean+0x151)
[0x7fc6f497aeb1] ) 0-management: argument invalid [Invalid argument]
[2016-06-17 13:44:22.723185] I [socket.c:454:ssl_setup_connection]
0-socket.management: peer CN = c1-m1.cluster1.backups.evermind.de
[2016-06-17 13:44:22.735305] I [MSGID: 106143]
[glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick
/vol/vol1/brick1 on port 49152
[2016-06-17 13:44:22.737439] I [rpc-clnt.c:991:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2016-06-17 13:44:22.737547] I [socket.c:4047:socket_init] 0-management:
SSL support for glusterd is ENABLED
[2016-06-17 13:44:22.801368] I [socket.c:454:ssl_setup_connection]
0-socket.management: peer CN = c1-m1.cluster1.backups.evermind.de
[2016-06-17 13:44:22.817745] I [MSGID: 106143]
[glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick
/vol/vol2/brick1 on port 49153
[2016-06-17 13:44:22.820079] I [rpc-clnt.c:991:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2016-06-17 13:44:22.820169] I [socket.c:4047:socket_init] 0-management:
SSL support for glusterd is ENABLED
[2016-06-17 13:44:22.884935] I [socket.c:454:ssl_setup_connection]
0-socket.management: peer CN = c1-m1.cluster1.backups.evermind.de
[2016-06-17 13:44:22.901411] I [MSGID: 106143]
[glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick
/vol/vol3/brick1 on port 49154
[2016-06-17 13:44:22.904488] I [rpc-clnt.c:991:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2016-06-17 13:44:22.904589] I [socket.c:4047:socket_init] 0-management:
SSL support for glusterd is ENABLED
[2016-06-17 13:44:22.968140] I [socket.c:454:ssl_setup_connection]
0-socket.management: peer CN = c1-m1.cluster1.backups.evermind.de
[2016-06-17 13:44:22.984827] I [MSGID: 106143]
[glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick
/vol/vol4/brick1 on port 49155
[2016-06-17 13:44:22.988199] I [rpc-clnt.c:991:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2016-06-17 13:44:22.988299] I [socket.c:4047:socket_init] 0-management:
SSL support for glusterd is ENABLED
[2016-06-17 13:44:24.352019] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already
stopped
[2016-06-17 13:44:24.352102] I [MSGID: 106568]
[glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is
stopped
[2016-06-17 13:44:24.353414] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd
already stopped
[2016-06-17 13:44:24.353460] I [MSGID: 106568]
[glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd
service is stopped
[2016-06-17 13:44:24.353523] I [MSGID: 106567]
[glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting
glustershd service
[2016-06-17 13:44:24.359687] W [socket.c:3131:socket_connect]
0-glustershd: Ignore failed connection attempt on
/var/run/gluster/6fe8cafd75bf10ffd386275f0bd67a06.socket, (No such file
or directory)
[2016-06-17 13:44:24.362092] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad
already stopped
[2016-06-17 13:44:24.362145] I [MSGID: 106568]
[glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service
is stopped
[2016-06-17 13:44:24.362207] I [MSGID: 106567]
[glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting
quotad service

==> /var/log/glusterfs/glustershd.log <==
[2016-06-17 13:44:24.365420] I [MSGID: 100030] [glusterfsd.c:2408:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.0
(args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd
-p /var/lib/glusterd/glustershd/run/glustershd.pid -l
/var/log/glusterfs/glustershd.log -S
/var/run/gluster/6fe8cafd75bf10ffd386275f0bd67a06.socket --xlator-option
*replicate*.node-uuid=7fa073a8-d641-43c8-b722-c260415a28d9)

==> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <==
[2016-06-17 13:44:24.368724] W [socket.c:3131:socket_connect] 0-quotad:
Ignore failed connection attempt on
/var/run/gluster/e3e8332cbfa06af217d491807ac6478e.socket, (No such file
or directory)
[2016-06-17 13:44:24.368876] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already
stopped
[2016-06-17 13:44:24.368937] I [MSGID: 106568]
[glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service
is stopped
[2016-06-17 13:44:24.369063] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already
stopped
[2016-06-17 13:44:24.369098] I [MSGID: 106568]
[glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service
is stopped

==> /var/log/glusterfs/glustershd.log <==
[2016-06-17 13:44:24.370214] I [socket.c:4047:socket_init]
0-socket.glusterfsd: SSL support for glusterd is ENABLED
[2016-06-17 13:44:24.371239] I [socket.c:4047:socket_init] 0-glusterfs:
SSL support for glusterd is ENABLED
[2016-06-17 13:44:24.372781] E [socket.c:3143:socket_connect]
0-glusterfs: connection attempt on  failed, (Connection refused)
[2016-06-17 13:44:24.372929] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2016-06-17 13:44:24.373051] W [socket.c:696:__socket_rwv] 0-glusterfs:
writev on ::1:24007 failed (Success)
[2016-06-17 13:44:24.373408] E [rpc-clnt.c:357:saved_frames_unwind] (-->
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f32827eb543]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f32825b250f]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f32825b262e]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f32825b3e2e]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f32825b4648]
))))) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake)
op(GETSPEC(2)) called at 2016-06-17 13:44:24.373080 (xid=0x1)
[2016-06-17 13:44:24.373441] E [glusterfsd-mgmt.c:1686:mgmt_getspec_cbk]
0-mgmt: failed to fetch volume file (key:gluster/glustershd)
[2016-06-17 13:44:24.373494] W [glusterfsd.c:1286:cleanup_and_exit]
(-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1fa)
[0x7f32825b253a] -->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x53a)
[0x7f3282ce6c1a] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57)
[0x7f3282ce08d7] ) 0-: received signum (0), shutting down

==> /var/log/glusterfs/quotad.log <==
[2016-06-17 13:44:24.374607] I [MSGID: 100030] [glusterfsd.c:2408:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.0
(args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p
/var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log
-S /var/run/gluster/e3e8332cbfa06af217d491807ac6478e.socket
--xlator-option *replicate*.data-self-heal=off --xlator-option
*replicate*.metadata-self-heal=off --xlator-option
*replicate*.entry-self-heal=off)
[2016-06-17 13:44:24.379241] I [socket.c:4047:socket_init]
0-socket.glusterfsd: SSL support for glusterd is ENABLED
[2016-06-17 13:44:24.380264] I [socket.c:4047:socket_init] 0-glusterfs:
SSL support for glusterd is ENABLED
[2016-06-17 13:44:24.381727] E [socket.c:3143:socket_connect]
0-glusterfs: connection attempt on  failed, (Connection refused)
[2016-06-17 13:44:24.381872] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2016-06-17 13:44:24.381982] W [socket.c:696:__socket_rwv] 0-glusterfs:
writev on ::1:24007 failed (Success)
[2016-06-17 13:44:24.382342] E [rpc-clnt.c:357:saved_frames_unwind] (-->
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f74fb650543]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1cf)[0x7f74fb41750f]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f74fb41762e]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7f74fb418e2e]
(-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f74fb419648]
))))) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake)
op(GETSPEC(2)) called at 2016-06-17 13:44:24.382010 (xid=0x1)
[2016-06-17 13:44:24.382377] E [glusterfsd-mgmt.c:1686:mgmt_getspec_cbk]
0-mgmt: failed to fetch volume file (key:gluster/quotad)
[2016-06-17 13:44:24.382432] W [glusterfsd.c:1286:cleanup_and_exit]
(-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1fa)
[0x7f74fb41753a] -->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x53a)
[0x7f74fbb4bc1a] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57)
[0x7f74fbb458d7] ) 0-: received signum (0), shutting down

==> /var/log/glusterfs/cmd_history.log <==
[2016-06-17 13:44:25.783916]  : v start backups : SUCCESS

==> /var/log/glusterfs/cli.log <==
[2016-06-17 13:44:25.784065] I
[cli-rpc-ops.c:1414:gf_cli_start_volume_cbk] 0-cli: Received resp to
start volume
[2016-06-17 13:44:25.784188] I [input.c:31:cli_batch] 0-: Exiting with: 0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160617/3dc68aa4/attachment-0001.html>


More information about the Gluster-devel mailing list