<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div>Hi Stefan,<br></div><div><br></div><div>Does the combined <span style="color: rgb(0, 0, 0); font-family: "Lato 2", system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">glusterfs.ca includes client nodes pem? Also this file need to be placed in Client node as well.</span></div><div><br></div><div>--</div><div id="Zm-_Id_-Sgn" data-sigid="3848334000000010003" data-zbluepencil-ignore="true"><div>Aravinda<br></div><div>Kadalu Technologies</div></div><div><br></div><div class="zmail_extra_hr" style="border-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px; margin-bottom: 10px; line-height: 0px;"><br></div><div class="zmail_extra" data-zbluepencil-ignore="true"><div><br></div><div id="Zm-_Id_-Sgn1">---- On Fri, 26 Jan 2024 15:14:39 +0530 <b>Stefan Kania <stefan@kania-online.de></b> wrote ---<br></div><div><br></div><blockquote id="blockquote_zmail" style="margin: 0px;"><div>Hi to all, <br>The system is running Debian 12 with Gluster 10. All systems are using <br>the same versions. <br> <br>I try to encrypt the communication between the peers and the clients via <br>TLS. The encryption between the peers works, but when I try to mount the <br>volume on the client I always get an error. <br> <br> <br>What have I done? <br> <br>1. all hosts and clients can resolve the name of all systems involved. <br> <br>2. the volume is running and all hosts and clients can mount the volume, <br>when TLS is not activated. <br> <br>To activate TLS I did in /usr/lib/ssl on all participating systems with <br> <br> openssl genrsa -out glusterfs.key 2048 <br> <br>openssl req -new -x509 -key glusterfs.key -subj "/CN=c01.gluster" -out <br>glusterfs.pem <br> <br>Keys and certificates created (CN customised) <br> <br>Then combine all certificates into one and copy them to /usr/lib/ssl/ as <br>glusterfs.ca to all hosts. <br> <br>Create the file /var/lib/glusterd/secure-access on the gluster peers. <br> <br>Gluster volume stopped and glusterd restarted. <br> <br>Then set the following parameters: <br> <br>gluster volume set gv1 auth.ssl-allow '*' <br> <br>gluster volume set gv1 client.ssl on <br> <br>gluster volume set gv1 server.ssl on <br> <br>When mounting the volume on the peers, I get the following messages: <br>------------------- <br>_64-linux-gnu/libglusterfs.so.0(runner_log+0x100) [0x7ffa11782640] ) <br>0-management: Ran script: <br>/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=gv1 <br>--first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd <br> <br>0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED <br>certificate depth is 1 for peer 192.168.57.42:49147 <br> <br>0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED <br>certificate depth is 1 for peer 192.168.57.43:49147 <br> <br>0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED <br>certificate depth is 1 for peer 192.168.57.41:49151 <br> <br>------------------- <br> <br>Looks good to me <br> <br>Now trying to mount on the client with: <br>--------------- <br>mount -t glusterfs c01.gluster:/gv1 /mnt <br>--------------- <br>Then I get the following messages: <br>On the gluster node in /var/log/gluster/glusterd <br>------ <br>[2024-01-26 09:27:34.987837 +0000] I <br>[socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL <br>support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 <br>for peer 192.168.57.51:49151 <br>[2024-01-26 09:27:34.991908 +0000] E [socket.c:224:ssl_dump_error_stack] <br>0-socket.management: error:0A00010B:SSL routines::wrong version number <br>------ <br> <br>On the client in /var/log/gluster/mnt.log <br>------- <br>[2024-01-26 09:30:06.673990 +0000] I [MSGID: 100030] <br>[glusterfsd.c:2767:main] 0-/usr/sbin/glusterfs: Started running version <br>[{arg=/usr/sbin/glusterfs}, {version=10.5}, <br>{cmdlinestr=/usr/sbin/glusterfs --process-name fuse <br>--volfile-server=c01.gluster --volfile-id=/gv1 /mnt}] <br>[2024-01-26 09:30:06.677184 +0000] I [glusterfsd.c:2447:daemonize] <br>0-glusterfs: Pid of current running process is 931 <br>[2024-01-26 09:30:06.685814 +0000] I [MSGID: 101190] <br>[event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread <br>with index [{index=1}] <br>[2024-01-26 09:30:06.686116 +0000] I [MSGID: 101190] <br>[event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread <br>with index [{index=0}] <br>[2024-01-26 09:30:06.690443 +0000] I <br>[glusterfsd-mgmt.c:2681:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected <br>from remote-host: c01.gluster <br>[2024-01-26 09:30:06.690512 +0000] I <br>[glusterfsd-mgmt.c:2720:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted <br>all volfile servers <br>[2024-01-26 09:30:06.691618 +0000] W <br>[glusterfsd.c:1458:cleanup_and_exit] <br>(-->/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfa35) [0x7f83ace13a35] <br>-->/usr/sbin/glusterfs(+0x14769) [0x55650549b769] <br>-->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x556505492447] ) 0-: <br>received signum (1), shutting down <br>[2024-01-26 09:30:06.691699 +0000] I [fuse-bridge.c:7065:fini] 0-fuse: <br>Unmounting '/mnt'. <br>[2024-01-26 09:30:06.694246 +0000] I [fuse-bridge.c:7069:fini] 0-fuse: <br>Closing fuse connection to '/mnt'. <br>[2024-01-26 09:30:06.694431 +0000] W <br>[glusterfsd.c:1458:cleanup_and_exit] <br>(-->/lib/x86_64-linux-gnu/libc.so.6(+0x89044) [0x7f83acc98044] <br>-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x556505499e05] <br>-->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x556505492447] ) 0-: <br>received signum (15), shutting down <br>------- <br> <br> <br>Testing with openssl on the client show: <br> <br>root@cluster-client:~# openssl s_client -CAfile <br>/usr/lib/ssl/glusterfs.ca -connect c01.gluster:24007 <br>CONNECTED(00000003) <br>depth=0 CN = c01.gluster <br>verify return:1 <br>--- <br>Certificate chain <br> 0 s:CN = c01.gluster <br> i:CN = c01.gluster <br> a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 <br> v:NotBefore: Jan 26 08:27:12 2024 GMT; NotAfter: Feb 25 08:27:12 <br>2024 GMT <br>--- <br>Server certificate <br>-----BEGIN CERTIFICATE----- <br>MIIDDTCCAfWgAwIBAgIULCwcIV9jWFzeZoeO1Xs5TJ9J5rkwDQYJKoZIhvcNAQEL <br>BQAwFjEUMBIGA1UEAwwLYzAxLmdsdXN0ZXIwHhcNMjQwMTI2MDgyNzEyWhcNMjQw <br>MjI1MDgyNzEyWjAWMRQwEgYDVQQDDAtjMDEuZ2x1c3RlcjCCASIwDQYJKoZIhvcN <br>AQEBBQADggEPADCCAQoCggEBANPQ+fSk2V+hAjSOViZJxDWEgkjO1g8r3JH47QmI <br>D8mhEAVoeUhzDdbDV2gWw26pgU1Z22cCQr72rnZaK9vV1xzvGVjdzbKwQU8NhqhR <br>XWGJVlHdc5LxcOXfU7FpY6XMDzDLvRuNTMzsc685vJ8hjMxMAZJSLMAXNmLPMPnW <br>NuaudBE+1f7oc9sdGWhUqmPcWXT6xUeEUEJCDbOrccH8nhUwBMbQFiU7S4pV3smB <br>bbYNHFtw7Liz9B/vMoX1HckUKAsWcaWqPlWYr1rFHHPneyuG2evVcfRDhGsA1Fmo <br>v7kamrGtXgEAdgXC6HdENFBJzdSSb77A89d8OSHOYNyEV5UCAwEAAaNTMFEwHQYD <br>VR0OBBYEFCFjInacsKnR6TuPf+BI30b8qWPtMB8GA1UdIwQYMBaAFCFjInacsKnR <br>6TuPf+BI30b8qWPtMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB <br>AKBZNCRxKO5rv4yezGZRa/SDdpEc/vrGD5jKbHxQjBP+0YX/hToOGt04oh48iNFT <br>A2vqUVby4JXml9FjPCNktHlRk/NYXIlQiTm//TBeG2D+HrAQRyLR6TXF62/4H3Pb <br>Yktzr+vNk/znd5AKv3g8kMMpAB0UGxjZ9CtMDTuAYrQPtFCgCy1rf6fvP3cKZwaK <br>kk/QjJyc9u6zTvL0ptOHdOdQbhrHjZHiQ1D6QvJu6LouMsY3gGlVXfh0rQHUzSvT <br>7MmDRb/l4jTs2sn/nexh9bpHUv/m3vzDWBbrWcwGzenKXR+lg1hvAZAP3Ds33S/+ <br>W7sfZVptCwBXbYK0bSh+KiU= <br>-----END CERTIFICATE----- <br>subject=CN = c01.gluster <br>issuer=CN = c01.gluster <br>--- <br>No client certificate CA names sent <br>Requested Signature Algorithms: <br>ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224 <br>Shared Requested Signature Algorithms: <br>ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512 <br>Peer signing digest: SHA256 <br>Peer signature type: RSA-PSS <br>Server Temp Key: ECDH, prime256v1, 256 bits <br>--- <br>SSL handshake has read 1534 bytes and written 777 bytes <br>Verification: OK <br>--- <br>New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 <br>Server public key is 2048 bit <br>Secure Renegotiation IS NOT supported <br>Compression: NONE <br>Expansion: NONE <br>No ALPN negotiated <br>Early data was not sent <br>Verify return code: 0 (ok) <br>--- <br>--- <br>Post-Handshake New Session Ticket arrived: <br>SSL-Session: <br> Protocol : TLSv1.3 <br> Cipher : TLS_AES_256_GCM_SHA384 <br> Session-ID: <br>A9CA3DA57FDA9BF9D9EFBBD0E5CE5D8F7A5DE091C10E54310D52A23DCB7DA95B <br> Session-ID-ctx: <br> Resumption PSK: <br>C7BA79D9FB045352371121AC97F891FBD4C2578AA48A7CD57747A941C6864CCE5163D5AF94BE01D75233148BD75E755E <br> PSK identity: None <br> PSK identity hint: None <br> SRP username: None <br> TLS session ticket lifetime hint: 7200 (seconds) <br> TLS session ticket: <br> 0000 - 6e fd 36 f6 0f 16 dc d0-f1 9f 02 4b 32 20 5e 4b <br>n.6........K2 ^K <br> 0010 - e4 98 1e 6f 4c 8d b3 71-c8 12 40 ed 75 3f f7 ce <br>...oL..q..@.u?.. <br> <br> Start Time: 1706261953 <br> Timeout : 7200 (sec) <br> Verify return code: 0 (ok) <br> Extended master secret: no <br> Max Early Data: 0 <br>--- <br>read R BLOCK <br>--- <br>Post-Handshake New Session Ticket arrived: <br>SSL-Session: <br> Protocol : TLSv1.3 <br> Cipher : TLS_AES_256_GCM_SHA384 <br> Session-ID: <br>42BA7A7BFC9B64C030DB99E2D12B060052F53B5A771826199868A6AE913ED245 <br> Session-ID-ctx: <br> Resumption PSK: <br>3E66E04230CDFDF569A87764318B3C0C67FEA910742784CBC31E0221C44DB4EB91C2EBCB471AEB31FFFD5AB452C899F3 <br> PSK identity: None <br> PSK identity hint: None <br> SRP username: None <br> TLS session ticket lifetime hint: 7200 (seconds) <br> TLS session ticket: <br> 0000 - 79 2a c8 0c 4c c7 2b f1-2d 3c 01 cf dd b3 e0 68 <br>y*..L.+.-<.....h <br> 0010 - 7c 19 e7 e3 96 d9 5d 77-19 a3 e1 a8 9e 6c 3a 37 <br>|.....]w.....l:7 <br> <br> Start Time: 1706261953 <br> Timeout : 7200 (sec) <br> Verify return code: 0 (ok) <br> Extended master secret: no <br> Max Early Data: 0 <br>--- <br>read R BLOCK <br>40D7F609527F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof <br>while reading:../ssl/record/rec_layer_s3.c:303: <br> <br>Any help? <br> <br>Thank's <br> <br>Stefan <br> <br>________ <br> <br> <br> <br>Community Meeting Calendar: <br> <br>Schedule - <br>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC <br>Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a> <br>Gluster-users mailing list <br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a> <br></div></blockquote></div><div><br></div></div><br></body></html>