[Bugs] [Bug 1210568] New: [GlusterFS 3.6.2 ] Brick goes down if there is incorrect SSL certificates are installed on the server nodes
bugzilla at redhat.com
bugzilla at redhat.com
Fri Apr 10 05:50:05 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1210568
Bug ID: 1210568
Summary: [GlusterFS 3.6.2 ] Brick goes down if there is
incorrect SSL certificates are installed on the server
nodes
Product: GlusterFS
Version: 3.6.2
Component: glusterd
Severity: high
Assignee: bugs at gluster.org
Reporter: ssamanta at redhat.com
CC: bugs at gluster.org, gluster-bugs at redhat.com
Description of problem:
After installing incorrect SSL/TLS certificates in one node the glusterd
crashes and after that bricks goes down for that node and cluster goes into
inconsistent state.
Version-Release number of selected component (if applicable):
[root at gqas009 ~]# rpm -qa | grep gluster
glusterfs-api-devel-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop_hive-0.1-11.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop_hbase-0.1-3.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_fs_counters-0.1-10.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_multiuser_support-0.1-3.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop_hadoop_hcfs_fileappend-0.1-4.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-setup_hadoop-0.1-121.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop_hadoop_hcfs_quota-0.1-5.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_multiple_volumes-0.1-17.noarch
glusterfs-libs-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-test_dfsio_io_exception-0.1-8.noarch
glusterfs-fuse-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-test_shim_access_error_messages-0.1-5.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_sqoop-0.1-1.noarch
glusterfs-devel-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-setup_gluster-0.2-77.noarch
glusterfs-resource-agents-3.5.3-1.fc20.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_brick_sorted_order_of_filenames-0.1-1.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-setup_bigtop-0.2.1-23.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_erroneous_multivolume_filepaths-0.1-3.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_gluster_selfheal-0.1-5.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_file_dir_permissions-0.1-8.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_selinux_persistently_disabled-0.1-1.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_user_mapred_job-0.1-4.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_generate_gridmix2_data-0.1-2.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-setup_hadoop_security-0.0.1-7.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_dfsio-0.1-1.noarch
glusterfs-api-3.6.2-1.fc20.x86_64
glusterfs-extra-xlators-3.6.2-1.fc20.x86_64
glusterfs-server-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-setup_common-0.2-111.noarch
glusterfs-hadoop-2.1.2-2.fc20.noarch
glusterfs-geo-replication-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-test_special_char_in_path-0.1-1.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_groovy_sync-0.1-23.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_gluster_quota_selfheal-0.2-10.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_multifilewc_null_pointer_exception-0.1-5.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop_pig-0.1-8.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_gridmix3-0.1-1.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_setting_working_directory-0.1-1.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-setup_rhs_georep-0.1-2.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_home_dir_listing-0.1-4.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop_hadoop_hcfs_testcli-0.2-6.noarch
glusterfs-hadoop-javadoc-2.1.2-2.fc20.noarch
glusterfs-debuginfo-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-test_missing_dirs_create-0.1-3.noarch
glusterfs-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop_hadoop_mapreduce-0.1-5.noarch
glusterfs-cli-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-test_append_to_file-0.1-5.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop_mahout-0.1-5.noarch
glusterfs-rdma-3.6.2-1.fc20.x86_64
glusterfs-hadoop-distribution-glusterfs-hadoop-test_bigtop-0.1-7.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_default_block_size-0.1-3.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_ldap-0.1-6.noarch
glusterfs-hadoop-distribution-glusterfs-hadoop-test_junit_shim-0.1-12.noarch
[root at gqas009 ~]#
[root at gqas005 ~]# yum info openssl
Installed Packages
Name : openssl
Arch : x86_64
Epoch : 1
Version : 1.0.1e
Release : 42.fc20
Size : 1.5 M
Repo : installed
>From repo : fedora-updates
Summary : Utilities from the general purpose cryptography library with TLS
implementation
URL : http://www.openssl.org/
License : OpenSSL
Description : The OpenSSL toolkit provides support for secure communications
between
: machines. OpenSSL includes a certificate management tool and
shared
: libraries which provide various cryptographic algorithms and
: protocols.
[root at gqas005 ~]#
How reproducible:
Tried once
Steps to Reproduce:
1.Install fedora-20 and GlusterFS3.6.2(4 server physical machines, 1 client
physical machines)
2.Create a volume and start it
3.Enable the SSL options(client.ssl and server.ssl)
4.Create separate private keys for all the server nodes and clients
5.Create the public key and CN and concatenate the public keys(client and
server) and create a glusterfs.ca file and copy to the server nodes(/etc/ssl)
and clients(/etc/ssl).
6.Add the CN's to ssl-auth-allow list for the volume
7.Restart the volume
8.Mount from the client using fuse
Actual results:
Bricks for the one node goes down because of one node has not having correct
certificates.
Expected results:
There should not be any crashes and it should be handled gracefully.
Additional info:
Doc reference:
(https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_ssl.md
[root at gqas009 ~]# gluster peer status
Number of Peers: 3
Hostname: 10.16.156.36
Uuid: 10490a3d-10d8-48ca-963c-a85a6a195d1a
State: Peer in Cluster (Connected)
Hostname: 10.16.156.45
Uuid: de2bdc1a-cf40-4b4f-bb6a-5a261cd90db1
State: Peer in Cluster (Connected)
Hostname: 10.16.156.42
Uuid: dd3509dd-4fa4-4b0b-ae42-440ba22d8ec2
State: Peer in Cluster (Connected)
[root at gqas009 ~]#
Enable Nested Virtualization
============================
cat /sys/module/kvm_intel/parameters/nested
N
Temporarily remove the KVM intel Kernel module, enable nested virtualization to
be persistent across reboots and add the Kernel module back:
sudo rmmod kvm-intel
sudo sh -c "echo 'options kvm-intel nested=y' >> /etc/modprobe.d/dist.conf"
sudo modprobe kvm-intel
Ensure the Nested KVM Kernel module parameter for Intel is enabled on the host:
cat /sys/module/kvm_intel/parameters/nested
Y
modinfo kvm_intel | grep nested
parm: nested:bool
Generate private keys for the gluster nodes and clients
=======================================================
[root at gqas009 ~]# openssl genrsa -out glusterfs.key 1024
Generating RSA private key, 1024 bit long modulus
...++++++
....................++++++
e is 65537 (0x10001)
[root at gqas009 ~]#
[root at gqas013 ~]# openssl genrsa -out glusterfs.key 1024
Generating RSA private key, 1024 bit long modulus
...++++++
...........................++++++
e is 65537 (0x10001)
[root at gqas013 ~]#
[root at gqas015 ~]# openssl genrsa -out glusterfs.key 1024
Generating RSA private key, 1024 bit long modulus
........................++++++
.....................................................................++++++
e is 65537 (0x10001)
[root at gqas015 ~]#
[root at gqas016 ~]# openssl genrsa -out glusterfs.key 1024
Generating RSA private key, 1024 bit long modulus
...........++++++
.............................................++++++
e is 65537 (0x10001)
[root at gqas016 ~]#
>From brick logs:
================
signal received: 11
time of crash:
2015-04-08 13:54:58
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.6.2
[2015-04-08 13:54:58.330721] E [socket.c:384:ssl_setup_connection]
0-tcp.testvol2-server: SSL connect error
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f6fe8782362]
/lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f6fe879985d]
/lib64/libc.so.6(+0x358f0)[0x7f6fe779c8f0]
/lib64/libcrypto.so.10(X509_subject_name_cmp+0x3)[0x7f6fe7c3f6c3]
/lib64/libcrypto.so.10(OBJ_bsearch_ex_+0x64)[0x7f6fe7b93614]
/lib64/libcrypto.so.10(+0xe34c5)[0x7f6fe7c084c5]
/lib64/libcrypto.so.10(+0x12067f)[0x7f6fe7c4567f]
/lib64/libcrypto.so.10(X509_STORE_CTX_get1_issuer+0xe7)[0x7f6fe7c465c7]
/lib64/libcrypto.so.10(X509_verify_cert+0x90b)[0x7f6fe7c427fb]
/lib64/libssl.so.10(ssl3_output_cert_chain+0x1a8)[0x7f6fdd83cb68]
/lib64/libssl.so.10(ssl3_send_server_certificate+0x35)[0x7f6fdd8303d5]
/lib64/libssl.so.10(ssl3_accept+0xd1d)[0x7f6fdd83184d]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x478a)[0x7f6fdda7f78a]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x5e70)[0x7f6fdda80e70]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0xb149)[0x7f6fdda86149]
/lib64/libpthread.so.0(+0x7ee5)[0x7f6fe7f14ee5]
/lib64/libc.so.6(clone+0x6d)[0x7f6fe785bd1d]
>From client:
===========
[root at gqas005 ssl]# openssl req -new -x509 -key glusterfs.key -subj
/CN=client1.example.com -out glusterfs.pem
[root at gqas005 ssl]# ls
Servers with same CN(client.example.com)
=======================================
[root at gqas005 ssl]# openssl req -new -x509 -key glusterfs.key -subj
/CN=client.example.com -out glusterfs.pem
[root at gqas005 ssl]#
[root at gqas005 ssl]# mount -t glusterfs 10.16.156.24:/testvol2 /mnt/test
Mount failed. Please check the log file for more details.
[root at gqas005 ssl]#
Crash from glusterd logs:
========================
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.6.2
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f60b99ed362]
/lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f60b9a0485d]
/lib64/libc.so.6(+0x358f0)[0x7f60b8a078f0]
/lib64/libssl.so.10(SSL_write+0x4)[0x7f60ac636a94]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x4602)[0x7f60ac865602]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x495e)[0x7f60ac86595e]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x4ff2)[0x7f60ac865ff2]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x54a4)[0x7f60ac8664a4]
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x2b2)[0x7f60b97c01a2]
/usr/lib64/glusterfs/3.6.2/xlator/mgmt/glusterd.so(glusterd_submit_request_unlocked+0x164)[0x7f60aec0e8a4]
/usr/lib64/glusterfs/3.6.2/xlator/mgmt/glusterd.so(glusterd_submit_request+0x7a)[0x7f60aec0ea1a]
/usr/lib64/glusterfs/3.6.2/xlator/mgmt/glusterd.so(glusterd_peer_dump_version+0x8e)[0x7f60aec49b1e]
/usr/lib64/glusterfs/3.6.2/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x2ee)[0x7f60aebfc58e]
/usr/lib64/glusterfs/3.6.2/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x4c)[0x7f60aebf501c]
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a0)[0x7f60b97c13d0]
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f60b97bd2f3]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x5977)[0x7f60ac866977]
/usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0xabff)[0x7f60ac86bbff]
/lib64/libglusterfs.so.0(+0x765f2)[0x7f60b9a425f2]
/usr/sbin/glusterd(main+0x502)[0x7f60b9e96012]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f60b89f3d65]
/usr/sbin/glusterd(+0x63b1)[0x7f60b9e963b1]
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list