[Bugs] [Bug 1677555] New: Glusterfs brick is crashed due to segfault caused by broken gfid symlink

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1677555

            Bug ID: 1677555
           Summary: Glusterfs brick is crashed due to segfault caused by
                    broken gfid symlink
           Product: GlusterFS
           Version: 4.1
          Hardware: x86_64
                OS: Linux
            Status: NEW
         Component: core
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: ruslanloman at gmail.com
                CC: bugs at gluster.org
  Target Milestone: ---
    Classification: Community



Description of problem:
Glusterfs brick is crashed due to segfault caused by broken gfid symlink.

# gdb /usr/sbin/glusterfsd core.12867

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfsd -s glusterv3-04.namecheapcloud.net
--volfile-id easywp_pro'.
Program terminated with signal 11, Segmentation fault.
#0  __strftime_internal (s=0x7f85e44863f0 "", maxsize=256,
format=0x7f861bb39dfb "%F %T", tp=0x7f85e44863b0, 
    tzset_called=tzset_called at entry=0x7f85e4486320, loc=0xce4bbb31a32ca014) at
strftime_l.c:472
472       struct __locale_data *const current = loc->__locales[LC_TIME];
Missing separate debuginfos, use: debuginfo-install
glusterfs-fuse-4.1.7-1.el7.x86_64
(gdb) bt
#0  __strftime_internal (s=0x7f85e44863f0 "", maxsize=256,
format=0x7f861bb39dfb "%F %T", tp=0x7f85e44863b0, 
    tzset_called=tzset_called at entry=0x7f85e4486320, loc=0xce4bbb31a32ca014) at
strftime_l.c:472
#1  0x00007f861a172423 in __GI___strftime_l (s=<optimized out>,
maxsize=<optimized out>, format=<optimized out>, tp=<optimized out>, 
    loc=<optimized out>) at strftime_l.c:459
#2  0x00007f861ba89414 in gf_glusterlog_log_repetitions.isra.6 () from
/lib64/libglusterfs.so.0
#3  0x00007f861ba89953 in gf_log_flush_message () from /lib64/libglusterfs.so.0
#4  0x00007f861ba89a39 in gf_log_flush_list () from /lib64/libglusterfs.so.0
#5  0x00007f861ba89cbd in gf_log_set_log_buf_size () from
/lib64/libglusterfs.so.0
#6  0x00007f861ba89d17 in gf_log_disable_suppression_before_exit () from
/lib64/libglusterfs.so.0
#7  0x00007f861ba905c5 in gf_print_trace () from /lib64/libglusterfs.so.0
#8  <signal handler called>
#9  __GI_____strtoul_l_internal (nptr=nptr at entry=0x7f85e449e18c
"0cee86d5-51c9-4094-a670-091ebc518c08", endptr=endptr at entry=0x0, 
    base=base at entry=16, group=group at entry=0, loc=0xce4bbb31a32ca014) at
../stdlib/strtol_l.c:241
#10 0x00007f861a0efe22 in __GI_strtoul (nptr=nptr at entry=0x7f85e449e18c
"0cee86d5-51c9-4094-a670-091ebc518c08", endptr=endptr at entry=0x0, 
    base=base at entry=16) at ../stdlib/strtol.c:103
#11 0x00007f861b20f4bf in uuid_parse (in=0x7f85e449e18c
"0cee86d5-51c9-4094-a670-091ebc518c08", 
    uu=0x7f85e44ab2f0
"\024\240,\243\061\273KΑg\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273KΑg\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273KΑg\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273KΑg\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273KΑg\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273KΑg\345`Q|*j\f\356\206\325Q\311@\224\246p\t\036\274Q\214\b\024\240,\243\061\273",
<incomplete sequence \316>...) at libuuid/src/parse.c:65
#12 0x00007f860e395f4f in posix_make_ancestryfromgfid () from
/usr/lib64/glusterfs/4.1.7/xlator/storage/posix.so
#13 0x088c51bc1e0970a6 in ?? ()

If we check this gfid we'll see that it's broken symlink:

# file 0cee86d5-51c9-4094-a670-091ebc518c08
0cee86d5-51c9-4094-a670-091ebc518c08: broken symbolic link to
`../../14/a0/14a02ca3-31bb-4bce-9167-e560517c2a6a/..'

# file 14a02ca3-31bb-4bce-9167-e560517c2a6a
14a02ca3-31bb-4bce-9167-e560517c2a6a: broken symbolic link to
`../../0c/ee/0cee86d5-51c9-4094-a670-091ebc518c08/wp-admin'


After removing those broken symlink and execution 'gluster volume start force',
brick is back online. We have another gluster cluster version 4.1.5, it has
broken symlinks as well, but brick is never crushed.


Version-Release number of selected component (if applicable):

# uname -r
4.18.16-1.el7.elrepo.x86_64

# rpm -qa|grep gluster
glusterfs-4.1.7-1.el7.x86_64
glusterfs-fuse-4.1.7-1.el7.x86_64
glusterfs-geo-replication-4.1.7-1.el7.x86_64
centos-release-gluster41-1.0-3.el7.centos.noarch
glusterfs-libs-4.1.7-1.el7.x86_64
glusterfs-client-xlators-4.1.7-1.el7.x86_64
glusterfs-extra-xlators-4.1.7-1.el7.x86_64
glusterfs-api-4.1.7-1.el7.x86_64
glusterfs-server-4.1.7-1.el7.x86_64
glusterfs-devel-4.1.7-1.el7.x86_64
python2-gluster-4.1.7-1.el7.x86_64
glusterfs-cli-4.1.7-1.el7.x86_64


rpm -qa|grep gcc
libgcc-4.8.5-36.el7.x86_64

rpm -qa|grep glibc
glibc-common-2.17-260.el7.x86_64
glibc-debuginfo-common-2.17-260.el7.x86_64
glibc-2.17-260.el7.x86_64
glibc-debuginfo-2.17-260.el7.x86_64


How reproducible:


Steps to Reproduce:
1. Deploy gluster 4.1.7 in distributed replicated mode
2. Find symlink in gfid directory (.glusterfs) for one of brick and break it 
3. Brick with broken symlink should be crashed

Actual results:
Crushed with segfault


Expected results:
Brick should ignore broken symlink

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.