[Bugs] [Bug 1456225] gluster-block is not working as expected when shard is enabled

bugzilla at redhat.com bugzilla at redhat.com
Sun May 28 01:34:18 UTC 2017


https://bugzilla.redhat.com/show_bug.cgi?id=1456225



--- Comment #1 from Pranith Kumar K <pkarampu at redhat.com> ---
Because gluster-block is storing metadata on the same volume as that of data
and since metadata updates are multi-client writes, gluster-block create hangs
and goes into a loop before it dies.
Reason is that the actual file size and filesize on the mount are differing and
gluster-block is not able to understand if the operation succeeded or not.

[root at localhost block-meta]# ls -l /brick1/
block-meta/  block-store/ .glusterfs/  .shard/      .trashcan/   
[root at localhost block-meta]# ls -l /brick1/block-meta/1
-rw-------. 2 root root 52304 May 20 19:36 /brick1/block-meta/1 <<<---- true
size.
[root at localhost block-meta]# ls -l 1
-rw-------. 1 root root 101 May 20 19:36 1 <<----- has truncated size.

When a file is opened with O_APPEND, offset gets ignored and the write buffer
is always appended to the file. Where as shard doesn't ignore offset when the
fd has O_APPEND. This is leading the size to be always stuck at 101 bytes
because that is the biggest write that comes on the file:

Thread 2 "gluster-blockd" hit Breakpoint 1, shard_writev (frame=0x61200005391c, 
    this=0x61f00001a4c0, fd=0x61100000b21c, vector=0x60800000cee0, count=1,
offset=0, 
    flags=0, iobref=0x60d00001d7c0, xdata=0x0) at shard.c:4827
4827            shard_common_inode_write_begin (frame, this, GF_FOP_WRITE, fd,
vector,
Missing separate debuginfos, use: dnf debuginfo-install
json-c-0.12-7.fc24.x86_64 libacl-2.2.52-11.fc24.x86_64
libattr-2.4.47-16.fc24.x86_64 libstdc++-6.2.1-2.fc25.x86_64
sssd-client-1.14.2-1.fc25.x86_64
(gdb) dis 1
(gdb) c
Continuing.
[Switching to Thread 0x7fffe565a700 (LWP 9037)]

Thread 10 "gluster-blockd" hit Breakpoint 2, trace_writev_cbk
(frame=0x612000053c1c, 
    cookie=0x61200005391c, this=0x61f0000196c0, op_ret=101, op_errno=0, 
    prebuf=0x61b00001a68c, postbuf=0x61b00001a6fc, xdata=0x611000052d9c) at
trace.c:232
232            char         preopstr[4096]  = {0, };
(gdb) p postbuf.ia_size
$1 = 101
(gdb) en 1
(gdb) c
Continuing.

Thread 10 "gluster-blockd" hit Breakpoint 1, shard_writev
(frame=0x61200002841c, 
    this=0x61f00001a4c0, fd=0x61100003cf9c, vector=0x608000020be0, count=1,
offset=0, 
    flags=0, iobref=0x60d00003d530, xdata=0x0) at shard.c:4827
4827            shard_common_inode_write_begin (frame, this, GF_FOP_WRITE, fd,
vector,
(gdb) c
Continuing.
[Switching to Thread 0x7fffe0f08700 (LWP 9038)]

Thread 11 "gluster-blockd" hit Breakpoint 2, trace_writev_cbk
(frame=0x61200002871c, 
    cookie=0x61200002841c, this=0x61f0000196c0, op_ret=21, op_errno=0,
prebuf=0x61b00000cd8c, 
    postbuf=0x61b00000cdfc, xdata=0x611000064bdc) at trace.c:232
232            char         preopstr[4096]  = {0, };
(gdb) p postbuf.ia_size
$2 = 101
(gdb) c
Continuing.
[New Thread 0x7fffe04e8700 (LWP 9040)]
[New Thread 0x7fffdfcc4700 (LWP 9041)]
[New Thread 0x7fffdf490700 (LWP 9042)]

Thread 11 "gluster-blockd" hit Breakpoint 1, shard_writev
(frame=0x61200003dd1c, 
    this=0x61f00001a4c0, fd=0x61100009479c, vector=0x608000032a60, count=1,
offset=0, 
    flags=0, iobref=0x60d00006c800, xdata=0x0) at shard.c:4827
4827            shard_common_inode_write_begin (frame, this, GF_FOP_WRITE, fd,
vector,
(gdb) c
Continuing.
[Switching to Thread 0x7fffe565a700 (LWP 9037)]

Thread 10 "gluster-blockd" hit Breakpoint 2, trace_writev_cbk
(frame=0x61200003e01c, 
    cookie=0x61200003dd1c, this=0x61f0000196c0, op_ret=33, op_errno=0,
prebuf=0x61b00002b78c, 
    postbuf=0x61b00002b7fc, xdata=0x61100007e5dc) at trace.c:232
232            char         preopstr[4096]  = {0, };
(gdb) p postbuf.ia_size
$3 = 101
(gdb) q
A debugging session is active.

    Inferior 1 [process 9024] will be killed.

After fixing the issue with:

[root at localhost r3]# gluster-block create r3/12 ha 3
192.168.122.61,192.168.122.123,192.168.122.113 1GiB
IQN: iqn.2016-12.org.gluster-block:1aef8052-2547-482e-9316-e41ba0e4b289
PORTAL(S):  192.168.122.61:3260 192.168.122.123:3260 192.168.122.113:3260
RESULT: SUCCESS
[root at localhost r3]# ls -l /brick1/block-meta/12
-rw-------. 2 root root 315 May 24 22:52 /brick1/block-meta/12
[root at localhost r3]# ls -l /mnt/block-meta/12
-rw-------. 1 root root 315 May 24 22:52 /mnt/block-meta/12

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list