[Gluster-devel] Toggle storage.linux-aio and volume restart

[Dmitry Antipov]

On 12/8/20 4:47 PM, Amar Tumballi wrote:

> Depends on if posix_reconfigure() handles this option or not. If it handles it, then without volume restart, we can use it.

Sure it handles. But I do not understand how the following becomes possible:

Thread 20 "glfs_iotwr003" hit Breakpoint 1, posix_writev (frame=0x7fb068003bf8, this=0x7fb09c008b10, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1,
     offset=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at posix-inode-fd-ops.c:1929
1929	    int32_t op_errno = 0;
(gdb) bt
#0  posix_writev (frame=0x7fb068003bf8, this=0x7fb09c008b10, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, offset=653123584, flags=0, iobref=0x7fb0a4088120,
     xdata=0x7fb090394ac8) at posix-inode-fd-ops.c:1929
#1  0x00007fb0bbd9b422 in default_writev (frame=frame at entry=0x7fb068003bf8, this=<optimized out>, fd=fd at entry=0x7fb0781ee528, vector=vector at entry=0x7fb090293750,
     count=count at entry=1, off=off at entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at defaults.c:2550
#2  0x00007fb0a90405fb in changelog_writev (frame=frame at entry=0x7fb068003ae8, this=0x7fb09c00ec80, fd=fd at entry=0x7fb0781ee528, vector=vector at entry=0x7fb090293750,
     count=count at entry=1, offset=offset at entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at changelog.c:1763
#3  0x00007fb0a90c9eaf in br_stub_writev (frame=0x7fb068019968, this=0x7fb09c0110b0, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, offset=653123584, flags=0,
     iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at bit-rot-stub.c:2112
#4  0x00007fb0bbd9b422 in default_writev (frame=frame at entry=0x7fb068019968, this=<optimized out>, fd=fd at entry=0x7fb0781ee528, vector=vector at entry=0x7fb090293750,
     count=count at entry=1, off=off at entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at defaults.c:2550
#5  0x00007fb0a8ffd4d4 in pl_writev (frame=0x7fb0680039d8, this=0x7fb09c014d70, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, offset=653123584, flags=0,
     iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at posix.c:2377
#6  0x00007fb0a8fcd99b in worm_writev (frame=0x7fb0680039d8, this=<optimized out>, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, offset=653123584, flags=0,
     iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at worm.c:429
#7  0x00007fb0a8fbed0f in ro_writev (frame=frame at entry=0x7fb0680039d8, this=<optimized out>, fd=fd at entry=0x7fb0781ee528, vector=vector at entry=0x7fb090293750,
     count=count at entry=1, off=off at entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at read-only-common.c:374
#8  0x00007fb0a8fa60a6 in leases_writev (frame=frame at entry=0x7fb0680038c8, this=0x7fb09c01ab10, fd=fd at entry=0x7fb0781ee528, vector=vector at entry=0x7fb090293750,
     count=count at entry=1, off=off at entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at leases.c:132
#9  0x00007fb0a8f90d4a in up_writev (frame=frame at entry=0x7fb068003338, this=0x7fb09c01c8d0, fd=fd at entry=0x7fb0781ee528, vector=vector at entry=0x7fb090293750,
     count=count at entry=1, off=off at entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at upcall.c:124
#10 0x00007fb0bbdb08c9 in default_writev_resume (frame=0x7fb09022fbc8, this=0x7fb09c01e640, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, off=653123584, flags=0,
     iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at defaults.c:1831
#11 0x00007fb0bbd38ba8 in call_resume_wind (stub=<optimized out>) at call-stub.c:1922
#12 0x00007fb0bbd38d95 in call_resume (stub=0x7fb0900b7a08) at call-stub.c:2392
#13 0x00007fb0a8f72740 in iot_worker (data=0x7fb09c05b490) at io-threads.c:232
#14 0x00007fb0bbab63f9 in start_thread (arg=0x7fb0a8200640) at pthread_create.c:463
#15 0x00007fb0bb6f5903 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) p this
$1 = (xlator_t *) 0x7fb09c008b10                                                           ;; xlator in subject
(gdb) p this->fops->readv
$2 = (fop_readv_t) 0x7fb0a90ff530 <posix_readv>
(gdb) p this->fops->writev
$3 = (fop_writev_t) 0x7fb0a91002e0 <posix_writev>
(gdb) p this->private
$4 = (void *) 0x7fb09c07ce40
(gdb) p ((struct posix_private *)(this->private))->aio_configured                          ;; has AIO configured
$5 = true
(gdb) p ((struct posix_private *)(this->private))->aio_init_done                           ;; and initialized (i.e. AIO processing thread is running)
$6 = true
(gdb) info threads
   Id   Target Id                                            Frame
   ...
   12   Thread 0x7fb0a8bf5640 (LWP 447802) "glfs_posixaio"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
   ...
(gdb) thr 12                                                                               ;; this is the one
[Switching to thread 12 (Thread 0x7fb0a8bf5640 (LWP 447802))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38		cmpq $-4095, %rax	/* Check %rax for error.  */
(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007fb0a90b530b in __io_getevents_0_4 (ctx=0x7fb0a8bf6000, min_nr=1, nr=16, events=0x7fb0a8bf3d60, timeout=0x0) at io_getevents.c:25
#2  0x00007fb0a90b5361 in io_getevents_0_4 (ctx=<optimized out>, min_nr=min_nr at entry=1, nr=nr at entry=16, events=events at entry=0x7fb0a8bf3d60, timeout=timeout at entry=0x0)
     at io_getevents.c:54
#3  0x00007fb0a90eab62 in posix_aio_thread (data=0x7fb09c008b10) at posix-aio.c:422        ;; xlator passed to AIO thread
#4  0x00007fb0bbab63f9 in start_thread (arg=0x7fb0a8bf5640) at pthread_create.c:463
#5  0x00007fb0bb6f5903 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) p (xlator_t *)0x7fb09c008b10                                                         ;; xlator in subject
$7 = (xlator_t *) 0x7fb09c008b10
(gdb) p ((xlator_t *)0x7fb09c008b10)->fops->readv
$8 = (fop_readv_t) 0x7fb0a90ff530 <posix_readv>
(gdb) p ((xlator_t *)0x7fb09c008b10)->fops->writev
$9 = (fop_writev_t) 0x7fb0a91002e0 <posix_writev>

IOW if aio_configured is true, fops->readv and fops->writev should be set to posix_aio_readv()
and posix_aio_writev(), respectively. But the whole picture looks like something in xlator
graph silently reverts fops->readv and fops->writev back to posix_xxxv() defaults.

Dmitry