[Gluster-devel] libgfapi threads

[Kelly Burkhart]

We've noticed that gfapi threads won't die until process exit, they aren't
joined to in glfs_fini().  Is that expected?  The following will create 4*N
threads:

for( idx=0; idx<N; ++idx) {
    glfs_new
    glfs_set_volfile_server
    glfs_init
    // pause a bit here
    glfs_fini
}

-K



On Fri, Jan 31, 2014 at 9:07 AM, Kelly Burkhart <kelly.burkhart at gmail.com>wrote:

> Thanks Anand,
>
> I notice three different kind of threads: gf_timer_proc and
> syncenv_processor in libglusterfs and glfs_poller in the api.  Right off
> the bat two syncenv threads are created and one each of the other two are
> created.  In my limited testing, it doesn't seem to take much for more
> threads to be created.
>
> The reason I'm concerned is that we intend to run our gluster client on a
> machine with all but one core dedicated to latency critical apps.  The
> remaining core will handle all other things.  In this scenario creating
> scads of threads seems likely to be a pessimization compared to just having
> one thread with an epoll loop handling everything.  Would any of you
> familiar with the guts of gluster predict a problem with pegging a gfapi
> client and all of it's child threads to a single core?
>
> BTW, attached is a simple patch to help me track what threads are created,
> it's linux specific, but I think it's useful.  It adds an identifier and
> instance count to each kind of child thread so I see this in top:
>
> top - 08:35:47 up 48 min,  3 users,  load average: 0.12, 0.07, 0.05
> Tasks:   9 total,   0 running,   9 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.2%us,  0.1%sy,  0.0%ni, 98.9%id,  0.0%wa,  0.0%hi,  0.7%si,
>  0.0%st
> Mem:     16007M total,     1372M used,    14634M free,       96M buffers
> Swap:     2067M total,        0M used,     2067M free,      683M cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
> 22979 kelly     20   0  971m 133m  16m S      0  0.8   0:00.06 tst
> 22987 kelly     20   0  971m 133m  16m S      0  0.8   0:00.00 tst/sp:0
> 22988 kelly     20   0  971m 133m  16m S      0  0.8   0:00.00 tst/sp:1
> 22989 kelly     20   0  971m 133m  16m S      0  0.8   0:00.03 tst/gp:0
> 22990 kelly     20   0  971m 133m  16m S      0  0.8   0:00.00 tst/tm:0
> 22991 kelly     20   0  971m 133m  16m S      0  0.8   0:00.00 tst/sp:2
> 22992 kelly     20   0  971m 133m  16m S      0  0.8   0:00.00 tst/sp:3
> 22993 kelly     20   0  971m 133m  16m S      0  0.8   0:01.98 tst/gp:1
> 22994 kelly     20   0  971m 133m  16m S      0  0.8   0:00.00 tst/tm:1
>
> Thanks,
>
> -K
>
>
>
> On Thu, Jan 30, 2014 at 4:38 PM, Anand Avati <avati at gluster.org> wrote:
>
>> Thread count is independent of number of servers. The number of
>> sockets/connections is a function of number of servers/bricks. There are a
>> minimum number of threads (like the timer threads, syncop exec threads,
>> io-threads, epoll thread, depending on interconnect RDMA event reaping
>> threads) and some of them (syncop and io-thread) count are dependent on the
>> work load. All communication with servers is completely asynchronous and we
>> do not spawn a new thread per server.
>>
>> HTH
>> Avati
>>
>>
>>
>> On Thu, Jan 30, 2014 at 1:17 PM, James <purpleidea at gmail.com> wrote:
>>
>>> On Thu, Jan 30, 2014 at 4:15 PM, Paul Cuzner <pcuzner at redhat.com> wrote:
>>> > Wouldn't the thread count relate to the number of bricks in the volume,
>>> > rather that peers in the cluster?
>>>
>>>
>>> My naive understanding is:
>>>
>>> 1) Yes, you should expect to see one connection to each brick.
>>>
>>> 2) Some of the "scaling gluster to 1000" nodes work might address the
>>> issue, as to avoid 1000 * brick count/perserver connections.
>>>
>>> But yeah, Kelly: I think you're seeing the right number of threads.
>>> But this is outside of my expertise.
>>>
>>> James
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at nongnu.org
>>> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140204/814abd9d/attachment-0001.html>