[Gluster-devel] GlusterD2 - 4.0.0rc1 (warning: we have a blocker for GD2)

[Kaushal M]

On Thu, Mar 1, 2018 at 6:14 PM, Kaushal M <kshlmster at gmail.com> wrote:
> On Thu, Mar 1, 2018 at 12:52 PM, Kaushal M <kshlmster at gmail.com> wrote:
>> On Wed, Feb 28, 2018 at 9:50 PM, Kaleb S. KEITHLEY <kkeithle at redhat.com> wrote:
>>> On 02/28/2018 10:49 AM, Kaushal M wrote:
>>>> We have a GlusterD2-4.0.0rc1 release.
>>>>
>>>> Aravinda, Prashanth and the rest of the GD2 developers have been
>>>> working hard on getting more stuff merged into GD2 before the 4.0
>>>> release.
>>>>
>>>> At the same time I have been working on getting GD2 packaged for Fedora.
>>>> I've been able to get all the required dependencies updated and have
>>>> submitted to the package maintainer for merging.
>>>> I'm now waiting on the maintainer to accept those updates. Once the
>>>> updates have been accepted, the GD2 spec can get accepted [2].
>>>> I expect this to take at least another week on the whole.
>>>>
>>>> In the meantime, I've been building all the updated dependencies and
>>>> glusterd2-v4.0.0rc1, on the GD2 copr [3].
>>>>
>>>> I tried to test out the GD2 package with the GlusterFS v4.0.0rc1
>>>> release from [4]. And this is where I hit the blocker.
>>>>
>>>> GD2 does not start with the packaged glusterfs-v4.0.0rc1 bits. I've
>>>> opened an issue on the GD2 issue tracker for it [5].
>>>> In short, GD2 fails to read options from xlators, as dlopen fails with
>>>> a missing symbol error.
>>>>
>>>> ```
>>>> FATA[2018-02-28 15:02:53.345686] Failed to load xlator options
>>>>         error="dlopen(/usr/lib64/glusterfs/4.0.0rc1/xlator/protocol/server.so)
>>>> failed; dlerror =
>>>> /usr/lib64/glusterfs/4.0.0rc1/xlator/protocol/server.so: undefined
>>>> symbol: glusterfs_mgmt_pmap_signout" source="[main.go:79:main.main]"
>>>
>>>
>>> see https://review.gluster.org/#/c/19225/
>>>
>>>
>>> glusterfs_mgmt_pmap_signout() is in glusterfsd. When glusterfsd dlopens
>>> server.so the run-time linker can resolve the symbol — for now.
>>>
>>> Tighter run-time linker semantics coming in, e.g. Fedora 28, means this
>>> will stop working in the near future even when RTLD_LAZY is passed as a
>>> flag. (As I understand the proposed changes.)
>>>
>>> It should still work, e.g., on Fedora 27 and el7 though.
>>>
>>> glusterfs_mgmt_pmap_signout() (and glusterfs_autoscale_threads()) really
>>> need to be moved to libglusterfs. ASAP. Doing that will resolve this issue.
>>
>> Thanks for the pointer Kaleb!
>>
>> But, I'm testing on Fedora 27, where this shouldn't theoretically happen.
>> So then, why am I hitting this. Is it something to do with the way the
>> packages are built?
>> Or is there some runtime ld configuration that has been set up.
>>
>> In any case, we should push and get the offending functions moved into
>> libglusterfs.
>> That should solve the problem for us.
>
> I took a shot at this, and it's not as easy simple as it appeared.
> I ended up in a recursive linking situation with libglusterfs,
> libgfxdr and libgfrpc.
> Looks like the solution is to create a libglusterfsd.

I see two ways to do this.
1. Make a library out of the whole glusterfsd.
Rename `main` to `init`
And then create a simple executable which loads this library and calls `init`.

Or,
2. Create a very small library with just the pmap_signout and
autoscale functions. And use that instead.

If anyone else has a better idea about how to do this please let me know.

>
>>
>>>
>>> --
>>>
>>> Kaleb