[Gluster-infra] About the recent regression test breakages

[Vijay Bellur]

On 02/24/2014 07:43 PM, Justin Clift wrote:
> (re-sending after subscribing to the list)
>
> On 22/02/2014, at 6:42 PM, Vijay Bellur wrote:
>> On 02/22/2014 02:54 PM, Niels de Vos wrote:
>>> Hi all,
>>>
>>> we recently have had an issue where regression tests were failing with
>>> strange errors like 'C compiler cannot create execytables'. Also, the
>>> nightly builds return the same error... Of course, when running the
>>> tests locally, this works all fine.
>>>
>>> The command that ./configure creates looks like this (from config.log):
>>>
>>>     gcc   /d/build/install/lib/ conftest.c  >&5
>>>
>>> This definitely is an incorrect command.
>>>
>>> Now, because both the regression test and nightly builds fail with the
>>> same error, the problem seems to be system-wide on build.gluster.org.
>>> The nightly builds run under my user-account, and only create a tarball
>>> and src.rpm which gets build on a different system. Running ./configure
>>> as my user, also has /d/build/install/lib/ in the gcc commandline. This
>>> is surely unexpected.
>>>
>>> It seems that there are two global environment variables set with this
>>> path. LD_LIBRARY_PATH and LDFLAGS.
>>>
>>>     LD_LIBRARY_PATH:
>>>     Setting a system-wide LD_LIBRARY_PATH is quite fragile. This
>>>     environment variable affects the loading of dynamically linked
>>>     libraries. The normal operating system services should NEVER try to
>>>     load any library from /d/build/install/lib which gets build during
>>>     each regression test. If the reviewer who starts the regression test
>>>     does not pay attention to the change, malicious libraries could be
>>>     build and installed in /d/build/install/lib, which then could affect
>>>     standard operating system behaviour.
>>>     LD_LIBRARY_PATH=/d/build/install/lib/ was set in
>>>     /etc/profile.d/jenkins.sh. This has now been commented out, and
>>>     a note in the file has been left behind for future reference. It
>>>     would be much better to only set LD_LIBRARY_PATH in the scripts that
>>>     execute the regression test runs (after building).
>>>
>>>     LDFLAGS:
>>>     Of course, LDFLAGS immediately affects any standard 'make' commands, and
>>>     also ./configure uses CFLAGS and LDFLAGS from the environment.
>>>     LDFLAGS=/d/build/install/lib/ was set (not a valid LDFLAGS option
>>>     string) in /etc/environment. This has now been commented out, and
>>>     a note in the file has been left behind for future reference.
>>>
>>>
>>> I have not restarted Jenkins (don't know how to do that). With the above
>>> changes the nightly builds work again, but regression tests still fail.
>>> I expect that there is a Jenkins process that has LDFLAGS still set in
>>> its environment.
>>>
>>> Two pending actions:
>>>
>>> 1. update the regression test script(s) to set LD_LIBRARY_PATH
>>> 2. restart the Jenkins service to clean the environment variables
>>>
>>> It would be much appreciated if system-wide changes like these at least
>>> have some notes in the configuration files on why these settings are
>>> needed, and who changed this. An email to this list would help others in
>>> case suddenly strange errors occur...
>>
>> Thanks, Niels for identifying this one.
>
> Fantastic stuff Niels! :)
>
>> I am the culprit here. These modifications to the build environment were meant to be temporary but I forgot to unset this after I was done debugging :-(.
>>
>> My sincere apologies for all the inconvenience and I'll ensure that I never attempt such kludges without notifying this list. A beer on me to both you & Justin when I meet you next!
>
> Nah, Niels did the work not me.  Give him mine as well. :)
>

Thanks Justin.

Niels - prepare for a double shot :).

-Vijay