[Gluster-infra] About the recent regression test breakages

[Vijay Bellur]

On 02/22/2014 02:54 PM, Niels de Vos wrote:
> Hi all,
>
> we recently have had an issue where regression tests were failing with
> strange errors like 'C compiler cannot create execytables'. Also, the
> nightly builds return the same error... Of course, when running the
> tests locally, this works all fine.
>
> The command that ./configure creates looks like this (from config.log):
>
>      gcc   /d/build/install/lib/ conftest.c  >&5
>
> This definitely is an incorrect command.
>
> Now, because both the regression test and nightly builds fail with the
> same error, the problem seems to be system-wide on build.gluster.org.
> The nightly builds run under my user-account, and only create a tarball
> and src.rpm which gets build on a different system. Running ./configure
> as my user, also has /d/build/install/lib/ in the gcc commandline. This
> is surely unexpected.
>
> It seems that there are two global environment variables set with this
> path. LD_LIBRARY_PATH and LDFLAGS.
>
>      LD_LIBRARY_PATH:
>      Setting a system-wide LD_LIBRARY_PATH is quite fragile. This
>      environment variable affects the loading of dynamically linked
>      libraries. The normal operating system services should NEVER try to
>      load any library from /d/build/install/lib which gets build during
>      each regression test. If the reviewer who starts the regression test
>      does not pay attention to the change, malicious libraries could be
>      build and installed in /d/build/install/lib, which then could affect
>      standard operating system behaviour.
>      LD_LIBRARY_PATH=/d/build/install/lib/ was set in
>      /etc/profile.d/jenkins.sh. This has now been commented out, and
>      a note in the file has been left behind for future reference. It
>      would be much better to only set LD_LIBRARY_PATH in the scripts that
>      execute the regression test runs (after building).
>
>      LDFLAGS:
>      Of course, LDFLAGS immediately affects any standard 'make' commands, and
>      also ./configure uses CFLAGS and LDFLAGS from the environment.
>      LDFLAGS=/d/build/install/lib/ was set (not a valid LDFLAGS option
>      string) in /etc/environment. This has now been commented out, and
>      a note in the file has been left behind for future reference.
>
>
> I have not restarted Jenkins (don't know how to do that). With the above
> changes the nightly builds work again, but regression tests still fail.
> I expect that there is a Jenkins process that has LDFLAGS still set in
> its environment.
>
> Two pending actions:
>
> 1. update the regression test script(s) to set LD_LIBRARY_PATH
> 2. restart the Jenkins service to clean the environment variables
>
> It would be much appreciated if system-wide changes like these at least
> have some notes in the configuration files on why these settings are
> needed, and who changed this. An email to this list would help others in
> case suddenly strange errors occur...
>


Thanks, Niels for identifying this one.

I am the culprit here. These modifications to the build environment were 
meant to be temporary but I forgot to unset this after I was done 
debugging :-(.

My sincere apologies for all the inconvenience and I'll ensure that I 
never attempt such kludges without notifying this list. A beer on me to 
both you & Justin when I meet you next!

I have attempted to get both pending actions right. Please let me know 
if you still encounter problems.

-Vijay