[Gluster-infra] About the recent regression test breakages

Niels de Vos ndevos at redhat.com
Sat Feb 22 09:24:10 UTC 2014


Hi all,

we recently have had an issue where regression tests were failing with 
strange errors like 'C compiler cannot create execytables'. Also, the 
nightly builds return the same error... Of course, when running the 
tests locally, this works all fine.

The command that ./configure creates looks like this (from config.log):

    gcc   /d/build/install/lib/ conftest.c  >&5

This definitely is an incorrect command.

Now, because both the regression test and nightly builds fail with the 
same error, the problem seems to be system-wide on build.gluster.org.  
The nightly builds run under my user-account, and only create a tarball 
and src.rpm which gets build on a different system. Running ./configure 
as my user, also has /d/build/install/lib/ in the gcc commandline. This 
is surely unexpected.

It seems that there are two global environment variables set with this 
path. LD_LIBRARY_PATH and LDFLAGS.

    LD_LIBRARY_PATH:
    Setting a system-wide LD_LIBRARY_PATH is quite fragile. This 
    environment variable affects the loading of dynamically linked 
    libraries. The normal operating system services should NEVER try to 
    load any library from /d/build/install/lib which gets build during 
    each regression test. If the reviewer who starts the regression test 
    does not pay attention to the change, malicious libraries could be 
    build and installed in /d/build/install/lib, which then could affect 
    standard operating system behaviour.
    LD_LIBRARY_PATH=/d/build/install/lib/ was set in 
    /etc/profile.d/jenkins.sh. This has now been commented out, and 
    a note in the file has been left behind for future reference. It 
    would be much better to only set LD_LIBRARY_PATH in the scripts that 
    execute the regression test runs (after building).

    LDFLAGS:
    Of course, LDFLAGS immediately affects any standard 'make' commands, and 
    also ./configure uses CFLAGS and LDFLAGS from the environment.
    LDFLAGS=/d/build/install/lib/ was set (not a valid LDFLAGS option 
    string) in /etc/environment. This has now been commented out, and 
    a note in the file has been left behind for future reference.


I have not restarted Jenkins (don't know how to do that). With the above 
changes the nightly builds work again, but regression tests still fail.  
I expect that there is a Jenkins process that has LDFLAGS still set in 
its environment.

Two pending actions:

1. update the regression test script(s) to set LD_LIBRARY_PATH
2. restart the Jenkins service to clean the environment variables

It would be much appreciated if system-wide changes like these at least 
have some notes in the configuration files on why these settings are 
needed, and who changed this. An email to this list would help others in 
case suddenly strange errors occur...

Thanks,
Niels


More information about the Gluster-infra mailing list