[Gluster-infra] About the recent regression test breakages
Niels de Vos
ndevos at redhat.com
Sat Feb 22 09:24:10 UTC 2014
we recently have had an issue where regression tests were failing with
strange errors like 'C compiler cannot create execytables'. Also, the
nightly builds return the same error... Of course, when running the
tests locally, this works all fine.
The command that ./configure creates looks like this (from config.log):
gcc /d/build/install/lib/ conftest.c >&5
This definitely is an incorrect command.
Now, because both the regression test and nightly builds fail with the
same error, the problem seems to be system-wide on build.gluster.org.
The nightly builds run under my user-account, and only create a tarball
and src.rpm which gets build on a different system. Running ./configure
as my user, also has /d/build/install/lib/ in the gcc commandline. This
is surely unexpected.
It seems that there are two global environment variables set with this
path. LD_LIBRARY_PATH and LDFLAGS.
Setting a system-wide LD_LIBRARY_PATH is quite fragile. This
environment variable affects the loading of dynamically linked
libraries. The normal operating system services should NEVER try to
load any library from /d/build/install/lib which gets build during
each regression test. If the reviewer who starts the regression test
does not pay attention to the change, malicious libraries could be
build and installed in /d/build/install/lib, which then could affect
standard operating system behaviour.
LD_LIBRARY_PATH=/d/build/install/lib/ was set in
/etc/profile.d/jenkins.sh. This has now been commented out, and
a note in the file has been left behind for future reference. It
would be much better to only set LD_LIBRARY_PATH in the scripts that
execute the regression test runs (after building).
Of course, LDFLAGS immediately affects any standard 'make' commands, and
also ./configure uses CFLAGS and LDFLAGS from the environment.
LDFLAGS=/d/build/install/lib/ was set (not a valid LDFLAGS option
string) in /etc/environment. This has now been commented out, and
a note in the file has been left behind for future reference.
I have not restarted Jenkins (don't know how to do that). With the above
changes the nightly builds work again, but regression tests still fail.
I expect that there is a Jenkins process that has LDFLAGS still set in
Two pending actions:
1. update the regression test script(s) to set LD_LIBRARY_PATH
2. restart the Jenkins service to clean the environment variables
It would be much appreciated if system-wide changes like these at least
have some notes in the configuration files on why these settings are
needed, and who changed this. An email to this list would help others in
case suddenly strange errors occur...
More information about the Gluster-infra