[Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #1611

[sankarshan]

Looking at https://fstat.gluster.org/summary I see quite a few tests
still failing - are these failures being looked into?

./tests/basic/fencing/afr-lock-heal-basic.t
./tests/basic/fencing/afr-lock-heal-advanced.t
./tests/bugs/glusterd/quorum-validation.t

[Multiple bits in-line]

On Mon, 17 Feb 2020 at 14:38, Deepshikha Khandelwal <dkhandel at redhat.com> wrote:
>
>
> 04:08:01 Skipping script  : #!/bin/bash
> 04:08:01
> 04:08:01 ARCHIVE_BASE="/archives"
> 04:08:01 ARCHIVED_LOGS="logs"
> 04:08:01 UNIQUE_ID="${JOB_NAME}-${BUILD_ID}"
> 04:08:01 SERVER=$(hostname)
> 04:08:01
> 04:08:01 filename="${ARCHIVED_LOGS}/glusterfs-logs-${UNIQUE_ID}.tgz"
> 04:08:01 sudo -E tar -czf "${ARCHIVE_BASE}/${filename}" /var/log/glusterfs /var/log/messages*;
> 04:08:01 echo "Logs archived in http://${SERVER}/${filename}"
> 04:08:01 sudo reboot
>
>
> FYI, here we are skipping the post build script!`echo "Logs archived in http://${SERVER}/${filename}"` is a  part of the script.
>
>
> You can find the logs in Build artifacts https://build.gluster.org/job/regression-test-with-multiplex/1611/
>
>
>

Sunil, were you able to access the logs - please acknowledge on this list.

>
> On Thu, Jan 16, 2020 at 2:24 PM Sunil Kumar Heggodu Gopala Acharya <sheggodu at redhat.com> wrote:
>>
>> Hi,
>>
>> Please have the log location fixed so that others can take look into the failures.
>>
>> https://build.gluster.org/job/regression-test-with-multiplex/1611/consoleFull
>>
>> 21:10:48 echo "Logs archived in http://${SERVER}/${filename}"
>>
>>
>>
>> On Thu, Jan 16, 2020 at 2:18 PM Sanju Rakonde <srakonde at redhat.com> wrote:
>>>
>>> The below glusterd test cases are constantly failing in brick-mux regression:
>>> ./tests/bugs/glusterd/bug-857330/normal.t
>>> ./tests/bugs/glusterd/bug-857330/xml.t
>>> ./tests/bugs/glusterd/quorum-validation.t
>>>
>>> ./tests/bugs/glusterd/bug-857330/normal.t and ./tests/bugs/glusterd/bug-857330/xml.t are timed-out after 200 seconds. I don't find any abnormality in the logs. we need to increase the time(?). I'm unable to run these tests in my local setup as they are always failing saying "ModuleNotFoundError: No module named 'xattr' " Is the same happening in CI as well?
>>>

I am not a big fan of arbitrarily increasing the time. Do we know from
the logs as to why it might need more than 200s (that's 3+ mins -
quite a bit of time)?

>>>
>>> Also, we don't print the output of "prove -vf  <test>" when the test gets timed-out. It will be great if we print the output. It will help us to debug and to check which step took more time.
>>>

Was there a patch merged to enable this?

>>>
>>> ./tests/bugs/glusterd/quorum-validation.t is failing because of the regression caused by https://review.gluster.org/#/c/glusterfs/+/21651/. Rafi is looking into this issue. To explain the issue in brief, "after a reboot, glusterd is spawning multiple brick processes for a single brick instance and volume status shows the brick as offline".
>>>

Was this concept not obvious during the review of the patch?



-- 
sankarshan at kadalu.io | TZ: UTC+0530
kadalu.io : Making it easy to provision storage in k8s!