[Bugs] [Bug 1224193] New: Scrub.log grows rapidly and the size increases upto 24GB in a span of 10 hours

Fri May 22 10:14:00 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1224193

            Bug ID: 1224193
           Summary: Scrub.log grows rapidly and the size increases upto
                    24GB in a span of 10 hours
           Product: Red Hat Gluster Storage
         Component: glusterfs
     Sub Component: bitrot
          Keywords: Triaged
          Severity: high
          Assignee: rhs-bugs at redhat.com
          Reporter: senaik at redhat.com
        QA Contact: rmekala at redhat.com
                CC: bugs at gluster.org, nsathyan at redhat.com,
                    vshankar at redhat.com
        Depends On: 1207134, 1221605
            Blocks: 1186580 (qe_tracker_everglades), 1219955
                    (glusterfs-3.7.1)

+++ This bug was initially created as a clone of Bug #1221605 +++

Description of problem:
=======================
Scrub.log increases rapidly and grows upto 24GB in a span of about 10 hours and
root filesystem gets full 

[root at rhs-arch-srv3 glusterfs]# du -sch scrub.log 
24G    scrub.log
24G    total

Version-Release number of selected component (if applicable):
=============================================================
gluster --version
glusterfs 3.7.0beta2 built on May 11 2015 01:27:46

How reproducible:
================
1/1

Steps to Reproduce:
==================
1.Created some EC and tiered volumes and had USS, quota and bitrot enabled.

2.Scheduled some snapshots on the volumes 

3.Some bricks crashed - tracked by BZ 1221577

4.[root at rhs-arch-srv3 glusterfs]# df -h 
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhsarchsrv3-lv_root
                       26G   26G     0 100% /

[root at rhs-arch-srv3 glusterfs]# du -sch *
3.6M    bitd.log
940K    bricks
8.0K    cli.log
4.0K    cmd_history.log
208K    etc-glusterfs-glusterd.vol.log
4.0K    gcron.log
4.0K    geo-replication
8.0K    geo-replication-slaves
448K    glustershd.log
680K    nfs.log
476K    quotad.log
24G    scrub.log 
72K    snaps
4.0K    snap_scheduler.log
16K    var-run-gluster-shared_storage.log
28K    vol0-quota-crawl.log
24K    vol0-rebalance.log
24K    vol1-quota-crawl.log
12K    vol2-quota-crawl.log
24G    total

----------------Part of scrub.log-----------------

[2015-05-14 08:42:50.132193] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132209] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132216] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132239] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e18 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132250] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132262] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132269] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132280] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e19 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132301] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132310] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132317] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132327] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1a Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132336] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132345] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132351] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132361] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1b Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132371] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132379] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132386] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132396] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1c Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132405] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132414] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132420] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132430] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1d Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132440] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132459] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132466] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132477] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1e Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132486] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132495] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132501] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132511] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1f Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132521] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132530] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132536] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132546] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e20 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132555] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132564] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132570] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132580] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e21 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132590] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132598] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132605] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132615] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e22 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132624] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132633] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132639] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132649] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e23 P

Actual results:

Expected results:

Additional info:

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1186580
[Bug 1186580] QE tracker bug for Everglades
https://bugzilla.redhat.com/show_bug.cgi?id=1207134
[Bug 1207134] BitRot :- bitd is not signing Objects if more than 3 bricks
are present on same node
https://bugzilla.redhat.com/show_bug.cgi?id=1219955
[Bug 1219955] GlusterFS 3.7.1 tracker
https://bugzilla.redhat.com/show_bug.cgi?id=1221605
[Bug 1221605] Scrub.log grows rapidly and the size increases upto 24GB in a
span of 10 hours
-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=cXqkcloWSe&a=cc_unsubscribe