[Bugs] [Bug 1221605] New: Scrub.log grows rapidly and the size increases upto 24GB in a span of 10 hours

Thu May 14 12:33:00 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1221605

            Bug ID: 1221605
           Summary: Scrub.log grows rapidly and the size increases upto
                    24GB in a span of 10 hours
           Product: GlusterFS
           Version: 3.7.0
         Component: bitrot
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: senaik at redhat.com
                CC: bugs at gluster.org
      Docs Contact: bugs at gluster.org

Description of problem:
=======================
Scrub.log increases rapidly and grows upto 24GB in a span of about 10 hours and
root filesystem gets full 

[root at rhs-arch-srv3 glusterfs]# du -sch scrub.log 
24G    scrub.log
24G    total

Version-Release number of selected component (if applicable):
=============================================================
gluster --version
glusterfs 3.7.0beta2 built on May 11 2015 01:27:46

How reproducible:
================
1/1

Steps to Reproduce:
==================
1.Created some EC and tiered volumes and had USS, quota and bitrot enabled.

2.Scheduled some snapshots on the volumes 

3.Some bricks crashed - tracked by BZ 1221577

4.[root at rhs-arch-srv3 glusterfs]# df -h 
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhsarchsrv3-lv_root
                       26G   26G     0 100% /

[root at rhs-arch-srv3 glusterfs]# du -sch *
3.6M    bitd.log
940K    bricks
8.0K    cli.log
4.0K    cmd_history.log
208K    etc-glusterfs-glusterd.vol.log
4.0K    gcron.log
4.0K    geo-replication
8.0K    geo-replication-slaves
448K    glustershd.log
680K    nfs.log
476K    quotad.log
24G    scrub.log 
72K    snaps
4.0K    snap_scheduler.log
16K    var-run-gluster-shared_storage.log
28K    vol0-quota-crawl.log
24K    vol0-rebalance.log
24K    vol1-quota-crawl.log
12K    vol2-quota-crawl.log
24G    total

----------------Part of scrub.log-----------------

[2015-05-14 08:42:50.132193] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132209] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132216] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132239] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e18 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132250] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132262] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132269] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132280] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e19 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132301] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132310] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132317] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132327] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1a Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132336] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132345] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132351] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132361] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1b Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132371] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132379] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132386] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132396] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1c Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132405] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132414] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132420] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132430] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1d Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132440] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132459] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132466] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132477] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1e Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132486] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132495] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132501] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132511] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1f Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132521] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132530] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132536] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132546] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e20 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132555] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132564] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132570] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132580] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e21 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132590] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132598] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132605] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132615] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e22 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132624] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132633] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132639] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132649] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e23 P

Actual results:

Expected results:

Additional info:

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
You are the Docs Contact for the bug.