[Bugs] [Bug 1221605] New: Scrub.log grows rapidly and the size increases upto 24GB in a span of 10 hours

bugzilla at redhat.com bugzilla at redhat.com
Thu May 14 12:33:00 UTC 2015


https://bugzilla.redhat.com/show_bug.cgi?id=1221605

            Bug ID: 1221605
           Summary: Scrub.log grows rapidly and the size increases upto
                    24GB in a span of 10 hours
           Product: GlusterFS
           Version: 3.7.0
         Component: bitrot
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: senaik at redhat.com
                CC: bugs at gluster.org
      Docs Contact: bugs at gluster.org



Description of problem:
=======================
Scrub.log increases rapidly and grows upto 24GB in a span of about 10 hours and
root filesystem gets full 

[root at rhs-arch-srv3 glusterfs]# du -sch scrub.log 
24G    scrub.log
24G    total


Version-Release number of selected component (if applicable):
=============================================================
gluster --version
glusterfs 3.7.0beta2 built on May 11 2015 01:27:46


How reproducible:
================
1/1

Steps to Reproduce:
==================
1.Created some EC and tiered volumes and had USS, quota and bitrot enabled.

2.Scheduled some snapshots on the volumes 

3.Some bricks crashed - tracked by BZ 1221577

4.[root at rhs-arch-srv3 glusterfs]# df -h 
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhsarchsrv3-lv_root
                       26G   26G     0 100% /


[root at rhs-arch-srv3 glusterfs]# du -sch *
3.6M    bitd.log
940K    bricks
8.0K    cli.log
4.0K    cmd_history.log
208K    etc-glusterfs-glusterd.vol.log
4.0K    gcron.log
4.0K    geo-replication
8.0K    geo-replication-slaves
448K    glustershd.log
680K    nfs.log
476K    quotad.log
24G    scrub.log 
72K    snaps
4.0K    snap_scheduler.log
16K    var-run-gluster-shared_storage.log
28K    vol0-quota-crawl.log
24K    vol0-rebalance.log
24K    vol1-quota-crawl.log
12K    vol2-quota-crawl.log
24G    total


----------------Part of scrub.log-----------------

[2015-05-14 08:42:50.132193] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132209] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132216] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132239] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e18 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132250] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132262] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132269] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132280] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e19 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132301] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132310] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132317] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132327] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1a Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132336] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132345] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132351] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132361] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1b Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132371] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132379] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132386] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132396] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1c Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132405] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132414] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132420] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132430] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1d Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132440] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132459] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132466] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132477] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1e Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132486] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132495] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132501] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132511] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1f Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132521] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132530] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132536] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132546] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e20 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132555] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132564] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132570] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132580] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e21 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132590] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132598] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132605] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132615] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e22 Program:
GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132624] W [client-rpc-fops.c:2826:client3_3_lookup_cbk]
0-vol0-client-2: remote operation failed: Transport endpoint is not connected.
Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132633] E [bit-rot.c:1252:br_brick_connect]
0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not
connected]
[2015-05-14 08:42:50.132639] E [bit-rot.c:1331:br_handle_events]
0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132649] W [rpc-clnt.c:1566:rpc_clnt_submit]
0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e23 P


Actual results:


Expected results:


Additional info:

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
You are the Docs Contact for the bug.


More information about the Bugs mailing list