[Gluster-users] intresting issue of replication and self-heal
Mingfan Lu
mingfan.lu at gmail.com
Thu Jan 23 09:04:32 UTC 2014
I profiled node22, I found that most latency comes from setxattr, where
node23 & node22 comes from lookup and locks. any one could help?
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 2437540
FORGET
0.00 0.00 us 0.00 us 0.00 us 252684
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2226292
RELEASEDIR
0.00 38.00 us 37.00 us 40.00 us 4
FGETXATTR
0.00 66.16 us 15.00 us 13139.00 us 596
GETXATTR
0.00 239.14 us 58.00 us 126477.00 us 1967
LINK
0.00 51.85 us 14.00 us 8298.00 us 19045
STAT
0.00 165.50 us 9.00 us 212057.00 us 20544
READDIR
0.00 1827.92 us 184.00 us 150298.00 us 2084
RENAME
0.00 49.14 us 12.00 us 5908.00 us 189019
STATFS
0.00 84.63 us 14.00 us 96016.00 us 163405
READ
0.00 29968.76 us 156.00 us 1073902.00 us 3115
CREATE
0.00 1340.25 us 6.00 us 7415357.00 us 248141
FLUSH
0.00 1616.76 us 32.00 us 13865122.00 us 229190
FTRUNCATE
0.01 1807.58 us 19.00 us 55480776.00 us 249569
OPEN
0.01 1875.11 us 10.00 us 8842171.00 us 465197
FSTAT
0.05 393296.28 us 52.00 us 56856581.00 us 9057
UNLINK
0.07 32291.01 us 192.00 us 9638107.00 us 156081
RMDIR
0.08 18339.18 us 140.00 us 5313885.00 us 337862
MKNOD
0.09 2904.39 us 18.00 us 51724741.00 us 2226290
OPENDIR
0.15 4708.15 us 27.00 us 55115760.00 us 2334864
SETXATTR
0.18 8965.91 us 68.00 us 26465968.00 us 1513280
FXATTROP
0.21 3465.29 us 74.00 us 58580783.00 us 4506602
XATTROP
0.28 4801.16 us 44.00 us 49643138.00 us 4436847
READDIRP
0.37 5935.92 us 7.00 us 56449083.00 us 4611760
ENTRYLK
1.02 4226.58 us 33.00 us 63494729.00 us 18092335
WRITE
1.50 2734.50 us 6.00 us 185109908.00 us 40971541
INODELK
4.75 348602.30 us 5.00 us 2185602946.00 us 1019332
FINODELK
* 14.98 33957.49 us 14.00 us 59261447.00 us 32998211
LOOKUP 26.30 807063.74 us 150.00 us 68086266.00 us
2438422 MKDIR 49.95 457402.30 us 20.00 us 67894186.00
us 8171751 SETATTR*
Duration: 353678 seconds
Data Read: 21110920120 bytes
Data Written: 2338403381483 bytes
here is node23
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 22125898
FORGET
0.00 0.00 us 0.00 us 0.00 us 89286732
RELEASE
0.00 0.00 us 0.00 us 0.00 us 32865496
RELEASEDIR
0.00 35.50 us 23.00 us 48.00 us 2
FGETXATTR
0.00 164.04 us 29.00 us 749181.00 us 39320
FTRUNCATE
0.00 483.71 us 8.00 us 2688755.00 us
39288 LK
0.00 419.61 us 48.00 us 2183971.00 us 274939
LINK
0.00 970.55 us 145.00 us 2471745.00 us 293435
RENAME
0.00 1346.63 us 35.00 us 4462970.00 us 243238
SETATTR
0.01 285.51 us 25.00 us 2588685.00 us 3459436
SETXATTR
0.03 323.11 us 5.00 us 2074581.00 us 6977304
READDIR
0.05 12200.60 us 84.00 us 3943421.00 us 287979
RMDIR
0.07 592.75 us 7.00 us 3592073.00 us 8129847
STAT
0.07 6938.50 us 49.00 us 3268036.00 us 705818
UNLINK
0.08 19468.78 us 149.00 us 3664022.00 us 276310
MKNOD
0.09 763.31 us 8.00 us 3396903.00 us 8731725
STATFS
0.09 1715.79 us 4.00 us 5626912.00 us 3902746
FLUSH
0.10 4614.74 us 9.00 us 5835691.00 us 1574923
FSTAT
0.10 1189.55 us 13.00 us 6043163.00 us 6129885
OPENDIR
0.10 19729.66 us 131.00 us 4112832.00 us 376286
CREATE
0.13 328.26 us 24.00 us 2410049.00 us 29091424
WRITE
0.20 2107.64 us 10.00 us 5765196.00 us 6675496
GETXATTR
0.28 5317.38 us 14.00 us 7549301.00 us 3798543
OPEN
0.71 7042.79 us 47.00 us 5848284.00 us 7125716
READDIRP
0.80 743.88 us 10.00 us 7979373.00 us 76781383
READ
0.93 1802.29 us 60.00 us 11040319.00 us 36501360
FXATTROP
1.76 36083.12 us 141.00 us 3548175.00 us 3458135
MKDIR
1.83 5046.35 us 70.00 us 8120221.00 us 25765615
XATTROP
11.74 12896.99 us 4.00 us 2141920969.00 us 64590600
FINODELK
15.43 11171.78 us 5.00 us 909115697.00 us 98040443
ENTRYLK
25.46 12945.21 us 5.00 us 110968164.00 us 139545956
INODELK
39.91 9656.48 us 10.00 us 8137517.00 us 293268060
LOOKUP
here is node24
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 22124594
FORGET
0.00 0.00 us 0.00 us 0.00 us 89290582
RELEASE
0.00 0.00 us 0.00 us 0.00 us 26657287
RELEASEDIR
0.00 47.00 us 47.00 us 47.00 us 1
FGETXATTR
0.00 308.67 us 8.00 us 1405672.00 us
39285 LK
0.00 745.82 us 32.00 us 1690066.00 us 86586
FTRUNCATE
0.00 388.58 us 49.00 us 1348668.00 us 274927
LINK
0.00 1008.11 us 158.00 us 2443763.00 us 293423
RENAME
0.01 1094.49 us 31.00 us 2857159.00 us 290615
SETATTR
0.02 304.24 us 24.00 us 2878581.00 us 3506688
SETXATTR
0.03 279.83 us 5.00 us 3716543.00 us 6977266
READDIR
0.05 10919.43 us 83.00 us 5075633.00 us 287979
RMDIR
0.05 692.45 us 12.00 us 3951452.00 us 4692109
OPENDIR
0.06 465.87 us 6.00 us 3726826.00 us 8238785
STAT
0.07 1187.15 us 14.00 us 5361516.00 us 3626802
GETXATTR
0.07 6308.14 us 50.00 us 4281153.00 us 705476
UNLINK
0.07 16729.47 us 148.00 us 3238674.00 us 276299
MKNOD
0.08 553.69 us 8.00 us 2721668.00 us 8744855
STATFS
0.09 1462.59 us 4.00 us 5488045.00 us 3903587
FLUSH
0.10 16979.85 us 130.00 us 3471136.00 us 376279
CREATE
0.12 4818.36 us 9.00 us 6101767.00 us 1577172
FSTAT
0.15 315.32 us 24.00 us 3801518.00 us 29090837
WRITE
0.19 2539.98 us 48.00 us 4657386.00 us 4586952
READDIRP
0.23 3794.04 us 15.00 us 6487700.00 us 3798788
OPEN
0.37 393.76 us 10.00 us 3284611.00 us 58491958
READ
0.88 1524.40 us 60.00 us 7456834.00 us 36097324
FXATTROP
1.63 4429.64 us 72.00 us 7194041.00 us 22984938
XATTROP
1.74 31485.11 us 143.00 us 4705647.00 us 3458000
MKDIR
2.08 2010.98 us 4.00 us 7669056.00 us 64626004
FINODELK
18.35 11708.39 us 4.00 us 7193745.00 us 98037767
ENTRYLK
31.62 14170.24 us 5.00 us 7194060.00 us 139544869
INODELK
41.94 9273.78 us 10.00 us 7193886.00 us 282853490
LOOKUP
On Wed, Jan 22, 2014 at 12:05 PM, Mingfan Lu <mingfan.lu at gmail.com> wrote:
> I have a volume (distribute-replica (*3)), today i found an interesting
> problem
>
> node22 node23 and node24 are the replica-7 from client A
> but the annoying thing is when I create dir or write file from client to
> replica-7,
>
> date;dd if=/dev/zero of=49 bs=1MB count=120
> Wed Jan 22 11:51:41 CST 2014
> 120+0 records in
> 120+0 records out
> 120000000 bytes (120 MB) copied, 1.96257 s, 61.1 MB/s
>
> but I could only find node23 & node24 have the find
> ---------------
> node23,node24
> ---------------
> /mnt/xfsd/test-volume/test/49
>
> in clientA, I use find command
>
> I use another machine as client B, and mount the test volume (newly
> mounted)
> to run* find /mnt/xfsd/test-volume/test/49*
>
> from Client A, the three nodes have the file now.
>
> ---------------
> node22,node23.node24
> ---------------
> /mnt/xfsd/test-volume/test/49
>
> but in Client A, I delete the file /mnt/xfsd/test-volume/test/49, node22
> still have the file in brick.
>
> ---------------
> node22
> ---------------
> /mnt/xfsd/test-volume/test/49
>
> but if i delete the new created files from Client B )
> my question is why node22 have no newly created/write dirs/files? I have
> to use find to trigger the self-heal to fix that?
>
> from ClientA's log, I find something like:
>
> I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-test-volume-replicate-7:
> no active sinks for performing self-heal on file /test/49
>
> It is harmless for it is information level?
>
> I also see something like:
> [2014-01-19 10:23:48.422757] E
> [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk]
> 0-test-volume-replicate-7: Non Blocking entrylks failed for
> /test/video/2014/01.
> [2014-01-19 10:23:48.423042] E
> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk]
> 0-test-volume-replicate-7: background entry self-heal failed on
> /test/video/2014/01
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140123/8fd7a25e/attachment.html>
More information about the Gluster-users
mailing list