[Gluster-users] intresting issue of replication and self-heal
Mingfan Lu
mingfan.lu at gmail.com
Sat Jan 25 16:20:41 UTC 2014
more hints:
I found node23 & node24 have many files in
.glusterfs/indices/xattrop
there should be some problem? who could give some suggestions to resolve it?
On Thu, Jan 23, 2014 at 5:04 PM, Mingfan Lu <mingfan.lu at gmail.com> wrote:
> I profiled node22, I found that most latency comes from setxattr, where
> node23 & node22 comes from lookup and locks. any one could help?
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 2437540
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 252684
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 2226292
> RELEASEDIR
> 0.00 38.00 us 37.00 us 40.00 us 4
> FGETXATTR
> 0.00 66.16 us 15.00 us 13139.00 us 596
> GETXATTR
> 0.00 239.14 us 58.00 us 126477.00 us 1967
> LINK
> 0.00 51.85 us 14.00 us 8298.00 us 19045
> STAT
> 0.00 165.50 us 9.00 us 212057.00 us 20544
> READDIR
> 0.00 1827.92 us 184.00 us 150298.00 us 2084
> RENAME
> 0.00 49.14 us 12.00 us 5908.00 us 189019
> STATFS
> 0.00 84.63 us 14.00 us 96016.00 us 163405
> READ
> 0.00 29968.76 us 156.00 us 1073902.00 us 3115
> CREATE
> 0.00 1340.25 us 6.00 us 7415357.00 us 248141
> FLUSH
> 0.00 1616.76 us 32.00 us 13865122.00 us 229190
> FTRUNCATE
> 0.01 1807.58 us 19.00 us 55480776.00 us
> 249569 OPEN
> 0.01 1875.11 us 10.00 us 8842171.00 us 465197
> FSTAT
> 0.05 393296.28 us 52.00 us 56856581.00 us 9057
> UNLINK
> 0.07 32291.01 us 192.00 us 9638107.00 us 156081
> RMDIR
> 0.08 18339.18 us 140.00 us 5313885.00 us 337862
> MKNOD
> 0.09 2904.39 us 18.00 us 51724741.00 us 2226290
> OPENDIR
> 0.15 4708.15 us 27.00 us 55115760.00 us 2334864
> SETXATTR
> 0.18 8965.91 us 68.00 us 26465968.00 us 1513280
> FXATTROP
> 0.21 3465.29 us 74.00 us 58580783.00 us 4506602
> XATTROP
> 0.28 4801.16 us 44.00 us 49643138.00 us 4436847
> READDIRP
> 0.37 5935.92 us 7.00 us 56449083.00 us 4611760
> ENTRYLK
> 1.02 4226.58 us 33.00 us 63494729.00 us 18092335
> WRITE
> 1.50 2734.50 us 6.00 us 185109908.00 us 40971541
> INODELK
> 4.75 348602.30 us 5.00 us 2185602946.00 us 1019332
> FINODELK
>
>
> * 14.98 33957.49 us 14.00 us 59261447.00 us 32998211
> LOOKUP 26.30 807063.74 us 150.00 us 68086266.00 us
> 2438422 MKDIR 49.95 457402.30 us 20.00 us 67894186.00
> <67894186.00> us 8171751 SETATTR*
>
> Duration: 353678 seconds
> Data Read: 21110920120 bytes
> Data Written: 2338403381483 bytes
>
> here is node23
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 22125898
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 89286732
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 32865496
> RELEASEDIR
> 0.00 35.50 us 23.00 us 48.00 us 2
> FGETXATTR
> 0.00 164.04 us 29.00 us 749181.00 us 39320
> FTRUNCATE
> 0.00 483.71 us 8.00 us 2688755.00 us
> 39288 LK
> 0.00 419.61 us 48.00 us 2183971.00 us 274939
> LINK
> 0.00 970.55 us 145.00 us 2471745.00 us 293435
> RENAME
> 0.00 1346.63 us 35.00 us 4462970.00 us 243238
> SETATTR
> 0.01 285.51 us 25.00 us 2588685.00 us 3459436
> SETXATTR
> 0.03 323.11 us 5.00 us 2074581.00 us 6977304
> READDIR
> 0.05 12200.60 us 84.00 us 3943421.00 us 287979
> RMDIR
> 0.07 592.75 us 7.00 us 3592073.00 us 8129847
> STAT
> 0.07 6938.50 us 49.00 us 3268036.00 us 705818
> UNLINK
> 0.08 19468.78 us 149.00 us 3664022.00 us 276310
> MKNOD
> 0.09 763.31 us 8.00 us 3396903.00 us 8731725
> STATFS
> 0.09 1715.79 us 4.00 us 5626912.00 us 3902746
> FLUSH
> 0.10 4614.74 us 9.00 us 5835691.00 us 1574923
> FSTAT
> 0.10 1189.55 us 13.00 us 6043163.00 us 6129885
> OPENDIR
> 0.10 19729.66 us 131.00 us 4112832.00 us 376286
> CREATE
> 0.13 328.26 us 24.00 us 2410049.00 us 29091424
> WRITE
> 0.20 2107.64 us 10.00 us 5765196.00 us 6675496
> GETXATTR
> 0.28 5317.38 us 14.00 us 7549301.00 us 3798543
> OPEN
> 0.71 7042.79 us 47.00 us 5848284.00 us 7125716
> READDIRP
> 0.80 743.88 us 10.00 us 7979373.00 us 76781383
> READ
> 0.93 1802.29 us 60.00 us 11040319.00 us 36501360
> FXATTROP
> 1.76 36083.12 us 141.00 us 3548175.00 us 3458135
> MKDIR
> 1.83 5046.35 us 70.00 us 8120221.00 us 25765615
> XATTROP
> 11.74 12896.99 us 4.00 us 2141920969.00 us 64590600
> FINODELK
> 15.43 11171.78 us 5.00 us 909115697.00 us 98040443
> ENTRYLK
> 25.46 12945.21 us 5.00 us 110968164.00 us 139545956
> INODELK
> 39.91 9656.48 us 10.00 us 8137517.00 us 293268060
> LOOKUP
>
> here is node24
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 22124594
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 89290582
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 26657287
> RELEASEDIR
> 0.00 47.00 us 47.00 us 47.00 us 1
> FGETXATTR
> 0.00 308.67 us 8.00 us 1405672.00 us
> 39285 LK
> 0.00 745.82 us 32.00 us 1690066.00 us 86586
> FTRUNCATE
> 0.00 388.58 us 49.00 us 1348668.00 us 274927
> LINK
> 0.00 1008.11 us 158.00 us 2443763.00 us 293423
> RENAME
> 0.01 1094.49 us 31.00 us 2857159.00 us 290615
> SETATTR
> 0.02 304.24 us 24.00 us 2878581.00 us 3506688
> SETXATTR
> 0.03 279.83 us 5.00 us 3716543.00 us 6977266
> READDIR
> 0.05 10919.43 us 83.00 us 5075633.00 us 287979
> RMDIR
> 0.05 692.45 us 12.00 us 3951452.00 us 4692109
> OPENDIR
> 0.06 465.87 us 6.00 us 3726826.00 us 8238785
> STAT
> 0.07 1187.15 us 14.00 us 5361516.00 us 3626802
> GETXATTR
> 0.07 6308.14 us 50.00 us 4281153.00 us 705476
> UNLINK
> 0.07 16729.47 us 148.00 us 3238674.00 us 276299
> MKNOD
> 0.08 553.69 us 8.00 us 2721668.00 us 8744855
> STATFS
> 0.09 1462.59 us 4.00 us 5488045.00 us 3903587
> FLUSH
> 0.10 16979.85 us 130.00 us 3471136.00 us 376279
> CREATE
> 0.12 4818.36 us 9.00 us 6101767.00 us 1577172
> FSTAT
> 0.15 315.32 us 24.00 us 3801518.00 us 29090837
> WRITE
> 0.19 2539.98 us 48.00 us 4657386.00 us 4586952
> READDIRP
> 0.23 3794.04 us 15.00 us 6487700.00 us 3798788
> OPEN
> 0.37 393.76 us 10.00 us 3284611.00 us 58491958
> READ
> 0.88 1524.40 us 60.00 us 7456834.00 us 36097324
> FXATTROP
> 1.63 4429.64 us 72.00 us 7194041.00 us 22984938
> XATTROP
> 1.74 31485.11 us 143.00 us 4705647.00 us 3458000
> MKDIR
> 2.08 2010.98 us 4.00 us 7669056.00 us 64626004
> FINODELK
> 18.35 11708.39 us 4.00 us 7193745.00 us 98037767
> ENTRYLK
> 31.62 14170.24 us 5.00 us 7194060.00 us 139544869
> INODELK
> 41.94 9273.78 us 10.00 us 7193886.00 us 282853490
> LOOKUP
>
>
>
> On Wed, Jan 22, 2014 at 12:05 PM, Mingfan Lu <mingfan.lu at gmail.com> wrote:
>
>> I have a volume (distribute-replica (*3)), today i found an interesting
>> problem
>>
>> node22 node23 and node24 are the replica-7 from client A
>> but the annoying thing is when I create dir or write file from client to
>> replica-7,
>>
>> date;dd if=/dev/zero of=49 bs=1MB count=120
>> Wed Jan 22 11:51:41 CST 2014
>> 120+0 records in
>> 120+0 records out
>> 120000000 bytes (120 MB) copied, 1.96257 s, 61.1 MB/s
>>
>> but I could only find node23 & node24 have the find
>> ---------------
>> node23,node24
>> ---------------
>> /mnt/xfsd/test-volume/test/49
>>
>> in clientA, I use find command
>>
>> I use another machine as client B, and mount the test volume (newly
>> mounted)
>> to run* find /mnt/xfsd/test-volume/test/49*
>>
>> from Client A, the three nodes have the file now.
>>
>> ---------------
>> node22,node23.node24
>> ---------------
>> /mnt/xfsd/test-volume/test/49
>>
>> but in Client A, I delete the file /mnt/xfsd/test-volume/test/49, node22
>> still have the file in brick.
>>
>> ---------------
>> node22
>> ---------------
>> /mnt/xfsd/test-volume/test/49
>>
>> but if i delete the new created files from Client B )
>> my question is why node22 have no newly created/write dirs/files? I have
>> to use find to trigger the self-heal to fix that?
>>
>> from ClientA's log, I find something like:
>>
>> I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-test-volume-replicate-7:
>> no active sinks for performing self-heal on file /test/49
>>
>> It is harmless for it is information level?
>>
>> I also see something like:
>> [2014-01-19 10:23:48.422757] E
>> [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk]
>> 0-test-volume-replicate-7: Non Blocking entrylks failed for
>> /test/video/2014/01.
>> [2014-01-19 10:23:48.423042] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk]
>> 0-test-volume-replicate-7: background entry self-heal failed on
>> /test/video/2014/01
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140126/acd2b04e/attachment.html>
More information about the Gluster-users
mailing list