[Bugs] [Bug 1472758] Running sysbench on vm disk from plain distribute gluster volume causes disk corruption

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1472758

Johan Bernhardsson <johan at kafit.se> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |johan at kafit.se



--- Comment #4 from Johan Bernhardsson <johan at kafit.se> ---
This also seem to affect disperse volumes.  Attaching logs from when we try to
copy an image with qemu-img from one storage to another.

We get this on vm storage configured like this:
Volume Name: fs02
Type: Disperse
Volume ID: 7f3d96e7-8d1e-48b8-bad0-dc5b3de13b38
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: vbgsan01:/gluster/fs02/fs02
Brick2: vbgsan02:/gluster/fs02/fs02
Brick3: vbgsan03:/gluster/fs02/fs02


from one brick log:
[2017-07-30 15:29:43.815362] E [MSGID: 113002] [posix.c:266:posix_lookup]
0-fs02-posix: buf->ia_gfid is null for
/gluster/fs02/fs02/.shard/fea7c5e9-f1a7-4684-bd5c-08383af3a2fb.5079 [No data
available]
[2017-07-30 15:29:43.815527] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-fs02-server: 400238: LOOKUP
/.shard/fea7c5e9-f1a7-4684-bd5c-08383af3a2fb.5079
(be318638-e8a0-4c6d-977d-7a937aa84806/fea7c5e9-f1a7-4684-bd5c-08383af3a2fb.5079)
==> (No data available) 
[No data available]
[2017-07-30 15:29:46.837927] W [MSGID: 113096]
[posix-handle.c:761:posix_handle_hard] 0-fs02-posix: link
/gluster/fs02/fs02/.shard/fea7c5e9-f1a7-4684-bd5c-08383af3a2fb.5180 ->
/gluster/fs02/fs02/.glusterfs/1e/2e/1e2e59bb-f8e3-41d1-93c6-db795bbc96d4failed 
[File exists]
[2017-07-30 15:29:46.837962] E [MSGID: 113020] [posix.c:1402:posix_mknod]
0-fs02-posix: setting gfid on
/gluster/fs02/fs02/.shard/fea7c5e9-f1a7-4684-bd5c-08383af3a2fb.5180 failed
[2017-07-30 15:29:51.418415] E [MSGID: 113002] [posix.c:266:posix_lookup]
0-fs02-posix: buf->ia_gfid is null for
/gluster/fs02/fs02/.shard/fea7c5e9-f1a7-4684-bd5c-08383af3a2fb.5315 [No data
available]
[2017-07-30 15:29:51.418471] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-fs02-server: 414456: LOOKUP
/.shard/fea7c5e9-f1a7-4684-bd5c-08383af3a2fb.5315
(be318638-e8a0-4c6d-977d-7a937aa84806/fea7c5e9-f1a7-4684-bd5c-08383af3a2fb.5315)
==> (No data available) 
[No data available]


>From mount log:
[2017-07-30 13:09:35.041843] I [MSGID: 109066] [dht-rename.c:1569:dht_rename]
0-fs02-dht: renaming
/0924ff77-ef51-435b-b90d-50bfbf2e8de7/images/cab4e8d0-a82a-4048-b0e8-a9c1bd2e38bf/797faedf-b7e2-4b3c-bff6-82264efa11f5.meta.new
(hash=fs02-disperse-0/cache=fs02-disperse-0) =>

/0924ff77-ef51-435b-b90d-50bfbf2e8de7/images/cab4e8d0-a82a-4048-b0e8-a9c1bd2e38bf/797faedf-b7e2-4b3c-bff6-82264efa11f5.meta
(hash=fs02-disperse-0/cache=fs02-disperse-0)
[2017-07-30 13:09:35.841507] I [MSGID: 109066] [dht-rename.c:1569:dht_rename]
0-fs02-dht: renaming
/0924ff77-ef51-435b-b90d-50bfbf2e8de7/images/cab4e8d0-a82a-4048-b0e8-a9c1bd2e38bf/4c3cc823-e977-4fa1-b233-718c445d632e.meta.new
(hash=fs02-disperse-0/cache=fs02-disperse-0) =>

/0924ff77-ef51-435b-b90d-50bfbf2e8de7/images/cab4e8d0-a82a-4048-b0e8-a9c1bd2e38bf/4c3cc823-e977-4fa1-b233-718c445d632e.meta
(hash=fs02-disperse-0/cache=<nul>)
[2017-07-30 13:36:57.236163] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-fs02-client-0: remote operation
failed. Path: /.shard/3997817a-4678-4e75-8131-438db9faca9a.1300
(00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-30 13:36:57.237191] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-fs02-client-2: remote operation
failed. Path: /.shard/3997817a-4678-4e75-8131-438db9faca9a.1300
(00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-30 13:36:57.237220] W [MSGID: 122053]
[ec-common.c:121:ec_check_status] 0-fs02-disperse-0: Operation failed on 1 of 3
subvolumes.(up=111, mask=111, remaining=000, good=101, bad=010)
[2017-07-30 13:36:57.237229] W [MSGID: 122002] [ec-common.c:71:ec_heal_report]
0-fs02-disperse-0: Heal failed [Invalid argument]
[2017-07-30 13:36:57.248807] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-fs02-client-0: remote operation
failed. Path: /.shard/3997817a-4678-4e75-8131-438db9faca9a.1301
(00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-30 13:36:57.248973] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-fs02-client-1: remote operation
failed. Path: /.shard/3997817a-4678-4e75-8131-438db9faca9a.1301
(00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-30 13:36:57.249502] W [MSGID: 122053]
[ec-common.c:121:ec_check_status] 0-fs02-disperse-0: Operation failed on 1 of 3
subvolumes.(up=111, mask=111, remaining=000, good=011, bad=100)
[2017-07-30 13:36:57.249524] W [MSGID: 122002] [ec-common.c:71:ec_heal_report]
0-fs02-disperse-0: Heal failed [Invalid argument]
[2017-07-30 13:36:57.249535] E [MSGID: 133010]
[shard.c:1725:shard_common_lookup_shards_cbk] 0-fs02-shard: Lookup on shard
1301 failed. Base file gfid = 3997817a-4678-4e75-8131-438db9faca9a [No data
available]
[2017-07-30 13:36:57.249585] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 84787: READ => -1 gfid=3997817a-4678-4e75-8131-438db9faca9a
fd=0x7f0cfa192210 (No data available)

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=nKmCXpQUVz&a=cc_unsubscribe