[Bugs] [Bug 1288352] Few snapshot creation fails with pre-validation failed message on tiered volume.

bugzilla at redhat.com bugzilla at redhat.com
Fri Dec 4 05:00:25 UTC 2015


https://bugzilla.redhat.com/show_bug.cgi?id=1288352



--- Comment #1 from Nithya Balachandran <nbalacha at redhat.com> ---
Description of problem:

Running snapshot creation in loop fails some of the snap creation with
pre-validation failed message on tiered volume while large file creation is in
progress.


Version-Release number of selected component (if applicable):

glusterfs-3.7.5-5

How reproducible:

Always

Steps to Reproduce:

1.Create a tiered volume with 2x(4+2) EC cold tier and 2x2 dist-rep hot tier
2.FUSE mount the volume on a client. 
3.Start creating 20000 files on the mount point each of 100KB.
4.Simultaneously start creating 100 snapshots in loop with sleep of 5 in 2 snap
creation.
5.Observe that few of the snapshot creation fails while the IO is in progress.
(in my case 11 out of 100 snapshots fails with pre-validation failed message).
6.Below are the failed messages in logs:

[2015-11-05 21:56:02.975294] E [MSGID: 106062]
[glusterd-snapshot.c:1504:glusterd_snap_create_clone_pre_val_use_rsp_dict]
0-management: failed to get the volume count
[2015-11-05 21:56:02.975451] E [MSGID: 106062]
[glusterd-snapshot.c:1813:glusterd_snap_pre_validate_use_rsp_dict]
0-management: Unable to use rsp dict
[2015-11-05 21:56:02.975459] E [MSGID: 106122]
[glusterd-mgmt.c:600:glusterd_pre_validate_aggr_rsp_dict] 0-management: Failed
to aggregate prevalidate response dictionaries.
[2015-11-05 21:56:02.975467] E [MSGID: 106108]
[glusterd-mgmt.c:701:gd_mgmt_v3_pre_validate_cbk_fn] 0-management: Failed to
aggregate response from  node/brick
[2015-11-05 21:56:02.975497] E [MSGID: 106116]
[glusterd-mgmt.c:134:gd_mgmt_v3_collate_errors] 0-management: Pre Validation
failed on 10.70.35.140. Please check log file for details.
[2015-11-05 21:56:05.833521] W [socket.c:588:__socket_rwv] 0-nfs: readv on
/var/run/gluster/11f5a41d4df7a19d42d4e641eb784bfa.socket failed (Invalid
argument)
The message "I [MSGID: 106006]
[glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: nfs has
disconnected from glusterd." repeated 28 times between [2015-11-05
21:54:39.740819] and [2015-11-05 21:56:05.833576]
[2015-11-05 21:56:08.702620] E [MSGID: 106122]
[glusterd-mgmt.c:883:glusterd_mgmt_v3_pre_validate] 0-management: Pre
Validation failed on peers
[2015-11-05 21:56:08.702694] E [MSGID: 106122]
[glusterd-mgmt.c:2164:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre
Validation Failed


[2015-11-05 22:04:59.652100] E [MSGID: 106572]
[glusterd-snapshot.c:1998:glusterd_snapshot_pause_tier] 0-management: Failed to
pause tier. Errstr=(null)
[2015-11-05 22:04:59.652159] E [MSGID: 106572]
[glusterd-snapshot.c:2592:glusterd_snapshot_create_prevalidate] 0-management:
Failed to pause tier in snap prevalidate.
[2015-11-05 22:04:59.652201] W [MSGID: 106030]
[glusterd-snapshot.c:8380:glusterd_snapshot_prevalidate] 0-management: Snapshot
create pre-validation failed
[2015-11-05 22:04:59.652215] W [MSGID: 106122]
[glusterd-mgmt.c:166:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot
Prevalidate Failed
[2015-11-05 22:04:59.652228] E [MSGID: 106122]
[glusterd-mgmt.c:820:glusterd_mgmt_v3_pre_validate] 0-management: Pre
Validation failed for operation Snapshot on local node
[2015-11-05 22:04:59.652247] E [MSGID: 106122]
[glusterd-mgmt.c:2164:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre
Validation Failed

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list