[Gluster-users] Unable to get volume stat after lots of small files created

[Fred Yang]

Hi,

I have three nodes cluster running GlusterFS v3.7.6 just built and are
in the progress running various tests.

For the small file test, after ~120,000 64K files written to one
volume, access to the volume slow down significantly. "ls -lR
/gluster_nj" took 3+ minutes to complete. 120,000 files under ~1000
directories under volume root.

Two questions :
1. Any tuning I can do to improve small file meta data query?
2. Right after creation of the 120,000 small files, the gluster CLI
function also get impacted:

sanlnjpnfsbu01 /etc/glusterfs # gluster volume profile rsvol1 info
Another transaction is in progress for rsvol1. Please try again after sometime.

I verified no profile command ran on anywhere else in the cluster.
Retried the command and getting same error for 30+ minutes.

At the time when "volume profile" command failed, I can see below
error in etc-glusterfs-glusterd.vol.log :

[2016-02-10 20:25:49.757758] W
[glusterd-locks.c:577:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(__glusterd_handle_cli_profil
e_volume+0x2eb) [0x7f401889796b]
-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_op_txn_begin+0x51a)
[0x7f40188973ba] -->/usr/lib64/glusterfs/3.
7.6/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_lock+0x1ed)
[0x7f4018938add] ) 0-management: Lock for rsvol1 held by
2e3e526a-1e9a-4f6d-abee-c42d84f3bb16
[2016-02-10 20:25:49.757776] E [MSGID: 106119]
[glusterd-handler.c:728:glusterd_op_txn_begin] 0-management: Unable to
acquire lock for rsvol1

Any idea what's preventing the profile command acquiring lock for the volume?

Thanks,
Fred