[Gluster-users] Issue with removing directories after disabling quotas
JF Le Fillâtre
jean-francois.lefillatre at uni.lu
Mon Mar 16 10:45:24 UTC 2015
Hello all,
So, another day another issue. I was trying to play with quotas on my volume:
================================================================================
[root at stor104 ~]# gluster volume status
Status of volume: live
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick stor104:/zfs/brick0/brick 49167 Y 13446
Brick stor104:/zfs/brick1/brick 49168 Y 13457
Brick stor104:/zfs/brick2/brick 49169 Y 13468
Brick stor106:/zfs/brick0/brick 49159 Y 14158
Brick stor106:/zfs/brick1/brick 49160 Y 14169
Brick stor106:/zfs/brick2/brick 49161 Y 14180
NFS Server on localhost 2049 Y 13483
Quota Daemon on localhost N/A Y 13490
NFS Server on stor106 2049 Y 14195
Quota Daemon on stor106 N/A Y 14202
Task Status of Volume live
------------------------------------------------------------------------------
Task : Rebalance
ID : 6bd03709-1f48-49a9-a215-d0a6e6f3ab1e
Status : completed
================================================================================
Not sure if the "Quota Daemon on localhost -> N/A" is normal, but that's another topic.
While the quotas were enabled, I did some test copying a whole tree of small files (the Linux kernel sources) to the volume to see which performance I would get, and it was really low. So I decided to disable quotas again:
================================================================================
[root at stor104 ~]# gluster volume status
Status of volume: live
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick stor104:/zfs/brick0/brick 49167 Y 13754
Brick stor104:/zfs/brick1/brick 49168 Y 13765
Brick stor104:/zfs/brick2/brick 49169 Y 13776
Brick stor106:/zfs/brick0/brick 49159 Y 14282
Brick stor106:/zfs/brick1/brick 49160 Y 14293
Brick stor106:/zfs/brick2/brick 49161 Y 14304
NFS Server on localhost 2049 Y 13790
NFS Server on stor106 2049 Y 14319
Task Status of Volume live
------------------------------------------------------------------------------
Task : Rebalance
ID : 6bd03709-1f48-49a9-a215-d0a6e6f3ab1e
Status : completed
================================================================================
I remounted the volume from the client and tried deleting the directory containing the sources, which gave me a very long list of this:
================================================================================
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/ftrace/test.d/kprobe’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/ptrace’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/rcutorture/configs/rcu/v0.0’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/rcutorture/configs/rcu/v3.5’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/powerpc’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/perf/scripts/python/Perf-Trace-Util’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/perf/Documentation’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/perf/ui/tui’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/perf/util/include’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/lib’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/virtio’: Directory not empty
rm: cannot remove ‘/glusterfs/live/linux-3.18.7/virt/kvm’: Directory not empty
================================================================================
I did my homework on Google, yet the information I found was that the cause for this is that the contents of the bricks have been modified locally. It's definitely not the case here, I have *not* touched the contents of the bricks.
So my question is: is it possible that disabling the quotas may have had some side effects on the metadata of Gluster? If so, what can I do to force Gluster to rescan all local directories and "import" local files?
GlusterFS version: 3.6.2
The setup of my volume:
================================================================================
[root at stor104 ~]# gluster volume info
Volume Name: live
Type: Distribute
Volume ID:
Status: Started
Number of Bricks: 6
Transport-type: tcp
Bricks:
Brick1: stor104:/zfs/brick0/brick
Brick2: stor104:/zfs/brick1/brick
Brick3: stor104:/zfs/brick2/brick
Brick4: stor106:/zfs/brick0/brick
Brick5: stor106:/zfs/brick1/brick
Brick6: stor106:/zfs/brick2/brick
Options Reconfigured:
features.quota: off
performance.readdir-ahead: on
nfs.volume-access: read-only
cluster.data-self-heal-algorithm: full
performance.strict-write-ordering: off
performance.strict-o-direct: off
performance.force-readdirp: off
performance.write-behind-window-size: 4MB
performance.io-thread-count: 32
performance.flush-behind: on
performance.client-io-threads: on
performance.cache-size: 32GB
performance.cache-refresh-timeout: 60
performance.cache-max-file-size: 4MB
nfs.disable: off
cluster.eager-lock: on
cluster.min-free-disk: 1%
server.allow-insecure: on
diagnostics.client-log-level: ERROR
diagnostics.brick-log-level: ERROR
================================================================================
It is mounted from the client with those fstab options:
================================================================================
stor104:live defaults,backupvolfile-server=stor106,direct-io-mode=disable,noauto
================================================================================
Attached are the log files from stor104
Thanks a lot for any help!
JF
--
Jean-François Le Fillâtre
-------------------------------
HPC Systems Administrator
LCSB - University of Luxembourg
-------------------------------
PGP KeyID 0x134657C6
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logs-glusterfs-stor104.tar.gz
Type: application/gzip
Size: 1250001 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150316/8f7791e0/attachment-0001.bin>
More information about the Gluster-users
mailing list