[Gluster-users] Issue with removing directories after disabling quotas
Vijaikumar M
vmallika at redhat.com
Mon Mar 16 12:00:41 UTC 2015
On Monday 16 March 2015 05:04 PM, JF Le Fillâtre wrote:
> Hello Vijay,
>
> Mmmm sorry, I jumped the gun and deleted by hand all the files present
> on the bricks, so I can't see if there's any link anywhere...
>
> It's not the first time I have seen this. Very early in my tests I had
> another case of files still present on the bricks but not visible in
> Gluster, which I solved the same way. At that point I chalked it up to
> my limited knowledge of GlusterFS and I assumed that I had misconfigured it.
>
> What do you suspect that it may be, and what do I have to look for if it
> ever happens again?
We wanted to check file attributes and xattrs on the bricks to see why
some files where not deleted.
Is this problem easily re-creatable? If yes, can you please provide the
test-case?
Thanks,
Vijay
> Thanks!
> JF
>
>
>
> On 16/03/15 12:25, Vijaikumar M wrote:
>> Hi JF,
>>
>> This may not be a quota issue. Can you please check if there are any
>> linkto files exists on the brick?
>> On node 'stor104/106', can we get the below output
>>
>> #find /zfs/brick0/brick | xargs ls -l
>> #find /zfs/brick1/brick | xargs ls -l
>> #find /zfs/brick2/brick | xargs ls -l
>>
>> #find /zfs/brick0/brick | xargs getfattr -d -m . -e hex
>> #find /zfs/brick1/brick | xargs getfattr -d -m . -e hex
>> #find /zfs/brick2/brick | xargs getfattr -d -m . -e hex
>>
>>
>> Thanks,
>> Vijay
>>
>>
>>
>> On Monday 16 March 2015 04:18 PM, JF Le Fillâtre wrote:
>>> Forgot to mention: the non-empty directories list files like this:
>>>
>>> -?????????? ? ? ? ? ? Hal8723APhyReg.h
>>> -?????????? ? ? ? ? ? Hal8723UHWImg_CE.h
>>> -?????????? ? ? ? ? ? hal_com.h
>>> -?????????? ? ? ? ? ? HalDMOutSrc8723A.h
>>> -?????????? ? ? ? ? ? HalHWImg8723A_BB.h
>>> -?????????? ? ? ? ? ? HalHWImg8723A_FW.h
>>> -?????????? ? ? ? ? ? HalHWImg8723A_MAC.h
>>> -?????????? ? ? ? ? ? HalHWImg8723A_RF.h
>>> -?????????? ? ? ? ? ? hal_intf.h
>>> -?????????? ? ? ? ? ? HalPwrSeqCmd.h
>>> -?????????? ? ? ? ? ? ieee80211.h
>>> -?????????? ? ? ? ? ? odm_debug.h
>>>
>>> Thanks,
>>> JF
>>>
>>>
>>> On 16/03/15 11:45, JF Le Fillâtre wrote:
>>>> Hello all,
>>>>
>>>> So, another day another issue. I was trying to play with quotas on my
>>>> volume:
>>>>
>>>> ================================================================================
>>>>
>>>> [root at stor104 ~]# gluster volume status
>>>> Status of volume: live
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>>
>>>> Brick stor104:/zfs/brick0/brick 49167 Y 13446
>>>> Brick stor104:/zfs/brick1/brick 49168 Y 13457
>>>> Brick stor104:/zfs/brick2/brick 49169 Y 13468
>>>> Brick stor106:/zfs/brick0/brick 49159 Y 14158
>>>> Brick stor106:/zfs/brick1/brick 49160 Y 14169
>>>> Brick stor106:/zfs/brick2/brick 49161 Y 14180
>>>> NFS Server on localhost 2049 Y 13483
>>>> Quota Daemon on localhost N/A Y 13490
>>>> NFS Server on stor106 2049 Y 14195
>>>> Quota Daemon on stor106 N/A Y 14202
>>>> Task Status of Volume live
>>>> ------------------------------------------------------------------------------
>>>>
>>>> Task : Rebalance
>>>> ID : 6bd03709-1f48-49a9-a215-d0a6e6f3ab1e
>>>> Status : completed
>>>> ================================================================================
>>>>
>>>>
>>>>
>>>> Not sure if the "Quota Daemon on localhost -> N/A" is normal, but
>>>> that's another topic.
>>>>
>>>> While the quotas were enabled, I did some test copying a whole tree
>>>> of small files (the Linux kernel sources) to the volume to see which
>>>> performance I would get, and it was really low. So I decided to
>>>> disable quotas again:
>>>>
>>>>
>>>> ================================================================================
>>>>
>>>> [root at stor104 ~]# gluster volume status
>>>> Status of volume: live
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>>
>>>> Brick stor104:/zfs/brick0/brick 49167 Y 13754
>>>> Brick stor104:/zfs/brick1/brick 49168 Y 13765
>>>> Brick stor104:/zfs/brick2/brick 49169 Y 13776
>>>> Brick stor106:/zfs/brick0/brick 49159 Y 14282
>>>> Brick stor106:/zfs/brick1/brick 49160 Y 14293
>>>> Brick stor106:/zfs/brick2/brick 49161 Y 14304
>>>> NFS Server on localhost 2049 Y 13790
>>>> NFS Server on stor106 2049 Y 14319
>>>> Task Status of Volume live
>>>> ------------------------------------------------------------------------------
>>>>
>>>> Task : Rebalance
>>>> ID : 6bd03709-1f48-49a9-a215-d0a6e6f3ab1e
>>>> Status : completed
>>>> ================================================================================
>>>>
>>>>
>>>>
>>>> I remounted the volume from the client and tried deleting the
>>>> directory containing the sources, which gave me a very long list of
>>>> this:
>>>>
>>>>
>>>> ================================================================================
>>>>
>>>> rm: cannot remove
>>>> ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/ftrace/test.d/kprobe’:
>>>> Directory not empty
>>>> rm: cannot remove
>>>> ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/ptrace’:
>>>> Directory not empty
>>>> rm: cannot remove
>>>> ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/rcutorture/configs/rcu/v0.0’:
>>>> Directory not empty
>>>> rm: cannot remove
>>>> ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/rcutorture/configs/rcu/v3.5’:
>>>> Directory not empty
>>>> rm: cannot remove
>>>> ‘/glusterfs/live/linux-3.18.7/tools/testing/selftests/powerpc’:
>>>> Directory not empty
>>>> rm: cannot remove
>>>> ‘/glusterfs/live/linux-3.18.7/tools/perf/scripts/python/Perf-Trace-Util’:
>>>> Directory not empty
>>>> rm: cannot remove
>>>> ‘/glusterfs/live/linux-3.18.7/tools/perf/Documentation’: Directory
>>>> not empty
>>>> rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/perf/ui/tui’:
>>>> Directory not empty
>>>> rm: cannot remove
>>>> ‘/glusterfs/live/linux-3.18.7/tools/perf/util/include’: Directory not
>>>> empty
>>>> rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/lib’: Directory
>>>> not empty
>>>> rm: cannot remove ‘/glusterfs/live/linux-3.18.7/tools/virtio’:
>>>> Directory not empty
>>>> rm: cannot remove ‘/glusterfs/live/linux-3.18.7/virt/kvm’: Directory
>>>> not empty
>>>> ================================================================================
>>>>
>>>>
>>>>
>>>> I did my homework on Google, yet the information I found was that the
>>>> cause for this is that the contents of the bricks have been modified
>>>> locally. It's definitely not the case here, I have *not* touched the
>>>> contents of the bricks.
>>>>
>>>> So my question is: is it possible that disabling the quotas may have
>>>> had some side effects on the metadata of Gluster? If so, what can I
>>>> do to force Gluster to rescan all local directories and "import"
>>>> local files?
>>>>
>>>> GlusterFS version: 3.6.2
>>>>
>>>> The setup of my volume:
>>>>
>>>>
>>>> ================================================================================
>>>>
>>>> [root at stor104 ~]# gluster volume info
>>>> Volume Name: live
>>>> Type: Distribute
>>>> Volume ID:
>>>> Status: Started
>>>> Number of Bricks: 6
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: stor104:/zfs/brick0/brick
>>>> Brick2: stor104:/zfs/brick1/brick
>>>> Brick3: stor104:/zfs/brick2/brick
>>>> Brick4: stor106:/zfs/brick0/brick
>>>> Brick5: stor106:/zfs/brick1/brick
>>>> Brick6: stor106:/zfs/brick2/brick
>>>> Options Reconfigured:
>>>> features.quota: off
>>>> performance.readdir-ahead: on
>>>> nfs.volume-access: read-only
>>>> cluster.data-self-heal-algorithm: full
>>>> performance.strict-write-ordering: off
>>>> performance.strict-o-direct: off
>>>> performance.force-readdirp: off
>>>> performance.write-behind-window-size: 4MB
>>>> performance.io-thread-count: 32
>>>> performance.flush-behind: on
>>>> performance.client-io-threads: on
>>>> performance.cache-size: 32GB
>>>> performance.cache-refresh-timeout: 60
>>>> performance.cache-max-file-size: 4MB
>>>> nfs.disable: off
>>>> cluster.eager-lock: on
>>>> cluster.min-free-disk: 1%
>>>> server.allow-insecure: on
>>>> diagnostics.client-log-level: ERROR
>>>> diagnostics.brick-log-level: ERROR
>>>> ================================================================================
>>>>
>>>>
>>>>
>>>> It is mounted from the client with those fstab options:
>>>>
>>>> ================================================================================
>>>>
>>>> stor104:live
>>>> defaults,backupvolfile-server=stor106,direct-io-mode=disable,noauto
>>>> ================================================================================
>>>>
>>>>
>>>> Attached are the log files from stor104
>>>>
>>>> Thanks a lot for any help!
>>>> JF
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
More information about the Gluster-users
mailing list