<div dir="ltr"><div><div><div><div><div><div>HI Mathew,<br><br></div>In order to do listing we use an auxiliary mount, It could be that this is returning cached values..<br></div>So please try the following.<br><br></div>1) unmount the auxiliary mount for the volume (would have "client pid -5" is its command line)<br>.......... /var/log/glusterfs/quota-mount-xyz.log -p /var/run/gluster/xyz.pid --client-pid -5 ....<br></div>2) do a quota list again<br></div><br>Regards,<br></div>Sanoj<br><div><div><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Sep 2, 2017 at 3:55 AM, Matthew B <span dir="ltr"><<a href="mailto:matthew.has.questions@gmail.com" target="_blank">matthew.has.questions@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Apologies - I copied and pasted the wrong ansible output: <br><br>matthew@laptop:~/playbooks$ ansible -i hosts gluster-servers[0:6] -u matthewb --ask-pass -m shell -b --become-method=sudo --ask-become-pass -a "getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/<wbr>data/projects/MEOPAR | egrep '^trusted.glusterfs.quota.<wbr>size'"<span class=""><br>SSH password: <br>SUDO password[defaults to SSH password]: <br>gluster02 | SUCCESS | rc=0 >><br></span><span class="">trusted.glusterfs.quota.size=<wbr>0x0000011ecfa56c00000000000005<wbr>cd6d000000000006d478<br>trusted.glusterfs.quota.size.<wbr>1=<wbr>0x0000010ad4a45200000000000001<wbr>2a0300000000000150fa<br><br>gluster05 | SUCCESS | rc=0 >><br></span>trusted.glusterfs.quota.size=<wbr>0x00000033b8e93800000000000004<wbr>cde9000000000006b1a4<br>trusted.glusterfs.quota.size.<wbr>1=<wbr>0x0000010dca277c00000000000001<wbr>297d0000000000015005<br><br>gluster04 | SUCCESS | rc=0 >><br>trusted.glusterfs.quota.size=<wbr>0xffffff396f3ec000000000000004<wbr>d7eb0000000000068c62<br>trusted.glusterfs.quota.size.<wbr>1=<wbr>0x00000106e6724800000000000001<wbr>138f0000000000012fb2<span class=""><br><br>gluster01 | SUCCESS | rc=0 >><br>trusted.glusterfs.quota.size=<wbr>0x0000003d4d434800000000000005<wbr>7616000000000006afd2<br>trusted.glusterfs.quota.size.<wbr>1=<wbr>0x00000133fe211e00000000000005<wbr>d161000000000006cfd4<br><br></span><span class="">gluster03 | SUCCESS | rc=0 >><br>trusted.glusterfs.quota.size=<wbr>0xfffffd02acabf000000000000003<wbr>599000000000000643e2<br>trusted.glusterfs.quota.size.<wbr>1=<wbr>0x00000114e20f5e00000000000001<wbr>13b30000000000012fb2<br><br>gluster06 | SUCCESS | rc=0 >><br>trusted.glusterfs.quota.size=<wbr>0xffffff0c98de4400000000000005<wbr>36e40000000000068cf2<br>trusted.glusterfs.quota.size.<wbr>1=<wbr>0x0000013532664e00000000000005<wbr>e73f000000000006cfd4<br><br>gluster07 | SUCCESS | rc=0 >><br></span>trusted.glusterfs.quota.size=<wbr>0x000001108e511400000000000003<wbr>27c6000000000006bf6d<br><br></div>Thanks,<br></div> -Matthew<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 1, 2017 at 3:22 PM, Matthew B <span dir="ltr"><<a href="mailto:matthew.has.questions@gmail.com" target="_blank">matthew.has.questions@gmail.<wbr>com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Thanks Sanoj, <br><br></div><div>Now the brick is showing the correct xattrs: <br></div><div>
</div><div><pre><span>[root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR
# file: /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR
security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a756e6<wbr>c6162656c65645f743a733000
trusted.gfid=0x7209b677f4b94d8<wbr>2a3820733620e6929
trusted.glusterfs.6f95525a-94d<wbr>7-4174-bac4-e1a18fe010a2.xtime<wbr>=0x599f228800088654
trusted.glusterfs.dht=0x000000<wbr>0100000000b6db6d41db6db6ee
</span><b>trusted.glusterfs.quota.d5a5ec<wbr>da-7511-4bbb-9b4c-4fcc84e3e1da<wbr>.contri=0x000001108e5114000000<wbr>0000000327c6000000000006bf6d</b>
trusted.glusterfs.quota.dirty=<wbr>0x3000
trusted.glusterfs.quota.limit-<wbr>set=0x0000088000000000ffffffff<wbr>ffffffff
<b>trusted.glusterfs.quota.size=0<wbr>x000001108e5114000000000000032<wbr>7c6000000000006bf6d</b></pre></div><div><br></div><div>However, the quota listing still shows the old (incorrect) value: <br></div><div></div><div><pre><br>[root@gluster07 ~]# gluster volume quota storage list | egrep "MEOPAR "
/data/projects/MEOPAR 8.5TB 80%(6.8TB) <b>16384.0PB</b> 10.6TB No No</pre></div><div><br></div><div>I've checked on each of the bricks and they look fine now - is there any way to reflect the new value in the quota itself? <br></div><div><br></div><div><pre>matthew@laptop:~/playbooks$ ansible -i hosts gluster-servers[0:6] -u matthewb --ask-pass -m shell -b --become-method=sudo --ask-become-pass -a "getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/dat<wbr>a/projects/comp_support | egrep '^trusted.glusterfs.quota.size<wbr>\=' | sed 's/trusted.glusterfs.quota.siz<wbr>e\=//' | cut -c 1-18 | xargs printf '%d\n'"
SSH password:
SUDO password[defaults to SSH password]:
gluster05 | SUCCESS | rc=0 >>
567293059584
gluster04 | SUCCESS | rc=0 >>
510784812032
gluster03 | SUCCESS | rc=0 >>
939742334464
gluster01 | SUCCESS | rc=0 >>
98688324096
gluster02 | SUCCESS | rc=0 >>
61449348096
gluster07 | SUCCESS | rc=0 >>
29252869632
gluster06 | SUCCESS | rc=0 >>
31899410944</pre></div><div><br></div>Thanks,<br></div> -Matthew<br></div><div class="m_-5194400459960243334HOEnZb"><div class="m_-5194400459960243334h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 1, 2017 at 4:33 AM, Sanoj Unnikrishnan <span dir="ltr"><<a href="mailto:sunnikri@redhat.com" target="_blank">sunnikri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi Mathew,<br></div><br></div>The other option is to explicitly remove the size and contri xattr at the brick path and then do a stat from the mount point.<br><br> #setfattr -x trusted.glusterfs.quota.000000<wbr>00-0000-0000-0000-000000000001<wbr>.contri.1 <brick path /dir><br> #setfattr -x trusted.glusterfs.quota.size.1<wbr> <brick path / dir><br> #stat <mount path /dir><br><br></div>Stat would heal the size and the contri xattr and the dirty xattr would heal only on the next operation on the directory.<br><br></div><div>After this you could set dirty bit and do a stat again.<br><pre>setxattr -n trusted.glusterfs.quota.dirty -v 0x3100 <brick path / dir><br></pre><pre>stat <mount path /dir></pre></div><div><br></div><div><br></div><div><div>Regards,<br></div><div>Sanoj<br></div></div></div><div class="m_-5194400459960243334m_-5496848955793637859HOEnZb"><div class="m_-5194400459960243334m_-5496848955793637859h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 31, 2017 at 9:12 PM, Matthew B <span dir="ltr"><<a href="mailto:matthew.has.questions@gmail.com" target="_blank">matthew.has.questions@gmail.c<wbr>om</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Raghavendra, <br><br></div>I didn't get a chance to implement your suggestions, however it looks like the dirty bit is no longer set - so presumably the quota should have been updated, however the quota.size attribute is still incorrect though slightly different than before. Any other suggestions? <br><span><br><tt>[root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR</tt><tt><br>
</tt><tt># file: /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR</tt><tt><br>
</tt><tt>security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a756e6<wbr>c6162656c65645f743a733000</tt><tt><br>
</tt><tt>trusted.gfid=0x7209b677f4b94d8<wbr>2a3820733620e6929</tt><tt><br>
</tt><tt>trusted.glusterfs.6f95525a-94d<wbr>7-4174-bac4-e1a18fe010a2.xtime<wbr>=0x599f228800088654</tt><tt><br>
</tt><tt>trusted.glusterfs.dht=0x000000<wbr>0100000000b6db6d41db6db6ee</tt><tt><br>
</tt></span><tt>trusted.glusterfs.quota.d5a5ec<wbr>da-7511-4bbb-9b4c-4fcc84e3e1da<wbr>.contri=0xfffffa3d7c28f6000000<wbr>0000000a9d0a000000000005fd2f</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.dirty=<wbr>0x3000</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.limit-<wbr>set=0x0000088000000000ffffffff<wbr>ffffffff</tt><tt><br>
</tt><tt>trusted.glusterfs.quota.size=0<wbr>xfffffa3d7c28f60000000000000a9<wbr>d0a000000000005fd2f</tt><br><br></div>Thanks,<br></div>-Matthew</div><div class="m_-5194400459960243334m_-5496848955793637859m_-7645030894149328762HOEnZb"><div class="m_-5194400459960243334m_-5496848955793637859m_-7645030894149328762h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 28, 2017 at 8:05 PM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
<br>
----- Original Message -----<br>
> From: "Matthew B" <<a href="mailto:matthew.has.questions@gmail.com" target="_blank">matthew.has.questions@gmail.c<wbr>om</a>><br>
</span><span>> To: "Sanoj Unnikrishnan" <<a href="mailto:sunnikri@redhat.com" target="_blank">sunnikri@redhat.com</a>><br>
> Cc: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>, "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
> Sent: Monday, August 28, 2017 9:33:25 PM<br>
> Subject: Re: [Gluster-devel] Quota Used Value Incorrect - Fix now or after upgrade<br>
><br>
> Hi Sanoj,<br>
><br>
> Thank you for the information - I have applied the changes you specified<br>
> above - but I haven't seen any changes in the xattrs on the directory after<br>
> about 15 minutes:<br>
<br>
</span>I think stat is served from cache - either gluster's md-cache or kernel attribute cache. For healing to happen we need to force a lookup (which we had hoped would be issued as part of stat cmd) and this lookup has to reach marker xlator loaded on bricks. To make sure a lookup on the directory reaches marker we need to:<br>
<br>
1. Turn off kernel attribute and entry cache (using --entrytimeout=0 and --attribute-timeout=0 as options to glusterfs while mounting)<br>
2. Turn off md-cache using gluster cli (gluster volume set performance.md-cache <volname> off)<br>
3. Turn off readdirplus in the entire stack [1]<br>
<br>
Once the above steps are done I guess doing a stat results in a lookup on the directory witnessed by marker. Once the issue is fixed you can undo the above three steps so that performance is not affected in your setup.<br>
<br>
[1] <a href="http://nongnu.13855.n7.nabble.com/Turning-off-readdirp-in-the-entire-stack-on-fuse-mount-td220297.html" rel="noreferrer" target="_blank">http://nongnu.13855.n7.nabble.<wbr>com/Turning-off-readdirp-in-th<wbr>e-entire-stack-on-fuse-mount-t<wbr>d220297.html</a><br>
<div class="m_-5194400459960243334m_-5496848955793637859m_-7645030894149328762m_4318700441570490995HOEnZb"><div class="m_-5194400459960243334m_-5496848955793637859m_-7645030894149328762m_4318700441570490995h5"><br>
><br>
> [root@gluster07 ~]# setfattr -n trusted.glusterfs.quota.dirty -v 0x3100<br>
> /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR/<br>
><br>
> [root@gluster07 ~]# stat /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR<br>
><br>
> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex<br>
> /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR<br>
> # file: /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR<br>
> security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a756e6<wbr>c6162656c65645f743a733000<br>
> trusted.gfid=0x7209b677f4b94d8<wbr>2a3820733620e6929<br>
> trusted.glusterfs.6f95525a-94d<wbr>7-4174-bac4-e1a18fe010a2.xtime<wbr>=0x599f228800088654<br>
> trusted.glusterfs.dht=0x000000<wbr>0100000000b6db6d41db6db6ee<br>
> trusted.glusterfs.quota.d5a5ec<wbr>da-7511-4bbb-9b4c-4fcc84e3e1da<wbr>.contri=0xfffffa3d7c1ba6000000<wbr>0000000a9ccb000000000005fd2f<br>
> trusted.glusterfs.quota.dirty=<wbr>0x3100<br>
> trusted.glusterfs.quota.limit-<wbr>set=0x0000088000000000ffffffff<wbr>ffffffff<br>
> trusted.glusterfs.quota.size=0<wbr>xfffffa3d7c1ba60000000000000a9<wbr>ccb000000000005fd2f<br>
><br>
> [root@gluster07 ~]# gluster volume status storage<br>
> Status of volume: storage<br>
> Gluster process TCP Port RDMA Port Online Pid<br>
> ------------------------------<wbr>------------------------------<wbr>------------------<br>
> Brick 10.0.231.50:/mnt/raid6-storage<wbr>/storag<br>
> e 49159 0 Y<br>
> 2160<br>
> Brick 10.0.231.51:/mnt/raid6-storage<wbr>/storag<br>
> e 49153 0 Y<br>
> 16037<br>
> Brick 10.0.231.52:/mnt/raid6-storage<wbr>/storag<br>
> e 49159 0 Y<br>
> 2298<br>
> Brick 10.0.231.53:/mnt/raid6-storage<wbr>/storag<br>
> e 49154 0 Y<br>
> 9038<br>
> Brick 10.0.231.54:/mnt/raid6-storage<wbr>/storag<br>
> e 49153 0 Y<br>
> 32284<br>
> Brick 10.0.231.55:/mnt/raid6-storage<wbr>/storag<br>
> e 49153 0 Y<br>
> 14840<br>
> Brick 10.0.231.56:/mnt/raid6-storage<wbr>/storag<br>
> e 49152 0 Y<br>
> 29389<br>
> NFS Server on localhost 2049 0 Y<br>
> 29421<br>
> Quota Daemon on localhost N/A N/A Y<br>
> 29438<br>
> NFS Server on 10.0.231.51 2049 0 Y<br>
> 18249<br>
> Quota Daemon on 10.0.231.51 N/A N/A Y<br>
> 18260<br>
> NFS Server on 10.0.231.55 2049 0 Y<br>
> 24128<br>
> Quota Daemon on 10.0.231.55 N/A N/A Y<br>
> 24147<br>
> NFS Server on 10.0.231.54 2049 0 Y<br>
> 9397<br>
> Quota Daemon on 10.0.231.54 N/A N/A Y<br>
> 9406<br>
> NFS Server on 10.0.231.53 2049 0 Y<br>
> 18387<br>
> Quota Daemon on 10.0.231.53 N/A N/A Y<br>
> 18397<br>
> NFS Server on 10.0.231.52 2049 0 Y<br>
> 2230<br>
> Quota Daemon on 10.0.231.52 N/A N/A Y<br>
> 2262<br>
> NFS Server on 10.0.231.50 2049 0 Y<br>
> 2113<br>
> Quota Daemon on 10.0.231.50 N/A N/A Y<br>
> 2154<br>
><br>
> Task Status of Volume storage<br>
> ------------------------------<wbr>------------------------------<wbr>------------------<br>
> There are no active volume tasks<br>
><br>
> [root@gluster07 ~]# gluster volume quota storage list | egrep "MEOPAR "<br>
> /data/projects/MEOPAR 8.5TB 80%(6.8TB) 16384.0PB<br>
> 17.4TB No No<br>
><br>
><br>
><br>
><br>
> Looking at the quota daemon on gluster07:<br>
><br>
> [root@gluster07 ~]# ps -f -p 29438<br>
> UID PID PPID C STIME TTY TIME CMD<br>
> root 29438 1 0 Jun19 ? 04:43:31 /usr/sbin/glusterfs -s<br>
> localhost --volfile-id gluster/quotad -p<br>
> /var/lib/glusterd/quotad/run/q<wbr>uotad.pid -l /var/log/glusterfs/quotad.log<br>
><br>
> I can see some errors on the log - not sure if those are related:<br>
><br>
> [root@gluster07 ~]# tail /var/log/glusterfs/quotad.log<br>
> [2017-08-28 15:36:17.990909] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:36:17.991389] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:36:17.992656] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:36:17.993235] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:45:51.024756] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:45:51.027871] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:45:51.030843] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:45:51.031324] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:45:51.032791] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
> [2017-08-28 15:45:51.033295] W [dict.c:592:dict_unref]<br>
> (-->/usr/lib64/glusterfs/3.7.1<wbr>3/xlator/features/quotad.so(qd<wbr>_lookup_cbk+0x35e)<br>
> [0x7f79fb09253e]<br>
> -->/usr/lib64/glusterfs/3.7.13<wbr>/xlator/features/quotad.so(quo<wbr>tad_aggregator_getlimit_cbk+0x<wbr>b3)<br>
> [0x7f79fb093333] -->/lib64/libglusterfs.so.0(di<wbr>ct_unref+0x99)<br>
> [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument]<br>
><br>
> How should I proceed?<br>
><br>
> Thanks,<br>
> -Matthew<br>
><br>
> On Mon, Aug 28, 2017 at 3:13 AM, Sanoj Unnikrishnan <<a href="mailto:sunnikri@redhat.com" target="_blank">sunnikri@redhat.com</a>><br>
> wrote:<br>
><br>
> > Hi Mathew,<br>
> ><br>
> > If you are sure that "/mnt/raid6-storage/storage/da<wbr>ta/projects/MEOPAR/"<br>
> > is the only directory with wrong accounting and its immediate sub<br>
> > directories have correct xattr values, Setting the dirty xattr and doing a<br>
> > stat after that should resolve the issue.<br>
> ><br>
> > 1) setxattr -n trusted.glusterfs.quota.dirty -v 0x3100<br>
> > /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR/<br>
> ><br>
> > 2) stat /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR/<br>
> ><br>
> > Could you please share what kind of operations that happens on this<br>
> > directory, to help RCA the issue.<br>
> ><br>
> > If you think this can be true elsewhere in filesystem as well,use the<br>
> > following script to identify the same.<br>
> ><br>
> > 1) <a href="https://github.com/gluster/glusterfs/blob/master/extras/" rel="noreferrer" target="_blank">https://github.com/gluster/glu<wbr>sterfs/blob/master/extras/</a><br>
> > quota/xattr_analysis.py<br>
> > 2) <a href="https://github.com/gluster/glusterfs/blob/master/extras/" rel="noreferrer" target="_blank">https://github.com/gluster/glu<wbr>sterfs/blob/master/extras/</a><br>
> > quota/log_accounting.sh<br>
> ><br>
> > Regards,<br>
> > Sanoj<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Mon, Aug 28, 2017 at 12:39 PM, Raghavendra Gowdappa <<br>
> > <a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<br>
> ><br>
> >> +sanoj<br>
> >><br>
> >> ----- Original Message -----<br>
> >> > From: "Matthew B" <<a href="mailto:matthew.has.questions@gmail.com" target="_blank">matthew.has.questions@gmail.c<wbr>om</a>><br>
> >> > To: <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
> >> > Sent: Saturday, August 26, 2017 12:45:19 AM<br>
> >> > Subject: [Gluster-devel] Quota Used Value Incorrect - Fix now or after<br>
> >> upgrade<br>
> >> ><br>
> >> > Hello,<br>
> >> ><br>
> >> > I need some advice on fixing an issue with quota on my gluster volume.<br>
> >> It's<br>
> >> > running version 3.7, distributed volume, with 7 nodes.<br>
> >> ><br>
> >> > # gluster --version<br>
> >> > glusterfs 3.7.13 built on Jul 8 2016 15:26:18<br>
> >> > Repository revision: git:// <a href="http://git.gluster.com/glusterfs.git" rel="noreferrer" target="_blank">git.gluster.com/glusterfs.git</a><br>
> >> > Copyright (c) 2006-2011 Gluster Inc. < <a href="http://www.gluster.com" rel="noreferrer" target="_blank">http://www.gluster.com</a> ><br>
> >> > GlusterFS comes with ABSOLUTELY NO WARRANTY.<br>
> >> > You may redistribute copies of GlusterFS under the terms of the GNU<br>
> >> General<br>
> >> > Public License.<br>
> >> ><br>
> >> > # gluster volume info storage<br>
> >> ><br>
> >> > Volume Name: storage<br>
> >> > Type: Distribute<br>
> >> > Volume ID: 6f95525a-94d7-4174-bac4-e1a18f<wbr>e010a2<br>
> >> > Status: Started<br>
> >> > Number of Bricks: 7<br>
> >> > Transport-type: tcp<br>
> >> > Bricks:<br>
> >> > Brick1: 10.0.231.50:/mnt/raid6-storage<wbr>/storage<br>
> >> > Brick2: 10.0.231.51:/mnt/raid6-storage<wbr>/storage<br>
> >> > Brick3: 10.0.231.52:/mnt/raid6-storage<wbr>/storage<br>
> >> > Brick4: 10.0.231.53:/mnt/raid6-storage<wbr>/storage<br>
> >> > Brick5: 10.0.231.54:/mnt/raid6-storage<wbr>/storage<br>
> >> > Brick6: 10.0.231.55:/mnt/raid6-storage<wbr>/storage<br>
> >> > Brick7: 10.0.231.56:/mnt/raid6-storage<wbr>/storage<br>
> >> > Options Reconfigured:<br>
> >> > changelog.changelog: on<br>
> >> > geo-replication.ignore-pid-che<wbr>ck: on<br>
> >> > geo-replication.indexing: on<br>
> >> > nfs.disable: no<br>
> >> > performance.readdir-ahead: on<br>
> >> > features.quota: on<br>
> >> > features.inode-quota: on<br>
> >> > features.quota-deem-statfs: on<br>
> >> > features.read-only: off<br>
> >> ><br>
> >> > # df -h /storage/<br>
> >> > Filesystem Size Used Avail Use% Mounted on<br>
> >> > 10.0.231.50:/storage 255T 172T 83T 68% /storage<br>
> >> ><br>
> >> ><br>
> >> > I am planning to upgrade to 3.10 (or 3.12 when it's available) but I<br>
> >> have a<br>
> >> > number of quotas configured, and one of them (below) has a very wrong<br>
> >> "Used"<br>
> >> > value:<br>
> >> ><br>
> >> > # gluster volume quota storage list | egrep "MEOPAR "<br>
> >> > /data/projects/MEOPAR 8.5TB 80%(6.8TB) 16384.0PB 17.4TB No No<br>
> >> ><br>
> >> ><br>
> >> > I have confirmed the bad value appears in one of the bricks current<br>
> >> xattrs,<br>
> >> > and it looks like the issue has been encountered previously on bricks<br>
> >> 04,<br>
> >> > 03, and 06: (gluster07 does not have a trusted.glusterfs.quota.size.1<br>
> >> as it<br>
> >> > was recently added)<br>
> >> ><br>
> >> > $ ansible -i hosts gluster-servers[0:6] -u <user> --ask-pass -m shell -b<br>
> >> > --become-method=sudo --ask-become-pass -a "getfattr --absolute-names -m<br>
> >> . -d<br>
> >> > -e hex /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR | egrep<br>
> >> > '^trusted.glusterfs.quota.size<wbr>'"<br>
> >> > SSH password:<br>
> >> > SUDO password[defaults to SSH password]:<br>
> >> ><br>
> >> > gluster02 | SUCCESS | rc=0 >><br>
> >> > trusted.glusterfs.quota.size=0<wbr>x0000011ecfa56c00000000000005c<br>
> >> d6d000000000006d478<br>
> >> > trusted.glusterfs.quota.size.1<wbr>=0x0000010ad4a4520000000000000<br>
> >> 12a0300000000000150fa<br>
> >> ><br>
> >> > gluster05 | SUCCESS | rc=0 >><br>
> >> > trusted.glusterfs.quota.size=0<wbr>x00000033b8e92200000000000004c<br>
> >> de8000000000006b1a4<br>
> >> > trusted.glusterfs.quota.size.1<wbr>=0x0000010dca277c0000000000000<br>
> >> 1297d0000000000015005<br>
> >> ><br>
> >> > gluster01 | SUCCESS | rc=0 >><br>
> >> > trusted.glusterfs.quota.size=0<wbr>x0000003d4d4348000000000000057<br>
> >> 616000000000006afd2<br>
> >> > trusted.glusterfs.quota.size.1<wbr>=0x00000133fe211e0000000000000<br>
> >> 5d161000000000006cfd4<br>
> >> ><br>
> >> > gluster04 | SUCCESS | rc=0 >><br>
> >> > trusted.glusterfs.quota.size=0<wbr>xffffff396f3e9400000000000004d<br>
> >> 7ea0000000000068c62<br>
> >> > trusted.glusterfs.quota.size.1<wbr>=0x00000106e672480000000000000<br>
> >> 1138f0000000000012fb2<br>
> >> ><br>
> >> > gluster03 | SUCCESS | rc=0 >><br>
> >> > trusted.glusterfs.quota.size=0<wbr>xfffffd02acabf0000000000000035<br>
> >> 99000000000000643e2<br>
> >> > trusted.glusterfs.quota.size.1<wbr>=0x00000114e20f5e0000000000000<br>
> >> 113b30000000000012fb2<br>
> >> ><br>
> >> > gluster06 | SUCCESS | rc=0 >><br>
> >> > trusted.glusterfs.quota.size=0<wbr>xffffff0c98de44000000000000053<br>
> >> 6e40000000000068cf2<br>
> >> > trusted.glusterfs.quota.size.1<wbr>=0x0000013532664e0000000000000<br>
> >> 5e73f000000000006cfd4<br>
> >> ><br>
> >> > gluster07 | SUCCESS | rc=0 >><br>
> >> > trusted.glusterfs.quota.size=0<wbr>xfffffa3d7c1ba60000000000000a9<br>
> >> ccb000000000005fd2f<br>
> >> ><br>
> >> > And reviewing the subdirectories of that folder on the impacted server<br>
> >> you<br>
> >> > can see that none of the direct children have such incorrect values:<br>
> >> ><br>
> >> > [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex<br>
> >> > /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR/*<br>
> >> > # file: /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR/<dir1 ><br>
> >> > ...<br>
> >> > trusted.glusterfs.quota.7209b6<wbr>77-f4b9-4d82-a382-0733620e6929<br>
> >> .contri=0x000000fb684182000000<wbr>0000000074730000000000000dae<br>
> >> > trusted.glusterfs.quota.dirty=<wbr>0x3000<br>
> >> > trusted.glusterfs.quota.size=0<wbr>x000000fb684182000000000000007<br>
> >> 4730000000000000dae<br>
> >> ><br>
> >> > # file: /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR/<dir2 ><br>
> >> > ...<br>
> >> > trusted.glusterfs.quota.7209b6<wbr>77-f4b9-4d82-a382-0733620e6929<br>
> >> .contri=0x0000000416d5f4000000<wbr>000000000baa0000000000000441<br>
> >> > trusted.glusterfs.quota.dirty=<wbr>0x3000<br>
> >> > trusted.glusterfs.quota.limit-<wbr>set=0x0000010000000000ffffffff<wbr>ffffffff<br>
> >> > trusted.glusterfs.quota.size=0<wbr>x0000000416d5f4000000000000000<br>
> >> baa0000000000000441<br>
> >> ><br>
> >> > # file: /mnt/raid6-storage/storage/dat<wbr>a/projects/MEOPAR/<dir3><br>
> >> > ...<br>
> >> > trusted.glusterfs.quota.7209b6<wbr>77-f4b9-4d82-a382-0733620e6929<br>
> >> .contri=0x000000110f2c4e000000<wbr>00000002a76a000000000006ad7d<br>
> >> > trusted.glusterfs.quota.dirty=<wbr>0x3000<br>
> >> > trusted.glusterfs.quota.limit-<wbr>set=0x0000020000000000ffffffff<wbr>ffffffff<br>
> >> > trusted.glusterfs.quota.size=0<wbr>x000000110f2c4e00000000000002a<br>
> >> 76a000000000006ad7d<br>
> >> ><br>
> >> ><br>
> >> > Can I fix this on the current version of gluster (3.7) on just the one<br>
> >> brick<br>
> >> > before I upgrade? Or would I be better off upgrading to 3.10 and trying<br>
> >> to<br>
> >> > fix it then?<br>
> >> ><br>
> >> > I have reviewed information here:<br>
> >> ><br>
> >> > <a href="http://lists.gluster.org/pipermail/gluster-devel/2016-Februa" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-devel/2016-Februa</a><br>
> >> ry/048282.html<br>
> >> > <a href="http://lists.gluster.org/pipermail/gluster-users.old/2016-" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users.old/2016-</a><br>
> >> September/028365.html<br>
> >> ><br>
> >> > It seems like since I am on Gluster 3.7 I can disable quotas and<br>
> >> re-enable<br>
> >> > and everything will get recalculated and increment the index on the<br>
> >> > quota.size xattr. But with the size of the volume that will take a very<br>
> >> long<br>
> >> > time.<br>
> >> ><br>
> >> > Could I simply mark the impacted directly as dirty on gluster07? Or<br>
> >> update<br>
> >> > the xattr directly as the sum of the size of dir1, 2, and 3?<br>
> >> ><br>
> >> > Thanks,<br>
> >> > -Matthew<br>
> >> ><br>
> >> > ______________________________<wbr>_________________<br>
> >> > Gluster-devel mailing list<br>
> >> > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> >> > <a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
> >><br>
> ><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>