[Gluster-users] NFS clients show missing files while gluster volume rebalanced

[Erik Jacobson]

> Can you provide the volume info.
> Also check the logs for the rebalance. There should be some valuable info on what is going on.


Hello thank you for replying.

The re-balance log is between 20M and 40M on all 9 nodes.

Even xz-compressing one gets us to 1.8M which I'm not sure I can attach
here. However, looking at one, I saw most lines were the same. So
I tried removing the repeating normal lines like this:

# cat cm_shared-rebalance.log | egrep -v "completed migration of|migrate data called on|attempting to move" > leader1-cm_shared-rebalance.log-filtered
# xz -9 leader1-cm_shared-rebalance.log-filtered

So I have attached the above and the volume info.

I'm not sure what I'm looking for exactly and I know the above is just
one of the 9 leaders. It's almost like the gluster NFS server was losing
track of the moves in terms of serving the clients.

Thanks again !!! Attached some details! Happy to attach more! Finally a
problem I can reproduce at will in the lab :) :)

PS: I can make stuff available on an ftp server too.


On Thu, Jan 30, 2020 at 05:50:30PM +0200, Strahil Nikolov wrote:
> On January 30, 2020 2:20:34 AM GMT+02:00, Erik Jacobson <erik.jacobson at hpe.com> wrote:
> >We are using gluster 4.1.6. We are using gluster NFS (not ganesha).
> >
> >Distributed/replicated with subvolume size 3 (6 total servers, 2
> >subvols).
> >
> >The NFS clients use this for their root filesystem.
> >
> >When I add 3 more gluster servers to add one more subvolume to the
> >storage volumes (so now subvolume size 3, 9 total servers, 3 total
> >subvolumes), the process gets started. 
> >
> >ssh leader1 gluster volume add-brick cm_shared
> >172.23.0.9://data/brick_cm_shared 172.23.0.10://data/brick_cm_shared
> >172.23.0.11://data/brick_cm_shared
> >
> >then
> >
> >ssh leader1 gluster volume rebalance cm_shared start
> >
> >The re-balance works. 'gluster volume status' shows re-balance in
> >progress.
> >
> >However, existing gluster-NFS clients now show missing files and I can
> >no longer log into them (since NFS is their root). If you are logged
> >in,
> >you can find that libraries are missing and general unhappiness with
> >random files now missing.
> >
> >Is accessing a volume that is in the process of being re-balanced not
> >supported from a gluster NFS client? Or have I made an error?
> >
> >Thank you for any help,
> >
> >Erik
> >________
> >
> >Community Meeting Calendar:
> >
> >APAC Schedule -
> >Every 2nd and 4th Tuesday at 11:30 AM IST
> >Bridge: https://bluejeans.com/441850968 
> >
> >NA/EMEA Schedule -
> >Every 1st and 3rd Tuesday at 01:00 PM EDT
> >Bridge: https://bluejeans.com/441850968 
> >
> >Gluster-users mailing list
> >Gluster-users at gluster.org
> >https://lists.gluster.org/mailman/listinfo/gluster-users 
> 
> Hi Erik,
> 
> Can you provide the volume info.
> Also check the logs for the rebalance. There should be some valuable info on what is going on.
> 
> Best Regards,
> Strahil Nikolov
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gluster-volinfo.txt.xz
Type: application/x-xz
Size: 748 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200130/3fcf851f/attachment-0003.xz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: leader1-cm_shared-rebalance.log-filtered.xz
Type: application/x-xz
Size: 998824 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200130/3fcf851f/attachment-0004.xz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: volume-get-all.txt.xz
Type: application/x-xz
Size: 3808 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200130/3fcf851f/attachment-0005.xz>