[Gluster-users] Sudden, dramatic performance drops with Glusterfs

[DUCARROZ Birgit]

Hi,

I had complete crash with gluster and ZFS. Installed ext4 instead and it 
worked much more better.

Regards,
Birgit

On 06/11/19 11:50, Michael Rightmire wrote:
> Hello list!
> 
> I'm new to Glusterfs in general. We have chosen to use it as our 
> distributed file system on a new set of HA file servers.
> 
> The setup is:
> 2 SUPERMICRO SuperStorage Server 6049PE1CR36L with 24-4TB spinning disks 
> and NVMe for cache and slog.
> HBA not RAID card
> Ubuntu 18.04 server (on both systems)
> ZFS filestorage
> Glusterfs 5.10
> 
> Step one was to install Ubuntu, ZFS, and gluster. This all went without 
> issue.
> We have 3 ZFS raidz2 identical on both servers
> We have three glusterfs mirrored volumes - 1 attached to each raidz on 
> each server. I.e.
> 
> And mounted the gluster volumes as (for example) "/glusterfs/homes -> 
> /zpool/homes". I.e.
> gluster volume create homes replica 2 transport tcp 
> server1:/zpool-homes/homes server2:/zpool-homes/homes force
> (on server1) server1:/homes     44729413504 16032705152 28696708352  36% 
> /glusterfs/homes
> 
> The problem is, the performance has deteriorated terribly.
> We needed to copy all of our data from the old server to the new 
> glusterfs volumes (appx. 60TB).
> We decided to do this with multiple rsync commands (like 400 simultanous 
> rsyncs)
> The copy went well for the first 4 days, with an average across all 
> rsyncs of 150-200 MBytes per second.
> Then, suddenly, on the fourth day, it dropped to about 50 MBytes/s.
> Then, by the end of the day, down to ~5MBytes/s (five).
> I've stopped the rsyncs, and Ican still copy an individual file across 
> to the glusterfs shared directory at 100MB/s.
> But actions such as "ls -la" or "find" take forever!
> 
> Are there obvious flaws in my setup to correct?
> How can I better troubleshoot this?
> 
> Thanks!
> -- 
> 
> Mike
> 
> 
> ________
> 
> Community Meeting Calendar:
> 
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/118564314
> 
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/118564314
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>