<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Gluster community:</span></div><div><br data-mce-bogus="1"></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Due to some recent issues with performance from one of our (internal) clients that uses several GlusterFS setups, this is mainly an open question for generic possible improvements.</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">(apart from the fact the users themselves do some 'less smart' things on it....)</span></div><div><br data-mce-bogus="1"></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">The setup used: a 3 node (replica=3) RHEL7 gluster (with RHGS, so currently that is GlusterFS 6.0-37.1.el7rhgs). Each Gluster has 1 volume, exported through NFS-Ganesha.</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Each node also has a virtual IP, managed by pacemaker/corosync following standard RedHat setup documentation</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Size is not that big, 600GB space with around half of that actually used. GlusterFS servers themselves each have 4 cores and 12GB memory. It might also be important to note that these are VMware hosted nodes that make use of SAN storage for the datastores.</span></div><div><br data-mce-bogus="1"></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Connected to that NFS (ganesha) exported share are just over 100 clients, all RHEL6 and RHEL7, some spanning 10 network hops away. All of those clients are (currently) using the same virtual-IP, so all end up on the same server.</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">(we did already advise them to spread that across the three servers).</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Certain subfolders of the share hold (at times) large numbers of (small) files that *should* peak at around 50.000 files, into a single, unhashed, directory (this will of course make simple ls and find commands via NFS quite slow).</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Note that I mentioned 'should', since at times it had anywhere between 250.000 and 1 million files in it (which of course is not advised). Using some kind of hashing (subfolders spread per day/hour etc) was also already advised.</span></div><div><br data-mce-bogus="1"></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Problems that are often seen:</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">- Any kind of operation on VMware such as a vMotion, creating a VM snapshot etc. on the node that has these 100+ clients connected causes such a temporary pause that pacemaker decides to switch the resources (causing a failover of the virtual IP address, thus clients connected suffer delay). One would expect this to last just shy under a minute, then clients would happily continue. However connected clients are stuck with a non-working mountpoint (commands as df, ls, find etc simply hang.. they go into an uninterruptible sleep).</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Mount are 'hard' mounts to insure guaranteed writes.</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">- Once the number of files are over the 100.000 mark (again into a single, unhashed, folder) any operation on that share becomes very sluggish (even a df, on a client, would take 20/30 seconds, a find command would take minutes to complete).</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"><br data-mce-bogus="1"></span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">If anyone can spot any ideas for improvement ?</span></div><div><br data-mce-bogus="1"></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">Some config info (below is from a sandbox setup using the same values as the affected gluster):</span></div><div><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">For Ganesha:</span></div><div><pre><span style="font-family: "lucida console", sans-serif; font-size: 10pt;" data-mce-style="font-family: 'lucida console', sans-serif; font-size: 10pt;">/etc/ganesha/ganesha.conf:</span><br><span style="font-family: "lucida console", sans-serif; font-size: 10pt;" data-mce-style="font-family: 'lucida console', sans-serif; font-size: 10pt;"># BEGIN ANSIBLE MANAGED BLOCK</span><br><span style="font-family: "lucida console", sans-serif; font-size: 10pt;" data-mce-style="font-family: 'lucida console', sans-serif; font-size: 10pt;">NFSv4 {</span><br><span style="font-family: "lucida console", sans-serif; font-size: 10pt;" data-mce-style="font-family: 'lucida console', sans-serif; font-size: 10pt;">minor_versions = 0;</span><br><span style="font-family: "lucida console", sans-serif; font-size: 10pt;" data-mce-style="font-family: 'lucida console', sans-serif; font-size: 10pt;">}</span><br><span style="font-family: "lucida console", sans-serif; font-size: 10pt;" data-mce-style="font-family: 'lucida console', sans-serif; font-size: 10pt;"># END ANSIBLE MANAGED BLOCK</span><br><span style="font-family: "lucida console", sans-serif; font-size: 10pt;" data-mce-style="font-family: 'lucida console', sans-serif; font-size: 10pt;">%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.BLAH.conf"</span></pre><div><br></div><div><span style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 10pt; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;">/var/run/gluster/shared_storage/nfs-ganesha/exports/export.BLAH.conf:</span></div><div><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;">EXPORT{</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> Export_Id = 2;</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> Path = "/BLAH";</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> FSAL {</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> name = GLUSTER;</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> hostname="localhost";</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> volume="BLAH";</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> }</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> Access_type = RW;</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> Disable_ACL = true;</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> Squash="No_root_squash";</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> Pseudo="/BLAH";</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> Protocols = "3", "4" ;</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> Transports = "UDP","TCP";</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> SecType = "sys";</span></div><div style=""><span style="font-size: 10pt;" data-mce-style="font-size: 10pt;"> }</span></div><div style=""><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important; font-size: 10pt;" data-mce-style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important; font-size: 10pt;"><br data-mce-bogus="1"></span></div><div style=""><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"># gluster v info BLAH</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Volume Name: BLAH</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Type: Replicate</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Volume ID: 6fee713c-4258-44d8-a849-f8d6b2991631</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Status: Started</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Snapshot Count: 0</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Number of Bricks: 1 x 3 = 3</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Transport-type: tcp</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Bricks:</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Brick1: tlrvrhgluster03:/gluster/BLAH/export</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Brick2: tlrvrhgluster02:/gluster/BLAH/export</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Brick3: tlrvrhgluster01:/gluster/BLAH/export</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">Options Reconfigured:</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">diagnostics.count-fop-hits: on</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">diagnostics.latency-measurement: on</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">ganesha.enable: on</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">features.cache-invalidation: on</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">performance.client-io-threads: off</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">nfs.disable: on</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">storage.fips-mode-rchecksum: on</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">transport.address-family: inet</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">cluster.server-quorum-type: server</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">cluster.quorum-count: 2</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">performance.cache-refresh-timeout: 10</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">cluster.quorum-type: fixed</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">cluster.enable-shared-storage: enable</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;">nfs-ganesha: enable</span></div><div><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important; font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important; font-size: 10pt; font-family: 'lucida console', sans-serif;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;"><br data-mce-bogus="1"></span></span></span></div><div><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"># lvs -a</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> BLAH gfsvg Vwi-aotz-- 10.00g gfspool 36.45</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> gfspool gfsvg twi-aotz-- 48.00g 7.59 1.76</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> [gfspool_tdata] gfsvg Twi-ao---- 48.00g</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> [gfspool_tmeta] gfsvg ewi-ao---- 1.00g</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> [lvol0_pmspare] gfsvg ewi------- 48.00m</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> auditlv systemvg -wi-ao---- 252.00m</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> homelv systemvg -wi-ao---- 1.00g</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> rootlv systemvg -wi-ao---- 16.00g</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> swaplv systemvg -wi-ao---- 2.00g</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> tmplv systemvg -wi-ao---- 2.00g</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> varcorelv systemvg -wi-ao---- 1.00g</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> varloglv systemvg -wi-ao---- 6.00g</span></div><div><span style="font-size: 10pt; font-family: "lucida console", sans-serif;" data-mce-style="font-size: 10pt; font-family: 'lucida console', sans-serif;"> varlv systemvg -wi-ao---- 6.00g</span></div><div><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></span></span></span></span></span></span></span></span></div></span></span></span></span></span></span></div></span></span></div></span></div></div></div></body></html>