[Bugs] [Bug 1200457] gluster performance issue as data is added to volume. tar extraction of files goes from 1-minute on empty volume to 20-minutes on volume with 40TB.

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1200457

Pranith Kumar K <pkarampu at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bfoster at redhat.com



--- Comment #22 from Pranith Kumar K <pkarampu at redhat.com> ---
Some of the findings based on the tests I did based on Brian's inputs:
We removed all the xlators between fuse and posix to make the volfile similar
to what we have here:
volume test-posix-0
    type storage/posix
    option directory /home/brick/test20
end-volume

Mounted this volfile using:
glusterfs --volfile=/root/volfile.vol /d

Small file CLI we used is: 
for i in {1..80}; do mkdir /mnt/smf/$i ; python smallfile_cli.py --top
/mnt/smf/$i --threads 4 --files 8192 --file-size 64 --record-size 64 --fsync Y
--response-times N --operation create >> /root/smf.out; done

1) In this test I ran the small file workload using the following CLI 25 times.
We observed that the elapsed time kept increasing.
These are the times:
[root at gprfs039 ~]# grep 'sec elapsed' just-posix-smf.out 
45.026515 sec elapsed time
45.630780 sec elapsed time
47.106841 sec elapsed time
48.994986 sec elapsed time
49.056766 sec elapsed time
47.511098 sec elapsed time
57.042012 sec elapsed time
57.128427 sec elapsed time
63.952732 sec elapsed time
77.634293 sec elapsed time
69.540255 sec elapsed time
84.495499 sec elapsed time
83.755443 sec elapsed time
76.225897 sec elapsed time
86.382487 sec elapsed time
113.626848 sec elapsed time
109.282297 sec elapsed time
95.230611 sec elapsed time
110.965802 sec elapsed time
90.876091 sec elapsed time
98.172049 sec elapsed time
106.055803 sec elapsed time
133.958628 sec elapsed time
104.454683 sec elapsed time
119.641330 sec elapsed time

2) In this test we did a remount and ran the test where we stopped from. The
first run took 97 seconds which was in the same range as before.

3) In this test we did a drop-caches. What we observed is that the elapsed time
started at around 440 seconds and came down to 95 seconds over time but hovered
again at around 95-120 seconds
442.857238 sec elapsed time
314.401744 sec elapsed time
260.829069 sec elapsed time
219.472399 sec elapsed time
201.450154 sec elapsed time
178.373704 sec elapsed time
158.367833 sec elapsed time
151.658923 sec elapsed time
144.989891 sec elapsed time
126.476212 sec elapsed time
133.021995 sec elapsed time
106.850061 sec elapsed time
105.401238 sec elapsed time
118.126849 sec elapsed time
115.904676 sec elapsed time
94.786972 sec elapsed time
119.076134 sec elapsed time

4) In this test we kept creating new bricks on the same XFS partitions by
generating new volfiles and running small file create using the following CLI:
for i in {2..26}; do echo "volume test-posix-0" > /root/volfile.vol ; echo "   
type storage/posix" >> /root/volfile.vol; echo "    option directory
/home/brick/test$i"  >>  /root/volfile.vol; echo "end-volume" >> 
/root/volfile.vol; mkdir /home/brick/test$i; glusterfs
--volfile=/root/volfile.vol /d; mkdir -p /d/smf/$i ; python smallfile_cli.py
--top /d/smf/$i --threads 4 --files 8192 --file-size 64 --record-size 64
--fsync Y --response-times N --operation create >> /root/separate-bricks.out;
umount /d; done

Run time didn't degrade in this case!!
Here are the times:
[root at gprfs039 ~]# grep 'sec elapsed' /root/separate-bricks.out 
48.763393 sec elapsed time
48.581379 sec elapsed time
48.777564 sec elapsed time
49.469651 sec elapsed time
48.915736 sec elapsed time
50.647200 sec elapsed time
48.117454 sec elapsed time
49.835503 sec elapsed time
48.549795 sec elapsed time
50.297137 sec elapsed time
51.053001 sec elapsed time
49.238791 sec elapsed time
49.562103 sec elapsed time
46.240161 sec elapsed time
50.360794 sec elapsed time
48.729912 sec elapsed time
51.763603 sec elapsed time
49.370971 sec elapsed time
48.930266 sec elapsed time
48.070540 sec elapsed time
48.472625 sec elapsed time
48.350298 sec elapsed time
47.316538 sec elapsed time
46.672402 sec elapsed time
47.304169 sec elapsed time

Will update with more info once we find more about the reason why this is the
case.

Pranith

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.