[Bugs] [Bug 1540759] New: Failure to demote tiered volume file that is continuously modified by client during hot tier detachment.
bugzilla at redhat.com
bugzilla at redhat.com
Wed Jan 31 23:49:30 UTC 2018
https://bugzilla.redhat.com/show_bug.cgi?id=1540759
Bug ID: 1540759
Summary: Failure to demote tiered volume file that is
continuously modified by client during hot tier
detachment.
Product: GlusterFS
Version: 3.12
Component: tiering
Severity: urgent
Assignee: bugs at gluster.org
Reporter: vnosov at stonefly.com
QA Contact: bugs at gluster.org
CC: bugs at gluster.org
Description of problem:
Enable CIFS export for tiered volume. Map the volume to drive on Windows. Run
IOMeter with writes and reads to the volume continuously during the test. Check
that file created by the IOMeter is on hot tier.
Start "detach" hot tier for the volume. Verify that status of the detachment is
"completed". Issue "commit" the detachment. Check file used for the IOMeter
testing. The file still has bit "T" at the permissions.
Version-Release number of selected component (if applicable):
glusterfs 3.12.3
How reproducible: 100%
Steps to Reproduce:
System: Centos 6.
SC182, glusterFS 3.12.3
Test volume: tiered "nas-volume-0003", with enabled CIFS export. Size of cold
tier is 1GB, size of hot tier is 1GB.
Client: Windows host.
Application: IOmeter, 64KB sequential, 50% reads and 50% writes. IOmeter
creates file with name "iobw.tst". Size of the file is 512000000 bytes.
[root at SC-182 log]# gluster volume info nas-volume-0003
Volume Name: nas-volume-0003
Type: Tier
Volume ID: 6ea89454-985f-4ad6-925d-4f82560961e9
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 1
Brick2: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003
Options Reconfigured:
cluster.tier-mode: cache
features.ctr-enabled: on
nfs.disable: on
transport.address-family: inet
nfs.addr-namelookup: off
performance.stat-prefetch: off
performance.quick-read: off
server.allow-insecure: on
[root at SC-182 log]# gluster volume status nas-volume-0003
Status of volume: nas-volume-0003
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003
62000 0 Y 10130
Cold Bricks:
Brick 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003
62001 0 Y 23421
Tier Daemon on localhost N/A N/A Y 10151
Task Status of Volume nas-volume-0003
------------------------------------------------------------------------------
There are no active volume tasks
Start detach:
13145 01/30/2018 14:33:18.715250327 1517351598 command:
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003
detach start 2>&1
13159 01/30/2018 14:33:18.780335608 1517351598 command: /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "start"
13159 01/30/2018 14:33:24.062642838 1517351604 status=0 /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "start"
13145 01/30/2018 14:33:24.078741347 1517351604 status=0
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003
detach start 2>&1
[root at SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/
total 500000
drwxr-xr-x 2 root root 6 Jan 29 16:29 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 14:37 iobw.tst
[root at SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/
total 904
drwxr-xr-x 2 root root 8192 Jan 29 16:29 dir1
---------T 2 root nas_users 453705728 Jan 30 14:34 iobw.tst
[root at SC-182 log]# /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach"
"status"
volume detach tier status: success
Node Rebalanced-files size
scanned failures skipped status run time in h:m:s
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 1 488.3MB
50 0 0 completed 0:00:08
Hot segment:
[root at SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/
total 500000
drwxr-xr-x 2 root root 6 Jan 29 16:29 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 14:41 iobw.tst
Cold segment:
[root at SC-182 log]#
[root at SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/
total 904
drwxr-xr-x 2 root root 8192 Jan 29 16:29 dir1
---------T 2 root nas_users 453705728 Jan 30 14:34 iobw.tst
[root at SC-182 log]# gluster volume profile nas-volume-0003 start
Starting volume profile on nas-volume-0003 has been successful
[root at SC-182 log]# gluster volume profile nas-volume-0003 info clear
Brick: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003
-------------------------------------------------------------
Cleared stats.
Brick: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003
-------------------------------------------------------------
Cleared stats.
[root at SC-182 log]# gluster volume profile nas-volume-0003 info
Brick: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003
=============> Hot segment
-------------------------------------------------------------
Cumulative Stats:
Block Size: 65536b+
No. of Reads: 0
No. of Writes: 365
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13 RELEASEDIR
0.71 70.43 us 32.00 us 140.00 us 14 OPENDIR
1.58 57.18 us 23.00 us 138.00 us 38 STAT
4.83 80.22 us 41.00 us 152.00 us 83 GETXATTR
5.53 61.54 us 28.00 us 152.00 us 124 STATFS
5.79 70.73 us 43.00 us 151.00 us 113 LOOKUP
81.56 308.24 us 227.00 us 479.00 us 365 WRITE
Duration: 29 seconds
Data Read: 0 bytes
Data Written: 23920640 bytes
Interval 0 Stats:
Block Size: 65536b+
No. of Reads: 0
No. of Writes: 365
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13 RELEASEDIR
0.71 70.43 us 32.00 us 140.00 us 14 OPENDIR
1.58 57.18 us 23.00 us 138.00 us 38 STAT
4.83 80.22 us 41.00 us 152.00 us 83 GETXATTR
5.53 61.52 us 28.00 us 152.00 us 124 STATFS
5.79 70.73 us 43.00 us 151.00 us 113 LOOKUP
81.56 308.24 us 227.00 us 479.00 us 365 WRITE
Duration: 29 seconds
Data Read: 0 bytes
Data Written: 23920640 bytes
Brick: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003
===================> Cold segment
-------------------------------------------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13 RELEASEDIR
1.65 50.62 us 34.00 us 71.00 us 8 STATFS
4.01 70.50 us 36.00 us 126.00 us 14 OPENDIR
7.43 48.13 us 24.00 us 197.00 us 38 STAT
12.21 214.71 us 161.00 us 255.00 us 14 READDIRP
21.04 77.27 us 40.00 us 210.00 us 67 GETXATTR
53.66 62.89 us 41.00 us 162.00 us 210 LOOKUP
Duration: 29 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Interval 0 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13 RELEASEDIR
1.65 50.62 us 34.00 us 71.00 us 8 STATFS
4.01 70.50 us 36.00 us 126.00 us 14 OPENDIR
7.43 48.13 us 24.00 us 197.00 us 38 STAT
12.21 214.71 us 161.00 us 255.00 us 14 READDIRP
21.04 77.27 us 40.00 us 210.00 us 67 GETXATTR
53.66 62.89 us 41.00 us 162.00 us 210 LOOKUP
Duration: 29 seconds
Data Read: 0 bytes
Data Written: 0 bytes
[root at SC-182 log]# gluster volume profile nas-volume-0003 stop
Stopping volume profile on nas-volume-0003 has been successful
Do commit:
22219 01/30/2018 14:48:51.919431953 1517352531 command:
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003
detach commit 2>&1
22233 01/30/2018 14:48:51.984308571 1517352531 command: /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "commit"
22233 01/30/2018 14:48:53.212079189 1517352533 status=0 /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "commit"
22219 01/30/2018 14:48:53.228757277 1517352533 status=0
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003
detach commit 2>&1
[root at SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/
total 455440
drwxr-xr-x 2 root root 8192 Jan 29 16:29 dir1
---------T 2 root nas_users 511967232 Jan 30 14:49 iobw.tst
[root at SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/
ls: cannot access /exports/nas-segment-0003/nas-volume-0003/cifs_share/: No
such file or directory
[root at SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/
total 499972
drwxr-xr-x 2 root root 4096 Jan 29 16:29 dir1
---------T 1 root nas_users 511967232 Jan 30 14:49 iobw.tst
CIFS client continue to do IO's!
[root at SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/
total 499972
drwxr-xr-x 2 root root 4096 Jan 29 16:29 dir1
---------T 1 root nas_users 511967232 Jan 30 14:51 iobw.tst
Actual results:
Check file "iobw.tst" on the volume:
[root at SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/
total 499972
drwxr-xr-x 2 root root 4096 Jan 29 16:29 dir1
---------T 1 root nas_users 511967232 Jan 30 14:51 iobw.tst
Size and file permissions are wrong.
Expected results:
The test was executed successfully on glusterfs 3.7.18. The correct results
have to be like the next:
[root at SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root 4096 Jan 30 15:11 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:31 iobw.tst
Additional info:
Results of successfull test on glusterFS 3.7.18:
Do the same on SC183 that runs glusterfs 3.7.18.
[root at SC-183 log]# gluster volume info tiered-volume-0015
Volume Name: tiered-volume-0015
Type: Distribute
Volume ID: aefc0a6e-2515-40f1-a888-2f3168e0fa07
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
Options Reconfigured:
server.allow-insecure: on
performance.quick-read: off
performance.stat-prefetch: off
nfs.disable: on
nfs.addr-namelookup: off
performance.readdir-ahead: on
snap-activate-on-create: enable
cluster.enable-shared-storage: disable
[root at SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root 4096 Jan 30 15:06 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:06 iobw.tst
[root at SC-183 log]# ls -l
/exports/cold-segment-0004/tiered-volume-0015/cifs_share/
total 500000
drwxr-xr-x 2 root root 141 Jan 30 15:06 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 15:07 iobw.tst
[root at SC-183 log]# gluster volume status tiered-volume-0015
Status of volume: tiered-volume-0015
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.10.60.183:/exports/cold-segment-00
04/tiered-volume-0015 62078 0 Y 22474
Task Status of Volume tiered-volume-0015
------------------------------------------------------------------------------
There are no active volume tasks
[root at SC-183 log]# gluster volume profile tiered-volume-0015 start
Starting volume profile on tiered-volume-0015 has been successful
[root at SC-183 log]# gluster volume profile tiered-volume-0015 info clear
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cleared stats.
[root at SC-183 log]# gluster volume profile tiered-volume-0015 info
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cumulative Stats:
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 9 0 37835
No. of Writes: 0 20334 9
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR
0.00 44.15 us 31.00 us 88.00 us 20 STATFS
0.01 53.50 us 39.00 us 150.00 us 30 OPENDIR
0.01 40.54 us 23.00 us 171.00 us 50 STAT
0.03 146.25 us 40.00 us 323.00 us 60 READDIRP
0.05 74.78 us 47.00 us 223.00 us 180 GETXATTR
0.08 74.15 us 56.00 us 186.00 us 280 LOOKUP
32.54 422.47 us 47.00 us 827002.00 us 20343 WRITE
67.28 469.44 us 31.00 us 826207.00 us 37851 READ
Duration: 21 seconds
Data Read: 4959404032 bytes
Data Written: 1333788672 bytes
Interval 0 Stats:
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 9 0 37835
No. of Writes: 0 20334 9
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR
0.00 44.15 us 31.00 us 88.00 us 20 STATFS
0.01 53.50 us 39.00 us 150.00 us 30 OPENDIR
0.01 40.54 us 23.00 us 171.00 us 50 STAT
0.03 146.25 us 40.00 us 323.00 us 60 READDIRP
0.05 74.78 us 47.00 us 223.00 us 180 GETXATTR
0.08 74.15 us 56.00 us 186.00 us 280 LOOKUP
32.54 422.47 us 47.00 us 827002.00 us 20343 WRITE
67.28 469.44 us 31.00 us 826207.00 us 37851 READ
Duration: 21 seconds
Data Read: 4959404032 bytes
Data Written: 1333788672 bytes
[root at SC-183 log]# gluster volume profile tiered-volume-0015 stop
Stopping volume profile on tiered-volume-0015 has been successful
Attach hot tier:
4780 01/30/2018 15:11:13.505967116 1517353873 command:
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015
attach 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 2>&1 |
/bin/grep "Tiering Migration Functionality: tiered-volume-0015: success"
4795 01/30/2018 15:11:13.604148675 1517353873 command: /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "attach"
"10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015"
4795 01/30/2018 15:11:19.087525009 1517353879 status=0 /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "attach"
"10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015"
4780 01/30/2018 15:11:19.105676210 1517353879 status=0
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015
attach 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 2>&1 |
/bin/grep "Tiering Migration Functionality: tiered-volume-0015: success"
[root at SC-183 log]# gluster volume info tiered-volume-0015
Volume Name: tiered-volume-0015
Type: Tier
Volume ID: aefc0a6e-2515-40f1-a888-2f3168e0fa07
Status: Started
Number of Bricks: 2
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 1
Brick2: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
Options Reconfigured:
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
cluster.tier-mode: cache
features.ctr-enabled: on
server.allow-insecure: on
performance.quick-read: off
performance.stat-prefetch: off
nfs.disable: on
nfs.addr-namelookup: off
performance.readdir-ahead: on
snap-activate-on-create: enable
cluster.enable-shared-storage: disable
[root at SC-183 log]# gluster volume status tiered-volume-0015
Status of volume: tiered-volume-0015
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.10.60.183:/exports/hot-segment-000
5/tiered-volume-0015 62079 0 Y 4824
Cold Bricks:
Brick 10.10.60.183:/exports/cold-segment-00
04/tiered-volume-0015 62078 0 Y 22474
Quota Daemon on localhost N/A N/A Y 4980
Task Status of Volume tiered-volume-0015
------------------------------------------------------------------------------
Task : Tier migration
ID : cf15d19a-8ba6-4cfd-9794-fc7a77050f45
Status : in progress
[root at SC-183 log]# ls -l
/exports/cold-segment-0004/tiered-volume-0015/cifs_share/
total 448
drwxr-xr-x 2 root root 141 Jan 30 15:11 dir1
---------T 2 root nas_users 236519424 Jan 30 15:12 iobw.tst
[root at SC-183 log]# ls -l
/exports/hot-segment-0005/tiered-volume-0015/cifs_share/
total 500080
drwxr-xr-x 2 root root 6 Jan 30 15:11 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 15:17 iobw.tst
[root at SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root 4096 Jan 30 15:11 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:17 iobw.tst
[root at SC-183 log]# gluster volume profile tiered-volume-0015 start
Starting volume profile on tiered-volume-0015 has been successful
[root at SC-183 log]# gluster volume profile tiered-volume-0015 info clear
Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
----------------------------------------------------------------
Cleared stats.
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cleared stats.
[root at SC-183 log]# gluster volume profile tiered-volume-0015 info
Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
----------------------------------------------------------------
Cumulative Stats:
Block Size: 65536b+ 131072b+
No. of Reads: 0 0
No. of Writes: 20702 364
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR
0.00 44.37 us 36.00 us 68.00 us 30 OPENDIR
0.00 33.45 us 26.00 us 65.00 us 49 STAT
0.03 648.21 us 36.00 us 1160.00 us 24 STATFS
0.04 77.75 us 45.00 us 160.00 us 218 GETXATTR
0.08 117.96 us 83.00 us 386.00 us 317 LOOKUP
99.85 2287.28 us 212.00 us 827377.00 us 21067 WRITE
Duration: 20 seconds
Data Read: 0 bytes
Data Written: 1404436480 bytes
Interval 0 Stats:
Block Size: 65536b+ 131072b+
No. of Reads: 0 0
No. of Writes: 20702 364
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR
0.00 44.37 us 36.00 us 68.00 us 30 OPENDIR
0.00 33.45 us 26.00 us 65.00 us 49 STAT
0.03 648.21 us 36.00 us 1160.00 us 24 STATFS
0.04 77.75 us 45.00 us 160.00 us 218 GETXATTR
0.08 117.96 us 83.00 us 386.00 us 317 LOOKUP
99.85 2287.28 us 212.00 us 827377.00 us 21067 WRITE
Duration: 20 seconds
Data Read: 0 bytes
Data Written: 1404436480 bytes
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR
1.67 61.13 us 41.00 us 97.00 us 30 OPENDIR
1.69 37.76 us 26.00 us 75.00 us 49 STAT
13.00 80.09 us 46.00 us 165.00 us 178 GETXATTR
14.22 779.65 us 619.00 us 1047.00 us 20 STATFS
14.76 269.72 us 67.00 us 815.00 us 60 READDIRP
54.66 108.40 us 81.00 us 351.00 us 553 LOOKUP
Duration: 20 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Interval 0 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR
1.67 61.13 us 41.00 us 97.00 us 30 OPENDIR
1.69 37.76 us 26.00 us 75.00 us 49 STAT
13.00 80.09 us 46.00 us 165.00 us 178 GETXATTR
14.22 779.65 us 619.00 us 1047.00 us 20 STATFS
14.76 269.72 us 67.00 us 815.00 us 60 READDIRP
54.66 108.40 us 81.00 us 351.00 us 553 LOOKUP
Duration: 20 seconds
Data Read: 0 bytes
Data Written: 0 bytes
[root at SC-183 log]# gluster volume profile tiered-volume-0015 stop
Stopping volume profile on tiered-volume-0015 has been successful
Detach hot tier:
15364 01/30/2018 15:24:09.119881869 1517354649 command:
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015
detach start 2>&1
15378 01/30/2018 15:24:09.196084465 1517354649 command: /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "start"
15378 01/30/2018 15:24:14.514959281 1517354654 status=0 /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "start"
15364 01/30/2018 15:24:14.533662479 1517354654 status=0
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015
detach start 2>&1
[root at SC-183 log]# /usr/sbin/gluster "volume" "tier" "tiered-volume-0015"
"detach" "status"
Node Rebalanced-files size
scanned failures skipped status run time in h:m:s
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 1 488.3MB
7 0 0 completed 0:0:19
[root at SC-183 log]# ls -l
/exports/hot-segment-0005/tiered-volume-0015/cifs_share/
total 0
drwxr-xr-x 2 root root 6 Jan 30 15:11 dir1
[root at SC-183 log]# ls -l
/exports/cold-segment-0004/tiered-volume-0015/cifs_share/
total 500064
drwxr-xr-x 2 root root 141 Jan 30 15:11 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 15:26 iobw.tst
[root at SC-183 log]# gluster volume profile tiered-volume-0015 start
Starting volume profile on tiered-volume-0015 has been successful
[root at SC-183 log]# gluster volume profile tiered-volume-0015 info clear
Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
----------------------------------------------------------------
Cleared stats.
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cleared stats.
[root at SC-183 log]# gluster volume profile tiered-volume-0015 info
Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
----------------------------------------------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 24 RELEASEDIR
2.34 57.12 us 38.00 us 84.00 us 24 OPENDIR
3.13 45.75 us 28.00 us 356.00 us 40 STAT
18.46 84.43 us 48.00 us 256.00 us 128 GETXATTR
21.07 770.94 us 556.00 us 1017.00 us 16 STATFS
55.00 118.36 us 84.00 us 261.00 us 272 LOOKUP
Duration: 17 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Interval 0 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 24 RELEASEDIR
2.34 57.12 us 38.00 us 84.00 us 24 OPENDIR
3.13 45.75 us 28.00 us 356.00 us 40 STAT
18.46 84.43 us 48.00 us 256.00 us 128 GETXATTR
21.07 770.94 us 556.00 us 1017.00 us 16 STATFS
55.00 118.36 us 84.00 us 261.00 us 272 LOOKUP
Duration: 17 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cumulative Stats:
Block Size: 65536b+ 131072b+
No. of Reads: 0 0
No. of Writes: 17104 275
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 24 RELEASEDIR
0.00 44.75 us 37.00 us 58.00 us 24 OPENDIR
0.00 33.20 us 26.00 us 71.00 us 40 STAT
0.04 767.94 us 603.00 us 958.00 us 16 STATFS
0.04 261.44 us 64.00 us 674.00 us 48 READDIRP
0.04 82.23 us 48.00 us 177.00 us 160 GETXATTR
0.17 112.10 us 80.00 us 554.00 us 448 LOOKUP
99.69 1678.12 us 214.00 us 629362.00 us 17378 WRITE
Duration: 17 seconds
Data Read: 0 bytes
Data Written: 1156972544 bytes
Interval 0 Stats:
Block Size: 65536b+ 131072b+
No. of Reads: 0 0
No. of Writes: 17104 275
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 24 RELEASEDIR
0.00 44.75 us 37.00 us 58.00 us 24 OPENDIR
0.00 33.20 us 26.00 us 71.00 us 40 STAT
0.04 767.94 us 603.00 us 958.00 us 16 STATFS
0.04 261.44 us 64.00 us 674.00 us 48 READDIRP
0.04 82.23 us 48.00 us 177.00 us 160 GETXATTR
0.17 112.10 us 80.00 us 554.00 us 448 LOOKUP
99.69 1678.12 us 214.00 us 629362.00 us 17378 WRITE
Duration: 17 seconds
Data Read: 0 bytes
Data Written: 1156972544 bytes
[root at SC-183 log]# gluster volume profile tiered-volume-0015 stop
Stopping volume profile on tiered-volume-0015 has been successful
[root at SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root 4096 Jan 30 15:11 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:30 iobw.tst
Commit:
19949 01/30/2018 15:31:07.145654679 1517355067 command:
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015
detach commit 2>&1
19982 01/30/2018 15:31:08.636386875 1517355068 command: /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "commit"
19982 01/30/2018 15:31:10.000126293 1517355070 status=0 /bin/echo "y" |
/usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "commit"
19949 01/30/2018 15:31:10.047054376 1517355070 status=0
/usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015
detach commit 2>&1
[root at SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root 4096 Jan 30 15:11 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:31 iobw.tst
[root at SC-183 log]# ls -l
/exports/cold-segment-0004/tiered-volume-0015/cifs_share/
total 500064
drwxr-xr-x 2 root root 141 Jan 30 15:11 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 15:32 iobw.tst
[root at SC-183 log]# gluster volume profile tiered-volume-0015 start
Starting volume profile on tiered-volume-0015 has been successful
[root at SC-183 log]# gluster volume profile tiered-volume-0015 info clear
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cleared stats.
[root at SC-183 log]# gluster volume profile tiered-volume-0015 info
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cumulative Stats:
Block Size: 65536b+ 131072b+
No. of Reads: 0 0
No. of Writes: 16722 29
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR
0.00 47.50 us 37.00 us 69.00 us 22 OPENDIR
0.00 45.85 us 26.00 us 379.00 us 40 STAT
0.02 552.44 us 490.00 us 614.00 us 16 STATFS
0.02 239.14 us 63.00 us 491.00 us 44 READDIRP
0.03 74.09 us 44.00 us 157.00 us 204 GETXATTR
0.05 102.15 us 78.00 us 447.00 us 240 LOOKUP
99.87 2793.68 us 61.00 us 1026245.00 us 16751 WRITE
Duration: 16 seconds
Data Read: 0 bytes
Data Written: 1099694080 bytes
Interval 0 Stats:
Block Size: 65536b+ 131072b+
No. of Reads: 0 0
No. of Writes: 16722 29
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR
0.00 47.50 us 37.00 us 69.00 us 22 OPENDIR
0.00 45.85 us 26.00 us 379.00 us 40 STAT
0.02 552.44 us 490.00 us 614.00 us 16 STATFS
0.02 239.14 us 63.00 us 491.00 us 44 READDIRP
0.03 74.09 us 44.00 us 157.00 us 204 GETXATTR
0.05 102.15 us 78.00 us 447.00 us 240 LOOKUP
99.87 2793.68 us 61.00 us 1026245.00 us 16751 WRITE
Duration: 16 seconds
Data Read: 0 bytes
Data Written: 1099694080 bytes
[root at SC-183 log]# gluster volume profile tiered-volume-0015 stop
Stopping volume profile on tiered-volume-0015 has been successful
--
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list