[Gluster-users] Getting glusterfs to expand volume size to brick size
Artem Russakovskii
archon810 at gmail.com
Tue Apr 17 04:33:44 UTC 2018
I just remembered that I didn't run
https://docs.gluster.org/en/v3/Upgrade-Guide/op_version/ for this test
volume/box like I did for the main production gluster, and one of these ops
- either heal or the op-version, resolved the issue.
I'm now seeing:
pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
3: option shared-brick-count 1
dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
3: option shared-brick-count 1
dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
3: option shared-brick-count 1
/dev/sdd 25071M
1491M 22284M 7% /mnt/pylon_block1
/dev/sdc 26079M
1491M 23241M 7% /mnt/pylon_block2
/dev/sde 25071M
1491M 22315M 7% /mnt/pylon_block3
localhost:/dev_apkmirror_data 25071M
1742M 22284M 8% /mnt/dev_apkmirror_data1
localhost:/dev_apkmirror_data 25071M
1742M 22284M 8% /mnt/dev_apkmirror_data2
localhost:/dev_apkmirror_data 25071M
1742M 22284M 8% /mnt/dev_apkmirror_data3
localhost:/dev_apkmirror_data 25071M
1742M 22284M 8% /mnt/dev_apkmirror_data_ganesha
Problem is solved!
Sincerely,
Artem
--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
<https://plus.google.com/+ArtemRussakovskii> | @ArtemR
<http://twitter.com/ArtemR>
On Mon, Apr 16, 2018 at 9:29 PM, Nithya Balachandran <nbalacha at redhat.com>
wrote:
> Ok, it looks like the same problem.
>
>
> @Amar, this fix is supposed to be in 4.0.1. Is it possible to regenerate
> the volfiles to fix this?
>
> Regards,
> Nithya
>
> On 17 April 2018 at 09:57, Artem Russakovskii <archon810 at gmail.com> wrote:
>
>> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
>> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
>> 3: option shared-brick-count 3
>>
>> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
>> 3: option shared-brick-count 3
>>
>> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
>> 3: option shared-brick-count 3
>>
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
>> On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran <nbalacha at redhat.com
>> > wrote:
>>
>>> Hi Artem,
>>>
>>> Was the volume size correct before the bricks were expanded?
>>>
>>> This sounds like [1] but that should have been fixed in 4.0.0. Can you
>>> let us know the values of shared-brick-count in the files in
>>> /var/lib/glusterd/vols/dev_apkmirror_data/ ?
>>>
>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880
>>>
>>> On 17 April 2018 at 05:17, Artem Russakovskii <archon810 at gmail.com>
>>> wrote:
>>>
>>>> Hi Nithya,
>>>>
>>>> I'm on Gluster 4.0.1.
>>>>
>>>> I don't think the bricks were smaller before - if they were, maybe 20GB
>>>> because Linode's minimum is 20GB, then I extended them to 25GB, resized
>>>> with resize2fs as instructed, and rebooted many times over since. Yet,
>>>> gluster refuses to see the full disk size.
>>>>
>>>> Here's the status detail output:
>>>>
>>>> gluster volume status dev_apkmirror_data detail
>>>> Status of volume: dev_apkmirror_data
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Brick : Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
>>>> TCP Port : 49152
>>>> RDMA Port : 0
>>>> Online : Y
>>>> Pid : 1263
>>>> File System : ext4
>>>> Device : /dev/sdd
>>>> Mount Options : rw,relatime,data=ordered
>>>> Inode Size : 256
>>>> Disk Space Free : 23.0GB
>>>> Total Disk Space : 24.5GB
>>>> Inode Count : 1638400
>>>> Free Inodes : 1625429
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Brick : Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
>>>> TCP Port : 49153
>>>> RDMA Port : 0
>>>> Online : Y
>>>> Pid : 1288
>>>> File System : ext4
>>>> Device : /dev/sdc
>>>> Mount Options : rw,relatime,data=ordered
>>>> Inode Size : 256
>>>> Disk Space Free : 24.0GB
>>>> Total Disk Space : 25.5GB
>>>> Inode Count : 1703936
>>>> Free Inodes : 1690965
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Brick : Brick pylon:/mnt/pylon_block3/dev_apkmirror_data
>>>> TCP Port : 49154
>>>> RDMA Port : 0
>>>> Online : Y
>>>> Pid : 1313
>>>> File System : ext4
>>>> Device : /dev/sde
>>>> Mount Options : rw,relatime,data=ordered
>>>> Inode Size : 256
>>>> Disk Space Free : 23.0GB
>>>> Total Disk Space : 24.5GB
>>>> Inode Count : 1638400
>>>> Free Inodes : 1625433
>>>>
>>>>
>>>>
>>>> What's interesting here is that the gluster volume size is exactly 1/3
>>>> of the total (8357M * 3 = 25071M). Yet, each block device is separate, and
>>>> the total storage available is 25071M on each brick.
>>>>
>>>> The fstab is as follows:
>>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1
>>>> ext4 defaults 0 2
>>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2
>>>> ext4 defaults 0 2
>>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3
>>>> ext4 defaults 0 2
>>>>
>>>> localhost:/dev_apkmirror_data /mnt/dev_apkmirror_data1 glusterfs
>>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>>> localhost:/dev_apkmirror_data /mnt/dev_apkmirror_data2 glusterfs
>>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>>> localhost:/dev_apkmirror_data /mnt/dev_apkmirror_data3 glusterfs
>>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>>> localhost:/dev_apkmirror_data /mnt/dev_apkmirror_data_ganesha
>>>> nfs4 defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5
>>>> 0 0
>>>>
>>>> The latter entry is for an nfs ganesha test, in case it matters (which,
>>>> btw, fails miserably with all kinds of stability issues about broken pipes).
>>>>
>>>> Note: this is a test server, so all 3 bricks are attached and mounted
>>>> on the same server.
>>>>
>>>>
>>>> Sincerely,
>>>> Artem
>>>>
>>>> --
>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>> beerpla.net | +ArtemRussakovskii
>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>>> <http://twitter.com/ArtemR>
>>>>
>>>> On Sun, Apr 15, 2018 at 10:56 PM, Nithya Balachandran <
>>>> nbalacha at redhat.com> wrote:
>>>>
>>>>> What version of Gluster are you running? Were the bricks smaller
>>>>> earlier?
>>>>>
>>>>> Regards,
>>>>> Nithya
>>>>>
>>>>> On 15 April 2018 at 00:09, Artem Russakovskii <archon810 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a 3-brick replicate volume, but for some reason I can't get it
>>>>>> to expand to the size of the bricks. The bricks are 25GB, but even after
>>>>>> multiple gluster restarts and remounts, the volume is only about 8GB.
>>>>>>
>>>>>> I believed I could always extend the bricks (we're using Linode block
>>>>>> storage, which allows extending block devices after they're created), and
>>>>>> gluster would see the newly available space and extend to use it.
>>>>>>
>>>>>> Multiple Google searches, and I'm still nowhere. Any ideas?
>>>>>>
>>>>>> df | ack "block|data"
>>>>>> Filesystem
>>>>>> 1M-blocks Used Available Use% Mounted on
>>>>>> /dev/sdd
>>>>>> 25071M 1491M 22284M 7% /mnt/pylon_block1
>>>>>> /dev/sdc
>>>>>> 26079M 1491M 23241M 7% /mnt/pylon_block2
>>>>>> /dev/sde
>>>>>> 25071M 1491M 22315M 7% /mnt/pylon_block3
>>>>>> localhost:/dev_apkmirror_data
>>>>>> 8357M 581M 7428M 8% /mnt/dev_apkmirror_data1
>>>>>> localhost:/dev_apkmirror_data
>>>>>> 8357M 581M 7428M 8% /mnt/dev_apkmirror_data2
>>>>>> localhost:/dev_apkmirror_data
>>>>>> 8357M 581M 7428M 8% /mnt/dev_apkmirror_data3
>>>>>>
>>>>>>
>>>>>>
>>>>>> gluster volume info
>>>>>>
>>>>>> Volume Name: dev_apkmirror_data
>>>>>> Type: Replicate
>>>>>> Volume ID: cd5621ee-7fab-401b-b720-08863717ed56
>>>>>> Status: Started
>>>>>> Snapshot Count: 0
>>>>>> Number of Bricks: 1 x 3 = 3
>>>>>> Transport-type: tcp
>>>>>> Bricks:
>>>>>> Brick1: pylon:/mnt/pylon_block1/dev_apkmirror_data
>>>>>> Brick2: pylon:/mnt/pylon_block2/dev_apkmirror_data
>>>>>> Brick3: pylon:/mnt/pylon_block3/dev_apkmirror_data
>>>>>> Options Reconfigured:
>>>>>> disperse.eager-lock: off
>>>>>> cluster.lookup-unhashed: auto
>>>>>> cluster.read-hash-mode: 0
>>>>>> performance.strict-o-direct: on
>>>>>> cluster.shd-max-threads: 12
>>>>>> performance.nl-cache-timeout: 600
>>>>>> performance.nl-cache: on
>>>>>> cluster.quorum-count: 1
>>>>>> cluster.quorum-type: fixed
>>>>>> network.ping-timeout: 5
>>>>>> network.remote-dio: enable
>>>>>> performance.rda-cache-limit: 256MB
>>>>>> performance.parallel-readdir: on
>>>>>> network.inode-lru-limit: 500000
>>>>>> performance.md-cache-timeout: 600
>>>>>> performance.cache-invalidation: on
>>>>>> performance.stat-prefetch: on
>>>>>> features.cache-invalidation-timeout: 600
>>>>>> features.cache-invalidation: on
>>>>>> performance.io-thread-count: 32
>>>>>> server.event-threads: 4
>>>>>> client.event-threads: 4
>>>>>> performance.read-ahead: off
>>>>>> cluster.lookup-optimize: on
>>>>>> performance.client-io-threads: on
>>>>>> performance.cache-size: 1GB
>>>>>> transport.address-family: inet
>>>>>> performance.readdir-ahead: on
>>>>>> nfs.disable: on
>>>>>> cluster.readdir-optimize: on
>>>>>>
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> Sincerely,
>>>>>> Artem
>>>>>>
>>>>>> --
>>>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>>>> beerpla.net | +ArtemRussakovskii
>>>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>>>>> <http://twitter.com/ArtemR>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180416/65fcd4c1/attachment.html>
More information about the Gluster-users
mailing list