[heketi-devel] heketi-cli kubernetes gluster pvc endpoint issue

Ercan Aydoğan ercan.aydogan at gmail.com
Wed Feb 7 15:20:53 UTC 2018


Yes, i delete heketi.db before every try


> On 7 Feb 2018, at 18:18, Jose A. Rivera <jarrpa at redhat.com> wrote:
> 
> When you deleted the cluster, did you also delete the heketi database?
> 
> --Jose
> 
> On Wed, Feb 7, 2018 at 3:30 AM, Ercan Aydoğan <ercan.aydogan at gmail.com> wrote:
>> Gluster cluster on ubuntu 16.04 and i remove with this commands
>> 
>> apt-get purge glusterfs-server -y  --allow-change-held-packages
>> rm -rf /var/lib/glusterd
>> rm -rf /var/log/glusterfs/
>> wipefs -a --force /dev/nbd1
>> wipefs -a --force /dev/nbd2
>> wipefs -a --force /dev/nbd3
>> 
>> after reboot  install with
>> 
>> apt-get install -y software-properties-common
>> add-apt-repository ppa:gluster/glusterfs-3.11
>> apt-get update
>> apt-get install -y glusterfs-server
>> 
>> after this i’m using
>> 
>>> /heketi-cli   topology  load --json=topology.json
>> 
>> 
>> but
>> 
>> i can’t create any volume with gluster cmd or heketi-cli maybe this is
>> hostname or /etc/hostname issue.
>> 
>> my current /etc/hosts is
>> 
>> node 1
>> 
>> 
>> root at pri:/var/log/glusterfs# cat /etc/hosts
>> #127.0.0.1       localhost
>> 127.0.0.1        pri.ostechnix.lan     pri
>> ::1             localhost ip6-localhost ip6-loopback
>> ff02::1         ip6-allnodes
>> ff02::2         ip6-allrouters
>> 
>> 51.15.90.60      sec.ostechnix.lan     sec
>> 163.172.151.120  third.ostechnix.lan   third
>> root at pri:/var/log/glusterfs#
>> 
>> on every node i set 127.0.0.1 matching hostname.
>> 
>> 
>> 
>> 
>> 
>> 
>> On 7 Feb 2018, at 11:51, Humble Chirammal <hchiramm at redhat.com> wrote:
>> 
>> true, storage should be IP address. However afaict, it failed in "Peer in
>> cluster" , because gluster cluster is formed with different IP/hostname and
>> its stored in metadata. If you can delete the cluster and recreate it with
>> "storage" in IP, it should work I believe.
>> 
>> On Wed, Feb 7, 2018 at 2:16 PM, Ercan Aydoğan <ercan.aydogan at gmail.com>
>> wrote:
>>> 
>>> Hello,
>>> 
>>> i have dedicated 3.11.3 version glusterfs 3 nodes. i can create volumes
>>> with both gluster own command line utility and heketi-cli it’s ok.
>>> 
>>> If i use  storage hostname FQDN i can create cluster with
>>> 
>>> /heketi-cli   topology  load --json=topology.json
>>> 
>>> after storageclass , secret and pvc creation i got this error.
>>> 
>>> kubectl get pvc claim1 returns
>>> 
>>> root at kubemaster ~ # kubectl describe pvc claim1
>>> Name:          claim1
>>> Namespace:     default
>>> StorageClass:  fast
>>> Status:        Pending
>>> Volume:
>>> Labels:        <none>
>>> Annotations:
>>> volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs
>>> Finalizers:    []
>>> Capacity:
>>> Access Modes:
>>> Events:
>>>  Type     Reason              Age   From                         Message
>>>  ----     ------              ----  ----                         -------
>>>  Warning  ProvisioningFailed  21s   persistentvolume-controller  Failed
>>> to provision volume with StorageClass "fast": create volume error: failed to
>>> create endpoint/service error creating endpoint: Endpoints
>>> "glusterfs-dynamic-claim1" is invalid: [subsets[0].addresses[0].ip: Invalid
>>> value: "pri.ostechnix.lan": must be a valid IP address, (e.g. 10.9.8.7),
>>> subsets[0].addresses[1].ip: Invalid value: "third.ostechnix.lan": must be a
>>> valid IP address, (e.g. 10.9.8.7), subsets[0].addresses[2].ip: Invalid
>>> value: "sec.ostechnix.lan": must be a valid IP address, (e.g. 10.9.8.7)]
>>> 
>>> 
>>> my topology.json content is
>>> 
>>> {
>>>  "clusters": [
>>>    {
>>>      "nodes": [
>>>        {
>>>          "node": {
>>>            "hostnames": {
>>>              "manage": [
>>>                "51.15.77.14"
>>>              ],
>>>              "storage": [
>>>                "pri.ostechnix.lan"
>>>              ]
>>>            },
>>>            "zone": 1
>>>          },
>>>          "devices": [
>>>            "/dev/nbd1",
>>>            "/dev/nbd2",
>>>    "/dev/nbd3"
>>>          ]
>>>        },
>>>        {
>>>          "node": {
>>>            "hostnames": {
>>>              "manage": [
>>>                "51.15.90.60"
>>>              ],
>>>              "storage": [
>>>                "sec.ostechnix.lan"
>>>              ]
>>>            },
>>>            "zone": 1
>>>          },
>>>          "devices": [
>>>            "/dev/nbd1",
>>>            "/dev/nbd2",
>>>            "/dev/nbd3"
>>>          ]
>>>        },
>>>        {
>>>          "node": {
>>>            "hostnames": {
>>>              "manage": [
>>>                "163.172.151.120"
>>>              ],
>>>              "storage": [
>>>                "third.ostechnix.lan"
>>>              ]
>>>            },
>>>            "zone": 1
>>>          },
>>>          "devices": [
>>>            "/dev/nbd1",
>>>            "/dev/nbd2",
>>>             "/dev/nbd3"
>>>          ]
>>>        }
>>> 
>>> 
>>>      ]
>>>    }
>>>  ]
>>> }
>>> 
>>> 
>>> Yes, it says storage must be ip for endpoint creation. But if i change
>>> 
>>> manage : hostname
>>> storage: ip address
>>> 
>>> {
>>>  "clusters": [
>>>    {
>>>      "nodes": [
>>>        {
>>>          "node": {
>>>            "hostnames": {
>>>              "manage": [
>>>                "pri.ostechnix.lan"
>>>              ],
>>>              "storage": [
>>>                "51.15.77.14"
>>>              ]
>>>            },
>>>            "zone": 1
>>>          },
>>>          "devices": [
>>>            "/dev/nbd1",
>>>            "/dev/nbd2",
>>>    "/dev/nbd3"
>>>          ]
>>>        },
>>>        {
>>>          "node": {
>>>            "hostnames": {
>>>              "manage": [
>>>                "sec.ostechnix.lan"
>>>              ],
>>>              "storage": [
>>>                "51.15.90.60"
>>>              ]
>>>            },
>>>            "zone": 1
>>>          },
>>>          "devices": [
>>>            "/dev/nbd1",
>>>            "/dev/nbd2",
>>>            "/dev/nbd3"
>>>          ]
>>>        },
>>>        {
>>>          "node": {
>>>            "hostnames": {
>>>              "manage": [
>>>                "third.ostechnix.lan"
>>>              ],
>>>              "storage": [
>>>                "163.172.151.120"
>>>              ]
>>>            },
>>>            "zone": 1
>>>          },
>>>          "devices": [
>>>            "/dev/nbd1",
>>>            "/dev/nbd2",
>>>             "/dev/nbd3"
>>>          ]
>>>        }
>>> 
>>> 
>>>      ]
>>>    }
>>>  ]
>>> }
>>> 
>>> i can not create volume with heketi-cli.
>>> 
>>> it says
>>> 
>>> root at kubemaster ~/heketi # ./heketi-cli volume create --size=3 --replica=3
>>> Error: volume create: vol_207bbf81f28b959c51448b919be3bb59: failed: Host
>>> 51.15.90.60 is not in 'Peer in Cluster’ state
>>> 
>>> i need advice how can fix this issue.
>>> 
>>> 
>>> _______________________________________________
>>> heketi-devel mailing list
>>> heketi-devel at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/heketi-devel
>>> 
>> 
>> 
>> 
>> --
>> Cheers,
>> Humble
>> 
>> Red Hat Storage Engineering
>> Mastering KVM Virtualization: http://amzn.to/2vFTXaW
>> Website: http://humblec.com
>> 
>> 
>> 
>> _______________________________________________
>> heketi-devel mailing list
>> heketi-devel at gluster.org
>> http://lists.gluster.org/mailman/listinfo/heketi-devel
>> 



More information about the heketi-devel mailing list