<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">if kubernetes gluster pvc requires ip address i need ip based gluster peer but it’s not working and i want to make this happen. I’m wondering how others use gluster pvc. <div class=""><br class=""></div><div class="">i can not create volume when i use storage address set to ip. </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Helvetica Neue"; color: rgb(69, 69, 69);" class="">Error: volume create: vol_207bbf81f28b959c51448b919be3bb59: failed: Host 51.15.90.60 is not in 'Peer in Cluster' state</div></div><div class=""><br class=""></div><div class="">I want to know how this can possible. last 2-3 days i am working this but no progress.</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 7 Feb 2018, at 19:49, Jose A. Rivera <<a href="mailto:jarrpa@redhat.com" class="">jarrpa@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">What is the output of all those things when you use the "proper"<br class="">configuration os manage and storage addresses?<br class=""><br class="">On Wed, Feb 7, 2018 at 10:46 AM, Ercan Aydoğan <<a href="mailto:ercan.aydogan@gmail.com" class="">ercan.aydogan@gmail.com</a>> wrote:<br class=""><blockquote type="cite" class="">When i change<br class=""><br class="">manage to ip address and<br class="">storage to FQDN / hostname i can create volume but kubernetes storage<br class="">cluster not accept FQDN it requires ip address.<br class=""><br class="">i can create with this topology.json<br class=""><br class="">{<br class=""> "clusters": [<br class=""> {<br class=""> "nodes": [<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "51.15.77.14"<br class=""> ],<br class=""> "storage": [<br class=""> "pri.ostechnix.lan"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> },<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "51.15.90.60"<br class=""> ],<br class=""> "storage": [<br class=""> "sec.ostechnix.lan"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> },<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "163.172.151.120"<br class=""> ],<br class=""> "storage": [<br class=""> "third.ostechnix.lan"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> }<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""> ]<br class=""> }<br class=""> ]<br class="">}<br class=""><br class=""><br class="">but with this one i can’t.<br class=""><br class="">{<br class=""> "clusters": [<br class=""> {<br class=""> "nodes": [<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "pri.ostechnix.lan"<br class=""> ],<br class=""> "storage": [<br class=""> "51.15.77.14"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> },<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "sec.ostechnix.lan"<br class=""> ],<br class=""> "storage": [<br class=""> "51.15.90.60"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> },<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "third.ostechnix.lan"<br class=""> ],<br class=""> "storage": [<br class=""> "163.172.151.120"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> }<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""> ]<br class=""> }<br class=""> ]<br class="">}<br class=""><br class=""><br class="">root@kubemaster ~/heketi # ./heketi-cli topology load<br class="">--json=topology_with_ip.json<br class="">Creating cluster ... ID: 522adced1b7033646f0196d538b1f093<br class="">Creating node 51.15.77.148 ... ID: 1c52608dd3f624ad32cb4d1d074613d7<br class="">Adding device /dev/nbd1 ... OK<br class="">Adding device /dev/nbd2 ... OK<br class="">Adding device /dev/nbd3 ... OK<br class="">Creating node 51.15.90.10 ... ID: 36eec4fa09cf572b2a0a11f65c43b706<br class="">Adding device /dev/nbd1 ... OK<br class="">Adding device /dev/nbd2 ... OK<br class="">Adding device /dev/nbd3 ... OK<br class="">Creating node 163.172.151.170 ... ID: da1bea3e71629b4f6f8ed1f1584f521c<br class="">Adding device /dev/nbd1 ... OK<br class="">Adding device /dev/nbd2 ... OK<br class=""><br class=""><br class="">root@pri:~# gluster peer status<br class="">Number of Peers: 2<br class=""><br class="">Hostname: sec.ostechnix.lan<br class="">Uuid: 2bfc4a96-66f5-4ff9-8ee4-5e382a711c3a<br class="">State: Peer in Cluster (Connected)<br class=""><br class="">Hostname: third.ostechnix.lan<br class="">Uuid: c3ae3a1e-d9d0-4675-bf0d-0f6cf7267b30<br class="">State: Peer in Cluster (Connected)<br class=""><br class=""><br class=""><br class="">root@kubemaster ~/heketi # ./heketi-cli volume create --size=3 --replica=3<br class="">Name: vol_a6a21750e64c5317e1f949baaac25372<br class="">Size: 3<br class="">Volume Id: a6a21750e64c5317e1f949baaac25372<br class="">Cluster Id: 522adced1b7033646f0196d538b1f093<br class="">Mount: pri.ostechnix.lan:vol_a6a21750e64c5317e1f949baaac25372<br class="">Mount Options: backup-volfile-servers=sec.ostechnix.lan,third.ostechnix.lan<br class="">Durability Type: replicate<br class="">Distributed+Replica: 3<br class="">root@kubemaster ~/heketi #<br class=""><br class="">root@kubemaster ~/heketi # ./heketi-cli topology info<br class=""><br class="">Cluster Id: 522adced1b7033646f0196d538b1f093<br class=""><br class=""> Volumes:<br class=""><br class="">Name: vol_a6a21750e64c5317e1f949baaac25372<br class="">Size: 3<br class="">Id: a6a21750e64c5317e1f949baaac25372<br class="">Cluster Id: 522adced1b7033646f0196d538b1f093<br class="">Mount: pri.ostechnix.lan:vol_a6a21750e64c5317e1f949baaac25372<br class="">Mount Options: backup-volfile-servers=sec.ostechnix.lan,third.ostechnix.lan<br class="">Durability Type: replicate<br class="">Replica: 3<br class="">Snapshot: Disabled<br class=""><br class="">Bricks:<br class="">Id: 05936fa978e7d9fb534c04b4e993fefb<br class="">Path:<br class="">/var/lib/heketi/mounts/vg_057848a621c23d381b086cf7898e58cc/brick_05936fa978e7d9fb534c04b4e993fefb/brick<br class="">Size (GiB): 3<br class="">Node: 1c52608dd3f624ad32cb4d1d074613d7<br class="">Device: 057848a621c23d381b086cf7898e58cc<br class=""><br class="">Id: 624b9813a4d17bbe34565bb95d9fe2b3<br class="">Path:<br class="">/var/lib/heketi/mounts/vg_d9ce655c3d31fa92f6486abc19e155d5/brick_624b9813a4d17bbe34565bb95d9fe2b3/brick<br class="">Size (GiB): 3<br class="">Node: 36eec4fa09cf572b2a0a11f65c43b706<br class="">Device: d9ce655c3d31fa92f6486abc19e155d5<br class=""><br class="">Id: a779eb5de0131ab97d8d17e8ddad4a3e<br class="">Path:<br class="">/var/lib/heketi/mounts/vg_2cb5b83b84bfd1c0e06ac99779f413d7/brick_a779eb5de0131ab97d8d17e8ddad4a3e/brick<br class="">Size (GiB): 3<br class="">Node: da1bea3e71629b4f6f8ed1f1584f521c<br class="">Device: 2cb5b83b84bfd1c0e06ac99779f413d7<br class=""><br class=""><br class=""> Nodes:<br class=""><br class="">Node Id: 1c52608dd3f624ad32cb4d1d074613d7<br class="">State: online<br class="">Cluster Id: 522adced1b7033646f0196d538b1f093<br class="">Zone: 1<br class="">Management Hostname: 51.15.77.14<br class="">Storage Hostname: pri.ostechnix.lan<br class="">Devices:<br class="">Id:057848a621c23d381b086cf7898e58cc Name:/dev/nbd3 State:online<br class="">Size (GiB):139 Used (GiB):3 Free (GiB):136<br class="">Bricks:<br class="">Id:05936fa978e7d9fb534c04b4e993fefb Size (GiB):3 Path:<br class="">/var/lib/heketi/mounts/vg_057848a621c23d381b086cf7898e58cc/brick_05936fa978e7d9fb534c04b4e993fefb/brick<br class="">Id:24f557371d0eaf2c72065f4220113988 Name:/dev/nbd1 State:online<br class="">Size (GiB):46 Used (GiB):0 Free (GiB):46<br class="">Bricks:<br class="">Id:670546b880260bb0240b9e0ac51bb82c Name:/dev/nbd2 State:online<br class="">Size (GiB):46 Used (GiB):0 Free (GiB):46<br class="">Bricks:<br class=""><br class="">Node Id: 36eec4fa09cf572b2a0a11f65c43b706<br class="">State: online<br class="">Cluster Id: 522adced1b7033646f0196d538b1f093<br class="">Zone: 1<br class="">Management Hostname: 51.15.90.60<br class="">Storage Hostname: sec.ostechnix.lan<br class="">Devices:<br class="">Id:74320e3bd92b7fdffa06499b17fb3c8f Name:/dev/nbd1 State:online<br class="">Size (GiB):46 Used (GiB):0 Free (GiB):46<br class="">Bricks:<br class="">Id:923f2d04b03145600e5fc2035b5699c0 Name:/dev/nbd2 State:online<br class="">Size (GiB):46 Used (GiB):0 Free (GiB):46<br class="">Bricks:<br class="">Id:d9ce655c3d31fa92f6486abc19e155d5 Name:/dev/nbd3 State:online<br class="">Size (GiB):139 Used (GiB):3 Free (GiB):136<br class="">Bricks:<br class="">Id:624b9813a4d17bbe34565bb95d9fe2b3 Size (GiB):3 Path:<br class="">/var/lib/heketi/mounts/vg_d9ce655c3d31fa92f6486abc19e155d5/brick_624b9813a4d17bbe34565bb95d9fe2b3/brick<br class=""><br class="">Node Id: da1bea3e71629b4f6f8ed1f1584f521c<br class="">State: online<br class="">Cluster Id: 522adced1b7033646f0196d538b1f093<br class="">Zone: 1<br class="">Management Hostname: 163.172.151.120<br class="">Storage Hostname: third.ostechnix.lan<br class="">Devices:<br class="">Id:23621c56af1237380bac9bb482d13859 Name:/dev/nbd2 State:online<br class="">Size (GiB):46 Used (GiB):0 Free (GiB):46<br class="">Bricks:<br class="">Id:2cb5b83b84bfd1c0e06ac99779f413d7 Name:/dev/nbd3 State:online<br class="">Size (GiB):139 Used (GiB):3 Free (GiB):136<br class="">Bricks:<br class="">Id:a779eb5de0131ab97d8d17e8ddad4a3e Size (GiB):3 Path:<br class="">/var/lib/heketi/mounts/vg_2cb5b83b84bfd1c0e06ac99779f413d7/brick_a779eb5de0131ab97d8d17e8ddad4a3e/brick<br class="">Id:89648509136a29dea7712733f1b91733 Name:/dev/nbd1 State:online<br class="">Size (GiB):46 Used (GiB):0 Free (GiB):46<br class="">Bricks:<br class=""><br class=""><br class="">this is ok.<br class=""><br class="">but this is not work with kubernetes storage class because it need’s ip<br class="">address on storage hostname side.<br class=""><br class="">Warning ProvisioningFailed 21s persistentvolume-controller Failed<br class="">to provision volume with StorageClass "fast": create volume error: failed to<br class="">create endpoint/service error creating endpoint: Endpoints<br class="">"glusterfs-dynamic-claim1" is invalid: [subsets[0].addresses[0].ip: Invalid<br class="">value: "pri.ostechnix.lan": must be a valid IP address, (e.g. 10.9.8.7),<br class="">subsets[0].addresses[1].ip: Invalid value: "third.ostechnix.lan": must be a<br class="">valid IP address, (e.g. 10.9.8.7), subsets[0].addresses[2].ip: Invalid<br class="">value: "sec.ostechnix.lan": must be a valid IP address, (e.g. 10.9.8.7)]<br class=""><br class=""><br class=""><br class=""><br class=""><br class="">On 7 Feb 2018, at 19:21, Jose A. Rivera <<a href="mailto:jarrpa@redhat.com" class="">jarrpa@redhat.com</a>> wrote:<br class=""><br class="">What's the output of the topology load command?<br class=""><br class="">Can you verify that glusterd is running and healthy on all the nodes?<br class="">I'm not sure about Ubuntu, but on Fedora we do "systemctl status<br class="">glusterd", so something like that.<br class=""><br class="">Does running "glusterfs pool list" show all the nodes? This only needs<br class="">to be run on one of the nodes.<br class=""><br class="">Finally, do you have a firewall up and does it have the requisite<br class="">ports open for GlusterFS?<br class=""><br class="">--Jose<br class=""><br class="">On Wed, Feb 7, 2018 at 9:20 AM, Ercan Aydoğan <<a href="mailto:ercan.aydogan@gmail.com" class="">ercan.aydogan@gmail.com</a>><br class="">wrote:<br class=""><br class="">Yes, i delete heketi.db before every try<br class=""><br class=""><br class="">On 7 Feb 2018, at 18:18, Jose A. Rivera <<a href="mailto:jarrpa@redhat.com" class="">jarrpa@redhat.com</a>> wrote:<br class=""><br class="">When you deleted the cluster, did you also delete the heketi database?<br class=""><br class="">--Jose<br class=""><br class="">On Wed, Feb 7, 2018 at 3:30 AM, Ercan Aydoğan <<a href="mailto:ercan.aydogan@gmail.com" class="">ercan.aydogan@gmail.com</a>><br class="">wrote:<br class=""><br class="">Gluster cluster on ubuntu 16.04 and i remove with this commands<br class=""><br class="">apt-get purge glusterfs-server -y --allow-change-held-packages<br class="">rm -rf /var/lib/glusterd<br class="">rm -rf /var/log/glusterfs/<br class="">wipefs -a --force /dev/nbd1<br class="">wipefs -a --force /dev/nbd2<br class="">wipefs -a --force /dev/nbd3<br class=""><br class="">after reboot install with<br class=""><br class="">apt-get install -y software-properties-common<br class="">add-apt-repository ppa:gluster/glusterfs-3.11<br class="">apt-get update<br class="">apt-get install -y glusterfs-server<br class=""><br class="">after this i’m using<br class=""><br class="">/heketi-cli topology load --json=topology.json<br class=""><br class=""><br class=""><br class="">but<br class=""><br class="">i can’t create any volume with gluster cmd or heketi-cli maybe this is<br class="">hostname or /etc/hostname issue.<br class=""><br class="">my current /etc/hosts is<br class=""><br class="">node 1<br class=""><br class=""><br class="">root@pri:/var/log/glusterfs# cat /etc/hosts<br class="">#127.0.0.1 localhost<br class="">127.0.0.1 pri.ostechnix.lan pri<br class="">::1 localhost ip6-localhost ip6-loopback<br class="">ff02::1 ip6-allnodes<br class="">ff02::2 ip6-allrouters<br class=""><br class="">51.15.90.60 sec.ostechnix.lan sec<br class="">163.172.151.120 third.ostechnix.lan third<br class="">root@pri:/var/log/glusterfs#<br class=""><br class="">on every node i set 127.0.0.1 matching hostname.<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><br class="">On 7 Feb 2018, at 11:51, Humble Chirammal <<a href="mailto:hchiramm@redhat.com" class="">hchiramm@redhat.com</a>> wrote:<br class=""><br class="">true, storage should be IP address. However afaict, it failed in "Peer in<br class="">cluster" , because gluster cluster is formed with different IP/hostname and<br class="">its stored in metadata. If you can delete the cluster and recreate it with<br class="">"storage" in IP, it should work I believe.<br class=""><br class="">On Wed, Feb 7, 2018 at 2:16 PM, Ercan Aydoğan <<a href="mailto:ercan.aydogan@gmail.com" class="">ercan.aydogan@gmail.com</a>><br class="">wrote:<br class=""><br class=""><br class="">Hello,<br class=""><br class="">i have dedicated 3.11.3 version glusterfs 3 nodes. i can create volumes<br class="">with both gluster own command line utility and heketi-cli it’s ok.<br class=""><br class="">If i use storage hostname FQDN i can create cluster with<br class=""><br class="">/heketi-cli topology load --json=topology.json<br class=""><br class="">after storageclass , secret and pvc creation i got this error.<br class=""><br class="">kubectl get pvc claim1 returns<br class=""><br class="">root@kubemaster ~ # kubectl describe pvc claim1<br class="">Name: claim1<br class="">Namespace: default<br class="">StorageClass: fast<br class="">Status: Pending<br class="">Volume:<br class="">Labels: <none><br class="">Annotations:<br class=""><a href="http://volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs" class="">volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs</a><br class="">Finalizers: []<br class="">Capacity:<br class="">Access Modes:<br class="">Events:<br class="">Type Reason Age From Message<br class="">---- ------ ---- ---- -------<br class="">Warning ProvisioningFailed 21s persistentvolume-controller Failed<br class="">to provision volume with StorageClass "fast": create volume error: failed to<br class="">create endpoint/service error creating endpoint: Endpoints<br class="">"glusterfs-dynamic-claim1" is invalid: [subsets[0].addresses[0].ip: Invalid<br class="">value: "pri.ostechnix.lan": must be a valid IP address, (e.g. 10.9.8.7),<br class="">subsets[0].addresses[1].ip: Invalid value: "third.ostechnix.lan": must be a<br class="">valid IP address, (e.g. 10.9.8.7), subsets[0].addresses[2].ip: Invalid<br class="">value: "sec.ostechnix.lan": must be a valid IP address, (e.g. 10.9.8.7)]<br class=""><br class=""><br class="">my topology.json content is<br class=""><br class="">{<br class="">"clusters": [<br class=""> {<br class=""> "nodes": [<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "51.15.77.14"<br class=""> ],<br class=""> "storage": [<br class=""> "pri.ostechnix.lan"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> },<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "51.15.90.60"<br class=""> ],<br class=""> "storage": [<br class=""> "sec.ostechnix.lan"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> },<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "163.172.151.120"<br class=""> ],<br class=""> "storage": [<br class=""> "third.ostechnix.lan"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> }<br class=""><br class=""><br class=""> ]<br class=""> }<br class="">]<br class="">}<br class=""><br class=""><br class="">Yes, it says storage must be ip for endpoint creation. But if i change<br class=""><br class="">manage : hostname<br class="">storage: ip address<br class=""><br class="">{<br class="">"clusters": [<br class=""> {<br class=""> "nodes": [<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "pri.ostechnix.lan"<br class=""> ],<br class=""> "storage": [<br class=""> "51.15.77.14"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> },<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "sec.ostechnix.lan"<br class=""> ],<br class=""> "storage": [<br class=""> "51.15.90.60"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> },<br class=""> {<br class=""> "node": {<br class=""> "hostnames": {<br class=""> "manage": [<br class=""> "third.ostechnix.lan"<br class=""> ],<br class=""> "storage": [<br class=""> "163.172.151.120"<br class=""> ]<br class=""> },<br class=""> "zone": 1<br class=""> },<br class=""> "devices": [<br class=""> "/dev/nbd1",<br class=""> "/dev/nbd2",<br class=""> "/dev/nbd3"<br class=""> ]<br class=""> }<br class=""><br class=""><br class=""> ]<br class=""> }<br class="">]<br class="">}<br class=""><br class="">i can not create volume with heketi-cli.<br class=""><br class="">it says<br class=""><br class="">root@kubemaster ~/heketi # ./heketi-cli volume create --size=3 --replica=3<br class="">Error: volume create: vol_207bbf81f28b959c51448b919be3bb59: failed: Host<br class="">51.15.90.60 is not in 'Peer in Cluster’ state<br class=""><br class="">i need advice how can fix this issue.<br class=""><br class=""><br class="">_______________________________________________<br class="">heketi-devel mailing list<br class="">heketi-devel@gluster.org<br class="">http://lists.gluster.org/mailman/listinfo/heketi-devel<br class=""><br class=""><br class=""><br class=""><br class="">--<br class="">Cheers,<br class="">Humble<br class=""><br class="">Red Hat Storage Engineering<br class="">Mastering KVM Virtualization: http://amzn.to/2vFTXaW<br class="">Website: http://humblec.com<br class=""><br class=""><br class=""><br class="">_______________________________________________<br class="">heketi-devel mailing list<br class="">heketi-devel@gluster.org<br class="">http://lists.gluster.org/mailman/listinfo/heketi-devel<br class=""><br class=""><br class=""><br class=""></blockquote></div></div></blockquote></div><br class=""></div></body></html>