[Gluster-users] [rhos-list] [gluster-swift] Gluster UFO 3.4 swift Multi tenant question

Tue Sep 17 20:38:48 UTC 2013

Luis
Thanks for the timely response.

On Tue, Sep 17, 2013 at 1:52 PM, Luis Pabon <lpabon at redhat.com> wrote:
>
> On 09/17/2013 11:13 AM, Paul Robert Marino wrote:
>>
>> Luis
>> well thats intresting because it was my impression that Gluster UFO
>> 3.4 was based on the Grizzly version of Swift.
>
> [LP] Sorry, the gluster-ufo RPM is Essex only.

[PRM] The source of my confusion was here
http://www.gluster.org/community/documentation/index.php/Features34
and here http://www.gluster.org/2013/06/glusterfs-3-4-and-swift-where-are-all-the-pieces/
These pages on the gluster site should probably be updated to reflect
the changes.

>
>
>> Also I was previously unaware of this new rpm which doesnt seem to be
>> in a repo any where.
>
> [LP] gluster-swift project RPMs have been submitted to Fedora and are
> currently being reviewed.

[PRM] Cool if they are in the EPEL testing repo Ill look for them
there because I would rather pull the properly EPEL signed RPMs if
they exist just to make node deployments easier. If not Ill ask some
of my friends offline if they can help expedite it.

>
>
>> also there is a line in this new howto that is extreamly unclear
>>
>> "
>> /usr/bin/gluster-swift-gen-builders test
>> "
>> in place of "test" what should go there is it the tenant ID string,
>> the tenant name, or just a generic volume you can name whatever you
>> want?
>> in other words how should the Gluster volumes be named?
>
> [LP] We will clarify that in the quick start guide.  Thank you for pointing
> it out.  While we update the community site, please refer to the
> documentation available here http://goo.gl/bQFI8o for a usage guide.
>
> As for the tool, the format is:
> gluster-swift-gen-buildes [VOLUME] [VOLUME...]
>
> Where VOLUME is the name of the GlusterFS volume to use for object storage.
> For example
> if the following two GlusterFS volumes, volume1 and volume2, need to be
> accessed over Swift,
> then you can type the following:
>
> # gluster-swift-gen-builders volume1 volume2

[PRM] That part I understood however it doesn't answer the question exactly.

Correct me if I'm wrong but looking over the code briefly it looks as
though the volume name needs to be the same as the tenant ID number
like it did with Gluster UFO 3.3.
so for example
if I do a " keystone tenant-list" and a see tenant1 with an id of
"f6da0a8151ff43b7be10d961a20c94d6" then I would need to create a
volume named f6da0a8151ff43b7be10d961a20c94d6

If I can name the volumes whatever I want or give them the same name
as the tenant that would be great because it makes it easier for other
SA's who are not directly working with OpenStack but may need to mount
the volumes to comprehend, but its not urgently needed.

One thing I was glad to see is that with Gluster UFO 3.3 I had to add
mount points to /etc/fstab for each volume and manually create the
directories for the mount points this looks to have been corrected in
Gluster-Swift.

>
> For more information please read: http://goo.gl/gd8LkW
>
> Let us know if you have any more questions or comments.

[PRM] I may fork the Github repo and add some changes that may be
beneficial so they can be reviewed and possibly merged.
for example it would be nice if the  gluster-swift-gen-buildes script
used the value of the mount_ip field in /etc/swift/fs.conf instead of
127.0.0.1 if its defined.
also I might make a more robust version that allows create, add,
remove, and list options.

Ill do testing tomorrow and let everyone know how it goes.

>
> - Luis
>
>>
>>
>> On Tue, Sep 17, 2013 at 10:10 AM, Luis Pabon <lpabon at redhat.com> wrote:
>>>
>>> First thing I can see is that you have Essex based gluster-ufo-* which
>>> has
>>> been replaced by the gluster-swift project.  We are currently in progress
>>> of
>>> replacing the gluster-ufo-* with RPMs from the gluster-swift project in
>>> Fedora.
>>>
>>> Please checkout the following quickstart guide which show how to download
>>> the Grizzly version of gluster-swift:
>>>
>>> https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md
>>> .
>>>
>>> For more information please visit: https://launchpad.net/gluster-swift
>>>
>>> - Luis
>>>
>>>
>>> On 09/16/2013 05:02 PM, Paul Robert Marino wrote:
>>>
>>> Sorry for the delay on reporting the details. I got temporarily pulled
>>> off the project and dedicated to a different project which was
>>> considered higher priority by my employer. I'm just getting back to
>>> doing my normal work today.
>>>
>>> first here are the rpms I have installed
>>> "
>>>   rpm -qa |grep -P -i '(gluster|swift)'
>>> glusterfs-libs-3.4.0-8.el6.x86_64
>>> glusterfs-server-3.4.0-8.el6.x86_64
>>> openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
>>> openstack-swift-proxy-1.8.0-2.el6.noarch
>>> glusterfs-3.4.0-8.el6.x86_64
>>> glusterfs-cli-3.4.0-8.el6.x86_64
>>> glusterfs-geo-replication-3.4.0-8.el6.x86_64
>>> glusterfs-api-3.4.0-8.el6.x86_64
>>> openstack-swift-1.8.0-2.el6.noarch
>>> openstack-swift-container-1.8.0-2.el6.noarch
>>> openstack-swift-object-1.8.0-2.el6.noarch
>>> glusterfs-fuse-3.4.0-8.el6.x86_64
>>> glusterfs-rdma-3.4.0-8.el6.x86_64
>>> openstack-swift-account-1.8.0-2.el6.noarch
>>> glusterfs-ufo-3.4.0-8.el6.noarch
>>> glusterfs-vim-3.2.7-1.el6.x86_64
>>> python-swiftclient-1.4.0-1.el6.noarch
>>>
>>> here are some key config files note I've changed the passwords I'm
>>> using and hostnames
>>> "
>>>   cat /etc/swift/account-server.conf
>>> [DEFAULT]
>>> mount_check = true
>>> bind_port = 6012
>>> user = root
>>> log_facility = LOG_LOCAL2
>>> devices = /swift/tenants/
>>>
>>> [pipeline:main]
>>> pipeline = account-server
>>>
>>> [app:account-server]
>>> use = egg:gluster_swift_ufo#account
>>> log_name = account-server
>>> log_level = DEBUG
>>> log_requests = true
>>>
>>> [account-replicator]
>>> vm_test_mode = yes
>>>
>>> [account-auditor]
>>>
>>> [account-reaper]
>>>
>>> "
>>>
>>> "
>>>   cat /etc/swift/container-server.conf
>>> [DEFAULT]
>>> devices = /swift/tenants/
>>> mount_check = true
>>> bind_port = 6011
>>> user = root
>>> log_facility = LOG_LOCAL2
>>>
>>> [pipeline:main]
>>> pipeline = container-server
>>>
>>> [app:container-server]
>>> use = egg:gluster_swift_ufo#container
>>>
>>> [container-replicator]
>>> vm_test_mode = yes
>>>
>>> [container-updater]
>>>
>>> [container-auditor]
>>>
>>> [container-sync]
>>> "
>>>
>>> "
>>>   cat /etc/swift/object-server.conf
>>> [DEFAULT]
>>> mount_check = true
>>> bind_port = 6010
>>> user = root
>>> log_facility = LOG_LOCAL2
>>> devices = /swift/tenants/
>>>
>>> [pipeline:main]
>>> pipeline = object-server
>>>
>>> [app:object-server]
>>> use = egg:gluster_swift_ufo#object
>>>
>>> [object-replicator]
>>> vm_test_mode = yes
>>>
>>> [object-updater]
>>>
>>> [object-auditor]
>>> "
>>>
>>> "
>>> cat /etc/swift/proxy-server.conf
>>> [DEFAULT]
>>> bind_port = 8080
>>> user = root
>>> log_facility = LOG_LOCAL1
>>> log_name = swift
>>> log_level = DEBUG
>>> log_headers = True
>>>
>>> [pipeline:main]
>>> pipeline = healthcheck cache authtoken keystone proxy-server
>>>
>>> [app:proxy-server]
>>> use = egg:gluster_swift_ufo#proxy
>>> allow_account_management = true
>>> account_autocreate = true
>>>
>>> [filter:tempauth]
>>> use = egg:swift#tempauth
>>> # Here you need to add users explicitly. See the OpenStack Swift
>>> Deployment
>>> # Guide for more information. The user and user64 directives take the
>>> # following form:
>>> #     user_<account>_<username> = <key> [group] [group] [...]
>>> [storage_url]
>>> #     user64_<account_b64>_<username_b64> = <key> [group] [group]
>>> [...] [storage_url]
>>> # Where you use user64 for accounts and/or usernames that include
>>> underscores.
>>> #
>>> # NOTE (and WARNING): The account name must match the device name
>>> specified
>>> # when generating the account, container, and object build rings.
>>> #
>>> # E.g.
>>> #     user_ufo0_admin = abc123 .admin
>>>
>>> [filter:healthcheck]
>>> use = egg:swift#healthcheck
>>>
>>> [filter:cache]
>>> use = egg:swift#memcache
>>>
>>>
>>> [filter:keystone]
>>> use = egg:swift#keystoneauth
>>> #paste.filter_factory = keystone.middleware.swift_auth:filter_factory
>>> operator_roles = Member,admin,swiftoperator
>>>
>>>
>>> [filter:authtoken]
>>> paste.filter_factory = keystone.middleware.auth_token:filter_factory
>>> auth_host = keystone01.vip.my.net
>>> auth_port = 35357
>>> auth_protocol = http
>>> admin_user = swift
>>> admin_password = PASSWORD
>>> admin_tenant_name = service
>>> signing_dir = /var/cache/swift
>>> service_port = 5000
>>> service_host = keystone01.vip.my.net
>>>
>>> [filter:swiftauth]
>>> use = egg:keystone#swiftauth
>>> auth_host = keystone01.vip.my.net
>>> auth_port = 35357
>>> auth_protocol = http
>>> keystone_url = https://keystone01.vip.my.net:5000/v2.0
>>> admin_user = swift
>>> admin_password = PASSWORD
>>> admin_tenant_name = service
>>> signing_dir = /var/cache/swift
>>> keystone_swift_operator_roles = Member,admin,swiftoperator
>>> keystone_tenant_user_admin = true
>>>
>>> [filter:catch_errors]
>>> use = egg:swift#catch_errors
>>> "
>>>
>>> "
>>> cat /etc/swift/swift.conf
>>> [DEFAULT]
>>>
>>>
>>> [swift-hash]
>>> # random unique string that can never change (DO NOT LOSE)
>>> swift_hash_path_suffix = gluster
>>> #3d60c9458bb77abe
>>>
>>>
>>> # The swift-constraints section sets the basic constraints on data
>>> # saved in the swift cluster.
>>>
>>> [swift-constraints]
>>>
>>> # max_file_size is the largest "normal" object that can be saved in
>>> # the cluster. This is also the limit on the size of each segment of
>>> # a "large" object when using the large object manifest support.
>>> # This value is set in bytes. Setting it to lower than 1MiB will cause
>>> # some tests to fail. It is STRONGLY recommended to leave this value at
>>> # the default (5 * 2**30 + 2).
>>>
>>> # FIXME: Really? Gluster can handle a 2^64 sized file? And can the
>>> fronting
>>> # web service handle such a size? I think with UFO, we need to keep with
>>> the
>>> # default size from Swift and encourage users to research what size their
>>> web
>>> # services infrastructure can handle.
>>>
>>> max_file_size = 18446744073709551616
>>>
>>>
>>> # max_meta_name_length is the max number of bytes in the utf8 encoding
>>> # of the name portion of a metadata header.
>>>
>>> #max_meta_name_length = 128
>>>
>>>
>>> # max_meta_value_length is the max number of bytes in the utf8 encoding
>>> # of a metadata value
>>>
>>> #max_meta_value_length = 256
>>>
>>>
>>> # max_meta_count is the max number of metadata keys that can be stored
>>> # on a single account, container, or object
>>>
>>> #max_meta_count = 90
>>>
>>>
>>> # max_meta_overall_size is the max number of bytes in the utf8 encoding
>>> # of the metadata (keys + values)
>>>
>>> #max_meta_overall_size = 4096
>>>
>>>
>>> # max_object_name_length is the max number of bytes in the utf8 encoding
>>> of
>>> an
>>> # object name: Gluster FS can handle much longer file names, but the
>>> length
>>> # between the slashes of the URL is handled below. Remember that most web
>>> # clients can't handle anything greater than 2048, and those that do are
>>> # rather clumsy.
>>>
>>> max_object_name_length = 2048
>>>
>>> # max_object_name_component_length (GlusterFS) is the max number of bytes
>>> in
>>> # the utf8 encoding of an object name component (the part between the
>>> # slashes); this is a limit imposed by the underlying file system (for
>>> XFS
>>> it
>>> # is 255 bytes).
>>>
>>> max_object_name_component_length = 255
>>>
>>> # container_listing_limit is the default (and max) number of items
>>> # returned for a container listing request
>>>
>>> #container_listing_limit = 10000
>>>
>>>
>>> # account_listing_limit is the default (and max) number of items returned
>>> # for an account listing request
>>>
>>> #account_listing_limit = 10000
>>>
>>>
>>> # max_account_name_length is the max number of bytes in the utf8 encoding
>>> of
>>> # an account name: Gluster FS Filename limit (XFS limit?), must be the
>>> same
>>> # size as max_object_name_component_length above.
>>>
>>> max_account_name_length = 255
>>>
>>>
>>> # max_container_name_length is the max number of bytes in the utf8
>>> encoding
>>> # of a container name: Gluster FS Filename limit (XFS limit?), must be
>>> the
>>> same
>>> # size as max_object_name_component_length above.
>>>
>>> max_container_name_length = 255
>>>
>>> "
>>>
>>>
>>> The volumes
>>> "
>>>   gluster volume list
>>> cindervol
>>> unified-storage-vol
>>> a07d2f39117c4e5abdeba722cf245828
>>> bd74a005f08541b9989e392a689be2fc
>>> f6da0a8151ff43b7be10d961a20c94d6
>>> "
>>>
>>> if I run the command
>>> "
>>>   gluster-swift-gen-builders unified-storage-vol
>>> a07d2f39117c4e5abdeba722cf245828 bd74a005f08541b9989e392a689be2fc
>>> f6da0a8151ff43b7be10d961a20c94d6
>>> "
>>>
>>> because of a change in the script in this version as compaired to the
>>> version I got from
>>> http://repos.fedorapeople.org/repos/kkeithle/glusterfs/ the
>>> gluster-swift-gen-builders script only takes the first option and
>>> ignores the rest.
>>>
>>> other than the location of the config files none of the changes Ive
>>> made are functionally different than the ones mentioned in
>>>
>>> http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/
>>>
>>> The result is that the first volume named "unified-storage-vol" winds
>>> up being used for every thing regardless of the tenant, and users and
>>> see and manage each others objects regardless of what tenant they are
>>> members of.
>>> through the swift command or via horizon.
>>>
>>> In a way this is a good thing for me it simplifies thing significantly
>>> and would be fine if it just created a directory for each tenant and
>>> only allow the user to access the individual directories, not the
>>> whole gluster volume.
>>> by the way seeing every thing includes the service tenants data so
>>> unprivileged users can delete glance images without being a member of
>>> the service group.
>>>
>>>
>>>
>>>
>>> On Mon, Sep 2, 2013 at 9:58 PM, Paul Robert Marino <prmarino1 at gmail.com>
>>> wrote:
>>>
>>> Well I'll give you the full details in the morning but simply I used the
>>> stock cluster ring builder script that came with the 3.4 rpms and the old
>>> version from 3.3 took the list of volumes and would add all of them the
>>> version with 3.4 only takes the first one.
>>>
>>> Well I ran the script expecting the same behavior but instead they all
>>> used
>>> the first volume in the list.
>>>
>>> Now I knew from the docs I read that the per tenant directories in a
>>> single
>>> volume were one possible plan for 3.4 to deal with the scalding issue
>>> with a
>>> large number of tenants, so when I saw the difference in the script and
>>> that
>>> it worked I just assumed that this was done and I missed something.
>>>
>>>
>>>
>>> -- Sent from my HP Pre3
>>>
>>> ________________________________
>>> On Sep 2, 2013 20:55, Ramana Raja <rraja at redhat.com> wrote:
>>>
>>> Hi Paul,
>>>
>>> Currently, gluster-swift doesn't support the feature of multiple
>>> accounts/tenants accessing the same volume. Each tenant still needs his
>>> own
>>> gluster volume. So I'm wondering how you were able to observe the
>>> reported
>>> behaviour.
>>>
>>> How did you prepare the ringfiles for the different tenants, which use
>>> the
>>> same gluster volume? Did you change the configuration of the servers?
>>> Also,
>>> how did you access the files that you mention? It'd be helpful if you
>>> could
>>> share the commands you used to perform these actions.
>>>
>>> Thanks,
>>>
>>> Ram
>>>
>>>
>>> ----- Original Message -----
>>> From: "Vijay Bellur" <vbellur at redhat.com>
>>> To: "Paul Robert Marino" <prmarino1 at gmail.com>
>>> Cc: rhos-list at redhat.com, "Luis Pabon" <lpabon at redhat.com>, "Ramana Raja"
>>> <rraja at redhat.com>, "Chetan Risbud" <crisbud at redhat.com>
>>> Sent: Monday, September 2, 2013 4:17:51 PM
>>> Subject: Re: [rhos-list] Gluster UFO 3.4 swift Multi tenant question
>>>
>>> On 09/02/2013 01:39 AM, Paul Robert Marino wrote:
>>>
>>> I have Gluster UFO installed as a back end for swift from here
>>> http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/
>>> with RDO 3
>>>
>>> Its working well except for one thing. All of the tenants are seeing
>>> one Gluster volume which is some what nice, especially when compared
>>> to the old 3.3 behavior of creating one volume per tenant named after
>>> the tenant ID number.
>>>
>>> The problem is I expected to see is sub directory created under the
>>> volume root for each tenant but instead what in seeing is that all of
>>> the tenants can see the root of the Gluster volume. The result is that
>>> all of the tenants can access each others files and even delete them.
>>> even scarier is that the tennants can see and delete each others
>>> glance images and snapshots.
>>>
>>> Can any one suggest options to look at or documents to read to try to
>>> figure out how to modify the behavior?
>>>
>>> Adding gluster swift developers who might be able to help.
>>>
>>> -Vijay
>>>
>>>
>