[Gluster-users] [rhos-list] [gluster-swift] Gluster UFO 3.4 swift Multi tenant question

[Paul Robert Marino]

Thank you every one for your help especially Louis

I tested the RPM and it went well every thing is working now.

I did have to use the tenant ID's as the volume names. I may submit an
update to the documentation to clarify this for people

So in other words the volume names have to match the output of

"
keystone tenant-list|grep -v + | \
grep -v -P '^\|\s+id\s+\|\s+name\s+\|\s+enabled\s+\|$' | \
grep -v -P '^\w+:' | awk '{print $2}'
"


I've created an updated copy of gluster-swift-gen-builders that grabs
the value of mount_ip from /etc/swift/fs.conf and posted it on github.
you should see a pull request on the site for the submission. of the
change

Thank you every one for your help

On Tue, Sep 17, 2013 at 4:38 PM, Paul Robert Marino <prmarino1 at gmail.com> wrote:
> Luis
> Thanks for the timely response.
>
> On Tue, Sep 17, 2013 at 1:52 PM, Luis Pabon <lpabon at redhat.com> wrote:
>>
>> On 09/17/2013 11:13 AM, Paul Robert Marino wrote:
>>>
>>> Luis
>>> well thats intresting because it was my impression that Gluster UFO
>>> 3.4 was based on the Grizzly version of Swift.
>>
>> [LP] Sorry, the gluster-ufo RPM is Essex only.
>
> [PRM] The source of my confusion was here
> http://www.gluster.org/community/documentation/index.php/Features34
> and here http://www.gluster.org/2013/06/glusterfs-3-4-and-swift-where-are-all-the-pieces/
> These pages on the gluster site should probably be updated to reflect
> the changes.
>
>
>>
>>
>>> Also I was previously unaware of this new rpm which doesnt seem to be
>>> in a repo any where.
>>
>> [LP] gluster-swift project RPMs have been submitted to Fedora and are
>> currently being reviewed.
>
> [PRM] Cool if they are in the EPEL testing repo Ill look for them
> there because I would rather pull the properly EPEL signed RPMs if
> they exist just to make node deployments easier. If not Ill ask some
> of my friends offline if they can help expedite it.
>
>>
>>
>>> also there is a line in this new howto that is extreamly unclear
>>>
>>> "
>>> /usr/bin/gluster-swift-gen-builders test
>>> "
>>> in place of "test" what should go there is it the tenant ID string,
>>> the tenant name, or just a generic volume you can name whatever you
>>> want?
>>> in other words how should the Gluster volumes be named?
>>
>> [LP] We will clarify that in the quick start guide.  Thank you for pointing
>> it out.  While we update the community site, please refer to the
>> documentation available here http://goo.gl/bQFI8o for a usage guide.
>>
>> As for the tool, the format is:
>> gluster-swift-gen-buildes [VOLUME] [VOLUME...]
>>
>> Where VOLUME is the name of the GlusterFS volume to use for object storage.
>> For example
>> if the following two GlusterFS volumes, volume1 and volume2, need to be
>> accessed over Swift,
>> then you can type the following:
>>
>> # gluster-swift-gen-builders volume1 volume2
>
> [PRM] That part I understood however it doesn't answer the question exactly.
>
> Correct me if I'm wrong but looking over the code briefly it looks as
> though the volume name needs to be the same as the tenant ID number
> like it did with Gluster UFO 3.3.
> so for example
> if I do a " keystone tenant-list" and a see tenant1 with an id of
> "f6da0a8151ff43b7be10d961a20c94d6" then I would need to create a
> volume named f6da0a8151ff43b7be10d961a20c94d6
>
> If I can name the volumes whatever I want or give them the same name
> as the tenant that would be great because it makes it easier for other
> SA's who are not directly working with OpenStack but may need to mount
> the volumes to comprehend, but its not urgently needed.
>
> One thing I was glad to see is that with Gluster UFO 3.3 I had to add
> mount points to /etc/fstab for each volume and manually create the
> directories for the mount points this looks to have been corrected in
> Gluster-Swift.
>
>>
>> For more information please read: http://goo.gl/gd8LkW
>>
>> Let us know if you have any more questions or comments.
>
> [PRM] I may fork the Github repo and add some changes that may be
> beneficial so they can be reviewed and possibly merged.
> for example it would be nice if the  gluster-swift-gen-buildes script
> used the value of the mount_ip field in /etc/swift/fs.conf instead of
> 127.0.0.1 if its defined.
> also I might make a more robust version that allows create, add,
> remove, and list options.
>
>
> Ill do testing tomorrow and let everyone know how it goes.
>
>
>>
>> - Luis
>>
>>>
>>>
>>> On Tue, Sep 17, 2013 at 10:10 AM, Luis Pabon <lpabon at redhat.com> wrote:
>>>>
>>>> First thing I can see is that you have Essex based gluster-ufo-* which
>>>> has
>>>> been replaced by the gluster-swift project.  We are currently in progress
>>>> of
>>>> replacing the gluster-ufo-* with RPMs from the gluster-swift project in
>>>> Fedora.
>>>>
>>>> Please checkout the following quickstart guide which show how to download
>>>> the Grizzly version of gluster-swift:
>>>>
>>>> https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md
>>>> .
>>>>
>>>> For more information please visit: https://launchpad.net/gluster-swift
>>>>
>>>> - Luis
>>>>
>>>>
>>>> On 09/16/2013 05:02 PM, Paul Robert Marino wrote:
>>>>
>>>> Sorry for the delay on reporting the details. I got temporarily pulled
>>>> off the project and dedicated to a different project which was
>>>> considered higher priority by my employer. I'm just getting back to
>>>> doing my normal work today.
>>>>
>>>> first here are the rpms I have installed
>>>> "
>>>>   rpm -qa |grep -P -i '(gluster|swift)'
>>>> glusterfs-libs-3.4.0-8.el6.x86_64
>>>> glusterfs-server-3.4.0-8.el6.x86_64
>>>> openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
>>>> openstack-swift-proxy-1.8.0-2.el6.noarch
>>>> glusterfs-3.4.0-8.el6.x86_64
>>>> glusterfs-cli-3.4.0-8.el6.x86_64
>>>> glusterfs-geo-replication-3.4.0-8.el6.x86_64
>>>> glusterfs-api-3.4.0-8.el6.x86_64
>>>> openstack-swift-1.8.0-2.el6.noarch
>>>> openstack-swift-container-1.8.0-2.el6.noarch
>>>> openstack-swift-object-1.8.0-2.el6.noarch
>>>> glusterfs-fuse-3.4.0-8.el6.x86_64
>>>> glusterfs-rdma-3.4.0-8.el6.x86_64
>>>> openstack-swift-account-1.8.0-2.el6.noarch
>>>> glusterfs-ufo-3.4.0-8.el6.noarch
>>>> glusterfs-vim-3.2.7-1.el6.x86_64
>>>> python-swiftclient-1.4.0-1.el6.noarch
>>>>
>>>> here are some key config files note I've changed the passwords I'm
>>>> using and hostnames
>>>> "
>>>>   cat /etc/swift/account-server.conf
>>>> [DEFAULT]
>>>> mount_check = true
>>>> bind_port = 6012
>>>> user = root
>>>> log_facility = LOG_LOCAL2
>>>> devices = /swift/tenants/
>>>>
>>>> [pipeline:main]
>>>> pipeline = account-server
>>>>
>>>> [app:account-server]
>>>> use = egg:gluster_swift_ufo#account
>>>> log_name = account-server
>>>> log_level = DEBUG
>>>> log_requests = true
>>>>
>>>> [account-replicator]
>>>> vm_test_mode = yes
>>>>
>>>> [account-auditor]
>>>>
>>>> [account-reaper]
>>>>
>>>> "
>>>>
>>>> "
>>>>   cat /etc/swift/container-server.conf
>>>> [DEFAULT]
>>>> devices = /swift/tenants/
>>>> mount_check = true
>>>> bind_port = 6011
>>>> user = root
>>>> log_facility = LOG_LOCAL2
>>>>
>>>> [pipeline:main]
>>>> pipeline = container-server
>>>>
>>>> [app:container-server]
>>>> use = egg:gluster_swift_ufo#container
>>>>
>>>> [container-replicator]
>>>> vm_test_mode = yes
>>>>
>>>> [container-updater]
>>>>
>>>> [container-auditor]
>>>>
>>>> [container-sync]
>>>> "
>>>>
>>>> "
>>>>   cat /etc/swift/object-server.conf
>>>> [DEFAULT]
>>>> mount_check = true
>>>> bind_port = 6010
>>>> user = root
>>>> log_facility = LOG_LOCAL2
>>>> devices = /swift/tenants/
>>>>
>>>> [pipeline:main]
>>>> pipeline = object-server
>>>>
>>>> [app:object-server]
>>>> use = egg:gluster_swift_ufo#object
>>>>
>>>> [object-replicator]
>>>> vm_test_mode = yes
>>>>
>>>> [object-updater]
>>>>
>>>> [object-auditor]
>>>> "
>>>>
>>>> "
>>>> cat /etc/swift/proxy-server.conf
>>>> [DEFAULT]
>>>> bind_port = 8080
>>>> user = root
>>>> log_facility = LOG_LOCAL1
>>>> log_name = swift
>>>> log_level = DEBUG
>>>> log_headers = True
>>>>
>>>> [pipeline:main]
>>>> pipeline = healthcheck cache authtoken keystone proxy-server
>>>>
>>>> [app:proxy-server]
>>>> use = egg:gluster_swift_ufo#proxy
>>>> allow_account_management = true
>>>> account_autocreate = true
>>>>
>>>> [filter:tempauth]
>>>> use = egg:swift#tempauth
>>>> # Here you need to add users explicitly. See the OpenStack Swift
>>>> Deployment
>>>> # Guide for more information. The user and user64 directives take the
>>>> # following form:
>>>> #     user_<account>_<username> = <key> [group] [group] [...]
>>>> [storage_url]
>>>> #     user64_<account_b64>_<username_b64> = <key> [group] [group]
>>>> [...] [storage_url]
>>>> # Where you use user64 for accounts and/or usernames that include
>>>> underscores.
>>>> #
>>>> # NOTE (and WARNING): The account name must match the device name
>>>> specified
>>>> # when generating the account, container, and object build rings.
>>>> #
>>>> # E.g.
>>>> #     user_ufo0_admin = abc123 .admin
>>>>
>>>> [filter:healthcheck]
>>>> use = egg:swift#healthcheck
>>>>
>>>> [filter:cache]
>>>> use = egg:swift#memcache
>>>>
>>>>
>>>> [filter:keystone]
>>>> use = egg:swift#keystoneauth
>>>> #paste.filter_factory = keystone.middleware.swift_auth:filter_factory
>>>> operator_roles = Member,admin,swiftoperator
>>>>
>>>>
>>>> [filter:authtoken]
>>>> paste.filter_factory = keystone.middleware.auth_token:filter_factory
>>>> auth_host = keystone01.vip.my.net
>>>> auth_port = 35357
>>>> auth_protocol = http
>>>> admin_user = swift
>>>> admin_password = PASSWORD
>>>> admin_tenant_name = service
>>>> signing_dir = /var/cache/swift
>>>> service_port = 5000
>>>> service_host = keystone01.vip.my.net
>>>>
>>>> [filter:swiftauth]
>>>> use = egg:keystone#swiftauth
>>>> auth_host = keystone01.vip.my.net
>>>> auth_port = 35357
>>>> auth_protocol = http
>>>> keystone_url = https://keystone01.vip.my.net:5000/v2.0
>>>> admin_user = swift
>>>> admin_password = PASSWORD
>>>> admin_tenant_name = service
>>>> signing_dir = /var/cache/swift
>>>> keystone_swift_operator_roles = Member,admin,swiftoperator
>>>> keystone_tenant_user_admin = true
>>>>
>>>> [filter:catch_errors]
>>>> use = egg:swift#catch_errors
>>>> "
>>>>
>>>> "
>>>> cat /etc/swift/swift.conf
>>>> [DEFAULT]
>>>>
>>>>
>>>> [swift-hash]
>>>> # random unique string that can never change (DO NOT LOSE)
>>>> swift_hash_path_suffix = gluster
>>>> #3d60c9458bb77abe
>>>>
>>>>
>>>> # The swift-constraints section sets the basic constraints on data
>>>> # saved in the swift cluster.
>>>>
>>>> [swift-constraints]
>>>>
>>>> # max_file_size is the largest "normal" object that can be saved in
>>>> # the cluster. This is also the limit on the size of each segment of
>>>> # a "large" object when using the large object manifest support.
>>>> # This value is set in bytes. Setting it to lower than 1MiB will cause
>>>> # some tests to fail. It is STRONGLY recommended to leave this value at
>>>> # the default (5 * 2**30 + 2).
>>>>
>>>> # FIXME: Really? Gluster can handle a 2^64 sized file? And can the
>>>> fronting
>>>> # web service handle such a size? I think with UFO, we need to keep with
>>>> the
>>>> # default size from Swift and encourage users to research what size their
>>>> web
>>>> # services infrastructure can handle.
>>>>
>>>> max_file_size = 18446744073709551616
>>>>
>>>>
>>>> # max_meta_name_length is the max number of bytes in the utf8 encoding
>>>> # of the name portion of a metadata header.
>>>>
>>>> #max_meta_name_length = 128
>>>>
>>>>
>>>> # max_meta_value_length is the max number of bytes in the utf8 encoding
>>>> # of a metadata value
>>>>
>>>> #max_meta_value_length = 256
>>>>
>>>>
>>>> # max_meta_count is the max number of metadata keys that can be stored
>>>> # on a single account, container, or object
>>>>
>>>> #max_meta_count = 90
>>>>
>>>>
>>>> # max_meta_overall_size is the max number of bytes in the utf8 encoding
>>>> # of the metadata (keys + values)
>>>>
>>>> #max_meta_overall_size = 4096
>>>>
>>>>
>>>> # max_object_name_length is the max number of bytes in the utf8 encoding
>>>> of
>>>> an
>>>> # object name: Gluster FS can handle much longer file names, but the
>>>> length
>>>> # between the slashes of the URL is handled below. Remember that most web
>>>> # clients can't handle anything greater than 2048, and those that do are
>>>> # rather clumsy.
>>>>
>>>> max_object_name_length = 2048
>>>>
>>>> # max_object_name_component_length (GlusterFS) is the max number of bytes
>>>> in
>>>> # the utf8 encoding of an object name component (the part between the
>>>> # slashes); this is a limit imposed by the underlying file system (for
>>>> XFS
>>>> it
>>>> # is 255 bytes).
>>>>
>>>> max_object_name_component_length = 255
>>>>
>>>> # container_listing_limit is the default (and max) number of items
>>>> # returned for a container listing request
>>>>
>>>> #container_listing_limit = 10000
>>>>
>>>>
>>>> # account_listing_limit is the default (and max) number of items returned
>>>> # for an account listing request
>>>>
>>>> #account_listing_limit = 10000
>>>>
>>>>
>>>> # max_account_name_length is the max number of bytes in the utf8 encoding
>>>> of
>>>> # an account name: Gluster FS Filename limit (XFS limit?), must be the
>>>> same
>>>> # size as max_object_name_component_length above.
>>>>
>>>> max_account_name_length = 255
>>>>
>>>>
>>>> # max_container_name_length is the max number of bytes in the utf8
>>>> encoding
>>>> # of a container name: Gluster FS Filename limit (XFS limit?), must be
>>>> the
>>>> same
>>>> # size as max_object_name_component_length above.
>>>>
>>>> max_container_name_length = 255
>>>>
>>>> "
>>>>
>>>>
>>>> The volumes
>>>> "
>>>>   gluster volume list
>>>> cindervol
>>>> unified-storage-vol
>>>> a07d2f39117c4e5abdeba722cf245828
>>>> bd74a005f08541b9989e392a689be2fc
>>>> f6da0a8151ff43b7be10d961a20c94d6
>>>> "
>>>>
>>>> if I run the command
>>>> "
>>>>   gluster-swift-gen-builders unified-storage-vol
>>>> a07d2f39117c4e5abdeba722cf245828 bd74a005f08541b9989e392a689be2fc
>>>> f6da0a8151ff43b7be10d961a20c94d6
>>>> "
>>>>
>>>> because of a change in the script in this version as compaired to the
>>>> version I got from
>>>> http://repos.fedorapeople.org/repos/kkeithle/glusterfs/ the
>>>> gluster-swift-gen-builders script only takes the first option and
>>>> ignores the rest.
>>>>
>>>> other than the location of the config files none of the changes Ive
>>>> made are functionally different than the ones mentioned in
>>>>
>>>> http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/
>>>>
>>>> The result is that the first volume named "unified-storage-vol" winds
>>>> up being used for every thing regardless of the tenant, and users and
>>>> see and manage each others objects regardless of what tenant they are
>>>> members of.
>>>> through the swift command or via horizon.
>>>>
>>>> In a way this is a good thing for me it simplifies thing significantly
>>>> and would be fine if it just created a directory for each tenant and
>>>> only allow the user to access the individual directories, not the
>>>> whole gluster volume.
>>>> by the way seeing every thing includes the service tenants data so
>>>> unprivileged users can delete glance images without being a member of
>>>> the service group.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Sep 2, 2013 at 9:58 PM, Paul Robert Marino <prmarino1 at gmail.com>
>>>> wrote:
>>>>
>>>> Well I'll give you the full details in the morning but simply I used the
>>>> stock cluster ring builder script that came with the 3.4 rpms and the old
>>>> version from 3.3 took the list of volumes and would add all of them the
>>>> version with 3.4 only takes the first one.
>>>>
>>>> Well I ran the script expecting the same behavior but instead they all
>>>> used
>>>> the first volume in the list.
>>>>
>>>> Now I knew from the docs I read that the per tenant directories in a
>>>> single
>>>> volume were one possible plan for 3.4 to deal with the scalding issue
>>>> with a
>>>> large number of tenants, so when I saw the difference in the script and
>>>> that
>>>> it worked I just assumed that this was done and I missed something.
>>>>
>>>>
>>>>
>>>> -- Sent from my HP Pre3
>>>>
>>>> ________________________________
>>>> On Sep 2, 2013 20:55, Ramana Raja <rraja at redhat.com> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> Currently, gluster-swift doesn't support the feature of multiple
>>>> accounts/tenants accessing the same volume. Each tenant still needs his
>>>> own
>>>> gluster volume. So I'm wondering how you were able to observe the
>>>> reported
>>>> behaviour.
>>>>
>>>> How did you prepare the ringfiles for the different tenants, which use
>>>> the
>>>> same gluster volume? Did you change the configuration of the servers?
>>>> Also,
>>>> how did you access the files that you mention? It'd be helpful if you
>>>> could
>>>> share the commands you used to perform these actions.
>>>>
>>>> Thanks,
>>>>
>>>> Ram
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Vijay Bellur" <vbellur at redhat.com>
>>>> To: "Paul Robert Marino" <prmarino1 at gmail.com>
>>>> Cc: rhos-list at redhat.com, "Luis Pabon" <lpabon at redhat.com>, "Ramana Raja"
>>>> <rraja at redhat.com>, "Chetan Risbud" <crisbud at redhat.com>
>>>> Sent: Monday, September 2, 2013 4:17:51 PM
>>>> Subject: Re: [rhos-list] Gluster UFO 3.4 swift Multi tenant question
>>>>
>>>> On 09/02/2013 01:39 AM, Paul Robert Marino wrote:
>>>>
>>>> I have Gluster UFO installed as a back end for swift from here
>>>> http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/
>>>> with RDO 3
>>>>
>>>> Its working well except for one thing. All of the tenants are seeing
>>>> one Gluster volume which is some what nice, especially when compared
>>>> to the old 3.3 behavior of creating one volume per tenant named after
>>>> the tenant ID number.
>>>>
>>>> The problem is I expected to see is sub directory created under the
>>>> volume root for each tenant but instead what in seeing is that all of
>>>> the tenants can see the root of the Gluster volume. The result is that
>>>> all of the tenants can access each others files and even delete them.
>>>> even scarier is that the tennants can see and delete each others
>>>> glance images and snapshots.
>>>>
>>>> Can any one suggest options to look at or documents to read to try to
>>>> figure out how to modify the behavior?
>>>>
>>>> Adding gluster swift developers who might be able to help.
>>>>
>>>> -Vijay
>>>>
>>>>
>>