[Gluster-users] [rhos-list] [gluster-swift] Gluster UFO 3.4 swift Multi tenant question

[Paul Robert Marino]

done it should have the tag GET-MOUNT_IP-FOR-GEN-BUILDERS-SCRIPT in Gerrit.


On Fri, Sep 20, 2013 at 4:11 PM, Luis Pabon <lpabon at redhat.com> wrote:
> This is great! Thank you. The only issue is that we use github as a public
> repo and we do not use github for our normal development workflow. Do you
> mind resubmitting the change using the development workflow as described
> here:https://github.com/gluster/gluster-swift/blob/master/doc/markdown/dev_guide.md.
> More information can be found at https://launchpad.net/gluster-swift
>
> - Luis
>
>
> On 09/19/2013 05:23 PM, Paul Robert Marino wrote:
>
> Thank you every one for your help especially Louis
>
> I tested the RPM and it went well every thing is working now.
>
> I did have to use the tenant ID's as the volume names. I may submit an
> update to the documentation to clarify this for people
>
> So in other words the volume names have to match the output of
>
> "
> keystone tenant-list|grep -v + | \
> grep -v -P '^\|\s+id\s+\|\s+name\s+\|\s+enabled\s+\|$' | \
> grep -v -P '^\w+:' | awk '{print $2}'
> "
>
>
> I've created an updated copy of gluster-swift-gen-builders that grabs
> the value of mount_ip from /etc/swift/fs.conf and posted it on github.
> you should see a pull request on the site for the submission. of the
> change
>
> Thank you every one for your help
>
> On Tue, Sep 17, 2013 at 4:38 PM, Paul Robert Marino <prmarino1 at gmail.com>
> wrote:
>
> Luis
> Thanks for the timely response.
>
> On Tue, Sep 17, 2013 at 1:52 PM, Luis Pabon <lpabon at redhat.com> wrote:
>
> On 09/17/2013 11:13 AM, Paul Robert Marino wrote:
>
> Luis
> well thats intresting because it was my impression that Gluster UFO
> 3.4 was based on the Grizzly version of Swift.
>
> [LP] Sorry, the gluster-ufo RPM is Essex only.
>
> [PRM] The source of my confusion was here
> http://www.gluster.org/community/documentation/index.php/Features34
> and here
> http://www.gluster.org/2013/06/glusterfs-3-4-and-swift-where-are-all-the-pieces/
> These pages on the gluster site should probably be updated to reflect
> the changes.
>
>
>
> Also I was previously unaware of this new rpm which doesnt seem to be
> in a repo any where.
>
> [LP] gluster-swift project RPMs have been submitted to Fedora and are
> currently being reviewed.
>
> [PRM] Cool if they are in the EPEL testing repo Ill look for them
> there because I would rather pull the properly EPEL signed RPMs if
> they exist just to make node deployments easier. If not Ill ask some
> of my friends offline if they can help expedite it.
>
>
> also there is a line in this new howto that is extreamly unclear
>
> "
> /usr/bin/gluster-swift-gen-builders test
> "
> in place of "test" what should go there is it the tenant ID string,
> the tenant name, or just a generic volume you can name whatever you
> want?
> in other words how should the Gluster volumes be named?
>
> [LP] We will clarify that in the quick start guide.  Thank you for pointing
> it out.  While we update the community site, please refer to the
> documentation available here http://goo.gl/bQFI8o for a usage guide.
>
> As for the tool, the format is:
> gluster-swift-gen-buildes [VOLUME] [VOLUME...]
>
> Where VOLUME is the name of the GlusterFS volume to use for object storage.
> For example
> if the following two GlusterFS volumes, volume1 and volume2, need to be
> accessed over Swift,
> then you can type the following:
>
> # gluster-swift-gen-builders volume1 volume2
>
> [PRM] That part I understood however it doesn't answer the question exactly.
>
> Correct me if I'm wrong but looking over the code briefly it looks as
> though the volume name needs to be the same as the tenant ID number
> like it did with Gluster UFO 3.3.
> so for example
> if I do a " keystone tenant-list" and a see tenant1 with an id of
> "f6da0a8151ff43b7be10d961a20c94d6" then I would need to create a
> volume named f6da0a8151ff43b7be10d961a20c94d6
>
> If I can name the volumes whatever I want or give them the same name
> as the tenant that would be great because it makes it easier for other
> SA's who are not directly working with OpenStack but may need to mount
> the volumes to comprehend, but its not urgently needed.
>
> One thing I was glad to see is that with Gluster UFO 3.3 I had to add
> mount points to /etc/fstab for each volume and manually create the
> directories for the mount points this looks to have been corrected in
> Gluster-Swift.
>
> For more information please read: http://goo.gl/gd8LkW
>
> Let us know if you have any more questions or comments.
>
> [PRM] I may fork the Github repo and add some changes that may be
> beneficial so they can be reviewed and possibly merged.
> for example it would be nice if the  gluster-swift-gen-buildes script
> used the value of the mount_ip field in /etc/swift/fs.conf instead of
> 127.0.0.1 if its defined.
> also I might make a more robust version that allows create, add,
> remove, and list options.
>
>
> Ill do testing tomorrow and let everyone know how it goes.
>
>
> - Luis
>
>
> On Tue, Sep 17, 2013 at 10:10 AM, Luis Pabon <lpabon at redhat.com> wrote:
>
> First thing I can see is that you have Essex based gluster-ufo-* which
> has
> been replaced by the gluster-swift project.  We are currently in progress
> of
> replacing the gluster-ufo-* with RPMs from the gluster-swift project in
> Fedora.
>
> Please checkout the following quickstart guide which show how to download
> the Grizzly version of gluster-swift:
>
> https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md
> .
>
> For more information please visit: https://launchpad.net/gluster-swift
>
> - Luis
>
>
> On 09/16/2013 05:02 PM, Paul Robert Marino wrote:
>
> Sorry for the delay on reporting the details. I got temporarily pulled
> off the project and dedicated to a different project which was
> considered higher priority by my employer. I'm just getting back to
> doing my normal work today.
>
> first here are the rpms I have installed
> "
>   rpm -qa |grep -P -i '(gluster|swift)'
> glusterfs-libs-3.4.0-8.el6.x86_64
> glusterfs-server-3.4.0-8.el6.x86_64
> openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
> openstack-swift-proxy-1.8.0-2.el6.noarch
> glusterfs-3.4.0-8.el6.x86_64
> glusterfs-cli-3.4.0-8.el6.x86_64
> glusterfs-geo-replication-3.4.0-8.el6.x86_64
> glusterfs-api-3.4.0-8.el6.x86_64
> openstack-swift-1.8.0-2.el6.noarch
> openstack-swift-container-1.8.0-2.el6.noarch
> openstack-swift-object-1.8.0-2.el6.noarch
> glusterfs-fuse-3.4.0-8.el6.x86_64
> glusterfs-rdma-3.4.0-8.el6.x86_64
> openstack-swift-account-1.8.0-2.el6.noarch
> glusterfs-ufo-3.4.0-8.el6.noarch
> glusterfs-vim-3.2.7-1.el6.x86_64
> python-swiftclient-1.4.0-1.el6.noarch
>
> here are some key config files note I've changed the passwords I'm
> using and hostnames
> "
>   cat /etc/swift/account-server.conf
> [DEFAULT]
> mount_check = true
> bind_port = 6012
> user = root
> log_facility = LOG_LOCAL2
> devices = /swift/tenants/
>
> [pipeline:main]
> pipeline = account-server
>
> [app:account-server]
> use = egg:gluster_swift_ufo#account
> log_name = account-server
> log_level = DEBUG
> log_requests = true
>
> [account-replicator]
> vm_test_mode = yes
>
> [account-auditor]
>
> [account-reaper]
>
> "
>
> "
>   cat /etc/swift/container-server.conf
> [DEFAULT]
> devices = /swift/tenants/
> mount_check = true
> bind_port = 6011
> user = root
> log_facility = LOG_LOCAL2
>
> [pipeline:main]
> pipeline = container-server
>
> [app:container-server]
> use = egg:gluster_swift_ufo#container
>
> [container-replicator]
> vm_test_mode = yes
>
> [container-updater]
>
> [container-auditor]
>
> [container-sync]
> "
>
> "
>   cat /etc/swift/object-server.conf
> [DEFAULT]
> mount_check = true
> bind_port = 6010
> user = root
> log_facility = LOG_LOCAL2
> devices = /swift/tenants/
>
> [pipeline:main]
> pipeline = object-server
>
> [app:object-server]
> use = egg:gluster_swift_ufo#object
>
> [object-replicator]
> vm_test_mode = yes
>
> [object-updater]
>
> [object-auditor]
> "
>
> "
> cat /etc/swift/proxy-server.conf
> [DEFAULT]
> bind_port = 8080
> user = root
> log_facility = LOG_LOCAL1
> log_name = swift
> log_level = DEBUG
> log_headers = True
>
> [pipeline:main]
> pipeline = healthcheck cache authtoken keystone proxy-server
>
> [app:proxy-server]
> use = egg:gluster_swift_ufo#proxy
> allow_account_management = true
> account_autocreate = true
>
> [filter:tempauth]
> use = egg:swift#tempauth
> # Here you need to add users explicitly. See the OpenStack Swift
> Deployment
> # Guide for more information. The user and user64 directives take the
> # following form:
> #     user_<account>_<username> = <key> [group] [group] [...]
> [storage_url]
> #     user64_<account_b64>_<username_b64> = <key> [group] [group]
> [...] [storage_url]
> # Where you use user64 for accounts and/or usernames that include
> underscores.
> #
> # NOTE (and WARNING): The account name must match the device name
> specified
> # when generating the account, container, and object build rings.
> #
> # E.g.
> #     user_ufo0_admin = abc123 .admin
>
> [filter:healthcheck]
> use = egg:swift#healthcheck
>
> [filter:cache]
> use = egg:swift#memcache
>
>
> [filter:keystone]
> use = egg:swift#keystoneauth
> #paste.filter_factory = keystone.middleware.swift_auth:filter_factory
> operator_roles = Member,admin,swiftoperator
>
>
> [filter:authtoken]
> paste.filter_factory = keystone.middleware.auth_token:filter_factory
> auth_host = keystone01.vip.my.net
> auth_port = 35357
> auth_protocol = http
> admin_user = swift
> admin_password = PASSWORD
> admin_tenant_name = service
> signing_dir = /var/cache/swift
> service_port = 5000
> service_host = keystone01.vip.my.net
>
> [filter:swiftauth]
> use = egg:keystone#swiftauth
> auth_host = keystone01.vip.my.net
> auth_port = 35357
> auth_protocol = http
> keystone_url = https://keystone01.vip.my.net:5000/v2.0
> admin_user = swift
> admin_password = PASSWORD
> admin_tenant_name = service
> signing_dir = /var/cache/swift
> keystone_swift_operator_roles = Member,admin,swiftoperator
> keystone_tenant_user_admin = true
>
> [filter:catch_errors]
> use = egg:swift#catch_errors
> "
>
> "
> cat /etc/swift/swift.conf
> [DEFAULT]
>
>
> [swift-hash]
> # random unique string that can never change (DO NOT LOSE)
> swift_hash_path_suffix = gluster
> #3d60c9458bb77abe
>
>
> # The swift-constraints section sets the basic constraints on data
> # saved in the swift cluster.
>
> [swift-constraints]
>
> # max_file_size is the largest "normal" object that can be saved in
> # the cluster. This is also the limit on the size of each segment of
> # a "large" object when using the large object manifest support.
> # This value is set in bytes. Setting it to lower than 1MiB will cause
> # some tests to fail. It is STRONGLY recommended to leave this value at
> # the default (5 * 2**30 + 2).
>
> # FIXME: Really? Gluster can handle a 2^64 sized file? And can the
> fronting
> # web service handle such a size? I think with UFO, we need to keep with
> the
> # default size from Swift and encourage users to research what size their
> web
> # services infrastructure can handle.
>
> max_file_size = 18446744073709551616
>
>
> # max_meta_name_length is the max number of bytes in the utf8 encoding
> # of the name portion of a metadata header.
>
> #max_meta_name_length = 128
>
>
> # max_meta_value_length is the max number of bytes in the utf8 encoding
> # of a metadata value
>
> #max_meta_value_length = 256
>
>
> # max_meta_count is the max number of metadata keys that can be stored
> # on a single account, container, or object
>
> #max_meta_count = 90
>
>
> # max_meta_overall_size is the max number of bytes in the utf8 encoding
> # of the metadata (keys + values)
>
> #max_meta_overall_size = 4096
>
>
> # max_object_name_length is the max number of bytes in the utf8 encoding
> of
> an
> # object name: Gluster FS can handle much longer file names, but the
> length
> # between the slashes of the URL is handled below. Remember that most web
> # clients can't handle anything greater than 2048, and those that do are
> # rather clumsy.
>
> max_object_name_length = 2048
>
> # max_object_name_component_length (GlusterFS) is the max number of bytes
> in
> # the utf8 encoding of an object name component (the part between the
> # slashes); this is a limit imposed by the underlying file system (for
> XFS
> it
> # is 255 bytes).
>
> max_object_name_component_length = 255
>
> # container_listing_limit is the default (and max) number of items
> # returned for a container listing request
>
> #container_listing_limit = 10000
>
>
> # account_listing_limit is the default (and max) number of items returned
> # for an account listing request
>
> #account_listing_limit = 10000
>
>
> # max_account_name_length is the max number of bytes in the utf8 encoding
> of
> # an account name: Gluster FS Filename limit (XFS limit?), must be the
> same
> # size as max_object_name_component_length above.
>
> max_account_name_length = 255
>
>
> # max_container_name_length is the max number of bytes in the utf8
> encoding
> # of a container name: Gluster FS Filename limit (XFS limit?), must be
> the
> same
> # size as max_object_name_component_length above.
>
> max_container_name_length = 255
>
> "
>
>
> The volumes
> "
>   gluster volume list
> cindervol
> unified-storage-vol
> a07d2f39117c4e5abdeba722cf245828
> bd74a005f08541b9989e392a689be2fc
> f6da0a8151ff43b7be10d961a20c94d6
> "
>
> if I run the command
> "
>   gluster-swift-gen-builders unified-storage-vol
> a07d2f39117c4e5abdeba722cf245828 bd74a005f08541b9989e392a689be2fc
> f6da0a8151ff43b7be10d961a20c94d6
> "
>
> because of a change in the script in this version as compaired to the
> version I got from
> http://repos.fedorapeople.org/repos/kkeithle/glusterfs/ the
> gluster-swift-gen-builders script only takes the first option and
> ignores the rest.
>
> other than the location of the config files none of the changes Ive
> made are functionally different than the ones mentioned in
>
> http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/
>
> The result is that the first volume named "unified-storage-vol" winds
> up being used for every thing regardless of the tenant, and users and
> see and manage each others objects regardless of what tenant they are
> members of.
> through the swift command or via horizon.
>
> In a way this is a good thing for me it simplifies thing significantly
> and would be fine if it just created a directory for each tenant and
> only allow the user to access the individual directories, not the
> whole gluster volume.
> by the way seeing every thing includes the service tenants data so
> unprivileged users can delete glance images without being a member of
> the service group.
>
>
>
>
> On Mon, Sep 2, 2013 at 9:58 PM, Paul Robert Marino <prmarino1 at gmail.com>
> wrote:
>
> Well I'll give you the full details in the morning but simply I used the
> stock cluster ring builder script that came with the 3.4 rpms and the old
> version from 3.3 took the list of volumes and would add all of them the
> version with 3.4 only takes the first one.
>
> Well I ran the script expecting the same behavior but instead they all
> used
> the first volume in the list.
>
> Now I knew from the docs I read that the per tenant directories in a
> single
> volume were one possible plan for 3.4 to deal with the scalding issue
> with a
> large number of tenants, so when I saw the difference in the script and
> that
> it worked I just assumed that this was done and I missed something.
>
>
>
> -- Sent from my HP Pre3
>
> ________________________________
> On Sep 2, 2013 20:55, Ramana Raja <rraja at redhat.com> wrote:
>
> Hi Paul,
>
> Currently, gluster-swift doesn't support the feature of multiple
> accounts/tenants accessing the same volume. Each tenant still needs his
> own
> gluster volume. So I'm wondering how you were able to observe the
> reported
> behaviour.
>
> How did you prepare the ringfiles for the different tenants, which use
> the
> same gluster volume? Did you change the configuration of the servers?
> Also,
> how did you access the files that you mention? It'd be helpful if you
> could
> share the commands you used to perform these actions.
>
> Thanks,
>
> Ram
>
>
> ----- Original Message -----
> From: "Vijay Bellur" <vbellur at redhat.com>
> To: "Paul Robert Marino" <prmarino1 at gmail.com>
> Cc: rhos-list at redhat.com, "Luis Pabon" <lpabon at redhat.com>, "Ramana Raja"
> <rraja at redhat.com>, "Chetan Risbud" <crisbud at redhat.com>
> Sent: Monday, September 2, 2013 4:17:51 PM
> Subject: Re: [rhos-list] Gluster UFO 3.4 swift Multi tenant question
>
> On 09/02/2013 01:39 AM, Paul Robert Marino wrote:
>
> I have Gluster UFO installed as a back end for swift from here
> http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/
> with RDO 3
>
> Its working well except for one thing. All of the tenants are seeing
> one Gluster volume which is some what nice, especially when compared
> to the old 3.3 behavior of creating one volume per tenant named after
> the tenant ID number.
>
> The problem is I expected to see is sub directory created under the
> volume root for each tenant but instead what in seeing is that all of
> the tenants can see the root of the Gluster volume. The result is that
> all of the tenants can access each others files and even delete them.
> even scarier is that the tennants can see and delete each others
> glance images and snapshots.
>
> Can any one suggest options to look at or documents to read to try to
> figure out how to modify the behavior?
>
> Adding gluster swift developers who might be able to help.
>
> -Vijay
>
>
>