[Gluster-devel] Gluster CLI for setting options for glusterd.vol

Thu Jun 27 09:35:08 UTC 2013

On Thu, Jun 27, 2013 at 2:30 PM, Deepak C Shetty <
deepakcs at linux.vnet.ibm.com> wrote:

>  On 06/27/2013 12:34 PM, Kanagaraj M wrote:
>
>
>
> On Thu, Jun 27, 2013 at 11:20 AM, Deepak C Shetty <
> deepakcs at linux.vnet.ibm.com> wrote:
>
>> On 06/24/2013 06:21 PM, Vijay Bellur wrote:
>>
>>> On 06/20/2013 07:28 PM, M. Mohan Kumar wrote:
>>>
>>>> Vijay Bellur <vbellur at redhat.com> writes:
>>>>
>>>>  On 06/19/2013 09:51 PM, M. Mohan Kumar wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> When qemu is invoked by a non-root user with -drive
>>>>>> file=gluster://server/volname/imagename option, unprivileged port is
>>>>>> used for gluster rpc and by default glusterd and gluster brick process
>>>>>> deny the request if the request is from a unprivileged port. The
>>>>>> option
>>>>>> "rpc-auth-allow-insecure" needs to be enabled in glusterd.vol so that
>>>>>> non privileged ports can be used to access Gluster volumes.
>>>>>>
>>>>>> In a typical environment VDSM might want to enable
>>>>>> rpc-auth-allow-insecure
>>>>>> option and the administrator has to edit the glusterd.vol manually and
>>>>>> restart glusterd process.
>>>>>>
>>>>>> CLI options available to enable volume specific options to work with
>>>>>> unprivileged ports by using gluster volume set <volname> <option>
>>>>>> <value>. For example per volume server.allow-insecure option can be
>>>>>> enabled so that unprivileged users can mount a GlusterFS volume.
>>>>>>
>>>>>> But as of now there is no CLI option available to set glusterd.vol
>>>>>> options. How about adding a gluster CLI set option to configure
>>>>>> glusterd.vol options? Can following CLI command line 'gluster volume
>>>>>> set
>>>>>> all <glusterd.option> <value>" be used for setting glusterd options?
>>>>>> IIUC "all" is a reserved volume name and we can use this reserved name
>>>>>> for setting glusterd option.
>>>>>>
>>>>>
>>>>> 'volume set all' is mostly used for options that are applicable to all
>>>>> volumes. Since glusterd options are beyond the scope of a volume, tying
>>>>> them to the peer entity might be a good idea. We can introduce 'peer
>>>>> set
>>>>> all <key> <value>' which sets a particular option on all peers.
>>>>>
>>>>>
>>>> You mean by 'gluster peer set all rpc-auth-allow-insecure on' will
>>>> enable insecured port access to all glusterds in the peer environment?
>>>>
>>>
>>>
>>> Yes.
>>>
>>
>> This still doesn't help the VDSM usecase, when VDSM host ( aka hypervisor
>> host ) is not part of gluster peer.
>> One of the goal here was to provide a cli way to modify glusterd options
>> so that VDSM can exploit it, when Gluster volume is used as a storage
>> domain, and VDSM needs rpc-auth-allow-insecure to be ON as VMs accessing
>> Gluster volume natively via libgfapi will be running as non-root.
>>
>> On the same lines.. how does oVirt Engine 'Volumes' GUI manage Gluster
>> volumes.... when the oVirt host is not part of the Gluster peer ? Just
>> wondering....
>>
>>
>  Why oVirt Engine host needs to be a gluster peer to be able to
> communicate with gluster(through vdsm)? As super-vdsm is running as root in
> the gluster node and it should be able to communicate with underlying
> gluster cli and can respond to oVirt engine.
>
>
> Right, I forgot the fact that gluster clis are executed thru super-vdsm,
> so non-priv / insecure issue won't be present. But the VDSM usecase still
> is a issue... when VDSM host (hypervisor node) is not a gluster peer, we
> need a way to set the glusterd option... currently one has to do that
> manually, which is not a good choice.
>
>
>
>  If not, are you talking about the scenario where gluster node doesn't
> have vdsm installed also the storage cluster is not managed through oVirt.
>
>
> So, as i see it, there are couple of scenarios.... from VDSM perspective...
>
> For the sake of discussion, lets say *VDSM (aka hypervisor host) is hostA
> & Gluster node/host is hostB**.*
>
> *Case 1:* hostA is not a gluster peer. hostB is a gluster peer
> (obviously) , part of Gluster volume which is managed by oVirt Engine.
>
>     hostA will access Gluster volume as non-root (as part of VM running on
> Gluster storage domain, using libgfapi). Since hostA is not a gluster peer
> but since the Gluster volume is managed by oVirt, it should be possible for
> the engine to first execute 'gluster peer set all rpc-auth-allow-insecure
> on the gluster volume, then ask VDSM to create the gluster storage domain.
> From the perspective of "User friendliness", its better to have User click
> on something like "Allow insecure access" or "Make this gluster volume work
> with VDSM Gluster storage domain" which will execute 'gluster peer set all
> ..insecure.. ' option, so that User is aware of enabling the insecure
> option, instead of engine doing it w/o User's knowledge as part of gluster
> storage domain flow.
>
> This should work and for this to happen seamless, the CLI support for
> setting glusterd is needed, so this usecase is covered.
>
> *Case 2:* hostA is not a gluster peer. hostB is a gluster peer
> (obviously), but the Gluster volume is not managed by oVirt.
>
>     The same as Case 1 applies, with the exception that, since the Gluster
> volume is not managed by oVirt, engine cannot do 'gluster peer set all
> rpc-auth-allow-insecure'. Ther are again 2 sub-cases here ...
>
>     *2a)*  Have the user import the gluster volume into oVirt, post which
> it becomes like Case 1... problem solved.
>
>     *2b)* For some reason, user doesn't want to import the gluster volume
> into oVirt.. in which case, the only way is for someone to _manually_ do
> 'gluster peer set all rpc-auth-allow-insecure' from one of the gluster
> peers, which is not a good thing since it involved manual intervention. One
> can argue that hostA can use --remote-host option and do 'gluster peer set
> all...', which will work today, but that would be insecure way of doing it,
> since you are allowing a non-peer to set insecure option for all peers.
> Also Mohan's another patch (http://review.gluster.org/#/c/5092/) adds
> support for IP based access control, which also disables any 'set' kind of
> actions using --remote-host, so --remote-host anyways won't be possible in
> VDSM once that patch is thru. Net, there is no way in VDSM to enable
> glusterd insecure option for this usecase.
>
> *Case 3: *hostA is a gluster peer. hostB is a gluster peer (obviously) ,
> part of Gluster volume which is managed by oVirt Engine.
>
>     Since oVirt is managing is gluster volume, the same as Case 1 applies
> *
> **Case 4: *hostA is a gluster peer. hostB is a gluster peer (obviously) ,
> but the Gluster volume is not managed by oVirt.
>
>     (Someone manually set up the Gluster volume, using hypervisor hosts as
> peers, for using the un-used disk space on hyp host for storage)
>     This is similar to Case 2, but since hostA and hostB are part of
> gluster peer, VDSM can do 'gluster peer set all ...insecure..' before
> creating storage domain. Problem happens, when new hosts are added to the
> datacentre which are not part of gluster peer. Note in a virt env. hosts
> can be dynamically added / remove. So when a VM backed by gluster storage
> domain migrates to a host that is not a gluster peer, this usecase is
> broken like in Case 2b
>

I am not much familiar with storage domain migrations, please correct me in
case i am wrong.

>From the above case 4, i assume

- cluster1 is having hostA and hostB
- SD is created on hostA
- a new cluster cluster2 is created and hostC is added to it
- SD is moved to hostC

in this hostC is not a peer. this is same as *case 2b. *Also if a new peer
is added to cluster1, insecure option will be automatically set in the new
node as well AFAIK.

>
> *To Summarize:*
>
> 1) 'gluster peer set all ...insecure...' cli option will definitely help
> and it will help more if the oVirt Volumes tab provides an UI way of
> invoking this on User request. This will cover some usecases as desc above
>
> 2) Case 2b & 4 - how to handle this scenario ?
>
> thanx,
> deepak
>
>
>
>
>  Thanks,
> Kanagaraj
>
>
>
>> thanx,
>> deepak
>>
>>
>>>
>>>>
>>>>>> IIUC glusterd.info file can be used to store about these parameters
>>>>>> similar to how volume specific options are stored in
>>>>>> vols/<volname>/info
>>>>>> file?
>>>>>>
>>>>>>
>>>>> We can persist this in glusterd.vol referred by the respective glusterd
>>>>> instance.
>>>>>
>>>>
>>>> So glusterd.vol is not [re]generated during glusterd init?
>>>>
>>>
>>>
>>> No, glusterd.vol does not get altered during init.
>>>
>>> -Vijay
>>>
>>>>
>>>>> -Vijay
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at nongnu.org
>>> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>>>
>>>
>>>
>>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20130627/240cc82a/attachment-0001.html>