<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 7, 2017 at 3:05 PM, Xavier Hernandez <span dir="ltr"><<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 07/07/17 11:25, Pranith Kumar Karampuri wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
On Fri, Jul 7, 2017 at 2:46 PM, Xavier Hernandez <<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>> wrote:<br>
<br>
On 07/07/17 10:12, Pranith Kumar Karampuri wrote:<br>
<br>
<br>
<br>
On Fri, Jul 7, 2017 at 1:13 PM, Xavier Hernandez<br>
<<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>>><br>
wrote:<br>
<br>
Hi Pranith,<br>
<br>
On 05/07/17 12:28, Pranith Kumar Karampuri wrote:<br>
<br>
<br>
<br>
On Tue, Jul 4, 2017 at 2:26 PM, Xavier Hernandez<br>
<<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>>>><br>
wrote:<br>
<br>
Hi Pranith,<br>
<br>
On 03/07/17 08:33, Pranith Kumar Karampuri wrote:<br>
<br>
Xavi,<br>
Now that the change has been reverted, we can<br>
resume this<br>
discussion and decide on the exact format that<br>
considers, tier, dht,<br>
afr, ec. People working geo-rep/dht/afr/ec had<br>
an internal<br>
discussion<br>
and we all agreed that this proposal would be a<br>
good way<br>
forward. I<br>
think once we agree on the format and decide on<br>
the initial<br>
encoding/decoding functions of the xattr and<br>
this change is<br>
merged, we<br>
can send patches on afr/ec/dht and geo-rep to<br>
take it to<br>
closure.<br>
<br>
Could you propose the new format you have in<br>
mind that<br>
considers<br>
all of<br>
the xlators?<br>
<br>
<br>
My idea was to create a new xattr not bound to any<br>
particular<br>
function but which could give enough information to<br>
be used<br>
in many<br>
places.<br>
<br>
Currently we have another attribute called<br>
glusterfs.pathinfo that<br>
returns hierarchical information about the location of a<br>
file. Maybe<br>
we can extend this to unify all these attributes<br>
into a single<br>
feature that could be used for multiple purposes.<br>
<br>
Since we have time to discuss it, I would like to<br>
design it with<br>
more information than we already talked.<br>
<br>
First of all, the amount of information that this<br>
attribute can<br>
contain is quite big if we expect to have volumes with<br>
thousands of<br>
bricks. Even in the most simple case of returning<br>
only an<br>
UUID, we<br>
can easily go beyond the limit of 64KB.<br>
<br>
Consider also, for example, what shard should return<br>
when<br>
pathinfo<br>
is requested for a file. Probably it should return a<br>
list of<br>
shards,<br>
each one with all its associated pathinfo. We are<br>
talking<br>
about big<br>
amounts of data here.<br>
<br>
I think this kind of information doesn't fit very<br>
well in an<br>
extended attribute. Another think to consider is<br>
that most<br>
probably<br>
the requester of the data only needs a fragment of<br>
it, so we are<br>
generating big amounts of data only to be parsed and<br>
reduced<br>
later,<br>
dismissing most of it.<br>
<br>
What do you think about using a very special virtual<br>
file to<br>
manage<br>
all this information ? it could be easily read using<br>
normal read<br>
fops, so it could manage big amounts of data easily.<br>
Also,<br>
accessing<br>
only to some parts of the file we could go directly<br>
where we<br>
want,<br>
avoiding the read of all remaining data.<br>
<br>
A very basic idea could be this:<br>
<br>
Each xlator would have a reserved area of the file.<br>
We can<br>
reserve<br>
up to 4GB per xlator (32 bits). The remaining 32<br>
bits of the<br>
offset<br>
would indicate the xlator we want to access.<br>
<br>
At offset 0 we have generic information about the<br>
volume.<br>
One of the<br>
the things that this information should include is a<br>
basic<br>
hierarchy<br>
of the whole volume and the offset for each xlator.<br>
<br>
After reading this, the user will seek to the<br>
desired offset and<br>
read the information related to the xlator it is<br>
interested in.<br>
<br>
All the information should be stored in a format easily<br>
extensible<br>
that will be kept compatible even if new information is<br>
added in the<br>
future (for example doing special mappings of the 32<br>
bits<br>
offsets<br>
reserved for the xlator).<br>
<br>
For example we can reserve the first megabyte of the<br>
xlator<br>
area to<br>
have a mapping of attributes with its respective offset.<br>
<br>
I think that using a binary format would simplify<br>
all this a<br>
lot.<br>
<br>
Do you think this is a way to explore or should I stop<br>
wasting time<br>
here ?<br>
<br>
<br>
I think this just became a very big feature :-). Shall<br>
we just<br>
live with<br>
it the way it is now?<br>
<br>
<br>
I supposed it...<br>
<br>
Only thing we need to check is if shard needs to handle this<br>
xattr.<br>
If so, what it should return ? only the UUID's corresponding<br>
to the<br>
first shard or the UUID's of all bricks containing at least one<br>
shard ? I guess that the first one is enough, but just to be<br>
sure...<br>
<br>
My proposal was to implement a new xattr, for example<br>
glusterfs.layout, that contains enough information to be<br>
usable in<br>
all current use cases.<br>
<br>
<br>
Actually pathinfo is supposed to give this information and it<br>
already<br>
has the following format: for a 5x2 distributed-replicate volume<br>
<br>
<br>
Yes, I know. I wanted to unify all information.<br>
<br>
<br>
root@dhcp35-190 - /mnt/v3<br>
13:38:12 :) ⚡ getfattr -n trusted.glusterfs.pathinfo d<br>
# file: d<br>
trusted.glusterfs.pathinfo="((<wbr><DISTRIBUTE:v3-dht><br>
(<REPLICATE:v3-replicate-0><br>
<POSIX(/home/gfs/v3_0):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_0/d><br>
<POSIX(/home/gfs/v3_1):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_1/d>)<br>
(<REPLICATE:v3-replicate-2><br>
<POSIX(/home/gfs/v3_5):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_5/d><br>
<POSIX(/home/gfs/v3_4):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_4/d>)<br>
(<REPLICATE:v3-replicate-1><br>
<POSIX(/home/gfs/v3_3):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_3/d><br>
<POSIX(/home/gfs/v3_2):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_2/d>)<br>
(<REPLICATE:v3-replicate-4><br>
<POSIX(/home/gfs/v3_8):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_8/d><br>
<POSIX(/home/gfs/v3_9):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_9/d>)<br>
(<REPLICATE:v3-replicate-3><br>
<POSIX(/home/gfs/v3_6):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_6/d><br>
<POSIX(/home/gfs/v3_7):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_7/d>))<br>
(v3-dht-layout (v3-replicate-0 0 858993458) (v3-replicate-1<br>
858993459<br>
1717986917) (v3-replicate-2 1717986918 2576980376) (v3-replicate-3<br>
2576980377 3435973835) (v3-replicate-4 3435973836 4294967295)))"<br>
<br>
<br>
root@dhcp35-190 - /mnt/v3<br>
13:38:26 :) ⚡ getfattr -n trusted.glusterfs.pathinfo d/a<br>
# file: d/a<br>
trusted.glusterfs.pathinfo="(<<wbr>DISTRIBUTE:v3-dht><br>
(<REPLICATE:v3-replicate-1><br>
<POSIX(/home/gfs/v3_3):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_3/d/a><br>
<POSIX(/home/gfs/v3_2):dhcp35-<wbr>190.lab.eng.blr.redhat.com:/ho<wbr>me/gfs/v3_2/d/a>))"<br>
<br>
<br>
<br>
<br>
The idea would be that each xlator that makes a significant<br>
change<br>
in the way or the place where files are stored, should put<br>
information in this xattr. The information should include:<br>
<br>
* Type (basically AFR, EC, DHT, ...)<br>
* Basic configuration (replication and arbiter for AFR, data and<br>
redundancy for EC, # subvolumes for DHT, shard size for<br>
sharding, ...)<br>
* Quorum imposed by the xlator<br>
* UUID data comming from subvolumes (sorted by brick position)<br>
* It should be easily extensible in the future<br>
<br>
The last point is very important to avoid the issues we have<br>
seen<br>
now. We must be able to incorporate more information without<br>
breaking backward compatibility. To do so, we can add tags<br>
for each<br>
value.<br>
<br>
For example, a distribute 2, replica 2 volume with 1 arbiter<br>
should<br>
be represented by this string:<br>
<br>
DHT[dist=2,quorum=1](<br>
AFR[rep=2,arbiter=1,quorum=2](<br>
NODE[quorum=2,uuid=<UUID1>](<<wbr>path1>),<br>
NODE[quorum=2,uuid=<UUID2>](<<wbr>path2>),<br>
NODE[quorum=2,uuid=<UUID3>](<<wbr>path3>)<br>
),<br>
AFR[rep=2,arbiter=1,quorum=2](<br>
NODE[quorum=2,uuid=<UUID4>](<<wbr>path4>),<br>
NODE[quorum=2,uuid=<UUID5>](<<wbr>path5>),<br>
NODE[quorum=2,uuid=<UUID6>](<<wbr>path6>)<br>
)<br>
)<br></blockquote></blockquote><div><br></div><div>Yes, this looks simpler for now.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Some explanations:<br>
<br>
AFAIK DHT doesn't have quorum, so the default is '1'. We may<br>
decide<br>
to omit it when it's '1' for any xlator.<br>
<br>
Quorum in AFR represents client-side enforced quorum. Quorum<br>
in NODE<br>
represents the server-side enforced quorum.<br>
<br>
The <path> shown in each NODE represents the physical<br>
location of<br>
the file (similar to current glusterfs.pathinfo) because<br>
this xattr<br>
can be retrieved for a particular file using getxattr. This<br>
is nice,<br>
but we can remove it for now if it's difficult to implement.<br>
<br>
We can decide to have a verbose string or try to omit some<br>
fields<br>
when not strictly necessary. For example, if there are no<br>
arbiters,<br>
we can omit the 'arbiter' tag instead of writing 'arbiter=0'. We<br>
could also implicitly compute 'dist' and 'rep' from the<br>
number of<br>
elements contained between '()'.<br>
<br>
What do you think ?<br>
<br>
<br>
Quite a few people are already familiar with path-info. So I am<br>
of the<br>
opinion that we give this information for that xattr itself.<br>
This xattr<br>
hasn't changed after quorum/arbiter/shard came in, so may be<br>
they should?<br>
<br>
<br>
Not sure how easy would it be to change the format of path-info to<br>
incorporate the new information without breaking existing features<br>
or even user scripts based on it. Maybe a new xattr would be easier<br>
to implement and adapt.<br>
<br>
<br>
Probably.<br>
<br>
<br>
<br>
I missed one important thing in the format: an xlator may have<br>
per-subvolume information. This information can be placed just<br>
before each subvolume information:<br>
<br>
DHT[dist=2,quorum=1](<br>
[hash-range=0x00000000-0x7ffff<wbr>fff]AFR[...](...),<br>
[hash-range=0x80000000-0xfffff<wbr>fff]AFR[...](...)<br>
)<br>
<br>
<br>
Yes, makes sense.<br>
<br>
In general I am better at solving problems someone faces, because things<br>
will be more concrete. Do you think it is better to wait until the first<br>
consumer of this functionality comes along and gives their inputs about<br>
what would be nice to have vs must have? At the moment I am not sure how<br>
to distinguish what must be there vs what is nice to have :-(.<br>
</blockquote>
<br>
The good thing is that using this format we can easily start with bare minimum information, like this:<br>
<br>
DHT(<br>
AFR(<br>
NODE[uuid=<UUID1>],<br>
NODE[uuid=<UUID2>],<br>
NODE[uuid=<UUID3>]<br>
),<br>
AFR(<br>
NODE[uuid=<UUID1>],<br>
NODE[uuid=<UUID2>],<br>
NODE[uuid=<UUID3>]<br>
)<br>
)<br>
<br>
And add more information as it is needed, since it won't break backward compatibility.<br>
<br>
Xavi<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Xavi<br>
<br>
<br>
<br>
<br>
Xavi<br>
<br>
<br>
<br>
<br>
Xavi<br>
<br>
<br>
<br>
<br>
On Wed, Jun 21, 2017 at 2:08 PM, Karthik Subrahmanya<br>
<<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> <mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>>><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a> <mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a> <mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>>>><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> <mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>>><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a> <mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>><br>
<mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a> <mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>>>><wbr>>> wrote:<br>
<br>
<br>
<br>
On Wed, Jun 21, 2017 at 1:56 PM, Xavier<br>
Hernandez<br>
<<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>>>>><br>
wrote:<br>
<br>
That's ok. I'm currently unable to write<br>
a patch for<br>
this on ec.<br>
<br>
Sunil is working on this patch.<br>
<br>
~Karthik<br>
<br>
If no one can do it, I can try to do it<br>
in 6 - 7<br>
hours...<br>
<br>
Xavi<br>
<br>
<br>
On Wednesday, June 21, 2017 09:48 CEST,<br>
Pranith<br>
Kumar<br>
Karampuri<br>
<<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>> <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a> <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a> <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a> <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>>><wbr>>> wrote:<br>
<br>
<br>
<br>
On Wed, Jun 21, 2017 at 1:00 PM, Xavier<br>
Hernandez<br>
<<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>>> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>>>>> wrote:<br>
<br>
I'm ok with reverting node-uuid<br>
content<br>
to the<br>
previous<br>
format and create a new xattr<br>
for the<br>
new format.<br>
Currently, only rebalance will<br>
use it.<br>
<br>
Only thing to consider is what can<br>
happen if we<br>
have a<br>
half upgraded cluster where some<br>
clients<br>
have<br>
this change<br>
and some not. Can rebalance work<br>
in this<br>
situation ? if<br>
so, could there be any issue ?<br>
<br>
<br>
I think there shouldn't be any problem,<br>
because this is<br>
in-memory xattr so layers below<br>
afr/ec will<br>
only see<br>
node-uuid<br>
xattr.<br>
This also gives us a chance to do<br>
whatever<br>
we want<br>
to do in<br>
future with this xattr without any<br>
problems<br>
about<br>
backward<br>
compatibility.<br>
<br>
You can check<br>
<br>
<br>
<br>
<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/c<wbr>/17576/3/xlators/cluster/afr/s<wbr>rc/afr-inode-read.c@1507</a><br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a>><br>
<br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a><br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a>>><br>
<br>
<br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a><br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a>><br>
<br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a><br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a>>>><br>
<br>
<br>
<br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a><br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a>><br>
<br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a><br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a>>><br>
<br>
<br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a><br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a>><br>
<br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a><br>
<<a href="https://review.gluster.org/#/c/17576/3/xlators/cluster/afr/src/afr-inode-read.c@1507" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/17576/3/xlators/cluster/afr/<wbr>src/afr-inode-read.c@1507</a>>>>><br>
for how karthik implemented this in AFR<br>
(this got merged<br>
accidentally yesterday, but looks<br>
like this<br>
is what<br>
we are<br>
settling on)<br>
<br>
<br>
<br>
Xavi<br>
<br>
<br>
On Wednesday, June 21, 2017<br>
06:56 CEST,<br>
Pranith<br>
Kumar<br>
Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a> <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a> <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a> <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>>><wbr>>> wrote:<br>
<br>
<br>
<br>
On Wed, Jun 21, 2017 at<br>
10:07 AM, Nithya<br>
Balachandran<br>
<<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a> <mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>>><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>>>> <mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a> <mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>>><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a><br>
<mailto:<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>>>><wbr>>> wrote:<br>
<br>
<br>
On 20 June 2017 at<br>
20:38, Aravinda<br>
<<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>>> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>>><wbr>>> wrote:<br>
<br>
On 06/20/2017 06:02<br>
PM, Pranith<br>
Kumar Karampuri<br>
wrote:<br>
<br>
Xavi, Aravinda<br>
and I had a<br>
discussion on<br>
#gluster-dev and we<br>
agreed to go<br>
with the format<br>
Aravinda<br>
suggested for<br>
now and<br>
in future we<br>
wanted some more<br>
changes<br>
for dht<br>
to detect which<br>
subvolume went<br>
down came<br>
back<br>
up, at that time<br>
we will revisit<br>
the solution<br>
suggested by Xavi.<br>
<br>
Susanth is doing<br>
the dht<br>
changes<br>
Aravinda is doing<br>
geo-rep changes<br>
<br>
Done. Geo-rep patch<br>
sent for<br>
review<br>
<br>
<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/175<wbr>82</a><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a>><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a>>><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a>><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a>>>><br>
<br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a>><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a>>><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a>><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a><br>
<<a href="https://review.gluster.org/17582" rel="noreferrer" target="_blank">https://review.gluster.org/17<wbr>582</a>>>>><br>
<br>
<br>
<br>
The proposed changes to the<br>
node-uuid<br>
behaviour<br>
(while good) are going<br>
to break<br>
tiering<br>
. Tiering<br>
changes will take a<br>
little more<br>
time to<br>
be coded and<br>
tested.<br>
<br>
As this is a regression<br>
for 3.11<br>
and a<br>
blocker for<br>
3.11.1, I suggest we go<br>
back to<br>
the original<br>
node-uuid behaviour for<br>
now so as to<br>
unblock the<br>
release and target the<br>
proposed<br>
changes<br>
for the next<br>
3.11 releases.<br>
<br>
<br>
Let me see if I understand<br>
the changes<br>
correctly. We are<br>
restoring the behavior of<br>
node-uuid<br>
xattr<br>
and adding a<br>
new xattr for parallel<br>
rebalance for<br>
both<br>
afr and ec,<br>
correct? Otherwise that is<br>
one more<br>
regression. If yes,<br>
we will also wait for Xavi's<br>
inputs.<br>
Jeff<br>
accidentally<br>
merged the afr patch<br>
yesterday which<br>
does<br>
these changes.<br>
If everyone is in agreement,<br>
we will<br>
leave<br>
it as is and<br>
add similar changes in ec as<br>
well.<br>
If we are<br>
not in<br>
agreement, then we will let the<br>
discussion<br>
progress :-)<br>
<br>
<br>
<br>
<br>
Regards,<br>
Nithya<br>
<br>
--<br>
Aravinda<br>
<br>
<br>
Thanks to all of you<br>
guys for<br>
the discussions!<br>
<br>
On Tue, Jun 20,<br>
2017 at<br>
5:05 PM,<br>
Xavier<br>
Hernandez<br>
<<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>>><br>
<br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a><br>
<mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>>>>> wrote:<br>
<br>
Hi Aravinda,<br>
<br>
On 20/06/17<br>
12:42,<br>
Aravinda<br>
wrote:<br>
<br>
I think<br>
following format<br>
can be easily<br>
adopted<br>
by all<br>
components<br>
<br>
UUIDs of a<br>
subvolume are<br>
seperated by<br>
space and<br>
subvolumes are<br>
separated<br>
by comma<br>
<br>
For example,<br>
node1 and<br>
node2 are replica<br>
with U1<br>
and U2 UUIDs<br>
<br>
respectively and<br>
node3<br>
and node4 are<br>
replica with U3 and<br>
U4 UUIDs<br>
respectively<br>
<br>
<br>
node-uuid can<br>
return "U1<br>
U2,U3 U4"<br>
<br>
<br>
While this<br>
is ok for<br>
current<br>
implementation,<br>
I think this<br>
can be<br>
insufficient if there<br>
are more<br>
layers of<br>
xlators<br>
that require to<br>
indicate<br>
some sort of<br>
grouping. Some<br>
<br>
representation that can<br>
represent hierarchy<br>
would be<br>
better. For<br>
example: "(U1 U2) (U3<br>
U4)" (we can use<br>
spaces or<br>
comma as a<br>
separator).<br>
<br>
<br>
<br>
Geo-rep can<br>
split by ","<br>
and then split<br>
by space and<br>
take first UUID<br>
DHT can<br>
split<br>
the value<br>
by space or<br>
comma<br>
and get unique<br>
UUIDs list<br>
<br>
<br>
This doesn't<br>
solve the<br>
problem I described<br>
in the previous<br>
email. Some<br>
more logic will<br>
need to be<br>
added to<br>
avoid<br>
more than one node<br>
from each<br>
replica-set to be<br>
active. If we<br>
have some<br>
explicit<br>
hierarchy<br>
information in<br>
the<br>
node-uuid value,<br>
more<br>
decisions can be<br>
taken.<br>
<br>
An initial<br>
proposal<br>
I made<br>
was this:<br>
<br>
<br>
DHT[2](AFR[2,0](NODE(U1),<br>
NODE(U2)),<br>
<br>
AFR[2,0](NODE(U1),<br>
NODE(U2)))<br>
<br>
This is<br>
harder to<br>
parse, but<br>
gives a lot of<br>
information:<br>
DHT with 2<br>
subvolumes, each<br>
subvolume is<br>
an AFR with<br>
replica 2 and no<br>
arbiters.<br>
It's also<br>
easily<br>
extensible with<br>
any new<br>
xlator that<br>
changes<br>
the layout.<br>
<br>
However<br>
maybe this<br>
is not<br>
the moment to do<br>
this, and<br>
probably<br>
we could<br>
implement this<br>
in a new<br>
xattr with<br>
a better<br>
name.<br>
<br>
Xavi<br>
<br>
<br>
<br>
Another<br>
question is<br>
about the behavior<br>
when a<br>
node is down,<br>
existing<br>
<br>
node-uuid xattr<br>
will not<br>
return that<br>
UUID if<br>
a node<br>
is down.<br>
What is the<br>
behavior<br>
with the<br>
proposed xattr?<br>
<br>
Let me<br>
know your<br>
thoughts.<br>
<br>
regards<br>
Aravinda VK<br>
<br>
On<br>
06/20/2017<br>
03:06 PM,<br>
Aravinda wrote:<br>
<br>
Hi Xavi,<br>
<br>
On<br>
06/20/2017 02:51<br>
PM, Xavier<br>
<br>
Hernandez wrote:<br>
<br>
<br>
Hi Aravinda,<br>
<br>
<br>
On 20/06/17<br>
11:05, Pranith Kumar<br>
<br>
Karampuri wrote:<br>
<br>
<br>
Adding more<br>
people to get a<br>
<br>
consensus<br>
about this.<br>
<br>
<br>
On<br>
Tue, Jun<br>
20, 2017 at 1:49<br>
<br>
PM,<br>
Aravinda<br>
<br>
<<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>>><br>
<br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>>><wbr>><br>
<br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>>><br>
<br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a> <mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a><br>
<mailto:<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>>>><wbr>>>><br>
<br>
wrote:<br>
<br>
<br>
<br>
regards<br>
<br>
Aravinda VK<br>
<br>
<br>
<br>
On<br>
06/20/2017 01:26 PM,</blockquote>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>