[Gluster-devel] DHT2 Status update
Shyam
srangana at redhat.com
Thu Nov 5 19:51:07 UTC 2015
Hi,
Here are the various artifacts from this meeting,
1) Recording:
https://drive.google.com/file/d/0B_1LWY0UTlNiQTI0ZU9hWDJGdEk/view?usp=sharing
2) Slides:
https://docs.google.com/presentation/d/1SNzak0lyEaKh2Zin-P1v9JUD_uSodiIVAssKOfJj7Ow/edit?usp=sharing
3) Attached please find steps to replay the demo, building from sources.
(Nov_5_2015_StatusUpdateDemo.log)
Thank you,
Shyam/Venky
On 11/03/2015 09:27 PM, Shyam wrote:
> Hi,
>
> Coming Thursday, i.e Nov-05-2015, @ 6:30 AM Eastern we are having a
> short call to present DHT2 status (ics attached)
>
> To join the meeting on a computer or mobile phone:
> https://bluejeans.com/371209168?src=calendarLink&g=onzgc3thmfxgcqdsmvsgqylufzrw63i=
>
>
> Agenda:
> - Current status update
> - Quick demo of some FOPs working (lookup, create)
> - A quick tour on how things look at the backend
> - Targets for next milestone
>
> We will record the meeting, so in case you are interested but are unable
> to make it, a recording will be posted.
>
> We will also post a mail update on the status, so finer details should
> be present in text form anyway.
>
> Thanks,
> Shyam
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
How/Where to get the code
=========================
1) git:
git clone ssh://ShyamsundarR@review.gerrithub.io:29418/ShyamsundarR/glusterfs
2) cd to cloned directory and cherry pick the following changes that are not
yet merged,
- https://review.gerrithub.io/#/c/251309/
(FYI, the above brings in 2 other changes when cherry picked)
- https://review.gerrithub.io/#/c/250059/
- https://review.gerrithub.io/#/c/251134/
- https://review.gerrithub.io/#/c/251377/
- https://review.gerrithub.io/#/c/250778/
NOTE: Post the above cherry picks the git log should look as below (changes may
have moved, so commit # may not match),
# git log -n8 --decorate --pretty=oneline --abbrev-commit
d485bdc (HEAD -> integration) dht2/glusterd: Added graph generation change to
include posix2-mds
20e7dab Integration: Fixes for remote inode lookup to work
ad4255b storage/posix2: Inode operations
4ba5572 posix, v2: Boilerplate for mds, ds split
2e6e550 (review/shyam/dht2fops-30) cluster/dht2: Add prototype open/stat/
setattr/flush FOPs
bee2a3e (review/shyam/dht2fops-20) cluster/dht2: Prototype implementation
of create
aa21512 (review/shyam/dht2fops-10) cluster/dht2: Prototype implementation
of lookup FOP
c67933b (origin/gl40_dht_playground, gerrit/gl40_dht_playground,
gl40_dht_playground) dht2: Initial conceptual design for the
second DHT2 prototype
3) Build (./autogen.sh; ./configure; make clean; make -j8)
-> Install (sudo make install)
4) Get to a root shell and play with the steps below...
GOTCHA'S:
- Create "/d/backends/" directory before proceeding with the steps.
- GFID generation would differ as the steps are run, as they are
randomly generated
- Mode bits may differ as per umask of systems used to run the commands
- Do not use tab completions (or ls like commands) for paths on the
gluster mount, readdir(p) is not implemented yet (the brick process
will crash)
Step 1: Gluster health checks
=============================
# glusterd
# gluster v status
No volumes present
Step 2: Creating a DHT2 volume
==============================
# gluster volume create patchy mds 2 data 1 \
marvin.bos.redhat.com:/d/backends/patchy1 \
marvin.bos.redhat.com:/d/backends/patchy2 \
marvin.bos.redhat.com:/d/backends/patchy3
# gluster v status
Volume patchy is not started
Inspecting the vol files for DHT2 and POSIX2 changes
----------------------------------------------------
# less /var/lib/glusterd/vols/patchy/trusted-patchy.tcp-fuse.vol
(Note distribute2/dht2 instead of regular dht in the vol file)
# less \
/var/lib/glusterd/vols/patchy/\
patchy.marvin.bos.redhat.com.d-backends-patchy1.vol
(Note posix2-mds instead of regular posix in the vol file)
Inspecting the bricks pre-volume start
--------------------------------------
# tree /d/backends/patchy?
/d/backends/patchy1
/d/backends/patchy2
/d/backends/patchy3
0 directories, 0 files
Starting the volume and inspecting the bricks
---------------------------------------------
# gluster v start patchy
volume start: patchy: success
# gluster v status
Status of volume: patchy
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick marvin.bos.redhat.com:/d/backends/pat
chy1 49152 0 Y 21075
Brick marvin.bos.redhat.com:/d/backends/pat
chy2 49153 0 Y 21090
Brick marvin.bos.redhat.com:/d/backends/pat
chy3 49154 0 Y 21105
NFS Server on localhost N/A N/A N N/A
Task Status of Volume patchy
------------------------------------------------------------------------------
There are no active volume tasks
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
??? 00
??? 00000000-0000-0000-0000-000000000001
/d/backends/patchy2
/d/backends/patchy3
3 directories, 0 files
(Note ['root'|'/'] GFID 0x00...01 created by default)
Step 3: Mounting and inspecting attributes of gluster root
==========================================================
# mount -t glusterfs marvin.bos.redhat.com:/patchy /mnt/glusterfs/0
# stat -c %A /mnt/glusterfs/0
drwx------
# stat -c %A /d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/
drwx------
Changing the mode for root on the brick and checking back on the client
-----------------------------------------------------------------------
# chmod 770 /d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/
# stat -c %A /mnt/glusterfs/0
drwxrwx---
Step 4: Creating a file and inspecting the bricks
=================================================
# touch /mnt/glusterfs/0/File1
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
??? 00
??? 00000000-0000-0000-0000-000000000001
? ??? File1
??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
/d/backends/patchy3
3 directories, 2 files
Demonstrating the relation between the file name and its inode/GFID
-------------------------------------------------------------------
# getfattr -e hex -n trusted.gfid \
/d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/File1
trusted.gfid=0x00003e561b944f99ac20dab615aa06e8
(Note File1 trusted.gfid xattr points to an on disk file of the same
hexadecimal name, on the same brick (this is the DHT2 co-location policy, inode
for files in a directory belong to the same brick))
Displaying the real meta-data store
-----------------------------------
# stat -c %A \
/d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/File1
-rwx------
# stat -c %A /d/backends/patchy1/00/00/00003e56-1b94-4f99-ac20-dab615aa06e8
-rw-r--r--
# stat -c %A /mnt/glusterfs/0/File1
-rw-r--r--
(Note that at the mount point the metadata for the object (a file here) is
displayed based on its GFID object in the backend and not its dentry/name
object)
Step 5: chown (or setattr) functionality
========================================
# stat -c %U /mnt/glusterfs/0/File1
root
# chown somari /mnt/glusterfs/0/File1
# stat -c %U /mnt/glusterfs/0/File1
somari
# stat -c %U \
/d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/File1
root
# stat /d/backends/patchy1/00/00/00003e56-1b94-4f99-ac20-dab615aa06e8
somari
(This is just to demonstrate that setattr works, and that the GFID is the one
that is modified, as that rather than the dentry/name is the real
metadata store)
Step 6: Directory distibution
=============================
NOTE: (synthetic creation as mkdir FOP is TBD)
----------------------------------------------
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
??? 00
??? 00000000-0000-0000-0000-000000000001
? ??? File1
??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
/d/backends/patchy3
3 directories, 2 files
# touch /d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/Dir1
# mkdir -p /d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001
# setfattr -n trusted.gfid -v 0x00010000dddd00000000000000000001 \
/d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/Dir1
(Note, use the same GFID in both 'mkdir -p' and 'setfattr' and further keep
the first 4 bytes the same, so that it falls logically into the second brick
as per the client DHT2 layout)
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
??? 00
??? 00000000-0000-0000-0000-000000000001
? ??? Dir1
? ??? File1
??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
??? 00
??? 01
??? 00010000-dddd-0000-0000-000000000001
/d/backends/patchy3
6 directories, 3 files
(Note, the above demonstrates lack of co-location for directory objects,
resulting in providing metadata distribution)
Inspecting the directory from the client
----------------------------------------
# stat -c %A /mnt/glusterfs/0/Dir1
drwxr-xr-x
# chmod 700 /d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/
# stat -c %A /mnt/glusterfs/0/Dir1
drwx------
Step 7: Creating files within sub-directories and inspecting the bricks
=======================================================================
# touch /mnt/glusterfs/0/Dir1/File2
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
??? 00
??? 00000000-0000-0000-0000-000000000001
? ??? Dir1
? ??? File1
??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
??? 00
??? 01
??? 00010000-dddd-0000-0000-000000000001
? ??? File2
??? 000175d5-2bc4-4f07-8bbb-27b42f30eaf4
/d/backends/patchy3
6 directories, 5 files
# getfattr -n trusted.gfid -e hex \
/d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/File2
trusted.gfid=0x000175d52bc44f078bbb27b42f30eaf4
(Note similar dentry/name and GFID relations and co-location property as
demonstrated for files within root, hold true for files within other
subdirectories, but do not need to hold for directories)
Step 8: Bonus demonstration of non co-located files (say due to rename)
=======================================================================
(NOTE: Synthetic splitting of file name and inode done as rename FOP is TBD)
----------------------------------------------------------------------------
# touch /d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/File3
# setfattr -n trusted.gfid -v 0x00000000ffff00000000000000000001 \
/d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/File3
# touch /d/backends/patchy1/00/00/00000000-ffff-0000-0000-000000000001
(Note, the above simulates a condition where File3 was originally created under
(say) root, and hence got a GFID heaader of 0x0000 to co-locate its inode on the
same MDS. Further to this, say this file was renamed under Dir1, so here name
moves, but inode/GFID does not change, causing a non co-located file name with
its inode)
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
??? 00
??? 00000000-0000-0000-0000-000000000001
? ??? Dir1
? ??? File1
??? 00000000-ffff-0000-0000-000000000001
??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
??? 00
??? 01
??? 00010000-dddd-0000-0000-000000000001
? ??? File2
? ??? File3
??? 000175d5-2bc4-4f07-8bbb-27b42f30eaf4
/d/backends/patchy3
6 directories, 7 files
# getfattr -n trusted.gfid -e hex \
/d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/File3
trusted.gfid=0x00000000ffff00000000000000000001
# stat -c %A /mnt/glusterfs/0/Dir1/File3
-rw-r--r--
[root at marvin glusterfs]# stat -c %A /mnt/glusterfs/0/Dir1/File3
-rw-r--r--
[root at marvin glusterfs]# chmod 777 \
/d/backends/patchy1/00/00/00000000-ffff-0000-0000-000000000001
[root at marvin glusterfs]# stat -c %A /mnt/glusterfs/0/Dir1/File3
-rwxrwxrwx
(Note, the above inspects the metadata for the file, and also demonstrates
changed inode (not entry/name metadata) information being reflected on
the mount)
<END>
More information about the Gluster-devel
mailing list