[Gluster-devel] DHT2 Status update

Shyam srangana at redhat.com
Thu Nov 5 19:51:07 UTC 2015


Hi,

Here are the various artifacts from this meeting,

1) Recording: 
https://drive.google.com/file/d/0B_1LWY0UTlNiQTI0ZU9hWDJGdEk/view?usp=sharing

2) Slides: 
https://docs.google.com/presentation/d/1SNzak0lyEaKh2Zin-P1v9JUD_uSodiIVAssKOfJj7Ow/edit?usp=sharing

3) Attached please find steps to replay the demo, building from sources. 
(Nov_5_2015_StatusUpdateDemo.log)

Thank you,
Shyam/Venky

On 11/03/2015 09:27 PM, Shyam wrote:
> Hi,
>
> Coming Thursday, i.e Nov-05-2015, @ 6:30 AM Eastern we are having a
> short call to present DHT2 status (ics attached)
>
> To join the meeting on a computer or mobile phone:
> https://bluejeans.com/371209168?src=calendarLink&g=onzgc3thmfxgcqdsmvsgqylufzrw63i=
>
>
> Agenda:
> - Current status update
> - Quick demo of some FOPs working (lookup, create)
>    - A quick tour on how things look at the backend
> - Targets for next milestone
>
> We will record the meeting, so in case you are interested but are unable
> to make it, a recording will be posted.
>
> We will also post a mail update on the status, so finer details should
> be present in text form anyway.
>
> Thanks,
> Shyam
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
How/Where to get the code
=========================
1) git: 
git clone ssh://ShyamsundarR@review.gerrithub.io:29418/ShyamsundarR/glusterfs

2) cd to cloned directory and cherry pick the following changes that are not
yet merged,
  - https://review.gerrithub.io/#/c/251309/
    (FYI, the above brings in 2 other changes when cherry picked)
  - https://review.gerrithub.io/#/c/250059/
  - https://review.gerrithub.io/#/c/251134/
  - https://review.gerrithub.io/#/c/251377/
  - https://review.gerrithub.io/#/c/250778/

NOTE: Post the above cherry picks the git log should look as below (changes may
have moved, so commit # may not match),
# git log -n8 --decorate --pretty=oneline --abbrev-commit
d485bdc (HEAD -> integration) dht2/glusterd: Added graph generation change to
          include posix2-mds
20e7dab Integration: Fixes for remote inode lookup to work
ad4255b storage/posix2: Inode operations
4ba5572 posix, v2: Boilerplate for mds, ds split
2e6e550 (review/shyam/dht2fops-30) cluster/dht2: Add prototype open/stat/
          setattr/flush FOPs
bee2a3e (review/shyam/dht2fops-20) cluster/dht2: Prototype implementation
          of create
aa21512 (review/shyam/dht2fops-10) cluster/dht2: Prototype implementation
          of lookup FOP
c67933b (origin/gl40_dht_playground, gerrit/gl40_dht_playground,
          gl40_dht_playground) dht2: Initial conceptual design for the
          second DHT2 prototype

3) Build (./autogen.sh; ./configure; make clean; make -j8)
    -> Install (sudo make install)

4) Get to a root shell and play with the steps below...
GOTCHA'S: 
  - Create "/d/backends/" directory before proceeding with the steps.
  - GFID generation would differ as the steps are run, as they are
  randomly generated
  - Mode bits may differ as per umask of systems used to run the commands
  - Do not use tab completions (or ls like commands) for paths on the
  gluster mount, readdir(p) is not implemented yet (the brick process
  will crash)

Step 1: Gluster health checks
=============================
# glusterd
# gluster v status
No volumes present

Step 2: Creating a DHT2 volume
==============================
# gluster volume create patchy mds 2 data 1 \
  marvin.bos.redhat.com:/d/backends/patchy1 \
  marvin.bos.redhat.com:/d/backends/patchy2 \
  marvin.bos.redhat.com:/d/backends/patchy3

# gluster v status
Volume patchy is not started

Inspecting the vol files for DHT2 and POSIX2 changes
----------------------------------------------------
# less /var/lib/glusterd/vols/patchy/trusted-patchy.tcp-fuse.vol
(Note distribute2/dht2 instead of regular dht in the vol file)

# less \
/var/lib/glusterd/vols/patchy/\
patchy.marvin.bos.redhat.com.d-backends-patchy1.vol
(Note posix2-mds instead of regular posix in the vol file)

Inspecting the bricks pre-volume start
--------------------------------------
# tree /d/backends/patchy?
/d/backends/patchy1
/d/backends/patchy2
/d/backends/patchy3

0 directories, 0 files

Starting the volume and inspecting the bricks
---------------------------------------------
# gluster v start patchy
volume start: patchy: success
# gluster v status
Status of volume: patchy
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick marvin.bos.redhat.com:/d/backends/pat
chy1                                        49152     0          Y       21075
Brick marvin.bos.redhat.com:/d/backends/pat
chy2                                        49153     0          Y       21090
Brick marvin.bos.redhat.com:/d/backends/pat
chy3                                        49154     0          Y       21105
NFS Server on localhost                     N/A       N/A        N       N/A  
 
Task Status of Volume patchy
------------------------------------------------------------------------------
There are no active volume tasks
 
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
    ??? 00
        ??? 00000000-0000-0000-0000-000000000001
/d/backends/patchy2
/d/backends/patchy3

3 directories, 0 files
(Note ['root'|'/'] GFID 0x00...01 created by default)

Step 3: Mounting and inspecting attributes of gluster root
==========================================================
# mount -t glusterfs marvin.bos.redhat.com:/patchy /mnt/glusterfs/0
# stat -c %A /mnt/glusterfs/0
drwx------
# stat -c %A /d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/
drwx------

Changing the mode for root on the brick and checking back on the client
-----------------------------------------------------------------------
# chmod 770 /d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/
# stat -c %A /mnt/glusterfs/0
drwxrwx---

Step 4: Creating a file and inspecting the bricks
=================================================
# touch /mnt/glusterfs/0/File1
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
    ??? 00
        ??? 00000000-0000-0000-0000-000000000001
        ?   ??? File1
        ??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
/d/backends/patchy3

3 directories, 2 files

Demonstrating the relation between the file name and its inode/GFID
-------------------------------------------------------------------
# getfattr -e hex -n trusted.gfid \
/d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/File1 
trusted.gfid=0x00003e561b944f99ac20dab615aa06e8
(Note File1 trusted.gfid xattr points to an on disk file of the same
hexadecimal name, on the same brick (this is the DHT2 co-location policy, inode
for files in a directory belong to the same brick))

Displaying the real meta-data store
-----------------------------------
# stat -c %A \
/d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/File1 
-rwx------
# stat -c %A /d/backends/patchy1/00/00/00003e56-1b94-4f99-ac20-dab615aa06e8 
-rw-r--r--
# stat -c %A /mnt/glusterfs/0/File1
-rw-r--r--
(Note that at the mount point the metadata for the object (a file here) is
displayed based on its GFID object in the backend and not its dentry/name
object)

Step 5: chown (or setattr) functionality
========================================
# stat -c %U /mnt/glusterfs/0/File1
root
# chown somari /mnt/glusterfs/0/File1
# stat -c %U /mnt/glusterfs/0/File1
somari
# stat -c %U \
/d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/File1 
root
# stat /d/backends/patchy1/00/00/00003e56-1b94-4f99-ac20-dab615aa06e8 
somari
(This is just to demonstrate that setattr works, and that the GFID is the one
that is modified, as that rather than the dentry/name is the real
metadata store)

Step 6: Directory distibution
=============================
NOTE: (synthetic creation as mkdir FOP is TBD)
----------------------------------------------
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
    ??? 00
        ??? 00000000-0000-0000-0000-000000000001
        ?   ??? File1
        ??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
/d/backends/patchy3

3 directories, 2 files

# touch /d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/Dir1
# mkdir -p /d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001
# setfattr -n trusted.gfid -v 0x00010000dddd00000000000000000001 \
/d/backends/patchy1/00/00/00000000-0000-0000-0000-000000000001/Dir1 
(Note, use the same GFID in both 'mkdir -p' and 'setfattr' and further keep
the first 4 bytes the same, so that it falls logically into the second brick
as per the client DHT2 layout)

# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
    ??? 00
        ??? 00000000-0000-0000-0000-000000000001
        ?   ??? Dir1
        ?   ??? File1
        ??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
??? 00
    ??? 01
        ??? 00010000-dddd-0000-0000-000000000001
/d/backends/patchy3

6 directories, 3 files

(Note, the above demonstrates lack of co-location for directory objects,
resulting in providing metadata distribution)

Inspecting the directory from the client
----------------------------------------
# stat -c %A /mnt/glusterfs/0/Dir1
drwxr-xr-x
# chmod 700 /d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/
# stat -c %A /mnt/glusterfs/0/Dir1
drwx------

Step 7: Creating files within sub-directories and inspecting the bricks
=======================================================================
# touch /mnt/glusterfs/0/Dir1/File2
# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
    ??? 00
        ??? 00000000-0000-0000-0000-000000000001
        ?   ??? Dir1
        ?   ??? File1
        ??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
??? 00
    ??? 01
        ??? 00010000-dddd-0000-0000-000000000001
        ?   ??? File2
        ??? 000175d5-2bc4-4f07-8bbb-27b42f30eaf4
/d/backends/patchy3

6 directories, 5 files
# getfattr -n trusted.gfid -e hex \
/d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/File2
trusted.gfid=0x000175d52bc44f078bbb27b42f30eaf4
(Note similar dentry/name and GFID relations and co-location property as
demonstrated for files within root, hold true for files within other
subdirectories, but do not need to hold for directories)

Step 8: Bonus demonstration of non co-located files (say due to rename)
=======================================================================
(NOTE: Synthetic splitting of file name and inode done as rename FOP is TBD)
----------------------------------------------------------------------------
# touch /d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/File3
# setfattr -n trusted.gfid -v 0x00000000ffff00000000000000000001 \
/d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/File3
# touch /d/backends/patchy1/00/00/00000000-ffff-0000-0000-000000000001

(Note, the above simulates a condition where File3 was originally created under
(say) root, and hence got a GFID heaader of 0x0000 to co-locate its inode on the
same MDS. Further to this, say this file was renamed under Dir1, so here name
moves, but inode/GFID does not change, causing a non co-located file name with
its inode)

# tree /d/backends/patchy?
/d/backends/patchy1
??? 00
    ??? 00
        ??? 00000000-0000-0000-0000-000000000001
        ?   ??? Dir1
        ?   ??? File1
        ??? 00000000-ffff-0000-0000-000000000001
        ??? 00003e56-1b94-4f99-ac20-dab615aa06e8
/d/backends/patchy2
??? 00
    ??? 01
        ??? 00010000-dddd-0000-0000-000000000001
        ?   ??? File2
        ?   ??? File3
        ??? 000175d5-2bc4-4f07-8bbb-27b42f30eaf4
/d/backends/patchy3

6 directories, 7 files
# getfattr -n trusted.gfid -e hex \
/d/backends/patchy2/00/01/00010000-dddd-0000-0000-000000000001/File3
trusted.gfid=0x00000000ffff00000000000000000001

# stat -c %A /mnt/glusterfs/0/Dir1/File3
-rw-r--r--
[root at marvin glusterfs]# stat -c %A /mnt/glusterfs/0/Dir1/File3
-rw-r--r--
[root at marvin glusterfs]# chmod 777 \
/d/backends/patchy1/00/00/00000000-ffff-0000-0000-000000000001 
[root at marvin glusterfs]# stat -c %A /mnt/glusterfs/0/Dir1/File3
-rwxrwxrwx
(Note, the above inspects the metadata for the file, and also demonstrates
changed inode (not entry/name metadata) information being reflected on
the mount)

<END>


More information about the Gluster-devel mailing list