[Gluster-devel] Improving Geo-replication Status and Checkpoints
Kotresh Hiremath Ravishankar
khiremat at redhat.com
Wed Apr 1 09:06:11 UTC 2015
Hi Aravinda,
Looks good to me.
Thanks and Regards,
Kotresh H R
----- Original Message -----
> From: "Aravinda" <avishwan at redhat.com>
> To: "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Wednesday, April 1, 2015 2:30:49 PM
> Subject: Re: [Gluster-devel] Improving Geo-replication Status and Checkpoints
>
> Hi,
>
> In each node of Master Cluster one Monitor process and one or more
> worker process for each brick in that node.
> Monitor will have status file, which will be updated by glusterd.
> Possible Status values in monitor_status file are Created, Started,
> Paused, Stopped.
>
> Geo-rep can not be paused if monitor status is not "Started".
>
> Based on monitor_status, we need to hide other information from brick
> status file from showing it to user. For example, If monitor status is
> "Stopped", it will not make sense to show "Crawl Status" in Geo-rep
> Status output. Created a Matrix of possible status values based on
> Status of Monitor. VALUE represents actual unchanged value from Brick
> status file.
>
> Monitor Status ---> Created Started Paused Stopped
> --------------------------------------------------------------------------
> session VALUE VALUE VALUE VALUE
> brick VALUE VALUE VALUE VALUE
> node VALUE VALUE VALUE VALUE
> node_uuid VALUE VALUE VALUE VALUE
> volume VALUE VALUE VALUE VALUE
> slave_user VALUE VALUE VALUE VALUE
> slave_node N/A VALUE VALUE N/A
> status Created VALUE Paused Stopped
> last_synced N/A VALUE VALUE VALUE
> crawl_status N/A VALUE N/A N/A
> entry N/A VALUE N/A N/A
> data N/A VALUE N/A N/A
> meta N/A VALUE N/A N/A
> failures N/A VALUE VALUE VALUE
> checkpoint_completed N/A VALUE VALUE VALUE
> checkpoint_time N/A VALUE VALUE VALUE
> checkpoint_completed_time N/A VALUE VALUE VALUE
>
> Where:
> session - only in XML output, Complete session URL which is used in
> Create command
> brick - Master Brick Node
> node - Master Node
> node_uuid - Master Node UUID, Only in XML output
> volume - Master Volume
> slave_user - Slave User
> slave_node - Slave node to which respective master worker is connected.
> status - Created/Initializing../Active/Passive/Faulty/Paused/Stopped
> last_synced - Last synced Time
> crawl_status - Hybrid/History/Changelog
> entry - Number of entry ops pending(per session, resets counter if
> worker restart)
> data - Number of data ops pending(per session, resets counter if worker
> restart)
> meta - Number of meta ops pending(per session, resets counter if worker
> restart)
> failures - Number of failures. (If count more than 0, then action item
> for admin to look in log files)
> checkpoint_completed - Checkpoint Status Yes/No/ N/A
> checkpoint_time - Checkpoint Set time or N/A
> checkpoint_completed_time - Checkpoint Completed Time or N/A
>
> Along with the monitor_status, if brick status is Faulty, following
> fields will be displayed as N/A.
> active, paused, slave_node, crawl_status, entry, data, metadata
>
> Let me know your thoughts.
>
> --
> regards
> Aravinda
>
>
> On 02/03/2015 11:00 PM, Aravinda wrote:
> > Today we discussed about Geo-rep Status design, summary of the
> > discussion.
> >
> > - No usecase for "Deletes pending" column, should we retain it?
> > - No separate column for Active/Passive. Worker can be Active/Passive
> > only when worker is Stable(It can't be Faulty and Active)
> > - Rename "Not Started" status as "Created"
> > - Checkpoint columns will be retained in the Status output till we
> > support Multiple checkpoints. Three columns instead of Single
> > column(Completed, Checkpoint time and Completion time)
> > - Still we have confusion about "Files Pending" and "Files Synced",
> > What numbers it has to show. Georep can't map the number to exact
> > count on disk.
> > Venky suggested to show Entry, Data and Metadata pending as three
> > columns. (Remove "Files Pending" and "Files Synced")
> > - Rename "Files Skipped" to "Failures"
> >
> > Status output proposed:
> > -----------------------
> > MASTER NODE - Master node hostname/IP
> > MASTER VOL - Master volume name
> > MASTER BRICK - Master brick path
> > SLAVE USER - Slave user to which geo-rep is established.
> > SLAVE - Slave host and Volume name(HOST::VOL format)
> > STATUS - Created/Initializing../Started/Active/Passive/Stopped/Faulty
> > LAST SYNCED - Last synced time(Based on stime xattr)
> > CRAWL STATUS - Hybrid/History/Changelog
> > CHECKPOINT STATUS - Yes/No/ N/A
> > CHECKPOINT TIME - Checkpoint Set Time
> > CHECKPOINT COMPLETED - Checkpoint Completion Time
> >
> > Not yet decided
> > ---------------
> > FILES SYNCD - Number of Files Synced
> > FILES PENDING - Number of Files Pending
> > DELETES PENDING- Number of Deletes Pending
> > FILES SKIPPED - Number of Files skipped
> > ENTRIES - Create/Delete/MKDIR/RENAME etc
> > DATA - Data operations
> > METADATA - SETATTR, SETXATTR etc
> >
> > Let me know your suggestions.
> >
> > --
> > regards
> > Aravinda
> >
> >
> > On 02/02/2015 04:51 PM, Aravinda wrote:
> >> Thanks Sahina, replied inline.
> >>
> >> --
> >> regards
> >> Aravinda
> >>
> >> On 02/02/2015 12:55 PM, Sahina Bose wrote:
> >>>
> >>> On 01/28/2015 04:07 PM, Aravinda wrote:
> >>>> Background
> >>>> ----------
> >>>> We have `status` and `status detail` commands for GlusterFS
> >>>> geo-replication, This mail is to fix the existing issues in these
> >>>> command outputs. Let us know if we need any other columns which
> >>>> helps users to get meaningful status.
> >>>>
> >>>> Existing output
> >>>> ---------------
> >>>> Status command output
> >>>> MASTER NODE - Master node hostname/IP
> >>>> MASTER VOL - Master volume name
> >>>> MASTER BRICK - Master brick path
> >>>> SLAVE - Slave host and Volume name(HOST::VOL format)
> >>>> STATUS - Stable/Faulty/Active/Passive/Stopped/Not Started
> >>>> CHECKPOINT STATUS - Details about Checkpoint completion
> >>>> CRAWL STATUS - Hybrid/History/Changelog
> >>>>
> >>>> Status detail -
> >>>> MASTER NODE - Master node hostname/IP
> >>>> MASTER VOL - Master volume name
> >>>> MASTER BRICK - Master brick path
> >>>> SLAVE - Slave host and Volume name(HOST::VOL format)
> >>>> STATUS - Stable/Faulty/Active/Passive/Stopped/Not Started
> >>>> CHECKPOINT STATUS - Details about Checkpoint completion
> >>>> CRAWL STATUS - Hybrid/History/Changelog
> >>>> FILES SYNCD - Number of Files Synced
> >>>> FILES PENDING - Number of Files Pending
> >>>> BYTES PENDING - Bytes pending
> >>>> DELETES PENDING - Number of Deletes Pending
> >>>> FILES SKIPPED - Number of Files skipped
> >>>>
> >>>>
> >>>> Issues with existing status and status detail:
> >>>> ----------------------------------------------
> >>>>
> >>>> 1. Active/Passive and Stable/faulty status is mixed up - Same
> >>>> column is used to show both active/passive status as well as
> >>>> Stable/faulty status. If Active node goes faulty then by looking at
> >>>> the status it is difficult to understand Active node is faulty or
> >>>> the passive one.
> >>>> 2. Info about last synced time, unless we set checkpoint it is
> >>>> difficult to understand till what time data is synced to slave. For
> >>>> example, if a admin want's to know all the files synced which are
> >>>> created 15 mins ago, it is not possible without setting checkpoint.
> >>>> 3. Wrong values in metrics.
> >>>> 4. When multiple bricks present in same node. Status shows Faulty
> >>>> when one of the worker is faulty in that node.
> >>>>
> >>>> Changes:
> >>>> --------
> >>>> 1. Active nodes will be prefixed with * to identify it is a active
> >>>> node.(In xml output active tag will be introduced with values 0 or 1)
> >>>> 2. New column will show the last synced time, which minimizes the
> >>>> use of checkpoint feature. Checkpoint status will be shown only in
> >>>> status detail.
> >>>> 3. Checkpoint Status is removed, Separate Checkpoint command will
> >>>> be added to gluster cli(We can introduce multiple Checkpoint
> >>>> feature with this change)
> >>>> 4. Status values will be "Not
> >>>> Started/Initializing/Started/Faulty/Stopped". Stable is changed to
> >>>> "Started"
> >>>> 5. Slave User column will be introduced to show to which user
> >>>> geo-rep session is established.(Useful in Non root geo-rep)
> >>>> 6. Bytes pending column will be removed. It is not possible to
> >>>> identify the delta without simulating sync. For example, we are
> >>>> using rsync to sync data from master to slave, If we need to know
> >>>> how much data to be transferred then we have to run the rsync
> >>>> command with --dry-run flag before running actual command. With
> >>>> tar-ssh we have to stat all the files which are identified to be
> >>>> synced to calculate the total bytes to be synced. Both are costly
> >>>> operations which degrades the geo-rep performance.(In Future we can
> >>>> include these columns)
> >>>> 7. Files pending, Synced, deletes pending are only session
> >>>> information of the worker, these numbers will not match with the
> >>>> number of files present in Filesystem. If worker restarts, counter
> >>>> will reset to zero. When worker restarts, it logs previous session
> >>>> stats before resetting it.
> >>>> 8. Files Skipped is persistent status across sessions, Shows exact
> >>>> count of number of files skipped(Can get list of GFIDs skipped from
> >>>> log file)
> >>>> 9. "Deletes Pending" column can be removed?
> >>>
> >>> Is there any way to know if there are errors syncing any of the
> >>> files? Which column would that reflect in?
> >> "Skipped" Column shows number of files failed to sync to Slave.
> >>
> >>> Is the last synced time - the least of the synced time across the
> >>> nodes?
> >> Status output will have one entry for each brick, so we are planning
> >> to display last synced time from that brick.
> >>>
> >>>
> >>>>
> >>>> Example output
> >>>>
> >>>> MASTER NODE MASTER VOL MASTER BRICK SLAVE USER
> >>>> SLAVE STATUS LAST SYNCED CRAWL
> >>>> ----------------------------------------------------------------------------------------------------------------
> >>>>
> >>>> * fedoravm1 gvm /gfs/b1 root fedoravm3::gvs
> >>>> Started 2014-05-10 03:07 pm Changelog
> >>>> fedoravm2 gvm /gfs/b2 root fedoravm4::gvs
> >>>> Started 2014-05-10 03:07 pm Changelog
> >>>>
> >>>> New Status columns
> >>>>
> >>>> ACTIVE_PASSIVE - * if Active else none.
> >>>> MASTER NODE - Master node hostname/IP
> >>>> MASTER VOL - Master volume name
> >>>> MASTER BRICK - Master brick path
> >>>> SLAVE USER - Slave user to which geo-rep is established.
> >>>> SLAVE - Slave host and Volume name(HOST::VOL format)
> >>>> STATUS - Stable/Faulty/Active/Passive/Stopped/Not Started
> >>>> LAST SYNCED - Last synced time(Based on stime xattr)
> >>>> CHECKPOINT STATUS - Details about Checkpoint completion
> >>>> CRAWL STATUS - Hybrid/History/Changelog
> >>>> FILES SYNCD - Number of Files Synced
> >>>> FILES PENDING - Number of Files Pending
> >>>> DELETES PENDING- Number of Deletes Pending
> >>>> FILES SKIPPED - Number of Files skipped
> >>>>
> >>>>
> >>>> XML output
> >>>> active
> >>>> master_node
> >>>> master_node_uuid
> >>>> master_brick
> >>>> slave_user
> >>>> slave
> >>>> status
> >>>> last_synced
> >>>> crawl_status
> >>>> files_syncd
> >>>> files_pending
> >>>> deletes_pending
> >>>> files_skipped
> >>>>
> >>>>
> >>>> Checkpoints
> >>>> ===========
> >>>> New set of Gluster CLI commands will be introduced for Checkpoints.
> >>>>
> >>>> gluster volume geo-replication <VOLNAME>
> >>>> <SLAVEHOST>::<SLAVEVOL> checkpoint create <NAME> <DATE>
> >>>> gluster volume geo-replication <VOLNAME> <SLAVEHOST>::<SLAVEVOL>
> >>>> checkpoint delete <NAME>
> >>>> gluster volume geo-replication <VOLNAME>
> >>>> <SLAVEHOST>::<SLAVEVOL> checkpoint delete all
> >>>> gluster volume geo-replication <VOLNAME>
> >>>> <SLAVEHOST>::<SLAVEVOL> checkpoint status [<NAME>]
> >>>> gluster volume geo-replication <VOLNAME> checkpoint status #
> >>>> For all geo-rep sessions for that volume
> >>>> gluster volume geo-replication checkpoint status # For all
> >>>> geo-rep sessions for all volumes
> >>>>
> >>>>
> >>>> Checkpoint Status:
> >>>>
> >>>> SESSION NAME Completed Checkpoint
> >>>> Time Completion Time
> >>>> -----------------------------------------------------------------------------------------
> >>>>
> >>>> gvm->root at fedoravm3::gvs Chk1 Yes 2014-11-30 11:30 pm
> >>>> 2014-12-01 02:30 pm
> >>>> gvm->root at fedoravm3::gvs Chk2 No 2014-12-01 10:00 pm N/A
> >>>
> >>> Can the time information have the timezone information as well? Or
> >>> is this UTC time?
> >>> (Same comment for last synced time)
> >> Sure. Will have UTC time in Status output.
> >>>
> >>>>
> >>>> XML output:
> >>>> session
> >>>> master_uuid
> >>>> name
> >>>> completed
> >>>> checkpoint_time
> >>>> completion_time
> >>>>
> >>>>
> >>>> --
> >>>> regards
> >>>> Aravinda
> >>>> _______________________________________________
> >>>> Gluster-devel mailing list
> >>>> Gluster-devel at gluster.org
> >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> >>>
> >>
> >> _______________________________________________
> >> Gluster-devel mailing list
> >> Gluster-devel at gluster.org
> >> http://www.gluster.org/mailman/listinfo/gluster-devel
> >
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
More information about the Gluster-devel
mailing list