<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p><br>
    </p>
    <div class="moz-cite-prefix">On 31/05/19 3:20 AM, Alan Orth wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAKKdN4W-wKsfx-G5Ji=hkLdw9WBqfdMkjceFrwgoibkp072w4A@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div>Dear Ravi,</div>
        <div><br>
        </div>
        <div>I spent a bit of time inspecting the xattrs on some files
          and directories on a few bricks for this volume and it looks a
          bit messy. Even if I could make sense of it for a few and
          potentially heal them manually, there are millions of files
          and directories in total so that's definitely not a scalable
          solution. After a few missteps with `replace-brick ... commit
          force` in the last week—one of which on a brick that was
          dead/offline—as well as some premature `remove-brick`
          commands, I'm unsure how how to proceed and I'm getting
          demotivated. It's scary how quickly things get out of hand in
          distributed systems...<br>
        </div>
      </div>
    </blockquote>
    Hi Alan,<br>
    The one good thing about gluster is it that the data is always
    available directly on the backed bricks even if your volume has
    inconsistencies at the gluster level. So theoretically, if your
    cluster is FUBAR, you could just create a new volume and copy all
    data onto it via its mount from the old volume's bricks.<br>
    <blockquote type="cite"
cite="mid:CAKKdN4W-wKsfx-G5Ji=hkLdw9WBqfdMkjceFrwgoibkp072w4A@mail.gmail.com">
      <div dir="ltr">
        <div><br>
        </div>
        <div>
          <div>I had hoped that bringing the old brick back up would
            help, but by the time I added it again a few days had passed
            and all the brick-id's had changed due to the replace/remove
            brick commands, not to mention that the
            trusted.afr.$volume-client-xx values were now probably
            pointing to the wrong bricks (?).<br>
          </div>
        </div>
        <div><br>
        </div>
        <div>Anyways, a few hours ago I started a full heal on the
          volume and I see that there is a sustained 100MiB/sec of
          network traffic going from the old brick's host to the new
          one. The completed heals reported in the logs look promising
          too:</div>
        <div><br>
        </div>
        <div>Old brick host:</div>
        <div><br>
        </div>
        <div># grep '2019-05-30' /var/log/glusterfs/glustershd.log |
          grep -o -E 'Completed (data|metadata|entry) selfheal' | sort |
          uniq -c<br>
           281614 Completed data selfheal<br>
               84 Completed entry selfheal<br>
           299648 Completed metadata selfheal</div>
        <div><br>
        </div>
        <div>New brick host:<br>
        </div>
        <div><br>
        </div>
        <div># grep '2019-05-30' /var/log/glusterfs/glustershd.log |
          grep -o -E 'Completed (data|metadata|entry) selfheal' | sort |
          uniq -c<br>
           198256 Completed data selfheal<br>
            16829 Completed entry selfheal<br>
           229664 Completed metadata selfheal</div>
        <div><br>
        </div>
        <div>So that's good I guess, though I have no idea how long it
          will take or if it will fix the "missing files" issue on the
          FUSE mount. I've increased cluster.shd-max-threads to 8 to
          hopefully speed up the heal process.<br>
        </div>
      </div>
    </blockquote>
    The afr xattrs should not cause files to disappear from mount. If
    the xattr names do not match what each AFR subvol expects (for eg.
    in a replica 2 volume, trusted.afr.*-client-{0,1} for 1st subvol,
    client-{2,3} for 2nd subvol and so on - ) for its children then it
    won't heal the data, that is all. But in your case I see some
    inconsistencies like one brick having the actual file (<span
      style="font-family:courier new,monospace">licenseserver.cfg) </span>and
    the other having a linkto file (the one with the<span
      style="font-family:courier new,monospace"> dht.linkto xattr)</span>
    <i>in the same replica pair</i>. <br>
    <blockquote type="cite"
cite="mid:CAKKdN4W-wKsfx-G5Ji=hkLdw9WBqfdMkjceFrwgoibkp072w4A@mail.gmail.com">
      <div dir="ltr">
        <div><br>
        </div>
        <div>I'd be happy for any advice or pointers,<br>
        </div>
      </div>
    </blockquote>
    <p>Did you check if the .glusterfs hardlinks/symlinks exist and are
      in order for all bricks?</p>
    <p>-Ravi<br>
    </p>
    <blockquote type="cite"
cite="mid:CAKKdN4W-wKsfx-G5Ji=hkLdw9WBqfdMkjceFrwgoibkp072w4A@mail.gmail.com"><br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Wed, May 29, 2019 at 5:20
          PM Alan Orth &lt;<a href="mailto:alan.orth@gmail.com"
            target="_blank" moz-do-not-send="true">alan.orth@gmail.com</a>&gt;
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div>Dear Ravi,</div>
            <div><br>
            </div>
            <div>Thank you for the link to the blog post series—it is
              very informative and current! If I understand your blog
              post correctly then I think the answer to your previous
              question about pending AFRs is: no, there are no pending
              AFRs. I have identified one file that is a good test case
              to try to understand what happened after I issued the
              `gluster volume replace-brick ... commit force` a few days
              ago and then added the same original brick back to the
              volume later. This is the current state of the replica 2
              distribute/replicate volume:<br>
            </div>
            <div><br>
            </div>
            <div><span style="font-family:courier new,monospace">[root@wingu0
                ~]# gluster volume info apps<br>
                 <br>
                Volume Name: apps<br>
                Type: Distributed-Replicate<br>
                Volume ID: f118d2da-79df-4ee1-919d-53884cd34eda<br>
                Status: Started<br>
                Snapshot Count: 0<br>
                Number of Bricks: 3 x 2 = 6<br>
                Transport-type: tcp<br>
                Bricks:<br>
                Brick1: wingu3:/mnt/gluster/apps<br>
                Brick2: wingu4:/mnt/gluster/apps<br>
                Brick3: wingu05:/data/glusterfs/sdb/apps<br>
                Brick4: wingu06:/data/glusterfs/sdb/apps<br>
                Brick5: wingu0:/mnt/gluster/apps<br>
                Brick6: wingu05:/data/glusterfs/sdc/apps<br>
                Options Reconfigured:<br>
                diagnostics.client-log-level: DEBUG<br>
                storage.health-check-interval: 10<br>
                nfs.disable: on</span></div>
            <div><span style="font-family:courier new,monospace"><br>
              </span></div>
            <div>I checked the xattrs of one file that is missing from
              the volume's FUSE mount (though I can read it if I access
              its full path explicitly), but is present in several of
              the volume's bricks (some with full size, others empty):<span
                style="font-family:courier new,monospace"><br>
              </span></div>
            <div><span style="font-family:courier new,monospace"><br>
              </span></div>
            <div><span style="font-family:courier new,monospace">[root@wingu0
                ~]# getfattr -d -m. -e hex
                /mnt/gluster/apps/clcgenomics/clclicsrv/licenseserver.cfg
              </span></div>
            <div>
              <pre class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-code"><span style="font-family:courier new,monospace">getfattr: Removing leading '/' from absolute path names
# file: mnt/gluster/apps/clcgenomics/clclicsrv/licenseserver.cfg
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.apps-client-3=0x000000000000000000000000
trusted.afr.apps-client-5=0x000000000000000000000000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x0200000000000000585a396f00046e15
trusted.gfid=0x878003a2fb5243b6a0d14d2f8b4306bd

[root@wingu05 ~]# getfattr -d -m. -e hex /data/glusterfs/sdb/apps/clcgenomics/clclicsrv/licenseserver.cfg
getfattr: Removing leading '/' from absolute path names
# file: data/glusterfs/sdb/apps/clcgenomics/clclicsrv/licenseserver.cfg
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x878003a2fb5243b6a0d14d2f8b4306bd
trusted.gfid2path.82586deefbc539c3=0x34666437323861612d356462392d343836382d616232662d6564393031636566333561392f6c6963656e73657365727665722e636667
trusted.glusterfs.dht.linkto=0x617070732d7265706c69636174652d3200

[root@wingu05 ~]# getfattr -d -m. -e hex /data/glusterfs/sdc/apps/clcgenomics/clclicsrv/licenseserver.cfg
getfattr: Removing leading '/' from absolute path names
# file: data/glusterfs/sdc/apps/clcgenomics/clclicsrv/licenseserver.cfg
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x878003a2fb5243b6a0d14d2f8b4306bd
trusted.gfid2path.82586deefbc539c3=0x34666437323861612d356462392d343836382d616232662d6564393031636566333561392f6c6963656e73657365727665722e636667

[root@wingu06 ~]# getfattr -d -m. -e hex /data/glusterfs/sdb/apps/clcgenomics/clclicsrv/licenseserver.cfg
getfattr: Removing leading '/' from absolute path names
# file: data/glusterfs/sdb/apps/clcgenomics/clclicsrv/licenseserver.cfg
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x878003a2fb5243b6a0d14d2f8b4306bd
trusted.gfid2path.82586deefbc539c3=0x34666437323861612d356462392d343836382d616232662d6564393031636566333561392f6c6963656e73657365727665722e636667
trusted.glusterfs.dht.linkto=0x617070732d7265706c69636174652d3200</span></pre>
            </div>
            <div>According to the <font face="courier new,monospace">trusted.afr.apps-client-xx<font
                  face="arial,sans-serif"> xattrs this particular file
                  should be on bricks with id "apps-client-3" and
                  "apps-client-5". It took me a few hours to realize
                  that the brick-id values are recorded in the volume's
                  volfiles in /var/lib/glusterd/vols/apps/bricks. After
                  comparing those brick-id values with a volfile backup
                  from before the replace-brick, I realized that the
                  files are simply on the wrong brick now as far as
                  Gluster is concerned. </font></font><font
                face="courier new,monospace"><font
                  face="arial,sans-serif">This particular file is now on
                  the brick for "apps-client-</font></font>4". As an
              experiment I copied this one file to the two bricks listed
              in the xattrs and I was then able to see the file from the
              FUSE mount (yay!).<br>
            </div>
            <div><br>
            </div>
            <div>Other than replacing the brick, removing it, and then
              adding the old brick on the original server back, there
              has been no change in the data this entire time. Can I
              change the brick IDs in the volfiles so they reflect where
              the data actually is? Or perhaps script something to reset
              all the xattrs on the files/directories to point to the
              correct bricks?<br>
            </div>
            <div><br>
            </div>
            <div>Thank you for any help or pointers,<br>
            </div>
          </div>
          <br>
          <div class="gmail_quote">
            <div dir="ltr" class="gmail_attr">On Wed, May 29, 2019 at
              7:24 AM Ravishankar N &lt;<a
                href="mailto:ravishankar@redhat.com" target="_blank"
                moz-do-not-send="true">ravishankar@redhat.com</a>&gt;
              wrote:<br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <p><br>
                </p>
                <div
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-cite-prefix">On
                  29/05/19 9:50 AM, Ravishankar N wrote:<br>
                </div>
                <blockquote type="cite">
                  <p><br>
                  </p>
                  <div
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-cite-prefix">On
                    29/05/19 3:59 AM, Alan Orth wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div>Dear Ravishankar,</div>
                      <div><br>
                      </div>
                      <div>I'm not sure if Brick4 had pending AFRs
                        because I don't know what that means and it's
                        been a few days so I am not sure I would be able
                        to find that information.</div>
                    </div>
                  </blockquote>
                  When you find some time, have a look at a <a
                    href="http://wp.me/peiBB-6b" target="_blank"
                    moz-do-not-send="true">blog</a> series I wrote about
                  AFR- I've tried to explain what one needs to know to
                  debug replication related issues in it.<br>
                </blockquote>
                <p>Made a typo error. The URL for the blog is <a
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-txt-link-freetext"
                    href="https://wp.me/peiBB-6b" target="_blank"
                    moz-do-not-send="true">https://wp.me/peiBB-6b</a></p>
                <p>-Ravi<br>
                </p>
                <blockquote type="cite">
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div><br>
                      </div>
                      <div>Anyways, after wasting a few days rsyncing
                        the old brick to a new host I decided to just
                        try to add the old brick back into the volume
                        instead of bringing it up on the new host. I
                        created a new brick directory on the old host,
                        moved the old brick's contents into that new
                        directory (minus the .glusterfs directory),
                        added the new brick to the volume, and then did
                        Vlad's find/stat trick¹ from the brick to the
                        FUSE mount point.</div>
                      <div><br>
                      </div>
                      <div>The interesting problem I have now is that
                        some files don't appear in the FUSE mount's
                        directory listings, but I can actually list them
                        directly and even read them. What could cause
                        that?<br>
                      </div>
                    </div>
                  </blockquote>
                  Not sure, too many variables in the hacks that you did
                  to take a guess. You can check if the contents of the
                  .glusterfs folder are in order on the new brick
                  (example hardlink for files and symlinks for
                  directories are present etc.) .<br>
                  Regards,<br>
                  Ravi<br>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div><br>
                      </div>
                      <div>Thanks,<br>
                      </div>
                      <div><br>
                      </div>
                      <div>¹ <a
href="https://lists.gluster.org/pipermail/gluster-users/2018-February/033584.html"
                          target="_blank" moz-do-not-send="true">https://lists.gluster.org/pipermail/gluster-users/2018-February/033584.html</a></div>
                    </div>
                    <br>
                    <div class="gmail_quote">
                      <div dir="ltr" class="gmail_attr">On Fri, May 24,
                        2019 at 4:59 PM Ravishankar N &lt;<a
                          href="mailto:ravishankar@redhat.com"
                          target="_blank" moz-do-not-send="true">ravishankar@redhat.com</a>&gt;
                        wrote:<br>
                      </div>
                      <blockquote class="gmail_quote" style="margin:0px
                        0px 0px 0.8ex;border-left:1px solid
                        rgb(204,204,204);padding-left:1ex">
                        <div bgcolor="#FFFFFF">
                          <p><br>
                          </p>
                          <div
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829moz-cite-prefix">On
                            23/05/19 2:40 AM, Alan Orth wrote:<br>
                          </div>
                          <blockquote type="cite">
                            <div dir="ltr">Dear list,
                              <div><br>
                              </div>
                              <div>I seem to have gotten into a tricky
                                situation. Today I brought up a shiny
                                new server with new disk arrays and
                                attempted to replace one brick of a
                                replica 2 distribute/replicate volume on
                                an older server using the
                                `replace-brick` command:</div>
                              <div><br>
                              </div>
                              <div># gluster volume replace-brick homes
                                wingu0:/mnt/gluster/homes
                                wingu06:/data/glusterfs/sdb/homes commit
                                force</div>
                              <div><br>
                              </div>
                              <div>The command was successful and I see
                                the new brick in the output of `gluster
                                volume info`. The problem is that
                                Gluster doesn't seem to be migrating the
                                data, </div>
                            </div>
                          </blockquote>
                          <p>`replace-brick` definitely must heal (not
                            migrate) the data. In your case, data must
                            have been healed from Brick-4 to the
                            replaced Brick-3. Are there any errors in
                            the self-heal daemon logs of Brick-4's node?
                            Does Brick-4 have pending AFR xattrs blaming
                            Brick-3? The doc is a bit out of date.
                            replace-brick command internally does all
                            the setfattr steps that are mentioned in the
                            doc.</p>
                          <p>-Ravi<br>
                          </p>
                          <p><br>
                          </p>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div>and now the original brick that I
                                replaced is no longer part of the volume
                                (and a few terabytes of data are just
                                sitting on the old brick):</div>
                              <div><br>
                              </div>
                              <div># gluster volume info homes | grep -E
                                "Brick[0-9]:"<br>
                                Brick1: wingu4:/mnt/gluster/homes<br>
                                Brick2: wingu3:/mnt/gluster/homes<br>
                                Brick3:
                                wingu06:/data/glusterfs/sdb/homes<br>
                                Brick4:
                                wingu05:/data/glusterfs/sdb/homes<br>
                                Brick5:
                                wingu05:/data/glusterfs/sdc/homes<br>
                                Brick6:
                                wingu06:/data/glusterfs/sdc/homes</div>
                              <br>
                              <div>I see the Gluster docs have a more
                                complicated procedure for replacing
                                bricks that involves getfattr/setfattr¹.
                                How can I tell Gluster about the old
                                brick? I see that I have a backup of the
                                old volfile thanks to yum's rpmsave
                                function if that helps.<br>
                              </div>
                              <div><br>
                              </div>
                              <div>We are using Gluster 5.6 on CentOS 7.
                                Thank you for any advice you can give.<br>
                              </div>
                              <div><br>
                              </div>
                              <div>¹ <a
href="https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-faulty-brick"
                                  target="_blank" moz-do-not-send="true">https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-faulty-brick</a></div>
                              <div><br>
                              </div>
                              <div>-- <br>
                                <div dir="ltr"
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829gmail_signature">Alan
                                  Orth<br>
                                  <a href="mailto:alan.orth@gmail.com"
                                    target="_blank"
                                    moz-do-not-send="true">alan.orth@gmail.com</a><br>
                                  <a href="https://picturingjordan.com"
                                    target="_blank"
                                    moz-do-not-send="true">https://picturingjordan.com</a><br>
                                  <a href="https://englishbulgaria.net"
                                    target="_blank"
                                    moz-do-not-send="true">https://englishbulgaria.net</a><br>
                                  <a href="https://mjanja.ch"
                                    target="_blank"
                                    moz-do-not-send="true">https://mjanja.ch</a><br>
                                  "In heaven all the interesting people
                                  are missing." ―Friedrich Nietzsche</div>
                              </div>
                            </div>
                            <br>
                            <fieldset
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829mimeAttachmentHeader"></fieldset>
                            <pre class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail-m_-6098182312870478829moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
                          </blockquote>
                        </div>
                      </blockquote>
                    </div>
                    <br clear="all">
                    <br>
                    -- <br>
                    <div dir="ltr"
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754gmail_signature">Alan
                      Orth<br>
                      <a href="mailto:alan.orth@gmail.com"
                        target="_blank" moz-do-not-send="true">alan.orth@gmail.com</a><br>
                      <a href="https://picturingjordan.com"
                        target="_blank" moz-do-not-send="true">https://picturingjordan.com</a><br>
                      <a href="https://englishbulgaria.net"
                        target="_blank" moz-do-not-send="true">https://englishbulgaria.net</a><br>
                      <a href="https://mjanja.ch" target="_blank"
                        moz-do-not-send="true">https://mjanja.ch</a><br>
                      "In heaven all the interesting people are
                      missing." ―Friedrich Nietzsche</div>
                  </blockquote>
                  <br>
                  <fieldset
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754mimeAttachmentHeader"></fieldset>
                  <pre class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail-m_-8560298453182323754moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
                </blockquote>
              </div>
            </blockquote>
          </div>
          <br clear="all">
          <br>
          -- <br>
          <div dir="ltr"
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail-m_4641490506639946635gmail_signature">Alan
            Orth<br>
            <a href="mailto:alan.orth@gmail.com" target="_blank"
              moz-do-not-send="true">alan.orth@gmail.com</a><br>
            <a href="https://picturingjordan.com" target="_blank"
              moz-do-not-send="true">https://picturingjordan.com</a><br>
            <a href="https://englishbulgaria.net" target="_blank"
              moz-do-not-send="true">https://englishbulgaria.net</a><br>
            <a href="https://mjanja.ch" target="_blank"
              moz-do-not-send="true">https://mjanja.ch</a><br>
            "In heaven all the interesting people are missing."
            ―Friedrich Nietzsche</div>
        </blockquote>
      </div>
      <br clear="all">
      <br>
      -- <br>
      <div dir="ltr"
class="gmail-m_6037636534267754189gmail-m_7953209464412382574gmail-m_-7848846245795047948gmail_signature">Alan
        Orth<br>
        <a href="mailto:alan.orth@gmail.com" target="_blank"
          moz-do-not-send="true">alan.orth@gmail.com</a><br>
        <a href="https://picturingjordan.com" target="_blank"
          moz-do-not-send="true">https://picturingjordan.com</a><br>
        <a href="https://englishbulgaria.net" target="_blank"
          moz-do-not-send="true">https://englishbulgaria.net</a><br>
        <a href="https://mjanja.ch" target="_blank"
          moz-do-not-send="true">https://mjanja.ch</a><br>
        "In heaven all the interesting people are missing." ―Friedrich
        Nietzsche</div>
    </blockquote>
  </body>
</html>