<div dir="ltr">Hi Amar,<div><br></div><div>Unfortunately I do not have the GlusterFS brick logs anymore - however I do have a hint:</div><div>I have 2 gluster (4.1.1) glusterfs volumes where I saw the issue - each has about 10-12 VMs active.</div><div>I also have 2 addl. gluster (4.1.1) glusterfs volumes, but with only 3-4 VMs, where I did not see the</div><div>issue (and they had been running for 1-2 months).</div><div><br></div><div>Thanx,</div><div><br></div><div>Claus.</div><div><br></div><div>P.S. We are talking about using Gluster "URI" with qemu - I hope - e.g. like</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)"> <disk type='network' device='disk'>
</span><br> <driver name='qemu' type='raw' cache='none' io='native'/>
<br> <source protocol='gluster' name='glu-vol03-lab/install3'>
<br> <host name='install2.vlan13' port='24007'/>
<br> </source>
<br> <target dev='vda' bus='virtio'/>
<br> </disk><br></span></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Aug 20, 2018 at 5:39 PM Amar Tumballi <<a href="mailto:atumball@redhat.com">atumball@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 20, 2018 at 6:20 PM, Walter Deignan <span dir="ltr"><<a href="mailto:WDeignan@uline.com" target="_blank">WDeignan@uline.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span style="font-size:12pt;font-family:Arial">I upgraded late last week
to 4.1.2. Since then I've seen several posix health checks fail and bricks
drop offline but I'm not sure if that's related or a different root issue.</span>
<br>
<br><span style="font-size:12pt;font-family:Arial">I haven't seen the
issue described below re-occur on 4.1.2 yet but it was intermittent to
begin with so I'll probably need to run for a week or more to be confident.</span>
<br><span>
<br></span></blockquote><div><br></div><div>Thanks for the update! We will be trying to reproduce the issue, and also root cause based on analysis of code, but if you get us brick logs around the time this happens, it may fasttrack the issue.</div><div><br></div><div>Thanks again,</div><div>Amar</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><span style="font-size:12pt;font-family:Arial">-Walter Deignan<br>
-Uline IT, Systems Architect</span>
<br>
<br>
<br>
<br></span><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style="font-size:9pt;font-family:sans-serif">"Claus
Jeppesen" <<a href="mailto:cjeppesen@datto.com" target="_blank">cjeppesen@datto.com</a>></span>
<br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:
</span><span style="font-size:9pt;font-family:sans-serif"><a href="mailto:WDeignan@uline.com" target="_blank">WDeignan@uline.com</a></span>
<br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc:
</span><span style="font-size:9pt;font-family:sans-serif"><a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a></span>
<br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:
</span><span style="font-size:9pt;font-family:sans-serif">08/20/2018
07:20 AM</span>
<br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style="font-size:9pt;font-family:sans-serif">Re:
[Gluster-users] KVM lockups on Gluster 4.1.1</span>
<br>
<hr noshade>
<br>
<br>
<br><span style="font-size:12pt">I think I have seen this also on our
CentOS 7.5 systems using GlusterFS 4.1.1 (*) - has an upgrade to 4.1.2
helped out ? I'm trying this now.</span>
<br>
<br><span style="font-size:12pt">Thanx,</span>
<br>
<br><span style="font-size:12pt">Claus.</span>
<br>
<br><span style="font-size:12pt">(*) libvirt/quemu log:</span>
<br><tt><span style="font-size:12pt">[2018-08-19 16:45:54.275830] E [MSGID:
114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 0-glu-vol01-lab-client-0:
remote operation failed [Invalid argument] <br>
[2018-08-19 16:45:54.276156] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-1: remote operation failed [Invalid argument] <br>
[2018-08-19 16:45:54.276159] E [MSGID: 108010] [afr-lk-common.c:284:afr_unlock_inodelk_cbk]
0-glu-vol01-lab-replicate-0: path=(null) gfid=00000000-0000-0000-0000-000000000000:
unlock failed on subvolume glu-vol<br>
01-lab-client-0 with lock owner 28ae497049560000 [Invalid argument] <br>
[2018-08-19 16:45:54.276183] E [MSGID: 108010] [afr-lk-common.c:284:afr_unlock_inodelk_cbk]
0-glu-vol01-lab-replicate-0: path=(null) gfid=00000000-0000-0000-0000-000000000000:
unlock failed on subvolume glu-vol<br>
01-lab-client-1 with lock owner 28ae497049560000 [Invalid argument] <br>
[2018-08-19 17:16:03.690808] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-0:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x3071a5
sent = 2018-08-19 16:45:54.276560. timeout = 1800 for</span></tt><tt><span style="font-size:12pt;color:blue"><u><br>
</u></span></tt><a href="http://192.168.13.131:49152/" target="_blank"><tt><span style="font-size:12pt;color:blue"><u>192.168.13.131:49152</u></span></tt></a><tt><span style="font-size:12pt">
<br>
[2018-08-19 17:16:03.691113] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-0: remote operation failed [Transport endpoint is
not connected] <br>
[2018-08-19 17:46:03.855909] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-1:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x301d0f
sent = 2018-08-19 17:16:03.691174. timeout = 1800 for</span></tt><tt><span style="font-size:12pt;color:blue"><u><br>
</u></span></tt><a href="http://192.168.13.132:49152/" target="_blank"><tt><span style="font-size:12pt;color:blue"><u>192.168.13.132:49152</u></span></tt></a><tt><span style="font-size:12pt">
<br>
[2018-08-19 17:46:03.856170] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-1: remote operation failed [Transport endpoint is
not connected] <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
... many repeats ... <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
[2018-08-19 18:16:04.022526] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-0:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x307221
sent = 2018-08-19 17:46:03.861005. timeout = 1800 for</span></tt><tt><span style="font-size:12pt;color:blue"><u><br>
</u></span></tt><a href="http://192.168.13.131:49152/" target="_blank"><tt><span style="font-size:12pt;color:blue"><u>192.168.13.131:49152</u></span></tt></a><tt><span style="font-size:12pt">
<br>
[2018-08-19 18:16:04.022788] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-0: remote operation failed [Transport endpoint is
not connected] <br>
[2018-08-19 18:46:04.195590] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-1:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x301d8a
sent = 2018-08-19 18:16:04.022838. timeout = 1800 for</span></tt><tt><span style="font-size:12pt;color:blue"><u><br>
</u></span></tt><a href="http://192.168.13.132:49152/" target="_blank"><tt><span style="font-size:12pt;color:blue"><u>192.168.13.132:49152</u></span></tt></a><tt><span style="font-size:12pt">
<br>
[2018-08-19 18:46:04.195881] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-1: remote operation failed [Transport endpoint is
not connected] <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
qemu: terminating on signal 15 from pid 507 <br>
2018-08-19 19:36:59.065+0000: shutting down, reason=destroyed <br>
2018-08-19 19:37:08.059+0000: starting up libvirt version: 3.9.0, package:
14.el7_5.6 (CentOS BuildSystem <</span></tt><a href="http://bugs.centos.org/" target="_blank"><tt><span style="font-size:12pt;color:blue"><u>http://bugs.centos.org</u></span></tt></a><tt><span style="font-size:12pt">>,
2018-06-27-14:13:57, </span></tt><a href="http://x86-01.bsys.centos.org/" target="_blank"><tt><span style="font-size:12pt;color:blue"><u>x86-01.bsys.centos.org</u></span></tt></a><tt><span style="font-size:12pt">),
qemu version: 1.5.3 (qemu-kvm-1.<br>
5.3-156.el7_5.3)</span></tt>
<br>
<br><tt><span style="font-size:12pt">At 19:37 the VM was restarted.</span></tt>
<br>
<br>
<br>
<br><span style="font-size:12pt">On Wed, Aug 15, 2018 at 8:25 PM Walter
Deignan <</span><a href="mailto:WDeignan@uline.com" target="_blank"><span style="font-size:12pt;color:blue"><u>WDeignan@uline.com</u></span></a><span style="font-size:12pt">>
wrote:</span>
<br><span style="font-size:12pt;font-family:Arial">I am using gluster
to host KVM/QEMU images. I am seeing an intermittent issue where access
to an image will hang. I have to do a lazy dismount of the gluster volume
in order to break the lock and then reset the impacted virtual machine.</span><span style="font-size:12pt">
<br>
</span><span style="font-size:12pt;font-family:Arial"><br>
It happened again today and I caught the events below in the client side
logs. Any thoughts on what might cause this? It seemed to begin after I
upgraded from 3.12.10 to 4.1.1 a few weeks ago.</span><span style="font-size:12pt">
<br>
</span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:22:15.549501] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
2-gv1-client-4: remote operation failed [Invalid argument]</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:22:15.549576] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
2-gv1-client-5: remote operation failed [Invalid argument]</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:22:15.549583] E [MSGID: 108010] [afr-lk-common.c:284:afr_unlock_inodelk_cbk]
2-gv1-replicate-2: path=(null) gfid=00000000-0000-0000-0000-000000000000:
unlock failed on subvolume gv1-client-4 with lock owner d89caca92b7f0000
[Invalid argument]</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:22:15.549615] E [MSGID: 108010] [afr-lk-common.c:284:afr_unlock_inodelk_cbk]
2-gv1-replicate-2: path=(null) gfid=00000000-0000-0000-0000-000000000000:
unlock failed on subvolume gv1-client-5 with lock owner d89caca92b7f0000
[Invalid argument]</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:52:18.726219] E [rpc-clnt.c:184:call_bail] 2-gv1-client-4:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0xc5e00
sent = 2018-08-14 14:22:15.699082. timeout = 1800 for </span><a href="http://10.35.20.106:49159/" target="_blank"><span style="font-size:12pt;color:blue;font-family:Arial"><u>10.35.20.106:49159</u></span></a><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:52:18.726254] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
2-gv1-client-4: remote operation failed [Transport endpoint is not connected]</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 15:22:25.962546] E [rpc-clnt.c:184:call_bail] 2-gv1-client-5:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0xc4a6d
sent = 2018-08-14 14:52:18.726329. timeout = 1800 for </span><a href="http://10.35.20.107:49164/" target="_blank"><span style="font-size:12pt;color:blue;font-family:Arial"><u>10.35.20.107:49164</u></span></a><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 15:22:25.962587] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
2-gv1-client-5: remote operation failed [Transport endpoint is not connected]</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 15:22:25.962618] W [MSGID: 108019] [afr-lk-common.c:601:is_blocking_locks_count_sufficient]
2-gv1-replicate-2: Unable to obtain blocking inode lock on even one child
for gfid:24a48cae-53fe-4634-8fb7-0254c85ad672.</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
[2018-08-14 15:22:25.962668] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse:
3715808: FSYNC() ERR => -1 (Transport endpoint is not connected)</span><span style="font-size:12pt">
<br>
</span><span style="font-size:12pt;font-family:Arial"><br>
Volume configuration -</span><span style="font-size:12pt"> <br>
</span><span style="font-size:12pt;font-family:Arial"><br>
Volume Name: gv1</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
Type: Distributed-Replicate</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
Volume ID: 66ad703e-3bae-4e79-a0b7-29ea38e8fcfc</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Status: Started</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
Snapshot Count: 0</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
Number of Bricks: 5 x 2 = 10</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
Transport-type: tcp</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
Bricks:</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
Brick1: dc-vihi44:/gluster/bricks/megabrick/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick2: dc-vihi45:/gluster/bricks/megabrick/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick3: dc-vihi44:/gluster/bricks/brick1/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick4: dc-vihi45:/gluster/bricks/brick1/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick5: dc-vihi44:/gluster/bricks/brick2_1/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick6: dc-vihi45:/gluster/bricks/brick2/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick7: dc-vihi44:/gluster/bricks/brick3/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick8: dc-vihi45:/gluster/bricks/brick3/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick9: dc-vihi44:/gluster/bricks/brick4/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Brick10: dc-vihi45:/gluster/bricks/brick4/data</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
Options Reconfigured:</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
cluster.min-free-inodes: 6%</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
performance.client-io-threads: off</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
nfs.disable: on</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
transport.address-family: inet</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
performance.quick-read: off</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
performance.read-ahead: off</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
performance.io-cache: off</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
performance.low-prio-threads: 32</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
network.remote-dio: enable</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
cluster.eager-lock: enable</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
cluster.server-quorum-type: server</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
cluster.data-self-heal-algorithm: full</span><span style="font-size:12pt">
</span><span style="font-size:12pt;font-family:Arial"><br>
cluster.locking-scheme: granular</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
cluster.shd-max-threads: 8</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
cluster.shd-wait-qlength: 10000</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
user.cifs: off</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
cluster.choose-local: off</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
features.shard: on</span><span style="font-size:12pt"> </span><span style="font-size:12pt;font-family:Arial"><br>
cluster.server-quorum-ratio: 51%</span><span style="font-size:12pt"> <br>
</span><span style="font-size:12pt;font-family:Arial"><br>
-Walter Deignan<br>
-Uline IT, Systems Architect</span><span style="font-size:12pt">_______________________________________________<br>
Gluster-users mailing list</span><span style="font-size:12pt;color:blue"><u><br>
</u></span><a href="mailto:Gluster-users@gluster.org" target="_blank"><span style="font-size:12pt;color:blue"><u>Gluster-users@gluster.org</u></span></a><span style="font-size:12pt;color:blue"><u><br>
</u></span><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank"><span style="font-size:12pt;color:blue"><u>https://lists.gluster.org/mailman/listinfo/gluster-users</u></span></a>
<br>
<br>
<br><span style="font-size:12pt">-- </span>
<table width="650" style="border-collapse:collapse">
<tbody><tr height="8">
<td width="650" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><span style="font-size:11pt;color:#2f2f2f;font-family:Arial"><b>Claus
Jeppesen</b></span>
</td></tr><tr height="8">
<td width="650" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><span style="font-size:11pt;color:#2f2f2f;font-family:Arial">Manager,
Network Services</span>
</td></tr><tr height="8">
<td width="650" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><span style="font-size:11pt;color:#2f2f2f;font-family:Arial">Datto,
Inc.</span>
</td></tr><tr height="8">
<td width="650" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><span style="font-size:11pt;color:#2f2f2f;font-family:Arial">p
+45 6170 5901 | Copenhagen Office</span>
</td></tr><tr height="8">
<td width="650" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><a href="http://www.datto.com/" target="_blank"><span style="font-size:11pt;color:#0060a0;font-family:Arial"><u>www.datto.com</u></span></a>
</td></tr><tr height="8">
<td width="650" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px"><span style="font-size:11pt;color:#0082bf;font-family:Arial"><u><br>
</u></span><a href="http://www.datto.com/datto-signature/" target="_blank"><img src="http://pages.datto.com/rs/572-ZRG-001/images/generic-sig.png" width="400" height="160" style="border:0px solid"></a></td></tr></tbody></table>
<p style="margin-top:0px;margin-Bottom:0px"></p>
<p style="margin-top:0px;margin-Bottom:0px"></p>
<p style="margin-top:0px;margin-Bottom:0px"></p><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_-2032326994069252517gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0" width="650" style="color:rgb(51,51,51);font-family:DattoDIN,Arial,sans-serif;font-size:14px"><tbody><tr><td height="17" width="270"><b>Claus Jeppesen</b></td></tr><tr><td height="16">Manager, Network Services<br></td></tr><tr><td height="16">Datto, Inc.</td></tr><tr><td height="16">p +45 6170 5901 | Copenhagen Office<br></td></tr><tr><td height="16"><a href="http://www.datto.com/" style="color:rgb(0,128,198)" target="_blank"><span style="color:rgb(11,83,148)">www.datto.com</span></a></td></tr><tr><td height="16"><br><a href="http://www.datto.com/datto-signature/" style="color:rgb(0,128,198)" target="_blank"><img alt="" src="http://pages.datto.com/rs/572-ZRG-001/images/generic-sig.png" style="margin:0px;padding:0px;border:0px"></a></td></tr></tbody></table></div></div>