[Gluster-users] Failed to get fd context for a non-anonymous fd

Tue Dec 18 11:16:54 UTC 2018

Hi again,

We see these type of errors on all our glusterfs servers : Failed to get fd context for a non-anonymous fd, file: (null), gfid

I even took some fresh VM, build from scratch and installed glusterfs 4.1.6 (2 node setup). I also connected 4 clients and loaded some data ( around 600 Mb with random files). Within a couple of hours the same errors in the logs.

Should we simply ignore these errors?

Thanks,

Max

Van: Graaf, Max de
Verzonden: vrijdag 14 december 2018 08:27
Aan: 'Raghavendra Gowdappa'
CC: Gluster-users
Onderwerp: RE: [Gluster-users] Failed to get fd context for a non-anonymous fd

Hi,

So messages like “: Failed to get fd context for a non-anonymous fd, file: (null), gfid: 2aed88da-0664-4070-9202-b856eb1e3c96” we can simply ignore?

Looking at the logs after the updrage this one keeps coming back:
grep null /var/log/glusterfs/bricks/apps-glusterfs-gst.log
2018-12-12 16:16:20.664304] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: d497d8e0-7246-4c97-beed-8d927a2242f1
[2018-12-12 20:25:20.793225] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: d497d8e0-7246-4c97-beed-8d927a2242f1
[2018-12-12 21:10:20.503040] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: d497d8e0-7246-4c97-beed-8d927a2242f1
[2018-12-13 20:41:19.462145] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: d497d8e0-7246-4c97-beed-8d927a2242f1
[2018-12-13 21:34:20.088353] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: d497d8e0-7246-4c97-beed-8d927a2242f1
[2018-12-13 21:56:19.957970] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: d497d8e0-7246-4c97-beed-8d927a2242f1
[2018-12-13 23:35:19.859388] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: d497d8e0-7246-4c97-beed-8d927a2242f1
[2018-12-14 00:45:20.160086] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: d497d8e0-7246-4c97-beed-8d927a2242f1

Tried to find the file with gfid: d497d8e0-7246-4c97-beed-8d927a2242f1 but it looks like it doesn’t exist.

Thanks,

Max

Van: Raghavendra Gowdappa [mailto:rgowdapp at redhat.com]
Verzonden: vrijdag 14 december 2018 07:32
Aan: Graaf, Max de
CC: Gluster-users
Onderwerp: Re: [Gluster-users] Failed to get fd context for a non-anonymous fd

On Fri, Dec 14, 2018 at 11:49 AM <max.degraaf at kpn.com<mailto:max.degraaf at kpn.com>> wrote:
Hi,

We updated our glusterfs servers and clients from 3.10 to 4.1.6. Although all seems to be working just fine we see errors like these on almost all glusterfs servers:

[2018-12-14 02:35:03.524378] I [addr.c:55:compare_addr_and_update] 0-/apps/glusterfs-gst: allowed = "*", received addr = "10.159.240.195"
[2018-12-14 02:35:03.524410] I [login.c:111:gf_auth] 0-auth/login: allowed user names: 9d79471c-ce22-4e48-ba0d-81f297ed5ba1
[2018-12-14 02:35:03.524419] I [MSGID: 115029] [server-handshake.c:495:server_setvolume] 0-gst-server: accepted client from CTX_ID:37e79003-38db-48bf-8408-f7e9feaaa44c-GRAPH_ID:0-PID:4012-HOST:GRPACCAALCGST01-PC_NAME:gst-client-1-RECON_NO:-0 (version: 4.1.6)
[2018-12-14 02:35:03.572494] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gst-server: disconnecting connection from CTX_ID:37e79003-38db-48bf-8408-f7e9feaaa44c-GRAPH_ID:0-PID:4012-HOST:GRPACCAALCGST01-PC_NAME:gst-client-1-RECON_NO:-0
[2018-12-14 02:35:03.572549] I [MSGID: 115013] [server-helpers.c:286:do_fd_cleanup] 0-gst-server: fd cleanup on <gfid:2aed88da-0664-4070-9202-b856eb1e3c96>

A cleanup was initiated for this fd as client got disconnected.

[2018-12-14 02:35:03.572697] I [socket.c:3844:socket_submit_reply] 0-tcp.gst-server: not connected (priv->connected = -1)
[2018-12-14 02:35:03.572718] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x18, Program: GlusterFS 3.3, ProgVers: 330, Proc: 42) to rpc-transport (tcp.gst-server)
[2018-12-14 02:35:03.572807] E [server.c:137:server_submit_reply] (-->/lib64/libgfrpc.so.0(+0x817e) [0x7f022c85717e] -->/usr/lib64/glusterfs/4.1.6/xlator/protocol/server.so(+0x335b9) [0x7f02179665b9] -->/usr/lib64/glusterfs/4.1.6/xlator/protocol/server.so(+0xae2e) [0x7f021793de2e] ) 0-: Reply submission failed
[2018-12-14 02:35:03.572823] E [server.c:144:server_submit_reply] (-->/lib64/libgfrpc.so.0(+0x817e) [0x7f022c85717e] -->/usr/lib64/glusterfs/4.1.6/xlator/protocol/server.so(+0x335b9) [0x7f02179665b9] -->/usr/lib64/glusterfs/4.1.6/xlator/protocol/server.so(+0xae96) [0x7f021793de96] ) 0-: Reply submission failed
[2018-12-14 02:35:03.572877] E [MSGID: 113040] [posix-helpers.c:1671:__posix_fd_ctx_get] 0-gst-posix: Failed to get fd context for a non-anonymous fd, file: (null), gfid: 2aed88da-0664-4070-9202-b856eb1e3c96

This, most likely is an operation racing with the cleanup initiated. Since this condition is triggered by a disconnect, this looks to be benign, unless you've seen any functional issues at application layer.

[2018-12-14 02:35:03.572889] W [MSGID: 113006] [posix-inode-fd-ops.c:2018:posix_flush] 0-gst-posix: pfd is NULL on fd=0x7f01e80092c8 [Invalid argument]
[2018-12-14 02:35:03.573034] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gst-server: Shutting down connection CTX_ID:37e79003-38db-48bf-8408-f7e9feaaa44c-GRAPH_ID:0-PID:4012-HOST:GRPACCAALCGST01-PC_NAME:gst-client-1-RECON_NO:-0
[2018-12-14 02:35:33.750415] I [addr.c:55:compare_addr_and_update] 0-/apps/glusterfs-gst: allowed = "*", received addr = "10.159.240.227"
[2018-12-14 02:35:33.750439] I [login.c:111:gf_auth] 0-auth/login: allowed user names: 9d79471c-ce22-4e48-ba0d-81f297ed5ba1
[2018-12-14 02:35:33.750447] I [MSGID: 115029] [server-handshake.c:495:server_setvolume] 0-gst-server: accepted client from CTX_ID:e76cbd89-2f9e-404e-887a-256783d2fc1f-GRAPH_ID:0-PID:5385-HOST:GRPACCAPDCGST01-PC_NAME:gst-client-1-RECON_NO:-0 (version: 4.1.6)
[2018-12-14 02:35:33.817381] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gst-server: disconnecting connection from CTX_ID:e76cbd89-2f9e-404e-887a-256783d2fc1f-GRAPH_ID:0-PID:5385-HOST:GRPACCAPDCGST01-PC_NAME:gst-client-1-RECON_NO:-0
[2018-12-14 02:35:33.818343] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gst-server: Shutting down connection CTX_ID:e76cbd89-2f9e-404e-887a-256783d2fc1f-GRAPH_ID:0-PID:5385-HOST:GRPACCAPDCGST01-PC_NAME:gst-client-1-RECON_NO:-0

We have 3 separate environments and all 3 have a similar setup with 2 glusterfs servers with 6 to 10 glusterfs clients connected. No heavy load at all, all running on CentOS 7.1 en 7.3.

Any thoughts?

Thanks,

Max

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181218/8882382c/attachment.html>