The Belt-and-Suspenders Check That Deadlocked My Nvim

A safety re-check I added to my nvim config ended up deadlocking every nvim launch in my terminal. Because the frozen process held a Unix socket, it cascaded into freezing the next launches too. The trigger was one commit; the symptom looked like “nvim and kitty-scrollback got slow”.

The Problem

The setup: I click hyperlinks in kitty and they open the file in the nvim instance that belongs to that kitty window. The glue is an RPC socket named after kitty’s PID, registered from init.lua, plus a small script (kitty-scrollback.nvim launches nvim the same way for scrollback popups).

One commit hardened the registration: if the socket file exists, verify it’s actually live before trusting it, and re-run the registration at VimEnter and on a deferred callback as belt-and-suspenders. That’s where the bug came in:

-- ~/.config/nvim/init.lua (simplified)
local function socket_is_live(sock)
  -- spawns a CHILD nvim that connects to the socket and waits for a reply
  local result = vim.system(
    { vim.v.progpath, "--server", sock, "--remote-expr", "1" },
    { text = true }
  ):wait() -- blocking
  return result and result.code == 0
end

local function register_kitty_socket()
  if vim.uv.fs_stat(kitty_socket) then
    if socket_is_live(kitty_socket) then
      return
    end
    pcall(vim.fn.delete, kitty_socket)
  end
  pcall(vim.fn.serverstart, kitty_socket)
end

register_kitty_socket() -- 1st call: no socket -> this nvim now OWNS it
vim.api.nvim_create_autocmd("VimEnter", { once = true, callback = register_kitty_socket })
vim.defer_fn(register_kitty_socket, 0) -- 2nd call: socket exists... because this instance owns it

The first call finds no socket and claims it via serverstart(). The belt-and-suspenders re-run at VimEnter finds the socket file, owned by this same instance, and probes its liveness. The probe is a child nvim doing --remote-expr, which connects to the socket and waits for a reply:

parent nvim ── blocked in :wait() for the child to finish
     ▲                                  │
     └──────── RPC reply never comes ───┘
child nvim  ── connected to parent's socket, waiting for reply

The parent can’t service the RPC because it’s blocked in :wait(). The child can’t finish because it’s waiting for the parent. Deadlock: both processes hang forever.

Worse, it cascades. The frozen nvim still owns the socket. The next nvim launch probes that socket, the frozen owner never replies, and that launch hangs too. From the outside: nvim and kitty-scrollback got “slow”, and orphaned nvim processes piled up. Each held language-server children that never get shut down, because VimLeave never fires for a process that never exits.

What Changed

Two guards. First: never probe a socket you own. serverlist() exposes this directly:

-- ~/.config/nvim/init.lua
local function register_kitty_socket()
  -- If this instance already owns the socket there is nothing to do.
  -- Probing our own server deadlocks: the spawned probe nvim waits for
  -- an RPC reply that the parent, blocked in :wait(), can never service.
  if vim.tbl_contains(vim.fn.serverlist(), kitty_socket) then
    return
  end

  if vim.uv.fs_stat(kitty_socket) then
    if socket_is_live(kitty_socket) then
      return
    end
    pcall(vim.fn.delete, kitty_socket)
  end

  pcall(vim.fn.serverstart, kitty_socket)
end

Second: bound the probe so even a frozen foreign owner cannot hang the launch. vim.system()’s :wait(timeout) kills the probe after the timeout:

local result = vim.system(
  { vim.v.progpath, "--server", sock, "--remote-expr", "1" },
  { text = true }
):wait(2000) -- frozen owner costs 2s max, never a hang

The shell wrapper had the same unbounded probe, and macOS ships no timeout(1), so there it’s a watchdog race:

# ~/.local/bin/kitty-open-nvim
"$nvim_bin" --server "$candidate" --remote-expr '1' >/dev/null 2>&1 &
local probe=$!
( sleep 2 && kill -KILL "$probe" ) >/dev/null 2>&1 & # watchdog
local watchdog=$!
wait "$probe" 2>/dev/null || rc=$?
kill -KILL "$watchdog" 2>/dev/null
wait "$watchdog" 2>/dev/null

A probe killed by the watchdog returns 137; the retry loop treats that as “owner is frozen, stop retrying” and falls back to opening a visible tab instead of burning 20 × 2s on a dead-on-arrival socket.

Results

The reproduction was one line: nvim hung indefinitely with KITTY_PID set and exited in 43ms without it.

The kitty-scrollback popups launch instantly again.

Two things I got wrong along the way, for the record. I initially chased the orphaned processes as the cause when they were a symptom, and I misread nvim’s --startuptime log as seconds when it’s milliseconds, which sent me hunting for a filesystem latency problem that didn’t exist. The log said 37ms; I read 37s. Checking units before building a theory would have saved an hour.

References

• Neovim remote docs (--server, --remote-expr, serverstart()): the RPC mechanism the probe and registration use
• vim.system() / SystemObj:wait(timeout): the blocking call at the center of the deadlock, and the timeout that bounds it
• kitty-scrollback.nvim: its launcher loads my config, so it inherited the freeze
• kitty open actions: the hyperlink-click feature the socket exists to support