Home

---

Background

I've recently been developing a maze puzzle for an initiative at work; the puzzle has a web frontend, a couple of REST endpoints, and a websocket endpoint to interact with the maze.

It's been running just fine on my home server as a systemd service, but I have noticed that whenever I update the build and restart the service it's been hanging for around 90 seconds before systemd gets tired of waiting and sends a SIGKILL.

I could replicate the issue locally in Rider (though shutdown was taking significantly less time), but not when running with the debugger attached. The application shut down immediately when there weren't any active websocket connections, but seemed to get stuck after logging Application is shutting down... to the console when there were.

The websocket connections were being stored in a static instance of a ConcurrentDictionary, mapping the id of a maze to a collection of websocket connections. This allowed a user to connect to their maze from their browser, and also connect from their chosen programming language to solve the maze programmatically.

This ConcurentDictionary is referred to as "Topics" as I've reused a load of code I'd written for a Pub/Sub service in the past. Each Topic has a Clients property with the type ConcurrentDictionary<string, WebSocket> that maps UUIDs to each websocket connection.

Fixing my issue

The websocket connection handler has a CancellationToken being passed in, but that appears to only fire when the client disconnects.

After googling around for a solution, I tried adding a new handler to AppDomain.CurrentDomain.ProcessExit, but wasn't able to get the event to fire.

I was also told I could register an IHostedService and cleanup using the StopAsync method, but also found I wasn't able to get it to fire at the right time.

I get the feeling that the active websocket connections might have been preventing these handlers to fire.

In the end, I discovered that IHostApplicationLifetime has a CancellationToken called ApplicationStopping, that I could wait on for the SIGINT or SIGTERM that systemd sends to restart the service.

The end of the main Program.cs now looks like

...

app.UseWebsockets();

var shutdownTask = Task.Run(async () =>
{
    app.Lifetime.ApplicationStopping.WaitHandler.WaitOne();
    await WebsocketController.Topics.Shutdown();
});

app.Run();

await shutdownTask;

The shutdownTask runs before the blocking call to app.Run(), with WaitHandle.WaitOne() blocking the task until SIGINT or SIGTERM has been received. At this point each of the client connections are sent a shutdown error message and are disconnected.

The disconnect code is roughly

public static async Task Shutdown(this ConcurrentDictionary<string, Topic> topics)
{
    var clients = topics.Values.SelectMany(topic => topic.Clients).ToList();
    
    foreach (var (id, ws) in clients)
    {
        try
        {
            await ws.SendMessage(new ErrorModel("Server shutting down"));
            await ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "Connection closed", CancellationToken.None);
            Console.WriteLine($"Disconnected {id}");
        }
        catch
        {
            // client may already have disconnected since calling Shutdown
            // do nothing
        }
    }
}

Shutdown is a static extension method called on the ConcurrentDictionary of topics.

I collect the client connections into a list at the start of the function to avoid iterating through the collection as it's being modified, as the disconnecting clients get removed from the ConcurrentDictionary by the websocket handler as they disconnect. I also wrap each disconnect in a try/catch block to ensure that issues disconnecting one client won't affect disconnecting other clients.

WebSocket.SendMessage<T> is another extension method that serialises a model, converts the json string to bytes, and sends it to over the websocket connection.

public static async Task SendMessage<T>(this WebSocket websocket, T model, CancellationToken token = default)
{
    var json = JsonSerializer.Serialize(model);
    var bytes = Encoding.UTF8.GetBytes(json);
    await websocket.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, token);
}

Conclusion

This solution seems to work pretty well for my use case, and means I can update and restart my service without it hanging for over a minute.

I could set off all the disconnects at once and collect a list of tasks to await with Task.WhenAll, but given this project was for a small audience, it's unlikely to need to disconnect a large number of clients and taking a few seconds to shut down is entirely acceptable.

I'm sure there are more correct ways to do handle the shutdown event too, but given this isn't a production-grade application and wasn't being written for a client I'm happy enough with how it works.

---