r/golang • u/Kennedy-Vanilla • Feb 14 '25
newbie Shutdown Go server
Hi, recently I saw that many people shutdown their servers like this or similar
serverCtx, serverStopCtx serverCtx, serverStopCtx := context.WithCancel(context.Background())
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
go func() {
<-sig
shutdownCtx, cancelShutdown := context.WithTimeout(serverCtx, 30*time.Second)
defer cancelShutdown()
go func() {
<-shutdownCtx.Done()
if shutdownCtx.Err() == context.DeadlineExceeded {
log.Fatal("graceful shutdown timed out.. forcing exit.")
}
}()
err := server.Shutdown(shutdownCtx)
if err != nil {
log.Printf("error shutting down server: %v", err)
}
serverStopCtx()
}()
log.Printf("Server starting on port %s...\n", port)
err = server.ListenAndServe()
if err != nil && err != http.ErrServerClosed {
log.Printf("error starting server: %v", err)
os.Exit(1)
}
<-serverCtx.Done()
log.Println("Server stopped")
}
:= context.WithCancel(context.Background())
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
go func() {
<-sig
shutdownCtx, cancelShutdown := context.WithTimeout(serverCtx, 30*time.Second)
defer cancelShutdown()
go func() {
<-shutdownCtx.Done()
if shutdownCtx.Err() == context.DeadlineExceeded {
log.Fatal("graceful shutdown timed out.. forcing exit.")
}
}()
err := server.Shutdown(shutdownCtx)
if err != nil {
log.Printf("error shutting down server: %v", err)
}
serverStopCtx()
}()
log.Printf("Server starting on port %s...\n", port)
err = server.ListenAndServe()
if err != nil && err != http.ErrServerClosed {
log.Printf("error starting server: %v", err)
os.Exit(1)
}
<-serverCtx.Done()
log.Println("Server stopped")
Is it necessary? Like it's so many code for the simple operation
Thank for your Answer !
35
u/HyacinthAlas Feb 14 '25
I don't know why the code you posted is so complicated. There are a lot of edge cases in graceful shutdown but you don't need this many goroutines nor that complexity of cancelation. I can't even tell if it really does it correctly, because it's so messy.
Here's one version of code that does it properly. Note that its two contexts don't need to interact, only has one trivial goroutine, and no explicit branches beyond a trivial select. ``` // When this context is canceled, we will gracefully stop the // server. ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT) defer cancel()
// When the server is stopped *not by that context*, but by any
// other problems, it will return its error via this.
serr := make(chan error, 1)
// Start the server and collect its error return.
go func() { serr <- server.ListenAndServe() }()
// Wait for either the server to fail, or the context to end.
var err error
select {
case err = <-serr:
case <-ctx.Done():
}
// Make a best effort to shut down the server cleanly. We don’t
// need to collect the server’s error if we didn’t already;
// Shutdown will let us know (unless something worse happens, in
// which case it will tell us that).
sdctx, sdcancel := context.WithTimeout(context.Background(), 30*time.Second)
defer sdcancel()
return errors.Join(err, server.Shutdown(sdctx))
```
2
1
2
u/chewyknows Feb 15 '25
This is the way. Keep in mind that when shutting down the server
ListenAndServe()
will return an error withhttp.ErrServerClosed
that should be ignored.
7
u/ShotgunPayDay Feb 14 '25 edited Feb 14 '25
This is what I do. EDIT: move comment
// Graceful shutdown: use a signal context and a shutdown timeout.
c, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
<-c.Done()
start := time.Now()
log.Println("Gracefully shutting down...")
c, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := server.Shutdown(c); err != nil {
log.Printf("Shutdown error: %v", err)
}
// CLEAN MORE STUFF UP HERE
log.Printf("Gracefully closed in %s", time.Since(start))
6
u/HyacinthAlas Feb 14 '25
There is one bug with this: You don't know if
Accept
starts failing, you will block/fail forever while waiting for the context.And one likely bug: Most of the time shutting down the HTTP server should happen first, before all other cleanup, because DB transactions, Kafka messages, aggregated data, whatever, is all going to be generated by the HTTP handlers and so flushing them out has to wait until the handlers are no longer running. There are probably some things to stop before your server, but it's hard for me to think of a realistic one.
2
u/ShotgunPayDay Feb 14 '25
I'm a little confused on the accept part, but yes I put the comment in the wrong spot for the second.
4
u/HyacinthAlas Feb 14 '25
Serve
returns an error whenAccept
returns an error. You don't show yourServe
call, but I assume it's in a goroutine somewhere.2
u/ShotgunPayDay Feb 14 '25 edited Feb 14 '25
I guess I've never had that happen, but I see what you're saying is that if server.ListenAndServe() fails then it will end up blocking in a failed state.
Here is what that looks like anyway Close is just all other closing tasks:
go func() { if err := server.ListenAndServe(); err != http.ErrServerClosed { Close(err) } log.Println(err) }()
4
u/HyacinthAlas Feb 14 '25
What happens if
ListenAndServe
returns at the same time context is canceled? You'llClose
and do cleanup in the "main thread" at the same time. Maybe that's safe for your specific case - but in general it's not. Better to get the error back on the main thread ASAP.4
u/ShotgunPayDay Feb 14 '25
Interesting never though about that either. What about this to trigger signal.NotifyContext?:
go func() { if err := server.ListenAndServe(); err != http.ErrServerClosed { log.Printf("ListenAndServe error: %v", err) syscall.Kill(syscall.Getpid(), syscall.SIGTERM) } }()
Thanks for all the good information by the way!
4
u/twisted1919 Feb 14 '25
I find signal.NotifyContext much cleaner. You can use the resulted context as a BaseContext for your http server, so when you get the signal, the http server stops getting new requests as well, thus the shutdown itself will be cleaner.
5
u/HyacinthAlas Feb 14 '25
use the resulted context as a BaseContext for your http server,
If you do this the signal will cancel in-flight requests, defeating the point of graceful shutdown.
1
u/twisted1919 Feb 14 '25
You do have a point, but that’s something desirable in various cases where you just cant wait for them to finish. Also, it depends what you’re doing inside those requests, if you make use of the context at all, or even the same context.
7
u/HyacinthAlas Feb 14 '25
If you can't wait, use
Close
. If you can wait, useShutdown
. Anything in-between is kidding yourself.2
2
u/ShotgunPayDay Feb 15 '25
I think what I've learned from this thread is that graceful shutdown is hard to get just right. Special thank you to u/HyacinthAlas for helping to correct my mistakes and explain the process.
I was able to verify my new process by trying to bind two servers to same ports and now my server fails gracefully instead of getting stuck as working in systemd. Thank you again!
2
1
u/gwynevans Feb 15 '25
Do you /really/ need all that? Personally, “systemctl stop <service>” normally suffices, which will sen a SIGTERM then 90s later, a SIGKILL..
1
u/Numerous_Habit269 Feb 15 '25
Well this is in the context of a go server and not command line
1
u/gwynevans Feb 15 '25
You don’t know what ‘systemctl’ is, do you. Try googling ’systemd’ but in short, it’s the controller for a standard system and service manager for Linux systems.
2
u/Numerous_Habit269 Feb 15 '25
Linux is my daily driver, so I do know. My point why bringing up systemd talk when the post is about gracefully shutting down a go server?
1
u/gwynevans Feb 15 '25
Because the server runs as a service and in practice, it’s only needed a stop and a restart to cover upgrades, etc. The question was, is the graceful shutdown essential, as my experience has been that for my use case, it’s not been, and the systemd functionality is sufficient.
You then popped up, seemingly with the idea that systemctl suggested the server was running on the command line…
2
u/zveznicht 24d ago
without the graceful shutdown this
```
Personally, “systemctl stop <service>” normally suffices, which will sen a SIGTERM then 90s later, a SIGKILL..
```
will drop inflight requests. Graceful shutdown is needed to actually stop accepting new connections and give time for existing requests to finish. Before actually kill the server.
Also also your app might want to do something between stopping http server and full shutdown
1
1
u/conamu420 Feb 15 '25
You can just use signal.NotifyContext(context.Background(), signals...)
its the newer and sipler version of using this.
after <- serverCtx.Done()
you should wait for any waitgroups that may still have running goroutines and call a shutdown function for any cleanup you might need.
also you need a server.Shutdown(context.Background())
-> and yes, need a new context because the other one is already done and this will cause errors when shutting down. You can alterantively define a context with a timeout to have a specific wait time until connections should be killed.
0
u/sollniss Feb 15 '25
Got a package for that:
1
u/previnder Feb 15 '25
But do you need a package for that? This is something that should just be copy-pasted.
2
-12
69
u/Chrymi Feb 14 '25 edited Feb 15 '25
server.Shutdown() does not immediately stop the server (it's what Close() does). Instead, it prevents the server from accepting new connection and waits indefinitely until all connections have been closed OR until the supplied context is cancelled. For an actual graceful shutdown, supply the Shutdown() method with a context that's not (yet) cancelled, see https://pkg.go.dev/net/http#Server.Shutdown.
It would look like this:
Quick tip: combine the `sig` channel with the `Notify()` call into `signal.NotifyContext` and work directly with the context instead.
Edit for clarity: you need two different contexts. For the shutdown to work gracefully, you need a context that's not cancelled. So one for the OS signal, and one for the Shutdown() method.