Spring Boot application freezes after a day

Question

I have a Spring Boot application running on two servers. It accepts file upload requests (approx 1 KB) via Controller from 1000s of IOT devices.

Additionally there is a TCP Listener which also accepts GPS data as a Stream from these IOT devices.

I have setup the Tomcat max thread count to 3000 on one server and 5000 on a larger VM.

After a day or two my Spring Boot application freezes, i.e. it stops responding to any further API calls.

One reason, I suspect is that the IOT devices are not closing the TCP socket connection after sending the data.

Where could I be going wrong? I am running out of threads to handle new incoming requests?

############################

EDIT 1:

After investigating the thread dump and adding logs during during Socket connection and closure, I have discovered that the IOT device isn't closing connections. Instead the device is creating a new connection every time it wants to send new data without closing the previous one.

Before the app froze, 22500 open connections were automatically closed.

How can I handle this behavior? Should I close the connections forcefully after a certain timeout?

Answer 1

You can mitigate the idle connections problem by hiding your service behind an Nginx or HAProxy which close idle connections by default after a given timeout.

Another option is to set server.connection-timeout in application properties (see server properties), but this appears to be applicable to HTTP only.

Get your Spring Certification today to become certified.