If you got a vulnerability of a remote system but can't get a shell, it probably falls under a number of reasons why any payload does not run successfully or can't even connect properly. Based on what I know, here's what I would look for:
1. First, firewall rules may be interfering with incoming connections, especially on non-standard ports. Sometimes, payloads which connect over common ports, like 80 or 443, can assist because those ports are often left open for web traffic. A reverse shell can sometimes bypass simple firewall restrictions on these ports.
2. Then there are the Intrusion Detection/Prevention Systems. These can be set to block a particular known exploit signature, and it'll terminate your attempted connection. To somewhat troubleshoot this, consider obfuscating your payload, making them custom, or both. For this you can use tools like msfvenom. Sometimes payloads that do not match any of these known IDS/IPS signatures are able to bypass basic system protections.
3. Next, there's the anti-virus and endpoint protection problem. Many security tools on the target system might detect your payload, especially if it is common, like a reverse shell. In some cases, running the payload in memory instead of on disk or using AV-evasion techniques might work for you. For example, you could try encoding the payload or even polymorphic techniques.
4. Another is network segmentation. If the vulnerable system is on an isolated network segment, you may not have a direct route to establish a shell. This can be very helpful to pivot through another compromised host that has network access to the target system.
5. You might have to consider if patches or privilege restrictions exist. And though the vulnerability is unpatched, you may already be restricted as to what a payload can do by the environment on the target. Ensure the payload you are using is for the OS and architecture of the system you are targeting is 32-bit or 64-bit. Another useful change in payload type, sometimes, gets you past a sticking point depending on the environment, such as switching from a meterpreter session to a simple command shell.
6. This way, you would troubleshoot properly by first running simple commands or even scripts through the exploit to see if anything is running. It may help prove whether you can run code at all. If there is some kind of access to logs on the target, go through them; at times they will reveal whether the system will detect and block your attempts. Finally, test different payloads and ports while attempting to bypass blocking rules.
