Can t recover docker swarm from pending state

Question

There was a crash and I have this issue now where it says docker swarm status is pending and the node status is UNKNOWN. This is my docker info result

swarm@swarm-manager-1:~$ docker info
Containers: 270
 Running: 0
 Paused: 0
 Stopped: 270
Images: 160
Server Version: 1.12.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1211
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge null overlay
Swarm: pending
 NodeID: d9hq8wzz6skh9pzrxzhbckm97
 Is Manager: true
 ClusterID: 5zgab5w50qgvvep35eqcbote2
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: HIDDEN
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-91-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.804 GiB
Name: swarm-manager-1
ID: AXPO:VFSV:TDT3:6X7Y:QNAO:OZJN:U23R:V5S2:FU33:WUNI:CRPK:2E2C
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

This is my docker node ls result:

ID                           HOSTNAME         STATUS   AVAILABILITY  MANAGER STATUS
9tlo3rg7tuc23xzc3am28lak1    swarm-worker-1   Unknown  Active        
d9hq8wzz6skh9pzrxzhbckm97 *  swarm-manager-1  Unknown  Active        Leader

Answer 1

First, note the node IDs

On manager node:

docker node update --availability drain $WORKER_ID

This is optional

docker swarm join-token manager

This command will give you the join command to run on each node after it's removed. I'll refer to it as $JOIN_COMMAND below. We will demote the worker once the manager re-joins.

On worker:

docker swarm leave
$JOIN_COMMAND

This node is now re-joined as a manager, but I'll continue calling it the 'worker' to avoid confusion.

On manager:

docker node rm $WORKER_ID
docker node update --availability drain $MANAGER_ID
docker swarm leave -f
$JOIN_COMMAND
docker node rm $MANAGER_ID
docker node ls

Find the worker's new id (pay attention to the hostname, not the role) -> $NEW_WORKER_ID

docker node demote $NEW_WORKER_ID

Your swarm should be refreshed - if there were more nodes, the services running on each would have migrated across the swarm when you drained each node.