I've recently built such an architecture, maybe it will help you as a reference. Our players - Field Gateway - A local hub that is strong enough to handle communication to the Iot Hub and receive data from edge sensors. Edge Sensors - devices that are unable to connect directly to the Iot Hub but implement some communication protocol that enable them to connect to the field gateway(Zwave, Zigbee...). Iot Hub - Handles the D2C and C2D between the Field Gateway and the Hub. Back end server - Receives the data from the Iot Hub.
The Edge Sensor sends their telemetry to the Field Gateway. The Field Gateway maintains a connection to the Iot Hub and the only device that the Iot Hub knows is the Field Gateway. Each telemetry that is being received in the Field Gateway from the Edge Sensors contains a unique id in the message payload. When the message is being received in the Iot Hub, The Iot Hub knows only about the Field Gateway. But When the message is being process by the Back End it takes the unique id from the payload and therefore knows which is the correct device that sent the telemetry. So we have multiple devices that all "riding" on one connection.
Hope it helps.