Is there a SignIn API for federated users or is just a hosted UI?
As far as I can tell you have to use the hosted UI when you federate a user pool to social IdPs.
How do I launch a browser on that Sign In URL?
This depends on the language and platform obviously, on Android with Xamarin you can use Xamarin.Auth.Presenters.OAuthLoginPresenter.Login() to launch a native browser Chrome at a URL specified by the OAuth2Authenticator you pass in. That OAuth2Authenticator does more than just craft the URL it gives Chrome, its stateful so when you get an answer back in the form of a code or token, you can then call methods on that object to proceed.
How can my app be called back when the user has finished signing in?
Specifically how can the browser redirecting a URL actually redirect you back inside the app. That's done via something called, Deep Links & App Links, here's that concept explained on Android.