To restrict the access to only the CloudFront, add an origin access identity to your distribution. Then go ahead and review your bucket policy for the following pointers:
- Only the OAI can access the bucket
- CloudFront can access the bucket
- Users cannot access the bucket in any way.