Find a good homography from different point of view of objects?

I am doing object detection using feature extraction (sift,orb).

I want to extract ORB feature from different point of view of the object (train images) and then matching all of them with a query image.

The problem I am facing is: how can I create a good homography from keypoint coming from different point of view of the image that have of course different sizes?

Edit

I was thinking to create an homography for each train images that got say 3-4 matches and then calculate some "mean" homography...

The probleam arise when you have for example say just 1-2 matches from each train image, at that point you cannot create not even 1 homography

Code for create homography

  //> For each train images with at least some good matches ??
  H = findHomography( train, scene, CV_RANSAC );
  perspectiveTransform( trainCorners, sceneCorners, H);