rectangle document detection using hough transform opencv android

I am trying to detect rectangle document using opencv 4 android sdk . First i tried to detect it by finding contours but it is not working with multi color documents.You can check this link to get better idea : detecting multi color document with OpenCV4Android

I researched a lot and found that it can be done using houghline transform.So i followed following way to detect document:

original image -> cvtColor -> GaussianBlur filter -> dilate it to sharpen edges -> applied watershed image segmentation algorithm -> canny edge detection with dynamic otsu's threshold -> then applied hough line transform

what i did for hough line transform is:

Imgproc.HoughLinesP(watershedMat, lines, 1, Math.PI / 180, 50, 100, 50);

    List<Line> horizontals = new ArrayList<>();
    List<Line> verticals = new ArrayList<>();
    for (int x = 0; x < lines.rows(); x++)
    {
        double[] vec = lines.get(x, 0);
        double x1 = vec[0],
                y1 = vec[1],
                x2 = vec[2],
                y2 = vec[3];
        Point start = new Point(x1, y1);
        Point end = new Point(x2, y2);
        Line line = new Line(start, end);
        if (Math.abs(x1 - x2) > Math.abs(y1-y2)) {
            horizontals.add(line);
        } else if (Math.abs(x2 - x1) < Math.abs(y2 - y1)){
            verticals.add(line);
        }
    }

and from above list of horizontal and vertical lines , i am finding intersection points as below:

protected Point computeIntersection (Line l1, Line l2) {
    double x1 = l1._p1.x, x2= l1._p2.x, y1 = l1._p1.y, y2 = l1._p2.y;
    double x3 = l2._p1.x, x4 = l2._p2.x, y3 = l2._p1.y, y4 = l2._p2.y;
    double d = (x1 - x2) * (y3 - y4) - (y1 - y2) * (x3 - x4);

   // double angle = angleBetween2Lines(l1,l2);
    Log.e("houghline","angle between 2 lines = "+angle);
    Point pt = new Point();
    pt.x = ((x1 * y2 - y1 * x2) * (x3 - x4) - (x1 - x2) * (x3 * y4 - y3 * x4)) / d;
    pt.y = ((x1 * y2 - y1 * x2) * (y3 - y4) - (y1 - y2) * (x3 * y4 - y3 * x4)) / d;


  return pt;
}

and from that four intersection points i am drawing lines . So, far i am able to detect document through it . see below image :

but , when other objects are concernced with the document ,it tries to detect them also. i am going top to down rows and left to right cols to find intersections of the largest rectangle . I am getting following issues :

As you can see in above images , when other object comes on the screen it is going to detect it too. How to detect only document ? and ignore other objects ? Here is my original image :

Any help will be highly appreciated !! thanks in advance