DESIGN PROBLEM: How to use composition the best

Since you use Set<Pixel> you have to create new Pixel instance to check if it exists in set or not.

If set contains N elements after calling B.function method you will create extra N Pixel nodes. If all elements are new, you just add them to set, in other case Garbage Collection needs to sweep them. One of drawbacks is we need to create m (wheren m <= N - number of Pixel-s which already exists in set) and later we need to collect them by GC. How big is m/N ratio depends from your algorithm and what you are actually doing.

Lets calculate how many memory we need to consume for N = 1_000_000 pixels in set. We know that int is a 4 bytes and double is 8 bytes, lets add extra 8 bytes for an object and 8 bytes for a reference. It gives 32 bytes for every instance of Pixel object. We need to create N objects which gives 32MB. Let's assume that our ratio is 50% so, 16MB we allocated just to check it is not needed.

If this is a cost you can not pay you need to develop algorithm which allows you iterate over Set<Pixel> in an order from left-to-right. So, left neighbour of Pixel X is before X.

Assume that left neighbour of Pixel X(x, y) is pixel X'(x - 1, y). Pixel B(0, y) does not have left neighbour. You need to use TreeSet and implement Comparable<Pixel> interface in Pixel class. Simple implementation could look like this:

@Override
public int compareTo(Pixel o) {
    return this.y == o.y ? this.x - o.x : this.y - o.y;
}

This allows you to iterate set in order from left to right: (0, 0), (1, 0), ...., (x - 1, y), (x, y), (x + 1, y), ... , (maxX, maxY). So, when you iterate it you can check whether previous element is a left neighbour of current Pixel. Example implementation could look like below:

void addNeighboursIfNeeded() {
    Set<Pixel> neighbours = new HashSet<>(pixels.size());
    Pixel last = null;
    for (Pixel p : pixels) {
        if (p.getX() == 0 || p.isLeftNeighbour(last)) {
            // a left border pixel
            // or last checked element is a left neighbour of current pixel.
            last = p;
            continue;
        }
        // last element was not our left-neighbour so we need to call b method
        Pixel left = b.getLeft(p);
        neighbours.add(left);
        last = p;
    }
    // add all new neigbours
    pixels.addAll(neighbours);
}

This should allow you to save this memory which is allocated for duplicated Pixel objects.

I can see here few concerns regarding object oriented programming.

1. Encapsulation violation: When you call function of B from A which operates on A's data (which you are avoiding by not sending HashMap), it violates encapsulation (If there is a reason, its acceptable though). Is it possible to move that function (operating on A's HashSet) to A? This will protect A's state from getting exposed.
2. Proliferation of classes: There is a possibility that there will be large number of objects of Point type. You can think of using Flyweight GOF design pattern, it will externalize the state of each point and will make it reusable. and will reduce number substantially.
3. Passing large collection of Points to method in B: If you can shift method from B to A, this point gets resolved. Anyway java will pass this collection by reference. But in that case its open for modifications from external classes (need to take care of this aspect).
4. Abstraction of type Point: If class Point has only state and no behavior, it will lead to violation of encapsulation. Can you shift the method getNeighbour() in to Point? as it will make Point immutable (which is essential). Off course actual algorithm can be delegated to another class (if its independently varying responsibility and has hierarchy of algorithms, think of GOF Strategy pattern here).
5. Uniqueness of points in collection: which your set will take care with due care about appropriate Hash and logical equality for class Point.