Another option is to use the Duration class to get the difference and change it accordingly:

Instant roundToHourIfWithin5Min(Instant instant) {
  final Instant now = Instant.now();
  final Duration absoluteDifference = Duration.between(now, instant).abs();
  // If the difference is less than 5 min then we round off
  if (absoluteDifference.compareTo(Duration.ofMinutes(5)) < 0) {
    final Instant truncated = now.truncatedTo(ChronoUnit.HOURS);
    if (instant.isBefore(now)) {
      return truncated.plus(1, ChronoUnit.HOURS);
    }
    else {
      return truncated;
    }
  }
  return instant;
}

You can simply check the minute part of the date-time and truncate it to hours as per the requirement.

Demo:

import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.time.temporal.ChronoUnit;

class Main {
    public static void main(String[] args) {
        // Test
        String[] arr = { "2021-02-08T19:02:49.594", "2021-02-08T19:56:49.594", "2021-02-08T19:54:49.594",
                "2021-02-08T19:06:49.594" };

        DateTimeFormatter dtf = DateTimeFormatter.ofPattern("uuuu-MM-dd'T'HH:mm:ss.SSS");

        for (String s : arr) {
            System.out.println(roundToNearestHour(s).format(dtf));
        }
    }

    static LocalDateTime roundToNearestHour(String str) {
        LocalDateTime ldt = LocalDateTime.parse(str);
        int minute = ldt.getMinute();
        return minute < 5 ? ldt.truncatedTo(ChronoUnit.HOURS)
                : (minute >= 55 ? ldt.plusHours(1).truncatedTo(ChronoUnit.HOURS) : ldt);
    }
}

Output:

2021-02-08T19:00:00.000
2021-02-08T20:00:00.000
2021-02-08T19:54:49.594
2021-02-08T19:06:49.594

If minute-of-hour is below 5 or over 55

Given that we are dealing with Instant, we know the values are in UTC. And in UTC every hour is 60 minutes long. So we merely need to look at the minute-of-hour to tell if we need to round up or down.

To get the minute-of-hour, we must convert the Instant to the more flexible OffsetDateTime class. We will assign ZoneOffset.UTC constant as the offset, to keep things predictable.

If the minute is under 5, we merely truncate the OffsetDateTime object to the hour.

If the minute is 55 or over, we truncate to hour, but then we add an hour to effectively round up.

In either case of truncation, we extract and return an Instant from our OffsetDateDate truncation-result.

Else we return the same Instant we received, with no rounding.

package work.basil.example;

import java.time.Duration;
import java.time.Instant;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.Objects;

public class InstantRounder
{
    public static void main ( String[] args )
    {
        Instant result = InstantRounder. roundInstantToHourIfWithin5Minutes( Instant.now() );
        System.out.println( "result = " + result );
    }

    public static Instant roundInstantToHourIfWithin5Minutes ( Instant instant )
    {
        OffsetDateTime odt = Objects.requireNonNull( instant ).atOffset( ZoneOffset.UTC );
        int minute = odt.getMinute();
        if ( minute < 5 )
        {
            return odt.truncatedTo( ChronoUnit.HOURS ).toInstant();
        } else if ( minute >= 55 ) // Else if in last 5 minutes, truncate to hour, then add an hour, to effectively round up.
        {
            return odt.truncatedTo( ChronoUnit.HOURS ).plus( Duration.ofHours( 1 ) ).toInstant();
        } else  // Else not within five minutes on either end of hour, so do no rounding. 
        {
            return instant;
        }
    }
}

You can use isBefore and isAfter methods to compare the Instant, with another instant of current hours or next hour

 Instant instant = Instant.now();

    if(instant.isAfter(instant.truncatedTo(ChronoUnit.HOURS)) &&
       instant.isBefore(instant.truncatedTo(ChronoUnit.HOURS).plus(5,ChronoUnit.MINUTES))){
        instant = instant.truncatedTo(ChronoUnit.HOURS);
        
    }else if(instant.isBefore(instant.truncatedTo(ChronoUnit.HOURS).plus(1,ChronoUnit.HOURS)) && 
           instant.isAfter(instant.truncatedTo(ChronoUnit.HOURS).plus(55,ChronoUnit.MINUTES))){

        instant = instant.truncatedTo(ChronoUnit.HOURS).plus(1,ChronoUnit.HOURS);
    } else {
       //do nothing
   }

use below function to achieve this

public static void RoundOffTime(int limit,Date date,Calendar cal2)
{
        int day_of_week=cal2.get(Calendar.DAY_OF_WEEK);
                    int date_of_month=cal2.get(Calendar.DAY_OF_MONTH);
                    int min=cal2.get(Calendar.MINUTE);
                    int hour=cal2.get(Calendar.HOUR);
                    int sec=cal2.get(Calendar.SECOND);
                    int millisec=cal2.get(Calendar.MILLISECOND);
                    int  year=cal2.get(Calendar.YEAR);
        int month=cal2.get(Calendar.MONTH);
        if(min<limit)
        {
        min=0;
        sec=0;
        millisec=0;

        }
        else if(min>60-limit)
        {
        min=0;
        sec=0;
        millisec=0;
        date.setTime(date.getTime() + TimeUnit.HOURS.toMillis(1));
        cal2.setTime(date);
        hour=cal2.get(Calendar.HOUR);
       }
cal2.set(Calendar.DATE,date_of_month);
cal2.set(Calendar.HOUR, hour);
cal2.set(Calendar.MINUTE, min);
cal2.set(Calendar.SECOND, sec);
cal2.set(Calendar.MILLISECOND, millisec);
cal2.set(Calendar.MONTH, month);
cal2.set(Calendar.YEAR, year);
cal2.setTimeZone(TimeZone.getDefault());
Date date3=cal2.getTime(); 
System.out.println("Rounded time is "+date3);

 }

where limit is roundoff limit say 5 min in this case