Neat implementation problem in python, robot not going straight

I have a problem with the results im getting in python with NEAT algorithm, using NEAT-python library. Im training robots to go in a straight line, but the problem is, at the end of a track image im using, my robot model seems to evolve to turn to the corner.

Here is a function I use to evaluate fitness with:

    def distance_x(self):
        distance_travelled_x = int(self.rect.center[0] - x_spawn)
        return distance_travelled_x

My robot model has 2 sensors in front, angled 30 and -30 degrees, measuring distance to wall. They are also inputs to my neural network. There are 2 outputs, 1st for going left-right, 2nd for going forward-backward.

        for i, robot in enumerate(robots):
            output = nets[i].activate(robot.sprite.data())
            if output[0] < 0.4:
                robot.sprite.direction = 1
            if output[0] > 0.6:
                robot.sprite.direction = -1
            if output[1] > 0.6:
                robot.sprite.state = 1
            if output[1] < 0.4:
                robot.sprite.state = -1

Here is a picture of my track:

The robot tries to go straight, and then goes to one of the corners when near the finish line...

I also tried this fitness function, which severly punishes the ones who take a turn:

    def distance_x(self):
        distance_travelled_x = int(self.rect.center[0] - x_spawn)
        distance travelled_y = int(self.rect.center[1] - y_spawn)**2
        fitness = distance_travelled_x - distance travelled_y
        return fitness

And this one:

 def measure_fitness(self):
         maximum = 1000000 #most fitness an individual can get
         distance_travelled = int(math.sqrt(math.pow(self.rect.center[0] - x_spawn,2)
                              + math.pow(self.rect.center[1] - y_spawn, 2)))
         angle = math.acos((self.rect.center[0]-x_spawn)/(distance_travelled+1))
    
         try:
             fitness = (distance_travelled/angle)
         except ZeroDivisionError:
             fitness = maximum
         return fitness

And they both work pretty similar to the 1st one (the robots still take a turn when near the end), which makes me think its not the problem in the fitness function, but the sensor inputs to the network... Any ideas?