Deep Q Network Reinforcement learning on RC car track - Understanding Rewards + Backpropagation

I have used openai’s gyms and done some basic reinforcement learning tutorials found on medium and YouTube. These are all using virtual environments.

I am now struggling to understand how to adapt calculating rewards to a real world scenario. My little RC car version is as follows:

• Have a car that drives on a track that is composed of N pieces. The car has built in software to keep it on the track and keep it in a "lane" on the track. These track pieces can be straight, or left/right curves, bridges, or ramps.
• At periodic timepoints, it up updates with the following:

- velocity
- track location (piece + location on piece)
- distance from center of the track (all track segments are uniform widths)
- current lap time in milliseconds
- how much drift (wobble say from taking turns too fast)

The goal is obviously to produce a model that can navigate the track the fastest. In my head, for deciding the reward, it would be based on time to complete the lap (could also include how much drift i.e. to encourage smooth driving but thats another story).

Stepping through the process in my head:

- The car would initialize at a random velocity
- periodically (say 50x / lap), the car would update its "state" including the metrics above
- At each state update, an action will then be taken as determined by feeding the current state to the network and determining the highest Q value. (Note it will sometimes do random actions based on epsilon)
- Each update will be stored in an experience replay. Once I have a set amount of "experiences", I will sample from that replay to update the neural networks weights.

Question: What I believe struggle with I think is around the reward, Q value + possible how back propagation works. Say I have the following rewards:

if (state == crossFinishLine)
    reward = (1 / lapTime) # Biggest reward should be from fastest time
else 
    reward = -1 # action penalty

If we then look at the Q-value update function (Source: Sentdex tutorial):

max_future_q = np.max(future_qs_list[index])
new_q = reward + DISCOUNT * max_future_q

Would the reward be back propagated along the previous decisions made? Like if in a given lap, there are "50" state updates + potential opportunities for actions, does a "laps" q-value get back propagated along all the various 50 state actions?