I am trying to understand this piece of Netsted objects in Javascript

"How does the variable yourScore pass the value into the rpsDatabase ?"

It doesn't – yourScore is the result of looking up a nested value in the database.

You can use a variable to lookup a property of an object using [] notation -

const myObject = { hello: "world", foo: "bar" }
const key = "hello"
console.log(myObject[key])
// "world"

You can sequence many [][]... to lookup a nested property -

const myObject = { hello: { world: "earth" }}
const key1 = "hello"
const key2 = "world"
console.log(myObject[key1][key2])
// "earth"

"And how are the functions being called ?"

The rpsGame function is called when one of the images are clicked. This happens because each img has onclick="rpsGame(this)"

improvements

I would like to offer several quality improvements to the code.

1. The rpsDatabase is redefined each time the decideWinner is called. This is unnecessary as the values of rpsDatabase do not (should not) ever change -

const rpsDatabase = {
  rock: {scissors: 1, rock: 0.5, paper: 0},
  paper: {rock: 1, paper: 0.5, scissors: 0},
  scissors: {paper: 1, scissors: 0.5, rock: 0},
}

function decideWinner (yourChoice, computerChoice) {
  var yourScore = rpsDatabase[yourChoice][computerChoice]
  var computerScore = rpsDatabase[computerChoice][yourChoice]
  return [yourScore, computerScore]
}

2. By separating rpsDatabase, we can use it to reduce code duplication in randToRpsInt and numberToChoice. Instead we end up with a simple randomChoice function -

function randomChoice () {
  const rand = Math.floor(Math.random() * 3)
  // no need to redefine ["rock", "paper", "scissors"] on each call!
  return Object.keys(rpsDatabase)[rand]
}

3. The values chose to encode a win, loss, or tie are arbitrary: 1, 0, or 0.5. Consider this alternative -

This new rpsDatabase would be written as -

const rpsDatabase = {
  rock: {scissors: 1, rock: 0, paper: -1},
  paper: {rock: 1, paper: 0, scissors: -1},
  scissors: {paper: 1, scissors: 0, rock: -1},
};

Now instead of testing for equality to arbitrary numbers, this allows you to write finalMessage in a more natural way using comparisons > and < -

function finalMessage ([yourScore, computerScore]) {
  if (yourScore > computerScore)
    return {message: 'You won!', color: 'green'}
  else if (computerScore > yourScore)
    return {message: 'You lost!', color: 'red'}
  else 
    return {message: 'You tied!', color: 'yellow'}
}

Or the equivalent using switch. Maybe you like this form more -

function finalMessage ([yourScore, computerScore]) {
  switch (true) {
    case yourScore > computerScore:
      return {message: 'You won!', color: 'green'}
    case computerScore > yourScore:
      return {message: 'You lost!', color: 'red'}
    default:
      return {message: 'You tied!', color: 'yellow'}
  }
}

4. Use addEventListener instead of onclick attribute. This creates cleaner separation of HTML and JavaScript, allowing greater portability of both -

<form id="game">
  <button type="button" value="rock"></button>
  <button type="button" value="paper"></button>
  <button type="button" value="scissors">✂️</button>
  <br>
  <output name="result"></output>
</form>

const game = document.forms.game

// attach `rpsGame` to `click` event on each `button`
for (const button of game.querySelectorAll("button"))
  button.addEventListener("click", rpsGame)

demo

Here's a functional demo that incorporates the suggestions above -

const game = document.forms.game

const rpsDatabase = {
  rock: {scissors: 1, rock: 0, paper: -1},
  paper: {rock: 1, paper: 0, scissors: -1},
  scissors: {paper: 1, scissors: 0, rock: -1},
}

function rpsGame (event) {
  const humanChoice = event.target.value
  const botChoice = randomChoice()
  const result = decideWinner(humanChoice, botChoice)
  game.result.value = JSON.stringify(finalMessage(result))
}

function randomChoice () {
  const rand = Math.floor(Math.random() * 3)
  return Object.keys(rpsDatabase)[rand]
}

function decideWinner (yourChoice, computerChoice) {
  const yourScore = rpsDatabase[yourChoice][computerChoice]
  const computerScore = rpsDatabase[computerChoice][yourChoice]
  return [yourScore, computerScore]
}

function finalMessage ([yourScore, computerScore]) {
  if (yourScore > computerScore)
    return {message: 'You won!', color: 'green'}
  else if (computerScore > yourScore)
    return {message: 'You lost!', color: 'red'}
  else 
    return {message: 'You tied!', color: 'yellow'}
}

for (const button of game.querySelectorAll("button"))
  button.addEventListener("click", rpsGame)

<form id="game">
  <button type="button" value="rock"></button>
  <button type="button" value="paper"></button>
  <button type="button" value="scissors">✂️</button>
  <br>
  <output name="result"></output>
</form>

From your code it is pretty clear that there are three clickable image, when user click either of these images, we are calling a method rpsGame and passing the whole image object as a parameter. This rpsGame method calls another method decideWinner to decide who is the winner by passing two parameters one selected by you(like rock, paper or scissors) and other selected by some random method like numberToChoice. Finally based on the the result returned by decideWinner method like 0, 0.5 or 1, you can call the finalMessage method and print whatever your results is, based on the condition given in finalMessage method.