Updating complicated Object with useState

Because the object is in state, you can't modify it directly. Instead, you have to create a copy of the object and any inner objects (including arrays) that you change. See comments:

// Use the callback form because you're updating state based on existing state
setTheObject(original => {
    // Shallow copy the object and book1
    const update = {...original, book1: [...original.book1]};
    // Replace the object at index 0 with a new updated object
    update.book1[0] = {...update.book1[0], selected: false};
    // return the update
    return update;
});

It is technically possible to do this in one nested operation, but it's much harder to read and debug:

// Use the callback form because you're updating state based on existing state
setTheObject(original => ({
    // Shallow copy the object
    ...original,
    // Shallow copy `book1` while updating index 0 with a copy of the object with the updated property
    book1: Object.assign([], original.book1, {0: {...original.book1[0], selected: false}}),
}));

I don't recommend it. :-D

const [books, setBooks] = useState({
      book1: [
           { name: 1, selected: true },
           { name: 2, selected: false },
           { name: 3, selected: false },
           { name: 4, selected: false },
           { name: 5, selected: true }
          ],
      book2: [
           { name: 1, selected: false },
           { name: 2, selected: false },
           { name: 3, selected: true },
           { name: 4, selected: false },
           { name: 5, selected: true }
          ]
 });

Assume you hold the books state like this. Now update the books state like this.

setBooks({
  ...books,
  book1: books.book1.map((book) =>
    book.name === 1 ? { ...book, selected: !book.selected } : book
  )
});

In order to check the effect after state is changed, you can add dependancy to useEffect hook and as soon the books are changed, you will see the updated state in useEffect.

useEffect(() => {
  console.log(books);
 }, [books]);