Foreign key object is getting "instance is already tracked" error in unit tests

In the screenshot below, I'm trying to attach an Answer object so that it can be updated. However, this causes the error: "The instance of entity type 'QuestionSet' cannot be tracked because another instance with the key value '{QuestionSetId: 3}' is already being tracked."

The QuestionSet is indeed being tracked, so I don't think I'm confused about that part. However this error doesn't make sense to me, since I'm trying to update the Answer record, not the QuestionSet record. Shouldn't the contents of the QuestionSet field be irrelevant? I'm not trying to perform an operation that affects the related QuestionSet record, so why am I getting this error?

It's important to note that this error is ONLY happening when I run unit tests. The code works just fine outside of unit tests.

Here's the context setup code I'm using for my unit tests:

DbContextOptions<SurveyApplicationDbContext> options = new DbContextOptionsBuilder<Infrastructure.Data.EntityFramework.SurveyApplicationDbContext>()
                    .EnableSensitiveDataLogging()
                    .UseQueryTrackingBehavior(QueryTrackingBehavior.NoTracking)
                    .UseInMemoryDatabase(databaseName: "SurveyApplicationDbContext" + dbNameSuffix).Options;

        var context = new SurveyApplicationDbContext(options);

        context.Survey.Add(new Survey
        {
            SurveyId = 123,
            ClubId = "c1",
            FormId = 1,
            CreateDateUtc = DateTime.Now
        });
        context.Answer.Add(new Answer
        {
            SurveyId = 123,
            QuestionSetId = 3,
            AnswerValue = "Answer Value",
        });
        context.QuestionSet.Add(new QuestionSet
        {
            QuestionSetId = 3,
            QuestionId = Constants.ScoreQuestionId,
            SectionId = Constants.SummarySectionId,
            FormId = 1
        });

        context.SaveChanges();

        _mockSurveyUnitOfWork.Setup(x => x.SurveyRepository).Returns(new SurveyRepository(context));
        _mockSurveyUnitOfWork.Setup(x => x.AnswerRepository).Returns(new AnswerRepository(context));
        _mockSurveyUnitOfWork.Setup(x => x.QuestionSetRepository).Returns(new QuestionSetRepository(context));
    }

And here is my DbContext model setup code for Answer:

modelBuilder.Entity<Answer>(entity =>
        {
            entity.ToTable("answer");

            entity.HasIndex(e => new { e.QuestionSetId, e.SurveyId })
                .HasName("UCanswer_questionSetId-surveyId")
                .IsUnique();

            entity.Property(e => e.AnswerId).HasColumnName("answerId");

            entity.Property(e => e.AnswerValue)
                .HasColumnName("answerValue")
                .HasMaxLength(800)
                .IsUnicode(false);

            entity.Property(e => e.CreateDateUtc)
                .HasColumnName("createDateUTC")
                .HasColumnType("datetime");

            entity.Property(e => e.ModifiedDateUtc)
                .HasColumnName("modifiedDateUTC")
                .HasColumnType("datetime");

            entity.Property(e => e.QuestionSetId).HasColumnName("questionSetId");

            entity.Property(e => e.ScoreValue)
                .HasColumnName("scoreValue")
                .HasColumnType("decimal(3, 2)");

            entity.Property(e => e.SurveyId).HasColumnName("surveyId");

            entity.HasOne(d => d.QuestionSet)
                .WithMany(p => p.Answer)
                .HasForeignKey(d => d.QuestionSetId)
                .OnDelete(DeleteBehavior.ClientSetNull)
                .HasConstraintName("FKanswer_questionSet_questionSetId");

            entity.HasOne(d => d.Survey)
                .WithMany(p => p.Answer)
                .HasForeignKey(d => d.SurveyId)
                .OnDelete(DeleteBehavior.ClientCascade)
                .HasConstraintName("FKanswer_survey_surveyId");
        });

Obviously there's more code involved that could be causing this error, but I didn't want to overload this post with too much code. Please let me know if I should provide more details. Mainly I want to understand this error on a conceptual level: why is it complaining about tracking QuestionSet even though I'm not trying to update it?