Sql multiple cascade paths/ Unable to determine the principal end of an association between the types entity framework code first

My issue is I keep getting this error using entity framework code first "Additional information: Introducing FOREIGN KEY constraint 'FK_dbo.Sets_dbo.ExerciseStats_ExerciseStatId' on table 'Sets' may cause cycles or multiple cascade paths. Specify ON DELETE NO ACTION or ON UPDATE NO ACTION, or modify other FOREIGN KEY constraints."

My tables are: workouts, exerciseStats, sets

public class Workout
    {
    [Key]
    public int WorkoutId { get; set; }
    public string UserId { get; set; }
    [Display(Name = "Workout Name")]
    public string Name { get; set; }
    public DateTime? Date { get; set; }
    public List<ExerciseStats> Exercises { get; set; }

    public bool IsBaseWorkout { get; set; }

    }

public class ExerciseStats
    {
    [Key]
    public int ExerciseStatsId { get; set; }
    [Required]
    public int ExerciseId { get; set; }
    [ForeignKey("ExerciseId")]
    public Exercise Exercise { get; set; }
    [Required]
    public int WorkoutId { get; set; }
    [ForeignKey("WorkoutId")]
    public Workout Workout { get; set; }

    public int DesiredSetId { get; set; }
    [ForeignKey("DesiredSetId")]
    public Set DesiredSet { get; set; }
    public int DesiredSetCount { get; set; }
    public List<Set> ActualSets { get; set; }


   }

public class Set
{

    [Key]
    public int SetId { get; set; }
    [Required]
    public int ExerciseStatId { get; set; }
    [ForeignKey("ExerciseStatId")]
    public ExerciseStats ExerciseStat { get; set; }
    public float? WeightInKg { get; set; }
    public int? Reps { get; set; }
    public float? Minutes { get; set; }
}

I've been looking at the code for a while and can not seem to identify where the problem is occurring. I have tried to disable cascade delete using this code:

protected override void OnModelCreating(DbModelBuilder modelBuilder)
    {
        modelBuilder.Entity<ExerciseStats>()
        .HasMany(p => p.ActualSets)
        .WithRequired()
        .HasForeignKey(c => c.ExerciseStatId)
        .WillCascadeOnDelete(false);
    }

This resulted in this error "Additional information: Unable to determine the principal end of an association between the types 'GymDiaryCodeFirst.Models.Set' and 'GymDiaryCodeFirst.Models.ExerciseStats'. The principal end of this association must be explicitly configured using either the relationship fluent API or data annotations."

Which doesn't make sense to me as I have used the annotations on the required properties.

What I am trying to achieve is a sets table which when data is deleted doesn't effect the exerciseStats table or any other table for that matter. Also the exerciseStats class contains a property called desiredSet which also appears in the sets table.

Been trying to figure this out all sunday :( Thanks for any help guys.