How do I Retry an observable returned via a SelectMany call from another observable

I want to retry an observable when an error occurs. In the two cases below, I only see the Retry work in case 1. However, in case 2, the Retry does not happen.

What is the difference between these two cases? Is it because the Retry in case 2 would just Retry if subject error'ed? If that is the case, then wouldn't the Subscribe in case 2 only be acting on subject, in which case I don't understand when ErrorProducer is actually subscribed to.

In case 2 my understanding was that SelectMany(_ => ErrorProducer(3)) returns an IObservable<int> and then the Retry and Subscribe calls act on that. Though it seems I need to add Retry directly to the ErrorProducer call as in case 1.

        public ViewModel()
        {
            var subject = new Subject<int>();

            //Case 1: comment out case 2 when running case 1
            subject
              .SelectMany(_ => ErrorProducer(3).Retry())
              .Subscribe();

            subject.OnNext(1);


            //Case 2: comment out case 1 when running case 2
            //subject
            //    .SelectMany(_ => ErrorProducer(3))
            //    .Retry()
            //    .Subscribe();

            //subject.OnNext(1);
        }

        private IObservable<int> ErrorProducer(int i)
        {
            int count = 0;
            return Observable.Create<int>(observer =>
            {
                Debug.WriteLine("Doing work");

                if (count++ < i)
                {
                    Debug.WriteLine("Failed");
                    observer.OnError(new Exception());
                }
                else
                {
                    Debug.WriteLine("Done");
                    observer.OnNext(count);
                    observer.OnCompleted();
                }
                return Disposable.Empty;
            });
        }

Output:

Case 1:
    Doing work
    Failed
    Doing work
    Failed
    Doing work
    Failed
    Doing work
    Done


Case 2:
    Doing work
    Failed