CompositeDisposable.clear() does not stop Observable execution

I use CompositeDisposable to observe a network call. The observable is inside a class that uses my activity context, so even after calling clear() in onDestroy(), the observable keeps running and its class holds a reference to my activity causing a memory leak. Is there any way to actually stop the process, instead of just stop observing it?

The class with the Observable

    public ProgressObservable(Context context) {
        super(context);
        createProgressObservable();
    }

    private void createProgressObservable() {
        progressObservable = authenticationObservable
                .flatMap((Function<Object, Observable<ProgressResponse>>)
                        Response -> Observable.fromCallable(() -> {
                            ...
                            ...
                            return pResponse;
                        })).cache();
    }

The creation of observable in my Activity

        Observable<ProgressResponse> prObservable = new ProgressObservable(context).getProgressObservable();
        disposable.add(prObservable
                .subscribe(progress -> mListener.onSilentRefreshComplete(),
                        error -> {
                        }));

    @Override
    public void onDestroy() {
        super.onDestroy();
        if (isFinishing()) {
            disposables.clear();
        }
    }