Recursively combining HTTP results based on response

user_3794740 0
Viewed 1192

There's an API(https://panelapp.genomicsengland.co.uk/api/v1/panels/?page=1) that I want to consume all the data to my angular apps. The problem is that their API have pagination, and I want to retrieve all of the content at once.

As you can see on the API, they actually have "next" attribute on their response which point to the next page. I want to be able to keep requesting from the API as long as "next" attribute is not null then combine all their response into one.

I have tried using recursive, but by the time it reach the 2nd loop I get undefined value. My guess is that its because async request, hence I get undefined.

Below is my code

@Injectable()
export class GenomicsEnglandService {
    panels = [];
    constructor(private http: HttpClient) {
    }

    getPanels(url): Observable<any>{
        const headers = new HttpHeaders()
        .append('Content-Type', 'application/json')
        .append('Accept', '*/*');

        return this.http.get(url, {headers: headers})
            .map((data) => {
                panels = panels.concat(data.results);
                if(data.next){
                    this.getPanels(data.next);
                }else{
                    return panels;
                }
            })
            .catch((e) => {
                Raven.captureMessage("GENOMICS ENGLAND ERROR: " + JSON.stringify(e));
                return of([]);
            });

    }

}

Then from my component I just called 

this.GenomicsEnglandService.getPanels('https://panelapp.genomicsengland.co.uk/api/v1/panels/?page=1').subscribe(data => {
  console.log(data);
})
angular-httpclient rxjs angular recursion
5 Answers

This can easily be done in rxjs by using the expand operator:

import {empty, Observable} from 'rxjs';
import {expand, map, reduce} from 'rxjs/operators';

export interface PanelResponse {
  results: object[];
  next: string|null;
}

@Injectable()
export class Service {
  private readonly baseUrl = 'https://panelapp.genomicsengland.co.uk/api/v1/panels/';

  constructor(private http: HttpClient) {
  }

  getPanels(): Observable<object[]>{
    return this.get(this.baseUrl).pipe(
      expand(({next}) => next ? get(next) : empty()),
      map(({results}) => results),
      // if you want the observable to emit 1 value everytime that
      // a page is fetched, use `scan` instead of `reduce`
      reduce((acc, val) => acc.concat(val), new Array<object>()),
    );
  }

  private get(url:string>):Observable<PanelResponse> => this.http.get<PanelResponse>(url);
}
user_53942200

Although this question has been answered, I would like to propose another approach by using expand operator [https://rxjs-dev.firebaseapp.com/api/operators/expand]. expand operator is made for such recursive purposes:

getResult() {

    const url = "https://panelapp.genomicsengland.co.uk/api/v1/panels/";

    return this.getResponse(url)
                    .pipe(
                      expand((res: any) => this.getResponse(res.next)),
                      takeWhile((res: any) => res.next, true),
                      concatMap((res: any) => res.results),
                      reduce((acc, val) => {
                        acc.push(val);
                        return acc;
                      }, []),
                      tap(_ => {
                        console.log(_);
                        this.loading = false;
                      })
                    )

  }

  getResponse(url) {
    return this.httpClient.get(url);
  }

See working stackblitz

user_22165840

If I've understood what you're trying to do, you want to be using mergeMap instead of map. Merge map lets you combine observables in sequence, like so:

getPanels(url): Observable<any> {
  return this.http.get(url)
    .pipe(
      mergeMap(data => {
        panels = panels.concat(data.results);
        if(data.next) {
          return this.getPanels(data.next);
        } else {
          return panels;
        }
      })
    );
}

Does that work?

user_87616600

Updated see answer from @user2216584. This answer is acepted, but it's better the indicate answer

see stackblitz

constructor(private httpClient:HttpClient){}
  ngOnInit()
  {

    this.getHttp('https://panelapp.genomicsengland.co.uk/api/v1/panels/',null).subscribe(res=>{
      this.data=res
    })

  }

  getHttp(url,fullData:any[]):Observable<any[]>
  {
    fullData=fullData || []
    return this.httpClient.get(url).pipe(
      switchMap((data:any)=>{
        fullData=fullData.concat(data.results);
        return !data.next? of(fullData):
               this.getHttp(data.next,fullData)
      })
    )
  }
user_85581860

I have done a similar implementation in one of my projects.

In my service (I am returning promise you can return observable if you want)

  getDataHttp(url) {
    return this.http.get(url).toPromise();
  }

  getData(url) {
    let response = {};
    let data = [];
    return new Promise(async (resolve, reject) => {
      try {
        do {
          response = await this.getDataHttp(url);
          data.push(response);
        } while(response['next']); 
        resolve(data);
      } catch(err) {
        reject(err);
      }
    })
  }

In my component 

this.service.getData(url).then((response) => {
  console.log(response);
}).catch(err => console.log(err))
user_79912600
Related