What is the best way to paginate a large table using React Table and Django? (closed)

Question

I have a table with 160k rows and 20+ columns. For now, I have configured a custom Django Rest Framework pagination so that I can use query string to set the page size as well as the page number:

http://127.0.0.1:8000/api/ix/?page=2&page_size=100

The JSON result looks like this:

{
    "count": 166710,
    "next": "http://127.0.0.1:8000/api/ix/?page=3&page_size=100",
    "previous": "http://127.0.0.1:8000/api/ix/?page_size=100",
    "results": [{...}]
}

For the frontend, I want to use React Query to get the JSON, and React Table to display and handle the pagination.

From React Table's offical example, it seems to me that it just get the unpaginated version of the data, then do a slicing on the client side:

const startRow = pageSize * pageIndex
const endRow = startRow + pageSize
setData(serverData.slice(startRow, endRow))

Possible Approaches

Don't use pagination on the server side, and slice the data as shown by the example
- Pro: I can just use the example codebase with a little modification to make this work
- Con: It will take quite while to load the data; the pagination might cause re-fetching and loading the data per pagination operation
Use custom pagination on the server side, and make my own pagination methods
- Pro: Since the data is already paginated, it's more efficient to load them
- Con: More codes are involved, which can be error-prone. Also, I'd have to redesign the components given React's top-down only data flow
Use server-side pagination AND client-side pagination
- This might be optimal, but I don't really know how to do this if the data is already paginated with fixed page size. Assuming I cannot query page_size anymore, how would the client-side fetch data when users set a different page size? For example, if my API is returning 100 results on page 1, and now the user wants to display 500 rows. Would React Table "magically" call the API until it gets all the 500 results?

What is the best approach here?

My Solution

Well, that was awkward... I ended up making this work by using my server-side pagination coupled with several useEffect hooks to pass page number and page size to the url query parameters.

This blog helped me a lot because the author tries to fetch data from Pokemon API, which is server-side paginated, very similar to my setup.

import React from "react";
import { useTable, usePagination } from "react-table";
import { useQuery } from "react-query";
import { fetchIXData, ColName } from "fetchData";

const fetchIXData = async (page, pageSize) => {
    try {
      const response = await fetch(
        `http://127.0.0.1:8000/api/ix/?page=${page + 1}&page_size=${pageSize}`
      );   // My API is 1-indexed but React Table is 0-index, hence the + 1
      const data = await response.json();
      return data;
    } catch (e) {
      throw new Error(`API error: !{e?.message}`);
    }
  };
  
  const ColName = {
    col: "col1",
    // and 19 more
};
  
const columns = Object.keys(ColName).map((key) => {
  return {
    Header: ColName[key],
    accessor: key,
  };
});

const initialState = {
  queryPageIndex: 0,
  queryPageSize: 20,
  totalCount: null,
};
const PAGE_CHANGED = "PAGE_CHANGED";
const PAGE_SIZE_CHANGED = "PAGE_SIZE_CHANGED";
const TOTAL_COUNT_CHANGED = "TOTAL_COUNT_CHANGED";

const reducer = (state, { type, payload }) => {
  switch (type) {
    case PAGE_CHANGED:
      return {
        ...state,
        queryPageIndex: payload,
      };
    case PAGE_SIZE_CHANGED:
      return {
        ...state,
        queryPageSize: payload,
      };
    case TOTAL_COUNT_CHANGED:
      return {
        ...state,
        totalCount: payload,
      };
    default:
      throw new Error(`Unhandled action type: ${type}`);
  }
};

function Table() {
  const [{ queryPageIndex, queryPageSize, totalCount }, dispatch] =
    React.useReducer(reducer, initialState);

  const { isLoading, error, data, isSuccess } = useQuery(
    ["ixData", queryPageIndex, queryPageSize],
    () => fetchIXData(queryPageIndex, queryPageSize),
    {
      keepPreviousData: true,
      staleTime: Infinity,
    }
  );

  const {
    getTableProps,
    getTableBodyProps,
    headerGroups,
    prepareRow,
    page,
    canPreviousPage,
    canNextPage,
    pageOptions,
    pageCount,
    gotoPage,
    nextPage,
    previousPage,
    setPageSize,
    // Get the state from the instance
    state: { pageIndex, pageSize },
  } = useTable(
    {
      columns,
      data: isSuccess ? data.results : [],
      initialState: {
        pageIndex: queryPageIndex,
        pageSize: queryPageSize,
      },
      manualPagination: true,
      pageCount: isSuccess ? Math.ceil(totalCount / queryPageSize) : null,
    },
    usePagination
  );

  React.useEffect(() => {
    dispatch({ type: PAGE_CHANGED, payload: pageIndex });
  }, [pageIndex]);

  React.useEffect(() => {
    dispatch({ type: PAGE_SIZE_CHANGED, payload: pageSize });
    gotoPage(1);
  }, [pageSize, gotoPage]);

  React.useEffect(() => {
    if (data?.count) {
      dispatch({
        type: TOTAL_COUNT_CHANGED,
        payload: data.count,
      });
    }
  }, [data?.count]);

  if (error) {
    return Error;
  }

  if (isLoading) {
    return Loading...;
  }
  return (
    <>
      {isSuccess ? (
        <>
        
            {" "}
            {" "}
            {" "}
            {" "}
            
              Page{" "}
              
                {pageIndex + 1} of {pageOptions.length}
              {" "}
            
            
              | Go to page:{" "}
               {
                  const page = Number(e.target.value) - 1;
                  if (e.target.value != null) gotoPage(page);
                }}
                style={{ width: "100px" }}
              />
            {" "}
            
          


          
              {headerGroups.map((headerGroup) => (
                
                  {headerGroup.headers.map((column) => (
                    
                  ))}
                
              ))}
            
              {page.map((row) => {
                prepareRow(row);
                return (
                  
                    {row.cells.map((cell) => (
                      
                    ))}
                  
                );
              })}
            
            
                      {column.render("Header")}
                    
            {cell.render("Cell")}
          
          
        
      ) : null}
    
  );
}

export default Table;

What is the best way to paginate a large table using React Table and Django? (closed)

Possible Approaches

My Solution

0 Answers