I structured my HOC so it could be invoked twice. How do I connect redux in this context

If you want to connect the wrapped component to redux within your Higher Order Component, then you can refactor your HOC to define an "inner" functional component that can then also be wrapped by the conenct Higher Order Component (Alternatively you could use react-redux's useDispatch and useSelector hooks in the component directly).

import React from "react"

export default (WrappedComponent) => (mapFunctions) => {
  const component = (props) => { // <-- will contain inject props from Redux
    const [open, setOpen] = React.useState(false);

    // typical functionality of the HOC
    //search function
    const handleSearch = (e) => {
      let filter,
        table,
        tableRow,
        tableCells,
        txtValue,
        results = [];

      filter = e.currentTarget.value.toLowerCase();
      table = document.querySelector(`#${mapFunctions.tableID}`);
      tableRow = table.getElementsByTagName("tr");

      Array.from(tableRow).forEach((row, index) => {
        tableCells = row.getElementsByTagName("td")[1];

        if (tableCells) {
          txtValue = tableCells.textContent || tableCells.innerText;
          if (txtValue.toLowerCase().indexOf(filter) > -1) {
            return mapFunctions.searchFuncReasult === "array"
              ? results.push(row)
              : (tableRow[index].style.display = "");
          } else {
            return mapFunctions.searchFuncReasult === "array"
              ? results.push(row)
              : (tableRow[index].style.display = "none");
          }
        }
      });
    };

    return <WrappedComponent {...props} handleSearch={handleSearch} />;
  };

  const mapStateToProps = (state) => ({
    // ... any redux state selectors
  });

  const mapDispatchToProps = {
    // ... any redux action creators
  };

  return connect(mapStateToProps, mapDispatchToProps)(component);
};

Another common pattern I use when combining several HOCs to decorate a component is to compose them together. If you are using react-redux then you likely are already using redux as a dependency, which exports such a compose utility. You likely already used it to compose all your reducers into a single root reducer. In my opinion this is the simpler solution, especially since it doesn't appear you actually need the injected redux props in your HOC wrapper.

import { compose } from 'redux';

...

export default compose(
  connect(mapStateToProps, mapDispatchToProps),
  YourCustomHOC
)(ComponentYouWantToWrap)(customMapFunctions);

If you invert the order of the arguments to your HOC

export default (mapFunctions) => (WrappedComponent) => {

then you can simplify the composition a bit

export default compose(
  connect(mapStateToProps, mapDispatchToProps),
  YourCustomHOC(customMapFunctions),
)(ComponentYouWantToWrap);