Importing files in subdirectories when using Yarn Workspaces and Typescript

I'm having trouble porting a Yarn Workspaces monorepo to TypeScript. In the previous working version of the monorepo, moduleA could import something from a subdirectory of moduleB.

The new ported code attempts the same thing...

/* monorepo/packages/moduleA/src/helpers/misc.ts */

export const someHelper = () => console.log('hello world')

/* monorepo/packages/moduleB/src/index.ts */

import {someHelper} from 'moduleA/src/helpers/index.js'

This yields the error:

Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'monorepo/node_modules/moduleA/src/helpers/misc.js' imported from monorepo/packages/moduleB/lib/index.js

My TsConfig for both packages:

{
  "compilerOptions": {
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true,
    "composite": true,
    "outDir": "lib",
    "rootDir": "src",
    "allowJs": true,
    "esModuleInterop": true,
    "moduleResolution": "node",
    "module": "es6"
  },
  "include": ["src"],
}

And added config for the moduleB package:

"references": [
  {
    "path": "../moduleA"
  }
]

Note that I am building moduleB with tsc --build.

One way to solve the issue would be to simply export everything I need from moduleA as named exports in the entry point of the module. The reason I don't want to do this is because they may not be needed in the corresponding distribution which gets built from various parts of the monorepo. Importing and exporting them in moduleA would mean bundling code and sometimes running code that didn't need to be run, or shouldn't be run due to the environment.

Another thing to note is that if I rename monorepo/packages/moduleA/src/helpers/misc.ts with a .js extension instead of typescript; it works.

I can also seemingly change the error by adding this to the "compilerOptions" in moduleB:

"baseUrl": "src",
"paths": {
  "moduleA/helpers/*": ["../moduleA/src/helpers/*"]
},

And changing the import statement to:

/* monorepo/packages/moduleB/src/index.ts */

import {someHelper} from 'moduleA/helpers/index.js'

But this gives me a type definition error:

src/index.ts:2:28 - error TS2307: Cannot find module 'moduleA/helpers/misc.js' or its corresponding type declarations.

Perhaps the above use of the paths in the TsConfig is not the way to go.

To summarise, the ultimate goal here is to allow importing of any of the files from moduleA's subdirectories to use in moduleB, without having to publish a separate module for that purpose.