Trying to build apollo-server with babel returns syntax error

I am trying to build my node app which depends on Apollo-server using babel. But when I try to build using babel I get the following error:

SyntaxError: /Users/saokanneh/Projects/memento/node_modules/@apollo/protobufjs/cli/wrappers/es6.js: Unexpected token (1:27)

> 1 | import * as $protobuf from $DEPENDENCY;
    |                            ^
  2 | 
  3 | $OUTPUT;
  4 | 
    at Parser._raise (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:723:17)
    at Parser.raiseWithData (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:716:17)
    at Parser.raise (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:710:17)
    at Parser.unexpected (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:8610:16)
    at Parser.parseImportSource (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:12265:41)
    at Parser.parseImport (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:12259:24)
    at Parser.parseStatementContent (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:11014:27)
    at Parser.parseStatement (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:10916:17)
    at Parser.parseBlockOrModuleBlockBody (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:11490:25)
    at Parser.parseBlockBody (/Users/saokanneh/Projects/memento/node_modules/@babel/parser/lib/index.js:11477:10) {
  loc: Position { line: 1, column: 27 },
  pos: 27,
  code: 'BABEL_PARSE_ERROR'
}

I have tried to look up this error but can't find a solution anywhere.

My babel setup in package.json looks like this:

"babel": {
    "presets": [
      "@babel/preset-env"
    ],
    "plugins": [
      "@babel/plugin-transform-modules-commonjs"
    ]
  }