TLDR: Using bundler is still the best way to ship/package your application.
It is not just the question of module. It has more to do with server topology and overall bundling.
For example, you have some third party npm module which is provides as ES module aka import/export syntax. It will be typically installed within the node_modules folders and so will its dependencies. Now, you are going to use this module in your code, you would want to use it like
import xyz from 'my-npm-module';
With the above simple statement, there are multiple issues. Above type of import is non-relative import. For Node.js or bundlers like Webpack or Rollup, this is a simple matter of looking into node_modules folder starting from current directory and finding all the way up till the root partition.
Now browsers have no such knowledge of non-relative import resolution (Browsers do not use Node style module resolution). So, first question is how would a browser know that where to look for such modules on the server. Second question, assume that somehow you teach browser to use non-relative modules and follow Node style resolution algorithm, you would still need to package entire node_modules folder including sub-directories. This is very cumbersome process and needlessly inflating your final build size of the application.
These are the reasons why you still need to use bundler like Webpack and Browserify. To keep things lean, you may skip Babel if you don't want to support old JS engines. However, if you really want to use and ship ES modules to the browser, consider using Rollup.js which supports output format as ES module as against Webpack's UMD or Common.js.
Another alternative that you can consider using System.js + JSPM way of bundling application where ES modules are defaults but it does not offer flexibility like that of Webpack or Rollup. Also, there is a new bundler - Parcel.js which aims to be zero config solution and is very easy to use. You won't notice all the complex belts and whistles.
In the end, real world front-end bundling is not just about JavaScript. There are other assets like TS files, ReasonML files, Images, SVG files which you might want to transpile and bundle with final bundle with using advanced optimizations like Tree shaking, HMR (dev mode) where bundles really excel at.