How to configure webpack to use babel for JSX with no transpiling to es5?

I am using webpack in my project, I use es6 so I don't need transpiling to es5. But I had to add add babel because I added some react components and I need babel for the jsx transpiling. Now when I try to use async/await the building is really huge, I suspect it is because babel now injects polyfills for that. I wonder if I can restrict babel to only process the JSX but don't do any transpiling to es5 nor inject polyfills? I am fine with using async/await in the browser.