We can fit a LinearRegression model on the regression dataset and retrieve the coeff_ property that contains the coefficients found for each input variable. These coefficients can provide the basis for a crude feature importance score. This assumes that the input variables have the same scale or have been scaled prior to fitting a model.
What about Bert? Can we get coef_ variable from the model and use it to calculate feature importance like LinearRegression model in text classification task?