warden authenticate recall not getting called

I'm trying to have ajax login using devise, but I can't seem to get this working. The problem is that I have confirmable strategy enabled on devise, warden.authenticate! should fail on inactive users and render the failure action, instead it renders the default action. I notice that many tutorials have the same set ups. I am using rails 4.0.0 and devise 3.2.2.

class SessionsController < Devise::SessionsController
   def create
     #problem: failure not getting called
     resource = warden.authenticate!(scope: resource_name, recall: "#{controller_path}#failure") 
     sign_in(resource_name, resource)
     render json: {success: true}
   end 

  def failure
     render json: {success: false, errors: ["Login failed!"]}
  end
end

The devise.rb is set up as follow,

config.http_authenticatable_on_xhr = false
config.navigational_formats = ['*/*', :'*/*', :html, :json]

The routes.rb has the following,

devise_for :users, controllers: {registrations: "registrations", sessions: "sessions"}

The signin form,

 <%= form_for(resource, as: resource_name, 
    url: session_path(resource_name), 
    format: :json,
    html:{id:"signin_form"},
    remote: true) do |f| %>
<%= f.label(:email, "Email") %> 
<%= f.text_field(:email, class: "field text") %> 
<%= f.label(:password, "Password") %> 
<%= f.text_field(:password, class: "field text", id: "user_signin_password") %>  

<%= f.submit "Sign in", class: "theme_color btn key" %>
<% end %>

Since the failure action is not getting called, the default new action is rendered with response header as follow,

Content-Type   text/html; charset=utf-8
Location       http://localhost:3000/users/sign_in.js

I am also curious why it renders .js instead of .json.

Can someone point me out what I missed? Thanks