What is the difference between NightwatchJS and WebdriverIO?

This question has become a real fork in the road for a lot of web-automation POC and/or MVP implementations. I don't want it to dictate the wrong direction!

Thus, I feel it needs a more thorough answer, with a broader reach (ease of installation, dependencies, important plugins, services, reporters, documentation, support, & other integrations) and hopefully a more subjective delivery than the accepted answer.

❒ TL;DR: (for the lazies!)

➮ if you're a beginner, starting out with web automation, or you want to build-up the test-coverage for a small/medium-sized web app, then choose any of the two! (might as well do a coin-flip!) The differences between the two are very slim at a lower level. The cracks become gaps only when you reach more complex challenges, later down the road.

➮ if you're looking into building a full-fledged, heavyweight automation harness in order to assure the quality for a big corporate web app, or a massive web portal, then read the entire post (or take my advice & go WDIO!)

❒ Credentials: (what recommends me to have a say in this?)

I have been working with NodeJS based test frameworks extensively since early 2013, having occupied various QA automation engineering roles from entry-level, to senior, to QA Lead. I'll summarize my working experience with these frameworks & web automation:

• WebdriverIO (+4 years)
• Nightwatch (+2 year)
• built, maintained & extended several frameworks using these tools, assuring quality over all major browsers & environments (desktop & mobile)
• kept several WebdriverIO & NightwatchJS talks & presentations at various meetups/conferences
• fairly familiar with the entire NodeJS testing environment (apart from WebdriverIO & Nightwatch, also worked with: WebdriverJS (original fork for WebdriverIO), Nightmare, Robot & as of recently, Puppeteer).

❒ Recommandation: If you have to pick between the two for the implementation of a POC, I recommend you read through this entire answer. Only then can you have a complete outlook on the entire tableau.

❒ Project Details:

.-------------.---------.-----------.----------.-------------.--------------.
|   project   | ⭐️stars | forks | issues|  open PRs |   updated  |
:-------------+---------+-----------+----------+-------------+--------------:
| webdriverio |    5050 |      1450 |       68 |           6 |  12.Jul.2019 |
:-------------+---------+-----------+----------+-------------+--------------:
| nightwatch  |    9450 |       930 |  116|⁺306|          15 |  01.Jul.2019 |
'-------------'---------'-----------'----------'-------------'--------------'

⁺ - the second value represents the sum of Open Issues ( open label ) & Stale Issues ( stale label, 238 of them ). If you're curious, (I know I was!) stale represents the label given to open tickets that have been inactive for a while, using a bot. (That's pretty suspicious!)

❖ Shallow Takeaways: (this is a personal interpretation of the above numbers, so please take them with a grain of salt)

➮ ( Strength ) NightwatchJS is a more widely used solution (idea enforced by the total number of Github stars), it also rapidly gained notoriety after being integrated and Evangelized upon by the QA teams in several big companies (e.g: LinkedIn, AirBnB, GoPro, etc.)

➮ ( Strength ) Due to its robustness, modularity & extensibility, WebdriverIO boasts a higher number of forks. Recent integrations with a lot of popular & on demand services (e.g: ChromeDevTools, React & VisualRegression plugins) have also strengthened the testing community's trust in the solution

➮ ( Weakness ) Even though this isn't a tech-specific critique, I lately have become displeased with the constant unprofessionalism surrounding the NightwatchJS team. So, apart from the stale tickets, you can constantly find your perfectly valid BUG, suddenly closed, with little to no information. Sometimes, you get something funny like this.

➮ ( Opportunity ) The next two categories (Open Issues, Open PRs) are actually a true image of the commit-to-contributor ratio: NightwatchJS is mainly maintained by Andrei Rusu and a handful of others, while WebdriverIO stands out as a project led by main contributor Christian Bromann, and backed by a very active & effervescent community.

➮ ( Subjective ) Personally, I sometimes get the feeling that Nightwatch, as a web-automation framework, has been semi-shelved already and the main contributor is more interested in generating revenue from the users, than fixing the current issues. Don't get me wrong... I have nothing against marketing & promoting oneself, but I would much rather put effort back into the product & the respect for my active users, FIRST!

❒ Disclaimer!

The following grades (on a scale of 1-5 ⭐️) are my personal assessment after having worked extensively with both frameworks. They don't reflect anything other than my own experience with the given tools.

❒ Setting up a Project:

• WebdriverIO (5/5 ⭐️)
• NightwatchJS (4.0/5 ⭐️)

Setting up a WDIO, or Nightwatch, or any NodeJS-based test framework from nowadays for that matter is pretty straightforward:

➊ install the package:

npm install --save webdriverio
// or
npm install --save nightwatch

➋ install & fire-up a Selenium Server package:

npm install --save-dev selenium-standalone
// don't forget to ...
cd ./node_modules/.bin
./selenium-standalone install
./selenium-standalone start

➌ create a quick test folder structure:

./
├── /test
|   ├── /features        (test-files go here)
|   |   └── test.js      (your first test here)
|   ├── /pages           (page-objects go here)
|   ├── /assets          (everything goes in here)
|   └── /custom_commands (custom methods go here)
└── package.json

➍ generating the configuration file (wdio.conf.js, or nightwatch.conf.js):

➮ ( Strength ) it's at this point that WebdriverIO gets an advantage, as it comes out-of-the-box with a custom-made CLI wizard for an easy & straightforward configuration of the wdio.conf.js file (you can start the wizard via the wdio config command)

➮ with Nightwatch, you have to copy-paste a mock nightwatch.conf.js file from the Getting Started section

➮ it's not that Nightwatch is lacking in this regard, just that I personally feel it leaves more room for interpretation on certain aspects, such as what config keys are mandatory, what is nightwatch.json and how is it different than nightwatch.conf.js, etc. It just seems that after reading the initial config documentation, I leave with more questions than answers.

❖ WebdriverIO's Configuration Helper (CLI wizard):

❒ Writing & Running our first Test:

• WebdriverIO (5/5 ⭐️)
• NightwatchJS (4.5/5 ⭐️)

OK, you have now finally set up your project, you have all your software dependencies in check, not it's time to write our first test. Your project at this current step should look like this:

./
├── /test
|   ├── /features
|   |   └── test.js
|   ├── /pages
|   ├── /assets
|   └── /custom_commands
├── wdio.conf.js or nightwatch.conf.js
└── package.json

I really like & appreciate that both Nightwatch & WebdriverIO have some nice, out-of-the-box, working examples on their respective homepages. Plug & play!

❖ NightwatchJS example: (tests a Google search for Rembrandt)

module.exports = {
  'Demo test Google' : function (client) {
    client
      .url('http://www.google.com')
      .waitForElementVisible('body', 1000)
      .assert.title('Google')
      .assert.visible('input[type=text]')
      .setValue('input[type=text]', 'rembrandt van rijn')
      .waitForElementVisible('button[name=btnG]', 1000)
      .click('button[name=btnG]')
      .pause(1000)
      .assert.containsText('ol#rso li:first-child',
        'Rembrandt - Wikipedia')
      .end();
  }
};

❖ WebdriverIO example: (tests a DuckDuckGo search for WebdriverIO)

❯ wdio-v5 example:

describe('DuckDuckGo - Search Test, () => {

    it('Should test the DuckDuckGo search page', async () => {

        await browser.url('https://duckduckgo.com/');
        await browser.setValue('#search_form_input_homepage', 'WebdriverIO');
        await $('#search_button_homepage').click();
        // 
        const title = await browser.getTitle();
        assert.equal(title, 'WebdriverIO at DuckDuckGo', `Checking the title ...`);
    });
});

❯ wdio-v4 example:

const webdriverio = require('webdriverio');
const options = { desiredCapabilities: { browserName: 'chrome' } };
const client = webdriverio.remote(options);

client
  .init()
  .url('https://duckduckgo.com/')
  .setValue('#search_form_input_homepage', 'WebdriverIO')
  .click('#search_button_homepage')
  .getTitle().then(function(title) {
     console.log('Title is: ' + title);
     // outputs: "Title is: WebdriverIO (Software) at DuckDuckGo"
  })
  .end();

This becomes especially handy when you want to show someone a fast test framework deployment, or teach an automation course & you're missing a working example to test the work at hand.

➮ ( Strength ) Right before writing your first test, WebdriverIO gives you a decision to make. Do you want to write your code synchronously, or asynchronously? This offers immense flexibility, off the bat, letting you choose the way in which you want to write your tests (using the sync flag).

❗Note: This was a wdio-v4 specific feature! The new WebdriverIO implementation (wdio-v5) focuses on a synchronous approach to test writing, whilst leveraging the asynchronous calls under the hood. For the sake of maintaining a faithful representation of the old version, I'll keep the below examples, though they are no longer valid for wdio-v4.

// By default WebdriverIO commands are executed in a synchronous way using
// the wdio-sync package. If you still want to run your tests in an async way
// e.g. using promises you can set the sync option to false.
sync: true,

❖ sync: true example:

describe('DuckDuckGo - Search Test, () => {

    it('Should test the DuckDuckGo search page', () => {

        browser.url('https://duckduckgo.com/');
        browser.setValue('#search_form_input_homepage', 'WebdriverIO');
        browser.click('#search_button_homepage');
        const title = browser.getTitle();
        console.log('Title is: ' + title);
    });
});

❖ sync: false example:

describe('DuckDuckGo - Search Test, () => {

    it('Should test the DuckDuckGo search page', () => {

        return browser
            .url('https://duckduckgo.com/')
            .setValue('#search_form_input_homepage', 'WebdriverIO')
            .click('#search_button_homepage')
            .getTitle().then(function(title) {
                console.log('Title is: ' + title)
                // outputs: "Title is: WebdriverIO (Software) at DuckDuckGo"
            });
    });
});

❒ Page Objects:

• WebdriverIO (5/5 ⭐️)
• NightwatchJS (5/5 ⭐️)

Nowadays it's pretty much impossible having a discussion about web automation & not ending up in a heated discussion about page objects, their usefulness, their implementation, or the Page Object Model in general.

Before we nosedive into how page-objects can be implemented in these two NodeJS test frameworks, I feel we have to understand WHY? we're using them.

WHY? (why are we using page-objects?)

There's a saying, don't reinvent the wheel, so I'm not going to. Instead, I'll quote ThoughtWork's Martin Fawler who IMHO said it best:

"When you write tests against a web page, you need to refer to elements within that web page in order to click links and determine what's displayed. However, if you write tests that manipulate the HTML elements directly your tests will be brittle to changes in the UI. A page object wraps an HTML page, or fragment, with an application-specific API, allowing you to manipulate page elements without digging around in the HTML."

WHAT? (what do page-objects provide us with?)

• they create a separation of concern: test vs. page
• they create a high-level abstraction of the app
• they (should) contains all static information about the page
• they offer the ability to redesign the app w/o changing the tests
• they can represent any element (object) on the page, or the whole page itself

HOW? (how do we create such page-objects?)

Enough blabbering! Let's see how we can easily implement some page-objects inside our test framework.

WebdriverIO page-object example:

❖ page.js (this is your page-object base)

export default class Page {
    open (path) {
        browser.url(path);
        // !Note: if you want to make sure your page was completely
        //        loaded, then maybe add some logic to wait for
        //        document.readyState = "complete", or "interactive"
        // Use a /custom_command for this! Go D.R.Y. principle! RAWWWWR!
    }
}

❖ form.page.js (this is a login form page-object example):

import Page from './page'

class FormPage extends Page {
    //
    // > define your elements here <
    //
    get username () { return $('#username') }
    get password () { return $('#password') }
    get submitButton () { return $('#login button[type=submit]') }
    get flash () { return $('#flash') }
    //
    // > define or overwrite page methods here <
    //
    open () {
        super.open('login')
    }

    submit () {
        this.submitButton.click()
    }
}

export default new FormPage()

NightwatchJS page-object example:

❖ homepage.js (this is a homepage page-object example):

const homepageCustomCommands = {

  checkHomePage: function() {
    this.api.perform((done) => {
      //
      // > do some homepage checks here <
      //
    });
    return this;
  }
};

const homepage = {
  url() {
    return `${this.api.globals.baseUrl}/homepage`;
  },
  elements: {
    'generic': '#generic',
    'elements': '#elements',
    'gohere': '#gohere',
    sections: {
      header: {
        selector: '#header',
        elements: {
          'these': '#these',
          'are': '#are',
          'your': '#your',
          'selectors': '#selectors'
        },
      },
      footer: {
        selector: '#footer',
        elements: {
          // > footer selectors go here <
        },
      },
    },
  }
  commands: [homepageCustomCommands]
};

module.exports = homepage;

❒ Documentation & Support:

• WebdriverIO (5/5 ⭐️)
• NightwatchJS (3/5 ⭐️)

Both NightwatchJS as well as WebdriverIO have great support in regards to documentation:

❯ WebdriverIO: Getting Started | API DOCs | Gitter Chat

❯ NightwatchJS: Getting Started | API DOCs

➮ ( Strength ) both projects have very clean & informative documentation to go along with great examples

➮ ( Opportunity ) as a remark, I sometimes found myself searching for solutions to one, or more Nightwatch feature issues, or framework limitations, only to find the solution on some back-alley gist, or backwater blog. I would be nice if such answers & contributions (like the feature PR submissions left open) would be centralized & documented

➮ ( Weakness ) I don't know for sure where the bottleneck resides regarding Nightwatch's slow development cycle & lackluster (at best) support for community BUGs, feature-requests, or even submitted PRs. In my eyes, this appears even more as a flaw in contrast to the vibrant development community around WDIO, or the helpful Gitter chat channel

❒ To be continued ...

❒ Reporting

❒ CI/CD System Integration

❒ Architectural Differences