What is the difference between Dialogflow bot framework vs Rasa nlu bot framework?

I think I can answer this without any bias, granted that overtime the answer will grow outdated as the two services evolve.

Cliffnotes version:

Dialogflow is a complete closed source product with a fully functional API and graphical web interface. Rasa (NLU + Core) are open source python libraries that require slightly lower level development. Both try to abstract some of the difficulty of working with Machine Learning to build a chatbot.

As of writing this however here is my comparison:

DialogFlow

• Is a mostly complete tool for the creation of a chatbot. Mostly complete meaning that it does almost everything you need for most chatbots.
• Specifically it can handle classification of intents and entities. It uses what it calls context to handle dialogue. It allows web hooks for fulfillment.
• One thing it does not have that is often desirable for chatbots is some form of end user management.
• It has a robust API, which allows you to define entities/intents/etc either via the API or with their web based interface.
• Formerly known as API.ai before being acquired by Google.
• Data is hosted in the cloud and any interaction with API.ai require cloud related communications.
• Cannot be operated on premise.

Rasa NLU + Core

• To get close to the same level of fucntionality as Dialogflow you have to use both Rasa NLU and Rasa Core. Rasa NLU handles projects/intents/entities whereas Rasa Core handles dialogue and fulfillment.
• Rasa doesn't provide a complete open source GUI leaving most of your interactions with NLU in JSON or markdown. And Rasa Core requires direct python development to customize your bot.
• Also does not directly offer any sort of user info management.
• The Rasa team does not provide hosting (at least outside of their enterprise offerings) and you will be responsible for hosting and thus ownership of the data.
• Can be operated on premise.

As far as other open source frameworks, I would say that it is very likely that most chatbot frameworks right now are built on a variety of open source tools, with some proprietary add-ons. So you can always start from the lower level open source tools like MITIE or spaCy.

Update:

The Smart Platform Group (of which I am a member) recently released a product in between Rasa NLU/Core and Dialogflow called Articulate.

Articulate is a full-featured bot framework, based on Rasa NLU, that lets you build Natural Language Agents effortlessly.

• Uses Rasa NLU for understanding and custom context based code for dialog. This makes it work closer to how Dialogflow does than Rasa Core.
• HTTP API for creating intents, entities, and interacting with agents.
• GUI similar to Dialogflow that is fully open source.
• Data and interface can be hosted in the cloud or on premise.

Dialogflow:

No installation, get started immediately

Easy to use, non-techies can also build bots

Closed system

Web-based interface for building bots

Data is hosted on the cloud

Can’t be hosted on your servers or on-premise

Out of box integration with Google Assistant, Skype, Slack, Fb messenger, etc

Rasa:

Requires installation of multiple components

Requires tech knowledge

Open-source, code available in Github

No interface provided, write JSON or markdown files

No hosting provided (at least in the free version) Host it on your server

No out of box integration

Source: https://www.kommunicate.io/blog/dialogflow-vs-rasa-which-one-to-choose/

The most important difference is, the entire NLU, NLP and NLG is not happening under the hood in case of Rasa. It's open source. You are the boss. In case of Dialogflow, you have all the functionalities but it has to send the data to cloud service every time a dialog transaction happens. Also some of the service providers have limits on number of dialogs per day.

However Dialogflow is flawless, simple to use and easy to model.

Microsoft's bot framework is also open source

https://github.com/microsoft/botframework-sdk

For nlp it is typically paired with LUIS, and LUIS is not open source.

SpaCy however is an open source nlp (the one that RASA uses also). It would be a completely valid workflow to create a IReconizer in bot framework to use SpaCy https://spacy.io/

There are a hand full of chat engines that also use SpaCy open source NLP that are linked on their site here https://spacy.io/universe/category/conversational.