Many to many filtering in Message Broker

user_945343 0
Viewed 87

I have a Person object in the system. When a Person does some action there is an Administrator actor who is interested in monitoring these kind of events.

Person
{
   Id: string
}
PersonAction
{
   ActionType: enum
   PersonId: string
}

Currently I have this subscription implemented throught ServiceBus topic and subscriptions: Administrators subscribe to actions of all Persons in the system:

  • Azure Service Bus broker has PersonActions topic.
  • Every time when Person does any action a PersonAction event is sent to the Topic.
  • Every Administrator creates it's own subscription to the topic and monitors all Persons actions.

Now I have a new requirement that introduces grouping of Persons and I need a way to allow Administrators to subscribe to PersonActions events based on groups they want to monitor:

  • Persons can be part of one ore more groups.
  • Administrators are interested in monitoring groups of Persons and, hence, receiving all PersonAction events for groups they are monitoring.
  • Administrators may subscribe to one or several groups.

Here are my thoughts how to do this:

  • Add to PersonAction a routing property that will contain information about groups this Person is member of
  • When Administrator creates new subscription he will specify a set of groups that he wants to monitor and it should be than somehow used in subscription filter to filter PersonAction messages in the Topic.

So, cutting to the case, I want to leverage Service Bus Topic filtering capabilities to deliver PersonAction messages specificaly to Administrators that are interested in them based on Groups.

In general this doesn't seem to be a straightforward task to do with ServiceBus(or any other message broker) because there is a many-to-many relation: one Person can be in multiple groups and Administrator may want to subscribe to multiple groups. Usually all filters support filtering when event has a single property(like "groupId=1234") and in my case it's an array.

So far, I've came up with two solutions but don't quite like any of them:

  1. Use LIKE SqlFilter. Concatenate all groups of the Person into a single comma-separated string (groups=1,2,5,8) and than have filter groups LIKE %1% OR groups LIKE %5% (in reality group ids will be guids, so don't mind the problem with one group id being a substring of another)

  2. Add each group id as a property with an empty value and than use EXISTS filter to check if event has this group id defined. Filter would be EXISTS(1) OR EXISTS(5) and PersonAction properties: {1:null, 2:null, 5:null, 8:null}

Is there a better way to do such filtering and how is many-to-many filtering rule done in message brokers?

Answers describing this for Any message broker(not only ServiceBus) will be also extremely helpful.

azureservicebus rabbitmq message-queue
1 Answers

I'm not really that familiar with other brokers but here is something that comes to mind for Azure Service Bus.

You could have 2 (3 with bonus) level of entities instead of 1 for such a scenario

  1. The first level is a topic where all the PersonAction messages come into and would have subscriptions for each group with auto-forward setup to their own topics
  2. The second level is where each group has its own topic and administrators would subscribe to multiple topics based on the groups they want to monitor but will have to de-duplicate messages

    You could remove this level and have direct subscriptions (one per group per administrator) but would likely hit the limit of 2000 subscriptions per topic

  3. (Bonus) Auto Forward the messages from the subscriptions into administrator queues and enable Duplicate Detection

Note that the number of operations billed would increase as mentioned in the Auto Forward Considerations section of the docs

Here is a more elaborate explanation for the same

1. Input Topic
This is where the PersonAction messages would first come in.

This topic would have subscriptions that filter messages based on the group (either of your approaches; I'd prefer using a Correlation Filter since its more efficient) and would auto-forward the messages into respective topics.

2. Topic per Group
This is where the PersonAction messages filtered by group go into.

At this point, there would copies of the same message in different topics based on all of the groups the user is part of.

Administrators would create subscriptions to the topics required depending on the groups they want to monitor but will have to handle the duplicate messages that they could potentially receive.

3. (Bonus) Administrator Queues
The subscriptions created by administrators could be setup to auto-forward messages into their personal queue and these queues could have duplicate detection enabled allowing the administrators to freely process the messages as-is without worrying about duplicates.

user_104178390
Related