Detect flickering in list of durations values

Question

I'm trying to get an average of flickering data that comes from a device that sends me a value periodically.

For instance, it sends me 5 values in a window of 1 minute, then the next value will come in one hour, and again one value in one minute, and the next value in several hours.

In terms of code, let's say that I have a List of Tuple. I've defined a threshold value that is, say, 15 minutes.

var flickeringThreshold = 15;

var flickeringList = new List<(DateTime, int)>(8);

// I'd like to regroup all these because the TimeSpan resulting of
// the substraction of two Dates Values  ElementAt(index n+1) - ElementAt(index n)
// is under the threshold.
// By regroup I mean weight average the values and take the date 
// when the flickering begins, but this not the issue here.
flickeringList.Add((new DateTime(2022, 3, 25, 9, 2, 5, DateTimeKind.Local), 3));
flickeringList.Add((new DateTime(2022, 3, 25, 9, 2, 10, DateTimeKind.Local), 5));
flickeringList.Add((new DateTime(2022, 3, 25, 9, 2, 15, DateTimeKind.Local), 3));
flickeringList.Add((new DateTime(2022, 3, 25, 9, 2, 20, DateTimeKind.Local), 5));
flickeringList.Add((new DateTime(2022, 3, 25, 9, 2, 25, DateTimeKind.Local), 3));
            
// This one lives on her own because the difference with the previous value in the list is over the threshold
flickeringList.Add((new DateTime(2022, 3, 25, 11, 4, 0, DateTimeKind.Local), 2));
            
// This one is alone no flickering
flickeringList.Add((new DateTime(2022, 3, 25, 11, 4, 30, DateTimeKind.Local), 3));

// This one lives on her own because the difference with the previous value in the list is over the threshold
flickeringList.Add((new DateTime(2022, 3, 25, 12, 7, 25, DateTimeKind.Local), 5));

My first approach to this problem was to use a for loop to compare elements. There would be a boolean value to signal that the flickering starts and stops...

My problem is that, with the example below, I can't seem to find a way to not take into account the 6th value when looping...

Pseudo code:

for (int i = 0; i < flickeringList.Count; i++)
{
    var level = flickeringList[i].Item2;
    var nextLevel = i < flickeringList.Count - 1 ? flickeringList[i + 1] : default;

    DateTime forDurationStart = flickeringList[i].Item1;
    DateTime forDurationEnd = i < flickeringList.Count - 1 ? (DateTime)flickeringList[i + 1].Item1 : default;

    if ((forDurationEnd - forDurationStart).TotalMinutes < flickeringThreshold)
    {
        // Flickering detected
        continue;
    }
    else
    {
        // We've gone past flickering...
    }
}

How can I solve my problem ?

I've found a way to filter out the undesired row (using Lag from MoreLINQ) but lost the data to weight average in the process:

var filteredList = flickeringList
    .OrderBy(e => e.Item1)
    .Lag(1, (e, lag) => new
    {
        Event = e,
        PreviousItem = lag,
    })
    .Where(x => x.PreviousItem == default || (x.Event.Item1 - x.PreviousItem.Item1).TotalMinutes > flickeringThreshold)
    .Select(x => x.Event);

user_559347 · Answer

If I've understood correctly, a period of flickering starts when we get a "reading" and includes all subsequent "readings" for a period of threshold. If there are no subsequent readings before threshold expires, then this is not a flicker.

This solution builds on your simple loop, though I think the queue based loop in @amir-keibi's solution might be better in the final analysis (you might want a timeout for the dequeue equivalent to threshold to detect the end of flickering before the next data item is received, rather than waiting an indeterminate amount of time).

I use a single Tuple to store enough information about a period of flickering that it is not necessary to keep a store the individual items.

Note that the question references a "weighted average", but there's no indication of how the weighting is calculated, so I've assumed it's time weighted.

// Sample data from original question
var flickeringList = new List<(DateTime, int)>
{
    (new DateTime(2022, 3, 25, 9, 2, 5, DateTimeKind.Local), 3),
    (new DateTime(2022, 3, 25, 9, 2, 10, DateTimeKind.Local), 5),
    (new DateTime(2022, 3, 25, 9, 2, 15, DateTimeKind.Local), 3),
    (new DateTime(2022, 3, 25, 9, 2, 20, DateTimeKind.Local), 5),
    (new DateTime(2022, 3, 25, 9, 2, 25, DateTimeKind.Local), 3),
    // This one lives on her own because the difference with the previous value in the list is over the threshold
    (new DateTime(2022, 3, 25, 11, 4, 0, DateTimeKind.Local), 2),
    // This one is alone no flickering
    (new DateTime(2022, 3, 25, 11, 4, 30, DateTimeKind.Local), 3),
    // This one lives on her own because the difference with the previous value in the list is over the threshold
    (new DateTime(2022, 3, 25, 12, 7, 25, DateTimeKind.Local), 5),
};


// local func to initialize a flicker period from an item.
(int itemCount, double weightedAvg, double totalWeight, DateTime firstItemDate, DateTime lastItemDate, int lastItemValue) 
    InitFlickerPeriod((DateTime dateTime, int value) valueTuple)
{
    return (1, 0, 0, valueTuple.dateTime, valueTuple.dateTime, valueTuple.value);
}


(int itemCount, double weightedAvg, double totalWeight, DateTime firstItemDate, DateTime lastItemDate, int lastItemValue )
    UpdateToFlickerPeriod(
        (int itemCount, double weightedAvg, double totalWeight, DateTime firstItemDate, DateTime lastItemDate, int lastItemValue)
            periodTuple, (DateTime dateTime, int value) valueTuple)
{
    var weightOfPriorPointInSeconds = (valueTuple.dateTime - periodTuple.lastItemDate).TotalSeconds;
    
    return (
            itemCount: periodTuple.itemCount+1,
            weightedAvg: ((periodTuple.weightedAvg*periodTuple.totalWeight)+(weightOfPriorPointInSeconds*periodTuple.lastItemValue)) / (weightOfPriorPointInSeconds+periodTuple.totalWeight),
            totalWeight: periodTuple.totalWeight+weightOfPriorPointInSeconds,
            firstItemDate: periodTuple.firstItemDate,
            lastItemDate: valueTuple.dateTime,
            lastItemValue: valueTuple.value
            );
}

void DoSomethingWithFlickerPeriod(
    (int itemCount, double weightedAvg, double totalWeight, DateTime firstItemDate, DateTime lastItemDate, int lastItemValue) flickerTuple)
{
    Console.WriteLine($"Flicker consisting {flickerTuple.itemCount} readings with a weightedAvg value of {flickerTuple.weightedAvg} started at {flickerTuple.firstItemDate} and ended at {flickerTuple.lastItemDate}.");    }



// Define the threshold
var threshold = TimeSpan.FromMinutes(15);


// In keeping with the question, use a Tuple to store details of a period of flickering.  The items represent:
// the number of data items in the weightedAvg;
// the 'weightedAvg';
// the total weight of the weighted average;
// the DateTime of the first item;
// the DateTime of the last item;
(int itemCount, double weightedAvg, double totalWeight, DateTime firstItemDate, DateTime lastItemDate, int lastItemValue) flickerPeriod = default;



foreach ((DateTime dateTime, int value) flickerDataPoint in flickeringList)
{
    if (flickerPeriod == default)
    {
        // this must be the first 'flicker' in the period of flickering.  Init the filckerPeriod.
        flickerPeriod = InitFlickerPeriod(flickerDataPoint);
        continue;
    }

    // If the current data point occurred within the threshold of time since the last one, then we add it to the flicker period.
    if ((flickerDataPoint.dateTime - flickerPeriod.firstItemDate) <= threshold)
    {
        flickerPeriod = UpdateToFlickerPeriod(flickerPeriod, flickerDataPoint);

        continue;
    }

    // If we get here, then the new data point does not belong with the prior period of flickering.
    // Assume that flickering occurs only if there is more than one value.
    if (flickerPeriod.itemCount > 1)
        DoSomethingWithFlickerPeriod(flickerPeriod);

    // ...and re-initialize the flickerPeriod.
    flickerPeriod = InitFlickerPeriod(flickerDataPoint);

}

// Deal with the final period if there is one.
if (flickerPeriod != default && flickerPeriod.itemCount > 1)
    DoSomethingWithFlickerPeriod(flickerPeriod);

Generates the following output (which is slightly at odds with the comments in the sample data - perhaps I misunderstood):

Flicker consisting 5 readings with a weightedAvg value of 4 started at 25/03/2022 09:02:05 and ended at 25/03/2022 09:02:25.
Flicker consisting 2 readings with a weightedAvg value of 2 started at 25/03/2022 11:04:00 and ended at 25/03/2022 11:04:30.

user_1307284 · Answer

It seems to me that you're processing a stream which means you don't know when the next value is coming. So the only thing you could do is decide based on what you already have.

Let's just say q is a blocking queue that waits for the next value to become available. Also, let's assume each incoming item has a Time and Data values (event time and the incoming item's value respectively).

var lastItem = default;
do
{
  var nextItem = q.Dequeue();

  DateTime forDurationStart = lastItem != default ? lastItem.Time : default;
  DateTime forDurationEnd = nextItem.Time;

  // keep track of the last incoming item
  // this should be moved if you still need to reference lastItem further down
  lastItem = nextItem;

  // you can extend this if-else to handle the first item differently
  // that is, when forDurationStart == default
  if (forDurationStart != default && (forDurationEnd - forDurationStart).TotalMinutes < flickeringThreshold)
  {
      // Flickering detected
      continue;
  }
  else
  {
      // We've gone past flickering...
  }
} while(lastItem.Data != "quit")

This way, we're always looking back to detect flickers. Also, for good measures, I introduced a quit command to get out of the loop. Hope this helps.

Detect flickering in list of durations values

2 Answers