Check two big objects for one property

user_8086644 0
Viewed 66

Description

I´ve got a big object from a database by using a sqlite database with entity framework. Then I need to check if some entry´s from the database are not exist anymore by the new dataset I got from a import file. The objects have a length by ~120.000. The only I found was with the where and any extensions but this take minutes of time. Is there any solution that will be more efficent?

Source

var delData = old.Where(o => !content.Data.Any(n => n.Row == o.Row));

Class of the list

namespace GCS.Db
{
    public class GcsContent
    {
        public int Id { get; set; }
        public int HeadlineId { get; set; }
        public int TableId { get; set; }
        public int Row { get; set; }
        public string Content { get; set; }
    }
}

Example of htm list (content and old variables)

<head>
    <meta http-equiv="Content-Type" content=''application/vnd.ms-excel; charset=utf-8"/>
</head>
<table>
    <tr>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
      <th>Header</th>
    </tr>
    <tr>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
      <td>Cell</td>
    </tr>
c#
2 Answers

if understand correctly, you compare each element of old object to each element of new object. This causes exponential growth of combinations. Do you want to detect only deleted rows ? if yes, having identification of such rows (as id autoincrement) i think is not sufficient because you still need to iterate over whole collection to detect missing elements.

I would mark deleted rows with additional html tag so detection would require only one pass

user_29637520

First Solution

Cause the Row column isn´t unique I could use the .distinct to get fast the new row size of the htm. After that I can delete all other datasets which have a higher row then the new dataset with RemoveRange.

Source

    var rows = content.Data.Select(x => x.Row).Distinct().Count() - 1;
    var delData = old.Where(x => x.Row > rows);

I would mark deleted rows with additional html tag so detection would require only one pass

If you mean edit the htm it´s not possible cause it´s a auto generated table. I don´t have access to the source there.

user_80866440
Related