MS Graph Beta API doesn't get full sign in logs using python and stops seeming randomly?

Viewed 260

We are trying to get the full signin logs using the following MS Graph Beta API request using Python (/auditLogs/signIns?$filter=signInEventTypes/any(t: t eq 'servicePrincipal')). According to Microsoft Documentation signin logs for the last 30 days are saved. Since only a certain amount of entries can be retrieved at once (1000 entries), we are using paging to get the next page using the link given in @odata.nextLink.

But here is the problem, it seems random how many pages the graph API retrieves after executing the code. Executing the same request after each other, it some times gives out around 50 pages, 70 pages and other times more or even less pages. It seems random how many pages are retrieved. Even filtering for days using for example createdDateTime ge 2021-02-25 and createdDateTime le 2021-02-27, it doesn't retrieve the whole logs for these two days. It also stops somewhere in the middle and it seems random how many pages it retrieves.

Another interesting issue is that during execution of the code, it sometimes doesn't retrieve the full 1000 entries for a page. For example on the first page 1000 entries are retrieved, for the second page only 978 entries and for the third page 1000 entries again. I don't quite understand why these issues arises but we can't reliably retrieve logs for certain timeframes. Anybody has a solution or know why it stops seeming randomly?

import os
import datetime
import json
import pandas as pd
from dotenv import load_dotenv
from azure.identity import ClientSecretCredential
from msgraph.core import GraphClient, APIVersion

load_dotenv()

tenant_id = os.getenv("AZURE_TENANT_ID")
client_id = os.getenv("AZURE_CLIENT_ID")
client_secret = os.getenv("AZURE_CLIENT_SECRET")

browser_credential=ClientSecretCredential(tenant_id, client_id, client_secret)
client=GraphClient(credential=browser_credential, api_version=APIVersion.beta)

def get_signins_info(signin_data, signin_df):
'''
Fill dataframe with signins data
signin_data:
signin_df:
'''
id_ = signin_data["id"]
app_id = signin_data["appId"]
signin = signin_data["createdDateTime"]
    
row = {"id": id_, \
       "app_id": app_id, \
       "signin": signin, \
      }

signin_df = signins_df.append(row, ignore_index=True)

return signins_df

signins_df =  pd.DataFrame(columns=["objectId","appDisplayName","appId","activity"])  
next_link = "/auditLogs/signIns?$filter=signInEventTypes/any(t: t eq 'servicePrincipal')&(createdDateTime ge 2021-02-25 and createdDateTime le 2021-02-27)"

 while next_link:  
    result = client.get(next_link)  
    json_data = result.json()  
    if "value" in json_data:  
        for log in json_data["value"]:  
            signins_df = get_signins_info(log, signins_df)  
    if "@odata.nextLink" in json_data:  
        next_link = json_data["@odata.nextLink"]  
    else:  
        next_link = False 
    print(len(signins_df))
0 Answers
Related