How can I extract an array from the console.log in selenium?

Question

When I try to get the console.log with

driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.log(values);')
time.sleep(4)
for entry in driver.get_log('browser'):
print(entry)

I just get the console output but not the actual contents of the array which should look like this screenhot of array

Instead I get:

{'level': 'INFO', 'message': 'console-api 2:121 Array(261)', 'source': 'console-api', 'timestamp': 1647448899327}

I´ve found a couple of suggestion on how to fix this but none of them worked for me unfortunately. Does anyone have a solution? Thanks a lot!

EDIT: I found this script (https://bgrins.github.io/devtools-snippets/#console-save) which saves the console output to a file when i execute this script manually in the browser but when I try to implement it in selenium like

driver.execute_script('(function(console){console.save=function(data,filename){if(!data){console.error("Console.save: No data")return}if(!filename)filename="console.json"if(typeof data==="object"){data=JSON.stringify(data,undefined,4)}var blob=new Blob([data],{type:"text/json"}),e=document.createEvent("MouseEvents"),a=document.createElement("a")a.download=filename a.href=window.URL.createObjectURL(blob)a.dataset.downloadurl=["text/json",a.download,a.href].join(":")e.initMouseEvent("click",true,false,window,0,0,0,0,0,false,false,false,false,0,null)a.dispatchEvent(e)}})(console)')

driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.save(values, add);')

I get this response:

[19980:18944:0316/183240.115:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 13 errno 112: database or disk is full sql: CREATE TABLE IF NOT EXISTS previews_v1 (host_name VARCHAR NOT NULL, time INTEGER NOT NULL, opt_out INTEGER NOT 
NULL, type INTEGER NOT NULL, PRIMARY KEY(host_name, time DESC, opt_out, type))
[19980:18944:0316/183240.121:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: enabled_previews_v1 sql: SELECT type, version FROM enabled_previews_v1
[19980:18944:0316/183240.132:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: enabled_previews_v1 sql: INSERT INTO enabled_previews_v1 (type, version) VALUES  (?, ?)
[19980:18944:0316/183240.180:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: previews_v1 sql: SELECT host_name, time, opt_out, type FROM previews_v1 ORDER BY host_name, time DESC
[15816:23948:0316/183240.237:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.259:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.340:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.359:ERROR:entry_impl.cc(999)] Failed to save user data
[19980:18944:0316/183240.607:ERROR:database.cc(1777)] Shortcuts SQLite error: code 13 errno 112: database or disk is full sql: COMMIT
[15816:23948:0316/183240.677:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.755:ERROR:entry_impl.cc(999)] Failed to save user data
[15816:23948:0316/183240.765:ERROR:entry_impl.cc(999)] Failed to save user data
[19980:18944:0316/183240.825:ERROR:database.cc(1777)] Shortcuts SQLite error: code 1 errno 112: no such table: omni_box_shortcuts sql: SELECT id, text, fill_into_edit, url, document_type, contents, contents_class, description, description_class, transition, type, keyword, last_access_time, number_of_hits FROM omni_box_shortcuts
[19980:19372:0316/183241.460:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: INSERT OR REPLACE INTO top_sites (url, url_rank, title) VALUES (?, ?, ?)
[19980:19372:0316/183241.468:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: INSERT OR REPLACE INTO top_sites (url, url_rank, title) VALUES (?, ?, ?)
[19980:19372:0316/183241.479:ERROR:database.cc(1777)] TopSites SQLite error: code 13 errno 0: database or disk is full sql: UPDATE top_sites SET url_rank = ? WHERE url == ?
[19980:19372:0316/183241.479:ERROR:database.cc(1777)] TopSites SQLite error: code 1 errno 0: cannot commit - no transaction is active sql: COMMIT
[19980:18944:0316/183241.550:ERROR:database.cc(1777)] Predictor SQLite error: code 13 errno 112: database or disk is full sql: COMMIT

user_1832058 · Answer

I think it is XYProblem.

You ask how to get output from console.log() but it seems real problem is how to get data from Highcharts and use in Python - and this doesn't need console.log().

You can use return values in JavaScript to get it in Python.

It doesn't even need JSON.stringify() in JavaScript and json.loads() in Python.

data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')

print(data)

EDIT:

Using .map() instead of forEarch() you can reduce JavaScript to one line:

return Highcharts.charts[0].series[0].data.map(d => d.y);

Minimal working example for https://www.highcharts.com/demo/line-basic

from selenium import webdriver
#from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager
import time

url = 'https://www.highcharts.com/demo/line-basic'

#driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())

driver.get(url)

time.sleep(5)

data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')

print(type(data), data)

# or shorter

data = driver.execute_script('return Highcharts.charts[0].series[0].data.map(d => d.y);')

print(type(data), data)

Result:

 [43934, 52503, 57177, 69658, 97031, 119931, 137133, 154175]

EDIT:

For example it can be used to convert x with timestamps (in milliseconds) to date like MM/DD/YY or YYYY-MM-DD

import datetime

data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]

new_data = []

for x in data:
    dt = datetime.datetime.fromtimestamp(x//1000)
    new_data.append(d.strftime('%D'))
    print(d.strftime('%D'), '|', d.strftime('%Y-%m-%d'))
        
print('new_data:', new_data)

Result:

03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
new_data: ['03/17/22', '03/17/22', '03/17/22', '03/17/22']

And if you keep data in pandas.DataFrame then you can use pd.to_datetime() and .dt.strftime('%D') but for calculations it can be better keep it as datetime

import pandas as pd

data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]

df = pd.DataFrame({'date': data})
print(df)

df['date'] = pd.to_datetime(df['date'], unit='ms')
df['date'] = df['date'].dt.strftime('%D')
print(df)

Result:

            date
0  1647212400000
1  1647298800000
2  1647385200000
3  1647471600000
       date
0  03/13/22
1  03/14/22
2  03/15/22
3  03/16/22

user_17616528 · Answer

I basically fixed it with an additional js

driver.execute_script("""(function(console){

    console.save = function(data, filename){

        if(!data) {
            console.error('Console.save: No data')
            return;
        }

        if(!filename) filename = 'console.json'

        if(typeof data === "object"){
            data = JSON.stringify(data, undefined, 4)
        }

        var blob = new Blob([data], {type: 'text/json'}),
            e    = document.createEvent('MouseEvents'),
            a    = document.createElement('a')

        a.download = filename
        a.href = window.URL.createObjectURL(blob)
        a.dataset.downloadurl =  ['text/json', a.download, a.href].join(':')
        e.initMouseEvent('click', true, false, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null)
        a.dispatchEvent(e)
    }
})(console)""")
driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });')
driver.execute_script('console.save(values, add);')

This will save the array to a json

How can I extract an array from the console.log in selenium?

2 Answers