How to get the text elements from python dictionaries

user_4903479 0
Viewed 62

I am trying to get some content from pandas column. The pandas dataframe is df and have a column named entities. This column seem to contain dictionaries. One of them is 'hashtags', which contains another dictionary 'text'. There can be multiple 'text' elements in a single row. Want to get the values for text field. The example below can help to understand. It will be helpful if I can get guidance on getting the hashtags values.

df['entities']

output =>

0  {'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': [{'url': '', 'expanded_url': 'https://twitter.com/i/web/status/1460865425911205891', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [117, 140]}]}
1  {'hashtags': [{'text': 'carbon', 'indices': [17, 24]}, {'text': 'nature', 'indices': [48, 55]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': '', 'expanded_url': 'https://rpujolviven.blogspot.com/2012/10/infinite-growth.html', 'display_url': 'rpujolviven.blogspot.com/2012/10/infini…', 'indices': [61, 84]}, {'url': '', 'expanded_url': 'https...
2  {'hashtags': [{'text': 'Lincoln', 'indices': [69, 77]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': '', 'expanded_url': 'https://thelincolnite.co.uk/2021/11/professor-duncan-french-the-juggernaut-of-climate-politics-rolls-on/', 'display_url': 'thelincolnite.co.uk/2021/11/profes…', 'indices': [78, 101]}]}
3  {'hashtags': [{'text': 'CBDC', 'indices': [107, 112]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': , 'expanded_url': 'https://twitter.com/i/web/status/1460865388699344900', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [115, 138]}]}

The expected output:

 0  []
 1  ['carbon', 'nature']
 2  ['Lincoln']
 3  ['CBDC']
tweepy dataframe python-3.x dictionary python
1 Answers

try df['entities'].map(lambda x:[i['text'] for i in x['hashtags']]).

Details (I used Google colab):

import pandas as pd
a=pd.Series([ {'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': [{'url': '', 'expanded_url': 'https://twitter.com/i/web/status/1460865425911205891', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [117, 140]}]}
,{'hashtags': [{'text': 'carbon', 'indices': [17, 24]}, {'text': 'nature', 'indices': [48, 55]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': '', 'expanded_url': 'https://rpujolviven.blogspot.com/2012/10/infinite-growth.html', 'display_url': 'rpujolviven.blogspot.com/2012/10/infini…', 'indices': [61, 84]}, {'url': '', 'expanded_url': 'https...'}]}
,{'hashtags': [{'text': 'Lincoln', 'indices': [69, 77]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': '', 'expanded_url': 'https://thelincolnite.co.uk/2021/11/professor-duncan-french-the-juggernaut-of-climate-politics-rolls-on/', 'display_url': 'thelincolnite.co.uk/2021/11/profes…', 'indices': [78, 101]}]}
,{'hashtags': [{'text': 'CBDC', 'indices':[107, 112]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': '', 'expanded_url': 'https://twitter.com/i/web/status/1460865388699344900', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [115, 138]}]}])
df = a.to_frame(name="entities")
df['entities']
df['entities'].map(lambda x:[i['text'] for i in x['hashtags']])

Output:

0    {'hashtags': [], 'symbols': [], 'user_mentions...
1    {'hashtags': [{'text': 'carbon', 'indices': [1...
2    {'hashtags': [{'text': 'Lincoln', 'indices': [...
3    {'hashtags': [{'text': 'CBDC', 'indices': [107...
Name: entities, dtype: object

0                  []
1    [carbon, nature]
2           [Lincoln]
3              [CBDC]
Name: entities, dtype: object

Reference: https://pandas.pydata.org/docs/reference/api/pandas.Series.map.html

user_113331720
Related