How to convert a list of dictionaries to a Pandas Dataframe with one of the values as column name?

From the column2, use tolist and recreate a dataframe that you stack to get one dictionary {'id':...,'value':...} per row.

s = pd.DataFrame(df['column2'].tolist()).stack()
print(s)
# 0  0    {'id': 1144801690551941, 'value': 20}
#    1    {'id': 8202109018383881, 'value': 26}
#    2    {'id': 3025222222235562, 'value': 37}
#    3    {'id': 5834245818862827, 'value': 35}
#    4    {'id': 4689782481420271, 'value': 27}
#    5    {'id': 7385168421196875, 'value': 56}
# 1  0    {'id': 1144801690551941, 'value': 25}
#    1    {'id': 8202109018383881, 'value': 26}

Then from there, use again tolist on this Series s and create a Dataframe, ensure to keep the original index. Append the column id just created with set_index, and unstack to get all id number as column header. You get the wanted shape for the id-value. Just need to join to column1.

res = (
    df[['column1']]
      .join(pd.DataFrame(s.tolist(), 
                         s.index.get_level_values(0)) # keep original index
              .set_index('id', append=True)
              ['value'].unstack()
              .rename_axis(columns=None))
)

and you get as expected

print(res)
   column1  1144801690551941  3025222222235562  4689782481420271  \
0        1                20                37                27   
1        2                25                38                21   
2        3                20                37                27   

   5834245818862827  7385168421196875  8202109018383881  
0                35                56                26  
1                35                53                26  
2                32                50                29  

You could explode the lists and then use pd.json_normalize:

df = df.explode('column2').reset_index()
df = df.join(pd.json_normalize(df['column2'])).drop('index', axis=1)
df = df.pivot_table(columns=['id'], index=['column1'], values=['value'])
df = df.droplevel(level=0, axis=1)

id       1144801690551941  3025222222235562  ...  7385168421196875  8202109018383881
column1                                      ...                                    
1                      20                37  ...                56                26
2                      25                38  ...                53                26
3                      20                37  ...                50                29

1. Convert column2 to dataframes in each cell:

>>> df['column2'] = [pd.DataFrame(col) for col in df.column2]
>>> df
   column1   column2
   <int64>  <object>
0        1  <DF 6x2>
1        2  <DF 6x2>
2        3  <DF 6x2>
>>> df.column2[0]
                 id   value
            <int64> <int64>
0  1144801690551941      20
1  8202109018383881      26
2  3025222222235562      37
3  5834245818862827      35
4  4689782481420271      27
5  7385168421196875      56

2. Unnest the dataframes:

>>> from datar.all import f, unnest, pivot_wider
>>> df = df >> unnest(f.column2)
>>> df
    column1                id   value
    <int64>           <int64> <int64>
0         1  1144801690551941      20
1         1  8202109018383881      26
2         1  3025222222235562      37
3         1  5834245818862827      35
4         1  4689782481420271      27
5         1  7385168421196875      56
6         2  1144801690551941      25
7         2  8202109018383881      26
8         2  3025222222235562      38
9         2  5834245818862827      35
10        2  4689782481420271      21
11        2  7385168421196875      53
12        3  1144801690551941      20
13        3  8202109018383881      29
14        3  3025222222235562      37
15        3  5834245818862827      32
16        3  4689782481420271      27
17        3  7385168421196875      50

3. pivot_wider():

>>> df >> pivot_wider(f.column1, names_from=f.id, values_from=f.value)
   column1  1144801690551941  3025222222235562  4689782481420271  5834245818862827  7385168421196875  8202109018383881
   <int64>           <int64>           <int64>           <int64>           <int64>           <int64>           <int64>
0        1                20                37                27                35                56                26
1        2                25                38                21                35                53                26
2        3                20                37                27                32                50                29

My solution is from 3 steps:

Step 1: Expand Data of column 2

So we are going to expand data in column2 by using recursive - .apply(pd.Series):

dfs = []

df_expand = df['column2'].apply(pd.Series)

for col in df_expand.columns:
    dfs.append(df_expand[col].apply(pd.Series))

df_temp = pd.concat(dfs)

This will result into:

Step 2: Expand Data from step 1

In this step we will use pd.pivot to convert row values to columns. Finally we will reset the MultiIndex:

df_final = pd.pivot(data=df_temp, columns=['id'], values=['value'])
df_final.columns = df_final.columns.droplevel(0)

This will result into:

Step 3: Form the final DataFrame

For the final DataFrame we will merge the result from Step 2 with column1 on index

pd.merge(df['column1'], df_final, right_index=True, left_index=True)