As indicated by your reply to my comment on the question, the following example strings are to be transformed as indicated:
'(aaa) (bbb (ccc)) ddd' => '(aaa) (bbb {ccc}) ddd'
'(aaa (eee)) (bbb ccc) ddd' => '(aaa {eee}) (bbb ccc) ddd'
'(aaa) (ee (ff (gg))) (bbb (ccc)) ddd' => '(aaa) (ee (ff {gg})) (bbb {ccc}) ddd'
We cannot obtain those results with a single regular expression but we can do so by executing a sequence of the regular expressions
r'\(([^()]*)\)(?=(?:[^()]*\)){n})'
for n = 0, 1, .... and substituting matches with
r'{\1}'
If n = N is the smallest value of n for which there is no match, the desired substitution is given by the string produced by n = N-1.
I have assumed the string has balanced parentheses.
The strings 'a(b X c)' and 'a(b(c(d X e)f)g)' have balanced parentheses; 'a(b(c(d X e)fg)' and 'a(b))cd X ((efg)' do not.
The nesting level of any character in the string equals the number of right parentheses that follow before a left parenthesis is encountered (equivalently, the number of left parentheses that precede before a right parenthesis is encountered). The nesting levels of 'X' in the following strings are as shown:
String Nesting level
_____________________________
a X b 0
a(b X c) 1
a(b(c X d)e)f 2
a(b(c(d X e)f)(g)) 3
Consider the string
'(aaa) (ee (ff (gg))) (bbb (ccc)) ddd'
We first set n = 0 to obtain
r'\(([^()]*)\)(?=(?:[^()]*\)){0})'
Demo 0 shows that the substitution of matches produces the string
{aaa} (ee (ff {gg})) (bbb {ccc}) ddd
Now set n = 1 to produce the regular expression
\(([^()]*)\)(?=(?:[^()]*\)){1})
Demo 1 shows that the substitution of matches produces the string
(aaa) (ee (ff {gg})) (bbb {ccc}) ddd
Next set n = 2 to produce the regular expression
\(([^()]*)\)(?=(?:[^()]*\)){2})
Demo 2 and Python demo show that the substitution of matches produces the string
(aaa) (ee (ff {gg})) (bbb (ccc)) ddd
Next set n = 3 to produce the regular expression
\(([^()]*)\)(?=(?:[^()]*\)){3})
Demo 3 shows that there are no matches. We therefore conclude that n = 2 is the greatest level of nested parentheses, so the desired substituted string must be that produced when n = 2:
(aaa) (ee (ff {gg})) (bbb (ccc)) ddd
Demo 4 illustrates that there may be ties.