# If you'd like to omit non-matching lines from the result; add ';d' to the end of the expression.
sed -E 's/(\(([^(){},]+|\((?:[^(),]+,[^(),]+)\)|({(?:[^{}]+|(?-1))++}))(?:,(?2))+),/($1),/gm;t' <<< "Should work for:
(1,2,3)
(1,2,3,4)
((1,2),3,4)
(((1,2,3),4),5)
({x},{y},{z})
(1,(2,3),4)
((1,2),(3,4),5)
((1,2),3,(4,5))
(1,(2,(3,4),5))
(1,(2,({3},4),5))
(1,{{1},{1,2}},3)
Should not work for:
(1,2)
(10,10)
(1,10)
((1,2),3)
(((1,1),1),1)
({0},1)
({{x},{x,y}},z)"
Please keep in mind that these code samples are automatically generated and are not guaranteed to work. If you find any syntax errors, feel free to submit a bug report. For a full regex reference for SED, please visit: https://www.gnu.org/software/sed/manual/html_node/The-_0022s_0022-Command.html