$re = '/(\{(?>[^{}]+|(?1))*\})/m';
$str = 'This is working: {test1|test2|test3}, this is working {test1|{test2|test3}}, this is not working: {test1|{test2|test3}|test4}';
preg_match_all($re, $str, $matches, PREG_SET_ORDER, 0);
// Print the entire match result
var_dump($matches);
Please keep in mind that these code samples are automatically generated and are not guaranteed to work. If you find any syntax errors, feel free to submit a bug report. For a full regex reference for PHP, please visit: http://php.net/manual/en/ref.pcre.php