$re = '/\b(?<![{}])(\w+)\{([^{}]*)\}/m';
$str = '# It should ignore the anotherone{test{nest}}
new{test}anotherone{test{nest}}new{test}new{test}';
preg_match_all($re, $str, $matches, PREG_SET_ORDER, 0);
// Print the entire match result
var_dump($matches);
Please keep in mind that these code samples are automatically generated and are not guaranteed to work. If you find any syntax errors, feel free to submit a bug report. For a full regex reference for PHP, please visit: http://php.net/manual/en/ref.pcre.php