regex101: toki pona tokenization

\ban ASCII word boundary

Non-Capturing Group(?:[ptkmnlswj]? (?:(?<=w)[aei]| (?<=[jt])[aeou]| (?<=[pkmnls])[aeiou]| (?<=\b)[aeiou]) (?:n(?![nm]?[aeiou]))?)+

+repeats the group contents below at least once:

Character Class[ptkmnlswj]?

?matches at most one character from the set below:

pthe literal p (112₁₀ / 160₈ / 70₁₆) (case insensitive)

tthe literal t (116₁₀ / 164₈ / 74₁₆) (case insensitive)

kthe literal k (107₁₀ / 153₈ / 6B₁₆) (case insensitive)

mthe literal m (109₁₀ / 155₈ / 6D₁₆) (case insensitive)

nthe literal n (110₁₀ / 156₈ / 6E₁₆) (case insensitive)

lthe literal l (108₁₀ / 154₈ / 6C₁₆) (case insensitive)

sthe literal s (115₁₀ / 163₈ / 73₁₆) (case insensitive)

wthe literal w (119₁₀ / 167₈ / 77₁₆) (case insensitive)

jthe literal j (106₁₀ / 152₈ / 6A₁₆) (case insensitive)

Non-Capturing Group(?:(?<=w)[aei]| (?<=[jt])[aeou]| (?<=[pkmnls])[aeiou]| (?<=\b)[aeiou])

1st Alternative(?<=w)[aei]

Positive Lookbehind(?<=w)

wthe literal w (119₁₀ / 167₈ / 77₁₆) (case insensitive)

Character Class[aei]

athe literal a (97₁₀ / 141₈ / 61₁₆) (case insensitive)

ethe literal e (101₁₀ / 145₈ / 65₁₆) (case insensitive)

ithe literal i (105₁₀ / 151₈ / 69₁₆) (case insensitive)

2nd Alternative (?<=[jt])[aeou]

Positive Lookbehind(?<=[jt])

Character Class[jt]

jthe literal j (106₁₀ / 152₈ / 6A₁₆) (case insensitive)

tthe literal t (116₁₀ / 164₈ / 74₁₆) (case insensitive)

Character Class[aeou]

athe literal a (97₁₀ / 141₈ / 61₁₆) (case insensitive)

ethe literal e (101₁₀ / 145₈ / 65₁₆) (case insensitive)

othe literal o (111₁₀ / 157₈ / 6F₁₆) (case insensitive)

uthe literal u (117₁₀ / 165₈ / 75₁₆) (case insensitive)

3rd Alternative (?<=[pkmnls])[aeiou]

Positive Lookbehind(?<=[pkmnls])

Character Class[pkmnls]

pthe literal p (112₁₀ / 160₈ / 70₁₆) (case insensitive)

kthe literal k (107₁₀ / 153₈ / 6B₁₆) (case insensitive)

mthe literal m (109₁₀ / 155₈ / 6D₁₆) (case insensitive)

nthe literal n (110₁₀ / 156₈ / 6E₁₆) (case insensitive)

lthe literal l (108₁₀ / 154₈ / 6C₁₆) (case insensitive)

sthe literal s (115₁₀ / 163₈ / 73₁₆) (case insensitive)

Character Class[aeiou]

athe literal a (97₁₀ / 141₈ / 61₁₆) (case insensitive)

ethe literal e (101₁₀ / 145₈ / 65₁₆) (case insensitive)

ithe literal i (105₁₀ / 151₈ / 69₁₆) (case insensitive)

othe literal o (111₁₀ / 157₈ / 6F₁₆) (case insensitive)

uthe literal u (117₁₀ / 165₈ / 75₁₆) (case insensitive)

4th Alternative (?<=\b)[aeiou]

Positive Lookbehind(?<=\b)

\ban ASCII word boundary

Character Class[aeiou]

athe literal a (97₁₀ / 141₈ / 61₁₆) (case insensitive)

ethe literal e (101₁₀ / 145₈ / 65₁₆) (case insensitive)

ithe literal i (105₁₀ / 151₈ / 69₁₆) (case insensitive)

othe literal o (111₁₀ / 157₈ / 6F₁₆) (case insensitive)

uthe literal u (117₁₀ / 165₈ / 75₁₆) (case insensitive)

Non-Capturing Group(?:n(?![nm]?[aeiou]))?

?repeats the group contents below at most once:

nthe literal n (110₁₀ / 156₈ / 6E₁₆) (case insensitive)

Negative Lookahead(?![nm]?[aeiou])

Character Class[nm]?

?matches at most one character from the set below:

nthe literal n (110₁₀ / 156₈ / 6E₁₆) (case insensitive)

mthe literal m (109₁₀ / 155₈ / 6D₁₆) (case insensitive)

Character Class[aeiou]

athe literal a (97₁₀ / 141₈ / 61₁₆) (case insensitive)

ethe literal e (101₁₀ / 145₈ / 65₁₆) (case insensitive)

ithe literal i (105₁₀ / 151₈ / 69₁₆) (case insensitive)

othe literal o (111₁₀ / 157₈ / 6F₁₆) (case insensitive)

uthe literal u (117₁₀ / 165₈ / 75₁₆) (case insensitive)

\ban ASCII word boundary

Global pattern flags

i modifier: insensitive. Matches without distinguishing uppercase and lowercase letters

x modifier: extended. Ignores unescaped whitespace and comments outside character classes

g modifier: global. Finds all matches instead of stopping after the first

Search reference

Regular Expression

\b(?:[ptkmnlswj]?
(?:(?<=w)[aei]|
(?<=[jt])[aeou]|
(?<=[pkmnls])[aeiou]|
(?<=\b)[aeiou])
(?:n(?![nm]?[aeiou]))?)+\b

Unit Tests

There are no unit tests, click to add some.