split() is a tokenizer: building cache paths from strings in Houdini VEX

TL;DR Houdini's VEX language runs a small function once per element of a 3D model, and it has strings, so every piece of a shattered object can compute its own output file path. VEX's split is a tokenizer rather than Python's str.split: it treats the separator argument as a set of characters, merges repeats and drops empty fields, including the one before a leading /. We build per-piece cache paths from the scene file's name and walk through the string rules that differ from Python and C, all checked in Houdini 22.0.368, including the $ that a wrangle expands before VEX ever sees it.

Houdini in 60 seconds

Houdini builds geometry with SOPs (surface operators), nodes that take geometry in and pass geometry out. Geometry is points, primitives (polygons, curves, packed pieces) and vertices, plus one "detail" record for the whole object. Each of these can carry named, typed attributes, including strings: s@name is a string attribute called name. An Attribute Wrangle runs a VEX snippet once per element. Its Run Over menu picks the element type, so with Primitives selected the snippet is the body of a loop over every primitive, run in parallel. Sliders are uniforms covers the longer version.

Pipelines run on strings: file paths, shot names, version tags, piece names. Doing that work in a wrangle means each element can carry its own path, which later nodes or a Python script can read.

The string rules a programmer would guess wrong

The SideFX strings page and the function docs give these rules, and Houdini 22.0.368 did what they say when we compiled and ran each one:

• Literals can use single or double quotes, plus Python-style r"..." and C++-style R"(...)" raw strings. + concatenates, and concat does the same for many arguments.
• s[i] indexes by byte, assuming UTF-8, with negative indices counting from the end. Out of range returns an empty string, not an error. You can't assign to s[i]: s[0] = "x"; stops the compiler with "Read-only expression on left side of assignment".
• len and strlen both count bytes, so "café" has length 5.
• split treats its second argument as a set of separator characters, and runs of separators merge into one. The docs say it "operates more as a tokenize method than as a split method." They don't say what happens at the ends of the string. In our tests leading and trailing separators left no empty tokens either, so split("/proj//shots/sh010/", "/") returns three tokens.
• Scalar find returns -len-1 when nothing matches (find("abc", "z") is -4), and searching for the empty string always fails, unlike Python.
• opdigits returns the last run of digits in a string as an int, which is how Houdini numbers things like piece17. A string with no digits gives 0.

A cache path per piece

Juraj Tomori's VEX tutorial rebuilds a render path by hand with split, pop, push and join. VEX also has splitpath, which hands back the directory and the file name in one call, so the example below uses that and keeps split for the file name.

// Attribute Wrangle. Run Over: Primitives.
// Input: Voronoi Fracture, which names each piece "piece0", "piece1", ...
// Click Create spare parameters, then type $HIPFILE into the new "hipfile"
// field. The parameter expands the variable; VEX only sees the resulting text.
string hipfile = chs("hipfile");          // "/proj/sh010/sh010_fx_v003.hipnc"

string dir, file;
splitpath(hipfile, dir, file);            // "/proj/sh010", "sh010_fx_v003.hipnc"
dir = rstrip(dir, "/");                   // only a file at the root leaves a "/"

string parts[] = split(file, ".");        // {"sh010_fx_v003", "hipnc"}
string stem = join(parts[:-1], ".");      // everything before the extension

string vtag = re_find("_v[0-9]+", stem);  // "_v003", or "" when there is no version
int version = vtag != "" ? opdigits(vtag) : -1;
int piece = opdigits(s@name);             // "piece17" -> 17

s@cachepath = sprintf("%s/cache/v%03d/%s.piece_%04d.bgeo.sc",
                      dir, version, stem, piece);
s@renderpath = sprintf("%s/render/%s.%04d.exr", dir, stem, int(@Frame));

Create spare parameters is the button to the right of the code editor; it adds a hipfile text field for the chs call. We cooked this on a box broken into 20 pieces, in a scene saved as sh010_fx_v003.hiplc (Houdini Indie's extension; Apprentice saves .hipnc, and the code doesn't care). Open the Geometry Spreadsheet, Houdini's table view of attributes, and every primitive has a cachepath. Voronoi Fracture builds each piece from several polygons, so our 185 primitives shared 20 distinct paths, one per piece. Here are three of them, with the start of each path (the folder the scene was saved in) cut:

piece0    .../sh010/cache/v003/sh010_fx_v003.piece_0000.bgeo.sc
piece1    .../sh010/cache/v003/sh010_fx_v003.piece_0001.bgeo.sc
piece17   .../sh010/cache/v003/sh010_fx_v003.piece_0017.bgeo.sc

The renderpath is the same on every row (.../sh010/render/sh010_fx_v003.0024.exr on frame 24), which is fine: the frame number belongs to the whole cook, not to a piece.

Two choices in there are deliberate. The version is found with re_find rather than calling opdigits on the whole stem, because opdigits("sh010_fx") returns 10, the shot number, when a file has no version tag. The frame goes through int() because VEX formats by the argument's type more than by the letter: in printf debugging a million points, %d on 3.14159 printed 3.14159. A whole-number float still pads, so the SideFX sprintf tip, sprintf("texture%04d.exr", @Frame), gives texture0001.exr on frame 1. A subframe doesn't: at frame 24.5, %04d on the float gives 24.5. With the cast every frame gets four digits, and frame 24.5 gives 0024 (so subframe renders need a pattern of their own).

The same rules in Python

Python makes the tokenizer behavior easy to see. We ran this with Python 3.12.10, and the comments show the output we got.

import re

def vex_split(s, seps=" \t\n"):
    # VEX split() per the H22 docs: a tokenizer. Each character in seps is a
    # separator and runs of separators merge, so no empty tokens survive.
    return [t for t in re.split("[" + re.escape(seps) + "]+", s) if t]

def opdigits(s):
    # VEX opdigits(): the last run of digits as an int, 0 if there is none
    runs = re.findall(r"[0-9]+", s)
    return int(runs[-1]) if runs else 0

path = "/proj//shots/sh010/"
print(path.split("/"))              # ['', 'proj', '', 'shots', 'sh010', '']
print(vex_split(path, "/"))         # ['proj', 'shots', 'sh010']
print(vex_split("a,b;c", ",;"))     # ['a', 'b', 'c']

print(opdigits("sh010_fx_v003"))    # 3: the version, as hoped
print(opdigits("sh010_fx"))         # 10: no version, so you get the shot number
m = re.search("_v[0-9]+", "sh010_fx")
print(opdigits(m.group()) if m else -1)   # -1: ask for the version explicitly

s = "café"
print(len(s), len(s.encode("utf-8")))     # 4 5: characters vs bytes

vex_split is our model, not SideFX's code, but for every input in this post it returned the same tokens as VEX's split in Houdini 22.0.368. The gap between the first two print lines is the reason Tomori's example puts the root / back by hand after join.

The wrangle rewrites $ and backslashes first

The strings page warns that in a wrangle "'$' will start an environment variable expansion and '\' will trigger escape sequences before VEX receives the code." To see what that means, we stored a few literals in string attributes from a Detail wrangle (Run Over: Detail, so it runs once) and read them back on frame 7:

"$HIPNAME"        ->  "sh010_fx_v003"
"$F4"             ->  "0007"
"$NOTAVAR_XYZ"    ->  ""
"cost $5"         ->  "cost "
"[0-9]+$"         ->  "[0-9]+$"
chr(36) + "F4"    ->  "$F4"

A $ followed by a name or a digit is replaced, and an unknown name becomes an empty string with no warning. A $ before a closing quote or a space survives, so a regex that ends in $ is safe. The case that bites is re_replace, whose docs use $1 for a captured group: in a wrangle, re_replace("piece([0-9]+)", "chunk_$1", "piece17") returned chunk_, because $1 had become nothing before VEX saw it. Build the dollar sign with chr(36) ("chunk_" + chr(36) + "1" returned chunk_17), or put the text in a parameter, as the hipfile field does.

The Attribute Wrangle help says $F variables are evaluated at frame 1. In Houdini 22.0.368 they weren't: "$F4" gave 0001, 0007, 0008 and 0030 on those frames, and the node warned "Text of snippet depends on time. This forces recompilation every frame and causes significant performance issues." For the frame, use @Frame.

Backslashes lose a level too. In a wrangle, re_find("\\d+", "ab12 cd") returned d, because the pattern arrived as d+. It takes "\\\\d+" to reach the regex engine as \d+, which is what the re_split docs mean when they say a wrangle needs \\\\s. A raw string skips the escape processing: re_find(r"\d+", "ab12 cd") returned 12. The cache-path example sidesteps all of it with [0-9].

What trips you up, and what clicks

1. The leading slash disappears. split on an absolute path drops the empty field before the first /, so join gives you a relative path. Tomori's example puts it back with "/" + join(...), which works for his /path/to/the/project/file/project_v3.hipnc. On Windows it doesn't: we ran his code as published with $HIPFILE on a Windows machine and got a path starting /C:/Users/. splitpath never takes the path apart, so it avoids both problems.
2. A multi-character separator isn't one separator. split(s, "::") splits on every :. On "std::vector::size" that happens to give the same three tokens, but split("xaybz", "ab") splits on every a and every b and returns x, y, z. For a real multi-character or regex separator, use re_split. Its docs warn that a trailing delimiter doesn't produce a trailing empty token (re_split(",", "a,b,") gave two tokens), but unlike split it keeps the empty tokens from leading and doubled delimiters: re_split(",", ",a,,b") gave "", "a", "", "b".
3. Byte offsets in a UTF-8 world. "café"[3] returns the two-byte é, but "café"[4] and "café"[-1] return empty strings, because those offsets land on the second byte of é. Slices work on raw bytes, so "café"[3:4] is one byte, half a character.
4. find returns a negative number on failure, and nothing stops you using it. The find docs say the -len-1 result "is intended to cause an error if you try to use it as an index". In Houdini 22.0.368 it didn't: on "sh010_fx", which has no dot, s[:find(s, ".")] returned an empty string, and reading an int array at a missed index returned 0, both without a warning. Python's s[:s.find(".")] fails quietly too, in a different way: it drops the last character. Test the result before you use it. One more compiler detail: pass find(...) straight to printf and it won't compile ("Ambiguous call to function find()"), because the int and int-array versions take the same arguments. Assign it to an int first.

What clicks: a string attribute is a column of text that the whole network can read. Once the path is on the geometry, every node downstream and any Python script that reads the geometry can use it without repeating the parsing.

Try it yourself

Houdini Apprentice is free for non-commercial learning. Save a scene as something like sh010_fx_v003.hipnc. Wire a Box into the first input of a Voronoi Fracture, and a Scatter (fed by the same Box, with Force Total Count set to 20) into its second input. Add the wrangle above with Run Over set to Primitives, click Create spare parameters, type $HIPFILE into hipfile, and check cachepath in the Geometry Spreadsheet.

1. Save the scene as sh010_fx.hipnc. The paths switch to cache/v-01/: version is -1, and %03d prints it as -01. Then switch the code back to opdigits(stem) and watch the folder turn into v010.
2. Add a Detail wrangle that prints split("/proj//shots/sh010/", "/") and len("café") once, and compare them with the Python output above. Ours printed {proj, shots, sh010} and 5.
3. No Houdini? Extend vex_split with VEX's third argument, maxsplits. To check your version: in Houdini 22.0.368, split("a//b//c", "/", 1) returned a and b//c, split("a/b/", "/", 1) returned a and b/, and a maxsplits of 0 didn't split at all.

Functions used: chs, splitpath, rstrip, split, join, re_find, re_split, re_replace, opdigits, sprintf, printf, len, strlen, find, concat, chr, pop, push. Each one links to its entry in the Simstim VEX reference.

Sources

• jtomori/vex_tutorial, Juraj Tomori's VEX tutorial ("Arrays and strings example")
• SideFX: Strings (byte indexing, raw strings, $ and backslashes in wrangles)
• SideFX VEX docs: split, join, splitpath, rstrip, sprintf, re_find, re_split, re_replace, opdigits, find, len, strlen, chs
• SideFX: Attribute Wrangle ($F in snippets) and Voronoi Fracture (the name attribute)