Wednesday, August 7, 2013

Smarty vs. Twig (a benchmark done poorly)

Note added 14 Feb 2014: I ended up settling on Twig without too much more rigorous testing.  This used to be titled "Part 1" but there is, and is never expected to be, a Part 2.  Now back to your original post....

I'm pretty cranky about what my templates do (I have been known to roll my own), so I have a small list of must-have features.  Listing in order by increasing likelihood that a template engine satisfies them:
  • Extensible functions for adding convenience from the host environment
  • Built-in functions for date formatting
  • No annoying <? ?> all over the place
  • Auto-escaping of variables when building HTML
  • Conditionals, expressions, and loops (if-elseif-else chains, boolean logic)
  • Access to array elements and object properties
That's pretty much it: I'm willing to let "presentation logic" get pretty heavyweight, as long as it involves read-only access to the variables given to the template.  Templates should in no way mutate their model or context (excepting functions they initiate, like cycle, for hopefully obvious reasons.)

With that, it looks like the contenders are Smarty and Twig, currently at versions 3.1.14 and 1.13.2, respectively.  How do they compare in a benchmark showdown?

To keep the table as narrow as possible, "Time" is Wall Time, and Engine and Total are both measurements of memory usage: Total is the result of memory_get_peak_usage() post-benchmark, and Engine is the difference between Total and the result of memory_get_peak_usage() before any engine code was invoked.

PHPRun TypeTimeEngineTotal
5.3.3 +xdebugTwig, Cached25 s1606 KiB2017 KiB
5.3.3 +xdebugSmarty, Cached30 s3214 KiB3631 KiB
5.3.3Twig, Cached3.8 s1216 KiB1610 KiB
5.3.27Twig, Cached3.2 s1218 KiB1620 KiB
5.4.17Twig, Cached2.2 s705 KiB869 KiB
5.5.0Twig, Cached2.2 s709 KiB873 KiB
5.3.3Smarty, Cached10.3 s918 KiB1317 KiB
5.3.27Smarty, Cachednot tested
5.4.17Smarty, Cached5.4 s509 KiB675 KiB
5.5.0Smarty, Cached5.4 s513 KiB680 KiB
5.3.3Twig, Compiling3.9 s1871 KiB2265 KiB
5.3.3Smarty, Compiling10.0 s2838 KiB3234 KiB
5.3.27Smarty, Compilingnot tested
5.4.17Smarty, Compiling5.6 s1939 KiB2106 KiB
5.5.0Smarty, Compiling5.5 s1947 KiB2114 KiB

One other metric: the twig directory appears to contain 960 K of code in a 2.8 M file tree; smarty has about 3.4 M of code in a 45 M tree.  Smarty's code is bigger than the entire twig distribution.

Caveats from the following paragraphs in mind, I'm going to place conclusions here, nearer the numbers table.  Outside of the xdebug case, twig 1.13.2 runs in under half the time of smarty 3.1.14.  For the kind of templates I'm working with here, compiled vs. cached doesn't make too much of a difference, and while Smarty achieves better memory use when building a cached template, server metrics don't indicate a shortage of RAM.

There are some obvious limitations here, most of which stem from the fact that I forgot that the system PHP had xdebug enabled.  That means the benchmark loop is tuned for xdebug's execution speed, and twig on other versions is finishing its benchmark within the time it takes PHP to warm up for benchmarking.  On the other hand, twig is still stable over two to three runs, so it doesn't seem like there's so much error that these numbers are completely useless.

The xdebug issue is also why I stopped testing 5.3.27 once I realized what was going on with 5.3.3.  I also tested Smarty cached vs. compiling when I saw the dramatic memory difference, but I didn't cover much for Twig compiling except the 5.3.3 test after I realized that xdebug was interfering.  That php version is more specifically known as 5.3.3-23-el6_4.i686; other versions are self-compiled with default flags from the tarballs.

On the other hand, this is not the usual Hello World or "assign MANY things!" test: this is based on a real email template extracted from work.  It's a pair of files, containing approximately 270 bytes of wrapper (the html/body goop, a signature, and an include directive) and about 700 bytes of actual template involving three simple variable substitutions… and one back-channel variable assignment.  To keep the subject as part of the template, the engines are extended with a custom function like...

{{ phpSetVar("subject", "Welcome to B2B Service Portal") }} for twig, or
{phpSetVar name="subject" value="Welcome to B2B Service Portal"} for smarty.

The phpSetVar implementation uses its arguments to set an array key, which the upper level code then picks out afterward to pass to the real email builder.  (I apologize for the names, but they're not what you're here for.)

The benchmark code requests both Smarty and Twig to render into a string, since my real code needs the string to set as the body of a MIME part.  Depending on how they're implemented, memory usage could have been misreported, or could be lower when calling output() instead of fetch() for smarty.

All in all, I'm probably dissatisfied enough with this run to try more testing tomorrow at lunch.

No comments: