The sea change to vendoring and containers

Not so long ago, we would install code to the global runtime path, where it would be available machine-wide.  CPAN was perhaps the earliest to do this, but PEAR, rubygems, and probably others followed.  After all, CPAN was seen as a great strength of Perl.

Users of such a library would say something like use Image::EXIF; or include_once "Smarty/Smarty.class.php"; Notice that these are a mechanism to follow some include path to resolve relative filenames.

But as servers got larger faster than applications, co-installing those applications became attractive.  The problem was that they can have different dependency requirements, which complicates the management of a shared include path.

This is about where Bundler happened for Ruby, and its ideas were brought to other ecosystems: Perl got carton, and PHP got composer. These systems build a directory within the project to hold the dependencies, and basically ignore the global include path entirely.

At the cost of disk space, this bypasses the problems of sharing dependencies between projects.  It bypasses all the problems of working with multiple versions of dependencies that may be installed in different hosts’ include paths.  It also makes the concept of the language’s include path obsolete, at least in PHP’s case: the Composer autoloader is a more-capable “include path” written in code.  Finally, it allows piecemeal upgrades—a small service can make a risky upgrade and gain experience, before a larger or more important service on the machine needs to upgrade.

Piecemeal upgrades also enable independent teams to make more decisions on their own.  Since their dependencies are not shared with other users, changes to them do not have to be coordinated with other users.

Containers are the next step in the evolution of this process.  Pulling even more dependencies into a container brings all the advantages of local management of packages into a broader scope.  It allows piecemeal upgrading of entire PHP versions.  And it makes an application cost an order of magnitude more space, once again.

In our non-containerized environment, I can switch the PHP version on a host pretty easily, using co-installable versions and either update-alternatives (for the CLI) or a2disconf/a2enconf (for FPM), but this means the services do not really have a choice about their PHP version.  It has been made before the application's entry point is reached.

Stability vs Churn Culture

I’m working on rewriting memcache-dynamo in Go.  Why?  What was wrong with Python?

The problem is that the development community has diverged from my goals.  I’m looking for a language that’s stable, since this isn’t the company’s primary language.  (memcache-dynamo is utility code.) I want to write the code and then forget about it, more or less.  Python has made that impossible.

A 1-year release cycle with “deprecated for 1 cycle before removal” as the policy means it’s possible for users on Ubuntu LTS to end up in a situation where their previous LTS, 2 versions behind, doesn’t provide an alternative for something that’s been removed in the next LTS / current Python.

But looking closer to home, it’s a trend that’s sweeping the industry as a whole, and fracturing communities in its wake.  Perl wants to do the same thing, if “Perl 7” ever lands.

Also, PHP 5.6 has been unsupported for two years, yet there are still code bases out there that support PHP 5, and people are (presumably) still running 5.6 in production.  With Enterprise Linux distributions, we will see this continue for years; RHEL 7 shipped PHP 5.4, with support through June 2024.

There’s a separate group of people that are moving ahead with the yearly updates.  PHPUnit for example; every year, a new major version comes out, dropping support for the now-unsupported PHP versions, and whatever PHPUnit functionality has been arbitrarily renamed in the meantime.  The people writing for 5.x are still using PHPUnit 4 or 5, which don’t support 8.0; it’s not until PHPUnit 8.5.12 that installation on PHP 8 is allowed, and that still doesn’t support PHP 7.0 or 7.1.

This is creating two ecosystems, and it’s putting pressure on the projects that value stability to stop doing that.  People will make a pull request, and write, “but actually, can you drop php 5 support? i had to work extra because of that.”

The instability of Linux libraries, from glibc on up, made building for Linux excessively complex, and AFAICT few people bother.  Go decided to skip glibc by default when building binaries, apparently because that was the easier path?

Now everyone thinks we should repeat that mistake in most of our other languages.

(Not) Learning Rust

I want Rust to succeed.  It seems like the best gambit for fixing C, where the slightest mistake is likely to be a security hole… or to be turned into one by a newer compiler version.

On the other hand, actually learning it has proven extremely difficult.  Very little of Rust is explained in terms of something else; and all of the basic mechanics of the language feel weird.

After the Rust Book and the Cookbook and maybe some other stuff from Awesome Rust, I went through the tutorial about writing very bad linked lists; there were places that were like, “Can we do X? [code that does X].”  I would stop there, run that code, and try to solve the error myself.  I never could. I would read on, and it would be, “The error tells us we need to call F” so I’d try to figure out how to call F.  I never could figure it out, even with that hint. And I still didn’t have the capability to go from error message to even discovering F, let alone understanding that F would apply to the error.

I threw myself at the wall several times, and failed to advance past the part where one writes a number-guessing game for the CLI.

Moreover, I don’t have a motivating project to “learn Rust” with.  It’s mostly an exercise in following other people’s code, breaking it, or trying to improve it, and ending up with incomprehensible and unactionable errors.  The compiler errors try to be helpful, but if the code starts out far enough off the rails, it’s just flailing, same as me.

When a survey comes in that says “Opinions of Rust are very favorable, but people aren’t using it,” I understand.  I understand completely.  We all want something to fix entire bug classes, but keep the bare-metal performance, and Rust is basically without competition there.  But we can’t all devote enough time and energy to learning Rust from scratch.

Smarty vs. Twig (a benchmark done poorly)

Note added 14 Feb 2014: I ended up settling on Twig without too much more rigorous testing.  This used to be titled "Part 1" but there is, and is never expected to be, a Part 2.  Now back to your original post....

I'm pretty cranky about what my templates do (I have been known to roll my own), so I have a small list of must-have features.  Listing in order by increasing likelihood that a template engine satisfies them:

• Extensible functions for adding convenience from the host environment
• Built-in functions for date formatting
• No annoying <? ?> all over the place
• Auto-escaping of variables when building HTML
• Conditionals, expressions, and loops (if-elseif-else chains, boolean logic)
• Access to array elements and object properties

That's pretty much it: I'm willing to let "presentation logic" get pretty heavyweight, as long as it involves read-only access to the variables given to the template.  Templates should in no way mutate their model or context (excepting functions they initiate, like cycle, for hopefully obvious reasons.)

With that, it looks like the contenders are Smarty and Twig, currently at versions 3.1.14 and 1.13.2, respectively.  How do they compare in a benchmark showdown?

Radical Clojure

Apparently, last month I missed Yegge's post and followup regarding software liberals and conservatives.

One of the things that caught my eye, that's clearly a Nerd Trap but so powerful that I need to answer anyway, is this little quote:

But the reality is that Clojure is quite conservative.  As far as I'm concerned, if you can't set the value of a memory variable without writing code to ask for a transaction, it's... it's... all I can think of is Morgan Freeman in the Shawshank Redemption, working at the grocery store, complaining "Forty years I been asking permission to piss. I can't squeeze a drop without say-so."

Emphasis and vivid FUD in original.

There's just one missing word, though, that makes all the difference:

You can't set the value of a shared memory variable outside of a transaction.

Shared.  Global.  Possibly still in use by someone else.

Clojure is utterly pointless without understanding that time is explicit and everything is an immutable value (unless you have a Java native thing.)  Values last, unmodified, until a reader asks for an update, and so a writer must be forbidden from modifying (destroying) that memory.  There's a whole paradigm hiding there, which you can see again if you look at Datomic.  Clojure without immutable values might as well be JRuby.

The other trick up Clojure's sleeve is that 'transaction' implies something rather more heavyweight outside of Clojure than in it.  Again, because immutability is pervasive, Clojure's transactions don't have to do read tracking.  When anyone reads an old value, it's going to stay what they read.  It's like RCU, conceptually, except that every read is protected and nobody needs to copy because the messy details are taken care of on write.

And if it's not actually shared?  Then you should use a var, which only one thread can see, and therefore doesn't need to be updated in a transaction!

Clojure is an interesting language.  I'd still recommend it.


Added 26 Sept: I think that liberal/conservative as applied by Yegge divide the languages according to how much a language insists on its own philosophy.  Things like Erlang and Clojure get ranked "conservative" right in with Pascal because they have Ways to Do Things, even if they are progressive ideas (functional and parallel/async are strongly encouraged.)  Then Perl is more "liberal" because it's equally well-suited to OO, procedural, functional, and concatenative programming, i.e. just barely, and if it didn't have regexes as syntax, it would be long dead already.  Python is more conservative than Ruby despite sharing characteristics because "there should be one—and preferably only one—obvious way to do it."  That's a clear Conservative value, right in the middle of the Zen of Python.

In any case, the insight that everyone thinks they're liberal is accurate.  Upon some thinking, which is hard and time-consuming and therefore not generally applied on the Internet, "conservative" is a fair enough label for my tendencies at this point in time—because I got burned throwing around too much fire, then came off that job a few months later to take positions in more conservative cultures.  But I've never thought of myself as a stodgy dinosaur programmer.

Language Driven

If templates weren’t languages, we wouldn’t call them template languages, nor have so many of them: HAML, HTML::Mason, TemplateToolkit, PHP, Smarty, pagelib and Output, and so forth.

(This is essentially a repost of something that used to be on my old blog.)

Plain Old Data

I’m coming to the conclusion that there’s actually no such thing as “plain data;” it always has some metadata attached.  If it doesn’t, it might be displayed incorrectly, and then a human needs to interfere to determine the correct metadata to apply to fix the problem.  (Example: View → Character Encoding in Firefox.)  Pushed to the extreme, even “just numbers” have metadata: they can be encoded as text, a binary integer/float (IEEE 754 or otherwise) of some size/endianness, or an ASN.1 encoding.

Another conclusion I’m reaching is that HTTP conflates all kinds of metadata.  Coupled with the lack of self-contained metadata in file formats and filesystems, things start to accumulate hacks.

PHP: retrospective

Once in a while, someone on Reddit asks for justification why everyone there hates PHP.  I never reply, because there's too much to list in a comment, but maybe I can write a definitive post here.

Most recently updated: 7 Feb 2014, for new features in PHP 5.6.0a1.

The Chariot

I read "Linking in JSON" the other day.  I knew someone had already gotten started on JSON Schema.  (A quick search shows JSON namespace ideas floating around.)  JSON as the lightweight alternative to XML being turned into XML?  This is beginning to sound familiar.

With lightweight formats, we tend to get a proliferation of variants for different uses.  (Not just images, naturally, and CPAN manages to use more than one.)  Heavier formats tend to have a problem I'm going to call "Accessories Not Included": they get sufficiently large and complex that not all readers support all format options.  If the growth is arrested early enough, you end up with a handful of profiles; if it gets out of hand, you have over ten of them.

I never expected XML to be so widely used for so much stuff, or spawn so many related specs.  After all, it was verbose!  And you could make up just any tag name you wanted!  But it turns out to scale well from a not-quite-simple tree data structure, with annotated nodes, all the way up to unfashionable Enterprisey uses.  But scalability bothers people, because who knows what wacky thing someone else on the team is going to foist on you, so more restricted alternatives rise to popularity.  If this is true, the rise of Java should correlate to (non-game) companies getting burned enough times on C++.  (If you think about Safety, it makes sense.  He wants you to use Java, because he can't hack reversing the polarity of the template stack flow.)

This sort of thing is practically destined to keep happening.  More features generally cost more memory and processing time, or some other inconvenience like a compilation step, which is against the religion of some developers.  Thus, lightweight versions of things spring up in opposition to whatever is perceived to be too heavy.  Sometimes compilation is considered the lightweight alternative, since it's not done on every request.

Though sometimes many similar projects proliferate because they just aren't that hard.  It's easier to write a web framework than to learn one, so there are a lot of them.


Made it out of the link forest and need something more to kill the time?  Maybe you want to subscribe to my feed, or follow me on twitter.

Programming Languages to Learn

Many languages these days are fairly Lispy, except for being homoiconic and thus having a full-strength macro system instead of C's token pasting or many other languages' nothing.

But which ones are absolutely vital to learn, and which ones are "just different languages"?