Converting the UselessCode.org Blog to a static site
If you've visited this blog before you might notice something is different about it. It's not just a new theme, previously it had been running on WordPress but has now been refactored into a static site generated with a Metalsmith based script.
Why change?
There are a number of reasons I decided to make the change. WordPress is a wonderful piece of software, it is quite powerful and allows you to do all sorts of things fairly easily. It is also quite popular. That popularity means it gets lots of attention from developers improving and extending it. But that also unfortunately means that it gets lots of bad attention too. You have to be pretty diligent about installing frequent updates due to security exploits being discovered in WordPress itself and even more so in the many popular plug-ins that can be used with it.
In addition to the update treadmill, the well know location of it's admin login leads to being a target for frequent brute-force login attacks. Luckily I've been pretty well protected from that thanks to my awesome webhost automatically disabling the page when they detect an attack. This site is mostly run as a hobby and keeping up with the constant updates can be time consuming.
Another reason to change is speed. There are things you can do to speed up a WordPress based site, smart caching, etc; but serving static files will always be faster. If I am fortunate enough to write something popular a static site is much more likely to withstand the traffic surge. This will also eventually help keep hosting costs down. Nothing on this site really needs to be dynamic.
My aforementioned awesome web host offers a static only hosting option that costs much less than one with access to PHP and other server-side tech. Eventually converting the rest of the site to be static will allow me to take advantage of that. (NFS recently changed their pricing structure, eliminating their distinction between static and dynamic sites, but they're still awesome 😉.) I've also become quite accustomed to writing in markdown over the last few years answering questions on Stack Overflow, a Metalsmith based site would allow me to use markdown more easily than I could with WordPress.
How the site was converted
A while ago I started investigating how I might go about this. After looking at several static site generators I settled on Metalsmith because it was both flexible and Node based. Initially I started to write my own script to convert a WXR file exported from WordPress to markdown but didn't get too far, being distracted by other projects.
Recently I decided to start work on converting the blog again. It had been a while and I wanted to make another post on the blog. When I logged into WordPress I was greeted by yet another please update to the new version message. That motivated me to see if anyone else had already created a better and more convenient method for converting WordPress data into something I could use to generate a static site. After a bit of searching I came across Jason Young's wordpress-to-markdown script, which takes an exported WXR file from WordPress and converts it into a directory structure of Markdown files with YAML style metadata. It was exactly what I was looking for. Unfortunately the directory structure it outputs isn't quite what I wanted. It sorts posts into a /yy/mm/slug structure. I wanted to mimic my existing url structure of /id/slug to avoid breaking any links. I was able to modify the script to do what I wanted. I also added some other enhancements such as saving the draft status of posts and including comments in the outputted metadata. I plan to do some more work on it and release my fork sometime in the future.
How it works now
Each post is kept in a separate .md file, the title and other metadata stored in the YAML header. Exported posts from WordPress are kept in a directory named wpposts and new posts are kept in posts. They are combined into a group via the metalsmith-collections plugin:
.use(collections({
posts: {
pattern: '**/*posts/**/*.html',
sortBy: 'pubDate',
reverse: true
}
}))
After working on the site for a while, I decided that instead of mimicking the old directory structure, I would move all of the posts and their sub-tree into a /posts directory of the blog, and then create 301 redirects to the new files so I wrote a small plugin:
.use(
function repathPosts(files, metalsmith, done) {
var file,
rxPosts = /^(?:wp)?posts(\\|\/)/,
rxIdPath = /^\d+(?:\\|\/)/,
newFiles = {},
originalPath,
slugs = [];
for (file in files) {
if (file.match(rxPosts)) {
originalPath = file;
file = file.replace(rxPosts, '');
if (file.match(rxIdPath)) { // have to use path match instead of .id so images are moved too
file = file.replace(rxIdPath, '');
}
file = 'posts/' + file.replace(/\\/g, '/');
// Because we are combining posts from different directories, check the slug,
// posts with duplicate slugs will end up having the same url,
// Stop with an error if a conflict is found.
if (slugs.find((slug) => slug === files[originalPath].slug)) {
throw ('Conflicting post path: "' + file + '" for "' + originalPath);
}
slugs.push(files[originalPath].slug);
newFiles[file] = files[originalPath];
newFiles[file].originalPath = originalPath;
}
}
// remove old files add new ones
for (file in newFiles) {
originalPath = newFiles[file].originalPath
delete files[originalPath];
files[file] = newFiles[file];
// building redirects object, used to add 301 redirects in .htaccess later
if (files[file].id !== undefined) { // use id so this doesn't apply to images or new content
let f = files[file],
oldPath = f.id + '/' + f.slug;
redirects[oldPath] = file;
}
}
done();
}
)
This keeps something at /posts/new-post at /posts/new-post, but something at /wpposts/100/old-post becomes /posts/old-post. If one of the new names conflicts with the name of an old post the script stops and complains.
The redirects object stores the path of old Wordpress posts and their new path. Later another plugin I wrote creates a .htaccess file that includes HTTP 301 redirects using this redirects object.
When I define the redirects object, I add a 'feed' property to redirect the old /feed location Wordpress used for rss to the new file that Metalsmith generates:
redirects = {
['feed']: 'rss.xml',
},
I had intended on getting around to packaging the .htaccess plugin code and publishing it publicly as a Metalsmith plugin before converting the blog, but that will have to wait for later. A new wave of Wordpress attacks prodded me to finally finish the conversion and publish the plugin later; this is too urgent to delay for finishing the plugin first.
I wanted to use metalsmith-rootpath to easily create relative path links. I had some difficulties installing it which are described below. Eventually I figured them out but in the meantime I wrote my own plugin with an equivalent function to what metalsmith-rootpath does:
.use(
(files, metalsmith, done) => {
var file,
slash = /\\|\//;
for (file in files) {
let root = '../'.repeat(file.split(slash).length - 1);
files[file].rootPath = root;
}
done();
}
)
I used metalsmith-tags to generate pages for each of my post tags.
.use(tags({
handle: 'tags',
path: 'tags/:tag/index.html',
layout: 'tag.jade'
}))
metalsmith-tags generates a page for each tag, however it does not generate an index page that lists all of the tags in one spot, so I wrote a plugin to create that page:
// create virtual file for tags index
// to list all available tags
.use((files, metalsmith, done) => {
var sortedTagNames = Object.keys(metalsmith._metadata.tags).sort();
files['tags/index.html'] = {
path: 'tags/index.html',
title: 'Tags for ' + metalsmith._metadata.site.title,
layout: 'tags.jade',
sortedTagNames,
contents: new Buffer(''),
mode: '0666',
};
done();
})
The template then uses sortedTagNames to list the tags in alphabetical order:
extends layout.jade
block main
h1 Tags
unless tags
p No posts are currently tagged.
else
ul.tags
for tag in sortedTagNames
li
a(href="#{tags[tag].urlSafe}")= tag
Now that the site is static I could no longer use WordPress' MYSQL based search. The search box now just does a site: search via DuckDuckgo. DDG is my favorite search engine due to their good results and focus on privacy. Since the new search box requires a little bit of JavaSript, it is initially hidden by CSS:
.nojs {
display: none;
}
and made visible by a small snippet of JS.
[...document.querySelectorAll('.nojs')].forEach(function (el) {
el.classList.remove('nojs');
});
Like search, I had to replace the comment system so I added Disqus comments.
Other plugins used by the build script include:
metalsmith-branchmetalsmith-excerptsmetalsmith-markdownmetalsmith-permalinksmetalsmith-layoutsmetalsmith-draftsmetalsmith-stylusmetalsmith-fingerprint-ignore(used to allow long caching of js and css files)metalsmith-babelmetalsmith-prism(used for highlighting code)metalsmith-if(used for conditionally building things via command-line options)metalsmith-gzipmetalsmith-watchmetalsmith-serve
metalsmith-watch and metalsmith-serve are used for development. yargs is used to parse command-line options. At present the only available option is --prod. Using --prod does not launch the server or watch process, inserts the Disqus code and gzips all .html, .css and .js files.
One thing that should be noted is that I used the metalsmith-layouts plugin instead of, the now deprecated, metalsmith-templates. Most tutorials you'll find out there haven't been updated and still use metalsmith-templates. The two work pretty much the same, you can generally just change template to layout in the examples and it will work.
I used Stylus for the CSS. I used the autoprefixer plugin for it as well as a great little unit conversion plugin. That allowed me to hide the menu behind the title bar with:
top: px(3em) - 150px;
and then slide it in via transition with:
top: 3em;
Difficulties in the process
I had some trouble installing some of the plugins I used due to dependency issues. I think what happened is that mid-project I upgraded my Node version. The new version of npm relies on node-gyp, which caused the problems. First I had to install Python, I installed Python 3. I then found out that it needed Python 2. I uninstalled 3 and installed 2. I was then informed that I needed the .NET Framework 2.0 SDK; I installed that and it still refused to install. Eventually I found my way to the node-gyp github page and followed the instructions in the "On Windows" section, first installing the windows-build-tools, having no success and then trying to install several different versions of Visual Studio Express. First I installed VS 2015, no success, it still complained about not finding msbuild.exe, possibly because msbuild.exe was renamed vcbuild.exe some versions ago. After finding some other information I then tried to install VS 2010, but couldn't find anywhere to download it on Microsoft's site. I then installed VS2012. This time it did not complain about msbuild.exe missing, it instead gave me a bunch of compilation errors.
Since npm now required node-gyp to install even non-native JS only packages I figured that it must have been installed with the update right? I did npm list -g --depth 0 to see if it was installed. Nope. Finally I figured out that I needed to install node-gyp globally: npm install -g node-gyp. Suddenly npm install started working fine again.
Another small bump in the road was caused by the aforementioned use of metalsmith-layouts. metalsmith-layouts is the officially supported plugin, it has replaced metalsmith-templates. Unfortunately most tutorials you will find out there still use metalsmith-templates. When I originally implemented pagination my index page and all of the archive pages it generated were completely blank. I eventually figured out that the code I had copied and pasted used template instead of layout, after changing template to layout things worked as expected.
UPDATE 2017-02-17
I added a plugin to create a recent-posts.json file that the main site can use to display recent posts after it is converted to a static site since it will no longer be able to query the Wordpress DB to do that like before:
.use((files, metalsmith, done) => {
let first3 = metalsmith._metadata.posts.slice(0, 3),
recent = [];
first3.forEach((item) => {
let post = {};
post.title = item.title;
post.url = `/blog/${item.path}`;
post.pubDate = item.pubDate;
post.excerpt = item.excerpt;
recent.push(post);
});
files['recent-posts.json'] = {
path: 'recent-posts.json',
contents: new Buffer(JSON.stringify(recent)),
mode: '0666',
};
done();
})