Unconstant Conjunction

Joining RStudio

2021-12-09T00:00:00Z

Over the last few years I’ve become interested in opportunities for using R in production, in the broad sense of code with users other than its original author. The reason is that this is an important avenue for R users to bring value to others. In the words of David Robinson, “anything still on your computer is, to a first approximation, useless.”

My public work in the R community has also largely fallen into this category, including production-grade metrics and logging packages, connecting R to RabbitMQ (which is very popular in enterprise production systems), and writing R’s first external profiler – a tool for answering questions like “why is my R code slow in production?”

My private work in R has included work on platforms and tools to build and host many dozens of internal packages as well as development of production R APIs and Shiny apps. I was also intimately involved in migrating many of these workloads to Kubernetes – increasingly the target for production workloads in industry, and a tool I feel strongly about as part of R’s future.

In my experience it can be empowering for R users to have a clear path from ad-hoc analysis to “data products” – graphics, emails, reports, APIs, or even interactive applications – as they need them. And so I’ve been advocating for these data products to have production-friendly defaults (or at least production-friendly stories), all while personally discovering what it means, operationally, to actually take R to production.

A few months ago I had the realisation that if I’m really serious about expanding the frontier for R in production, the best place to do so is at RStudio. No organisation is as plugged into the broader ecosystem as they are, and no organisation has as broad a reach to improve these tools and techniques for the wider data science community, in R and beyond.

So I’m happy to announce that I have joined RStudio, at least in part to work on improving the story for R in production, and to help out the folks making that leap.

Shipping Application Logs from RStudio Connect

2021-04-29T00:00:00Z

RStudio has an enterprise offering called RStudio Connect that is designed to host R (and now Python) content. In my experience it’s a great platform for R users using the RStudio IDE, and works particularly well for frictionless deployment of internal Shiny apps or RMarkdown reports.

But one of the things we’ve struggled with in using Connect is making Shiny app logs useful. You can view logs for individual sessions in the browser (if you know what you’re looking for), but there’s no searching or aggregation of any kind. This is the existing interface:

This is doubly frustrating for users with experience with any other log management system – a crowded field, with many commercial and open-source solutions available. These solutions typically offer not only searching but complex querying, visualisation, and ad-hoc dashboard capabilities.¹ For example, check out the landing page for Elasticsearch/Kibana, which is the most popular open-source option. Common commercial choices include Splunk and Datadog, and all of the cloud vendors have highly-integrated platforms of their own.

Unfortunately, Connect won’t work natively with any of these tools. Log management is focused on the per-session GUI.

Motivated by our own internal desire to get Shiny logs into Splunk, I wrote a Connect-specific plugin for the open-source Fluent Bit project. Fluent Bit itself is a fast, lightweight log forwarding and aggregation program with a vibrant plugin community.

The plugin works by watching for new logs written to the filesystem and querying the Connect API for additional metadata to link these logs back to specific applications. Fluent Bit can then be used to forward the results to any of its supported backends, including Elasticsearch/Kibana, Splunk, Stackdriver, Cloudwatch Logs, Datadog, Loki, and many others.

This should allow R users to easily export logs to their organisation’s existing log management system without needing to modify their reports or Shiny apps to do so. For example, log entries look like this in our Splunk instance:

You can see from the above that entries contain not only the log message, “Warning in readLines…”, but also the session identifier, the name of the app or report, and the app mode – a Shiny application, in this case. Support for Connect’s tag system will probably be added in the future.

The plugin is open-source under the LGPL 2.1 and available on GitHub.

Shoutout to Cole Arendt for his advice on using Connect’s APIs to make this work.

As an aside: originally we pushed Shiny app logs into our Splunk instance by having them write to the local syslog via the rsyslog package, which was then independently forwarded on to Splunk. My coworker Nicholas Jhirad mentioned some of this history in a talk at eRum last year. A huge downside of this approach is that it requires everyone who writes a Shiny app to opt in to using this package, and any message not sent through it is lost. ↩︎

.NET, rsync, and the Linux Page Cache: A Kubernetes War Story

2021-02-12T00:00:00Z

Everyone gets a Kubernetes war story, right? Well, here’s mine.

At work we have a huge, business-critical C# service that I helped to port to Linux (from the Windows-only .NET framework) to run under Kubernetes. This service emits a large number of logs.

Like, a lot of logs. Many gigabytes of logs per day, in sequentially-numbered text files of 50MB each.

For various reasons the developers of this application preferred to be able to look at these logs as actual files for post-mortem analysis rather than through our existing centralised logging tools. Under Windows they would remote into a server and inspect a mounted drive with the logs.

As part of the porting effort they asked me to emulate this workflow.

Log to the Network Fileshare Like It’s 1999

Since we’re running Azure’s Kubernetes offering, I had the pods mount an Azure Files share as a persistent volume – which is basically a classic network fileshare – and write their logs to that. Developers could then mount that fileshare to their local Windows computers via Azure’s SMB support. Very convenient.

However, it turns out that writing all of these logs to this network fileshare introduced so much latency to normal operations that the application ran about 3x slower on Kubernetes than it ran on Windows, which really undermined the effort we’d made porting it.

To “fix” this I introduced a very simple workaround: I had the app write logs to the local disk, and then I ran a dead-simple rsync command (in a sidecar container) every minute or so to sync these logs to the network fileshare for later inspection.

That’s how we did it in the old days.

– Our CTO, amused.

This worked nicely and brought the performance up to par (better, thankfully, than it had been on Windows).

Time passed.

Eviction Notice

Soon, we started having a problem with this application: Kubernetes would frequently evict the service’s pods for consuming too much memory and destabilising the cluster, essentially terminating them without warning.

Evictions might be fine for misbehaving stateless applications, but this service was heavily, expensively stateful. Unexpected shutdowns lost us real revenue.

What was puzzling about this was that Kubernetes was claiming that our service was using far more memory than I’d anticipated – think 4GB to an expected 1.5. During the porting effort I’d carefully set the pods' memory requests to match observed memory usage on Windows and test runs on Linux.

At first I wondered if Kubernetes was measuring something incorrectly, so I found a pod with high memory usage that would likely be evicted soon and opened a shell into it. You can check a container’s memory usage once inside by inspecting Sysfs directly:

root@<container># cat /sys/fs/cgroup/memory/memory.usage_in_bytes
4248424448

So it clearly wasn’t measurement error. To dig a little deeper I looked at the full memory usage statistics for the container:

root@<container># cat /sys/fs/cgroup/memory/memory.stat
cache 2910048256
rss 1322893312
<snip>
active_file 2884849664
<snip>

The rss entry (for resident set size) is close to what we had used to measure memory usage on Windows, and looked like what I would expect for the service under that particular configuration.

So what are those cache and active_file entries?

It turns out that Linux aggressively caches recently-used files in memory, in what is called the Page Cache, in order to improve filesystem performance. Normally this does not constrain applications from using memory, because when the system starts to run low Linux will happily flush older parts of this cache to make the reclaimed RAM available.

But where could 2.8GB of recently-used file memory be coming from?

Oh. Right.

I was able to confirm (using a handy tool I found on GitHub) that all of the individual log files were indeed in the page cache (about 55 of them in the case above).

Caches to Caches

With this in hand a picture of what was happening emerges:

The log files for the service are cached in memory by Linux, but rarely flushed unless the machine it’s running on happens to be low on total memory.
Kubernetes (rightly) views page cache usage as part of a container’s workload, so it counts when making resource allocation/scheduling decisions.
Scheduling decisions also include kicking overly greedy pods off the cluster, but in this case it’s actually a Linux kernel optimisation that makes the application look more memory-hungry than it really is.

I discovered much later that we’re not the only ones with this problem: including the page cache in the memory calculations for an eviction is considered a longstanding bug in Kubernetes itself that has received ample commentary. As you’ll discover in that thread, it’s a tricky issue to solve, which is probably why it hasn’t been.

It does turn out to be possible to remove individual files from the page cache manually by using a Linux-specific posix_fadvise(2) flag. Years ago Tobi Oetiker actually proposed modifying rsync itself to do this so that the files it copied did not end up in the page cache – but this was never merged upstream.

We briefly considered using a patched version of rsync or even running a second sidecar to periodically posix_fadvise() the log files to get them out of the cache.

Eventually we settled on a less obscure method: we configured the C# service to keep only the three most recent log files (though files still have a unique suffix to prevent collisions). Provided it doesn’t log more than 150MB in the one-minute window between rsync runs, this means we end up with all of the logs on the network fileshare, while still deleting them on the local disk to free up space in the page cache.

All of this just reinforced for me the wisdom of Charity Majors’s collection of lies we believe about logging:

“Logs are cheap.”

“They’re just logs, they can’t impact production reliability.”

“They’re just logs, they can’t possibly impact my code efficiency.”

Pushing Prometheus Metrics from R Scripts and Reports

2020-09-15T00:00:00Z

The openmetrics R package now supports pushing metrics to a Prometheus Pushgateway instance, which is useful for short-lived batch scripts or RMarkdown reports.

You might want to expose metrics from these scripts or reports to Prometheus in order to improve monitoring and alerting on failures, but many of these processes are not around long enough to run a webserver that Prometheus can pull from.

This is where the Pushgateway comes in. It allows you to push metrics to a centralised location where they can be aggregated and then scraped by Prometheus itself. But beware: there are a limited number of use cases for pushing metrics, and you should always prefer pull-based methods when possible.

Instrumenting an R ETL Job

Often R users find themselves writing an R script (or RMarkdown report) that runs regularly, pulling data from a database and creating a new dataset for further reporting, visualisation or modelling. This is the “extract, transform, load” (ETL) pattern.

There are two natural metrics that we might want to expose from such a script to Prometheus:

Whether or not the ETL succeeded; and
How long it took.

For example:

success <- gauge_metric(
  "etl_outcome", "Whether or not the ETL process succeeded."
)
duration <- gauge_metric(
  "etl_duration_seconds", "Total running time of the ETL process."
)

We might also add more granular timing metrics, parcelling out the database pull (so it could be tracked over time), and more domain-specific metrics (such as the number of users in the resulting dataset):

db_pull_duration <- gauge_metric(
  "etl_db_pull_duration_seconds",
  "Total time spent pulling data into the ETL process."
)
users <- gauge_metric(
  "etl_users_processed", "Number of unique users processed by the ETL."
)

These are all gauges, because we are not taking repeated measures and none of them are expected to increase monotonically.

Actually using these metrics in an ETL script might look as follows:

start <- Sys.time()
elapsed <- function() {
  unclass(difftime(Sys.time(), start, units = "secs"))
}

df <- pull_from_db()
db_pull_duration$set(elapsed())

clean <- process_raw_data(df)
write_cleaned_data()

users$set(length(unique(clean$user_id)))
success$set(1)
duration$set(elapsed())

These metrics can then be pushed to a Pushgateway instance at the end of the script:

push_to_gateway("http://localhost:9091", job = "etl-job-1")

You can also use on.exit() to ensure metrics are pushed even if the script fails.

If you have parameterised scripts or reports (say, running one job for each language group of your users), you can pass these parameters through as labels on the job in Pushgateway:

push_to_gateway("http://localhost:9091", job = "etl-job-1", lang = "en")

These metrics would look like the following when rendered:

# HELP etl_outcome Whether or not the ETL process succeeded.
# TYPE etl_outcome gauge
etl_outcome 1
# HELP etl_duration_seconds Total running time of the ETL process.
# TYPE etl_duration_seconds gauge
etl_duration_seconds 1.21224451065063
# HELP etl_db_pull_duration_seconds Total time spent pulling data into the ETL process.
# TYPE etl_db_pull_duration_seconds gauge
etl_db_pull_duration_seconds 1.18743181228638
# HELP etl_users_processed Number of unique users processed by the ETL.
# TYPE etl_users_processed gauge
etl_users_processed 100
# EOF

Bonus: Timing Chunks with knitr Hooks

The knitr package used to render RMarkdown reports sports a feature called hooks that can be used to run code before and after a chunk is executed. This can be used to time the chunk without cluttering the code with metric-related boilerplate.

To do this, we need to define a hook in the RMarkdown document:

knit_hooks$set(duration_metric = function(before, options, envir) {
  if (before) {
    envir$.start <- Sys.time()
  } else {
    if (exists(options$duration_metric, envir = envir)) {
      elapsed <- unclass(difftime(Sys.time(), envir$.start, units = "secs"))
      envir[[options$duration_metric]]$set(elapsed)
    } else {
      stop("No object matching duration_metric found.")
    }
  }
})

And then use it as follows:

```{r chunk-1, duration_metric = "db_pull_duration"}
df <- pull_from_db()
```

Annotating Deployments in Grafana Using the Process Start Time Metric

2020-05-22T00:00:00Z

Grafana sports a feature called Annotations that allow you to label a timestamp on a dashboard with meaningful events – most commonly deployments, campaigns, or outages:

(In this case annotating the simulated deployment of a Fluent Bit container, which I’ve used to forward container logs out of the cluster.)

Annotations can be input manually, but the only recommendations I’ve seen to generate them automatically is to use something like Loki, or teaching your CI/CD system to interact with Grafana’s web API. However, if you’re running a simple Prometheus + Grafana stack (say, using the Prometheus Operator on Kubernetes), you might be reticent to add more complexity to your setup just to get deployment annotations.

Fortunately, there’s a simpler alternative for this narrow case: you can use the process_start_time_seconds metric from Prometheus to get an approximate idea of when apps or pods were started. I haven’t seen this approach recommended elsewhere, which is the purpose of this post.

It turns out that process_start_time_seconds is exposed by almost all applications because it’s one of the standard metrics recommended by Prometheus itself and is exported by most client libraries automatically (including my own).

You can add annotations like the one in the image above as follows, assuming you have a namespace and pod template variable defined:

It’s important to understand that these annotations will show only when new processes are started, most likely because of a deployment but also during rescheduling, scaling, or a pod failure – but since it is likely that you’d want to know about those as well, perhaps that’s a good thing.

Introducing openmetrics: A Opinionated Prometheus Client for R

2020-05-17T00:00:00Z

My openmetrics package is now available on CRAN. The package makes it possible to add predefined and custom “metrics” to any R web application and expose them on a /metrics endpoint, where they can be consumed by Prometheus.

Prometheus itself is a hugely popular, open-source monitoring and metrics aggregation tool that is widely used in the Kubernetes ecosystem, usually alongside Grafana for visualisation.

To illustrate, the following is a real Grafana dashboard built from the default metrics exposed by the package for Plumber APIs:

Adding these to an existing Plumber API is extremely simple:

library(openmetrics)

srv <- plumber::plumb("plumber.R")
srv <- register_plumber_metrics(srv)
srv$run()

There is also built-in support for Shiny:

app <- shiny::shinyApp(...)
app <- register_shiny_metrics(app)
app

openmetrics is designed to be “batteries included” and offer good built-in metrics for existing applications, but it is also possible (and encouraged!) to add custom metrics tailored to your needs, and to expose them to Prometheus even if you are not using Plumber or Shiny.

More detailed usage information is available in the package’s README.

Three Useful Endpoints for Any Plumber API

2020-04-21T00:00:00Z

The Plumber package is a popular way to make R models or other code accessible to others with an HTTP API. It’s easy to get started using Plumber, but it’s not always clear what to do after you have a basic API up and running.

This post shares three simple endpoints I’ve used on dozens of Plumber APIs to make them easier to debug and deploy in development and production environments: /_ping, /_version, and /_sessioninfo.

Most Plumber users will be familiar with the special plumber.R files that can be used to generate an API. However, for the examples in this post it is convenient to add them programmatically. Programmatic endpoints are less user-friendly, but also less magic.

The “Ping” or “Healthcheck” Endpoint

Healthcheck endpoints (often called /status, /healthz, or my personal favourite, /_ping) give an outside observer the answer to a simple question: is the API up and running?

To implement it, just return an empty “OK”:

srv <- plumber::plumb("plumber.R")
srv$handle("GET", "/_ping", function(req, res) {
  res$setHeader("Content-Type", "application/json")
  res$status <- 200L
  res$body <- ""
  res
})
# ...
srv$run()

This allows you to check if you API is up and running from a browser (by visiting http://myapi.host/_ping) or the comfort of your R console (with httr::GET(), say). It also makes your API visible to any monitoring tools used by teams in your organisation to keep tabs on running services.

A healthcheck endpoint also integrates well with many tools in the Docker ecosystem, which is a common way to deploy Plumber APIs. For example, a Dockerfile can contain a HEALTHCHECK directive. Here’s one from an internal API:

# Check the /_ping endpoint every 30 seconds.
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s \
  CMD curl --silent --fail http://0.0.0.0:8080/_ping || exit 1

These healthcheck commands are understood by the Docker daemon and used to determine whether containers are “healthy” (or “unhealthy” and in need of a restart) – which you can see in commands like docker ps:

$ docker ps --format "table {{.ID}}\t{{.RunningFor}}\t{{.Status}}"
CONTAINER ID        CREATED             STATUS
7cedb56515a8        6 hours ago         Up 6 hours (healthy)

If you don’t want to modify the image directly, the Docker compose file format (also used by Docker Swarm) supports adding healthchecks as well:

healthcheck:
  test: curl --silent --fail http://0.0.0.0:8080/_ping || exit 1
  interval: 30s
  timeout: 5s

Both Docker Swarm and Kubernetes automatically use the healthchecks to understand if a deployment was successful, and also to implement zero-downtime rolling updates.

Over in the Kubernetes camp, you can also use the built-in liveness (and/or readiness) probes in Pods and Deployments. Here is one from ours:

livenessProbe:
  httpGet:
    path: /_ping
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30
  timeoutSeconds: 5

Kubernetes’s built-in probes have the advantage of not requiring the Docker image to bundle curl or a Windows equivalent, too.

On a related note, Kelsey Hightower has a talk on healthcheck endpoints where he suggests that applications use them to report various kinds of “readiness” measures, for example whether they can successfully connect to an underlying database.

The “Version” Endpoint

Much like the healthcheck endpoint, which answers the question “is my API running?”, it is often very useful to know “what version of my API is running?”.

As with the healthcheck endpoint, this can be added programmatically to any existing Plumber API:

version <- "2.1.1"

srv$handle("GET", "/_version", function(req, res) {
  res$setHeader("Content-Type", "application/json")
  res$status <- 200L
  res$body <- sprintf('{"version":"%s"}', version)
  res
})

Of course, this requires you to version your API to begin with, but you should be doing that anyway.

I find myself checking these endpoints all the time to verify that an API has deployed correctly, particularly for rolling deployments, which might take some time to converge to the new version.

The “SessionInfo” Endpoint

The last group of questions I find myself asking about Plumber APIs are variations on “what version of R (or a package) is it using?”.

R users often post the result of the sessionInfo() command when filing bugs for a package or posting on Stackoverflow, because it can help point to issues that only show up or newer or older versions of R itself or any of the packages in use. This information is similarly useful when debugging bad or inconsistent behaviour with a Plumber API.

Unfortunately, it can be hard to decipher R’s sessionInfo() results when directly serialised to JSON, because they contain deeply nested DESCRIPTION files for each loaded package.

Instead, I recommend using the sessioninfo package:

srv$handle("GET", "/_sessioninfo", function(req, res) {
  res$setHeader("Content-Type", "application/json")
  res$status <- 200L
  res$body <- jsonlite::toJSON(
    sessioninfo::session_info(), auto_unbox = TRUE, null = "null"
  )
  res
})

Coda

Implementing these three endpoints in all our internal APIs has helped diagnose and solve countless problems. But I’ve glossed over how to actually implement them in practice.

The approach used by my team is to have a common entrypoint.R script (recently reimplemented as an internal package) used across all our Plumber APIs. This ensures that even new projects will get these endpoints automatically.

Custom entrypoint support is an extremely useful but almost totally undocumented feature of Plumber. Hopefully more examples can be shared in the future to remedy this.

A Bayesian Estimate of BackBlaze's Hard Drive Failure Rates

2020-02-13T00:00:00Z

Each quarter the backup service BackBlaze publishes data on the failure rate of its hundreds of thousands of hard drives, most recently on February 11th. Since the failure rate of different models can vary widely, these posts sometimes make a splash in the tech community. They’re also notable as the only large public dataset on drive failures:

One of the things that strikes me about the presentation above is that BackBlaze uses simple averages to compute the “Annualized Failure Rate” (AFR), despite the fact that the actual count data vary by orders of magnitude, down to a single digit. This might lead us to question the accuracy for smaller samples; in fact, the authors are sensitive to this possibility and suppress data from drives with less than 5,000 days of operation in Q4 2019 (although they are detailed in the text of the article and available in their public datasets).

This looks like a perfect use case for a Bayesian approach: we want to combine a prior expectation of the failure rate (which might be close to the historical average across all drives) with observed failure events to produce a more accurate estimate for each model.

Re-estimating Failure Rates using Empirical Bayes

First, we can extract the data that is missing from the table but mentioned in the text:

(omitted <- tibble::tribble(
  ~mfg, ~name, ~size, ~days, ~failures,
  "Seagate", "ST16000NM001G", "16TB", 1440, 0,
  "Toshiba", "HDWF180", "8TB", 13994, 1,
  "HGST", "HUH721010ALE600", "10TB", 8042, 0,
  "Toshiba", "MG08ACA16TA", "16TB", 100, 0,
))
#> # A tibble: 4 x 5
#>   mfg     name            size   days failures
#>   <chr>   <chr>           <chr> <dbl>    <dbl>
#> 1 Seagate ST16000NM001G   16TB   1440        0
#> 2 Toshiba HDWF180         8TB   13994        1
#> 3 HGST    HUH721010ALE600 10TB   8042        0
#> 4 Toshiba MG08ACA16TA     16TB    100        0

You’ll notice that there are zero failures for most of these drives, even though it seems implausible that they will never fail. This approach will allow us to fill in reasonable estimates.

The remaining data come from the CSV download available in the original post, cleaned up a little:

fname <- "path/to/lifetime_chart_as_of_Q4_2019.csv"

library(dplyr, warn.conflicts = FALSE)

hdds <- read.csv(
  fname, skip = 4, row.names = NULL, stringsAsFactors = FALSE,
  nrows = 14
) %>%
  tibble::as_tibble() %>%
  select(
    mfg = MFR, name = Models, size = Drive.Size, days = Drive.Days,
    failures = Drive.Failures
  ) %>%
  mutate(
    name = trimws(gsub(",", "", name, fixed = TRUE)),
    days = as.integer(gsub(",", "", days, fixed = TRUE)),
    failures = as.integer(gsub(",", "", failures, fixed = TRUE))
  ) %>%
  bind_rows(omitted) %>%
  mutate(
    # Compute BackBlaze's "Annualized Failure Rate".
    afr = failures / (days / 365)
  ) %>%
  arrange(desc(afr))

hdds
#> # A tibble: 18 x 6
#>    mfg     name            size      days failures     afr
#>    <chr>   <chr>           <chr>    <dbl>    <dbl>   <dbl>
#>  1 Seagate ST4000DM000     4TB   52597595     3843 0.0267 
#>  2 Toshiba HDWF180         8TB      13994        1 0.0261 
#>  3 Seagate ST12000NM0007   12TB  20853714     1466 0.0257 
#>  4 Seagate ST8000NM0055    8TB   13172942      439 0.0122 
#>  5 Seagate ST12000NM0008   12TB    321275       10 0.0114 
#>  6 Seagate ST8000DM002     8TB   11807158      351 0.0109 
#>  7 Seagate ST6000DX000     6TB    2902718       84 0.0106 
#>  8 Toshiba MG07ACA14TA     14TB    673365       16 0.00867
#>  9 HGST    HUH728080ALE600 8TB     838384       18 0.00784
#> 10 Toshiba MD04ABA400V     4TB     234748        5 0.00777
#> 11 Seagate ST10000NM0086   10TB   1006626       19 0.00689
#> 12 HGST    HUH721212ALE600 12TB    327080        5 0.00558
#> 13 HGST    HMS5C4040ALE640 4TB   11674201      165 0.00516
#> 14 HGST    HMS5C4040BLE640 4TB   19578610      245 0.00457
#> 15 HGST    HUH721212ALN604 12TB   2919243       32 0.00400
#> 16 Seagate ST16000NM001G   16TB      1440        0 0      
#> 17 HGST    HUH721010ALE600 10TB      8042        0 0      
#> 18 Toshiba MG08ACA16TA     16TB       100        0 0

David Robinson has a well-known blog post on an analogous problem in baseball where he takes an empirical Bayes approach and estimates a reasonable prior distribution from the original data – which works here as well. To get a more stable estimate of the distribution, we can omit AFRs computed for drives with less than 1 million days of service:

afr_sample <- hdds$afr[hdds$days > 1e6]
(afr_beta <- MASS::fitdistr(
  afr_sample, dbeta, start = list(shape1 = 1.5, shape2 = 100),
  lower = 0.01
))
#>      shape1       shape2  
#>     2.379848   198.621649 
#>  (  1.052762) ( 97.652075)

# Empirical beta prior parameters.
alpha0 <- afr_beta$estimate[1]
beta0 <- afr_beta$estimate[2]

This models the failure rate of each model as fixed but drawn from a shared beta distribution (which is a natural distribution for probabilities), and individual days as a Bernoulli trial that can either succeed or fail based on this underlying rate. This is a contestable model – for instance, you might think that workload or other environmental factors are the true cause of failures – but I believe it’s a reasonable approximation.

With the parameters in hand we can then use Bayes rule to update the rates¹, combining the empirical prior with the observed data:

mutate(
  hdds, eb_afr = (failures + alpha0) / ((days / 365) + alpha0 + beta0)
)
#> # A tibble: 18 x 7
#>    mfg     name            size      days failures     afr  eb_afr
#>    <chr>   <chr>           <chr>    <dbl>    <dbl>   <dbl>   <dbl>
#>  1 Seagate ST4000DM000     4TB   52597595     3843 0.0267  0.0266 
#>  2 Toshiba HDWF180         8TB      13994        1 0.0261  0.0141 
#>  3 Seagate ST12000NM0007   12TB  20853714     1466 0.0257  0.0256 
#>  4 Seagate ST8000NM0055    8TB   13172942      439 0.0122  0.0122 
#>  5 Seagate ST12000NM0008   12TB    321275       10 0.0114  0.0115 
#>  6 Seagate ST8000DM002     8TB   11807158      351 0.0109  0.0109 
#>  7 Seagate ST6000DX000     6TB    2902718       84 0.0106  0.0106 
#>  8 Toshiba MG07ACA14TA     14TB    673365       16 0.00867 0.00898
#>  9 HGST    HUH728080ALE600 8TB     838384       18 0.00784 0.00816
#> 10 Toshiba MD04ABA400V     4TB     234748        5 0.00777 0.00874
#> 11 Seagate ST10000NM0086   10TB   1006626       19 0.00689 0.00723
#> 12 HGST    HUH721212ALE600 12TB    327080        5 0.00558 0.00673
#> 13 HGST    HMS5C4040ALE640 4TB   11674201      165 0.00516 0.00520
#> 14 HGST    HMS5C4040BLE640 4TB   19578610      245 0.00457 0.00459
#> 15 HGST    HUH721212ALN604 12TB   2919243       32 0.00400 0.00419
#> 16 Seagate ST16000NM001G   16TB      1440        0 0       0.0116 
#> 17 HGST    HUH721010ALE600 10TB      8042        0 0       0.0107 
#> 18 Toshiba MG08ACA16TA     16TB       100        0 0       0.0118

If you look carefully, you’ll see that drives with smaller samples are more strongly influenced by the prior: all of the drives that were originally omitted have failure rates close to the mean of the prior, and the model suggests that some of the large HGST drives have had a run of good luck as well.

Of course, a plot (see the top of the page) is more compelling than a table.

There is some evidence in the table above that different manufacturers may have slightly different base failure rates, and BackBlaze has in the past speculated that different drive sizes may have different failure characteristics as well. In this case, you could try to recover manufacturer- or drive size-specific parameters by using a hierarchical model. This approach might bring the estimate for the HGST 10TB drive down, for example, since the other HGST models all have low failure rates.

Finally, it’s worth mentioning that failure rates in and of themselves may not be the right metric against which to make a purchasing decision – the drives vary in size and price, so a more useful measure might be an estimate of the annual per-TB replacement cost for each model.

This simple update formula for the beta-binomial is due to the highly convenient fact that the posterior distribution is also beta-binomial, which is partly why this distribution is so popular. ↩︎

Browsing Twitch.tv From Emacs

2020-01-11T00:00:00Z

A video of my presentation from EmacsConf 2019 is now available. You can check out the recording below or see the slides here.

Browser-based applications can sometimes punish even new machines. In 2015, due to limited hardware, I was no longer able to use the popular video streaming site Twitch.tv to follow eSports. I investigated some alternatives at the time, but they lacked discovery and curation features, and so I decided to write a full-fledged Twitch client in my favourite text editor, Emacs. Years later, I still use this little bit of Emacs Lisp almost every day.

The talk discusses how I was able to use the richness of the built-in Emacs features and some community packages to build this client, as well as the various bumps along the way.

The code is available on GitHub.

Structured Errors in Plumber APIs

2019-12-07T00:00:00Z

If you’ve used the Plumber package to make R models or other code accessible to others via an API, sooner or later you will need to decide how to handle and report errors.

By default, Plumber will catch R-level errors (like calls to stop()) and report them to users of your API as a JSON-encoded error message with HTTP status code 500 – also known as Internal Server Error. This might look something like the following from the command line:

$ curl -v localhost:8000/
> GET /status HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Date: Sun, 24 Mar 2019 22:56:27 GMT
< Content-Type: application/json
< Date: Sun, 24 Mar 2019 10:56:27 PM GMT
< Connection: close
< Content-Length: 97
< 
* Closing connection 0
{"error":["500 - Internal server error"],"message":["Error: Missing required 'id' parameter.\n"]}

There are two problems with this approach: first, it gives you almost zero control over how errors are reported to real users, and second, it’s badly behaved at the protocol level – HTTP status codes provide for much more granular and semantically meaningful error reporting.

In my view, the key to overcoming these problems is treating errors as more than simply a message and adding additional context when they are emitted. This is sometimes called structured error handling, and although it has not been used much historically in R, this may be changing. As you’ll see, we can take advantage of R’s powerful condition system to implement rich error handling and reporting for Plumber APIs with relative ease.

But first, it’s worth asking precisely what we want to get out of such an error handling system – that is, how can we distinguish errors we want our users to see?

Operational vs. Programmer Errors

Part of the issue here is that R (and Plumber) treat all errors as essentially the same, when in practice this is not the case.

The folks at Joyent coined the terms operational and programmer errors in the context of Javascript, and I think this distinction is apt for R as well. To quote the article at length:

People use the term “errors” to talk about both operational and programmer errors, but they’re really quite different. Operational errors are error conditions that all correct programs must deal with, and as long as they’re dealt with, they don’t necessarily indicate a bug or even a serious problem. “File not found” is an operational error, but it doesn’t necessarily mean anything’s wrong. It might just mean the program has to create the file it’s looking for first.

By contrast, programmer errors are bugs. They’re cases where you made a mistake, maybe by forgetting to validate user input, mistyping a variable name, or something like that. By definition there’s no way to handle those. If there were, you would have just used the error handling code in place of the code that caused the error!

In the context of Plumber APIs, we want to notify users of operational errors, because we require that they address these errors in order to use the API correctly. Programmer errors, on the other hand, might generate bizarre or misleading messages – so it’s not clear we want users to see them at all. At the same time, it is very important that we see them so that we can start to track down the underlying bugs that caused them.

Operational and programmer errors also have a very natural expression in terms of HTTP status codes; for the most part, 4xx codes are for client (operational) errors, and 5xx codes are for server (programmer) errors.

By design, Plumber assumes that any error it encounters while running your code is a programmer error. This is the right default, but it does mean that you need to go out of your way to report operational errors instead. You can see this clearly by attempting to use R-style errors in a Plumber API.

R-Style Error Handling in Plumber

Suppose we have the following simple plumber.R file, which allows users to query the status of some familiar institutional patients:

records <- data.frame(
  id = 1:3,
  name = c("George", "Sally", "Michael"),
  admitted = c("2018-01-03", "2018-04-14", "2018-05-26"),
  released = c("2018-11-27", "2018-12-25", NA)
)

#* @param id:numeric* The patient's ID number.
#* @serializer unboxedJSON
#* @get /status
status <- function(id = NULL) {
  id <- as.integer(id)
  record <- records[records$id == id,]
  record$status <- if (!is.na(record$released)) "Released" else "Admitted"

  unclass(record)
}

You can run this in the usual way with

server <- plumber::plumber("plumber.R")
server$run(port = 8000, debug = TRUE, swagger = FALSE)

And normal queries should look like the following from the command line (with curl and jq):

$ curl -s localhost:8000/status?id=2 | jq
{
  "id": 2,
  "name": "Sally",
  "admitted": "2018-04-14",
  "released": "2018-12-25",
  "status": "Released"
}
$ curl -s localhost:8000/status?id=3 | jq
{
  "id": 3,
  "name": "Michael",
  "admitted": "2018-05-26",
  "released": null,
  "status": "Admitted"
}

Of course, there are a number of ways the endpoint could fail, so let’s add some R-style error handling:

status <- function(id) {
  if (missing(id)) {
    stop("Missing required 'id' parameter.", call. = FALSE)
  }
  id <- suppressWarnings(as.integer(id))
  if (is.na(id)) {
    stop("The 'id' parameter must be a positive integer.", call. = FALSE)
  }

  record <- records[records$id == id,]
  if (nrow(record) == 0) {
    stop("No patient found with id: ", id, ".", call. = FALSE)
  }
  record$status <- if (!is.na(record$released)) "Released" else "Admitted"

  unclass(record)
}

We can then test some error conditions:

$ curl localhost:8000/status | jq
{
  "error": "500 - Internal server error",
  "message": "Error: Missing required 'id' parameter.\n"
}
$ curl localhost:8000/status?id=cats | jq
{
  "error": "500 - Internal server error",
  "message": "Error: The 'id' parameter must be a positive integer.\n"
}
$ curl localhost:8000/status?id=4 | jq
{
  "error": "500 - Internal server error",
  "message": "Error: No patient found with id: 4.\n"
}

You might notice that I passed debug = TRUE to the run() method above; this is because Plumber will only show the message field in the error responses in “debug” mode. This is partly for privacy – error messages could expose internal state you’d prefer users not to see – but it also in recognition of the point I made above: random R error messages are rarely helpful to users.

Plumber’s default error handler does a few useful things:

Prints the error to the console, so we can see it on the server side. This is absolutely essential for tracking down bugs.
Sets the status code to 500; and
Adds the error message to the response (as the message field you see above), but only when running in debug mode.

Unfortunately, this all means that we can’t use the default handler to send operational error messages back to the user. Instead, we can circumvent it by constructing error responses manually, or override it with smarter code.

Manual Error Reporting

To generate useful operational errors for users, we need to do two things: first, come up with a meaningful payload for errors; and second, ensure that errors set an appropriate HTTP status code. Both of these can be accomplished by manually modifying the response object that Plumber exposes as the magic parameter res.

There are many, many different takes on how to report errors in JSON; I’m going to use a pretty simple one here and include just a status code¹ and a message. For example:

{
  "status": 400,
  "message": "Missing required parameter."
}

Similarly, there is some debate on how to map errors like “invalid parameter” to HTTP status codes, but here I’ll use 400. Both 422 and 409 are common alternatives. For the case when a patient can’t be found, I also think it make sense to use 404.

status <- function(id, res) {
  if (missing(id)) {
    res$status <- 400
    res$body <- jsonlite::toJSON(auto_unbox = TRUE, list(
      status = 400,
      message = "Missing required 'id' parameter."
    ))
    return(res)
  }
  id <- suppressWarnings(as.integer(id))
  if (is.na(id)) {
    res$status <- 400
    res$body <- jsonlite::toJSON(auto_unbox = TRUE, list(
      status = 400,
      message = "The 'id' parameter must be a positive integer."
    ))
    return(res)
  }

  record <- records[records$id == id,]
  if (nrow(record) == 0) {
    res$status <- 404
    res$body <- jsonlite::toJSON(auto_unbox = TRUE, list(
      status = 404,
      message = paste0("No patient found with id: ", id, ".")
    ))
    return(res)
  }
  record$status <- if (!is.na(record$released)) "Released" else "Admitted"

  unclass(record)
}

This gives us much nicer, more meaningful errors we can safely pass down to users of the API:

$ curl -s localhost:8000/status | jq
{
  "status": 400,
  "message": "Missing required 'id' parameter."
}
$ curl -s localhost:8000/status?id=moose | jq
{
  "status": 400,
  "message": "The 'id' parameter must be a positive integer."
}
$ curl -s localhost:8000/status?id=4 | jq
{
  "status": 404,
  "message": "No patient found with id: 4."
}

The code to manipulate res objects for error handling ends up involving a lot of copy & paste, especially for larger APIs where you want to report certain classes of errors in a standard way. Ideally, we want to provide some helper functions so that API authors do the right thing without needing to copy so much code.

Emitting Errors via Custom Conditions

The underlying machinery that powers R’s stop(), warning(), and message() is the concept of a condition. We can construct and “signal” error-like conditions using a simple S3 object that inherits from the "error" class:

api_error <- function(message, status) {
  err <- structure(
    list(message = message, status = status),
    class = c("api_error", "error", "condition")
  )
  signalCondition(err)
}

# Works like stop():
api_error("Bad request.", 400)
#> Error: Bad request.

Moreover, since these are S3 objects, we can use the class attribute to sort out which errors are purposeful, operational errors that need to be reported to the user, and those that are not:

error_handler <- function(req, res, err) {
  if (!inherits(err, "api_error")) {
    res$status <- 500
    res$body <- "{\"status\":500,\"message\":\"Internal server error.\"}"
    
    # Print the internal error so we can see it from the server side. A more
    # robust implementation would use proper logging.
    print(err)
  } else {
    # We know that the message is intended to be user-facing.
    res$status <- err$status
    res$body <- sprintf(
      "{\"status\":%d,\"message\":\"%s\"}", err$status, err$message
    )
  }
  res
}

# Add this to the server with
# server$setErrorHandler(error_handler)

I’d also advise writing some helper methods, like the following:

not_found <- function(message = "Not found.") {
  api_error(message = message, status = 404)
}

missing_params <- function(message = "Missing required parameters.") {
  api_error(message = message, status = 400)
}

invalid_params <- function(message = "Invalid parameter value(s).") {
  api_error(message = message, status = 400)
}

These helper functions allow us to simplify and clarify the code so that it is as concise and familiar looking as it was when we were using stop():

status <- function(id, res) {
  if (missing(id)) {
    missing_params("Missing required 'id' parameter.")
  }
  id <- suppressWarnings(as.integer(id))
  if (is.na(id)) {
    invalid_params("The 'id' parameter must be a positive integer.")
  }

  record <- records[records$id == id,]
  if (nrow(record) == 0) {
    not_found(paste0("No patient found with id: ", id, "."))
  }
  record$status <- if (!is.na(record$released)) "Released" else "Admitted"

  unclass(record)
}

Using a custom error handler and the structured error support of S3 conditions, we now have a way to emit operational errors with ease and a consistent JSON error reporting format. This is an essential piece of providing a robust, user-friendly Plumber API.

I like having the original status code as part of the error payload. That way, even if I don’t have access to the full original request (e.g. someone just copy & pasted the error message to me, or it’s not in the logs, or a proxy along the way did not forward it appropriately), I still have a good idea where to look. ↩︎

Writing Proprietary R Packages

2019-11-26T00:00:00Z

Author’s note: this is a lightly modified version of the talk I gave at the GTA R User’s Group in May of this year. You can find the original slides here. Unfortunately, the talk was not recorded.

As I have noted before, most resources for R package authors are pitched at those writing open-source packages — usually hosted on GitHub, and with the goal of ending up on CRAN.

These are valuable resources, and reflect the healthy free and open-source (FOSS) R package ecosystem. But it is not the whole story. Many R users, especially those working as data scientists in industry, can and should be writing packages for internal use within their company or organisation.

Yet there is comparatively little out there about how to actually put together high-quality packages in these environments.

This post is my attempt to address that gap.

At work we have more than 50 internal R packages, and I have been heavily involved in building up the culture and tooling we use to make managing those packages possible over the last two years.

I’ll focus on three major themes: code, tooling, and culture.

Why Should You Write R Packages for Internal Use?

At the outset, it is worth repeating that R packages are the best way to share R code and keep it well-maintained and reliable. This matters even more inside an organisation or when you are part of a team.

The most salient reason why this is the case is that common tools to make your R code robust, portable, and well-documented are only available for use with packages: R CMD check, testthat, and roxygen are all good examples.

I would push this further than most, and suggest you put as much R code as you can get away with inside packages. For instance, we put all production models in R packages, much of our ETL, and a good portion of our Shiny apps – which has recently become a lot easier due to the excellent golem package.

The Code: What Can Internal Packages Contain?

It is a oft-repeated observation that when you find yourself copy/pasting the same function or snippet of R code between projects, it might be time to push that code into a package.

Inside a business or organisation, functions that I’ve seen generally fall into a small number of categories:

Tools for making data easy to access and use correctly. For example, accessing the right database with the right credentials.
Plot themes and other internal conventions ported to R, such as presentation templates.
Business logic, which are conversions and routines that are highly specific to your organisation or industry; and
Encoding process in code – e.g. automation of team or company-specific tasks.

As an illustration, all of the following are real functions from our internal packages:

# Accessing data.
pull_data(...)
mongo_collection(...)

# Plot themes and templates.
theme_pinnacle(...)
pinnacle_presentation(...)

# Business logic.
vig_cents_to_percent(...)
get_clp_est(..)

# Encoding process.
send_to_slack(...)
rnd_release(...)

There could be other good candidates. As I mentioned, we put models, Shiny applications, and ETL into packages when possible, some of which turns into reusable code.

The Tooling: Limitations and Opportunities

Unlike the CRAN/FOSS world, in an organisation you’ll have limited power to choose the tools your organisation (already) uses for collaboration and development. And, unless you’ve got a compelling reason, you should adopt your organisation’s existing tools.

For example, my organisation uses TeamCity for continuous integration, which does not support R out of the box. In order to get access to shared CI resources, we had to make this possible (which involved using Docker and some custom scripts).

The upside of this is that we could then hook into integrations used by the rest of the organisation. For instance, Slack alerts for R packages that pass or fail their CI tests:

Because everyone is required to use the same tools, you can sometimes turn this to your advantage and leverage a shared tool for opportunities you might not otherwise have.

For example, since everyone is required to be in the same Slack channel or read their corporate email, you can ensure that everyone is notified of important R package releases – ie. this is our Slack bot that posts R package releases:

This kind of broadcast is genuinely impossible for CRAN packages, because there is simply no medium by which to contact all R users. FOSS communities are and always will be more decentralised and heterogeneous.

Moreover, tools can create data (e.g. releases, downloads, commit activity, email messages, Slack messages) that can be analysed to help you measure and understand bottlenecks or problems with your internal processes, since you know they represent the whole picture. I’ve used these data to decide what would help my team by more productive on several occasions.

The Culture: Authorship vs. Maintainership

Ultimately, I think the biggest difference between FOSS and proprietary R packages is a cultural one.

Most CRAN packages are written by someone trying to scratch an itch. Maybe that’s a new statistical method or data transformation, or a new approach to a older ones; maybe it’s a new data format that you need access to from R; or maybe it’s just a bundle of cool stuff you’ve done that you want to make more widely accessible.

There are some consequences of this model. Even if the package is released to CRAN, the expectation is that the original author will (1) design the APIs; (2) write the code; and (3) maintain the package. The author will make all the major decisions about where the package goes and be the authority on why code works the way it does.

The author unconsciously acts as though they will maintain the code forever (or more likely: until the package is abandoned).

I call this an Authorship paradigm.

These packages have a bus factor of exactly 1 – that is, if the author gets hit by a bus, that’s probably the end of the project. It is not an usual state of affairs for FOSS software, by any means.

Things are dramatically different for proprietary code:

You likely inherited a codebase you did not design, and it is your job to maintain it, to understand it, and to be the person to ask about bugs and new features.
You will not maintain new code you have designed/written forever. At some point it will likely be someone else’s job.

I call this a Maintainership paradigm.

What does this imply about writing R packages? Well, you should write the kind of R code you’d like to maintain.

What Kind of R Package Would You Like to Maintain?

This is not a trick question. You want to see

Clear R source code with helpful comments.
Good documentation, with clear explanations and examples, and an overview of the main package features in a README file.
A test suite, both as additional examples and also to help prevent you from introducing regressions.
A documented history: an up-to-date NEWS file and clear git commit history.

In other words, pretty much all of the usual advice. The difference is that you are primarily motivated by collaboration with current and future coworkers.

In sum, I think that there is quite a lot of overlap between “best practice” package development in FOSS and proprietary environments. The main differences arise from the motivations of package authors, the tools and ecosystems they might operate in, and the nature of the code they might write.

An Autoconf Primer for R Package Authors

2019-11-09T00:00:00Z

Have you ever noticed something like the following when you’re installing an R package?

checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
...
configure: creating ./config.status
config.status: creating src/Makevars

on Windows, users install pre-built packages, but on all other platforms (including macOS and Linux), they are built from source, including any native C, C++, or Fortran code. Often it’s enough to use the default R settings to build these packages, but sometimes you might need know a bit more about a user’s system in order to get things working correctly.

For this reason, R permits packages to have a ./configure shell script to make these checks before a package is installed.

Far and away the most common reason you’d need a ./configure script is to check for the presence of a system library that your package needs. You can’t be sure in advance what this library will be called – it may have a different name on Linux and macOS, for example – or where it will be located on a user’s machine, so you need at least a little bit of logic to try and track it down.

Some package authors like to write ./configure scripts by hand (for example, data.table and protolite). But many package authors instead opt to use Autoconf, which is an arcane but highly useful language and tool for generating ./configure scripts that will work on most systems. It also comes with a ton of built-in functions for the kinds of checks you are likely to make.

It is Autoconf that generates output like the snippet at the start of the post.

Unfortunately, there are few resources out there on how to write Autoconf scripts for R packages.¹ Partly as a result, these scripts can be messy and inconsistent in practice. This post aims to provide the boilerplate you need to get started with some “best practices” in place.

Getting Started

To use Autoconf, you’ll generally need the following:

A configure.ac script.
Template files, most often a src/Makevars.in.
Various entries in your .gitignore or .Rbuildignore; and
Optionally, a ./cleanup script to delete various intermediate files.

First, create an src/Makevars.in file with the following contents:²

PKG_CPPFLAGS = -I. @PKG_CPPFLAGS@
PKG_LIBS = @PKG_LIBS@

This is a template file that Autoconf will fill in to produce a final Makevars file telling R how to compile your package correctly. As such, you should add src/Makevars.in to your .Rbuildignore and src/Makevars to your .gitignore.

Second, create a file called configure.ac like the one below:

AC_INIT([<package>],[<version>])

# Find the compiler and compiler flags used by R.
: ${R_HOME=`R RHOME`}
if test -z "${R_HOME}"; then
  echo "could not determine R_HOME"
  exit 1
fi
CC=`"${R_HOME}/bin/R" CMD config CC`
CFLAGS=`"${R_HOME}/bin/R" CMD config CFLAGS`
CPPFLAGS=`"${R_HOME}/bin/R" CMD config CPPFLAGS`

# Search for a system library, in this case the zlib compression library (which
# contains the function 'deflate'). This sets the variable 'ac_cv_search_deflate'
# to what must be passed to ${PKG_LIBS}.
AC_SEARCH_LIBS(deflate, z, [], [AC_ERROR([The zlib library is required.])])
AC_CHECK_HEADERS(zlib.h, [], [AC_ERROR([The zlib library headers are required.])])

# Write the flags into the src/Makevars file.
AC_SUBST([PKG_CPPFLAGS], ["${PKG_CPPFLAGS}"])
AC_SUBST([PKG_LIBS], ["${LIBS} ${PKG_LIBS} ${ac_cv_search_deflate}"])
AC_CONFIG_FILES([src/Makevars])
AC_OUTPUT

echo "
  --------------------------------------------------
  Configuration for ${PACKAGE_NAME} ${PACKAGE_VERSION}

    cppflags: ${CPPFLAGS} ${PKG_CPPFLAGS}
    libs:     ${PKG_LIBS}

  --------------------------------------------------
"

I was not kidding about the “arcane” bit. But you can now run the command autoconf (or, better yet autoreconf) in your package directory to generate a configure script that R can understand.

Now, the contents of this generated script are scary and near-unreadable, but that doesn’t really matter, since you will never edit the file by hand anyway. It is good practice to check both the configure.ac and configure files into version control, and in fact R CMD check will even warn you if you forget.

Finally, you might also notice that autoreconf produces a bunch of intermediate files that you will want to add you your .gitignore:

autom4te.cache
config.log
config.status

This should give you a functional Autoconf setup that will check for the zlib library on the user’s system and link it to your compiled code.

More Common Patterns

1. Using C++ Instead of C

For a project using C++ code instead of C – for example, if you are using Rcpp – you can make the following small adjustment. Replace

CC=`"${R_HOME}/bin/R" CMD config CC`
CFLAGS=`"${R_HOME}/bin/R" CMD config CFLAGS`
CPPFLAGS=`"${R_HOME}/bin/R" CMD config CPPFLAGS`

in the example above with

CXX=`"${R_HOME}/bin/R" CMD config CXX`
CXXFLAGS=`"${R_HOME}/bin/R" CMD config CXXFLAGS`
CPPFLAGS=`"${R_HOME}/bin/R" CMD config CPPFLAGS`
AC_LANG(C++)
AC_PROG_CPP

Autoconf will then run automated checks for C++ features (instead of C). Note that it is important to set CXX, CXXFLAGS and CPPFLAGS before you make these checks, to ensure that Autoconf will check these features using the C++ toolchain that R will actually use to compile your package.

2. Finding Libraries Using `pkg-config`

Most Linux distributions³ come with a program called pkg-config that allows you to query installed libraries and the compiler flags used to build them.

I would suggest using pkg-config by default, falling back on AC_CHECK_HEADERS otherwise. Autoconf actually comes with a PKG_PROG_PKG_CONFIG macro that will set $PKG_CONFIG if it is available, so you can set up these checks fairly easily:

have_zlib=no
ZLIB_CXXFLAGS=""
ZLIB_LIBS="-lz"

PKG_PROG_PKG_CONFIG

if test [ -n "$PKG_CONFIG" ] ; then
  AC_MSG_CHECKING([pkg-config for zlib])
  if $PKG_CONFIG --exists zlib; then
    have_zlib=yes
    ZLIB_CXXFLAGS=`"${PKG_CONFIG}" --cflags zlib`
    ZLIB_LIBS=`"${PKG_CONFIG}" --libs zlib`
  fi
  AC_MSG_RESULT([${have_zlib}])
fi

if test "x${have_zlib}" = xno; then
  AC_CHECK_HEADERS(zlib.h, [have_zlib=yes], [AC_ERROR(
    [The zlib library headers are required.]
  )])
fi

AC_SUBST([PKG_LIBS], ["${LIBS} ${PKG_LIBS} ${ZLIB_LIBS}"])

3. Showing Users What They Need to Install

The examples above will print

The zlib library headers are required.

in case of failure. This isn’t a terribly helpful message for users, so an emerging best practice for R packages is to provide more actionable messages explaining what users need to install on various systems to get things working.

For example, you could remove the AC_ERROR message above add the following block below it:

if test "x${have_zlib}" = xno; then
  AC_MSG_FAILURE([
  ---------------------------------------------
   'zlib' and its header files are required.

   Please install:

   * zlib1g-dev (on Debian and Ubuntu)
   * zlib-devel (on Fedora, CentOS and RHEL)
   * zlib (via Homebrew on macOS)
   * libz1 (on Solaris)

   and try again.

   If you believe this library is installed on your system but
   this script is simply unable to find it, you can specify the
   include and lib paths manually:

   R CMD INSTALL ${PACKAGE_NAME} \\
     --configure-vars='LIBS=-L/path/to/libs CPPFLAGS=-I/path/to/headers'
  ---------------------------------------------])
fi

A few quick searches should turn up the various package names on each platform. The ones in the example above are the main platforms that CRAN checks against, although instructions for Solaris are seldom worth the bother.

4. Using a Cleanup Script

R permits packages to have a ./cleanup script. From Writing R Extensions:

Under a Unix-alike only, an executable (Bourne shell) script cleanup is executed as the last thing by R CMD INSTALL if option –clean was given, and by R CMD build when preparing the package for building from its source.

This can be a useful way to automatically clean up generated Autoconf files. Here is one to get you started:

#!/bin/sh

rm -f config.*
rm -f src/Makevars
rm -rf autom4te.cache

5. Windows Support

Windows users generally install binary packages, and will not run the ./configure script during installation. However, you might want to help out the CRAN maintainers who create these binary packages by creating a static, Windows-oriented src/Makevars.win file:

PKG_CPPFLAGS = -I. -lz

This will obviously break when it cannot find zlib, but we’re punting that responsibility to the binary package maintainers in this case.

For and Against data.table

2019-06-27T00:00:00Z

I was reminded by Megan Stodel’s recent post – “Three reasons why I use data.table” – of the enduring data.table vs. dplyr argument.

To Stodel’s points in favour of data.table, which I would summarise as (1) performance, (2) good design, and (3) that it makes you feel a bit iconoclastic in these tidyverse times, I would add the following:

A very stable, actively-maintained, and battle-tested codebase. This is also true of both dplyr and base R, of course, but it is valuable nonetheless.
Zero third-party dependencies. In contrast, dplyr has, at the time of writing, 22 direct or indirect package dependencies. In combination with the choice of C over Rcpp, this leads to very fast compilation and installation times.

Both of these are highly valued in production environments, which happens to be the focus of my current work.

However, I would usually advocate against using data.table unless you really, really do need its performance benefits.

Much of the tibble/data.table debate seems to be over peoples' preferred syntax,¹ or the degree to which base R, data.table, or the tidyverse produces more understandable code – and, relatedly, whether it is easier to teach to newcomers. All of these are valuable discussions.

However, I’d like to articulate another reason why I’m always cautious about adopting data.table: modify-by-reference can lead to surprising and non-idiomatic R code.

Every data.table user probably knows that the reason the package is fast is that it discards R’s usual copy-on-modify behaviour, instead modifying objects “by reference”.² It is worth explaining in some detail what this means.

Copy-on-Modify vs. Modify-by-Reference

R users are accustomed to the fact that objects will not change unless you reassign to them with the <- operator. And so we can look at code like

df <- data.frame(x = rnorm(100), y = runif(100) * 5)
custom_summary(df)
df <- create_features(df)

and expect that the call to custom_summary() will not modify df, while the call to create_features() clearly does. We would know this without needing to look at the source code for these functions, even if they were given unhelpful names like fn1() and fn2(). Now suppose that custom_summary() looked like the following:

custom_summary <- function(df) {
  df$summary_measure <- df$x * floor(df$y)
  df$x <- ifelse(df$x < -1e-4, -1e-4, df$x)
  df$x <- ifelse(df$x > 1e-4, 1e-4, df$x)
  summary(df)
}

That is, the function cleans up the data a little before presenting a summary. Even though the function is modifying df internally, we know that these changes will not “leak” out of the scope of the function. This is what is meant by copy-on-modify: R will copy objects as necessary to prevent modifications from manifesting in the original.

Now suppose that for performance reasons we decide to translate this code to use data.table objects. The result might look like the following:

df <- data.table(x = rnorm(100), y = runif(100) * 5)
custom_summary(df)
df[, y := floor(y)]
df[, feature1 := x * 2 + y]
df[, feature2 := x * x]

Even though there are no further uses of <-, these last lines all modify df in-place. This is what is meant by “modify-by-reference”. We could also move these into a new create_features() function, and this function would not need to use <- to modify the original data.table object.

A naive refactor could be carried out for the summary function, as well:

custom_summary <- function(df) {
  df[, summary_measure := x * floor(y)]
  df[x < -1e-4, x := -1e-4]
  df[x > 1e-4, x := 1e-4]
  summary(df)
}

This new version introduces an subtle bug: if the call to custom_summary() is later removed or commented out, df$x may have larger or smaller entries than expected (since they are no longer capped). This is a direct consequence of modify-by-reference, and could never happen with native data.frame or tibble objects.

It’s true that this is a contrived example, but I think it’s a useful illustration nonetheless. Experienced data.table users might argue that functions should “clean up” after themselves if they make use of modify-by-reference features, or perhaps that sometimes-surprising side effects are merely the cost of using the package to begin with. The authors of data.table present an analogous scenario in one of the vignettes; they suggest that the function call copy() before modifying the input. Yet this is likely to erase the performance benefits of using data.table objects in the first place.

Reference Semantics are Not Idiomatic in User Code

As Bob Rudis remarks, what constitutes “idiomatic” R can vary, but in practice there are very few aspects of base R that have “reference semantics” – i.e. that break copy-on-modify rules in favour of modify-by-reference.

An important set of exceptions to this general rule are the functions that load packages, print to the console, or plot. All of these modify in place (e.g. without using <-) important objects: the search path, the console, and the graphics device. What these exceptions have in common is precisely that they manipulate well-known global objects that are baked into R itself.

In contrast, user-created objects – the usual vectors, data frames, and functions – almost never have reference semantics. This is not to say that you are categorically unable to use them. There are even explicit language features with reference semantics, namely environments, reference classes, and external pointers – or even when writing functions that use the <<- operator. Yet it is exceedingly rare to see these in the usual data analysis code; they are mostly used by packages to implement design patterns that would otherwise be very difficult in a copy-on-modify world.

You can see echos of this in Hadley Wickham’s warning to users about R6 classes (which have reference semantics) in the new edition of Advanced R:

[I]f you use R6 it’s very easy to create a non-idiomatic API that will feel very odd to native R users, and will have surprising pain points because of the reference semantics.

It’s my guess that concern about these pain points is the primary reason why dplyr never adopted the modify-by-reference approach, even though it has obviously been shown by data.table to be more performant.

Does This Mean You Should Use dplyr?

To be clear, data.table has its place. All of the arguments in favour of using it I mentioned at the outset remain in good standing. And reference semantics are not the only reason for the performance of data.table operations.

Nor does this mean that I would wholeheartedly recommend using dplyr and friends at every opportunity. Most data wrangling can be done in base R,³ and I often argue that this is the right choice, for instance, inside packages.

If anything, what I would suggest is that if you are going to use data.table, try to ensure that you write an API that exposes as little as possible of its reference semantics. Avoid creating situations that will surprise users expecting <- as the signal for modification.

Atrebas has produced an excellent side-by-side comparison of some common analysis patterns with data.table and dplyr. ↩︎
I’m a bit critical of this language because all R objects are, at the C level, pointers/references. Copy-on-modify is actually a contract adhered to by the parts of R written in C; it is not enforced by the language and can be broken. This is how it was possible to actually implement data.table in the first place.

Better terms are “reference semantics” or “modify-by-reference”, because they separate behaviour from implementation. ↩︎
For instance, Jozef Hajnala has a neat collection of articles demonstrating how common dplyr and data.table patterns can be implemented in base R. ↩︎

Copyright in Closed-Source R Packages: The Right Way

2019-03-18T00:00:00Z

There are a wealth of excellent resources for R users on creating and maintaining R packages, which remain the best way to share your code within your organisation or the larger community. However, almost all of these resources tend to focus on open-source packages, tools, and workflows. As a result of this, they tend to skim over more corporate issues like copyright assignment.

Yet many R users are working on their code in proprietary environments, creating packages for internal use by their company. That code is closed source, with the copyright belonging to their organisation. If you fall into this category, how should you communicate that in your R package? This post is intended to provide a very clear answer to that question.

It turns out that R has excellent support for embedding copyright information in the metadata of the package itself. This makes it easy to precisely communicate both open- and closed-source ownership in a few well-defined places: your DESCRIPTION and LICENSE files.

The DESCRIPTION File

It is very likely that if you are writing R packages for your employer, they are the ultimate copyright owner of the software. Fortunately it’s very easy to lay out this fact in your DESCRIPTION file – there is a special copyright holder role (cph) that can be used in the Authors fields. For example, suppose that I worked for Widgets Incorporated. In that case, my DESCRIPTION file would contain something like the following:

Authors@R: c(
  person("Aaron", "Jacobs", role = c("cre", "aut")),
  person("Widgets Incorporated", role = "cph")
  )

This is not quite the end of the story, because R also requires that you to lay out what open-source license your code is available under in the License field, or use the special value file LICENSE to point users to a file explaining it. Since your code is not open-source, your DESCRIPTION file must take this route and contain the following:

License: file LICENSE

But this just punts on the issue – what do you put in that LICENSE file?

The LICENSE File

Hadley Wickham’s highly influential R Packages book makes the following suggestion for the contents of a proprietary LICENSE file:

Proprietary 

Do not distribute outside of Widgets Incorporated.

In my view, this is a mistake. Why? Because this file says absolutely nothing about copyright. It bears no resemblance to a valid software license; notably, it does not detail any of the rights or permissions granted to those in possession of the source code.

But wait, you say, my company doesn’t want to grant any rights or permissions! If this is the case – and likely it is – your LICENSE file should contain a familiar legal phrase: “All rights reserved”. In full:

Copyright 2019 Widgets Incorporated. All rights reserved.

This is a true proprietary software license.¹

Relicensing

For most closed source code, this is the end of the copyright story. But some initially-internal packages are later released under an open source license, and you might be curious as to how you should go about doing this.

Fortunately the answer is quite simple if you’ve got things set up as I outlined above: all you have to do is change the license. If you are using one of the open-source licenses understood by R itself, you can delete the LICENSE file and change the License field in your DESCRIPTION file. If you are using a more uncommon license, just replace the content of the LICENSE file in its entirety with the text of that license.

(You can see examples of this on GitHub in the commit history of the internal projects an employer permitted me to open-source: here and here.)

It is important to note what you should not change here, namely the copyright role (cph) for your employer in the DESCRIPTION file. Open-sourcing a package does not transfer copyright to you or anyone else – that is a wholly separate process.

TL;DR

To sum up, put this in your DESCRIPTION file:

Authors@R: c(
  person("<My>", "<Name>", role = c("cre", "aut")),
  person("<My Company>", role = "cph")
  )
License: file LICENSE

And this in a LICENSE file:

Copyright 2019 <My Company>. All rights reserved.

And that’s it.

Fair warning: I’m not a copyright lawyer. If you are, and have an issue with the content or arguments presented in this post, please reach out to me and I will attempt to address it. ↩︎

What Does R Value?

2018-12-13T00:00:00Z

I’m fan of Bryan Cantrill’s argument that we ought to think about “platform values” when assessing technologies, highlighted in a recent talk, but explained more thoroughly in an earlier one on his experience with the node.js community.

In my reading, he argues that of the many values (such as performance, security, or expressiveness) that programming languages or platforms may hold, many are in conflict, and the platform inevitably emphasises some of these values over others. These decisions are a reflection of explicit or implicit “platform values”.

Cantrill illustrates this with a series of examples, but, no surprise, the R platform does not make his shortlist. I couldn’t resist trying to cook up my own taxonomy of platform values for the R language and its community.

More broadly, though, Cantrill believes that the values of a platform affect the projects that adopt them, and conflicts can arise between the values of a project and its respective choice of platform. This strongly echoes my own experience in the R community, and I’ve gotten a measure of clarity for future and existing projects by learning to articulate these conflicts.

R’s Values: The List

Note: The effect of the following tables will be diminished without this site’s CSS.

Cantrill gives the following selection of values that we might claim as “core” to a platform, although he doesn’t consider them exhaustive:

• Approachability	• Integrity	• Robustness
• Availability	• Maintainability	• Safety
• Compatibility	• Measurability	• Security
• Composability	• Operability	• Simplicity
• Debuggability	• Performance	• Stability
• Expressiveness	• Portability	• Thoroughness
• Extensibility	• Resiliency	• Transparency
• Interoperability	• Rigour	• Velocity

He also gives a few illustrations of platform values; for example, he formulates the core values of the C programming language as

• Approachability	• Integrity	• Robustness
• Availability	• Maintainability	• Safety
• Compatibility	• Measurability	• Security
• Composability	• Operability	• Simplicity
• Debuggability	• Performance	• Stability
• Expressiveness	• Portability	• Thoroughness
• Extensibility	• Resiliency	• Transparency
• Interoperability	• Rigour	• Velocity

This is not to say that these values preclude us from writing safe, robust, or composable C code – in fact, this is expected of expert C programmers – but that the language itself does not offer strong support for doing so. Many features that would make C programs safer have been sacrificed at the altar of performance.

In contrast, these are what he sees as the core values of the Javascript community in general and node.js in particular:

• Approachability	• Integrity	• Robustness
• Availability	• Maintainability	• Safety
• Compatibility	• Measurability	• Security
• Composability	• Operability	• Simplicity
• Debuggability	• Performance	• Stability
• Expressiveness	• Portability	• Thoroughness
• Extensibility	• Resiliency	• Transparency
• Interoperability	• Rigour	• Velocity

There are some additional examples for C++, Awk, K, OpenBSD – which values only Security – and others in his talks.

I would formulate the core values of the R platform as follows:

• Approachability	• Integrity	• Robustness
• Availability	• Maintainability	• Safety
• Compatibility	• Measurability	• Security
• Composability	• Operability	• Simplicity
• Debuggability	• Performance	• Stability
• Expressiveness	• Portability	• Thoroughness
• Extensibility	• Resiliency	• Transparency
• Interoperability	• Rigour	• Velocity

The presence of Approachability and Velocity in this list should be no surprise: R is and always has been a language heavily geared to those learning a bit of programming to fulfil their need to do data analysis and statistics. It has a long history of efforts to get that group up and running as fast as possible. I also think that the recent push to build tools and packages that make working with R “fun” or “easy” – complete with the requisite emoji – is a re-expression of these values as core to the platform.

Similarly, it is clear to me that R holds Extensibility by its users, through the development of functions and then packages, in very high regard. It is a widely-held view that any non-trivial R code ought to be bundled up into a package so that it can be distributed painlessly to others. In fact, this view is so pervasive that the traditional “scripting” options for R are limited: Rscript did not become part of the language until 2007, and none of the built-in tooling (e.g. NAMESPACE or DESCRIPTION files, or R CMD check) works with standalone R scripts.

It’s worth explaining why I have highlighted Integrity. What I have in mind here is R’s language level support for the notion of “missing values” (NA). All functions must handle missing values, and function authors are constantly on the hook for deciding how their function ought to behave in the absence of data (na.rm, anyone?). This represents a remarkable commitment to ensuring the integrity of R’s many mathematical programs.¹

R’s Values: The Consequences

Part of the larger argument about why platform values are important is that they affect the nature of the projects that are built on these platforms. Some of this is unequivocally positive: for instance, I would argue that efforts by folks like DataCamp to teach R as widely as possible have been much helped by the platform’s preexisting commitment to Approachability.

Similarly, one might well attribute the enormous popularity of R as an implementation language for cutting-edge statistical methods to the platform’s focus on Velocity and Extensibility, at least in the case of the academic statistical community.

And I have no doubt that R’s focus on Integrity has had a quiet but meaningful impact on the many statistical analyses that are undertaken with it.

Still, we can’t have everything, and in the case of the R platform, there is no doubt that you can encounter hurdles if you pursue projects that align with different values. In my view, the most pressing concern is the way R’s platform values diverge from those held by those running applications of one kind or another.

Much of my recent professional life has been focused on divining ways for my team to run R code in “production” environments, where values like Stability, Resiliency, Robustness, Measurability, and Debuggability become dramatically more important. This has had a substantial impact on the nature of the R code we write, and the kinds of tools we build to help us write it better.

With the advent of Shiny and Plumber, these past few years have witnessed a sea change in the ability for R users to tackle writing true applications. It remains to be seen whether R’s platform values – as they exist today – will curtail the growth of these projects, or whether these projects will have some success in changing the culture of R itself.

One might also argue that the popular tidyverse ecosystem attempts to bring Composability and Compatibility more into the foreground. You can see this in the current mission statement of the project:

The tidyverse is an opinionated collection of R packages designed for data science. All packages share an underlying design philosophy, grammar, and data structures.

↩︎

Why There is No importFrom() Function in R

2018-08-27T00:00:00Z

For those coming to R from other languages, it seems very odd that R users import code from packages the way that they do. For instance, if you are used to Python, this is the general pattern of “using code from elsewhere”:

import math
import numpy as np
from random import randint

# Usage:
math.floor(3.2)
np.array(...)
randint()

Meanwhile, in R, you’re likely to see

library(dplyr)

data_frame(x = 1, y = "A") %>%
  mutate(z = TRUE)

… with the subtext being that users must simply know that the data_frame(), mutate(), and %>% functions are actually all from the dplyr package. Why is R so unusual here?

Unlike C or some familiar Lisps, R has proper namespace support, so it’s not as though conflicting symbols are simply overwritten when you use library() in the way they would be when using #include in C. Moreover, there is one place that R users do work with a highly granular mechanism to control how symbols are imported into an environment: when working with the NAMESPACE file included with every R package. If you crack open one of these files, you’ll see lines like

importFrom(dplyr, mutate, data_frame, %>%)

which look a bit like pseudo-R code. Yet there is no importFrom() function in R itself. It’s not hard to write one if we make use of the not-widely-advertised namespace API, either:

importFrom <- function(pkg, ...) {
  pkg <- as.character(substitute(pkg))
  call <- match.call()
  symbols <- vapply(3:length(call), function(i) {
    as.character(call[[i]])
  }, character(1))
  ns <- loadNamespace(pkg)
  importIntoEnv(parent.frame(), symbols, ns, symbols)
  invisible(NULL)
}

# Confirm that it works:
importFrom(tibble, is.tibble, as_tibble)

df <- dplyr::data_frame(x = 1, y = "A")
is.tibble(df)
#> [1] TRUE
is.tibble
#> function(x) {
#>   "tbl_df" %in% class(x)
#> }
#> <bytecode: 0x55c14454ca60>
#> <environment: namespace:tibble>

So that’s clearly not the barrier. In fact if this appeals to you there is a fully-realized vision of an import*() function for R available in the import package. But there is still a question of why the language itself does not include such a function.

I was curious about whether anything had been written about this before, so I asked about the historical reason that R does not have an importFrom() function on the r-devel mailing list. While I didn’t get an answer from those involved, those who did comment raised some interesting points.

The first is that larger R programs tend to get refactored into R packages, which do have careful management of imported functions and packages. (This largely aligns with my own experience.) And the second is that R has a strong culture of development-by-interpreter, where library() is a useful shortcut to getting the third-party functions that you want.

I also suspect that the history of the language plays a part as well. When namespace support was added to the R language in 2003, the language had a strong emphasis on maintaining compatibility with the S language, so that new users would not need to rewrite much of their existing code. S used library(), and thus R provided library(). And insofar as there was a design principle for the S language, it was that users could begin using it without thinking of themselves as “programmers”, and gradually slide into programming as their analyses demanded it. The library() approach is in line with this philosophy, in my view.

And of course, there is also the reality that R has almost thirty years of development and design decisions to account for – almost 50 if we include its S ancestors dating back to 1976. By this measure, R is contemporary with the very first versions of Unix (1975), Emacs (1976), and TeX (1978). I’m sure that some of these decisions might be regretted or resented by the core R developers, but most of these now-features have users expecting at least some semblance of backwards compatibility.

In the end I think that the main reason that we don’t see mass migration to solutions like the import package speaks to a deeper division in the R community. The R users who are likely to raise concerns over using library() in scripts – those worried about code hygiene or dependency management, for example – are likely willing and able to solve these problems in other ways, notably by refactoring code into packages. And as Roger Peng argued in his recent keynote at useR! 2018, R has focused on programmer-oriented features at its peril.

For the large contingent of R users writing run-once data analysis scripts – still the key constituency of the language – library() is exactly what you want: no hassle access to third-party functions. That this is the default recommendation, as opposed to some version of importFrom(), reflects the nature of the R community.

Communicating with UDP Sockets from R

2018-07-06T00:00:00Z

Recently I wanted to send some Shiny usage data from R to a certain metrics server. Since R includes write.socket() and friends for opening arbitrary network sockets, it seemed at the outset that this would be quite simple. However, I ran into an interesting roadblock along the way.

It turns out that R’s socket API only supports TCP connections, which you can confirm by looking at the source code – and in this case I needed to send UDP packets instead. This was a little surprising to me, since most other languages would include UDP support out of the box; it’s a core internet protocol, after all. For whatever reason, this seems not to be the case with R, and even after searching CRAN and GitHub I wasn’t able to find an existing package that provides UDP socket support.

To remedy this, I put together a simple way to write messages to UDP sockets from R.

Since working with sockets is done at the system level, it makes sense to use R’s C API in this case. The following is a self-contained implementation that you can drop into an R package’s src directory as udp.c:

#include <Rinternals.h>
#include <R_ext/Rdynload.h>

#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/socket.h>
#include <arpa/inet.h>

SEXP udp_send_impl(SEXP message, SEXP host, SEXP port) {
  const char* msg = CHAR(asChar(message));
  const char* host_ = CHAR(asChar(host));
  int port_ = asInteger(port);

  // Open the socket.
  int sock;
  struct sockaddr_in server;

  if ((sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)) < 0) {
    Rf_error("Failed to create UDP socket.");
    return R_NilValue;
  }

  server.sin_family = AF_INET;
  server.sin_addr.s_addr = INADDR_ANY;
  server.sin_port = htons((short) port_);

  if (inet_aton(host_, &server.sin_addr) == 0) {
    Rf_error("Failed to parse host address.");
    close(sock);
    return R_NilValue;
  }

  // Send message.
  if (sendto(sock, msg, strlen(msg), 0, (struct sockaddr *) &server,
             sizeof(server)) < 0) {
    Rf_error("Failed to send message.");
  }

  close(sock);
  return R_NilValue;
}

static const R_CallMethodDef udp_entries[] = {
  {"udp_send_impl", (DL_FUNC) &udp_send_impl, 3},
  {NULL, NULL, 0}
};

void R_init_udp(DllInfo *info) {
  R_registerRoutines(info, NULL, udp_entries, NULL, NULL);
  R_useDynamicSymbols(info, FALSE);
}

Note that you may need to adjust the R_registerRoutines() code if you have existing C or C++ code in your package.

To use this C function from R, you’ll need something like the following in your package:

#' @export
#' @useDynLib udp udp_send_impl
udp_send <- function(message, host = "localhost", port) {
  stopifnot(is.character(message))
  stopifnot(is.character(host))
  stopifnot(is.numeric(port))
  .Call(udp_send_impl, message, host, as.integer(port))
  invisible(NULL)
}

To confirm that this works, you can listen for UDP packets with from a terminal with

$ nc -lu 1001

and you can send UDP packets to that server from R with

udp_send("Hello from R!", host = "localhost", port = 1001)

Principles for Pipe-Friendly APIs in R Packages

2018-03-28T00:00:00Z

In the past few years the APIs of the wildly popular tidyverse packages have coalesced around the pipe operator (%>%). R users familiar with this ecosystem of packages are now accustomed to writing “pipelines” where code is expressed as a series of steps composed with the pipe operator.

For example, plenty of data science code basically boils down to:

data() %>% transform() %>% summarise()

(This is even more evident if you subscribe to Wickham & Grolemund’s view that models are a “low-dimensional summary of your data”.)

There are plenty of resources on how to use pipes to re-write you R code. There are fewer on the implications of pipes on how R functions and package APIs are designed. What should package authors should keep in mind, knowing that their users are likely to use their functions in pipelines?

My own experience has led me to compile four principles for writing pipe- friendly APIs for R packages:

Only one argument is going to be “piped” into your functions, so you should design them to accommodate this. The first argument should be what you’d expect to be piped in; all other arguments should be parameters with meaningful default values. If you think you’ll always need more than one argument, consider wrapping them up in a lightweight S3 class.
Think carefully about the output of your functions, since this will be what is passed down the pipeline. Strive to always return a single type of output (a data frame, a numeric vector, etc). Never return NULL, because very few functions can take NULL as an input. (To signify empty values, use zero-row data frames or vectors with NA.)
Prefer “pure” functions – e.g. those that have no “side effects” that mutate global state – whenever possible. Users think about pipelines as a linear progression of steps. The transparency of pure functions make them easy to reason about in this fashion.
When your functions must have side effects (e.g. when printing or plotting), return the original object (instead of the conventional NULL) so that pipelines can continue.

I’ve found that being explicit about these design goals has improved the quality of my own package interfaces; perhaps they can be of use to others.

A Note on Migrating Packages from rjson to jsonlite

2018-02-19T00:00:00Z

Recently I was working on an R package that had historically used both rjson and jsonlite to serialize R objects to JSON (before sending it off to an API). In this case we wanted to remove the rjson depdency, which was only used in a few places.

The most noticable hiccup I encountered while porting the code was during encoding of parameter lists generated in R, which looked something like

params <- list(key1 = "param", key2 = NULL,
               key3 = c("paired", "params"))

In this case, rjson produced exactly what our API was looking for:

cat(rjson::toJSON(params))
#> {"key1":"param","key2":null,"key3":["paired","params"]}

But by default the jsonlite package will behave very differently:

jsonlite::toJSON(params)
#> {"key1":["param"],"key2":{},"key3":["paired","params"]}

(That is, the NULL has been converted to an empty dictionary and the key1 parameter has been turned into a one-element array.)

Fortunately, jsonlite is quite configurable. Vectors of a single element can be converted into bare values by passing the auto_unbox parameter to toJSON() (or handled individually with unbox()), and handling of NULL values can be customized with the null paramter.

To get the jsonlite equivalent to the rjson output above, we used

jsonlite::toJSON(params, auto_unbox = TRUE, null = "null")
#> {"key1":"param","key2":null,"key3":["paired","params"]}

During the transition, I ran a number of tests to ensure that the conversions were equivalent. Since jsonlite produces an object of class "json", you’ll need to coerce to character to make things like all.equal() work correctly:

a <- rjson::toJSON(params)
b <- jsonlite::toJSON(params, auto_unbox = TRUE, null = "null")

all.equal(a, b)
#> [1] "Attributes: < target is NULL, current is list >"
#> [2] "target is character, current is json"
all.equal(a, as.character(b))
#> [1] TRUE

Technical merits of each respective package aside, I think it’s likely that some existing rjson code will migrate to jsonlite in the future. Hopefully this post helps make that process a smooth one.

Don't Forget to Reproject Spatial Data when Exporting to GeoJSON

2017-07-05T00:00:00Z

In the process of working on a recent choropeth piece for work, I discovered that it’s easy to stumble when moving spatial data out of R and onto the web. It’s a poorly-documented reality that many web-based mapping libraries (including both D3 and Leaflet) expect GeoJSON data to be in EPSG:4326, and it is by no means a given that that your spatial data will start off in this projection.

If you’re like me and do your spatial data pre-processing in R before exporting to GeoJSON, you may have to re-project your data before these libraries will handle them properly. Thankfully, this is fairly easy to do with the modern spatial packages.

The next-gen spatial package sf provides an st_transform() function for reprojecting spatial objects. My current workflow for taking shapefiles, adding/removing features and data, and writing out a GeoJSON file that can be read in D3 or Leaflet looks something like the following:

library(dplyr)
library(sf)

sf::st_read("path/to/shapefile", layer = "layer") %>%
  # Filter, mutate, join, select on this spatial data.
  ... %>%
  # Re-project the map into EPSG:4326 before writing to GeoJSON.
  sf::st_transform(crs = 4326) %>%
  sf::st_write("map.json", driver = "GeoJSON")

This GeoJSON data can then be re-projected using the Javascript library into something more appropriate than EPSG:4326. In D3 this looks something like the following:

// This example projection works well for Canadian maps. I
// cribbed it from Statistics Canada:
// http://www.statcan.gc.ca/pub/82-402-x/2015001/gui-eng.htm#a6
var proj = d3.geoConicConformal()
    .parallels([49, 77])
    .rotate([91.86, -63.390675]);

var path = d3.geoPath()
    .projection(proj);

d3.json("map.json", function(error, map) {
    if (error) throw error;

    // Protip: use `fitSize()` set the projection scale, center,
    // and translation automatically to fit in the SVG element.
    proj.fitSize([780, 500], map.features);

    d3.select("svg")
        .attr("width", 780)
        .attr("height", 500)
        .selectAll("path")
        .data(map.features)
        .enter()
        .append("path")
        .attr("d", path)
        .style("stroke", "white");
});

I did report this issue to sf itself, but the maintainer indicated that the library ought to be sticking to the standard, regardless of existing implementation quirks. In the meantime, if you plan to export GeoJSON data for use with Javascript libraries, don’t forget to reproject as necessary.

Forecasting YYZ Passengers in the Tidyverse

2017-03-02T00:00:00Z

Buried in the Toronto Economic Bulletin (warning: Excel document) is a column listing the number of passengers passing through Pearson International Airport (YYZ) each month, going back more than fifteen years. There’s a story to tell in the forecast, too:

Flight data are a great forecasting example because they display such clear seasonal patterns, in this case peaking in the summer months and falling off in the winter. R has excellent tools for working with time series data and whipping up simple forecasts like this one. But there’s some friction with the modern tidyverse tools, because the latter expect a data.frame as the common interchange format.

In this post, I’ll outline an approach to fitting many time series models using the tidyverse tools, including model selection for out-of-sample performance. To ease the transition between these two worlds I make extensive use of list columns and the broom package.

The Set Up

For this post I’ve extracted the data into a vector of R’s built-in ts class, which is the common interchange format for most time series packages. As of writing, I have data from January 2000 through August 2016.

# load("path/to/passengers.rda")
str(passengers)

##  Time-Series [1:200] from 2000 to 2017: 2072798 2118150 2384907 2260620 2386165 2635018 2788843 2942082 2477000 2527969 ...

You’ll also need the following packages, all of which are staples of the tidyverse except for forecast, which is a prominent time series forecasting package for R:

suppressMessages({
    library(dplyr)
    library(tidyr)
    library(purrr)
    library(broom)
    library(ggplot2)
    library(forecast)
})

Time Series Forecasting in R

There are an overwhelming number of time series models used for forecasting out there (see the CRAN task view on time series for a sense of what’s available in R alone), but the most common ones are probably ARIMA and ETS, both of which can handle seasonal data sets like this one.

In choosing a model for forecasting, what we care about is the accuracy of the model on unseen (that is, out-of-sample) data. Since we don’t have a way of measuring that directly, we simulate the experience by holding out a test set.

The nature of the autocorrelated errors in time series data mean that randomly selecting training and test sets (or, indeed, using traditional bootstrap or cross-validation) will erase much of the information in the data, so common practice with time series data is to hold out the last few observations, ideally the same interval as you’d like to forecast over.¹

I’ll mention that this risks overfitting to this particular test set, something I’ll return to below. (As the statistician Andrew Gelman has remarked, it is important to realise that test sets and cross validation are estimates of out-of-sample performance, not a direct measure.)

For this post I’m going to hold out the two most recent years' worth of data as a test set. R provides a convenient window function for breaking up time series:

train <- window(passengers, end = c(2014, 8))
test <- window(passengers, start = c(2014, 9))

We can use this training set to fit some ARIMA and ETS models, and see how they compare.

Fitting ARIMA Models in R

R has built in support for fitting ARIMA models with, you guessed it, the arima() function. However, there are a number of shortcomings to this implementation, so I’d recommend using the Arima() version from the forecast package instead. You can also try your luck with the package’s auto.arima() function, which will try and find an optimal model automatically.

Choosing the order of an ARIMA model usually benefits from some manual exploration of the autocorrelated errors, especially using the forecast::tsdisplay() plot. For brevity I won’t walk you through the process here, but I’d strongly recommend taking a look at Rob Hyndman and George Athanasopoulos' chapter on forecasting seasonal data using ARIMA, which is freely available online.

For now, auto.arima() should do just fine. It’s worth experimenting with the approximations disabled, as follows:

arima.fit <- forecast::auto.arima(train, d = 1, D = 1,
                                  stepwise = FALSE,
                                  approximation = FALSE)
arima.fit

## Series: train 
## ARIMA(0,1,1)(0,1,1)[12]                    
## 
## Coefficients:
##           ma1     sma1
##       -0.1927  -0.6731
## s.e.   0.0774   0.0675
## 
## sigma^2 estimated as 6.925e+09:  log likelihood=-2080.57
## AIC=4167.15   AICc=4167.3   BIC=4176.43

The automatic process selects an ARIMA model with low order, indicating that a simple model might fit this data reasonably well.

Fitting ETS Models in R

The forecast package also provides an ets function, which by default will choose an optimal model:

ets.fit <- forecast::ets(train, model = "ZZZ")
ets.fit

## ETS(A,N,A) 
## 
## Call:
##  forecast::ets(y = train, model = "ZZZ") 
## 
##   Smoothing parameters:
##     alpha = 0.7467 
##     gamma = 1e-04 
## 
##   Initial states:
##     l = 2373863.6415 
##     s=-155048.7 -371915.5 -66744.27 -9469.378 562836.6 411528.9
##            104613.3 -2894.734 -101326.8 65840.67 -269851.6 -167568.5
## 
##   sigma:  79047.8
## 
##      AIC     AICc      BIC 
## 4909.794 4912.794 4957.351

Notice that this (ostensibly optimal) model has an AICc of 4912.794, which is much higher than the ARIMA models explored above. This may be an indication that ETS models are not a good fit for this data set.

Fitting Many Models using List-Columns and Purrr

One can choose both an ARIMA and ETS model based on how well it fits the training data. However, when forecasting, better out-of-sample performance is the true goal. For this reason, I’m going to take a more systematic approach, fitting a wide variety of both ETS and ARIMA models and comparing their out-of-sample performance.

Inspired by the general coherence of the tidyverse, I believe all of this can be organized into a tidy data frame, with one model per row. I use the following approach:

Create columns for training data, modelling functions, and any other model parameters.
Wrap these training data and model parameters into a single param column.
Use purrr::invoke_map() to apply the modelling function to these model parameters; and
Use purrr::map() and friends (or broom::glance()) to extract goodness of fit measures.

To start with, we create data sets of models and their parameters. The following will generate a data frame of all possible ETS models (check out ?ets for details):

ets.params <- tidyr::crossing(
    error = c("A", "M"), trend = c("N", "A", "M"),
    seasonal = c("N", "A", "M"), damped = c(TRUE, FALSE)
) %>%
    # Drop combinations with a damped non-trend.
    mutate(drop = ifelse(trend == "N" & damped, TRUE, FALSE)) %>%
    filter(!drop) %>%
    # Create labels for the models out of these parameters.
    mutate(kind = "ETS",
           desc = paste0("(", error, ",", trend,
                         ifelse(damped, "d", ""),
                         ",", seasonal, ")"),
           model = paste0(error, trend, seasonal)) %>%
    # Drop nonsensical models (these are flagged by `ets` anyway).
    filter(!model %in% c("MMA", "AMN", "AMA", "AMM",
                         "ANM", "AAM")) %>%
    select(kind, desc, model, damped)

ets.params

## # A tibble: 19 × 4
##     kind     desc model damped
##    <chr>    <chr> <chr>  <lgl>
## 1    ETS  (A,A,A)   AAA  FALSE
## 2    ETS (A,Ad,A)   AAA   TRUE
## 3    ETS  (A,A,N)   AAN  FALSE
## 4    ETS (A,Ad,N)   AAN   TRUE
## 5    ETS  (A,N,A)   ANA  FALSE
## 6    ETS  (A,N,N)   ANN  FALSE
## 7    ETS  (M,A,A)   MAA  FALSE
## 8    ETS (M,Ad,A)   MAA   TRUE
## 9    ETS  (M,A,M)   MAM  FALSE
## 10   ETS (M,Ad,M)   MAM   TRUE
## 11   ETS  (M,A,N)   MAN  FALSE
## 12   ETS (M,Ad,N)   MAN   TRUE
## 13   ETS  (M,M,M)   MMM  FALSE
## 14   ETS (M,Md,M)   MMM   TRUE
## 15   ETS  (M,M,N)   MMN  FALSE
## 16   ETS (M,Md,N)   MMN   TRUE
## 17   ETS  (M,N,A)   MNA  FALSE
## 18   ETS  (M,N,M)   MNM  FALSE
## 19   ETS  (M,N,N)   MNN  FALSE

With this set of model parameters in hand, we can create list columns containing the training data and the function to compute the model (ets in this case). We can also wrap up the parameters into a param list column by using the curious but useful transpose() function.

ets.models <- ets.params %>%
    # Add in the training set and the modelling function.
    mutate(fn = replicate(forecast::ets, n = n()),
           train = replicate(list(train), n = n())) %>%
    # Create a "param" column to pass to `fn`.
    mutate(params = purrr::transpose(list(
        "y" = train, "model" = model, "damped" = damped
    ))) %>%
    select(kind, desc, train, fn, params)

ets.models

## # A tibble: 19 × 5
##     kind     desc    train     fn     params
##    <chr>    <chr>   <list> <list>     <list>
## 1    ETS  (A,A,A) <S3: ts>  <fun> <list [3]>
## 2    ETS (A,Ad,A) <S3: ts>  <fun> <list [3]>
## 3    ETS  (A,A,N) <S3: ts>  <fun> <list [3]>
## 4    ETS (A,Ad,N) <S3: ts>  <fun> <list [3]>
## 5    ETS  (A,N,A) <S3: ts>  <fun> <list [3]>
## 6    ETS  (A,N,N) <S3: ts>  <fun> <list [3]>
## 7    ETS  (M,A,A) <S3: ts>  <fun> <list [3]>
## 8    ETS (M,Ad,A) <S3: ts>  <fun> <list [3]>
## 9    ETS  (M,A,M) <S3: ts>  <fun> <list [3]>
## 10   ETS (M,Ad,M) <S3: ts>  <fun> <list [3]>
## 11   ETS  (M,A,N) <S3: ts>  <fun> <list [3]>
## 12   ETS (M,Ad,N) <S3: ts>  <fun> <list [3]>
## 13   ETS  (M,M,M) <S3: ts>  <fun> <list [3]>
## 14   ETS (M,Md,M) <S3: ts>  <fun> <list [3]>
## 15   ETS  (M,M,N) <S3: ts>  <fun> <list [3]>
## 16   ETS (M,Md,N) <S3: ts>  <fun> <list [3]>
## 17   ETS  (M,N,A) <S3: ts>  <fun> <list [3]>
## 18   ETS  (M,N,M) <S3: ts>  <fun> <list [3]>
## 19   ETS  (M,N,N) <S3: ts>  <fun> <list [3]>

And now we can make use of invoke_map() to fit all of these models. I’ve also extracted the AICc values of each for comparison.

ets.fits <- ets.models %>%
    # Fit the models, and extract the AICc.
    mutate(fit = purrr::invoke_map(fn, params),
           aicc = purrr::map_dbl(fit, "aicc")) %>%
    arrange(aicc)

ets.fits

## # A tibble: 19 × 7
##     kind     desc    train     fn     params       fit     aicc
##    <chr>    <chr>   <list> <list>     <list>    <list>    <dbl>
## 1    ETS  (A,N,A) <S3: ts>  <fun> <list [3]> <S3: ets> 4912.794
## 2    ETS  (M,N,M) <S3: ts>  <fun> <list [3]> <S3: ets> 4914.492
## 3    ETS  (A,A,A) <S3: ts>  <fun> <list [3]> <S3: ets> 4923.181
## 4    ETS (A,Ad,A) <S3: ts>  <fun> <list [3]> <S3: ets> 4925.173
## 5    ETS (M,Ad,M) <S3: ts>  <fun> <list [3]> <S3: ets> 4925.813
## 6    ETS (M,Md,M) <S3: ts>  <fun> <list [3]> <S3: ets> 4926.635
## 7    ETS  (M,N,A) <S3: ts>  <fun> <list [3]> <S3: ets> 4928.246
## 8    ETS (M,Ad,A) <S3: ts>  <fun> <list [3]> <S3: ets> 4941.445
## 9    ETS  (M,A,A) <S3: ts>  <fun> <list [3]> <S3: ets> 4942.699
## 10   ETS  (M,A,M) <S3: ts>  <fun> <list [3]> <S3: ets> 4977.796
## 11   ETS  (M,M,M) <S3: ts>  <fun> <list [3]> <S3: ets> 4985.782
## 12   ETS  (M,M,N) <S3: ts>  <fun> <list [3]> <S3: ets> 5283.217
## 13   ETS  (M,N,N) <S3: ts>  <fun> <list [3]> <S3: ets> 5283.227
## 14   ETS  (M,A,N) <S3: ts>  <fun> <list [3]> <S3: ets> 5285.426
## 15   ETS (M,Ad,N) <S3: ts>  <fun> <list [3]> <S3: ets> 5285.783
## 16   ETS (M,Md,N) <S3: ts>  <fun> <list [3]> <S3: ets> 5286.098
## 17   ETS  (A,N,N) <S3: ts>  <fun> <list [3]> <S3: ets> 5301.575
## 18   ETS (A,Ad,N) <S3: ts>  <fun> <list [3]> <S3: ets> 5305.353
## 19   ETS  (A,A,N) <S3: ts>  <fun> <list [3]> <S3: ets> 5308.133

Note that the best-performing model here — ETS(A,N,A) — is the same one arrived at automatically above.

We can do the same thing with the ARIMA models: generate a list of model parameters as a data frame, collect these into a param list-column, and then fit the function using invoke_map(). Unlike ETS, there are an infinite number of possible ARIMA models. However, we can guess from above that good models will have low order.

arima.params <- tidyr::crossing(ar = 0:2, d = 1:2, ma = 0:2,
                                sar = 0:1, sma = 0:1) %>%
    # Create a label for the model.
    mutate(kind = "ARIMA",
           desc = paste0("(", ar, ",", d, ",", ma, ")(",
                         sar, ",1,", sma, ")")) %>%
    group_by(kind, desc) %>%
    # This has to be done rowwise to avoid the flattening
    # behaviour of c().
    do(order = c(.$ar, .$d, .$ma),
       seasonal = c(.$sar, 1, .$sma)) %>%
    ungroup()

arima.models <- arima.params %>%
    # Add in the training set and the modelling function.
    mutate(fn = replicate(forecast::Arima, n = n()),
           train = replicate(list(train), n = n())) %>%
    # Create a "param" column to pass to `fn`.
    mutate(params = purrr::transpose(list(
        "y" = train, "order" = order, "seasonal" = seasonal
    ))) %>%
    select(kind, desc, train, fn, params)

arima.fits <- arima.models %>%
    # Fit the models, and extract the AICc.
    mutate(fit = purrr::invoke_map(fn, params),
           aicc = purrr::map_dbl(fit, "aicc")) %>%
    arrange(aicc)

arima.fits

## # A tibble: 72 × 7
##     kind           desc    train     fn     params         fit     aicc
##    <chr>          <chr>   <list> <list>     <list>      <list>    <dbl>
## 1  ARIMA (0,2,2)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4157.528
## 2  ARIMA (1,2,1)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4157.796
## 3  ARIMA (0,2,2)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4158.203
## 4  ARIMA (1,2,1)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4158.223
## 5  ARIMA (2,2,1)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4159.600
## 6  ARIMA (1,2,2)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4159.658
## 7  ARIMA (2,2,1)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4160.292
## 8  ARIMA (1,2,2)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4160.306
## 9  ARIMA (0,2,1)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4160.888
## 10 ARIMA (0,2,1)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4161.533
## # ... with 62 more rows

Since this data frame has the same form as the one used for the ETS models, we can concatenate them into a master list of models. The forecast() function is its namesake package’s universal forecasting function. We can use it to add a column of the two-year forecast for all of these models.

yyz.forecasts <- dplyr::bind_rows(ets.fits, arima.fits) %>%
    mutate(forecast = map(fit, forecast::forecast, h = 24)) %>%
    arrange(aicc)

yyz.forecasts

## # A tibble: 91 × 8
##     kind           desc    train     fn     params         fit     aicc
##    <chr>          <chr>   <list> <list>     <list>      <list>    <dbl>
## 1  ARIMA (0,2,2)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4157.528
## 2  ARIMA (1,2,1)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4157.796
## 3  ARIMA (0,2,2)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4158.203
## 4  ARIMA (1,2,1)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4158.223
## 5  ARIMA (2,2,1)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4159.600
## 6  ARIMA (1,2,2)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4159.658
## 7  ARIMA (2,2,1)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4160.292
## 8  ARIMA (1,2,2)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4160.306
## 9  ARIMA (0,2,1)(0,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4160.888
## 10 ARIMA (0,2,1)(1,1,1) <S3: ts>  <fun> <list [3]> <S3: ARIMA> 4161.533
## # ... with 81 more rows, and 1 more variables: forecast <list>

Measuring Forecast Accuracy

The AICc is not a direct measure of forecast accuracy. Traditional root mean squared error (RMSE) and mean absolute error (MAE) are both options, although the author of the forecast package recommends mean absolute scaled error (MASE). The package provides the accuracy() function to compute all of these measures (and a few others) for any forecast object, but unfortunately it returns a matrix with meaningful row names, which would be hard to wrap up inside a list column. What we need is an API that returns a data frame.

It turns out that there is a project devoted specifically to this problem. The broom package provides a standard interface for “tidying” model summaries using the S3 glance() method. At the time of writing, the package doesn’t contain an implementation for forecast objects, but it isn’t too hard to write one using the output of forecast::accuracy():

glance.forecast <- function(x, newdata = NULL, ...) {
    res <- if (is.null(newdata)) {
        acc <- forecast::accuracy(x)
        tibble::as_tibble(t(acc[1,]))
    } else {
        acc <- forecast::accuracy(x, newdata)
        tibble::as_tibble(t(acc[2,]))
    }
    # Format the names of the measures to suit broom::glance().
    names(res) <- tolower(names(res))
    if (length(names(res)) > 7) names(res)[8] <- "thielsu"
    res
}

This can now be combined with tidyr::unnest() to get columns of RMSE, MAE, MASE, and so on using map():

acc <- yyz.forecasts %>%
    mutate(glance = map(forecast, broom::glance)) %>%
    unnest(glance) %>%
    arrange(rmse, mae, mase)

select(acc, kind, desc, rmse, mae, mase)

## # A tibble: 91 × 5
##     kind           desc     rmse      mae      mase
##    <chr>          <chr>    <dbl>    <dbl>     <dbl>
## 1  ARIMA (2,1,2)(1,1,1) 76061.98 51352.61 0.3289412
## 2    ETS        (M,N,M) 76143.55 49990.36 0.3202153
## 3    ETS       (M,Ad,M) 76387.04 50008.08 0.3203288
## 4    ETS       (M,Md,M) 76595.67 50443.22 0.3231161
## 5    ETS        (A,N,A) 79047.80 53876.34 0.3451071
## 6    ETS        (M,N,A) 79174.58 54396.73 0.3484405
## 7  ARIMA (2,2,1)(1,1,1) 79294.88 53378.97 0.3419212
## 8  ARIMA (1,2,2)(1,1,1) 79303.39 53397.52 0.3420400
## 9  ARIMA (1,1,2)(1,1,1) 79305.25 53167.62 0.3405674
## 10 ARIMA (0,2,2)(1,1,1) 79307.55 53368.28 0.3418527
## # ... with 81 more rows

Each accuracy measure provides a slightly different ranking of models. Does this list lay out which model is best? If what you are after is forecast accuracy, the answer is no. It is overwhelmingly likely that some models which perform exceptionally well on this training data may do poorly when confronted with new data. This is precisely why we left a “test” series aside earlier.

Measuring Out-of-Sample Performance Using a Test Set

You might have noticed above that the broom::glance() implementation above accepts a newdata parameter for out-of-sample accuracy assessment. We can pass the test data left aside at the start of the post in the same fashion as above, and lay out the in-sample and out-of-sample measures side by side.

This is also a great opportunity for visual exploration! I’m going to use the ggrepel package to add cleaner labels to the top in-sample and test-set performers, although you can also substitute the built-in ggplot2::geom_label() here.

perf <- yyz.forecasts %>%
    mutate(test = replicate(list(test), n = n()),
           glance = map2(forecast, test, broom::glance)) %>%
    tidyr::unnest(glance) %>%
    # Pull out the out-of-sample RMSE, MAE, and MASE.
    select(kind, desc, oos.rmse = rmse,
           oos.mae = mae, oos.mase = mase) %>%
    # Join the in-sample RMSE, MAE, and MASE columns.
    inner_join(acc, by = c("kind", "desc")) %>%
    rename(is.rmse = rmse, is.mae = mae, is.mase = mase) %>%
    arrange(oos.rmse, is.rmse) %>%
    select(kind, desc, is.rmse, oos.rmse, is.mae, oos.mae,
           is.mase, oos.mase)

# Find the top four models by in-sample and out-of-sample MASE.
top.perf <- perf %>%
    mutate(top.is = rank(is.mase), top.oos = rank(oos.mase)) %>%
    filter(top.is <= 4 | top.oos <= 4)

library(ggrepel)

# Graph in-sample vs. out-of-sample MASE for the top 50% of each.
perf %>%
    filter(oos.mase < quantile(oos.mase, 0.5) |
           is.mase < quantile(is.mase, 0.5)) %>%
    ggplot(aes(y = oos.mase, x = is.mase)) +
    geom_point(aes(colour = kind)) +
    # Vertical/horizontal lines at the minimum MASE for each sample.
    geom_vline(aes(xintercept = min(is.mase)),
               linetype = 5, colour = "gray50") +
    geom_hline(aes(yintercept = min(oos.mase)),
               linetype = 5, colour = "gray50") +
    # Label the top models.
    ggrepel::geom_label_repel(aes(label = paste(kind, desc)),
                             size = 2.5, segment.colour = "gray50",
                             data = top.perf) +
    labs(x = "In-Sample MASE", y = "Out-of-Sample MASE",
         colour = "Family")

![plot of chunk forecast-perf-plot](/images/r-6c067198/forecast-perf-plot-1.png)

This plot reveals some interesting patterns. First, it looks as though some of the multiplicative ETS models do quite well on measures of in-sample accuracy, but poorly when confronted with the test set (top left). Second, there is an ETS (M,M,M) model that does very well on the test set, but quite unremarkably on the training data (bottom right). To mirror this, the best in-sample performer has only middling test set performance (middle right). Third, the models that do well on both measures tend to be ARIMA models of low order (bottom left).

What conclusions can we draw from this? Test set performance isn’t the ultimate arbiter of forecasting performance. As I mentioned above, you can overfit to this specific test set, which is probably what is going on with the high-flying ETS (M,M,M) model. The low-order ARIMA models strike me as a better choice.

Forecasting, Finally

Okay, now that we’ve settled on a model, it’s time to actually forecast the series forward.

passengers.fc <- passengers %>%
    Arima(order = c(2, 1, 2), seasonal = c(1, 1, 0)) %>%
    forecast(h = 24, level = c(50, 95))

All forecast accuracy measures aside, there’s no substitute for the sense of model performance one gets by visual inspection. The forecast object has an implementation for the ggplot2::autoplot() S3 method in addition to the base graphics plot(), so we can get a ggplot2 graphic very easily. The autoplot() output doesn’t include the fitted values by default, but we can add them manually:

autoplot(passengers.fc) +
    geom_path(aes(y = as.vector(passengers.fc$fitted),
                  x = time(passengers.fc$fitted)),
              linetype = 1, alpha = 0.5, colour = "red")

![plot of chunk forecast-plot](/images/r-6c067198/forecast-plot-1.png)

It looks like the model fits the data quite well, aside from some trouble around 2003. The forecast output also gives us a nice story to tell: monthly passengers at YYZ looks likely to break 5 million for the first time in summer 2017.

With this story in hand, you can wrap up the output of autoplot() in a pretty theme, add some bells and whistles, and publish a nice little graphic on this data. The code below generates the plot at the top of this post:

millions <- function(x) paste0(format(x / 1e6, digits = 2), "m")

autoplot(passengers.fc, shadecols = c("#cccccc", "#eeeeee")) +
    geom_segment(aes(y = 5e6, yend = 5e6, x = 2016.5, xend = 2019),
                 size = 0.5, colour = "#888888", linetype = 5) +
    scale_x_continuous(breaks = seq(2000, 2018, by = 2), expand = c(0, 1)) +
    scale_y_continuous(labels = millions) +
    theme_minimal() +
    theme(legend.position = "none",
          panel.grid.minor = element_blank(),
          panel.grid.major.x = element_blank(),
          axis.ticks.x = element_line(colour = "#333333", size = 0.5),
          axis.ticks.length = grid::unit(2, units = "pt"),
          text = element_text(family = "Arial Narrow", size = 8),
          plot.title = element_text(size = 12, face = "bold")) +
    labs(title = paste("Monthly Passengers at Pearson International Airport",
                       "to Top 5 Million in 2017"),
         subtitle = paste("Incoming and outgoing passengers in millions,",
                          "with forecast data in blue."),
         caption = paste("A. Jacobs :: unconj.ca\n",
                         "Data: Toronto Open Data (2016)\n",
                         "You may redistribute this graphic under",
                         "the terms of the CC-by-SA license."),
         x = NULL, y = NULL)

## Scale for 'x' is already present. Adding another scale for 'x', which
## will replace the existing scale.

![plot of chunk narrative-plot](/images/r-6c067198/narrative-plot-1.png)

The author of the forecast package has recently revealed that future versions of the package will provide support for computing cross validation-like residuals. I may discuss incorporating this function in a future post. ↩︎

Using hledger with ledger-mode

2017-02-10T00:00:00Z

Last summer I landed a few patches in Emacs’s ledger-mode that make it easier to use with alternative implementations of Ledger, such as hledger. Since the competing hledger-mode garnered some attention last week on Hacker News, I thought these new features might be worth highlighting to those interested in plain-text accounting in Emacs.

Why choose ledger-mode over hledger-mode?

ledger-mode is the original major mode for the Ledger file format, first penned by current Emacs maintainer and Ledger author John Wiegley himself. It’s generally less opinionated than the newer hledger-mode and is less tied to the specifics of hledger’s command line output, but each has some nice features the other lacks.

Unfortunately, up until last summer certain commands (such as register reports) expected command line flags that hledger lacks, and so several things were broken if you tried to use hledger to run your reports instead.

To summarise the changes, you can now use hledger pretty seamlessly by adding the following somewhere in your init.el file:

;; Required to use hledger instead of ledger itself.
(setq ledger-mode-should-check-version nil
      ledger-report-links-in-register nil
      ledger-binary-path "hledger")

If you write your own reports, you might also be interested in the new %(binary) format specifier (demo below), which will substitute in whatever you have set as the binary path. This makes it a little easier to switch back and forth, too.

Why use hledger at all?

Again, this is mostly a matter of taste, but hledger and Ledger do offer slightly different features. I’m particularly fond of hledger’s column format for periodic reports. I find it useful to compare expenses from one month to the next, and this makes it easy to do so.

For example, you could add the following to your ledger-reports:

("monthly expenses"
 (concat "%(binary) -f %(ledger-file) balance expenses"
         "--tree --no-total --row-total --average --monthly"))

It should produce output like the following:

Balance changes in 2016/12/01-2017/02/28:

               ||  2016/12  2017/01  2017/02    Total  Average 
===============++==============================================
 Expenses      || $1977.97 $1713.99 $1402.97 $5094.93 $1698.31 
   Bills       ||   $90.00   $90.00   $90.00  $270.00   $90.00 
     Internet  ||   $35.00   $35.00   $35.00  $105.00   $35.00 
     Phone     ||   $55.00   $55.00   $55.00  $165.00   $55.00 
   Cash        ||   $41.00   $27.50   $60.00  $128.50   $42.83 
   Food        ||  $549.21  $679.52  $443.09 $1671.82  $557.27 
     Dining    ||  $247.67  $302.31  $186.77  $736.75  $245.58 
     Groceries ||  $301.54  $377.21  $256.32  $935.07  $311.69 
   FunTimes    ||  $177.76  $210.21  $145.56  $533.53  $177.84 
   Misc        ||  $500.00   $86.76   $44.32  $631.08  $210.36 
     Books     ||        0   $34.99        0   $34.99   $11.66 
     Car       ||        0   $51.77   $44.32   $96.09   $32.03 
     Gifts     ||  $340.00        0        0  $340.00  $113.33 
     Travel    ||  $160.00        0        0  $160.00   $53.33 
   Rent        ||  $500.00  $500.00  $500.00 $1500.00  $500.00 
   Transit     ||  $120.00  $120.00  $120.00  $360.00  $120.00

(No, this is not my real data.)

You can get ledger-mode on MELPA.

An Update to the Choropleth Post

2017-02-06T00:00:00Z

A few years ago I published a post outlining how to make nice-looking choropleth maps in R, and this piece still draws a reasonable share of my hits each month. Unfortunately, some of the techniques I used at the time are now quite out of date, and I was starting to feel bad for anyone taking my advice.

As of today the post has received a makeover, and takes a more modern approach. For any returning readers, the changes are explained in a series of HTML <ins> tags — which I have only recently discovered.

When do RcppEigen 'expressions' matter?

2016-09-27T00:00:00Z

Some time ago I discovered a very surprising performance issue while working with the Rcpp package, and thought I’d share the example I discovered.

At the time, I was interested in computing the vector of leverages for a linear model (which you need for certain covariance matrices). This vector is constructed from the diagonal elements of the “hat” (or “influence”) matrix H, such that

$$ H = X (X^T X)^{-1} X^T$$

In base R, you might write the following function to compute this for you:

leverage <- function(X) {
    diag(X %*% solve(t(X) %*% X) %*% t(X))
}

In other words, it’s a simple matrix calculation: the exact kind of thing you might want to optimize with Rcpp.

Computing Leverages using Rcpp

For fun, I decided to implement this same computation in Rcpp code, specifically using the RcppEigen package, since having matrix classes is really nice in this case.

My first version looked like the following:

// [[Rcpp::depends(RcppEigen)]]
#include <RcppEigen.h>

using namespace Rcpp;
using Eigen::Map;
using Eigen::VectorXd;
using Eigen::MatrixXd;

typedef Map<MatrixXd> MapMatd;

// [[Rcpp::export]]
VectorXd leverage_cpp(const NumericMatrix& XX) {
    const MapMatd X(as<MapMatd>(XX));
    const MatrixXd H = X * (X.adjoint() * X).inverse() * X.adjoint();
    return H.diagonal();
}

But later I was feeling strongly about brevity, and opted for this instead:

// [[Rcpp::export]]
VectorXd leverage_cpp_short(const NumericMatrix& XX) {
    const MapMatd X(as<MapMatd>(XX));
    return (X * (X.adjoint() * X).inverse() * X.adjoint()).diagonal();
}

You can load these into R by using the sourceCpp function

library(RcppEigen)

#sourceCpp("code above")

And we can confirm that they produce the same results as the pure R version by using one of the built-in data sets, called attitude:

data("attitude", package = "datasets")
model <- lm(rating ~ ., data = attitude)
X <- model.matrix(model)

all.equal(leverage(X), leverage_cpp(X))

## [1] "names for target but not for current"

all.equal(leverage(X), leverage_cpp_short(X))

## [1] "names for target but not for current"

Notice that RcppEigen does not set the names of these vectors in the same way that diag does, so the vectors are not precisely identical.

Benchmarking R Against Rcpp Functions

The first thing that comes to mind when rewriting any R code using Rcpp is: how much faster is it? The brillaint microbenchmark package provides an easy way to test this:

library(microbenchmark)
library(ggplot2)
 
# Benchmark the results.
test <- microbenchmark(
    leverage_r         = leverage(X),
    leverage_cpp       = leverage_cpp(X),
    leverage_cpp_short = leverage_cpp_short(X),
    times = 1000L
)

test

## Unit: microseconds
##                expr    min     lq      mean median      uq      max neval
##          leverage_r 73.378 87.126 103.80609 91.880 100.933 3683.704  1000
##        leverage_cpp 10.542 13.503  18.59291 16.767  22.383   83.863  1000
##  leverage_cpp_short  7.857  9.182  12.03752 10.254  13.728  128.881  1000

The microbenchmark package also supports the ggplot2::autoplot API, if you prefer a plot:

library(ggplot2)

ggplot2::autoplot(test)

![plot of chunk unnamed-chunk-5](./images/r-7797a4c4/unnamed-chunk-5-1.png)

As expected, the Rcpp code is 5 to 10 times faster, even on this small data set. But there’s a small mystery here as well: why is the second piece of C++ code faster than the first?

If you’re operating in R world, it’s not all that surprising that two different pieces of code to do the same thing have different performance characteristics. This is, after all, the main reason that the microbenchmark package comes in handy.

But since C++ is a compiled language, I generally expect the compiler to figure out that two pieces of code that do the same thing ought to boil down to the same machine code. Why is that not the case here?

The answer is probably to do with how Eigen (the library that RcppEigen wraps) defers computation of an expression in intelligent ways. My guess is that Eigen figures out at compile time that there’s a really simple loop for computing this matrix expression, and pares your high-level description down to a simple statement. By breaking the computation in to two parts (as in the first C++ function), you hamper the ability of the library (not the compiler!) to help you out.

This whole experience has made me much more cautious about the C++ code that I write for R, to the point that I’m now prone to writing the function a number of different ways, and then doing some tests with microbenchmark to see what ends up being faster. Of course, it’s arguable that I should have been taking that kind of empirical approach to begin with.

If anyone can shed some light on this by looking at the assembly output (which is beyond my own abilities), I would be quite grateful.

Exporting Clock Entries from org-mode to CSV

2016-09-05T00:00:00Z

If you’ve used the clocking features of org-mode, you’re no doubt familiar with the clock table, which allows you to summarise time spent on different tasks. This is great for getting an overview of projects, but it’s not a very flexible tool if you want to have a more detailed idea of how you spend your time.

At this point I’ve accumulated about a year’s worth of clocked work time in org, and while clock tables have served me well so far, eventually I just wanted to get my data into R or Python for more minute analysis, and charts like the following:

![Calendar heatmap example](/images/calendar-clock-heatmap.png)

However, I haven’t come across a reliable¹ way to get individual clock entries out of org-mode files and into a more widely readable format. So I’ve written one.

The org-clock-csv package, available on GitHub and through MELPA, parses all of the clock entries in an org-mode file and exports them to CSV format. At present the columns include start and end dates, the task’s category, effort, and tags. Habits are also given a flag.

You can invoke the parser by calling org-clock-csv in interactive mode, in which case it will parse whatever you’ve got in org-agenda-files and pop up a buffer with the results. The function can also be called from lisp code with a file argument, and there’s an org-clock-csv-batch version that will dump the CSV results to standard output (for use with Emacs in batch mode).

If you’re looking for inspiration on what to do with your shiny new data set, here’s a recent post (though it’s not org-mode based) that poses some good questions.

I did find the in-progress R package orgclockr, but its home-grown org parser (in R) failed on my files. Since the best org parser is already avaiable in Emacs, it made more sense to me to write an Emacs package that outputs a universal format. ↩︎

Custom Hexbin Functions with ggplot

2016-08-13T00:00:00Z

Recently, I wanted to create a map similar to James Cheshire’s crime map of London, which shows the most common crimes commited in a rectangular grid of points laid over London. Instead of using a rectangular grid, I wanted to use hexbins, but it turns out that ggplot needs a bit of prodding to do anything other than simply count the number of observations in each bin.

At the time I couldn’t find a good tutorial on writing custom hexbin functions, so this post is a reasonably thorough explanation of what I’ve made work.

Getting Spatial Data

To get these examples up and running, you’ll need some spatial data. I’m going to use something similar to the data used in the original crime map of London: crimes reported to the Metropolitan Police Department (of London) from May 2012 to April 2013. You can get the exact same extract I have used from the excellent data.police.uk website by following this link. The files come in one-month increments, each in their own folder. There are a number of interesting columns, but for our purposes we only need the coordinates and the type of crime. I’ve appended a quick bit of code to concatenate all of them together and extract the relevant columns at the end of this post.

You will also need a fairly recent (post version 2.0) of ggplot2. To use geom_hex and related functions, you will also require the hexbin package, which I’ve loaded explicitly here. I’m personally a fan of dplyr and the %>% operator, but its use here is purely to make working with the data easier.

library(dplyr, warn.conflicts = FALSE)
library(hexbin)
library(ggplot2)

The data looks as follows:

# load("path/to/crimes.rda")
head(crimes, 10)

## # A tibble: 10 x 5
##     Year Month Longitude Latitude                 Crime
##    <int> <int>     <dbl>    <dbl>                <fctr>
## 1   2012     5  0.135866 51.58734 Anti-social behaviour
## 2   2012     5  0.134947 51.58806 Anti-social behaviour
## 3   2012     5  0.135866 51.58734 Anti-social behaviour
## 4   2012     5  0.134947 51.58806 Anti-social behaviour
## 5   2012     5  0.137065 51.58367 Anti-social behaviour
## 6   2012     5  0.134947 51.58806 Anti-social behaviour
## 7   2012     5  0.140192 51.58231 Anti-social behaviour
## 8   2012     5  0.134947 51.58806 Anti-social behaviour
## 9   2012     5  0.134947 51.58806 Anti-social behaviour
## 10  2012     5  0.134947 51.58806 Anti-social behaviour

Creating a Basic Spatial Hexbin Map

To start with, we can produce a traditional hexbin overlaid on a basemap (that is, using something like Open Street Map or Google Maps as a “background”). The ggmap package is a handy way to get a basemap into your ggplot maps. You can get a basemap for London (as a ggplot object) as follows:

library(ggmap)

# These are approximate coordinates for London.
london.ish <- c(`left` = -0.25, `bottom` = 51.4,
                `right` = 0, `top` = 51.6)
london.map <- ggmap::get_map(location = london.ish,
                             maptype = "toner", source = "stamen")

basemap <- ggmap::ggmap(london.map) +
    coord_equal() +
    labs(x = NULL, y = NULL) +
    theme(axis.text = element_blank())

# The data set actually contains a lot of crime reports from outside
# of London itself. We're not interested in these observations.
crimes <- crimes %>%
    filter(Longitude > london.ish[1],
           Longitude < london.ish[3],
           Latitude > london.ish[2],
           Latitude < london.ish[4])

Which allows us to create a nice, traditional hexbin map:

basic.hexmap <- basemap +
    geom_hex(aes(x = Longitude, y = Latitude,
                 fill = cut(..value.., c(0, 100, 250, 500, 1000,
                                         1500, 2000, 2500, Inf))),
             colour = NA, bins = 30, alpha = 0.75,
             data = crimes) +
    scale_fill_brewer(palette = "OrRd",
                      labels = c("<100", "100-250", "250-500",
                                 "500-1000", "1000-1500",
                                 "1500-2000", "2000-2500",
                                 ">2500")) +
    labs(fill = NULL,
         title = "Reported Crimes in London, by Location (2012-2013)")

print(basic.hexmap)

![plot of chunk unnamed-chunk-4](./images/r-b6720d/unnamed-chunk-4-1.png)

I’ve binned the count data so that the map is a little more informative. (You can read more about using the cut function for this in my earlier post on hexbins.) If you look carefully you might notice that there are very few crimes in the City of London — I’m assuming that this is because they have their own police force.

The geom_hex geometry works by using stat_bin_hex to transform the source data frame into another data frame with binned data. (This is where the magic ..value.. aesthetic that I used above comes from: it refers to a column in the data frame that stat_bin_hex outputs.) In fact you can use the statistic directly to do some interesting things. Say, by using dots instead of hexagons, but still maintaining the hexagonal tiling:

dot.hexmap <- basemap +
    stat_bin_hex(aes(x = Longitude, y = Latitude, fill = NULL,
                     colour = cut(..value.., c(0, 100, 250, 500, 1000,
                                               1500, 2000, 2500, Inf))),
                 geom = "point", size = 3, bins = 30, alpha = 0.75,
                 data = crimes) +
    scale_colour_brewer(palette = "OrRd",
                        labels = c("<100", "100-250", "250-500",
                                   "500-1000", "1000-1500",
                                   "1500-2000", "2000-2500",
                                   ">2500")) +
    labs(colour = NULL,
         title = "Reported Crimes in London, by Location (2012-2013)")

print(dot.hexmap)

![plot of chunk unnamed-chunk-5](./images/r-b6720d/unnamed-chunk-5-1.png)

In fact this is precisely what the original crime map of London does — only with rectangular binning instead of hexagonal binning. Since Cheshire is a self-admitted ggplot2 user himself, I’m assuming the original map used stat_bin_2d.

Note that stat_bin_hex gets a little confused when you use a different geometry, which is why I’ve explicitly set fill = NULL here. You get strange legends otherwise.

Using Custom Binning Functions

But what if we want to do something other than just count up the number of crimes in each hexagon? Suppose that in our data set, each row corresponded to an arbitrary number of crimes of a given type at a given location. In other words, each observation has a weight accompanying it. We can’t really use stat_bin_hex for this, but ggplot provides another, more powerful interface to computing statistics on a hexagonal grid: stat_summary_hex.

The stat_summary_hex statistic expects two new things. First, it requires that we explicitly identify the third dimension our data is organized by, the z aesthetic. And second, it requires us to pass a function that will be used to compute the value at each hexagon in the grid from this z aesthetic. This also means that the stat_bin_hex statistic we used above is equivalent to running stat_summary_hex with fun = sum and a z = 1L aesthetic.

For example, suppose each row in our crime dataset referred to between one and three individual crimes at a given location. We could aggregate into a traditional hexbin map as follows:

crimes$Count <- sample(1:3, nrow(crimes), replace = TRUE)
summary.hexmap <- basemap +
    stat_summary_hex(aes(x = Longitude, y = Latitude, z = Count,
                         fill = cut(..value..,
                                    c(0, 100, 250, 500, 1000,
                                      1500, 2000, 2500, Inf))),
                     fun = sum,
                     colour = NA, bins = 30, alpha = 0.75,
                     data = crimes) +
    scale_fill_brewer(palette = "OrRd",
                      labels = c("<100", "100-250", "250-500",
                                 "500-1000", "1000-1500",
                                 "1500-2000", "2000-2500",
                                 ">2500")) +
    labs(fill = NULL,
         title = "Reported Crimes in London, by Location (2012-2013)")

print(summary.hexmap)

![plot of chunk unnamed-chunk-6](./images/r-b6720d/unnamed-chunk-6-1.png)

When `stat_summary_hex` Fails, Use `hexbin` Directly

There some limitations to stat_summary_hex. For instance, suppose we were just interested in non-drug crimes. The intuitive approach will fail:

broken.hexmap <- basemap +
    stat_summary_hex(aes(x = Longitude, y = Latitude, z = Crime,
                         fill = cut(..value..,
                                    c(0, 100, 250, 500, 1000,
                                      1500, 2000, 2500, Inf))),
                     fun = function(z) sum(z != "Drugs"),
                     colour = NA, bins = 30, alpha = 0.75,
                     data = crimes) +
    scale_fill_brewer(palette = "OrRd",
                      labels = c("<100", "100-250", "250-500",
                                 "500-1000", "1000-1500",
                                 "1500-2000", "2000-2500",
                                 ">2500")) +
    labs(fill = NULL,
         title = "Reported Crimes in London, by Location (2012-2013)")

print(broken.hexmap)

![plot of chunk unnamed-chunk-7](./images/r-b6720d/unnamed-chunk-7-1.png)

What’s going on here? It turns out that ggplot’s implementation (which you can see here) uses tapply, so it actually creates one geometry for each level of our Crime factor. You can see this in that the alpha setting seems to be ignored. In fact if you look closely you can see the layer below in some places.

Thankfully, ggplot is flexible enough for us to completely circumvent this problem. Internally, geom_hex calls out to the hexbin function through stat = "hex" and then draws hexagons according to the resulting data frame. We can work around this by computing the intermediate statistic ourselves, and then pass stat = "identity" to geom_hex instead (this is similar to how you would disable geom_bar’s binning if you were starting out with aggregate data).

So what we really need is to join our data set with a column indicating which hexagon a given row would fall in. That would allow us to compute arbitrary contents for the hexagons from the original data set. Thankfully, the hexbin function provides a straightforward way of doing this by passing IDs = TRUE (otherwise the cID slot is optional, since it can be very large).

The hexbin function itself returns an S4 object with a number of other interesting fields, chiefly those that can be used to create a data frame containing the hexagon coordinates.

crimes.hex <- hexbin::hexbin(crimes$Longitude, crimes$Latitude,
                           xbins = 30, IDs = TRUE)

# Deconstruct the relevant hexbin information into a dataset:
hexagons <- data.frame(hexbin::hcell2xy(crimes.hex),
                       cell = crimes.hex@cell,
                       count = crimes.hex@count)

crimes$cell = crimes.hex@cID
nondrug.crime.hexagons <- crimes %>%
    filter(Crime != "Drugs") %>%
    group_by(cell) %>%
    summarise(crimes = n()) %>%
    ungroup() %>%
    # Cut here, instead of in the ggplot call (as above).
    mutate(crime.level = cut(crimes, c(0, 100, 250, 500,
                                       1000, 1500, 2000,
                                       2500, Inf),
                             labels = c("<100", "100-250",
                                        "250-500", "500-1000",
                                        "1000-1500", "1500-2000",
                                        "2000-2500", ">2500"))) %>%
    right_join(hexagons, by = "cell") %>%
    select(cell, x, y, count, crimes, crime.level)

head(nondrug.crime.hexagons, 10)

## # A tibble: 10 x 6
##     cell          x        y count crimes crime.level
##    <int>      <dbl>    <dbl> <int>  <int>      <fctr>
## 1      1 -0.2499860 51.40001    77     77        <100
## 2      2 -0.2416534 51.40001   117    116     100-250
## 3      3 -0.2333209 51.40001    54     50        <100
## 4      4 -0.2249883 51.40001    26     26        <100
## 5      5 -0.2166557 51.40001    29     29        <100
## 6      6 -0.2083232 51.40001    87     86        <100
## 7      7 -0.1999906 51.40001   136    132     100-250
## 8      8 -0.1916580 51.40001   336    318     250-500
## 9      9 -0.1833255 51.40001    23     23        <100
## 10    10 -0.1749929 51.40001   102    100        <100

We can now plot the nondrug.crime.hexagons data frame directly, using geom_hex.

manual.hexmap <- basemap +
    geom_hex(aes(x = x, y = y, fill = crime.level),
             stat = "identity", colour = NA, alpha = 0.75,
             data = nondrug.crime.hexagons) +
    scale_fill_brewer(palette = "OrRd") +
    labs(fill = NULL,
         title = "Non-drug Crimes in London, by Location (2012-2013)")

print(manual.hexmap)

![plot of chunk unnamed-chunk-9](./images/r-b6720d/unnamed-chunk-9-1.png)

Using this approach it is possible to compute arbitrary data for each hex bin.

Postscript: Cleaning up the Crime Data

This snippet of code should extract all of the relevant data from the raw, unzipped data extract:

library(dplyr, warn.conflicts = FALSE)

# Point R at the unzipped files:
setwd("~/Downloads/b9490c7923e00e0e56ac3d2aa9f2b2a968935b15.zip Folder/")

csv.files <- data_frame(
    month = c(5:12, 1:4),
    year = c(rep(2012L, times = 8), rep(2013L, times = 4))
) %>% mutate(
    fmt = paste0(year, ifelse(month < 10, "-0", "-"), month),
    file = paste0(fmt, "/", fmt, "-metropolitan-street.csv")
) %>% apply(1, function(row) {
    cat("Reading file:", row['file'], "\n")
    raw.data <- read.csv(
        row['file'], stringsAsFactors = FALSE, na.strings = "",
        colClasses = c("NULL", "NULL", "NULL", "NULL", "numeric", "numeric",
                       "NULL", "NULL", "NULL", "factor", "NULL", "NULL")
    ) %>% mutate(Year = as.integer(row['year']),
                 Month = as.integer(row['month'])) %>%
        select(Year, Month, Longitude, Latitude, Crime = Crime.type)
    raw.data
})

crimes <- do.call(rbind, csv.files) %>% tbl_df()

save(crimes, file = "~/Code/R/data/london-crimes/crimes.rda",
     compress = "xz")

For reference, the following is the output of my sessionInfo() at the time of writing:

## R version 3.3.1 (2016-06-21)
## Platform: x86_64-apple-darwin14.3.0 (64-bit)
## Running under: OS X 10.10.3 (Yosemite)
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] methods   stats     graphics  grDevices utils     datasets  base     
## 
## other attached packages:
## [1] ggmap_2.6.1   ggplot2_2.1.0 hexbin_1.27.1 dplyr_0.5.0   stringr_1.0.0
## [6] digest_0.6.10 knitr_1.13   
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_0.12.6         magrittr_1.5        maps_3.1.1         
##  [4] munsell_0.4.3       colorspace_1.2-6    geosphere_1.5-5    
##  [7] lattice_0.20-33     rjson_0.2.15        R6_2.1.2           
## [10] jpeg_0.1-8          plyr_1.8.4          tools_3.3.1        
## [13] grid_3.3.1          gtable_0.2.0        png_0.1-7          
## [16] DBI_0.5             assertthat_0.1      tibble_1.1         
## [19] RJSONIO_1.3-0       reshape2_1.4.1      mapproj_1.2-4      
## [22] formatR_1.4         codetools_0.2-14    evaluate_0.9       
## [25] sp_1.2-3            stringi_1.1.1       RgoogleMaps_1.2.0.7
## [28] scales_0.4.0        proto_0.3-10

Since this post relies on certain implementation details of ggplot2 and hexbin, it may break with future versions of either library.

Linting Prose in Emacs

2016-02-10T00:00:00Z

Recently, I came across the Proselint project, which aspires to be something like a “linter” for written English prose. It differs from spelling or grammar checker in that it is primarily focused on logic and style, taking its advice from the work of many prominent writers on writing.

The project seems to be a little rough around the edges (code blocks seem to make it struggle, for example), but it’s also perfectly usable in its current form. It’s available as a Python package, so you can pip install proselint to get the command-line tool, proselint.

Thanks to the Flycheck framework it’s comically easy to get this imported into a Emacs writing environment. The following snippet will get you up and running in text and markdown modes:

(flycheck-define-checker proselint
  "A linter for prose."
  :command ("proselint" source-inplace)
  :error-patterns
  ((warning line-start (file-name) ":" line ":" column ": "
	    (id (one-or-more (not (any " "))))
	    (message) line-end))
  :modes (text-mode markdown-mode gfm-mode))

(add-to-list 'flycheck-checkers 'proselint)

Proselint won’t focus on checking your grammar and spelling. But it will point out that referring to my very unique sense of humour commits both a subtle language error, and a not-so-subtle error of style:

Every lint in Proselint is named after the authority to whom it appeals: in these two examples, Mark Twain and David Foster Wallace. While my sentence is grammatically correct, these two authors would take issue with it. Wallace points out that “very unique” doesn’t really make sense — a thing is either unique or not. And Twain seems to take issue with the usage of “very” in general.

If this kind of feedback appeals to you, Proselint might be worth a try. Given the simplicity of the Flycheck-based approach, I don’t think there’s a need for a seperate proselint.el package at the moment, but perhaps someone should submit a small one to the Proselint project itself, which appears to accept plugins.

I’ve also heard of the writegood-mode and artbollocks-mode packages, which address similar issues.

Expressing L-systems in Rust

2015-06-06T00:00:00Z

I thought I’d jump on the Rust blogging bandwagon. Specifically, I’ve been toying with a new library to define and iterate over L-systems in a unique and Rust-inspired fashion. It’s been very rewarding to fool around with a type system so foreign to my roots in duck- and weakly-typed languages.

If you haven’t heard of L-systems before, they’re mostly used to make pretty pictures. I was able to create a little animation of the first seven iterations of an L-system that draws Penrose tiles (below).

![Penrose Tile L-system](/images/penrose.gif)

L-systems have put them to a variety of different uses, including modelling plants and trees, buildings and cities, and all manner of interesting geometric constructions. They hold their fascination in part because of the way in which a very small number of basic components and rules can evolve in startling ways over time. In other words, L-systems put emergence on display.

Leveraging Rust’s Syntax for Implementation

Formally, an L-system consists of three things:

An alphabet of symbols,
An intial starting point (called an axiom) composed of letters of this alphabet; and
A set of rules (called productions) for transforming sets of letters into one another.

I thought it might be neat to see what it looked like if one were to use a type as the alphabet, and pattern matching (inside of a closure) to express the productions. It turns out that Rust’s syntax makes this is a nice way to build up not only the kinds of L-systems I defined above, but also more advanced parametric and stochastic L-systems as well.

As an L-system evolves over time, each successive iteration uses the productions to replace the symbols in the axiom with new symbols, and the result then becomes the new axiom for the next iteration. The concept of iteration in L-systems also suggests a natural expression in Rust: an iterator type, which responds to calls to next() by internally mutating a vector according to a production function.

This is the basic outline of the implementation: start with a vector of some arbitrary type, use pattern matching to express productions, and then allow iteration over the vector to produce successive axioms.

Lindenmayer’s Algae

The original application that inspired the invention of L-systems is modelling the growth of algae cells. According to Lindenmayer, these cells are in one of two distinct states, reproduction (usually denoted “A”) or growth (usually denoted “B”). In Rust, we can represent these states with a type, specifically an enum:

#[deriving(Clone, Show, Eq, PartialEq)]
enum Algae {
    /** Reproduction State */ A,
    /** Growth State */ B
}

In order to capture how a group of algae cells grows and changes over time, Lindenmayer proposed that algae cells that are in the growth state transition to being in the reproductive state after a set period. In this same period, a cell in the reproductive state will birth a new cell, which starts in the growth state. These two processes can be represented using a simple function that makes use of Rust’s match construct:

fn algae_rule(input: Algae) -> Vec<Algae> {
    match input {
        Algae::A => vec!(Algae::A, Algae::B),
        Algae::B => vec!(Algae::A)
    }
}

If we take the Algae type as representing the L-system’s alphabet, these two pieces are all that we need to define an L-system:

use lsystem::LSystem;

let algae = LSystem::new(vec!(Algae::B), |input| match input {
    Algae::A => vec!(Algae::A, Algae::B),
    Algae::B => vec!(Algae::A)
});

Although this time I’ve used a closure instead of referencing the algae_rule function directly. In the current version of the library, the LSystem type itself is an Iterator, which means we can capture the notion of iteration as it pertains to L-systems in a very idiomatic way. For instance, to check that the n = 4 case matches Lindenmayer’s “ABAAB” string:

let algae_n4 = algae.nth(4).unwrap();

// Confirm that it matches Lindenmayer's fifth iteration.
assert_eq!(algae_n4,
           vec!(Algae::A, Algae::B, Algae::A, Algae::A, Algae::B));

Of course, you could also use some form of for n in algae { ... }, but keep in mind that this is an infinite loop when dealing with LSystem, so you’d have to break out of it somehow.

An Implementation

It actually took me quite a while to hammer out an exact implementation for these ideas — all of the backpedalling in the commit history is a testament to this. But it ended up being quite simple now that Rust has the FnMut trait:

pub struct LSystem<T, F: FnMut(T) -> Vec<T>> {
    axiom: Vec<T>,
    rules: F
}

impl<T, F> LSystem<T, F> where F: FnMut(T) -> Vec<T> {
    /// Creates a new representation of an L-system with the given axiom and
    /// production rules.
    pub fn new(axiom: Vec<T>, rules: F) -> LSystem<T, F> {
        LSystem { axiom: axiom, rules: rules }
    }
}

The closure (or fn) is actually called during iteration. I experimented with a number of ways to turn an LSystem into a iterator, including a top-level function and an LSystemIterator that was created with a call to an iter() method. In the end, I decided just to implement the Iterator trait for the key struct directly:

impl<T, F> Iterator for LSystem<T, F> where T: Clone, F: FnMut(T) -> Vec<T> {
    type Item = Vec<T>;

    /// Yield the next iteration of the L-system by rewriting the current
    /// axiom's contents using the productions.
    fn next(&mut self) -> Option<Vec<T>> {
        // To handle the "n = 0" case, we return the current state of the
        // iterator by value, and then rewrite the internal vector in
        // preparation of the next state.
        let result = self.axiom.clone();

        // Apply the production rules to the axiom to produce a new axiom for
        // the iteration level.
        let mut new_axiom = Vec::new();
        for element in self.axiom.drain(..) {
            new_axiom.extend((self.rules)(element).into_iter());
        }
        self.axiom = new_axiom;

        Some(result)
    }
}

Update: Thanks to u/matthiem for some nice improvements over the original implementation.

The “Turtle” Interpretation of L-systems

You might be asking yourself “what do I do with an L-system?” at this point. That is, what does the vector of symbols that each interation produces actually mean? In Lindenmayer’s algae model, we could look at the vector of symbols and see it as a culture of cells, but that doesn’t hold for other examples.

If you followed any of the links above, you might have noticed that L-systems are often associated with turtle graphics. The essential idea is that a string of symbols can be sequentially “interpreted” as a series of drawing commands¹.

Once again, this has a very nice representation in Rust:

pub mod turtle {
	// Note that this is only a small subset of more complete Turtle graphics
	// implementations.
	pub enum Turtle {
	    Forward(u32), Left(u32), Right(u32), Push, Pop
	}

	pub trait TurtleInterpretation {
	    fn to_turtle(&self) -> Turtle;
	}
}

That is, the Turtle algebraic type represents turtle graphics commands, and any type can have a “turtle interpretation” by implementing a simple trait.

For example, we can create a version of the Sierpinski triangle with the following:

extern crate lsystem;

use lsystem::LSystem;
use lsystem::turtle::{Turtle, TurtleInterpretation};

fn draw<T: TurtleInterpretation>(v: Vec<T>) { /* see below */ }

#[derive(Clone)]
enum Sierpinski {
	A, B, Plus, Minus
}

impl TurtleInterpretation for Sierpinski {
	fn to_turtle(&self) -> Turtle {
		match *self {
			Sierpinski::A     => Turtle::Forward(10),
			Sierpinski::B     => Turtle::Forward(10),
			Sierpinski::Plus  => Turtle::Left(60),
			Sierpinski::Minus => Turtle::Right(60)
		}
	}
}

fn main() {
	let mut s = LSystem::new(vec!(Sierpinski::A), |x| match x {
		// A -> B-A-B
		Sierpinski::A => vec!(Sierpinski::B, Sierpinski::Minus, Sierpinski::A,
		                      Sierpinski::Minus, Sierpinski::B),
		// B -> A+B+A
		Sierpinski::B => vec!(Sierpinski::A, Sierpinski::Plus, Sierpinski::B,
		                      Sierpinski::Plus, Sierpinski::A),
		// + and - are constants.
		c => vec!(c)
	});

	draw(s.nth(3).unwrap());
}

The last piece of code we need is a function that will take a Vec<T: Turtle> and draw() it to the screen. I didn’t really want to write any graphics code, so here’s a way to cheat:

fn draw<T: TurtleInterpretation>(v: Vec<T>) {
    println!("import turtle\n\nturtle.speed(0)\n");

    for command in v.iter() {
        match command.to_turtle() {
            Turtle::Forward(val) => println!("turtle.forward({})", val),
            Turtle::Left(val)    => println!("turtle.left({})", val),
            Turtle::Right(val)   => println!("turtle.right({})", val),
            _ => ()
        }
    }

    println!("\nturtle.exitonclick()\n");
}

It turns out that there’s a turtle graphics implementation distributed with Python — so this little draw() function will write a complete Python program to draw your L-system to stdin (which you can pipe to python). I usually just run something like the following in my shell to test them:

$ ./sierpinski | python -

Sneaky, eh? There’s a much, much more complete version of such a draw() function included in the script I used to create the Penrose tiles above, which can be found in the examples directory of the project. All of the heavy lifting for creating the .gif itself is done by ImageMagick’s convert utility, because I don’t think there’s a Rust image library that will handle Postscript files at the moment.

A possible improvement to the library in the future might be output to the new turtle graphics implementation in Rust, which was posted to Reddit a few days ago.

Comments on the implementation details are very welcome! There were also some good comments on the Reddit thread.

L-systems are often expressed only in this form. This is a bit of a mistake, as the algae example illustrates. ↩︎

Not All Population Maps Are Boring

2014-12-02T00:00:00Z

As the venerable xkcd comic points out, unadjusted geographic data often ends up looking like a population map. Normally, this makes it kind of boring, since it doesn’t tell you anything new.

Except, of course, when a visualizing the population is exactly what you had in mind. I discovered recently that the Australian government keeps track of every public toilet in the country, for example — and what better way is there to learn about Australian geography than through such an important public utility?

![Australia, In Public Toilets](/images/australia-in-public-toilets.png)

As usual, the remainder of this post is a technical discussion of how I created the graphic above. It was a neat opportunity to make use of hexbinning, and I’m quite fond of the end result. I haven’t yet figured out a way to add a “shadow” made of hexagons to indicate the overall shape of Australia, though I would like to. The fully reproducible code can be found in my visualization repository on Github.

Using Hex Bins in ggplot2

You can grab the data on Australia toilets here. Each entry in the .csv file contains quite a lot of other interesting information, but for now I’m most interested in the Longitude and Latitude fields. I’ve used the brilliant dplyr package to rearrange and select what I need, as follows:

require(dplyr, warn.conflicts = FALSE, quiet = TRUE)

toilets <-
    read.csv("~/Code/R/data/toiletmap.csv",
             header = TRUE, stringsAsFactors = FALSE) %>%
    mutate(long = Longitude, lat = Latitude) %>%
    select(ToiletID, long, lat)

The hex binning features of ggplot2 rely on the hexbin package, so you may need to run install.packages("hexbin") before the following code will work. But it’s actually quite simple to get them working:

require(ggplot2, quiet = TRUE)

p <-
    ggplot(toilets, aes(x = long, y = lat)) +
    geom_hex() +
    coord_equal()

print(p)

![plot of chunk toilet-hexbin](images/r-b27dc3c2/toilet-hexbin.png)

This looks pretty good. But it can be reasonaby difficult to compare different places on a continuous scale, so I decided to break the bins in levels. A very useful base R function for me here was cut(), which does exactly this: it breaks a continuous variable into a factor. I chose seven levels, because I think that’s about the most one can digest really easily. To do this, I made use of stat_binhex as opposed to geom_hex (much as you might use stat_density instead of geom_density when you want more control). It’s easy to add a nicer colour scale as well using the RColorBrewer palettes, too.

p <-
    ggplot(toilets, aes(x = long, y = lat)) +
    stat_binhex(
        colour = NA,
        aes(fill = cut(..count.., c(0, 5, 10, 50, 100,
                                    500, 1000, Inf)))
    ) +
    coord_equal() +
    labs(fill = NULL) +
    scale_fill_brewer(
        palette = "OrRd",
        labels = c("<5 ", "5-9 ", "10-49 ", "50-99 ",
                   "100-499 ", "500-999 ", "1000+  ")
    )

print(p)

![plot of chunk toilet-hexbin2](images/r-b27dc3c2/toilet-hexbin2.png)

Adding Points of Interest

Not surprisingly, the highest concentrations of toilets are found in the largest urban areas. I thought the map might be more interesting if I layered over the labels for these cities, so I looked up the list of the largest cities in Australia (via Wikipedia) and then extracted the longitude and latitude information from the site’s links to GeoHack. This gave me the following data frame:

poi <- data.frame(
    name = c("Perth", "Sydney", "Melbourne", "Brisbane", "Adelaide",
             "Alice\nSprings", "Darwin", "Hobart\n(Tasmania)"),
    long = c(115.858889, 151.209444, 144.963056, 153.027778, 138.601, 133.87,
             130.833333, 147.325),
    lat  = c(-31.952222, -33.859972, -37.813611, -27.467917, -34.929, -23.7,
             -12.45, -42.880556),
    vjust = c(1, 1, 0, 0.5, 0, 0.5, 0.5, 0),
    hjust = 0.5
)

Which can just be overlaid on the previous ggplot object using geom_text. To get away from the “plot” feeling, I also removed most of the theme elements.

require(grid, quiet = TRUE) # For the unit() function.

p <- p +
    geom_text(aes(x = long, y = lat, label = name, vjust = vjust,
                  hjust = hjust), data = poi, size = 4) +
    theme(panel.background = element_rect(fill = "gray90", colour = NA),
          plot.background = element_rect(fill = "gray90", colour = NA),
          # Remove titles, ticks, gridlines, and borders.
          axis.text = element_blank(),
          axis.title = element_blank(),
          axis.ticks = element_blank(),
          panel.grid = element_blank(),
          panel.border = element_blank(),
          # Set the legend background.
          legend.background = element_rect(fill = NA, colour = NA),
          legend.key = element_rect(fill = NA, colour = NA),
          # Set margins so that the graphic fills the whole space.
          plot.margin = unit(c(0, 0, -0.5, -0.5), "line")
    ) 

print(p)

![plot of chunk toilet-hexbin3](images/r-b27dc3c2/toilet-hexbin3.png)

There were some other minor adjustments I made (to the legend’s position and the transparency of the hexbins, for example), but that’s the gist of my approach. To get a sense of how I incorporated fonts and created the title/subtitle, you can check out the fully reproducible code on Github.

Visualizing Health Expenditure using Spie Charts (and R)

2014-10-11T00:00:00Z

Continuing with the theme of my last post, I wanted to put an interesting visualization together using some publically available data. The result is the following (somewhat surprising) graphic:

![health-spending-infovis](/images/spending-spie-final.png)

One of the looming issues in Canadian public policy is how to address the fact that our population is ageing, and that this will mean a larger burden on many of the social services that are more heavily consumed by those who are older. But just how uneven is the consumption of health services? The above should give you some idea of why this is viewed as a looming problem.

The remainder of this post is a technical discussion of how I created the visualization. I’m not quite satisfied with the overall approach (I think it takes quite a while before you can really read the graphic), but it does serve as a good technical demonstration of what can be accomplished in R.

Creating a “Spie” Chart with ggplot2

A spie chart is really just a way of comparing two pie charts. The width of each slice corresponds to its share by one measure, and its area (or, in some formulations, its height, corresponds to its share by another). I originally found the concept while exploring DataVis. A version of the the paper that introduced the concept is available here.

In order to get the Canadian health spending data into a format suitable for comparison with this chart, we need need to play around with it a bit. CIHI publishes their complete tables for free here, but none of it was in precisely the format that I wanted, so I extracted the relevant information from Table E.1.14 and Appendix C.17 and created the following pop_exp_canada_2011.csv file to work with:

bin,pop,exp,sex
<1,192.49,1737.42,F
1-4,738.24,1117.42,F
5-9,885.96,1091.74,F
10-14,924.04,1200.01,F
15-19,1072.69,1823.61,F
20-24,1169.13,2345.44,F
25-29,1193.84,3176.77,F
30-34,1174.74,3504.91,F
35-39,1140.41,2994.82,F
40-44,1189.54,2860.62,F
45-49,1364.98,3522.20,F
50-54,1334.70,3963.40,F
55-59,1193.13,4113.49,F
60-64,1040.40,4473.18,F
65-69,790.72,4687.53,F
70-74,603.58,4842.44,F
75-79,503.05,5587.23,F
80-84,410.18,6525.53,F
85-89,283.02,7068.76,F
90+,172.18,4698.45,F
<1,201.30,2055.33,M
1-4,777.30,1251.80,M
5-9,939.64,1253.51,M
10-14,975.94,1303.69,M
15-19,1123.67,1656.40,M
20-24,1235.04,1770.19,M
25-29,1229.32,1853.98,M
30-34,1174.23,1898.48,M
35-39,1147.82,2026.09,M
40-44,1204.91,2454.87,M
45-49,1384.03,3280.49,M
50-54,1332.97,3942.38,M
55-59,1161.09,4361.40,M
60-64,999.08,4889.40,M
65-69,744.85,5187.12,M
70-74,539.84,5026.25,M
75-79,416.40,5207.42,M
80-84,294.40,4821.01,M
85-89,158.38,3526.42,M
90+,66.76,1594.15,M

The first thing to do is reorganize the data. We want the data for women to run in the reverse order, so that Women <1 are next to Men <1. This is purely an aesthetic consideration; you can experiment with the spie chart that is produced when skipping this step.

# Data culled form Appendix C.17 and Table E.1.14 of the CIHI report's data
spie.data <- read.csv("~/Code/R/data/pop_exp_canada_2011.csv", header = TRUE)

# (Optional) Change the levels from "F" and "M" to more complete labels:
levels(spie.data$sex) <- c("Women", "Men")

# Reorganize the data (we can skip the factor column "sex", since it does not
# make a difference):
spie.data$bin[1:20] <- rev(spie.data$bin[1:20])
spie.data$exp[1:20] <- rev(spie.data$exp[1:20])
spie.data$pop[1:20] <- rev(spie.data$pop[1:20])

Now we need to convert the population data into shares (percentages). We also need to compute the cumulative population share so that we can properly position the entries along the x-axis. Finally, we calculate the radius of the expenditure for a given slice of the population using the formula from the original paper:

$$ r_{outer}^2 = \frac{outer_i}{\sum outer_i} \frac{inner_i}{\sum inner_i} $$

Which implies an inner radius of 1. (To understand where this number comes from, remember that the area of a slice of a circle is given by (\theta r^2), where (\theta) is the angle.)

# Calculate the population percentage and cumulative percentage for each entry:
spie.data$popshare <- spie.data$pop / sum(spie.data$pop)
spie.data$cumpop <- cumsum(spie.data$popshare)

# And the expenditure share as well
spie.data$expshare <- spie.data$exp / sum(spie.data$exp)

# Compute the radius
spie.data$r_exp <- sqrt(spie.data$expshare / spie.data$popshare)

And now we create the plot itself. A pie chart is really just a bar chart in polar coordinates with the width of the bars equal to the share of the value for that category, so we can start with a bar chart:

require(ggplot2)

bars <-
    ggplot(spie.data, aes(x = cumpop - 0.5 * popshare, fill = sex)) +
    # The "inner" pie plot, with radius 1, is washed out by setting the alpha
    geom_bar(aes(width = popshare, y = 1),
             color = NA, stat = "identity", alpha = 0.2) +
    # The "outer" pie plot uses the radius calculated earlier
    geom_bar(aes(width = popshare, y = expshare / popshare), #y = r_exp),
             color = "grey10", size = 0.1, stat = "identity") +
    # Add a dotted line at the radius = 1 point to emphasize it
    annotate("segment", linetype = 3, size = 0.5, lineend = "round",
             x = -Inf, xend = Inf, y = 1.0, yend = 1.0)

print(bars)

This really illustrates how a spie chart is constructed. The dotted line is the “break-even” point: below it, the expenditure of those groups is less than their share of the population, and above it those groups are consuming a share larger than their representation in the population. Note also that their are in fact two bar charts here, a pale one where all bars of of height 1 and a solid one over top.

To transform this into a spie chart we need to make two adjustments. The first is the conversion to polar coordinates (imagine taking each end of the x-axis and squeezing it to a point), and the second is to rescale the y-axis so that the area (and not the height) of the slices corresponds to the relative shares of the groups. Fortunately, this second step is also quite simple: it boils down to taking the square root of each y entry.

We can do this quite simply using ggplot:

spie <-
    bars +
    coord_polar(theta = "x") +
    scale_y_sqrt() +
    scale_x_continuous(labels = spie.data$bin,
                       breaks = spie.data$cumpop - 0.5 * spie.data$popshare) +
    labs(x = "Age Group and Sex",
         y = "Ratio of Expenditure to Population Size",
         fill = "Sex") +
    theme_bw() +
    theme(axis.text.x = element_text(angle = 0, size = 8))

print(spie)

And with the addition of the labels we’ve created a simple spie chart. Of course it is quite ugly — there are far too many labels for the groups — but this is the starting point.

Creating an “Information Visualization”

Moving away from a statistical graphic to a pretty infographic, or information visualization, requires some modifications to the plot. Traditionally, the advice was to export this from R and into Inkscape or some other drawing program in order to add annotations, nicer titles, and so on. But if you’re like me, and enjoy the idea of creating visualizations purely programmatically, then there are ways to accomplish this tidying up in R alone.

To start with, I’ve used the same approach as before as the base plot, but removed all of the axes and labels. This gives us a base object with which to build up the visualization:

library(grid) # For the units() function
library(ggplot2)

base <-
    ggplot(spie.data, aes(x = cumpop - 0.5 * popshare, y = r_exp, fill = sex)) +
    geom_bar(aes(width = popshare, y = 1),
             color = NA, stat = "identity", alpha = 0.2) +
    geom_bar(aes(width = popshare),
             color = "grey10", size = 0.1, stat = "identity") +
    annotate("segment", linetype = 3, size = 0.25, lineend = "round",
             x = -Inf, xend = Inf, y = sqrt(c(0.5, 1.0)),
             yend = sqrt(c(0.5, 1.0))) +
    coord_polar(theta = "x") +
    scale_x_continuous(labels = spie.data$bin,
                       breaks = spie.data$cumpop - 0.5 * spie.data$popshare) +
    scale_fill_brewer(palette = "Pastel1") +
    labs(x = NULL, y = NULL, title = NULL) +
    theme_bw() +
    theme(panel.grid = element_blank(),
          axis.title = element_blank(),
          axis.text = element_blank(),
          axis.ticks = element_blank(),
          legend.position = "none",
          panel.border = element_rect(colour = NA, fill = NA),
          plot.margin = unit(c(0, 0, -0.5, -0.5), units = "line"))

print(base)

Since I wanted to construct something of a narrative for this information, I decided to annotate some of the interesting features of the plot. When using ggplot2, the annotate function facilitates this. Unfortunately, because the plot is in polar (square root) coordinates, this complicates placement. I just tweaked values by eye instead of doing things systematically. This gives

final <- base +
    # Arrow to the 90+ group, and a text blurb:
    annotate("segment", linetype = 1, size = 0.25, x = 0.94, xend = 0.9925,
             y = sqrt(6.0) - 0.05, yend = sqrt(6.0),
             arrow = arrow(ends = "last", length = unit(0.3, "line"))) +
    annotate("text", size = 2, x = 0.935, y = sqrt(6.0) - 0.05,
             hjust = 1, vjust = 1,
             label = paste("If we take the ratio of the share of public health",
                           "spending to the share of the population for",
                           "five-year age groups, we spend at more",
                           "than 6:1 on those over 90.", sep = "\n")) +
    # Arrow to the 80-84 group, and a text blurb:
    annotate("segment", linetype = 1, size = 0.25, x = 0.03, xend = 0.095,
             y = sqrt(4.0), yend = sqrt(4.0),
             arrow = arrow(ends = "first", length = unit(0.3, "line"))) +
    annotate("text", size = 2, x = 0.1, y = sqrt(4.0), hjust = 0, 
             label = "And more than 4:1 on those over 80.") +
    # Arrow to the 65-69 group, and a text blurb:
    annotate("segment", linetype = 1, size = 0.25, x = 0.855, xend = 0.93,
             y = sqrt(2.0), yend = sqrt(2.0) - 0.1,
             arrow = arrow(ends = "last", length = unit(0.3, "line"))) +
    annotate("text", size = 2, x = 0.85, y = sqrt(2.0), hjust = 1, 
             label = "In fact by 65 the ratio is already 2:1.") +
    # Text blurbs:
    annotate("segment", linetype = 1, size = 0.25, x = 0.31, xend = 0.275,
             y = 1.05, yend = 1.5,
             arrow = arrow(ends = "first", length = unit(0.3, "line"))) +
    annotate("text", size = 2, x = 0.27, y = 1.16, hjust = 0, vjust = 0,
             label = paste("Only below age 55 is the ratio less than 1:1.",
                           "(The break-even point is illustrated by the",
                           "pale inner circle.)", sep = "\n")) +
    # Arrow to the inner circle and text blurb:
    annotate("segment", linetype = 1, size = 0.25, x = 0.65, xend = 0.62,
             y = 1.25, yend = sqrt(0.5) + 0.05,
             arrow = arrow(ends = "last", length = unit(0.3, "line"))) +
    annotate("text", size = 2, x = 0.7, y = 1.2, hjust = 1, vjust = 1,
             label = paste("In fact most of the remainder cost less than",
                           "50% of their share. (Women are more",
                           "expensive in their youth primarily",
                           "because of childbirth.)", sep = "\n")) +
    # Babies text and arrow:
    annotate("segment", linetype = 1, size = 0.25, x = 0.495, xend = 0.455,
             y = 1.53, yend = 1.83,
             arrow = arrow(ends = "first", length = unit(0.3, "line"))) +
    annotate("text", size = 2, x = 0.45, y = 1.8, hjust = 0, vjust = 1,  
             label = "(Babies themselves are expensive, too.)") +
    # Explaining the graph itself:
    annotate("text", size = 2, x = 0.55, y = 2.2, hjust = 1, vjust = 1,  
             fontface = "italic",
             label = paste("Age increases in five-year intervals as you move",
                           "up the graph, with the addition of 90+ and <1",
                           "groups at the top and bottom, respectively.",
                           sep = "\n")) +
    annotate("text", size = 2, x = 0.05, y = 3.0, hjust = 0, vjust = 1,
             fontface = "italic",
             label = "Women are in red, men are in purple.")

print(final)

The other obvious addition is to add a proper title and a caption on the bottom of the image (containing, for example, sources and authorship). A good way of doing this is with the arrangeGrob provided by the gridExtra package:

library(gridExtra)

out <- arrangeGrob(
    final,
    main = textGrob(
        paste("The Pressures of an Ageing Population:",
              "Public Health Spending in Canada"),
        hjust = 0, vjust = 1, x = unit(0.01, "npc"),
        gp = gpar(fontsize = 12, fontface = "bold")),
    sub = textGrob(
        x = unit(0.01, "npc"), y = unit(0.2, "npc"),
        hjust = 0, vjust = 0,
        gp = gpar(fontsize = 7, lineheight = 0.9),
        label = paste(
            "A. Jacobs :: unconj.ca\n",
            "Data: Canadian Institute for Health Information, 2013\n",
            "You may redistribute this graphic under the terms of ",
            "the CC-by-SA license.", sep = ""))
)

print(out)

Fonts

The nature of working with the PNG drivers when producing images is that you are severely limited by the selection of fonts available. One clever way of alleviating this is provided by the showtext and sysfonts packages, which simply draw text objects as polygons and lines instead of text — and thus avoid the font problem. I have not yet got showtext working with knitr, so the following standalone code is used to produce the image at the top of this post. Since the scaling is a little different, I’ve had to tweak the final object slightly. I’ve also added a subtitle. It’s true that Minion and Myriad are a little cliché, but they seemed like the most obvious choices to start with.

require(sysfonts)
require(showtext)

font.add("myriad", regular = "MyriadPro-Regular.otf",
         bold = "MyriadPro-Bold.otf", italic = "MyriadPro-It.otf")
font.add("minion", regular = "MinionPro-Regular.otf",
         bold = "MinionPro-Bold.otf", italic = "MinionPro-It.otf")

final <- base +
    # Arrow to the 90+ group, and a text blurb:
    annotate("segment", linetype = 1, size = 0.25, x = 0.94, xend = 0.9925,
             y = sqrt(6.0) - 0.05, yend = sqrt(6.0),
             arrow = arrow(ends = "last", length = unit(0.3, "line"))) +
    annotate("text", size = 3, x = 0.935, y = sqrt(6.0) - 0.05,
             hjust = 1, vjust = 1, family = "minion",
             label = paste("If we take the ratio of the share of public health",
                           "spending to the share of the population for",
                           "five-year age groups, we spend at more",
                           "than 6:1 on those over 90.", sep = "\n")) +
    # Arrow to the 80-84 group, and a text blurb:
    annotate("segment", linetype = 1, size = 0.25, x = 0.03, xend = 0.095,
             y = sqrt(4.0), yend = sqrt(4.0),
             arrow = arrow(ends = "first", length = unit(0.3, "line"))) +
    annotate("text", size = 3, x = 0.1, y = sqrt(4.0), hjust = 0, 
             family = "minion",
             label = "And more than 4:1 on those over 80.") +
    # Arrow to the 65-69 group, and a text blurb:
    annotate("segment", linetype = 1, size = 0.25, x = 0.855, xend = 0.93,
             y = sqrt(2.0), yend = sqrt(2.0) - 0.1,
             arrow = arrow(ends = "last", length = unit(0.3, "line"))) +
    annotate("text", size = 3, x = 0.85, y = sqrt(2.0), hjust = 1, 
             family = "minion",
             label = "In fact by 65 the ratio is already 2:1.") +
    # Text blurbs:
    annotate("segment", linetype = 1, size = 0.25, x = 0.31, xend = 0.275,
             y = 1.05, yend = 1.5,
             arrow = arrow(ends = "first", length = unit(0.3, "line"))) +
    annotate("text", size = 3, x = 0.27, y = 1.16, hjust = 0, vjust = 0,
             family = "minion",
             label = paste("Only below age 55 is the ratio less than 1:1.",
                           "(The break-even point is illustrated by the",
                           "pale inner circle.)", sep = "\n")) +
    # Arrow to the inner circle and text blurb:
    annotate("segment", linetype = 1, size = 0.25, x = 0.65, xend = 0.62,
             y = 1.25, yend = sqrt(0.5) + 0.05,
             arrow = arrow(ends = "last", length = unit(0.3, "line"))) +
    annotate("text", size = 3, x = 0.7, y = 1.2, hjust = 1, vjust = 1,
             family = "minion",
             label = paste("In fact most of the remainder cost less than",
                           "50% of their share. (Women are more",
                           "expensive in their youth primarily",
                           "because of childbirth.)", sep = "\n")) +
    # Babies text and arrow:
    annotate("segment", linetype = 1, size = 0.25, x = 0.495, xend = 0.455,
             y = 1.53, yend = 1.83,
             arrow = arrow(ends = "first", length = unit(0.3, "line"))) +
    annotate("text", size = 3, x = 0.45, y = 1.8, hjust = 0, vjust = 1,  
             family = "minion",
             label = "(Babies themselves are expensive, too.)") +
    # Explaining the graph itself:
    annotate("text", size = 3, x = 0.55, y = 2.2, hjust = 1, vjust = 1,  
             family = "minion", lineheight = 0.95, fontface = "italic",
             label = paste("Age increases in five-year intervals as you move",
                           "up the graph, with the addition of 90+ and <1",
                           "groups at the top and bottom, respectively.",
                           sep = "\n")) +
    annotate("text", size = 3, x = 0.05, y = 3.0, hjust = 0, vjust = 1,
             family = "minion", lineheight = 0.95, fontface = "italic",
             label = "Women are in red, men are in purple.")

png("output5.png", width = 7 * 300, height = 7 * 300, res = 300)
showtext.begin()
out <- arrangeGrob(
    final + theme(text = element_text(family = "minion", size = 8)),
    main = textGrob(
        label = c(paste("The Pressures of an Ageing Population:",
                        "Public Health Spending in Canada"),
                  paste("The width of the slice corresponds to the population",
                        "size, and its area to the share of government health",
                        "spending devoted to that group.")),
        hjust = c(0, 0), vjust = c(1, 1), x = unit(c(0.01, 0.01), "npc"),
        y = unit(c(0, -2.1), "line"),
        gp = gpar(fontsize = c(15, 8), fontfamily = c("myriad", "minion"),
                  fontface = c("bold", "italic"), lineheight = 0.9)),
    sub = textGrob(
        x = unit(0.01, "npc"), y = unit(0.2, "npc"),
        hjust = 0, vjust = 0,
        gp = gpar(fontsize = 8, fontfamily = "minion", lineheight = 0.9),
        label = paste(
            "A. Jacobs :: unconj.ca\n",
            "Data: Canadian Institute for Health Information, 2013\n",
            "You may redistribute this graphic under the terms of ",
            "the CC-by-SA license.", sep = ""))
)
print(out)
showtext.end()
dev.off()

Choropleth Maps with R and ggplot2

2014-09-05T00:00:00Z

This post is meant to be a short intro on how to create visualizations like the following using R and ggplot2:

![intro-map](/images/r-f380bbe4/final-map-1.png)

Update (February 6, 2017): I’ve updated the content of this post to be much more modern, taking advantage of developments in the spatial package ecosystem and in the capabilities of ggplot2.

To get through this tutorial, you’ll need the following packages:

suppressMessages({
    library(rgeos)
    library(maptools)
    library(ggplot2)
})

The first thing you need to get your hands on is some representation of the polygons on a map. There are many formats for these polygons, but one of the more common ones is the .shp format used by ArcGIS. The maptools package enables opening and manipulating objects created from this format, so you’ll need to get your hands on both that and its prerequisite, rgeos.

Update (February 6, 2017): In the past I recommended gpclib, but the rgeos library is now the preferred approach. It also has a more permissive license.

A nice scale to work at in Canada is the level of the forward sortation areas (FSAs) defined by the first three digits of a postal code. Until very recently the exact borders of these areas was data you had to pay for, but now Statistics Canada makes it available on the Census website, here.

Fair warning: it’s quite a large file, and will take some time to downlaod. Of course, all of the FSAs in Canada is a bit too much to deal with, so I’ll focus just on those that encompass the city of Toronto, which all begin with the letter “M”.

# fsas <- maptools::readShapeSpatial("path/to/gfsa000b11a_e.shp")

toronto.fsas <- fsas[substr(fsas$CFSAUID, 1, 1) == 'M',]

# The shapefile takes up quite a lot of memory, and we no longer need it all,
# so you can delete it if you like:
rm(fsas)

We have to turn the object that maptools produces into something that can be plotted in the ggplot2 paradigm, which reguires a data.frame object. To do this, we can use the ggplot2::fortify() function:

data <- ggplot2::fortify(toronto.fsas, region = "CFSAUID")

# Turn the generated "id" chr column into a correctly-labelled factor variable
data$fsa <- factor(data$id)
data$id <- NULL

Toronto is actually pretty great about making lots of interesting data available online. They provide the number of licensed dogs and cats in the city by FSA here, although it needs to be cleaned up a little before you can use it directly. It needs to be merged into the map data:

# pets <- read.csv("path/to/pets.csv")

# Merge the map data and the pet data into something we'll plot
plot.data <- merge(data, pets, by = "fsa")

Now we can use ggplot2 to plot the polygons, and fill them with a gradient based on the number of dogs. Note that you must group the polygons, otherwise they might not be drawn out in the correct order (try omitting it and see). Another key parameter is the coord_equal() coordinate modification: since we’re dealing with map data, if the scaling of the x and y axis are not the same the map will be arbitrarily distorted.

I’m glossing over the details of map projections here — in this case, the city is small enough that a latitude = longitude presentation will look just fine.

ggplot(plot.data, aes(x = long, y = lat, group = group, fill = dog)) +
    geom_polygon() +
    coord_equal()

![plot of chunk first-map](/images/r-f380bbe4/first-map-1.png)

It’s that easy! Of course, the default ggplot2 theme is really meant for presenting data, and might not look that great with a map. The built-in theme_void() will strip away most of this extraneous framing, although I tend to tweak it a little as well.

Update (February 6, 2017): Bin your spatial data, and use viridis.

I strongly believe that you usually want to bin data for choropleth maps, since it can be very difficult to judge fine colour differences. The ggplot2::cut_number() function will find bins roughly equal in size, which is a good place to start.

Unfortunately, the built-in colours for discrete scales are horrible not only for a final presentation, but also for your own exploratory work. You’re better off using one of the ColorBrewer palettes (available through the built-in scale_fill_brewer() function), and you should also check out the carefully tuned palettes from the viridis package.

Now let’s try this again, with these changes:

library(viridis)

ggplot(plot.data, aes(x = long, y = lat, group = group,
       fill = cut_number(dog, 5))) +
    geom_polygon(color = "grey10", size = 0.2) +
    coord_equal() +
    viridis::scale_fill_viridis(discrete = TRUE) +
    labs(title = "Registered Dogs (Toronto, 2013)",
         fill = NULL) +
    theme_void() +
    theme(legend.position = "bottom",
          panel.background = element_rect(fill = NA, colour = "#cccccc"))

![plot of chunk second-map](/images/r-f380bbe4/second-map-1.png)

Which looks much better. Normally we’d be done here, but the sage advice of Randall Munroe is worth keeping in mind:

So we can improve the meaningfulness of this map by adjusting for population. Luckily, Statistics Canada actually publishes population data at the FSA-level, and even more luckily they provide data on the number of private dwellings that are occupied by their usual owners. This last number is particularly interesting because it is the level at which you would expect pet ownership to occur. You can find the data here.

As with most StatsCan-provided CSV files, you will need to clean it up a bit before using it.

# pop <- read.csv("path/to/fsa_pop.csv")

# We don't really need the rest of this...
pop$population <- NULL
pop$indian <- NULL
pop$dwellings <- NULL

# Merge the data into the dataset we're already using
plot.data <- merge(plot.data, pop, by = "fsa")

# Calculate the density of dogs per private dwelling
plot.data$dog.density <- plot.data$dog / plot.data$dwellings.usual

And now we can plot our population-adjusted map using the theme from before:

dog.map <- ggplot(plot.data, aes(x = long, y = lat, group = group,
                      fill = cut_number(dog.density, 5))) +
    geom_polygon(color = "gray10", size = 0.2) +
    coord_equal() +
    viridis::scale_fill_viridis(discrete = TRUE) +
    labs(title = "Registered Dogs per Private Dwelling (Toronto, 2013)",
         fill = NULL) +
    theme_void() +
    theme(legend.position = "bottom",
          panel.background = element_rect(fill = NA, colour = "#cccccc"))

dog.map

![plot of chunk third-map](/images/r-f380bbe4/third-map-1.png)

Notice that the numbers are quite small. We might want to convert to per-1000 units later.

Style Points: Better Longitude & Latitude Labels

In case you need to explicitly show the coordinates of your spatial data, we can ensure that they are formatted nicely by passing a custom formatting function to the labels option on the x and y scale. These functions must take a vector of inputs and return a vector of labels, and we can write one to handle each of longitude and latitude quite easily:

# Converts a numeric array to an array of latitude labels
latitude <- function(x)
    paste0(format(x, digits = 3), "° ", ifelse(x > 0, "N", "S"))

# Converts a numeric array to an array of longitude labels
longitude <- function(x) 
    paste0(format(x, digits = 3), "° ", ifelse(x > 0, "E", "W"))

dog.map +
    scale_y_continuous(labels = latitude) +
    scale_x_continuous(labels = longitude) +
    theme(axis.text = element_text(size = 8, colour = "#333333"))

![plot of chunk labelled-map](/images/r-f380bbe4/labelled-map-1.png)

Style Points: Adding a Compass Rose

Update (February 6, 2017): Before ggsn existed, I used to do this manually by creating a grob from a vector image using the grImport package. This is no longer necessary.

The ggsn package makes it possible to add scale bars and compass rose symbols to ggplot objects.

library(ggsn)

dog.map +
    ggsn::north(plot.data, location = "bottomright", symbol = 10)

![plot of chunk rose-map](/images/r-f380bbe4/rose-map-1.png)

Style Points: Adding a Drop Shadow

There’s a little trick for adding a drop shadow: we just draw the polygons filled with grey and shifted slightly southeast, and then draw the actual map polygons with the correct fill on top (which I learned here).

ggplot(plot.data, aes(x = long, y = lat, group = group,
                      fill = cut_number(dog.density, 5))) +
    geom_polygon(aes(x = long + 0.005, y = lat - 0.002),
                 color = "grey50", size = 0.2, fill = "grey50") +
    geom_polygon(color = "gray10", size = 0.2) +
    coord_equal() +
    viridis::scale_fill_viridis(discrete = TRUE) +
    labs(title = "Registered Dogs per Private Dwelling (Toronto, 2013)",
         fill = NULL) +
    theme_void() +
    theme(legend.position = "bottom",
          panel.background = element_rect(fill = NA, colour = "#cccccc"))

![plot of chunk shadow-map](/images/r-f380bbe4/shadow-map-1.png)

Finishing Touches: Better Fonts and Nicer Labels

Update (February 6, 2017): As of ggplot2 version 2.2, subtitles and captions are officially supported.

For standalone graphics, it is conventional to point to your data sources somewhere, often below. You can do this fairly easily with the caption label. I also like to add authorship and licensing information.

In order to get “nicer” labels, I’m also going to bin the data manually.

Adopting Arial Narrow as the font-of-the-day, the image at the top of this post can be generated as follows:

plot.data$dogs.per.1000 <- factor(
    cut(plot.data$dog.density * 1000, c(0, 30, 45, 60, 90, 120)),
    labels = c("Under 30", "30 to 44", "45 to 59", "60 to 89", "90 to 120")
)

ggplot(plot.data, aes(x = long, y = lat, group = group, fill = dogs.per.1000)) +
    geom_polygon(aes(x = long + 0.005, y = lat - 0.002),
                 color = "grey50", size = 0.2, fill = "grey50") +
    geom_polygon(color = "grey10", size = 0.2) +
    coord_equal() +
    # Add the compass rose.
    ggsn::north(plot.data, location = "bottomleft", symbol = 10) +
    viridis::scale_fill_viridis(option = "viridis", discrete = TRUE) +
    # Set the labels to NULL so that no space is allocated for them
    labs(x = NULL, y = NULL, fill = NULL,
         title = "Registered Dogs per Toronto Home",
         subtitle = "Density per 1000 Private Dwellings, 2013",
         caption = paste("A. Jacobs :: unconj.ca",
"Data: Toronto Animal Services (2014) & Statistics Canada (2013)",
"You may redistribute this graphic under the terms of the CC-by-SA license.",
            sep = "\n")) +
    theme_void() +
    theme(text = element_text(family = "Arial Narrow", size = 8),
          plot.title = element_text(size = 12, face = "bold"),
          plot.margin = unit(c(0, 0.25, 0.0, 0.25), "in"),
          panel.border = element_rect(fill = NA, colour = "#cccccc"),
          legend.text = element_text(size = 8),
          legend.position = c(0.9, 0.25))

![plot of chunk final-map](/images/r-f380bbe4/final-map-1.png)

Which looks pretty good to me!

Fiddling Around with DEA Models

2014-05-12T00:00:00Z

Measuring performance is a tricky business. Two of the most annoying drawbacks of applying regression analysis (as used in most simple econometric approaches) to these kinds of problems are the requirement to specify a functional form for the production function (which we can wave away by admitting that our results are only a “best linear approximation” that work around the mean) and the inability to measure the impact of inputs on several outputs similtaneously.

These are two problems that data envelopment analysis (DEA) tries to address. In essence, it looks at the relative ability of actors (or “decision-making units”) to turn inputs into outputs, and then measures how far away from the “frontier” of the most efficient actors the rest of the lot are. This avoids making many of the very restrictive assumptions (Cobb-Douglas, anyone?) we usually employ when discussing production functions.

All of this sounded pretty interesting, so I thought I’d try out some of these models. The FEAR package for the R programming language seems like a popular tool to for this kind of analysis; however, I also discovered the newer Benchmarking package, which has the added benefit of being free even for commercial use.

… but mostly it was because FEAR does not yet compile properly on OS X for R vesion 3.1.0+, and I didn’t want to try and patch it. So:

install.packages("Benchmarking")

It wasn’t entirely clear how to actually check that I was computing efficiencies correctly, so I pulled some data out of a textbook¹ that also provided the “answer”:

dmu	input1	input2	output1	output2	ccr
1	0.7150	1.7996	0.7618	1.2239	0.432
2	1.0036	1.2616	1.5441	0.6716	0.709
3	0.0126	0.9276	1.3270	0.3433	1.000
4	0.4339	0.7792	0.3600	0.5075	0.419
5	1.7455	0.2783	1.0560	1.0000	1.000
6	1.5061	0.9276	0.7990	0.5821	0.448
7	0.5313	1.2059	1.1862	1.7612	0.952
8	1.8513	0.7978	1.0885	1.9104	0.865
9	1.9796	1.6698	1.6517	1.1493	0.570
10	0.2211	0.3525	0.2258	0.8507	1.000

And then ran a very simple script to see if I could reproduce the results of a DEA model with constant returns to scale (this is often called the CCR model, hence the name of the last column above):

library(Benchmarking)

test1 = read.table("dea-test1.tsv", header = T)
x = matrix(c(test1$input1, test1$input2), nrow = 10, ncol = 2)
y = matrix(c(test1$output1, test1$output2), nrow = 10, ncol = 2)

result = dea(x, y, RTS = "crs", ORIENTATION = 1)
print(result$eff)

## [1] 0.4319693 0.7091446 1.0000000 0.4194153 1.0000000 0.4485119 0.9519665 0.8653230 0.5701125 1.0000000

Which seems to match up with the final column in the example data nicely.

A few minor details about the script: the dea function provided by the Benchmarking package defaults to much more nuanced assumptions about returns to scale, etc, so you need to set the RTS argument explicitly for simpler models; further, the ORIENTATION = 1 arguments request the computation of output-based efficiencies. I’ve obviously saved the tab-separated data above as dea-test1.tsv.

Specifically, this is from Table 3 in Sarkis, Joseph. 2002. “Preparing Your Data for DEA” in Productivity Analysis in the Service Sector with Data Envelopment Analysis, 2nd Edition, ed. Necmi Avkiran. ↩︎

Managing Complex Research Workflows with Make

2014-04-26T00:00:00Z

If you’re doing any kind of empirical work in Economics, you probably have a huge, messy folder containing a mix of

Data files (.csv, .dta, .xlsx, etc.) in various states of merge-ness and cleanliness.
Scripts for creating graphs & figures, producing summary statistics, and computing models. Probably written for Stata, R, or the Pandas data stack¹.
Files containing written work. These are usually .doc(x) files, but I’ve seen lots of LaTeX lately as well, and being a plain-text format, this is a huge boon to reproducible research.

A really simple research workflow (start with data, make some figures, make some summary statistics, and run some models) might look like the following:

![An Econ Workflow](./images/econ-workflow.png)

But of course that’s not clear when looking at the .zip file you send your coauthor.

(In case anyone is wondering about the file extensions, the .do extensions are for Stata, which is a popular statistics package among economists², and the .R extensions are for R, which is a very popular programming language/statistics package for everyone else. R probably makes the nicer graphs of the two³.)

I have a good friend who deals with all of this by creating a single .do file to run all of the other Stata, R, and pdflatex scripts⁴, and then simply commenting out the sections she doesn’t want to run. I used to do essentially the same thing with a Bash script, and back when I used R exclusively I used knitr (which I’ll discuss a little bit at the end).

The problem with these approaches (despite being somewhat inelegant, except possibly in the case of knitr) is that they don’t really incorporate the notion of dependencies. If you look back at the diagram above, you’ll see that if, say, you modify the script to produce the tables, then everything that relies on the tables must be compiled/re-run. So in this example, table1-data.tex, paper.tex, and paper.pdf — and, importantly, only those things — would need to be updated. That kind of interdependency is very hard to capture in a simple master “build” script.

An alternative that I’ve been toying with recently is to use the venerable Unix tool “make”⁵. Make requires that you write a small file (almost always called Makefile, with no extension) outlining the “targets” your project produces, and the “dependencies” each of them require. The syntax looks like the following:

VARIABLE = 0
# Comment
[target]: [dependency1 dependency2 ...]
	[command to run] $(VARIABLE) [etc]

So for our project above, you might have a simple Makefile containing

# First, clean the data using the cleanup script:
data-cleaned.dta: data.csv cleanup.do
	stata-se -b do "cleanup.do"

# Next, produce the two tables from the cleaned data:
table1-data.tex: data-cleaned.dta sumstats.do
	stata-se -b do "sumstats.do"

table2-data.tex: data-cleaned.dta estimates.do
	stata-se -b do "estimates.do"

# Next, produce the three figures. The '%' character can be used as a wildcard:
f%.pdf: data-cleaned.dta make_figs.R
	Rscript "make_figs.R"

# Finally, produce the paper (which relies on complete figures and tables):
paper.pdf: paper.tex f1.pdf f2.pdf f3.pdf table1-data.tex table2-data.tex
	pdflatex "paper.tex"

As you can see, the structure of “targets” and “dependencies” is very explicit. If we want to produce one (or more) of the targets we can call Make with, for example:

$ make f1.pdf

This will check the rule for producing f1.pdf, which relies on data-cleaned.dta and make_figs.R. Since there is a rule for the former, it will try to produce data-cleaned.dta first, then produce f1.pdf.

What happens if we call make f1.pdf again? Make checks if the file exists (it does, we just created it), and then exits without doing any redundant work.

This is all quite convenient, but hardly worth switching to Make for. But wait! It gets better. Suppose we edit the cleanup.do file. And ask Make to produce f1.pdf again. Make will check if f1.pdf exists. It does (we just made it). Next, it will check if the files that were used to produce f1.pdf have been updated since f1.pdf was created. They were not. Next, Make will check if any of the files used to make those files were updated since f1.pdf. And therein lies the magic of Make’s dependency-based structure: Make will realize that cleanup.do is newer than f1.pdf, and it will re-generate data-cleaned.dta as a result before re-generating f1.pdf, because these are the files that depend on cleanup.do.

By convention, most Makefiles also include an all target to create the “default” output (for us, the final paper), and a clean target that removes all generated files. So in this case we should probably add the following to the top of the file:

all: paper.pdf

clean:
	rm *.pdf table*-data.tex data-cleaned.dta

.PHONY: clean all

There are a couple of neat features of Make on display here⁶. Notice that not all targets require dependencies (in this case, clean), and that not all targets have to specify a command (in this case, all does not). In addition, I’ve added the special .PHONY target, which tells Make that the clean and all targets don’t actually produce files called all and clean.

Now, to check if everything works from scratch.

$ make clean
$ make

When you don’t specify a target for Make, it looks for the first target specified (that doesn’t start with .). Since we added all at the top, it process this target.

More Examples: Reproducible Data Sources

In the first example, I sort of assumed you just started with your data.csv. More often than not, however, you have to make it from a variety of sources, and part of doing reproducible research is documenting your sources in the code. So you might have something that can be visualized with

![A Date Workflow](./images/data-workflow.png)

One way you might encode this is with

# Get first data source from the internet
data-1.dta: csv_to_dta.py
	wget http://www.somedatasource.org/datsets/awesome.csv
	python csv_to_dta.py --output=data-1.dta awesome.csv

# Scrape the second data source using some Python script
data-2.dta: scrape_site.py
	python scrape_site.py --output=data-2.dta "http://www.somedatasource.org/info/results.html"

# Merge the datasets using a Stata script
data-merged.dta: data-1.dta data-2.dta merge.do
	stata-se -b do "merge.do"

Wget is an useful tool for downloading files from the command line, by the way. Also, while you don’t have to write web scrapers (or file format converts) in Python, it’s probably the easiest way to do so.

More Examples: Making Your Project (More) Portable

One problem you might encounter with Makefiles is hard-coding in the programs to compile the scrips. For example, while Stata is called stata-se on my machine, it may be simply stata on your machine. A good way of simplifying the find-and-replace problem this entails is to define a variable at the top of the file, say STATA, that anyone can use to adapt the file to their environment with minimal effort, e.g.

# Change this to the location of your Stata executable:
STATA = stata-se

...

# First, clean the data using the cleanup script:
data-cleaned.dta: data.csv cleanup.do
	$(STATA) -b do "cleanup.do"

# Next, produce the two tables from the cleaned data:
table1-data.tex: data-cleaned.dta sumstats.do
	$(STATA) -b do "sumstats.do"

table2-data.tex: data-cleaned.dta estimates.do
	$(STATA) -b do "estimates.do"

...

And so on with Python, etc. If your research works with the same script over and over again (say, a web scraper), but another researcher might replace just this file (say, replacing scrape.py with fastscrape.py), then you could have something like

PYTHON = python
SCRAPER = fastscrape.py

data-1.dta: $(SCRAPER)
	$(PYTHON) $(SCRAPER) --output=data-1.dta --source="http://www.somedatasoure.org/datasets/awesome1.dta"

data-2.dta: $(SCRAPER)
	$(PYTHON) $(SCRAPER) --output=data-2.dta --source="http://www.somedatasoure.org/datasets/awesome2.dta"

And so on.

A Word About Make vs. Knitr

Knitr is a tool that allows you to embed runnable R code in LaTeX or Markdown files so that the code is run and the figures/tables/etc are produced when the document is compiled (LaTeX to .pdf, Markdown to .html). It’s a widely-used tool for literate programming and research papers. I think this is really neat, but eventually I switched away from it and towards using things like Make because literate programming/knitr is sort of limited to linear approaches, and might not reflect more modular projects. It also tends to hang if part of your project takes a while to run (Matlab simulations, anyone?), which can be a pain in the ass when you need to wait 10 minutes to see what the new paragraph you wrote actually looks like in the final PDF. This is exactly the same problem iPython notebooks have — great for a step-by-step explanation, but not so great for a paper where it might actually take a while for things to run.

It is possible that you are using excel for all of this, but if that’s the case you’re probably not interested in this article to begin with. ↩︎
On a related Stata note, if you’re trying to output tables and regression results from Stata, the user-written esttab package can be quite useful. Running esttab or estout with the booktabs fragment options work very nicely in my opinion. ↩︎
At least in conjunction with ggplot2, anyway. install.packages("ggplot2"), please. ↩︎
Pro Stata tip: you can use shell to exectue unix-style commands from within Stata. So for example to compile a downstream LaTeX document you can use shell pdflatex paper.tex. ↩︎
Make is available on all Unix-like systems (Mac OS X, Linux, BSDs), and there is also a version available for Windows. ↩︎
Also, note that rm is the Unix command for removing files, and the * character serves as a wildcard when using that program. ↩︎

Parsing tape data with Python

2014-02-18T00:00:00Z

Recently I had to use an older dataset that hadn’t been nicely sanitized into something that Stata or Pandas could understand. In particular, I was working with the NHANES I Epidemiologic Follow-up Study¹, which has data available only in the original tape format from the 1980s and early 1990s. An individual record from one of the smaller files looks like this (scroll right for the full effect):

9220809          511     12           0112         996102410232091442411  1 2      9                                                         486 21400520   230031142750215185031486 0                                                                                                                                                           03 42750486 051850

Where every number might be a single variable or part of a multi-column variable. For example, the first two digits are the year the individual died, followed by a five-digit identification number. In other words, it’s impossible to decipher without digging into the original documentation. This kind of data is very hard to read into a statistics package (or Excel, for the less ambitious) in this format — but never to fear! It turns out to be quite simple to parse these files in Python and output something that more high-level tools can understand.

For reference, I’m going to be using the public data files and documentation provided by the Centers for Disease Control on their website. In particular, I’ll use the data from the 1992 NHEFS Mortality survey and its accompanying documentation.

Extracting a simple variable

What we really want to do is convert the column(s) that correspond to a particular variable into something that we can move around in Python (or output so we can use it with another program). The documentation for the education field, for example, looks like this::

   Tape  Field
   Pos.  Size   Variable Description and Codes

52-53      2    Education

                  00        ...  No formal education
                  01-08     ...  Years of elementary school
                  09        ...  1 year of high school
                  10        ...  2 years of high school
                  11        ...  3 years of high school
                  12        ...  4 years of high school
                  13        ...  1 year of college
                  14        ...  2 years of college
                  15        ...  3 years of college
                  16        ...  4 years of college
                  17        ...  5 or more years of college
                  99        ...  Not stated

                  (Note: This information is only available for deaths
                  occurring in 1989 and later.  For earlier years, this
                  field will be blank.)

Parsing a line with this kind of documentation in Python is really quite easy if you make use of slicing. It’s likely that you don’t need to differentiate between ‘Not stated’ and ‘missing due to being before 1989’, so the blank and 99 entries are the same from your perspective. Since a CSV file usually represents missing data with an empty string, we can extract the education variable from a line in the tape with:

educ = line[51:53]
if educ == '  ' or educ == '99':
    educ = ''
elif educ == '00':
    educ = '0'
else:
    educ = educ.lstrip('0')

Although there are of other ways to accomplish this:

# If you need to remove the leading zeros and don't mind converting str -> int -> str
educ = line[51:53]
educ = str(int(educ)) if educ != '  ' and educ != '99' else ''

# Using `replace`; leaves in the leading zeros
educ = line[51:53].replace(' ', '').replace('99', '')

Another simple example might be the sex variable:

   Tape  Field
   Pos.  Size   Variable Description and Codes

   59      1    Sex

                  1         ...  Male
                  2         ...  Female

which can be extracted from a line and converted to a dummy variable for female participants with

# Represent with 0 = male, 1 = female
female = int(line[58]) - 1

Converting categorical variables to dummy variables

Often it may be of interest to convert a categorical variable like race:

   Tape  Field
   Pos.  Size   Variable Description and Codes

   62      1    Race Recode #1 (*)

                  1         ...  White
                  2         ...  Races other than white or black
                  3         ...  Black

Into a series of dummy variables (i.e. white, black, other_race). This is also fairly easy to do with some ternary operators:

white = 1 if line[61] == '1' else 0
black = 1 if line[61] == '3' else 0
other_race = 1 if line[61] == '2' else 0

As a slightly more complex example, consider the entries for marital status:

   Tape  Field
   Pos.  Size   Variable Description and Codes

   77      1    Marital Status

                  1         ...  Never married, single
                  2         ...  Married
                  3         ...  Divorced
                  4         ...  Widowed
                  8         ...  Marital status not on certificate
                  9         ...  Marital status not stated

We might want the dummies for married, widowed, divorced, and single to be empty (as opposed to zero) when we have either of the last two conditions.

if line[76] == '8' or line[76] == '9':
    married = single = divorced = widowed = ''
else:
    single = '1' if line[76] == '1' else '0'
    married = '1' if line[76] == '2' else '0'
    divorced = '1' if line[76] == '3' else '0'
    widowed = '1' if line[76] == '4' else '0'

The full example

If we stuff all of this into a single function that will extract variables from a line, it might look something like the following:

def convert_record_to_csv(line):
    # Identifier used by the survey
    seqnum = line[2:7].lstrip('0')
    
    # Year and month of death (day is not always provided)
    y_death = '19%s' % line[:2]
    m_death = line[54:56]
    
    # Sex
    female = int(line[58]) - 1

    # Marriage
    if line[76] == '8' or line[76] == '9':
        married = single = divorced = widowed = ''
    else:
        single = '1' if line[76] == '1' else '0'
        married = '1' if line[76] == '2' else '0'
        divorced = '1' if line[76] == '3' else '0'
        widowed = '1' if line[76] == '4' else '0'

    # Education
    educ = line[51:53]
    educ = educ if educ != '  ' and educ != '99' else ''

    # Autopsy
    autopsy = '1' if line[83] == '1' else '0'
    autopsy = autopsy if line[83] != '8' and line[83] != '9' else ''

    # Race
    white = 1 if line[61] == '1' else 0
    black = 1 if line[61] == '3' else 0
    other_race = 1 if line[61] == '2' else 0

    return ', '.join([str(x) for x in (seqnum, y_death, m_death, female, educ, autopsy, white, black, other_race, single, married, divorced, widowed)]) + '\n'

We can then write a pretty simple script to convert the raw tape data (provided by the CDC as N92mort.txt) into a .csv file:

with open('N92mort.txt', 'r') as source, open('out.csv', 'w') as dest:
    
    # Write column names, as per CSV convention
    dest.write('seqnum, y_death, m_death, female, educ, autopsy, white, black, other_race, single, married, divorced, widowed\n')
    
    # Loop over each line in the source file
    for i, line in enumerate(source):
        try:
            dest.write(convert_record_to_csv(line))
        except Exception as e:
            # It may be helpful for debugging to know which line the parser failed at
            print 'Failed at record %d.' % i
            raise e

Of course, I’m not parsing every single variable in the data set (there are over 400 columns in the mortality data alone, and more than 4000 in the interview data), but in principle this same method could be used.

National Center for Health Statistics. Data File Documentation, NHANES I Epidemiologic Followup Study (NHEFS), National Center for Health Statistics. Hyattsville, Maryland. ↩︎

Moving to Pelican

2014-02-12T00:00:00Z

I recently moved the site over to Pelican. Although I liked Octopress, it wasn’t working exactly like I wanted it to, and I’m not comfortable enough working with Ruby to modify it. Since Python has become my go-to language for most things these days, it made sense to move to what seems to be the most popular static site generator written in that language. For posteriority’s sake, the following is a guide based on the steps I took to get it working. I’ve also included most of my configuration files and a short program for creating new posts.

Installation

First, you will need to install Python. Then, you will need pip, which is used to install third-party Python programs and modules. Once you have both of these, you can run:

$ pip install pelican

from a terminal. This shouldn’t take too long, since Pelican is written in pure Python and has few dependencies. Now, navigate to where you’d like to put your blog (say, the directory myblog) and run::

$ pelican-quickstart

This will ask you several questions. Generally, if you don’t understand the question, just answer n.

Configuration & Theme

My configuration file at the moment looks something like the following

AUTHOR = u'Aaron Jacobs'
AUTHOR_EMAIL = 'atheriel@gmail.com'
SITENAME = u'Unconstant Conjunction'
SITEURL = ''
TAGLINE = u'most python and pcg'
DEFAULT_CATEGORY = 'Uncategorized'
DEFAULT_LANG = u'en'
SUMMARY_MAX_LENGTH = 120

# Feed generation is usually not desired when developing
FEED_ALL_ATOM = None
CATEGORY_FEED_ATOM = None
TRANSLATION_FEED_ATOM = None

# Social widget
SOCIAL = (('github', 'https://github.com/atheriel'),)

DEFAULT_PAGINATION = 5

# Uncomment following line if you want document-relative URLs when developing
RELATIVE_URLS = True

# Theme -- Pure Single
THEME = 'pure-single'
CSS_FILE = 'pure.css'
COVER_IMG_URL = 'http://dnqgz544uhbo8.cloudfront.net/_/fp/img/home/11.WVgzJf1Jz5x9uuR7KjWjXw.jpg'
PROFILE_IMG_URL = 'https://1.gravatar.com/avatar/eee1ad9206b3858a0ff49ade9983d8f4?s=128'
MENUITEMS = [
    ('About', '/pages/about-me.html'),
    ('Archives', 'archives.html')
]

I’m using a modified version of the Pure theme, which I’ve placed in the pure-single directory. Note that you’ll need to fill in the SITEURL with your own domain.

Creating New Posts

Unlike Octopress, Pelican doesn’t come with a prepackaged solution to creating new posts and pages. To remedy this, I whipped up a small program to do so, which I’ve also published as a gist. The script expects to reside in the root directory of your blog; the code should be pretty self-explanatory if you want to modify this, however. Notice that the configuration file above contains an “About Me” page – you can create one for yourself with:

$ python newpost.py 'About Me' --page

The full script is as follows. Note that it uses Docopt instead of the hell that is argparse, which you may need to install beforehand.

#!/usr/bin/env python

__doc__ = '''
Create a new post/page.

Usage:
  newpost.py <title> [--page] [-t <tag>...]
  newpost.py -h | --help | --version

Options:
  --page           Create a 'page' as opposed to a 'post'.
  -t, --tags       Specify the tags for the page/post.
  -v, --version    Show version information and exit.
  -h, --help       Show this help and exit.
'''

template = '''{title}
{title_underline}

:date: {date}
:tags: {tags}
:author: {author}
'''

import os
import sys
import os.path
import datetime 
from docopt import docopt

import pelicanconf  # To get the blog author

def main():
    args = docopt(__doc__)
    today = datetime.date.today()
    
    # Locate the blog directory and check that `content` exists
    cwd = os.path.dirname(os.path.realpath(__file__))
    assert os.path.exists(cwd + '/content')
    
    # Generate a file from title + date in the blog's content directory
    if not args['--page']:
        filename = os.path.join(cwd, 'content/{0}-{1}.rst'.format(today, args['<title>'].lower().replace(' ', '-').replace('.', '')))
    else:
        if not os.path.exists(cwd + '/content/pages'):
            os.mkdir(cwd + '/content/pages')
        filename = os.path.join(cwd, 'content/pages/{0}.rst'.format(args['<title>'].lower().replace(' ', '-').replace('.', '')))
    
    if os.path.exists(filename):
        print('Error: A file for that title already exists.')
        sys.exit(1)

    with open(filename, 'w') as f:
        f.write(template.format(
            title = args['<title>'],
            title_underline = '#' * len(args['<title>']),
            tags = ', '.join([t.lower() for t in args['<tag>']]) if args['<tag>'] is not None else '',
            date = today,
            author = pelicanconf.AUTHOR,
        ))

if __name__ == '__main__':
    main()

Writing a Web Scraper

2014-01-27T00:00:00Z

Okay, so I know everyone’s written some kind of web scraper in their time, but I’m still proud of myself for my own take on the subject.

Recently, I saw an interesting post on Reddit that provided some source code for scraping images based on the similarity of their names. This might be useful for downloading all of the images in a gallery (for example, if you want to keep something posted on Imgur) without having to follow all of the links by hand. However, there was quite a lot of it missing, it relied on some random site, and it was bad at handling all kinds of common use cases: for example, thumbnails.

So I rewrote it. And expanded it. Massively. The current usage output is

Usage:
  galleryscraper.py URL DIR [--threads N --log-level N -q -s]
  galleryscraper.py -h | --help | --version

Options:
      --threads N        the number of threads to use [default: 4]
  -V, --log-level N      the level of info logged to the console, which can be
                         one of INFO, DEBUG, or WARNING [default: INFO]
  -s, --skip-duplicates  ignore files that have been downloaded already
  -q, --quiet            suppress output to console
  -v, --version          show program's version number and exit
  -h, --help             show this help message and exit

In particular, I worked out a clever way of dealing with thumbnails. Often, a gallery of thumbnails each link to a page containing ads, etc, with one large image (the one you actually want) in the centre. So I figured if one found the image on the page with the largest content-length, this would be the image in mind. This technique seems to work extremely well.

Mostly, this was an opportunity to learn about some gaps in my programming experience. First and foremost is how HTML requests actually work (success), but I also learned a lot of parallelization and Python decorators. I made use of the excellent (and pretty standard at this point) Python libraries Requests and BeautifulSoup.

The script itself has a couple of interesting features:

I write really, really well-documented code. It helps me keep track of what goes on, explain (and the improve) my approach, and it often comes in handy when I got back and try to figure out what’s going on. This should make it really easy to read the code itself.
The script contains no classes. Everything is a function, and functions are designed to have manageable side effects. I certainly haven’t been converted to functional programming, but I have become pretty irritated by the everything-must-be-a-class paradigm. This script is essentially a big function that turns a web page into a directory of images. If anyone’s interested in the things that go into accomplishing that, they might just benefit from the functions in the script itself. I’ve also seen the light that is namedtuples, which makes the return value of more complex functions much more manageable.
I learned to love decorators. No, seriously. I had never really thought of a use for them before, but here such uses turned up in abundance. I’m under the impression that this will have a big impact on how I think about writing code in Python going forward.

My favourite decorator at the moment:

def sessional(func):
    """
    Decorator that maintains the same session for URL requests within the given
    function. It relies on the wrapped function taking a ``session`` keyword
    argument. At the moment this is only used for the safe_request function,
    but if timeouts are not a problem it can be used to wrap all of the other
    functions that make requests instead.
    """
    session = requests.Session()
    session.mount('http://', HTTPAdapter(max_retries=5))
    session.headers['User-Agent'] = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11'
    
    @wraps(func)
    def newfunc(*args, **kwargs):
        try:
            # Replace sessions in the function's (kw) arguments
            kwargs['session'] = session
        except KeyError:
            pass
        return func(*args, **kwargs)
    
    return newfunc

Of course, there are still some edge cases. The script works very well most of the time, but some things are still tripping it up. Usually this is due to the images you want having different names, for example when some kind of random hash is appended to them.

In the future, I may make the script accept a file of URLs and directories, similar to how crontab works. I’m also looking in to working out when the page linked to by the thumbnail has multiple sizes of image – i.e. the Small - Medium - Large - Original header – and getting the image of the desired size.

IRC Hybrid on Arch Linux

2013-12-01T00:00:00Z

I recently tried to get an IRC server working on an installation of Arch Linux, and it failed to work “out of the box”. Below are the steps I took to get it working, including the work I did to get an SSL connection up and running.

First, get the package from the AUR::

$ cd ~/builds # or wherever else you like to put builds
$ curl -O https://aur.archlinux.org/packages/ir/ircd-hybrid/ircd-hybrid.tar.gz
$ tar -xvzf ircd-hybrid.tar.gz && cd ircd-hybrid

As of writing, simply installing this will cause the daemon to fail because it cannot find the .la files for the modules. To fix this you need to prevent the .la files from being removed. Open the PKGBUILD file and add:

options=("libtool")

then build and install the package:

$ makepkg -s
$ pacman -U ircd-hybrid-8.1.1-5-x86_64.pkg.tar.xz

you should now be able to run the service with systemctl start ircd-hybrid.

Setting up SSL

I mostly lifted this from the guide here.

To set up the server to run over SSL, you need to create the certificates and keys as follows:

$ cd /etc/ircd-hybrid/ssl
$ openssl genrsa -out rsa.key 2048
$ openssl req -new -days 365 -x509 -key rsa.key -out cert.pem
$ openssl dhparam -out dhparam.pem 2048

Fair warning: the last command can take several minutes to run. Afterwards you will need to make these files visible to the ircd user, since this is the user associated with the daemon::

$ cd ..
$ chown -hR ircd ssl

The next is to enable SSL correctly in the /etc/ircd-hybrid/ircd.conf file. In the listen section make sure to have the following:

host = "0.0.0.0";
#port = 6667; /* Only include this line if you want to allow non-SSL connections on another port. */
flags = ssl;
port = 6697;

You can modify the port number; 6697 seems to be the standard, and is used by Freenode. In the serverinfo section, make sure to indicate the location of the files that you’ve just created::

rsa_private_key_file = "/etc/ircd-hybrid/ssl/rsa.key";
ssl_certificate_file = "/etc/ircd-hybrid/ssl/cert.pem";
ssl_dh_param_file = "/etc/ircd-hybrid/ssl/dhparam.pem";
ssl_server_method = tlsv1, sslv3;

Once these changes have been made, save the configuration file and restart the daemon with systemctl start ircd-hybrid. You can check if the service is properly exposed to the outside world with netstat -an | grep -i '6697', which should give a result like:

tcp        0      0 0.0.0.0:6697            0.0.0.0:*               LISTEN

if the port you are using is indeed 6697.

Using More Interesting Random Values in Procedural Content

2013-10-16T00:00:00Z

This post is an inquiry into some of the drawbacks with using the random() function to generate all your random values, and discusses the circumstances in which the Beta distribution might prove a compelling alternative. It is written with procedural games in mind, there is no expectation that the reader know what a ‘distribution’ is, and all of the code examples are written in Python.

The Shortcomings of the Uniform Distribution

Most of the time I see tutorials on generating procedural content, whenever a random value is needed a call is made to the default generator (in Python, this is probably the one from the random module). So if we want to have a random number of chests in a room, you might see something like

import random as r
r.seed(101)
r.randint(1, 3)
## 2

Which gives a number of chests in {1, 2, 3}, with an equal chance of each. Or, we might trigger an event with 90% probability with something like

r.seed(101)
if r.random() >= 0.1:
	print 'Event happened.'
else:
	print 'Event did not happen.'
## Event happened.

In a setting where we might want to have an enemy with a variable amount of health, you could see something like

r.seed(102)
monster_health = 3000 + r.randint(1000, 2000)
## monster_health = 4148

But inevitably, these examples are constructed to use random values from uniform distributions. That is, every integer in the range of a to b has an equal chance of being output by the randint(a,b) function, and the mean value is exactly a + b / 2. So, in the example above, we can expect the average monster to have about 4500 health, and expect to see monsters with 4000-4500 health exactly as often as those with 4500-5000. But what if you wanted most monsters to have a health value close to 4500, and only a few with low values near 4000 or high values near 5000? This becomes a very difficult thing to do with only the uniformly-distributed r.random() and r.randint() functions very quickly.

Maybe the Normal Distribution is the Answer?

If you’ve ever been exposed to any introductory statistics, you’ve probably heard of the normal (or Gaussian) distribution, which might get us exactly what we want. A random function that gets its values from a normal distribution has most values clustered close to a specified mean, with a specified spread (or variance). It is quite easy to ensure that, for example, 99.7% of the values will fall within the range 4000-5000, with 68% of them falling within the range of about 4350-4650. Sounds great, right?

r.seed(103)
monster_health_list = list()
for i in xrange(0, 10):
	monster_health_list.append(int(r.gauss(mu = 4500, sigma = 500 / 3)))
print monster_health
## [4681, 4475, 4447, 4234, 4514, 4565, 4432, 4449, 4242, 4735]

The reason the sigma parameter is 500 / 3 in this case is because for normal distributions, 99.7% of values far within three ‘sigmas’ of the mean (which is called mu).

So far, so good. However, normal distributions have the inconvenient property of returning any value on occasion. It would be quite possible (although improbable) for a player to run into a monster with negative health, or a health value of 50,000. So if you’re going to use the normal distribution, make sure to have some checks for these unlikely cases.

Yet a normal distribution may leave some things to be desired. Say we wanted monsters to have health values in the range 4000-4500 more often than 4500-5000? Or we wanted most monsters to have health values in the low 4000s or high 4000s, but very few in between? If the latter scenario doesn’t seem that desirable, consider an analogous problem: we want to generate a random number of monsters for each room in our dungeon, but want most rooms to be either empty or relatively full – with few ‘easy’ rooms in between.

All of these problems can be solved using the remarkable Beta distribution.

The Beta Distribution and Why You Should Use It

The Beta distribution takes two parameters – alpha and beta – and outputs values between 0 and 1. This means we don’t need to worry about occasionally producing extremely large or small values. Also, the mean is always alpha / (alpha + beta), so it is easy to figure out what the average value will be.

What makes the Beta distribution interesting is that the two parameters give it a wide variety of ‘shapes’. It is because of the flexibility of the Beta distribution to take on different ‘shapes’ that it can be so useful:

When alpha = beta = 1, the Beta distribution is identical to the uniform distribution. That is, it will produce the same values as the random() function.
When alpha = beta, the mean is exactly 0.5, and there is a equal chance that values will fall above and below the mean. As alpha and beta grow larger, values move closer and closer to the mean.
When alpha > beta, more values will be generated below the mean, and vice versa.
When alpha < 1 and/or beta < 1, more values will be generated near 0 and 1 than near the mean. As alpha or beta gets closer and closer to zero, it becomes more and more unlikely that values away from 0 and 1 will be generated.

The Beta distribution is extremely popular in some fields of (social) science, and so it is no surprise that the standard library has an implementation in the random module, the function betavariate(alpha, beta).

The problem I proposed with filling rooms in a dungeon with monsters could be solved with code as simple as the following:

# A dungeon has ten rooms, with up to 50 monsters in each.
# We want most rooms to have a large number of monsters in
# them, or be almost empty.
r.seed(107)
monsters_in_rooms = list()
for i in xrange(0, 10):
	monsters_in_rooms.append(int(r.betavariate(0.1, 0.1) * 50))
print monsters_in_rooms
## [0, 49, 45, 49, 5, 49, 0, 48, 0, 25]

Notice that since betavariate() returns values between 0 and 1, I simply multiplied by the number I actually wanted. The parameters are alpha = beta = 0.1.

To get more monsters with 4000-4250 health than 4250-5000 health, try:

# Monsters have an average of 4500 health. With alpha = 1
# and beta = 3, the betavariate() will return values with
# a mean of 0.25. So we multiply by 1000, and add 4000.
r.seed(105)
health_list = list()
for i in xrange(0, 15):
	health_list.append(int(r.betavariate(1, 3) * 1000) + 4000)
print health
## [4054, 4023, 4430, 4172, 4415, 4253, 4045, 4135, 4056, 4035, 4005, 4172, 4072, 4296, 4395]
print sum(health) / 15
## 4170

If you wanted values closer to 4250 itself, simply increase alpha and beta without changing their ratio.

And finally, to get something like the normal distribution solution we had earlier, try:

# Monsters have an average of 4500 health, with a range of
# 4000-5000. So we multiply by 1000, and add 4000.
r.seed(115)
health_list = list()
for i in xrange(0, 10):
	health_list.append(int(r.betavariate(3, 3) * 1000) + 4000)
print health_list
## [4194, 4571, 4162, 4334, 4564, 4551, 4324, 4363, 4450, 4193]
print sum(health_list) / 15
## 4370

In a future post I will show how to use the Dirichlet distribution to solve another kind problem that shows up in procedural content generation.

An Update on the Libnoise Wrapper

2013-10-02T00:00:00Z

![NoisyPy Perlin Demo](./images/noisypy-perlin.png)

I’ve gotten a good chunk of the Libnoise wrapper working. And despite several sessions of head-against-wall bugs, my SIP and disutils setup seems to be working nicely. Some of the C++ code in noiseutils.h (which comes with the Libnoise examples) is quite outdated, so I wrote several of my own output functions in pure C++. This also gave me the opportunity to write exporters to OpenGL textures, which seem to work seamlessly with Pyglet. I should have a whole directory of examples up and running shortly, drawing from the standard Libnoise examples and as well as demonstrations of OpenGL textures.

At the moment, the code to produce the image in this post looks like this:

import noisypy.modules as nmods
import noisypy.util as nutil

p = nmods.Perlin()
p.seed = 113

s = nmods.ScaleBias()
s.SetSourceModule(0, p)
s.scale = 0.5
s.bias = 0.5

c = nmods.Clamp()
c.bounds = 0, 1.0
c.SetSourceModule(0, s)

nutil.noisemap_image(512, 512, c, filename='lolnoise.pgm')

which produces identical results to analagous code written in C++. This is an illustration of the wonderful interface that libnoise provides, and of how I think it has translated nicely to Python. I’ve decided to call the library noisypy, and it should appear on GitHub in the near future.

Setting up GDB for Debugging Python on OS X

2013-09-30T00:00:00Z

Note: this post can be read as a series of terminal commands if you’re not interested in the intervening explanations.

Recently, I started writing a Python extension that wraps the procedural generation library libnoise using SIP-generated code, and encountered the inevitable segfaults. It didn’t take long to establish that the error was somewhere in my C++ code, but I couldn’t figure out where exactly it was. A couple of searches online revealed that the way to go about finding these things was to use a proper debugger, and because the GNU Debugger – ``gdb` – supports Python, that seemed to be the way to go.

Getting it working was not, however, particularly straightforward. gdb supports debugging Python code as of version 7. Unfortunately, Apple’s version of gdb is currently sitting at 6.3. So all I had to do was get a newer version, right? Wrong.

Assuming you have Homebrew – if you don’t, go read about it – you can get a newer version of gdb from there. Since gdb is sort of a compiler, it’s not available in the main Homebrew repository (there’s a policy you can read on the wiki), so you’ll have to:

$ brew tap homebrew/dupes

first, and then run:

$ brew install gdb

You’ll notice immediately that if you run gdb --version in a terminal you’ll see:

GNU gdb 6.3.50-20050815 (Apple version gdb-1822) (Sun Aug  5 03:00:42 UTC 2012)
Copyright 2004 Free Software Foundation, Inc.
...
This GDB was configured as "x86_64-apple-darwin".

(That’s right, the Apple version is circa 2004…) Since you now have two versions of gdb installed, the shell will by default start the one in /usr/bin, while your Homebrew’d version is located in /usr/local/bin. There are a few ways to fix this. One is to edit the order of the paths in /etc/paths; you could, if you wanted, copy over the system version of gdb, too. However, both of these solutions may run you into other problems, particularly if you use XCode, which depends on Apple’s version of gdb for debugging. I haven’t actually tried deleting the system gdb, but I suspect you could screw some related things up this way. Instead, I prefer the simple renaming method::

$ mv /usr/local/bin/gdb /usr/local/bin/gdb7

Which allows you to run gdb7 --version and get the expected::

GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
...
This GDB was configured as "x86_64-apple-darwin12.5.0".

So far, so good. Unfortunately, debuggers need to inspect the internals of other processes, which programs aren’t normally allowed to do. So in order to really use gdb you need to give it permission to do this. Follow the instructions on the gdb website here to create a certificate, and then run::

$ codesign -s gdb-cert gdb7

Now it’s time to test debugging Python code, which should be as easy as gdb7 python. However, this gives:

"/usr/bin/python": not in executable format: File format not recognized

on OS X Mountain Lion. The reason? The system python executable is a ‘fat’ binary, meaning it can run on multiple system archtectures, i.e.:

$ file /usr/bin/python
/usr/bin/python: Mach-O universal binary with 2 architectures
/usr/bin/python (for architecture i386):	Mach-O executable i386
/usr/bin/python (for architecture x86_64):	Mach-O 64-bit executable x86_64

Unfortunately, gdb does not support debugging fat binaries (at least according to their mailing list). So you’ll have to extract the right system architecture using the lipo (hah!) program. If you have an Intel Mac running a 64-bit operating system (the mostly likely), that means the “x86_64” one. As a side note, the python program will actually call on the specific version of Python when you launch it, meaning that the executable you’re really interested in is /usr/bin/python2.7:

$ file /usr/bin/python2.7
/usr/bin/python2.7: Mach-O universal binary with 2 architectures
/usr/bin/python2.7 (for architecture i386):	Mach-O executable i386
/usr/bin/python2.7 (for architecture x86_64):	Mach-O 64-bit executable x86_64

From which we can extract a “thin”, architecture-specific binary (I’ve called python64) as follows::

$ lipo /usr/bin/python2.7 -thin x86_64 -output /usr/bin/python64

You can now run:

$ gdb7 python64

And see that:

Reading symbols from /usr/bin/python64...(no debugging symbols found)...done.

As expected. I’m not quite sure what gdb means when it says that python does not have debugging symbols – backtraces seem to work fine anyway. To verify that it works, try something like:

$ gdb7 python64
...
(gdb) run test.py
Starting program: /usr/bin/python64 test.py

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007fff5fc01028 in ?? ()
(gdb) continue
Continuing.
Hello, world!

Note that for some reason, Python always sends a SIGTRAP before it begins executing the script. Don’t worry, just continue. If you want to find out why your program segfaults, you can run bt:

Program received signal SIGSEGV, Segmentation fault.
0x000000010003441a in PyErr_Occurred () from /System/Library/Frameworks/Python.framework/Versions/2.7/Python
(gdb) bt
#0  0x000000010003441a in PyErr_Occurred () from /System/Library/Frameworks/Python.framework/Versions/2.7/Python
#1  0x000000010003a81b in _PyArg_NoKeywords () from /System/Library/Frameworks/Python.framework/Versions/2.7/Python
#2  0x0000000100039c91 in _PyArg_NoKeywords () from /System/Library/Frameworks/Python.framework/Versions/2.7/Python
#3  0x0000000100038c43 in PyArg_Parse () from /System/Library/Frameworks/Python.framework/Versions/2.7/Python
#4  0x0000000100038f0d in PyArg_ParseTuple () from /System/Library/Frameworks/Python.framework/Versions/2.7/Python
#5  0x0000000100493335 in meth_Clamp_SetBounds () from /Users/Aaron/Code/Python/simplex-test/noisypy/noisypy.so
#6  0x00000001000136c6 in PyObject_Call () from /System/Library/Frameworks/Python.framework/Versions/2.7/Python
...

Which located the error pretty quickly my wrapper of Clamp.SetBounds().

Happy debugging!

Building Libnoise on OS X

2013-09-29T00:00:00Z

Although it is very old (and unmaintained since 2007), Libnoise is a great library for procedural generation of pseudorandom noise. It provides a wonderfully intuitive way of chaining together various layers and styles of noise that I’ve never seen matched, even in newer implementations. It also supports seeds (which I require), and is largely feature-complete. I’ve used some of the Java and XNA ports in the past, but in the last few days started getting interested in using the original C++ code, partly because I’m wondering if it can be converted into a Python extension.

Since the original Makefile is problematic, someone has gone to the trouble of converting it to cmake, so you can run:

$ git clone git@github.com:eXpl0it3r/libnoise.git
$ cd libnoise
$ cmake .
$ make && make install

To install the static library on your system. This seems to work on OS X Mountain Lion. You can compile a few of the examples (in the examples directory) to try them out. Everyone’s favourite is the worms.cpp demo, but since it uses OpenGL, you’ll have to change

#include <gl/gl.h>
#include <glut.h>

at the top of the file to

#include <OpenGL/gl.h>
#include <GLUT/glut.h>

due to the locations of those libraries on OS X. You can then compile this using (yes, I know it’s a hell of a command):

$ g++ /System/Library/Frameworks/GLUT.framework/GLUT /System/Library/Frameworks/OpenGL.framework/OpenGL -o worms.o worms.cpp -llibnoise

Run it (you may have to chmod +x worms.o first) with:

$ ./worms.o

which opens up a lovely window with some wriggling worms. It should look something like this:

![The Libnoise Worms]({filename}/images/libnoise-worms.png)

Most of the other examples make use of the noiseutils.cpp file, so they can be compiled and run with something like:

$ g++ -o wood.o texturewood.cpp noiseutils.cpp -llibnoise && ./wood.o

The most interesting example is probably to be found in complexplanet.cpp. Although it takes about three full minutes to run on my computer (poor performance has always been the criticism of libnoise), it does produce the kind of image that got me interested in procedural generation to begin with:

![Libnoise Terrain]({filename}/images/libnoise-terrain.png)

Logic in Maven Archetypes

2013-04-18T00:00:00Z

Time for a real first post. Recently, I’ve been working on creating a Maven archetype for a certain open-source project, and discovered to my delight that archetypes actually support a limited kind of logic. If it’s not immediately obvious why that would be useful, consider that you can make use of this logic to, for example, optionally add a dependency based on a custom parameter.

Making use of the fact that Maven builds its archetypes using Apache’s Velocity template engine, you can embed statements reminiscent of C-style preprocessor directives in any file in the archetype. To add an optional dependency, try embedding the logic below in the pom.xml file

<dependencies>
	...
#if (${includeNetty} == 'true' || ${includeNetty} == 'yes' || ${includeNetty} == 'y')
	<dependency>
		<groupId>io.netty</groupId>
		<artifactId>netty</artifactId>
		<version>3.4.6.Final</version>
		<scope>compile</scope>
	</dependency>
#end
</dependencies>

This will work provided, of course, that you’ve defined includeNetty like so in your archetype-metadata.xml file.

<requiredProperties>
    <requiredProperty key="includeNetty"> 
     	<defaultValue>false</defaultValue>
    </requiredProperty> 
</requiredProperties>

The above snippet becomes

<dependencies>
	...
	<dependency>
		<groupId>io.netty</groupId>
		<artifactId>netty</artifactId>
		<version>3.4.6.Final</version>
		<scope>compile</scope>
	</dependency>
</dependencies>

when -DincludeNetty is set to true (or yes, or y, in keeping with Maven-style commands), and

<dependencies>
	...
</dependencies>

when it is false (or, really, anything but true, yes, or y). Unfortunately, this trick doesn’t work to optionally include whole files – there isn’t currently a way to do that at all in Maven, although the feature has been suggested.

Avoid leading whitespace

Just a tip: when embedding these preprocessing commands, don’t put any whitespace before the # character, otherwise it will be included in the output. This snippet

<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<currentYear>2013</currentYear>
	#if (${includeNetty} == "true")
	<nettyVersion>3.4.6.Final</nettyVersion>
	#end	
<properties>

harmless though it looks, will actually output

<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<currentYear>2013</currentYear>
		<nettyVersion>3.4.6.Final</nettyVersion>
	<properties>

due to the leading whitespace, which can be extremely annoying if you actually care about nicely formatted pom.xml files.