RStudio has an enterprise offering called RStudio Connect that is designed to host R (and now Python) content. In my experience it’s a great platform for R users using the RStudio IDE, and works particularly well for frictionless deployment of internal Shiny apps or RMarkdown reports.
But one of the things we’ve struggled with in using Connect is making Shiny app logs useful. You can view logs for individual sessions in the browser (if you know what you’re looking for), but there’s no searching or aggregation of any kind. This is the existing interface:
This is doubly frustrating for users with experience with any other log management system – a crowded field, with many commercial and open-source solutions available. These solutions typically offer not only searching but complex querying, visualisation, and ad-hoc dashboard capabilities.1 For example, check out the landing page for Elasticsearch/Kibana, which is the most popular open-source option. Common commercial choices include Splunk and Datadog, and all of the cloud vendors have highly-integrated platforms of their own.
Unfortunately, Connect won’t work natively with any of these tools. Log management is focused on the per-session GUI.
Motivated by our own internal desire to get Shiny logs into Splunk, I wrote a Connect-specific plugin for the open-source Fluent Bit project. Fluent Bit itself is a fast, lightweight log forwarding and aggregation program with a vibrant plugin community.
The plugin works by watching for new logs written to the filesystem and querying the Connect API for additional metadata to link these logs back to specific applications. Fluent Bit can then be used to forward the results to any of its supported backends, including Elasticsearch/Kibana, Splunk, Stackdriver, Cloudwatch Logs, Datadog, Loki, and many others.
This should allow R users to easily export logs to their organisation’s existing log management system without needing to modify their reports or Shiny apps to do so. For example, log entries look like this in our Splunk instance:
You can see from the above that entries contain not only the log message, “Warning in readLines…”, but also the session identifier, the name of the app or report, and the app mode – a Shiny application, in this case. Support for Connect’s tag system will probably be added in the future.
The plugin is open-source under the LGPL 2.1 and available on GitHub.
Shoutout to Cole Arendt for his advice on using Connect’s APIs to make this work.
-
As an aside: originally we pushed Shiny app logs into our Splunk instance by having them write to the local syslog via the rsyslog package, which was then independently forwarded on to Splunk. My coworker Nicholas Jhirad mentioned some of this history in a talk at eRum last year. A huge downside of this approach is that it requires everyone who writes a Shiny app to opt in to using this package, and any message not sent through it is lost. ↩︎