Building a Gantt Chart from Github Issues: Caching w/ Realm

Gantt Chart from Github Issues

Creating complex software like Realm requires excellent project management tools. While there are lots of different solutions to manage this process, we are heavily invested in Github because this is where we host all of our open-source code. This means we go beyond just using Github for source code management, we also actively develop new functionality through PRs and track new features and bugs with issues. We love Github and the community around it, but from a project management perspective we have faced challenges managing our development process within it.

One aspect of Github we found limiting for us was visualizing work in the issues. To combat this, we use Waffle.io to create a board layout of our issues. While Github recently added support for projects, it still has limitations such as combining multiple repos into one board. Having a board view helped us understand the priorities and work in progress, but it still left us needing another view to represent task lengths and dependencies.

The ideal tool for us would create a chronological timeline of issues to represent our product roadmap. We surveyed the space, as there are lots of charting or roadmapping tools, but all of them failed to robustly sync with Github. Specifically, the best tools are the ones that fade into existing processes rather than create new ones. Given that our engineering organization lives in Github, for us the ideal tool is one that allows our engineering team to make changes in Github, which updates the resulting views (similar to how Waffle updates in response to issues closing). Evaluating existing products showed that all of them would require leaving Github to update date estimates.

What We Built

We then stumbled upon a project called GanttlLab-Live, which creates a Gantt chart from GitLab/Github issues directly. It does this by parsing the issues to find data tags, such as GanttStart:, which act as the source of truth for the chart data. This satisfied our desire to allow data updates in Github to propagate into the view. However, in testing the code we ran into issues with Github support and found the chart UI limiting as it is a static chart built via D3. Given the project is open-source we could have committed changes to it, but we assessed it would be a lot of work. Instead, we found another full-featured Gantt chart framework, dhtmlxGantt that looked simpler to integrate with and would offer advanced functionality out-of-the-box.

Get more development news like this

Building off the pattern with GanttlLab-Live, we quickly sketched out the requirements to make everything work: Integration logic to pull issue data from Github APIs Data layer to map issues to Gantt chart tasks Webserver to serve the Gantt chart

Given that Realm supports Javascript, it seemed easiest to use realm-js as the data layer within a Node application to handle the data integration and use Express as the webserver. While Realm is commonly known as a mobile database, with our recent launch of Realm Mobile Platform we added support for Linux with realm-js to support server-side use-cases. The side-effect of using Realm is that it offers persistence, allowing us to cache the Github API responses so the webserver can respond immediately to incoming requests.

Pulling it altogether was very quick. We used Octokat.js to make the Github API calls, saving the data as a Task object in the Realm. Correspondingly, we created a /data endpoint that would lookup the Task data in Realm and return JSON to be used by the Gantt chart itself. To keep the Realm data up-to-date, a background process triggers a full refresh from Github APIs on each request.

The tool quickly proved to be extremely helpful for us internally and we hope it can help others that are heavily invested in Github as well. The code is available on Github here. To get started, just run the setup.sh script to configure your Github credentials and the corresponding data tags.

// Your Github Access Token
GITHUB_API_TOKEN: ""
  
// The name of the Github organization
GITHUB_ORG_NAME: ""
// The repo name in the Github organization
GITHUB_REPO_NAME: ""
  
// Configuration for the labels in Github issues to search for
START_DATE_STRING: "#### 🗓 Start Date:"
DUE_DATE_STRING: "#### 🗓 Expected Date:"
LABEL_STRING: "#### 🏷 Label:"
PROGRESS_STRING: "#### 📈 Progress (0-1):"

We recommend creating an issue template in Github that will ensure that the data tags are always present in the issue. We have continued to add new features, such as the ability use the Gantt chart’s drag-and-drop functionality to makes changes to dates, which subsequently get written back to the Github issue. The chart offers many more features, such as subtasks and dependencies, that we would love to incorporate as well.

Now you might be wondering “this is cool, but why?” We realize there is a common trap companies can fall into by building unnecessary tooling themselves. Admittedly though there was another reason to create this tool and that was to showcase a new use-case for Realm. While Realm is a great replacement for Core Data or SQLite in mobile applications, the power of having a high-performance embedded object-database extends to other applications as well.

The Github-Gantt tool demonstrates a common pattern where a caching mechanism is needed to fulfill backend requests quickly. Furthermore, while this tool only had to rely on Github APIs, there still was a data mapping step. In more complex deployments involving several backend system-of-records, this can be complex and require data virtualization middleware.

Caching Architecture

Realm’s unique characteristics make it well suited to solve these challenges. As a caching system, we offer competitive read performance due to our column-based internal architecture, memory-mapping, and lazy loading. Furthermore, unlike existing caching systems, our robust type and query system goes beyond a basic key/value store. As a true object-database, this means that Realm as a data layer can blend seamlessly with your application logic.

Looking at the architecture of the Github-Gantt tool, Realm is being used as a “write-behind cache” within the Express webserver. Reads are handled directly from the cache and changes from the chart are written directly to the cache first, then asynchronously applied to Github. The benefit from this architecture is that it reduces the latency from the perspective of the webserver. From a high-level perspective, this latency reduction is the same benefit that Realm brings mobile applications as well. By having your data synchronously accessible in your application, latency is reduced and you can deliver a better user-experience.

Perhaps a better way to think about Realm is not just as a mobile database, but as an edge data layer. Shifting your data to the edge of your infrastructure can mean the mobile device, but in the case of web applications, the edge is the webserver itself. This capability is even more powerful with the launch of Realm Mobile Platform and the concept of synchronized Realms.

Now your edge data layer isn’t just a basic cache requiring complex machinery to keep up-to-date, but rather a realtime database. Multiple copies of a Realm seamlessly update, which can be used to simplify scaling out your caching infrastructure, in addition to managing mobile device data synchronization. Github-Gantt doesn’t need scale out, but it does support configuring the Realm as a synchronized Realm by simply providing a few extra parameters in the configuration:

// Configuration to use Realm Mobile Platform
RMP_ADMIN_TOKEN: ""
RMP_SYNC_URL: "" // realm://127.0.0.1:9080/github-gantt

This enables opening the data in the Realm Browser to view or make live-updates. Furthermore, if we wanted to extend the web application, we could export the model definition in the Realm Browser to Swift, Objective-C, Java, or C# and immediately have the synced data accessible in a mobile application.

Today, we only offer our Node SDK for server-side use-cases, but we plan to launch other languages, like Java and C#, as well. If this sounds interesting, we would love to hear from you!

Next Up: Ready for Realtime and Scale: Announcing Realm Mobile Platform 1.0

General link arrow white

Adam Fish

Adam is the Director of Product at Realm, where he manages the product development for currently supported mobile platforms and upcoming new products. He has a strong background in entrepreneurship and software development, having previously co-founded Roobiq, a mobile-first sales productivity app used by sales teams worldwide.