In this second instalment of the Forging Dropfort series, we will look at the Dropfort release manager. This Dropfort module is an integral part of Dropfort and performs many functions including responding to project repository callbacks (or webhooks), queueing project releases, and generating all the data required by Feature Server (see Forging Dropfort - Feature Server for details).

When you connect a project to Dropfort, webhooks are automatically set on the project repository on GitHub and/or GitLab. Webhooks are used as a way of notifying a web service that an action has been performed in a given system. GitLab and GitHub offer webhooks that trigger whenever a push is made against a repository. On the Dropfort side, the release manager defines a few web service endpoints – using the Services module – to listen for webhook events. This way, Dropfort knows when a commit or tag is pushed to project a project.

When Dropfort receives a push notification, it finds the project corresponding to that notification and, given certain conditions, queues a release to be created for that project. Dropfort does this by employing the use of the Drupal Queue API. We decided to make use of the Queue API because of its flexibility in the way it can be implemented, as well as the fact that it leverages cron to batch process the queue items and to allow hundreds or thousands of releases to be created at once without affecting front-end performance.

Below is a very high-level diagram outlining the major flow of operations performed by the Dropfort release manager:

1. The push notification webhook is triggered on the project repository and handled by the release manager service endpoint.
2. A queue item is created.
3. Cron processes the queue item.
4. A release file is created, along with a Feature Server release entity and release page node.

Setting up a basic queue is straightforward, but we had specific requirements for Dropfort requiring us to change the default behaviour of some of the basic queue operations. Thankfully, Drupal’s queue implementations allows us to override certain methods in a custom queue subclass.

Let’s take for example the case where many commits are pushed to a project (that has been connected to Dropfort) in a short span of time. That is to say, several web hooks are triggered for a single project before a release has time to be created. The first step in the flow, Dropfort would be notified of the first push, perform some validation, and add an item to the release manager queue. Under normal circumstances, in the time between that first push, and when the queue item is processed any other push notifications from that project would create a new queue item. Thanks to our Queue API subclass, instead of this behavior, the data in the queue is updated. This also saves the item from being sent to the back of the queue and losing its place in line.

Here’s a summary of the requirements for the release manager queue:

• persistent queue (DB rather than in-memory)
• no duplicate releases for the same project in the queue at any given time
• subsequent push notifications update (rather than requeue) existing item

The requirement to have the queue persist in the database rather than in-memory is satisfied by using the SystemQueue class, which implements the DrupalReliableQueueInterface. The methods of this class do not satisfy the requirements to update existing items while retaining their place in the queue. To solve this problem, we subclassed SystemQueue in order to provide a custom implementation for the createItem($data) method. Here’s what the implementation looks like (using pseudo code):

function createItem($data) {
    query DB for a queue item for the project;
    if a queue item exists:
        update queue item DB fields using new data;
    else:
        create queue item normally;
}

One last queue class we created was a subclass of QueueUISystemQueue. This simple class has the same name as our queue class prefixed with “QueueUI” and only implements a basic constructor which calls the parent constructor. This allows us to use the QueueUI module to manage the release manager queue using an administrative interface.

The last step in the process is actually packaging the files into zip or tgz files for distribution. But we’ll cover that in a later post on the “Workflow” system inside Dropfort.

There you have it. From commit to webhook to release, this is how how we package your custom modules with Dropfort.