Live Updates With Queues, WebSockets, and Push Notifications. Part 4: Webhooks

So far in this series we’ve set up a React Native frontend and Node backend to receive notifications from external services and deliver live updates to our client app. But we haven’t done any real service integrations yet. Let’s fix that! Our app can be used to notify us of anything; in this post we’ll hook it up to GitHub. We’ll also have some tips for if you want to hook it up to Netlify. In a future post we’ll provide tips for hooking up to Heroku as well, after we’ve deployed the backend to run on Heroku.

If you like, you can download the completed server project and the completed client project for part 4.

Setup

Before we create any of these webhook integrations, let’s refactor how our webhook code is set up to make it easy to add additional integrations. We’ll keep our existing “test” webhook for easy experimentation; we’ll just add webhooks for real services alongside it.

We could set up multiple webhook endpoints in a few different ways. If we were worried about having too much traffic for one application server to handle we could run each webhook as a separate microservice so that they could be scaled independently. Alternatively, we could run each webhook as a separate function on a function-as-a-service platform like AWS Lambda. Each microservice or function would need to have access to send messages to the same queue, but other than that they could be totally independent.

In our case, we’re going to deploy our app on Heroku. That platform only allows us to expose a single service to HTTP traffic, so let’s make each webhook a separate route within the same Node server.

Create a web/webhooks folder. Move web/webhook.js to web/webhooks/test.js. Make the following changes to only export the route, not to set up a router:

-const express = require('express');
-const bodyParser = require('body-parser');
-const queue = require('../lib/queue');
+const queue = require('../../lib/queue');

 const webhookRoute = (req, res) => {
...
 };

-const router = express.Router();
-router.post('/', bodyParser.text({ type: '*/*' }), webhookRoute);
-
-module.exports = router;
+module.exports = webhookRoute;

We’ll define the router in a new web/webhooks/index.js file instead. Create it and add the following:

const express = require('express');
const bodyParser = require('body-parser');
const testRoute = require('./test');

const router = express.Router();
router.post('/test', bodyParser.text({ type: '*/*' }), testRoute);

module.exports = router;

Now we just need to make a tiny change to web/index.js to account for the fact that we’ve pluralized “webhooks”:

 const express = require('express');
-const webhookRouter = require('./webhook');
+const webhookRouter = require('./webhooks');
 const listRouter = require('./list');
...
 app.use('/list', listRouter);
-app.use('/webhook', webhookRouter);
+app.use('/webhooks', webhookRouter);

 const server = http.createServer(app);

This moves our webhook endpoint from /webhook to /webhooks/test. Now any future webhooks we add can be at other paths under /webhooks/.

If your node web process is running, stop and restart it. Make sure node workers is running as well. You’ll then need to reload your Expo app to re-establish the WebSocket connection.

Now you can send a message to the new path and confirm our test webhook still works:

$ curl http://localhost:3000/webhooks/test -d "this is the new endpoint"

That message should show up in the Expo app as usual.

Making Our Local Server Accessible

We need to do another preparatory step as well. Because we’ve been sending webhooks from our local machine, we’ve been able to connect to localhost. But external services don’t have access to our localhost. One way to get around this problem is ngrok, a great free tool to give you a publicly-accessible URL to your local development machine. Create an ngrok account if you don’t already have one, then sign in.

Install ngrok by following the instructions on the dashboard to download it, or, if you’re on a Mac and use Homebrew, you can run brew cask install ngrok. Provide ngrok with your auth token as instructed on the ngrok web dashboard.

Now you can open a public tunnel to your local server. With node web running, in another terminal run:

$ ngrok http 3000

You should see output like the following:

In the output, look for the lines that start with “Forwarding” – these show the .ngrok.io subdomain that has been temporarily set up to access your service. Note that there is an HTTP and HTTPS one; you may as well use the HTTPS one.

To confirm it works, send a POST to your test webhook using the ngrok URL instead of localhost. Be sure to fill in your domain name instead of the sample one I’m using here:

$ curl https://abcd1234.ngrok.io/webhooks/test -d "this is via ngrok"

The message should appear in the client as usual.

Building the GitHub Webhook

Now that we’ve got a subdomain that can be accessed from third-party services, we’re ready to build out the webhook endpoint for GitHub to hit. Create a web/webhooks/github.js file and add the following:

const queue = require('../../lib/queue');

const webhookRoute = (req, res) => {
  console.log(JSON.stringify(req.body));

  const {
    repository: { name: repoName },
    pull_request: { title: prTitle, html_url: prUrl },
    action,
  } = req.body;

  const message = {
    text: `PR ${action} for repo ${repoName}: ${prTitle}`,
    url: prUrl,
  };

  console.log(message);
  queue
    .send('incoming', message)
    .then(() => {
      res.end('Received ' + JSON.stringify(message));
    })
    .catch(e => {
      console.error(e);
      res.status(500);
      res.end(e.message);
    });
};

module.exports = webhookRoute;

In our route, we do a few things:

• We log out the webhook request body we received as a JSON string, in case it’s useful.
• We pull the pertinent fields out of the request body: the repository name, pull request title and URL, and the action that was taken (opened, closed, etc.)
• We construct a message object in the standard format our app uses: with a text field describing it and a related url the user can visit.
• As in our test webhook, we send this message to our incoming queue to be processed.

Connect this new route in web/webhooks/index.js:

 const testRoute = require('./test');
+const githubRoute = require('./github');

 const router = express.Router();
 router.post('/test', bodyParser.text({ type: '*/*' }), testRoute);
+router.post('/github', express.json(), githubRoute);

 module.exports = router;

Note that in this case we aren’t using the bodyParser.text() middleware, but instead Express’s built-in express.json() middleware. This is because we’ll be receiving JSON data instead of plain text.

Restart node web to pick up these changes. You don’t need to restart ngrok.

Testing the Integration

Now let’s create a new repo to use for testing. Go to github.com and create a new repo; you could call it something like notifier-test-repo. We don’t care about the contents of this repo; we just need to be able to open PRs. So choose the option to “Initialize this repository with a README”.

When the repo is created, go to Settings > Webhooks, then click “Add webhook”. Choose the following options

• Payload URL: your ngrok domain, with /webhooks/github appended.
• Content type: change this to application/json
• Secret: leave this blank. We aren’t using it for this tutorial, but you can use this field to confirm your webhook traffic is coming from a trusted source
• SSL verification: leave as “Enable SSL verification”
• Which events would you like to trigger this webhook? Choose “Let me select individual events”

• Scroll down and uncheck “Pushes” and anything else if it’s checked by default, and check “Pull requests”.
• Active: leave this checked

Note that your ngrok URL will change every time you restart ngrok. You will need to update any testing webhook configuration in GitHub and other services to continue receiving webhooks.

Now we just need to create a pull request to test out this webhook. The easiest way is to click the edit icon at the top right of our readme on GitHub’s site. Add some text to the readme, then at the bottom choose “Create a new branch for this commit and start a pull request,” and click “Commit changes,” then click “Create pull request.”

In your client app you should see a new message “PR opened for repo notifier-test-repo: Update README.md:”

If you want to see more messages, or if something went wrong and you need to troubleshoot, you can repeatedly click “Close pull request” then “Reopen pull request;” each one will send a new event to your webhook.

Our test webhook didn’t pass along any URLs. Now that we have messages from GitHub with URLs attached, let’s update our client app to allow tapping on an item to visit its URL. Open src/MessageList.js and make the following change:

 import React, { useState, useEffect, useCallback } from 'react';
-import { FlatList, Platform, View } from 'react-native';
+import { FlatList, Linking, Platform, View } from 'react-native';
 import { ListItem } from 'react-native-elements';
...
       <FlatList
         data={messages}
         keyExtractor={item => item._id}
         renderItem={({ item }) => (
           <ListItem
             title={item.text}
             bottomDivider
+            onPress={() => item.url && Linking.openURL(item.url)}
           />
         )}
       />

Reload the client app, tap on one of the GitHub notifications, and you’ll be taken to the PR in Safari. Pretty nice!

Heroku and Netlify

Now we’ve got a working GitHub webhook integration. We’ll wait a bit to set up the webhook integration with Heroku; first we’ll deploy our app to Heroku. That way we’ll be sure we have a Heroku app to receive webhooks for!

Netlify is another deployment service with webhook support; it’s extremely popular for frontend apps. We won’t walk through setting up Netlify webhooks in detail, but here are a few pointers if you use that service and would like to try integrating.

To configure webhooks, open your site in the Netlify dashboard, then click Settings > Build & deploy > Deploy notifications. Click Add notification > Outgoing webhook. Netlify requires you to set up a separate hook for each event you want to monitor. You may be interested in “Deploy started,” “Deploy succeeded,” and “Deploy failed.”

The webhook route code itself should be very similar to the GitHub one. The following lines can be used to construct a message from the request body:

const { state, name, ssl_url: url } = req.body;

const message = {
  text: `Deployment ${state} for site ${name}`,
  url,
};

What’s Next?

Now we’ve got our first real service sending notifications to our app. But the fact that we’re dependent on a changeable ngrok URL feels a bit fragile. So we can get this running in a stable way, in our next post we’ll deploy our app to production on a free Heroku account.