Jason St-Cyr

Posted on Apr 25 • Originally published at jasonstcyr.com on Apr 9

Using Webhooks to Update an Algolia Index on Every Sanity Edit

#cms #algolia #sanity

While exploring Sanity and Algolia, I decided to investigate how to keep my Algolia index updated as edits were made in the Sanity Studio. It turned out that this was super simple to set up by using webhooks in the Sanity Studio configuration. I followed a great guide on Sanity io by Irina Blumenfeld on How to implement front-end search with Sanity which provided me with a lot of the steps and code that I needed to get going. Most of this article is a subset of that article, with specific breakdowns of what I had to learn as I went through setting this up. My order of operations differs a little bit from the official guide, as well. The order below was what worked for me to get things up and running and testing as I went along.

Okay, let's do a step-by-step walkthrough of how this works! Here are the steps we'll cover:

ℹ️ The following steps assume you have a Next.js TypeScript app router application connected to Sanity and Algolia.

Before doing anything else, your application needs to know about your Algolia and Sanity endpoints. In your local environment, you will want to have environment variables setup in your .env file, and in your hosting environment (like Vercel) you’ll need to set the values you will be using there. In this tutorial, these are the environment variables you will need:

NEXT_PUBLIC_ALGOLIA_INDEX_NAME=myIndexName
ALGOLIA_API_KEY=YourAPIKeyGoesHere
NEXT_PUBLIC_ALGOLIA_APP_ID=YourAlgoliaAppID
SANITY_WEBHOOK_SECRET=my-algolia-secret

NEXT_PUBLIC_ALGOLIA_INDEX_NAME: When you first setup Algolia, you created your first index. This environment variable should be configured with the name of your index where you will store the data.
ALGOLIA_API_KEY: You can see your list of API keys on the Algolia API key dashboard. Because this webhook is writing to the index, you'll want the Write API Key from the list.
NEXT_PUBLIC_ALGOLIA_APP_ID: This can also be found on your API Keys dashboard where there's an easy "copy" button. If you go to your Algolia applications dashboard it is also displayed there. It should look like a 10-character code like JMUBOVIBSE.
SANITY_WEBHOOK_SECRET: This can be any value you want to create that will act as the secret handshake between your Sanity Webhook and the API endpoint.

Step Two: Create an API Endpoint to Receive Webhook Requests

The first step is the most important clinching piece of the entire flow. The API endpoint that you create will receive the requests from your Sanity webhook and will then invoke the Algolia client to update the index. You can find my full example file on GitHub that is based on the Sanity guide code, but I will break it down in this section.

Install npm packages. Install the algoliasearch and Sanity packages that will be needed to work with the client endpoints.
npm install algoliasearch next-sanity @sanity/webhook
Create folder. In your web application project, create a folder for your route: /app/api/algolia
Create Route file. In this folder, create a new route.ts file. This will handle the requests to this route.

Import types. Start building out your route handler with imports.

import { algoliasearch } from "algoliasearch";
import { client } from "@/sanity/client";
import { SanityDocument } from "next-sanity";
import { isValidSignature, SIGNATURE_HEADER_NAME } from "@sanity/webhook";

Load environment variable values. Before running any logic, the API endpoint needs to read your environment variables to load up the Algolia and Sanity connection information and establish a connection to Algolia.

const algoliaAppId = process.env.NEXT_PUBLIC_ALGOLIA_APP_ID!;
const algoliaApiKey = process.env.ALGOLIA_API_KEY!;
const indexName = process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME!;
const webhookSecret = process.env.SANITY_WEBHOOK_SECRET!;

const algoliaClient = algoliasearch(algoliaAppId, algoliaApiKey);

Create skeleton performInitialIndexing. Create a method to do an initial population of the index. We will leave this empty in this step, but if you want to drop something in at this stage you can grab a sample from the full example file on GitHub

// Function to perform initial indexing
async function performInitialIndexing() {
  console.log("Starting initial indexing...");

  console.log("Initial indexing completed.");
  return {
    message: "Successfully completed initial indexing!",
  };
}

Create a POST method to receive requests from the Sanity Webhook. Again, we'll leave this as mostly a skeleton function in this tutorial step, but a full example file is on GitHub. In this skeleton, we extract a search parameter to see if it's the initial indexing call. If so, call our empty initial indexing function, and otherwise proceed to close out the function.

The try/catch here handles the various errors that can come up as we communicate with both Sanity and Algolia endpoints.

export async function POST(request: Request) {
  try {
    const { searchParams } = new URL(request.url);
    const initialIndex = searchParams.get("initialIndex") === "true";

    // Perform initial indexing
    if (initialIndex) {
      const response = await performInitialIndexing();
      return Response.json(response);
    }
    console.log(`Indexed/Updated object`);
    return Response.json({
      message: `Successfully processed document!`,
    });
  } catch (error) {
    console.error("Error indexing objects:", error instanceof Error ? error.message : String(error));
    return Response.json(
      { error: "Error indexing objects", details: error instanceof Error ? error.message : String(error) },
      { status: 500 }
    );
  }
}

TEST IT! At this point, you should run an npm run build for your basic route and make sure you pass linting. Then, deploy to your host. Once your API route is in your Node.js host (like a Vercel or Netlify), you can open up the API route and check to see if it can run and get you a success message. Since there is no security at this stage (secrets or signatures in the headers) you should be able to directly access the endpoint.

Step Three: Initial Indexing

It's now time to add the initial indexing logic into the route.ts file and flesh out that performInitialIndexing function. This tutorial has been broken out as a separate step for that function so we can walk through what the pieces of the code are doing.

ℹ️ If you want to skip ahead, again reminding that there is a full example file on GitHub!

Load Sanity documents. The very first thing we're going to add in is a call to the Sanity client to get the data we want to put in Algolia. In my case, I'm reading my post records so that I can import it into Algolia in my articles index.

// Fetch all documents from Sanity
const sanityData = await client.fetch(`*[_type == "post"]{
    _id,
    title,
    slug,
    "body": pt::text(content)[0..2000],
    _type,
    "coverImage": coverImage.asset->url,
    date,
    _createdAt,
    _updatedAt
  }`);

Map to Algolia Index structure. Once these are loaded from Sanity, we need to map the data from the SanityDocument structure into the record structure we need for Algolia. This is defining what pieces we want to send into the index.

const records = sanityData.map((doc: SanityDocument) => ({
    objectID: doc._id,
    title: doc.title,
    slug: doc.slug.current,
     /**
     *  Truncating the body if it's too long. 
     *  Another approach: defining multiple records:
     *  https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/how-to/indexing-long-documents/
     */
    body: doc.body?.slice(0, 9500),
    image: doc.image,
    publishedAt: doc.publishedAt,
    _createdAt: doc._createdAt,
    _updatedAt: doc._updatedAt,
    tags: doc.tags,
    categories: doc.categories,
  }));

Save to Algolia. Then we save the objects in Algolia!

// Save all records to Algolia
 await algoliaClient.saveObjects({
   indexName,
   objects: records,
 });

TEST IT! Now you have a function you can call and you should be able to test that it can connect to Sanity and populate your Algolia index with an initial set of data.

After having validated that all your connections to Sanity and Algolia are working and you have an initial index population, now we can add the code that is going to handle the webhook calls from Sanity. The intention is that as editors change data in Sanity, the webhook will fire and tell your API route to update Algolia to match with the new changes.

This tutorial step is going to fill out the POST method so that the API can handle those updates. We will break down each logical chunk of the function.

ℹ️ If you want to skip ahead, again reminding that there is a full example file is on GitHub!

Validate signature. First up is to validate that the request is originating from a Sanity webhook. If the header for the signature isn't there, then we block the request with a 401 (unauthorized). This is a first layer of security to only execute Algolia updates from the Sanity webhook. We then validate the signature and the webhook secret to make sure a valid signature was provided. This is our second layer of security.

// Validate webhook signature
const signature = request.headers.get(SIGNATURE_HEADER_NAME);
if (!signature) {
    return Response.json(
    { success: false, message: "Missing signature header" },
    { status: 401 }
    );
}
// Get request body for signature validation
const body = await request.text();
const isValid = await isValidSignature(body, signature, webhookSecret);

if (!isValid) {
    return Response.json(
    { success: false, message: "Invalid signature" },
    { status: 401 }
    );
}

Parse the payload. Once the signature has been validated in the header, the payload from Sanity needs to be parsed out so that we can begin extracting values.

// Incremental updates based on webhook payload
let payload;
try {
  payload = JSON.parse(body);
  console.log("Parsed Payload:", JSON.stringify(payload));
} catch (jsonError) {
  console.warn("No JSON payload provided." + jsonError);
  return Response.json({ error: "No payload provided" }, { status: 400 });
}

Extract payload info. From this payload we need to find out what object was updated, what operation was done on the object, and what value we need to send to the index. These all come from the Sanity projection in the webhook that will be defined later.
```
const { _id, operation, value } = payload;

if (!operation || !_id || !value) {
  return Response.json(
    { error: "Invalid payload, missing required fields" },
    { status: 400 }
  );
}
```

Update index. Now the index needs to be updated. If the operation was actually a "delete" operation, we don't want to change the data in the index, we want to call the Algolia client and delete the object with the matching ID to the ID that was passed in. If it was a create or update, then we need to call the Algolia client to save the object value that was provided.

if (operation === "delete") {
  // Handle delete operation
  await algoliaClient.deleteObject({
    indexName,
    objectID: _id,
  });
  console.log(`Deleted object with ID: ${_id}`);
  return Response.json({
    message: `Successfully deleted object with ID: ${_id}`,
  });
} else {
  // Add or update the document in Algolia
  await algoliaClient.saveObject({
    indexName,
    body: {
      ...value,
      objectID: _id,
    },
  });

  console.log(`Indexed/Updated object with ID: ${_id}`);
  return Response.json({
    message: `Successfully processed document with ID: ${_id}!`,
  });
}

Step Five: Create a New Sanity Webhook

The first thing that needs to be done is accessing the area where webhooks are defined in your Sanity project. Follow these steps:

Log in to your Sanity account (https://www.sanity.io/manage)
Select the project from your dashboard
From the menu provided, click on the API option. By default this will load up the GROQ-powered webhooks section.

Now we build up the definition of the webhook, but we'll leave out some of the full pieces (like Projection) for later. When you are doing this in the future, you'll probably do all of it in one step, but for the purpose of this tutorial we'll walk through it a little more slowly.

In the GROQ-powered webhooks section, click on "Create webhook" button. This will take you to the webhook creation form.
Name: Provide a descriptive name that will be used when displaying the webhook in your Sanity settings.
- Example: "Algolia Indexing".
Description: Like the name, this helps users identify what the webhook is for when looking at a list of web hooks.
- Example: "This hooks into a serverless function to keep an Algolia index up to date"
URL: This is the endpoint route that you created earlier. This will receive the information from Sanity and then upload the information to Algolia.
- Example: https://example.vercel.app/api/algolia
Dataset: This specifies which of your collections of data you want this webhook to execute on. In the context of Algolia, you might have different indexes with different data sets to represent dev or testing environments. However, in my case, I only want this running against my production data.
- Example: production
Trigger on: This is the type of action that will trigger the webhook. The default is only set to have "Create" enabled. In our scenario, with Algolia indexing, we need it updated on Create, Update, and Delete to make sure our index removes elements or updates them whenever a change is made.
- Example: ✅ Create ✅ Update ✅ Delete
Filter: This restricts which types of documents in your data store will trigger the webhook. For example, you might only be searching blog articles and products and don't want marketing landing pages returned.
- Example: _type == 'post'
Projection: This is how we'll send the data over to our API endpoint, we'll set that up later.
Status: Because we don't have the projection configured yet, we should disable this for now.
Advanced Settings: For most learning scenarios, you won't need this. However, if you have any sort of security on your API endpoint you'll likely need to add some headers or a secret to lock down your webhook. In this tutorial, we will be setting up a secret for the webhook.
- Example: my-algolia-secret
Hit the Save button to create your webhook.

Step Six: Configure the Webhook Projection

As mentioned previously, you can setup the Projection field on the webhook during initial creation, but for the purpose of the tutorial I wanted to break it out so we can step through all the pieces of the projection. The Projection takes the data from Sanity and transforms it into the format that your API will accept.

In the GROQ-powered webhooks section, click the Edit webhook button next to your Algolia Indexing webhook that you just created. Scroll down to the Projection field and paste in this code as your starting point.

``` javascript
{
  "transactionId": _rev,
  "projectId": sanity::projectId(),
  "dataset": sanity::dataset(),
  _id,
  // Returns a string value of "create", "update" or "delete" according to which operation was executed
  "operation": delta::operation(),
  // Define the payload
  "value": {
    "objectID": _id,
    "title": title,
    "slug": slug.current,
    // Portable text
    "body": pt::text(body[0...15]),
    "_type": _type,
    "image": image.asset->url,
    "tags": tags,
    "categories": categories,
    "publishedAt": publishedAt,
    "_createdAt": _createdAt,
    "_updatedAt": _updatedAt
  }
}
```

***ℹ️ *In your solution, you may want to put your webhook projection in source control to keep track of change history in case something happens in your project.

Breaking down the Projection

We're going to break down the different pieces of this code to understand what's happening.

transactionId, projectId, dataset, _id: These are all the basic record and database identifiers that identify what triggered the webhook to fire. You don't have to edit these from the samples and can use as is.
operation: This identifies what type of operation (a create, update, or delete) triggered the webhook. This is helpful in case you have different logic to perform. In our case for the Algolia index, we'll use this to know whether to update the index or to remove a record from the index.
value: This is where the real meat of the projection happens. The keys on the left are the fields in the Algolia index that you want to update, and the values on the right are the data pieces you will pull from the Sanity content lake record. Note, for example, that I am truncating the 'body' field and only grabbing a portion of the content to keep my record from getting too big in the index. Also, for the image, I am only sending over the URL and not other information about the asset. This is where you'll be manipulating the projection the most, especially as you add more fields to Sanity that you want to also add to your Algolia index.

Step Seven: Add the Secret and Enable the Webhook

Before closing down the webhook, we should also secure it and turn it on.

Status: Make sure that 'Enable webhook' is checked so that the webhook will be turned on.
Secret: At the very beginning, when we setup environment variables, there is a secret value that was put into the SANITY_WEBHOOK_SECRET setting. This should be dropped in here.
- Example: my-algolia-secret (please don't use this value, try to have something much more unique and secure)
Save the webhook.

Step Eight: Test the Webhook

️It's important to test your webhook after installing and configuring and make sure it is actually running and successfully updating Algolia. Don't just hope!

In your Sanity Studio, find one of your documents that is supposed to be listened to by your webhook.
Make a change to the document and then publish it.
Switch back to your Sanity management dashboard (https://www.sanity.io/manage) and navigate to the API settings section, if you aren't already there.
Next to your Algolia Indexing webhook is the "Edit webhook" button and an ellipse menu. Click on the ellipse to expand the additional options for the web hook.

Click on the "Show attempt log" menu option. This will pop open a new tab that shows a raw log of the webhook runs. If your webhook succeeded its call, you should see a message that contains "Successfully processed document with ID:".
You can also go to your application host and view the logs there to see how your API route logged the request. For example, I use Vercel for my hosting, so in my Project's Logs section I filter by 'algolia' to find the Algolia API route. I can then inspect the details of the payload sent over the wire to validate that the projection is working correctly and sending the correct information.

Credits and References

How to implement front-end search with Sanity (sanity.io): This guide really helped me and informed a lot of the steps that are seen here. Many of the example code pieces in my article are from this Sanity guide.
Sanity Algolia Next.js Example (GitHub): This is the code that I started from (matches to the tutorial). I found it didn't quite pass my linting so my version has some slight differences, but you might be able to work directly from this.
sanity-algolia plugin: While you won't need this plugin in more recent Sanity versions and webhooks, this page has a decent walkthrough of some pieces you'll be working with.
Banner image credit: Banner created by Jason St-Cyr using logos from Sanity, Algolia, and Twilio's webhooks logo.