Handling Failures
Define any failure handlers for your function with the onFailure
option. This function will be automatically called when your function fails after it's maximum number of retries. Alternatively, you can use the "inngest/function.failed"
system event to handle failures across all functions.
import { inngest } from "./client";
export default inngest.createFunction(
{
id: "import-product-images",
onFailure: async ({ error, event, step }) => {
// This is the failure handler which can be used to
// send an alert, notification, or whatever you need to do
},
},
{ event: "shop/product.imported" },
async ({ event, step, runId }) => {
// This is the main function handler's code
}
);
The failure handler is very useful for:
- Sending alerts to your team
- Sending metrics to a third party monitoring tool (e.g. Datadog)
- Send a notification to your team or user that the job has failed
- Perform a rollback of the transaction (i.e. undo work partially completed by the main handler)
Failures should not be confused with Errors which will be retried. Read the error handling & retries documentation for more context.
How onFailure
works
The onFailure
handler is a helper that actually creates a separate Inngest function used specifically for handling failures for your main function handler.
The separate Inngest function utilizes an "inngest/function.failed"
system event that gets sent to your account any time a function fails. The function created with onFailure
will appear as a separate function in your dashboard with the name format: "<Your function name> (failure)"
.
onFailure({ error, event, step, runId })
The onFailure
handler function has the same arguments as the main function handler when creating a function, but also received an error
argument.
error
The JavaScript Error
object as thrown from the last retry in your main function handler.
The Inngest SDK attempts to serialize and deserialize the Error
object to the best of its ability and any custom error classes (e.g. Prisma.PrismaClientKnownRequestError
or MyCustomErrorType
) that may be thrown will be deserialized as the default Error
object. This means you cannot use instance
of within onFailure
to infer the type of error.
event
The "inngest/function.failed"
system event payload object. This object is similar to any event payload, but it contains data specific to the failed function's final retry attempt. See the complete reference for this event payload below.
step
See the step
reference in the create function documentation.
runId
This will be the function run ID for the error handling function, not the function that failed. To get the failed function's run ID, use event.data.run_id
. Learn more about runId
here.
The "inngest/function.failed"
event
Inngest sends an "inngest/function.failed"
event to your environment whenever any function fails after exhausting all of it's retry attempts. You can use this event as an event
trigger within your own function or use the onFailure
helper option as documented above.
- Name
name
- Type
- string: "inngest/function.failed"
- Description
The
inngest/
event prefix is reserved for system events in each environment.
- Name
data
- Type
- object
- Description
The event payload data.
Properties- Name
error
- Type
- object
- Description
Data about the error payload as returned from the failed function.
- Name
event
- Type
- string
- Description
The failed function's original event payload.
- Name
function_id
- Type
- string
- Description
The failed function's
id
- Name
run_id
- Type
- string
- Description
The failed function's function run ID.
- Name
ts
- Type
- number
- Description
The timestamp integer in milliseconds at which the failure occurred.
Example "inngest/function.failed" event payload
{
"name": "inngest/function.failed",
"data": {
"error": {
"__serialized": true,
"error": "invalid status code: 500",
"message": "taylor@ok.com is already a list member. Use PUT to insert or update list members.",
"name": "Error",
"stack": "Error: taylor@ok.com is already a list member. Use PUT to insert or update list members.\n at /var/task/.next/server/pages/api/inngest.js:2430:23\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async InngestFunction.runFn (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestFunction.js:378:32)\n at async InngestCommHandler.runStep (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestCommHandler.js:459:25)\n at async InngestCommHandler.handleAction (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestCommHandler.js:359:33)\n at async ServerTiming.wrap (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/helpers/ServerTiming.js:69:21)\n at async ServerTiming.wrap (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/helpers/ServerTiming.js:69:21)"
},
"event": {
"data": { "billingPlan": "pro" },
"id": "01H0TPSHZTVFF6SFVTR6E25MTC",
"name": "user.signup",
"ts": 1684523501562,
"user": { "external_id": "6463da8211cdbbcb191dd7da" }
},
"function_id": "my-gcp-cloud-functions-app-hello-inngest",
"run_id": "01H0TPSJ576QY54R6JJ8MEX6JH"
},
"id": "01H0TPW7KB4KCR739TG2J3FTHT",
"ts": 1684523589227
}
Examples
Send a Slack notification when a function fails
In this example, the function attempts to sync all products from a Shopify store, and if it fails, it sends a message to the team's #eng-alerts Slack channel using the Slack Web Api's chat.postMessage
(docs) API.
import { client } from "@slack/web-api";
import { inngest } from "./client";
export default inngest.createFunction(
{
id: "sync-shopify-products",
// Your handler should be an async function:
onFailure: async ({ error, event }) => {
const originalEvent = event.data.event;
// Post a message to the Engineering team's alerts channel in Slack:
const result = await client.chat.postMessage({
token: process.env.SLACK_TOKEN,
channel: "C12345",
blocks: [
{
type: "section",
text: {
type: "mrkdwn",
text: `Sync Shopify function failed for Store ${
originalEvent.storeId
}: ${error.toString()}`,
},
},
],
});
return result;
},
},
{ event: "shop/product_sync.requested" },
async ({ event, step, runId }) => {
// This is the main function handler's code
const products = await step.run("fetch-products", async () => {
const storeId = event.data.storeId;
// The function might fail here or...
});
await step.run("save-products", async () => {
// The function might fail here after the maximum number of retries
});
}
);
Track all function failures in Datadog
You can use the "inngest/function.failed"
event to handle all failed functions in a single place. This can enable you to send metrics, alerts, or events to external systems like Datadog or Sentry for all of your Inngest functions. Here's an example using Datadog's Events API to send all failures the Datadog event stream.
import { client, v1 } from "@datadog/datadog-api-client";
import { inngest } from "./client";
const configuration = client.createConfiguration();
const apiInstance = new v1.EventsApi(configuration);
export default inngest.createFunction(
{
name: "Send failures to Datadog",
id: "send-failed-function-events-to-datadog"
},
{ event: "inngest/function.failed" },
async ({ event, step }) => {
// This is a normal Inngest function, so we can use steps as we normally do:
await step.run("send-event-to-datadog", async () => {
const error = event.data.error;
// Create the Datadog event body using information about the failed function:
const params: v1.EventsApiCreateEventRequest = {
body: {
title: "Inngest Function Failed",
alert_type: "error",
text: `The ${event.data.function_id} function failed with the error: ${error.message}`,
tags: [
// Add a tag with the Inngest function id:
`inngest_function_id:${event.data.function_id}`,
],
},
};
// Send to Datadog:
const data = await apiInstance.createEvent(params);
// Return the data to Inngest for viewing in function logs:
return { message: "Event sent successfully", data };
});
}
);
Capture all failure errors with Sentry
Similar to the above example, you can capture and all failed functions' errors and send them to a singular place. Here's an example using Sentry's node.js library to capture and send all failure errors to Sentry.
import * as Sentry from "@sentry/node";
import { inngest } from "./client";
Sentry.init({
dsn: "https://examplePublicKey@o0.ingest.sentry.io/0",
});
export default inngest.createFunction(
{
name: "Send failures to Sentry",
id: "send-failed-function-errors-to-sentry"
},
{ event: "inngest/function.failed" },
async ({ event, step }) => {
// The error is serialized as JSON, so we must re-construct it for Sentry's error handling:
const error = event.data.error;
const reconstructedEvent = new Error(error.message);
// Set the name in the newly created event:
// You can even customize the name here if you'd like,
// e.g. `Function Failure: ${event.} - ${error.name}`
reconstructedEvent.name = error.name;
// Add the stack trace to the error:
reconstructedEvent.stack = error.stack;
// Capture the error with Sentry and append any additional tags or metadata:
Sentry.captureException(reconstructedEvent,{
extra: {
function_id,
},
});
// Flush the Sentry queue to ensure the error is sent:
return await Sentry.flush();
}
);