Use webhooks with camera inference¶

Glossary

Webhook is a mechanism for transmitting real-time data from one application to another in response to specific events. This way, every time a camera inference makes a prediction, Robovision AI automatically communicates with a third-party service or custom application via HTTP POST requests.

Key features¶

Event-driven – Webhooks are triggered by events, which eliminates the need for continuous polling.
Efficient – Only relevant data is sent, making webhooks lightweight and fast.
Customizable – Webhooks can be configured to send specific event data (for example, when an inference detects a specific class).

How it works¶

Setup

Using webhooks is optional. If you choose to use a webhook, you need to provide a webhook URL (endpoint) for your camera inference. You can use a third-party service or create a custom application. For testing, you can use tools like Pipedream RequestBin, which will generate a webhook URL for you and allow you to view the data sent by Robovision AI.

Optionally, you can send the image along with the prediction. For more details, see Set up camera inference.
Trigger

When the camera detects an event, Robovision AI sends an HTTP POST request containing the event data to the specified webhook URL. Currently, each prediction made by the camera inference triggers the webhook.
Response

The receiving application processes the data and returns a response, such as an HTTP 200 OK status. Robovision AI logs all responses received from the webhook.

Limitations¶

If the requests to the webhook fail with error codes, these failures are logged but not otherwise acted upon. You can download the inference logs to help troubleshoot the issue.
If the receiving application cannot process requests quickly enough, the inference may begin to stall while waiting for the receiver to catch up. This can result in some frames being missed. This issue can be addressed by reducing the camera's FPS or optimizing the receiving application's performance.
For the instance segmentation tasks with high throughput and a large number of predictions, sending masks via webhook may be inefficient.

Webhook formats¶

During camera inference, the webhook endpoint receives a request for each image. The request's body is HTTP multipart encoded. Note that sending the image with the prediction is optional, so the image file component may be missing from the request body. Each project type has its specific request body.

Object detection¶

Components of the multipart request body:

(Optional) Image file analyzed by the algorithm.
- File name: image.jpeg
- MIME type: image/jpeg

Predictions file with metadata about the detected objects.

File name: predictions.json
MIME type: application/json

Example of JSON content:

{
    "kind": "ObjectDetection.v1",
    "detections": [
        {
            "centerX": 0.0,
            "centerY": 0.0,
            "orientation": 0.0,
            "height": 128.0,
            "width": 128.0,
            "classId": "detection",
            "confidence": 0.42
        }
    ]
}

Image classification¶

Components of the multipart request body:

(Optional) Image file analyzed by the algorithm.
- File name: image.jpeg
- MIME type: image/jpeg
Predictions file with metadata about the classification result.
- File name: predictions.json
- MIME type: application/json
Example of JSON content:
```
{
    "kind": "ImageClassification.v1",
    "classId": "detection",
    "confidence": 0.42
}
```

Multilabel image classification¶

Components of the multipart request body:

(Optional) Image file analyzed by the algorithm.
- File name: image.jpeg
- MIME type: image/jpeg

Predictions file with the detected labels for the image.

File name: predictions.json
MIME type: application/json

Example of JSON content:

{
    "kind": "MultiLabelImageClassification.v1",
    "labels": [
        {
            "classId": "detection",
            "confidence": 0.42
        }
    ]
}

Instance segmentation¶

Components of the multipart request body:

(Optional) Image file analyzed by the algorithm.
- File name: image.jpeg
- MIME type: image/jpeg
Mask files highlighting specific areas of the image corresponding to the detected instances (one file per mask).
- File name: mask_0000.png
- MIME type: image/png

Predictions file with metadata about each segmented instance: the reference to its mask file, the class label, and the confidence score.

File name: predictions.json
MIME type: application/json

Example of JSON content:

{
    "kind": "InstanceSegmentation.v1",
    "segments": [
        {
            "fileKey": "mask_0000",
            "fileName": "mask_0000.png",
            "classId": "detection",
            "confidence": 0.42
        }
    ]
}

Semantic segmentation¶

Components of the multipart request body:

(Optional) Image file analyzed by the algorithm.
- File name: image.jpeg
- MIME type: image/jpeg
Mask files indicating regions in the image corresponding to specific classes (one file per mask). Each pixel in the mask is associated with a class label.
- File name: mask_0000.png
- MIME type: image/png

Predictions file with metadata about each segmented region: the reference to its mask file and the class label.

File name: predictions.json
MIME type: application/json

Example of JSON content:

{
    "kind": "SemanticSegmentation.v1",
    "segments": [
        {
            "fileKey": "mask_0000",
            "fileName": "mask_0000.png",
            "classId": "detection"
        }
    ]
}