Skip to Main Content

MongoByte MongoDB Logo

Welcome to the new MongoDB Feedback Portal!

{Improvement: "Your idea"}
We’ve upgraded our system to better capture and act on your feedback.
Your feedback is meaningful and helps us build better products.

Status Submitted
Categories Kafka Connector
Created by Guest
Created on Jan 4, 2021

Get schema validation "feedback" in Kafka Mongo Sink Connector

Objective : We want to be able to validate that data matches some requirements. We would like to to perform this data validation by adding a JSON schema in Mongo (such as it is described here : https://docs.mongodb.com/manual/core/schema-validation/). Problem is that current implementation of the current Mongo DB Kafka Sink connector does not implement the required elements to benefit from features brought by this KIP : https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors So if we define such a validation on Mongo, if a message has a value that does not match the definition, it would not go in the dead letter queue, and the connector would "die". We would like to enhance the current implementation of the Mongo Sink connector to benefit from it.
  • Guest
    Jan 11, 2021
    Hi Sebastien, we filed https://jira.mongodb.org/browse/KAFKA-183. This improvement will allow any errors including those caused by $jsonSchema validation to put placed in the DLQ. Thank you for your feedback!