librdkafka not able to consume zstd compressed messages

[saksham-arya-z]

While sending messages, to librdkafka v1.6.2 zstd compression causes messages not be consumed while snappy works
logs:

%7|1736951549.320|FETCH|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Topic .ads-service.ad-billing [0] in state active at offset 167 (0/100000 msgs, 0/1048576 kb queued, opv 2) is fetchable: 
%7|1736951549.320|FETCHADD|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Added .ads-service.ad-billing [0] to fetch list (1 entries, opv 2)
%7|1736951549.320|FETCH|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Fetch topic .ads-service.ad-billing [0] at offset 167 (v2)
%7|1736951549.320|FETCH|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Fetch 1/1/1 toppar(s)
%7|1736951549.320|SEND|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Sent FetchRequest (v4, 93 bytes @ 0, CorrId 28)
%7|1736951549.321|RECV|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Received FetchResponse (v4, 420 bytes, CorrId 28, rtt 1.04ms)
%7|1736951549.321|FETCH|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Topic .ads-service.ad-billing [0] MessageSet size 347, error "Success", MaxOffset 168, Ver 2/2
%7|1736951549.321|PROTOERR|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Protocol parse failure at 142/420 (rd_kafka_msgset_reader_msg_v2:643) (incorrect broker.version.fallback?)
%7|1736951549.321|PROTOERR|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: expected 18446744073709551607 bytes > 278 remaining bytes
%7|1736951549.321|CONSUME|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Enqueue 0 message(s) (0 ops) on .ads-service.ad-billing [0] fetch queue (qlen 0, v2, last_offset -1)
%7|1736951549.321|BACKOFF|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: .ads-service.ad-billing [0]: Fetch backoff for 500ms: Local: Bad message format
%7|1736951549.321|FETCH|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Topic .ads-service.ad-billing [0] in state active at offset 167 (0/100000 msgs, 0/1048576 kb queued, opv 2) is not fetchable: fetch backed off
%7|1736951549.321|FETCHADD|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Removed .ads-service.ad-billing [0] from fetch list (0 entries, opv 2)
%7|1736951549.321|FETCH|rdkafka#consumer-1| [thrd:b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2]: b-2.broker.jhr5t6.c4.kafka.ap-south-1.amazonaws.com:9092/2: Fetch backoff for 499ms