React Native Support about kafkajs

[Nevon]

So the simple answer is no. kafkajs will not work in react native, as it uses Node modules such as net, which cannot be fully used in a browser context.

This begs the question though - what problem are you actually trying to solve by using kafkajs on the client side? We did a quick proof of concept by replacing net with a browserified version that relies on a proxy server for socket support, just because the idea was so crazy, and it actually worked. This still requires a server to act as a proxy, as browser don't offer support for raw sockets yet (though there is a draft), but it proves that it isn't impossible to achieve.

[tommymsw]

We are currently encounter the same problem, our front-end is React , which need browser as client to consume kafka topics, and I think it is very nice if the kafkajs can support this function. We want to know do you have any plans to implement this feature? thanks very much.

[Nevon]

I personally think it's a cool idea, but this is not something that is being actively worked on.

We did a hacky PoC just to see if it's possible, and got it working. To elaborate on my previous comment, the problem is that the Kafka protocol is a custom binary protocol over TCP, which requires the client to have a raw socket connection. This is not a capability that the web has. This means that, as @zluo001 mentions, you need to run a proxy server that your client can establish a websocket connection to, which then proxies the messages to Kafka using regular TCP sockets.

For Android and iOS, you do have access to raw TCP sockets, AFAIK, although you'd have to expose that capability to react-native. For web it simply is not possible.

I think it's unlikely that we will support this use-case in core KafkaJS, but if someone who has a vested interest in this establishes and commits to a realistic plan for how to achieve this, I believe that we would just need to expose a pluggable interface for the network layer, which we could theoretically support. Given that, target-specific packages that just swap out the network layer could be built outside of KafkaJS core.

[tulios]

Hi folks, I think this is how we push technology forward, and it is really interesting to see this need. Just highlighting some of the comments, we can make the network module swappable, so it's easier to achieve such a thing, and we can support you with any necessary steps related to KafkaJS.

It shouldn't be too hard on native, but on the web, until we find a way to make TCP requests directly from the browser we will need the proxy to convert the requests, which is totally feasible but requires some more effort.

In 2019 we expect to catch-up with the Kafka APIs, and we are committed to keeping KafkaJS "dependency free" and purely written in javascript.

[joaomilho]

Brilliant

[zluo001]

Maybe it's crazy for running kafka client in browser, but it's really useful. Now days, all the solutions of produce or consume topics in browser require a proxy server. It makes the architecture become complicated and more latency of message delivery, while it led to consume more hardware resources also more money. It should be a tremendous progress in web client if this can be made. Thanks for your great job.

[csnvrl]

Hello, any updates on this ? We are using React Native for mobile development. It would be very nice to implement kafkajs features on this platform.

[Nevon]

We are using React Native for mobile development. It would be very nice to implement kafkajs features on this platform.

That is pretty wild. I would love to hear what use-cases you have that you think directly connecting to Kafka from a mobile device would be a good solution for (this sounds like I'm being sarcastic, but I'm really not).

To my knowledge, no one is working or has offered to work on building a react-native bridge to KafkaJS. In #263 I added the ability to inject your own socket implementation, which should make this possible to do.

[MrHazimAli]

@Nevon , I just started to do some research on integrating kafka with my mobile application. What I need basically just produce and send message to broker which will be handle by server. for that will I able to use this?

I believe it can be done if we use https but that would create another request after a request, that would defeat the benefit of kafka. I would love to know or have a tips to able to product/send message to topic in broker from react-native side.

[Nevon]

No need to overcomplicate things. Just make a request to your back end using whatever protocol you are currently using, and have the server produce the message before responding. It's consuming that's the tricky part.

[MrHazimAli]

I see.. alright. noted @Nevon , will try the suggestion first and see from there. thanks!

[michaelkleinhenz]

@Nevon Maybe I don't get some basic arch considerations on Kafka (yet, I am just starting out with Kafka), but wouldn't a mobile app not treated like any other consumer component in a Kafka setup? Sending and receiving messages directly without using a middleware sounds like a resonable idea. The Kafka use cases also mention mobile apps as consumers.

[Nevon]

wouldn't a mobile app not treated like any other consumer component in a Kafka setup

Yes, they could be. Anything that can create a TCP socket can use Kafka. The issue is that client-side Javascript is not able to create a raw TCP socket. In the case of React Native, you probably could write a module that binds to the native OS to create the socket for you, but it's not something I've seen done (at least not open source).

[tulios]

@michaelkleinhenz, a mobile app could work. This issue points out that even though kafkajs is purely written in javascript and doesn't have any dependencies, it was written to run on nodejs. It depends on standard libraries only available on the node runtime. The biggest barrier is the lack of socket support, but this could work on mobile apps as they have access to real sockets. Another point is that we as the kafkajs core maintainers don't want to add support for browsers or mobile frameworks, at least in the core library, so we usually incentivize people to work with us to achieve something pluggable, that's, for example, how we ended up with a pluggable socket factory among other things.

[michaelkleinhenz]

@tulios ok, thanks. In our use case, the app is done using React Native, so it is not "really" a native app nor a browser application. But thanks for the clarification that is not a fundamental architectural antipattern.