May be a noob question, but I am not totally convinced every I talk about Queuing service like kafka, SQS. Why do I not tocuh the source itself to pull into map reduce or spark streaming engine etc.

For eg, if I am ingesting logs into Queue which then consumed by spark streaming consumer, why can't I do that directly between spark and log source itself.

I understand it when there are multiple consumers and Queue acts as interface to offload processing from the source but what's the selling point if it is point-to-point communication.

submitted by /u/abhi5025

Source link

No tags for this post.


Please enter your comment!
Please enter your name here