Story :- To persist the list of b2c user information with card details who have opted "VISA" cards for purchasing the products.
Algorithm:
a.) Read user information from the MongoDB (DB#1) collection named "users_info" and the collection named "users_cards_info"
b.) Join the two collections information - collection named "users_info" and the collection named "users_cards_info" and then publish the
information into kafka topic.
c.) Spark Structured Streaming as a consumer consumes the information from the kafka topic.
d.) Apply the transformation at the spark layer to filter the user data who are having only of type VISA cards.
e.) Via, Mongo Spark Helper, Filtered user data would be persisted into separate collection of different MongoDB (#2).
Requirement :-
List of b2c user with cards information to be retrieved from MongoDB collection(s) - from the (MongoDB - Database#1) collection named "users_info" and the collection named "users_cards_info". Then, Persist the output (user data who are having only of type VISA cards) into (MongoDB - Database#2 ) collection named - "users_cards_info_list"
Provided algorithm outlines the steps for achieving the desired result of persisting user information with VISA card details from MongoDB Database #1 to MongoDB Database #2 using Kafka and Spark Streaming. Here's a breakdown of each step:
a.) Read user information from MongoDB (DB#1) collection named "users_info" and the collection named "users_cards_info":
In this step, you retrieve user information from two collections in MongoDB Database #1. These collections are "users_info" and "users_cards_info". The user information likely includes details such as names, addresses, and other relevant data.
b.) Join the two collections' information and publish it into a Kafka topic:
After retrieving the user information from both collections, you perform a join operation to combine the relevant data. The purpose of this join is to associate the user information with their corresponding card details. Once the join is complete, you publish the combined information into a Kafka topic. This allows other systems or processes to consume the data from the Kafka topic.
c.) Spark Structured Streaming consumes the information from the Kafka topic:
Spark Structured Streaming, which is a real-time processing engine, acts as a consumer in this step. It consumes the data that was published to the Kafka topic in the previous step. Spark Structured Streaming provides the capability to process the data in a streaming fashion, allowing for real-time analysis and transformations.
d.) Apply transformations at the Spark layer to filter the user data with only VISA cards:
In this step, you apply transformations to the data received from the Kafka topic using Spark. The purpose is to filter out users who only have VISA cards. The filtering condition is likely based on a specific field or attribute in the user's card details. Once the filtering is applied, the resulting dataset will only contain users who meet the criteria of having VISA cards.
e.) Persist the filtered user data into a separate collection in MongoDB Database #2:
Finally, using the Mongo Spark Helper library, you persist the filtered user data into a separate collection in MongoDB Database #2. This separate collection will contain the user information of individuals who have opted for VISA cards for their purchases. The Mongo Spark Helper library provides a convenient way to interact with MongoDB from Spark and enables seamless data transfer between the two systems.
By following these steps, you can achieve the goal of persisting user information with VISA card details from MongoDB Database #1 to MongoDB Database #2, leveraging Kafka for data transportation and Spark Streaming for real-time processing and filtering.
MongoDB configurations, Kafka setup, and data schema. Make sure to replace the placeholder values (localhost:27017, DB1, DB2, /path/to/checkpoint, etc.) with the appropriate values for your setup.
This code assumes that you have the necessary dependencies and libraries set up correctly, including the MongoDB Spark Connector and Kafka integration with Spark. You may need to add the required dependencies to your project configuration.
