Move data by using the Azure Cosmos DB Spark Connector

Azure Cosmos DB is a globally distributed, multi-model database service provided by Microsoft Azure. It offers a variety of APIs and connectors to enable seamless integration with different data sources and platforms. In this article, we will explore how to move data using the Azure Cosmos DB Spark Connector, which allows us to connect Azure Cosmos DB with Apache Spark.

Prerequisites

To get started, you need to have the following prerequisites:

1. An Azure Cosmos DB account: Create a Cosmos DB account using the Azure portal. Make sure to select SQL API as the API type while creating the account.
2. Apache Spark: Install Apache Spark on your development machine or cluster. You can download it from the Apache Spark website.

Using the Azure Cosmos DB Spark Connector

Once you have the prerequisites in place, follow the steps below to use the Azure Cosmos DB Spark Connector:

Step 1: Include the Azure Cosmos DB Spark Connector library

Include the Azure Cosmos DB Spark Connector library in your Spark application. You can add the dependency in your build file or specify it using the --packages option while submitting your Spark job.

For example, if you are using Maven, add the following dependency to your pom.xml file:

com.microsoft.azure
azure-cosmos-spark_2.4.0_2.11
2.4.0

Step 2: Configure the connection settings

Configure the connection settings for Azure Cosmos DB. Specify the Cosmos DB account endpoint, master key, and database name.

val endpoint = “[Cosmos DB account endpoint]”
val masterKey = “[Cosmos DB account master key]”
val database = “[Cosmos DB database name]”

You can find these values in the Azure portal under “Keys” and “Settings” for your Cosmos DB account.

Step 3: Read data from Azure Cosmos DB

Read data from Azure Cosmos DB into Spark RDD or DataFrame using the com.microsoft.azure.cosmosdb.spark package.

Read data into an RDD:

import com.microsoft.azure.cosmosdb.spark._
import com.microsoft.azure.cosmosdb.spark.schema._

val rdd = spark.sparkContext.loadFromCosmosDB(Map(
“Endpoint” -> endpoint,
“Masterkey” -> masterKey,
“Database” -> database,
“Collection” -> “[Cosmos DB collection name]”,
“query_custom” -> “[SQL query to filter data]”
))

Read data into a DataFrame:

import com.microsoft.azure.cosmosdb.spark.config.Config
import com.microsoft.azure.cosmosdb.spark.CosmosDBSpark

val df = spark.read.cosmosDB(Map(
“Endpoint” -> endpoint,
“Masterkey” -> masterKey,
“Database” -> database,
“Collection” -> “[Cosmos DB collection name]”,
“query_custom” -> “[SQL query to filter data]”
))

Step 4: Write data to Azure Cosmos DB

Write data from Spark RDD or DataFrame to Azure Cosmos DB using the saveToCosmosDB method.

Write an RDD to Cosmos DB:

rdd.saveToCosmosDB(Map(
“Endpoint” -> endpoint,
“Masterkey” -> masterKey,
“Database” -> database,
“Collection” -> “[Cosmos DB collection name]”
))

Write a DataFrame to Cosmos DB:

df.write.cosmosDB(Map(
“Endpoint” -> endpoint,
“Masterkey” -> masterKey,
“Database” -> database,
“Collection” -> “[Cosmos DB collection name]”
))

That’s it! You have successfully moved data using the Azure Cosmos DB Spark Connector. You can now leverage the power of Apache Spark to process and analyze the data stored in Azure Cosmos DB.

In conclusion, the Azure Cosmos DB Spark Connector provides a seamless integration between Azure Cosmos DB and Apache Spark, allowing you to read and write data efficiently. By following the steps outlined in this article, you can easily move data between Azure Cosmos DB and Spark RDDs or DataFrames. Start leveraging the combined capabilities of Azure Cosmos DB and Apache Spark to build powerful data-driven applications in the cloud.