Spark 3.5 - Cosmos OLAP read fails with "Session isn't active" while working in Spark 3.4

Question

Spark 3.5 - Cosmos OLAP read fails with "Session isn't active" while working in Spark 3.4

Victor Buzy 20

🔹 Description

We are currently migrating workloads from Azure Synapse Spark 3.4 to Spark 3.5.

We are experiencing an issue when reading data from Azure Cosmos DB using Synapse Link (cosmos.olap).

The exact same code works perfectly in Spark 3.4, but consistently fails in Spark 3.5.

🔹 Minimal Repro Code

df = spark.read \

🔹 Observed Behavior

Spark session starts successfully:
- Apache Spark session started... Running
- Immediately after executing the first action:

InvalidHttpRequest:

🔹 Expected Behavior

The notebook should successfully read data from Cosmos DB as it does in Spark 3.4.
df.limit(1).show() should return data.

🔹 Additional Context

The issue occurs even with minimal code (no transformations).
The issue seems specific to cosmos.olap (Synapse Link).
SQL-based notebooks work correctly in Spark 3.5.
The same Cosmos notebook works without any issue in Spark 3.4.
Spark session appears to start but becomes inactive immediately afterward.
🔹 Description
We are currently migrating workloads from Azure Synapse Spark 3.4 to Spark 3.5. We are experiencing an issue when reading data from Azure Cosmos DB using Synapse Link (cosmos.olap). The exact same code works perfectly in Spark 3.4, but consistently fails in Spark 3.5.
🔹 Minimal Repro Code
```
  df
```
🔹 Observed Behavior
- Spark session starts successfully:
- Apache Spark session started... Running
Immediately after executing the first action:

  InvalidHttpRequest:

## 🔹 Expected Behavior
```  -  The notebook should successfully read data from Cosmos DB as it does in Spark 3.4. 

- `df.limit(1).show()` should return data. 

    ## 🔹 Additional Context

  -  The issue occurs even with minimal code (no transformations). 
  
  -  The issue seems specific to `cosmos.olap` (Synapse Link). 
  
  -  SQL-based notebooks work correctly in Spark 3.5. 
  
  -  The same Cosmos notebook works without any issue in Spark 3.4. 
  
  -  Spark session appears to start but becomes inactive immediately afterward.
  
## 🔹 Error Trace ID


```dockerfile
e9c2162f-93e2-46a0-ac70-91bc47fde292

🔹 Environment Details

Spark Version: 3.5
Node Size: (ex: Medium 8 vCores)
Autoscale: *(ex: 1–10 nodes)
*See picture attached

🔹 Impact

This issue is blocking migration of Cosmos-based workloads to Spark 3.5.

To recap: To clarify the issue, here is the clean minimal repro and context.

Description:

We are migrating workloads from Azure Synapse Spark 3.4 to Spark 3.5.

We are experiencing an issue when reading data from Azure Cosmos DB using Synapse Link (cosmos.olap).

The exact same code works correctly in Spark 3.4, but consistently fails in Spark 3.5.

Minimal repro code:

df = spark.read \

.format("cosmos.olap") \

.option("spark.synapse.linkedService", "DATABASE") \

.option("spark.cosmos.container", "TABLE") \

.load()
```df.limit(1).show()

Observed behavior:

- Spark session starts successfully and shows as Running

- Immediately after executing the first action, the notebook fails with:

InvalidHttpRequest: Submission failed due to error content =["requirement failed: Session isn't active."]

HTTP status code: 400

Trace ID: e9c2162f-93e2-46a0-ac70-91bc47fde292

Expected behavior:

- The notebook should successfully read data from Cosmos DB in Spark 3.5, as it does in Spark 3.4

- df.limit(1).show() should return data

Additional context:

- The issue occurs even with minimal code and no transformations

- The issue seems specific to cosmos.olap (Synapse Link)

- SQL-based notebooks work correctly in Spark 3.5

- The same Cosmos notebook works without any issue in Spark 3.4

- Spark session appears to start, then becomes inactive immediately afterward

Environment:

- Spark version: 3.5

- Node size: Medium (8 vCores / 64 GB)

- Autoscale: Enabled (3 to 10 nodes)

- Dynamic allocation: Enabled

- Default executors: 1 to 4

- Python version: 3.11

- Delta Lake version: 3.2

Impact:

This issue is blocking migration of Cosmos-based workloads to Spark 3.5.

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Pilladi Padma Sai Manisha 6,580 Reputation points Microsoft External Staff Moderator

2026-04-01T20:11:24.5033333+00:00

Hi Victor Buzy,
Based on what you’ve described, this behavior appears to be related to a compatibility gap between Spark 3.5 and the Cosmos DB OLAP (cosmos.olap) connector used through Synapse Link. The key observation is that the same code executes successfully in Spark 3.4 but fails in Spark 3.5 with the “Session isn’t active” error as soon as an action is triggered. This typically indicates that the Spark session is being marked inactive during execution, which points to a runtime or connector-level issue rather than anything in your code or configuration.

Given this, the most reliable approach at the moment is to continue running Cosmos OLAP workloads on Spark 3.4, where the connector integration is stable and functioning as expected. This ensures continuity while avoiding the session instability observed in Spark 3.5.

For additional context on how Synapse Link integrates with Spark and Cosmos DB, you can refer to the following documentation:

https://dori-uw-1.kuma-moon.com/azure/cosmos-db/synapse-link
https://dori-uw-1.kuma-moon.com/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark https://dori-uw-1.kuma-moon.com/azure/synapse-analytics/spark/apache-spark-version-support

Your implementation and repro steps look correct, and the behavior you’re seeing aligns with a version-specific limitation rather than a usage issue. Please feel free to share if you notice any changes or need help validating alternative approaches.
Pilladi Padma Sai Manisha 6,580 Reputation points Microsoft External Staff Moderator

2026-04-03T21:53:00.7766667+00:00

Hi Victor Buzy,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Victor Buzy 20 Reputation points

2026-04-06T13:14:38.18+00:00
Hi @Anonymous

Thank you for your response.

However, this situation is blocking our migration strategy.

On one hand, Spark 3.4 is reaching end of support and we are encouraged to migrate to Spark 3.5.

On the other hand, Cosmos OLAP workloads are currently not working on Spark 3.5 and you are advised to remain on Spark 3.4.

This creates a contradiction, as we cannot fully migrate our platform while Cosmos OLAP is not supported or stable in Spark 3.5.

Could you please clarify:

Is this a known limitation or a confirmed bug in Spark 3.5 with cosmos.olap?

Is there an official ETA or roadmap for a fix or support in Spark 3.5?

Are there any recommended workarounds or alternative approaches for Cosmos workloads in Spark 3.5?

Should we expect an upcoming patch or runtime update?

This issue is critical for us as it directly blocks the migration of Cosmos-based workloads.

Thank you for your support.
Pilladi Padma Sai Manisha 6,580 Reputation points Microsoft External Staff Moderator

2026-04-09T13:20:27.87+00:00

Hi Victor Buzy,
Thanks for calling this out , your concern makes complete sense, and I agree this creates a difficult situation from a migration standpoint.

Based on what you’ve shared and the behavior observed, this is best treated as a current limitation with Spark 3.5 when used with the Cosmos OLAP (cosmos.olap) connector via Synapse Link, rather than an issue in your code or configuration. The fact that the same workload works in Spark 3.4 and fails consistently in Spark 3.5 right at execution strongly points to a gap in compatibility at the runtime/connector level.

At the moment, there isn’t any public documentation that confirms full support of Cosmos OLAP workloads on Spark 3.5, and there’s no officially published ETA for a fix yet. These kinds of issues are typically addressed through runtime updates, so it’s reasonable to expect improvements over time, but there isn’t a committed timeline available right now.

In terms of how to move forward, the most practical approach today is to continue running Cosmos OLAP workloads on Spark 3.4 while moving other workloads to Spark 3.5 where possible. I understand that’s not ideal, but it does allow you to keep progressing your migration instead of blocking everything on this dependency.

If you need to explore options within Spark 3.5, you could look at alternative access patterns depending on your use case, for example, using Cosmos DB through transactional APIs or leveraging SQL-based paths that are already working in your environment. This won’t be a like-for-like replacement for Synapse Link OLAP scenarios, but in some cases it can help bridge the gap temporarily.

For background and current support details, these docs may be helpful: https://dori-uw-1.kuma-moon.com/azure/cosmos-db/synapse-link https://dori-uw-1.kuma-moon.com/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark https://dori-uw-1.kuma-moon.com/azure/synapse-analytics/spark/apache-spark-version-support

I know this isn’t the smooth migration path you were expecting, and your feedback here is important. If you want to walk through your workload and see what can safely move to 3.5 versus what should stay on 3.4 for now, I’m happy to help with that.

2 answers

Your answer

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Pilladi Padma Sai Manisha 6,580 Reputation points Microsoft External Staff Moderator

2026-04-01T20:11:24.5033333+00:00

Hi Victor Buzy,
Based on what you’ve described, this behavior appears to be related to a compatibility gap between Spark 3.5 and the Cosmos DB OLAP (cosmos.olap) connector used through Synapse Link. The key observation is that the same code executes successfully in Spark 3.4 but fails in Spark 3.5 with the “Session isn’t active” error as soon as an action is triggered. This typically indicates that the Spark session is being marked inactive during execution, which points to a runtime or connector-level issue rather than anything in your code or configuration.

Given this, the most reliable approach at the moment is to continue running Cosmos OLAP workloads on Spark 3.4, where the connector integration is stable and functioning as expected. This ensures continuity while avoiding the session instability observed in Spark 3.5.

For additional context on how Synapse Link integrates with Spark and Cosmos DB, you can refer to the following documentation:

https://dori-uw-1.kuma-moon.com/azure/cosmos-db/synapse-link
https://dori-uw-1.kuma-moon.com/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark https://dori-uw-1.kuma-moon.com/azure/synapse-analytics/spark/apache-spark-version-support

Your implementation and repro steps look correct, and the behavior you’re seeing aligns with a version-specific limitation rather than a usage issue. Please feel free to share if you notice any changes or need help validating alternative approaches.
Pilladi Padma Sai Manisha 6,580 Reputation points Microsoft External Staff Moderator

2026-04-03T21:53:00.7766667+00:00

Hi Victor Buzy,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Victor Buzy 20 Reputation points

2026-04-06T13:14:38.18+00:00

Hi @Anonymous

Thank you for your response.

However, this situation is blocking our migration strategy.

On one hand, Spark 3.4 is reaching end of support and we are encouraged to migrate to Spark 3.5.

On the other hand, Cosmos OLAP workloads are currently not working on Spark 3.5 and you are advised to remain on Spark 3.4.

This creates a contradiction, as we cannot fully migrate our platform while Cosmos OLAP is not supported or stable in Spark 3.5.

Could you please clarify:

Is this a known limitation or a confirmed bug in Spark 3.5 with cosmos.olap?

Is there an official ETA or roadmap for a fix or support in Spark 3.5?

Are there any recommended workarounds or alternative approaches for Cosmos workloads in Spark 3.5?

Should we expect an upcoming patch or runtime update?

This issue is critical for us as it directly blocks the migration of Cosmos-based workloads.

Thank you for your support.

Answer 1

Hi @Anonymous

Thank you for your response.

However, this situation is blocking our migration strategy.

On one hand, Spark 3.4 is reaching end of support and we are encouraged to migrate to Spark 3.5.

On the other hand, Cosmos OLAP workloads are currently not working on Spark 3.5 and you are advised to remain on Spark 3.4.

This creates a contradiction, as we cannot fully migrate our platform while Cosmos OLAP is not supported or stable in Spark 3.5.

Could you please clarify:

Is this a known limitation or a confirmed bug in Spark 3.5 with cosmos.olap?
Is there an official ETA or roadmap for a fix or support in Spark 3.5?
Are there any recommended workarounds or alternative approaches for Cosmos workloads in Spark 3.5?
Should we expect an upcoming patch or runtime update?
What is the expected sunset date of spark 3.4, will it be removed from the cloud at what point of time?

This issue is critical for us as it directly blocks the migration of Cosmos-based workloads.

Thank you for your support.

Answer 2

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Share via

Spark 3.5 - Cosmos OLAP read fails with "Session isn't active" while working in Spark 3.4

🔹 Description

🔹 Minimal Repro Code

🔹 Observed Behavior

🔹 Expected Behavior

🔹 Additional Context

🔹 Description

🔹 Minimal Repro Code

🔹 Observed Behavior

🔹 Environment Details

🔹 Impact

2 answers

Your answer