Hot Iterative Read Operations : Definition and how to manage them?

Question

Hot Iterative Read Operations : Definition and how to manage them?

javier cohenar 211

Posted this question on SO but with no answers so far, so I try it here hoping more luck.

We are trying to identify the operations behind the item 'Hot Iterative Read Operations' in our Azure Data Lake Gen2 containers as this seems to be our main cost driver.

What is the definition of Hot Iterative Read Operations? Our understanding is that the more files are being read (for instance when appending data to an existing file, or just simply reading it), the higher the number of Hot Iterative Read Operations. Is this correct? However we noticed that Hot iterative Read Operations are also performed during Sundays and bank holidays (null user activity) which led me to think that they are related to non user activity
When exploring the $logs container we found that most records refer to two operations: ReadFile and GetFileProperties. Are those operations linked to Hot Iterative Read Operations? Are there any other?
Is there any other way to identify the operations under the 'Hot Iterative Read Operations' invoice item?
When we use the Copy Activity in ADF (filtered by last modified date), are we incurring in Hot Iterative Read Operations?

ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator

2023-10-23T07:30:59.8833333+00:00

Hi javier cohenar, Just checking if below answer helps. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.

1 answer

Your answer

ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator

2023-10-23T07:30:59.8833333+00:00

Hi javier cohenar, Just checking if below answer helps. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.

Answer 1

Vahid Ghafarpour 23,595 Volunteer Moderator

I hope this KQL can help you to find the reason for HIRO.

// Query to identify operations contributing to Hot Iterative Read Operations
AzureDiagnostics
| where Category == "AzureDataLakeStorage"
| where OperationName == "ReadFile" or OperationName == "GetFileProperties" // Operations of interest
| summarize TotalOperations = count() by OperationName, bin(TimeGenerated, 1d)
| order by TimeGenerated asc

ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator

2023-10-20T17:33:22.1766667+00:00

Hi javier cohenar, Just checking if above answer helps. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.
Andrew Freeman 0 Reputation points

2025-12-10T13:19:13.62+00:00
Is the above query correct? According to the MS documentation I've seen, interactive read operations have an operation name of List* :

https://azure.github.io/Storage/docs/analytics/azure-storage-data-lake-gen2-billing-faq/#what-apis-are-considered-iterative-read-operations
What APIs are considered Iterative read operations? #

Iterative read operations are operations performed on a folder that requires the system to iterate through all the subfolders and files in that folder to complete. Examples: ListFileSystem, ListFileSystemDir,ListPath and all List* operations

Share via

Hot Iterative Read Operations : Definition and how to manage them?

1 answer

Your answer