Edit

Share via

Determine integration requirements

Successful integration design starts with understanding three foundational dimensions: volume and frequency, directionality, and capability. These dimensions help you evaluate business requirements, system constraints, and scalability needs.

For example, imagine that you have a high-level goal such as connecting SAP to Dataverse or sending a notification every time there's an update to a case a user works on. What is the starting point to design the integration?

The first step is to deconstruct the requirements into the three key components of an integration:

  • Volume and frequency is the first major component in the decision-making process. It helps determine the kinds of tools you need to use to implement the business requirements.

  • Directionality, the next component, addresses where data flows from and to. Understanding directionality helps you set up the pattern for a successful integration.

  • Capability, or the ability of each system to receive, process, and send data, is the final step. Use the "weakest link" approach to evaluate capabilities to identify limitations and possibilities.

Volume and frequency

This dimension defines how much data is transferred and how often. Volume and frequency work together to shape integration architecture. While they might appear similar, they influence solution design in distinct ways. The following components explain in detail how volume and frequency interact and affect integration decisions.

Compare volume versus frequency

Two integration scenarios might involve the same total volume, such as 60,000 records per hour and 1,000 records per minute, but differ in frequency. While both equal the same hourly volume, the minute-by-minute expectation changes the solution design.

  • Don't assume one solution fits both.
  • Validate the system’s ability to handle higher-frequency load.
  • Consider building separate solutions if one pattern is resource-intensive or rarely used.

Trigger types

Triggers define how and when integrations run. Choose the right trigger based on predictability and system load.

Scheduled triggers (also known as "Batch"):

  • Run at fixed intervals.
  • Easier to predict and manage.
  • Suitable for stable data growth patterns.

Event-driven triggers: An event can be a button select, a change in a record in one of the systems, or an API call.

  • Launch based on user actions or system events.
  • Harder to predict.
  • Can spike unexpectedly, especially in public-facing systems.

Seasonality

Data volume fluctuates with business cycles. Plan for seasonal spikes in both scheduled and event-driven integrations.

  • Monthly or quarterly billing cycles might cause predictable surges.
  • Tax season or public service deadlines can create unpredictable spikes.
  • Implement safeguards to prevent overload during peak periods.

Stakeholder collaboration

Discuss volume and frequency with process owners and business users. Validate assumptions against actual workflows.

  • Business users might not know the full process.
  • Architects must investigate and confirm operational realities.

Plan for the future

Design integration solutions with growth in mind.

  • Define operating conditions clearly.
  • Include long-term scalability plans.
  • Estimate when scaling is required.

Directionality

Directionality defines the flow of data between systems. Define where data originates and where it's delivered to shape how you configure and execute the integration. When determining the directionality of data flow, account for system availability, compliance requirements, and security measures to ensure reliable and secure operations. For example, the data might come from a private system that isn't always available or might be subject to strict compliance and security regulations.

Stakeholders and compliance

Compliance plays a critical role in integration design and varies across systems. Consult with infrastructure architects and security officers to ensure connections meet organizational security and regulatory standards.

  • High-security environments often impose strict access controls that influence integration architecture.
  • Legacy on-premises systems might restrict inbound connections. In such cases, design the integration so the legacy system initiates communication with the cloud application.

Capability

Integration performance depends on the capability of each system involved. The weakest system in the chain limits the overall result.

  • Evaluate system capabilities against business requirements.
  • Identify bottlenecks that might affect high-frequency or large-volume data transfers.
  • Consider enhancements if a system can't meet performance expectations.

Capability and frequency

Frequency impacts how well a system handles data transfers. A system that performs well once a day might fail under multiple daily loads.

  • Match system capability to the required frequency.
  • Don't assume that volume alone determines feasibility.

Caching

Caching is a common solution when a system can't meet performance requirements.

  • Use tools like Azure Synapse Link for Dataverse to replicate data into scalable storage.
  • Understand the trade-off: caching improves response time but might deliver outdated data.
  • Ensure data remains fresh to prevent inaccurate results in real-time processes.

Transformation and business logic

System capability includes the ability to perform necessary transformations and business logic to satisfy business requirements.

  • Assess what each system can do before, during, and after data transfer.
  • Consider the complexity of source data, transformation needs, and target system processing.

For example, exporting a SQL view with stored procedures to Dataverse might require mid-flight adaptation and post-arrival plugin execution.

Capability stakeholders

System administrators provide insight into system capabilities. Engage with centralized or decentralized IT teams to validate assumptions.

  • Evaluate each system before selecting an integration pattern.
  • Confirm that technical capabilities align with business expectations.

Put it all together

Effective integration design begins with understanding three core components. To summarize:

  • Volume and frequency define how much data is transferred and how often. These metrics influence the choice of tools, performance expectations, and scalability planning.
  • Directionality identifies the source and destination of data. It helps determine how data flows between systems and ensures compliance with security and regulatory requirements.
  • Capability measures each system’s ability to send, receive, and process data. It highlights performance limitations and helps identify potential bottlenecks in the integration process.

Each component maps directly to initial business requirements. Analyze, together with stakeholders, how volume, frequency, directionality, and capability affect the overall integration process.

Stakeholder collaboration is essential during analysis. Their input can reshape the integration approach.

  • Process owners provide initial business requirements.
  • Infrastructure architects and security officers ensure compliance and secure connectivity.
  • System administrators evaluate system capabilities and constraints.

Example scenario

Let's put everything together through an example scenario. Imagine that the business requirement is to create an integration process that keeps case information in sync between an external customer and internal service engineers who work on cases. Customers can add comments to a case through a website, while engineers can add case information through a Power App.

Request volume and trigger frequency

Volume and frequency determine how much data the system transfers and how often it does so. In this scenario, customers primarily drive case creation, so the volume depends on the number of customers the company serves and its projected growth trajectory.

The total volume of updates might be calculated as:

[Customers] × [Cases per customer] × [Average updates per case]

Visualize this number on a chart to show how it grows over time. For example, if you start with 10 million updates per year and expect a 20% increase each year, the chart should show a steady rise in updates year over year.

Diagram of a line chart showing the number of requests per year, with a steady upward trend based on projected 20% annual growth.

Use historical data and growth projections to estimate future load. For example, if the system handles 10 million updates per year today and grows at 20% annually, the integration must support 25 million updates per year over five years.

Frequency analysis shows monthly peaks. If current demand is 3.2 million requests per month, future demand might reach 8 million per month. Design the integration to meet these performance thresholds.

To ensure the integration remains effective over a typical five-year return-on-investment (ROI) period, design the solution to support at least 25 million requests per year. This capacity planning accounts for projected growth and helps the solution remain scalable and reliable as business needs evolve.

The frequency part of volume is the ability of the systems involved handling information within a year. Again, we can chart historical data to understand how frequency applies.

Diagram of request density over a year, highlighting monthly peaks and projected growth trends.

Directionality and data flow

Directionality defines the flow of data between systems. This scenario includes four distinct data streams:

  • A data stream from the website to write case updates into Dataverse
  • Another stream for the website to read updates from Dataverse
  • A third data stream where engineers write updates into Dataverse from the Power App
  • A final data stream to read updates in the Power App

This diagram illustrates the direct integration pattern, showing how data moves between the website, Dataverse, and the Power App through four distinct data streams:

Diagram illustrating four data streams: website to Dataverse, Dataverse to website, engineers to Dataverse, and Dataverse to the Power App.

Understanding these flows helps you configure secure and efficient integrations. Use direct or decoupled patterns based on system capabilities and performance needs.

Capability in action

In this example integration, built-in connectors streamline the process. When retrieving case information from Dataverse, apply filters and set request limits to optimize data retrieval and display only the necessary data in the app. For the website, publish endpoints using Power Automate HTTP triggers to enable reading and writing of data. Evaluate the capacity of both Power Automate flows and Dataverse to ensure they support projected loads. Review the limits of automated, scheduled, and instant flows to avoid exceeding platform constraints.

Use Dataverse Analytics to monitor current usage. If Dataverse approaches its projected request load, consider adding a protective buffer in the form of Azure Data Lake.

This diagram shows the decoupled read pattern, where a Data Lake is introduced between Dataverse and the website to offload read traffic and improve scalability:

Diagram of a website integration pattern showing a decoupled read pattern with the addition of Azure Data Lake.

This strategy helps reduce read volume from Dataverse and prevent throttling errors (such as HTTP 429 Too Many Requests).

To decrease the dependency even further, decouple the create and update requests from the website using a queuing service such as Azure Service Bus.

This diagram shows the fully decoupled integration pattern, where both reads and writes are offloaded through a Data Lake and a queue to maximize reliability and protect Dataverse from spikes in demand:

Diagram of a website integration pattern showing a fully decoupled design with the addition of a queuing service.

Design cloud flows to handle errors, implement retry logic, and follow best practices for reliability. When selecting an integration pattern, prioritize solutions that meet business needs with minimal complexity. Balance technical capability with cost, licensing, and maintenance requirements. Choose the simplest approach that fulfills requirements and avoids unnecessary investment.

Next step

Explore common patterns to translate your requirements analysis into practical, scalable integration architectures.

Explore integration patterns

  • Last updated on