Edit

Share via


Audit columns in Copy job

This article describes the audit columns capability in Copy job and how to use it.

What are audit columns?

Audit columns are additional metadata columns that Copy job can automatically append to every row it writes to the destination. These columns aren't from your source data—they're generated by the platform to describe the data movement itself.

When you enable audit columns in Copy job, each row in your destination table can be enriched with information such as:

Audit column What it captures
Data extraction time The timestamp when the row was extracted from the source by a Copy job run.
File path The source file path the row was read from (applicable for file-based sources).
Workspace ID The Fabric workspace ID where the Copy job resides.
Copy job ID The unique identifier of the Copy job item.
Copy job run ID The unique identifier of the specific Copy job execution.
Copy job name The name of the Copy job that moved the row.
Lower bound The lower bound value of the incremental window for the current run.
Upper bound The upper bound value of the incremental window for the current run.
Custom A user-defined static value. Add any additional context your team needs, such as a source server name.

With audit columns enabled, you can answer the following questions for any row in your destination table:

  • When was this data extracted? Exact timestamp from when the row was read from the source.
  • Where did it come from? Which file path and which data store.
  • Which job moved it? Which Copy job from which workspace, which specific run, by name and ID.
  • What is the incremental scope? Lower and upper bounds tell you exactly what slice of data a run covers.

No custom code or expression authoring is required. Add as many audit columns as you want, and every row in every table your Copy job writes includes this metadata automatically.

Note

Audit columns are supported on all Copy job connectors except Snowflake, Office 365, and Databricks Delta Lake.

Why audit columns matter

Row-level data lineage without the plumbing

Data lineage—knowing where data came from and how it got to its current state—is a foundational requirement for any serious data platform. But traditional approaches to lineage tracking typically operate at the job level: you know that a job ran at a certain time and wrote to a specific table. What you don't know is which specific rows were affected, when each row was written, or whether a particular row came from this run or a previous one.

Audit columns close this gap by embedding lineage metadata directly into the data. Every row carries its own provenance. This is row-level lineage—and it's fundamentally more useful than job-level metadata for debugging, auditing, and data quality.

Compliance and regulatory reporting

For organizations in regulated industries—financial services, healthcare, insurance, and government—the ability to trace data back to its source isn't optional. Auditors need to answer questions like:

  • "When was this customer record last updated in our analytics system?"
  • "Can you prove this financial transaction data was sourced from the production ERP system?"
  • "Which data movement job brought this patient record into the data warehouse, and when?"

Without audit columns, answering these questions requires correlating external monitoring logs with destination table contents—a manual, error-prone, and time-consuming process. With audit columns, the answers are in the data itself. A simple SELECT query on the destination table tells you everything you need.

Data quality and debugging

When data quality issues appear—duplicate rows, stale data, or missing records—the first question is always: when did this row arrive, and where did it come from? Audit columns answer that instantly.

Without audit columns, you'd need to cross-reference workspace monitoring logs, match timestamps against row counts, and hope the correlation holds. With audit columns, the metadata is right there in the row.

Downstream analytics and freshness tracking

Audit columns enable a class of downstream analytics that would otherwise require custom engineering:

  • Source file traceability: Use the file path audit column to trace any row back to its exact source file.
  • Ingestion SLA monitoring: Compare the data extraction time against expected schedules to detect ingestion delays.
  • Incremental window auditing: Use the lower bound and upper bound to verify that every incremental slice was processed and nothing was missed.

How to use audit columns

Enabling audit columns in a Copy job is straightforward and only takes a few steps.

Step 1: Create or open a Copy job

In your Fabric workspace, start by creating a new Copy job or opening an existing one. Choose your source tables as you normally would.

Step 2: Add audit columns

In the Copy job setup, after selecting the source tables or folders to be copied, add audit columns. This automatically adds metadata columns to every destination table in your job.

Screenshot of configuring Audit Columns in Copy job.

Step 3: Run your Copy job

Run the Copy job. With each execution, every row written to the destination table includes audit column values such as extraction time, workspace ID, Copy job name, run ID, and any custom metadata you've defined.

Step 4: Query your data and build reports

Open your destination table and query the audit columns along with your business data for a complete context. Audit columns are standard table fields, so they work seamlessly with Power BI, KQL queries, and other tools. You can build dashboards for data freshness, monitor ingestion SLAs, and create compliance lineage reports without relying on external metadata stores.

Screenshot of showing Audit Columns created by Copy job.