The Data Management Framework (DMF) in Dynamics 365 Finance and Operations (FnO) is a comprehensive toolset designed to simplify data transfer, integration, and management. It serves as a vital mechanism for importing, exporting, transforming, and synchronizing data, enabling businesses to handle large data volumes while ensuring accuracy and consistency efficiently.
Key Features of DMF
Streamlined Data Handling: DMF uses data entities as intermediaries, making it easier to migrate, integrate, and manage data across environments.
Versatile Functionality: DMF facilitates:
Data migration between companies or environments.
Real-time service-based integration.
Exporting data for advanced reporting and analysis.
User-Friendly Access: Available via Workspaces > Data Management or System Administration > Workspaces > Data Management, users must “Refresh” the entity list on first use to populate entities for import/export operations.
Data Validation and Mapping: Ensures data integrity by validating field mappings and verifying data types. Errors in staging data, such as mismatches or invalid values, are flagged for correction.
Entity Templates: Predefined configurations bundle related entities for everyday tasks. For example, a customer data template includes customer details, addresses, and contacts.
Managing Data Projects
Creating and Executing Data Projects
DMF allows users to define, manage, and execute data import/export processes through data projects. Key steps include:
Setup:
Define project categories.
Identify and sequence entities logically.
Decide on the use of staging tables.
Execution:
Validate source-target mappings.
Import/export jobs.
Review job history to confirm success.
Use Cases:
Data migration during implementation phases.
Synchronizing data between disparate systems.
Composite Entities for Complex Scenarios
Composite entities combine multiple related entities into a single structure for:
Batch processing of hierarchical data structures.
Simplified handling of complex datasets, such as sales orders with headers, lines, and attachments.
Example: A composite entity for sales orders bundles order headers, line items, and shipping details for streamlined data management.
Steps to Build Composite Entities
Create Individual Entities: Define related entities.
Establish Relationships: Link parent-child entities.
Create Composite Entity: Use the designer mode to embed child entities under a parent entity.
Set Up Relationships: Define runtime relationships and indexes for performance.
Test and Validate: Import/export data to ensure accuracy.
Virtual Entities for Real-Time Integration
Virtual entities dynamically fetch data from external sources, enabling seamless real-time integration. Key benefits include:
No Data Duplication: Access external systems directly.
Unified Views: Display data from multiple sources in one interface.
Enhanced Analytics: Combine external and Dynamics 365 data for comprehensive dashboards and reports.
Use Cases:
Real-time ERP data access.
Displaying inventory from external warehouse systems.
Creating real-time data-driven dashboards.
Advanced Integration Scenarios
Azure Integration
Integrating Dynamics 365 with Azure enables advanced analytics, real-time synchronization, and external data enrichment. For example, Azure Functions can export data from Dynamics 365 to Azure Blob Storage or perform transformations before reimporting data.
Steps:
Configure Azure Functions with HTTP triggers.
Authenticate with Dynamics 365 using connection strings.
Process data dynamically for seamless operations.
BYOD (Bring Your Own Database)
BYOD enables exporting Dynamics 365 data to external SQL databases for reporting and integration purposes. Features include:
Full or incremental data exports.
Batch scheduling for periodic exports.
Access via T-SQL for custom reporting.
Steps:
Create an SQL database in Azure.
Configure export settings and validate connections in DMF.
Publish entity schemas and set up change tracking for incremental updates.
Considerations:
Avoid concurrent exports for the same entity to prevent data loss.
Optimize performance using clustered column store indexes.
Address limitations such as exporting composite entities individually.
Conclusion
The Data Management Framework in Dynamics 365 Finance and Operations is an indispensable tool for efficient data management. Its robust capabilities in data import/export, integration, and migration ensure accuracy and consistency across systems. Enhanced by virtual entities and Azure integration, DMF provides real-time data access and seamless interoperability.
Mastering DMF empowers businesses to unlock the full potential of their Dynamics 365 environment, driving efficiency and enabling data-driven decisions. For expert assistance or further guidance, reach out to our team. We’re here to help!
Microsoft Fabric: A Transformative Analytics Platform for Power BI Users
What is Microsoft Fabric for Power BI service users?
As a Power BI user, you might have looming questions about Microsoft Fabric and how they work together. This blog answers all your questions and offers a comprehensive overview of Fabric and the Power BI along with their integration. The blog also includes insights into migration, licensing, and how Microsoft Fabric complements Power BI.
You will also find answers to burning questions like:
Are these two different tools?
Do I have to migrate from Power BI to Fabric?
Is Fabric complicated?
How do I buy Fabric?
How do I log in?
Power BI as a standalone service
The Microsoft Power BI service started as a standalone SaaS offering to combine, explore, and visualize data. The same service is now a component of Microsoft Fabric. You can continue to use Power BI by itself or use the new capabilities of Fabric with Power BI to do even more with your data.
Overview of Power BI
Power BI is a business intelligence platform used to visualize data to discover insights and make business decisions. Power BI simplifies data analysis and decentralizes BI. Creators ingest, integrate, combine, and model data. When data is shared with consumers, consumers use visualizations to make business decisions, track key metrics, and dig into the data to discover other business insights.
Overview of Fabric
Fabric is an all-in-one data analytics solution that covers everything from data movement to data science, real-time analytics, and business intelligence. Fabric consolidates several Azure data workloads with Power BI so that you and your team can work with different workloads in different roles in the same environment. With Fabric, you can go from raw, fragmented data to meaningful insights across your organization and in other clouds in seconds.
The data for these workloads is consolidated in a multi-cloud data lake called OneLake. Each Fabric tenant has one OneLake, and all Fabric services work natively with data in OneLake. This copy of the data can be used across Fabric. This centralized storage eliminates duplication, maintains a single data source, simplifies data discovery, and enforces security and data protection. The data is saved in a way that makes it accessible to the different workloads in the necessary format.
AI capabilities are seamlessly embedded within Fabric, eliminating the need for manual integration. With Fabric, you can easily transition your raw data into actionable insights.
What about licenses?
If you have a Power BI or Microsoft 365 account, you can sign in to Fabric. New users can create the first item in Fabric, and Fabric auto-turns you into a free trial. You only need a Fabric (Free) license for all Fabric. Fabric is yours to explore at no cost. Signing up takes only a few clicks.
It’s important to note that while a Fabric (Free) license provides access to many features and specific capabilities, especially those related to Power BI may require additional licensing.
Migrating from Power BI to Fabric.
If you already use the Power BI service, you can now access Microsoft Fabric and its additional workloads and capabilities. There’s no need to migrate; all your existing Power BI content is automatically available in Fabric. Workspaces and reports remain intact and accessible as before. The Power BI user interface and capabilities remain unchanged.
Start exploring Fabric – Azure services users will find Fabric familiar as it incorporates Azure Data Factory and Azure Data Explorer within the Fabric SaaS platform. Existing Power BI users will also recognize the familiar foundation, user interface, and navigation of Power BI within Fabric.
Shared concepts and administration – Fabric and Power BI share foundational elements like workspaces, reports, and capacities. Power BI and Fabric administration are centralized in the Fabric admin center.
How Microsoft Fabric works with Power BI
Microsoft Fabric provides a unified platform that integrates data, roles, and workloads seamlessly. Power BI is one of the key workloads within this ecosystem. Central to Microsoft Fabric is OneLake, a single data repository for each tenant. This centralized data store enables effortless use, analysis, and visualization of data across all workloads and roles, streamlining analytics across multiple datasets.
Large organizations benefit significantly from Fabric’s ability to manage vast data volumes and make them accessible across its suite of tools, including Power BI. While Microsoft Fabric now oversees Power BI administration, familiar tools like the Power BI service and Desktop remain unchanged.
They remain robust solutions for transforming data from sources like OneLake or Excel into actionable business intelligence insights. Power BI solutions consist of numerous development stages and efforts, all performed within the Power BI Desktop or the Power BI service. It is worth noting that the data ingestion and transformation stages are all contained within Power BI. Various practices and techniques to optimize and then scale a solution that processes these actions and runs entirely within the power BI service.
By introducing Fabric, some processing can occur outside of Power BI, earlier in the process – in a tightly integrated and managed environment. In this diagram, the data ingestion and transformations occur before the data gets to Power BI. Data shaping and transformation could happen in a Spark notebook or Gen2 dataflow, where it is written to a lakehouse. A semantic model is built from the lakehouse structured data storage. In this case, the semantic model (what we used to call a Power BI “dataset”) exists both in Fabric and Power BI.
Every Fabric lakehouse has a corresponding semantic model that utilizes the Power BI in-memory analytic engine, so in a sense, the model is both a Fabric and a Power BI asset. The semantic model, report management, and user access are managed in Fabric along with everything else using role-based Entra security (aka “AAD or Windows security”).
Do you Need to Make Changes?
While it’s true that you don’t need to make any changes, it’s worth considering the new and improved features offered by Fabric. Power BI can continue to function as it has in the past, with Import mode models and DirectQuery mode being fully supported. However, transitioning to the Fabric platform opens up a world of new possibilities. Let’s explore how this transition can benefit you. Consider the solution components in the following diagram encompassing Power BI portions that we might consider transitioning to Fabric.
The source connections enable connectivity from data sources to the Power Query queries that run within our Power BI model. After executing all the transformation steps in these queries, the resulting tables are loaded into an Import-mode data model. Measures in the model perform calculations for the visuals and reports pages on the reports.
When moving the Power Query queries from Power BI to a Fabric Gen2 Dataflow in the Fabric solution, you’ll find that the transition is smoother than you might expect. The new Dataflows in Fabric use the same query structures, language, and visual designer as Power BI, ensuring that little, if anything, might need to change. If the data sources are the same, queries could be copied and pasted from Power BI to the data flow, making the transition even easier.
A significant difference between Power Query in Power BI Desktop and Gen2 Dataflows in Fabric is that each query can have a destination. A query can be outputted to an Azure SQL database, a Fabric lakehouse, a KQL database, or a Fabric warehouse.
Conclusion
Microsoft Fabric represents a transformative step for Power BI users, offering an integrated environment that combines advanced data analytics capabilities with familiar tools. With the inclusion of features like OneLake for centralized data storage and seamless AI integration, Fabric empowers users to unlock deeper insights while maintaining the simplicity and efficiency of Power BI. Existing users benefit from a smooth transition, as there is no migration required—workspaces, reports, and tools remain consistent and accessible.
