Virtual Entity : Chapter 1 – Definition and Usage

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,

• Chapter 1 - Core concept and definition (Current)
• Chapter 2 - Setting up custom data source and custom provider and hands on code
• Chapter 3 – Introducing searching capability
• Chapter 4 – Storing the settings for the api.

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter 1

Virtual Entities have been available for quite some time now and although I knew conceptually what they were I had not had a chance to use them. Recently I had to look at them for a project at work and I was a bit surprised to find that there aren’t many blogs about them. I will rectify that.

In the first part of the blog, I will start by defining various components of the virtual entity stack. I will then follow up the core concepts with some actual real-life examples. I will include links to a code repository and a packaged solution that could be imported into Dynamics 365 to see the inner workings in action.

What is a Virtual Entity?

The actual definition of a Virtual Entity can be found in this MS document. My own definition of it is not very different. However, before starting to define it in my own terms, let me give you common case study:

Dave the developer, comes to the office one morning. Before going to his desk, he makes himself the best mug of coffee. He is thrilled about the promise of the day. Today he is going to finish the project that he started few weeks ago but never got around to finishing. He even charged his noise cancelling headphone to max so that he can exclude himself from all the usual distractions of the office. Alas the solace never lasts that long, in comes the project manager.

PM : Hey Dave, I have this ultra-super max priority work that I need you look at atm.

Dave : Err… did you not want me to finish this work that I started looking at couple of days ago.

PM : Yeah! But this work is even a bigger priority.

Dave : Huh! Okay, so what’s the work?

PM : Management wants to show a list of ingredients for cakes in the brand new cake web side.

Dave : But the ingredients database in not in Dynamics 365.

PM : Okay, can you not import the data in to Dynamics?

Dave : Of course I can. I can create an ingredients table in Dynamics and create a SSIS package to import the data from the ingredients database into Dynamics.

PM : Oh I forgot, as you know ingredient changes all the time for cakes. So we need the data to be always updated.

Dave : Okay, I can run the SSIS package every 5 mins and scan for updated, newly created ingredients and update Dynamics 365 with the data. But for that we need an SQL database server to run the SSIS package in timely manner. You can also do it in azure but it is not free.

PM : Oh I forgot, since they are now going through this security stuff now, the ingredients organization will not permit us to use their database directly. They are creating this API and we can only get data from the API.

Dave : Okay, SSIS will not be a wise idea then. Maybe Azure Data Factory or Azure Logic Apps then?

PM : Erm one tiny problem. We do not have money to have either and management wants this done tomorrow.

Dave gets up bashes his head against the wall.

Except for the extremities, above is a classic example for a data migration/integration project. Even if you take the complexity and technology out of the equation you still need to answer questions like How often do you keep the data updated? What happens if an update sequence fails? It all boils down to the fact that maintaining one source of truth is far better than maintaining multiple sources. If you want to consider a single source of truth as a rule of thumb for your project, then all you have to answer is how do you present the data with possibly limited customization.

Virtual Entities answer the above question with relative ease. My definition of a Virtual Entity is as follows:

A Virtual Entity is a process by which Dynamics 365 can present data located in a third-party data store to the Dynamics’ application users. This means with a Virtual Entity the data remains in its original location but gets a 'view' inside Dynamics 365. Since the data is not getting copied over there is no need to run expensive integration that synchronizes data. With certain limitations the data is query-able through advanced find, through SDK calls and meta data calls. However, the data remains in a read only state while using virtual entity through Dynamics 365. 

Components

There are three major components of Virtual Entities.

Data Source

From a high-level point of view data source denotes the source of data. However, for Dynamics 365 a Data Source is basically the information about the source of the data. Such information might include, for OData service, the base url, query string, api header. For a custom provider it may include the column set, connection string etc. Interestingly you cannot create a data source from within a solution. You can only add an existing data source. A Data Source can be created by going to Settings > Administration > Virtual Entity Data Sources. Once the Data Source is created in can be added to a solution and exported to different instances.

Data Provider

This is defined as the component the connects to a Data Source, fetches the data, transforms the data and supplies it to Dynamics. By default, Dynamics 365 ships with ODataProvider. There is a Cosmos Data Provider that is available on the Microsoft App Source which I have not tested. Developers can also create custom Data Providers of their own. Regardless of the provider, the data that gets presented to the Dynamics needs to have a key field and key needs to have a globally unique Id. If you are working on a non OData API and need to give a user access to any data using a Virtual Entity, an option can be to use Custom Data Provider as a transformation layer and add a Guid column on each row.

Virtual Entity

A Virtual entity is the record type that has the same columns as those that come from the Data Provider. A Dynamics user can flag an Entity to be a Virtxual Entity when they are creating an Entity in customization. For a Virtual Entity that depends on an out of the box OdataProvider by Microsoft, mapping for entity name, fields and data type are very strict. I am going to discuss custom data provider in following chapters. For custom data provider some of the rules does not apply. Refer to this document for more information know how on OOB odata provider.

One thing worth noting, while creating a custom data provider both data source and Virtual Entity are created with the Virtual Entity flag set to true.