Virtual Entity : Chapter 4 – Pulling API Endpoint Settings from Data Source Entity

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,

• Chapter 1 - Core concept and definition
• Chapter 2 - Setting up custom data source and custom provider and hands on code
• Chapter 3 – Introducing searching capability
• Chapter 4 – Storing the settings for the api. (Current)

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter 4

The last piece of the puzzle would be how to have dynamic API settings for different environments without changing anything in the code.

In Chapter 2 of our journey we added some custom fields in our Data Source, but we have not used them yet. Now since we have connectivity between the API and Dynamics and we can search in advanced find, let’s make the Data Source truly dynamic so that we can set all the API settings and configure them on a per instance basis.

Just to jog your memory our data source record now looks like below,

The Dynamics SDK provides an interface IEntityDataSourceRetrieverService that gives you access to the source of the Entity the Custom Provider that it is attached to. The retriever service is available within the service provider of the plugin. 

First of all in my plugin code I add the following two lines,

I can then update my method that sets the settings for the api. It changes like below

So there you have it, we have created a Virtual Entity, Custom Provider, Data Source and then learned how to query and the configure the source so that we can have different API connections in our different instances. The code is available in this Github repo along with the solution. Have fun!

Virtual Entity : Chapter 3 – Translating an advanced find query to a query understood by the Data Source

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,

• Chapter 1 - Core concept and definition
• Chapter 2 - Setting up custom data source and custom provider and hands on code
• Chapter 3 – Introducing searching capability (Current)
• Chapter 4 – Storing the settings for the api.

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter - 3

So now we have set up our Virtual Entity and it is connected to a Custom Data Provider. The Data Provider is getting data from third-party Data Source in an advanced find view. We can also open the record and see the details in a form.

Here are some problems that we have not yet discussed 

• Can we run a query on the advanced find view so that only selected records are returned?
• What happens if we have different Dynamics instance (like dev-test-uat-production) and we have custom API environment for each? Do we need to register a Custom Provider like I did in Part 1 and Part 2 all over again for each environment?

This chapter will answer the first of the above questions. Ready?

Visitor Pattern and Query Expression Visitor

I am not going to go into huge detail about the visitor pattern. I found that this article explains the whole pattern very clearly. Let me, however, give you some idea of the problem we are trying to solve here, in our case we are exposing external data from and API. You might want to expose data directly from an SQL server or may be from even an azure logic app. So your users in CRM need be able to construct a query in Dynamics in fetch xml in advanced find. When the user clicks on search this comes to your retrieve multiple plugin. It becomes the responsibility of your plug in to translate the fetch xml (or query expression) to something that your data source can understand. This is where visitor pattern comes it. Newly introduced Query Expression Visitor class in the CRM SDK leverages this pattern to translate fetch queries to the query that the data source API will understand. The translation code still needs to be written by you. The pattern is there to encapsulate it. A similar example of this can be found in this MS doc I recommend reading and understanding the article in the first link for the explanation of the pattern before proceeding any further with this.

Code Example

So first up I am going introduce two new methods in my CountryGet class that will search countries by capital and search countries by region. They are quite self-explanatory. The former searches all the countries by given capital while the later searches country by region. After adding the code to call the apis my class for calling the API looks like below,


Now comes the visitor class – very straight forward nothing super crafty about it.


My Retrieve Multiple now ties everything up. Notice that I have also introduced tracing.


I have cleaned the code just to be very nit-picking, I introduced factory pattern in the country get like so,


For that my visitor is slightly changed as well.


And my Retrieve Multiple is a breeze


As you could rightly say by now formatting the query for a Custom Provider is not that hard, but it is a bit of work nonetheless. You should consider allowing support only for certain operator like I did on the code above. Like you should consider rejecting any queries that has date operator. The problem also become some order of magnitude harder when the user use combination of filter criteria.
You should also consider allowing search on the field that you support. In this way you can give searchability when it is relevant.

Virtual Entity : Chapter 1 – Definition and Usage

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,

• Chapter 1 - Core concept and definition (Current)
• Chapter 2 - Setting up custom data source and custom provider and hands on code
• Chapter 3 – Introducing searching capability
• Chapter 4 – Storing the settings for the api.

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter 1

Virtual Entities have been available for quite some time now and although I knew conceptually what they were I had not had a chance to use them. Recently I had to look at them for a project at work and I was a bit surprised to find that there aren’t many blogs about them. I will rectify that.

In the first part of the blog, I will start by defining various components of the virtual entity stack. I will then follow up the core concepts with some actual real-life examples. I will include links to a code repository and a packaged solution that could be imported into Dynamics 365 to see the inner workings in action.

What is a Virtual Entity?

The actual definition of a Virtual Entity can be found in this MS document. My own definition of it is not very different. However, before starting to define it in my own terms, let me give you common case study:

Dave the developer, comes to the office one morning. Before going to his desk, he makes himself the best mug of coffee. He is thrilled about the promise of the day. Today he is going to finish the project that he started few weeks ago but never got around to finishing. He even charged his noise cancelling headphone to max so that he can exclude himself from all the usual distractions of the office. Alas the solace never lasts that long, in comes the project manager.

PM : Hey Dave, I have this ultra-super max priority work that I need you look at atm.

Dave : Err… did you not want me to finish this work that I started looking at couple of days ago.

PM : Yeah! But this work is even a bigger priority.

Dave : Huh! Okay, so what’s the work?

PM : Management wants to show a list of ingredients for cakes in the brand new cake web side.

Dave : But the ingredients database in not in Dynamics 365.

PM : Okay, can you not import the data in to Dynamics?

Dave : Of course I can. I can create an ingredients table in Dynamics and create a SSIS package to import the data from the ingredients database into Dynamics.

PM : Oh I forgot, as you know ingredient changes all the time for cakes. So we need the data to be always updated.

Dave : Okay, I can run the SSIS package every 5 mins and scan for updated, newly created ingredients and update Dynamics 365 with the data. But for that we need an SQL database server to run the SSIS package in timely manner. You can also do it in azure but it is not free.

PM : Oh I forgot, since they are now going through this security stuff now, the ingredients organization will not permit us to use their database directly. They are creating this API and we can only get data from the API.

Dave : Okay, SSIS will not be a wise idea then. Maybe Azure Data Factory or Azure Logic Apps then?

PM : Erm one tiny problem. We do not have money to have either and management wants this done tomorrow.

Dave gets up bashes his head against the wall.

Except for the extremities, above is a classic example for a data migration/integration project. Even if you take the complexity and technology out of the equation you still need to answer questions like How often do you keep the data updated? What happens if an update sequence fails? It all boils down to the fact that maintaining one source of truth is far better than maintaining multiple sources. If you want to consider a single source of truth as a rule of thumb for your project, then all you have to answer is how do you present the data with possibly limited customization.

Virtual Entities answer the above question with relative ease. My definition of a Virtual Entity is as follows:

A Virtual Entity is a process by which Dynamics 365 can present data located in a third-party data store to the Dynamics’ application users. This means with a Virtual Entity the data remains in its original location but gets a 'view' inside Dynamics 365. Since the data is not getting copied over there is no need to run expensive integration that synchronizes data. With certain limitations the data is query-able through advanced find, through SDK calls and meta data calls. However, the data remains in a read only state while using virtual entity through Dynamics 365. 

Components

There are three major components of Virtual Entities.

Data Source

From a high-level point of view data source denotes the source of data. However, for Dynamics 365 a Data Source is basically the information about the source of the data. Such information might include, for OData service, the base url, query string, api header. For a custom provider it may include the column set, connection string etc. Interestingly you cannot create a data source from within a solution. You can only add an existing data source. A Data Source can be created by going to Settings > Administration > Virtual Entity Data Sources. Once the Data Source is created in can be added to a solution and exported to different instances.

Data Provider

This is defined as the component the connects to a Data Source, fetches the data, transforms the data and supplies it to Dynamics. By default, Dynamics 365 ships with ODataProvider. There is a Cosmos Data Provider that is available on the Microsoft App Source which I have not tested. Developers can also create custom Data Providers of their own. Regardless of the provider, the data that gets presented to the Dynamics needs to have a key field and key needs to have a globally unique Id. If you are working on a non OData API and need to give a user access to any data using a Virtual Entity, an option can be to use Custom Data Provider as a transformation layer and add a Guid column on each row.

Virtual Entity

A Virtual entity is the record type that has the same columns as those that come from the Data Provider. A Dynamics user can flag an Entity to be a Virtxual Entity when they are creating an Entity in customization. For a Virtual Entity that depends on an out of the box OdataProvider by Microsoft, mapping for entity name, fields and data type are very strict. I am going to discuss custom data provider in following chapters. For custom data provider some of the rules does not apply. Refer to this document for more information know how on OOB odata provider.

One thing worth noting, while creating a custom data provider both data source and Virtual Entity are created with the Virtual Entity flag set to true. 

Customer Data type in Dynamics 365

Dynamics 365 has introduced a new data type for custom fields. its called Customer. When new field is created with this data type, the new field will act as parent customer id field in account/contact. Meaning there will be a look up created on the entity that can look up to both contact and account. Sweetness!!