Virtual Entity : Chapter 4 – Pulling API Endpoint Settings from Data Source Entity

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,

• Chapter 1 - Core concept and definition
• Chapter 2 - Setting up custom data source and custom provider and hands on code
• Chapter 3 – Introducing searching capability
• Chapter 4 – Storing the settings for the api. (Current)

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter 4

The last piece of the puzzle would be how to have dynamic API settings for different environments without changing anything in the code.

In Chapter 2 of our journey we added some custom fields in our Data Source, but we have not used them yet. Now since we have connectivity between the API and Dynamics and we can search in advanced find, let’s make the Data Source truly dynamic so that we can set all the API settings and configure them on a per instance basis.

Just to jog your memory our data source record now looks like below,

The Dynamics SDK provides an interface IEntityDataSourceRetrieverService that gives you access to the source of the Entity the Custom Provider that it is attached to. The retriever service is available within the service provider of the plugin. 

First of all in my plugin code I add the following two lines,

I can then update my method that sets the settings for the api. It changes like below

So there you have it, we have created a Virtual Entity, Custom Provider, Data Source and then learned how to query and the configure the source so that we can have different API connections in our different instances. The code is available in this Github repo along with the solution. Have fun!

Virtual Entity : Chapter 3 – Translating an advanced find query to a query understood by the Data Source

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,

• Chapter 1 - Core concept and definition
• Chapter 2 - Setting up custom data source and custom provider and hands on code
• Chapter 3 – Introducing searching capability (Current)
• Chapter 4 – Storing the settings for the api.

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter - 3

So now we have set up our Virtual Entity and it is connected to a Custom Data Provider. The Data Provider is getting data from third-party Data Source in an advanced find view. We can also open the record and see the details in a form.

Here are some problems that we have not yet discussed 

• Can we run a query on the advanced find view so that only selected records are returned?
• What happens if we have different Dynamics instance (like dev-test-uat-production) and we have custom API environment for each? Do we need to register a Custom Provider like I did in Part 1 and Part 2 all over again for each environment?

This chapter will answer the first of the above questions. Ready?

Visitor Pattern and Query Expression Visitor

I am not going to go into huge detail about the visitor pattern. I found that this article explains the whole pattern very clearly. Let me, however, give you some idea of the problem we are trying to solve here, in our case we are exposing external data from and API. You might want to expose data directly from an SQL server or may be from even an azure logic app. So your users in CRM need be able to construct a query in Dynamics in fetch xml in advanced find. When the user clicks on search this comes to your retrieve multiple plugin. It becomes the responsibility of your plug in to translate the fetch xml (or query expression) to something that your data source can understand. This is where visitor pattern comes it. Newly introduced Query Expression Visitor class in the CRM SDK leverages this pattern to translate fetch queries to the query that the data source API will understand. The translation code still needs to be written by you. The pattern is there to encapsulate it. A similar example of this can be found in this MS doc I recommend reading and understanding the article in the first link for the explanation of the pattern before proceeding any further with this.

Code Example

So first up I am going introduce two new methods in my CountryGet class that will search countries by capital and search countries by region. They are quite self-explanatory. The former searches all the countries by given capital while the later searches country by region. After adding the code to call the apis my class for calling the API looks like below,


Now comes the visitor class – very straight forward nothing super crafty about it.


My Retrieve Multiple now ties everything up. Notice that I have also introduced tracing.


I have cleaned the code just to be very nit-picking, I introduced factory pattern in the country get like so,


For that my visitor is slightly changed as well.


And my Retrieve Multiple is a breeze


As you could rightly say by now formatting the query for a Custom Provider is not that hard, but it is a bit of work nonetheless. You should consider allowing support only for certain operator like I did on the code above. Like you should consider rejecting any queries that has date operator. The problem also become some order of magnitude harder when the user use combination of filter criteria.
You should also consider allowing search on the field that you support. In this way you can give searchability when it is relevant.

Virtual Entity : Chapter 2 – Create a Custom Data Provider for a Virtual Entity

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,

• Chapter 1 - Core concept and definition
• Chapter 2 - Setting up custom data source and custom provider and hands on code (Current)
• Chapter 3 – Introducing searching capability
• Chapter 4 – Storing the settings for the api.

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter - 2

Recap and Some More Explanation

Following on from the definition page, in this chapter I would like to discuss how can we create and register a Custom Data Provider. Before that let me define a Custom Data Provider in more detail.

As discussed in the previous chapter that Dynamics 365 ships with an ODataProvider for Virtual Entities. If your API does not happen to support OData, you can create your own provider that reads, transforms and present the data to Dynamics. These are called Custom Data Providers.

What are the components of a Custom Data Provider? Excellent question! Under the hood Custom Data Providers are Pre-Operation plugins that have steps for Retrieve and Retrieve Multiple. Consider you are opening a first class Entity record in Dynamics CE. In order to find the record in question you send in the primary key to database via a Retrieve message. In the case of advanced find or looking for more than one records, you come up with a query and send query wrapped inside a Retrieve Multiple. Your Virtual Entities will follow exactly the same mechanism, the only difference here is that while doing the Retrieve or Retrieve Multiple call, you have code inside a plugin step that performs an API call to a third party datastore. Once the query returns you map it to your Virtual Entity and finish your plugin. It’s a very straight forward process, if you think about it. Makes you wonder why no one thought of it before!

Hands on Tutorial

Registration and configuration of Virtual Entity Data Provider requires a lot of back forth work between the plugin registration tool and Dynamics 365. Here are the rough steps that we need to follow in order to register a working Custom Data Provider.

• Step 1 : Create and register Retrieve and Retrieve Multiple plugin signature steps for Custom Provider. Although we cannot create a full working provider yet as the column definition of the Virtual Entity does not exist, this is where we start.
• Step 2 : Register the Custom Data Provider and Data Source with Plugin registration tool.
• Step 3 : Make additional configuration change for the source in Dynamics 365.
• Step 4 : Create the Virtual Entity in the Dynamics 365. Also create the columns in the Dynamics
• Step 5 : Finish the Custom Data Provider code and update the assembly in plugin registration tool.

First of all, Even before we begin, if you are using older version of the plugin registration tool, it will not work. It is crucial to have the latest version of the tool found on the Microsoft docs site. The one that comes with the SDK download did not work for me. I went to the above-mentioned site and ran the code in Powershell and after a while all the tools were installed in the stipulated directory.

Also all the code used in the following sections is available for you in the Github repository. I have also included the Dynamics 365 with the code in the release section.

Step 1 : Creation and Registration of the plugin

This part is easy. I have created the shell of two plug in classes. One plugin is for Retrieve and the other for Retrieve Multiple. Apart from their names, they look identical.

We will have to register the plugin as normal assembly in the plugin registration tool. I am expecting the readers will have understanding of how to register a plugin. If this is something very new to you, please refer to this ms doc. It lists down every little detail for plugin registration.

Step 2 : Register Custom Data Provider and Data Source

Microsoft need to do a bit of work here (but stick with it!) Occasionally the plugin registration tool sometimes breaks even when the registration is successful! I have pointed out where it breaks and what to do when the registration tool crashes. The crashing was very frequent when I was trying custom provider couple of months ago. However, when at the time of writing this demo the registration tool has not crashed once. May be the trial instance that I was preparing this demo in has some updates that my organisation’s develop instance do not have. Refers to this document for more information for crash.

To register a new Custom Data Provider a click on Register button on the plugin registration tool again. And now select Register New Data Provider.

The below screen will be rendered. Complete all the fields (they are all mandatory)

My settings for Provider registration was as follows.

Settings Name	My Value	Explanation
Data Provider Name	Country Provider	Free text. Think of something that sensibly represents your Data Provider.
Solution	Look for the solution in question, My setting was Custom Provider	You can create your own solution in this screen but I prefer to create the solution beforehand.
Assembly	Codebug.CountryProvider	The newly registered assembly in the Step 1
Retrieve		The class in the above assembly meant to handle Retrieve message.
Retrieve Multiple		The class in the above assembly meant to handle Retrieve Multiple message

I have skipped the settings while setting up data set registration. When you load the screen for the first time the look up will be blank and you can select the look up and can add a new entity in the following screen. 

The new screen contains nothing but prompts for Name(s) of the data source entity and prompt for the solution where the user would like the data source entity to sit. Again, the names fields are free text and user can put anything sensible in the fields.

In most cases at this point you will be presented with an error when they click on create button. In addition to previous link that I shared on this topic it is explained beautifully in the following blog post. I am not going to bore you with details but long story short the data source entity has actually been got created. You only need to restart the plugin registration tool and go back to the Register New Data Provider screen again. This time you do not have to create a data source entity since you could see the data source entity in the drop down. You now just need to fill in the rest of the details. 

Step 3 : Make Additional Change to the Data Source

I have discussed that the Data Source entity contains information about the source of the data rather than data itself. In the previous step we have created data source entity. We now need to make some additional customization in datasource in order to make everything work. These customizations will basically be new field creation. So that we can, for instance, store the main part of the url of the API that we are using. The API that we will be using using for this example uses a couple of set headers as well. We will create fields in the custom datasource so that we can store the keys and values for it. Last customization will be changing the form of the custom data source so that we can view the information. Let us go and do that.

Data source entity is created as a Virtual Entity in Dynamics meaning all the constraints for the Virtual Entity exists when you create a Data Source as well. When you create new field on the above Data Source entity you need to provide an external name as well. These external names for Custom Data Source fields do not have any usage but are required.

After the customization the Data Source Entity will be ready to use and you can set one up. You can view the data source by going to,

Setting > Administration > Virtual Entity Data Sources

Once there click on new button and select the Data Provider that you have registered with plugin registration tool. The record will look like below for our use case (I have added data to the page as well),

Step 4 : Create the Virtual Entity

I am planning to get country data from third party API. I have added the important information that I need in the previous screen shot. In case if you missed it the details can be found in the api description page. As you might have guessed, I am planning to expose list of countries in Dynamics through virtual entity. For the rest of the blog I am using two API end points,

• Get All countries – which will show all countries in an advanced find and in associated view
• Get by Country code – this will get a single country. This will be called when an entity form is opened. 
• Search by Region – it will bring up all the countries in a region. I will use it to show advanced find capability.
• Search By Capital - another search functionality. This time with capital of a country.

The primary object (you guessed it right) is a country.  Here’s how one object looks like on the API.

As you can see, the data structure is quite complex. I am going to normalize the object a little bit so that the deserialization becomes an easy task. One of the key things you might also have noticed is that this data structure does not give us an identifiable key. I am going to use Alpha3Code as the key and will hold a key-value dictionary inside the plugin which will give me a GUID when a code is supplied. I am sure you could think of ten other ways to craftily achieve the same thing! For this example, I consider having a dictionary inside the plugin code as simple enough. While we are on the topic, here’s how my model class look like inside the plug-in assembly,


Based on the above model I am going to create a new Entity, country as a Virtual Entity. My Entity definition looks like below. 

For Custom Data Provider and Data Source External Name and External Collection name do not matter, the only requirement is they need to be populated. This is true for the external name for the field definition as well.

Here is how the Entity field look like when all the field creation is finished.

Step 5 : Finish the Plugin Provider Code

With Retrieve Multiple, you can do some nifty things with a visitor pattern to actually query for specific country. However, to keep the discussion on point, in this example, I am not including anything on that yet. So we will just be running a GET request and displaying the result in a Retrieve Multiple or a Retrieve step. The detailed code example is available on github.

Below is the class that is responsible for the api requests,

And below is how the plugins look like after we have done the code change

As mentioned before, RetrieveMultipleCountry is getting called when we are opening and advanced find view and trying to search for all country. RetrieveCountry will get called when user tries to open a single record. We are taking the entity id (Guid) translating it to 3 character country code, searching the country the code and then displaying the result in Dynamics.

On both the plugin you see we are initializing the countryGet object and below is the source code for it as it stands now,

There static settings for all the end points in an api. They are same for all the api endpoints used in a Dynamics 365 instance. However in a production scenario the likelihood is they won't be same for dev-test-production instance. We can configure this settings so that plugin code does not need to be changed every time you import the solution in a new instance. The current functionality mentioned above will soon change. In the following chapters we will look at how we can save the API specific information in secure way inside your Data Source.

Up to now we have seen how to get all the records in an advanced find. There is a way we can do queries that can bring up only selected records. This will be discussed on the following chapter of this blog.

Hope you have enjoyed (even better understood..) what we have discussed up to now. If you have any question, please do drop me line.

Virtual Entity : Chapter 1 – Definition and Usage

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,

• Chapter 1 - Core concept and definition (Current)
• Chapter 2 - Setting up custom data source and custom provider and hands on code
• Chapter 3 – Introducing searching capability
• Chapter 4 – Storing the settings for the api.

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter 1

Virtual Entities have been available for quite some time now and although I knew conceptually what they were I had not had a chance to use them. Recently I had to look at them for a project at work and I was a bit surprised to find that there aren’t many blogs about them. I will rectify that.

In the first part of the blog, I will start by defining various components of the virtual entity stack. I will then follow up the core concepts with some actual real-life examples. I will include links to a code repository and a packaged solution that could be imported into Dynamics 365 to see the inner workings in action.

What is a Virtual Entity?

The actual definition of a Virtual Entity can be found in this MS document. My own definition of it is not very different. However, before starting to define it in my own terms, let me give you common case study:

Dave the developer, comes to the office one morning. Before going to his desk, he makes himself the best mug of coffee. He is thrilled about the promise of the day. Today he is going to finish the project that he started few weeks ago but never got around to finishing. He even charged his noise cancelling headphone to max so that he can exclude himself from all the usual distractions of the office. Alas the solace never lasts that long, in comes the project manager.

PM : Hey Dave, I have this ultra-super max priority work that I need you look at atm.

Dave : Err… did you not want me to finish this work that I started looking at couple of days ago.

PM : Yeah! But this work is even a bigger priority.

Dave : Huh! Okay, so what’s the work?

PM : Management wants to show a list of ingredients for cakes in the brand new cake web side.

Dave : But the ingredients database in not in Dynamics 365.

PM : Okay, can you not import the data in to Dynamics?

Dave : Of course I can. I can create an ingredients table in Dynamics and create a SSIS package to import the data from the ingredients database into Dynamics.

PM : Oh I forgot, as you know ingredient changes all the time for cakes. So we need the data to be always updated.

Dave : Okay, I can run the SSIS package every 5 mins and scan for updated, newly created ingredients and update Dynamics 365 with the data. But for that we need an SQL database server to run the SSIS package in timely manner. You can also do it in azure but it is not free.

PM : Oh I forgot, since they are now going through this security stuff now, the ingredients organization will not permit us to use their database directly. They are creating this API and we can only get data from the API.

Dave : Okay, SSIS will not be a wise idea then. Maybe Azure Data Factory or Azure Logic Apps then?

PM : Erm one tiny problem. We do not have money to have either and management wants this done tomorrow.

Dave gets up bashes his head against the wall.

Except for the extremities, above is a classic example for a data migration/integration project. Even if you take the complexity and technology out of the equation you still need to answer questions like How often do you keep the data updated? What happens if an update sequence fails? It all boils down to the fact that maintaining one source of truth is far better than maintaining multiple sources. If you want to consider a single source of truth as a rule of thumb for your project, then all you have to answer is how do you present the data with possibly limited customization.

Virtual Entities answer the above question with relative ease. My definition of a Virtual Entity is as follows:

A Virtual Entity is a process by which Dynamics 365 can present data located in a third-party data store to the Dynamics’ application users. This means with a Virtual Entity the data remains in its original location but gets a 'view' inside Dynamics 365. Since the data is not getting copied over there is no need to run expensive integration that synchronizes data. With certain limitations the data is query-able through advanced find, through SDK calls and meta data calls. However, the data remains in a read only state while using virtual entity through Dynamics 365. 

Components

There are three major components of Virtual Entities.

Data Source

From a high-level point of view data source denotes the source of data. However, for Dynamics 365 a Data Source is basically the information about the source of the data. Such information might include, for OData service, the base url, query string, api header. For a custom provider it may include the column set, connection string etc. Interestingly you cannot create a data source from within a solution. You can only add an existing data source. A Data Source can be created by going to Settings > Administration > Virtual Entity Data Sources. Once the Data Source is created in can be added to a solution and exported to different instances.

Data Provider

This is defined as the component the connects to a Data Source, fetches the data, transforms the data and supplies it to Dynamics. By default, Dynamics 365 ships with ODataProvider. There is a Cosmos Data Provider that is available on the Microsoft App Source which I have not tested. Developers can also create custom Data Providers of their own. Regardless of the provider, the data that gets presented to the Dynamics needs to have a key field and key needs to have a globally unique Id. If you are working on a non OData API and need to give a user access to any data using a Virtual Entity, an option can be to use Custom Data Provider as a transformation layer and add a Guid column on each row.

Virtual Entity

A Virtual entity is the record type that has the same columns as those that come from the Data Provider. A Dynamics user can flag an Entity to be a Virtxual Entity when they are creating an Entity in customization. For a Virtual Entity that depends on an out of the box OdataProvider by Microsoft, mapping for entity name, fields and data type are very strict. I am going to discuss custom data provider in following chapters. For custom data provider some of the rules does not apply. Refer to this document for more information know how on OOB odata provider.

One thing worth noting, while creating a custom data provider both data source and Virtual Entity are created with the Virtual Entity flag set to true.