Sunday 8 September 2019

Virtual Entity : Chapter 2 – Create a Custom Data Provider for a Virtual Entity

Overview

This is of the four-part series tutorial where I am trying to explain virtual entity in Dynamics 365 CE. The chapters are,
  • Chapter 1 - Core concept and definition
  • Chapter 2 - Setting up custom data source and custom provider and hands on code (Current)
  • Chapter 3 – Introducing searching capability
  • Chapter 4 – Storing the settings for the api.

The demo is created on a 30 day trial edition of Dynamics 365. All the code mentioned in the tutorial is available on my Github Repo. All the customization used in the tutorial is found on releases section of the same Github repo. I have tagged the binaries as pre-release as I have not used the code in any production scenario. Although I am keeping an eye out for any possible bug reported in Github or to my email but please consider all the code as is. In the code I have integrated country Api found in this page along with the description of the each end point. A huge shout out to ApiLayer. A big thanks to Mark Findlater for reviewing the blog.

Chapter - 2

Recap and Some More Explanation

Following on from the definition page, in this chapter I would like to discuss how can we create and register a Custom Data Provider. Before that let me define a Custom Data Provider in more detail.

As discussed in the previous chapter that Dynamics 365 ships with an ODataProvider for Virtual Entities. If your API does not happen to support OData, you can create your own provider that reads, transforms and present the data to Dynamics. These are called Custom Data Providers.

What are the components of a Custom Data Provider? Excellent question! Under the hood Custom Data Providers are Pre-Operation plugins that have steps for Retrieve and Retrieve Multiple. Consider you are opening a first class Entity record in Dynamics CE. In order to find the record in question you send in the primary key to database via a Retrieve message. In the case of advanced find or looking for more than one records, you come up with a query and send query wrapped inside a Retrieve Multiple. Your Virtual Entities will follow exactly the same mechanism, the only difference here is that while doing the Retrieve or Retrieve Multiple call, you have code inside a plugin step that performs an API call to a third party datastore. Once the query returns you map it to your Virtual Entity and finish your plugin. It’s a very straight forward process, if you think about it. Makes you wonder why no one thought of it before!


Hands on Tutorial

Registration and configuration of Virtual Entity Data Provider requires a lot of back forth work between the plugin registration tool and Dynamics 365. Here are the rough steps that we need to follow in order to register a working Custom Data Provider.

  • Step 1 : Create and register Retrieve and Retrieve Multiple plugin signature steps for Custom Provider. Although we cannot create a full working provider yet as the column definition of the Virtual Entity does not exist, this is where we start.
  • Step 2 : Register the Custom Data Provider and Data Source with Plugin registration tool.
  • Step 3 : Make additional configuration change for the source in Dynamics 365.
  • Step 4 : Create the Virtual Entity in the Dynamics 365. Also create the columns in the Dynamics
  • Step 5 : Finish the Custom Data Provider code and update the assembly in plugin registration tool.
First of all, Even before we begin, if you are using older version of the plugin registration tool, it will not work. It is crucial to have the latest version of the tool found on the Microsoft docs site. The one that comes with the SDK download did not work for me. I went to the above-mentioned site and ran the code in Powershell and after a while all the tools were installed in the stipulated directory.

Also all the code used in the following sections is available for you in the Github repository. I have also included the Dynamics 365 with the code in the release section.

Step 1 : Creation and Registration of the plugin


This part is easy. I have created the shell of two plug in classes. One plugin is for Retrieve and the other for Retrieve Multiple. Apart from their names, they look identical.




We will have to register the plugin as normal assembly in the plugin registration tool. I am expecting the readers will have understanding of how to register a plugin. If this is something very new to you, please refer to this ms doc. It lists down every little detail for plugin registration.

Step 2 : Register Custom Data Provider and Data Source

Microsoft need to do a bit of work here (but stick with it!) Occasionally the plugin registration tool sometimes breaks even when the registration is successful! I have pointed out where it breaks and what to do when the registration tool crashes. The crashing was very frequent when I was trying custom provider couple of months ago. However, when at the time of writing this demo the registration tool has not crashed once. May be the trial instance that I was preparing this demo in has some updates that my organisation’s develop instance do not have. Refers to this document for more information for crash.

To register a new Custom Data Provider a click on Register button on the plugin registration tool again. And now select Register New Data Provider.




The below screen will be rendered. Complete all the fields (they are all mandatory)


My settings for Provider registration was as follows.
Settings Name
My Value           
Explanation      
Data Provider Name
Country Provider
Free text. Think of something that sensibly represents your Data Provider.
Solution
Look for the solution in question, My setting was Custom Provider
You can create your own solution in this screen but I prefer to create the solution beforehand.
Assembly
Codebug.CountryProvider
The newly registered assembly in the Step 1
Retrieve

The class in the above assembly meant to handle Retrieve message.
Retrieve Multiple

The class in the above assembly meant to handle Retrieve Multiple message

I have skipped the settings while setting up data set registration. When you load the screen for the first time the look up will be blank and you can select the look up and can add a new entity in the following screen. 



The new screen contains nothing but prompts for Name(s) of the data source entity and prompt for the solution where the user would like the data source entity to sit. Again, the names fields are free text and user can put anything sensible in the fields.

In most cases at this point you will be presented with an error when they click on create button. In addition to previous link that I shared on this topic it is explained beautifully in the following blog post. I am not going to bore you with details but long story short the data source entity has actually been got created. You only need to restart the plugin registration tool and go back to the Register New Data Provider screen again. This time you do not have to create a data source entity since you could see the data source entity in the drop down. You now just need to fill in the rest of the details. 


Step 3 : Make Additional Change to the Data Source

I have discussed that the Data Source entity contains information about the source of the data rather than data itself. In the previous step we have created data source entity. We now need to make some additional customization in datasource in order to make everything work. These customizations will basically be new field creation. So that we can, for instance, store the main part of the url of the API that we are using. The API that we will be using using for this example uses a couple of set headers as well. We will create fields in the custom datasource so that we can store the keys and values for it. Last customization will be changing the form of the custom data source so that we can view the information. Let us go and do that.
Data source entity is created as a Virtual Entity in Dynamics meaning all the constraints for the Virtual Entity exists when you create a Data Source as well. When you create new field on the above Data Source entity you need to provide an external name as well. These external names for Custom Data Source fields do not have any usage but are required.

After the customization the Data Source Entity will be ready to use and you can set one up. You can view the data source by going to,
Setting > Administration > Virtual Entity Data Sources


Once there click on new button and select the Data Provider that you have registered with plugin registration tool. The record will look like below for our use case (I have added data to the page as well),



Step 4 : Create the Virtual Entity

I am planning to get country data from third party API. I have added the important information that I need in the previous screen shot. In case if you missed it the details can be found in the api description page. As you might have guessed, I am planning to expose list of countries in Dynamics through virtual entity. For the rest of the blog I am using two API end points,
  • Get All countries – which will show all countries in an advanced find and in associated view
  • Get by Country code – this will get a single country. This will be called when an entity form is opened. 
  • Search by Region – it will bring up all the countries in a region. I will use it to show advanced find capability.
  • Search By Capital - another search functionality. This time with capital of a country.


The primary object (you guessed it right) is a country.  Here’s how one object looks like on the API.


As you can see, the data structure is quite complex. I am going to normalize the object a little bit so that the deserialization becomes an easy task. One of the key things you might also have noticed is that this data structure does not give us an identifiable key. I am going to use Alpha3Code as the key and will hold a key-value dictionary inside the plugin which will give me a GUID when a code is supplied. I am sure you could think of ten other ways to craftily achieve the same thing! For this example, I consider having a dictionary inside the plugin code as simple enough. While we are on the topic, here’s how my model class look like inside the plug-in assembly,


Based on the above model I am going to create a new Entity, country as a Virtual Entity. My Entity definition looks like below. 


For Custom Data Provider and Data Source External Name and External Collection name do not matter, the only requirement is they need to be populated. This is true for the external name for the field definition as well.
Here is how the Entity field look like when all the field creation is finished.



Step 5 : Finish the Plugin Provider Code

With Retrieve Multiple, you can do some nifty things with a visitor pattern to actually query for specific country. However, to keep the discussion on point, in this example, I am not including anything on that yet. So we will just be running a GET request and displaying the result in a Retrieve Multiple or a Retrieve step. The detailed code example is available on github.

Below is the class that is responsible for the api requests,



And below is how the plugins look like after we have done the code change


As mentioned before, RetrieveMultipleCountry is getting called when we are opening and advanced find view and trying to search for all country. RetrieveCountry will get called when user tries to open a single record. We are taking the entity id (Guid) translating it to 3 character country code, searching the country the code and then displaying the result in Dynamics.

On both the plugin you see we are initializing the countryGet object and below is the source code for it as it stands now,


There static settings for all the end points in an api. They are same for all the api endpoints used in a Dynamics 365 instance. However in a production scenario the likelihood is they won't be same for dev-test-production instance. We can configure this settings so that plugin code does not need to be changed every time you import the solution in a new instance. The current functionality mentioned above will soon change. In the following chapters we will look at how we can save the API specific information in secure way inside your Data Source.

Up to now we have seen how to get all the records in an advanced find. There is a way we can do queries that can bring up only selected records. This will be discussed on the following chapter of this blog.



Hope you have enjoyed (even better understood..) what we have discussed up to now. If you have any question, please do drop me line.

No comments: