This is my third blog in a series concerning data integration. In my first two blog entries we overviewed some of the data integration hurdles as well as some of the common methods used to discover the net-change data that will need to be translated. Here in my third blog I’ll discuss some of the benefits of not integrating in real-time, but creating a batch job to perform the integration. We’ll also look at the Business Rules that may need to be applied within the integration process.
Business processes are not the only consideration when designing your integration. The environment, and specifically the hardware that the integrated systems reside on, can play a key role in determining the integration process to use. For example, if your integration design requires polling a record set for the net-change data, that polling can effect performance. But to a greater extent, the record set that is returned will typically populate any RAM that is available, and if there is not enough RAM to hold the entire return record set, then it will occupy static drive space. Depending on how large the net-change record set is, that’s been returned, stealing all available RAM can seriously impact performance of the system. Conversely, you may have designed a multi-threaded integration process, such as utilizing a message queue as a pickup point for the extracted net- change data. Where running a multi-threaded process, you are able to translation a larger volume of data in a shorter period of time, but, that process will be very CPU intensive.
So, when developing your integration design, keep in mind that batch processing is memory intensive and multi-threaded processes are CPU intensive. Depending on the environment you’re working in, you may have the inclination to build one type of integration process, but the impact of that process would be too costly in terms of system performance and end-user satisfaction.
This is a very typical scenario. The CRM system breaks down ‘customers’ into different customer types. Only ‘customers’ that have actually bought something are to be integrated into the ERP system. So, you need to be able to filter records in order to meet the requirements of this business rule . That filtering can take place in two different places, a) at the time of discovery of the net-change data or, b) during the translation process of the net-change data. If you have used a query to discover the net-change data, you may only need to add something to the WHERE clause of the query, to ensure that only records that meet the business rule criteria are discovered. If the application has it’s own net-change method, but cannot be modified to filter the records, you will need to build the filtering into the translation process rather than in the discovery process. There can, however, be some advantages to filtering at the integration process level. Let’s say that you only want purchasing customers being integrated into you ERP system, but you would like to see an aggregated view of all new customers that have been added to the either system. When you filter out the customers at the translation point rather than the discovery point, you have the records discovered in the net-change process, so, the entire record set can be used to create customer, by type, reporting.
Another. more complex example would be; sales orders being placed in the ERP system and then translated to one of several warehouse systems for processing. In this case, you not only have to filter the record set, but you also have to determine what translation processes will be used to ensure the sales order record is consumed by the correct warehouse system. As you can see, business rules will have a huge impact on how you might design your integration process. The entire solution may incorporate many different net-change discovery methods as well as many different data translation methods. Don’t get caught building a useless solution, wasting time and money. Dig deep into the requirements in order to develop the best integration processes for the given scenario your are working under.
Trackback from C5 Insight Blog
We recently published a blog on all of the updates available when upgrading from CRM 2011 to CRM 2015 ...
The complementary paper includes over 12 years of research, recent survey results, and CRM turnaround success stories.
This 60-second assessment is designed to evaluate your organization's collaboration readiness.
Learn how you rank compared to organizations typically in years 1 to 5 of implementation - and which areas to focus on to improve.
This is a sandbox solution which can be activated per site collection to allow you to easily collect feedback from users into a custom Feedback list.
Whether you are upgrading to SharePoint Online, 2010, 2013 or the latest 2016, this checklist contains everything you need to know for a successful transition.