Salesforce dynamic mapping and the awesomeness of Custom Metadata Types

So, you are a company that uses Salesforce, you have an already existing app that works outside of Salesforce, you want to have data from the external app inside Salesforce - “what do?” well… of course, you use Heroku Connect, right? Not so fast! 

From here you simply create a Console App in Visual Studio and put this in the App.config file:

<appSettings>
    <add key="ConsumerKey" value="[CONSUMER_KEY]" /> <!-- Setup -> Create -> Apps -> Connected Apps -> New -> Check 'Enable OAuth and set the OAuth Scope(s). Fill remaining required fields. -->
    <add key="ConsumerSecret" value="[CONSUMER_SECRET]" /> <!-- Secret is generated along with the Consumer Key -->
    <add key="SecurityToken" value="[SECURITY_TOKEN]" /> <!-- My Settings -> Personal -> Reset My Security Token  -->
    <add key="Username" value="[USERNAME]" /> <!-- My Settings -> Personal -> Personal Information  -->
    <add key="Password" value="[PASSWORD]" /> <!-- My Settings -> Personal -> Change My Password  -->

    <!--set to true to point  SimpleConsole at a Sandbox environment (test.salesforce.com) -->
    <add key="IsSandboxUser" value="false" />
  </appSettings>

(with your creds, key, and secret from the Connected App)

Then if you simply follow the examples from the Force.com Toolkit’s GitHub page you’ll push some data to Salesforce in no time. Awesome!

Ok, you’ve pushed some data, let’s say you created some Accounts or Contacts, maybe you even created some custom object records, that’s nice but, let’s take a specific scenario into focus. Like I mentioned in the intro, maybe you already have some external app and let’s assume that your app deals with customers and bookings. How do you translate that into Salesforce Objects? You make custom objects or translate (hardcode) your customers and bookings into Contacts and Activities.

That’s a good option, of course, but let’s go even deeper with my wild assumptions. What if the requirements change at some point (they do) OR even something crazier - what if you want to distribute your app on Salesforce’s AppExchange? Do you really want to force your potential clients to use Contacts + Activities? You could, but why would you? keeping in mind that you’re reading this blog post.

Let’s take things into perspective, you need to build a ‘data translator’. Where do you do this? You could do it outside of Salesforce or directly in Apex. I chose the latter for multiple reasons. 

No how do you build said ‘data translator’? Do you hard code some mappings (I did this)?
Of course not! You’re not an idiot! You use Custom Metadata Types.

You can look it up on Salesforce’s docs but for me they are these magical custom objects which you can query to your heart’s content and you can deploy both the definition plus their records to other orgs.
That’s right, you can deploy records. Think of that - you can create some metadata types that influence your Salesforce App’s logic (think config files) and deploy those to a customer’s organization.  

Now to put that into our context, we can build our mapper using custom metadata types for mapping configurations - yay! 

1. Build the Custom Object

At the time of writing this, Salesforce doesn’t allow us to build triggers directly on Custom Metadata Types so we still need one custom object to trigger the mapper at the time the data arrives in Salesforce. This is fine, we can use that sole custom object for everything, store the data, results, debugging stuff, as long as we take into account the usual Salesforce limitations.

Ok, so let’s make that object. I’m going with the client + booking case from above so I’m making a Booking_Data custom object with two text fields (Long Text Area) with max length. Here it is:

We can push some data from .NET to Salesforce.

2. The external Console App

If you followed the article you know I chose to use the Force.com .NET Toolkit which lists some useful examples on how to do CRUD on Salesforce. I followed the simplest example there and I managed to push Booking Data records.

Here’s one key point though - you need to serialize the data in some way, this is good for multiple reasons. I went with the popular JSON format:

var book = new Booking
            {
                Id = 1,
                full_describe = "Testing NET",
                Date_Time = DateTime.Now,
                Duration = 60
            };
            var bookObj = Newtonsoft.Json.JsonConvert.SerializeObject(book);
            var sfBook = new { stefanistest__Booking__c = bookObj };
            var createBookingDataResponse = await client.CreateAsync("stefanistest__Booking_Data__c", sfBook);
            Console.WriteLine("BData response:" + createBookingDataResponse);

            var customer = new Client
            {
                id = 1,
                first_name = "John",
                last_name = "The Revelator",
                mobile_phone = "+4402233224",
                email = "trulymyemail@noscam.com"
            };
            var clientJson = Newtonsoft.Json.JsonConvert.SerializeObject(customer);
            var sfClient = new { stefanistest__Client__c = clientJson };

            var createClientDataResponse = await client.CreateAsync("stefanistest__Booking_Data__c", sfClient);
            Console.WriteLine("BData response:" + createClientDataResponse);

3. Our Custom Metadata Type

Now this here is the key point of this article, la crème de la crème,  this awesome, albeit simple, piece of the Salesforce Platform will help us with our quest to map stuff. 

Here’s mine. Don’t worry, I’ll go through each field individually and explain things.

4. The fields

• ‘Active’ it simply states if a mapping will be active or not. Let’s say that for one Client we’d like to have both a Contact and Custom Object created every time we pushed a Customer from the external app. What I’m trying to say here is that we can have multiple mappings for one type of record and we can enable or disable them at will. 
• ‘API Name’ it’s the name of the target object (eg: Contact or MyCustomObject__c), simple enough.
• ‘Field Map Definition’ where we’ll store the actual mappings. I’ll show some examples to better understand it later on. 
• ‘Record Type’ (of the target object).
• ‘Type’ very important field here. For this example the values in the picklist are ‘Booking’ and ‘Client’ - can you guess what they’re used for? That’s right, for telling our code what data to grab and map. 
• ‘Unique Field’ - the field by which you upsert or update your target records. That’s right, we don’t only create, we can upsert or just update fields. This is good if we want to avoid duplicate records. 
• ‘Update Only’ - we use this field to force update only. Find records that match our unique field values, update those records with new data, disregard the rest. 

5. Some mappings

Now I’ll present you some sample mappings so that you get a better grasp of what I was rambling above. It will all become clear.

First off we have a simple example where we map some Booking Data to the Event standard object.

So, when we push data and have this mapping active, the Mapper will create Event records with the fields from above filled. As you can see the Type field is Booking - this means it won’t randomly grab Client data and try to make Events, we explicitly specify that we want Bookings. Simple enough, right?

Lets fully use the Mapper’s functionality in the next example:

The API Name is Contact, the fields are mapped, and it’s Active - all good. In this case, let's pretend we have some Contacts that already exist in Salesforce, so we don’t want duplicate data, and we have them under the ‘Special_Contacts’ Record Type. Notice that we have the Unique Field field filled (that’s a mouthful). This means the Mapper will query Contacts by that field’s value, compare it with the data that comes from the external app and if it’s a match, update that Contact with new data (only what is specified in the Field Map Definition, it will not touch anything else). I say update because the Update Only field is checked. So if it won’t find a match it won’t do anything. If we had this unchecked, it would create new Contacts if a match wasn’t found. 

Goes without saying that you can map to any other object in Salesforce, both standard, and custom, as long as you look out for mandatory fields or any other rules that apply to the said object.

As you can imagine, when writing a mapper, you need to be able to deserialize the data. You can do it manually by parsing the JSON/XML or you can write a class and have Apex do it automatically for you. Apart from this, I ran into the weird situation when I needed to grab class fields dynamically (you will encounter this word a lot when writing a mapper); think grab x field from json and assign it to x field of an object.

As it happens with every weird situation in programming, other people encountered it before you. Therefore, if you’re thinking to write a mapper you’ll need this gem.
The author explains its behavior quite well. Simple and effective! 

Applying this in our case:

global class Client extends CoreObject{ 
    public Integer id {get; set;}
    public String first_name {get; set;}
    public String last_name {get; set;}
    public String mobile {get; set;}
    public String email {get; set;}
}

global class Booking extends CoreObject{
    public Integer id {get; set;}  
    public String full_describe {get; set;}  
    public String date_time {get; set;}
    public Integer duration {get; set;}  
    public String end_date_time {get; set;}
}

public static Client deserializeClient(String JSONString) {
        Client member = new Client();
        member = (Client)JSON.deserialize(JSONString, Client.class);
        return member;
    }

If you’ve done your reading, you could say that triggers are better off being logicless, this means that our trigger can be a one liner. How so? By using a Trigger Framework of course.

There are a few out there but most were overkill in my case, so I went with Kevin O’hara’s one (click here to see the example). Again, simple and effective, that is my target. 

trigger BookingDataTrigger on Booking_Data__c (before insert, before update, before delete,
                                                    after insert, after update, after delete, after undelete) {
    new BookingDataTriggerHandler().run();
}

public class BookingDataTriggerHandler extends TriggerHandler {

	private List<Booking_Data__c> newDataList;
	
	
	public BookingDataTriggerHandler() {
		//prevent recurison
		this.setMaxLoopCount(2);
	}

	public override void afterInsert()
	{
		newDataList = Trigger.new;
		for(Booking_Data__c bdata : newDataList) {
			Mapper m = new Mapper();
			if(String.isNotBlank(bdata.Client__c)) {
					Client newClient = new Client(); 
					newClient = Mapper.deserializeClient(bdata.Client__c);
					Map<Booking_Data_Mapping, SObject> cList = Mapper.createClientMapping(newClient);
					Util.makeDMLOperation(cList);
			}
        }
    }

Then in the handler, you can write all the logic you want. I advise to keep it clean even there and read about Apex Enterprise Patterns, maybe even do the Trails. It will help you write clean code.

 public static List<SObject> createBookingMapping(Booking book) {
        List<SObject> bookingObjects = new List<SObject>();
        SObject bookingObject;
        List<Booking_Mapping__mdt> mappings = getActiveBookingRecords();
        Map<String, Object> bookingMap;

        for(Booking_Mapping__mdt m : mappings) {
            bookingMap = deserializeMapping(m);
            bookingObject = Schema.getGlobalDescribe().get(m.API_Name__c).newSObject();
            Schema.SobjectType bookingObjType = bookingObject.getSObjectType();
            Map<String, Schema.SObjectField> bookingObjectFieldMap = bookingObjType.getDescribe().fields.getMap();
            for(String s : bookingMap.keyset()) {
                Object val = (Object)book.get((String)bookingMap.get(s));
                assignFieldVal(bookingObjectFieldMap, bookingObject, s, val);
            }
            bookingObjects.add(bookingObject);
        }
        return bookingObjects;
    }

This is it - it’s not long, it’s not complicated. 
I simply: 

• grab all the Custom Metadata Records for Bookings in this case (see the type field) with the `getActiveBookingRecords()`; 
• go through each, deserialize the actual mapping (I chose JSON, you can choose a format or do something custom and more friendly);
• create a new SObject based on the API_Name field;
• determine what object that is;
• get its fields map;
• get values from the Booking object (remember CoreObject?);
• assign the correct values to the correct fields via `assignFieldVal(...)`;

The `assignFieldVal(...)` does exactly what the name says with a bit of null and error checking. I also have a date time format inconsistency in the original code and that is the method by which I can resolve it (you could also do it in the mutators).

public static void makeDMLOperation(Map<Booking_Mapping__mdt, SObject> objects) {
		for(Booking_Mapping__mdt m : objects.keySet()) {
			String sobjectName = objects.get(m).getSObjectType().getDescribe().getName();
			if(String.isNotBlank(m.Unique_Field__c)) {
				Schema.DescribeFieldResult uniqueField = getFieldDetails(objects.get(m), m.Unique_Field__c);
				if(m.Update_Only__c) { //find and update only objects that match the value for the unique field
					Boolean foundObj = updateObjectByUniqueField(objects.get(m), uniqueField);
					if(foundObj) { 
						Database.SaveResult result = database.update(objects.get(m), false);
						system.debug('==>Unique field update only:' + result);
					} else {
						system.debug('==>Unique field update only: RECORD NOT FOUND');
					}
				} else { //find objects that match and insert(or upsert if possible) the rest
					if(uniqueField.isExternalID()) {
						//tricks to get upsert with no concrete sobject
						String listType = 'List<' + sobjectName + '>';
						List<SObject> castRecord = (List<SObject>)Type.forName(listType).newInstance();
						castRecord.add(objects.get(m));
						Database.UpsertResult[] result = database.upsert(castRecord, uniqueField.getSObjectField());
						system.debug('==>Unique field external ID:' + result[0]);
					} else {
						Boolean foundObj = updateObjectByUniqueField(objects.get(m), uniqueField);
						if(foundObj) {
							Database.SaveResult result = database.update(objects.get(m), false);
							system.debug('==>Unique field not external ID and found:' + result);
						} else {
							Database.SaveResult result = database.insert(objects.get(m), false);
							system.debug('==>Unique field update only not found:' + result);
						}
					}
				}
			} else { //if unique field is blank
				Database.SaveResult result = database.insert(objects.get(m), false);
				system.debug('==>No unique Field just insert:' + result);
			}
		}
	}

Seems long but only because of debugging messages. So, again, it’s short and not complicated at all. 
At the time of writing this, there isn’t a way to upsert a list of  sobjects of which you don’t know the type. You will see that there’s a trick in there, therefore, we can do exactly like in here.

The author documented it pretty well and specifies in what cases it doesn’t work. For us it’s fine!

But, if you are curious you will see someone also created this idea here.

As you will see, it covers three cases:

Case #1

• Unique_Field__c is filled and update only is selected;
• Query database for given sobject and the given Unique_Field value. If found, update all fields specified in mapping;
• This includes the SF ID;
• Unique_Field must also be included in mapping;
• If not found do nothing;

Case #2

• Unique_Field__c is filled and update only is not selected;
• Query database for given sobject and the given Unique_Field value. If found, update all fields specified in mapping;
• Test if Unique_Field is external ID. If true then classic upsert by external ID;
• This includes the SF ID;
• Unique_Field must also be included in mapping;
• If not found insert a new sobject with fields specified in mapping;

Case #3

• Unique_Field__c is not filled;
• Insert a new sobject;
• is not recommended as it may cause duplicates;
• I advise that you always fill the unique_field value;

When I started writing The Mapper I wasn’t aware what I was getting myself into. I was aware of Custom Metadata Types but wasn’t completely aware of their usefulness. 
My idea is just a starting point - you can expand way more, cover more cases, make it friendlier, doing it prettier. You can add more filters/criteria and more importantly think of some dynamic way of linking target objects. You don’t want orphaned Events. 

Thank you for reading, I really hope you got something out of it!