Over a year ago we posted Retrieving customer and prospect information from Marketo using the API. That post presented a code sample that could be run on a recurring basis to poll Marketo for updates. The idea was to use Marketo APIs to identify changes to lead data, and extract the lead data that had changed. This data could then be pushed to an external system for synchronization purposes.
The code sample presented used our SOAP API. Well we have a new way to walk, and that way is using the Marketo REST API. This post will show you how to accomplish that same goal using two REST endpoints: Get Lead Changes, Get Lead by Id.
The program contains 2 main steps:
- Call Get Lead Changes to generate a list of all Lead Ids that either had specific lead fields changed, or were added during a given time period.
- Call Get Lead Id for each Lead Id in the list to retrieve field data from the lead record.
We’ll take the data retrieved in step 2 and format it for consumption by an external system.
Program Input
By default the program “goes back” one day from the current date to look for changes. So you could run this program at the same time each day for example. To go farther back in time, you can specify the number of days as a command line argument, effectively increasing the time window.
The program contains several variables that you can modify:
CUSTOM_SERVICE_DATA – This contains your Marketo Custom Service data (Account Id, Client Id, Client Secret).
LEAD_CHANGE_FIELD_FILTER – This contains a comma seperated list of lead fields that we will inspect for changes.
READ_BATCH_SIZE – This is the number of records to retrieve at a time. Use this to tune the response body size.
Program Output
The program gathers up all of the changed lead records and formats them in JSON as an array of lead objects as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
{ "result": [ { "leadId": "318592", "updatedAt": "2015-07-22T19:19:07Z", "firstName": "David", "lastName": "Everly", "email": "deverly@marketo.com" }, ...more lead objects here... ] } |
The idea is that you could then pass this JSON as the request payload to an external web service to synchronize the data.
Program Logic
First we establish our time window, compose our REST endpoint URLS, and obtain our Authentication access token. Next we fire up a Get Paging Token/Get Lead Changes loop in which runs until we exhaust the supply of lead changes. The purpose of this loop is to accumulate a list of unique Lead Ids so that we can pass them to Get Lead by Id later in the program. For this example, we tell Get Lead Changes to look for changes to the following fields: firstName, lastName, email. You are free to select any combination of fields for your purposes.
Get Lead Changes returns “result” objects that contain an Activity Type Id, which we can use to filter results. Note: You can get a list of Activity Types by calling Get Activity Types REST endpoint. We are interested in 2 activity types that are returned:
1. New Lead (12)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
{ "id": 12024682, "leadId": 318581, "activityDate": "2015-03-17T00:18:41Z", "activityTypeId": 12, "primaryAttributeValueId": 318581, "primaryAttributeValue": "David Everly", "attributes": [ { "name": "Created Date", "value": "2015-03-16" }, { "name": "Source Type", "value": "New lead" } ] } |
2. Change Data Value (13)
We can tell which field changed by inspecting the “name” property in the Change Data Value response.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
{ "id": 12024689, "leadId": 318581, "activityDate": "2015-03-17T22:58:18Z", "activityTypeId": 13, "fields": [ { "id": 31, "name": "lastName", "newValue": "Evely", "oldValue": "Everly" } ], "attributes": [ { "name": "Source", "value": "Web form fillout" } ] } |
When either of these two types of activities are returned, we store the associated Lead Id in a list. Once we have our list, we can iterate through it calling Get Lead by Id for each item. This will retrieve the latest lead data for each lead in the list. For this example we retrieve the following lead fields: leadId, updatedAt, firstName, lastName, email. You are free to select any combination of fields for your purposes. This is done by specifying the fields parameter to Get Lead by Id. And finally we JSONify the results as an array of lead objects as described above.
Program Code
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 |
package com.marketo; // minimal-json library (https://github.com/ralfstx/minimal-json) import com.eclipsesource.json.JsonArray; import com.eclipsesource.json.JsonObject; import com.eclipsesource.json.JsonValue; import java.io.*; import java.net.MalformedURLException; import java.net.URL; import java.text.DateFormat; import java.text.ParseException; import java.text.SimpleDateFormat; import java.util.*; import javax.net.ssl.HttpsURLConnection; public class LeadChanges { // // Define Marketo REST API access credentials: Account Id, Client Id, Client Secret. For example: // public static String CUSTOM_SERVICE_DATA[] = // {"111-AAA-222", "2f4a4435-f6fa-4bd9-3248-098754982345", "asdf6IVE9h4Jjcl59cOMAKFSk78ut12W"}; // private static final String CUSTOM_SERVICE_DATA[] = {INSERT YOUR CUSTOM SERVICE DATA HERE}; // Lead fields that we are interested in private static final String LEAD_CHANGE_FIELD_FILTER = "firstName,lastName,email"; // Number of lead records to read at a time private static final String READ_BATCH_SIZE = "200"; // Activity type ids that we are interested in private static final int ACTIVITY_TYPE_ID_NEW_LEAD = 12; private static final int ACTIVITY_TYPE_ID_CHANGE_DATA_VALE = 13; public static void main(String[] args) { // Command line argument to set how far back to look for lead changes (number of days) int lookBackNumDays = 1; if (args.length == 1) { lookBackNumDays = Integer.parseInt(args[0]); } // Establish "since date" using current timestamp minus some number of days (default is 1 day) Calendar cal = Calendar.getInstance(); cal.add(Calendar.DAY_OF_MONTH, -lookBackNumDays); SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ"); String sinceDateTime = sdf.format(cal.getTime()); // Compose base URL String baseUrl = String.format("https://%s.mktorest.com", CUSTOM_SERVICE_DATA[0]); // Compose Identity URL String identityUrl = String.format("%s/identity/oauth/token?grant_type=%s&client_id=%s&client_secret=%s", baseUrl, "client_credentials", CUSTOM_SERVICE_DATA[1], CUSTOM_SERVICE_DATA[2]); // Call Identity API JsonObject identityObj = JsonObject.readFrom(getData(identityUrl)); String accessToken = identityObj.get("access_token").asString(); // Compose URLs for Get Lead Changes, and Get Paging Token String leadChangesUrl = String.format("%s/rest/v1/activities/leadchanges.json?access_token=%s&fields=%s&batchSize=%s", baseUrl, accessToken, LEAD_CHANGE_FIELD_FILTER, READ_BATCH_SIZE); String pagingTokenUrl = String.format("%s/rest/v1/activities/pagingtoken.json?access_token=%s&sinceDatetime=%s", baseUrl, accessToken, sinceDateTime); HashSet leadIdList = new HashSet(); // Call Get Paging Token API JsonObject pagingTokenObj = JsonObject.readFrom(getData(pagingTokenUrl)); if (pagingTokenObj.get("success").asBoolean()) { String nextPageToken = pagingTokenObj.get("nextPageToken").asString(); boolean moreResult; do { moreResult = false; // Call Get Lead Changes API JsonObject leadChangesObj = JsonObject.readFrom(getData(String.format("%s&nextPageToken=%s", leadChangesUrl, nextPageToken))); if (leadChangesObj.get("success").asBoolean()) { moreResult = leadChangesObj.get("moreResult").asBoolean(); nextPageToken = leadChangesObj.get("nextPageToken").asString(); if (leadChangesObj.get("result") != null) { JsonArray resultAry = leadChangesObj.get("result").asArray(); for (JsonValue resultObj : resultAry) { int activityTypeId = resultObj.asObject().get("activityTypeId").asInt(); // Store lead ids for later use boolean storeThisId = false; if (activityTypeId == ACTIVITY_TYPE_ID_NEW_LEAD) { storeThisId = true; } else if (activityTypeId == ACTIVITY_TYPE_ID_CHANGE_DATA_VALE) { // See if any of the changed fields are of interest to us JsonArray fieldsAry = resultObj.asObject().get("fields").asArray(); for (JsonValue fieldsObj : fieldsAry) { String name = fieldsObj.asObject().get("name").asString(); if (LEAD_CHANGE_FIELD_FILTER.contains(name)) { storeThisId = true; } } } if (storeThisId) { leadIdList.add(resultObj.asObject().get("leadId").toString()); } } } } } while (moreResult); } JsonObject result = new JsonObject(); JsonArray leads = new JsonArray(); for (Object o : leadIdList) { String leadId = o.toString(); // Compose Get Lead by Id URL String getLeadUrl = String.format("%s/rest/v1/lead/%s.json?access_token=%s", baseUrl, leadId, accessToken); // Call Get Lead by Id API JsonObject leadObj = JsonObject.readFrom(getData(getLeadUrl)); if (leadObj.get("success").asBoolean()) { if (leadObj.get("result") != null) { JsonArray resultAry = leadObj.get("result").asArray(); for (JsonValue resultObj : resultAry) { // Create lead object JsonObject lead = new JsonObject(); lead.add("leadId", leadId); lead.add("updatedAt", resultObj.asObject().get("updatedAt").asString()); lead.add("firstName", resultObj.asObject().get("firstName").asString()); lead.add("lastName", resultObj.asObject().get("lastName").asString()); lead.add("email", resultObj.asObject().get("email").asString()); // Add lead object to leads array leads.add(lead); } } } } // Add leads array to result object result.add("result", leads); // Print out result object System.out.println(result); System.exit(0); } // Perform HTTP GET request private static String getData(String endpoint) { String data = ""; try { URL url = new URL(endpoint); HttpsURLConnection urlConn = (HttpsURLConnection) url.openConnection(); urlConn.setRequestMethod("GET"); urlConn.setAllowUserInteraction(false); urlConn.setDoOutput(true); int responseCode = urlConn.getResponseCode(); if (responseCode == 200) { InputStream inStream = urlConn.getInputStream(); data = convertStreamToString(inStream); } else { System.out.println(responseCode); data = "Status:" + responseCode; } } catch (MalformedURLException e) { System.out.println("URL not valid."); } catch (IOException e) { System.out.println("IOException: " + e.getMessage()); e.printStackTrace(); } return data; } private static String convertStreamToString(InputStream inputStream) { try { return new Scanner(inputStream).useDelimiter("A").next(); } catch (NoSuchElementException e) { return ""; } } } |
So there you have it, life’s a happy song. Enjoy!