Synchronizing Lead Data Changes using REST API

July 31, 2015 | by

Over a year ago we posted Retrieving customer and prospect information from Marketo using the API.  That post presented a code sample that could be run on a recurring basis to poll Marketo for updates.  The idea was to use Marketo APIs to identify changes to lead data, and extract the lead data that had changed.  This data could then be pushed to an external system for synchronization purposes.

The code sample presented used our SOAP API.  Well we have a new way to walk, and that way is using the Marketo REST API.  This post will show you how to accomplish that same goal using two REST endpoints: Get Lead Changes, Get Lead by Id.

The program contains 2 main steps:

  1. Call Get Lead Changes to generate a list of all Lead Ids that either had specific lead fields changed, or were added during a given time period.
  2. Call Get Lead Id for each Lead Id in the list to retrieve field data from the lead record.

We’ll take the data retrieved in step 2 and format it for consumption by an external system.


Program Input

By default the program “goes back” one day from the current date to look for changes.  So you could run this program at the same time each day for example.  To go farther back in time, you can specify the number of days as a command line argument, effectively increasing the time window.

The program contains several variables that you can modify:

CUSTOM_SERVICE_DATA – This contains your Marketo Custom Service data (Account Id, Client Id, Client Secret).

LEAD_CHANGE_FIELD_FILTER – This contains a comma seperated list of lead fields that we will inspect for changes.

READ_BATCH_SIZE – This is the number of records to retrieve at a time.  Use this to tune the response body size.

Program Output

The program gathers up all of the changed lead records and formats them in JSON as an array of lead objects as follows:

The idea is that you could then pass this JSON as the request payload to an external web service to synchronize the data.

Program Logic

First we establish our time window, compose our REST endpoint URLS, and obtain our Authentication access token.  Next we fire up a Get Paging Token/Get Lead Changes loop in which runs until we exhaust the supply of lead changes.  The purpose of this loop is to accumulate a list of unique Lead Ids so that we can pass them to Get Lead by Id later in the program.  For this example, we tell Get Lead Changes to look for changes to the following fields: firstName, lastName, email.  You are free to select any combination of fields for your purposes.

Get Lead Changes returns “result” objects that contain an Activity Type Id, which we can use to filter results.  Note: You can get a list of Activity Types by calling Get Activity Types REST endpoint.  We are interested in 2 activity types that are returned:

1. New Lead (12)

2. Change Data Value (13)

We can tell which field changed by inspecting the “name” property in the Change Data Value response.

When either of these two types of activities are returned, we store the associated Lead Id in a list.  Once we have our list, we can iterate through it calling Get Lead by Id for each item.  This will retrieve the latest lead data for each lead in the list.  For this example we retrieve the following lead fields: leadId, updatedAt, firstName, lastName, email.  You are free to select any combination of fields for your purposes.  This is done by specifying the fields parameter to Get Lead by Id.  And finally we JSONify the results as an array of lead objects as described above.


Program Code

So there you have it, life’s a happy song.  Enjoy!