Polling for Activities using REST API

Earlier this year we published this post that described how to poll for lead changes within Marketo. This post is similar, but this time we’ll poll for activities.

Activities are a core object in the Marketo Platform. Activities are the behavioral data that is stored about every webpage visit, email open, webinar attendance, or trade show attendance. A common use case is combining activity data with data from other parts of an organization.

This sample program contains 6 steps:

Call Get Lead Activities to generate a list of all activity records created since a given date/time. We use a filter to limit the type of activity records that are returned.
Extract fields of interest from each activity record.
Call Get Multiple Leads by Filter Type to generate a list of lead records that correspond with the activities from Step 1. We use the leadId field extracted from activity record in Step 2 as our filter to specify which leads are returned.
Extract fields of interest from each lead record.
Join the activity data from step 2 with the lead data from Step 4.
Transform the data from Step 5 into a format that is consumable by an external system.

As the diagram below shows, for this example we have chosen to capture email-related activities.

Program Input

By default the program goes back in time one day from the current date to look for changes. So you could run this program at the same time each day for example. To go farther back in time, you can specify the number of days as a command line argument, effectively increasing the window of time.

The program contains several variables that you can modify:

CUSTOM_SERVICE_DATA – This contains your Marketo Custom Service data (Account Id, Client Id, Client Secret).

READ_BATCH_SIZE – This is the number of records to retrieve at a time. Use this to tune the response body size.

LEAD_FIELDS – This contains a list of lead fields that we want to collect.

ACTIVITY_TYPES – This contains a list of activity types that we want to collect.

Program Logic

First we establish our time window, compose our REST endpoint URLS, and obtain our Authentication access token. Next we fire up a Get Paging Token/Get Lead Activities loop which runs until we exhaust the supply of activities. The purpose of this loop is to retrieve activity records, and extract fields of interest from those records.

We tell Get Lead Activities to look only for the following activity types:

Email Delivered
Unsubscribe Email
Open Email
Click Email.

We extract the following fields of interest from each activity record:

leadId
activityType
activityDate
primaryAttributeValue

You are free to select any combination of activity types and activity fields to suit your purpose.

Next we fire up a Get Multiple Leads by Filter Type loop which runs until we exhaust the supply of leads. Note that we use the “filterType=id” parameter in combination with a series of “filterValues” parameters to limit the lead records retrieved to only those leads associated with activities that we retrieved earlier.

We extract the following fields of interest from each lead record:

firstName
lastName
email

Again, you are feel to select any lead fields that you like.

Next we join the lead fields with the activity fields by using the lead id to link them together.

Finally, we loop through all of the data, transform it into JSON format, and print it to the console.

Program Output

Here is an example of output from the sample program. This shows the activity fields and the lead fields combined together as JSON “result” objects. The idea here is that you could pass this JSON as a request payload to an external web service.

{

    "result": [
        {
            "leadId": 318581,
            "activityType": "Email Delivered",
            "activityDate": "2015-03-17T20:00:06Z",
            "primaryAttributeValue": "My Email Program",
            "firstName": "David",
            "lastName": "Everly",
            "email": "everlyd@marketo.com"
        },
        {
            "leadId":318581,
            "activityType":"Open Email",
            "activityDate":"2015-03-17T23:23:12Z",
            "primaryAttributeValue":"My Email Program - Auto Response",
            "firstName":"David",
            "lastName":"Everly",
            "email":"everlyd@marketo.com"
       },
        ... more result objects here...
    ]
}

{

"result": [

{

"leadId": 318581,

"activityType": "Email Delivered",

"activityDate": "2015-03-17T20:00:06Z",

"primaryAttributeValue": "My Email Program",

"firstName": "David",

"lastName": "Everly",

"email": "everlyd@marketo.com"

{

"leadId":318581,

"activityType":"Open Email",

"activityDate":"2015-03-17T23:23:12Z",

"primaryAttributeValue":"My Email Program - Auto Response",

"firstName":"David",

"lastName":"Everly",

"email":"everlyd@marketo.com"

... more result objects here...

]

}

Program Code

package com.marketo;

// minimal-json library (https://github.com/ralfstx/minimal-json)
import com.eclipsesource.json.JsonArray;
import com.eclipsesource.json.JsonObject;
import com.eclipsesource.json.JsonValue;

import java.io.*;
import java.net.MalformedURLException;
import java.net.URL;
import java.text.DateFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.*;

import javax.net.ssl.HttpsURLConnection;

public class LeadActivities {
//
// Define Marketo REST API access credentials: Account Id, Client Id, Client Secret.  For example:
    private static Map<String, String> CUSTOM_SERVICE_DATA;
    static {
        CUSTOM_SERVICE_DATA = new HashMap<String, String>();
//        CUSTOM_SERVICE_DATA.put("accountId", "111-AAA-222");
//        CUSTOM_SERVICE_DATA.put("clientId", "2f4a4435-f6fa-4bd9-3248-098754982345");
//        CUSTOM_SERVICE_DATA.put("clientSecret", "asdf6IVE9h4Jjcl59cOMAKFSk78ut12W");
    }

    // Number of lead records to read at a time
    private static final String READ_BATCH_SIZE = "200";

    // Lookup lead records using lead id as primary key
    private static final String LEAD_FILTER_TYPE = "id";

    // Lead record lookup returns these fields
    private static final String LEAD_FIELDS = "firstName,lastName,email";

    // Lookup activity records for these activity types
    private static Map<Integer, String> ACTIVITY_TYPES;
    static {
        ACTIVITY_TYPES = new HashMap<Integer, String>();
        ACTIVITY_TYPES.put(7, "Email Delivered");
        ACTIVITY_TYPES.put(9, "Unsubscribe Email");
        ACTIVITY_TYPES.put(10, "Open Email");
        ACTIVITY_TYPES.put(11, "Click Email");
    }

    public static void main(String[] args) {
        // Command line argument to set how far back to look for lead changes (number of days)
        int lookBackNumDays = 1;
        if (args.length == 1) {
            lookBackNumDays = Integer.parseInt(args[0]);
        }

        // Establish "since date" using current timestamp minus some number of days (default is 1 day)
        Calendar cal = Calendar.getInstance();
        cal.add(Calendar.DAY_OF_MONTH, -lookBackNumDays);
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");
        String sinceDateTime = sdf.format(cal.getTime());

        // Compose base URL
        String baseUrl = String.format("https://%s.mktorest.com",
            CUSTOM_SERVICE_DATA.get("accountId"));

        // Compose Identity URL
        String identityUrl = String.format("%s/identity/oauth/token?grant_type=%s&client_id=%s&client_secret=%s",
                baseUrl, "client_credentials", CUSTOM_SERVICE_DATA.get("clientId"), CUSTOM_SERVICE_DATA.get("clientSecret"));

        // Call Identity API
        JsonObject identityObj = JsonObject.readFrom(getData(identityUrl));
        String accessToken = identityObj.get("access_token").asString();

        // Compose URLs for Get Lead Changes, and Get Paging Token
        String activityTypesUrl = String.format("%s/rest/v1/activities/types.json?access_token=%s",
                baseUrl, accessToken);
        String pagingTokenUrl = String.format("%s/rest/v1/activities/pagingtoken.json?access_token=%s&sinceDatetime=%s",
                baseUrl, accessToken, sinceDateTime);

        // Use activity ids to create filter parameter
        String activityTypeIds = "";
        for (Integer id : ACTIVITY_TYPES.keySet()) {
            activityTypeIds += "&activityTypeIds=" + id.toString();
        }

        // Compose URL for Get Lead Activities
        // Only retrieve activities that match our defined activity types
        String leadActivitiesUrl = String.format("%s/rest/v1/activities.json?access_token=%s%s&batchSize=%s",
                baseUrl, accessToken, activityTypeIds, READ_BATCH_SIZE);

        Map<Integer, List> activitiesMap = new HashMap<Integer, List>();
        Set leadsSet = new HashSet();

        // Call Get Paging Token API
        JsonObject pagingTokenObj = JsonObject.readFrom(getData(pagingTokenUrl));
        if (pagingTokenObj.get("success").asBoolean()) {
            String nextPageToken = pagingTokenObj.get("nextPageToken").asString();
            boolean moreResult;
            do {
                moreResult = false;

                // Call Get Lead Activities API to retrieve activity data
                JsonObject leadActivitiesObj = JsonObject.readFrom(getData(String.format("%s&nextPageToken=%s",
                        leadActivitiesUrl, nextPageToken)));
                if (leadActivitiesObj.get("success").asBoolean()) {
                    moreResult = leadActivitiesObj.get("moreResult").asBoolean();
                    nextPageToken = leadActivitiesObj.get("nextPageToken").asString();

                    if (leadActivitiesObj.get("result") != null) {
                        JsonArray activitiesResultAry = leadActivitiesObj.get("result").asArray();
                        for (JsonValue activitiesResultObj : activitiesResultAry) {

                            // Extract activity fields of interest (leadID, activityType, activityDate, primaryAttributeValue)
                            JsonObject leadObj = new JsonObject();
                            int leadId = activitiesResultObj.asObject().get("leadId").asInt();
                            leadObj.add("activityType", ACTIVITY_TYPES.get(activitiesResultObj.asObject().get("activityTypeId").asInt()));
                            leadObj.add("activityDate", activitiesResultObj.asObject().get("activityDate").asString());
                            leadObj.add("primaryAttributeValue", activitiesResultObj.asObject().get("primaryAttributeValue").asString());

                            // Store JSON containing activity fields in a map using lead id as key
                            List leadLst = activitiesMap.get(leadId);
                            if (leadLst == null) {
                                activitiesMap.put(leadId, new ArrayList());
                                leadLst = activitiesMap.get(leadId);
                            }
                            leadLst.add(leadObj);

                            // Store unique lead ids for use as lead filter value below
                            leadsSet.add(leadId);
                        }
                    }
                }
            } while (moreResult);
        }

        // Use unique lead id values to create filter parameter
        String filterValues = "";
        for (Object object : leadsSet) {
            if (filterValues.length() > 0) {
                filterValues += ",";
            }
            filterValues += String.format("%s", object);
        }

        // Compose Get Multiple Leads by Filter Type URL
        // Only retrieve leads that match the list of lead ids that was accumulated during activity query
        String getMultipleLeadsUrl = String.format("%s/rest/v1/leads.json?access_token=%s&filterType=%s&fields=%s&filterValues=%s&batchSize=%s",
                baseUrl, accessToken, LEAD_FILTER_TYPE, LEAD_FIELDS, filterValues, READ_BATCH_SIZE);
        String nextPageToken = "";

        do {
            String gmlUrl = getMultipleLeadsUrl;

            // Append paging token to Get Multiple Leads by Filter Type URL
            if (nextPageToken.length() > 0) {
                gmlUrl = String.format("%s&nextPageToken=%s", getMultipleLeadsUrl, nextPageToken);
            }

            // Call Get Multiple Leads by Filter Type API to retrieve lead data
            JsonObject multipleLeadsObj = JsonObject.readFrom(getData(gmlUrl));
            if (multipleLeadsObj.get("success").asBoolean()) {
                if (multipleLeadsObj.get("result") != null) {
                    JsonArray multipleLeadsResultAry = multipleLeadsObj.get("result").asArray();

                    // Iterate through lead data
                    for (JsonValue leadResultObj : multipleLeadsResultAry) {
                        int leadId = leadResultObj.asObject().get("id").asInt();

                        // Join activity data with lead fields of interest (firstName, lastName, email)
                        List leadLst = activitiesMap.get(leadId);
                        for (JsonObject leadObj : leadLst) {
                            leadObj.add("firstName", leadResultObj.asObject().get("firstName").asString());
                            leadObj.add("lastName", leadResultObj.asObject().get("lastName").asString());
                            leadObj.add("email", leadResultObj.asObject().get("email").asString());
                        }
                    }
                }
            }

            nextPageToken = "";
            if (multipleLeadsObj.asObject().get("nextPageToken") != null) {
                nextPageToken = multipleLeadsObj.get("nextPageToken").asString();
            }
        } while (nextPageToken.length() > 0);

        // Now place activity data into an array of JSON objects
        JsonArray activitiesAry = new JsonArray();
        for (Map.Entry<Integer, List> activity : activitiesMap.entrySet()) {
            int leadId = activity.getKey();
            for (JsonObject leadObj : activity.getValue()) {
                // do something with leadId and each leadObj
                leadObj.add("leadId", leadId);
                activitiesAry.add(leadObj);
            }
        }

        // Print out result objects
        JsonObject result = new JsonObject();
        result.add("result", activitiesAry);
        System.out.println(result);

        System.exit(0);
    }

    // Perform HTTP GET request
    private static String getData(String endpoint) {
        String data = "";
        try {
            URL url = new URL(endpoint);
            HttpsURLConnection urlConn = (HttpsURLConnection) url.openConnection();
            urlConn.setRequestMethod("GET");
            urlConn.setAllowUserInteraction(false);
            urlConn.setDoOutput(true);
            int responseCode = urlConn.getResponseCode();
            if (responseCode == 200) {
                InputStream inStream = urlConn.getInputStream();
                data = convertStreamToString(inStream);
            } else {
                System.out.println(responseCode);
                data = "Status:" + responseCode;
            }
        } catch (MalformedURLException e) {
            System.out.println("URL not valid.");
        } catch (IOException e) {
            System.out.println("IOException: " + e.getMessage());
            e.printStackTrace();
        }
        return data;
    }

    private static String convertStreamToString(InputStream inputStream) {
        try {
            return new Scanner(inputStream).useDelimiter("A").next();
        } catch (NoSuchElementException e) {
            return "";
        }
    }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

package com.marketo;

// minimal-json library (https://github.com/ralfstx/minimal-json)

import com.eclipsesource.json.JsonArray;

import com.eclipsesource.json.JsonObject;

import com.eclipsesource.json.JsonValue;

import java.io.*;

import java.net.MalformedURLException;

import java.net.URL;

import java.text.DateFormat;

import java.text.ParseException;

import java.text.SimpleDateFormat;

import java.util.*;

import javax.net.ssl.HttpsURLConnection;

public class LeadActivities {

// Define Marketo REST API access credentials: Account Id, Client Id, Client Secret. For example:

private static Map<String, String> CUSTOM_SERVICE_DATA;

static {

CUSTOM_SERVICE_DATA = new HashMap<String, String>();

// CUSTOM_SERVICE_DATA.put("accountId", "111-AAA-222");

// CUSTOM_SERVICE_DATA.put("clientId", "2f4a4435-f6fa-4bd9-3248-098754982345");

// CUSTOM_SERVICE_DATA.put("clientSecret", "asdf6IVE9h4Jjcl59cOMAKFSk78ut12W");

}

// Number of lead records to read at a time

private static final String READ_BATCH_SIZE = "200";

// Lookup lead records using lead id as primary key

private static final String LEAD_FILTER_TYPE = "id";

// Lead record lookup returns these fields

private static final String LEAD_FIELDS = "firstName,lastName,email";

// Lookup activity records for these activity types

private static Map<Integer, String> ACTIVITY_TYPES;

static {

ACTIVITY_TYPES = new HashMap<Integer, String>();

ACTIVITY_TYPES.put(7, "Email Delivered");

ACTIVITY_TYPES.put(9, "Unsubscribe Email");

ACTIVITY_TYPES.put(10, "Open Email");

ACTIVITY_TYPES.put(11, "Click Email");

}

public static void main(String[] args) {

// Command line argument to set how far back to look for lead changes (number of days)

int lookBackNumDays = 1;

if (args.length == 1) {

lookBackNumDays = Integer.parseInt(args[0]);

}

// Establish "since date" using current timestamp minus some number of days (default is 1 day)

Calendar cal = Calendar.getInstance();

cal.add(Calendar.DAY_OF_MONTH, -lookBackNumDays);

SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");

String sinceDateTime = sdf.format(cal.getTime());

// Compose base URL

String baseUrl = String.format("https://%s.mktorest.com",

CUSTOM_SERVICE_DATA.get("accountId"));

// Compose Identity URL

String identityUrl = String.format("%s/identity/oauth/token?grant_type=%s&client_id=%s&client_secret=%s",

baseUrl, "client_credentials", CUSTOM_SERVICE_DATA.get("clientId"), CUSTOM_SERVICE_DATA.get("clientSecret"));

// Call Identity API

JsonObject identityObj = JsonObject.readFrom(getData(identityUrl));

String accessToken = identityObj.get("access_token").asString();

// Compose URLs for Get Lead Changes, and Get Paging Token

String activityTypesUrl = String.format("%s/rest/v1/activities/types.json?access_token=%s",

baseUrl, accessToken);

String pagingTokenUrl = String.format("%s/rest/v1/activities/pagingtoken.json?access_token=%s&sinceDatetime=%s",

baseUrl, accessToken, sinceDateTime);

// Use activity ids to create filter parameter

String activityTypeIds = "";

for (Integer id : ACTIVITY_TYPES.keySet()) {

activityTypeIds += "&activityTypeIds=" + id.toString();

}

// Compose URL for Get Lead Activities

// Only retrieve activities that match our defined activity types

String leadActivitiesUrl = String.format("%s/rest/v1/activities.json?access_token=%s%s&batchSize=%s",

baseUrl, accessToken, activityTypeIds, READ_BATCH_SIZE);

Map<Integer, List> activitiesMap = new HashMap<Integer, List>();

Set leadsSet = new HashSet();

// Call Get Paging Token API

JsonObject pagingTokenObj = JsonObject.readFrom(getData(pagingTokenUrl));

if (pagingTokenObj.get("success").asBoolean()) {

String nextPageToken = pagingTokenObj.get("nextPageToken").asString();

boolean moreResult;

do {

moreResult = false;

// Call Get Lead Activities API to retrieve activity data

JsonObject leadActivitiesObj = JsonObject.readFrom(getData(String.format("%s&nextPageToken=%s",

leadActivitiesUrl, nextPageToken)));

if (leadActivitiesObj.get("success").asBoolean()) {

moreResult = leadActivitiesObj.get("moreResult").asBoolean();

nextPageToken = leadActivitiesObj.get("nextPageToken").asString();

if (leadActivitiesObj.get("result") != null) {

JsonArray activitiesResultAry = leadActivitiesObj.get("result").asArray();

for (JsonValue activitiesResultObj : activitiesResultAry) {

// Extract activity fields of interest (leadID, activityType, activityDate, primaryAttributeValue)

JsonObject leadObj = new JsonObject();

int leadId = activitiesResultObj.asObject().get("leadId").asInt();

leadObj.add("activityType", ACTIVITY_TYPES.get(activitiesResultObj.asObject().get("activityTypeId").asInt()));

leadObj.add("activityDate", activitiesResultObj.asObject().get("activityDate").asString());

leadObj.add("primaryAttributeValue", activitiesResultObj.asObject().get("primaryAttributeValue").asString());

// Store JSON containing activity fields in a map using lead id as key

List leadLst = activitiesMap.get(leadId);

if (leadLst == null) {

activitiesMap.put(leadId, new ArrayList());

leadLst = activitiesMap.get(leadId);

}

leadLst.add(leadObj);

// Store unique lead ids for use as lead filter value below

leadsSet.add(leadId);

}

} while (moreResult);

}

// Use unique lead id values to create filter parameter

String filterValues = "";

for (Object object : leadsSet) {

if (filterValues.length() > 0) {

filterValues += ",";

}

filterValues += String.format("%s", object);

}

// Compose Get Multiple Leads by Filter Type URL

// Only retrieve leads that match the list of lead ids that was accumulated during activity query

String getMultipleLeadsUrl = String.format("%s/rest/v1/leads.json?access_token=%s&filterType=%s&fields=%s&filterValues=%s&batchSize=%s",

baseUrl, accessToken, LEAD_FILTER_TYPE, LEAD_FIELDS, filterValues, READ_BATCH_SIZE);

String nextPageToken = "";

do {

String gmlUrl = getMultipleLeadsUrl;

// Append paging token to Get Multiple Leads by Filter Type URL

if (nextPageToken.length() > 0) {

gmlUrl = String.format("%s&nextPageToken=%s", getMultipleLeadsUrl, nextPageToken);

}

// Call Get Multiple Leads by Filter Type API to retrieve lead data

JsonObject multipleLeadsObj = JsonObject.readFrom(getData(gmlUrl));

if (multipleLeadsObj.get("success").asBoolean()) {

if (multipleLeadsObj.get("result") != null) {

JsonArray multipleLeadsResultAry = multipleLeadsObj.get("result").asArray();

// Iterate through lead data

for (JsonValue leadResultObj : multipleLeadsResultAry) {

int leadId = leadResultObj.asObject().get("id").asInt();

// Join activity data with lead fields of interest (firstName, lastName, email)

List leadLst = activitiesMap.get(leadId);

for (JsonObject leadObj : leadLst) {

leadObj.add("firstName", leadResultObj.asObject().get("firstName").asString());

leadObj.add("lastName", leadResultObj.asObject().get("lastName").asString());

leadObj.add("email", leadResultObj.asObject().get("email").asString());

}

nextPageToken = "";

if (multipleLeadsObj.asObject().get("nextPageToken") != null) {

nextPageToken = multipleLeadsObj.get("nextPageToken").asString();

}

} while (nextPageToken.length() > 0);

// Now place activity data into an array of JSON objects

JsonArray activitiesAry = new JsonArray();

for (Map.Entry<Integer, List> activity : activitiesMap.entrySet()) {

int leadId = activity.getKey();

for (JsonObject leadObj : activity.getValue()) {

// do something with leadId and each leadObj

leadObj.add("leadId", leadId);

activitiesAry.add(leadObj);

}

// Print out result objects

JsonObject result = new JsonObject();

result.add("result", activitiesAry);

System.out.println(result);

System.exit(0);

}

// Perform HTTP GET request

private static String getData(String endpoint) {

String data = "";

try {

URL url = new URL(endpoint);

HttpsURLConnection urlConn = (HttpsURLConnection) url.openConnection();

urlConn.setRequestMethod("GET");

urlConn.setAllowUserInteraction(false);

urlConn.setDoOutput(true);

int responseCode = urlConn.getResponseCode();

if (responseCode == 200) {

InputStream inStream = urlConn.getInputStream();

data = convertStreamToString(inStream);

} else {

System.out.println(responseCode);

data = "Status:" + responseCode;

}

} catch (MalformedURLException e) {

System.out.println("URL not valid.");

} catch (IOException e) {

System.out.println("IOException: " + e.getMessage());

e.printStackTrace();

}

return data;

}

private static String convertStreamToString(InputStream inputStream) {

try {

return new Scanner(inputStream).useDelimiter("A").next();

} catch (NoSuchElementException e) {

return "";

}

That’s it. Happy coding!

Latest Blog Posts