Create a Dashboard for API Usage and Error Counts

As mentioned in an earlier post, you can view API usage data for the prior week by going to the Admin > Web Services panel in Marketo. This gives you a daily rollup of the number of calls made by user. As a Marketo API consumer, this is useful information that you should keep an eye on.

What if you could get historical usage data to help detect trends over time?

What if you could get a summary of API error codes to help measure the health of your integration?

As a Marketo Technology Partner, what if you could get usage and error data across all of your customer accounts in one dashboard?

This post will provide an approach to answering the questions above.

Buckle up, here we go!

Diagram

Here is a high level diagram of the dashboard system.

Scheduled Job for Stats Retrieval

Let’s create an app that retrieves usage and error data using Get Daily Usage and Get Daily Errors endpoints. The app is designed to be scheduled to run once per day. Each time the app runs, it appends a day’s worth of usage data to one file, and a day’s worth of error data to another file. At the beginning of each month, a new pair of files are created. These files will serve as a historical record that we can access at any time.

Here is the app logic …

Read Marketo account information (Munchkin id and Client credentials) from an external source . Note: This source must be secure to keep others from accessing account data.
Iterate through each account and…
- Call Get Daily Usage to retrieve usage data for one day
- Append daily usage data to a montly “usage” file
- Call Get Daily Errors to retrieve errors data for one day
- Append daily errors data to a monthly “errors” file

Output File Format

The format for the output files is JSON which matches up with the “result” array returned from the respective API calls (Usage and Error). Each element of the “result” array is a JSON object that contains data for one day.

Output File Naming

The output files are named as follows:

<type>_<yyyy>_<mm>_<account>.json

where,

<type> – The type of data (“usage” or “errors”)
<yyyy> – The year (4-digits)
<mm> – The month (2-digits)
<account> – The account id (Munchkin id)

Output File Examples

usage_2015_10_111-AAA-222.json

[ 
    { "date": "2015-10-15", "total": 0, "users" : [] }, 
    { "date": "2015-10-16", "total": 9, "users": [ { "userId": "some.body@yahoo.com", "count": 9 } ] }, 
    { "date": "2015-10-17", "total": 1120, "users": [ { "userId": "some.body@yahoo.com", "count": 200 }, { "userId": "some.body@marketo.com", "count": 200 }, { "userId": "some.body@gmail.com", "count": 720 } ] },
<span class="pun">]</span>

[

{ "date": "2015-10-15", "total": 0, "users" : [] },

{ "date": "2015-10-16", "total": 9, "users": [ { "userId": "some.body@yahoo.com", "count": 9 } ] },

{ "date": "2015-10-17", "total": 1120, "users": [ { "userId": "some.body@yahoo.com", "count": 200 }, { "userId": "some.body@marketo.com", "count": 200 }, { "userId": "some.body@gmail.com", "count": 720 } ] },

errors_2015_10_111-AAA-222.json

[
    { "date": "2015-10-15", "total":80, "errors": [ { "errorCode":"1003", "count":80 } ] },
    { "date": "2015-10-16", "total":148, "errors": [ { "errorCode":"612", "count":40 }, { "errorCode":"609", "count":70 }, { "errorCode":"1008", "count":38 } ] },
    { "date": "2015-10-17", "total":73, "errors": [ { "errorCode":"604", "count":1 }, { "errorCode":"609", "count":56 }, { "errorCode":"610", "count":16 } ] },
]

[

{ "date": "2015-10-15", "total":80, "errors": [ { "errorCode":"1003", "count":80 } ] },

{ "date": "2015-10-16", "total":148, "errors": [ { "errorCode":"612", "count":40 }, { "errorCode":"609", "count":70 }, { "errorCode":"1008", "count":38 } ] },

{ "date": "2015-10-17", "total":73, "errors": [ { "errorCode":"604", "count":1 }, { "errorCode":"609", "count":56 }, { "errorCode":"610", "count":16 } ] },

]

Code for this app is presented at the end of this post (Stats.java).

Stats Web Service

So now we need a way to get this data into our browser. The proposal is to create a web service to deliver the data. The web service will consume the app’s output files and then return data to the browser in a form that can be readily presented. For simplicity’s sake, and to get around same-origin policy restrictions, we will leverage the JSONP pattern.

Here is the proposed REST endpoint specification for the external web service:

URI: /stats

Method: GET

Parameter	Description	Example
month	Retrieve data for this month. Comma seperated list of months to include (2-digit representation). Default to all months.	10,11
year	Retrieve data for this year. Comma seperated list of years to include (4-digit representation). Default to all years.	2015
account	Retrieve data for this account (Munchkin id).	111-AAA-222
callback	Name of function to wrap JSON content with.	processStats

Example Request

GET //localhost:8080/stats?month=10&year=2015&account=111-AAA-222&callback=processStats

The web service reads “usage” and “error” files, and combines them together and returns them in this format:

<Name of Callback here>(

<Contents of Usage file here>,

);

This is a JSONP callback with 2 arguments. First argument is usage “result” array. Second argument is errors “result” array.

Example Response

<strong>processStats(</strong>
    [
        { "date": "2015-10-15", "total": 0, "users" : [] },
        { "date": "2015-10-16", "total": 9, "users": [ { "userId": "some.body@yahoo.com", "count": 9 } ] },
        { "date": "2015-10-17", "total": 1120, "users": [ { "userId": "some.body@yahoo.com", "count": 200 }, { "userId": "some.body@marketo.com", "count": 200 }, { "userId": "some.body@gmail.com", "count": 720 } ] }
    ]<strong>,</strong>
    [
        { "date": "2015-10-15", "total":80, "errors": [ { "errorCode":"1003", "count":80 } ] },
        { "date": "2015-10-16", "total":148, "errors": [ { "errorCode":"612", "count":40 }, { "errorCode":"609", "count":70 }, { "errorCode":"1008", "count":38 } ] },
        { "date": "2015-10-17", "total":73, "errors": [ { "errorCode":"604", "count":1 }, { "errorCode":"609", "count":56 }, { "errorCode":"610", "count":16 } ] },
    ]
<strong>);</strong>

<strong>processStats(</strong>

[

{ "date": "2015-10-15", "total": 0, "users" : [] },

{ "date": "2015-10-16", "total": 9, "users": [ { "userId": "some.body@yahoo.com", "count": 9 } ] },

]<strong>,</strong>

[

{ "date": "2015-10-15", "total":80, "errors": [ { "errorCode":"1003", "count":80 } ] },

{ "date": "2015-10-16", "total":148, "errors": [ { "errorCode":"612", "count":40 }, { "errorCode":"609", "count":70 }, { "errorCode":"1008", "count":38 } ] },

{ "date": "2015-10-17", "total":73, "errors": [ { "errorCode":"604", "count":1 }, { "errorCode":"609", "count":56 }, { "errorCode":"610", "count":16 } ] },

]

As you can see, the web service has simply wrapped the contents of the two output files from our app. We have created a mock web service response using Mocky. An example of the web service the mock is here.

Creation of this web service is left as an exercise for the reader 🙂

Dashboard Web Page

So now all we need is a web page that calls our web service and formats the data.

To use the JSONP pattern we just need to add a <script> tag that invokes the web service:

This will inject the web service response body directly into the HTML page.

We then add the JSONP callback function :

function processStats(usage, errors) {
    var cfg = { maxDepth: 5};
    document.write("<h2>Usage</h2>");
    document.body.appendChild(prettyPrint(usage, cfg));
    document.write("<h2>Errors</h2>");
    document.body.appendChild(prettyPrint(errors, cfg));
;

function processStats(usage, errors) {

var cfg = { maxDepth: 5};

document.write("<h2>Usage</h2>");

document.body.appendChild(prettyPrint(usage, cfg));

document.write("<h2>Errors</h2>");

document.body.appendChild(prettyPrint(errors, cfg));

;

This function is automatically called after the web service call. In this example, we call a simple JavaScript “variable dumper” called prettyPrint.js on each array. The prettyPrint function simply produces an HTML table using the contents of the array.

Here is a screen shot of the HTML tables:

Voilà, that is our our dashboard!

Granted this isn’t very elegant, but it should give you an idea of what is possible. There is nothing stopping you from transforming the data any way you like to make your own eye catching visualizations.

The HTML page is below (Index.html)

You can view the tables above live in your browser using the following steps:

Save a local copy of Index.html
Save a local copy of prettyPrint.js
Open Index.html in your browser

So that’s it. Hopefully this post has given you some ideas about monitoring your Marketo API stats. Happy coding!

Stats.java

package com.marketo;

// minimal-json library (https://github.com/ralfstx/minimal-json)
import com.eclipsesource.json.JsonArray;
import com.eclipsesource.json.JsonObject;

import java.io.*;
import java.lang.reflect.Array;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.*;

import java.nio.file.Files;
import java.nio.file.Paths;

import javax.net.ssl.HttpsURLConnection;

public class Stats {
    //
    // Define Marketo instance meta data here. Each row contains three elements: Account Id, Client Id, Client Secret.
    //
    // Note: that this information would typically be stored read from an external source (i.e. database)
    //
    // For example:
    //
    // private static String final CUSTOM_SERVICE_DATA[][] = {
    // {"111-AAA-222", "2f4a4435-f6fa-4bd9-3248-098754982345", "asdf6IVE9h4Jjcl59cOMAKFSk78ut12W"},
    // {"222-BBB-333", "5f4a6657-f6fa-4cd9-4356-123083238821", "gfjgfIVE9h4Jjcl59cOMAKFSk78ut12W"},
    // {"444-CCC-444", "9f4a4678-f6fa-4dd9-7735-908713247721", "xzcxvbVE9h4Jjcl59cOMAKFSk78ut12W"}
    // };
    //
    private static final String CUSTOM_SERVICE_DATA[][] = {
    };

    // Output directory for stats files
    private static final String OUTPUT_DIR = "C:stats";

public static void main(String[] args) {

    // Loop through each Marketo instance
    for (int i = 0; i < CUSTOM_SERVICE_DATA.length; i++) {

        // Compose base URL
        String baseUrl = String.format("https://%s.mktorest.com",
            CUSTOM_SERVICE_DATA[i][0]);

        // Compose Identity URL
        String identityUrl = String.format("%s/identity/oauth/token?grant_type=%s&client_id=%s&client_secret=%s",
            baseUrl, "client_credentials", CUSTOM_SERVICE_DATA[i][1], CUSTOM_SERVICE_DATA[i][2]);

        // Call Identity API
        JsonObject identityObj = JsonObject.readFrom(httpRequest("GET", identityUrl, null));
        String accessToken = identityObj.get("access_token").asString();

        // Compose Get Last 7 Days Usage URL
        String usageUrl = String.format("%s/rest/v1/stats/usage/last7days.json?access_token=%s",
            baseUrl, accessToken);

        // Compose Get Last 7 Days Errors URL
        String errorsUrl = String.format("%s/rest/v1/stats/errors/last7days.json?access_token=%s",
            baseUrl, accessToken);

        // Process usage data
        JsonObject usageObj = JsonObject.readFrom(httpRequest("GET", usageUrl, null));
        if (usageObj.get("success").asBoolean()) {
            if (usageObj.get("result") != null) {
                updateFile(usageObj, "usage", CUSTOM_SERVICE_DATA[i][0]);
            }
        }

        // Process errors data
        JsonObject errorsObj = JsonObject.readFrom(httpRequest("GET", errorsUrl, null));
        if (usageObj.get("success").asBoolean()) {
            if (errorsObj.get("result") != null) {
                updateFile(errorsObj, "errors", CUSTOM_SERVICE_DATA[i][0]);
            }
        }
    }
    System.exit(0);
}

// Write yesterday's data to output file
private static void updateFile(JsonObject usageObj, String statsType, String account){

    JsonArray usageResultAry = usageObj.get("result").asArray();
    JsonObject yesterdayUsageResultObj = usageResultAry.get(1).asObject();

    String yesterdayDate = yesterdayUsageResultObj.get("date").asString();
    String[] yesterdayDateAry = yesterdayDate.split("[-]+");

    // "C:statsstats_yyyy_mm_account.json"
    String statsFile = String.format("%s%s_%s_%s_%s.json",
        OUTPUT_DIR, statsType, yesterdayDateAry[0], yesterdayDateAry[1], account);

    // Create file
    File file = new File(statsFile);
    try {
        if (file.createNewFile()) {
            // created new file, seed with empty array
            FileWriter fw = new FileWriter(file.getAbsoluteFile());
            BufferedWriter bw = new BufferedWriter(fw);
            bw.write("[n]");
            bw.close();
        }
    } catch (IOException e) {
        e.printStackTrace();
    }

    // Read file
    String content = null;
    try {
        content = new String(Files.readAllBytes(Paths.get(statsFile)));
    } catch (IOException e) {
        e.printStackTrace();
    }

    // Remove trailing "]", append new record, append trailing "]"
    content = content.substring(0, content.length() - 1);
    content += yesterdayUsageResultObj.toString();
    content += "n]";

    // Write file
    FileWriter fw = null;
    try {
        fw = new FileWriter(file.getAbsoluteFile());
        BufferedWriter bw = new BufferedWriter(fw);
        bw.write(content);
        bw.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

// Perform HTTP request
private static String httpRequest(String method, String endpoint, String body) {
    String data = "";
    try {
        URL url = new URL(endpoint);
        HttpsURLConnection urlConn = (HttpsURLConnection) url.openConnection();
        urlConn.setDoOutput(true);
        urlConn.setRequestMethod(method);
        switch (method) {
            case "GET":
                break;
            case "POST":
                urlConn.setRequestProperty("Content-type", "application/json");
                urlConn.setRequestProperty("accept", "text/json");
                OutputStreamWriter wr = new OutputStreamWriter(urlConn.getOutputStream());
                wr.write(body);
                wr.flush();
                break;
            default:
                System.out.println("Error: Invalid method.");
                return data;
        }
        int responseCode = urlConn.getResponseCode();
        if (responseCode == 200) {
            InputStream inStream = urlConn.getInputStream();
            data = convertStreamToString(inStream);
        } else {
            System.out.println(responseCode);
            data = "Status:" + responseCode;
        }
    } catch (MalformedURLException e) {
        System.out.println("URL not valid.");
    } catch (IOException e) {
        System.out.println("IOException: " + e.getMessage());
        e.printStackTrace();
    }
    return data;
}

private static String convertStreamToString(InputStream inputStream) {
    try {
        return new Scanner(inputStream).useDelimiter("A").next();
    } catch (NoSuchElementException e) {
        return "";
    }
}
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

package com.marketo;

// minimal-json library (https://github.com/ralfstx/minimal-json)

import com.eclipsesource.json.JsonArray;

import com.eclipsesource.json.JsonObject;

import java.io.*;

import java.lang.reflect.Array;

import java.net.MalformedURLException;

import java.net.URL;

import java.util.*;

import java.nio.file.Files;

import java.nio.file.Paths;

import javax.net.ssl.HttpsURLConnection;

public class Stats {

// Define Marketo instance meta data here. Each row contains three elements: Account Id, Client Id, Client Secret.

// Note: that this information would typically be stored read from an external source (i.e. database)

// For example:

// private static String final CUSTOM_SERVICE_DATA[][] = {

// {"111-AAA-222", "2f4a4435-f6fa-4bd9-3248-098754982345", "asdf6IVE9h4Jjcl59cOMAKFSk78ut12W"},

// {"222-BBB-333", "5f4a6657-f6fa-4cd9-4356-123083238821", "gfjgfIVE9h4Jjcl59cOMAKFSk78ut12W"},

// {"444-CCC-444", "9f4a4678-f6fa-4dd9-7735-908713247721", "xzcxvbVE9h4Jjcl59cOMAKFSk78ut12W"}

// };

private static final String CUSTOM_SERVICE_DATA[][] = {

};

// Output directory for stats files

private static final String OUTPUT_DIR = "C:stats";

public static void main(String[] args) {

// Loop through each Marketo instance

for (int i = 0; i < CUSTOM_SERVICE_DATA.length; i++) {

// Compose base URL

String baseUrl = String.format("https://%s.mktorest.com",

CUSTOM_SERVICE_DATA[i][0]);

// Compose Identity URL

String identityUrl = String.format("%s/identity/oauth/token?grant_type=%s&client_id=%s&client_secret=%s",

baseUrl, "client_credentials", CUSTOM_SERVICE_DATA[i][1], CUSTOM_SERVICE_DATA[i][2]);

// Call Identity API

JsonObject identityObj = JsonObject.readFrom(httpRequest("GET", identityUrl, null));

String accessToken = identityObj.get("access_token").asString();

// Compose Get Last 7 Days Usage URL

String usageUrl = String.format("%s/rest/v1/stats/usage/last7days.json?access_token=%s",

baseUrl, accessToken);

// Compose Get Last 7 Days Errors URL

String errorsUrl = String.format("%s/rest/v1/stats/errors/last7days.json?access_token=%s",

baseUrl, accessToken);

// Process usage data

JsonObject usageObj = JsonObject.readFrom(httpRequest("GET", usageUrl, null));

if (usageObj.get("success").asBoolean()) {

if (usageObj.get("result") != null) {

updateFile(usageObj, "usage", CUSTOM_SERVICE_DATA[i][0]);

}

// Process errors data

JsonObject errorsObj = JsonObject.readFrom(httpRequest("GET", errorsUrl, null));

if (usageObj.get("success").asBoolean()) {

if (errorsObj.get("result") != null) {

updateFile(errorsObj, "errors", CUSTOM_SERVICE_DATA[i][0]);

}

System.exit(0);

}

// Write yesterday's data to output file

private static void updateFile(JsonObject usageObj, String statsType, String account){

JsonArray usageResultAry = usageObj.get("result").asArray();

JsonObject yesterdayUsageResultObj = usageResultAry.get(1).asObject();

String yesterdayDate = yesterdayUsageResultObj.get("date").asString();

String[] yesterdayDateAry = yesterdayDate.split("[-]+");

// "C:statsstats_yyyy_mm_account.json"

String statsFile = String.format("%s%s_%s_%s_%s.json",

OUTPUT_DIR, statsType, yesterdayDateAry[0], yesterdayDateAry[1], account);

// Create file

File file = new File(statsFile);

try {

if (file.createNewFile()) {

// created new file, seed with empty array

FileWriter fw = new FileWriter(file.getAbsoluteFile());

BufferedWriter bw = new BufferedWriter(fw);

bw.write("[n]");

bw.close();

}

} catch (IOException e) {

e.printStackTrace();

}

// Read file

String content = null;

try {

content = new String(Files.readAllBytes(Paths.get(statsFile)));

} catch (IOException e) {

e.printStackTrace();

}

// Remove trailing "]", append new record, append trailing "]"

content = content.substring(0, content.length() - 1);

content += yesterdayUsageResultObj.toString();

content += "n]";

// Write file

FileWriter fw = null;

try {

fw = new FileWriter(file.getAbsoluteFile());

BufferedWriter bw = new BufferedWriter(fw);

bw.write(content);

bw.close();

} catch (IOException e) {

e.printStackTrace();

}

// Perform HTTP request

private static String httpRequest(String method, String endpoint, String body) {

String data = "";

try {

URL url = new URL(endpoint);

HttpsURLConnection urlConn = (HttpsURLConnection) url.openConnection();

urlConn.setDoOutput(true);

urlConn.setRequestMethod(method);

switch (method) {

case "GET":

break;

case "POST":

urlConn.setRequestProperty("Content-type", "application/json");

urlConn.setRequestProperty("accept", "text/json");

OutputStreamWriter wr = new OutputStreamWriter(urlConn.getOutputStream());

wr.write(body);

wr.flush();

break;

default:

System.out.println("Error: Invalid method.");

return data;

}

int responseCode = urlConn.getResponseCode();

if (responseCode == 200) {

InputStream inStream = urlConn.getInputStream();

data = convertStreamToString(inStream);

} else {

System.out.println(responseCode);

data = "Status:" + responseCode;

}

} catch (MalformedURLException e) {

System.out.println("URL not valid.");

} catch (IOException e) {

System.out.println("IOException: " + e.getMessage());

e.printStackTrace();

}

return data;

}

private static String convertStreamToString(InputStream inputStream) {

try {

return new Scanner(inputStream).useDelimiter("A").next();

} catch (NoSuchElementException e) {

return "";

}

Index.html

<html>
<head>
<title>Marketo API Stats</title>
<!-- Browser JavaScript variable dumper                  -->
<!-- https://github.com/padolsey-archive/prettyprint.js  -->
<script src="prettyPrint.js"></script>
</head>
<body>

<h1>Marketo API Stats</h1>

<script>
// JSONP callback that uses prettyPrint to format API stats
function processStats(usage, errors) {
    var cfg = { maxDepth: 5};
    document.write("<h2>Usage</h2>");
    document.body.appendChild(prettyPrint(usage, cfg));
    document.write("<h2>Errors</h2>");
    document.body.appendChild(prettyPrint(errors, cfg));
};
</script>

<!-- Web service for you to implement as an exercise                                                        -->
<!-- <script src="http://localhost:8080/stats?month=10&account=111-AAA-222&callback=processStats"></script> -->

<!-- Mock web service that returns sample payload -->
<!-- http://www.mocky.io/                         -->
<script src="http://www.mocky.io/v2/5627b2f9270000f2226eec63?month=10&year=2015&account=111-AAA-222&callback=processStats"></script>"
</body>
</html>

<html>

<head>

<title>Marketo API Stats</title>

</head>

<body>

<h1>Marketo API Stats</h1>

// JSONP callback that uses prettyPrint to format API stats

function processStats(usage, errors) {

var cfg = { maxDepth: 5};

document.write("<h2>Usage</h2>");

document.body.appendChild(prettyPrint(usage, cfg));

document.write("<h2>Errors</h2>");

document.body.appendChild(prettyPrint(errors, cfg));

};

</script>

</body>

</html>

Latest Blog Posts