Bulk Custom Object Import

Bulk Custom Object Import Endpoint Reference

When you have a large number of custom object records to  import, it is best practice to import them asynchronously using the bulk API.  This is done by importing a flat file that contains delimited records (comma, tab, or semicolon).  The file can contain any number of records, provided it’s size is less than 10MB (otherwise an HTTP  413 status code is returned).  The contents of the file will depend on your custom object definition.  The first row always contains a header that lists the fields to map values of each row into.  All field names in header must match an API name (as discussed below).  Remaining rows contain the data to import, one record per row.

Processing Limits

You are allowed to submit more than one bulk import request, within limits.  Each request is added as a job to a FIFO queue to be processed.  A maximum of 2 jobs are processed at the same time.  A maximum of 10 jobs are allowed in the queue at any given time (including the 2 currently being processed).  If you exceed the 10 job maximum, then a “1016, Too many imports” error is returned.

Custom Object Example

Before using the bulk API, you must first use the Marketo Admin UI to create your custom object.  As an example, suppose that we created a “Car” custom object with “Color”, “Make”, “Model”, and “VIN” fields. Below are Admin UI screens showing the custom object.  You can see that we used VIN field for deduplication.  The API names are highlighted because they must be used when calling bulk API-related endpoints.

doc-rest-api-bulk-insert-co-car

Here are the custom object fields as presented in the Admin UI.

doc-rest-api-bulk-insert-co-car-fields

API Names

Note that you can retrieve API names programatically by passing the custom object API name to the Describe Custom Object endpoint.

Import File

Now suppose that you want to import three “Car” custom object records.  Using comma-delimited format (CSV), the file could look like this:

Line 1 is the header, and lines 2-4 are the custom object data records.

Creating a Job

To make the bulk import request, you must include the API name of the custom object in the path to the Import Custom Objects endpoint.  You must also include a “file” parameter that references the name of your import file, and a”format” parameter that specifies how your import file is delimited (“csv”, “tsv”, or “ssv”).

In this example, we specified “csv” format and named our import file “custom_object_import.csv”.

Notice in the response to our call, here is no listing of successes or failures like you would get back from Sync Custom Objects endpoint.  Instead, you receive a “batchId”.  This is because the call is asynchronous, and can return a status of “Queued”, “Importing”, or “Failed”.  You should retain the “batchId” so that you can get status of the import job, or retrieve failures and/or warnings upon completion.  Note that the “batchId” remains valid for 7 days.

A simple way to replicate the bulk import request is to use curl from the command line:

Polling Job Status

Once the import job has been created, you need to query it’s status.  It is best practice to poll the import job every 5-30 seconds.  Do this by passing the API name of the custom object and the “batchId” in the path to the Get Import Custom Object Status endpoint.

This response shows a completed import, but the “status” can be one of:

  • Complete
  • Queued
  • Importing
  • Failed

If the job has completed, you will have a listing of the number of rows processed, with failures, and with warnings.  The message attribute is also a good place to look for additional job information.

Failures

Failures are indicated by the “numOfRowsFailed” attribute in Get Import Custom Object Status response.  If “numOfRowsFailed” is greater than zero, then that value indicates the number of failures that occurred.  Call Get Import Custom Object Failure File endpoint to obtain a file with failure detail.  Again, you must pass the custom object API name and “batchId” in the path.  If no failure file exists, an HTTP 404 status code is returned.

Continuing with the example, we can force a failure by modifying the header and change “vin” to ” vin” (by adding a space between the comma and “vin”).

When we re-import and check the status we see this response with “numRowsFailed” : 3.  This indicates three failures.

Now we’ll make Get Import Custom Object Failure File endpoint call to get additional failure detail:

And we can see that we’re missing our deduplication field “vin”.

Warnings

Warnings are indicated by the “numOfRowsWithWarning” attribute in Get Import Custom Object Status response.  If “numOfRowsWithWarning” is greater than zero, then that value indicates the number of warnings that occurred.  Call Get Import Custom Object Warning File endpoint to obtain a file with warning detail.  Again, you must pass the custom object API name and “batchId” in the path.  If no warning file exists, an HTTP 404 status code is returned.