Data Streams

Our customer’s marketing organizations rely on timely and focused Marketing Campaigns to stay on top of their business and be competitive. In order to support fast paced decisions and enable strategic change at speed, it’s important to have data to support and drive those key decisions that deliver focused and targeted Campaigns. There are also some customers that perform Marketing efforts at levels of their customer segments both inside and outside of Marketo Engage. To support these different efforts Adobe Marketo has created the ability to acquire large volumes of data in near real-time thru Data Streams.

Aside from the benefit of near real-time data, there are Product related benefits is it relieves the bottleneck of API limits because streaming is used instead. Reducing the scenario of API limits generates less alert messaging. Also, not having the need to perform bulk exports to extract data due to the Data Streaming capability.

Overviews

Lead Activity Data Stream Overview

Lead Activity Data Stream provides near real-time streaming of audit tracking Lead Activities where large volumes of Lead Activities can be sent to a customer’s external system. Streams enable customers to effectively audit Lead related events, usage patterns, provide views into Lead changes and trigger processes and workflows based upon the different types of Lead Events. There are over 144 Activity Types that can be subscribed to and sent thru the stream.

Types of Lead Data Streamed:

1. Lead Changes – all changes on all fields and new Leads
2. Lead Activities – all Lead activity types described in the document
3. Deleted Leads
4. All Custom Objects on the Lead (if requested). It’s all or nothing at this time.

By providing views into Lead changes, this allows customers to make faster decisions on their overall marketing strategies and create more focused targeted campaigns. Some popular use cases would be:

• Custom Alerting: When certain Leads are found with inconsistent conditions, they can be added to the list. Activity streams can pick these up and push the “Add to List” activity for customers to any follow on action.
• Powering ML Models: Some customers plan to build scoring models that use Activity insights and feed them back to Marketo or use in their own internal Scoring models as desired. By scoring a Lead, customers can then inform Marketo to add customers to Nurture campaigns increase their scoring.

List of Streamed Activities:

One point to note is that if custom objects are desired to be streamed, it would have to be all of the Lead related custom objects. There isn’t a way at the present time to select which ones are desired.

User Audit Data Stream Overview

User Audit Data Stream provides near real-time audit tracking of asset changes by users​. This enables a customer to effectively audit asset events, provide a view into user changes, and trigger processes or workflows based upon different types of audit events. Near real-time asset changes are sent via Adobe I/O events to a configurable endpoint. Audit events are broken down by Asset type and can subscribe to audit events that are important to them.

A good use case for subscribing to this stream would be:

• Tracking changes when using multiple Marketing Systems: There are some customers who also perform some level of marketing activities in another system such as a CRM like Salesforce and then pass the Lead to Marketo. The Lead at times gets updated and synched back and forth so it is important to track which system has made recent changes.

List of Streamed User Audit Events:

Example of User Audit Event:

Notification Data Stream Overview

Currently, the notification center in Marketo can be configured to send notifications to an email address. Notification Data Stream enables the notifications to be sent directly to a configurable endpoint via Adobe I/O events. Notifications are provided thru the UI today and can be referenced by the orange bell in the top right of the screen and this stream takes those notifications and sends them thru a stream.

List of Notification Events:

Example of Notification Event:

Technical Details

This section provides guidelines on what is needed, how to connect and receive streaming data for each of the streams. There is some level of coding and setup involved for each.

Lead Activity Data Stream

The Lead Activity Stream provides near real-time streaming of Marketo Lead Activity events and sends subscribed activity type changes with configurable attributes:

• Frequency of data pushes every 2 seconds by default.
• Batches from 100 to 500 per subscription.
• Timeout for customer REST service 20 seconds with 3 retries every 3 minutes and auto enabled upon success, otherwise after which will be paused. Once its paused, the service retries every 3 minutes in an attempt to re-enable unless de-provisioned manually.
• Data retention in a queue for up to 7 days.

To implement the Lead Activity Data Stream, here are the steps for customers to follow:

1. Expose an HTTP endpoint that can receive POST requests with a JSON body from the public internet, that the Activity Push Data Stream will send requests to
2. Provide Adobe with the following:
   1. Marketo Munchkin ID for their subscription
   2. The URL of the endpoint in step 1
   3. The Activity types they wish to receive (complete list above)
   4. A means of authentication, so that the customer can verify that the requests are legitimate. Either:
      1. An identity provider URL, Client ID, and Client Secret for OAuth Client Credentials Authentication (https://www.oauth.com/oauth2-servers/access-tokens/client-credentials/)
      2. An API token, which can be included in requests sent by the Lead Activity Datastream either in query parameters, or in an Authorization header (customer’s choice)
3. Adobe will enable the datastream, at which point customers will begin to receive data.

UML diagram of a typical Lead Activity Data Stream call:

Example of URL Endpoint Creation:

A code sample for an application that consumes the Marketo Lead Activity Data Stream can be found here.

User Audit Data Stream and Notification Data Stream

User Audit events are sent to Adobe IO and can be consumed by logging in with an Adobe ID. Here are the steps to follow:

1. Customers provide Adobe with the following:
   1. Adobe ID
   2. Marketo Munchkin ID for their subscription
2. Customer exposes a REST endpoint to consume events normally in the form of a webhook.
3. Once that is provided, Adobe will enable the stream for the customer’s subscription.
4. Customer will then setup the stream in Adobe IO (instructions to be provided)
   1. This step requires an Adobe Org
   2. Requires Adobe Org User to have Developer or System Admin Role

To setup Adobe IO follow the instructions on page Setting up Marketo User Audit Data Streams with Adobe IO in the Public Documentation section.

Setting Up The User Audit Data Stream in Marketo

The User Audit Data Stream is currently available as part of the Performance packages along with the other 3 Data Streams. For more information on the Packages, refer to the Product Description Page for Product limits and features.

Setting up Adobe I/O

See Getting Started with Adobe I/O Events

For basic instructions for this use case, starting from console.adobe.io:

When prompted, click the designated button to proceed  either

Select  -> Create New Project Or

Select ->  Add Event.

Get started with your new project

Welcome to your new empy project. To start using Adobe services, add an API, events or runtime. For further guidance on projects and available services, view our documentation.

Public Documentation

• Marketo Data Streams
• Intro to Adobe IO Events & Webhooks
• Data Streams Blog