Data Streams

Our customer’s marketing organizations rely on timely and focused Marketing Campaigns to stay on top of their business and be competitive. In order to support fast paced decisions and enable strategic change at speed, it’s important to have data to support and drive those key decisions that deliver focused and targeted Campaigns. There are also some customers that perform Marketing efforts at levels of their customer segments both inside and outside of Marketo Engage. To support these different efforts Adobe Marketo has created the ability to acquire large volumes of data in near real-time thru Data Streams.

Aside from the benefit of near real-time data, there are Product related benefits is it relieves the bottleneck of API limits because streaming is used instead. Reducing the scenario of API limits generates less alert messaging. Also, not having the need to perform bulk exports to extract data due to the Data Streaming capability.

Overviews

Lead Activity Data Stream Overview

Lead Activity Data Stream provides near real-time streaming of audit tracking Lead Activities where large volumes of Lead Activities can be sent to a customer’s external system. Streams enable customers to effectively audit Lead related events, usage patterns, provide views into Lead changes and trigger processes and workflows based upon the different types of Lead Events. There are over 144 Activity Types that can be subscribed to and sent thru the stream.

Types of Lead Data Streamed:

1. Lead Changes – all changes on all fields and new Leads
2. Lead Activities – all Lead activity types described in the document
3. Deleted Leads
4. All Custom Objects on the Lead (if requested). It’s all or nothing at this time.

By providing views into Lead changes, this allows customers to make faster decisions on their overall marketing strategies and create more focused targeted campaigns. Some popular use cases would be:

• Custom Alerting: When certain Leads are found with inconsistent conditions, they can be added to the list. Activity streams can pick these up and push the “Add to List” activity for customers to any follow on action.
• Powering ML Models: Some customers plan to build scoring models that use Activity insights and feed them back to Marketo or use in their own internal Scoring models as desired. By scoring a Lead, customers can then inform Marketo to add customers to Nurture campaigns increase their scoring.

Current List of Streamed Activities:

One point to note is that if custom objects are desired to be streamed, it would have to be all of the Lead related custom objects. There isn’t a way at the present time to select which ones are desired.

User Audit Data Stream Overview

User Audit Data Stream provides near real-time audit tracking of asset changes by users​. This enables a customer to effectively audit asset events, provide a view into user changes, and trigger processes or workflows based upon different types of audit events. Near real-time asset changes are sent via Adobe I/O events to a configurable endpoint. Audit events are broken down by Asset type and can subscribe to audit events that are important to them.

A good use case for subscribing to this stream would be:

• Tracking changes when using multiple Marketing Systems: There are some customers who also perform some level of marketing activities in another system such as a CRM like Salesforce and then pass the Lead to Marketo. The Lead at times gets updated and synched back and forth so it is important to track which system has made recent changes.

Current List of Streamed Audit Events:

Example of an Audit Event:

Notification Data Stream Overview

Currently, the notification center in Marketo can be configured to send notifications to an email address. Notification Data Stream enables the notifications to be sent directly to a configurable endpoint via Adobe I/O events. Notifications are provided thru the UI today and can be referenced by the orange bell in the top right of the screen and this stream takes those notifications and sends them thru a stream.

Current Notification Events:

Example Event:

Technical Details

This section provides guidelines on what is needed, how to connect and receive streaming data for each of the streams. There is some level of coding and setup involved for each.

Lead Activity Data Stream

The Lead Activity Stream provides near real-time streaming of Marketo Lead Activity events and sends subscribed activity type changes with configurable attributes:

• Frequency of data pushes every 2 seconds by default.
• Batches from 100 to 500 per subscription.
• Timeout for customer REST service 20 seconds with 3 retries after which will be paused.
• Data retention in a queue for up to 7 days.

To implement the Lead Activity Data Stream, here are the steps for customers to follow:

1. Expose an HTTP endpoint that can receive POST requests with a JSON body from the public internet, that the Activity Push Data Stream will send requests to
2. Provide Adobe with the following:
   1. Marketo Munchkin ID for their subscription
   2. The URL of the endpoint in step 1
   3. The Activity types they wish to receive (complete list above)
   4. A means of authentication, so that the customer can verify that the requests are legitimate. Either:
      1. An identity provider URL, Client ID, and Client Secret for OAuth Client Credentials Authentication (https://www.oauth.com/oauth2-servers/access-tokens/client-credentials/)
      2. An API token, which can be included in requests sent by the Lead Activity Datastream either in query parameters, or in an Authorization header (customer’s choice)
3. Adobe will enable the datastream, at which point customers will begin to receive data.

UML diagram of a typical Lead Activity Data Stream call:

Example of URL Endpoint Creation

Here is a link that provides sample code to create a URL Endpoint for assistance.

https://git.corp.adobe.com/OpenSourceAdvisoryBoard/opensource_submission_process/issues/1348

User Audit Data Stream and Notification Data Stream

User Audit events are sent to Adobe IO and can be consumed by logging in with an Adobe ID. Here are the steps to follow:

1. Customers provide Adobe with the following:
   1. Adobe ID
   2. Marketo Munchkin ID for their subscription
2. Customer exposes a REST endpoint to consume events normally in the form of a webhook.
3. Once that is provided, Adobe will enable the stream for the customer’s subscription.
4. Customer will then setup the stream in Adobe IO (instructions to be provided)
   1. This step requires an Adobe Org
   2. Requires Adobe Org User to have Developer or System Admin Role

To setup Adobe IO follow the instructions on page Setting up Marketo User Audit Data Streams with Adobe IO in the Public Documentation section.

Public Documentation

• Data Streams Blog
• Setting up Marketo User Audit Data Stream with Adobe IO Events
• Marketo REST API
• Marketo Endpoint Reference
• Intro to Adobe IO Events & Webhooks
• Setting up Marketo User Audit Data Stream with Adobe IO Events