Bulk Import API overview

The bulk Import API allows you to import supporter or transactional data using a token, from outside the system. If you wish to import data using the front-end functions, then click here.

Overview

An example implementation of the import API as an HTML form can be accessed at the following URLs:

Canadian server:

https://ca.engagingnetworks.app/ea-dataservice/import.jsp

U.S. server:

https://us.engagingnetworks.app/ea-dataservice/import.jsp

The import API is set up so that clients can “push” their user data files to the Engaging Networks software automatically, by submitting standard https multipart POST method, using the URLs:

https://ca.engagingnetworks.app/ea-dataservice/import.service

https://us.engagingnetworks.app/ea-dataservice/import.service

The required parameters for the service are:

token – a private token set up by the client using the admin dashboard
name – a reference name for their import
upload – the supporter data file you wish to upload (csv format)
formatName -This depends on whether you are importing supporter or transactional data:

(a) For a supporter data import: A saved format name, created using the standard manual user data import process in your account (which fields and in what order should be imported).

(b) For a transactional data import: The parameter should be given the fixed value importTransactions

Clients can write an application in the language of their choice to submit import files to the service. When the file is imported, the client will receive an email regarding the status of the import.

There are also a number of optional parameters:

• segmentId
• segmentName

These parameters are intended for integration between Engaging Networks and an external CRM (constituent relationship management) system. They are used to assign a batch of supporters to a segment within a client’s list. The details are explained below in the section ‘Using CRM segments to help integrate with an external CRM’.

Using the import API to import supporter data

Preparing the supporter import file

Before submitting any files to the import service, a few things must be in place:

1. You must know a private token key for the subaccount. If you do not have a private token, log into the admin dashboard for your subaccount and navigate to Hello Your name > Account settings > Tokens. Here you can either check your private key if one already exists, or generate a new one. One private key can be generated for each dashboard user. Import jobs created via the API will be assigned to the user associated with the private token used to invoke the API, and will be visible in the jobs monitor in the corresponding user dashboard.
2. For each data column you want to import, there must be a corresponding field in your account’s default supporter record. You do not have to import every column in the csv file – the saved format you specify determines which columns are imported and which are ignored. See below for an example. Note: Files should not exceed 20MB
3. You must have manually uploaded some user data through the client dashboard, with the same data structure as the one you want with the import API, and have created a saved format in the process. The saved format tells the software how to map columns in the imported file to fields in the default user record in your account.

Submitting the supporter import file

Once this is done, you can submit import files to the service by following these steps:

1. Go to https://ca.engagingnetworks.app/ea-dataservice/import.jsp (Canada server) or https://us.engagingnetworks.app/ea-dataservice/import.jsp (U.S. server)
2. Enter the private token for your user account
3. Enter the location of your csv import file
4. Enter the name of the saved format you wish to use (N.B. this must match the format of the import file)

Once the file is received it will be scheduled for import to your account – a notification email will be sent when the import is complete. You can check the status of the job by logging into the user account associated with the private token you have used. As mentioned in the overview, you can also write an application in the language of your choice to submit to the above URL.

Note that an email address column is always required, but you can import rows with a blank email address field. For example, if you want to import data for offline supporters. The software will randomly generate a dummy email address that will be used as a unique key for that offline supporter in the future.

Example supporter data import format

Suppose you wanted to import a csv file of supporter data in the following format:

First Name	Last Name	Email Address	PostCode	OPT-IN #1	Campaign ABC
Supporter	One	[email protected]	123ABC	Y	P
Supporter	Two	[email protected]	456DEF	N	P

Before importing, you would need to make sure that a field existed in the default user record for every ‘user data’ field you wanted to import (ie the first four columns.), and that a saved format had been created that mapped each column to the corresponding user record field.

If you also wished to import the question and campaign data in the last two columns, your specified saved format would need to map the ‘OPT-IN #1’ column to an existing question in your account, and ‘Campaign ABC’ to an existing campaign.

For more details on preparing your import files, consult this page.

Using CRM segments to help integrate with an external CRM

When importing supporter data, the supporters in a particular batch can be associated with an external CRM by submitting the segment parameters segmentId and segmentName as part of your form post.

The values of these parameters should be the desired ID and reference name of the CRM segment, respectively. If segmentName is left blank, the reference name of the created segment will be the same as the ID.

Existing CRM segments can be viewed under Data & Reports > External CRM Segments and can be queried on in the query builder.

For example: I have 20,000 supporters imported from a segment called “2014-07-01-SEGMENT”. Here I would include that name in my import as “segmentName”. This would later allow me to select all supporters who formed part of that import.

In the following video, Ben Childers, Director of Account Services walks you through using the External CRM Segment tools.

Troubleshooting importing supporter data

If you have successfully imported data but cannot see it in the platform, note that it can take up to 20 minutes before the data is available for live use.

If you see the error “Error: reference name is a required parameter”, ensure you have specified a valid saved import name and included it in formatName. Saved imports are created when using the front-end Data & Reports > Import tool.

Using the import API to import transactional data

Preparing the transactional import file

As with a supporter data import, before submitting any files to the import service, a few steps must be taken:

1. Any supporter you’re importing transactional data for needs to already have an existing constituent record in the database. If they don’t exist, you’ll need to create them via lookup supporters or by supporter data import. You can reference them with either their email address or their supporter ID.
2. You must know a private token key for the subaccount. If you do not have a private token, log into the admin dashboard for your subaccount and navigate to Hello Your name > Account settings > Tokens. Here you can either check your private key if one already exists, or generate a new one. One private key can be generated for each dashboard user. Import jobs created via the API will be assigned to the user associated the private token used to invoke the API, and will be visible in the jobs monitor in the corresponding user dashboard.
3. Your import file must be formatted in a standard way, as defined in the Transactional Data Import Template spreadsheet (see attachment). Each upload file should only contain rows for a single campaign ID. If you want to import data for several campaigns, this needs to be done in separate files.
4. The campaign ID column in the import file needs to be the numerical ID of the campaign in the Engaging Networks system. For example, for an advocacy campaign, the ID is the ea.campaign.id parameter that appears in the bottom of the source code of your advocacy page. You therefore will need to have a pre-created campaign in the Engaging Networks dashboard with which to associate the imported transactions.

   At the bottom of the page source you will see:

   <!– Page Name:[page id]:[campaign id]:[client id] –>

    

   questionId

   In order to obtain the correct ID number for a question, it is necessary to use the browser’s Developer Tools ‘Network’ tab; this will allow you to inspect the response from calls that are made to page-builder while editing pages. To locate the data structure containing the questions, edit a page and open any ‘Form block’ for editing, then select the Questions tab. You should now see a call to:question?sharedOnly=false&_= By viewing the Response for this call, you will be able to see the JSON data structure which contains the questionId. The data structure is an array containing both regular questions and so called ‘opt-in’ questions, however each index also contains the question ‘name’, which should help in identifying the correct question.

   

5. If you are providing a date for the transaction it must be in the YYYY-MM-DD format. If you are providing a time, this should be in the hhhh:mm:ss format

Submitting the transactional import file

Once these steps are completed, you can submit import files to the service by following these steps:

1. Go to Go to https://ca.engagingnetworks.app/ea-dataservice/import.jsp (Canada server) or https://us.engagingnetworks.app/ea-dataservice/import.jsp (U.S. server)
2. Enter the private token for your user account
3. Enter the location of your csv import file
4. Enter importTransactions as the format name

Once the file is received it will be scheduled for import to your account – a notification email will be sent when the import is complete. You can check the status of the job by logging into the user account associated with the private token you have used.

As mentioned in the overview, you can also write an application in the language of your choice to submit to the above URL.

Troubleshooting importing transactional data

1. Check that you have covered all the points above (preparing the transaction import file).
2. If you have successfully imported data but cannot see it in the platform, note that it can take up to 20 minutes before the data is available for live use.
3. If you see the error “Error: The format you selected does not match the first line of your upload file. Please verify that the header row in the import is exactly the same as the import format fields.“, check that the file is formatted in the format as given by the template (see attachment). 
4. If you see the error “Error: Import format parameter not found or is invalid” when importing supporters, check that you are using the correct name for the “format name”. The format name is created as part of the front-end import process when mapping fields. 
5. If you have confirmed this is correct, if you are using Excel ensure you are saving it as the file format “Comma Separated Values (.csv)”, and not “CSV UTF-8 (Comma delimited) (.csv)”. If using Notepad, Notepad++, or Sublime Text, save the file with “UTF-8” encoding – do not choose “UTF-8 with BOM”.