Script agent

For ERP systems that do not presently have an agent, you can use the Script agent to add Javascripts to connect and retrieve data.

The Script agent is specifically designed for scenarios where you need to:

  • Call external APIs
  • Manipulate or transform data using code (JavaScript)
  • Handle advanced logic like dynamic URL construction, conditional flows, and retries

A Script agent can only be used as a source and can connect to the following destinations:

  • Excel
  • MySQL
  • ODBC/ODBC32
  • SQL Server
  • Text

Add a connection

To add a connection for a Script agent:

  1. In the navigation panel, select Setup.
  2. Select a Script agent or peer connection, and click > Add Connection.
  3. Connection Name must be unique for the agent. Up to 40 characters.
  4. To add a parameter, in the Parameters table click in the blank row and complete the following:
    • Key: The parameter name. (No character restrictions or length limit.)
    • Value: The parameter's value. (No character restrictions or length limit.)

    • Secret: Select to hide and encrypt Value.

       Note:  After you click Save, you cannot select/deselect Secret.

  5. Click Save.

Add a query

  1. Select a Script connection or peer query, and click > Add Query
  2. Complete Query name.
  3. In the Query tab's text area, type or paste your Javascript. (No character restrictions or length limit.)
  4. Optionally, to add filter parameters to the query, select Filters; to add a parameter click in the blank row and complete the following:

    • Name: The parameter name. Up to 50 characters.

       Tip:  In the filter table, don't prepend "@" to the parameter names.

    • Type: The parameter's data type (text or numeric). (For date values choose text.)
    • Default value: What the user sees in the Data Integration process and can edit. Up to 50 characters.
    • Sample value: What the user sees when hovering over the cell in the Data Integration process. Use it to show the expected input/format. Up to 50 characters.
    • Hide: Turn on if you want to keep users of the Data Integration process from seeing the parameter, while using its Default value in the query.
  5. Click Save.

Use HTTP response headers

When integrating with external APIs, response headers often contain critical metadata. Whether you're tracking response times, inspecting caching policies, or managing authentication tokens, having access to these headers is essential for building robust and reliable integrations.

Integration experts and technical users who want to build custom integrations can use JavaScript in Script agent to access HTTP response headers.

The HTTP response headers feature is also useful for teams that rely on header-level data for diagnostics, security, or performance tuning.

This feature is exposed through the http object used in JavaScript scripts run by the Script agent.

You can use the feature to map header values into variables or use them as parameters in your integration workflows, to build more intelligent and responsive integration logic.

For example, you can use header values to:

  • Track API response times

  • Validate or refresh authenitcation tokens

  • Analyze caching behavior or rate-limiting policies

 Note:
  • The HTTP response headers feature is available in all environments where Data Integration is enabled.
  • No additional configuration is required to enable header access.

How to do it

To access HTTP headers in your script:

  1. Use the built-in HTTP service to make your API call.
  2. In your JavaScript script, retrieve the response headers using the header access method.
  3. Parse and use the header values as needed for your integration logic.

 Tip:   Common headers to inspect include Authorization, Cache-Control, ETag, and X-RateLimit-Remaining.

For example, imagine you're connecting to a REST API to retrieve customer orders. The API returns a 429 Too Many Requests header with a Retry-After: 30 header. You can:

  1. Read this header using the HTTP Response Headers feature.

  2. Pause or retry the integration after 30 seconds programmatically.

Example

When making an API call using http.get() or http.post(), the response object includes a headers field that lets you programmatically access and use HTTP response headers. For example:

Copy 
var response = http.get({
  url: "https://api.example.com/data",
  headers: {
    "Authorization": "Bearer " + token
  }
});

var content = response.body;
var rateLimit = response.headers["x-ratelimit-remaining"];
var retryAfter = response.headers["retry-after"];

log.info("Rate limit remaining: " + rateLimit);
log.info("Retry-After: " + retryAfter);

// Conditional logic based on headers
if (retryAfter) {
  // Pause or trigger a retry mechanism
}

 

Response header scenarios

The HTTP response headers feature in Data Integration can be particularly useful in several scenarios involving API-based data retrieval or interaction. Here’s a detailed breakdown of where and why you might use it:

API Debugging and Validation

When working with REST APIs, response headers often contain critical metadata such as (with examples):

  • Status codes (200 OK, 404 Not Found)
  • Rate-limiting info (X-RateLimit-Remaining)
  • Pagination details (Link headers)
  • Content-type or encoding (application/json, gzip)

 Use Case:  You can capture and inspect these headers to troubleshoot failed requests or validate that your integration is interacting correctly with third-party services.

Dynamic Flow Control Based on Headers

Some APIs include operational hints or flags in the headers:

  • Retry-After for throttling
  • ETag/Last-Modified for conditional requests or caching
  • Custom headers indicating data readiness or job status

 Use Case:  You can program logic to delay, repeat, or change behavior in subsequent steps depending on values found in the response headers.

Logging and Monitoring

For compliance, auditing, or simply tracking API behavior over time, you may want to log response metadata alongside the data payload.

 Use Case:   Save headers in a log table or send them to a monitoring tool to track changes in API performance or behavior.

Token or Session Management

Some services return tokens (like JWT or session IDs) in response headers rather than the body.

 Use Case:  Extract the token from the header and use it in downstream steps (for example, follow-up requests, auth flows).

Pagination Handling

If an API uses response headers to signal the next page of data (for example, a Link: <...>; rel="next" header), you can extract the next page URL and use it to continue fetching data.

 Use Case:  Automate multi-page data pulls without relying on URL patterns in the body.