Data Mapping

This section will cover how Data Mapping is managed. This concept is used for file imports, data transformation, and mapping between different existing Data Schemas.

Data model

Data mapping is defined with this structure. If you exclude functions, it's a simple array of connections between the source and target Data Schemas.

interface IDataMapping {
    // Array of connections to define how data is mapped.
    "Connections": {
        // Indicates this connection is defining a foreign key.
        // This is a requirement for imports with an 'update' strategy.
        // Multiple keys can be defined as a foreign-key, it's recommended to define a single key for a stable result.
        "IsForeignKey": boolean;

        // The left side of the connection.
        // This is the source Data Schema or an output of a Data Mapping function.
        "Left": {
            // Set to 'out' to indicate an output from the source Data Schema.
            // For functions, set to a function's output value path defined in its documentation.
            "Key": "out" | string;

            // Type of panel we're getting the value from.
            "Panel": "left" | "function";

            // Path to the attribute within the source panel.
            // Attribute path segments are separated by a forward slash. Eg: 'address/city'.
            // For functions, this would be the function's ID.
            "Path": string;
        };

        // The right side of the connection.
        // This is the target Data Schema or input to a Data Mapping function.
        "Right": {
            // Set to 'in' to indicate an input to the target Data Schema.
            // For functions, set to a function's input value path defined in its documentation.
            "Key": "in" | string;

            // Type of panel we're setting the value to.
            "Panel": "right" | "function";

            // Path to the attribute within the target panel.
            // Attribute path segments are separated by a forward slash. Eg: 'address/city'.
            // For functions, this would be the function's ID.
            "Path": string;
        };
    }[];

    // Array of referenced functions in the Data Mapping connections.
    "Functions": IFunction[]; // See the Function section for more details.
}

Mapping functions

Mapping functions let you augment the input value before assigning it. You can chain multiple functions together to create complex transformations.

Mapping functions can have multiple inputs/outputs. For example you can concat or split a string.

interface IFunction {
    // A unique identifier for the function within the Data Mapping.
    // You can assign this as complex or simple as you want as long as it's unique within the Data Mapping.
    "ID": string;

    // The function to perform.
    // See documented functions for the different options.
    "Function" string;

    // Differs based on the function.
    // See individual function definitions for more details.
    "Settings": any;
}

Mapping functions: Concatenate

Combine multiple input attributes into a single output string.

The value will be available if you use the connection Key: Result in combination with this function's ID as the connection path.

interface IFunction {
    "ID": string;
    "Function": "Concatenate";
    "Settings": {
        // Array of inputs to concatenate.
        // If not specified, we assume "A" and "B" as two default inputs.
        // Set the right-side connection "Key" to one of these inputs to use them.
        "Inputs": string[];

        // The delimiter to use between the inputs.
        // When not specified a space is used.
        // You can set an empty string to concatenate without a delimiter.
        "Delimiter": string;
    };
}

Mapping functions: Split

Split an input string value into multiple outputs.

The value's Key will be available as a string index of the split result. Eg: splitting "Hello World" by space would give you "0" as "Hello" and "1" as "World".

If the split fails for any reason, then the "0" output will be set to the original input value. Other outputs are set to empty strings.

interface IFunction {
    "ID": string;
    "Function": "Concatenate";
    "Settings": {
        // This is set to an array of string indexes, eg: ["0", "1", "2"].
        // This is not a requirement for the function to run.
        // It acts as a guide for the UI to draw the number of outputs and related connections.
        "Outputs": string[];

        // Regex to split by.
        // Most of the time you can just enter plain-text, eg: " " or "," to split against.
        "splitRegex": string;
    };
}

Mapping functions: Replace

Run a string replace on an input value to get a new value.

Set the input value to the connection Key: Input to use this function.

The value will be available if you use the connection Key: Output in combination with this function's ID as the connection path.

interface IFunction {
    "ID": string;
    "Function": "StringReplace";
    "Settings": {
        // Regex to match against.
        // Most of the time you can just enter plain-text, eg: " " or "_" to replace.
        "replaceRegex": string;

        // The string to replace the matched value with.
        // This can be an empty string to remove the matched value.
        "replaceWith": string;
    };
}

Mapping functions: DateTime conversion.

Specify how your date string should be interpreted so we can store a valid ISO 8601 string.

If your strings are already in ISO 8601 format, you can skip this function and assign the value directly.

We store all dates in UTC so specifying a timezone that isn't UTC will result in a conversion to UTC.

interface IFunction {
    "ID": string;
    "Function": "DateTimeConversion";
    "Settings": {
        // Format to match. Eg: 'YYYY/MM/dd HH:mm:ss'.
        "format": string;

        // Optional timezone to specify if your data is not in UTC.
        // You can specify "UTC" as a default to avoid conversion.
        // This supports common timezone names, however it is highly recommended to use offsets instead.
        // Eg: '+13" or "-5".
        "timezoneCode"?: string;

        // Default value to output if the calculation fails or no input is provided.
        "defaultValue"?: string;
        // Set to true if you alternatively want to output the original value if the calculation fails.
        "defaultToOriginal"?: boolean;

        // Optional regex to match against to run a replace prior to interpreting the date.
        // Most of the time you can just enter plain-text, eg: " " or "," to split against.
        "replaceRegex"?: string;
        // Value to replace with if a regex is specified.
        // This defaults to an empty string.
        "replaceValue"?: string;
    };
}

Mapping functions: Calculate

This is a general-use calculation function where you can run Javascript eval code to get a new value.

The code will use $A, $B, etc as placeholders that will be replaced with your input values.

The value will be available if you use the connection Key: Result in combination with this function's ID as the connection path.

WARNING: Nulls are not currently handled gracefully. Ensure all your inputs are at least an empty string.

interface IFunction {
    "ID": string;
    "Function": "Calc";

    // Array of inputs to concatenate.
    // If not specified, we assume "A" and "B" as two default inputs.
    // Set the right-side connection "Key" to one of these inputs to use them.
    // $ is prepended and used as a placeholder for the input value within the formula.
    "Inputs": string[];

    // The formula to calculate.
    // Note that if a string is replaced, it won't be automatically quoted.
    // Eg: '$A + $B' will become 'abc + def' if $A and $B are strings.
    // Ensure to quote strings in the formula if needed.
    "Formula": string;
}

API utilities

We're working on improving our utilities to help generate these mappings.

Here is an experimental endpoint you can use.

Analyze Source

Request urlhttps://

{accountId}

.api.nextspace-dev.net/source/analyze

Request type

POST

Requirements

Account ID

Logged in user auth token

Power user auth token

Request body

interface IRequest {
    // Array of JSON data samples to analyze.
    "DataSamples": any[];

    // Optional source schema to use instead of generating one based on the data samples.
    // This is typically used when you make modifications in UI and want to re-analyze.
    "SchemaSource": IDataSchema; // See schema docs.

    // Optional target schema we want to suggest mappings into.
    // If not supplied, we'll assume it matches the source schema with minor modifications
    // Modifications are done to avoid a clash with internal attributes.
    "SchemaTarget"?: IDataSchema; // See schema docs.

    // Optional existing Data Mapping to supplement.
    "DataMapping"?: IDataMapping;
}

Response

interface IResponse {
    // Number of samples that were analyzed.
    // Helps determine if any issues occurred by ones being skipped.
    // Lower number could also indicate that we hit a limit.
    "CountSampled": number;

    // Source and target schemas if they are available.
    // If not found, it's likely there was an issue generating them.
    "SchemaSource"?: IDataSchema; // See schema docs.
    "SchemaTarget"?: IDataSchema; // See schema docs;

    // Suggestions for Data Mapping connections.
    "DataMapping"?: IDataMapping;

    // Errors if any were encountered.
    "Errors"?: string[];
}