One of the most requested features since we launched System Initiative has been import. People need to be able to bring in their existing infrastructure when they start using System Initiative - and today, they can.

Full fidelity import

In existing IaC tools (like Terraform and Pulumi), import is a disjointed, and potentially destructive, process. IaC import commands will import the resource to state and will give a best attempt at generating the code that the user can paste into their configuration files. It's a best attempt because it can only add the required fields to the generated code.

In System Initiative, import is part of the modeling flow in a change set. To import an AWS VPC, you would add the VPC asset to the canvas, set the VPC ID as the Resource ID in the new management functions tab, and then run the import function. The AWS API will retrieve the details of the AWS VPC and inject it back into the component. Change sets allow you to review your work in real-time, so if you are not happy with what was imported, you can abandon the change set without the risk of deleting the upstream resource. Only when the change set is applied to head will System Initiative sync the digital twin's real-world side using a refresh func, thus bringing it under management.

Import functions

In order to import resources, we introduced a new style of function, management functions. A management function takes the component to be managed as a parameter and allows the component properties to be set.

The basic structure of an import function:

1. Get the resourceId
2. Validate the correct parameters
3. Get the resource from the upstream API
4. Set the component properties for use within the model
5. Enqueue the refresh action to run when the change set is applied to HEAD.

Let's look at a working example of an import function used for importing a GitHub Repository into System Initiative:

async function main({ thisComponent }: Input): Promise<Output> {
  const component = thisComponent.properties;

  // 1. Get the resourceId
  let repositoryId = _.get(component, ["si", "resourceId"]);
  if (!repositoryId) {
    return {
      status: "error",
      message: "No Repository ID or full name set, cannot import resource",
    };
  }

  // 2. Validate the correct parameters
  // Fetch the GitHub API Token from secret storage
  const githubApiToken = requestStorage.getEnv("GITHUB_PAT");

  if (!githubApiToken) {
    return {
      status: "error",
      message: "GitHub API token is missing.",
    };
  }

  // 3. Get the resource from the upstream API
  // Define the GitHub API endpoint for fetching repository details
  const githubRepoUrl = `https://api.github.com/repos/${repositoryId}`;
  try {
    const response = await fetch(githubRepoUrl, {
      method: "GET",
      headers: {
        Authorization: `Bearer ${githubApiToken}`,
        Accept: "application/vnd.github+json",
      },
    });

    // Parse the response as JSON
    const jsonResponse = await response.json();

    if (response.status !== 200) {
      return {
        status: "error",
        message: `Failed to fetch GitHub repository: ${githubRepoUrl} ${response.status} ${response.statusText}`,
        payload: jsonResponse, // Include the full JSON response in case of an error
      };
    }

    // Repository details returned from GitHub API
    const repository = jsonResponse;

    // 4. Set the component properties
    // Set component properties from the repository's attributes
    component["domain"]["Repository Name"] = repository.name || "";
    component["domain"]["Description"] = repository.description || "";
    component["domain"]["Repository Owner"] =
      repository.owner.login || "not set";

    return {
      status: "ok",
      message: JSON.stringify(repository),
      ops: {
        update: {
          self: {
            properties: {
              ...component,
            },
          },
        },
        // 5. Enqueue a refresh function to run on change set merge.
        actions: {
          self: {
            remove: ["create"],
            add: ["refresh"],
          },
        },
      },
    };
  } catch (error) {
    return {
      status: "error",
      message: `Error fetching GitHub repository: ${error.message}`,
    };
  }
}

In this example, we make an API request to GitHub to get the details of the repository, we compute the list of properties we want to set on our component and we return those properties and the list of associated actions to perform on the component.

When can I use this?

Import functionality is available today! Anything that System Initiative can manage can support import functions. Imagine you want to manage your AWS SSO Users users with System Initiative, you could write an asset to represent a user and then attach an import function to that asset to pull all the properties of the user into the graph. That is exactly what we did in our infrastructure engineering team here at System Initiative.

At launch, import functions are available for AWS VPC and AWS Subnets but over the coming weeks we will be working our way through adding import functions to the full suite of assets available System Initiative. Hit us up on Discord if you want to help add support for a specific asset.