Generating a Crossplane Provider

February 23, 2022 · View on GitHub

In this guide, we will generate a Crossplane provider based on an existing Terraform provider using Terrajet.

We have chosen Terraform GitHub provider as an example, but the process will be quite similar for any other Terraform provider.

Generate

Generate a GitHub repository for the Crossplane provider by hitting the "Use this template" button in provider-jet-template repository.
Clone the repository to your local and cd into the repository directory. Fetch the upbound/build submodule by running the following:
```
make submodules
```
Replace template with your provider name.
1. Export ProviderName:
```
export ProviderNameLower=github
export ProviderNameUpper=GitHub
```
2. Run the ./hack/prepare.sh script from repo root to prepare the repo, e.g., to replace all occurrences of template with your provider name:
```
./hack/prepare.sh
```
To configure the Terraform provider to generate from, update the following variables in Makefile:
```
export TERRAFORM_PROVIDER_SOURCE := integrations/github
export TERRAFORM_PROVIDER_VERSION := 4.19.2
export TERRAFORM_PROVIDER_DOWNLOAD_NAME := terraform-provider-github
export TERRAFORM_PROVIDER_DOWNLOAD_URL_PREFIX := https://releases.hashicorp.com/terraform-provider-github/4.19.2
```
You can find TERRAFORM_PROVIDER_SOURCE and TERRAFORM_PROVIDER_VERSION in Terraform GitHub provider documentation by hitting the "USE PROVIDER" button. Check this line in controller Dockerfile to see how these variables are used to build the provider plugin binary.

Implement ProviderConfig logic. In provider-jet-template, there is already a boilerplate code in file internal/clients/${ProviderNameLower}.go which takes care of properly fetching secret data referenced from ProviderConfig resource.

For our GitHub provider, we need to check Terraform documentation for provider configuration and provide the keys there:

const (
  keyBaseURL = "base_url"
  keyOwner = "owner"
  keyToken = "token"

  // GitHub credentials environment variable names
  envToken = "GITHUB_TOKEN"
)

func TerraformSetupBuilder(version, providerSource, providerVersion string) terraform.SetupFn {
  ...
  // set provider configuration
  ps.Configuration = map[string]interface{}{}
  if v, ok := githubCreds[keyBaseURL]; ok {
      ps.Configuration[keyBaseURL] = v
  }
  if v, ok := githubCreds[keyOwner]; ok {
      ps.Configuration[keyOwner] = v
  }
  // set environment variables for sensitive provider configuration
  ps.Env = []string{
      fmt.Sprintf(fmtEnvVar, envToken, githubCreds[keyToken]),
  }
  return ps, nil
}

Before generating all resources that the provider has, let's go step by step and only start with generating CRDs for github_repository and github_branch Terraform resources.

To limit the resources to be generated, we need to provide an include list option with tjconfig.WithIncludeList in file config/provider.go:
```
pc := tjconfig.NewProviderWithSchema([]byte(providerSchema), resourcePrefix, modulePath,
    tjconfig.WithDefaultResourceFn(defaultResourceFn),
    tjconfig.WithIncludeList([]string{
        "github_repository$",
        "github_branch$",
    }))
```

Finally, we would need to add some custom configurations for these two resources as follows:

# Create custom configuration directory for whole repository group
mkdir config/repository
# Create custom configuration directory for whole branch group
mkdir config/branch

cat <<EOF > config/repository/config.go
package repository

import "github.com/crossplane/terrajet/pkg/config"

// Configure configures individual resources by adding custom ResourceConfigurators.
func Configure(p *config.Provider) {
    p.AddResourceConfigurator("github_repository", func(r *config.Resource) {

        // we need to override the default group that terrajet generated for
        // this resource, which would be "github"  
        r.ShortGroup = "repository"
    })
}
EOF

cat <<EOF > config/branch/config.go
package branch

import "github.com/crossplane/terrajet/pkg/config"

func Configure(p *config.Provider) {
    p.AddResourceConfigurator("github_branch", func(r *config.Resource) {

        // we need to override the default group that terrajet generated for
        // this resource, which would be "github" 
        r.ShortGroup = "branch"

        // Identifier for this resource is assigned by the provider. In other
        // words it is not simply the name of the resource.
        r.ExternalName = config.IdentifierFromProvider

        // This resource need the repository in which branch would be created
        // as an input. And by defining it as a reference to Repository
        // object, we can build cross resource referencing. See 
        // repositoryRef in the example in the Testing section below.
        r.References["repository"] = config.Reference{
            Type: "github.com/crossplane-contrib/provider-jet-github/apis/repository/v1alpha1.Repository",
        }
    })
}
EOF

And register custom configurations in config/provider.go:

import (
    ...

    tjconfig "github.com/crossplane/terrajet/pkg/config"
    "github.com/crossplane/terrajet/pkg/types/conversion/cli"
    "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"

+   "github.com/crossplane-contrib/provider-jet-github/config/branch"
+   "github.com/crossplane-contrib/provider-jet-github/config/repository"
 )

 func GetProvider() *tjconfig.Provider {
    ...
    for _, configure := range []func(provider *tjconfig.Provider){
            // add custom config functions
+           repository.Configure,
+           branch.Configure,
    } {
            configure(pc)
    }

To learn more about custom resource configurations (in step 7), please see the Configuring a Resource document.

Now we can generate our Terrajet Provider:
```
make generate
```

Adding New Resources

To add more resources, please follow the steps between 6-8 for each resource.

Alternatively, you can drop the tjconfig.WithIncludeList option in provider Configuration which would generate all resources, and you can add resource configurations as a next step.

Test

Now let's test our generated resources.

First, we will create example resources under the examples directory:

Create example directories for repository and branch groups:

mkdir examples/repository
mkdir examples/branch

# remove the sample directory which was an example in the template
rm -rf examples/sample

Create a provider secret template:

cat <<EOF > examples/providerconfig/secret.yaml.tmpl
apiVersion: v1
kind: Secret
metadata:
  name: example-creds
  namespace: crossplane-system
type: Opaque
stringData:
  credentials: |
    {
      "token": "y0ur-t0k3n"
    }
EOF

Create example for repository resource, which will use provider-jet-template repo as template for the repository to be created:

cat <<EOF > examples/repository/repository.yaml
apiVersion: repository.github.jet.crossplane.io/v1alpha1
kind: Repository
metadata:
  name: hello-crossplane
spec:
  forProvider:
    description: "Managed with Crossplane Github Provider (generated with Terrajet)"
    visibility: public
    template:
      - owner: crossplane-contrib
        repository: provider-jet-template
  providerConfigRef:
    name: default
EOF

Create branch resource which refers to the above repository managed resource:

cat <<EOF > examples/branch/branch.yaml
apiVersion: branch.github.jet.crossplane.io/v1alpha1
kind: Branch
metadata:
  name: hello-terrajet
spec:
  forProvider:
    branch: hello-terrajet
    repositoryRef:
      name: hello-crossplane
  providerConfigRef:
    name: default
EOF

Generate a Personal Access Token for your Github account with repo/public_repo and delete_repo scopes.

Create examples/providerconfig/secret.yaml from examples/providerconfig/secret.yaml.tmpl and set your token in the file:

GITHUB_TOKEN=<your-token-here>
cat examples/providerconfig/secret.yaml.tmpl | sed -e "s/y0ur-t0k3n/${GITHUB_TOKEN}/g" > examples/providerconfig/secret.yaml

Apply CRDs:
```
kubectl apply -f package/crds
```
Run the provider:
```
make run
```

Apply ProviderConfig and example manifests (In another terminal since the previous command is blocking):

# Create "crossplane-system" namespace if not exists
kubectl create namespace crossplane-system --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f examples/providerconfig/
kubectl apply -f examples/repository/repository.yaml
kubectl apply -f examples/branch/branch.yaml

Observe managed resources and wait until they are ready:

watch kubectl get managed

NAME                                                   READY   SYNCED   EXTERNAL-NAME                     AGE
branch.branch.github.jet.crossplane.io/hello-terrajet   True    True     hello-crossplane:hello-terrajet   89s

NAME                                                             READY   SYNCED   EXTERNAL-NAME      AGE
repository.repository.github.jet.crossplane.io/hello-crossplane   True    True     hello-crossplane   89s

Verify that repo hello-crossplane and branch hello-terrajet created under your GitHub account.

Cleanup

kubectl delete -f examples/branch/branch.yaml
kubectl delete -f examples/repository/repository.yaml

Verify that the repo got deleted once deletion is completed on the control plane.