Prepare a shared RP development environment

June 26, 2026 ยท View on GitHub

Follow these steps to build a shared RP development environment and secrets file. A single RP development environment can be shared across multiple developers and/or CI flows. It may include multiple resource groups in multiple locations.

Prerequisites

  1. You will need Contributor and User Access Administrator roles on your Azure subscription, as well as the ability to create and configure AAD applications.

  2. Set the az account

    az account set -n "<your-azure-subscription>"

  3. Set the location for the environment. This will be used throughout the setup:

    LOCATION=eastus

  4. You will need a resource group for global infrastructure

    GLOBAL_RESOURCEGROUP=global-infra
az group create -n $GLOBAL_RESOURCEGROUP --location $LOCATION

  5. You will need a publicly resolvable DNS Zone resource in your Azure subscription. Set PARENT_DOMAIN_NAME and PARENT_DOMAIN_RESOURCEGROUP to the name and resource group of the DNS Zone resource.

    If a child DNS zone $LOCATION.<parent-domain> already exists (e.g. from a previous setup), you can derive the parent domain from it:

    CHILD_ZONE_NAME=$(az network dns zone list \
  --query "[?starts_with(name, '$LOCATION.')].name | [0]" -o tsv)
PARENT_DOMAIN_NAME=${CHILD_ZONE_NAME#$LOCATION.}
PARENT_DOMAIN_RESOURCEGROUP=$(az network dns zone list \
  --query "[?name=='$PARENT_DOMAIN_NAME'].resourceGroup | [0]" -o tsv)
echo "PARENT_DOMAIN_NAME=$PARENT_DOMAIN_NAME PARENT_DOMAIN_RESOURCEGROUP=$PARENT_DOMAIN_RESOURCEGROUP"

    If no child zone exists yet, list available zones and pick the parent manually:

    az network dns zone list --query "[].{Name:name, ResourceGroup:resourceGroup}" -o table

    Or create a new zone:

    PARENT_DOMAIN_NAME=<your-dns-parent-domain>
PARENT_DOMAIN_RESOURCEGROUP=$GLOBAL_RESOURCEGROUP

az network dns zone create --name $PARENT_DOMAIN_NAME -g $PARENT_DOMAIN_RESOURCEGROUP

  6. You will need a storage account in your Azure subscription in which to store shared development environment secrets. The storage account must contain a private container named secrets. All team members must have Storage Blob Data Reader or Storage Blob Data Contributor role on the storage account. Set SECRET_SA_ACCOUNT_NAME to the name of the storage account. Set SECRET_STORAGE_RESOURCEGROUP to the name of the resourcegroup to deploy the storage into.

    export SECRET_SA_ACCOUNT_NAME=<your-storage-account-name>
export SECRET_STORAGE_RESOURCEGROUP=<resource-group-for-secret-storage>
# When sourcing deploy-shared-env.sh environment variables may not be set,
# that is okay for the purpose of just deploying storage.
. ./hack/devtools/deploy-shared-env.sh 

deploy_global_secret_storage

  7. You will need an AAD object (this could be your AAD user, or an AAD group of which you are a member) which will be able to administer certificates in the development environment key vault(s). Set ADMIN_OBJECT_ID to the object ID.

    ADMIN_OBJECT_ID="$(az ad group show -g '<your-az-group>' --query id -o tsv)"

  8. You will need the ARO RP-specific pull secret (ask one of the @azure-red-hat-openshift GitHub team for this):

    PULL_SECRET=...

  9. Install Go matching the version in go.mod, if you haven't already.

  10. Install the Azure CLI, if you haven't already.

  11. Log in to Azure:

    az login

AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)

  12. Git clone this repository to your local machine:

    go get -u github.com/Azure/ARO-RP/...
cd ${GOPATH:-$HOME/go}/src/github.com/Azure/ARO-RP

  13. Prepare the secrets directory:

    mkdir -p secrets

AAD applications

  1. Set a prefix variable used for naming apps/sp

    # for PR E2E Environment
PREFIX=aro-v4-e2e

  2. Create an AAD application which will fake up the ARM layer:

    This application requires client certificate authentication to be enabled. A suitable key/certificate file can be generated using the following helper utility:

    go run ./hack/genkey -client arm
mv arm.* secrets

    > __NOTE:__: for macos change the -w0 option for base64 to -b0
AZURE_ARM_CLIENT_ID="$(az ad app create \
  --display-name ${PREFIX}-arm-shared \
  --query appId \
  -o tsv)"
az ad app credential reset \
  --id "$AZURE_ARM_CLIENT_ID" \
  --cert "$(base64 -w0 <secrets/arm.crt)" >/dev/null
az ad sp create --id "$AZURE_ARM_CLIENT_ID" >/dev/null

    Later this application will be granted:

    • User Access Administrator on your subscription.

  3. Create an AAD application which will fake up the first party application.

    This application requires client certificate authentication to be enabled. A suitable key/certificate file can be generated using the following helper utility:

    go run ./hack/genkey -client firstparty
mv firstparty.* secrets

    Now create the application:

    > __NOTE:__: for macos change the -w0 option for base64 to -b0
AZURE_FP_CLIENT_ID="$(az ad app create \
  --display-name ${PREFIX}-fp-shared \
  --query appId \
  -o tsv)"
az ad app credential reset \
  --id "$AZURE_FP_CLIENT_ID" \
  --cert "$(base64 -w0 <secrets/firstparty.crt)" >/dev/null
az ad sp create --id "$AZURE_FP_CLIENT_ID" >/dev/null

    Later this application will be granted:

    • ARO v4 FP Subscription on your subscription.
    • DNS Zone Contributor on the DNS zone in RESOURCEGROUP.
    • Network Contributor on RESOURCEGROUP.

  4. Create an AAD application which will fake up the RP identity.

    AZURE_RP_CLIENT_ID="$(az ad app create \
 --display-name ${PREFIX}-rp-shared \
 --query appId \
 --output tsv)"
AZURE_RP_CLIENT_SECRET="$(az ad app credential reset \
  --id "$AZURE_RP_CLIENT_ID" \
  --end-date '2299-12-31T11:59:59+00:00' \
  --query password \
  --output tsv)"
az ad sp create --id "$AZURE_RP_CLIENT_ID" >/dev/null

    Later this application will be granted:

    • Reader on RESOURCEGROUP.
    • Secrets / Get on the key vault in RESOURCEGROUP.
    • DocumentDB Account Contributor on the CosmosDB resource in RESOURCEGROUP.

  5. Create an AAD application which will fake up the gateway identity.

    
AZURE_GATEWAY_CLIENT_ID="$(az ad app create \
  --display-name ${PREFIX}-gateway-shared \
  --query appId \
  --output tsv)"
AZURE_GATEWAY_CLIENT_SECRET="$(az ad app credential reset \
  --id "$AZURE_GATEWAY_CLIENT_ID" \
  --end-date '2299-12-31T11:59:59+00:00' \
  --query password \
  --output tsv)"
az ad sp create --id "$AZURE_GATEWAY_CLIENT_ID" >/dev/null

  6. Create an AAD application which will be used by E2E and tooling.

    AZURE_CLIENT_ID="$(az ad app create \
  --display-name ${PREFIX}-tooling-shared \
  --query appId \
  --output tsv)"
AZURE_CLIENT_SECRET="$(az ad app credential reset \
  --id "$AZURE_CLIENT_ID" \
  --end-date '2299-12-31T11:59:59+00:00' \
  --query password \
  --output tsv)"
az ad sp create --id "$AZURE_CLIENT_ID" >/dev/null

    Later this application will be granted:

    • Contributor on your subscription.
    • User Access Administrator on your subscription.

    You must also manually grant this application the Microsoft.Graph/Application.ReadWrite.OwnedBy permission, which requires admin access, in order for AAD applications to be created/deleted on a per-cluster basis.

    • Go into the Azure Portal
    • Go to Azure Active Directory
    • Navigate to the aro-v4-tooling-shared app registration page
    • Click 'API permissions' in the left side pane
    • Click 'Add a permission'.
    • Click 'Microsoft Graph'
    • Select 'Application permissions'
    • Search for 'Application' and select Application.ReadWrite.OwnedBy
    • Click 'Add permissions'
    • This request will need to be approved by a tenant administrator. If you are one, you can click the Grant admin consent for <name> button to the right of the Add a permission button on the app page

    You can also do the same with following command:

    az ad app permission add \
  --id $AZURE_CLIENT_ID \
  --api 00000003-0000-0000-c000-000000000000 \
  --api-permissions $(az ad sp show --id 00000003-0000-0000-c000-000000000000 --query "appRoles[?value=='Application.ReadWrite.OwnedBy'].id" --output tsv)=Role

    But, this request will still need to be approved by a tenant administrator. If you are one, you can click the Grant admin consent for <name> button to the right of the Add a permission button on the app page

  7. Set up the RP role definitions and subscription role assignments in your Azure subscription. The usage of "uuidgen" for fpRoleDefinitionId is simply there to keep from interfering with any linked resources and to create the role net new. This mimics the RBAC that ARM sets up. With at least User Access Administrator permissions on your subscription, do:

    az deployment sub create \
  -l $LOCATION \
  --template-file pkg/deploy/assets/rbac-development.json \
  --parameters \
    "armServicePrincipalId=$(az ad sp list --filter "appId eq '$AZURE_ARM_CLIENT_ID'" --query '[].id' -o tsv)" \
    "fpServicePrincipalId=$(az ad sp list --filter "appId eq '$AZURE_FP_CLIENT_ID'" --query '[].id' -o tsv)" \
    "fpRoleDefinitionId"="$(uuidgen)" \
    "devServicePrincipalId=$(az ad sp list --filter "appId eq '$AZURE_CLIENT_ID'" --query '[].id' -o tsv)" \
  >/dev/null

  8. Create an AAD application which will fake up the portal client.

    This application requires client certificate authentication to be enabled. A suitable key/certificate file can be generated using the following helper utility:

    go run ./hack/genkey -client portal-client
mv portal-client.* secrets

    > __NOTE:__: for macos change the -w0 option for base64 to -b0
AZURE_PORTAL_CLIENT_ID="$(az ad app create \
  --display-name ${PREFIX}-portal-shared \
  --query appId \
  -o tsv)"

OBJ_ID="$(az ad app show --id $AZURE_PORTAL_CLIENT_ID --query id -o tsv)"

az rest --method PATCH \
 --uri "https://graph.microsoft.com/v1.0/applications/$OBJ_ID" \
 --headers 'Content-Type=application/json' \
 --body '{"web":{"redirectUris":["https://localhost:8444/callback"]}}'

az ad app credential reset \
  --id "$AZURE_PORTAL_CLIENT_ID" \
  --cert "$(base64 -w0 <secrets/portal-client.crt)" >/dev/null

Certificates

Generate new key/certificate files using an helper utility, and when these files already exists (and they need to be recreated), then rotate the certificates.

  1. Create the VPN CA key/certificate:

    go run ./hack/genkey -ca vpn-ca
mv vpn-ca.* secrets

  2. Create the VPN client key/certificate:

    go run ./hack/genkey -client -keyFile secrets/vpn-ca.key -certFile secrets/vpn-ca.crt vpn-client
mv vpn-client.* secrets

  3. Create the proxy serving key/certificate:

    go run ./hack/genkey proxy
mv proxy.* secrets

  4. Create the proxy client key/certificate:

    go run ./hack/genkey -client proxy-client
mv proxy-client.* secrets

  5. Create the proxy ssh key/certificate:

    ssh-keygen -f secrets/proxy_id_rsa -N ''

  6. Create an RP serving key/certificate:

    go run ./hack/genkey localhost
mv localhost.* secrets

  7. Create the dev CA key/certificate:

    go run ./hack/genkey -ca dev-ca
mv dev-ca.* secrets

  8. Create the dev client key/certificate:

    go run ./hack/genkey -client -keyFile secrets/dev-ca.key -certFile secrets/dev-ca.crt dev-client
mv dev-client.* secrets

  9. Create the cluster-mdsd CA key/certificate:

    go run ./hack/genkey cluster-mdsd
mv cluster-mdsd.* secrets

Environment file

  1. Choose the resource group prefix. The resource group location will be appended to the prefix to make the resource group name. If a v4-prefixed environment exists in the subscription already, use a unique prefix.

    RESOURCEGROUP_PREFIX=<your-rg-prefix>

  2. Choose the proxy domain name label. This final proxy hostname will be of the form vm0.$PROXY_DOMAIN_NAME_LABEL.$LOCATION.cloudapp.azure.com.

    PROXY_DOMAIN_NAME_LABEL=<your-proxy-domain-name-label>

  3. Export resourceGroup

    export RESOURCEGROUP="$RESOURCEGROUP_PREFIX-\$LOCATION"

  4. Create the secrets/env file:

    # use a unique prefix for Azure resources when it is set, otherwise use your user's name
cat >secrets/env <<EOF
######## Prior to sourcing the file the following env vars must be set:    ########
######## AZURE_PREFIX, LOCATION, ADMIN_OBJECT_ID, RESOURCEGROUP, and PARENT_DOMAIN_NAME   ########
export AZURE_PREFIX="${AZURE_PREFIX:-$USER}"
export ADMIN_OBJECT_ID="$ADMIN_OBJECT_ID"
export AZURE_TENANT_ID="$AZURE_TENANT_ID"
export AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID"
export AZURE_ARM_CLIENT_ID="$AZURE_ARM_CLIENT_ID"
export AZURE_FP_CLIENT_ID="$AZURE_FP_CLIENT_ID"
export AZURE_FP_SERVICE_PRINCIPAL_ID="$(az ad sp list --filter "appId eq '$AZURE_FP_CLIENT_ID'" --query '[].id' -o tsv)"
export AZURE_PORTAL_CLIENT_ID="$AZURE_PORTAL_CLIENT_ID"
export AZURE_PORTAL_ACCESS_GROUP_IDS="$ADMIN_OBJECT_ID"
export AZURE_PORTAL_ELEVATED_GROUP_IDS="$ADMIN_OBJECT_ID"
export AZURE_CLIENT_ID="$AZURE_CLIENT_ID"
export AZURE_SERVICE_PRINCIPAL_ID="$(az ad sp list --filter "appId eq '$AZURE_CLIENT_ID'" --query '[].id' -o tsv)"
export AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET"
export AZURE_RP_CLIENT_ID="$AZURE_RP_CLIENT_ID"
export AZURE_RP_CLIENT_SECRET="$AZURE_RP_CLIENT_SECRET"
export AZURE_GATEWAY_CLIENT_ID="$AZURE_GATEWAY_CLIENT_ID"
export AZURE_GATEWAY_SERVICE_PRINCIPAL_ID="$(az ad sp list --filter "appId eq '$AZURE_GATEWAY_CLIENT_ID'" --query '[].id' -o tsv)"
export AZURE_GATEWAY_CLIENT_SECRET="$AZURE_GATEWAY_CLIENT_SECRET"
export PROXY_HOSTNAME="vm0.$PROXY_DOMAIN_NAME_LABEL.\$LOCATION.cloudapp.azure.com"
export DATABASE_NAME="\$AZURE_PREFIX"
export RP_MODE="development"
export PULL_SECRET="$PULL_SECRET"
export SECRET_SA_ACCOUNT_NAME="$SECRET_SA_ACCOUNT_NAME"
export DATABASE_ACCOUNT_NAME="\$RESOURCEGROUP"
export KEYVAULT_PREFIX="\$RESOURCEGROUP"
export PARENT_DOMAIN_NAME="$PARENT_DOMAIN_NAME"
export PARENT_DOMAIN_RESOURCEGROUP="$PARENT_DOMAIN_RESOURCEGROUP"
export DOMAIN_NAME="\$LOCATION.\$PARENT_DOMAIN_NAME"
export AZURE_ENVIRONMENT="AzurePublicCloud"
export OIDC_STORAGE_ACCOUNT_NAME="\${RESOURCEGROUP//-}oic"
EOF

After creating this file, run make secrets-update to upload it to your storage account so other people on your team can access it via make secrets.

Deploy shared RP development environment (once per location)

Look at the helper file to understand each of the bash functions below.

  1. Copy, edit (if necessary) and source your environment file. The required environment variable configuration is documented immediately below:

    cp env.example env
vi env
. ./env
    • LOCATION: Location of the shared RP development environment (default: eastus).

  2. Create AzSecPack managed Identity https://msazure.visualstudio.com/ASMDocs/_wiki/wikis/ASMDocs.wiki/234249/AzSecPack-AutoConfig-UserAssigned-Managed-Identity (required for deploy_env_dev)

  3. Enable EncryptionAtHost for subscription.

    az feature register --namespace Microsoft.Compute --name EncryptionAtHost

  4. Create the resource group and deploy the RP resources:

    . ./hack/devtools/deploy-shared-env.sh
# Create the RG
create_infra_rg
# Deploy the predeployment ARM template
deploy_rp_dev_predeploy
# Deploy the infrastructure resources such as Cosmos, KV, Vnet...
deploy_rp_dev
# Deploy RP MSI for aks/hive
deploy_rp_managed_identity
# Deploy the proxy and VPN
deploy_env_dev
# Deploy AKS resources for Hive
deploy_aks_dev
# Deploy additional infrastructure required for workload identity clusters
deploy_miwi_infra_dev

  5. Get the AKS kubeconfig and upload it to the storage account:

    make aks.kubeconfig
mv aks.kubeconfig secrets/
make secrets-update

  6. Install Hive on the new AKS

  7. Load the keys/certificates into the key vault:

    import_certs_secrets

    NOTE:: in production, three additional keys/certificates (rp-mdm, rp-mdsd, and cluster-mdsd) are also required in the $KEYVAULT_PREFIX-svc key vault. These are client certificates for RP metric and log forwarding (respectively) to Geneva.

    If you need them in development:

    ! DO NOT RUN THIS IN E2E, INT! !

    az keyvault certificate import \
     --vault-name "$KEYVAULT_PREFIX-svc" \
     --name rp-mdm \
     --file secrets/rp-metrics-int.pem
az keyvault certificate import \
     --vault-name "$KEYVAULT_PREFIX-gwy" \
     --name gwy-mdm \
     --file secrets/rp-metrics-int.pem
az keyvault certificate import \
     --vault-name "$KEYVAULT_PREFIX-svc" \
     --name rp-mdsd \
     --file secrets/rp-logging-int.pem
az keyvault certificate import \
     --vault-name "$KEYVAULT_PREFIX-gwy" \
     --name gwy-mdsd \
     --file secrets/rp-logging-int.pem
az keyvault certificate import \
     --vault-name "$KEYVAULT_PREFIX-svc" \
     --name cluster-mdsd \
     --file secrets/cluster-logging-int.pem

    NOTE:: in development, if you don't have valid certs for these, you can just upload localhost.pem as a placeholder for each of these. This will avoid an error stemming from them not existing, but it will result in logging pods crash looping in any clusters you make. Additionally, no gateway resources are created in development so you should not need to execute the cert import statement for the "-gwy" keyvault.

  8. In pre-production (int, e2e) certain certificates are provisioned via keyvault integration. These should be rotated and generated in the keyvault itself:

    Vault Name: "$KEYVAULT_PREFIX-svc"
Certificate: rp-firstparty
Development value: secrets/firstparty.pem

Vault Name: "$KEYVAULT_PREFIX-svc"
Certificate: cluster-mdsd
Development value: secrets/cluster-logging-int.pem

    NOTE:: in the new tenant OneCert is not available, therefore firstparty and cluster-mdsd are self signed.

       az keyvault certificate import \
      --vault-name "$KEYVAULT_PREFIX-svc" \
      --name rp-firstparty \
      --file secrets/firstparty.pem

   az keyvault certificate import \
      --vault-name "$KEYVAULT_PREFIX-svc" \
      --name cluster-mdsd \
      --file secrets/cluster-mdsd.pem

  9. Create nameserver records in the parent DNS zone:

    update_parent_domain_dns_zone

  10. Store the VPN client configuration:

    vpn_configuration

Non-required Shared RP Procedures

Below are non-required Shared RP procedures that might be needed (e.g., only for PR E2E or to rotate old certs).

Certificate Rotation

This section documents the steps taken to rotate certificates in dev and INT subscriptions

  1. Generate new certificates like we did in AAD application and certificate sections above

  2. Import newly generated certificates to keyvault. Note that this does not include firstparty certificates

    source hack/devtools/deploy-shared-env.sh
import_certs_secrets

  3. Update the Azure VPN Gateway configuration. To do this, go to Virtual Network Gateways > Point-to-site configuration and the public cert data from vpn-ca.pem. Delete the old expired root certificate

  4. The OpenVPN configuration file needs to be manually updated. To achieve this, edit the vpn-<region>.ovpn file and add the vpn-client certificate and private key

  5. Next, we need to update certificates owned by FP Service Principal. Current configuration in DEV and INT is listed below. You can get the AAD APP ID from the secrets/env file

    VariableCertificate ClientSubscription TypeAAD App NameKey Vault Name
    AZURE_FP_CLIENT_IDfirstpartyDEVaro-v4-fp-shared-devv4-eastus-dev-svc
    AZURE_ARM_CLIENT_IDarmDEVaro-v4-arm-shared-devv4-eastus-dev-svc
    AZURE_PORTAL_CLIENT_IDportal-clientDEVaro-v4-portal-shared-devv4-eastus-dev-svc
    AZURE_FP_CLIENT_IDfirstpartyINTaro-int-sparo-int-eastus-svc
    # Import firstparty.pem to keyvault v4-eastus-svc
az keyvault certificate import --vault-name <kv_name>  --name rp-firstparty --file firstparty.pem

# Rotate certificates for SPs ARM, FP, and PORTAL (wherever applicable)
az ad app credential reset \
   --id "$AZURE_ARM_CLIENT_ID" \
   --cert "$(base64 -w0 <secrets/arm.crt)" >/dev/null

az ad app credential reset \
   --id "$AZURE_FP_CLIENT_ID" \
   --cert "$(base64 -w0 <secrets/firstparty.crt)" >/dev/null

az ad app credential reset \
   --id "$AZURE_PORTAL_CLIENT_ID" \
   --cert "$(base64 -w0 <secrets/portal-client.crt)" >/dev/null

  6. The RP makes API calls to kubernetes cluster via a proxy VMSS agent. For the agent to get the updated certificates, this vm needs to be deleted & redeployed. Proxy VM is currently deployed by the deploy_env_dev function in deploy-shared-env.sh. It makes use of env-development.json

  7. Run [rharosecretsdev|e2earosecrets|e2earoclassicsecrets] make secrets-update to upload it to your storage account so other people on your team can access it via make secrets

Append Resource Group to Subscription Cleaner DenyList

PR E2E Only - Create the global keyvault, ADO Library Variable Group

  1. Create E2E global keyvault

    AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
ARO_E2E_GLOBAL_VAULT_NAME=<your-global-keyvault>

deploy_aro_e2e_global_keyvault

  2. Upload Keyvault Secrets and Certificates

  3. Give List/Get permissions to Azure DevOps Connection SPN

  4. Set up Library Variable group in ADO and connect it to keyvault

PR E2E Only - Setup ACR Credentials

Due to cross tenant ACR access, token credentials must be generated for arointsvc

  1. Login to MSIT tenant and navigate to arointsvc

  2. Under "Repository permissions -> Tokens" add a new token and generate a password

  3. Add username and password to json file formatted like below and convert it to base64

    {
   "username": "<username>",
   "password": "<Password>"
}

  4. convert to base 64, copy the output and add it to aro-e2e-global keyvault

    cat <my-acr-cred-file>.json | base64 -w0

  5. Add the secret to the Library variable group that is connected to the global keyvault

Deploy Holmes Azure OpenAI (optional)

The Holmes admin API requires an Azure OpenAI resource for cluster investigation.

Prerequisites: Azure storage account access for make secrets command.

# Ensure secrets/env exists (the deploy script writes credentials there)
# Note: Requires Azure storage account access
make secrets

source env
./hack/devtools/deploy-holmes-aoai.sh

# Push updated secrets to shared storage
make secrets-update

This creates:

  • Azure OpenAI account: ${RESOURCEGROUP}-holmes-aoai
  • GPT model deployment: gpt-5.2

The script writes HOLMES_AZURE_API_BASE and HOLMES_AZURE_API_VERSION to secrets/env, assigns the Cognitive Services OpenAI User role to the RP service principal, and disables local (API key) auth on the Azure OpenAI resource. Authentication uses Entra ID tokens acquired at runtime by the RP. Non-secret config (HOLMES_IMAGE, HOLMES_MODEL, etc.) is in env.example.

PR E2E Only - Add keyvault permissions to aro-v4-e2e-devops-spn

  • assign 'Keyvault Secrets User' to aro-v4-e2e-devops-spn