Coder Social home page Coder Social logo

parfuemerie-douglas / scaffolder-backend-module-azure-repositories Goto Github PK

View Code? Open in Web Editor NEW
8.0 1.0 21.0 838 KB

A collection of Backstage scaffolder backend modules for Azure repositories.

Home Page: https://www.npmjs.com/package/@parfuemerie-douglas/scaffolder-backend-module-azure-repositories

License: Apache License 2.0

JavaScript 0.37% TypeScript 99.63%

scaffolder-backend-module-azure-repositories's Introduction

scaffolder-backend-module-azure-repositories

Welcome to the Microsoft Azure repository actions for the scaffolder-backend.

This plugin contains a collection of actions:

  • azure:repo:clone
  • azure:repo:push
  • azure:repo:pr

Getting started

Create your Backstage application using the Backstage CLI as described here: https://backstage.io/docs/getting-started/create-an-app.

Note: If you are using this plugin in a Backstage monorepo that contains the code for @backstage/plugin-scaffolder-backend, you need to modify your internal build processes to transpile files from the node_modules folder as well.

You need to configure the actions in your backend:

From your Backstage root directory

# From your Backstage root directory
yarn add --cwd packages/backend @parfuemerie-douglas/scaffolder-backend-module-azure-repositories

Configure the actions (you can check the docs to see all options):

// packages/backend/src/plugins/scaffolder.ts

import { CatalogClient } from '@backstage/catalog-client';
import { ScmIntegrations } from "@backstage/integration";

import {
  cloneAzureRepoAction,
  pushAzureRepoAction,
  pullRequestAzureRepoAction,
} from "@parfuemerie-douglas/scaffolder-backend-module-azure-repositories";

import { Router } from 'express';

import type { PluginEnvironment } from '../types';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  const catalogClient = new CatalogClient({
    discoveryApi: env.discovery,
  });

const integrations = ScmIntegrations.fromConfig(env.config);

const actions = [
  cloneAzureRepoAction({ integrations }),
  pushAzureRepoAction({ integrations, config: env.config }),
  pullRequestAzureRepoAction({ integrations }),
  ...createBuiltInActions({
    containerRunner,
    catalogClient,
    integrations,
    config: env.config,
    reader: env.reader,
  }),
];

return await createRouter({
  containerRunner,
  catalogClient,
  actions,
  logger: env.logger,
  config: env.config,
  database: env.database,
  reader: env.reader,
});

The Azure repository actions use an Azure PAT (personal access token) for authorization. The PAT requires Read permission for Code for the azure:repo:clone action. For the azure:repo:push action the PAT requires Read & write permission for Code. Simply add the PAT to your app-config.yaml:

# app-config.yaml

integrations:
  azure:
    - host: dev.azure.com
      credentials:
        - personalAccessToken: ${AZURE_TOKEN}

Read more on integrations in Backstage in the Integrations documentation.

Using the template

After loading and configuring the Azure repository template actions, you can use the actions in your template:

# template.yaml

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: azure-repo-demo
  title: Azure Repository Test
  description: Clone and push to an Azure repository example.
spec:
  owner: parfuemerie-douglas
  type: service

  parameters:
    - title: Fill in some steps
      required:
        - name
        - owner
      properties:
        name:
          title: Project name
          type: string
          description: Choose a unique project name.
          ui:field: EntityNamePicker
          ui:autofocus: true
        owner:
          title: Owner
          type: string
          description: Select an owner for the Backstage component.
          ui:field: OwnerPicker
          ui:options:
            allowedKinds:
              - Group

  steps:
    - id: cloneAzureRepo
      name: Clone Azure Repo
      action: azure:repo:clone
      input:
        remoteUrl: "https://<MY_AZURE_ORGANIZATION>@dev.azure.com/<MY_AZURE_ORGANIZATION>/<MY_AZURE_PROJECT>/_git/<MY_AZURE_REPOSITORY>"
        branch: "main"
        targetPath: ./sub-directory

    - id: fetch
      name: Template Skeleton
      action: fetch:template
      input:
        url: ./skeleton
        targetPath: ./sub-directory
        values:
          name: ${{ parameters.name }}
          owner: ${{ parameters.owner }}

    - id: pushAzureRepo
      name: Push to Remote Azure Repo
      action: azure:repo:push
      input:
        branch: <MY_AZURE_REPOSITORY_BRANCH>
        sourcePath: ./sub-directory
        gitCommitMessage: Add ${{ parameters.name }} project files

    - id: pullRequestAzureRepo
      name: Create a Pull Request to Azure Repo
      action: azure:repo:pr
      input:
        sourceBranch: <MY_AZURE_REPOSITORY_BRANCH>
        targetBranch: "main"
        repoId: <MY_AZURE_REPOSITORY>
        title: ${{ parameters.name }}
        project: <MY_AZURE_PROJECT>
        supportsIterations: false

    - id: register
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: "dev.azure.com?owner=<MY_AZURE_PROJECT>&repo=<MY_AZURE_REPOSITORY>&organization=<MY_AZURE_ORGANIZATION>&version=<MY_AZURE_REPOSITORY_BRANCH>"
        catalogInfoPath: "/catalog-info.yaml"

  output:
    links:
      - title: Repository
        url: "dev.azure.com?owner=<MY_AZURE_PROJECT>&repo=<MY_AZURE_REPOSITORY>&organization=<MY_AZURE_ORGANIZATION>"
      - title: Open in catalog
        icon: catalog
        entityRef: ${{ steps.register.output.entityRef }}

Replace <MY_AZURE_ORGANIZATION> with the name of your Azure DevOps organization, <MY_AZURE_PROJECT> with the name of your Azure DevOps project, <MY_AZURE_REPOSITORY_BRANCH with the name of the desired Azure DevOps repository branch, and <MY_AZURE_REPOSITORY> with the name of your Azure DevOps repository.

NOTE: You will not be able to register the Pull Request since the file will not exist from the main branch!

You can find a list of all registred actions including their parameters at the /create/actions route in your Backstage application.

scaffolder-backend-module-azure-repositories's People

Contributors

certainty3452 avatar clemensheithecker avatar fosterm-mw avatar hevans-dglcom avatar jettebke avatar jmtpt avatar sanderaernouts avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

scaffolder-backend-module-azure-repositories's Issues

Add support for Azure Devops Integration properties to be compatible with version 1.18 of Backstage

When using the software template actions the cloneAzureRepo (and I'm assuming the subsequent actions relating to Azure repo) fails with: "No token provided for Azure Integration".
After checking this comes due to the new Backstage release version 1.18 and the deprecation of the token field for the Azure Devops Integration. The new structure is as follows:

integrations:
  azure:
    - host: dev.azure.com
      credentials:
      - organizations:
         - organizationName
        personalAccessToken: ${PAT}

But the backend logic as shown here still refers to the old structure and this brings misleading errors when in fact the authorisation should work.
I believe updating the integrationConfig variable to mirror the new structure should fix it.

Move to Backstage official repo

Why don't you move it to the official Backstage repo?
That might help get more contributions.
We are also looking for this functionality and wonder why it is not part of Backstage itself.

Pull request action requires organization name

While testing out these actions I ran into an issue with the azure:repo:pr action. It failed with this rather cryptic error message:

[1]2023-11-10T03:19:35.000Z Beginning step Create a Pull Request to Azure Repo
[2]2023-11-10T03:19:35.000Z TypeError: Cannot read properties of null (reading 'value')
[3]    at <anonymous> (/Users/jmezach/Code/rr/Backstage/node_modules/@antoniobergas/scaffolder-backend-module-azure-repositories/node_modules/azure-devops-node-api/VsoClient.js:124:52)
[4]    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

I managed to fix the issue by passing in the organization parameter, so it seems that that is actually required. But this is not described in the README, nor does it state so in the schema. As also mentioned in #23 I think these actions should maybe accept the output of the RepoPicker directly and then build the appropriate URL's internally.

Clone action requires repoUrl AND remoteUrl

Hey everyone,

the action azure:repo:clone for some reason requires both repoUrl and remoteUrl. However when i look at the action's code, repoUrl seems to not be used.
The repoUrl parameter is also not documented in the README.md.

Possible solutions could be:

  • Removing the repoUrl parameter from the requirements, since it is not used
  • Adding the repoUrl to the code example in the README.md, since it is required (in my case i set the repoUrl and the remoteUrl to the same value)
  • rename remoteUrl to repoUrl ( more consistency to code of original backstage actions )

if you don't have the time to change that, let me know whats the best option for you so i could create a PR with the changes for this issue for you.

Type 'ScmIntegrations' is not assignable to type 'ScmIntegrationRegistry'.

There is a type error because of type incompatibility between the current version of @backstage/integration and the one you are using.

When initializing the scaffolder router in backend it can be patched with:

cloneAzureRepoAction({ integrations: integrations as any }),
pushAzureRepoAction({
  integrations: integrations as any,
  config: env.config,
}),
pullRequestAzureRepoAction({ integrations: integrations as any }),

To insure type-safety and future compatibility the packages should be updated.

Clone action should support RepoPicker's RepoURL

While trying out these actions in a template we've noticed that the clone action doesn't support the URL format that the RepoPicker component generates. Looking at the code it seems to use the provided URL directly, while the built-in publish:azure action actually processes that URL through the parseRepoUrl function from the scaffolder-backend package as can be seen here. I think these actions should do the same.

Can't add actions to scaffolder.ts

Hello! I'm following the instructions in the readme, but when I add the actions to my existing array of actions in scaffolder.ts, I get the following error:

[1] C:\backstage\packages\backend\dist\main.js:444
[1] /******/                    throw e;
[1]                             ^
[1]
[1] TypeError: (0 , smithy_client_1.withBaseException) is not a function
[1]     at Object.<anonymous> (C:\backstage\node_modules\@parfuemerie-douglas\scaffolder-backend-module-azure-repositories\node_modules\@aws-sdk\client-sso\dist-cjs\protocols\Aws_restJson1.js:263:65)
[1]     at Module._compile (node:internal/modules/cjs/loader:1254:14)
[1]     at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
[1]     at Module.load (node:internal/modules/cjs/loader:1117:32)
[1]     at Module._load (node:internal/modules/cjs/loader:958:12)
[1]     at Module.require (node:internal/modules/cjs/loader:1141:19)
[1]     at require (node:internal/modules/cjs/helpers:110:18)
[1]     at Object.<anonymous> (C:\backstage\node_modules\@parfuemerie-douglas\scaffolder-backend-module-azure-repositories\node_modules\@aws-sdk\client-sso\dist-cjs\commands\GetRoleCredentialsCommand.js:9:25)
[1]     at Module._compile (node:internal/modules/cjs/loader:1254:14)
[1]     at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
[1]     at Module.load (node:internal/modules/cjs/loader:1117:32)
[1]     at Module._load (node:internal/modules/cjs/loader:958:12)
[1]     at Module.require (node:internal/modules/cjs/loader:1141:19)
[1]     at require (node:internal/modules/cjs/helpers:110:18)
[1]     at Object.<anonymous> (C:\backstage\node_modules\@parfuemerie-douglas\scaffolder-backend-module-azure-repositories\node_modules\@aws-sdk\client-sso\dist-cjs\SSO.js:5:37)
[1]     at Module._compile (node:internal/modules/cjs/loader:1254:14)
[1]
[1] Node.js v18.15.0

Here's the dependencies in my package.json:

  "dependencies": {
    "@backstage/backend-common": "^0.19.0",
    "@backstage/backend-tasks": "^0.5.3",
    "@backstage/catalog-client": "^1.4.2",
    "@backstage/catalog-model": "^1.4.0",
    "@backstage/config": "^1.0.8",
    "@backstage/integration": "^1.5.0",
    "@backstage/plugin-app-backend": "^0.3.46",
    "@backstage/plugin-auth-backend": "^0.18.4",
    "@backstage/plugin-azure-devops-backend": "^0.3.25",
    "@backstage/plugin-catalog-backend": "^1.10.0",
    "@backstage/plugin-catalog-backend-module-azure": "^0.1.17",
    "@backstage/plugin-permission-common": "^0.7.6",
    "@backstage/plugin-permission-node": "^0.7.9",
    "@backstage/plugin-proxy-backend": "^0.2.40",
    "@backstage/plugin-scaffolder-backend": "^1.15.0",
    "@backstage/plugin-search-backend": "^1.3.2",
    "@backstage/plugin-search-backend-module-pg": "^0.5.7",
    "@backstage/plugin-search-backend-node": "^1.2.2",
    "@backstage/plugin-techdocs-backend": "^1.6.3",
    "@mdude2314/backstage-plugin-scaffolder-git-actions": "1.1.0",
    "@mdude2314/backstage-plugin-scaffolder-json-merge-actions": "^1.1.0",
    "@mdude2314/backstage-plugin-scaffolder-npm-actions": "^1.1.0",
    "@mdude2314/backstage-plugin-search-backend-module-azure-devops-wiki": "^0.4.0",
    "@parfuemerie-douglas/scaffolder-backend-module-azure-pipelines": "^1.0.0-next.5",
    "@parfuemerie-douglas/scaffolder-backend-module-azure-repositories": "^0.2.4-next.2",
    "@plusultra/plugin-scaffolder-dotnet-backend": "^1.1.1",
    "app": "link:../app",
    "dockerode": "^3.3.1",
    "express": "^4.17.1",
    "express-promise-router": "^4.1.0",
    "json-merger": "^1.1.9",
    "pg": "^8.3.0",
    "winston": "^3.2.1"
  }

And the code in scaffolder, obfuscated for brevity/secrets:

/// scaffolder.ts
import {
  cloneAzureRepoAction,
  pushAzureRepoAction,
  pullRequestAzureRepoAction,
} from '@parfuemerie-douglas/scaffolder-backend-module-azure-repositories';

 const builtInActions = createBuiltinActions({
    catalogClient,
    integrations,
    config: env.config,
    reader: env.reader,
  });

  const actions = [
    ...builtInActions,
    cloneAzureRepoAction({ integrations }),
    pushAzureRepoAction({ integrations, config: env.config }),
    pullRequestAzureRepoAction({ integrations }),
    dotnetNewAction(),
    dotnetBuildAction(),
    gitCloneAction(),
    gitAction(),
    dotnetInstallTemplateAction(),
    createNpmExecAction(),
    createNpmInitAction(),
    createNpmInstallAction(),
    createNpmConfigAction(),
    createJsonMergeFilesAction(),
    createJsonMergeFileAction(),
  ];

  return await createRouter({
    logger: env.logger,
    config: env.config,
    database: env.database,
    reader: env.reader,
    catalogClient,
    actions,
  });

Any ideas what I'm missing here?

Azure Clone Repo unable to checkout to branch other than master

Problem

While trying to clone a repo using the azure:repo:clone action, I noticed that I am not able to use the branch input successfully (unless we specify the default branch).

Code

id: cloneRepo
name: Clone Azure Repo
action: azure:repo:clone
input:
  remoteUrl: ...
  branch: new-branch
  targetPath: ./new-folder

Error message

2024-06-12T14:11:59.000Z info: Creating new remote...
2024-06-12T14:11:59.000Z info: Checking out branch...
2024-06-12T14:11:59.000Z NotFoundError: Could not find origin/new-branch

For some reason, only using master works. Is this perhaps because the code clones the repository with only the default branch?

Anyone experienced the same issue and have workarounds to this?

Planned support for the new AzureDevOpsCredentialProvider provider?

Hi!

We are currently running the 1.18.1.next release of backstage, and with the release of 1.18 there has been major work done to the support multiple organizations and by so they have implemented a AzureDevOpsCredentialProvider. Will there be any will to carry this plugin to support the 1.18 backstage version?

Since a beginner at best at typescript a PR from me would probably hurt more then it would help

//Johannes

Custom PR message

Hello.
Today i know is possible to put a CUSTOM TITLE for the PR, will be nice to have a field to put a custom description to.
Screenshot 2023-11-08 at 17 13 52
The result should be something like that:
Screenshot 2023-11-08 at 17 15 58

input: organization: <> sourceBranch: <> targetBranch: "develop" repoId: <> title: <> description: MY DESCRIPTION project: <> supportsIterations: false
Tanks for the time...

Issue: Error when cloning and pushing to the same branch

Currently I am getting an error when I try to run the azure:repo:push action due to the commitAndPush function in the plugin. I looked at the code and it looks like the branch is being hardcoded to be scaffolder. Could we change this functionality? I'd be happy to open a PR!

feature: allow push to current branch

Hello everybody,

We are using trunk based development and want to allow pushing the changes made by the template to the main branch.
I want to push to the current branch from the azure:repo:push action.
Currently, this fails with the branch already exists because it tries to create a new one.
I have created a fix for it but am not allowed to create branches in this project.
Please find the code snippet below in case you prefer to add it yourself. Otherwise, I am happy to submit a PR if you give me permission.

helpers.ts

... 
 const currentBranch = await git.currentBranch({dir})

  if (currentBranch !== branch) {
    await git.branch({
      dir,
      ref: branch,
    })
  
    await git.checkout({
      dir,
      ref: branch,
    });
  }
...

Let me know if you need more info :)
Thanks!

Support for Azure DevOps Server

Hi!

We are trying to use this backstage scaffolder actions, unfortunately, I had and error that make me think that somehow the implementation is not compatible with Azure DevOps Server

image

Is this possible to implement? Will be awesome to have as the pipelines actions works well with Azure DevOps Server and we want to be able to make PRs with the scaffolder ๐Ÿ˜Š

Thanks!

pullRequestAzureRepoAction does not work

Hello everybody,
I am trying to implement the pullRequestAzureRepoAction in my template but it doesn't work for me.
I am using "@parfuemerie-douglas/scaffolder-backend-module-azure-repositories": "^0.2.2" and implemented the code as described in the docs adding in the scaffolder.ts with the imports and

  const actions = [
    ...builtInActions,
    cloneAzureRepoAction({ integrations }),
    pushAzureRepoAction({ integrations, config: env.config }),
    pullRequestAzureRepoAction({ integrations }),
  ];

Now, If I start the app I receive

[1] Backend failed to start up TypeError: (0 , _parfuemerie_douglas_scaffolder_backend_module_azure_repositories__WEBPACK_IMPORTED_MODULE_2__.pullRequestAzureRepoAction) is not a function

If I change the version to 0.2.4-next.2, the app starts normally.
In my template yaml, I added

    - id: pullRequestAzureRepo
      name: Create a Pull Request
      action: azure:repo:pr
      input:
        sourceBranch: <BRANCH>
        targetBranch: main
        repoId: <REPO>
        title: 'Create ${{ parameters.name }}'
        project: <PROJECT>

The previous steps of clone and push work fine and the last step fails with

[1]2023-07-18T13:05:43.247Z Beginning step Create a Pull Request 
[2]2023-07-18T13:05:43.685Z TypeError: Cannot read properties of null (reading 'value')
[3]   at /backstage/node_modules/azure-devops-node-api/VsoClient.js:124:52
[4]   at runMicrotasks (<anonymous>)
[5]   at processTicksAndRejections (node:internal/process/task_queues:96:5)

I have read the docs multiple times and tried changing to use other repos or branches but I end up here again. Could you have a look and maybe help me out?
Happy to submit more info if necessary.

Other dependency versions

    "@backstage/backend-common": "^0.19.0",
    "@backstage/backend-tasks": "^0.5.3",
    "@backstage/catalog-client": "^1.4.2",
    "@backstage/catalog-model": "^1.4.0",
    "@backstage/config": "^1.0.8",
    "@backstage/integration": "^1.5.0",
    "@backstage/plugin-adr-backend": "^0.3.4",
    "@backstage/plugin-app-backend": "^0.3.46",
    "@backstage/plugin-auth-backend": "^0.18.4",
    "@backstage/plugin-auth-node": "^0.2.15",
    "@backstage/plugin-azure-devops-backend": "^0.3.25",
    "@backstage/plugin-catalog-backend": "^1.10.0",
    "@backstage/plugin-catalog-backend-module-azure": "^0.1.17",
    "@backstage/plugin-permission-common": "^0.7.6",
    "@backstage/plugin-permission-node": "^0.7.9",
    "@backstage/plugin-proxy-backend": "^0.2.40",
    "@backstage/plugin-scaffolder-backend": "^1.15.0",
    "@backstage/plugin-search-backend": "^1.3.2",
    "@backstage/plugin-search-backend-module-pg": "^0.5.7",
    "@backstage/plugin-search-backend-node": "^1.2.2",
    "@backstage/plugin-techdocs-backend": "^1.6.3",
    "@parfuemerie-douglas/scaffolder-backend-module-azure-repositories": "^0.2.4-next.2",
    "app": "link:packages/app",
    "dockerode": "^3.3.5",
    "express": "^4.17.1",
    "express-promise-router": "^4.1.0",
    "pg": "^8.11.1",
    "winston": "^3.10.0"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.