stevetweeddale / gatsby-source-git Goto Github PK

View Code? Open in Web Editor NEW

69.0 69.0 29.0 125 KB

Source plugin for pulling data into Gatsby from Git repositories (hosted anywhere)

License: MIT License

JavaScript 100.00%

gatsby-source-git's People

Contributors

Stargazers

Watchers

gatsby-source-git's Issues

Clone fails when the remote is ssh and option is specified in https

It took me a while to figure this one out, so bear with the long description.

I am currently using gatsby-source-git to source MD files from another Git repo. We'll call this Content repo. Now I've already cloned the Content repo on my local.

I've configured gatsby-source-git as follows:

{
	resolve: `gatsby-source-git`,
	options: {
        name: 'blog-content',
        remote: 'https://github.com/PsyGik/content.git',
        local: '/Development/Personal/blog-content/',
        branch: 'master',
        patterns: 'content/**',
    },
},

When running gatsby develop, it fails with an error:

 ERROR 
localPath is not defined

This kind of throws a wrench into the whole debugging process. Like, WTF is localPath. Thankfully, it took little time to figure that out.

    throw new Error(`Can't clone to target destination: ${localPath}`);

localPath is being used outside of it's scope. Classic JS scope issue! 🤦🏽 Fixed this line, and now started getting the expected output, Can't clone to target destination: ...

Now, since I already have the repo cloned, it shouldn't fail with the clone again. I was stumped as to why the clone was failing. So I started debugging isAlreadyCloned method. Lo and Behold, the remote were different. Same, but different.

Existing Remote: [email protected]:PsyGik/content.git
Remote: https://github.com/PsyGik/content.git

So obviously, the equality check failed, because I am a dumb genius who prefers ssh over https remotes. 🥇

I also found out the hard way, that the plugin overrides local changes and always prefers remote changes. I am not sure why it does that, but I'll be investigating more on this.

What should be done?

Mention something in the README that this plugin clones using whatever remote option is specified. So it's upto the user to make sure that the remote urls are in sync. And that the plugin overrrides the content on every gatsby develop

Kick-ass plugin btw. Thank you! 🍻

Requires gatsby-source-filesystem in gatsby-config.js

Hi Stephen,

Might just want to update the readme that this requires gatsby-source-filesystem be in the gatsby-config.
Without it I was getting.

Error: Could not find owner plugin of node for loadNodeContent with owner `gatsby-source-filesystem`

It's pretty clear the cause, but I am assuming something bad happens if you change the name here ?

// Then create the node, as if it were created by the gatsby-source
// filesystem plugin.
return createNode(fileNode, {
        name: `gatsby-source-filesystem`
      });

Working fine now.

Thanks

any advice on auto pull from repository?

hi，there:
Is there any way to set up such that it listens for updates from responsitory and will pull whenever something changes?
Or any advice?

Install fails with [email protected]

Is this package compatible with Gatsby V3? I searched around but couldn't find anything. I'm super new to gatsby, node, and all of this.

Upon running:

npm install --save gatsby-source-git

I get:

npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR! 
npm ERR! Found: [email protected]
npm ERR! node_modules/gatsby
npm ERR!   gatsby@"^3.4.1" from the root project
npm ERR!   peer gatsby@">2.0.0-alpha" from [email protected]
npm ERR!   node_modules/gatsby-source-git
npm ERR!     gatsby-source-git@"*" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer gatsby@"^2.2.0" from [email protected]
npm ERR! node_modules/gatsby-source-git/node_modules/gatsby-source-filesystem
npm ERR!   gatsby-source-filesystem@"^2.1.19" from [email protected]
npm ERR!   node_modules/gatsby-source-git
npm ERR!     gatsby-source-git@"*" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR! 
npm ERR! See /home/node/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2021-06-01T02_00_06_888Z-debug.log```

shallow

FYI: there is some movement about shallow on the issue: libgit2/libgit2#3058

gitlab clone

Can't clone internal company GitLAB

nodegit support

I plan on using this plugin in a few projects and noticed this note:

Ideally we'd use nodegit, but it doesn't support shallow clones (see libgit2/libgit2#3058) which would have a significant effect on build times if you wanted to read files from git repositories with large histories.

The issue linked seems to be closed, perhaps indicating that shallow clones are now supported, enabling the backend of this project to changed.

Create and publish a new release.

Time for a 2.0.0.

can you provide a sample site?

It would be nice to have a full sample website

Regards
Matteo

this package should be on NPM

Hey Stephen! (:

I know we can install packages directly from git, and I'm using it right now,
but it makes me uneasy that there's not a immutable/versioned/canonical distribution.

maybe @theresama can transfer you the package name, it seems to be unused right now.

Read files and folders that start with a dot

Hi,

Thanks for creating the plugin.

Is there a way to read files and folder from the repository that start with a dot? Fast-glob by default ignores these files and folders but it allows you to pass an option that includes them to be read. I can't figure out how to pass the option through to fast-glob via the "gatsby-source-git" plugin.

Edit:

What I mean is that file nodes are not being created from the files/folders that start with a dot. For example, file nodes are not being created for all the images in my ".attachments" folder.

Enable cloning a repo from a specific commit SHA

I would like to have the ability to clone a repo from a predefined commit

    {
      resolve: `gatsby-source-git`,
      options: {
        name: `repo-one`,
        remote: `https://bitbucket.org/stevetweeddale/markdown-test.git`,
        commit: `e05270aa0927691d797edcebbf1d7fa33fc91516`,
      }
    }

Gatsby always fetch cached data

I use gatsby-source-git to pull files from Github repository. It worked well, but I added some files into the repository and I am not able to pull new files into my Gatsby project.

If I run this query in http://localhost:8000/___graphql:

query MyQuery {
  allMarkdownRemark {
    edges {
      node {
        frontmatter {
          title
        }
        fileAbsolutePath
      }
    }
  }
}

I get this result:

{
  "data": {
    "allMarkdownRemark": {
      "edges": [
        {
          "node": {
            "frontmatter": {
              "title": "Introduction"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/index.md"
          }
        },
        {
          "node": {
            "frontmatter": {
              "title": "Conceptual Guide"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/tau-conceptual-guide.md"
          }
        },
        {
          "node": {
            "frontmatter": {
              "title": "Tau & Agoras Overview"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/what-is-tauchain-tau.md"
          }
        },
        {
          "node": {
            "frontmatter": {
              "title": "FAQs"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/tauchain-agoras-faqs.md"
          }
        },
        {
          "node": {
            "frontmatter": {
              "title": "Tutorials"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/Tutorials/index.md"
          }
        },
        {
          "node": {
            "frontmatter": {
              "title": "Analysis of TauBot.TML"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/Tutorials/analysis-of-taubot-tml.md"
          }
        },
        {
          "node": {
            "frontmatter": {
              "title": "TML Bot Tutorial"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/Tutorials/tml-Bot-tutorial.md"
          }
        },
        {
          "node": {
            "frontmatter": {
              "title": "Understanding TML"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/Tutorials/understanding-tml-prolog-datalog-tau.md"
          }
        },
        {
          "node": {
            "frontmatter": {
              "title": "Quick Start"
            },
            "fileAbsolutePath": "dev/tau-guide-website/.cache/gatsby-source-git/tau-guide-documents/docs/quick-start.md"
          }
        }
      ]
    }
  }
}

However, Tutorials folder contains 6 files, which aren't included in query result. I don't know why.

gatsby-config.js

require("dotenv").config({
  path: `.env.${process.env.NODE_ENV}`
});

module.exports = {
  siteMetadata: {
    title: "Fan Site"
  },
  plugins: [
    "gatsby-plugin-react-helmet",
    "svgo",
    "gatsby-plugin-sass",
    "gatsby-plugin-postcss",
    {
      resolve: "gatsby-source-filesystem",
      options: {
        name: "data",
        path: `${__dirname}/src/data/`
      }
    },
    {
      resolve: "gatsby-source-filesystem",
      options: {
        path: `${__dirname}/src/pages`,
        name: "pages"
      }
    },
    {
      resolve: `gatsby-transformer-remark`,
      options: {
        plugins: [`gatsby-remark-autolink-headers`]
      }
    },
    "gatsby-transformer-json",
    "gatsby-plugin-dark-mode",
    {
      resolve: "gatsby-source-prismic-graphql",
      options: {
        repositoryName: "funsite",
        accessToken: `${process.env.API_KEY}`
      }
    },
    {
      resolve: `gatsby-source-git`,
      options: {
        name: `tau-guide-documents`,
        remote: `https://github.com/TauGuide/tau-guide-documents.git`,
        branch: `master`,
        // Only import the docs folder from a codebase.
        patterns: `docs/**`
      }
    }
  ]
};

I tried to add local: "/dev/tauguide/tau-guide-documents", as gatsby-source-git option, but it didn't help.

I tried to run gatsby clean which removed .cache folder and gatsby develop, but I got the same result.

I tried to remove the project and pull it from git again, but it didn't help.

package.json

{
  "name": "fan-site",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "develop": "gatsby develop",
    "build": "gatsby build",
    "serve": "gatsby serve",
    "start": "node server.js",
    "gh-pages": "gatsby build --prefix-paths && gh-pages -d public",
    "lint": "eslint src --fix",
    "dev": "(shx --silent rm -rf public .cache || shx true) && gatsby develop",
    "server": "cross-env NODE_ENV=development DEBUG=api nodemon server.js",
    "postinstall": "npm rebuild node-sass"
  },
  "author": "Prototype Interactive",
  "license": "MIT",
  "dependencies": {
    "@fortawesome/fontawesome-svg-core": "^1.2.25",
    "@fortawesome/free-brands-svg-icons": "^5.12.0",
    "@fortawesome/free-solid-svg-icons": "^5.11.2",
    "@fortawesome/react-fontawesome": "^0.1.7",
    "@kunukn/react-collapse": "1",
    "@material-ui/core": "^4.9.5",
    "@popperjs/core": "^2.3.3",
    "add": "^2.0.6",
    "axios": "^0.19.0",
    "basic-auth": "^2.0.1",
    "bootstrap": "4.2.1",
    "chart.js": "^2.9.3",
    "d3-node": "^2.2.1",
    "debug": "^4.1.1",
    "font-awesome": "4.7.0",
    "framer-motion": "^1.10.3",
    "gatsby": "^2.18.12",
    "gatsby-plugin-canonical-urls": "^2.3.0",
    "gatsby-plugin-dark-mode": "^1.1.0",
    "gatsby-remark-autolink-headers": "^2.3.3",
    "gatsby-source-git": "^1.0.2",
    "gatsby-source-prismic-graphql": "3.6.2",
    "gatsby-transformer-remark": "^2.6.53",
    "lodash.get": "^4.4.2",
    "lodash.groupby": "^4.6.0",
    "lodash.pickby": "^4.6.0",
    "lodash.set": "^4.3.2",
    "lodash.update": "^4.10.2",
    "marked": "^0.8.0",
    "moment": "^2.24.0",
    "prismic-reactjs": "^1.3.1",
    "prop-types": "^15.7.2",
    "react-chartjs-2": "^2.8.0",
    "react-collapsible": "^2.6.3",
    "react-d3-components": "^0.9.1",
    "react-d3-library": "^1.1.8",
    "react-headroom": "^3.0.0",
    "react-helmet": "^5.2.0",
    "react-lazyload": "^2.6.5",
    "react-moment": "^0.9.6",
    "react-onclickout": "^2.0.8",
    "react-popper": "^2.2.3",
    "react-popper-tooltip": "^2.11.1",
    "react-responsive": "^8.0.1",
    "react-scroll-to": "^3.0.0-beta.3",
    "react-select": "^3.1.0",
    "react-sidebar": "^3.0.2",
    "react-slick": "^0.25.2",
    "react-svg-donuts": "^1.0.0",
    "react-telegram-embed": "^0.0.10",
    "react-toastify": "^5.4.1",
    "react-twitter-embed": "^3.0.3",
    "react-window": "^1.8.5",
    "reactstrap": "^8.4.1",
    "slick-carousel": "^1.8.1",
    "underscore": "^1.9.1",
    "yarn": "^1.21.1"
  },
  "devDependencies": {
    "@prototype-interactive/eslint-config": "^0.1.1",
    "autoprefixer": "^9.4.4",
    "cross-env": "^5.2.0",
    "dotenv": "^8.2.0",
    "eslint": "^5.12.0",
    "gatsby-plugin-google-analytics": "^2.3.0",
    "gatsby-plugin-postcss": "^2.0.2",
    "gatsby-plugin-postcss-sass": "^1.0.22",
    "gatsby-plugin-react-helmet": "^3.0.5",
    "gatsby-plugin-sass": "^2.0.7",
    "gatsby-source-filesystem": "^2.2.2",
    "gatsby-transformer-json": "^2.1.7",
    "gh-pages": "^2.0.1",
    "husky": "^1.3.1",
    "prettier": "^1.15.3",
    "pretty-quick": "^1.8.0",
    "react": "^16.7.0",
    "react-dom": "^16.7.0",
    "shx": "^0.3.2",
    "svg-sprite-loader": "^4.1.3"
  },
  "resolutions": {
    "gatsby-source-graphql-universal": "3.3.0"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/PrototypeInteractive/gatsby-react-boilerplate.git"
  },
  "bugs": {
    "url": "https://github.com/PrototypeInteractive/gatsby-react-boilerplate/issues"
  },
  "homepage": "https://github.com/PrototypeInteractive/gatsby-react-boilerplate#readme"
}

gatsby-node.js

const path = require(`path`);
const { createFilePath } = require(`gatsby-source-git`);

exports.onCreateNode = ({ node, getNode, actions }) => {
  const { createNodeField } = actions;
  if (node.internal.type === `MarkdownRemark`) {
    const slug = createFilePath({ node, getNode, basePath: `pages` });
    createNodeField({
      node,
      name: `slug`,
      value: slug
    });
  }
};

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions;
  const result = await graphql(`
    query {
      allMarkdownRemark {
        edges {
          node {
            fields {
              slug
            }
          }
        }
      }
    }
  `);
  result.data.allMarkdownRemark.edges.forEach(({ node }) => {
    createPage({
      path: node.fields.slug,
      component: path.resolve(`src/templates/tau-document.js`),
      context: {
        // Data passed to context is available
        // in page queries as GraphQL variables.
        slug: node.fields.slug
      }
    });
  });
};

Unable to use gatsby environment variable with gataby-source-git patterns

Gatsby-source-git plugin expects the patterns to be defined as [`*`, `!*.md`] (without grave accent symbols) where as gatsby environment variable inserts the patterns as `[`*`, `!*.md`]`

Steps to reproduce
gatsby-config.js
{
resolve: gatsby-source-git,
options: {
name: ``,
remote: ${process.env.GATSBY_SOURCE},
// Optionally supply a branch. If none supplied, you'll get the default branch.
branch: ${process.env.GATSBY_SOURCE_BRANCH},
// Tailor which files get imported eg. import the docs folder from a codebase.
patterns: process.env.GATSBY_SOURCE_PATTERNS,
},
},

Expected result
Source files from Github repository as per the patterns.
The build should complete successfully.

Actual result
Because of extra `` the patterns are not recognized and the build fails.

Environment
Production

Is the project active maintained?

Hello @stevetweeddale,
thanks for this useful gatsby plugin. I use it on my documentation site.
Some great features are waiting in the Pull requests list like #12 #26.
Can you merge this PR? If you need help with testing, let me know.

Greetings Ben

Cloning part of the remote repo

Hi there,
I'm cloning a pretty huge repo from github, from which I only need one specific folder called 'content/articles/'. I can see the patterns fast-glob is telling which files to process and create as nodes, but at this stage all of my unnecessary content is already imported into the .cache. Is there a built-in solution in your plugin for a selective cloning of the remote repo?
Thanks!

Git clone fails when building from Gatsby cache

There's a fair chance I'm taking the wrong approach here, so let me know if I'm doing something wrong somewhere...

I have 2 GitHub repos, one is for a Gatsby site at http://forcepush.tech/ and the other is for my resume. I have my Gatsby site deployed on Netlify, and I've set up build optimizations to speed up my build, so my build command in package.json is currently GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages and my Netlify build command is npm run build. In my gatsby-config.js, I have the gatsby-source-git plugin configured as follows:

    {
      resolve: `gatsby-source-git`,
      options: {
        name: `jidicula-resume`,
        remote: `https://github.com/jidicula/jidicula-resume.git`,
        branch: `master`,
        local: "./public/jidicula-resume",
        // Only import the compiled PDF.
        patterns: `**.pdf`,
      },

This clones the resume repo into the site's build directory (public/). However, when the build is triggered (by a hook in my resume repo for example) and is building from cache, I see the following error from the gatsby-source-filesystem plugin, and the repository doesn't get cloned again:

1:38:45 PM:   Error: fatal: No remote configured to list refs from.
1:38:45 PM:   
1:38:45 PM:   - promise.js:90 toError
1:38:45 PM:     [repo]/[simple-git]/promise.js:90:14
1:38:45 PM:   
1:38:45 PM:   - promise.js:61 
1:38:45 PM:     [repo]/[simple-git]/promise.js:61:36
1:38:45 PM:   
1:38:45 PM:   - git.js:725 Git.<anonymous>
1:38:45 PM:     [repo]/[simple-git]/src/git.js:725:18
1:38:45 PM:   
1:38:45 PM:   - git.js:1475 Function.Git.fail
1:38:45 PM:     [repo]/[simple-git]/src/git.js:1475:18
1:38:45 PM:   
1:38:45 PM:   - git.js:1433 fail
1:38:45 PM:     [repo]/[simple-git]/src/git.js:1433:20
1:38:45 PM:   
1:38:45 PM:   - git.js:1442 
1:38:45 PM:     [repo]/[simple-git]/src/git.js:1442:16
1:38:45 PM:   
1:38:45 PM:   - task_queues.js:97 processTicksAndRejections
1:38:45 PM:     internal/process/task_queues.js:97:5
1:38:45 PM:

You can see the entire build log here: https://app.netlify.com/sites/forcepush/deploys/5f357a83404af32690b43c31

I've since set up a workaround build command to remove the repo first before attempting a build: rm -rf public/jidicula-resume && npm run build

Feature request : Fetching data from private repository

Processing remote images

Hey @stevetweeddale, thanks for working on this plugin. I just started playing around with it and I can convert all of my remote .md files from github nicely but I'm having issues with the images. Do you have any examples or hints there?

Consider exposing created/edited timestamps from git

Hi there,

Does this plugin support folders? For example, if I have files in folders in my git repository. If yes, will the path be generated automatically based on these folders?

Also, can I get the date of creation and modification of the file with this plugin?

Thank you very much in advance!

Looks valuable beyond gatsby

Hi, this looks like a valuable module for any react-based server, for example nextjs. Would it be too weird to consider that? :-)

What's preventing private repos?

Hey, thanks for this handy source plugin 🙏
Just stumbled upon it and it could replace my hacky GitLab solution, yet my repo is private..
I tried to just provide it with

https://gitlab.com:<personal_access_token>@gitlab.com/user/repo.git

replaced the token, user and repo name of course and it's sourcing for a pretty long time, mostly because my internet is slow, anyway it's doing something, ~~yet the .cache/caches/gatsby-source-git dir stays empty~~ just found out it's all in .cache/gatsby-source-git still no nodes are available in my graphiql
You've wrote it wouldn't be hard to support it, my first peeks into the code didn't unveiled anything obvious to me 🤓
If you've got a hint for me, I might be able to prepare a PR..

Edit: or I just use a small package.json or shell script to clone the repo 🤔

Update: got it working combining it with gatsby-source-filesystem right after gatsby-source-git something is preventing the node creation?! 🤔
I'm a little lazy digging more into debugging cause my internet is pretty slow, maybe I could copy paste the cloned repo back in after every change 🤔

Update: Okay so it might be due to my wacky internet connection or out dated machine, something seems odd locally..on Netlify's CI the aforementioned solution just generated duplicate content, from the git and fs sources 🤦‍♂️ so using git online and fs locally seems to be my goto for now..

Thanks for the plugin, closing because it's working for me with a private GitLab repo now

No helper functions

When you are writing documentation please be specific. I cannot find anywhere on internet information what to put in gatsby-node.js configuration file.

const { createFilePath } = require(gatsby-source-filesystem) on top will not work as your plugin do not contain createFilePath helper function, so saying "it should work same way as git-source-filesystem' is an overstatement.

Unsure about the local flag

Hi there,

the remote repo I clone is quite heavy and I often need to gatsby clean in order to make it work.

Usually when not clearing the cache I get this type of errors, for the record if it helps!

ERROR #85901  GRAPHQL

There was an error in your GraphQL query:

Field "video" is not defined by type MdxFrontmatterFilterInput.

File: src/pages/articles.js:47:46


 ERROR #85901  GRAPHQL

There was an error in your GraphQL query:

Unknown argument "fields" on field "mdx" of type "Query".

GraphQL request:3:9
2 |   query($slug: String!) {
3 |     mdx(fields: { slug: { eq: $slug } }) {
  |         ^
4 |         frontmatter {

File: src/templates/article.js:38:9


 ERROR #85923  GRAPHQL

There was an error in your GraphQL query:

Cannot query field "desc" on type "MdxFrontmatter".

If you don't expect "desc" to exist on the type "MdxFrontmatter" it is most likely a typo.

Anyway as explained, it seems like this error goes away as soon as I gatsby clean. My problem is that it takes around 10minutes recloning and building given the size of the repo (and media). For this reason the local option looks exactly what I need. I would like to have that remote repo content cloned into a "/content" folder at the root of my gatsby directory.

Issue: I've added the local: '/content' option but when doing my gatsby build the repo still gets cloned inside of the cache folder.

Expected behavior: The cloned repo shoudl be cloned into the local: "/content" folder as indicated in the options.

Could you be a bit more explicit about how to use that option and have the repo cloned into the relevant local repo as indicated in the options?
Can we improve the doc in order to explain how one should go about updating that repo to reflect state of the cloned branch on github?

Thanks!

npm bump please

Eager to use the cache option. Could you push that to npm please?

stevetweeddale / gatsby-source-git Goto Github PK

gatsby-source-git's People

Contributors

Stargazers

Watchers

Forkers

gatsby-source-git's Issues

What should be done?

Recommend Projects

Recommend Topics

Recommend Org