contentful / contentful-batch-libs

Library modules used by contentful batch utility CLI tools.

Contentful Batch Libs

This repository contains shared methods used within Contentful's import and export tools.

Even when these parts are fairly small and independent, they might not be easy to understand and use for your own projects right now.

Each module has more extensive documentation in the code regarding their own purpose.

About

Contentful provides a content infrastructure for digital teams to power content in websites, apps, and devices. Unlike a CMS, Contentful was built to integrate with the modern software stack. It offers a central hub for structured content, powerful management and delivery APIs, and a customizable web app that enable developers and content creators to ship digital products faster.

"Get" Modules

getSourceSpace

Used to get content from a space, intended to be copied somewhere else, or manipulated.

getOutdatedDestinationContent

Gets content from a space which will have content copied to it, based on a collection of existing content.

"Push" Modules

creation

Entity creation methods.

assets

Asset processing methods.

deletion

Entity deletion methods.

publishing

Entity publishing methods.

pushToSpace

Pushes all changes, including deletions, to a given space. Handles (un)publishing as well as delays after creation and before publishing.

Creates everything in the right order so that a content type for a given entry is there when entry creation for that content type is attempted.

Allows only content model or only content pushing.

"Transform" Modules

transformSpace

Transforms all the content from a space, in order to ready it to be sent to another space.

transformers

Transformer methods for each kind of entity.

Utils

createClients

Creates delivery and management client instances for both source and destination spaces.

errorBuffer

Gathers errors in an internal buffer which can then be drained in order to output them to a user.

Changelog

Check out the releases page.

contentful-batch-libs's Issues

Misleading error message in getFullSourceSpace (copy-paste?)

I was looking at the source code and noticed a weird error message.

The error message talks about destination space but we are dealing with source space here, right? See https://github.com/contentful/contentful-batch-libs/blob/master/lib/get/get-full-source-space.js#L47-L49

I think this is just a copy-paste from https://github.com/contentful/contentful-batch-libs/blob/master/lib/get/get-outdated-destination-content.js#L26-L28

shasum Check Failed

Attempting to install contentful-space-sync: npm install -g contentful-space-sync.
Getting an error with contentful-batch-libs.

I think there is an issue with shasum

npm ERR! Expected: bee150adf008c0a2544e7a203adec020d5c6d395

npm ERR! Actual:  f821a6015008988eca701d2d2301cf801ac4106a

I manually downloaded the https://registry.npmjs.org/contentful-batch-libs/-/contentful-batch-libs-1.0.1.tgz and checked it too
The actual value I found is - f821a6015008988eca701d2d2301cf801ac4106a
but the value at https://registry.npmjs.org/contentful-batch-libs is - bee150adf008c0a2544e7a203adec020d5c6d395

Is this an issue with contentful-batch-libs or NPM itself?

Importing of roles from contentful-export script

Push-to-space does not create roles that are included in contentful-export json output file.

includeDrafts is ignored for Assets (drafts are returned, but shouldn't)

👋

I was using contentful-export and did not want to include drafts. That works just fine for content, but not for assets. Looking at the code, it's kinda obvious, why:

For content-types there's code to filter the drafts, based on the setting:

contentful-batch-libs/lib/get/get-full-source-space.js

Line 73 in 6420238

.then((items) => filterDrafts(items, includeDrafts))

There is no code to do the same for assets:

contentful-batch-libs/lib/get/get-full-source-space.js

Lines 94 to 98 in 6420238

	return pagedGet({space: ctx.space, method: 'getLocales'})
	.then(extractItems)
	.then((items) => {
	ctx.data.locales = items
	})

I was under the impression that when I don't want to include drafts, it also applies to assets, but maybe I'm understanding something wrong. So is that a bug / oversight, or is there a reasoning behind it?

My use case is exporting / importing content & assets, but I ran into validation errors because there was a draft asset lingering around.

Thanks for reading and have a great day!

Broken handling of skipContentModel in getFullSourceSpace

Hi,

I'm trying to build something with these libs. When using getFullSourceSpace with option skipContentModel set to true I encountered the following error:

info Getting content from source space
(node:24538) UnhandledPromiseRejectionWarning: Unhandled
promise rejection (rejection id: 1): TypeError: Cannot read property 'filter' of undefined

Looking at the code I believe this to be due to lines #39-41:

response.editorInterfaces = response.editorInterfaces.filter((editorInterface) => {
  return editorInterface !== null
})

In the preceding callback, response.editorInterfaces is only set if response.contentTypes.length !== 0. Thus, when response.contentType.length === 0, the call to response.editorInterfaces.filter fails, because response.editorInterFaces is undefined.

I don't know what the editor interfaces are being used for and whether it would make more sense to remove the condition on setting the property. After all, this code in lines #55-64

function getEditorInterfaces (contentTypes) {
  const editorInterfacePromises = contentTypes.map((contentType) => {
    // old contentTypes may not have an editor interface but we'll handle in a later stage
    // but it should not stop getting the data process
    return contentType.getEditorInterface().catch(() => {
      return Promise.resolve(null)
    })
  })
  return Promise.all(editorInterfacePromises)
}

will simply return a promise resolving to an empty array. Right? My gut feeling tells me

  return managementClient.getSpace(spaceId)
  .then((space) => {
    return Promise.props({
      contentTypes: skipContentModel ? [] : pagedGet(space, 'getContentTypes').then(extractItems),
      entries: skipContent ? [] : pagedGet(deliveryClient || space, 'getEntries').then(extractItems),
      assets: skipContent ? [] : pagedGet(space, 'getAssets').then(extractItems),
      locales: skipContentModel ? [] : pagedGet(space, 'getLocales').then(extractItems),
      webhooks: skipWebhooks ? [] : pagedGet(space, 'getWebhooks').then(extractItems),
      roles: skipRoles ? [] : pagedGet(space, 'getRoles').then(extractItems)
    }).then((response) => {
      response.editorInterfaces = getEditorInterfaces(response.contentTypes)
      return Promise.props(response)
    }).then((response) => {
      response.editorInterfaces = response.editorInterfaces.filter((editorInterface) => {
        return editorInterface !== null
      })
      return response
    })
  }, (err) => {
    log.error(`
The destination space was not found. This can happen for multiple reasons:
- If you haven't yet, you should create your space manually.
- If your destination space is in another organization, and your user from the source space does not have access to it, you'll need to specify separate sourceManagementToken and destinationManagementToken

Full error details below.
`)
    throw err
  })

should run just fine.

What do you think?

batchedIdQuery inconsistent

I've been working on my own export/import tool, using a lot of contentful-batch-libs. One issue I'm seeing is with getOutdatedDestinationContent and the batchedIdQuery. The ids are batched correctly, but not all the entries (or assets) are consistently found in the destination space. But, if I query for each id individually, the entry/asset is returned. I also experimented w/reducing the BATCH_CHAR_LIMIT and at some number lower than 1990, for me around 1200, the batches would find all the entries/assets.

I'm running the getOutdatedDestinationContent process on 250 assets/entries at a time.

Any ideas why the inclusion query is dropping entries/assets? Is reducing the BATCH_CHAR_LIMIT the best way to go?

Thanks for your feedback.

Asset transformation is returning an asset with empty file field

the sync tool is not able to process assets correctly due to a wrong transformation done the asset

Enable async transforms

Hey,

I'm looking to transform some of our content in a way that fields of a content type are converted into references to entries of a content type on the destination space. The referenced entries, however, don't exist on the destination space, yet. I would like to create them on-the- fly, if they don't exist.

Currently, the corresponding code in transform-space.js looks like it would support async operations within the transformer functions — good thing that TODO hasn't been investigated further, yet 😉.

What's missing, is a way to access the space's methods for interacting with content (e.g., access to createEntry etc.). To maintain API compatibility, I would propose a change along the lines of this:

export default function (
  space, destinationSpace, customTransformers, entities = spaceEntities
) {
  const transformers = defaults(customTransformers, defaultTransformers)
  // TODO maybe we don't need promises here at all
  const newSpace = omit(space, ...entities)
  return Promise.reduce(entities, (newSpace, type) => {
    return Promise.map(
      space[type],
      (entity) => Promise.resolve(transformers[type](entity, destinationEntities, destinationSpace))
        .then(transformed => ({
        original: entity,
        transformed 
      }))
    )
      .then((entities) => {
        newSpace[type] = entities
        return newSpace
      })
  }, newSpace)
}

This should allow you to access destinationSpace's methods, if an actual instance of Space from the management API client is provided to transformSpace. The transformer functions could interact with the space, this way. In addition because the entire destination space is now available within a transformer, transformers can now do things like creating references to other entity types (e.g., add a link to an asset in an entry).

I'd be willing to implement this and would also try to provide a cleaner solution than what I wrote down above.

@Khaledgarbaya, WDYT?

Request size exceeds 8 MB

right now the default item per page when requesting itmes is 1000, but this could cause a problem when entries have a lot of text content, and cause the Error, maximum request size exceeded.
To fix this we should make the limit configurable by the user when using contentful-export.

Breaking Node Version Update Added in 9.5.1 Patch

Release 9.5.1 dropped support for all node versions below 18. However, this was done in a patch release, which breaks all consumers who that are currently on a semver that is supposed to support those older versions. It doesn't leave the option for Node 16 consumers to stay on the current contentful version until they are able to upgrade to Node 18, and then upgrade to a new version of the package that targets Node 18. These breaking changes should really be done in a major release.

Same thing happened in contentful/contentful-export#1526

Pull request which calls out that it is a breaking change

The automated release is failing 🚨

🚨 The automated release from the master branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you can benefit from your bug fixes and new features again.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can fix this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here are some links that can help you:

• Usage documentation
• Frequently Asked Questions
• Support channels

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

---

Invalid npm token.

The npm token configured in the NPM_TOKEN environment variable must be a valid token allowing to publish to the registry https://registry.npmjs.org/.

If you are using Two Factor Authentication for your account, set its level to "Authorization only" in your account settings. semantic-release cannot publish with the default "
Authorization and writes" level.

Please make sure to set the NPM_TOKEN environment variable in your CI with the exact value of the npm token.

---

Good luck with your project ✨

Your semantic-release bot 📦🚀

Draft handling in pushToSpace is wrong

Hi,

As part of an investigation into a problem with entries not getting published anymore as part of pushToSpace() I discovered a problem with the drafts handling introduced by this change to push-to-space.js in #82. Specifically,

...
const drafts = sourceContent.entries.filter((entry) => entry.isDraft)
sourceContent.entries = sourceContent.entries.filter((entry) => !entry.isDraft)
result = result
  .then(partial(creation.createEntries,
    {space: space, skipContentModel: skipContentModel}, drafts, []))
...

At this stage sourceContent.entries is a collection of objects with just two properties: original and transformed. The first being the actual source space entry and the second being the output of the entry transform function. Hence, entry.isDraft is undefined in the filter predicate. Moreover, isDraft is a method of Entry and not a boolean. Even if the above code were correct and entry were the original entry, the current code would identify every entry as draft -- entry.isDraft would be truthy, because isDraft would be a method.

To fix this entire part I would suggest something along the lines of

const drafts = sourceContent.entries.filter(({original: entry}) => entry.isDraft())
const publishedEntries = sourceContent.entries.filter(({original: entry}) => !entry.isDraft())

I will open a PR with a fix, probably today. First I need to figure out why my entries are not getting published, though.

Allow optional property for locales