hawker-localization

Purpose

The purpose of this repo is to provide tools for managing Contentful. MAKE SURE to read the documentation before running any of these scripts. Some of these scripts are destructive. If the wrong space is written into the config file, content can be permanently lost.

Setup

Make config.json file on the root with this template

{
  "cloneFromContentfulSpace": "",
  "cloneFromContentfulToken": "",
  "uploadToContentfulSpace": "",
  "uploadToContentfulToken": "",
  "uploadToContentfulManagementToken": "",
  "cloneToExistingSpace": "",
  "cloneToExistingToken": "",
  "deleteContentsSpace": "",
  "deleteContentsToken": "",
  "uploadSpaceName" : ""
}

Set the proper variables
Get your managementToken here at the bottom of the page
uploadSpaceName is the name of the space that you will be creating
Run npm install to get all node modules
Run npm run babel to build everything. This should be done after any change to the code inside of main.js, but is not necessary in other files

Copying content

Contentful documentation is currently wrong. It does not handle rate-limit well at this point.
Go into node_modules/contentful-export/dist/run-contentful-export.js
Change the code one lines 62 - 68 to the following:

var clients = (0, _createClients2.default)(opts);
return (0, _getFullSourceSpace2.default)({
  managementClient: clients.source.management,
  spaceId: opts.spaceId,
  maxAllowedLimit: 100,
  includeDrafts: opts.includeDrafts
}).then(function (response) {

This is hard-coding maxAllowedLimit to prevent rate-limit errors from occuring

Copying to an existing space

Be sure the config file is up-to-date
First run npm run deleteAllEntries (see below on how to run this file if an error occurs) this will remove all entries from the space, but not delete the space
Next, run npm run cloneToExistingSpace. This will copy all content from cloneFromContentfulSpace into a JSON file located in staticContentfulFiles/. It will then attempt to upload all content from this file
These two together will remove all content from one space, then upload all entries from another

This script will now save the information of conflicted/non-working entires

When cloneToExistingSpace runs, it will attempt to upload all entries
When it runs into a non-working entry, which throws a 409, the script saves this entry into the missingFromExistingImport.txt file inside of the missingEntries/ directory
The script then removes this entry from the upload content and tries again
This may happen several times. Each time it hits a 409, it will save the new non-working entry and try again. You may hit a request limit. If this happens, just wait a few minutes and try again

To copy to a new space

WARNING this will completely remove and recreate a space with the name specified in uploadSpaceName
If there already exists a space with this name THIS SPACE WILL BE ERASED and recreated. The space ID and token will be new. Content will be copied to this new space from the space specified in cloneFromContentfulSpace
Because of Contentful's rate limits, this script had to be slowed down. Expect it to take ~30 minutes
To run: npm run cloneToNewSpace

To run Contentful upload - Localization

The script looks for headers with the underscore character. It assumes that the first column with an underscore "bg_BG" and all following columns are locales. This means that if a column that is not a locale contains an underscore "test_header", it will assume "test_header" is a locale, and all columns after "test_header" are locales too.
The excel sheet must contain 'fieldID', 'entryID', and 'message' within the non-locale columns. Their order does not matter. It also does not matter how many non-locale headers there are, so long as the three required headers exist
If you want to updated en-US, you must include it in one of the headers. Remember, if it is the first header, it must have an underscore "en_US"
If the sheet contains two of the same header, then the second occurrence will overwrite the first
Place this excel sheet into excelDocs directory
Be sure all locales in this document are available in the given space. If the document has just one locale that is not available, or not written in the same way as they are in Contentful, the script will not work.
One commonly mistaken locale is Serbian Cyrillic. Commonly mistaken as 'sr-RS' the correct format is 'sr-Cyrl-RS'
The script will read every sheet in the excel doc. If there is one sheet that does not conform to these standards, the script will not work. Sometimes a sheet can be hidden. Be sure to unhide all sheets to verify that they conform to these standards
Writing these locales with either underscores or hyphens is fine 'en_US' or 'en-US'
Be sure to specify the correct space in the config file. If the incorrect space is written, content could be over written, or the script may just not work.
Add the file path to main.js then run run npm run babel
To run upload: npm run uploadContent

Remove all content from a given space

This tool is required when copying to an existing space
WARNING this will remove all entries from the space specified in deleteContentsSpace. But it will not delete the space itself. So the spaceId and access token will remain the same
To run: npm run deleteAllEntries

To compare the two spaces

TODO: this has not been updated in awhile and is probably depricated

This will compare all entries from the cloned space, to the 'upload space'
At first, the newly created generated from 'cloneToNewSpace' will not have a spaceID or token
To generate these, go into the space and click on APIs in the navigation panel
To run npm run compareSpaces
This will output a stringified json with the test results
Suggested to use a json parser tool to view these results
TODO: Make a better visualization of these results

cloneToNewSpace fails due to a locale validation

Observed:

Error in uploading the space:  ValidationFailed: {
  "request": {
    "url": "https://api.contentful.com:443/spaces/<space>/locales",
    "headers": {
      "Accept": "application/json, text/plain, */*",
      "Content-Type": "application/vnd.contentful.management.v1+json",
      "X-Contentful-User-Agent": "contentful-management.js/1.3.1",
      "Authorization": "Bearer CFPAT-<token>",
      "user-agent": "node.js/v7.7.4",
      "Accept-Encoding": "gzip",
      "Content-Length": 119
    },
    "method": "post",
    "payloadData": "{\"code\":\"es-US\",\"name\":\"Spanish (United States)\",\"contentManagementApi\":true,\"contentDeliveryApi\":true,\"optional\":true}"
  },
  "status": 422,
  "statusText": "Unprocessable Entity",
  "requestId": "<id>",
  "message": "The resource you sent in the body is invalid.",
  "details": {
    "errors": [
      {
        "name": "taken",
        "path": "display_code",
        "value": "es-US"
      }
    ]
  }
}

Desired:
422 errors of this sort, which occurs at

spaceImport(uploadOptions)
      .then((finalOutput) => {
        console.log(chalk.green('\n *** All Data Imported successfully *** \n')) 
      })
      .catch((err) => {
        console.log(chalk.red('Error in uploading the space: ', err))
      })

Should be handled so that instead of catch, reject is handled in the spaceImport promise.

Steps to replicate:
Run the scripts

npm run cloneToNewSpace

physicalgraph / hawker-localization Goto Github PK