arranf / hugotojson

A Rust utility to produce a JSON representation of the contents of a Hugo static site

HugoToJSON

A utility to produce a JSON representation of the key front matter and contents of Hugo documents. It's main intent is to produce JSON to be used by Lunr (and Lunr-like packages) to support search on a static Hugo site. It's designed to be a fast and modern alternative to the now unsupported hugo_lunr Node tool.

Pull requests are welcome. A list of goals and work to be done is available in ToDo.txt.

It currently supports .md files and both YAML and TOML front matter.

Using

hugo_to_json HUGO_CONTENT_DIRECTORY -o OUTPUT_LOCATION

Example usage is shown below. hugo_to_json example/blog/content -o example/blog/static/index.json

Defaults to ./content for the content directory and stdout for the index output.

Fetching the Latest Version

If you want to use the latest version of this tool as part of a CI build process the following script should work.

#!/usr/bin/env bash
set -e

# Based on
#https://blog.markvincze.com/download-artifacts-from-a-latest-github-release-in-sh-and-powershell/

LATEST_RELEASE=$(curl -L -s -H 'Accept: application/json' https://github.com/arranf/HugoToJSON/releases/latest)
# The releases are returned in the format {"id":3622206,"tag_name":"hello-1.0.0.11",...}, we have to extract the tag_name.
LATEST_VERSION=$(echo $LATEST_RELEASE | sed -e 's/.*"tag_name":"\([^"]*\)".*/\1/')
ARTIFACT_URL="https://github.com/arranf/HugoToJSON/releases/download/$LATEST_VERSION/hugo_to_json"
INSTALL_DIRECTORY="."
INSTALL_NAME="hugo_to_json"
DOWNLOAD_FILE="$INSTALL_DIRECTORY/$INSTALL_NAME"

 echo "Fetching $ARTIFACT_URL.."
if test -x "$(command -v curl)"; then
    code=$(curl -s -w '%{http_code}' -L "$ARTIFACT_URL" -o "$DOWNLOAD_FILE")
elif test -x "$(command -v wget)"; then
    code=$(wget -q -O "$DOWNLOAD_FILE" --server-response "$ARTIFACT_URL" 2>&1 | awk '/^  HTTP/{print $2}' | tail -1)
else
    echo "Neither curl nor wget was available to perform http requests."
    exit 1
fi

if [ "$code" != 200 ]; then
    echo "Request failed with code $code"
    exit 1
fi

chmod +x "$DOWNLOAD_FILE"

hugotojson's Issues

Could not read slug from front matter

Hello
I tried to use your tool, but unfortunately I get an error message

[2019-10-27T19:12:40Z ERROR hugo_to_json::traverse::traverser] Error at site/content/en//blog/Its-a-breeze-to-develop-apache-airflow4.md. Could not read slug from front matter. Skipping.

Sample file content:

---
title: "It's a breeze to develop Apache Airflow"
linkTitle: "It's a breeze to develop Apache Airflow"
author: "Jarek Potiuk"
description: "Working on an Open Source project such as Apache Airflow is very demanding but also equally rewarding when you realize how many businesses use it every day."
tags: ["Community"]
date: 2019-09-24
---

##### My journey to developer productivity
It started at the beginning of my career with my older and more experienced colleague—one of the first “real” software engineers I had interacted with by then. I learned a lot from him. I specifically remember him saying: “When I develop code, I spend the first half of my time on developing tools to make my work 10x faster—that’s the key reason why I always deliver on time.” His words stuck with me. I’ve been a part of many successful projects in my career but they pretty much always looked like this: when I first joined a project I’d look around, trying to learn the ropes and identify bottlenecks, such as the lack of tools or what part of the development process could slow me down.

As an All-Stack-Engineer, I believe the best use of my time is to put my efforts not only to the code that I deliver but mainly to develop all the tools, DevOps setup, continuous integration, automated code analysis and everything essential for the developers. The best-case scenario is when your developers can use them independently and you don’t need to worry about them.

{{< highlight js >}}
// Synchronize user data and set up
var currentUser = new Talk.User({
    id: 79302,                      // user id
    name: "George Looney",          // full name
    email: "[email protected]"      // for offline email fallback
});
var session = new Talk.Session({
    appId: "9352938974",            // your TalkJS account id
    me: currentUser                 // make George the active user
});
{{< /highlight >}}

##### What do I mean by productivity
How do you define productivity though? And how can you measure it? I have a possibly surprising answer—it’s not about a number of lines of code you’ve written or features you’ve delivered. Productivity is something rather different. It’s all about focusing on the present and the future rather than the past. Productivity is more of a bet that in the future your team will be able to deliver more at the same time. It has way more to do with your perceived speed and enjoying work as a developer than the outcome of work you’ve already done. It is about building the potential for future gains while focusing on delivering the project.

I could think of a number of projects where I followed that philosophy of work and where it boosted the productivity of my team. Today, however, I’d like to focus on Apache Airflow.

##### The Apache Airflow project’s setup
Initially, we started contributing to this fantastic open-source project with a team of three which then grew to five. When we kicked it off a year ago, I realized pretty soon where the biggest bottlenecks and areas for improvement in terms of productivity were. Even with the help of our client, who provided us with a “homegrown” development environment it took us literally days to set it up and learn some basics.

Apache Airflow is a thoroughly tested project—it has almost 4,000 tests with around 80% coverage and varying complexity (from simple unit tests to end-to-end system tests). Airflow follows a modern software project philosophy: every single Pull Request can only be merged if all the tests pass. But that creates another problem—having to run all the tests for all the backends (there are 3 of them) and different python versions. At the time of starting the project we still supported 2.7, 3.5 and 3.6—so you had to wait sometimes for several hours to have the results of the CI build, depending on the delays and queues. At times, you have no control over the CI system you use. For example, recently we experienced hours of delays before we could even see the build starting.

This is hardly acceptable to any developer. Usually you can work on several issues in parallel. Such waiting is not the worst thing that can happen, but there are always costs of context-switching, distractions, getting out of the flow and the good old “I already forgot what I was doing” phase by the time the build is completed. It’s even worse when you want to make a small Pull Request. For example, when you find a bug and have just one line of fix for it. You submit the Pull Request and… half a day later you see that your single line of code change is badly formatted.

I hope this information will help you improve your program.