ringsaturn / tzf

Get timezone via longitude and latitude in Go in a fast way

TZF: a fast timezone finder for Go.

TZF is a fast timezone finder package designed for Go. It allows you to quickly find the timezone for a given latitude and longitude, making it ideal for geo queries and services such as weather forecast APIs. With optimized performance and two different data options, TZF is a powerful tool for any Go developer's toolkit.

---

Note

Here are some language or server which built with tzf or it's other language bindings:

Language or Sever	Link	Note
Go	ringsaturn/tzf
Ruby	HarlemSquirrel/tzf-rb
Rust	ringsaturn/tzf-rs
Python	ringsaturn/tzfpy
HTTP API	ringsaturn/tzf-server	build with tzf
HTTP API	racemap/rust-tz-service	build with tzf-rs
Redis Server	ringsaturn/tzf-server	build with tzf
Redis Server	ringsaturn/redizone	build with tzf-rs

Quick Start

To start using TZF in your Go project, you first need to install the package:

go get github.com/ringsaturn/tzf

Then, you can use the following code to locate:

// Use about 150MB memory for init, and 60MB after GC.
package main

import (
	"fmt"

	"github.com/ringsaturn/tzf"
)

func main() {
	finder, err := tzf.NewDefaultFinder()
	if err != nil {
		panic(err)
	}
	fmt.Println(finder.GetTimezoneName(116.6386, 40.0786))
}

If you require a query result that is 100% accurate, use the following to locate:

// Use about 900MB memory for init, and 660MB after GC.
package main

import (
	"fmt"

	"github.com/ringsaturn/tzf"
	tzfrel "github.com/ringsaturn/tzf-rel"
	"github.com/ringsaturn/tzf/pb"
	"google.golang.org/protobuf/proto"
)

func main() {
	input := &pb.Timezones{}

	// Full data, about 83.5MB
	dataFile := tzfrel.FullData

	if err := proto.Unmarshal(dataFile, input); err != nil {
		panic(err)
	}
	finder, _ := tzf.NewFinderFromPB(input)
	fmt.Println(finder.GetTimezoneName(116.6386, 40.0786))
}

Best Practice

It's expensive to init tzf's Finder/FuzzyFinder/DefaultFinder, please consider reuse it or as a global var. Below is a global var example:

package main

import (
	"fmt"

	"github.com/ringsaturn/tzf"
)

var f tzf.F

func init() {
	var err error
	f, err = tzf.NewDefaultFinder()
	if err != nil {
		panic(err)
	}
}

func main() {
	fmt.Println(f.GetTimezoneName(116.3883, 39.9289))
	fmt.Println(f.GetTimezoneName(-73.935242, 40.730610))
}

CLI Tool

In addition to using TZF as a library in your Go projects, you can also use the tzf command-line interface (CLI) tool to quickly get the timezone name for a set of coordinates. To use the CLI tool, you first need to install it using the following command:

go install github.com/ringsaturn/tzf/cmd/tzf@latest

Once installed, you can use the tzf command followed by the latitude and longitude values to get the timezone name:

tzf -lng 116.3883 -lat 39.9289

Data

You can download the original data from https://github.com/evansiroky/timezone-boundary-builder.

The preprocessed protobuf data can be obtained from https://github.com/ringsaturn/tzf-rel, which has Go's embedded support. These files are Protocol Buffers messages for more efficient binary distribution, similar to Python wheels. You can view the pb/tzinfo.proto file or its HTML format documentation for information about the internal format.

The data pipeline for tzf can be illustrated as follows:

graph TD
    Raw[GeoJSON from evansiroky/timezone-boundary-builder]
    Full[Full: Probuf based data]
    Lite[Lite: smaller of Full data]
    Compressed[Compressed: Lite compressed via Polyline]
    Preindex[Tile based data]

    Finder[Finder: Polygon Based Finder]
    FuzzyFinder[FuzzyFinder: Tile based Finder]
    DefaultFinder[DefaultFinder: combine FuzzyFinder and Compressed Finder]

    Raw --> |cmd/geojson2tzpb|Full
    Full --> |cmd/reducetzpb|Lite
    Lite --> |cmd/compresstzpb|Compressed
    Lite --> |cmd/preindextzpb|Preindex

    Full --> |tzf.NewFinderFromPB|Finder
    Lite --> |tzf.NewFinderFromPB|Finder
    Compressed --> |tzf.NewFinderFromCompressed|Finder --> |tzf.NewDefaultFinder|DefaultFinder
    Preindex --> |tzf.NewFuzzyFinderFromPB|FuzzyFinder --> |tzf.NewDefaultFinder|DefaultFinder

Loading

The complete dataset (~80MB) can be used anywhere, but requires higher memory usage.

The lightweight dataset (~10MB) may not function optimally in some border areas.

You can observe points with different outcomes on this page.

If a slightly longer initialization time is tolerable, the compressed dataset (~5MB) derived from the lightweight dataset will be more suitable for binary distribution.

The pre-indexed dataset (~1.78MB) consists of multiple tiles. It is used within the DefaultFinder, which is built on FuzzyFinder, to reduce execution times of the raycasting algorithm.

I have written an article about the history of tzf, its Rust port, and its Rust port's Python binding; you can view it here.

Performance

The tzf package is intended for high-performance geospatial query services, such as weather forecasting APIs. Most queries can be returned within a very short time, averaging around 2000 nanoseconds.

Here is what has been done to improve performance:

1. Using pre-indexing to handle most queries takes approximately 1000 nanoseconds.
2. Using an RTree to filter candidate polygons, instead of iterating through all polygons, reduces the execution times of the Ray Casting algorithm.
3. Using a finely-tuned Ray Casting algorithm package https://github.com/tidwall/geojson to verify whether a polygon contains a point.

That's all. There are no black magic tricks inside the tzf package.

The benchmark was conducted using version https://github.com/ringsaturn/tzf/releases/tag/v0.10.0

goos: darwin
goarch: amd64
pkg: github.com/ringsaturn/tzf
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkDefaultFinder_GetTimezoneName_Random_WorldCities-16              441309              2778 ns/op              1000 ns/p50            10000 ns/p90            19000 ns/p99
BenchmarkFuzzyFinder_GetTimezoneName_Random_WorldCities-16               1000000              1077 ns/op              1000 ns/p50             2000 ns/p90             2000 ns/p99
BenchmarkGetTimezoneName-16                                               226834              5190 ns/op              5000 ns/p50             5000 ns/p90            22000 ns/p99
BenchmarkGetTimezoneNameAtEdge-16                                         211555              5606 ns/op              5000 ns/p50             6000 ns/p90            23000 ns/p99
BenchmarkGetTimezoneName_Random_WorldCities-16                            163000              7279 ns/op              7000 ns/p50            10000 ns/p90            29000 ns/p99
BenchmarkFullFinder_GetTimezoneName-16                                    212896              5556 ns/op              5000 ns/p50             6000 ns/p90            22000 ns/p99
BenchmarkFullFinder_GetTimezoneNameAtEdge-16                              195381              6262 ns/op              6000 ns/p50             7000 ns/p90            23000 ns/p99
BenchmarkFullFinder_GetTimezoneName_Random_WorldCities-16                 116652              9354 ns/op              8000 ns/p50            15000 ns/p90            31000 ns/p99
PASS
ok      github.com/ringsaturn/tzf       18.321s

• https://ringsaturn.github.io/tzf/ displays continuous benchmarking results.
• https://ringsaturn.github.io/tz-benchmark/ displays a continuous benchmark comparison with other packages.

Related Repos

• https://github.com/ringsaturn/tzf-rel Preprocessed probuf data release repo
• https://github.com/ringsaturn/tz-benchmark Continuous Benchmark Compared with other packages
• https://github.com/ringsaturn/tzf-rs Rust port of tzf
• https://github.com/ringsaturn/tzfpy Rust port's Python binding
• https://github.com/ringsaturn/tzf-server HTTP&Redis server build with tzf
• https://github.com/ringsaturn/redizone Redis compatible server build with tzf-rs

Thanks

• https://github.com/paulmach/orb
• https://github.com/tidwall/geojson
• https://github.com/jannikmi/timezonefinder
• https://github.com/evansiroky/timezone-boundary-builder

tzf's Issues

Tile key based speed up

• use x,y,z to store Polygon, and store small polygon for only needed
• reduce memory usage
• speed up query

TODO

• add preindex cli tool
• add DefaultFinder with both fuzzy&finder support
• update tzf-rel after #55 merged

Speed up preindex date generation time

Use multi goroutine to build data.

Hanoi is being mapped to Asia/Bangkok, not Asia/Ho_Chi_Minh

While they are effectively in the same relative time zone wrt UTC or GMT, I'd expect the correct match here would be Ho Chi Minh and not Bangkok, keeping the match to a location in the same country.

Is this an error or is there a particular reason for this mapping?

Reduce binary size

I am using tzf in a project and I am trying to reduce the size of the output binary.

I tried to minimize the binary size by letting the program download the dataset and then load it into memory.

Example:

import (
	"os"
	"sync"

	"github.com/ringsaturn/tzf"
	"github.com/ringsaturn/tzf/pb"
	"google.golang.org/protobuf/proto"
)

var TZFinder tzf.F
var onceInitTZFinder sync.Once

// load data to match timezones to coordinates
func InitTZFinder() {
	onceInitTZFinder.Do(func() {
		// filesize: ~ 90 MiB (2024-05-29)
		url := "https://raw.githubusercontent.com/ringsaturn/tzf-rel/main/combined-with-oceans.pb"
		path := "./tzf_complete.pb"

		// download database if necessary
		if !FileExists(path) {
			err := DownloadFile(url, path)
			if err != nil {
				panic(err)
			}
		}

		// read db
		rawDb, err := os.ReadFile(path)
		if err != nil {
			// TODO: assume it is corrupted and download again?
			panic(err)
		}

		// unmarshall
		db := &pb.Timezones{}
		err = proto.Unmarshal(rawDb, db)
		if err != nil {
			panic(err)
		}

		// init tzfinder
		TZFinder, err = tzf.NewFinderFromPB(db)
		if err != nil {
			panic(err)
		}
	})
}

According to goweight I still get 18 MB from github.com/ringsaturn/tzf-rel-lite.

Is there a way to fix this?

Check nearby point if no timezone found

Add method to return all matched timezones

Auto build new data in tzf-rel

• Add version to tzf-rel https://github.com/ringsaturn/tzf/releases/tag/v0.13.0
• Check upstream tag and run gen cmd in GitHub Actions with cron setting
  • #114
  • ringsaturn/tzf-rel#4

Release Python

Reduce memory usage

• The DefaultFinder will not use probuf timezone data, so just not save it when necessary
• After NewFinder from bin data, the allocs/heap will increase to a large number, but after GC, heap will reduce to far less. Maybe it’s possible to use sync.Pool to reduce init memory usage.

TODO

• Not save probuf timezone in memory for DefaultFinder #60
• Add sync.Pool for init

Release 1.0 stable version for Go/Rust/Python

1.0.0 check lists

• tzf-rel add more meta data #110
  • Release Go https://github.com/ringsaturn/tzf/releases/tag/v0.13.0
  • Release Rust https://github.com/ringsaturn/tzf-rs/releases/tag/v0.4.0
  • Release Python https://github.com/ringsaturn/tzfpy/releases/tag/v0.15.0
• Clean up before 1.0.0 https://github.com/ringsaturn/tzf/releases
  • Mark funcs as deprecated and remove before 1.0 release. Since breaking changes from upstream, like new timezone name will cause tzf init failed. tzf will only return string name, as tzf-rs&tzfpy. Refer #103
    • GetTimezoneLoc
    • GetTimezone
    • GetTimezoneShapeByName
    • GetTimezoneShapeByShift
    • #111
  • Move to internal
    • GetMinMax #104
  • Return interface for finder, instead of pointer of structs. Refer #109
• #30
• ringsaturn/geometry-rs#3
• ringsaturn/geometry-rs#6

Refactor reduce

https://github.com/rubenv/topojson

• Release new reduce tool #21
• Release new tzf-rel https://github.com/ringsaturn/tzf-rel/releases/tag/v0.0.2021-c1
• Rerun tzf’s lite/full compare and update README #22

Publish Python binding to PyPI

Refer:

• pypa/cibuildwheel#1192

Probuf polygon missing holes

• Release fixed tools #26
• Release fixed data in tzf-rel https://github.com/ringsaturn/tzf-rel/releases/tag/v0.0.2021-c2
• Release updated tzf/tzfpy #27 https://github.com/ringsaturn/tzf/releases/tag/v0.7.0

Bypass preindex for some locations

• Vatican City
• San Marino
• ...

---

• Release preindex tool fix this issue
• Release fixed data
• Release tzf
• Release tzf-rs
• Release tzfpy