Bug: Download failed: usearch_sqlite_macos_arm64_2.10.0.dylib could not be found about usearch

[simonw]

The error is coming from here:

usearch/python/usearch/__init__.py

Lines 103 to 110 in 708fede

	try:
	binary_path = self.download_binary(self.sqlite_download_url(), temp_dir)
	except HTTPError as e:
	if e.code == 404:
	warnings.warn(f"Download failed: {e.url} could not be found.", UserWarning)
	else:
	warnings.warn(f"Download failed with HTTP error: {e.code} {e.reason}", UserWarning)
	return None

Here's where that download URL is constructed:

usearch/python/usearch/__init__.py

Lines 25 to 51 in 708fede

	def sqlite_download_url(self) -> str:
	"""
	Constructs a download URL for the `usearch_sqlite` binary based on the operating system, architecture, and version.
	Args:
	version (str): The version of the binary to download.
	Returns:
	A string representing the download URL.
	"""
	version = self.version
	base_url = "https://github.com/unum-cloud/usearch/releases/download"
	os_map = {"Linux": "linux", "Windows": "windows", "Darwin": "macos"}
	arch_map = {
	"x86_64": "amd64" if platform.system() != "Darwin" else "x86_64",
	"AMD64": "amd64",
	"arm64": "arm64",
	"aarch64": "arm64",
	"x86": "x86",
	}
	os_part = os_map.get(platform.system(), "")
	arch = platform.machine()
	arch_part = arch_map.get(arch, "")
	extension = {"Linux": "so", "Windows": "dll", "Darwin": "dylib"}.get(platform.system(), "")
	filename = f"usearch_sqlite_{os_part}_{arch_part}_{version}.{extension}"
	url = f"{base_url}/v{version}/{filename}"
	return url

[simonw]

That worked perfectly!

$ pip install usearch
Collecting usearch
  Downloading usearch-2.10.3-cp311-cp311-macosx_11_0_arm64.whl (400 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 400.3/400.3 kB 2.8 MB/s eta 0:00:00
Requirement already satisfied: numpy in /Users/simon/.local/share/virtualenvs/foop-q39PfZK-/lib/python3.11/site-packages (from usearch) (1.24.3)
Requirement already satisfied: tqdm in /Users/simon/.local/share/virtualenvs/foop-q39PfZK-/lib/python3.11/site-packages (from usearch) (4.65.0)
Installing collected packages: usearch
Successfully installed usearch-2.10.3
$ python

Python 3.11.8 (main, Feb  6 2024, 21:21:21) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import usearch
>>> usearch.sqlite_path()
/Users/simon/.local/share/virtualenvs/foop-q39PfZK-/lib/python3.11/site-packages/usearch/__init__.py:93: UserWarning: Will download `usearch_sqlite` binary from GitHub.
  warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)
'/Users/simon/.local/share/virtualenvs/foop-q39PfZK-/bin/usearch_binaries/usearch_sqlite'
>>> import sqlite3
>>> conn = sqlite3.connect(":memory:")
>>> conn.enable_load_extension(True)
>>> conn.load_extension(usearch.sqlite_path())
>>> [fn[0] for fn in conn.execute('pragma function_list').fetchall()]
['pow', 'group_concat', 'group_concat', 'json_type', 'json_type', 'jsonb_set', 'julianday', 'ntile', 'nullif', 'sqlite_compileoption_get', 'json_valid', 'json_valid', 'json_quote', 'json_patch', '->', 'json_array', 'current_timestamp', 'power', 'sqlite_compileoption_used', 'json_remove', 'jsonb_patch', 'json_object', 'json_insert', '->>', 'jsonb_array', 'sin', 'sum', 'quote', 'printf', 'likelihood', 'json_replace', 'jsonb_remove', 'jsonb_object', 'jsonb_insert', 'json_extract', 'last_value', 'rank', 'sign', 'sqrt', 'sinh', 'tan', 'round', 'round', 'rtrim', 'rtrim', 'jsonb_replace', 'jsonb_extract', 'nth_value', 'tanh', 'random', 'trim', 'trim', 'time', 'radians', 'trunc', 'total', 'substr', 'substr', 'replace', 'unhex', 'unhex', 'upper', 'subtype', 'typeof', 'load_extension', 'load_extension', 'json_group_array', 'avg', 'abs', 'octet_length', 'json_group_object', 'jsonb_group_array', 'json_array_length', 'json_array_length', 'strftime', 'atan', 'asin', 'acos', 'substring', 'substring', 'randomblob', 'unicode', 'jsonb_group_object', 'timediff', 'percent_rank', 'row_number', 'atanh', 'asinh', 'acosh', 'cos', 'atan2', 'string_agg', 'last_insert_rowid', 'sqlite_log', 'unlikely', 'json_error_position', 'cosh', 'ceil', 'char', 'unixepoch', 'exp', 'count', 'count', 'date', 'concat', 'ceiling', 'total_changes', 'changes', 'sqlite_version', 'degrees', 'floor', 'coalesce', 'glob', 'zeroblob', 'hex', 'iif', 'sqlite_source_id', 'concat_ws', 'format', 'datetime', 'cume_dist', 'ln', 'instr', 'json', 'dense_rank', 'log', 'log', 'ifnull', 'jsonb', 'current_date', 'current_time', 'lag', 'lag', 'lag', 'mod', 'log2', 'like', 'like', 'max', 'max', 'min', 'min', 'lead', 'lead', 'lead', 'log10', 'lower', 'ltrim', 'ltrim', 'first_value', 'pi', 'length', 'likely', 'json_set', 'distance_inner_i8', 'distance_cosine_i8', 'distance_inner_f16', 'distance_sqeuclidean_f16', 'distance_divergence_f32', 'fts3_tokenizer', 'fts3_tokenizer', 'distance_divergence_i8', 'bm25', 'distance_jaccard_binary', 'distance_sqeuclidean_i8', 'highlight', 'rtreenode', 'rtreecheck', 'geopoly_regular', 'matchinfo', 'matchinfo', 'geopoly_debug', 'geopoly_bbox', 'distance_hamming_unicode', 'fts5_source_id', 'geopoly_contains_point', 'geopoly_svg', 'distance_haversine_meters', 'snippet', 'geopoly_overlap', 'distance_sqeuclidean_f64', 'distance_hamming_bytes', 'distance_levenshtein_unicode', 'distance_cosine_f32', 'distance_cosine_f16', 'rtreedepth', 'geopoly_area', 'match', 'geopoly_blob', 'optimize', 'geopoly_within', 'distance_inner_f64', 'geopoly_json', 'offsets', 'distance_cosine_f64', 'geopoly_xform', 'distance_divergence_f16', 'fts5', 'distance_sqeuclidean_f32', 'geopoly_ccw', 'distance_hamming_binary', 'distance_levenshtein_bytes', 'distance_divergence_f64', 'geopoly_group_bbox', 'distance_inner_f32']
>>> conn.execute('select distance_cosine_f32(?, ?)', (repr([1.1, 2, 3]), repr([3.1, 1, 2]))).fetchall()
[(0.20811247825622559,)]

[ashvardanian]

I am trying to avoid a breaking change in how the SQLite module is loaded, and it really overcomplicates things. Working on a hot-fix now 🤗

[ashvardanian]

@simonw there are a couple of issues here.

1. SQLite will add .dylib suffix at the end of the name looking for a .dylib.dylib file.
2. Even after succesfull download, my M2 Mac complains about the build compatibility.

>>> conn.load_extension(usearch.sqlite.removesuffix('.dylib'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlopen(/var/folders/r0/mw8phgys4lg_60kgh8xm_x780000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib, 0x000A): tried: '/var/folders/r0/mw8phgys4lg_60kgh8xm_x780000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/var/folders/r0/mw8phgys4lg_60kgh8xm_x780000gn/T/usearch_

That doesn't happen with local builds, so it's somewhere in the CI. Still checking 😉

[ashvardanian]

🎉 This issue has been resolved in version 2.10.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[ashvardanian]

I am gonna change the import method, which is technical a breaking change and should trigger a major release, but assuming the low adoption of the SQLite API, I hope it's fine 🤗

[simonw]

It's not quite working for me still:

```pycon
>>> import usearch
/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/usearch/__init__.py:100: UserWarning: Will download `usearch_sqlite` binary from GitHub.
  warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)
>>> import sqlite3
>>> conn = sqlite3.connect(":memory:")
>>> conn.enable_load_extension(True)
>>> conn.load_extension(usearch.sqlite_path())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'usearch' has no attribute 'sqlite_path'
>>> usearch.sqlite
'/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib'
>>> conn.load_extension(usearch.sqlite)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlopen(/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib.dylib, 0x000A): tried: '/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib.dylib' (no such file), '/var
>>> conn.lead_extension(usearch.sqlite.replace('.dylib', ''))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'sqlite3.Connection' object has no attribute 'lead_extension'. Did you mean: 'load_extension'?
>>> conn.load_extension(usearch.sqlite.replace('.dylib', ''))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlopen(/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib, 0x000A): tried: '/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_

It says it got x86_64 but needed arm64.

Also note I had to strip the .dylib from the end of the path - plus usearch.sqlite_path() has been renamed usearch.sqlite` - that's a bug in this README:

usearch/sqlite/README.md

Lines 28 to 33 in 3fd5184

	import sqlite3
	import usearch
	conn = sqlite3.connect(":memory:")
	conn.enable_load_extension(True)
	conn.load_extension(usearch.sqlite_path())

[simonw]

I tried downloading this: https://github.com/unum-cloud/usearch/releases/download/v2.10.1/usearch_sqlite_macos_arm64_2.10.1.dylib

Then opening it like this:

>>> conn.load_extension("/tmp/usearch_sqlite_macos_arm64_2.10.1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlopen(/tmp/usearch_sqlite_macos_arm64_2.10.1.dylib, 0x000A): tried: '/tmp/usearch_sqlite_macos_arm64_2.10.1.dylib' (code signature in <DCF25A47-045B-3D73-BE3E-6EF40C87A897> '/private/tmp/usearch_sqlite_macos_arm64_2.10.1.dylib' not valid for use in process: library load disallowed by system policy), '/System/Volumes/Preboot/Cryptexes

I fixed that error by opening up the macOS System Preferences and going to Security and allowing that binary. On a second attempt:

>>> conn.load_extension("/tmp/usearch_sqlite_macos_arm64_2.10.1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlsym(0x81325550, sqlite3_usearchsqlitemacosarm_init): symbol not found

[ashvardanian]

Yes, the right architecture wasn't selected in CI for the 2.10.0 build, that you are using. But the 2.10.1 should be fixing that. The CI can take a couple of hours to complete all the tests and I will try again afterwards.

---

@simonw, your log, however, is quite handy, as it highlights another issue - SQLite is searching for a weird symbol name when loading the library - sqlite3_usearchsqlitemacosarm_init. That will be an issue even with the new installation. If you git clone and then pip install -e . locally, the issue won't happen. Alternatively, you can rename the binary to usearch_sqlite.dylib locally, it may pass.

---

Give me a few hors to polish this. Sorry for inconvenience, debugging GitHub CI is an absolute nightmare 🧠 🔫

[ashvardanian]

@simonw, can you please check the 2.10.3? It should work fine.