paulbricman / conceptarium Goto Github PK

A fluid medium for storing, relating, and surfacing thoughts.

Home Page: https://paulbricman.com/thoughtware/conceptarium

License: Mozilla Public License 2.0

Python 98.88% Dockerfile 1.12%

pkm knowledge-base knowledge-management tools-for-thought thoughtware

conceptarium's Introduction

title	emoji	colorFrom	colorTo	sdk	app_file	pinned
conceptarium	💡	green	gray	streamlit	frontend/main.py	false

screenshot 1	screenshot 2

💡 Conceptarium

The conceptarium is an experimental personal knowledge base designed to weave AI capabilities into knowledge work. Its main features include:

powerful multi-modal search across ideas
sharing microverses of knowledge with peers
ranking items by Anki-like activation, so as to promote serendipity

Installation

Docker

After installing docker and docker-compose, run:

# install with:
curl -fsS https://raw.githubusercontent.com/paulbricman/conceptarium/main/docker-compose.yml -o docker-compose.yml
mkdir knowledge
docker-compose up -d

# stop with:
docker-compose stop

# update with:
docker-compose stop
docker-compose rm -f
docker-compose pull
docker-compose up -d

Note that you'll have to wait a bit initially for the models to be downloaded in the docker container. Use docker logs <backend container ID> or watch the process's memory for feedback on that. Or just try using it until it via the API or UI until it works (see usage).

Source

After pulling this repo run:

python3 -m pip install -r frontend/requirements.txt
python3 -m pip install -r backend/requirements.txt
streamlit run frontend/main.py

# in a separate session:
cd backend
python3 -m uvicorn main:app --reload

# update by pulling from repo again

Missing dependencies? Please have a look at frontend/Dockerfile and backend/Dockerfile. ARM architecture (e.g. Raspberry Pi)? Remove the torch entries from requirements.txt, and install a custom-built version.

Usage

The web app should then be available at localhost:8501, while the API at localhost:8000 (with docs at localhost:8000/docs). The backend component takes a few minutes to get the ML models at first.

To access your local instance, enter the conceptarium URL (i.e. localhost:8000 if you ran from source, backend.docker:8000 if you used docker), and your desired token. Remember your token, as you'll have to use it to authenticate in future sessions.

conceptarium's People

Contributors

Stargazers

Watchers

Forkers

s-oravec amar-laksh tkukurin trajjan codexmonk lalax-systems maaliy primedeviation

conceptarium's Issues

Unable to search for thoughts

Unfortunately I have to open another issue, but we're getting there.

my token works now
I can save thoughts, or at least I get a response with a filename, metadata, and the embeddings

However, I cannot query for those thoughts. Through the API I just get an internal server error, but through the interface there are more details:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback:
File "/usr/local/lib/python3.9/site-packages/streamlit/script_runner.py", line 379, in _run_script
exec(code, module.dict)
File "/app/main.py", line 19, in
component.paint()
File "/app/components/navigator.py", line 25, in paint
st.session_state['authorized_thoughts'] = knowledge.load(
File "/app/components/knowledge.py", line 45, in load
content = json.loads(response.content)
File "/usr/local/lib/python3.9/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None

Add export functionality as API endpoint and UI button

Inspector component doesn't recognize image query as past item

Where is the docker data stored?

Hey @paulbricman - really neat work! I'm deploying the conceptarium, ideoscope and lexiscore in docker via docker-compose.

Quick question: where is the core data stored?

I'd like to add a volume mount my config so that I can take backups of the data. I looked at the source code and saw the metadata pickle, but this seems to live in /app.

compress input image during find/save

Installation on Raspberry Pi OS?

I'd first tried to install the conceptarium on the preinstalled 32-bit Pi OS, which failed because I could not get Streamlit to run.

Then I've tried it with the new 64-bit version of Pi OS. Streamlit works now, but the Pytorch-version linked in the readme could not be installed.

Have you managed to install the conceptarium on Raspberry Pi OS? If not, which OS have you been using, so that I can try that out?

Long texts crash the encoder

Some truncation strategy should help.

Move auth token from params to request header

Persist microverses and layout config using cookie

API docs now incomplete

Since you've changed the way the authorization works, the automatically generated API requests at localhost:8000/docs are not correct, because the token is not included anywhere in the output. After some trial and error, I've found out what works now (add -H 'authorization: Bearer <token>'\ to the curl request), but this might be an issue for someone just trying it out and relying on the requests provided by the docs.

probem with Docker installation

I've installed the new version of the conceptarium on MacOS following the Docker approach. During the installation there were no errors, but now I cannot get the Docker image to run.

The frontend image exits immediately after starting it, with the following error in the log:
Error: Invalid value: File does not exist: main.py
Usage: streamlit run [OPTIONS] TARGET [ARGS]...
Try 'streamlit run --help' for help.

The backend image does show as "Running", but looking at the log there is also an error:
INFO: Will watch for changes in these directories: ['/app']
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: Started reloader process [8] using watchgod
ERROR: Error loading ASGI app. Could not import module "main".

Make layout tweakable in terms of columns (especially for mobile)

Missing dependencies during install

On Mac (11.6), during installation:

Knowledge folder not created when installed from source

My imported knowledge folder should have gone into the base conceptarium directory, not the main user folder as was the case with the Docker version.

Still: Shouldn't it have created such a folder for me when I first try to log in?

Based on #25

split html header into separate file for easier customization

Unable to access conceptarium via the API

I've installed it via Docker and entered a token at localhost:8501. But I can't access it through the Curl or Request URL provided in the docs. Check custodian results in "{"custodian":false,"authorized_microverse":[]} and /save in "Only the conceptarium's custodian can save thoughts in it."

One thing that came to mind: I've used special characters in the token. Could the issue be with the URL encoding?

Cache images in viewport

Create docker image installation option

It might make it easier to install, though on systems with low memory (i.e. Raspberry Pi), it might not be the best idea.

Microverse cookies don't update on mobile

Illegal log values in activation expression mess up ranking, even if activation := 0

Batch import via adding notes in knowledge folder

I'm now curious how to save these notes. I was initially assuming that all text files in the knowledge folder would show up as notes here, but it seems at least the note I placed there is not showing, again I'm sure I'm doing something wrong.

Online demo instance showing error - No such file or directory: components

Your online instance hosted on Hugging Face is currently facing an issue, the one you link from your blog:

FileNotFoundError: [Errno 2] No such file or directory: 'components'
Traceback:

File "/home/user/.local/lib/python3.8/site-packages/streamlit/script_runner.py", line 354, in _run_script
    exec(code, module.__dict__)
File "/home/user/app/frontend/main.py", line 12, in <module>
    microverses.paint()
File "frontend/components/microverses.py", line 24, in paint
    components = [e.split('.')[0] for e in os.listdir('components') if e.endswith(

Unrelated, but thank you for the interesting write-ups!

Add separate mounted volume option for core data in Docker deployment

TLDR: backups and upgrades that preserve data

Right now, the data for the conceptarium is stored in the folder tree of the docker image. By definition, docker images are temporary and transient. If I were to pull a new image, that folder would get overwritten as there would be a new image and new filesystem in use.

By decoupling the data location, I can either store my data in a dedicated docker volume, or I can use my local filesystem and run my usual filesystem based backups on that folder.

If you plan to add full support for docker, it might be useful to take a $CONCEPTARIUM_DATA_PATH variable, such that it can be set in the docker environment field and pulled with os.getenv('CONCEPTARIUM_DATA_PATH'). This way your data won't live inside the app, and you can decouple source code and generated data.

Suggested in #7

Help in installation

Hi, I'm someone from a non-coding background so I have some difficulties following the instructions in the installation guide. Can you elaborate more on the installation guide (a step-by-step installation procedure would be great) in readme so that individuals from non-technical backgrounds can also make use of this wonderful resource.

Additionally can you explain how I can self-host this application using a salvaged desktop?