Coder Social home page Coder Social logo

thomashirtz / notion-scholar Goto Github PK

View Code? Open in Web Editor NEW
141.0 5.0 21.0 1.46 MB

Reference management solution using Python and Notion.

License: Apache License 2.0

Python 89.89% TeX 10.11%
notion notion-api notion-database notion-py bibliography reference-manager publications research-paper

notion-scholar's Introduction

notion-scholar

Reference management solution using Python and Notion.

The main idea of this app is to allow to furnish a Notion database using a BibTex formatted string or file. It also helps you to organize papers, create a bibliography and give you the opportunity to annotate the publications directly on their Notion's pages.

publication-database

Example of a database obtained using notion-scholar.

Installation

pip install git+https://github.com/thomashirtz/notion-scholar#egg=notion-scholar

You can now call the application using notion-scholar or ns

Never used python nor github before ?
  1. Install Python 3.7 or above (tutorial available online)
  2. Install pip (Generally included with Python)
  3. Run the command pip install ... above using your machine command line (Maybe need to launch the command prompt as administrator on Windows)

Setting up

1. Creation of a notion database

You can either duplicate the page template using the link below, or create a new database with the properties listed in requirements.

Template

Preview images

dashboard


category-page

Link to the template

Requirements

The properties necessary to import publications in a database are the following:

Property Name Property Type
Title Title
Authors Text
Year Number
Journal Text
Filename Text
URL URL
Abstract Text
DOI Text
Type Select
Bibtex Text
Inbox Checkbox

No space should be in the names of the properties.

2. Creation of an integration

Create an integration for the notion-scholar database. The integration needs to target the workplace containing the publication database.

Option needed:

  • Internal Integration
  • Read content
  • Update content
  • Insert content

Copy the Internal Integration Token for the step 4.

3. Share the database with the Integration

Go to your database in notion => Click on ... (top right) => + Add connections => Select the integration that you just created.

Copy the link of the database (simply the URL on a browser, on the application => Click on ... => Copy Link) for the step 4.

It is the database page and not the dashboard page that needs to be integrated. Moreover, it is also the URL of the database page that needs to be set into notion-scholar.

4. Set the token and database_id in notion-scholar

For the first use, it is recommended to set up the configuration file. The main parameters to save are the token (which will be securely saved using the keyring library) and the database-url.

ns set-config --token <token> --database-id <database_id>

or

ns set-config -t <token> -db <database_id>

The database_id is one part of the URL: https://www.notion.so/<workspace_name>/<database_id>?v=<view_id>

For example, if the database url is :
https://www.notion.so/thomashirtz/be31bb3f190f46c69e804785cc83c7ce?v=5674957e71e24aeb91d6db65f7978f0c
The command that needs to be used to set the database is :
ns set-config -db be31bb3f190f46c69e804785cc83c7ce

5. Set the bib file path (not required, but recommended)

If you want to set the default bib path that will be used when the ns run is called, you can set it by typing:

ns set-config --bib-file-path <bib_file_path>

or

ns set-config -f <bib_file_path>

The bib file needs to exist and the file path needs to be absolute.

How to use ?

There is three main way to use this application:

  1. Setting up the bib-file-path and call the run mode (It will automatically upload the new publications from the bib file into the database):
ns run
  1. Giving the bib-file-path argument:
ns run -f <bib-file-path>
  1. Giving a bib-string argument (/!\ be careful to type three quotes """ before pasting the bib-string) :
ns run -s """<bib-string>"""

The --token (-t) and the --database_url (-db) are also required, however, if they have been set, they don't need to be added again to the argument list.

Help

For getting help it is possible to call:

ns --help
usage: Use "notion-scholar --help" or "ns --help" for more information

notion-scholar

positional arguments:
  {run,download,clear-config,inspect-config,set-config}
                        Selection of the action
    run                 Run notion-scholar.
    download            Download the bibtex entries present in the database.
    clear-config        Clear the notion-scholar config.
    inspect-config      Inspect the notion-scholar config.
    set-config          Save the default values of notion-scholar.

optional arguments:
  -h, --help            show this help message and exit

Five main modes exist: run, download, set-config, inspect-config and clear-config. The help can be called this way:

ns <mode> --help
Example:
ns run --help
usage: Use "notion-scholar --help" or "ns --help" for more information run [-h] [-t] [-db] [-f] [-s]

optional arguments:
  -h, --help            show this help message and exit
  -t , --token          Token used to connect to Notion. (default: None)
  -db , --database-id   Database that will be furnished. The database_id can be found in the url of the database:
                        https://www.notion.so/{workspace_name}/{database_id}?v={view_id} (default: None)
  -f , --bib-file-path
                        Bib file that will be used. This argument is required if the bib file is not saved in the
                        config and no bib-string is passed. (default: None)
  -s , --bib-string     Bibtex entries to add (must be in-between three quotes """<bib-string>"""). By default, the
                        entries will be saved to the bib file from the config. It is possible to disable this behavior
                        by changing the "save" option: "ns setup -save false".
ns set-config --help
usage: Use "notion-scholar --help" or "ns --help" for more information set [-h] [-f] [-s] [-t] [-db]

optional arguments:
  -h, --help            show this help message and exit
  -f , --bib-file-path
                        Save the input file path in the user config using "platformdirs". The path must be absolute
                        and the file need to exist. (current: None)
  -s , --save           Set whether the entries from "bib-string" will be saved in the bib file. (current: True)
  -t , --token          Save the Notion token using "keyring". (current: None)
  -db , --database-id   Save the database-id in the user config using the library "platformdirs". The database_id can
                        be found in the url of the database:
                        https://www.notion.so/{workspace_name}/{database_id}?v={view_id} (current: None)```

Tips

Bibtex keys

When uploading a BibTex file into the database, the BibTex citekey will be mapped to the property "filename". This is because it seems convenient to have the same name for the key and the filename. It is therefore advised to change the citekey before uploading to Notion. (If you want to change it manually afterwards you would need to change both the Bibtex and the Filename property, which is not very convinient.)

This is ~ the nomenclature I used for the citekey (source). Feel free to take inspiration:

Nomenclature

Filename: aaaayyyyxxxx.pdf

Where:

  • aaaa: Name of the first author (variable length)
  • yyyy: Year of publication (fixed length)
  • xxxx: First word of title, minus articles and other small words (variable length)

For example: Attention Is All You Need, Ashish Vaswani et al., 2017

Would give: vaswani2017attention.pdf

If an author publish two papers the same year with the same beginning first word, it is possible to add an 1 and 2 depending on which one was the first released.

For example:

  • Soft Actor-Critic Algorithms and Applications, Tuomas Haarnoja et al., 2018 haarnoja2018soft1.pdf
  • Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Tuomas Haarnoja et al., 2018 haarnoja2018soft2.pdf

Copy equation properties

It is possible to copy the equation in the table view. Here is a comment to explain how, it can be very useful.

Feedbacks

If you want to see a new feature, or you have some feedback to give, feel free to email me at [email protected].

Buy me a coffee ツ

If this repository helped you or you if you like this project, feel free to support me!

Donate with PayPal

License

 Copyright 2021 Thomas Hirtz

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.

notion-scholar's People

Contributors

georg-wolflein avatar jordanc17 avatar thomashirtz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

notion-scholar's Issues

400 Bad Request error when loading in a BibTex file

Hi! I'm not sure what exactly I am doing wrong, but it seems like it isn't porting in the link for my Notion database URL. Any help would be appreciated! It also seems like the truncation issue that was talked about in a previous post is popping up here as well. Maybe it will still work if I fix the first issue, but just thought I should ask.

1/23: Publication(key="10.1039/c5ra11101c", title="{A review on natural surfactants}")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/notion_client/client.py", line 118, in _parse_response
    response.raise_for_status()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.notion.com/v1/pages'
For more information check: https://httpstatuses.com/400

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/ns", line 33, in <module>
    sys.exit(load_entry_point('notion-scholar==0.2.0', 'console_scripts', 'ns')())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/notion_scholar/main.py", line 143, in main
    return run(**config_manager.get_run_kwargs())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/notion_scholar/run.py", line 41, in run
    add_publications_to_database(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/notion_scholar/notion_api.py", line 59, in add_publications_to_database
    client.pages.create(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/notion_client/api_endpoints.py", line 199, in create
    return self.parent.request(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/notion_client/client.py", line 194, in request
    return self._parse_response(response)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/notion_client/client.py", line 126, in _parse_response
    raise APIResponseError(response, body["message"], code)
notion_client.errors.APIResponseError: body failed validation: body.properties.Bibtex.rich_text[0].text.content.length should be ≤ `2000`, instead was `2071`.


After following installation guide, DB links still point to thomashirtz.notion.so

This is a cool library -- it's always nice when you are looking for a functionality and it turns out someone has already implemented it. So first of all, thanks!

A quick note on usability of installation. The tl;dr is that after following the installation instructions, the Publication and Categories links still point to your Notion; I suggest adding a step in the tutorial explaining how to create new databases and link to them. In some more detail:

Steps to reproduce

  • Follow the installation guide, and specifically duplicate the page template as linked therein.

Expected outcome

  • Everything works correctly.

Actual outcome

  • All the views (Publication DB and Categories DB) link to [thomashirtz.notion.so].

Analysis and possible solution

  • Duplicating the template does not modify the links inside the template. It would be cool if there was a recursive "duplicate" functionality in Notion, but I'm not sure that exists. Instead, one could simply add a step a final step to the installation guide directing the user how to create their own Publication DB and Categories DB (e.g., by duplicating the respective pages in thomashirtz.notion.site ...).

The "-s" option is not working

Thank you for providing this useful Python package!

The bib-string command line option "-s" does not work as described in the README. It requires you to additionally also pass a bibliography file via "-f". I will provide a fix in a PR.

Feature request: automatically truncate strings that exceed 2000 characters

It seems that Notion's API restricts us to 2000 characters max. For example, trying to push an entry with a long abstract produces the error:

notion_client.errors.APIResponseError: body failed validation: body.properties.Bibtex.rich_text[0].text.content.length should be ≤ 2000, instead was 2123.

What are your thoughts on automatically truncating long strings? I've been running a preprocessing script before each notion-scholar run to do this, but figure it might be useful as a built-in feature for others.

HTTPStatusError

Hello.

I tried

$ pip uninstall notion-scholar
$ pip uninstall -r requirement.txt
$ pip install git+https://github.com/thomashirtz/notion-scholar#egg=notion-scholar

And I also refresh the token and reset the config.
But still I have the same problem below:

$ ns run -f NIPS-2017-attention-is-all-you-need-Bibtex.bib

Traceback (most recent call last):
  File "/home/hyeyun/anaconda3/lib/python3.8/site-packages/notion_client/client.py", line 118, in _parse_response
    response.raise_for_status()
  File "/home/hyeyun/anaconda3/lib/python3.8/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.notion.com/v1/pages'
For more information check: https://httpstatuses.com/400

During handling of the above exception, another exception occurred:
...

Thank you.

Invalid request URL

First of all, thanks so much for creating this amazing tool! This is exactly what I was looking for to easily manage all my references when writing my masters thesis, but unfortunately I can't get it to work.

I have went over the installation procedure several times, and think I am doing everything correctly.
I have installed the ns package with pip, created the integration, set the token, database and bib-file, but when running ns run, I get an 'notion_client.errors.APIResponseError: Invalid request URL.'

The connection to ns-scholar is in the Publication DB (I copied the template), and the database-id is set to the URL of this page.
Am I doing something wrong, or are there any solutions to this?

I am doing this on an M2 Mac, from the default zsh terminal. My Notion account is also accessed through my University email, as this grants me a free Education Plus Plan.

Here is the output I get in the terminal, I have also ran ns inspect-config first if the information should be useful:

erlendkopperud@eduroam-193-157-176-218 ~ % ns inspect-config

config_file_path: /Users/erlendkopperud/Library/Application Support/notion-scholar/config.ini
config_file_exist: True

token: secret_NIA0cY0UAJzxnt5rK1WeNJ4mMhK5FCAOEXpeflefq6A
database_id: https://www.notion.so/erlendkopp/199bf7f29eb64de3b439ec2796584249?v=ac9d18439dbd4b97ad0d9a38316f9a6b
file_path: /Users/erlendkopperud/Documents/Master/refs.bib
erlendkopperud@eduroam-193-157-176-218 ~ %    
erlendkopperud@eduroam-193-157-176-218 ~ % ns run
Traceback (most recent call last):
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/notion_client/client.py", line 118, in _parse_response
    response.raise_for_status()
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.notion.com/v1/databases/https://www.notion.so/erlendkopp/199bf7f29eb64de3b439ec2796584249?v=ac9d18439dbd4b97ad0d9a38316f9a6b/query'
For more information check: https://httpstatuses.com/400

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/bin/ns", line 8, in <module>
    sys.exit(main())
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/notion_scholar/main.py", line 143, in main
    return run(**config_manager.get_run_kwargs())
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/notion_scholar/run.py", line 35, in run
    key_list = get_publication_key_list_from_database(
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/notion_scholar/notion_api.py", line 104, in get_publication_key_list_from_database
    return get_property_list_from_database(
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/notion_scholar/notion_api.py", line 74, in get_property_list_from_database
    query = notion.databases.query(
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/notion_client/api_endpoints.py", line 132, in query
    return self.parent.request(
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/notion_client/client.py", line 194, in request
    return self._parse_response(response)
  File "/Users/erlendkopperud/.pyenv/versions/3.9.16/lib/python3.9/site-packages/notion_client/client.py", line 126, in _parse_response
    raise APIResponseError(response, body["message"], code)
notion_client.errors.APIResponseError: Invalid request URL.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.