Canonical (probably?) way to set file ID in Org about denote

[kaushalmodi]

[I might have gone overboard with this much chatter on the GitHub Issues in a single day. I will stop now.]

[protesilaos]

So I couldn't help but draw a parallel between the #+identifier
property created by denote (when denote-file-type is nil or Org) and
the :ID: property used by Org Roam.

Yes, this is a good use-case and I was exploring it myself. But...

Another benefit: Enables linking of files using Org ID

Have you tried it? I ran a test this morning and did it again. Instead
of visiting the file with the given id, it tells me that no heading
exists and asks to create one in the current buffer.

Maybe I am doing something wrong.

[I might have gone overboard with this much chatter on the GitHub
Issues in a single day. I will stop now.]

Haha, no worries! As you will notice from the times of the commits I
made this day, I am excited to contribute further to this project. And
I thank you once again for taking the time to share all your
knowledge---it helps a lot!

[kaushalmodi]

Have you tried it? I ran a test this morning and did it again. Instead of visiting the file with the given id, it tells me that no heading exists and asks to create one in the current buffer.

Yes.

May be you need to update the org-id-extra-files var? See that note I had in my earlier comment.

After you create the 2 Org files I saw in that example. eval that lisp form and then try to C-c C-o on that id: link.

I am excited to contribute further to this project. And I thank you once again for taking the time to share all your knowledge---it helps a lot!

I am enjoying following the activity in this project! And you are welcome!

[kaushalmodi]

Here's the entire recipe to make the Org ID linking work in a single Org file:

:PROPERTIES:
:ID: 20220610T134422
:END:
#+title:      idtest2
#+date:       [2022-06-10 Fri 13:44]
#+filetags:   org-id

1. The assumption is that another Org file with the below contents
   exists in the same directory (name of that file is not important):
   #+name: code__testfile1
   #+begin_src org
   :PROPERTIES:
   :ID: 20220610T134323
   :END:
   ,#+title:      idtest1
   ,#+date:       [2022-06-10 Fri 13:43]
   ,#+filetags:   org-id
   #+end_src

   - Note :: When you copy/paste the above snippet, remove the leading
     commas meant for escaping the keywords in there.
2. Then evaluate this (~C-c C-c~), and hit ~y~ at the next prompt:
   #+begin_src emacs-lisp :results none
   (require 'org-id)
   (setq org-id-extra-files (directory-files-recursively default-directory "\.org$"))
   #+end_src
3. Now with the cursor on the following ~id:~ link, hit ~C-c C-o~
   (bound by default to ~org-open-at-point~): [[id:20220610T134323]]
4. Your point should jump to the Org file where you saved [[code__testfile1][the above
   code snippet]].

[protesilaos]

Here's the entire recipe to make the Org ID linking work in a single
Org file:

Okay, I can reproduce it now. The downside is that we will be depending
on org-id to make this work, whereas now the package has no
dependencies of the sort (and is why I added the other file types).

Maybe we can provide this an an option? It will need more work though.

Also, we need to be careful not to re-invent org-roam: it already is
excellent, has a vibrant community, and so on. This is a tiny project
by comparison.

---

Since we are considering deeper intergration with Org, what are your
thoughts on an alternative approach. Instead of IDs, add a custom
hyperlink type. This is what I collected earlier today but abandonned
it to work on more pressing issues (I plan to revisit it):

[[info:org#Adding Hyperlink Types][org#Adding Hyperlink Types]]

When we start with something like this:

#+begin_src emacs-lisp
(org-link-set-parameters
 "denote"
 :follow #'denote-org-link-open
 :export #'denote-org-link-export
 :store #'denote-org-link-store)

(defun denote-link--find-file (identifier)
  "Visit file with IDENTIFIER.
Uses the function `denote-directory' to establish the path to the
file."
  (find-file (file-name-completion identifier (denote-directory))))

(defun denote-org-link-open (identifier _)
  (funcall #'testing identifier))
#+end_src

We can define links with the following syntax:

#+begin_example
[[denote:20220606T164519]]
#+end_example

I am not sure I like this approach.  It ultimately obfuscates the file
path and thus makes us dependent on Emacs/Org.

[kaushalmodi]

I am not sure I like this approach. It ultimately obfuscates the file path and thus makes us dependent on Emacs/Org.

That's true. It would be good to not have custom links for this because the id: links from org-id do the same thing.

The downside is that we will be depending on org-id to make this work

The linking using id: was just an example of added benefits if the files had the :ID: properties. I haven't read through the entire code, but would denote-link work with the proposed ID property if denote-link--identifier-regexp were changed? Then the link generation would be using the same file: approach that you have now, and there won't be any org-id dependency.

The upside of this ID format is that if user wants to use org-id and denote is already creating these ID properties, the org-id API for id: links will work out of the box.

Also, we need to be careful not to re-invent org-roam: it already is excellent, has a vibrant community, and so on. This is a tiny project by comparison.

That was my fear as well as I was suggesting this .. I am being careful so that the features of denote don't converge with that of Org Roam.

[kaushalmodi]

If the identifier is internally set to the ID property for the default file type (Org), will that need updating just the regexp for identifer search? If so, can we make it a default so that the ID goes where Org users would typically expect it?

This is of course given that it doesn't overly complicate your plan for creating links and updating backlinks.

[protesilaos]

If the identifier is internally set to the ID property for the default file type (Org), will that need updating just the regexp for identifer search? If so, can we make it a default so that the ID goes where Org users would typically expect it?

This is the same as what we discussed in #10, right?

This is of course given that it doesn't overly complicate your plan for creating links and updating backlinks.

I think we should do the right thing based on what the user expects.

Note though that the linking/backlinking facility still needs work. Damien Cassou raised some good points on the mailing about not automatically inserting backlinks, but using a dynamically generated list instead: https://lists.sr.ht/~protesilaos/denote/%3C87edzvd5oz.fsf%40cassou.me%3E#%[email protected]%3E. In the meantime, I wrote a proof-of-concept denote-link-backlinks command to generate such a buffer.

[protesilaos]

The principle I want to abide by is to have a solid but mostly generic linking facility which can work with all file formats. We can then add extensions to this core, depending on the case. I am open to ideas though and am willing to make breaking changes (as we did when we revised the file-naming scheme).

[kaushalmodi]

This is the same as what we discussed in #10, right?

Yes, denote-link and backlink insertion worked after the regexp update yesterday, without needing to add #+identifier to Org files.

I can test this out a bit more thoroughly tomorrow and update the status here.

Note though that the linking/backlinking facility still needs work.

Yes, I also don't like the concept of modifying the user-controlled file. I'll try out your proof of concept tomorrow.

The principle I want to abide by is to have a solid but mostly generic linking facility which can work with all file formats.

I believe that linking across multiple file formats already works (based on the code that I have read so far). I'll confirm this as the part of my more thorough testing tomorrow.

[protesilaos]

This is the same as what we discussed in #10, right?

Yes, denote-link and backlink insertion worked after the regexp update
yesterday, without needing to add #+identifier to Org files.

I can test this out a bit more thoroughly tomorrow and update the status here.

Thank you!

Note though that the linking/backlinking facility still needs work.

Yes, I also don't like the concept of modifying the user-controlled
file. I'll try out your proof of concept tomorrow.

Good! I think we will reach consensus here. As Damien argued on the
mailing list, editing files directly is a bad user experience and will
make our life difficult in the future if we ever want to make changes to
how we handle things.

The proof-of-concept feels like a step in the right direction. We are,
of course, about to change the technicalities but the core idea of
dynamically finding the backlinks and putting them in a buffer is what
will always work with standard tools.

We can even document CLI methods of getting the same results with
standard 'find' (if we end up using it ourselves). This would further
reinforce the notion that one's notes are not tied to Emacs.

The principle I want to abide by is to have a solid but mostly
generic linking facility which can work with all file formats.

I believe that linking across multiple file formats already works
(based on the code that I have read so far). I'll confirm this as the
part of my more thorough testing tomorrow.

The linking itself works. Otherwise we need to tweak the formats. I am
not sure about the style I chose for plain text links. It works, but
maybe it does not look nice. Though this is easy to change.

[jerm-the-wyrm]

I was thinking that instead of having the link be like [[denote:20220606T164519]] we could instead have it show up as [[denote:20220612T144708--test-note__testing.org:20220612T144708][Test Note]] allowing the identifier after the filename to be used in a similar manner to =org-id= which will help with #13

Doing this could open up the possibility of allowing identifiers to be placed at other points in the note perhaps under headings allowing for deeper notes while still maintaining the ability to link to individual ideas, these links could take the form of [[denote:20220612T144708--test-note__testing.org:20220612T145534][Test Note]] or perhaps to make the location of the inner link less obfuscated at the expense of longer links we could include the headline outline in the link like so
[[denote:20220612T144708--test-note__testing.org:test-heading/test-subheading/example-idea][Test Note]] or even
[[denote:20220612T144708--test-note__testing.org:test-heading/test-subheading/example-idea:20220612T145534][Test Note]] with a top level note appearing as [[denote:20220612T144708--test-note__testing.org::20220612T144708[Test Note]]

[protesilaos]

Hi @jerm-the-wyrm!

I was thinking that instead of having the link be like [[denote:20220606T164519]] we could instead have it show up as [[denote:20220612T144708--test-note__testing.org:20220612T144708][Test Note]] allowing the identifier after the filename to be used in a similar manner to =org-id= which will help with #13

Just for me to be sure I understand your suggestion: we should visually
show the whole file name but internally only read the identifier part?

[glechismi]

we should visually show the whole file name but
internally only read the identifier part?

Yes, that's exactly what i was going for!

Edit: I'm @jerm-the-wyrm but using my old account that I forgot my login to and can't reset my password, but I'm still logged in on my phone

[kaushalmodi]

or perhaps to make the location of the inner link less obfuscated at the expense of longer links we could include the headline outline in the link like so
[[denote:20220612T144708--test-note__testing.org:test-heading/test-subheading/example-idea][Test Note]] or even
[[denote:20220612T144708--test-note__testing.org:test-heading/test-subheading/example-idea:20220612T145534][Test Note]] with a top level note appearing as [[denote:20220612T144708--test-note__testing.org::20220612T144708[Test Note]]

It's very common to tweak/refactor sub-headings in posts. Having denote-link insert file names and/or sub-headings from the files will make the links break a lot more.

[jerm-the-wyrm]

@kaushalmodi

It's very common to tweak/refactor sub-headings in posts. Having denote-link insert file names and/or sub-headings from the files will make the links break a lot more.

You make a great point about renaming headings, so perhaps it is better to just use the file name plus the identifier like this [[denote:20220612T144708--test-note__testing.org:20220612T145534][Test Note]]
to your point in #13

I didn't understand how that suggestion will solve this issue with renaming. Your suggestion instead
hard-codes the file names in links which are prone to being renamed.
The link will follow the identifier itself, so even if the name changes the link will always work.
One of the ideas for the denote system is to remain non-dependant on emacs and org mode, thus we want to keep the name in the link to allow for note lookup that doesn't require fancy id finding functions.
Sure renaming files will break that, but with my link style, the identifier will still pull up the right note, and can be used with grep to find the correct note without emacs.

Perhaps though the better solution is to create a function that runs when you invoke denote-dired-rename-file it updates any links to that file.
And if we did go with my link style, we could also make a function denote-update-links that checks all links in the denote-directory and updates the file name.

[protesilaos]

Hello again folks!

An update on how things stand. I have created a custom Org hyperlink
type which is denote:. I have been testing it for the last few days.
It works like the file: type for following and completing links.

It also is like file: for specifying search options for it, as
described in the Org manual. Evaluate:

(info "(org) Search Options")

And it works like file: while exporting the document to HTML,
Markdown, LaTeX, and Info/Texinfo.

The denote: link accepts an identifier. Internally, that is used to
derive the file path relative to the current directory.

These behave in Org buffers just like ordinary file: links. For plain
text and Markdown, the command org-open-at-point-global also does the
trick.

I do not know of a method to define custom links for Markdown and there
probably isn't one for plain text either. The solution for Markdown and
plain text is probably a minor mode that buttonises those links, but
that is for version 0.2.0.

Perhaps though the better solution is to create a function that runs
when you invoke denote-dired-rename-file it updates any links to that
file.

With the current denote: link type, renaming files should not break
links (renaming always preserves the identifier).

And if we did go with my link style, we could also make a function
denote-update-links that checks all links in the denote-directory and
updates the file name.

I think what we will need in the future is a function to expand
denote:IDEENTIFIER into path/to/file links. Users may have
practical requirements to record the full path.

If we do that, we need another command to re-read and try to update any
such full path links.

[kaushalmodi]

I'm being a devil's advocate here ..

• How does the denote: link differ from the id: link? Looking at the manual, the denote: links always take an identifier. When I tested earlier, [[id:DENOTE_IDENTIFIER]] worked well with opening other files in Emacs and exporting.
• denote: link will not work with exporters other than the ones listed above out of the box, but id: does because it has been around for a while.
• id: links will face the same limitation as denote: when using with Markdown as plain text right now. The planned support for buttonizing them in 0.2.0 in a minor mode .. that could apply equally well to id: links too.

[protesilaos]

I'm being a devil's advocate here ..

That's a good thing. We need to assess this properly.

How does the denote: link differ from the id: link?

For the end-user, there should be no difference between file: and
denote:. The id: needs a :PROPERTIES: drawer. Then it has to be
set up, as you shared before:

(require 'org-id)
(setq org-id-extra-files (directory-files-recursively default-directory "\.org$"))

This means that we make org-id a dependency and then have to handle it
from there.

For context, I use Org for a lot of things on Emacs master but
(feature 'org-id) returns nil: it is not loaded here. As such, if I
am in an Org buffer and type C-c C-l there is no id: type among the
completion candidates.

We then have to consider org-open-at-point-global in plain text and
markdown files: will we need :PROPERTIES: drawers for those files too?
They look out-of-place even if they would work, meaning that I would not
like my .txt files to look like .org. Furthermore, I am not even
sure how such a drawer would be incorporated in YAML/TOML notation.

• denote: link will not work with exporters other than the ones
  listed above out of the box, but id: does because it has been
  around for a while.

Indeed. My expectation/hope is that we can cover those use-cases as
they arise.

[kaushalmodi]

This means that we make org-id a dependency and then have to handle it from there.

Yes, org-id will be needed as a dependency. My concern is that if we don't do that, the denote: support (portion supporting that in Org mode) will eventually grow up as a clone of the org-id.

(feature 'org-id) returns nil: it is not loaded here

Correct. It needs to be required.

As such, if I am in an Org buffer and type C-c C-l there is no id: type among the completion candidates.

I am not very familiar with that flow. We will need to figure that out.

will we need :PROPERTIES: drawers for those files too?

If you decide to support org-id, then the missing parting is extending that feature to Markdown and plain text. And in that, that org-id extension can be designed to support the existing identifier: or identifier = syntax we have right now.

My expectation/hope is that we can cover those use-cases as they arise.

Thank you.

[protesilaos]

My concern is that if we don't do that, the denote: support (portion
supporting that in Org mode) will eventually grow up as a clone of the
org-id.

That would be undesirable, for sure.

Looking at org-id.el, there is a lot of code not specific to what we
call the "identifier", as it relates to per-heading IDs. Other parts
handle the creation of IDs, which we also don't need.

If you decide to support org-id, then the missing parting is
extending that feature to Markdown and plain text. And in that, that
org-id extension can be designed to support the existing
identifier: or identifier = syntax we have right now.

In principle, I am fine with this. I just have no idea how to proceed.
Maybe things will become clear once I start experimenting with the code.

---

By the way, do you know if id: link types support search options the
way file: types do? I cna't find documentation about it. Explained
here:

(info "(org) Search Options")

Or online: https://orgmode.org/manual/Search-Options.html.

[protesilaos]

Hello again!

Just to let you know that today I will start working on an implementation with id: to replace denote:. This will be the final breaking change before version 0.1.0. I will put the code on a separate branch and will then discuss with you my findings.

[kaushalmodi]

@protesilaos Thank you for considering the use of id:! I won't be able to try out that branch for next 3 days (out vacationing, away from computer). But I'll look forward to trying it out as soon as I get a chance.

[nowislewis]

@protesilaos Recently I fell in love with denote and decided to make a migration from org-roam to note. But the work is really cumbersome(I have thousands of notes), and it would be very convenient if denote can support org-id.

I really like denote's reuse of existing ecosystems, which is concise and user-friendly. It's very customizable and extensible.

Thank you Protesilaos, you got me a great present.

[protesilaos]

Thanks @nowislewis! I am still working on this. It has taken me longer than expected but the plan has not changed.

[protesilaos]

Just pushed support for org-id to a new branch:

The gist is this:

(defcustom denote-use-org-id nil
  "When non-nil use the ID property and link type in Org files.
To use the ID property, a PROPERTIES drawer is added to the top
of newly created notes.  Furthermore, newly created links will
use the standard 'id:' hyperlink type instead of the custom
'denote:' type.

In practical terms, the ID ensures maximum compatibility with the
Org ecosystem.

When the value is nil, Denote uses a generic front matter for new
notes and links rely on the custom 'denote:' type (which should
behave the same as the standard 'file:' type).

Other files types beside Org always use the 'denote:' links and
keep their generic front matter."
  :type 'boolean
  :group 'denote)

What do you think?

Furthermore, do you think we can give this a thorough round of testing or should I release version 0.1.0 and then we can have as much time as we need for 0.2.0?

[protesilaos]

Sorry forgot to link to the branch: https://github.com/protesilaos/denote/tree/org-id

[protesilaos]

One problem I see with the id: link is that we need to handle links to other file types specially, as plain text and Markdown do not get a PROPERTIES drawer. I have not included such special casing yet.

I would prefer not to add such a drawer to plain text and Markdown: it should remain Org-specific.

[nowislewis]

I test it, it works fine and even works on my origin org-roam files, the only thing I need to change is to add #+date and then denote will discover the note. And I will use denote-dired-rename to rename all files.

[nowislewis]

I have to say, all of it works surprisingly well, the org-id support reduces me a lot of org-roam work that needs to be relinked

[nowislewis]

I found a thing which is a bit repetitive, if I want to modify the tag of a file, I can't modify it directly in the file with #+filetag, but need to go to the dired and execute denote-dired-rename-file, and then select the title and modify tags in order, even though I don't want to change the tile and just want to add one more tags.

I think it would be much more convenient to modify the file name or tags directly based on the file content (title, date, tag, etc.), or even a func which will refresh all file names under the path according to the content (title, date, tag, etc.).

But now it's quite good that this kind of thing doesn't happen very often

[kaushalmodi]

One problem I see with the id: link is that we need to handle links to other file types specially, as plain text and Markdown do not get a PROPERTIES drawer.

Can Org format use the property drawer format and rest others use the identifier: or identifier = format? I need to verify that again but I had seen that they used to get interlinked correctly.

---

Looking at the commit, I noticed that a new Org format, with the ID property drawer is supported. Do we need that additional option?

I'll try this out today. Thanks!

[kaushalmodi]

do you think we can give this a thorough round of testing?

For 0.1.0, may be allow a week's testing time?

[protesilaos]

Thanks @nowislewis for the feedback! I see that @kaushalmodi has a PR open, so we move to the technicalities. I am happy with the progress!

do you think we can give this a thorough round of testing?

For 0.1.0, may be allow a week's testing time?

I already asked on emacs-devel if Denote can be added to GNU ELPA. Assuming the patch is applied right now, it will take another day or a bit more before the package is available.

No pressure though. I will follow-up with version 0.2.0 shortly afterwards. But we might even manage to fit everything into 0.1.0, so let's see.

[protesilaos]

All the changes are now in main. The user option is denote-link-use-org-id, which you will need to set to non-nil to get the id: link type.

When we are in an Org file and link to, say, Markdown the link is always denote:. This also works when inserting links in bulk with denote-link-add-links. Basically, we do the right thing.

Please test it. I will now review the documentation for links and update the sample configuration.

[protesilaos]

I should add here that if we use id: to link to a note that does not have a PROPERTIES drawer, we do not get what we would expect. I think we do not need to account for this case, as the notes that were created prior to version 0.1.0 should not be considered immutable---we still were in the development phase.

What do you think?

[kaushalmodi]

as the notes that were created prior to version 0.1.0 should not be considered immutable---we still were in the development phase.

👍