Discussion: Grab Bag of Topics, Partially in Response to Issue #9 about zettlr

[adunning]

As a simpler method of working with Zotero, have you looked a integrating with https://github.com/retorquere/zotero-better-bibtex and simply using the built-in pandoc-citeproc support? He has a Zotero add-on that works quite nicely in Atom, simply filling in a citation key for Pandoc to process.

[nathanlesage]

Yes, you're right! I've seen that and decided that this is way better, therefore it'll come in one of the future versions!

[nathanlesage]

Hey,

these are definitely topics worth focusing upon, and some of them are already planned in some way similar to your ideas. Let me flesh out/categorize these ideas and merge them with my own ideas I've had.

This is a really long post, but I've tried to make it readable using headings.

Tabbed/pinned documents

What you basically mean, if I understand you correctly, is not that you open documents as mad to be added to some tab interface, but to define what documents should be automatically suggested to you based on your own choices. I've had something similar in mind: Filters.

What filters are doing is basically to create filters in certain folders (i.e. you right-click a folder and select "create new filter"). Then these filters are twofold. First, you can create something like "saved searches", i.e. you enter a search you would also enter into the global search field. This way you can dynamically generate a list of documents based on such a search — simply click on the filter to execute the search and only display the documents based on your criteria. The second option would be to create a so-called outline, i.e. a rather static filter. In this case, you could drag and drop files onto the filter, which would not result in the files being moved but added to the outline. Then, when you edit the filter (e.g. right-click on the filter and select "Edit" or something like that) you could even re-arrange the files. A second option would be to "export outline" which would basically simply pull in all files defined in the outline and concatenate them to one single document. This would be great for simple notes and short documents you've created somewhere and simply make one big document from them.

File metadata

The question of file metadata had appeared to me as well, but I am still reluctant to implement it because I want the files to still be accessible with ease from within other apps, because I do not want to enforce using Zettlr on the people using it. (Besides, Zettlr is designed to work with whole project folders, but if you want to simply preview a single markdown file — at least this is how I'm doing it currently — you could simply assign a single-page-markdown-app to opening markdown files directly, such as Abricotine.)

So what this boils down to is that I'm currently hesitant to writing additional metadata directly in the file, because that would look ugly in other text editors (yes, I'm one of these people even offended by the metadata vim writes to the bottom of text files). On the other hand, if you set yourself this barrier, you encounter the problem that metadata is handled differently by all operating systems and depend heavily on the file system (e.g. you'd need different approaches for macOS, Windows and Linux-based distributions mostly do not even support file metadata out of the box).

In short: Before thinking about metadata we would need to find a way to store it persistently across systems. Also, I'd like to point out that this would also be a precondition for filters to work properly (because currently I use a small hash based on the absolute path of a file as the identifier for files, which, of course, varies from system to system).

Workflow and time management

Concerning your ideas on workflow and time management practices, these sound really nice. To have something like "aims" for files (such as "Write xy words until yyyy-mm-dd") and even a calender parsing this information would be a really nice addon and I'll be willing to implement it. Still, we would first need to flesh out precisely how this should work, because this involves a massive overhaul of much of the core functionality to work smooth.

Also, I'm currently remembering the idea by a friend to develop a plugin API to move out heavy functionality to plugins so that the users may decide which functions they need and which they don't. This would also be good given the fact that the app is still based on Electron, which is pretty energy consuming. Therefore I'd like to maintain an approach that tries to level out energy consumption and functionality (especially in an era where many people use laptops, and believe me, at least in Germany there aren't so many power outlets).

Still, I am completely with you concerning the timer, that this functionality can definitely be improved a lot. But there's some ground work to do before tackling this …

Files as tasks

Now to your idea to handle files as tasks. I'm currently trying to implement more and more functionality that would shift the bias of the app to an equilibrium between long reads/long texts, such as whole research papers in one file and small notes that mostly don't contain more than 100 words (to use Zettlr both as a full-fledged writing app and a note box such as Evernote). This would give more freedom to use the app as one wishes without interfering with other workflows. Treating files as task directly would interfere with this as it would bind the functionality to the files.

What would be more appropriate, at least how I perceive it currently, would be to integrate such a management into Zettlr itself without touching files. This would mean that, instead of attaching tasks to files, one could attach files to tasks. In this way, you could globally define your goals, tasks and todos and could attach the files to them. In this respect, we would be able to solve several problems at once: First the metadata problem, because if we only write a unique ID to the file and, potentially tags, which I wanted to implement in the future then the amount of data appended to a file would be limited to a manageable size, while specific functionality would be stored in the app itself.

Current Roadmap

While I am really interested in developing these ideas from now on, their implementation may take some time, because I've got some other features I wanted to add to Zettlr, and which I am going to add first (and which also in some respect may also already include some of your thoughts). Therefore I'd like to implement the current roadmap in the discussion as well, not only to save time by not doubly developing something, but also to merge your workflow with mine. So what I'm proposing is: While we're developing these ideas, I'll simultaneously continue to add the functionality I had in mind for the app version 1.0 (so, what I'd call "base" functionality) and we can also discuss their specific implementation:

Zotero integration

Because I want Zettlr to be primarily an app for academics in the arts and humanities to some day even replace Microsoft Office (I hate editing those documents at work), I wanted to add a Zotero connector to Zettlr to make it easy to cite in your documents. This would be some more work. What I had in mind was the following:

• The app polls Zotero to check whether it is online or not.
• If it is online, one may press any button (maybe Cmd/Ctrl+Alt+Z or something like that) to open a drop down to search for a document to be cited.
• Once it is found, one can select it, press enter and Zettlr will insert a field (similar to LibreOffice fields) indicating that this is dynamically generated content.
• On export, one could be able to automatically generate a references dictionary based on what you've cited in the text. This brings me to the next roadmap point:

Enhanced PDF export

I want to enable users to export to common file types including PDF. While ODT-, DOCX-, and HTML-files are somewhat static using pandoc (and these are only meant for previewing or sharing the files with people that do not have Markdown experience), PDF offers a lot of flexibility, because Zettlr employs LaTeX to do so. As the template is easily editable, I wanted to enable two things:

• First offer something like the page-setup dialog of LibreOffice for the general export of files. So if you select the Share-button (e.g. what is currently named "Export" in the app), Zettlr would pull in the default PDF configuration, write a custom template and point pandoc to it to use for creating the final PDF file.
• Second be able to convert folders into something like sub-"Project Folder", that have two benefits: First, defining a custom PDF export, and second enable the export of the whole folder, so that you can use folders to categorize your files (e.g. use folders to indicate sections and then one file for each chapter in these sections) and Zettlr will take care of exporting the structure just as you defined it. This could be also applicable to the above mentioned outline-filters.

As configuration options one could think of author fields, a custom title, page margins, font options (size, family, alignment, spacing), whether to use footnotes or endnotes, if it should create a references chapter, a title page, etc.—basically everything the LaTeX-configuration has to offer.

Filters

As I've described them above.

Semi-automatic rendering of Markdown

With Zettlr, I try to combine the features I found interesting in other apps I've used in the past. There is the image previewing, which I have taken from Abricotine, the general layout with the file tree and file list which I have taken from Ulysses, but something that is still missing is something Typora implemented: There, all Markdown-Tags were immediately rendered as HTML, which means that the files looked even better than the default markdown. When clicking into an element, you could edit it as well as add/remove the tags. As I have currently implemented this functionality for images, I want to extend it to links, bold and italic text as well, and maybe footnote references. (Not all tags should be replaced, because I really like the heading-tags and it should still feel like markdown).

General improvements in the app

As of now there are still some bugs and inconsistent behavior in the app, which I want to purge until Zettlr reaches v1.0. Among them are mainly small things that—still—can be really annoying, such as an inconsistent italics/bold behaviour, or some really strange things the watchdog I am using to watch for external changes to files and folders does. This would be done during some time when all major features have been implemented that would be necessary to version 1.

Conclusion

To sum up, we have the following topics to cover:

• Contextuality — which files and folders are important at which times?
• Workflow — how can the app used for which tasks?
• Time Management — Organize your time using the app

I'm looking forward to discussing these issues further, because I think they really are what matter in an age where there are myriad markdown editors which all try to do something but not try to be as universal and usable as possibly. Yet, as I mentioned, there are some features which I wanted to implement in the next time as outlined in the roadmap. So maybe also have a look at it so that you may offer some thoughts on these, because they will be the first to be implemented!

[sahil48]

So, interestingly enough, the filter idea you mentioned was originally part of my grand plan for supercharging Matcha quite a few months ago. My original idea to simplify the interface Zettlr and Matcha currently use is to do away with the preview pane and have a long list of documents in a vertical and scrollable card-like format. They would be shown in their entirety in a list, with a toggle to collapse them to skim through more documents. If you needed to focus on a document, you made it full screen. A search bar was always on top and center, and typing in it would search the text of documents in the active view, and show just the relevant matching lines, sentences, or paragraphs to make it easier to find anything in a large set of documents. This approach was basically built around the idea of filtering documents and documents in sub folders of the folder selected. My intention was always to make documents more accessible to one-another, and to contribute to each other. Our intention is a little bit different from your own. To facilitate the above interface, and focus on developing a WYSIWYM editor great for devices of all sizes (even phones) for writing complex documents for the web (something I despise about word), we have been interested in devising a file format and folder structure that is both human readable and easy to understand by a computer, that largely adhered to the multimarkdown processor (i.e. it wouldn’t violate it’s existing syntax and simply render as plain text if necessary). I’m planning to release these ideas in a repo at some point (probably in May or August once my life settles down), but until then, we can definitely discuss these ideas further over email if you don’t mind letting me know your email address, so that we don’t clutter the issues section of Zettlr. From: Nathan LeSage <[email protected]> <[email protected]> Reply: Zettlr/Zettlr <[email protected]> <[email protected]> Date: February 26, 2018 at 7:25:33 PM To: Zettlr/Zettlr <[email protected]> <[email protected]> CC: sahil48 <[email protected]> <[email protected]>, Author <[email protected]> <[email protected]> Subject: Re: [Zettlr/Zettlr] Discussion: Grab Bag of Topics, Partially in Response to Issue #9 (#11) Hey,

…

these are definitely topics worth focusing upon, and some of them are already planned in some way similar to your ideas. Let me flesh out/categorize these ideas and merge them with my own ideas I've had. This is a really long post, but I've tried to make it readable using headings. Tabbed/pinned documents What you basically mean, if I understand you correctly, is not that you open documents as mad to be added to some tab interface, but to define what documents should be automatically suggested to you based on your own choices. I've had something similar in mind: *Filters*. What filters are doing is basically to create filters in certain folders (i.e. you right-click a folder and select "create new filter"). Then these filters are twofold. First, you can create something like "saved searches", i.e. you enter a search you would also enter into the global search field. This way you can dynamically generate a list of documents based on such a search — simply click on the filter to execute the search and only display the documents based on your criteria. The second option would be to create a so-called *outline*, i.e. a rather static filter. In this case, you could drag and drop files onto the filter, which would not result in the files being moved but added to the outline. Then, when you edit the filter (e.g. right-click on the filter and select "Edit" or something like that) you could even re-arrange the files. A second option would be to "export outline" which would basically simply pull in all files defined in the outline and concatenate them to one single document. This would be great for simple notes and short documents you've created somewhere and simply make one big document from them. File metadata The question of file metadata had appeared to me as well, but I am still reluctant to implement it because I want the files to still be accessible with ease from within other apps, because I do not want to enforce using Zettlr on the people using it. (Besides, Zettlr is designed to work with whole project folders, but if you want to simply preview a single markdown file — at least this is how I'm doing it currently — you could simply assign a single-page-markdown-app to opening markdown files directly, such as Abricotine <https://github.com/brrd/Abricotine>.) So what this boils down to is that I'm currently hesitant to writing additional metadata directly in the file, because that would look ugly in other text editors (yes, I'm one of these people even offended by the metadata vim writes to the bottom of text files). On the other hand, if you set yourself this barrier, you encounter the problem that metadata is handled differently by all operating systems and depend heavily on the file system (e.g. you'd need different approaches for macOS, Windows and Linux-based distributions mostly do not even support file metadata out of the box). *In short: Before thinking about metadata we would need to find a way to store it persistently across systems.* Also, I'd like to point out that this would also be a precondition for filters to work properly (because currently I use a small hash based on the absolute path of a file as the identifier for files, which, of course, varies from system to system). Workflow and time management Concerning your ideas on workflow and time management practices, these sound really nice. To have something like "aims" for files (such as "Write xy words until yyyy-mm-dd") and even a calender parsing this information would be a really nice addon and I'll be willing to implement it. Still, we would first need to flesh out precisely how this should work, because this involves a massive overhaul of much of the core functionality to work smooth. Also, I'm currently remembering the idea by a friend to develop a plugin API to move out heavy functionality to plugins so that the users may decide which functions they need and which they don't. This would also be good given the fact that the app is still based on Electron, which is pretty energy consuming. Therefore I'd like to maintain an approach that tries to level out energy consumption and functionality (especially in an era where many people use laptops, and believe me, at least in Germany there aren't so many power outlets). Still, I am completely with you concerning the timer, that this functionality can definitely be improved a lot. But there's some ground work to do before tackling this … Files as tasks Now to your idea to handle files as tasks. I'm currently trying to implement more and more functionality that would shift the bias of the app to an equilibrium between long reads/long texts, such as whole research papers in one file and small notes that mostly don't contain more than 100 words (to use Zettlr both as a full-fledged writing app and a note box such as Evernote). This would give more freedom to use the app as one wishes without interfering with other workflows. Treating files as task directly would interfere with this as it would bind the functionality to the files. What would be more appropriate, at least how I perceive it currently, would be to integrate such a management into Zettlr itself without touching files. This would mean that, instead of attaching tasks to files, one could attach files to tasks. In this way, you could globally define your goals, tasks and todos and could attach the files to them. In this respect, we would be able to solve several problems at once: First the metadata problem, because if we only write a unique ID to the file and, potentially tags, which I wanted to implement in the future then the amount of data appended to a file would be limited to a manageable size, while specific functionality would be stored in the app itself. Current Roadmap While I am really interested in developing these ideas from now on, their implementation may take some time, because I've got some other features I wanted to add to Zettlr, and which I am going to add first (and which also in some respect may also already include some of your thoughts). Therefore I'd like to implement the current roadmap in the discussion as well, not only to save time by not doubly developing something, but also to merge your workflow with mine. So what I'm proposing is: While we're developing these ideas, I'll simultaneously continue to add the functionality I had in mind for the app version 1.0 (so, what I'd call "base" functionality) and we can also discuss their specific implementation: *Zotero integration* Because I want Zettlr to be primarily an app for academics in the arts and humanities to some day even replace Microsoft Office (I hate editing those documents at work), I wanted to add a Zotero connector to Zettlr to make it easy to cite in your documents. This would be some more work. What I had in mind was the following: - The app polls Zotero to check whether it is online or not. - If it is online, one may press any button (maybe Cmd/Ctrl+Alt+Z or something like that) to open a drop down to search for a document to be cited. - Once it is found, one can select it, press enter and Zettlr will insert a field (similar to LibreOffice fields) indicating that this is dynamically generated content. - On export, one could be able to automatically generate a references dictionary based on what you've cited in the text. This brings me to the next roadmap point: *Enhanced PDF export* I want to enable users to export to common file types including PDF. While ODT-, DOCX-, and HTML-files are somewhat static using pandoc (and these are only meant for previewing or sharing the files with people that do not have Markdown experience), PDF offers a lot of flexibility, because Zettlr employs LaTeX to do so. As the template is easily editable, I wanted to enable two things: - First offer something like the page-setup dialog of LibreOffice for the general export of files. So if you select the Share-button (e.g. what is currently named "Export" in the app), Zettlr would pull in the default PDF configuration, write a custom template and point pandoc to it to use for creating the final PDF file. - Second be able to convert folders into something like sub-"Project Folder", that have two benefits: First, defining a custom PDF export, and second enable the export of the whole folder, so that you can use folders to categorize your files (e.g. use folders to indicate sections and then one file for each chapter in these sections) and Zettlr will take care of exporting the structure just as you defined it. This could be also applicable to the above mentioned outline-filters. As configuration options one could think of author fields, a custom title, page margins, font options (size, family, alignment, spacing), whether to use footnotes or endnotes, if it should create a references chapter, a title page, etc.—basically everything the LaTeX-configuration has to offer. *Filters* As I've described them above. *Semi-automatic rendering of Markdown* With Zettlr, I try to combine the features I found interesting in other apps I've used in the past. There is the image previewing, which I have taken from Abricotine, the general layout with the file tree and file list which I have taken from Ulysses, but something that is still missing is something Typora implemented: There, all Markdown-Tags were immediately rendered as HTML, which means that the files looked even better than the default markdown. When clicking into an element, you could edit it as well as add/remove the tags. As I have currently implemented this functionality for images, I want to extend it to links, bold and italic text as well, and maybe footnote references. (Not all tags should be replaced, because I really like the heading-tags and it should still feel like markdown). *General improvements in the app* As of now there are still some bugs and inconsistent behavior in the app, which I want to purge until Zettlr reaches v1.0. Among them are mainly small things that—still—can be really annoying, such as an inconsistent italics/bold behaviour, or some really strange things the watchdog I am using to watch for external changes to files and folders does. This would be done during some time when all major features have been implemented that would be necessary to version 1. Conclusion To sum up, we have the following topics to cover: - Contextuality — which files and folders are important at which times? - Workflow — how can the app used for which tasks? - Time Management — Organize your time using the app I'm looking forward to discussing these issues further, because I think they really are what matter in an age where there are myriad markdown editors which all try to do something but not try to be as universal and usable as possibly. Yet, as I mentioned, there are some features which I wanted to implement in the next time as outlined in the roadmap. So maybe also have a look at it so that you may offer some thoughts on these, because they will be the first to be implemented! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF0XJjSFQtBnGaBbH0HxtwrjlOh7jthUks5tY0t8gaJpZM4STWCr> .

[nathanlesage]

"[…] we can definitely discuss these ideas further over email if you don’t mind letting me know your email address, so that we don’t clutter the issues section of Zettlr."

I'd really like to keep the discussion open to the public. I mean, we'll only clutter this one issue, but not the complete section, and by doing it publicly, other people may also comment on this!

[nathanlesage]

This issue has been open for quite a bit. In the meantime, Zettlr has advanced to a different direction. I don't think it is a good idea to treat files as tasks in some way anymore. Most of the other features discussed here are either definitely coming (Zotero Integration, Filters), or have already been implemented.

Therefore I'm going to close the issue now.