First impressions with FORD about ford

[cmacmackin]

Thank you for your interest.

1. Yes, the Github issues is the right place to ask questions. As I'm the only one contributing code at this point, an mailing list seems like overkill.
2. I've been putting some thought into how comments could be placed before the code them are documenting. Your approach might work, but it wouldn't be my preference. FORD has been designed so that only one symbol designates documentation and, furthermore, this symbol can be specified by the user. Giving a special meaning to !> would conflict with the user's ability to chose what marks documentation. What I'd prefer to do would be to give the user a switch which would allow them to have comments come before code. This may be more achievable than I thought when I wrote the documentation. However, I can't give any estimate of when this behaviour will be implemented.

[mrestelli]

OK, thank you for the clarification.

I have done a simple attempt to include the functionality, the idea is
introduce another marker, predocmark = '>' (which can then be set by
the user analogously to docmark, or unset if this functionality is not
needed), and accumulate the corresponding lines in self.docbuffer in
reader.py.

The predocmark is converted into a docmark inside reader.py, so that
the rest of the code does not need any modification; essentially, they
are treated as the "online" comments already supported by FORD.

The disadvantage is that multiline anticipated comments must be
buffered before they can be processed; I don't know whether this is a
real limitation in practice.

I am sure that there are still many problems, and this patch does not
work for multiline anticipated comments, but before spending time on
this I would like to ask whether this makes sense to you at all.

I have tested it on this example:

!> A simple test module.
module m
!! Some more details concerning the module `m`: set
!! $$
!!  x_i = 5
!! $$

implicit none

 type :: t_x
  !! Some documentation for the type `t_x`

  !> Here is the first field in `t_x`.
  integer :: i !! It is an integer counter;
  !! the field `i` can be useful when a counter is needed.
 end type t_x

contains

 !> A function to set \(x_i=5\)
 pure subroutine s(x)
  !> the function argument;
  type(t_x), intent(out) :: x !! it will be set to 5
   x%i = 5
 end subroutine s

end module m

Here is the patch:

Index: reader.py
===================================================================
--- reader.py   (revision 35)
+++ reader.py   (working copy)
@@ -49,10 +49,14 @@
     def __init__(self,filename,docmark='!'):
         self.name = filename
         self.reader = open(filename,'r')
-        self.docbuffer = ""
+        #self.docbuffer = ""
+        self.docbuffer = []
         self.pending = []
         self.prevdoc = False
+        self.docmark = docmark
         self.doc_re = re.compile("^([^\"'!]|('[^']*')|(\"[^\"]*\"))*(!{}.*)$".format(docmark))
+        predocmark = '>'
+        self.predoc_re = re.compile("^([^\"'!]|('[^']*')|(\"[^\"]*\"))*(!{}.*)$".format(predocmark))

     def __iter__(self):
         return self
@@ -68,8 +72,8 @@
         # If there was documentation at the end of the previous line, return
         # it as the next line.
         elif len(self.docbuffer) != 0:
-            tmp = self.docbuffer
-            self.docbuffer = ""
+            #tmp = self.docbuffer
+            tmp = self.docbuffer.pop(0)
             self.prevdoc = True
             return tmp

@@ -81,10 +85,23 @@
         while not done:
             line = self.reader.next()
             if len(line.strip()) > 0 and line.strip()[0] == '#': continue
+            # Capture any anticipated documenation comments
+            match = self.predoc_re.match(line)
+            if match:
+                # Substitute predocmark with docmark
+                tmp = match.group(4)
+                tmp = tmp[:1] + self.docmark + tmp[2:]
+                print tmp
+                self.docbuffer.append(tmp)
+                line = line[0:match.start(4)]
+                if len(line.strip()) > 0:
+                    raise Exception("Anticipated documentation lines can not be online")
+                continue
             # Capture any documentation comments
             match = self.doc_re.match(line)
             if match:
-                self.docbuffer = match.group(4)
+                #self.docbuffer = match.group(4)
+                self.docbuffer.append(match.group(4))
                 line = line[0:match.start(4)]
             # Remove any regular comments
             match = self.COM_RE.match(line)
@@ -96,7 +113,8 @@
             if len(line) == 0:
                 if self.prevdoc and len(self.docbuffer) == 0:
                     #~ self.prevdoc = False
-                    self.docbuffer = "!!"
+                    #self.docbuffer = "!!"
+                    self.docbuffer.append("!"+self.docmark)
             else:
                 # Check if line is immediate continuation of previous
                 if line[0] == '&':
@@ -109,7 +127,7 @@
                 # Check if line will be continued
                 if line[-1] == '&':
                     continued = True
-                    self.docbuffer = ""
+                    #self.docbuffer = ""
                     line = line[0:-1]
                 else:
                     continued = False
@@ -143,8 +161,8 @@
             else:
                 return linebuffer
         else:
-            tmp = self.docbuffer
-            self.docbuffer = ""
+            tmp = self.docbuffer.pop(0)
+            #self.docbuffer = ""
             if tmp != "!!":
                 self.prevdoc = True;
             return tmp

[cmacmackin]

I'm slightly embarrassed that you were working with reader.py, to be honest, as there are some parts of that code which I want to clean up a bit--I feel like there are some unnecessary steps in there. In particular, I'd like to see if there is any way that I can get rid of docbuffer and just use the pending list instead. If you don't mind, before I even consider merging in these sorts of changes, I'd like to get reader.py tidied up.

[mrestelli]

Oh, sorry, I saw the comment about docbuffer but I did not know how
high it was on the todo list.

Anyway, meanwhile I did a few more twists to allow multiple
anticipated lines, accumulating them in the docbuffer. I am attaching
them here just in case you feel that you want to consider them when
restructuring the code; they are done anyway.

I am not going to touch FORD anymore for the time being.

In any case, it's nice that you published this code: it provides high
quality output, relies on a standardized syntax for comments and the
code looks much simpler compared to Doxygen. I hope the project can
keep evolving.

Here is the patch; as I said, I am only posting it since it is done
anyway, I don't mind if you ignore it.

I have tested it with the attached fortran code.

Index: reader.py
===================================================================
--- reader.py   (revision 35)
+++ reader.py   (working copy)
@@ -49,10 +49,14 @@
     def __init__(self,filename,docmark='!'):
         self.name = filename
         self.reader = open(filename,'r')
-        self.docbuffer = ""
+        self.docbuffer = []
         self.pending = []
         self.prevdoc = False
+        self.docmark = docmark
         self.doc_re = re.compile("^([^\"'!]|('[^']*')|(\"[^\"]*\"))*(!{}.*)$".format(docmark))
+        predocmark = '>'
+        self.predocmark = predocmark
+        self.predoc_re = re.compile("^([^\"'!]|('[^']*')|(\"[^\"]*\"))*(!{}.*)$".format(predocmark))

     def __iter__(self):
         return self
@@ -65,27 +69,46 @@
         if len(self.pending) != 0:
             self.prevdoc = False
             return self.pending.pop(0)
-        # If there was documentation at the end of the previous line, return
-        # it as the next line.
+        # If there are any documentation lines waiting to be returned, return them
+        # this can happen for online and anticipated docs
         elif len(self.docbuffer) != 0:
-            tmp = self.docbuffer
-            self.docbuffer = ""
             self.prevdoc = True
-            return tmp
+            return self.docbuffer.pop(0)

         # Loop through the source code until you have a complete line (including
-        # all line continuations)
+        # all line continuations), or a complete anticipated doc block
         done = False
         continued = False
+        reading_predoc = False
         linebuffer = ""
         while not done:
             line = self.reader.next()
             if len(line.strip()) > 0 and line.strip()[0] == '#': continue
+
+            # Capture any anticipated documenation comments
+            match = self.predoc_re.match(line)
+            if match:
+                # switch to predoc: following comment lines are predoc until the end of the block
+                reading_predoc = True
+                # substitute predocmark with docmark
+                tmp = match.group(4)
+                tmp = tmp[:1] + self.docmark + tmp[1+len(self.predocmark):]
+                self.docbuffer.append(tmp)
+                if len(line[0:match.start(4)].strip()) > 0:
+                    raise Exception("Anticipated documentation lines can not be online")
+                continue
+
             # Capture any documentation comments
             match = self.doc_re.match(line)
             if match:
-                self.docbuffer = match.group(4)
+                self.docbuffer.append(match.group(4))
                 line = line[0:match.start(4)]
+                if len(line.strip()) == 0 and reading_predoc:
+                    # complete doc comment lines belong to the predoc block
+                    continue
+
+            already_reading_postdoc = True
+
             # Remove any regular comments
             match = self.COM_RE.match(line)
             if match:
@@ -96,7 +119,7 @@
             if len(line) == 0:
                 if self.prevdoc and len(self.docbuffer) == 0:
                     #~ self.prevdoc = False
-                    self.docbuffer = "!!"
+                    self.docbuffer.append("!"+self.docmark)
             else:
                 # Check if line is immediate continuation of previous
                 if line[0] == '&':
@@ -109,14 +132,16 @@
                 # Check if line will be continued
                 if line[-1] == '&':
                     continued = True
-                    self.docbuffer = ""
                     line = line[0:-1]
                 else:
                     continued = False
-                # Add this line to the buffer then check whether we're done here
+            # Add this line to the buffer then check whether we're done here
             linebuffer += line
-            done = (len(self.docbuffer) > 0) or ((not continued) and 
-                   (len(linebuffer) > 0))
+            # Note: if we are here, the predoc block is completed
+            done = (not continued) and \
+                   ( (len(self.docbuffer) > 0) or (len(linebuffer) > 0) )
+            #done = (len(self.docbuffer) > 0) or ((not continued) and 
+            #       (len(linebuffer) > 0))

         # Split buffer with semicolons
         match = self.SC_RE.match(linebuffer)
@@ -143,9 +168,8 @@
             else:
                 return linebuffer
         else:
-            tmp = self.docbuffer
-            self.docbuffer = ""
-            if tmp != "!!":
+            tmp = self.docbuffer.pop(0)
+            if tmp != "!"+self.docmark:
                 self.prevdoc = True;
             return tmp

!> A simple test module.
module m
!! Some more details concerning the module `m`: set
!! $$
!!  x_i = 5
!! $$

implicit none

 type :: t_x
  !! Some documentation for the type `t_x`.
  !!
  !! Additional details concerning this type (FORD does not show
  !! this!).

  !> Here is the first field in `t_x`.
  integer :: i !! It is an integer counter;
  !! the field `i` can be useful when a counter is needed.
 end type t_x

contains

 !> A function to set \(x_i=5\)
 pure subroutine s(x,y)
  !> the function argument
  !! This argument is set to the proper value;
  type(t_x), intent(out) :: x !! in fact, it will be set to 5
  !! @note This might change in the future, see also Eq. (\ref{eq:fy}).
  !> Another argument
  !! with
  !! a
  !! rather
  !! long
  !! documentation
  real, & !! (further
    intent(in), & !! details
    target, optional & !! concerning
    !! the optional
    !! argument
    :: & !! `y`).
    y
  !! Still documenting `y`: this is the solution of
  !! \begin{equation}
  !!  f(y) = 0.
  !!  \label{eq:fy}
  !! \end{equation}

   x%i = 5
 end subroutine s

end module m

[cmacmackin]

No problem. And obviously, what with this being a GPL code, you're welcome to use your modified version yourself.

[cmacmackin]

I've concluded that I won't be changing the fundamentals of how reader.py works any time soon. There was some logic behind why I had the docbuffer and, while I'm still not convinced it's ideal, it would require too much work to refactor the code without it. I have, however, made a slight change to how lines with semicolons are split (it shouldn't effect any of your work). If you want to put together a pull request, I'll probably merge your changes in. Ideally it would be good if you could add the the information in init.py to look for a predocmark and specify a default value. It would also need to be passed to the constructor for the Reader object (preferably as a keyword argument with a default value, as was done with docmark). It would be great if you could add documentation for that option to the Project File Options page of the wiki, as well (just below the information about docmark).

However, I know I haven't left nearly as many comments in init.py, so I understand if you can't figure out how that works; it wouldn't be a big problem for me to make any necessary changes. reader.py was written early on when I was still vaguely sticking with my resolution to write detailed comments, which slipped as the volume of my code grew.