Design Description¶

Purpose¶

An implementation of the System Requirements is described in a short way with links to motivation further down in the document.

For an illustrative implementation of the following conventions see the end of dcx.py or the files generated by rstdcx --rest/--stpl smpl, and this documentation itself.

Documentation Format¶

dje:	s9v, san, stq, sed, sgt, s8c

Documentation files are text files.

dio:	dt7, dbz

Use pure RST as documentation format. Don’t use Sphinx extension.

dld:	doe, d03

Place a target ID before paragraphs, figures, list-tables, code listings, maths:

.. _`dx8`:

:dx8: <optionally key words here>

``:dx8:`` is a RST definition that will get special formatting in the generated document.
Else, ``dx8:`` would do, too.
Omitting it, if the ID is not wanted in the generated document.

|dx8| is an example ID.

.. _`dz3`:

.. figure:: _images/smpl.png
   :name:

   |dz3|: Caption here.

Reference via |dz3|.

.. _`dta`:

|dta|: Table legend

.. list-table::
   :name:
   :widths: 20 80
   :header-rows: 1

   * - Bit
     - ...

.. _`dyi`:

|dyi|: Listing showing struct.

.. code-block:: cpp
   :name:

   struct xxx{
      int yyy; //yyy for zzz
   }

.. _`d9x`:

.. math::
   :name:

If headers are referenced across files, add an ID before them, too:

.. _`d99`:

header
------

Inside the file, headers are automatically targets, by the definition of RST.

The ID¶

dfq:	san

The ID within a file

has only word characters: easily search word under cursor in editor
is short: easy to memorize
is lowercase: some editors are case insensitive and it is easier to write in lowercase
is random, not ordered: no problem when reordering documentation parts
is not derived from the topic of the paragraph, which is often not well defined and would make the ID long

All IDs that end up in a top level file should start with the same letter. This way one can immediately tell which top level file the ID is from. dcx.links_and_tags uses this to color a graph of all the ID dependencies.

Files¶

d75:	scs, sim, seo, sil, dt7

Top level files use the extension .rest.

The last line of a top level .rest file must be the following (d1z):

``.. include:: _links_sphinx.rst``

For image substitutions to work place the .. |xxx| image:: _images/xxx.jpg into the top level .rest file.

Sphinx uses an index.rest that become the entry point to the project site. It includes all top level .rest files for a project-wide table of contents:

.. toctree::
   readme.rest
   ra.rest
   sr.rest
   dd.rest
   tp.rest
   rstdoc.rest

The names of the files are an example. See Overview for their purpose.

dfy:	dt7

Included files use the extension .rst. Include them with:

.. include:: somefile.rst

d0t:	s9v, s8c

Images (.png, .jpg, .svg,…) are in the _images, or ..\_images folder.

Images can be generated from tikz files (.tikz) or templates of it (.tikz.stpl).

dyn:	s8c, dv6

.rest, .rst, .tikz files can be SimpleTemplate templates.

Then they have the additional extension

.stpl, if they can be expanded by themselves
.tpl, if they are only used by other template files (%include('some.rst.tpl')). .tpl files can have parameters, which are provided via %include('some.rst.tpl',aparam="test") .tpl files can be in the same or the parent folder, without the need for a path in %include().

In this documentation ra.rest.stpl includes utility.rst.tpl.

Tools¶

dru:	scf

The tools are all open source and community driven.

dmm:	scf

Pandoc is used to convert to HTML, PDF, DOCX, ODT

dsn:	Editor, san

A text editor is used for writing. It should support CTags’s .tags files

d13:	sgt, s8c

Python is used

for scripting and (da0)
for templates (dv6)

d23:	sgt

Data is preferably written directly in Python. Any text format that is readable in Python works, too. Very table-like data is written in yaml or json (dg8).

dwm:	doe, san, sgt, s8c

rstdoc’s rstdcx is used to

create .tags and _links_xxx.rst files to support hypertext (d03)
generated files from source code using the gen file (dhy)
expand template files .stpl (dv6)
convert .tikz files to .png and place into ./_images or ../_images

dqf:	sxr, scf, sgt, s8c

As build system waf is used (dw8).

d7o:	Sphinx, Project Site, scf

Optionally Sphinx can be used to create a central HTML site for the project with links to all the top level files (.rest) (d1w)

df3:	Latex, scf

LaTex to

convert RST directly to PDFs or
create graphics using tikz

Motivation¶

Light Markup¶

dt7:	light markup

LaTex and other document markup languages are not easy to learn (s45) and have not enough constraints.

The alternatives are light markup formats:

sgt, s8c: It is easier to generate parts of the documentation with scripts from source code or source code comments. It allows mixing source code with documentation for better cohesion and less redundancy.
s45: It can be easily learned, because it restricts itself to essential elements.
s45: The elements are of conceptual nature (header, list item, ) not actual formatting. The formatting is done when creating the final document. This makes it easier to keep a consistent formatting when more people work on the documentation.
s9v: As text, it is perfect for version control systems. One can commit documentation changes together with the according source code changes. It allows to keep outdated information hidden in the VCS history and not lying around and messing up. It is easy to review documentation changes. To make it even more easy one should try to have
- one sentence per line
- one clause per line or
- one list item per line
s0t: It is easier to extract, which items link to which other ones, especially if the team agrees on facilitating conventions.
san, stq, sed: It can be edited with a text editor, i.e. the same tool developers work with all the time.
stq: It is accessible to grep.
san: Ctags can be used to jump around while editing.
sed: It is very readable.
scf: It can be translated to several final formats, e.g.
- HTML (Pandoc, Sphinx)
- PDF (Pandoc, Sphinx)
- ODT, DOCX (Pandoc): Sphinx extension to RST cannot be converted to these formats.
Sphinx converts all files with an extension provided in conf.py. .rest is chosen for such main files. .rst is then for included files.

RST¶

dbz:	RST

There are many light markup formats. But especially restructuredText (RST)

s45: has rich format support (e.g. table formats)
s45: is extensible
sgt, s8c: is best for python scripting
scf: has a very good tooling support
- Pandoc
- Sphinx
- Docutils
- rstdoc

http://rst.ninjs.org can be used to play with RST.

For the conversion from RST to DOCX currently the best tool is Pandoc, Pandoc only takes pure RST and does not know about the Sphinx extensions like :ref:.

Hypertext¶

doe:	hypertext in text, r33

rstdoc’s rstdcx generates a .tags file for target IDs CTags’s .tags files are supported by many editors. With .tags files your editor can jump around in RST files, as if marked up hypertext (HTML).

d03:	hypertext in HTML, PDF and DOCX, r33

To reference paragraphs, figures, mathematics and tables use RST’s replacement substitutions:

This is text that references |dx8|.

As the RST’s link format differs between HTML, PDF and DOCX, rstdoc’s rstdcx generates separate files with definitions for the replacement substitutions:

_links_docx.rst:

.. |targetid| replace:: `targetid <file.docx#targetid>`_

_links_pdf.rst:

.. |targetid| replace:: `targetid <file.pdf#targetid>`_

_links_sphinx.rst:

.. |targetid| replace:: :ref:`targetid <file.html#targetid>`

d1z:	Pandoc bug

Substitutions cannot be in included files, until the Pandoc include bug is corrected. For the _links_docx.txt this helps:

cat file.rst _links_docx.txt | sed -e's/.. include:: _links_sphinx.txt//g' | pandoc -f rst -t docx -o file.docx

The last line of a top level .rest file must be:

``.. include:: _links_sphinx.rst``

For image substitutions to work place the .. |xxx| image:: xxx.jpg into the top level .rest file.

Editor¶

If most work is done on text, it becomes important to have a very flexible and powerful editor. One should invest a some time to know the editor well.

There are a lot of editors that work well with RST, e.g. Emacs.

Vim¶

vim_py3_rst
vim-table-mode
riv.vim

Atom¶

atom-ctags       #better: https://github.com/rpuntaie/atom-ctags
language-restructuredtext
rst-preview-pandoc
table-editor
rst-snippets
atom-build       #better: https://github.com/rpuntaie/atom-build
atom-build-waf
find-and-replace-under-cursor

atom-build and atom-ctags were modified to allow finding files by putting the relevant subdirectory into Atom’s project paths.

Scripting¶

da0:	scripting

Python is a good choice as a scripting language, because it

is easy
is powerful
has many libraries
has a huge community

Build System¶

dw8:	waf

waf (dqf) is a good choice as build system:

It is python
It is made part of the project, i.e. not an external dependency
It supports many computer languages
rstdoc’s dcx.py is a waf plugin

As a general automation tool to doit would be a good choice, but it does not provide abstraction for compiler handling.

make works, too, but it is less flexible.

Generated documentation¶

dhy:	gen

To generate documentation files from source code, rstdoc’s rstdcx

Looks into a gen file, to see which source files lead to which target file. The gen file can be
- python code defining from_to_fun_kw = [[fromfile,tofile,fun,kw],...] or
- from|to|fun|kwargs lines
fun means gen_fun() (or just gen() if empty) that can be found commented #gen_fun in fromfile.
Gets the # def gen_fun(lns,**kw) python function. fun comes from the gen file, the rest must match exactly. # def gen_fun marks the end of the function.
Executes the def gen_fun(lns,**kw) function with the lines of the source file and the kw from the gen file
Saves the result in the target files

In waf’s wscript_build, call gen_files() to initiate interpretation of the gen file.

See the gen file of this documentation as an example.

Templating¶

dv6:	templating text

Python has many template libraries. An important one is Jinja2. But they are targeted especially to HTML and consider aspects (e.g. security), that are not of relevance for technical documentation.

bottle’s SimpleTemplate is inverted Python (text un-enclosed, code enclosed) without any further restrictions to the Python code.

It is easy to learn
Very powerful

dpv:	templating tikz

One can use it to replace native control structures.

This smpl.tikz:

[thick]
\draw (0,0) grid (3,3);
\foreach \c in {(0,0), (1,0), (2,0), (2,1), (1,2)}
    \fill \c + (0.5,0.5) circle (0.42);

can become smpl.tikz.stpl:

[thick]
\draw (0,0) grid (3,3);
%for cx,cy in {(0,0), (1,0), (2,0), (2,1), (1,2)}:
    \fill ({{cx+0.5}},{{cy+0.5}}) circle (0.42);
%end

dhl:	defines for code and documentation

One can have

limits defined in a python file (specifications.py) and use them in
a code file template (e.g. specifications.h.stpl)
and in a documentation file template (e.g. specifications.rest.stpl)

Alternatively one can have

a code file with limits (e.g. specifications.h)
parse that code file in the python code and use the values in the text of a documentation file template (all in e.g. specifications.rest.stpl)

drz:	mathematics

One can use

python mathematics (e.g. sympy)
use it for actual calculations
expand the formulas in the text
use the calculated values in the text

%from sympy.abc import *
%from sympy import Eq, latex
%pythagoras=Eq(c**2,a**2+b**2)
Pythagorean theorem

.. math:

    {{latex(pythagoras)}}

is very important.

becomes:

Pythagorean theorem

.. math:

    c^{2} = a^{2} + b^{2}

is very important.

Data¶

dg8:	data

Data needs to be text, because then changes can be traced via VCS.

Since the scripting and templating is done with Python, data written in Python is most easily accessible, e.g. via and import.

If the data is more table-like:

yaml is good for direct editing, because it is

very non-verbose and
almost can be used directly as RST
changes can be best followed without too much syntax

json is appropriate, if the data

is generated
needs to be read and possibly edited by humans
needs to be rendered on HTML via javascript

XML is only appropriate for a predefined schema, that is unlikely to change throughout the development time.

There are alternatives for XML’s XPath in yaml and json:

Python and yaml: dpath
json: jsonpath

Project Site¶

d1w:	project site

A central HTML project site can be a central point of team coordination. One can have

an issue file
minutes file for meetings
a project coordination file with plans, deadlines
a progress file per developer
…

This keeps all the data together in

one format and
one tool chain
one repository