\input texinfo
@comment %**start of header
@setfilename pokeint.info
@include versionint.texi
@settitle GNU poke Internals and Development Guide
@afourpaper
@defcodeindex pv
@comment %**end of header

@copying
This manual describes the internals of GNU poke (version @value{VERSION},
@value{UPDATED}).

Copyright @copyright{} 2024, 2025 The poke authors.

@quotation
You can redistribute it and/or modify this manual under the terms of
the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any
later version.
@end quotation
@end copying

@dircategory GNU poke
@direntry
* pokeint: (poke). Internals of the GNU poke binary editor.
@end direntry

@titlepage
@title GNU poke Internals and Development Guide
@subtitle for version @value{VERSION}, @value{UPDATED}
@author by Jose E. Marchesi et al.
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@ifnottex
@node Top
@top GNU poke Internals

@insertcopying
@end ifnottex

GNU poke is an interactive, extensible editor for binary data.  Not
limited to editing basic entities such as bits and bytes, it provides
a full-fledged procedural, interactive programming language designed
to describe data structures and to operate on them.

This manual is mainly a reference manual rather than a tutorial.  It
discusses how to contribute to GNU poke, the source tree structure and
build system, the overall architecture and a detailed description of
the internals of the program.  The target audience for this manual are
GNU poke developers, testers, and people interested in extending or
customizing the program to their needs.

@menu
* Nomenclature::

Contributing
* Development environment::
* Version control system::
* Running an uninstalled poke::
* Picking something to work on::
* Submitting a patch::
* Notifying about pushed commits::

Building poke
* Building basics::
* Partial builds::
* Building after a git pull::
* Building with local changes in gnulib or Jitter::
* Updating gettext::

Testing poke
* Writing tests::
* Continuous Integration::

Fuzzing poke
* Grammarinator::

Debugging poke
* Building with debugging support::
* Invoking GDB to debug an uninstalled poke::
* Debugging poke with GDB::
* Valgrind and poke::
* Debugging PVM assembly code::

Profiling poke
* Building poke with profiling support::

Hacking poke
* Maintenance targets::
* Writing compiler diagnostics::

Coding Style and Conventions
* Writing C::
* Writing Poke::
* Writing RAS::

Writing Documentation
* Documenting pickles::

Internals
* The Poke compiler::
* The Poke Virtual Machine::
* Memory management::

Indices
* Concept Index::               General concept index
@end menu

@node Nomenclature
@chapter Nomenclature

We call @dfn{poke} the program.  When the context may induce confusion
(since ``poke'' is a pretty common word) then we use @dfn{GNU poke}.

@dfn{Poke} (with upper case P) is the name of the domain-specific
language implemented by poke, the program.

A @dfn{pickle} is a Poke source file containing definitions of types,
variables, functions, @i{etc}, that conceptually apply to some definite
domain.  For example, @file{elf.pk} is a pickle that provides
facilities to poke ELF object files.  Pickles are not necessarily
related to file formats: a set of functions to work with bit patterns,
for example, could be implemented in a pickle @file{bitpatterns.pk}.

@node Development environment
@chapter Development environment

The packages listed in the file @file{DEPENDENCIES} are needed.
Additionally:

@menu
* Autotools::
* Dejagnu::
* Flex::
* Bison::
* GNU Jitter::
@end menu

@node Autotools
@section Autotools

This distribution uses whatever versions of Automake, Autoconf, and
Gettext are listed in NEWS; usually the latest ones released.  If you
are getting the sources from git (or change configure.ac), you'll need
to have these tools installed to (re)build.  You'll also need
help2man.  All of these programs are available from
@url{ftp://ftp.gnu.org/gnu}.

@node Dejagnu
@section Dejagnu

The poke testsuite uses DejaGNU.  Please install it if you intend to
run the tests.  If you want to hack poke, you definitely want to run
the tests :)

@node Flex
@section Flex

You will need a recent version of flex, since we are using some recent
options like @code{reentrant} or @code{bison-bridge}.  flex version
2.6.1 works fine.

@node Bison
@section Bison

In order to improve the user's experience, we are using fairly recent
versions of bison.  Bison 3.6.0 or later is required to bootstrap and
build poke from git.

@node GNU Jitter
@section GNU Jitter

In order to build and run poke, you need Luca Saiu's GNU Jitter.
Jitter's git source repository, publicly readable over HTTP, is at
@url{http://git.ageinghacker.net/jitter} and its home page is
@url{https://www.gnu.org/software/jitter/}.

The appropriate version of Jitter is downloaded and bootstrapped
automatically by Poke's @file{bootstrap} script, which frees the user
from the annoyance of installing Jitter as a dependency.

Configuring and compiling Poke will also compile and configure Jitter
in a subdirectory.  Jitter, when configured in @dfn{sub-package mode}
as Poke does, only generates static libraries and requires no
installation.

@node Version control system
@chapter Version control system

@menu
* Git repos::
* Personal branches::
@end menu

@node Git repos
@section Git repos

This section lists the different repositories used in the development
of GNU poke.  The @code{https} handlers provide read-only.  The ssh
handlers are for write access, where @var{username} is the handler of
your user in @url{https://savannah.gnu.org}.

@table @url
@item https://https.git.savannah.gnu.org/git/poke.git
@itemx @var{username}@@git.savannah.gnu.org/srv/git/poke.git
Main repository, containing the source code of libpoke, the poke
application, documentation, poked, pokefmt, and testsuite.  The GNU
poke source distribution is compiled from this repo.

@item https://https.git.savannah.gnu.org/git/poke/poke-mode.git
@itemx @var{username}@@git.savannah.gnu.org/srv/git/poke/poke-mode.git
Repository for @code{poke-mode}, an Emacs major mode for editing Poke
source code.  The poke-mode ELPA package is built from this repo.

@item https://https.git.savannah.gnu.org/git/poke/poke-el.git
@itemx @var{username}@@git.savannah.gnu.org/srv/git/poke/poke-el.git
Repository for @code{poke.el}, an Emacs interface to GNU poke.  The
poke ELPA package is built from this repo.

@item https://https.git.savannah.gnu.org/git/poke/poke/poke-elf.git
@itemx @var{username}@@git.savannah.gnu.org/srv/git/poke/poke-elf.git
Repository for the poke ELF pickles
project@footnote{https://jemarch.net/poke-elf}.

@item https://https.git.savannah.gnu.org/git/poke/poke/poke-dwarf.git
@itemx @var{username}@@git.savannah.gnu.org/srv/git/poke/poke-dwarf.git
Repository for the poke DWARF pickles
project@footnote{https://jemarch.net/poke-dwarf}.
@end table

@node Personal branches
@section Personal branches

Anyone having write access to the git repository is allowed to push
and maintain personal branches.  These branches should be called
@code{WHO/WHAT}, where @code{WHO} is the nick identifying the owner of
the branch and @code{WHAT} a description of what it contains.

Example:

@example
jemarch/hyperlinks-server
@end example

Personal branches are intended to ease the interaction between
developers, and to provide a convenient basis for testing large
changes.

Personal branches can be rebased, and deleted.  Please do not write
into a personal branch unless you have the explicit approval of the
branch owner.

@node Running an uninstalled poke
@chapter Running an uninstalled poke

Once poke is compiled, you can run it before installing by using the
@command{run} script built into the root of your build tree; this
takes care of setting variables such as @code{PATH},
@code{POKEDATADIR}, @code{POKESTYLESDIR}, and @code{POKEPICKLESDIR} to
point at appropriate locations.

For example:

@example
$ pwd
/home/jemarch/gnu/hacks/poke/build/
$ ./run poke
@end example

@node Picking something to work on
@chapter Picking something to work on

We maintain a categorized and quite big list of tasks and ideas in the
@file{TODO} file which is located at the top level of the sources
distribution.  Some of the tasks listed there are simple, well
defined, and straightforward: these are ideal for first contributions.
Other entries, however, describe full-fledged projects or even half
baked ideas that are only sketched barely.  These later will require
additional discussion in the mailing list.

In any case, if you start working on some project, be it listed in
@file{TODO} or not, we highly recommend you let us know in the
poke-devel mailing list.

If you would prefer to work on bug fixing, you can browse the open
bugs at
@url{https://sourceware.org/bugzilla/describecomponents.cgi?product=poke}.

@node Submitting a patch
@chapter Submitting a patch

If you hack a feature/improvement/bugfix for poke and want to get it
integrated upstream, please keep the following points in mind:

@itemize @minus
@item If your patch changes the user-visible characteristics of poke,
please include an update for the user manual.
@item If your patch adds or changes the way poke works internally,
in a significant way, please consider including an update for the @file{HACKING} file.
@item Please include a GNU-style ChangeLog in the patch description, but
do not include it in the thunks.  This is to ease reviewers to apply
your patch for testing.  Of course, include the thunk in the final
push!  (We will get rid of manual ChangeLog entries soon.)
@item Make sure to run =make syntax-check= before submitting the patch,
and fix any reported problem.  Note that the maintainer reviewing your
patch will also do this, so this is a great time to save an iteration
;)
@item  Let's keep poke.git master linear... no merges please.  Pull with
@command{--ff-only}.
@item Send the patch to the @url{poke-devel@@gnu.org} mailing list.
@item Use text email only.  No html please.
@item Inline the patch in the body of your email, or alternatively
attach it as @code{text/x-diff} or @code{text/x-patch}.  This is
to ease reviewers to quote parts of the patch.
@end itemize

@node Notifying about pushed commits
@chapter Notifying about pushed commits

Once you push one or more commits to some branch in the public git
repo, please send each patch as an email to the
@email{poke-devel@@gnu.org} mailing list with a subject line like:

@example
[COMMITTED @var{branch}] Summary of the change
@end example

@noindent
where @var{branch} is the name of the branch to which you pushed the
associated commit.  This applies to both patches applies as obvious
and to commits pushed by maintainers.

This will help the other hackers subscribed to the mailing list to
follow the on-going developments, and even to catch some problem in
any of the pushed commits.

@node Building basics
@chapter Building basics

After getting the git sources, and installing the tools above, you can
run:

@example
$ ./bootstrap --skip-po
@end example

Then, you can run @command{configure}, as described in the files
@file{INSTALL} and @file{INSTALL.generic}.  For example:

@example
$ mkdir build/ && cd build
$ ../configure
@end example

@noindent
Finally:

@example
$ make
$ make check
@end example

@node Partial builds
@chapter Partial builds

The poke source distribution encompasses several components: libpoke,
the poke application, the poked daemon, @i{etc}.  By default all the
components will be built provided their requirements are found at
configure time, but it is also possible to explicitly disable them
when invoking configure.

For example, this is how you would build a poke without
@command{poked} and without @command{pokefmt}:

@example
$ ../configure --disable-poked --disable-pokefmt
@end example

Note however that if some of the components are disabled, the
resulting tree cannot be used to run @command{make dist} on it.  It
will fail.

@node Building after a git pull
@chapter Building after a git pull

Usually issuing a @command{make} at the top build directory is enough
to rebuild poke after changes have been fetched from the source repo.
The Automake maintainer-mode rules will make sure that autoconf and
friends are run again and do the right thing.

However, when the Jitter submodule is updated, you have to first
bootstrap:

@example
$ ./bootstrap
@end example

Then go to the top build directory, and issue a configure explicitly
before rebuilding:

@example
$ cd build && ../configure
$ make
@end example

You can't rely on the maintainer-mode rules there, because of certain
problem with the sub-package mode of Jitter.

@node Building with local changes in gnulib or Jitter
@chapter Building with local changes in gnulib or Jitter

If you have local changes to gnulib or jitter, that are not yet
registered as submodule reference commits, the way to invoke bootstrap
such that it doesn't delete your changes is:

@example
$ ./bootstrap --skip-po --no-git --gnulib-srcdir=gnulib --jitter-srcdir=jitter
@end example

@node Updating gettext
@chapter Updating gettext

When updating gettext, besides the normal installation on the system,
it is necessary to run @command{gettextize -f} in this hierarchy to
update the @file{po/} infrastructure.  After doing so, rerun
@command{gnulib-tool --import} since otherwise older files will have
been imported.  See the Gnulib manual for more information.

@node Writing tests
@chapter Writing tests

The poke testsuites live in the @file{testsuite/} subdirectory.  This
section contains useful hints for adding tests there.

@menu
* Test framework::
* Run selected tests::
* Unit tests::
* Naming tests::
* Always set obase::
* Put each test in its own file::
* List the test files in @file{testsuite/Makefile.am}::
* dg-output may require a newline::
* Using data files in tests::
* Using NBD connections in tests::
* Writing tests that depend on a certain capability::
* Writing REPL tests::
* Command REPL tests::
* General REPL tests::
* Testing pickles::
@end menu

@node Test framework
@section Test framework

The test suite is based on DejaGnu, for which you find the general
documentation at
@url{https://www.gnu.org/software/dejagnu/manual/index.html}.

The documentation of specific DejaGnu directives is at
@url{https://gcc.gnu.org/onlinedocs/gccint/Directives.html} and
@url{https://gcc.gnu.org/wiki/HowToPrepareATestcase}.

@node Run selected tests
@section Run selected tests

Tests are categorized in different sub-directories. You can run
run tests of a single category by running

@example
make check RUNTESTFLAGS=CATEGORY.exp
@end example

For example, to run tests of =pkl= in =poke.pkl= sub-directory, run:

@example
make check RUNTESTFLAGS=pkl.exp
@end example

You can also select tests using @code{TESTS} environment variable.  As
an example, to run all test files which start with @file{array}, you
can run:

@example
make check TESTS=array*.pk
@end example

@node Unit tests
@section Unit tests

We are using DejaGnu's support for unit tests.  See
@file{testsuite/poke.libpoke} for an example on how to write C unit
tests.

@node Naming tests
@section Naming tests

For testing a functionality =foo=, name your test =foo.pk= or
@file{foo-@var{N}.pk} where @var{N} is a number.

If the test is a =do-compile= whose compilation is expected to fail,
name the test @file{func-diag.pk} or @file{func-diag-N.pk}.  Here
``diag'' means diagnostic.

@node Always set obase
@section Always set obase

If your test relies on printing integer values in the REPL (or using
the @code{%v} formatting tag in a @code{printf}) please make sure to
set an explicit output numerical base, like in:

@example
/* @{ dg-command @{.set obase 10@} @}  */
@end example

This way, we won't have to change the tests if at some point we change
the default obase.

@node Put each test in its own file
@section Put each test in its own file

If you are writing tests for a specific functionality, like for
example a standard function @code{foo}, it may seem logical to put all
the tests in a single file @file{foo.pk} like:

@example
/* @{ dg-do run @} */

/* @{ dg-command @{foo (1)@} @} */
/* @{ dg-output "expected result" @} */

/* @{ dg-command @{foo (1)@} @} */
/* @{ dg-output "\nexpected result" @} */

[... and so on ...]
@end example

However, this is not a good idea.  If some of the "subtests" fail, it
becomes difficult to determine which one is the culprit looking at the
test log file.

It is better to put each test in its own file: @file{foo-1.pk}, @file{foo-2.pk}
and so on.

@node List the test files in @file{testsuite/Makefile.am}
@section List the test files in @file{testsuite/Makefile.am}

So that the tests get distributed, you need to list each newly added
test file in @file{testsuite/Makefile.am}.

We have a syntax-check target that checks that @code{EXTRA_DIST}
contains all the test files under @file{testsuite/}, and that there is
no test listed in @code{EXTRA_DIST} that doesn't exist in the file
system.

@node dg-output may require a newline
@section dg-output may require a newline

If despite the advise above you really need to put more than dg-output
in a dg-run test file, please be aware you need to prefix all of them
(but the first one) with a newline, like in:

@example
/* @{ dg-output "foo" @} */
/* @{ dg-output "\nbar" @} */
/* @{ dg-output "\n baz" @} */
@end example

@node Using data files in tests
@section Using data files in tests

If your test requires mapping data, the dg-data directive is what you
need.  It has two forms.

The first form looks like:

@example
/* @{ dg-data @{c*@} @{0x10 0x20 0x30 0x40 ...@} @} */
@end example

This creates a temporary file (whose name is irrelevant) and makes it
the current IO space at poke initialization time.  The test body, and
dg-commands, can then map on it.

However, sometimes you need to map on several files.  In this case,
you can have several dg-data entries with an additional argument,
which is the name of the temporary file to create:

@example
/* @{ dg-data @{c*@} @{0x01 0x02 ...@} foo.data @} */
/* @{ dg-data @{c*@} @{0x11 0x12 ...@} bar.data @} */

[...]

/* @{ dg-command @{ var foo = open ("foo.data") @} @} */
@end example

The file created by the last dg-data (be it anonymous or named) is the
current IO space.

@node Using NBD connections in tests
@section Using NBD connections in tests

If your test requires an NBD server (only useful when poke is compiled
with libnbd), the dg-nbd directive is what you need.  It has one form:

@example
/* @{ dg-nbd @{ 0x1 0x2 ...@} [dg-tmpdir]/sock @} */
@end example

This utilizes nbdkit to serve an in-memory disk with initial contents
over a named Unix socket.  Note that the data argument is not the same
as for dg-data (which uses tcl's binary command), but rather the
syntax accepted by nbdkit-data-plugin's data= argument, although the
two are similar.  nbdkit then creates a Unix socket server for the
data, and will be shut down gracefully when the testsuite completes.
Use of the utility directive [dg-tmpdir] ensures that the socket name
will not be too long while still respecting @code{$TMPDIR} (defaulting
to a new subdirectory of @file{/tmp}), since @code{$objdir} may be
arbitrarily deep.

To use the server as an IO space, your test will then follow up with:

@example
/* @{ dg-command "open (\"nbd+unix:///?socket=[dg-tmpdir]/sock\")" @} */
@end example

@node Writing tests that depend on a certain capability
@section Writing tests that depend on a certain capability

Sometimes the presence of the functionality tested may be optional.
An example is testing the support for styling, which depends on poke
to be built with libtextstyle support.

The dg-require directive can be used for this purpose.  For example,
the following test will be skipped (with status UNSUPPORTED) if the
capability @code{libtextstyle} is not found in poke:

@example
/* @{ dg-do run @} */
/* @{ dg-options "--color=html" @} */
/* @{ dg-require libtextstyle @} */

/* @{ dg-command @{printf "%<foo:%i32d%>", 10@} @} */
/* @{ dg-output "<span class=\"foo\">10</span>" @} */
@end example

IMPORTANT NOTE: dg-require should appear before any dg-command
directive in the test file.

The supported capabilities are:

@table @code
@item libtextstyle
poke is built with libtextstyle support.
@item nbd
poke is built with NBD io space support, and dg-nbd works.
@end table

@node Writing REPL tests
@section Writing REPL tests

The @file{poke.repl} testsuite is intended to test features in the
interactive usage of poke.  Therefore, it is not dg-based.  Instead,
it uses the services provided by @file{testsuite/lib/poke.exp}.  In a
nutshell, these services are:

@table @code
@item poke_start
Run a new poke process and wait at the prompt.
@item poke_exit
Exit poke.
@item poke_test_cmd @var{cmd} @var{pattern}
Send @var{cmd} to poke, and expect the result @var{pattern}.
@var{cmd} is sent as virtual keystrokes.  Therefore, sending @code{\t}
or @code{\n} has the same effect on the REPL than typing @key{TAB} or
@key{RET} in the keyboard, respectively.  Note that @var{pattern}
shouldn't include the prompt.
@item poke_send @var{input} @var{pattern}
Send @var{input} to poke, and expect @var{pattern} as output.
@item poke_send_signal @var{signal}
Send @var{signal} to poke.
@end table

Adding a new test to @file{poke.repl} involves editing
@file{poke.repl/repl.exp} and adding some content there.  The
following subsections detail how.

@node Command REPL tests
@section Command REPL tests

Some REPL tests need to check whether poke replies properly to some
sent command.  The @file{poke_test_cmd} procedure defined in
@file{testsuite/lib/poke.exp} provides a convenient interface for
this.

For example:

@example
set test "slashes are preserved"
poke_start
poke_test_cmd @{4 / 2@} @{2@}
poke_exit
@end example

The snippet above implements a test named "slashes are preserved" that
runs poke and sends the command @code{4 / 2} with expected result
@code{2}.

The dialogue when the test above is executed is:

@example
(poke) 4 / 2
2
(poke)
@end example

@node General REPL tests
@section General REPL tests

Other REPL tests are not about executing commands.  Suppose for
example we want to test whether tab-completion works.  We would write
something like:

@example
set test "tab-completion-2"
poke_start
poke_send ".e\t\t" "\r\n.editor  .exit *\r\n$poke_prompt .e"
poke_exit
@end example

The test above uses the @code{poke_send} procedure, defined in
@file{testsuite/lib/poke.exp}.  This procedure gets two arguments: the
input that is sent to poke, and the expected output.  Note how using
@code{poke_send} doesn't require poke to execute any command.

Using @code{poke_send} is more laborious than using
@code{poke_test_cmd}: it is necessary to explicitly include the prompt
in the expected output whenever needed.

Note also how newlines are perceived by expect as the sequence
@code{\r\n}.

@node Testing pickles
@section Testing pickles

Each pickle in @file{pickles/@var{foo}.pk} shall have a test file
@file{testsuite/poke.pickles/@var{foo}-test.pk} written using pickle
pktest.

If there are some features in the pickle that cannot be tested using
pktest (e.g., pickle uses @code{print}), that pickle also shall have a
testsuite in @file{testsuite/poke.@var{foo}} with a driver
@file{testsuite/poke.FOO/FOO.exp}.

@node Continuous Integration
@chapter Continuous Integration

The package is built automatically, at regular intervals.  You find
the latest build results at
@url{https://gitlab.com/gnu-poke/ci-distcheck/pipelines}.

We also have a set of "buildbots" hosted at Sourceware.  See
@url{https://builder.sourceware.org/}.  The sources for this are in
the Sourceware @file{builder.git} repository at
@url{https://sourceware.org/git/?p=builder.git}. The @file{README}
file contains a good reference of how to hack on the build machinery.
The poke homepage contains little graphics indicating the last
execution results of these buildbots.

Parts relevant to poke are named @code{/.*gnupoke.*/} in the
Sourceware buildbot configuration.  Please see the repository linked
above for details.

@node Grammarinator
@chapter Grammarinator

@dfn{grammarinator} is a random test generator/fuzzer that creates
test cases according to an input antlr4 grammar.

In order to generate Poke programs with grammarinator, follow the
following steps.

First of all, install grammarinator:

@example
$ git clone https://github.com/renatahodovan/grammarinator
$ cd grammarinator
$ sudo python3 setup.py install
@end example

Then, use @command{grammarinator-process} in order to create the ``unparser''
and ``unlexer'' for poke:

@example
$ grammarinator-process ../path/to/poke.g4
@end example

This will create two files in the current working directory:
@file{pokeUnlexer.py} and @file{pokeUnparser.py}.  Next step is to use
@command{grammarinator-generate} in order to create tests:

@example
$ grammarinator-generate -l pokeUnlexer.py -p pokeUnparser.py \
                         -r program -n 100 -d 10 \
                         -t grammarinator.runtime.simple_space_transformer
@end example

The option @command{-n} specifies the number of tests (Poke programs)
to generate.

The option @command{-d} specifies the maximum recursive level.  The
bigger the recursive level, the bigger the test files will be.  If you
don't specify a maximum recursive level then
@command{grammarinator-generate} will crash.

The option @command{-t grammarinator.runtime.simple_space_transformer}
makes sure that whitespaces get inserted between literal tokens.

@node Building with debugging support
@chapter Building with debugging support

Short summary: at the present time Poke and its Jittery VM are not
especially difficult to debug; however the situation is going to
change as Jitter evolves and the following details will become more
important to Poke hackers.

In order to make debugging easier you may want to disable advanced
dispatches in Jitter, which make the generated code harder to follow
and confuse GDB.

Jitter by default will use the most efficient dispatch which is both
stable and available for the current configuration; the most efficient
dispatch is no-threading, which requires assembly support in Jitter
for the specific architecture and a supported binary format.
minimal-threading is more portable than no-threading, even if less
efficient.  Jitter supports two alternative dispatches, slower but
very stable, portable and friendly to debugging: @dfn{switch} and
direct-threading.

Any dispatch can be selectively enabled or disabled from the Poke
@file{configure} script, by passing the following options which will
be automatically relayed to Jitter's @file{configure} script:

@table @command
@item --enable-dispatch-switch
@item --disable-dispatch-switch
@item --enable-dispatch-direct-threading
@item --disable-dispatch-direct-threading
@item --enable-dispatch-minimal-threading
@item --disable-dispatch-minimal-threading
@item --enable-dispatch-no-threading
@item --disable-dispatch-no-threading
@end table

When configured in sub-package mode, as is the case when used with
Poke, Jitter will only actually compile the single most efficient
enabled dispatch.  This means that in order to select some sub-optimal
dispatch method, one has to disable the more efficient ones.  For
example, this is the best options for debugging:

@example
$ ../configure [...] --disable-dispatch-no-threading \
                     --disable-dispatch-minimal-threading \
                     --disable-dispatch-direct-threading
@end example

Other than the configure options above, you will probably want to
configure the build to generate debugging information, and disable
compiler optimizations, using something like:

@example
../configure CFLAGS="-O0 -g3" [...]
@end example

Note that @command{-g3} is particularly useful to debug the Poke
compiler, because it allows to refer by cpp macros by name in GDB and
the compiler uses macros very extensively.

@node Invoking GDB to debug an uninstalled poke
@chapter Invoking GDB to debug an uninstalled poke

Since we are using libtool, @file{$(builddir)/poke/poke} is a shell
script.  In order to debug the uninstalled poke, you can do:

@example
$ ./libtool --mode=execute ./run gdb poke/poke
@end example

@node Debugging poke with GDB
@chapter Debugging poke with GDB

Once GDB is invoked in either an installed or uninstalled poke, you
can debug it normally.

Something to keep in mind, however, is that the Boehm GC we use uses
SIGPWR and SIGXCPU for internal purposes, in order to stop all threads
before starting collecting memory.  Proper handlers are installed by
the GC for these signals.  However, when you run poke under GDB, GDB
will stop once SIGPWR or SIGXCPU are signaled to the current thread.

You can teach GDB to ignore these signals by issuing this command:

@example
handle SIGPWR noprint pass
handle SIGXCPU noprint pass
@end example

Alternatively, you can just @command{(c)ontinue}.

@node Valgrind and poke
@chapter Valgrind and poke

The PVM uses the Boehm conservative garbage collector in order to
manage the memory used by the VM values.  Other parts of poke, such as
the PKL compiler, manage their own memory.

Valgrind gets easily confused by the GCs tampering with the stack, and
emits a lot of spurious warnings.  Fortunately it is possible to tell
memcheck to omit these warnings: the file
@file{etc/boehm-gc.suppressions} contains a list of suppressions.

Invoke valgrind with
@command{--suppressions=etc/boehm-gc.suppressions}.

If despite using the suppressions file you see some spurious warning,
please use:

@example
$ valgrind --tool=memcheck --gen-suppressions=all --log-file=raw.log
@end example

Then process @file{raw.log} with the
@file{etc/parse-valgrind-suppressions.sh} script, wildcard the result
as much as you can, and append the results to
@file{etc/boehm-gc.suppressions}.

In order to run the testsuite with valgrind, edit
@file{lib/poke-dg.exp} and uncomment the lines:

@example
set VALGRIND "valgrind --quiet \
   --suppressions=$@{srcdir@}/../etc/boehm-gc.suppressions \
   --tool=memcheck --gen-suppressions=all"
@end example

Finally run @command{make check} as usual.

@node Debugging PVM assembly code
@chapter Debugging PVM assembly code

Hacking some areas of the compiler, such as the code generator pass,
involves meta-programming PVM assembler.  It is easy to find examples
anywhere in @file{libpoke/pkl-gen.c}:

@example
pkl_asm_insn (pasm, PKL_INSN_ROT);
pkl_asm_insn (pasm, PKL_INSN_MULLU);
pkl_asm_insn (pasm, PKL_INSN_NIP2);
@end example

Or, alternatively, the code may be written in RAS in a @file{.pks} file.
Like:

@example
.loop:
   bz @@type, .endloop      ; ... A B
   mod @@type               ; ... A B A%B
   rot                     ; ... B A%B A
   drop                    ; ... B A%B
   ba .loop
@end example

Often, a run-time problem becomes apparent while the PVM executes the
generated code.  Typical cases are when a PVM value doesn't contain
what it's supposed to contain, and accessing the wrong boxed value
causes a segmentation fault (if we are lucky) or a non-crashing
invalid memory access (if we are very unlucky.)  Whenever that kind of
crap happens, we find ourselves in the need of debugging the PVM code,
which is a big PITA.

Bad news are: we don't have a PVM debugger (yet).  Good news are: we
have a couple of tools that may help.

The first of such tools is the @code{prints} instruction.  This
instructions basically prints in the standard output the string value
on the TOS, and then drops it.  It is a wonderful way to trace PVM
code.

For example, let's say we are trying to find out how many times the
loop above gets executed.  We can install traces like:

@example
      push "XXX entering loop\n"
      prints
.loop:
      push "XXX in loop\n"
      prints
      mod @@type
      rot
      drop
      ba .loop
@end example

The other tool is the @code{strace} instruction.  It prints the
contents of the stack (one value per line) from the TOS.  It gets the
number of stack values to print as an argument, 0 meaning all of them.
It is very useful in many situations, like when a loop is composing
values in the stack and something is going banana.  It is also useful
to determine what kind of value is being accessed by a given
instruction.

For example, let's say that we are hunting some segmentation fault.
We highly suspect the code generated in the first example in this
section, above.  Of the three instructions, @code{mullu} is the only
one that could conceivably generate a segfault, so we add a stack
trace instruction right before it to inspect its two arguments:

@example
pkl_asm_insn (pasm, PKL_INSN_ROT);
pkl_asm_insn (pasm, PKL_INSN_STRACE, 2); /* XXX remove me */
pkl_asm_insn (pasm, PKL_INSN_MULLU);
pkl_asm_insn (pasm, PKL_INSN_NIP2);
@end example

We recompile, re-run, and we find out that the elements at the TOS
when @code{mullu} is executed are a pair of stupid signed integers,
which are not boxed and not what the instruction expects.  Mystery
solved.

@section PVM definition tips

When using advanced Jitter dispatches it becomes important that every
C function or global variable accessed by a relocatable VM instruction
is @dfn{wrapped}: see the Jitter documentation [not yet existing at
the time of writing] for more information.  Non-relocatable
instructions can access any global without need for wrapping: the
problem is strictly with VM instruction whose native executable code
gets copied to different memory addresses, therefore breaking
PC-relative access to globals.  Wrapping prevents this kind of access,
replacing it with a indirect access thru a local pointer.

It is important to remark that macros cannot be wrapped as functions:
rather it is necessary to consider any global variable or function
containsed in a macro expansion.

Notice that high-latency VM instructions (for example instructions
performing I/O) and infrequently run instructions (for example an
instruction which raises an error) can be made non-relocatable without
loss of efficiency; in fact making large VM instructions
non-relocatable might improve code locality at run time.

@node Building poke with profiling support
@chapter Building poke with profiling support

poke supports a couple of commands that can be used to profile Poke
programs:

@example
.vm profile reset
.vm profile show
@end example

For these commands to work you need to build poke with profiling
support.  This is done by passing the @command{--enable-pvm-profiling}
command line option to @command{configure}.

@node Maintenance targets
@chapter Maintenance targets

This section describes @command{make} targets that performs several
maintenance tasks.

@table @command
@item syntax-check
Run several syntax-related checks in the source files.  It is useful
to run this target before submitting code to be reviewed, and while
reviewing other people's code.

Note that sometimes the results have to be taken with a pinch of salt.
This happens, for example, when a rule oriented to C is applied to,
say, an AWK file.  In these cases, consider adding a @file{.x-sc_*}
fine-tuning file.  But please ask in poke-devel first.

We provide a pre-push git hook that performs the syntax check right
before pushing. If the check fails, the push is aborted. You should
consider enabling this check by coping @file{etc/git-hooks/pre-push}
to @file{.git/hooks/}.

@item coverage
This target builds *poke* with code coverage support, runs the
testsuite, and generates a nice html report under
@file{$(top_builddir)/doc/coverage/}.  It is necessary to have the
@command{lcov} program for this to work.  Note also that you need to
have jitter configured to use a dispatch model other than
no-threading.

@item cyclo-poke.html
This target runs the pmccabe utility to calculate the modified
cyclomatic complexity of the functions composing the poke sources, and
generates a nice html report.

@item refresh-po
This target download the latest available translations from the
translation project and installs them in the source tree.

@item update-copyright
Run this rule once per year (usually early in January) to update all
the copyright years in the project.  By default this excludes all
variants of COPYING.  Exceptions to this procedure (such as
@file{ChangeLog..*} for rotated change logs) can be added in the file
@file{.x-update-copyright}.
@end table

@node Writing compiler diagnostics
@chapter Writing compiler diagnostics

Please follow the following guidelines when adding new errors or
warnings to the poke compiler:

@itemize @minus
@item
Make the diagnostic messages self-contained, @i{i.e.} do not rely on
the extra location information to make the error intelligible.  This
is mainly for the sake of accessibility, and also for when poke is run
in quiet mode.

This is an example of a bad diagnostic:

@example
error: field doesn't exist in struct
@end example

Write something like this instead:

@example
error: field `foo' doesn't exist in struct
@end example

@item
When checking for the type of some ENTITY, please use a two-lines
error message like:

@example
error: invalid ENTITY
error: expected EXPECTED_TYPE, got ACTUAL_TYPE
@end example

this is to keep some uniformity.  You can get the written
representation of a type using the =pkl_type_str= utility function.

@item
When referring to integer types in diagnostic messages, please refer
to them as @dfn{integrals}.

@item
Always pass 1 for @code{use_given_name} when invoking
@code{pkl_type_str} unless you need the expanded definition of the
type for whatever reason (like to show the arguments taken by a
function type.)
@end itemize

@node Writing C
@chapter Writing C


In Poke we follow the GNU Coding Standards.  Please see
@url{https://www.gnu.org/prep/standards/}.

Additionally, we have a few conventions that we ask you to follow when
hacking poke, in order to keep a coherent style when possible.  These
are documented in the following subsections.

@section Avoid Tabs


Please format your code using spaces.  Tabs (Ascii 9) should not
appear anywhere except in Makefiles.

@section Avoid @code{bool}

C99 defines the type @code{bool}.  However, in poke we prefer to use
variables of type @code{int} in order to hold truth values.  We kindly
ask you to do the same.

When documenting functions that receive or return boolean values
stored in integer variables, please remember that the interpretation
of the truth values in C is: zero means false, and any value other
than zero means true.  Avoid referring to 1 to mean true.

Also, please never rely on a boolean to hold exactly 1 or any other
particular non-zero value, to be interpreted as true.

@section Use @code{_p} for predicates

We follow the venerable convention of using a suffix @code{_p} (for
predicate) when naming variables and struct fields that contain
booleans.

We do not always do this, but it is especially important in cases
where the name of the variable is not indicative enough.

@section Documenting functions in public headers

Function prototypes in headers must be documented using a comment
block preceding the prototype.

Start the comment block with a terse short sentence that states what
the function does.  Then, document the arguments that the function
takes, and the value it returns, if any.  When referring to arguments
and other variables in the comment block, please use caps.

It is ok for single comment block to document more than one prototype.

Please leave an empty line between the comment block and the function
prototype.

Example:

@example
/* Frobnicate a frob.

   FROB is a frob that should have been previously initialized using
   frob_new.

   LAZY_P is a boolean.  If true, FROB won't be frobnicated
   immediately, but instead marked for later frobnication.

   If the frob cannot be frobnicated, this function returns NULL.
   Otherwise the function returns the given frob.  */

frob_t frobnicate (frob_t frob, int lazy_p);
frob_t forbnicate_r (frob_t frob, int lazy_p);
@end example

@node Writing Poke
@chapter Writing Poke

We recommend to use the Emacs mode in @file{etc/poke-mode.el} to write
@file{.pk} files.


@section Writing offset units

Do not separate magnitudes and units when writing offsets.  Do it like
this:

@example
16#B
@end example

@noindent
instead of:

@example
16 #B
@end example

@section Use @code{Camel_Case} for type names

Use @code{Camel_Case} for type names, but do not use @code{Camel_Case}
for variable/function names!

@section Formatting conventions in pretty-printers

Surround pretty-printed values with @code{#<} and @code{>}. This is to
notify the reader that the value has been pretty-printed.

@node Writing RAS
@chapter Writing RAS

We recommend to use the Emacs mode in @file{etc/poke-ras-mode.el} to
write @file{.pks} files.

@section Calling compiled Poke from RAS

A big part of the code generated by the PKL code generator is written
by hand, in @file{.pks} files.  Sometimes it is convenient to call
compiled Poke code from that code; for example, to execute complex
formulae or to perform some operation defined recursively.

In these situations, the solution is to first write a Poke function in
the compiler's runtime library, @file{pkl-rt.pk}, like:

@example
fun _pkl_foo = (uint<64> ival, uint<64> eval,
                uint<32> ivalw, uint<32> fieldw) uint<64>:
@{
  return ival | (eval <<. (ivalw - fieldw));
@}
@end example

Then, the function can be invoked from the hand-written RAS file using
the @code{.call} directive:

@example
.call _pkl_foo
@end example

Note how the compiler internal functions use the @code{_pkl_} prefix.
Also note that @code{.call} can only invoke functions defined in the
global environment.

@node Documenting pickles
@chapter Documenting pickles

The chapter @code{Pickles} in the poke user manual contains several
sections organized in thematic areas.  When adding a new pickle, find
the area appropriate for the abstractions provided by the pickle (or
create one if it doesn't exist already) and add a section for it.

Please make sure to document any dependency of your pickle to other
pickles.

Keep a tutorial-like tone: for fine details, the user can always go
and look to the pickle directly.

@node The Poke compiler
@chapter The Poke compiler

@menu
* Compiler overview::
* The bison parser in @file{pkl-tab.y}::
* The AST::
* Lexical handling::
* Compiler passes and phases::
@end menu

@node Compiler overview
@section Compiler overview

This figure depicts the architecture of the compiler:

@example
      /--------\
      | source |
      \---+----/
          |
          v
  +-----------------+
  |      Parser     |
  +-----------------+
  |  analysis and   |
  | transformation  |
  |     phases      |
  +-----------------+
  | code generation |
  |      phase      |
  +-----------------+
  | Macro assembler |
  +-----------------+
          |
          v
     /---------\
     | program |
     \---------/
@end example

@node The bison parser in @file{pkl-tab.y}
@section The bison parser in @file{pkl-tab.y}

The only purpose of the bison parser in pkl-tab.y is to do the
syntactic analysis, build the initial AST, and set the locations of
the AST nodes.

Unfortunately, currently it also does some extra work, due to
limitations in the LARL parser:

- It builds the compile-time environment and register type, variable
  and function declarations.
- It links return statements with their containing functions.

As we shall see below, any further analysis and transformations on the
AST are performed by the compiler phases, which are implemented
elsewhere.  This greatly helps to keep the parser code clean and easy
to read, and also eases changing the syntactic structure of poke
programs.

@node The AST
@section The AST

The compiler operates on an abstract syntax tree that represent the
program being compiled.  The following subsections discuss some useful
aspects of the structure of the AST.

@menu
* loop_stmt::
* try_stmt::
* exp::
* cond_exp::
@end menu

@node loop_stmt
@subsection loop_stmt

Loop statements are represented with trees like:

@example
loop_stmt
|
+-- [iterator]
|       |
|       +-- decl
|       +-- container
|
+-- [condition]
+-- body
@end example

Different kind of loop statements supported in Poke are represented
including or not including optional fields.

@code{while} statements with a condition:

@example
while (CONDITION) BODY
@end example

@noindent
are represented as:

@example
loop_stmt
    |
    +-- condition
    +-- body
@end example

@code{for} statements without a condition:

@example
for (;;) BODY
@end example

@noindent
are represented as:

@example
loop_stmt
    |
    +-- body
@end example

@code{for-in} statements:

@example
for (VAR in CONTAINER) BODY
@end example

@noindent
are represented as:

@example
loop_stmt
    |
    +-- iterator
    |      |
    |      +-- decl (VAR
    |      +-- container
    |
    +-- body
@end example

@code{for-in-where} statements:

@example
for (VAR in CONTAINER where SELECTOR) BODY
@end example

@noindent
are represented as:

@example
loop_stmt
    |
    +-- iterator
    |      |
    |      +-- decl (VAR)
    |      +-- container
    |
    +-- condition (SELECTOR)
    +-- body
@end example

@node try_stmt
@subsection try_stmt

There are several syntactic forms of this statement:

@example
try CODE catch HANDLER
try CODE catch if EXP HANDLER
try CODE catch (ARG) HANDLER
try CODE until EXP
@end example

Try statements are represented with trees like:

@example
try_stmt
|
+-- try_stmt_body
|   |
|   +-- code
|
+-- try_stmt_handler
|   |
|   +-- code
|
+-- [arg]
+-- [exp]
@end example

where @code{arg} and @code{exp} are exclusive.

@node exp
@subsection exp

Expressions are the application of some @dfn{operator} to one or more
operators.  The Poke language supports unary, binary and ternary
operators.

Expressions are represented with trees like:

@example
exp
|
+-- code
+-- attr
+-- flag
+-- numops
+-- operands[3]
    |
    +-- [operand]
    +-- [operand]
    +-- [operand]
@end example

Where @code{code} identifies the operator of the expression.  Operators
are defined in the @file{libpoke/pkl-ops.def} file.

If the operator is an attribute, then @code{attr} identifies the invoked
attribute.  Attributes are defined in the @file{libpoke/pkl-attrs.def}
file.  This field @code{attr} is initialized in the @code{trans1} phase.


The @code{exp} AST node is used for all the Poke unary and binary
operators, which only use two of the operands.

Additionally, the application of an attribute to which an argument is
passed, like in:

@example
foo'elem (1)
@end example

Is encoded as an @code{exp} node with three operands: the first
operand holds @code{foo}, the second operand an identifier for the
attribute @code{elem} and the third operand the expression passed to
the attribute (in this case 1).  A handler in @code{trans1} transforms
this ternary expression node in a binary expression, filling
@code{attr} in the process.

Note that the Poke ternary conditional operator @code{? :} is
@emph{not} implemented using @code{exp} nodes, but @code{cond_exp} AST
nodes.

@node cond_exp
@subsection cond_exp

This AST node is used to encode Poke's ternary conditional operator,
which looks like:

@example
cond_exp ? then_exp : else_exp
@end example

It uses trees like this:

@example
cond_exp
|
+-- cond
+-- thenexp
+-- elseexp
@end example

@node Lexical handling
@section Lexical handling


The parser needs to build the lexical enviroment as it works through
the source code.  This is because it needs to find the declarations of
all the entities referenced by variable names.  Also, the Poke syntax
requires to determine whether a given identifier is a type name or
not.  It uses a @code{pkl_env} for this, and it pushes and pops frames
according to the language rules.  The parser annotates each variable
reference with the declaration of the entity being referred, but does
not set the variable's lexical address.  This is because this address
may not be final, as we shall see below.  Once the parsing is done
successfully the lexical environment should consist on just the
top-level definitions, which is kept for the rest of the compilation.
But the inner frames are no longer available.

After the parser, the front-end compiler passes are run.  This
includes analysis and checking of language rules, transformation, type
analysis, constant folding, @i{etc}.  These passes can do arbitrary
transformations of the tree, and some of these transformations can
alter the lexical structure of the program.  The only exception is the
top-level environment.  Therefore, these passes shall not rely on the
lexical structure.

Finally, just before the code generator, a lexical pass is run.  This
pass contains a @code{transl} phase.  At this point no further
transformations of the AST will be performed and it is final.  The
@code{transl} phase reconstructs the lexical environment (the
top-level environment is preserved) and assigns the lexical addresses
(back, over) to all variable references.  It also performs a few
language rules checks that rely on the relative lexical position of
entities.

@node Compiler passes and phases
@section Compiler passes and phases

@menu
* The @file{pkl-passes.def} file::
* Naming conventions for phases::
* Naming conventions for phase handlers::
@end menu

@node The @file{pkl-passes.def} file
@subsection The @file{pkl-passes.def} file

The @file{pkl-passes.def} file describes the different @dfn{passes}
performed by the compiler front-end, along with its constituent
@dfn{phases}.  This is an example of a pass definition:

@example
PKL_PASS(PKL_PASS_F_TYPES,1)
     PKL_PHASE(trans1)
     PKL_PHASE(anal1)
     PKL_PHASE(typify1)
     PKL_PHASE(promo)
     PKL_PHASE(trans2)
     PKL_PHASE(fold)
     PKL_PHASE(trans3)
     PKL_PHASE(typify2)
     PKL_PHASE(anal2)
PKL_END_PASS
@end example

When running a pass, each phase will be applied in order to each node
of the abstract syntax tree. See the documentation in
@file{pkl-pass.h} for more detailed information about the compiler's
pass manager.

@node Naming conventions for phases
@subsection Naming conventions for phases

We use the following convention to name phases:

@example
@{NAME@}@{SUFFIX@}
@end example

Where @code{NAME} reflects a phase category (see below) and
@code{SUFFIX} is usually an integer that specifies the order in which
the phases are applied.  Thus, for example, @code{name4} is performed
after @code{name1}.  Sometimes, @code{SUFFIX} is @code{f} (meaning
``final'').

The suffix is not used if there is only one phase in the given
category.

We use the following phase categories:

@table @code
@item anal
For phases whose main purpose is to perform checks on the AST, and/or
the contents of the AST nodes, and emit errors/warnings.
@item trans
For phases whose main purpose is to alter the structure of the AST,
and/or the contents of the AST nodes.
@item typify
For phases whose main purpose is to perform type checks, and otherwise
do work on types.
@item promo
For phases whose main purpose is to perform coercions wherever
appropriate.  Currently there is only one phase in this category.
@item fold
For phases whose main purpose is to pre-compute areas of the AST
whenever it is possible to do so at compile-time.  Currently there is
only one phase in this category, that performs constant folding.
@item gen
For phases whose main purpose is to generate PVM code.  Currently
there is only one phase in this category.
@end table

The phases in category @code{NAME} are implemented in the source files
@file{libpoke/pkl-NAME.[c]}.

@node Naming conventions for phase handlers
@subsection Naming conventions for phase handlers

We use the following convention to name phase handlers:

@example
pkl_PHASE_@{ps,pr@}_NODE
@end example

Where @code{PHASE} can be a complete phase name (like @code{typify1})
if the handler is to be installed in that phase only, or a phase
category name (like @code{typify}) if the handler is to be installed
in several phases in that category.  If the phase is to be executed in
pre-order, @code{pr} follows, otherwise, @code{ps}.  Finally,
@code{NODE} is the name of the AST node.

For example, the handler:

@example
pkl_anal1_ps_comp_stmt
@end example

@noindent
is installed in the phase @code{anal1}, executes in post-order, and
serves the AST nodes with code @code{PKL_AST_COMP_STMT}.

@node The Poke Virtual Machine
@chapter The Poke Virtual Machine

@menu
* PVM types::
* Exception handling::
* Signal handling::
* Offsets and bit-offsets in the PVM::
* Writing relocatable instructions::
* Avoiding branching instructions::
* Wrapping globals::
@end menu

@node PVM types
@section PVM types

@subsection How @code{any} is handled in the PVM

The Poke type @code{any} is not implemented in the PVM as a PVM type
value.  Instead, @code{null} is used to denote it, by convention.

Note that it is not possible to use @code{void} to denote the Poke
@code{any} type, because of closure types: a closure returning
@code{void} is not the same than a closure returning @code{any}.

@node Exception handling
@section Exception handling

Exception types or codes are signed 32-bit integers, and are defined
in @file{libpoke/pkl-rt.pk} as @code{EC_*} variables.

The exceptions themselves are struct values of type @code{Exception},
also defined in @file{libpoke/pkl-rt.pk}.

There are two ways an exception can be raised in the PVM:

@itemize @minus
@item Explicitly, when the instruction =raise= is executed.
@item Implicitly, when some instruction needs to fail.
For example, an integer division instruction divides by zero.
@end itemize

In either case, the treatment of a raised exception is the same:

@enumerate
@item Pop an exception handler from the exception handler stack.
@item If the exception handler matches the raised exception type, then
@enumerate
@item Restore the heights of the main and return stacks.
@item Restore the dynamic environment.
@item Push the cached exception type to the stack.
@item Branch to the exception handler.
@end enumerate
@item Repeat.
@end enumerate

The default exception handler, which catches ``unhandled'' exceptions,
is installed by the macro-assembler in
@file{libpoke/pkl-asm.c:pkl_asm_new} and
@file{libpoke/pkl-asm.c:pkl_asm_finish}.  It calls the function
@code{_pkl_exception_handler}, that is defined in the compiler runtime
in @file{libpoke/pkl-rt.pk}.

@node Signal handling
@section Signal handling

When the PVM is about to execute a program, it installs a signal
handler that, at the moment, handles @code{SIGINT}.  This signal
handler is defined in the @code{late-c} block in pvm.jitter, and
records the signal in the PVM internal state.

Then, when a @code{sync} instruction is executed, it checks the PVM
internal state and raises a @code{PVM_E_SIGNAL} exception if there is
some pending signal.  The default signal handler just ignores this
signal, but the user can also intercept it if desired.  The PKL
compiler generates @code{sync} instructions in strategic places, such
as before backward jumps in loops.

The @code{exit} PVM instruction clears the pending signals in the
internal PVM state.

@node Offsets and bit-offsets in the PVM
@section Offsets and bit-offsets in the PVM

The PVM supports a @code{pvm_off} boxed value, to denote pairs of
magnitudes and units.  Both accessor macros (in @file{pvm-val.h}) and
PVM instructions (@code{ogetm}, @code{ogetu}) are provided to access
their components.

Many other PVM entities need to denote offsets in a way or another.
For example, struct fields in @code{pvm_struct} values need to record
their relative offset with respect the beginning of the struct.

It may come to mind, quite naturally, to use @code{pvm_off} values to
denote these offsets.  It is very elegant.  However, we decided to use
@dfn{bit offsets} instead, stored in 64-bit @code{pvm_long} values.

There are two reasons for this:

@itemize @minus
@item
First of all, performance.  It is fairly common to operate with the
absolute value of these offsets, in bits.  In fact, in most cases that
is the only purpose of maintaining them.  Having them stored in
@code{pvm_off} values means we have to multiply every time we want to
get their magnitude.  This is a waste, for no good reason.
@item
To avoid code coupling.  PVM offsets are very cool, but they are also
complex: the unit is arbitrary.  This means in many cases we have to
assume the nature of the unit, mainly bits.  This is very fragile.
@end itemize

So, the take-home message is: in the PVM, restrict the presence of
@code{pvm_off} values to the ones generated by the code generator.
Whenever an offset is needed in some internal PVM structure, use
bit-offsets instead encoded as @code{ulong<64>} values.

@node Writing relocatable instructions
@section Writing relocatable instructions

When operating in non-threaded mode, Jitter likes to move the code
compiled for instruction bodies around.  This imposes several
restrictions on what can you have in the instruction bodies:

@itemize @minus
@item
If you call a global function from your instruction, please add it to
the @code{wrapped-functions} section in the file.
@item
Likewise, if you use a global variable, please add the variable to the
@code{wrapped-globals} section.
@item
Do not use any constant C string like @code{"foo"}; these are global
variables in disguise and you cannot wrap them unless you give them a
name.
@end itemize

If you cannot follow the guidelines above for whatever reason, please
mark the instruction as @code{non-relocatable} like this:

@example
instruction foo ()
  non-relocatable
  code
    ...
  end
end
@end example

This will prevent Jitter to relocate the instruction body and crash in
many platforms and non-threading modes.

@node Avoiding branching instructions
@section Avoiding branching instructions

Another cause for a Jitter instruction to be non-relocatable is when
it branches mid-instructions.  In practice this often happens when an
instruction can raise an exception.  The typical example would be a
division instruction @code{div}, which may be defined as raising a
``division by zero'' exception if division by zero is found.  Jitter
requires such an instruction to be marked as @dfn{branching} like
this:

@example
instruction div
  branching
  code
   ...
  end
end
@end example

In the PVM we always must avoid having branching instructions like the
above, since it has a performance impact.  A way to avoid it is having
the instruction push some value on the stack that signifies the
execution found a problem, then get the compiler check for that value
and raise the exception.  As an example, if the @code{div} instruction
pushes @code{PVM_NULL} on the stack if division by zero is attempted,
the compiler can generate this to check for it and raise the
corresponding exception:

@example
  div
  bnn .ok
  pushe E_division_by_zero
  raise
.ok:
  drop
@end example

@node Wrapping globals
@section Wrapping globals

The non-threaded Jitter operation modes require every global (variable
or function) to be wrapped, @i{i.e.} specified in sections
@code{wrapped-functions} or @code{wrapped-globals} in
@file{pvm.jitter}.

Since it is not trivial to keep track of every used global, having
hundred of instructions, we are providing a syntax-check that looks
for this automatically.

In order to avoid false positives, there are two files that we
maintain by hand:

@table @file
@item etc/pvm-wraps-whitelist
This file contains a list of @emph{global} symbols that are used in
pvm.jitter but are not required to be listed in neither
@code{wrapped-function} nor @code{wrapped-globals}.  These are
variables and functions that, despite being global and undefined, are
not used @emph{within} PVM instructions.
@item etc/pvm-wraps-static-whitelist
This file contains a list of @emph{static} symbols that are defined in
@file{pvm.jitter} and also listed in either @code{wrapped-functions}
or @code{wrapped-globals}.  This includes for example the @code{pvm_*}
wrappers we define in the file, to be used in PVM instructions.
@end table

@subsection Wrapping standard function calls

If we need to call to a standard library function provided by the
system (such as @code{printf}, @code{strstr}, or the like) from the
body of a PVM instruction, we have to consider the possibility of
gnulib replacing such function with another having a name like
@code{rpl_printf} or @code{rpl_strstr}.

Since the @code{wrapped-functions} list in @file{pvm.jitter} is static
and the referred functions must exist, just wrapping the @code{rpl_*}
symbols doesn't work.

So instead, for every standard function @code{FOO} we call from PVM
instructions, we:

@itemize @minus
@item
Create a wrapper for it, called @code{pvm_FOO}, and put it in the
@file{pvm-wrappers.[ch]} source files.
@item
Use @code{pvm_FOO} in the PVM instruction.
@item
Add @code{pvm_FOO} to the wrapped-functions section in
@file{pvm.jitter}.
@end itemize

@node Memory management
@chapter Memory management

Different parts of poke use different strategies for memory
management:

@itemize @minus
@item
The compiler front-end uses reference counting to handle AST nodes.
@item
The PVM uses the Boehm GC collector for values and the run-time
environment.
@item
Everything else uses @code{malloc}/@code{free}.
@end itemize

This sometimes leads to tricky situations, some of which are
documented in the sections below.

@menu
* Using @code{ASTREF}::
* Using @code{ASTDEREF}::
* PVM values in PVM programs::
* PVM values in AST nodes::
@end menu

@node Using @code{ASTREF}
@section Using @code{ASTREF}

The AST uses reference counting in order to manage the memory used by
the nodes.  Every time you store a pointer to an AST node, you should
use the @code{ASTREF} function in order to increase its counter:

@example
pkl_ast_node foo = ASTREF (node);
@end example

Note that the @code{pkl_ast_make_*} constructors do @code{ASTREF}
internally, so you don't need to use it in calls like:

@example
pkl_ast_node new = pkl_ast_make_struct (ast, 5, elems_node);
@end example

To make sure you get the reference counting right, it's required to
use the return value of @code{ASTREF} function. The compiler will warn
you otherwise.

@node Using @code{ASTDEREF}
@section Using @code{ASTDEREF}

@code{ASTDEREF} decreases the reference counter of the provided AST
node. It's required to use the return value of @code{ASTDEREF}. The
compiler will warn you otherwise.

In practice you will seldom find yourself in the need to use
@code{ASTDEREF}.  Just make sure that every @code{ASTREF} is paired
with a @code{pkl_ast_node_free}.

However, there are situations where @code{ASTDEREF} is necessary in
order to avoid a memory leak.  For example, consider transformations
like @code{a -> b} to @code{a -> x -> b}.  In that case, you should
use something like:

@example
b = PKL_AST_KIND_WHAT (node);
x = pkl_ast_make_xxx (ast, ASTDEREF (b));
PKL_AST_KIND_WHAT (node) = ASTREF (x);
@end example

This works because @code{pkl_ast_make_xxx} does an @code{ASTREF} to
@code{b} internally.  The final result is that the reference counter
of @code{b} doesn't change at all.

@node PVM values in PVM programs
@section PVM values in PVM programs

PVM routines (data structures of type =pvm_routine=) are allocated by
Jitter in complicated data structures, internally relying on
@code{malloc}.  Their content is therefore not automatically visible
to the GC.

Now, the instructions in a routine can contain literal PVM values, and
some of these values will be boxed.  For example, the following
routine contains a pointer to a @code{pvm_val_box}:

@example
;; Initialize the element index to 0UL, and put it
;; in a local.
push ulong<64>0
regvar $eidx
@end example

There is only one place where jitter routines are stored in other
data structures: the PVM program abstraction.

PVM programs are allocated using @code{pvm_alloc}, @i{i.e.} by the GC.
They are composed by a PVM routine and other fields:

@example
struct pvm_program
@{
  pvm_routine routine;
  [...]
@}
@end example

However, since @code{routine} is malloc-allocated, the GC can't
traverse it.  Consequently, the references to contained boxed values
won't be accounted for, and these values will be collected if there
are no more references to them!

The solution, recommended by Luca Saiu, is to keep an array of
pointers in the PVM program structure, containing the pointers to
every boxed value used in @code{routine}:

@example
struct pvm_program
@{
  pvm_routine routine;
  void **pointers;
  [...]
@};
@end example

The pointers are collected in the @code{pvm-program.c} functions that
append parameters to instructions.

@node PVM values in AST nodes
@section PVM values in AST nodes

Storing a PVM value (whose memory is handled by GC) in an AST node
(reference-counted) leads to a problem: the GC is unable to see the
reference to values, and will therefore collect the memory if there
are no other reachable references.

Let's see an example of this.  Both array and struct types contain
closures for mappers, writers, bounders, and the like.  For example,
let's take arrays:

@example
struct pkl_ast_type
@{
    ...
    struct
    @{
      union pkl_ast_node *bound;
      union pkl_ast_node *etype;
      pvm_val mapper;
      pvm_val writer;
      pvm_val bounder;
    @} array;
@}
@end example

In this case, @code{mapper}, @code{writer} and @code{bounder} are
managed by GC.  However, the contained @code{struct pkl_ast_type} is
not.  There is no way the GC can find these nodes thru the AST node.

The solution is to allocate memory for these values as
@dfn{uncollectable}.  This is done with
@code{pvm_alloc_uncollectable}.  For example:

@example
pkl_ast_node
pkl_ast_make_array_type (pkl_ast ast, pkl_ast_node etype, pkl_ast_node bound)
@{
  ...
  /* Prevent GC to collect these PVM values.  */
  PKL_AST_TYPE_A_CLOSURES (type)
      = pvm_alloc_uncollectable (nclosures * sizeof (pvm_val));
  PKL_AST_TYPE_A_MAPPER (type) = PVM_NULL;
  PKL_AST_TYPE_A_WRITER (type) = PVM_NULL;
  PKL_AST_TYPE_A_BOUNDER (type) = PVM_NULL;
  PKL_AST_TYPE_A_CONSTRUCTOR (type) = PVM_NULL;
  PKL_AST_TYPE_A_INTEGRATOR (type) = PVM_NULL;
  PKL_AST_TYPE_A_FORMATER (type) = PVM_NULL;
  PKL_AST_TYPE_A_PRINTER (type) = PVM_NULL;

  return type;
@}
@end example

And of course, before the memory of the AST node is freed, the
uncollectable memory shall be explicitly fred in
@code{pkl_ast_node_free}:

@example
void
pkl_ast_node_free (pkl_ast_node ast)
@{
  ...
  case PKL_AST_TYPE:
    switch (PKL_AST_TYPE_CODE (ast))
      @{
      ...
      case PKL_TYPE_ARRAY:
        pvm_free_uncollectable (PKL_AST_TYPE_A_CLOSURES (ast));
      @}
@}
@end example

If you add PVM values or PVM environments to an AST node, please
follow the strategy above.

@node Concept Index
@appendix Concept Index

@c Merge the findex and vindex into the cpindex.
@syncodeindex fn cp
@syncodeindex vr cp

@printindex cp

@bye
