Oyranos/Code Generator

From ColourWiki

How it works

Overview

The oyAPIGenerator executable takes 3 arguments as input.

• A directory the templates files using the grantlee format.
• A source directory with all included code from the templates.
• An output directory were all auto-generated code is created.

All are optional, and in an in source build none of them is actually used, because they get the default values (respectively):

• ./templates
• ./sources
• ./API_generated

Actually, the extract_sources.sh shell script is used for convenience. It cleans the API_generated/ directory, creates the code, and then builds it. The paths work for an in source build.

The details

Template system

The templates are written in the Django template language (http://docs.djangoproject.com/en/dev/topics/templates/). A Django template is a plain text file with some specially formatted tags and variables. So, a template is just a C or C++ file that contains some extra {% tag %} and {{ variable }} embedded code that is replaced by the code generator. This is much like php code is replaced in html files. The built in tags and variables are documented here (http://docs.djangoproject.com/en/dev/ref/templates/builtins/). The Django template engine is written in python and can be used in stand alone mode (http://docs.djangoproject.com/en/dev/ref/templates/api/#configuring-the-template-system-in-standalone-mode). Instead, the generator is using the Grantlee template system (http://www.grantlee.org/), which uses the same template language, but is written in C++ and depends on Qt (http://qt.nokia.com/).

So, how is a new source code file auto-generated?

• First the generator scans the templates/ directory for template files. They have a special file name in the form of file_name.template.ext, e.g. oyranos_module.template.h
• Then the various template variables are loaded (taken from the source code metadata) and the template is rendered in memory
• A new file is created in API_generated/ with the .template. removed, e.g. oyranos_module.h

Template files

There are mainly two kinds of template files.

• A base template.
• A child template that extends a base template. Most templates are child templates.

The child templates use the {% extends %} tag, which is how template inheritance (http://docs.djangoproject.com/en/dev/topics/templates/#template-inheritance) is implemented in the django language. There is a 1-1 relationship between the class inheritance of the Oyranos object system and the django template inheritance. For example, that means if oyCMMapi10_s inherits from oyCMMapiFilter_s, then CMMapi10_s.template.c will extend CMMapiFilter_s.template.c. And if oyCMMapiFilter_s extends oyCMMapi_s, then CMMapiFilters_s.template.h will extend CMMapi_s.template.h.

The example above shows the inheritance graph for a public header file,
of a common Oyranos class.

The example table above shows the inheritance for a private implementation file,
of a special kind of Oyranos class - a "list" class. Note that the Base_s_.c
template does not extend a supposed oyStruct_s_.template.c file, because oyStruct_s
only has a public interface and so no private oyStruct_s_.* implementation files.

If in doubt, each generated source file has it's inheritance graph embedded, at the top of the @file doxygen comment.

Template file auto-generation

As stated above, there are two types of template files.

• The base template files: Base_s*[ch] and BaseList_s*[ch]. These templates contain all the generic template code. All template files that are used to generate a C source file, inherit (directly or indirectly) one of the respective Base* files. These files should not be edited, unless a global change to all generated source files is needed.
• Files named as <class_name>*.template.* . Each one of these files is used to auto-generate a source file in API_Generated/ (with the ".template" stripped from the filename). Each time the generator is run, it checks the sources/<class_name>.dox file for the [notemplates] tag. If it's not there, the <class_name>*.template.* file is auto-generated from the respective Class_s.[ch] file. Their usage is for overriding the inherited blocks, eg to insert include files, etc.

So, the second type of templates is autogenerated at first, and then edited by the user (if needed).

Code Organisation

File Structure

API_generated/

All the files in this directory are auto-generated from the templates.

Classes

Each class is implemented by four files.

oyClass_s.h

This is the public header file that exports the class API

oyClass_s.c

The implementation of all the public class parts.

oyClass_s_.h

Private declarations

oyClass_s_.c

Private definitions

Misc

oyranos_object.h

This file includes some vital oyranos headers and also contains

definitions necessary for, but not part of the object system.

It is included by all the oyClass_s.h headers.

oyranos_object_internal.h

This file includes some private oyranos headers and definitions necessary for,

but not part of the object system.

It is included by the private oyClass_s_.c implementation files.

oyTest.[cc|h]

Tests using the Qt test framework. For every generated class the basic functionality

like create, copy destroy is tested.

CMakeLists.txt

The generated file for building the code with cmake.

Modules

oyranos_generic.h

For everything that belongs to the Generic Objects API and is not part of a specific class.

oyranos_generic_internal.h

All Generic Objects API helper code, that is for internal usage only.

It is included by all the private implementations (oyClass_s_.c sources) that belong to the Generic Objects group.

oyranos_generic.c

For private/public functions that belong to the Generic Objects API but do not belong to a specific class.

oyranos_devices.h

Exports all public declarations that are part of the Device API.

oyranos_devices.c

The implementation for the Device API.

oyranos_devices_internal.[ch]

All Device API code that is for internal usage.

oyranos_module.h

All declarations that are part of the Module APIs, but not part of a specific class, and should be exported.

oyranos_module_internal.h

Used for all members of the Module APIs that are not part of any class and should not be exported.

It is included by all the private implementations (oyClass_s_.c sources) that belong to the Module APIs group.

oyranos_profile.h

All declarations that are are part of the Profile API, but not part of a specific class, and should be exported.

oyranos_conversion.h

All public declarations of the Conversion API's that do not belong to a specific class, go here.

oyranos_image.h

All public definitions of the Image API that do not belong to a specific class, go here.

oyranos_image_internal.h

All private definitions of the Image API that do not belong to a specific class, go here.

sources/

Here is all the source code that does not need to be inside the templates.

<class>.dox

This is the Doxygen description for the class, with some additional tags.

/** @struct  oyClass_s
 *  @ingroup some_group
 *  @extends oyStruct_s
 *  @brief   Brief description
 *  @internal
 *
 *  Multi line description
 *  @note New templates will not be created automaticly [notemplates]
 *  @note Create templates using "opaque pointer" [opaquepointer]
 *  @note This class holds a list of objects [list]
 *
 *  @version Oyranos: x.x.x
 *  @since   YYYY/MM/DD (Oyranos: x.x.x)
 *  @date    YYYY/MM/DD
 */

The basic idea is that the generator needs to know all kinds of information (metadata) about the class and these are provided here. Along with the doxygen tags, a few additional are also needed and are put in the @note tag.

[notemplates]

Each class has a template file for each generated source file.

At class creation, these are also created automaticly and are read-only.

When for any reason you want to override some default template block and

edit the class templates, change their permissions to read-write and

remove the [notemplates] tag

[list]

This tag specifies that the class is a special kind of class, a list of values. The

convention is that the class name is in plural (ends with s) and the list item type

is the class with the same name without the s. E.g. oyFilterPlugs_s -> oyFilterPlug_s

[opaquepointer]

This is an alternative way of the API and should not be used now.

NOTE: This file will be embedded in the oyClass_s.h file.

<class>.members.h

A list of the class members, but not inherited members from parent classes, e.g. for CMMapi6.members.h

 /** oyCMMapi4_s::context_type typic data; e.g. "oyDL" */
 char           * data_type_in;
 /** oyCMMapi7_s::context_type specific data; e.g. "lcCC" */
 char           * data_type_out;
 oyCMMdata_Convert_f oyCMMdata_Convert;

NOTE: This file will be embedded in the oyClass_s_.h file.

<class>.public.h

Here goes code for the oyClass_s.h public header file, e.g. for Options.public.h

typedef oyStruct_s * (*oyStruct_Copy_f ) ( oyStruct_s *, oyPointer );
typedef int       (*oyStruct_Release_f ) ( oyStruct_s ** );
typedef oyPointer (*oyStruct_LockCreate_f)(oyStruct_s * obj );
...
extern oyStruct_LockCreate_f   oyStruct_LockCreateFunc_;
extern oyLockRelease_f         oyLockReleaseFunc_;
extern oyLock_f                oyLockFunc_;
extern oyUnLock_f              oyUnLockFunc_;
...

NOTE: This file will be embedded in the oyClass_s.h file.

<class>.public_methods_declarations.h

All declarations of the class's public interface, that is not generated automaticly. E.g, for Profile.public_methods_declarations.h:

...
OYAPI oyProfile_s * OYEXPORT
                   oyProfile_FromMD5(  uint32_t          * md5,
                                       oyObject_s          object );
OYAPI int OYEXPORT
         oyProfile_GetChannelsCount ( oyProfile_s * colour );
OYAPI icSignature OYEXPORT
             oyProfile_GetSignature (  oyProfile_s       * profile,
                                       oySIGNATURE_TYPE_e  type );
...

NOTE: This file will be embedded in the oyClass_s.h file.

<class>.public_methods_definitions.c

The implementation file of the above public interface.

NOTE: This file will be embedded in the oyClass_s.c file.

<class>.private.h

All class definitions, enums, structs, etc that should not be exported should go here.

NOTE: This file will be embedded in the oyClass_s_.h file.

<class>.private_methods_declarations.h

NOTE: This file will be embedded in the oyClass_s_.h file.

<class>.private_methods_definitions.c

NOTE: This file will be embedded in the oyClass_s_.c file.

<class>.private_custom_definitions.c

NOTE: This file will be embedded in the oyClass_s_.c file.

templates/

Files

Base_s.h

Base_s.c

Base_s_.h

Base_s_.c

BaseList_s.h

BaseList_s.c

BaseList_s_.h

BaseList_s_.c

CMakeLists.template.txt

oyTest.template.h / oyTest.template.cc

Directories

The directories are named by the group name (@ingroup tag) and hold the template files of the classes that belong to that group.

Naming conventions

Function prototypes

Public member functions

All input/output variables use the public class interface. E.g:

OYAPI oyProfile_s* OYEXPORT
  oyProfile_Copy( oyProfile_s *profile, oyObject_s obj );

Private member functions

The input/output variables of the function class type use the private class interface. E.g:

oyProfile_s_*
  oyProfile_Copy_( oyProfile_s_ *profile, oyObject_s object);

The rest variables use their public interface. E.g:

oyProfileTag_s * oyProfile_GetTagByPos_ ( oyProfile_s_    * profile,
                                          int                 pos );
int                oyProfile_TagMoveIn_ ( oyProfile_s_      * profile,
                                          oyProfileTag_s   ** obj,
                                          int                 pos );

How to import a new class

Steps to use

1. Initial commit

Directory

sources/

• (a) cp Class.dox <class>.dox & edit
• (b) run generator
• (c) Edit <class>.members.h
• (h) run generator

NOTE Skip steps (c)&(h) for list classes.

git short comment

* Create skeleton files for oyClass_s

2. Import class members like enums,typedefs,...

Search for them in oyranos sources

grep 'memberof *oy<class>_s' * -B3

• Private ones in sources/<class>.private.h

  git short comment

  [sources] Import oy<class>_s private [enums,typedefs,...]

• Public ones in sources/<class>.public.h

  git short comment

  [sources] Import oy<class>_s public [enums,typedefs,...]

• Add proper include files in oyClass_s.h and oyClass_s_.h

  Find them by trying to compile the object files

  cd /API_generated/

  make oyClass_s.o

  make oyClass_s_.o

  git short comment

  [templates] Add include files to oyClass_s.h

3. Implement constructor [oyClass_New]

files

sources/<class>.private_custom_definitions.c

git short comment

[review] [sources] Implement the constructor for oyClass_s

4. Implement copy constructor [oyClass_Copy]

Files:

sources/<class>.private_custom_definitions.c

git short comment

[review] [sources] Implement the copy constructor for oyClass_s

5. Implement destructor [oyClass_Release]

sources/

.private_custom_definitions.c

[review] [sources] Implement the destructor for oyClass_s

6. Import private methods for oyClass_s

sources/

.private_methods_declarations.h
.private_methods_definitions.c

[sources] Import private methods for oyClass_s

7. Adopt oyClass_s private methods

sources/

.private_methods_declarations.h
.private_methods_definitions.c

Make a different commit for each refactored function:

[review] [sources] Adopt oyClass_XXX_() to "hidden struct" interface.

8. Import public methods for oyClass_s

sources/

.public_methods_declarations.h
.public_methods_definitions.c

[sources] Import public methods for oyClass_s

9. Adopt oyClass_s public methods

sources/

.public_methods_declarations.h
.public_methods_definitions.c

Make a different commit for each refactored function:

[review] [sources] Adopt oyClass_XXX() to "hidden struct" interface.

Tips n' Tricks

How to insert include files

To use include files in e.g. oyClass_s.c override the LocalIncludeFiles block in the Class_s.template.c template file. There are 2 blocks available for every template.

LocalIncludeFiles

For local tree include files

GlobalIncludeFiles

For system files

For example, to include a system icc34.h file in oyProfile_s_.h, add the following in Profile_s_.template.h:

{% block GlobalIncludeFiles %}
{{ block.super }}
#include <icc34.h>
{% endblock %}

To also include oyStructList_s.h, oyProfileTag_s.h, oyConfig_s.h:

{% block LocalIncludeFiles %}
{{ block.super }}
#include "oyStructList_s.h"
#include "oyProfileTag_s.h"
#include "oyConfig_s.h"
{% endblock %}

The above blocks will render as (in oyProfile_s_.h):

...
#define oyProfilePriv_m( var ) ((oyProfile_s_*) (var))
#include <icc34.h>
#include <oyranos_object.h>
#include "oyStructList_s.h"
#include "oyProfileTag_s.h"
#include "oyConfig_s.h"
#include "oyProfile_s.h"
typedef struct oyProfile_s_ oyProfile_s_;
...

How to use the f_compare.sh script

This script can show you the differences of a function definition between two branches. The function should have a doxygen header (even an empty one).

Usage: extras/f_compare.sh [-h|-s] <function> [branch:]<file1> [branch:]<file2>
-h      Show this help message
-s      Show just the diff output

If branch is ommited, the current branch is assumed. Two files are created at the working directory, each having the function body as it's contents. Without the -s switch, vimdiff is run on those two files. They are named as: branchname-filename-funcname.c

So, why is this script useful? When a function is imported to the template system, it is copied from the oyranos_alpha.c or oyranos_cmm.c files, into the relevant one in the sources/ directory. That happens in the gsoc2011 branch. If the function is changed in master, before the working branch gets merged back, then the copied function in sources/ has to be updated, too.

For example, oyConfig_Find() is imported from oyranos_alpha.c in sources/Config.public_methods_definitions.c. If we are in the gsoc2011 branch and want to check if the function has been updated in master, we have to call the f_compare.sh script like:

 ./extras/f_compare.sh oyConfig_Find master:oyranos_alpha.c sources/Config.public_methods_definitions.c

Now you can add the updates to the function in vimdiff, save the file, and then replace the function in sources/Config.public_methods_definitions.c with the gsoc2011-Config.public_methods_definitions.c-oyConfig_Find.c file.