Merge branch 'develop' into main

# Conflicts:
#	.gitignore

Author: francisco-ortin

.gitignore

@@ -20,10 +20,6 @@ parts/
 sdist/
 var/
 wheels/
-<<<<<<< HEAD
-=======
-pip-wheel-metadata/
->>>>>>> 892da8f00d4ce0b5f981ca1329f6b076693c0ad7
 share/python-wheels/
 *.egg-info/
 .installed.cfg
@@ -53,10 +49,7 @@ coverage.xml
 *.py,cover
 .hypothesis/
 .pytest_cache/
-<<<<<<< HEAD
 cover/
-=======
->>>>>>> 892da8f00d4ce0b5f981ca1329f6b076693c0ad7
 
 # Translations
 *.mo
@@ -75,14 +68,8 @@ instance/
 # Scrapy stuff:
 .scrapy
 
-# Sphinx documentation
-docs/_build/
-
 # PyBuilder
-<<<<<<< HEAD
 .pybuilder/
-=======
->>>>>>> 892da8f00d4ce0b5f981ca1329f6b076693c0ad7
 target/
 
 # Jupyter Notebook
@@ -93,7 +80,6 @@ profile_default/
 ipython_config.py
 
 # pyenv
-<<<<<<< HEAD
 #   For a library or package, you might want to ignore these files since the code is
 #   intended to run in multiple environments; otherwise, check them in:
 # .python-version
@@ -144,7 +130,6 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
-<<<<<<< HEAD
 
 # pytype static type analyzer
 .pytype/
@@ -161,3 +146,8 @@ out/
 # Pycharm config dir
 .idea/
 
+# Sphinx documentation
+sphinx/_build/
+
+scratch.py
+out/

.nojekyll

@@ -0,0 +1 @@
+pip install singledispatch

INSTALL

@@ -3,48 +3,115 @@
 
 [![License](https://img.shields.io/github/license/ComputationalReflection/cnerator)](LICENSE)
 [![Latest release](https://img.shields.io/github/v/release/computationalreflection/cnerator?include_prereleases)](https://github.com/ComputationalReflection/cnerator/releases)
+[![Documentation](https://img.shields.io/static/v1?label=docs&message=Sphinx&color=blue)](https://computationalreflection.github.io/Cnerator/)
 <img alt="Code size" src="https://img.shields.io/github/languages/code-size/computationalreflection/cnerator">
 <img alt="Repo size" src="https://img.shields.io/github/repo-size/computationalreflection/cnerator">
 
 
+Cnerator is a C source code generation tool. It can be used to generate large amounts of 
+[standard ANSI/ISO C source code](https://www.iso.org/standard/74528.html), ready to be compiled 
+by any standard language implementation. 
+Cnerator is highly customizable to generate all the syntactic constructs of the C language, necessary to build 
+accurate predictive models with machine learning algorithms. 
 
-Cnerator is a C source code generation tool. Generated programs can be compiled with any standard ANSI C compiler. 
-The user may define different parameters such as the number of function to be generated or the probabilities of all
-the different syntactic constructs (e.g., the average number of statements in a function, expression types, 
-number and types of local variables, and the kind of syntactic constructs to be generated). 
+## Functionalities
 
-Generated programs fulfill the type rules of the C programming language, so they are compiled without errors. 
-Using the different parameters, the user can utilize Cnerator to create a huge amount of synthetic C programs,
-necessary in common "Big Code" scenarios.
+These are the main functionalities provided by Cnerator: 
 
-## Command line options:
+1. _ANSI/ISO standard C_. All the source code generated by Cnerator follows the ISO/IEC 9899:2018 (C17) 
+standard specification.
+ 
+2. _Probabilistic randomness_. C language constructs are randomly generated, following different probability 
+distributions specified by the user. For example, it is possible to describe the probability of each statement 
+and expression construct, the number of statements in a function, and the types of their arguments and return values. 
+To this aim, the user can specify fixed probabilities of each element, or use different probability distributions, 
+such as normal, uniform and direct and inverse proportional. 
+
+3. _Compilable code_. The generated code strictly follows the syntax grammar, type system and semantic 
+rules of the C programming language. In this way, the generated code has been checked to be compilable 
+by any standard compiler implementation. 
+
+4. _Highly customizable_. Many features of the programs to be generated are customizable. 
+Some examples include the types of each language construct, array dimensions and sizes, struct fields, 
+maximum depth of expression and statement trees, number of function parameters and statements, 
+global and local variables, structures of control flow statements and type promotions, 
+among others –see the detailed [documentation](user-manual.md). 
+
+5. _Large amounts of code_. Cnerator is designed to allow generating large amounts of C source code. 
+A parameter indicates the number of independent compilation units to be created for the output application, 
+so that each unit could be treated as an independently compilable module. This feature, together with the 
+probabilistic randomness, make Cnerator an ideal tool to create predictive models of source code, because 
+the input programs used to train such models comprise abundant and varied code patterns. 
+
+## Usage
+
+To run Cnerator, you need to install the `singledispatch` Python package first:
+
+
+``` text
+pip install singledispatch
+```
+
+Then, you may just run Cnerator with no arguments to generate a random C program:
+
+
+``` text
+python cnerator.py
+```
+
+There are plenty of options to customize Cnerator. To specify the probability of a particular language
+construct, you can use the `-p` or `--probs` option. 
+The following command sets to 20% the probability of
+generating a function invocation when a new expression is required:
 
 ``` text
-usage: cnerator.py [-h] [-w PATH] [-o NAME] [-p AMOUNT] [-r RECURSION]
-                   [-vst VISITORS] [-v] [-d]
-
-Generates a compilable C program
-
-optional arguments:
-  -h, --help            show this help message and exit
-  -w PATH, --working-dir PATH
-                        Working directory (default: out)
-  -o NAME, --output NAME
-                        C output file name, without the .c extension (default:
-                        main)
-  -p AMOUNT, --parts AMOUNT
-                        Split the program in different C files (default: 2)
-  -r RECURSION, --recursion RECURSION
-                        Python recursion limit (default: 50000)
-  -vst VISITORS, --visitors VISITORS
-                        Semicolon-separated list of visitors, in order (e.g.,
-                        visitors.func_to_proc;visitors.return_instrumentation)
-                        (default: )
-  -v, --verbose         Verbose messages (default: False)
-  -d, --debug           Generate debug info (call graph and struct structure)
-                        in .dot files (default: False)
+python cnerator.py -p "call_prob = {True: 0.2, False: 0.8}"
 ```
 
+If more sophisticated probabilities are required, you can specify them in a JSON file and pass it as
+a parameter (see the [documentation](user-manual.md#probability-specification-files) to know the JSON file format). 
+The following line passes an example JSON file in the `json/probabilities` folder where
+different probability distributions are specified for some syntactic constructs:
+
+``` text
+python cnerator.py -P json/probabilities/example_probs.json
+```
+
+Cnerator also provides allows the user to control the number and characteristics of 
+all the functions to be generated. A JSON file is used for that purpose 
+(see the [documentation](user-manual.md#function-generation-files)). 
+The following command makes Cnerator generate one function for each high-level return
+type in the C programming language:
+
+
+``` text
+python cnerator.py -f json/functions/1-function-each-type.json
+```
+
+Sometimes, the user needs the output program to fulfill some requirements not guaranteed by the 
+stochastic generation process.
+To this aim, Cnerator allows the specification of an ordered collection of Python 
+post-process specification files (see the [documentation](user-manual.md#post-processing-specification-files)). 
+These post-processing files can modify the generated code to satisfy those requirements. 
+The following execution generates a random program and then executes two visitors: 
+one to rename `func` functions to `proc` (and their invocations) when they return `void`;
+and another one to add a `__RETURN__` label before each return statement:
+
+``` text
+python cnerator.py -V visitors.func_to_proc:visitors.return_instrumentation
+```
+
+To see all the options, just run the `-h` or `--help` options.
+For more information, please check the [documentation](user-manual.md).
+Developer documentation is also provided [here](https://computationalreflection.github.io/Cnerator).
+
+
+## Documentation
+
+* Check out the [user manual](user-manual.md) about how to use Cnerator.
+ 
+* A [developer manual](https://computationalreflection.github.io/Cnerator) is also available 
+if you want to modify the source code.
 
 ## License
 

README.md

@@ -1,6 +1,12 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+"""
+This is the entry point to the Cnerator source code generation tool.
+The main function runs Cnerator.
+To see the different arguments provided, run Cnerator with either ``-h`` or ``--help`` command line arguments.
+"""
+
 from __future__ import print_function
 
 import os
@@ -11,26 +17,32 @@
 from params.parameters import parse_args, get_modules_to_import, get_probs_to_override
 from params.writter import write_in_files
 from params import json_probs
-import cnerator
+import core
 
 
 def run(args):
+    """
+    This function runs Cnerator, receiving the command line arguments.
+
+    :param args: The command line arguments returned by ``argparse.ArgumentParser.parse_args``.
+    :return: None.
+    """
     # Set the recursion limit
     sys.setrecursionlimit(args.recursion)
 
-    # Get the probabilites from the command line arguments and modify the default ones
-    cnerator.probs.set_probabilites(get_probs_to_override(args.probs))
+    # Get the probabilities from the command line arguments and modify the default ones
+    core.probs.set_probabilities(get_probs_to_override(args.probs))
     if args.probsfile:
-        from cnerator import probs
-        probabilites = json_probs.parse_probabilites_specification_json_file(args.probsfile)
-        probs.set_probabilites(probabilites)
+        from core import probs
+        probabilities = json_probs.parse_probabilities_specification_json_file(args.probsfile)
+        probs.set_probabilities(probabilities)
 
     # Process the json file to create functions, or generates a program using the probabilities defined
     if args.functions:  # if a json file was passed, it defines the functions to be generated
         dictionary = parameters.parse_function_specification_json_file(args.functions)
-        program = cnerator.generators.generate_program_with_function_distribution(dictionary, args, remove_unwanted_functions=True)
+        program = core.generators.generate_program_with_function_distribution(dictionary, args, remove_unwanted_functions=True)
     else:  # otherwise, a random program is generated, considering the specified probabilities
-        program = cnerator.generators.generate_program()
+        program = core.generators.generate_program()
 
     # Load all the visitor modules and run them, in the same order as specified by the user
     modules = get_modules_to_import(args.visitors)
@@ -54,6 +66,9 @@ def run(args):
 
 
 def main():
+    """Main function.
+    To see the different arguments provided, run Cnerator with either ``-h`` or ``--help`` command line arguments.
+    """
     exit(run(parse_args()))