Merge 1.3.9 from RasaHQ

Author: Philippe Cote-Boucher

.travis.yml

@@ -114,4 +114,4 @@ jobs:
       distributions: "sdist bdist_wheel"
       password:
         secure: "MeL1Ve97eBY+VbNWuQNuLzkPs0TPc+Zh8OfZkhw69ez5imsiWpvp0LrUOLVW3CcC0vNTANEBOVX/n1kHxfcqkf/cChNqAkZ6zTMmvR9zHDwQxXVGZ3jEQSQM+fHdQpjwtH7BwojyxaCIC/5iza7DFMcca/Q6Xr+atdTd0V8Q7Nc5jFHEQf3/4oIIm6YeCUiHcEu981LRdS04+jvuFUN0Ejy+KLukGVyIWyYDjjGjs880Mj4J1mgmCihvVkJ1ujB65rYBdTjls3JpP3eTk63+xH8aHilIuvqB8TDYih8ovE/Vv6YwLI+u2HoEHAtBD4Ez3r71Ju6JKJM7DhWb5aurN4M7K6DC8AvpUl+PsJbNP4ZeW2jXMH6lT6qXKVaSw7lhZ0XY3wunyVcAbArX4RS0B9pb1nHBYUBWZjxXtr8lhkpGFu7H43hw63Y19qb8z4+1cGnijgz1mqXSAssuc+3r0W0cSr+OsCjmOs7cwT6HMQvPEKxLohwBOS/I3EbuKQOYMjFN5BWP5JXbsG45awV9tquxEW8zxjMetR+AOcYoyrDeiR8sAnj1/F99DE0bL1KyW/G5VNu2Xi/c+0M3KvP3+F8XTCuUY/5zTvqh1Qz1jcdiwsiAhO4eBQzQnjeFlxdiVeue2kmD5qsh+VLKKuKLfyVoaV7b1kBlAtBDu7+hDpA="
-    after_deploy: bash scripts/ping_slack_about_package_release.sh
+    after_deploy: ./scripts/ping_slack_about_package_release.sh

CHANGELOG.rst

@@ -7,7 +7,106 @@ Rasa Change Log
 All notable changes to this project will be documented in this file.
 This project adheres to `Semantic Versioning`_ starting with version 1.0.
 
+[Unreleased 1.3.10]
+^^^^^^^^^^^^^^^^^^^
 
+Added
+-----
+
+Fixed
+-----
+
+Changed
+-------
+
+Removed
+-------
+
+[1.3.9] - 2019-10-10
+^^^^^^^^^^^^^^^^^^^^
+
+Added
+-----
+- Port of 1.2.10 (support for RabbitMQ TLS authentication and ``port`` key in
+  event broker endpoint config).
+- Port of 1.2.11 (support for passing a CA file for SSL certificate verification via the
+  --ssl-ca-file flag).
+
+Fixed
+-----
+- Fixed the hanging HTTP call with ``ner_duckling_http`` pipeline.
+- Fixed text processing of ``intent`` attribute inside ``CountVectorFeaturizer``.
+
+[1.3.8] - 2019-10-08
+^^^^^^^^^^^^^^^^^^^^
+
+Changed
+-------
+- Policies now only get imported if they are actually used. This removes
+  TensorFlow warnings when starting Rasa X
+
+Fixed
+-----
+- Fixed error ``Object of type 'MaxHistoryTrackerFeaturizer' is not JSON serializable``
+  when running ``rasa train core``
+- Default channel ``send_`` methods no longer support kwargs as they caused issues in incompatible channels
+- Fixed ``argument of type 'NoneType' is not iterable`` when using ``rasa shell``,
+  ``rasa interactive`` / ``rasa run``
+
+[1.3.7] - 2019-09-27
+^^^^^^^^^^^^^^^^^^^^
+
+Fixed
+-----
+- re-added TLS, SRV dependencies for PyMongo
+- socketio can now be run without turning on the ``--enable-api`` flag
+- MappingPolicy no longer fails when the latest action doesn't have a policy
+
+[1.3.6] - 2019-09-21
+^^^^^^^^^^^^^^^^^^^^
+
+Added
+-----
+- Added the ability for users to specify a conversation id to send a message to when
+  using the ``RasaChat`` input channel.
+
+[1.3.5] - 2019-09-20
+^^^^^^^^^^^^^^^^^^^^
+
+Fixed
+-----
+- Fixed issue where ``rasa init`` would fail without spaCy being installed
+
+[1.3.4] - 2019-09-20
+^^^^^^^^^^^^^^^^^^^^
+
+Added
+-----
+- Added the ability to set the ``backlog`` parameter in Sanics ``run()`` method using
+  the ``SANIC_BACKLOG`` environment variable. This parameter sets the
+  number of unaccepted connections the server allows before refusing new
+  connections. A default value of 100 is used if the variable is not set.
+- Status endpoint (``/status``) now also returns the number of training processes currently running
+
+Fixed
+-----
+- Added the ability to properly deal with spaCy ``Doc``-objects created on
+  empty strings as discussed `here <https://github.com/RasaHQ/rasa/issues/4445>`_.
+  Only training samples that actually bear content are sent to ``self.nlp.pipe``
+  for every given attribute. Non-content-bearing samples are converted to empty
+  ``Doc``-objects. The resulting lists are merged with their preserved order and
+  properly returned.
+- asyncio warnings are now only printed if the callback takes more than 100ms
+  (up from 1ms).
+- ``agent.load_model_from_server`` no longer affects logging.
+
+Changed
+-------
+- The endpoint ``POST /model/train`` no longer supports specifying an output directory
+  for the trained model using the field ``out``. Instead you can choose whether you
+  want to save the trained model in the default model directory (``models``)
+  (default behavior) or in a temporary directory by specifying the
+  ``save_to_default_model_directory`` field in the training request.
 
 [1.3.3] - 2019-09-13
 ^^^^^^^^^^^^^^^^^^^^
@@ -18,6 +117,7 @@ Fixed
 - Default one-hot representation for label featurization inside ``EmbeddingIntentClassifier`` if label features don't exist.
 - Policy ensemble no longer incorrectly wrings "missing mapping policy" when
   mapping policy is present.
+- "test" from ``utter_custom_json`` now correctly saved to tracker when using telegram channel
 
 Removed
 -------
@@ -38,6 +138,7 @@ Changed
 -------
 - Pin gast to == 0.2.2
 
+
 [1.3.0] - 2019-09-05
 ^^^^^^^^^^^^^^^^^^^^
 
@@ -125,6 +226,47 @@ Removed
 -------
 - Removed ``--report`` argument from ``rasa test nlu``. All output files are stored in the ``--out`` directory.
 
+[1.2.11] - 2019-10-09
+^^^^^^^^^^^^^^^^^^^^^
+
+Added
+-----
+- Support for passing a CA file for SSL certificate verification via the
+  --ssl-ca-file flag
+
+[1.2.10] - 2019-10-08
+^^^^^^^^^^^^^^^^^^^^^
+
+Added
+-----
+- Added support for RabbitMQ TLS authentication. The following environment variables
+  need to be set:
+  ``RABBITMQ_SSL_CLIENT_CERTIFICATE`` - path to the SSL client certificate (required)
+  ``RABBITMQ_SSL_CLIENT_KEY`` - path to the SSL client key (required)
+  ``RABBITMQ_SSL_CA_FILE`` - path to the SSL CA file (optional, for certificate
+  verification)
+  ``RABBITMQ_SSL_KEY_PASSWORD`` - SSL private key password (optional)
+- Added ability to define the RabbitMQ port using the ``port`` key in the
+  ``event_broker`` endpoint config.
+
+[1.2.9] - 2019-09-17
+^^^^^^^^^^^^^^^^^^^^
+
+Fixed
+-----
+- Correctly pass SSL flag values to x CLI command (backport of
+
+
+[1.2.8] - 2019-09-10
+^^^^^^^^^^^^^^^^^^^^
+
+Fixed
+-----
+- SQL tracker events are retrieved ordered by timestamps. This fixes interactive
+  learning events being shown in the wrong order. Backport of ``1.3.2`` patch
+  (PR #4427).
+
+
 [1.2.7] - 2019-09-02
 ^^^^^^^^^^^^^^^^^^^^
 

docs/_static/spec/rasa.yml

@@ -69,9 +69,9 @@ paths:
       operationId: getStatus
       tags:
       - Server Information
-      summary: Status of the currently loaded Rasa model
+      summary: Status of the Rasa server
       description: >-
-        Information about the currently loaded Rasa model.
+        Information about the server and the currently loaded Rasa model.
       responses:
         200:
           description: Success
@@ -98,6 +98,10 @@ paths:
                     type: string
                     description: Path of the loaded model
                     example: 20190429-103105.tar.gz
+                  num_active_training_jobs:
+                    type: integer
+                    description: Number of running training processes
+                    example: 2
         401:
           $ref: '#/components/responses/401NotAuthenticated'
         403:
@@ -1229,15 +1233,16 @@ components:
           $ref: '#/components/schemas/NLUTrainingData'
         stories:
           $ref: '#/components/schemas/StoriesTrainingData'
-        out:
-          type: string
-          description: Output directory
-          example: models
         force:
           type: boolean
           description: >-
             Force a model training even if the data has not changed
           example: false
+        save_to_default_model_directory:
+          type: boolean
+          description:  >-
+            If `true` (default) the trained model will be saved in the default model
+            directory, if `false` it will be saved in a temporary directory
       required: ["config"]
 
     NLUTrainingData:

docs/api/lock-stores.rst

@@ -0,0 +1,68 @@
+:desc: Messages that are being processed lock Rasa for a given conversation ID to
+  ensure that multiple incoming messages for that conversation do not interfere with
+  each other. Rasa provides multiple implementations to maintain conversation locks.
+
+.. _lock-stores:
+
+Lock Stores
+===========
+
+.. edit-link::
+
+Rasa uses a ticket lock mechanism to ensure that incoming messages for a given
+conversation ID are processed in the right order, and locks conversations while
+messages are actively processed. This means multiple Rasa servers can
+be run in parallel as replicated services, and clients do not necessarily need to
+address the same node when sending messages for a given conversation ID.
+
+.. contents::
+
+InMemoryLockStore (default)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+:Description:
+    ``InMemoryLockStore`` is the default lock store. It maintains conversation locks
+    within a single process.
+
+    .. note::
+      This lock store should not be used when multiple Rasa servers are run
+      parallel.
+
+:Configuration:
+    To use the ``InMemoryTrackerStore`` no configuration is needed.
+
+RedisLockStore
+~~~~~~~~~~~~~~
+
+:Description:
+    ``RedisLockStore`` maintains conversation locks using Redis as a persistence layer.
+    This is the recommended lock store for running a replicated set of Rasa servers.
+
+:Configuration:
+    To set up Rasa with Redis the following steps are required:
+
+    1. Start your Redis instance
+    2. Add required configuration to your ``endpoints.yml``
+
+        .. code-block:: yaml
+
+            lock_store:
+                type: "redis"
+                url: <url of the redis instance, e.g. localhost>
+                port: <port of your redis instance, usually 6379>
+                password: <password used for authentication>
+                db: <number of your database within redis, e.g. 0>
+
+    3. To start the Rasa Core server using your Redis backend, add the ``--endpoints``
+    flag, e.g.:
+
+        .. code-block:: bash
+
+            rasa run -m models --endpoints endpoints.yml
+
+:Parameters:
+    - ``url`` (default: ``localhost``): The url of your redis instance
+    - ``port`` (default: ``6379``): The port which redis is running on
+    - ``db`` (default: ``0``): The number of your redis database
+    - ``password`` (default: ``None``): Password used for authentication
+      (``None`` equals no authentication)

docs/core/retrieval-actions.rst

@@ -13,6 +13,12 @@ Retrieval Actions
    We introduce experimental features to get feedback from our community, so we encourage you to try it out!
    However, the functionality might be changed or removed in the future.
    If you have feedback (positive or negative) please share it with us on the `forum <https://forum.rasa.com>`_.
+   Also, currently we do not support adding new annotations in Rasa X if your training data contains retrieval actions.
+   Once we have gathered enough feedback and we're happy with the training data format, we'll add support for training response retrieval models in Rasa X.
+
+.. note::
+   There is an in-depth blog post `here <https://blog.rasa.com/response-retrieval-models/>`_ about how to use retrieval
+   actions for handling single turn interactions.
 
 .. contents::
    :local:
@@ -52,6 +58,9 @@ You can cover all of these with a single story where the above intents are group
 A retrieval action uses the output of a :ref:`response-selector` component from NLU which learns a
 retrieval model to predict the correct response from a list of candidate responses given a user message text.
 
+
+.. _retrieval-training-data:
+
 Training Data
 ^^^^^^^^^^^^^
 
@@ -95,6 +104,10 @@ This is a key difference to the response templates in your domain file.
     to the training process. The contents of it cannot be a part of the file which contains training data for other
     components of NLU.
 
+.. note::
+    As shown in the above examples, ``/`` symbol is reserved as a delimiter to separate retrieval intents from response text identifier. Make sure not to
+    use it in the name of your intents.
+
 Config File
 ^^^^^^^^^^^
 
@@ -227,4 +240,4 @@ Example result:
     }
 
 If the ``retrieval_intent`` parameter of a particular response selector was left to its default value,
-the corresponding response selector will be identified as ``default`` in the returned output.
+the corresponding response selector will be identified as ``default`` in the returned output.

docs/core/stories.rst

@@ -74,6 +74,12 @@ to predict the next action based on a *combination* of both the intent and
 entities (you can, however, change this behavior using the
 :ref:`use_entities <use_entities>` attribute).
 
+.. warning::
+    ``/`` symbol is reserved as a delimiter to separate retrieval intents from response text identifiers.
+    Refer to ``Training Data Format`` section of :ref:`retrieval-actions` for more details on this format.
+    If any of the intent names contain the delimiter, the file containing these stories will be considered as a training
+    file for :ref:`response-selector` model and will be ignored for training Core models.
+
 Actions
 ~~~~~~~
 While writing stories, you will encounter two types of actions: utterances

docs/index.rst

@@ -88,6 +88,7 @@ Understand messages, hold conversations, and connect to messaging channels and A
    api/tracker
    api/tracker-stores
    api/event-brokers
+   api/lock-stores
    api/training-data-importers
    api/featurization
    migration-guide

docs/migration-guide.rst

@@ -50,6 +50,8 @@ General
   an entity set, this will influence the weighted precision and f1-score quite a bit. From now on we
   exclude ``no-entity`` from the evaluation. The overall metrics now only include proper entities. You
   might see a drop in the performance scores when running the evaluation again.
+- ``/`` is reserved as a delimiter token to distinguish between retrieval intent and the corresponding response text
+  identifier. Make sure you don't include ``/`` symbol in the name of your intents.
 
 .. _migration-to-rasa-1.0:
 

docs/nlu/components.rst

@@ -847,5 +847,6 @@ DucklingHTTPExtractor
           # if not set the default timezone of Duckling is going to be used
           # needed to calculate dates from relative expressions like "tomorrow"
           timezone: "Europe/Berlin"
-
-
+          # Timeout for receiving response from http url of the running duckling server
+          # if not set the default timeout of duckling http url is set to 3 seconds. 
+          timeout : 3

docs/nlu/training-data-format.rst

@@ -72,6 +72,10 @@ Lookup tables may be specified either directly as lists or as txt files containi
 .. note::
     The common theme here is that common examples, regex features and lookup tables merely act as cues to the final NLU model by providing additional features to the machine learning algorithm during training. Therefore, it must not be assumed that having a single example would be enough for the model to robustly identify intents and/or entities across all variants of that example.
 
+.. note::
+    ``/`` symbol is reserved as a delimiter to separate retrieval intents from response text identifiers. Make sure not to
+    use it in the name of your intents.
+
 JSON Format
 -----------