Add detailed documentation of API

- Added detailed documentation about installation, request results,
    and internal working of API.
- Added some more comments where ever necessary.
- Added an original_lodbrok weight for backup.
- Removed unnecessary files and images.

Author: me-diru

README.md

@@ -1,21 +1,40 @@
-# SB_API
-Editor account prediction using LodBrok model in the backend.
+# SpamBrainz API
+An API to classify editor account using LodBrok(keras model) in the backend. Also has an option to retrain the model for future enhancement by taking incorrect predictions into consideration as per SpamNinja feedback.
 
-Install all the dependencies needed in virtual environment: 
+### Steps to run the API: 
+
+1) Install all the dependencies needed in virtual environment: 
 
 ```
-pip install -r requirements.txt
+        pip install -r requirements.txt
+```
+2) In spambrainz folder, set certain environmental variables in the terminal to run the API:
+```
+         $ export FLASK_APP=sb_api.py
+         $ export FLASK_DEBUG=1 # only for development purpose
+         $ export FLASK_RUN_PORT=4321 # the api requests are sent to this port number
 ```
 
-Run the flask application from root folder in virutal environment:
-
+3) Install Redis: 
 ```
-python run_keras_server.py
+        $ wget http://download.redis.io/redis-stable.tar.gz
+        $ tar xvzf redis-stable.tar.gz
+        $ cd redis-stable
+        $ make
+        $ sudo make install
+```
+4) Run redis separate terminal to store the get the data sent to SB_API
+```  
+        $ redis-server 
+```
+
+5) With this all the dependencies are set to run the server, now simply run the server by
+```  
+        $ flask run
 ```
-Go to http://localhost:5000/static/editor.html
 
-Enter editor account detials, press prediction to get prediciton.
+This should run the API in the specified port in debug mode
 
-This is a sample prediction done using LodBrok: 
+The detailed internal functioning of API is present [here](spambrainz/app/README.md)
 
-![](/spambrainz/static/images/prediciton.png)
+The request used their details and the output are present [here](spambrainz/README.md)

spambrainz/README.md

@@ -0,0 +1,11 @@
+## Functioning of requests:
+
+1) There are two request codes namely classify_request.py and train_request.py for /predict and /train respectively.
+2) The classify_request.py sents a spam editor account to be classified by the lodbrok model running in the backend. The command ```python classify_request.py```  gives the following output: 
+
+![](static/images/classify_request.png)
+3) The train_request.py sents a spam editor accounts along with **verdict** given by SpamNinja as spam or not. The command ```python train_request.py```  gives the following output: 
+
+![](static/images/train_request.png)
+
+More details regarding the API functioning is written [here](app/README.md).

spambrainz/app/README.md

@@ -0,0 +1,46 @@
+## Internal Functioning of API:
+
+This is the structure in the API: 
+
+![](../static/images/api_structure.png)
+
+- The initialization of the application is done in **__ init __.py** where the flask, redis, and model instance are initialized. 
+
+- The **. ./sb_api.py** takes the above instances and runs the application. 
+   - (Note: This is to avoid circular imports in Flask)
+   
+- The **classify.py** contains classify_process() which classifies by retrieving the data stored in redis.
+    - First, it converts the given data JSON data to be ready for the model to predict on, for this:
+        - It converts the datetime object stored as string in JSON back to a datetime object with the help of string_to_dateitime function.
+        - Converts the JSON data to an np array with the help of preprocess_editor function. 
+    - After the preprocessed data is obtained, it performs the necessary predictions and stores them back into redis to be retrieved later.
+    - The editor details stored in redis are removed form the queue
+
+- The **routes.py** contains all the necessary API endpoints /predict and /train which are called when post requests are made to the API.
+    - The **/predict** endpoint:
+        - In this endpoint the input JSON data is pushed into the redis queue after converting into compatible form.
+        - The editor ids are stored before hand to retrieve the results mapped with ids stored in redis by classify_process, later on.
+        - Unfilled details such as area or bio are set as None to be compatible later on.
+        - classify_process() with size is called to load the model and classify the editor accounts retrived back from redis.
+        - The results stored in redis by classify_process are retrieved back and sent back to SpamNinja in a JSON format.
+    - The **/train** endpoint: 
+        - In this endpoint the input JSON data with additional **verdict** parameter and editor details are directly converted into compatible format(np array) to be used by model to retrain.
+        - Unfilled details such as area or bio are set as None to be compatible later on and also the datetime objects stored as string in JSON back to a datetime object with the help of string_to_dateitime function.
+        - Using preprocess_editor converts finally into an np array.
+        - The preprocessed data is sent to retrain_model function to retrain the model.
+        - If successfully retrained, a success JSON message is sent.
+
+-   The **preprocessing.py** is used to preprocess the JSON/Dict editor data into proper np array for the model to predict/train on, this uses the initial tokenizers created to convert each parameter properly.
+
+-   The **train.py** contains the retraining part of the model.
+    - The data sent to retrain_model is used to retrain the model.
+    - The learning rate of the optimizer(Adam) is set to 0.001(very low static value) to learn from new patterns while also keeping in my mind not to forget old learnings(drastic forgetting).
+    - The current model is saved as previous_lodbrok.h5 in ../statis/weights/ for future reference if we wanted to come back.
+    - Then it calls the train_model function which continues the model learning with new data, The batch is set to only 1 with 2 epochs so as to keep balanced learning.
+    - After the training is done, the new model is saved with current_lodbrok.h5 to overwrite over the old model and continue classifying new data over time.
+    - The original lodbrok weights are saved in original_lodbrok.h5 to come back and trace the progress done so far.
+    - **Benefits of using this method are:**
+        - No need to maintain any extra database to store new data sent by SpamNinja and maintain the old data on which the model is trained.
+        - There is no fixed size to the batch of data sent to the model, it can be any number of models.
+        - The old learnings of the model won't be forgotten soon due to **slow static learning rate** and also the structure of data being the same forever.
+    

spambrainz/app/classify.py

@@ -47,7 +47,9 @@ def classify_process():
 
     # defining the structure
     queue = np.array([queue])
-            
+    
+    # only data from index 1 is considered while predicting, thus 
+    # not taking the spam value into consideration
     predict_data = {
         "main_input": np.array(queue[:,1:10]),
         "email_input": np.array(queue[:,10]),

spambrainz/app/preprocessing.py

@@ -84,7 +84,7 @@ def preprocess_editor(editor, spam=None):
         bio = np.zeros(512)
 
     data = np.array([
-        spam, # spam classification
+        spam, # spam classification (used only during training the model)
         editor["area"] is not None, # Area Set
         editor["gender"] is not None, # Gender
         editor["birth_date"] is not None, # Birth date set

spambrainz/app/routes.py

@@ -42,10 +42,10 @@ def predict():
 
             # the classification done is retrived form redis
             output = db.get(editor_id)
-              
+            output =  json.loads(output)
+            output["id"] = editor_id
             if output is not None:
-
-                data["predictions"] = json.loads(output)                   
+                data["predictions"] = output              
                 db.delete(editor_id)                    
                 data["success"] = True