Merge pull request #1 from diru1100/sb-api

Create Initial core PR for SpamBrainz API

Author: me-diru

README.md

@@ -1,2 +1,44 @@
-# SB_API
-SpamBrainz API using LodBrok model.
+# SpamBrainz API
+An API to classify editor account using LodBrok(keras model) in the backend. Also has an option to retrain the model for future enhancement by taking incorrect predictions into consideration as per SpamNinja feedback.
+
+### Steps to run the API: 
+
+1) Install all the dependencies needed in virtual environment: 
+
+```
+        pip install -r requirements.txt
+```
+2) In spambrainz folder, set certain environmental variables in the terminal to run the API:
+```
+         $ export FLASK_APP=sb_api.py
+         $ export FLASK_DEBUG=1 # only for development purpose
+         $ export FLASK_RUN_PORT=4321 # the api requests are sent to this port number
+```
+
+3) Install Redis: 
+```
+        $ wget http://download.redis.io/redis-stable.tar.gz
+        $ tar xvzf redis-stable.tar.gz
+        $ cd redis-stable
+        $ make
+        $ sudo make install
+```
+4) Run redis separate terminal to store the get the data sent to SB_API
+```  
+        $ redis-server 
+```
+
+5) With this all the dependencies are set to run the server, now simply run the server by
+```  
+        $ flask run
+```
+
+This should run the API in the specified port in debug mode
+
+
+
+![](spambrainz/static/images/sb_api_working.png)
+
+The detailed internal functioning of API is present [here](spambrainz/app/README.md)
+
+The request used, their details and the output are present [here](spambrainz/README.md)

requirements.txt

@@ -0,0 +1,58 @@
+absl-py==0.9.0
+aniso8601==8.0.0
+appdirs==1.4.4
+astor==0.8.1
+astunparse==1.6.3
+cachetools==4.1.1
+certifi==2020.6.20
+chardet==3.0.4
+click==7.1.2
+DateTime==4.3
+filelock==3.0.12
+Flask==1.1.2
+Flask-RESTful==0.3.8
+gast==0.3.3
+gevent==20.6.2
+google-auth==1.20.0
+google-auth-oauthlib==0.4.1
+google-pasta==0.2.0
+greenlet==0.4.16
+grpcio==1.30.0
+h5py==2.10.0
+idna==2.10
+importlib-metadata==1.7.0
+itsdangerous==1.1.0
+Jinja2==2.11.2
+Keras==2.3.1
+Keras-Applications==1.0.8
+Keras-Preprocessing==1.1.2
+Markdown==3.2.2
+MarkupSafe==1.1.1
+mbdata==25.0.4
+numpy==1.18.5
+oauthlib==3.1.0
+opt-einsum==3.3.0
+protobuf==3.12.4
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pytz==2020.1
+PyYAML==5.3.1
+redis==3.5.3
+requests==2.24.0
+requests-oauthlib==1.3.0
+rsa==4.6
+scipy==1.4.1
+six==1.15.0
+tensorboard==2.3.0
+tensorboard-plugin-wit==1.7.0
+tensorflow==2.3.0
+tensorflow-estimator==2.3.0
+termcolor==1.1.0
+uritools==3.0.0
+urlextract==1.0.0
+urllib3==1.25.10
+Werkzeug==1.0.1
+wrapt==1.12.1
+zipp==3.1.0
+zope.event==4.4
+zope.interface==5.1.0

spambrainz/README.md

@@ -0,0 +1,11 @@
+## Functioning of requests:
+
+1) There are two request codes namely classify_request.py and train_request.py for /predict and /train respectively.
+2) The classify_request.py sents a spam editor account to be classified by the lodbrok model running in the backend. The command ```python classify_request.py```  gives the following output: 
+
+![](static/images/classify_request.png)
+3) The train_request.py sents a spam editor accounts along with **verdict** given by SpamNinja as spam or not. The command ```python train_request.py```  gives the following output: 
+
+![](static/images/train_request.png)
+
+More details regarding the API functioning is written [here](app/README.md).

spambrainz/app/README.md

@@ -0,0 +1,46 @@
+## Internal Functioning of API:
+
+This is the structure in the API: 
+
+![](../static/images/api_structure.png)
+
+- The initialization of the application is done in **__ init __.py** where the flask, redis, and model instance are initialized. 
+
+- The **. ./sb_api.py** takes the above instances and runs the application. 
+   - (Note: This is to avoid circular imports in Flask)
+   
+- The **classify.py** contains classify_process() which classifies by retrieving the data stored in redis.
+    - First, it converts the given data JSON data to be ready for the model to predict on, for this:
+        - It converts the datetime object stored as string in JSON back to a datetime object with the help of string_to_dateitime function.
+        - Converts the JSON data to an np array with the help of preprocess_editor function. 
+    - After the preprocessed data is obtained, it performs the necessary predictions and stores them back into redis to be retrieved later.
+    - The editor details stored in redis are removed form the queue
+
+- The **routes.py** contains all the necessary API endpoints /predict and /train which are called when post requests are made to the API.
+    - The **/predict** endpoint:
+        - In this endpoint the input JSON data is pushed into the redis queue after converting into compatible form.
+        - The editor ids are stored before hand to retrieve the results mapped with ids stored in redis by classify_process, later on.
+        - Unfilled details such as area or bio are set as None to be compatible later on.
+        - classify_process() with size is called to load the model and classify the editor accounts retrived back from redis.
+        - The results stored in redis by classify_process are retrieved back and sent back to SpamNinja in a JSON format.
+    - The **/train** endpoint: 
+        - In this endpoint the input JSON data with additional **verdict** parameter and editor details are directly converted into compatible format(np array) to be used by model to retrain.
+        - Unfilled details such as area or bio are set as None to be compatible later on and also the datetime objects stored as string in JSON back to a datetime object with the help of string_to_dateitime function.
+        - Using preprocess_editor converts finally into an np array.
+        - The preprocessed data is sent to retrain_model function to retrain the model.
+        - If successfully retrained, a success JSON message is sent.
+
+-   The **preprocessing.py** is used to preprocess the JSON/Dict editor data into proper np array for the model to predict/train on, this uses the initial tokenizers created to convert each parameter properly.
+
+-   The **train.py** contains the retraining part of the model.
+    - The data sent to retrain_model is used to retrain the model.
+    - The learning rate of the optimizer(Adam) is set to 0.001(very low static value) to learn from new patterns while also keeping in my mind not to forget old learnings(drastic forgetting).
+    - The current model is saved as previous_lodbrok.h5 in ../statis/weights/ for future reference if we wanted to come back.
+    - Then it calls the train_model function which continues the model learning with new data, The batch is set to only 1 with 2 epochs so as to keep balanced learning.
+    - After the training is done, the new model is saved with current_lodbrok.h5 to overwrite over the old model and continue classifying new data over time.
+    - The original lodbrok weights are saved in original_lodbrok.h5 to come back and trace the progress done so far.
+    - **Benefits of using this method are:**
+        - No need to maintain any extra database to store new data sent by SpamNinja and maintain the old data on which the model is trained.
+        - There is no fixed size to the batch of data sent to the model, it can be any number of models.
+        - The old learnings of the model won't be forgotten soon due to **slow static learning rate** and also the structure of data being the same forever.
+    

spambrainz/app/__init__.py

@@ -0,0 +1,12 @@
+import flask
+import redis
+import keras
+from threading import Thread
+
+# initialize our Flask application, Redis server, and Keras model
+
+app = flask.Flask(__name__)
+db = redis.StrictRedis(host="localhost", port=6379, db=0)
+model = None
+
+from app import routes

spambrainz/app/classify.py

@@ -0,0 +1,94 @@
+from app import model, db
+from tensorflow.keras.models import load_model
+from .preprocessing import preprocess_editor
+import json
+import datetime
+import numpy as np
+import flask 
+model = model
+
+
+# initialize constants used for redis server
+EDITOR_QUEUE = "editor_queue"
+SERVER_SLEEP = 0.25
+CLIENT_SLEEP = 0.25
+
+# function used to convert string to datetime, used before preprocessing
+def string_to_datetime(string_dt):
+    if string_dt is None:
+        return None
+    return datetime.datetime(*[int(v) for v in string_dt.replace('T', '-').replace(':', '-').split('-')])
+
+# function used to retrive editor_data from redis and store the results back 
+def classify_process(size):
+    
+    print("* Loading model...")
+    global model
+    model = load_model('static/models/weights/current_lodbrok.h5')
+    print("* Model loaded")
+
+    BATCH_SIZE = size
+
+    # All the editor detials are retrived here from redis
+    queue = db.lrange(EDITOR_QUEUE, 0, BATCH_SIZE - 1)
+    
+    
+    for q in queue:
+
+        q = json.loads(q)
+        editor_id = q["id"]
+    
+        # changing string datetime to datetime objects
+        q["birth_date"] = string_to_datetime(q["birth_date"])
+        q["member_since"] = string_to_datetime(q["member_since"])
+        q["email_confirm_date"] = string_to_datetime(q["email_confirm_date"])
+        q["last_updated"] = string_to_datetime(q["last_updated"])
+        q["last_login_date"] = string_to_datetime(q["last_login_date"])
+        
+        # preprocessing the given input to get prediction
+        q = preprocess_editor(q)
+
+        # defining the structure
+        q = np.array([q])
+        
+        # only data from index 1 is considered while predicting, thus 
+        # not taking the spam value into consideration
+        predict_data = {
+            "main_input": np.array(q[:,1:10]),
+            "email_input": np.array(q[:,10]),
+            "website_input": np.array(q[:,11]),
+            "bio_input": np.array(q[:,12:]),
+        }
+
+    
+        result = model.predict(x = [
+            predict_data["main_input"], 
+            predict_data["email_input"],
+            predict_data["website_input"],
+            predict_data["bio_input"],
+        ])
+
+        # identifying the prediction done by lodbrok
+        # if closer to 1 it's a spam account else a non spam account
+        if(result[0][1]>result[0][0]) :
+            prediction = {
+                'result' : "Spam Editor Account"
+            }
+
+        else:
+            prediction = {
+                'result' : "Non Spam Editor Account"
+            }        
+
+        # converting to fit in redis
+        prediction = json.dumps(prediction)        
+
+        #storign the result in redis
+        db.set(str(editor_id), prediction)
+    
+    # remove the set of editor from our queue
+    db.ltrim(EDITOR_QUEUE, size, -1)
+
+  
+
+

spambrainz/app/preprocessing.py

@@ -0,0 +1,103 @@
+import os
+import pickle
+import numpy as np
+from keras.preprocessing.text import Tokenizer
+from urllib.parse import urlparse
+from datetime import timedelta
+from urlextract import URLExtract
+from keras.preprocessing.text import Tokenizer
+
+extractor = URLExtract()
+one_hour = timedelta(hours=1)
+
+bio_tokenizer, website_tokenizer, email_tokenizer = None, None, None
+
+# The tokenizers build by the model are loaded here to convert 
+# given input to vectors to train on
+def load_tokenizers():
+    global bio_tokenizer, website_tokenizer, email_tokenizer
+
+    with open("static/models/others/bio_tokenizer.pickle", "rb") as f:
+        bio_tokenizer = pickle.load(f)
+
+    with open("static/models/others/website_tokenizer.pickle", "rb") as f:
+        website_tokenizer = pickle.load(f)
+
+    with open("static/models/others/email_tokenizer.pickle", "rb") as f:
+        email_tokenizer = pickle.load(f)
+
+# editor preprocessing
+extractor = URLExtract()
+one_hour = timedelta(hours=1)
+
+# preprocess_editor is used to convert given data to np array for
+# the model to train on
+def preprocess_editor(editor, spam=None):    
+
+    load_tokenizers()
+    # Apparently there are users with unset member_since
+    if editor["member_since"] is not None:
+        # These shouldn't be none but you can't trust the database
+        if editor["last_updated"] is not None:
+            update_delta = (editor["last_updated"] - editor["member_since"]) / one_hour
+        else:
+            update_delta = -1
+        
+        if editor["last_login_date"] is not None:
+            login_delta = (editor["last_login_date"] - editor["member_since"]) / one_hour
+        else:
+            login_delta = -1
+        
+        # Confirm date may be None
+        if editor["email_confirm_date"] is not None:
+            conf_delta = (editor["email_confirm_date"] - editor["member_since"]) / one_hour
+        else:
+            conf_delta = -1
+    else:
+        update_delta, login_delta, conf_delta = -2, -2, -2
+    
+    # Email domain
+    email_domain = email_tokenizer.texts_to_sequences([editor["email"].split("@")[1]])[0]
+    if len(email_domain) == 0:
+        email_token = 1024
+    else:
+        email_token = email_domain[0]
+    
+    # Website domain
+    domain = urlparse(editor["website"]).hostname
+    if domain is not None:
+        website_domain = website_tokenizer.texts_to_sequences(urlparse(editor["website"]).hostname)[0]
+        if len(website_domain) == 0:
+            website_token = 1023
+        else:
+            website_token = email_domain[0]
+    else:
+        website_token = 1024
+    
+    # Bio metadata
+    if editor["bio"] is not None:
+        bio_len = len(editor["bio"])
+        bio_urls = extractor.has_urls(editor["bio"])        
+        bio = bio_tokenizer.texts_to_matrix([editor["bio"]], mode="tfidf")[0]
+    else:
+        bio_len, bio_urls = 0, 0
+        bio = np.zeros(512)
+    
+    data = np.array([
+        spam, # spam classification (used only during training the model)
+        editor["area"] is not None, # Area Set
+        editor["gender"] is not None, # Gender
+        editor["birth_date"] is not None, # Birth date set
+        editor["privs"] != 0, # Nonzero privs
+        bio_len, # Bio length
+        bio_urls, # URLs in bio
+        conf_delta, # Confirmation delta
+        update_delta, # Last updated delta
+        login_delta, # Last login delta
+        email_token, # Email domain
+        website_token, # Website domain
+    ], dtype=np.float32)
+        
+    data = np.concatenate((data, bio))
+    # print("Preproccessed data")    
+    return data