Skip to content

Commit c033a36

Browse files
committed
feat: initialize ML pipeline project with API, training, and monitoring components
- Add core pipeline scripts for ingestion, training, scoring, deployment, diagnostics, and reporting - Implement Flask API (`app.py`, `wsgi.py`) and API calling utilities (`apicalls.py`) - Include configuration file (`config.json`) and project dependencies (`requirements.txt`) - Add datasets for source, practice, and test scenarios - Add end-to-end pipeline runner (`fullprocess.py`) - Include project metadata files (`.gitignore`, LICENSE)
0 parents  commit c033a36

22 files changed

Lines changed: 1035 additions & 0 deletions

.gitignore

Lines changed: 591 additions & 0 deletions
Large diffs are not rendered by default.

LICENSE

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
Copyright © 2012 - 2026, Udacity, Inc.
2+
3+
Udacity hereby grants you a license in and to the Educational Content, including
4+
but not limited to homework assignments, programming assignments, code samples,
5+
and other educational materials and tools (as further described in the Udacity
6+
Terms of Use), subject to, as modified herein, the terms and conditions of the
7+
Creative Commons Attribution-NonCommercial- NoDerivs 3.0 License located at
8+
http://creativecommons.org/licenses/by-nc-nd/4.0 and successor locations for
9+
such license (the "CC License") provided that, in each case, the Educational
10+
Content is specifically marked as being subject to the CC License.
11+
12+
Udacity expressly defines the following as falling outside the definition of
13+
"non-commercial":
14+
(a) the sale or rental of (i) any part of the Educational Content, (ii) any
15+
derivative works based at least in part on the Educational Content, or (iii)
16+
any collective work that includes any part of the Educational Content;
17+
(b) the sale of access or a link to any part of the Educational Content without
18+
first obtaining informed consent from the buyer (that the buyer is aware
19+
that the Educational Content, or such part thereof, is available at the
20+
Website free of charge);
21+
(c) providing training, support, or editorial services that use or reference the
22+
Educational Content in exchange for a fee;
23+
(d) the sale of advertisements, sponsorships, or promotions placed on the
24+
Educational Content, or any part thereof, or the sale of advertisements,
25+
sponsorships, or promotions on any website or blog containing any part of
26+
the Educational Material, including without limitation any "pop-up
27+
advertisements";
28+
(e) the use of Educational Content by a college, university, school, or other
29+
educational institution for instruction where tuition is charged; and
30+
(f) the use of Educational Content by a for-profit corporation or non-profit
31+
entity for internal professional development or training.
32+
33+
THE SERVICES AND ONLINE COURSES (INCLUDING ANY CONTENT) ARE PROVIDED "AS IS" AND
34+
"AS AVAILABLE" WITH NO REPRESENTATIONS OR WARRANTIES OF ANY KIND, EITHER
35+
EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
36+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. YOU
37+
ASSUME TOTAL RESPONSIBILITY AND THE ENTIRE RISK FOR YOUR USE OF THE SERVICES,
38+
ONLINE COURSES, AND CONTENT. WITHOUT LIMITING THE FOREGOING, WE DO NOT WARRANT
39+
THAT (A) THE SERVICES, WEBSITES, CONTENT, OR THE ONLINE COURSES WILL MEET YOUR
40+
REQUIREMENTS OR EXPECTATIONS OR ACHIEVE THE INTENDED PURPOSES, (B) THE WEBSITES
41+
OR THE ONLINE COURSES WILL NOT EXPERIENCE OUTAGES OR OTHERWISE BE UNINTERRUPTED,
42+
TIMELY, SECURE OR ERROR-FREE, (C) THE INFORMATION OR CONTENT OBTAINED THROUGH
43+
THE SERVICES, SUCH AS CHAT ROOM SERVICES, WILL BE ACCURATE, COMPLETE, CURRENT,
44+
ERROR- FREE, COMPLETELY SECURE OR RELIABLE, OR (D) THAT DEFECTS IN OR ON THE
45+
SERVICES OR CONTENT WILL BE CORRECTED. YOU ASSUME ALL RISK OF PERSONAL INJURY,
46+
INCLUDING DEATH AND DAMAGE TO PERSONAL PROPERTY, SUSTAINED FROM USE OF SERVICES.

apicalls.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import requests
2+
3+
#Specify a URL that resolves to your workspace
4+
URL = "http://127.0.0.1/"
5+
6+
7+
8+
#Call each API endpoint and store the responses
9+
response1 = #put an API call here
10+
response2 = #put an API call here
11+
response3 = #put an API call here
12+
response4 = #put an API call here
13+
14+
#combine all API responses
15+
responses = #combine reponses here
16+
17+
#write the responses to your workspace
18+
19+
20+

app.py

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
from flask import Flask, session, jsonify, request
2+
import pandas as pd
3+
import numpy as np
4+
import pickle
5+
import create_prediction_model
6+
import diagnosis
7+
import predict_exited_from_saved_model
8+
import json
9+
import os
10+
11+
12+
13+
######################Set up variables for use in our script
14+
app = Flask(__name__)
15+
app.secret_key = '1652d576-484a-49fd-913a-6879acfa6ba4'
16+
17+
with open('config.json','r') as f:
18+
config = json.load(f)
19+
20+
dataset_csv_path = os.path.join(config['output_folder_path'])
21+
22+
prediction_model = None
23+
24+
25+
#######################Prediction Endpoint
26+
@app.route("/prediction", methods=['POST','OPTIONS'])
27+
def predict():
28+
#call the prediction function you created in Step 3
29+
return #add return value for prediction outputs
30+
31+
#######################Scoring Endpoint
32+
@app.route("/scoring", methods=['GET','OPTIONS'])
33+
def stats():
34+
#check the score of the deployed model
35+
return #add return value (a single F1 score number)
36+
37+
#######################Summary Statistics Endpoint
38+
@app.route("/summarystats", methods=['GET','OPTIONS'])
39+
def stats():
40+
#check means, medians, and modes for each column
41+
return #return a list of all calculated summary statistics
42+
43+
#######################Diagnostics Endpoint
44+
@app.route("/diagnostics", methods=['GET','OPTIONS'])
45+
def stats():
46+
#check timing and percent NA values
47+
return #add return value for all diagnostics
48+
49+
if __name__ == "__main__":
50+
app.run(host='0.0.0.0', port=8000, debug=True, threaded=True)

config.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{ "input_folder_path": "practicedata", "output_folder_path": "ingesteddata", "test_data_path": "testdata", "output_model_path": "practicemodels", "prod_deployment_path": "production_deployment"}

deployment.py

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
from flask import Flask, session, jsonify, request
2+
import pandas as pd
3+
import numpy as np
4+
import pickle
5+
import os
6+
from sklearn import metrics
7+
from sklearn.model_selection import train_test_split
8+
from sklearn.linear_model import LogisticRegression
9+
import json
10+
11+
12+
13+
##################Load config.json and correct path variable
14+
with open('config.json','r') as f:
15+
config = json.load(f)
16+
17+
dataset_csv_path = os.path.join(config['output_folder_path'])
18+
prod_deployment_path = os.path.join(config['prod_deployment_path'])
19+
20+
21+
####################function for deployment
22+
def store_model_into_pickle(model):
23+
#copy the latest pickle file, the latestscore.txt value, and the ingestfiles.txt file into the deployment directory
24+
25+
26+
27+

diagnostics.py

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
2+
import pandas as pd
3+
import numpy as np
4+
import timeit
5+
import os
6+
import json
7+
8+
##################Load config.json and get environment variables
9+
with open('config.json','r') as f:
10+
config = json.load(f)
11+
12+
dataset_csv_path = os.path.join(config['output_folder_path'])
13+
test_data_path = os.path.join(config['test_data_path'])
14+
15+
##################Function to get model predictions
16+
def model_predictions():
17+
#read the deployed model and a test dataset, calculate predictions
18+
return #return value should be a list containing all predictions
19+
20+
##################Function to get summary statistics
21+
def dataframe_summary():
22+
#calculate summary statistics here
23+
return #return value should be a list containing all summary statistics
24+
25+
##################Function to get timings
26+
def execution_time():
27+
#calculate timing of training.py and ingestion.py
28+
return #return a list of 2 timing values in seconds
29+
30+
##################Function to check dependencies
31+
def outdated_packages_list():
32+
#get a list of
33+
34+
35+
if __name__ == '__main__':
36+
model_predictions()
37+
dataframe_summary()
38+
execution_time()
39+
outdated_packages_list()
40+
41+
42+
43+
44+
45+

fullprocess.py

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
2+
3+
import training
4+
import scoring
5+
import deployment
6+
import diagnostics
7+
import reporting
8+
9+
##################Check and read new data
10+
#first, read ingestedfiles.txt
11+
12+
#second, determine whether the source data folder has files that aren't listed in ingestedfiles.txt
13+
14+
15+
16+
##################Deciding whether to proceed, part 1
17+
#if you found new data, you should proceed. otherwise, do end the process here
18+
19+
20+
##################Checking for model drift
21+
#check whether the score from the deployed model is different from the score from the model that uses the newest ingested data
22+
23+
24+
##################Deciding whether to proceed, part 2
25+
#if you found model drift, you should proceed. otherwise, do end the process here
26+
27+
28+
29+
##################Re-deployment
30+
#if you found evidence for model drift, re-run the deployment.py script
31+
32+
##################Diagnostics and reporting
33+
#run diagnostics.py and reporting.py for the re-deployed model
34+
35+
36+
37+
38+
39+
40+

ingestion.py

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
import pandas as pd
2+
import numpy as np
3+
import os
4+
import json
5+
from datetime import datetime
6+
7+
8+
9+
10+
#############Load config.json and get input and output paths
11+
with open('config.json','r') as f:
12+
config = json.load(f)
13+
14+
input_folder_path = config['input_folder_path']
15+
output_folder_path = config['output_folder_path']
16+
17+
18+
19+
#############Function for data ingestion
20+
def merge_multiple_dataframe():
21+
#check for datasets, compile them together, and write to an output file
22+
23+
24+
25+
if __name__ == '__main__':
26+
merge_multiple_dataframe()

practicedata/Icon

Whitespace-only changes.

0 commit comments

Comments
 (0)