Skip to content

INTEGRITY: Continuing set.dat's processing #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 32 commits into from
Jul 18, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
161abd5
INTEGRITY: Increase character limit size for log text
ShivangNagta Jul 4, 2025
94cd1d9
INTEGRITY: Separate the additional checksum add logic from insert_fil…
ShivangNagta Jul 4, 2025
cc82c6f
INTEGRITY: Add filtering by platform for set.dat to reduce manual merge.
ShivangNagta Jul 4, 2025
599d2f1
INTEGRITY: Avoid adding a fileset as candidate if it was marked as pa…
ShivangNagta Jul 7, 2025
6ea2bba
INTEGRITY: Add additional filtering logic for glk engines
ShivangNagta Jul 7, 2025
3028d19
INTEGRITY: Add timestamp field in scan.dat and filtering support via …
ShivangNagta Jul 7, 2025
19b19c0
INTEGRITY: Add all size variants to scan.dat - size, size-r and size-rd.
ShivangNagta Jul 7, 2025
074da92
INTEGRITY: Fix clear database hang issue. Now the database is dropped…
ShivangNagta Jul 8, 2025
31b7d4f
INTEGRITY: Remove global database connection object from fileset.py, …
ShivangNagta Jul 8, 2025
e125227
INTEGRITY: Filter manual merge candidates if size mismatch.
ShivangNagta Jul 9, 2025
898ffd0
INTEGRITY: Add metadata for set.dat
ShivangNagta Jul 10, 2025
8970cd6
INTEGRITY: Add navbar with logo.
ShivangNagta Jul 10, 2025
bd3f2f4
INTEGRITY: Add modification timestamps for macfiles
ShivangNagta Jul 10, 2025
8575f8e
INTEGIRTY: Add punycode encoding for scan utlity.
ShivangNagta Jul 10, 2025
df41d7a
INTEGRITY: Fix the navbar on top.
ShivangNagta Jul 10, 2025
c38881c
INTEGRITY: Limit match fileset to 1 in remove_manual_merge_if_size_mi…
ShivangNagta Jul 14, 2025
e86f982
INTEGRITY: Improve console logging with progress update.
ShivangNagta Jul 14, 2025
7dcb20b
INTEGRITY: Remove custom recursive path split function.
ShivangNagta Jul 14, 2025
96d9cf4
INTEGRITY: Use INFORMATION_SCHEMA.COLUMNS instead of relying on error…
ShivangNagta Jul 14, 2025
4c9a5e7
INTEGRITY: Add scan processing logic.
ShivangNagta Jul 14, 2025
33cac5a
INTEGRITY: Add additional modification-time column in file table.
ShivangNagta Jul 14, 2025
90ffe1a
INTEGRITY: Additional error handling while extracing keys from scummv…
ShivangNagta Jul 14, 2025
493acb5
INTEGRITY: Traverse set.dat instead of candidate fileset while search…
ShivangNagta Jul 14, 2025
ff9934f
INTEGRITY: Add checksum based filtering in set.dat, when possible.
ShivangNagta Jul 16, 2025
ca9d4a7
INTEGRITY: Remove 'obsolete' fileset status entirely.
ShivangNagta Jul 17, 2025
a303645
INTEGRITY: Add checksum based filtering before filtering by maximum n…
ShivangNagta Jul 17, 2025
4dd7e29
INTEGRITY: Merge one of the entries from dropped duplicate entries. D…
ShivangNagta Jul 17, 2025
57df340
INTEGRITY: Merge filtering logic for glk with existing set.dat filter…
ShivangNagta Jul 17, 2025
c9dca04
INTEGRITY: Add checksum filtering before max files filtering in scan.…
ShivangNagta Jul 17, 2025
70e5e92
INTEGRITY: Parameterising all sql queries in db_functions.py
ShivangNagta Jul 17, 2025
e0a1de3
INTEGRITY: Parametrise sql queries in all files.
ShivangNagta Jul 17, 2025
d1e8466
INTEGRITY: Run ruff formatter on compute_hash.py.
ShivangNagta Jul 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions clear.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,26 +7,30 @@
import json
import os


def truncate_all_tables(conn):
# fmt: off
tables = ["filechecksum", "queue", "history", "transactions", "file", "fileset", "game", "engine", "log"]
cursor = conn.cursor()

# fmt: on

# Disable foreign key checks
cursor.execute("SET FOREIGN_KEY_CHECKS = 0")

for table in tables:
try:
cursor.execute(f"TRUNCATE TABLE `{table}`")
cursor.execute("TRUNCATE TABLE %s", (table,))
print(f"Table '{table}' truncated successfully")
except pymysql.Error as err:
print(f"Error truncating table '{table}': {err}")

# Enable foreign key checks
cursor.execute("SET FOREIGN_KEY_CHECKS = 1")


if __name__ == "__main__":
base_dir = os.path.dirname(os.path.abspath(__file__))
config_path = os.path.join(base_dir, 'mysql_config.json')
config_path = os.path.join(base_dir, "mysql_config.json")
with open(config_path) as f:
mysql_cred = json.load(f)

Expand All @@ -41,9 +45,9 @@ def truncate_all_tables(conn):
user=username,
password=password,
db=dbname, # Specify the database to use
charset='utf8mb4',
charset="utf8mb4",
cursorclass=pymysql.cursors.DictCursor,
autocommit=True
autocommit=True,
)

# Check connection
Expand All @@ -55,4 +59,4 @@ def truncate_all_tables(conn):
truncate_all_tables(conn)

# Close connection
conn.close()
conn.close()
Loading