Database preparation

This part of the pipeline is responsible for preparing the database. The database is created using the SQLite library. The database gets the data from the CSV files and stores it in the database, which is then used by the dashboard.

database.process_lstm_files(file_pattern)

Reads and processes LSTM files matching a pattern.

Parameters:

file_pattern (str) – The file pattern to match.

Returns:

The combined DataFrame.

Return type:

pd.DataFrame

database.read_and_process_excel(file_pattern, columns_to_keep, extract_item_type=False)

Reads and processes Excel files matching a pattern, with optional extraction of item type.

Parameters:

• file_pattern (str) – The file pattern to match.

• columns_to_keep (list) – The columns to keep.

• extract_item_type (bool) – Whether to extract item type and location.

Returns:

The combined DataFrame.

Return type:

pd.DataFrame

database.save_to_sqlite(df, db_name, table_name)

Saves a DataFrame to SQLite.

Parameters:

• df (pd.DataFrame) – The DataFrame to save.

• db_name (str) – The database name.

• table_name (str) – The table name.