Currently we either throw away a table before inserting new data, or we naively append data when that data already exists. There exists an UPSERT operator via sqlachemy for some databases that implicitly updates existing data or inserts new data as needed as a single operation. We need to explore if sqlachemy supports this for sqlite3 database operations so that we can safely ingest the same data multiple times without creating duplicates.
Currently we either throw away a table before inserting new data, or we naively append data when that data already exists. There exists an UPSERT operator via sqlachemy for some databases that implicitly updates existing data or inserts new data as needed as a single operation. We need to explore if sqlachemy supports this for sqlite3 database operations so that we can safely ingest the same data multiple times without creating duplicates.