In case of a sudden failure and crash, we don't want the process to restart and start scrapping from the beggining.
** REQUIREMENT **
1 . Add method of Database class inside db.py to store check point information
2. Let user implementing plugin decide how to use the checkpointing system.
Note: Database class uses pymongo and contains no data that is modified after creation of object. So it's safe to say it is thread safe.
In case of a sudden failure and crash, we don't want the process to restart and start scrapping from the beggining.
** REQUIREMENT **
1 . Add method of Database class inside db.py to store check point information
2. Let user implementing plugin decide how to use the checkpointing system.
Note: Database class uses pymongo and contains no data that is modified after creation of object. So it's safe to say it is thread safe.