-
Notifications
You must be signed in to change notification settings - Fork 13
Use a context manager for all write actions to prevent indefinite database locks #77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #77 +/- ##
==========================================
- Coverage 88.71% 87.35% -1.37%
==========================================
Files 6 6
Lines 567 585 +18
Branches 69 67 -2
==========================================
+ Hits 503 511 +8
- Misses 43 53 +10
Partials 21 21 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this PR! I will offer an explanation as to why you are encountering this error, and a proposal to handle this more appropriately.
There are three important pieces of context:
- The DB is locked if there are any pending (i.e. un-committed) transactions.
- In the current implementation, we only commit at the end of each public method for a performance benefit. This is because committing several transactions at once is faster than committing after each transaction, as each commit triggers a disk write op.
- However, in the
main
branch, if an exception is raised in a private method, then the public method calling the private method immediately exits, and theself.con.commit()
line is never reached. This is means that after an exception is raised, the DB is still locked as there are pending transactions.
While your current implementation does fix the issue, it is essentially ignoring failed transactions and committing all the successful ones anyways. Ideally, each set of transactions produced by a public method are either all committed or all rolled back; no in-between. I expect this will prevent subtle DB integrity issues that are tricky to debug.
With this info, I believe this proposal would handle the error more correctly:
- Wrap all public methods in the Connection context manager, e.g.
def index(self, path: str) -> str:
with self.con:
...
- Remove all calls to
self.con.commit()
at the end of each public method.
This means that public methods will still raise a sqlite3.IntegrityError
exception if the operation was invalid, but should also ensure the DB is not locked after the exception is raised. I believe this is preferable to the current PR, which never raises the exception back to the consumer.
Thanks, @dlqqq! That all sounds good.
I actually did this originally (hence the branch name), but felt I was changing things too much. I'm glad you proposed going this direction. I've updated the PR, incorporating all of your suggestions. Thank you for the detailed explanation! Let me know if I missed anything or you'd like additional changes. |
for more information, see https://pre-commit.ci
#76 uses a unit test to expose an issue we see when a "write" to the database raises an exception. The database becomes locked indefinitely, preventing any other connects from talking to the database.
In this PR, I wrap all "write" actions with try/except clauses, log any errors, and free up the lock.