1. Problem: Does on_commit() Execute Too Early?
Recently, in a Django project, I configured transaction.on_commit()
to execute a Celery task after the transaction is successful.
However, upon checking the logs, I noticed an unusual phenomenon.
Even though operations like
post.categories.add(...)
had not yet completed,
the Celery task insideon_commit()
was being executed.
2. Looking at the Code...
def view():
post = Post.objects.create(...)
def post_process():
with transaction.atomic():
post.categories.add(...)
post.tags.add(...)
def run_async():
translate_post.delay(post.id)
transaction.on_commit(run_async)
threading.Thread(target=post_process).start()
At first, it didn't seem like a problem...
But this threading.Thread()
was the trap.
3. Cause: Transactions are Thread Local
Django's transactions are dependent on the current thread.
This means that transaction.atomic()
and on_commit()
only apply to the transaction opened in this thread right now.
A
transaction.atomic()
opened in a new thread
has no relation to the main transaction.
4. What Happened As a Result
- The Celery task was executed "after the transaction commit," but
- that transaction was merely a small transaction within a sub-thread.
post.categories.add(...)
and others were still in progress
In other words, Celery was executed too early than intended, leading to issues with referencing incomplete data.
5. Solution
It’s simple to fix:
def view():
post = Post.objects.create(...)
with transaction.atomic():
post.categories.add(...)
post.tags.add(...)
def run_async():
translate_post.delay(post.id)
transaction.on_commit(run_async)
🚫 threading.Thread(...)
✅ Just register on_commit()
in the main thread
6. Conclusion
- Django's transactions are thread local
on_commit()
executes only after the transaction commits in the current thread- Registering in another thread will only respond to that thread's transaction
This experience was a powerful lesson.
When using transactions + Celery + threads together,
I realized I need to be very cautious.
Add a New Comment