core: worker: retry if the queue doesn't exist when ALL_INFRA is set#16726
Draft
eckter wants to merge 1 commit into
Draft
core: worker: retry if the queue doesn't exist when ALL_INFRA is set#16726eckter wants to merge 1 commit into
eckter wants to merge 1 commit into
Conversation
Signed-off-by: Eloi Charpentier <eloi.charpentier.42@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sometimes, especially when the queues haven't been used in a while, starting a
coreworker can result in an error where it doesn't seem to find the queue. In can be annoying when running core outside of docker compose, as it doesn't restart automatically. We generally need to put messages in the queue before starting the core process.The error is
com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'core-req-all'This PR adds a simple retry mechanism when core is running in
ALL_INFRA. Otherwise we can rely on osrdyne to restart the worker anyway.But I realized when testing the fix that I don't entirely understand why it fails. Deleting the queue manually leads to entirely different results than waiting for a while without message, it isn't recreated automatically.
It seems that the queue isn't actually deleted.I'm also surprised in that it only seem to need a single retry, the second attempt generally works.This PR would still be an improvement but there might be a better solution, I'd like to discuss this a bit further.
The fact that I haven't found a way to reproduce without waiting for ~15 minutes makes it a bit difficult to test.