-
-
Notifications
You must be signed in to change notification settings - Fork 109
docs: improve queueName docs to avoid high cardinality #558
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| * queue to run serially). (Default: null) | ||
| * queue to run serially). Avoid using high cardinality values (e.g., random | ||
| * strings, UUIDs, timestamps) as this degrades performance and requires | ||
| * periodic database cleanup. (Default: null) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feedback is welcome on the phrasing here - I don't feel strongly!
benjie
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
website/docs/library/add-job.md
Outdated
|
|
||
| :::warning | ||
|
|
||
| Avoid using high cardinality values (e.g., random strings, UUIDs, | ||
| timestamps) for queue names as this will create many dead queues that | ||
| degrade performance and require | ||
| [periodic database cleanup](../admin-functions.md#gc_job_queues). If you | ||
| find yourself needing to run the `GC_JOB_QUEUES` cleanup task regularly, | ||
| you're likely using queue names incorrectly. | ||
|
|
||
| ::: | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jemgillam Please confirm this renders okay. If not, move it to below the list of options.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The lines are all spaced out?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.


Description
We recently ran into an issue where our serverless database had to dramatically scale up capacity. Upon reviewing those queries, we realized that the issue was caused by Graphile Worker query:
Specifically the subquery against the
_private_job_queuestable.We were creating lots of job queues using high-cardinality values (e.g., database row IDs), which caused many thousands of dead queues to build up in this table.
Because the graphql worker polling job necessarily does a table scan on
_private_job_queues, this caused a big performance issue for us, forcing the DB to scale up.When I reviewed the documentation, I found a warning about not using randomly-generated values in
queueNamesworker/website/docs/admin-functions.md
Lines 127 to 140 in ae40946
This was helpful. However, I think it makes sense to also warn about this on the
queueNameparameter itself, so people know before there's a problem. This PR adds those warnings.Performance impact
N/A - docs only
Security impact
N/A - docs only
Checklist
yarn lint:fixpasses.yarn testpasses.RELEASE_NOTES.mdfile (if one exists).