Define "GPU" as a worker resource - #1401
Conversation
| resources = dict(pair.split("=") for pair in resources) | ||
| resources = valmap(float, resources) | ||
| gpu_resources = valmap(int, itemfilter(lambda x: x != "GPU", resources)) | ||
| resources = valmap(float, itemfilter(lambda x: x == "GPU", resources)) |
There was a problem hiding this comment.
@jacobtomlinson you have written this section originally, I presume mapping values to float was done to support a definition such as "MEMORY" but since I don't see any tests or any other explicit mention in Dask-CUDA, could you comment if that's right and whether you can think of more robust ways for us to handle types here other than what I wrote above as "GPU" or NOT "GPU"?
There was a problem hiding this comment.
Honestly I don't remember the reasoning. I think dask handles these values as floats, as you say it's to support values like MEMORY or other arbitrary quantities.
pentschev
added
3 - Ready for Review
|
@pentschev - Sorry this fell through the cracks. Do you think this is good to go for 24.12? |
|
@rjzamora I actually don't know how useful this is, @jacobtomlinson point out offline that tasks ignore annotations after the dask-expr work. Doing this seems like it may resolve the original ask, but at the same time it is very misleading to users who may attempt to use those annotations for their actual purpose with Dask. I don't have a strong opinion on not going ahead, so maybe @VibhuJawa and @ryantwolf who originally requested this functionality can comment on whether it solves their issue nevertheless and whether they think it's useful to introduce this even knowing it can mislead users (ourselves included). |
Okay, fair enough. I'll just assume we are holding off on this unless the Curator team tells us otherwise.
Yes, annotations are not supported by dask-expr, and I don't personally expect that they will ever be supported. With that said, I do think worker-resources are still useful. In the long run, it may become possible to add the resource restriction to the new TaskSpec
Users who try to apply annotations should get a warning. So, I don't think it hurts much if resources are labeled or not. |
4 participants
Add "GPU" as a worker resource to each CUDA worker. This should support users in identifying whether the workers available contain a GPU resource.