Replace manual ref counting with shared_ptr/weak_ptr

[bibrakc]

Description of changes:

Replace manual reference counting between domain, endpoint, and communicator with shared_ptr/weak_ptr. Tables hold weak_ptr as non-owning caches, comms hold shared_ptr, endpoints hold shared_ptr. Destruction cascades automatically via the shared_ptr destructor chain without requiring locks.

This eliminates release_ep(), release_domain(), the skip_lock and force_cleanup parameters, and the domain/device lock interlock during destruction. Destructors now call cleanup_resources() directly instead of relying on the manual release path.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[bibrakc]

The domain/device cleanup_resources() prints INFO messages like:

NET/OFI 1 RDMA endpoints still active at domain cleanup
NET/OFI 1 RDMA domains still active at close

This is for example in :

int nccl_net_ofi_rdma_domain_t::cleanup_resources()
{


if (!this->ep_table.empty()) {
		NCCL_OFI_INFO(NCCL_NET, "%zu RDMA endpoints still active at domain cleanup",
					 this->ep_table.size());
		err_code = this->release_all_ep();
		if (err_code != 0) {
			NCCL_OFI_WARN("Cleanup of RDMA domain failed. RC: %d, ERROR: %s",
				       err_code, fi_strerror(-err_code));
			ret = -EINVAL;
		}
	}

With the old manual ref counting, these indicated endpoints that hadn't been properly released. With the new weak_ptr design, these just mean there are stale non-owning cache entries in the table and are completely harmless. So the messages are misleading now since they suggest a problem where none exists.

Need input on what to do:

1- Either remove the check or
2- Change the message.

What do you prefer?
@bwbarrett @rauteric

Update: we discussed offline and made changes to the PR.

[bwbarrett]

Love to see all the code removal. This seems way simpler.

[bwbarrett]

I'd probably squash the 4 commits. I probably would have done the GIN EP pointer cleanup in a second commit, but your GIN EP pointer cleanup ALSO has a bunch of other things in it, which really should just be part of the first commit.

Given where you are, i'd just squash rather than try to rework some of the commits to make more sense.

[bibrakc]

@bwbarrett squash done.