[int] Add cleanDirectory utility function for general directory clean-up

[sfayer]

Hi,

Here is the reworked version of this to create a new cleanDirectory utility function and use it in a few places where it makes sense (including the HTCondorCE cleanup).

Regards,
Simon

BEGINRELEASENOTES
*Core
CHANGE: Add cleanDirectory utility function for general directory clean-up
ENDRELEASENOTES

[aldbr]

For my education, what are you trying to solve here exactly?

[sfayer]

Here is my draft for this: We're testing it on one of our test servers before I'll mark it for review. Please feel free to have a quick look and let me know if it needs any major changes before we finish testing it!

It didn't make sense for the TransformationAgent.cleanOldTransformationCache case: This would need an extra filter (to check for files that are still being used) and the whole thing would be much more complicated that the existing code.

Regards,
Simon

[sfayer]

We've done some tests of this, particularly the securityLog seems fine. It didn't cause any issue for our HTCondorCE submission either, but we use the remote schedd mode, so I'm not sure it's actually activated there.

[sfayer]

I've addressed the review comments and fixed-up one of the tests which was doing a slightly broad mock causing the latest version to fail (the is_dir check was failing as the test was mocking the parent directory to look like a file!).

I think this is ready for a (hopefully) final review now.