[Bug]: Training Fails on DataBricks Due to symlink not supported in DBFS about anomalib

[blaz-r]

Hi, this is a bit unfortunate and also causes issues on Windows. A workaround current is to comment out this section if symlink is not supported:

anomalib/src/anomalib/utils/path.py

Lines 59 to 65 in c3e0b38

	# Update the 'latest' symbolic link to point to the new version directory
	latest_link_path = root_dir / "latest"
	if latest_link_path.is_symlink() or latest_link_path.exists():
	latest_link_path.unlink()
	latest_link_path.symlink_to(new_version_dir, target_is_directory=True)
	return latest_link_path

and return new_version_dir

[CarlosNacher]

Hey, I have a related issue. It is not exactly the same but related with symlink: if I fit N models parallelly (as I have > 1 GPU and I run in different terminals the train code with different CUDA_VISIBLE_DEVICES env variable), as for each a new versioned_dir is created and this is symlinked with "latest" dir, my checkpoints don't know where to log, as each fit proccess unlinks the "latest" symlinked dir from the previous one, thus all checkpoints are logged in "latest" dir which is linked with the latests version of directories.

Maybe is it better to always return new_version_dir for the checkpoints to be logged always in numbered versiones dir? This will end up with a "latest" dir with only the actually "latests" process logged stuff, but each process will have logged in its versioned dir.