When a user HEADS an object in CEPH that does not exist and the user has path specific ListBucket conditions that allow listing the object's location, CEPH returns a 403 vs. a 404.
This is a problem because
- when Apache Spark starts, it HEADs the path at
spark.eventLog.dirand expects a 404; a 403 causes startup to fail. PUTing an object at the path also causes startup to fail because Spark expects a "directory". - DetlaLake fails to query due to the same issue; it HEADs the
_delta_log"directory" and fails when CEPH returns a 403.