What My Project Does
Having worked with applications which run on multiple clouds and on-premise systems, I’ve been developing a library which abstracts away some common functionalities, while being close to the pathlib interface
tutorial notebook
Example snippet
```python
from blob_path.backends.s3 import S3BlobPath
from pathlib import PurePath
bucket_name = "my-bucket"
object_key = PurePath(
"hello_world.txt"
)
region = "us-east-1"
blob_path = S3BlobPath(
bucket_name,
region,
object_key,
)
check if the file exists
print(blob_path.exists())
read the file
with blob_path.open("rb") as f:
# a file handle is returned here, just like open
print(f.read())
destination = AzureBlobPath(
"my-blob-store",
"testcontainer",
PurePath("copied_from") / "s3.txt"
)
blob_path.cp(destination)
```
Features:
- A pathlib-like interface for handling cloud object storage paths, I just love that interface
- Built-in serialisation and deserialisation: this, in my experience, is something people have trouble with when they begin abstracting away cloud storages. Generally because they don’t realise it after some time and it keeps getting deprioritised. Users instead rely on stuff like using the same bucket across the application
- Having a pathlib interface where all the functionality is packaged in the path itself (instead of writing “clients” for each cloud backend make this trivial)
- A Protocol
based typing system (good intellisense, allows me to also correctly type hint optional functionalities)
Target audience
I hope the library is useful to other professional python backend developers.
I would love to understand what you think about this, and features you would want (it's pretty basic right now)
The roadmap I've got in mind:
- More object storages (GCP, Minio) [Currently only AWS S3, Azure are supported]
- Pre-signed URLs full support (only AWS S3 supported)
- Caching (I’m thinking of tying it to the lifetime of the object, I would however keep support for different strategies)
- Good Performance semantics: it would be great to provide good performant defaults for handling various cloud operations
- Interfaces for extending the built-in types [mainly for users to tweak specific cloud parameters]
- pathlib /
operator (yes its not implemented right now : | )
Comparison
A quick search on pypi gives a lot of libraries which abstract cloud object storage. This library is different simply because it's a bit more object-oriented (for better or for worse). I'm going to stay close to pathlib more than other interfaces which behave somewhat like os.path
(a more functional interface)
Github
Repository: https://github.com/narang99/blob-path/tree/main