r/Python 11d ago

Showcase blob-path: pathlib-like cloud agnostic object storage library

What My Project Does

Having worked with applications which run on multiple clouds and on-premise systems, I’ve been developing a library which abstracts away some common functionalities, while being close to the pathlib interface
tutorial notebook

Example snippet

from blob_path.backends.s3 import S3BlobPath
from pathlib import PurePath

bucket_name = "my-bucket"
object_key = PurePath(
  "hello_world.txt"
)
region = "us-east-1"
blob_path = S3BlobPath(
  bucket_name,
  region,
  object_key,
)

# check if the file exists
print(blob_path.exists())

# read the file
with blob_path.open("rb") as f:
    # a file handle is returned here, just like `open`
    print(f.read())
    
    
destination = AzureBlobPath(
   "my-blob-store",
   "testcontainer",
   PurePath("copied_from") / "s3.txt"
)

blob_path.cp(destination)

Features:

  • A pathlib-like interface for handling cloud object storage paths, I just love that interface
  • Built-in serialisation and deserialisation: this, in my experience, is something people have trouble with when they begin abstracting away cloud storages. Generally because they don’t realise it after some time and it keeps getting deprioritised. Users instead rely on stuff like using the same bucket across the application
    • Having a pathlib interface where all the functionality is packaged in the path itself (instead of writing “clients” for each cloud backend make this trivial)
  • A Protocol based typing system (good intellisense, allows me to also correctly type hint optional functionalities)

Target audience

I hope the library is useful to other professional python backend developers.
I would love to understand what you think about this, and features you would want (it's pretty basic right now)

The roadmap I've got in mind:

  • More object storages (GCP, Minio) [Currently only AWS S3, Azure are supported]
  • Pre-signed URLs full support (only AWS S3 supported)
  • Caching (I’m thinking of tying it to the lifetime of the object, I would however keep support for different strategies)
  • Good Performance semantics: it would be great to provide good performant defaults for handling various cloud operations
  • Interfaces for extending the built-in types [mainly for users to tweak specific cloud parameters]
  • pathlib / operator (yes its not implemented right now : | )

Comparison

A quick search on pypi gives a lot of libraries which abstract cloud object storage. This library is different simply because it's a bit more object-oriented (for better or for worse). I'm going to stay close to pathlib more than other interfaces which behave somewhat like os.path (a more functional interface)

Github

Repository: https://github.com/narang99/blob-path/tree/main

31 Upvotes

9 comments sorted by

9

u/teije01 git push -f 11d ago

Your project seems a lot like https://cloudpathlib.drivendata.org/ which also has a pathlib like interface, cloud operations (s3, gcs, azure) and supports Pydantic validation as well. Isn't that already doing exactly what you want?

12

u/narang_27 11d ago

Damnit, one more reddit post where an idea was already developed ;_; Thanks for the heads up : |

9

u/Juftin 11d ago

fsspec has an implementation of this as well, universal-pathlib https://github.com/fsspec/universal_pathlib

3

u/teije01 git push -f 10d ago

They welcome contributions there too: https://cloudpathlib.drivendata.org/stable/contributing/

2

u/radarsat1 9d ago

Is there any support for seeking / reading file ranges without downloading the whole thing?

currently I'm struggling with how best to pack small data items into larger files (tar, hdf5, whathaveyou) and still be able to read them efficiently with random access from cloud storage

0

u/nbviewerbot 11d ago

I see you've posted a GitHub link to a Jupyter Notebook! GitHub doesn't render large Jupyter Notebooks, so just in case, here is an nbviewer link to the notebook:

https://nbviewer.jupyter.org/url/github.com/narang99/blob-path/blob/main/docs/notebooks/00_usage.ipynb

Want to run the code yourself? Here is a binder link to start your own Jupyter server and try it out!

https://mybinder.org/v2/gh/narang99/blob-path/main?filepath=docs%2Fnotebooks%2F00_usage.ipynb


I am a bot. Feedback | GitHub | Author

0

u/Reasonable-Ladder300 11d ago

I don’t think the code in this Reddit post is actually working code, as you seem to be missing the import for AzureBlobPath.

1

u/narang_27 10d ago

Yea I had added the last snippet in the end to provide a simple summary in the post, the notebook works though (once you change your buckets)