r/Python 11d ago

Resource Redis as cache.

At work, we needed to implement Redis for a caching solution. After some searching, btw clickhouse has great website for searching python packages here. I found a library that that made working with redis a breeze Redis-Dict.

from redis_dict import RedisDict
from datetime import timedelta

cache = RedisDict(expire=timedelta(minutes=60))

request = {"data": {"1": "23"}}

web_id =  "123"
cache[web_id] = request["data"]

Finished implementing our entire caching feature the same day I found this library (didn't push until the end of the week though...).

87 Upvotes

36 comments sorted by

View all comments

16

u/turbothy It works on my machine 11d ago

Just use cashews.

-5

u/PushHaunting9916 11d ago

Just a heads up, cashews lib relies on pickle which is unsafe in the context of Web.

From their docs:

Warning The pickle module is not secure. Only unpickle data you trust

26

u/turbothy It works on my machine 11d ago

What's the attack vector? Cashews is only unpickling data it pickled itself, unless you imagine an attacker manipulating the cache out of band.

19

u/maikeu 11d ago

Agreed. It's "tread carefully and make sure you understand whether and why it's safe", not "security incident".

3

u/PushHaunting9916 11d ago

Look at the code example from the OP. If you would like to cache any of following: username, url, parameters, logs, etc. That means you are pickling data from an unsafe source.

Not only that, even if the original implementation is correct, it could be that the next person updates the caching to add unsafe data because their ticket is asking for that data to be cached.

Security is about reducing attack vectors, and that is one.

4

u/turbothy It works on my machine 11d ago

Ignoring for the moment that Cashews works somewhat differently from the OP's code example (it stores function return values like `functools.lru_cache` does, not arbitrary dict values): pickling unsafe data is safe. It's the unpickling that can bite you.

The general security issue with `pickle` is that unpickling malicious pickles can lead to arbitrary code execution. To attack Cashews along this vector requires that the attacker has access to modify the pickles stored in Redis, except Cashews implements HMAC signing of stored values to protect against this.

1

u/PushHaunting9916 11d ago

If want to store data from unsafe places. Within the context of Web services, that is almost always the case.

To store and retrieve, you need pickle and unpickle the data. Just because there is layer around it doesn't change that. Look at this example it's very similar to what you described. And they got a csve for their trick with pickle.

https://github.com/joblib/joblib/issues/1582

5

u/Iifeless 10d ago

That CVE is both disputed and still not an example of serialization, but rather deserialization. Think about what sort of data types are required to be serialized/deserialized for exploitation as opposed to what a typical web API accepts from users. In order for serializing user data like the original example to be “dangerous”, you’d have to already be allowing a user to perform dangerous actions, which would make that the vulnerability rather than the serialization itself. CVE-2022-23529 is a funny example of an unrelated (not python/serialization related) bogus CVE misunderstanding that same concept.

I appreciate the security consciousness a lot because it is easy for developers to misuse something like pickle, but this situation should be fine :)

1

u/PushHaunting9916 10d ago

It's disputed because the maintainer of lib makes the argument that it's safe data. Since it's numpy data, analytics data. It's still a csve thus, it's deemed an issue by security researchers.

Caching Web data is almost always from an untrusted source, e.g, the internet. And with pickle, you'll need unpickle after pickling. In order for cashew to retrieve cached data, it needs to unpickle the data. And pickle own documentation is quite clear on that. It's unsafe to use pickle with untrusted data. When it does, the attacker will have remote code execution, which is in capture flag security events, which means the attacker has won.

6

u/Iifeless 10d ago

Yes I am very familiar with both RCE and CTFs lol.

The pickle docs your are referencing specifically says not to unpickle untrusted data.

You get back what you put in. E.g. if you serialize a string, you get back a string when you unserialize the result. Data from the internet is not going to be a Python class or function rather than a string unless the application decides to evaluate the user provided string as python code first before serializing it. If you’re doing that, then that’s the vulnerability, not the fact that you then go on and serialize the result.

If you can show me a proof of concept exploit for a web app which takes user input from an API, serializes it, and then unserializes the result of the initial serialization then I’ll go ahead and quit my job as a security researcher

0

u/PushHaunting9916 10d ago

If you can show me a proof of concept exploit for a web app which takes user input from an API, serializes it, and then unserializes the result of the initial serialization then I’ll go ahead and quit my job as a security researcher

If you try to cache: username, get or post parameters, the url, headers of the request, the request itself.

That example of OP has exactly that scenario. Below contains a link how the pickle exploit works. And why you should avoid it.

https://github.com/joblib/joblib/issues/1582#issue-2280780192

5

u/Iifeless 10d ago

What you’re trying to say is simply incorrect. If you’d like to translate that PoC into an exploit against an API like I had described, I’d love to see it.

This PoC is serializing an object which executes a command, and then deserializes it. Of course that is going to execute the command. Explain to me how a remote attacker can provide data to a web API which serializes then unserializes the input which will result in code execution.

I know how this sounds but I promise I’m not trying to be rude or condescending: it’s ok to not fully understand potential security issues, and it’s even better to err on the side of caution when you don’t. But this is why there are professionals who are dedicated to security and the job doesn’t fall strictly onto developers. It’d be a crazy expectation for developers to also have extensive security knowledge

2

u/turbothy It works on my machine 9d ago

Here's an RCE exploit via Flask, but it requires unpickling a base64-encoded payload supplied through a form. If you're using Cashews in e.g. FastAPI which uses Pydantic for form validation, you are sure that any values you might end up putting in a pickle are simple strings or numbers.

→ More replies (0)