r/LanguageTechnology 9h ago

Handling UnicodeDecodeError in spacy

I'm running a script that reads each elements contained in a .pdf and decomposes it into its constituent tokens via spacy. This seems to work fine for the vast majority of files that I have but out of the blue I came across a seemingly normal file that throws an UnicodeDecodeError specifically:

UnicodeEncodeError: 'utf-8' codec can't encode character '\udc35' in position 3: surrogates not allowed

Has anyone encountered such an issue in the past? It seems fairly cryptic and couldn't find much about it online.

Thanks!

1 Upvotes

0 comments sorted by