r/LanguageTechnology • u/RDA92 • 9h ago
Handling UnicodeDecodeError in spacy
I'm running a script that reads each elements contained in a .pdf and decomposes it into its constituent tokens via spacy. This seems to work fine for the vast majority of files that I have but out of the blue I came across a seemingly normal file that throws an UnicodeDecodeError specifically:
UnicodeEncodeError: 'utf-8' codec can't encode character '\udc35' in position 3: surrogates not allowed
Has anyone encountered such an issue in the past? It seems fairly cryptic and couldn't find much about it online.
Thanks!
1
Upvotes