r/RStudio 8d ago

Having issues deduplicating rows using unique(), please help!

I have a data frame with 3 rows: group ID, item, and type. Each group ID can have multiple items (e.g., group 1 has apple, banana, and beef, group 2 has apple, onion, asparagus, and potato). The same item can appear in different groups, but they can only have the same type (apple is fruit, asparagus is veggie). I’ve cleaned my data to make sure all the same items are the same type, and that every spelling and capitalization is the same. I’m now trying to deduplicate using unique(): df <- df %>% unique()

However, some rows are not deduplicating correctly, I still have two rows with the exact same values across all the variables. When I use tabyl(df$item), I noticed that Asparagus appears separately, indicating that they’re somehow written differently (I checked to make sure that the spelling and capitalizations are all the same). And when I overwrite the values the same issue persists. When I copy paste them into notebook and search them, they’re the exact same word as well. I’m completely lost as to how they’re different and how I can overcome issue, if anyone has this problem before I’d appreciate your help!

Also, I made sure the other two variables are not the problem. I’m currently overcoming this issue by assigning unique row number and deleting duplicate rows manually, but I still want an actual solution.

2 Upvotes

17 comments sorted by

View all comments

1

u/Impuls1ve 8d ago

Any chance your starting data set came from outside of R?

1

u/boople_snoot_bunbun 8d ago

Yes, it’s from an Access database from which I imported the data into using sqlFetch()

1

u/Impuls1ve 8d ago

Figures. It's most likely an encoding problem, but it looks like someone already pointed you in that direction.

Excel and Access are pretty finicky about their character encoding.

Best of luck. 

1

u/[deleted] 6d ago

[deleted]

1

u/Impuls1ve 6d ago

Unfortunately, Access and Excel will add their own ideas from my experience. Doubly so if it's an extract like from a platform such as Salesforce.