Feb 15, 2018 · text.decode("utf-8").encode("windows-1252").decode("utf-8") Both of these will give you a unicode string. By the way - to discover how a piece of text like this has been mangled due to encoding issues, you can use chardet :

Apr 7, 2019 · I'm voting to delete this because it is not the same issue as in the OP. The question in the OP is about decoding the content of the file UnicodeDecodeError: 'utf-8' codec can't decode byte, while this answer is for SyntaxError: 'unicodeescape'. Two completely different issues. The issue in this answer is already addressed here and here. As ...

Oct 25, 2013 · public static string Decode(string path) { // This StreamReader constructor defaults to UTF-8 using (StreamReader reader = new StreamReader(path)) return reader.ReadToEnd(); } I'm not sure what your Encode() method is supposed to do, since the intent seems to be to read a file as UTF-8 and then write the text back to the exact same file as UTF-8.

Aug 12, 2012 · import codecs f = codecs.open(dir+location, 'r', encoding='utf-8') txt = f.read() from that moment txt is in unicode format and you can use it everywhere in your code. If you want to generate UTF-8 files after your processing do:

Aug 23, 2013 · How to solve UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte in python Hot Network Questions An alternative proof that the reals are uncountably infinite.

In sublime, Click File -> Save with encoding -> UTF-8; VS Code: In the bottom bar of VSCode, you'll see the label UTF-8. Click it. A popup opens. Click Save with encoding. You can now pick a new encoding for that file. Then, you could read your file as usual: import pandas as pd data = pd.read_csv('file_name.csv', encoding='utf-8')

Nov 24, 2013 · When you want to change the encoding you always go from one into another. So you might go from Mac Roman to UTF-8 or from ASCII to UTF-8. It's as important to know the desired output encoding as the current source encoding. For example if you have Mac Roman and you decode it from UTF-16 to UTF-8 you'll just make it garbled.

Sep 18, 2012 · Yes, though this is usually bad practice/dangerous, because you'll just lose characters. Better to determine or detect the encoding of the input string and decode it to unicode first, then encode as UTF-8, for example: str.decode('cp1252').encode('utf-8') –

As you can see, the string "\xc3" has been turned into a single character. This is now an 8-bit string, UTF-8 encoded. To get Unicode: >>> x.decode('utf-8') u'Capit\xe1n\n' Gregg Lind asked: I think there are some pieces missing here: the file f2 contains: hex: 0000000: 4361 7069 745c 7863 335c 7861 316e Capit\xc3\xa1n

May 3, 2018 · Second, UTF-8 is an encoding standard to encode Unicode string to bytes. There are many encoding standards out there (e.g. UTF-16 , ASCII , SHIFT-JIS , etc.). When the client sends data to your server and they are using UTF-8 , they are sending a bunch of bytes not str .