Overview
Example errors:
Traceback (most recent call last):
File "unicode_ex.py", line 3, in
print str(a) # this throws an exception
UnicodeEncodeError: 'ascii' codec can't encode character u'xa1' in position 0: ordinal not in range(128)
This issue happens when Python can’t correctly work with a string variable.
Strings can contain any sequence of bytes, but when Python is asked to work with the string, it may decide that the string contains invalid bytes.
In these situations, an error is often thrown that mentions ordinal not in range, or codec can't encode character, or codec can't decode character.
Here’s a bit of code that may reproduce the error in Python 2:
a='xa1'
print(a + ' <= problem')
unicode(a)
Initial Steps Overview
-
Check Python version
-
Determine codec and character
Detailed Steps
1) Check Python version
The Python version you are using is significant.
You can determine the Python version by running:
python --version
or, if you have access to the running code, by logging it:
print(sys.version)
The major number (2 or 3) is the number you are interested in.
It is expected that you are using Python2.
2) Determine interpreting codec and character
Get this from the error message:
UnicodeEncodeError: 'ascii' codec can't encode character u'xa1' in position 0: ordinal not in range(128)
In this case, the code is ascii and the character is the hex character A1.
What is happening here is that Python is trying to interpret a string, and expects that the bytes in that string are legal for the format it’s expecting. In this case, it’s expecting a string composed of ASCII bytes. These bytes are in the range 0-127 (ie 8 bytes). The hex byte A1 is 161 in decimal, and is therefore out of range.
When Python comes to interpret this string in a context that requires a codec (for example, when calling the unicode function), it tries to ‘encode’ it with the codec, and can hit this problem.
3) Determine desired codec
You need to figure out how the bytes should be interpreted.
Most often in everyday use (eg web scraping or document ingestion), this is utf-8.
Once you have determined the desired codec, solution A may help you.
Solutions List
A) Decode the string
Solutions Detail
A) Decode the string
If you have a string s that you want to interpret as utf-8 data, you can try:
s = s.decode('utf-8')
to re-encode the string with the appropriate codec.
Further Information
Owner
Ian Miell
Иногда на нашем сервере выскакивает следующая ошибка:
UnicodeEncodeError: ‘ascii’ codec can’t encode character u’u200e’ in position 13: ordinal not in range(128)
Ошибка: порядковый номер вне диапазона (128)
Причина: это ошибка, вызванная проблемой с кодировкой китайских символов в Python, в основном вызванной символом u200e
естьУправляющие символы обозначают надписи слева направо, Это не пробел, полностью невидимый, символ без ширины, мы обычно не видим его на веб-страницах.
аналогичен управляющим символам формата Unicode, таким как «писать метку справа налево» ( u200F) и «писать метку слева направо» ( u200E), нулевая ширина Соединитель ( u200D) и не-коннектор нулевой ширины ( uFEFF) управляют визуальным отображением текста, что важно для правильного отображения некоторых неанглийских текстов.
Решение: добавьте следующий блок операторов в заголовок файла, в котором расположен код Python.
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
Если вы добавите приведенный выше блок кода, чтобы представить проблему сбоя функции печати в python,Затем замените приведенный выше блок кода следующим блоком кода
import sys # здесь просто ссылка на sys, перезагружается только перезагрузка
stdi,stdo,stde=sys.stdin,sys.stdout,sys.stderr
reload(sys) # При ссылке при импорте,Функция setdefaultencoding удаляется после вызова системой, поэтому ее необходимо перезагрузить один раз.
sys.stdin,sys.stdout,sys.stderr=stdi,stdo,stde
Several errors can arise when an attempt to change from one datatype to another is made. The reason is the inability of some datatype to get casted/converted into others. One of the most common errors during these conversions is Unicode Encode Error which occurs when a text containing a Unicode literal is attempted to be encoded bytes. This article will teach you how to fix UnicodeEncodeError in Python.
Why does the UnicodeEncodeError error arise?
An error occurs when an attempt is made to save characters outside the range (or representable range) of an encoding scheme because code points outside the encoding scheme’s upper bound (for example, ASCII has a 256 range) do not exist. An error would be produced by values greater than +127 or -128. To solve the issue, the string would need to be encoded using an encoding technique that permitted representation of that code point. UTF-8 (Unicode Transformation-8-bit), UTF-16, UTF-32, ASCII, and others are examples of frequently used encodings. UTF-8 would often fix this problem.
For demonstration, the same error would be reproduced and then fixed:
Python3
a = 'geeksforgeeks1234567xa0'.encode("ASCII")
print(a)
Output:
Traceback (most recent call last):
File “C:/Users/test.py”, line 1, in <module>
b = ‘geeksforgeeks1234567xa0’.encode(“ASCII”)
UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘xa0’ in position 20: ordinal not in range(128)
How to solve this UnicodeEncodeError?
The error is the same as the one in hand. The error arose as an attempt to represent a character was made, which was outside the range of the ASCII encoding system. i.e., ASCII could only represent character values between the range -128 to 127, but xa0 = 128, which is outside the range of ASCII. This led to the error. To rectify this error, we have to encode the text in a scheme that allows more code points (range) than ASCII. UTF-8 would serve this purpose.
Python3
a = 'geeksforgeeks1234567xa0'.encode("UTF-8")
print(a)
Output:
b'geeksforgeeks1234567xc2xa0'
The program was executed this time because the string was encoded by a standard that allowed encoding code points greater than 128. Due to this, the character xa0 (code point 128) got converted to xc2xa0, a two-byte representation.
Similarly, the error UnicodeEncodeError could be resolved by encoding to a format such as UTF-16/32, etc.
Python3
a = 'geeksforgeeks1234567xa0'.encode("UTF-16")
print(a, end="nnn")
a = 'geeksforgeeks1234567xa0'.encode("UTF-32")
print(a)
Output:
b’xffxfegx00ex00ex00kx00sx00fx00ox00rx00gx00ex00ex00kx00sx001x002x003x004x005x006x007x00xa0x00′
b’xffxfex00x00gx00x00x00ex00x00x00ex00x00x00kx00x00x00sx00x00x00fx00x00x00ox00x00x00rx00x00x00gx00x00x00ex00x00x00ex00x00x00kx00x00x00sx00x00x001x00x00x002x00x00x003x00x00x004x00x00x005x00x00x006x00x00x007x00x00x00xa0x00x00x00′
Error Description:
UnicodeEncodeError: ‘ascii’ codec can’t encode character u’xa0′ in position 20: ordinal not in range(128)
Solution 1:
UnicodeEncodeError: 'ascii' codec can't encode character u'xa1'
in position 0: ordinal not in range(128) click below button to copy the code. By — python tutorial — team
- This error occurs when we pass a Unicode string containing non-English characters (Unicode characters beyond 128) to something that expects an ASCII bytestring.
- The default encoding for a Python bytestring is ASCII, «which handles exactly 128 (English) characters».
- This is why trying to convert Unicode characters beyond 128 produces the error.
- The good news is that we can encode Python bytestrings in other encodings besides ASCII.
- Django’s smart_str function in the django.utils.encoding module, converts a Unicode string to a bytestring using a default encoding of UTF-8.
Here is an example using the built-in function, str:
a = u'xa1'
print str(a) # this throws an exception click below button to copy the code. By — python tutorial — team
Traceback (most recent call last):
File "unicode_ex.py", line 3, in
print str(a) # this throws an exception
UnicodeEncodeError: 'ascii' codec can't encode character u'xa1' in position 0: ordinal not in range(128)Solution 2:
- It will try to read file in ascii resulting in all too common: ‘ascii’ codec can’t encode character u’xa0′ in position 111: ordinal not in range(128)
- After lots of trial and error we found a workaround that works. First of all check if we have this problem by executing:
import sys sys.getdefaultencoding()
click below button to copy the code. By — python tutorial — team
- if it comes back with ‘ascii’ then read on.
- Default encoding need to be changed. However this is only possible when sys module is reloaded.
Here is a complete solution:
import sys; reload(sys); sys.setdefaultencoding("utf8") click below button to copy the code. By — python tutorial — team
Solution 3:
- We need to read the Python Unicode HOWTO This error is the very first example ,
- Basically, stop using str to convert from unicode to encoded text / bytes.
- Instead, properly use .encode() to encode the string:
p.agent_info = u' '.join((agent_contact, agent_telno)).encode('utf-8').strip()
click below button to copy the code. By — python tutorial — team
- Or work entirely in unicode.
Solution 4:
- After googling around we figured the following and it helped. python 2.7 is in use.
# encoding=utf8
import sys
reload(sys)
sys.setdefaultencoding('utf8') click below button to copy the code. By — python tutorial — team
Solution 5:
- We just had this problem, and Google led me here, so just to add to the general solutions here, this is what worked for us:
# 'value' contains the problematic data
unic = u''
unic += value
value = unic click below button to copy the code. By — python tutorial — team
Learn python — python tutorial — python unicode encode error — python examples — python programs