Utf 8 python ошибка

PEP: Python3 and UnicodeDecodeError

This is a PEP describing the behaviour of Python3 on UnicodeDecodeError. It’s a draft, don’t hesitate to comment it. This document suppose that my patch to allow bytes filenames is accepted which is not the case today.

While I was writing this document I found poential problems in Python3. So here is a TODO list (things to be checked):

• FIXME: When bytearray is accepted or not?
• FIXME: Allow bytes/str mix for shutil.copy*()? The ignore callback will get bytes or unicode?

Can anyone write a section about bytes encoding in Unicode using escape sequence?

What is the best tool to work on a PEP? I hate email threads, and I would prefer SVN / Mercurial / anything else.

Python3 and UnicodeDecodeError for the command line, environment variables and filenames

Introduction

Python3 does its best to give you texts encoded as a valid unicode characters strings. When it hits an invalid bytes sequence (according to the used charset), it has two choices: drops the value or raises an UnicodeDecodeError. This document present the behaviour of Python3 for the command line, environment variables and filenames.

Example of an invalid bytes sequence: ::

>>> str(b'xff', 'utf8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff (...)

whereas the same byte sequence is valid in another charset like ISO-8859-1: ::

>>> str(b'xff', 'iso-8859-1')
'ÿ'

Default encoding

Python uses «UTF-8» as the default Unicode encoding. You can read the default charset using sys.getdefaultencoding(). The «default encoding» is used by PyUnicode_FromStringAndSize().

A function sys.setdefaultencoding() exists, but it raises a ValueError for charset different than UTF-8 since the charset is hardcoded in PyUnicode_FromStringAndSize().

Command line

Python creates a nice unicode table for sys.argv using mbstowcs(): ::

$ ./python -c 'import sys; print(sys.argv)' 'Ho hé !'
['-c', 'Ho hé !']

On Linux, mbstowcs() uses LC_CTYPE environement variable to choose the encoding. On an invalid bytes sequence, Python quits directly with an exit code 1. Example with UTF-8 locale:

$ python3.0 $(echo -e 'invalid:xff')
Could not convert argument 1 to string

Environment variables

Python uses «_wenviron» on Windows which are contains unicode (UTF-16-LE) strings.  On other OS, it uses «environ» variable and the UTF-8 charset. It drops a variable if its key or value is not convertible to unicode. Example:

env -i HOME=/home/my PATH=$(echo -e "xff") python
>>> import os; list(os.environ.items())
[('HOME', '/home/my')]

Both key and values are unicode strings. Empty key and/or value are allowed.

Python ignores invalid variables, but values still exist in memory. If you run a child process (eg. using os.system()), the «invalid» variables will also be copied.

Filenames

Introduction

Python2 uses byte filenames everywhere, but it was also possible to use unicode filenames. Examples:

• os.getcwd() gives bytes whereas os.getcwdu() always returns unicode

• os.listdir(unicode) creates bytes or unicode filenames (fallback to bytes on UnicodeDecodeError), os.readlink() has the same behaviour

• glob.glob() converts the unicode pattern to bytes, and so create bytes filenames
• open() supports bytes and unicode

Since listdir() mix bytes and unicode, you are not able to manipulate easily filenames:

>>> path=u'.'
>>> for name in os.listdir(path):
...  print repr(name)
...  print repr(os.path.join(path, name))
...
u'valid'
u'./valid'
'invalidxff'
Traceback (most recent call last):
...
File "/usr/lib/python2.5/posixpath.py", line 65, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff (...)

Python3 supports both types, bytes and unicode, but disallow mixing them. If you ask for unicode, you will always get unicode or an exception is raised.

You should only use unicode filenames, except if you are writing a program fixing file system encoding, a backup tool or you users are unable to fix their broken system.

Windows

Microsoft Windows since Windows 95 only uses Unicode (UTF-16-LE) filenames. So you should only use unicode filenames.

Non Windows (POSIX)

POSIX OS like Linux uses bytes for historical reasons. In the best case, all filenames will be encoded as valid UTF-8 strings and Python creates valid unicode strings. But since system calls uses bytes, the file system may returns an invalid filename, or a program can creates a file with an invalid filename.

An invalid filename is a string which can not be decoded to unicode using the default file system encoding (which is UTF-8 most of the time).

A robust program will have to use only the bytes type to make sure that it can open / copy / remove any file or directory.

Filename encoding

Python use:

• «mbcs» on Windows
• or «utf-8» on Mac OS X
• or nl_langinfo(CODESET) on OS supporting this function
• or UTF-8 by default

«mbcs» is not a valid charset name, it’s an internal charset saying that Python will use the function MultiByteToWideChar() to decode bytes to unicode. This function uses the current codepage to decode bytes string.

You can read the charset using sys.getfilesystemencoding(). The function may returns None if Python is unable to determine the default encoding.

PyUnicode_DecodeFSDefaultAndSize() uses the default file system encoding, or UTF-8 if it is not set.

On UNIX (and other operating systems), it’s possible to mount different file systems using different charsets. sys.getdefaultencoding() will be the same for the different file systems since this encoding is only used between Python and the Linux kernel, not between the kernel and the file system which may uses a different charset.

Display a filename

Example of a function formatting a filename to display it to human eyes: ::

from sys import getfilesystemencoding
def format_filename(filename):
    return str(filename, getfilesystemencoding(), 'replace')

Example: format_filename(‘rxffport.doc’) gives ‘r�port.doc’ with the UTF-8 encoding.

Functions producing filenames

Policy: for unicode arguments: drop invalid bytes filenames; for bytes arguments: return bytes

• os.listdir()
• glob.glob()

This behaviour (drop silently invalid filenames) is motivated by the fact to if a directory of 1000 files only contains one invalid file, listdir() fails for the whole directory. Or if your directory contains 1000 python scripts (.py) and just one another document with an invalid filename (eg. r�port.doc), glob.glob(‘*.py’) fails whereas all .py scripts have valid filename.

Policy: for an unicode argument: raise an UnicodeDecodeError on invalid filename; for an bytes argument: return bytes

• os.readlink()

Policy: create unicode directory or raise an UnicodeDecodeError

• os.getcwd()

Policy: always returns bytes

• os.getcwdb()

Functions for filename manipulation

Policy: raise TypeError on bytes/str mix

• os.path.*(), eg. os.path.join()
• fnmatch.*()

Functions accessing files

Policy: accept both bytes and str

• io.open()
• os.open()
• os.chdir()
• os.stat(), os.lstat()
• os.rename()
• os.unlink()
• shutil.*()

os.rename(), shutil.copy*(), shutil.move() allow to use bytes for an argment, and unicode for the other argument

bytearray

In most cases, bytearray() can be used as bytes for a filename.

Unicode normalisation

Unicode characters can be normalized in 4 forms: NFC, NFD, NFKC or NFKD. Python does never normalize strings (nor filenames). No operating system does normalize filenames. So the users using different norms will be unable to retrieve their file. Don’t panic! All users use the same norm.

Use unicodedata.normalize() to normalize an unicode string.

Summary: The UnicodeEncodeError generally occurs while encoding a Unicode string into a certain coding. Only a limited number of Unicode characters are mapped to strings. Thus, any character that is not-represented / mapped will cause the encoding to fail and raise UnicodeEncodeError. To avoid this error use the encode(utf-8) and decode(utf-8) functions accordingly in your code.

You might be using handling an application code that needs to deal with multilingual data or web content that has plenty of emojis and special symbols. In such situations, you will possibly come across numerous problems relating to Unicode data. But python has well-defined options to deal with Unicode characters and we shall be discussing them in this article.

What is Unicode?

Unicode is a standard that facilitates character encoding using variable bit encoding. I am sure, you must have heard of ASCII if you are into the world of computer programming. ASCII represents 128 characters while Unicode defines 2²¹ characters. Thus, Unicode can be regarded as a superset of ASCII. If you are interested in having an in-depth look at Unicode, please follow this link. 
Click on Unicode:- U+1F40D to find out what it represents! (Try it!!!?)

What is a UnicodeEncodeError?

The best way to grasp any concept is to visualize it with an example. So let us have a look at an example of the UnicodeEncodeError.

u = 'é'
print("Integer value for é: ", ord(u))
print("Converting the encoded value of é to Integer Equivalent: ", chr(233))
print("UNICODE Representation of é: ", u.encode('utf-8'))
print("ASCII Representation of é: ", u.encode('ascii'))

Output

Integer value for é:  233
Converting the encoded value of é to Integer Equivalent:  é
UNICODE Representation of é:  b'xc3xa9'
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    print("ASCII Representation of é: ",u.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character 'xe9' in position 0: ordinal not in range(128)

In the above code, when we tried to encode the character é to its Unicode value we got an output but while trying to convert it to the ASCII equivalent we encountered an error. The error occurred because ASCII only allows 7-bit encoding and it cannot represent characters outside the range of [0..128]. 

You now have an essence of what the UnicodeEncodeError looks like. Before discussing how we can avoid such errors, I feel that there is a dire need to discuss the following concepts:

Encoding and Decoding

The process of converting human-readable data into a specified format, for the secured transmission of data is known as encoding. Decoding is the opposite of encoding that is to convert the encoded information to normal text (human-readable form).

In Python, 

• encode() is an inbuilt method used for encoding. Incase no encoding is specified, UTF-8 is used as default. 
• decode() is an inbuilt method used for decoding. 

Example:

u = 'Πύθωνος'
print("UNICODE Representation of é: ", u.encode('utf-8'))

Output:

UNICODE Representation of é:  b'xcexa0xcfx8dxcexb8xcfx89xcexbdxcexbfxcfx82'

The following diagram should make things a little easier:

Codepoint

Unicode maps the codepoint to their respective characters. So, what do we mean by a codepoint? 

• Codepoints are numerical values or integers used to represent a character. 
• The Unicode code point for é is U+00E9 which is integer 233. When you encode a character and print it, you will generally get its hexadecimal representation as an output instead of its binary equivalent (as seen in the examples above).
• The byte sequence of a code point is different in different encoding schemes. For eg: the byte sequence for é in UTF-8 is xc3xa9 while in UTF-16 is xffxfexe9x00.

Please have a look at the following program to get a better grip on this concept:

u = 'é'
print("INTEGER value for é: ", ord(u))
print("ENCODED Representation of é in UTF-8: ", u.encode('utf-8'))
print("ENCODED Representation of é in UTF-16: ", u.encode('utf-16'))

Output

INTEGER value for é:  233
ENCODED Representation of é in UTF-8:  b'xc3xa9'
ENCODED Representation of é in UTF-16:  b'xffxfexe9x00'

Now that we have an overview of Unicode and UnicodeEncodeError, let us discuss how we can deal with the error and avoid it in our program.

➥ Problem: Given a string/text to be written in a text File; how to avoid the UnicodeEncodeError and write given text in the text file.

Example:

f = open('demo.txt', 'w')
f.write('να έχεις μια όμορφη μέρα')
f.close()

Output:

Traceback (most recent call last):
  File "uniError.py", line 2, in <module>
    f.write('να έχεις μια όμορφη μέρα')
  File "C:UsersShubham-PCAppDataLocalProgramsPythonPython38-32libencodingscp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>

✨ Solution 1: Encode String Before Writing To File And Decode While Reading

You cannot write Unicode to a file directly. This will raise an UnicodeEncodeError. To avoid this you must encode the Unicode string using the encode() function and then write it to the file as shown in the program below:

text = u'να έχεις μια όμορφη μέρα'
# write in binary mode to avoid TypeError
f = open('demo.txt', 'wb')
f.write(text.encode('utf8'))
f.close()
f = open('demo.txt', 'rb')
print(f.read().decode('utf8'))

Output:

να έχεις μια όμορφη μέρα

✨ Solution 2: Open File In utf-8

If you are using Python 3 or higher, all you need to do is open the file in utf-8, as Unicode string handling is already standardized in Python 3.

text = 'να έχεις μια όμορφη μέρα'
f = open('demo2.txt', 'w', encoding="utf-8")
f.write(text)
f.close()

Output:

✨ Solution 3: Using The Codecs Module

Another approach to deal with the UnicodeEncodeError is using the codecs module. 

Let us have a look at the following code to understand how we can use the codecs module:

import codecs

f = codecs.open("demo3.txt", "w", encoding='utf-8')
f.write("να έχεις μια όμορφη μέρα")
f.close()

Output:

✨ Solution 4: Using Python’s unicodecsv Module

If you are dealing with Unicode data and using a csv file for managing your data, then the unicodecsv module can be really helpful. It is an extended version of Python 2’s csv module and helps the user to handle Unicode data without any hassle.

Since the unicodecsv module is not a part of Python’s standard library, you have to install it before using it. Use the following command to install this module:

$ pip install unicodecsv

Let us have a look at the following example to get a better grip on the unicodecsv module:

import unicodecsv as csv

with open('example.csv', 'wb') as f:
    writer = csv.writer(f, encoding='utf-8')
    writer.writerow(('English', 'Japanese'))
    writer.writerow((u'Hello', u'こんにちは'))

Output:

Conclusion

In this article, we discussed some of the important concepts regarding Unicode character and then went on to learn about the UnicodeEncodeError and finally discussed the methods that we can use to avoid it. I hope by the end of this article you can handle Unicode characters in your python code with ease. 

Please subscribe and stay tuned for more interesting articles!

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

🚀 If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!