94

As we use # for inserting comments in Python, then how does Python treat:

# -*- coding: utf-8 -*-

differently?

Improve this question
8
  • 1
    this actually happens often with software. They actually do some elementary parsing of comments and look for specific commands. Another example i have in mind is Hypermesh but i am sure there are many more. Commented Jan 16, 2017 at 16:12
  • 1
    You can think of it as a preprocessor that runs before the parser that peeks at the file and decides how it should be decoded. Then the parser itself starts and skips the line because its a comment. Some unixy text editors do the same thing to know how the editor should open the file. Commented Jan 16, 2017 at 16:16
  • 5
    @Ev.Kounis Perhaps the most prominent example: en.wikipedia.org/wiki/Shebang_(Unix) Commented Jan 16, 2017 at 16:17
  • 8
    It's worth noting that the -*- parts are completely optional, as far as Python is concerned, but including them seems to be customary. The docs say it "is recognized also by GNU Emacs", which suggests that that's where it comes from (an example of what @tdelaney was saying about text editors), but I've seen it (and used it myself) in code that was never touched by Emacs. Commented Jan 16, 2017 at 16:43
  • 3
    @MartijnPieters: I've got it! If you're using IDLE on Python 2, and if your file contains non-ASCII characters (as mine often did if I added a copyright line), then it will prompt you to add an encoding declaration, using the Emacs -*- style. So that's where I picked it up from. Commented Jan 16, 2017 at 19:17

2 Answers 2

Reset to default
81

Yes, it is also a comment. And the contents of that comment carry special meaning if located at the top of the file, in the first two lines.

From the Encoding declarations documentation:

If a comment in the first or second line of the Python script matches the regular expression coding[=:]\s*([-\w.]+), this comment is processed as an encoding declaration; the first group of this expression names the encoding of the source code file. The encoding declaration must appear on a line of its own. If it is the second line, the first line must also be a comment-only line.

Note that it doesn't matter what codec should be used to read the file, as far as comments are concerned. Python would normally ignore everything after the # token, and in all accepted source code codecs the #, encoding declaration and line separator characters are encoded exactly the same as they are all supersets of ASCII. So all the parser has to do is read one line, scan for the special text in the comment, read another if needed, scan for the comment, then configure the parser to read data according to the given codec.

Given that the comment is required to be either the first or second in the file (and if it is the second line, the first line must be a comment too), this is entirely safe, as the configured codec can only make a difference to non-comment lines anyway.

Improve this answer
Sign up to request clarification or add additional context in comments.

10 Comments

So the real question becomes: why do we use # -*- coding: X -*- instead of # coding: X?
@J.C.Leitão: you don't have to. Anything that matches the regular expression would work. But if you are using Emacs as your editor, then that comment also informs that editor what codec to use.
@MartijnPieters ... are there any other like this, which adds special meaning to the comment line?
@ShravilPotdar: There's loads. There is the shebang line that Unix systems use, and the Windows py launcher will look at the same info. As mentioned, many editors can be configured using text in comments (not just what codec to use, but many other aspects as well, see the emacs and vim docs). There are probably more.
@R.M.: no, multi-byte codecs are not supported, for this very reason. From PEP 263: Any encoding which allows processing the first two lines in the way indicated above is allowed as source code encoding, this includes ASCII compatible encodings as well as certain multi-byte encodings such as Shift_JIS. It does not include encodings which use two or more bytes for all characters like e.g. UTF-16. The reason for this is to keep the encoding detection algorithm in the tokenizer simple.
|
23

See encoding declarations in the Python Reference Manual:

If a comment in the first or second line of the Python script matches the regular expression coding[=:]\s*([-\w.]+), this comment is processed as an encoding declaration; the first group of this expression names the encoding of the source code file.

(Emphasis mine)

So yes, it is a comment, a special one. It is special in that the parser will try and act on it and not ignore it as it does for comments not in the first or second line. Take, for example, an unregistered encoding declaration in a sample file decl.py:

# # -*- coding: unknown-encoding -*-
print("foo")

If you try and run this, Python will try and process it, fail and complain:

python decl.py 
  File "decl.py", line 1
SyntaxError: encoding problem: unknown-encoding
Improve this answer

3 Comments

But if you were to register unkown-encoding as an encoding, say, with a .pth file, then that codec is actually loaded and used. This provides a very nice and interesting opportunity for pre-parse code processing.
Indeed @MartijnPieters I mainly added that as a code example that Python processes the declaration, not to make any other claims for it.
github.com/dropbox/pyxl would be an example of what @MartijnPieters is referring to.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.