Statistics

# All commits
$ git log --after=2016-12-31 --before=2017-04-01 --reverse --branches='*' --author=Stinner > 2017Q1
$ grep '^commit ' 2017Q1|wc -l
121

# Exclude merges
$ git log --no-merges --after=2016-12-31 --before=2017-04-01 --reverse --branches='*' --author=Stinner|grep '^commit '|wc -l
105

# master branch (excluding merges)
$ git log --no-merges --after=2016-12-31 --before=2017-04-01 --reverse --author=Stinner origin/master|grep '^commit '|wc -l
98

# Only merges
$ git log --merges --after=2016-12-31 --before=2017-04-01 --reverse --branches='*' --author=Stinner|grep '^commit '|wc -l
16

Statistics: 98 commits in the master branch, 16 merge commits (done using Mercurial before the migration to GitHub, and then converted to Git), and 7 other commits (likely backports), total: 121 commits.

Optimization

With the work done in 2016 on FASTCALL, it became much easier to optimize code by using the new FASTCALL API.

Median +- std dev: 121 ns +- 5 ns -> 82.8 ns +- 1.0 ns: 1.46x faster (-31%)

$ ./python -m perf timeit \
    -s 'import struct; pack=struct.pack' 'pack("i", 1)' \
    --compare-to=../default-ref/python
Median +- std dev: 119 ns +- 1 ns -> 76.8 ns +- 0.4 ns: 1.56x faster (-36%)
Significant (t=295.91)

Tricky bug

try:
    return context.handle('assertRaises', args, kwargs)
finally:
    # bpo-23890: manually break a reference cycle
    context = None

FASTCALL optimizations

FASTCALL is my project to avoid temporary tuple to pass positional arguments and avoid temporary dictionary to pass keyword arguments when calling a function. It optimizes function calls in general.

I continued work on FASTCALL to optimize code further and use FASTCALL in more cases.

Stack consumption

A FASTCALL micro-optimization was blocked by Serhiy Storchaka because it increased the C stack consumption. In the past, I never analyzed the C stack consumption. Since I wanted to get this micro-optimization merged, I tried to reduce the consumption.

At the beginning, I wrote a function to measure the C stack consumption in a reliable way. It took me a few iterations.

Table showing the C stack consumption in bytes, and the difference compared to Python 3.5 (last release before I started working on FASTCALL):

Table showing the number of function calls before a stack overflow, and the difference compared to Python 3.5:

Python 3.7 is the best of 2.7, 3.5, 3.6 and 3.7: lowest stack consumption and maximum number of calls (before a stack overflow) ;-)

Changes:

• call_method() now uses _PyObject_FastCall(). Issue #29233: Replace the inefficient _PyObject_VaCallFunctionObjArgs() with _PyObject_FastCall() in call_method() and call_maybe().
• Issue #29227: Inline call_function() into _PyEval_EvalFrameDefault() using Py_LOCAL_INLINE to reduce the stack consumption.
• Issue #29234: Inlining _PyStack_AsTuple() into callers increases their stack consumption, Disable inlining to optimize the stack consumption. Add _Py_NO_INLINE: use __attribute__((noinline)) of GCC and Clang.

Contributions

• Issue #28961: Fix unittest.mock._Call helper: don't ignore the name parameter anymore. Patch written by Jiajun Huang.
• Prohibit implicit C function declarations. Issue #27659: use -Werror=implicit-function-declaration when possible (GCC and Clang, but it depends on the compiler version). Patch written by Chi Hsuan Yen.

os.urandom() and getrandom()

As usual, I had fun with os.urandom() in this quarter (see my previous article on urandom: PEP 524: os.urandom() now blocks on Linux in Python 3.6).

The glibc developers succeeded to implement a function getrandom() in glibc 2.25 (February 2017) to expose the "new" Linux getrandom() syscall which was introduced in Linux 3.17 (August 2014). Read the LWN article: The long road to getrandom() in glibc.

I created the issue #29157 because my os.urandom() implementation wasn't ready for the addition of a getrandom() function on Linux. My implementation using the getrandom() function didn't handle the ENOSYS error (syscall not supported), when Python is compiled on a recent kernel and glibc, but run on an older kernel and glibc.

I rewrote the code to prefer getrandom() over getentropy():

• dev_urandom() now calls py_getentropy(). Prepare the fallback to support getentropy() failure and falls back on reading from /dev/urandom.
• Simplify dev_urandom(). pyurandom() is now responsible to call getentropy() or getrandom(). Enhance also dev_urandom() and pyurandom() documentation.
• getrandom() is now preferred over getentropy(). The glibc 2.24 now implements getentropy() on Linux using the getrandom() syscall. But getentropy() doesn't support non-blocking mode. Since getrandom() is tried first, it's not more needed to explicitly exclude getentropy() on Solaris. Replace: "if defined(HAVE_GETENTROPY) && !defined(sun)" with "if defined(HAVE_GETENTROPY)"
• Enhance py_getrandom() documentation. py_getentropy() now supports ENOSYS, EPERM & EINTR

IMHO the main enhancement was the documentation (comments) of the code. The main function pyrandom() now has this long comment:

Read random bytes:

• Return 0 on success
• Raise an exception (if raise is non-zero) and return -1 on error

Used sources of entropy ordered by preference, preferred source first:

• CryptGenRandom() on Windows
• getrandom() function (ex: Linux and Solaris): call py_getrandom()
• getentropy() function (ex: OpenBSD): call py_getentropy()
• /dev/urandom device

Read from the /dev/urandom device if getrandom() or getentropy() function is not available or does not work.

Prefer getrandom() over getentropy() because getrandom() supports blocking and non-blocking mode: see the PEP 524. Python requires non-blocking RNG at startup to initialize its hash secret, but os.urandom() must block until the system urandom is initialized (at least on Linux 3.17 and newer).

Prefer getrandom() and getentropy() over reading directly /dev/urandom because these functions don't need file descriptors and so avoid ENFILE or EMFILE errors (too many open files): see the issue #18756.

Only the getrandom() function supports non-blocking mode.

Only use RNG running in the kernel. They are more secure because it is harder to get the internal state of a RNG running in the kernel land than a RNG running in the user land. The kernel has a direct access to the hardware and has access to hardware RNG, they are used as entropy sources.

Note: the OpenSSL RAND_pseudo_bytes() function does not automatically reseed its RNG on fork(), two child processes (with the same pid) generate the same random numbers: see issue #18747. Kernel RNGs don't have this issue, they have access to good quality entropy sources.

If raise is zero:

• Don't raise an exception on error
• Don't call the Python signal handler (don't call PyErr_CheckSignals()) if a function fails with EINTR: retry directly the interrupted function
• Don't release the GIL to call functions.

Migration to GitHub

In February 2017, the Mercurial repository was converted to Git and the development of CPython moved to GitHub at https://github.com/python/cpython/. I helped to polish the migration in early days:

• Rename README to README.rst and enhance formatting
• bpo-29527: Don't treat warnings as error in Travis docs job
• Travis CI: run rstlint.py in the docs job. Currently, http://buildbot.python.org/all/buildslaves/ware-docs buildbot is only run as post-commit. For example, bpo-29521 (PR#41) introduced two warnings, unnotified by the Travis CI docs job. Modify the docs job to run toosl/rstlint.py. Fix also the two minor warnings which causes the buildbot slave to fail. Doc/Makefile: set PYTHON to python3.
• Add Travis CI and Codecov badges to README.
• Exclude myself from mention-bot. I made changes in almost all CPython files last 5 years, so mention-bot asks me to review basically all pull requests. I simply don't have the bandwidth to review everything, sorry! I prefer to select myself which PR I want to follow.
• bpo-27425: Add .gitattributes, fix Windows tests. Mark binary files as binay in .gitattributes to not translate newline characters in Git repositories on Windows.

Enhancements

• Issue #29259: python-gdb.py now also looks for PyCFunction in the current frame, not only in the older frame. python-gdb.py now also supports method-wrapper (wrapperobject) objects (Issue #29367).
• Issue #26273: Document the new TCP_USER_TIMEOUT and TCP_CONGESTION constants
• bpo-29919: Remove unused imports found by pyflakes. Make also minor PEP8 coding style fixes on modified imports.
• bpo-29887: Test normalization now fails if download fails; fix also a ResourceWarning.

Security

• Backport for Python 3.4. Issues #27850 and #27766: Remove 3DES from ssl default cipher list and add ChaCha20 Poly1305. See the CVE-2016-2183: Sweet32 attack (DES, 3DES) vulnerability.

regrtest

regrtest is the runner of the Python test suite. Changes:

• regrtest: don't fail immediately if a child does crash. Issue #29362: Catch a crash of a worker process as a normal failure and continue to run next tests. It allows to get the usual test summary: single line result (OK/FAIL), total duration, etc.
• Fix regrtest -j0 -R output: write also dots into stderr, instead of stdout.

Bugfixes

• Issue #29140: Fix hash(datetime.time). Fix time_hash() function: replace DATE_xxx() macros with TIME_xxx() macros. Before, the hash function used a wrong value for microseconds if fold is set (equal to 1).
• Issue #29174, #26741: Fix subprocess.Popen.__del__() on Python shutdown. subprocess.Popen.__del__() now keeps a strong reference to warnings.warn() function. The change allows to log the warning late at Python finalization. Before the warning was ignored or logged an error instead of the warning.
• Issue #25591: Fix test_imaplib if the module ssl is missing.
• Fix script_helper.run_python_until_end(): copy the SYSTEMROOT environment variable. Windows requires at least the SYSTEMROOT environment variable to start Python. If run_python_until_end() doesn't copy SYSTEMROOT, the function always fail on Windows.
• Fix datetime.fromtimestamp(): check bounds. Issue #29100: Fix datetime.fromtimestamp() regression introduced in Python 3.6.0: check minimum and maximum years.
• Fix test_datetime on system with 32-bit time_t. Issue #29100: Catch OverflowError in the new test_timestamp_limits() test.
• Fix test_datetime on Windows. Issue #29100: On Windows, datetime.datetime.fromtimestamp(min_ts) fails with an OSError in test_timestamp_limits().
• bpo-29176: Fix the name of the _curses.window class. Set name to _curses.window instead of _curses.curses window with a space!?
• bpo-29619: os.stat() and os.DirEntry.inodeo() now convert inode (st_ino) using unsigned integers to support very large inodes (larger than 2^31).