I'm trying to enhance and to fix the Python C API for 5 years. My first goal was to shrink the C API without breaking third party C extensions. I hid many private functions from the public functions: I moved them to the "internal C API". I also deprecated and removed many functions.
Between Python 3.5 and 3.10, 80 symbols have been removed. Python 3.10 is the first Python version exporting less symbols than its previous version!
Since Python 3.8, the C API is organized as 3 parts:
- Include/ directory: Limited API
- Include/cpython/ directory: CPython implementation details
- Include/internal/ directory: The internal API
The devguide Changing Python’s C API documentation now gives guidelines for C API additions, like avoiding borrowed references.
The limited C API got a few more functions, whereas broken and private functions have been removed. The Stable ABI is now explicitly defined and documented in the C API Stability page.
This article lists all C API changes, not only the ones done by me.
Shrink the the C API
Between Python 3.5 and 3.10, 80 symbols (functions or variables) have been removed, 3 structures have been removed, and 21 functions have been deprecated. In meanwhile, other symbols have been added to implement new Python features at each Python version.
Python 3.10 is the first Python version exporting less symbols than its previous version.
Python 3.6
Deprecate 4 functions:
- PyUnicode_AsDecodedObject()
- PyUnicode_AsDecodedUnicode()
- PyUnicode_AsEncodedObject()
- PyUnicode_AsEncodedUnicode()
Python 3.7
- Deprecate PyOS_AfterFork()
- Remove PyExc_RecursionErrorInst singleton (also removed in Python 3.6.4).
Python 3.8
Remove 3 functions:
- PyByteArray_Init()
- PyByteArray_Fini()
- PyEval_ReInitThreads()
Remove 1 structure:
- PyInterpreterState (moved to the internal C API)
Python 3.9
Remove 32 symbols:
- PyAsyncGen_ClearFreeLists()
- PyCFunction_ClearFreeList()
- PyCmpWrapper_Type
- PyContext_ClearFreeList()
- PyDict_ClearFreeList()
- PyFloat_ClearFreeList()
- PyFrame_ClearFreeList()
- PyFrame_ExtendStack()
- PyList_ClearFreeList()
- PyMethod_ClearFreeList()
- PyNoArgsFunction type
- PyNullImporter_Type
- PySet_ClearFreeList()
- PySortWrapper_Type
- PyTuple_ClearFreeList()
- PyUnicode_ClearFreeList()
- Py_UNICODE_MATCH()
- _PyAIterWrapper_Type
- _PyBytes_InsertThousandsGrouping()
- _PyBytes_InsertThousandsGroupingLocale()
- _PyDebug_PrintTotalRefs()
- _PyFloat_Digits()
- _PyFloat_DigitsInit()
- _PyFloat_Repr()
- _PyThreadState_GetFrame() (and _PyRuntime.getframe)
- _PyUnicode_ClearStaticStrings()
- _Py_AddToAllObjects()
- _Py_InitializeFromArgs()
- _Py_InitializeFromWideArgs()
- _Py_PrintReferenceAddresses()
- _Py_PrintReferences()
- _Py_tracemalloc_config
Remove 1 structure:
- PyGC_Head (moved to the internal C API)
Deprecate 15 functions:
- PyEval_CallFunction()
- PyEval_CallMethod()
- PyEval_CallObject()
- PyEval_CallObjectWithKeywords()
- PyNode_Compile()
- PyParser_SimpleParseFileFlags()
- PyParser_SimpleParseStringFlags()
- PyParser_SimpleParseStringFlagsFilename()
- PyUnicode_AsUnicode()
- PyUnicode_AsUnicodeAndSize()
- PyUnicode_FromUnicode()
- PyUnicode_WSTR_LENGTH()
- Py_UNICODE_COPY()
- Py_UNICODE_FILL()
- _PyUnicode_AsUnicode()
Python 3.10
Remove 44 symbols:
- PyAST_Compile()
- PyAST_CompileEx()
- PyAST_CompileObject()
- PyAST_Validate()
- PyArena_AddPyObject()
- PyArena_Free()
- PyArena_Malloc()
- PyArena_New()
- PyFuture_FromAST()
- PyFuture_FromASTObject()
- PyLong_FromUnicode()
- PyNode_Compile()
- PyOS_InitInterrupts()
- PyObject_AsCharBuffer()
- PyObject_AsReadBuffer()
- PyObject_AsWriteBuffer()
- PyObject_CheckReadBuffer()
- PyParser_ASTFromFile()
- PyParser_ASTFromFileObject()
- PyParser_ASTFromFilename()
- PyParser_ASTFromString()
- PyParser_ASTFromStringObject()
- PyParser_SimpleParseFileFlags()
- PyParser_SimpleParseStringFlags()
- PyParser_SimpleParseStringFlagsFilename()
- PyST_GetScope()
- PySymtable_Build()
- PySymtable_BuildObject()
- PySymtable_Free()
- PyUnicode_AsUnicodeCopy()
- PyUnicode_GetMax()
- Py_ALLOW_RECURSION
- Py_END_ALLOW_RECURSION
- Py_SymtableString()
- Py_SymtableStringObject()
- Py_UNICODE_strcat()
- Py_UNICODE_strchr()
- Py_UNICODE_strcmp()
- Py_UNICODE_strcpy()
- Py_UNICODE_strlen()
- Py_UNICODE_strncmp()
- Py_UNICODE_strncpy()
- Py_UNICODE_strrchr()
- _Py_CheckRecursionLimit
Remove 1 structure:
- _PyUnicode_Name_CAPI
Deprecate 1 function:
- PyUnicode_InternImmortal()
Moreover, PyUnicode_FromStringAndSize(NULL, size) and PyUnicode_FromUnicode(NULL, size) have been deprecated.
Statistics
Public Python symbols exported with PyAPI_FUNC() and PyAPI_DATA():
| Python | Symbols |
|---|---|
| 2.7 | 891 |
| 3.6 | 1041 (+150) |
| 3.7 | 1068 (+27) |
| 3.8 | 1105 (+37) |
| 3.9 | 1115 (+10) |
| 3.10 | 1080 (-35) |
Command used to count public symbols:
grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l
Reorganize header files
Since Python 3.8, the C API is organized as 3 parts:
- Include/ directory: Limited API
- Include/cpython/ directory: CPython implementation details
- Include/internal/ directory: The internal API
The intent is to help developers to think about if their additions must be part of the limited C API, the CPython C API or the internal C API.
Python 3.7
Creation on the Include/internal/ directory.
Python 3.8
Creation on the Include/cpython/ directory.
Python 3.10
Move 8 header files from Include/ to Include/cpython/:
- odictobject.h
- parser_interface.h
- picklebufobject.h
- pyarena.h
- pyctype.h
- pydebug.h
- pyfpe.h
- pytime.h
Python 3.10 added a Include/README.rst documentation to explain this organization and give guidelines for adding new functions. For example, new functions in the public C API must not steal references nor return borrowed references. In the meanwhile, this documentation moved to the devguide: Changing Python’s C API.
Statistics
Number of C API line numbers per Python version:
| Python | Limited API | CPython API | Internal API | Total |
|---|---|---|---|---|
| 2.7 | 12,686 (100%) | 0 | 0 | 12,686 |
| 3.6 | 16,011 (100%) | 0 | 0 | 16,011 |
| 3.7 | 16,517 (96%) | 0 | 705 (4%) | 17,222 |
| 3.8 | 13,160 (70%) | 3,417 (18%) | 2,230 (12%) | 18,807 |
| 3.9 | 12,264 (62%) | 4,343 (22%) | 3,066 (16%) | 19,673 |
| 3.10 | 10,305 (52%) | 4,513 (23%) | 5,092 (26%) | 19,910 |
Commands:
- Limited: wc -l Include/*.h
- CPython: wc -l Include/cpython/*.h
- Internal: wc -l Include/internal/*.h
Changes in the Limited C API
Between Python 3.8 and 3.10, 4 new functions have been and 14 symbols (functions or variables) have been removed from the limited C API.
The trashcan API was excluded from the limited C API since it never worked. The implementation accessed directly PyThreadState members, whereas this structure is opaque in the limited C API.
On the other side, Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() functions have been added to the limited C API. In Python 3.8, they were defined as macros accessing directly PyThreadState members. In Python 3.9, they became opaque function calls and so are now compatible with the stable ABI.
Python 3.9
Add 3 functions to the limited C API:
- Py_EnterRecursiveCall()
- Py_LeaveRecursiveCall()
- PyFrame_GetLineNumber()
Remove 14 symbols from the limited C API:
- PyFPE_START_PROTECT()
- PyFPE_END_PROTECT()
- PyThreadState_DeleteCurrent()
- PyTrash_UNWIND_LEVEL
- Py_TRASHCAN_BEGIN
- Py_TRASHCAN_BEGIN_CONDITION
- Py_TRASHCAN_END
- Py_TRASHCAN_SAFE_BEGIN
- Py_TRASHCAN_SAFE_END
- _PyTraceMalloc_NewReference()
- _Py_CheckRecursionLimit
- _Py_GetRefTotal()
- _Py_NewReference()
- _Py_ForgetReference()
Python 3.10
Add 1 function to the limited C API:
- PyUnicode_AsUTF8AndSize()
PEP 652: Maintaining the Stable ABI
Petr Viktorin wrote and implemented the PEP 652: Maintaining the Stable ABI in Python 3.10.
The Stable ABI (Application Binary Interface) for extension modules or embedding Python is now explicitly defined. The C API Stability documentation describes C API and ABI stability guarantees along with best practices for using the Stable ABI.