Python 3.13 alpha 1 removes _PyLong_New()

In August 2023, I removed the _PyLong_New() function as part of My plan to clarify private vs public C API functions in Python 3.13.

In October, Python 3.13.0 alpha 1 was released without this function. Sergey B Kirpichev reported that the gmpy2 project uses _PyLong_New() and asked how to replace the removed function. He created issue gh-111415: Consider restoring _PyLong_New() function as public.

Python 3.13 alpha 2 restores _PyLong_New()

In November, the private _PyLong_New() function has been restored in Python 3.13 alpha 2 which was released at November 22.

Add public function PyLong_GetDigits()

In June 2024, Sergey B Kirpichev opened the [C API Working Group] decision issue #31: Add public function PyLong_GetDigits(). API:

const digits* PyLong_GetDigits(PyObject* obj, Py_ssize_t *ndigits)

I disliked this API since it's too close to the exact implementation. The API cannot be implemented in an efficient way if implementation details change.

For example, in the future, CPython might adopt tagged pointers for small integers and so don't have a concrete array of digits.

There was a call for a different API to address these issues.

PyLong_Export() and PyLong_Import() functions

// Layout API
typedef struct PyLongLayout {
    uint8_t bits_per_digit;
    uint8_t digit_size;
    int8_t word_endian;
    int8_t array_endian;
} PyLongLayout;

const PyLongLayout PyLong_LAYOUT;

// Export API
typedef struct PyLong_DigitArray {
    PyObject *obj;
    int negative;
    Py_ssize_t ndigits;
    const Py_digit *digits;
} PyLong_DigitArray;

int PyLong_AsDigitArray(PyObject *obj, PyLong_DigitArray *array)
void PyLong_FreeDigitArray(PyLong_DigitArray *array)

// Import API
PyLongWriter* PyLongWriter_Create(int negative, Py_ssize_t ndigits, Py_digit **digits)
PyObject* PyLongWriter_Finish(PyLongWriter *writer)

PEP 757

In September, Sergey and me wrote PEP 757 – C API to import-export Python integers: the discussion got 80 messages.

It was proposed to use an union for PyLongExport with a kind member to select the format (small integer or digit array). The idea was abandonned in the meanwhile.

For small integers, there is no warranty that they will be stored as a digit array in the future. The PyLongExport.value member (int64_t) was added to store the value of small integers.

There were two open questions:

• Should we add digits_order and endian members to sys.int_info and remove PyLong_GetNativeLayout()? The PyLong_GetNativeLayout() function returns a C structure which is more convenient to use in C than sys.int_info which uses Python objects.
• Should we use anonymous union.

It was decided to leave sys.int_info unchanged and keep PyLong_GetNativeLayout().

It was also decided to avoid anonymous union to avoid any risk of compatibility issue with old C versions.

Benchmarks measured the abstraction cost: it is between 1.04x slower and 1.27x faster. It means that the abstraction has no significant impact on the performance. In short (geometric mean):

• Export: 1.05x faster
• Import: 1.03x slower

Overwhelmed

After months of discussions and many back and forth on the API, I got overwhelmed and close to give up. Hopefully, I didn't give up.

C API Working Group and Steering Council

In October, I opened a C API Working Group vote on PEP 757: decision issue #45 which got 40 messages.

At November 28, 2024, the C API WG accepted the PEP and I submitted the PEP to the Steering Council.

One week later, at December 8, the Steering Council accepted PEP 757 as well!

Final API

After many iterations, the final API is:

// Layout API
typedef struct PyLongLayout {
    uint8_t bits_per_digit;
    uint8_t digit_size;
    int8_t digits_order;
    int8_t digit_endianness;
} PyLongLayout;

const PyLongLayout* PyLong_GetNativeLayout(void)

// Export API
typedef struct PyLongExport {
    int64_t value;
    uint8_t negative;
    Py_ssize_t ndigits;
    const void *digits;
    Py_uintptr_t _reserved;
} PyLongExport;

int PyLong_Export(PyObject *obj, PyLongExport *export_long)
void PyLong_FreeExport(PyLongExport *export_long)

// Import API
PyLongWriter* PyLongWriter_Create(int negative, Py_ssize_t ndigits, void **digits)
PyObject* PyLongWriter_Finish(PyLongWriter *writer)
void PyLongWriter_Discard(PyLongWriter *writer)

The decimal extension and Python/marshal.c have been modified to use this API.