What's New in Pyscript in 2024 Q1

Life moves fast! In the midst of buying a house, a rapid public launch at my day job, and the other vagaries of real life, I haven't kept up with all the stunning releases that PyScript has had in the first quarter of 2024.

That changes today!

This post will cover all of the new and changed features since version 2024.1.1. That includes PyScript versions: 2024.1.2, 2024.1.3, 2024.2.1, 2024.3.1, and 2024.3.2. Specifically, we'll be looking at the current state of all the new features as of 2024.3.2 - I'm not going to try to fully document every incremental change in the intervening versions. The goal here is to catch up to the present, and share all the great new PyScript features, not fully rehash history.

So without further ado, let's get started!

Multiple Worker Terminals

It is now possible to have multiple terminals (associated with multiple interpreters) single page:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<script type="py" worker terminal>
    print("\033[31mThis is one terminal with one interpreter\033[39m")
    import code
    code.interact()
</script>
<script type="py" worker terminal>
    print("\033[32mThis is a totally separate terminal\033[39m")
    import code
    code.interact()
</script>

Each interpreter is still limited to having a single terminal - there's no way to have multiple terminals pointing at the same worker/interpreter. That's also true of the unique main-thread interpreter (of a given type). However, each worker interpreter can now have it's own terminal.

__terminal__ attribute

Speaking of the terminal - as soon as that made it's way back into PyScript late last year, users have been asking how they can simply access the underlying xterm.js instance to tweak it. Xtermjs provides a flexible API for adjusting things like column widths, key handlers, autocomplete, etc, and users wanted to tap into those underneath PyScript.

Now, they can! The __terminal__ global variable is a deference to the xtermjs instance (when the appropriate <script> tag has the terminal attribute). Here's an example that grabs the terminal and resizes its row and column count after startup:

1
2
3
4

<script type="py" worker terminal >
    __terminal__.resize(40, 6)
    print("Lorem ipsum dolor sit amet consectetur, adipisicing elit. Exercitationem laboriosam ex porro, aliquid aperiam maiores sunt similique natus accusantium adipisci odio cupiditate quidem quos impedit ratione, fuga autem numquam accusamus. ")
</script>

py-editor setup attribute

Another longstanding feature request, particularly by users in the educational space, is the desire to have py-editors that are are not just blank Python interpreters, but rather have some site-authored code run before their turned over to the user to play in. The new setup of <py-editor> tags makes this possible.

As you'll recall, py-editor tags can have an optional env attribute, where all editors with the same env share the same underlying interpreter instance. Now, each env can also optionally have a single py-editortag on the page with the setup attribute. Any code in the "setup tag" with run before the editors in that env are enabled. The code in a "setup tag" is not visible on the page.

1
2
3
4
5
6
7

<script type="py-editor" env="one" setup>
    # Preset a variable that users might want to use
    x = 42
</script>
<script type="py-editor" env="one">
    print(x)
</script>

As a more complex example, you might include whole functions before the user gets to interact

Note: Users, myself included, have asked for the ability to have a py-config associated to an editor environment. That changed has been merged the development branch, and so is coming soon, but is NOT present in the 2024.3.2 release. Similarly, not even the pyscript package is currently present in py-editors, but that's coming too.

pyscript.fetch

One of the neat parts of writing Python for the web is the ability to grab data from the web and make use of it. To that end, PyScript now includes its own fetch() utility that makes it simple and Pythonic to to interact with Web APIs.

pyscript.fetch() takes a URL as a string, and optionally some additional keyword arguments. It then uses the browsers fetch() mechanism to grab results from the destination URL. Any keyword arguments are packaged up as a JSON object and passed along to the JavaScript fetch call.

The results of the fetch are wrapped up as a _Reponse object, which provides pythonic access to the results via a variety of (mostly async) attributes. For example, to grab the results from the fetch as text, use await my_result.text(); for a dictionary that represents JSON, use .json(). The full list includes:

Here's a working example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<script type="py" worker>
    import asyncio
    import pyscript

    async def process_data(url):
        response = await pyscript.fetch(url)
        pyscript.display(await response.text())

    t = asyncio.create_task(process_data("https://reqres.in/api/users?page=2"))
</script>

You can also chain the awaitable which retrieves the data type, as in t = await pyscript.fetch(url).json(). This only succeeds if the result of the fetch() was in the HTTP Success Range (200-299).

Of course, not everyone loves asyncio, even though it pretty much Just Works in Pyodide. But this is a prime place to make use of the async attribute of PyScript tags, which allows the use of Top-Level Await. The code below works the same as the block above, but I think you'll agree is more readable:

1
2
3
4
5
6

<script type="py" worker async>
    import pyscript

    response = await pyscript.fetch("https://reqres.in/api/users/2")
    pyscript.display(await response.json())
</script>

As a bonus - this snippet works in both Pyodide and Micropython, even though asyncio looks quite different in Micropython. Similarly, pyscript.fetch works in both Pyodide and Micropython tags.

Automatic .zip and .tar.gz unzipping

For applications involving numerous data files, or a build step involving a whole folder of data, it's handy to not have to manually unzip those files on the PyScript side on every upload. PyScript now has the ability to detect and automatically unzip these assets within the [files] section of <py-config>.

As a refresher, the [files] section of <py-config> takes a list of key/value pairs, where the key is the source URL and the value is a location in the in-memory virtual filesystem that PyScript runs in. For instance, the following takes the contents of the file at the relative URL content/posts/datafile1.txt" and places them in a file at /data/data1.txt in the virtual filesystem:

1
2
3
4

<py-config>
    [files]
    "content/posts/datafile1.txt" = "/data/data1.txt"
</py-config>

The automatic-unzip feature triggers whenever the source URL ends in .zip or .tar.gz and the destination ends in "*". As an example, the following fetches a zip file from assets/images/compressed_images.zip and unzips them to the ./imagedata folder in the virtual filesystem:

1
2
3
4

<py-config>
    [files]
    "assets/images/compressed_images.zip" = "./imagedata/*"
</py-config>

pyscript.ffi module adds standardization

PyScript is a singular project which straddles two (quite different) interpreters - Pyodide/Cpython and Micropython. While the internal architectures of both are dissimilar, it's useful for PyScript to present a somewhat-unified interface for both when it comes to web interactivity.

To that end, the pyscript package now includes the ffi module, which is starting to collect common interface features to standarize them between the two interpreters. So far, this contains only two functions:

pyscript.ffi.to_js

The to_js utility manually converts Python references to their JavaScript counterparts (where possible). Typically this is not necessary, as passing proxy objects back and forth will do. The primary use case is that pyscript.ffi.to_js automatically converts all Mapping objects to JavaScript literal objects, which is the preferred way to pass around configurations in JavaScript.

(For advanced users of Pyodide, when used in a Pyodide-backed interpreter, pyscript.ffi.to_js is equivalent to pyodide.ffi.to_js with a default dict_converter of Object.from_entries.)

The key here is that the behavior of this utility is the same in both Pyodide and Micropython PyScript tags. By starting to flesh out our own FFI, we can create standard expectations between both runtime types.

pyscript.ffi.create_proxy

Another Pyodide-inspired utility that PyScript is adapting to be uniform across runtimes - this forably keeps a borrowed reference to whatever Python object it is handed, to prevent that object/Proxy from being garbage collected before its time. For more context on the challenges of cross-language garbage collection, see my earlier writies on Why create_proxy() is Necessary.

This is one solution to an error which you may have seen if you've been working with event handlers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<button id="btn">Click Me</button>
<script type="py">
    # This breaks when the event handler is called
    from pyscript import document

    def print_foo(evt):
        print("FOO!")

    document.getElementById("btn").addEventListener("click", print_foo)
</script>

But this works fine:

1
2
3
4
5
6
7
8
9

<button id="btn">Click Me</button>
<script type="py">
    from pyscript import document
    from pyscript.ffi import create_proxy
    def print_foo(evt):
        print("FOO!")

    document.getElementById("btn").addEventListener("click", create_proxy(print_foo))
</script>

Of course, there are other ways of hooking up event handlers that don't create this issue, like PyScripts @when decorator:

1
2
3
4
5
6
7
8

<button id="btn">Click Me</button>
<script type="py">
    from pyscript import document, when

    @when('click', 'button')
    def print_foo(evt):
        print("FOO!")
</script>

Accurate Error Line Handling

PyScript users can now expect the line number of the errors that appear in tracebacks to be accurate. While this seems like a trivial (and necessary!) detail, it actually involved a fair amount of complexity to ensure that the various bits of Python code that PyScript runs at startup aren't included in the line-number count.

1
2
3
4
5
6
7
8
9

<script type="py" worker>
    import sys                      # 1
    import traceback                # 2
    from pyscript import display    # 3
    try:                            # 4
        print(1/0)                  # 5 - Error is here
    except ZeroDivisionError as err:
        display(''.join(traceback.format_exception(err)))
</script>

CORS-Less Workers

As thrilled as we were to add the ability to run code in Worker threads in PyScript, it did force users to wade into the messy lands of server permissions, CORS, and COOP to avoid a baffling error message:

The long and short of this error is: PyScript workers make use of a feature of web browsers called a SharedArrayBuffer to pass data, signals, and proxies back and forth between the browser's main thread and the worker thread. But as of 2018, this structure is only available within secure environments. I.e., you have to either configure your server with specific headers, or use some shim-javascript like mini-coi.js. Neither is particularly obvious or intituive for new web users, let alone folks who just want to stick some Python on the Web.

Thankfully, a new PyScript configuration option makes this somewhat better. If you add sync_main_only = true to your py-config tag, your worker tags will lose access to the DOM/Window, but you also won't require SharedArrayBuffer anymore, which obviates the need for special server settings.

1
2
3
4
5
6
7
8

<py-config>
    sync_main_only = true
</py-config>
<script type="py" worker>
    from js import console
    for i in range(10):
        console.log(i)
</script>

Of course, without the ability to interact with the DOM, such workers are significantly more limited in how they can affect the page. They're best used for offloading long-running Python calculations to the worker thread, for when you need to crunch data in Numpy or similar. Here's a working example, which generates a pseudorandom number in a worker thread:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

    <label for="seed">Seed</label>
    <input type="text" name="seed" id="seed">
    <br><label for="iterations">Iterations</label>
    <input type="text" name="iterations" id="iterations">
    <br><button id="btn">Calculate</button>
    <div id="result"></div>
    <script type="module">
        import { PyWorker } from 'https://pyscript.net/releases/2024.3.2/core.js';
    
        const { sync } = await PyWorker(
          './worker.py',
          { config: { sync_main_only: true }}
        );
    
        document.getElementById('btn').addEventListener('click', async () => {
            const seed = parseInt(document.getElementById("seed").value)
            const iters = parseInt(document.getElementById("iterations").value)
            const result = await sync.pseudorandom(seed, iters)
            document.getElementById("result").innerText += result + "\n"
        })
        </script>
    </body>
    

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

    # worker.py
    from pyscript import sync
    
    def pseudorandom(seed: int, iters: int):
        """Linear congruential generator. Parameters borrowed from POSIX and are probably bad"""
        modulus = 2**48
        a = 25214903917
        increment = 11
    
        for _ in range(iters):
            seed = (a * seed + increment) % modulus
        return result
    
    sync.pseudorandom = ps
</div>eudorandom