Whats New in Pyscript 2022.09.1

PyScript Version 2022.09.1 was just released, and just as tech lead Fabio Pliger said in proposing the versioning scheme:

"...An important aspect to keep in mind is that PyScript is still in its very early stages. So, we should highlight that the expectations should be that think can often break until we reach a level of maturity and stability."

And wow, are there a lot of new things in this version of PyScript. What's more, the default Pyodide runtime has been upgraded to the recently-released version 21.2, which itself provides many new features and improved functionality to PyScript.

I want to specificically highlight new features, breaking changes, and neat behind-the-scenes work. The full details of what's changed are captured in the PyScript Release Changelog

PyScript

<py-env> Will Be Going Away

Previously, the <py-env> tag was where one would specify additional libraries to download from PyPI, as well as URL's to load into the local filesystem. Now, those options are being folded into <py-config>, alongside other options like plugins and runtimes and metadata like the pages name and version number. The use of <py-env> is deprecated and will be removed in a future release.

Additionally, <py-config> can now accept configurations in JSON in addition to TOML. Creators using build systems that strip out whitespace (which isn't very kind to TOML) may find this especially useful.

1
2
3
4

<py-config>
  packages: ["rich", "faker"]
  paths: ["./data_file.txt"]
</py-config>

py-* Events

The alpha and 2022.06.1 releases supported a couple of special attributes on HTML tags - pys-onClick and pys-onKeyDown - that PyScript hooked into to allow the running of Python code in response to a couple of common browser interactions.

Release 2022.09.1 radically expands this capability with many, many more browser events supported.

The syntax of py-* events has also changed to more closely match JavaScripts event syntax. Previously, you supplied a Callable which was called with no arguments. Now you write a line of code (optionally broken up with ; symbols) which is run when the event triggers. The correct usage is now:

1
2
3
4
5
6

<py-script>
    from js import console as jsconsole
    def say_hi(name):
        jsconsole.log("Hi, " + name)
</py-script>
<p id="my-paragraph" py-mouseover="say_hi('Jeff'); jsconsole.log('I did it!')">Mouse Over Me</p>

Note that, unlike JavaScripts event syntax, the value of the py-* attribute can be any valid Python code, not just a single function call.

Better Input/Output Escaping

Embedding something that looks like HTML inside of Python inside of HTML is... well, even just saying it is a mouthful, and it comes with its own pitfalls. Previously, PyScript tags like the following would fail in a couple of ways:

1
2
3

print("<b>A bold tag!</b>")
tag_name = "i"
print("I'm pretty sure 1 < 2 but 2 > 0")

First, the Browser needs to be prevented from interpretting the <b> tag as internal HTML, and second, the output needs to recognize that the < > symbols are not an HTML tag. These issues have been solved by a pair of changes.

Better Logging

Logging to the Developer Console that PyScript does is now much cleaner, and annotated by what file the log line is generated in. This makes it easier to see what's logged by the user's program and what's being logging by the PyScript mechanisms themselves.

Framework for Multiple Runtimes

The use of a specific version of Pyodide is no longer hardcoded into a PyScript release - users may now opt to supply a URL and name for a 'runtime' in the <py-config> tag. If one is not supplied, the default is still to load the version of Pyodide that PyScript has been most recently tested against, which should be the right option for most users. But this does open the door to future improvements like:

• Running py-script blocks in different versions of Pyodide
• Running py-script blocks in runtimes that are not Pyodide (Micropython??)
• Running py-script blocks in a self-built/custom build of Pyodide for experimentation or demonstrating new features

Documentation

Try PyScript

A new Try PyScript section now leads the main ReadMe on the PyScript GitHub, to more quickly get new users up to speed on how to try out PyScript in their browser.

Contributing

The CONTRIBUTING guide has been fleshed out with more guidance on developing submitting useful issues, forking the repository for local building and setting up the a development environment, and more. Both Mariana Meireles and Fabio Rosado have contributed excellent information on how to build PyScript and how to create and submit a Pull Request - every open source project should be so lucky!

How Tos

Two new How-To guides were added to the documentation. The first covers how to make HTTP requests in pure Python by using pyodide's pyfetch method. The second illustrates the techniques for passing objects between JavaScript and Python (in PyScript), including some slightly-cursed uses of JavaScript's eval().

Getting Started

The Getting Started guide got a huge update to reflect the new <py-config> changes (see above).

Pyodide

It's no secret that the beating heart of the PyScript project is the Pyodide project, which makes it possible to run Python in the browser by compiling the CPython runtime to Web Assembly. (This is now nicely highlighted at the top of the PyScript readme.) Which means that improvements to Pyodide are big boons for PyScript!

While PyScript's Alpha and 2022.06.1 were designed around Pyodide 20, PyScript 2022.09.1 fully embraces Pyodide 21.2 and the many changes and improvements it brings. We'll only hit the highlights here; for more details, see the Pyodide 21 Release Post and Change Log

API Changes

This is probably the most visible change for the casual PyScriptian - the functionality of the Pyodide Python API has been divied up into a number of individual packages for clarity and namespace control. So now, rather than using from pyodide import create_proxy, one would use from pyodide.ffi import create_proxy, and so on.

1
2
3
4
5
6
7

<py-script>
    from pyodide import create_proxy
    def say_hi(name):
        print(f"Hi, {name}")
</py-script>

>>> FutureWarning: pyodide.create_proxy has been moved to pyodide.ffi.create_proxy Accessing it through the pyodide module is deprecated.  

The old locations of the functions are still present but deprecated in version 21, so this change alone won't break code written for Pyodide 20. But you will see a deprecation warning, and any new code should obey the new namespacing as the deprecation

Python Wrappers for addEventListener, setTimeout, and more

As part of the reorganization mentioned above, the Pyodide API added a bunch of Python utility functions that handle common JS actions. Specifically, within the a[href="https://pyodide.org/en/stable/usage/api/python-api/ffi.html">pyodide.ffi.wrappers namespace, we now have functions for add_event_listener, remove_event_listener, set_timeout, clear_timeout, set_interval, and clear_interval. This avoids the need import those JS functions directly from js.document, and since the Python functions automatically wrap passed functions with create_proxy, that can be left out as well.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<py-script>
    from js import document
  
    from pyodide.ffi.wrappers import add_event_listener
    def say_bye(*args):
        print(f"Goodbye!")
    
    tag = document.getElementById("my-div")
    add_event_listener(tag, "click", say_bye)
</py-script>   

pyodide.code.run_js

Yet another API addition is pyodide.code.run_js, which evaluates the passed JavaScript code and returns the result as a JSProxy object. This removes the need to, for example, import eval from JavaScript to execute JS within Python. A nice clean feature.

New Packages

A whole load of new packages have now been bundled with Pyodide, including opencv-python, ffmpeg, svgwrite, sqlite3, python-magic, and many more. See the full list to see if your favorite package is now included.

Improved Build Process for Binary Wheels

For those looking to integrate their own Python wheels into apps built with Pyodide, the process for building binary wheels for Pyodide has been significantly improved. See the Pyodide team's blog post on Binary Wheels for more information.

JavaScript Array Slicing

There are lots of little edge cases and behaviors where JavaScript's and Python's behaviors are different, and the Pyodide team is constantly working on new ways to make that interface less painful. Recently, they've implemented slicing on JavaScript array objects that obeys the same syntax as Python lists, which is a neat feature for those passing data from the browser into Python for processing.

There's been some additional work and corrections to this process, and I'd imagine we'll continue to see it evolve and refine.

Correct Handling of Objects with Null Constructor

This is a small but necessary improvement - previously, it was difficult (if not impossible) to import a javascript module into Python-in-Pyodide, since JS modules don't have constructors, but Python expects everything (including modules) to be an object, and so would try to "construct" them. Now, JavaScript module imports work as expected.

Emscripten

Just as PyScript uses Pyodide as its primary runtime to run Python in the browser, so Pyodide relies on Emscripten to compile CPython for the browser. Pyodide 21 now moves from using Emscripten version 2.0.27 to version 3.1.14

To be honest, I'm not well enough versed in EmScripten to be able to parse the changelog details enough to highlight them. If you're more familiar with that program and its capabilities, let me know!

Testing

The last two categories of changes really shouldn't impact end-users of PyScript much, but they're already making a huge difference to the PyScript devs and maintainers. Prior to this release, there wasn't much of a testing regimin. Now there's multiple different means of testing the Python and TypeScript code that make up PyScript, as well as integration tests that test them both, making it easier and faster to tell when something's going to break. The testing methods are:

• Integration Tests with Playwright - loads HTML pages in the browser and checks that PyScript works as intended.
• TypeScript Testing with Jest - Testing whether Py-Script elements load correctly via TS/jS
• Python Tests with PyTest - Test functionality in PyScript.py, Python things like Element, PyScript, ContextManager, etc.

Infrastructure

Finally, there's the bounty of little improvements that make the codebase stronger and the dev process more repeatable.

Continuous Deployment

The CI/CD pipeline continues to get refined and grow more resiliant - there have been some improvements to the CD process to ensure PyScript is rebuilt with every commit and pushed to Unstable.

Type Annotations

PyScript is being developed in TypeScript, which has the nice property of allowing quick prototyping with loose typing and gradually refining the typing to make the Linter/compiler happier. Several users, especially contributor Woxtu have been hard at work makign sure types line up, Promises are resolved, and type signatures are accurate.

Though end-users don't see the results directly, having thorough and consistant type signatures makes it easy to spot smelly code when adding new features. Does this function really need to return two different types of thing, or should we be rethinking the code structure? Why is this any necessary?

What's Next?

So what's coming down the pipe next for PyScript? Frankly, a ton, and that work is largely visible in the open PR's and issues on the PyScript GitHub.

New Output and Rendering Design

One of the largest overhauls coming to a near-future version of PyScript is a total rethink of how PyScript renders to the browser window. print() is the right output method for a terminal, but it doesn't quite make sense in the context of a browser window, where the world of UI is much much larger.

To that end, there's a large project in the works that, among other things:

• Introduces a new display() function, which is the preferred way of outputting to the browser window
• Routes stdout to the developer console by default
• Improves escaping of HTML-like text included inside PyScript source

The exact syntax and methodology of display() is still being hashed out, but work is proceding a breakneck pace, and it'll be exciting to see where it ends up.

PyScript Lifecycle Changes

Antonio Cuni et. al. have laid the groundwork for a sweeping rethink of how PyScript manages the lifecylce of initializing, loading Pyodide, processing tags into custom elements, and more. It includes provisions for user-created plugins (to extend functionality) and widgets (essentially custom tags) on the page, in more-or-less a plug-and-play fashion.

This isn't the kind of dish that can be cooked up in 20 minutes - it'll touch almost every part of the PyScript codebase. But it's exciting to see the code moving in a direction that's more flexible and understandable, which will only make it more expansible.

Async Behavior

This one is near and dear to my heart. Exactly how asynchronous code should function in PyScript has been a hot topic for some of the maintainers, considering we broke it and had to fix it again.

Pyodide has a curious feature that allows you to run code with Top Level Await, since there's always an event loop running (the browser event loop). This is both handy and confusing, since it doesn't correspond to the experience of running Python in a terminal at all. So what's to be done? Perhaps we need to further specify the execution order of async tags? Or preclude top-level-await entirely? Do we even entirely understand how Pyodide is interacting with the browser event loop? What if an async task never terminates? Lots to be done here.

And More

There's plenty more swirling around in the PyScript ecosystem - web workers, arrow functions, further documentation, a file API...

Of course, not all ideas become plans, and not all plans become reality, but there's no lack of great ideas to keep pushing PyScript forward.