[pydeck] Create a prebuilt extension for JupyterLab #7030

dakoop · 2022-06-11T00:47:33Z

The general idea is to create a prebuilt extension that can be distributed as part of the pydeck python package. There is no need for a separate @deck.gl/jupyter-widget install as it is bundled in the pydeck python package, and the extension does not require jupyterlab to be rebuilt as it uses webpack 5's module federation. Just run pip install pydeck. This is WIP as it doesn't currently check for compatibility with classic notebook support, etc. To build,

yarn bootstrap
Install build and jupyter_packaging for python
Run python -m build in bindings/pydeck
a. If this doesn't work, try jlpm build in modules/jupyter-widget
b. This should create a wheel in dist
pip install dist/*.whl

Change List

Integrates @jupyterlab/builder and webpack 5 into the jupyter-widget build process. This is a one-step build now as the jupyter labextension build uses webpack to create a federated module (no separate dist/index.js)
Removes acorn resolution in the root package.json. Some part of the new build process requires a newer version of acorn, as the build just hangs otherwise. I am not sure what the effect is on other parts of deck.gl.
Writes the generated labextension to pydeck/labextension in bindings/pydeck so it can be distributed as part of the python wheel.
Supports the python -m build method for building pydeck and the labextension
Updates some metadata suggested in jupyterlab extensions
Updates pydeck version to 0.8.0
Builds on [pydeck] Update jupyter-widget to work with JupyterLab 3 #7026

Todo

Fix issue with jupyter labextension develop . (something with pyproject.toml?)
Check whether other pieces of pydeck and jupyter-widget function as before
Determine impact of acorn resolution
Fix other pydeck setup/build code (e.g. Makefile)
Update documentation

See https://ipywidgets.readthedocs.io/en/stable/changelog.html#updates-for-widget-maintainers

Note that this ties it to 8.8.0-alpha.6; see visgl#7025 for comments

bump_version.py generates these files so could be updated

The main thing is modifying package.json and webpack.config.js to work with the jupyterlab builder. Since the previous build was just using webpack, we let the builder integrate the webpack config from that build with the labextension webpack. Required some changes to remove the original dist directory so everything just goes to a pydeck/labextension directory now.

kylebarron · 2022-06-13T18:36:08Z

install as it is bundled in the pydeck python package

iirc this is only valid for specific jupyter versions. Only for notebook as of version 5.3 and Jupyter lab as of version 3.

kylebarron · 2022-06-13T18:42:12Z

bindings/pydeck/pydeck/_version.py

@@ -1 +1 @@
-__version__ = "0.7.1"
+__version__ = "0.8.0"


Nit: I'd bump the version in a follow-up PR

kylebarron · 2022-06-13T18:43:22Z

bindings/pydeck/pyproject.toml

-  "setuptools",
-  "wheel",
-]
+requires = ["jupyter_packaging~=0.10,<2", "jupyterlab~=3.1"]


Why do you include jupyterlab as a build dependency? The jupyter-packaging docs don't seem to require it: https://github.com/jupyter/jupyter-packaging/#as-a-build-backend

kylebarron · 2022-06-13T18:43:47Z

bindings/pydeck/pyproject.toml


-[tool.black]


Why delete the black config section?

kylebarron · 2022-06-13T18:47:10Z

bindings/pydeck/RELEASE.md

+
+```bash
+npm login
+npm publish --access public


Nit: we should be able to put --access public in the package.json I think so we don't need to specify it manually. Ref
https://docs.npmjs.com/cli/v8/configuring-npm/package-json#publishconfig and https://docs.npmjs.com/cli/v8/using-npm/config#access

kylebarron · 2022-06-13T18:48:15Z

bindings/pydeck/RELEASE.md

+python -m build
+```
+
+> `python setup.py sdist bdist_wheel` is deprecated and will not work for this package.


Can you update the Makefile with the new commands as well?

kylebarron · 2022-06-13T18:53:00Z

modules/jupyter-widget/pyproject.toml

+
+[tool.jupyter-packaging.build-args]
+build_cmd = "build:prod"
+source_dir = "../../modules/jupyter-widget"


This path doesn't make sense here or at least is unnecessary

kylebarron · 2022-06-13T18:54:35Z

bindings/pydeck/pyproject.toml

+build_dir = "pydeck/labextension"
+npm = ["jlpm"]
+
+[tool.check-manifest]


The check-manifest config is defined here but it seems there's instructions on how to use it.

kylebarron · 2022-06-13T18:55:00Z

bindings/pydeck/pyproject.toml

+factory = "jupyter_packaging.npm_builder"
+
+[tool.jupyter-packaging.build-args]
+build_cmd = "build:labextension"


This only builds the labextension?

kylebarron · 2022-06-13T18:58:37Z

bindings/pydeck/setup.py


+    logging.basicConfig(format="%(levelname)s: %(message)s")
+    logging.warning(
+        "Build tool `jupyter-packaging` is missing. Install it with pip or conda."


It's not clear to me why this should be a warning not an error. It seems like it could lead to hard-to-debug issues where the extension isn't updating as expected. Additionally, I would expect that jupyter-packaging is always installed given that it's defined in build-requirements in the pyproject.toml.

kylebarron · 2022-06-13T19:01:54Z

bindings/pydeck/setup.py

+    author_email=pkg_json["author"]["email"],
+    description=pkg_json["description"],
+    license=pkg_json["license"],
+    license_file="LICENSE.txt",


Given that in this PR you're now defining more of the build process in pyproject.toml, move the non-dynamic parts of the setuptools config there too? https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

kylebarron · 2022-06-13T19:04:24Z

package.json

@@ -20,7 +20,6 @@
    "prettier: 2.3.2 breaks on `import type`, remove when ocular bumps"
  ],
  "resolutions": {
-    "acorn": "7.4.1",


cc @Pessimistress on this. I don't know why it was previously pinned

kylebarron · 2022-06-13T19:05:20Z

modules/jupyter-widget/style/index.js

@@ -0,0 +1 @@
+import './base.css';


Is this folder used? Seems like it just imports an empty CSS file.

dakoop · 2022-06-13T19:10:53Z

Correct, although even notebook will now be built on jupyterlab infrastructure (ref). I think you could keep the other method (current install process) working for the other modes, but these would have to be tested. JupyterLab developers are aware of some difficulties for users finding and installing the new prebuilt extensions: jupyterlab/jupyterlab#10554

Perhaps a better approach in structuring the code would be to structure the jupyter-widget components as currently done, and create a separate build process for the JupyterLab 3 plugin so that different dependency versions (specific to JupyterLab's build/plugin process) could be integrated without impacting the deck.gl's current dependencies. I didn't end up going that route because I couldn't figure out what was going on with the original approach that just included dist/index.js, but in retrospect, this may have been due to the dependency issues (acorn).

davidbrochart · 2022-07-25T18:25:36Z

@dakoop do you plan on continuing that work? It would be awesome.

donmccurdy · 2024-03-29T15:12:00Z

My impression (not a regular Jupyter user!) is that nbextensions for Jupyter are dying a slow death at this point, and with Jupyter Notebooks v7+ (released July 2023), you'll get errors like ...

Jupyter command `jupyter-nbextension` not found.

... for which old solutions will no longer work. Likely a prebuilt extension for JupyterLab is becoming necessary. Are there other alternatives we should consider, or anything I can do to help land this PR?

Otherwise, I think pydeck may be stuck in Jupyter Notebook 6.4 or lower, or nbclassic.

kylebarron · 2024-03-29T15:21:55Z

I think the main difference is that Jupyter Notebook v7 is built upon JupyterLab architecture, but in a way that looks like the classic notebook. So I think you need to make JupyterLab extensions to support the latest notebook

dakoop · 2024-03-29T15:45:24Z

As @kylebarron noted, Notebook 7 introduces extension changes as it moved to lab architecture. This PR was a push to update the pydeck extension a while back, but I haven't had time to reexamine this. There is other work on deck.gl Jupyter integration: lonboard and ipydeck.

jtpio · 2024-12-11T21:29:51Z

At this point, maybe it could make sense to consider building the pydeck widget on top of anywidget?

This could help reduce maintenance, and would make pydeck usable with the latest versions of JupyterLab and Jupyter Notebook.

kylebarron · 2024-12-13T17:07:47Z

I'm not sure there's anyone with the motivation to rewrite pydeck's API on top of anywidget. As mentioned above, lonboard and ipydeck are newer bindings to deck.gl that do use anywidget.

jtpio · 2024-12-13T17:32:56Z

Oh, I hadn't noticed, thanks @kylebarron!

I was curious if pydeck was usable in JupyterLite, which brought me to this PR.

But it's good to see that ipydeck does seem to work out of the box (or at least the default example), when testing on a deployment which already has anywidget installed:

https://jupyter-widgets-contrib.github.io/anywidget-lite/lab/index.html?tempNotebook=1&path=temp.ipynb&openAsNotebook=1#notebook=N4IgtgpgLghgJjWIBcoDWEBOA7CAbAZwAcIBjFUbGSFEIgTygAsB7bEAGhDgEti8Y9APpUayEAAVGrbAAIAFFJa84EAJScQA7AHMArjB0RaDZmxABfLtv2GIQntgBmLCiFLKIYHpkwtMQmCebqLG4qYymgBuWAQ85sgAzFYgTjx49hAAHlAQ2HEJIAB0DJrekFD0JLS5OQD0WQC0EeZcoSbSrSDYAEYe2DGYUELZRP65mB1m7FwMOpDYUARCGVlYtDwt2InRsfHs4olFAByWKb0umGCIgY7+KAAsbT2X11CPXKT4hCgA2qBfPB4ISVariDyqTQEFh6TBfWgAUiIm1kjgIsCBqIYqlIaDK0HgiBgbigmD06IgcBQpL0EBSMKgRD0Sz+AF0uNkyMz9kIPHpFihsHogVYAd8QVUwu5glxobD4eInH4wFj6Di0KiwGMhrIACJkNAcWQAGUEWCNADUeBAAO4AZVguQAOtgXQJ6LFZABeWS-F2yAMms2YeT+wPh0EQL1OkAAcQgLAAUtDsKaPZgYxww+GAwhYNGQEwoIyCMg6nVMDAbUUdDxmHoeuSsP1cosih4wHUonwdHg6uqa3hGnmYHVrhTMHVstQiBkCHUjCwAFYprswbB8waNHp4Fi4ghFFdsTPZnMsIgwUh1+hegAMJyzchzAfRfgwcC9ADEYIQII-nwGaRApSXoACpkn+p7htkNKqB+4G0v+AE2j4EBKtQUYIZBT7PvgEBRIgPIEKQP5RveACMSHPkYwx4QRUD7AWAACTFekQfgkEM1oHgReC0hIWB2gAjmAJ44TmNFCEBwIeLumBer85EAJxKUaABMiSJEa5HHLe7JQYGkl4I49iyf4ClqQArJZ6nWbZln6eJaguqyLout2tpCOiiAQN6shWraDo+aG4kCAxUB6KoXoPEpRRWU8si7rodaRVGjTkRpRTkVpsgAF4sCwYBeuRlGyNcWRCPlhXFQAbEayJQKQTDRTZsg9BAMCYI4Oh3i6zmutg6p+fquIheG7qxF6E2YAQVGotgdY8D+QgeTaXmOlGq3rT5c3XEQ630BkBbGToRZif1LrqvisAjiSZIUlSyA0nSXAMkyLLIL87IgJypDcmwvIwgKyBCiKHBikCEpgtKkKyjCcJStdhJIKgIA0g91IQfSzLvQQbIcmsf0MQDfLA6DeAWKyFhAA

kylebarron · 2024-12-13T17:39:41Z

I also have an example with Lonboard in JupyterLite rendering 3.2 million points: https://jupyterlite.ds.io/lab/index.html?path=lonboard_example%2Fdata-filter-extension.ipynb

ajduberstein · 2024-12-13T17:41:57Z

pydeck does work via JupyterLite! https://duberste.in/pydeck-in-pyodide/lab/index.html

…

On Fri, 13 Dec 2024 at 9:40 am, Kyle Barron ***@***.***> wrote: I also have an example with Lonboard in JupyterLite rendering 3.2 million points: https://jupyterlite.ds.io/lab/index.html?path=lonboard_example%2Fdata-filter-extension.ipynb image.png (view on web) <https://github.com/user-attachments/assets/4d53b398-f29f-4ea7-bd51-0581a20519c6> — Reply to this email directly, view it on GitHub <#7030 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ2IVNCOVOAWWMRYBNGUF32FMLXLAVCNFSM6AAAAABTOMWJZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBRHE2DAMJQGM> . You are receiving this because your review was requested.Message ID: ***@***.***>

ibgreen · 2024-12-13T18:34:10Z

At this point, maybe it could make sense to consider building the pydeck widget on top of anywidget?

Just some context: I can confirm that the deck.gl TSC (technical steering committee) is talking about the value of a refresh of pydeck - and is strongly in favor of adopting anywidget in pydeck. It is not a lack of faith in the value of pydeck, it is mainly an issue of (Python capable) maintainers not being available.

As mentioned above, lonboard and ipydeck are newer bindings to deck.gl that do use anywidget.

Regarding ipydeck, the deck.gl team did reach out to the author of ipydeck about potential collaboration / merge of the projects, and while the interest was there from both sides, the big time zone differences (Japan) got in the way.
lonboard is an extremely impressive effort that deck.gl would love to align further with. However, in my view, it was kind of set up to enable maximum iteration speed for a specific set of use cases and a specific author and while @kylebarron is always open to collaboration, so far he has not signalled readiness to e.g. take steps into open governance etc. And that is probably the right decision for lonboard until it matures a bit more.

All of this leaves pydeck with very interesting options for the future, but unfortunately without a very clear immediate path forward (though happy to see original author @adjuberstein chiming in!)

dakoop added 7 commits June 10, 2022 16:30

Make changes to support JupyterLab 3

90d724c

See https://ipywidgets.readthedocs.io/en/stable/changelog.html#updates-for-widget-maintainers

Bump version so pydeck is tied to current jupyter-widget version

e1b18d4

Note that this ties it to 8.8.0-alpha.6; see visgl#7025 for comments

Run black on auto-generated version files

a246e84

bump_version.py generates these files so could be updated

Don't resolve acorn to allow jupyterlab builder to work

954598d

Integrate jupyter-widget with pydeck as prebuilt extension

cddac6e

Update package version in toml

b49b646

Pessimistress requested review from ajduberstein and kylebarron June 11, 2022 04:35

kylebarron reviewed Jun 13, 2022

View reviewed changes

This was referenced Mar 29, 2024

Fix Python CI #6777

Merged

[Bug]: build failure pydeck #8469

Open

donmccurdy added feature pydeck labels Mar 29, 2024

donmccurdy added this to the v9.0 milestone Mar 29, 2024

Pessimistress force-pushed the master branch 4 times, most recently from 0a083b6 to eb85da3 Compare April 25, 2024 03:09

donmccurdy mentioned this pull request Aug 1, 2024

[Bug] Two-way Jupyter binding support (i.e., ipywidgets) #9064

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pydeck] Create a prebuilt extension for JupyterLab #7030

[pydeck] Create a prebuilt extension for JupyterLab #7030

dakoop commented Jun 11, 2022

kylebarron commented Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

dakoop commented Jun 13, 2022

davidbrochart commented Jul 25, 2022

donmccurdy commented Mar 29, 2024

kylebarron commented Mar 29, 2024

dakoop commented Mar 29, 2024

jtpio commented Dec 11, 2024

kylebarron commented Dec 13, 2024

jtpio commented Dec 13, 2024

kylebarron commented Dec 13, 2024

ajduberstein commented Dec 13, 2024 via email

ibgreen commented Dec 13, 2024

		@@ -1 +1 @@
		__version__ = "0.7.1"
		__version__ = "0.8.0"


		[tool.black]

[pydeck] Create a prebuilt extension for JupyterLab #7030

Are you sure you want to change the base?

[pydeck] Create a prebuilt extension for JupyterLab #7030

Conversation

dakoop commented Jun 11, 2022

Change List

Todo

kylebarron commented Jun 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakoop commented Jun 13, 2022

davidbrochart commented Jul 25, 2022

donmccurdy commented Mar 29, 2024

kylebarron commented Mar 29, 2024

dakoop commented Mar 29, 2024

jtpio commented Dec 11, 2024

kylebarron commented Dec 13, 2024

jtpio commented Dec 13, 2024

kylebarron commented Dec 13, 2024

ajduberstein commented Dec 13, 2024 via email

ibgreen commented Dec 13, 2024