USB transport

[phoddie]

This issue is for discussion of using runmod with WebUSB.

I pushed changes to runmod today that provide the same functionality over USB as is available using WebSockets. The documentation has been updated with the relevant details throughout. The xsbug.js example has been updated with WebUSB support including separate new XsbugUSB and XsbugWebSocket classes to manage the two connection types.

[FWeinb]

@phoddie
I looked into it but I can't get it to work. I reapplied the patches and build runmod new.

I tested it with the provided example but noticed that there is a typo in line 79. But that is not the problem. I enabled the tracePacket function (changing the data here to tracePacket("> ", new Uint8Array(result.data.buffer));)

After the above fixes where applied I got this trace output.

I don't know if this is related but I get this message when unloading the driver but the WebUSB connection is established. If I load the driver again I can't connect to the device, so unloading seams to work.

sudo kextunload -b com.silabs.driver.CP210xVCPDriver
(kernel) Can't unload kext com.silabs.driver.CP210xVCPDriver; classes have instances:
(kernel)     Kext com.silabs.driver.CP210xVCPDriver class com_silabs_CP210xGPIO has 2 instances.
Failed to unload com.silabs.driver.CP210xVCPDriver - (libkern/kext) kext is in use or retained (cannot unload). 

[phoddie]

Thanks for giving it a try. Sorry about the typo. I'll fix that. It must have crept in when merging.

The driver isn't unloaded if there are still instances reported by kextunload. Let's fix that first. Sometimes there are zombie instances in the terminal. Try quitting terminal, restarting it, and then unloading again. If that fails, reboot and try again.

Once you are running, make sure the set the baud rate for your microcontroller at line 608. It is configured for ESP32 here. If you are using ESP8266 you will need to remove the divide by two.

[FWeinb]

@phoddie
Right after I hit send I noticed that it was configured for the ESP32. I tested with the moddable zero. Changeing to 921600 fixed everything. The kextunload seams not to be the issue.

[FWeinb]

@phoddie
There seams to be another xsbug message type called parsererror that is not implemented yet. I am getting TypeError: XsbugMessage[node.nodeName] is not a function

[Edit]
I see. This is the behaviour of the DOMParser. There seams to be a malformed XML document. the DOMParser will insert the parsererror node into the document.

[Edit2]
I am getting malformed usb transmission. There are intermitted random numbers or missing text in the XML.

[phoddie]

@FWeinb - as noted in the code, WebUSB itself doesn't seem to do much buffering itself. That can lead to dropped characters. One option is to slow down the connection, which is easy but undesirable as it makes everything slower. Another is to increase the buffer size. It is reliable for me as committed but your computer may be different. The revive buffer is set to 32 KB at line 234. The USB read size is set to 32 KB at line 328. Try making both of them much bigger, say 128 KB.

[FWeinb]

@phoddie
That had no effect. Still getting corrupt USB data.

[Edit]
Here is an example document (formatted)

[FWeinb]

@phoddie
That is strange! Commenting out the tracePacket function solved the issue. But not completely. Sometimes I still get broken data.

[FWeinb]

@phoddie
Okay. Sorry for spamming this thread, figured it out. I compiled the runmod without WiFi support but mdns is still trying to publish the hostname. That lead to UDP send errors that crashed the device and broke the USB connection.

[phoddie]

Good catch. I'll look at fixing runmod so it doesn't assume there is a network connection. Until yesterday, that was a safe assumption. ;)

[FWeinb]

@phoddie
Just FYI the arguments of padStart must be swapped here.

[phoddie]

Update to main.js pushed to only start the mDNS and WebSocket services if there is a Wi-Fi connection. You'll need to update the Moddable SDK for that to work properly on ESP8266.

[FWeinb]

@phoddie
I pushed a new version that supports both WiFi and USB connections. After some initial problems everything seems to work fine. It is a little strange that one has to unload the driver but understandable as the WebUSB spec seams very low level in regards to handling devices.

The best part is that WebUSB is working over https without any warnings!

Could not test with the latest changes you committed. It is way to late (early) here.

[phoddie]

That was very quick! I will check it out later. Get some rest.

It is a little strange that one has to unload the driver but understandable as the WebUSB spec seams very low level in regards to handling devices.

It isn't perfect. For beginning users, that may be just fine. The IDE may be able to provide guidance to users when the driver is installed. I have a trace in the example in that case, for example.

The best part is that WebUSB is working over https without any warnings!

Wahoo!

[FWeinb]

The IDE may be able to provide guidance to users when the driver is installed

That sounds like a great idea. I copied your error message and show it in the Log panel but that message could be improved.

Any idea what the implementation status of WebUSB is for Firefox? It seams that Chrome is the only browser shipping this API.

[phoddie]

I copied your error message and show it in the Log panel but that message could be improved.

Since the browser knows the host OS, it could provide pretty specific advice.

Any idea what the implementation status of WebUSB is for Firefox? It seams that Chrome is the only browser shipping this API.

As far as I know, no other browser has indicated an intent to support it. And all the editors of the WebUSB specification are from Google. ;)

[phoddie]

@FWeinb I tried out the WebUSB support. It is working very nicely. Congratulations. The cycle for making a change, compiling, downloading, and running is remarkably fast now.

Using ESP32 I was able to exercise all features. Getting and setting runtime breakpoints works well too. On ESP8266, the mod upload reported success but it did find the mod on restart. I expect that is a problem with my local set-up (too many locally modified files...). I will sort that out.

[FWeinb]

@phoddie
Great! Glad it is working for you. I still have some problems with the communication over usb being strange on the moddable zero. Sometimes the data received broken (broken xml), I don't know why that would be though.

It might be better to lower the baud rate on the ESP8266 too? As I am understanding it WebUSB isn't supported in workers yet, I might suspect that there are some problems reading the data on the main thread. That would explain why removing the tracePaket function from that loop made the connection a little more reliable, just a guess.

[phoddie]

Maybe. I hate to lower the baud rate. I'd rather increase it on ESP32. ;) Let me look into it a bit further.

[phoddie]

I committed a couple changes to xsbug.js for WebUSB.

The first is work that @lprader and I did this afternoon to use the default endpoints rather than hardcoding them. This allows the code to work with boards from oddWires, and presumably others that use the same CP2104 part from SiLabs. (Moddable Zero uses CP2102).

The second is a rework of readLoop. I thought the problem with dropped input might be due to the gap between reads. Between reads the original implementation does a fair amount of buffer parsing work and tracePacket adds more. The rework keeps multiple read requests active at the same time. The idea is that when one buffer fills up, WebUSB can begin filling the next. I am able to run with smaller buffers than buffer. Still, I'm not confident the approach is correct. Even if it is correct, it ma need tuning (e.g. more buffers or bigger buffers). Please let me know what you find.

[FWeinb]

@phoddie
Great work, I merged @lprader PR (thanks) but had to fix some minor things. I still get malformed data sometimes right after a restart.

Here are the issues I had:

• xsbug.js#259 should be usb.configurations[0] I guess because usb.configuration is null for me there.
• xsbug.js#376 should use this.outEndpoint instead of hardcoding it, @lprader did that in the PR.

I added some error handling yesterday to notify the user to please restart the debugging connection if malformed data was received.

[phoddie]

@FWeinb - thank you for the corrections. I'll apply those. For whatever reason, usb.configuration is non-null on my computer.

I still get malformed data sometimes right after a restart.

You will always receive that. The microcontroller ROM sends some basic information at start-up. On ESP8266 that is sent at 76800 baud and ESP32 at 115200. Since that is a different baud rate than the WebUSB serial port is configured for, it appears as garbage. The Moddable SDK sets the expected baud rate when it starts.

The implementation defends against unexpected characters. We use the same filtering in serial2xsbug and it works reliably.

[FWeinb]

@phoddie
Yeah that usb.configuration thing is very strange.

Regarding the malformed data. I get corrupt xml data from xsbug after a restart The readLoop is breaking on an parsererror node that will be inserted by the DOMParser if the XML was corrupted.

[phoddie]

Maybe the parseerror is caused by data in buffers from before the restart? I added a reset call to reinitialize the local receive buffer and call that immediately after restarting. With that in place, I was able to install/restart modBalls dozens of times in a row on both ESP8266 and ESP32 without a problem. That change is committed for your reference.

[FWeinb]

@phoddie
Must be my machine then. I don't know if it has something to do with the usb-c port, or the fact that I can't sudo kextunload -b com.silabs.driver.CP210xVCPDriver without getting the (kernel) Kext com.silabs.driver.CP210xVCPDriver class com_silabs_CP210xGPIO has 1 instance. response. Even after a fresh restart and only plugging in the device. I have this version installed: com.silabs.driver.CP210xVCPDriver (5.0.10)

Here is an example of the kind of xml I get.

[phoddie]

The XML fragment is clearly missing data. That's not good.

It could be configuration dependent. Still, let's not give up as there's some chance others will have the same configuration. FWIW - I'm using a USB-C connection too.

At one point, I had two copies of the SiLabs driver installed. That caused some bizarre problems. Maybe you do as well? In my notes, I have these two paths as places where the driver may be installed: /Library/Extensions/SiLabsUSBDriver.kext and /System/Library/Extensions/SiLabsUSBDriver.kext.

[FWeinb]

@phoddie
I only have the driver in /Library/Extensions/SiLabsUSBDriver.kext I just tested on my old machine and there everything is working fine, kext unloading is returning no error. No idea what this machine is doing with the USB driver stuff.

[FWeinb]

@phoddie
Uninstalling the driver completely seams to be working but is very uncomfortable.

[phoddie]

Well, at least it can work. That's reassuring. But, yes, it is inconvenient for development to have to reboot to update the host.

[FWeinb]

@phoddie
Heh might be a little risky but

sudo kextunload -b com.silabs.driver.CP210xVCPDriver
sudo mv /Library/Extensions/SiLabsUSBDriver.kext /Library/Extensions/SiLabsUSBDriver.kext_disabled

is doing the trick. So I don't have to reboot all the time.

[phoddie]

Curious. There is something to be learned here.

[FWeinb]

@phoddie
Had some time on the train and could circle in on the persisting connection problem via usb. I used the XsbugLogin event to determine when the device was ready. That was a mistake. The problem I am running into is that this.currentMachine is undefined thus sending the wrong preamble to the device. When executing a debug command.