-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
draftsman-update fails if mod's lua files begin with UTF-8 Byte Order Mark #84
Comments
Sorry about the lateness, thanks for the report. Going to take a look at this in a bit. May be related/adjacent to this issue, or maybe I need to be more strict with the way that files are read and given to lupa (explicit encoding?). And yeah, if you run into other issues with the load process, make some more issues for em', easier to track and resolve on my end when that's the case. |
Oh, and I should probably write this down somewhere for issue contributing, but an exact mod list would be helpful for reproductions. I'm looking into putting in a special command line argument for |
Alright, I think I've found the problem. The trouble is that most of the time mod lua files are located in zip archives, which when we read from them return raw byte strings instead of performing the necessary conversion we expect. We can get around this by using def file_to_string(filepath):
"""
Simply grabs a file's contents and returns it as a string. Ensures that the
returned string is stripped of special unicode characters that Lupa dislikes.
"""
# "utf-8-sig" makes sure to strip BOM tokens if they exist
with open(filepath, mode="r", encoding="utf-8-sig") as file:
return file.read()
def archive_to_string(archive, filepath):
"""
Simply grabs a file with the specified name from an archive and returns it
as a string. Ensures that the returned string is stripped of special
unicode characters that Lupa dislikes.
"""
with archive.open(filepath) as target_file:
# "utf-8-sig" makes sure to strip BOM tokens if they exist
formatted = io.TextIOWrapper(target_file, encoding="utf-8-sig")
return formatted.read() Replacing all instances of |
Awesome. Thanks for the fix. It seems like you've found a solution that should be robust as well. I've edited the first comment of this issue to include a full mod list. |
Alright, I think I finally got it working. Turns out there was some very strange code like: if ... ~= "__mod-name__/file" then
require("__mod-name__/file")
end ... which if you normalize the names to use proper paths instead of the double underscore syntax, this obviously no longer works and causes an infinite recursion. I looked everywhere for a simpler solution, but the only method I could get working required overwriting the default Lua file searcher, which was a pain. Fortunately, this means that I can now use python itself for all the file requiring, which is precisely what now is done: instead of using Lua file loading, python passes a file handle from which data is properly formatted and read, meaning that BOM encoding issues should (hopefully) never happen again under any circumstance. I didn't do much rigorous testing on whether or not all the data is correct, but I have no real reason to doubt that it would be and would probably fall outside the scope of this particular issue anyway. A cursory glance seems convincing enough: >>> from draftsman.data import entities
>>> entities.assembling_machines
['assembling-machine-1', 'assembling-machine-2', 'assembling-machine-3', 'oil-refinery', 'chemical-plant', 'centrifuge', 'glassworks-mk01', 'advanced-foundry-mk01', 'automated-factory-mk01', 'ground-borer', 'wpu', 'ball-mill-mk01', 'chemical-plant-mk01', 'carbon-filter', 'sand-extractor', 'classifier', 'soil-extractormk01', 'desulfurizator-unit', 'distilator', 'evaporator', 'fluid-separator', 'fts-reactor', 'gasifier', 'hpf', 'methanol-reactor', 'olefin-plant', 'jaw-crusher', 'power-house', 'quenching-tower', 'rectisol', 'solid-separator', 'tar-processing-unit', 'washer', 'glassworks-mk02', 'advanced-foundry-mk02', 'automated-factory-mk02', 'ground-borer-mk02', 'wpu-mk02', 'ball-mill-mk02', 'chemical-plant-mk02', 'carbon-filter-mk02', 'sand-extractor-mk02', 'classifier-mk02', 'soil-extractormk02', 'desulfurizator-unit-mk02', 'distilator-mk02', 'evaporator-mk02', 'fluid-separator-mk02', 'fts-reactor-mk02', 'gasifier-mk02', 'hpf-mk02', 'methanol-reactor-mk02', 'olefin-plant-mk02', 'jaw-crusher-mk02', 'power-house-mk02', 'quenching-tower-mk02', 'rectisol-mk02', 'solid-separator-mk02', 'tar-processing-unit-mk02', 'washer-mk02', 'glassworks-mk03', 'advanced-foundry-mk03', 'automated-factory-mk03', 'ground-borer-mk03', 'wpu-mk03', 'ball-mill-mk03', 'chemical-plant-mk03', 'carbon-filter-mk03', 'sand-extractor-mk03', 'classifier-mk03', 'soil-extractormk03', 'desulfurizator-unit-mk03', 'distilator-mk03', 'evaporator-mk03', 'fluid-separator-mk03', 'fts-reactor-mk03', 'gasifier-mk03', 'hpf-mk03', 'methanol-reactor-mk03', 'olefin-plant-mk03', 'jaw-crusher-mk03', 'power-house-mk03', 'quenching-tower-mk03', 'rectisol-mk03', 'solid-separator-mk03', 'tar-processing-unit-mk03', 'washer-mk03', 'glassworks-mk04', 'advanced-foundry-mk04', 'automated-factory-mk04', 'ground-borer-mk04', 'wpu-mk04', 'ball-mill-mk04', 'chemical-plant-mk04', 'carbon-filter-mk04', 'sand-extractor-mk04', 'classifier-mk04', 'soil-extractormk04', 'desulfurizator-unit-mk04', 'distilator-mk04', 'evaporator-mk04', 'fluid-separator-mk04', 'fts-reactor-mk04', 'gasifier-mk04', 'hpf-mk04', 'methanol-reactor-mk04', 'olefin-plant-mk04', 'jaw-crusher-mk04', 'power-house-mk04', 'quenching-tower-mk04', 'rectisol-mk04', 'solid-separator-mk04', 'tar-processing-unit-mk04', 'washer-mk04', 'mukmoux-pasture', 'cooling-tower-mk01', 'cooling-tower-mk02', 'agitator-mk01', 'nmf-mk01', 'secondary-crusher-mk01', 'thickener-mk01', 'gas-separator-mk01', 'hydrocyclone-mk01', 'vacuum-pump-mk01', 'automated-screener-mk01', 'centrifugal-pan-mk01', 'compressor-mk01', 'jig-mk01', 'grease-table-mk01', 'mixer-mk01', 'py-heat-exchanger', 'agitator-mk02', 'nmf-mk02', 'secondary-crusher-mk02', 'thickener-mk02', 'gas-separator-mk02', 'hydrocyclone-mk02', 'vacuum-pump-mk02', 'automated-screener-mk02', 'centrifugal-pan-mk02', 'compressor-mk02', 'jig-mk02', 'grease-table-mk02', 'mixer-mk02', 'py-heat-exchanger-mk02', 'agitator-mk03', 'nmf-mk03', 'secondary-crusher-mk03', 'thickener-mk03', 'gas-separator-mk03', 'hydrocyclone-mk03', 'vacuum-pump-mk03', 'automated-screener-mk03', 'centrifugal-pan-mk03', 'compressor-mk03', 'jig-mk03', 'grease-table-mk03', 'mixer-mk03', 'py-heat-exchanger-mk03', 'agitator-mk04', 'nmf-mk04', 'secondary-crusher-mk04', 'thickener-mk04', 'gas-separator-mk04', 'hydrocyclone-mk04', 'vacuum-pump-mk04', 'automated-screener-mk04', 'centrifugal-pan-mk04', 'compressor-mk04', 'jig-mk04', 'grease-table-mk04', 'mixer-mk04', 'py-heat-exchanger-mk04', 'fusion-reactor-mk01', 'fusion-reactor-mk02', 'centrifuge-mk01',
'neutron-absorber-mk01', 'neutron-moderator-mk01', 'nuclear-reactor-mk01', 'py-biomass-powerplant-mk01', 'py-coal-powerplant-mk01', 'py-gas-powerplant-mk01', 'py-oil-powerplant-mk01', 'centrifuge-mk02', 'neutron-absorber-mk02', 'neutron-moderator-mk02', 'nuclear-reactor-mk02', 'py-biomass-powerplant-mk02', 'py-coal-powerplant-mk02', 'py-gas-powerplant-mk02', 'py-oil-powerplant-mk02', 'centrifuge-mk03', 'neutron-absorber-mk03', 'neutron-moderator-mk03', 'nuclear-reactor-mk03', 'py-biomass-powerplant-mk03', 'py-coal-powerplant-mk03', 'py-gas-powerplant-mk03', 'py-oil-powerplant-mk03', 'centrifuge-mk04', 'neutron-absorber-mk04', 'neutron-moderator-mk04', 'nuclear-reactor-mk04', 'py-biomass-powerplant-mk04', 'py-coal-powerplant-mk04', 'py-gas-powerplant-mk04', 'py-oil-powerplant-mk04', 'lrf-building-mk01', 'solar-concentrator', 'atomizer-mk01', 'bio-printer-mk01', 'bio-reactor-mk01', 'biofactory-mk01', 'creature-chamber-mk01', 'genlab-mk01', 'incubator-mk01', 'micro-mine-mk01', 'research-center-mk01', 'slaughterhouse-mk01', 'spore-collector-mk01', 'botanical-nursery', 'rc-mk01', 'plankton-farm', 'guar-gum-plantation', 'fwf-mk01', 'cadaveric-arum-mk01', 'grods-swamp-mk01', 'kicalk-plantation-mk01', 'ralesia-plantation-mk01', 'rennea-plantation-mk01', 'sap-extractor-mk01', 'tuuphra-plantation-mk01', 'yotoi-aloe-orchard-mk01', 'bhoddos-culture-mk01', 'fawogae-plantation-mk01', 'navens-culture-mk01', 'yaedols-culture-mk01', 'arqad-hive-mk01', 'arthurian-pen-mk01', 'auog-paddock-mk01', 'cridren-enclosure-mk01', 'dingrits-pack-mk01', 'kmauts-enclosure-mk01', 'mukmoux-pasture-mk01', 'phadai-enclosure-mk01', 'phagnot-corral-mk01', 'prandium-lab-mk01', 'scrondrix-pen-mk01', 'simik-den-mk01', 'ulric-corral-mk01', 'vonix-den-mk01', 'vrauks-paddock-mk01', 'xenopen-mk01', 'zungror-lair-mk01', 'ez-ranch-mk01', 'dhilmos-pool-mk01', 'fish-farm-mk01', 'numal-reef-mk01', 'trits-reef-mk01', 'xyhiphoe-pool-mk01', 'zipir-reef-mk01', 'seaweed-crop-mk01', 'moss-farm-mk01', 'sponge-culture-mk01',
'atomizer-mk02', 'bio-printer-mk02', 'bio-reactor-mk02', 'biofactory-mk02', 'creature-chamber-mk02', 'genlab-mk02', 'incubator-mk02', 'micro-mine-mk02', 'research-center-mk02', 'slaughterhouse-mk02', 'spore-collector-mk02', 'botanical-nursery-mk02', 'rc-mk02', 'plankton-farm-mk02', 'guar-gum-plantation-mk02', 'fwf-mk02', 'cadaveric-arum-mk02', 'grods-swamp-mk02', 'kicalk-plantation-mk02', 'ralesia-plantation-mk02', 'rennea-plantation-mk02', 'sap-extractor-mk02', 'tuuphra-plantation-mk02', 'yotoi-aloe-orchard-mk02', 'bhoddos-culture-mk02', 'fawogae-plantation-mk02', 'navens-culture-mk02', 'yaedols-culture-mk02', 'arqad-hive-mk02', 'arthurian-pen-mk02', 'auog-paddock-mk02', 'cridren-enclosure-mk02', 'dingrits-pack-mk02', 'kmauts-enclosure-mk02', 'mukmoux-pasture-mk02', 'phadai-enclosure-mk02', 'phagnot-corral-mk02', 'prandium-lab-mk02', 'scrondrix-pen-mk02', 'simik-den-mk02', 'ulric-corral-mk02', 'vonix-den-mk02', 'vrauks-paddock-mk02', 'xenopen-mk02', 'zungror-lair-mk02', 'ez-ranch-mk02', 'dhilmos-pool-mk02', 'fish-farm-mk02', 'numal-reef-mk02', 'trits-reef-mk02', 'xyhiphoe-pool-mk02', 'zipir-reef-mk02', 'seaweed-crop-mk02', 'moss-farm-mk02', 'sponge-culture-mk02', 'atomizer-mk03', 'bio-printer-mk03', 'bio-reactor-mk03', 'biofactory-mk03', 'creature-chamber-mk03', 'genlab-mk03', 'incubator-mk03', 'micro-mine-mk03',
'research-center-mk03', 'slaughterhouse-mk03', 'spore-collector-mk03', 'botanical-nursery-mk03', 'rc-mk03', 'plankton-farm-mk03', 'guar-gum-plantation-mk03', 'fwf-mk03', 'cadaveric-arum-mk03', 'grods-swamp-mk03', 'kicalk-plantation-mk03', 'ralesia-plantation-mk03', 'rennea-plantation-mk03', 'sap-extractor-mk03', 'tuuphra-plantation-mk03', 'yotoi-aloe-orchard-mk03', 'bhoddos-culture-mk03', 'fawogae-plantation-mk03', 'navens-culture-mk03', 'yaedols-culture-mk03', 'arqad-hive-mk03', 'arthurian-pen-mk03', 'auog-paddock-mk03', 'cridren-enclosure-mk03', 'dingrits-pack-mk03', 'kmauts-enclosure-mk03', 'mukmoux-pasture-mk03', 'phadai-enclosure-mk03', 'phagnot-corral-mk03', 'prandium-lab-mk03', 'scrondrix-pen-mk03', 'simik-den-mk03', 'ulric-corral-mk03', 'vonix-den-mk03', 'vrauks-paddock-mk03', 'xenopen-mk03', 'zungror-lair-mk03', 'ez-ranch-mk03', 'dhilmos-pool-mk03', 'fish-farm-mk03', 'numal-reef-mk03', 'trits-reef-mk03', 'xyhiphoe-pool-mk03', 'zipir-reef-mk03', 'seaweed-crop-mk03', 'moss-farm-mk03', 'sponge-culture-mk03', 'atomizer-mk04', 'bio-printer-mk04', 'bio-reactor-mk04', 'biofactory-mk04', 'creature-chamber-mk04', 'genlab-mk04', 'incubator-mk04', 'micro-mine-mk04', 'research-center-mk04', 'slaughterhouse-mk04', 'spore-collector-mk04', 'botanical-nursery-mk04', 'rc-mk04', 'plankton-farm-mk04', 'guar-gum-plantation-mk04', 'fwf-mk04', 'cadaveric-arum-mk04', 'grods-swamp-mk04', 'kicalk-plantation-mk04', 'ralesia-plantation-mk04', 'rennea-plantation-mk04', 'sap-extractor-mk04', 'tuuphra-plantation-mk04', 'yotoi-aloe-orchard-mk04', 'bhoddos-culture-mk04', 'fawogae-plantation-mk04', 'navens-culture-mk04', 'yaedols-culture-mk04', 'arqad-hive-mk04', 'arthurian-pen-mk04', 'auog-paddock-mk04', 'cridren-enclosure-mk04', 'dingrits-pack-mk04', 'kmauts-enclosure-mk04', 'mukmoux-pasture-mk04', 'phadai-enclosure-mk04', 'phagnot-corral-mk04', 'prandium-lab-mk04', 'scrondrix-pen-mk04', 'simik-den-mk04', 'ulric-corral-mk04', 'vonix-den-mk04', 'vrauks-paddock-mk04', 'xenopen-mk04', 'ez-ranch-mk04', 'dhilmos-pool-mk04', 'fish-farm-mk04', 'numal-reef-mk04', 'trits-reef-mk04', 'xyhiphoe-pool-mk04', 'zipir-reef-mk04', 'seaweed-crop-mk04', 'moss-farm-mk04', 'sponge-culture-mk04', 'simik-boiler', 'data-array', 'antelope-enclosure-mk01', 'pyphoon-bay', 'vat-brain', 'dino-dig-site', 'clay-pit-mk01', 'chipshooter-mk01', 'pcb-factory-mk01', 'fbreactor-mk01', 'particle-accelerator-mk01', 'electronics-factory-mk01', 'pulp-mill-mk01', 'nano-assembler-mk01', 'clay-pit-mk02', 'chipshooter-mk02', 'pcb-factory-mk02', 'fbreactor-mk02', 'particle-accelerator-mk02', 'electronics-factory-mk02', 'pulp-mill-mk02', 'nano-assembler-mk02', 'clay-pit-mk03', 'chipshooter-mk03', 'pcb-factory-mk03', 'fbreactor-mk03', 'particle-accelerator-mk03', 'electronics-factory-mk03', 'pulp-mill-mk03', 'nano-assembler-mk03', 'clay-pit-mk04', 'chipshooter-mk04', 'pcb-factory-mk04', 'fbreactor-mk04', 'electronics-factory-mk04', 'pulp-mill-mk04', 'kicalk-plantation', 'moondrop-greenhouse-mk01', 'moondrop-greenhouse-mk02', 'moondrop-greenhouse-mk03', 'moondrop-greenhouse-mk04', 'zipir', 'auog-paddock', 'quantum-computer', 'coalbed-mk01', 'cracker-mk01', 'gas-refinery-mk01', 'heavy-oil-refinery-mk01', 'lor-mk01', 'pumpjack-mk01', 'reformer-mk01', 'tholin-atm-mk01', 'tholin-plant-mk01', 'upgrader-mk01', 'coalbed-mk02', 'cracker-mk02', 'gas-refinery-mk02', 'heavy-oil-refinery-mk02', 'lor-mk02', 'pumpjack-mk02', 'reformer-mk02', 'tholin-atm-mk02', 'tholin-plant-mk02', 'upgrader-mk02', 'coalbed-mk03', 'cracker-mk03', 'gas-refinery-mk03', 'heavy-oil-refinery-mk03', 'lor-mk03', 'pumpjack-mk03', 'reformer-mk03', 'tholin-atm-mk03', 'tholin-plant-mk03', 'upgrader-mk03', 'coalbed-mk04', 'cracker-mk04', 'gas-refinery-mk04', 'heavy-oil-refinery-mk04', 'lor-mk04', 'pumpjack-mk04', 'reformer-mk04', 'tholin-atm-mk04', 'tholin-plant-mk04', 'upgrader-mk04', 'fracking-rig', 'retorter', 'rhe', 'bof-mk01', 'smelter-mk01', 'casting-unit-mk01', 'eaf-mk01', 'electrolyzer-mk01', 'flotation-cell-mk01', 'hydroclassifier-mk01',
'impact-crusher-mk01', 'leaching-station-mk01', 'scrubber-mk01', 'wet-scrubber-mk01', 'bof-mk02', 'smelter-mk02', 'casting-unit-mk02', 'eaf-mk02', 'electrolyzer-mk02', 'flotation-cell-mk02', 'hydroclassifier-mk02', 'impact-crusher-mk02', 'leaching-station-mk02', 'scrubber-mk02', 'wet-scrubber-mk02', 'bof-mk03', 'smelter-mk03', 'casting-unit-mk03', 'eaf-mk03', 'electrolyzer-mk03', 'flotation-cell-mk03', 'hydroclassifier-mk03', 'impact-crusher-mk03', 'leaching-station-mk03', 'scrubber-mk03', 'wet-scrubber-mk03', 'bof-mk04', 'smelter-mk04', 'casting-unit-mk04', 'eaf-mk04', 'electrolyzer-mk04', 'flotation-cell-mk04', 'hydroclassifier-mk04', 'impact-crusher-mk04',
'leaching-station-mk04', 'scrubber-mk04', 'wet-scrubber-mk04', 'drp', 'sinter-unit', 'bitumen-seep-mk01-base', 'bitumen-seep-mk02-base', 'bitumen-seep-mk03-base', 'bitumen-seep-mk04-base', 'natural-gas-seep-mk01-base', 'tar-seep-mk01-base'] In addition, I added a prototype version of that mod-list printing command, currently prototyped as
Gimme a bit to clean up and then I'll push this for |
In case it matters this is in PyCharm IDE on Windows 10.
I was making an attempt to setup draftsman with most of the Pyanodons factorio mods installed.
Running draftsman-update resulted in the following stack trace
After some debugging, it looks like the root cause is that lupa (at least as it is currently invoked) does not handle executing when passed bytes that begin with the UTF-8 Byte Order Mark (BOM):
EF BB BF
(in hex). EF = 239The source of the BOM appears to be the
pypostprocessing_0.2.8
mod'ssettings-updates.lua
file. I had to open the file with a hex editor for the BOM to be visible.I was able to fix this by locally modifying draftsman's env.py to wrap lua file loads with the following
I'm sure there is a better way to solve the issue, but providing above as proof of concept.
P.S. That wasn't the only error, but I'll create separate issues as I make further attempts to get draftsman to update with py mods :)
Full mod list
Installed and active
AdditionalPasteSettings_1.6.4.zip
Automatic_Train_Painter_1.2.0.zip
AutoTrash_5.3.13.zip
BottleneckLite_1.2.4.zip
cybersyn_1.2.16.zip
even-distribution_1.0.10.zip
factoryplanner_1.1.66.zip
flib_0.12.9.zip
Milestones_1.3.18.zip
PickerDollies_1.2.6.zip
pushbutton_1.1.1.zip
pyalienlifegraphics2_2.0.6.zip
pyalienlifegraphics3_2.0.5.zip
pyalienlifegraphics_2.0.3.zip
pyalienlife_2.1.5.zip
pyalternativeenergygraphics_1.0.5.zip
pyalternativeenergy_1.2.3.zip
pycoalprocessinggraphics_2.0.6.zip
pycoalprocessing_2.1.2.zip
pyfusionenergygraphics_2.0.0.zip
pyfusionenergy_2.0.5.zip
pyhightechgraphics_2.0.1.zip
pyhightech_2.0.8.zip
pyindustry_2.0.8.zip
pypetroleumhandlinggraphics_2.0.0.zip
pypetroleumhandling_2.1.8.zip
pypostprocessing_0.2.8.zip
pyraworesgraphics_2.0.1.zip
pyrawores_2.4.8.zip
RecipeBook_3.5.5.zip
Shuttle_Train_Continued_1.1.0.zip
Squeak Through_1.8.2.zip
stdlib_1.4.8.zip
textplates_0.6.10.zip
VehicleSnap_1.18.5.zip
Installed and inactive
(alien-biomes-hr-terrain_0.6.1.zip)
(alien-biomes_0.6.8.zip)
(EditorExtensions_2.1.5.zip)
(me playing around with modding)
The text was updated successfully, but these errors were encountered: