Failure to build the example due to asdf search paths

[fakedrake]

I was unable to run the example unfortunately. I use roswell so I cloned into ~/.roswell/local-projects. Then I navigated to ~/.roswell/local-projects/sbcl-librarian/example and

make SBCL_SRC=$HOME/.roswell/src/sbcl-2.2.2-arm64-darwin

But it couldn't find the '#:libcalc system so I changed line 3 in example/script.lisp to

(asdf:load-asd (truename "./libcalc.asd"))

Because sbcl still couldn't find sbcl-librarian on its own I tried runing script.lisp with roswell directly

$ ros run -l script.lisp -q  
...

debugger invoked on a PACKAGE-DOES-NOT-EXIST in thread
#<THREAD "main thread" RUNNING {700863FBB3}>:
  The name "SBCL-LIBRARIAN/EXAMPLE/LIBCALC" does not designate any package.

I don't know how to deal with this one. The readme gives some instructions but they are all the way at the end of the process, it would be nice to have some instructions that start at "where do I clone this project?".

[karlosz]

We haven't tried this with Roswell. Have you been able to reproduce this issue vanilla?

[fakedrake]

I had essentially the same problem

$ make SBCL_SRC=$HOME/Projects/sbcl                        
/Users/cperivol/Projects/sbcl/run-sbcl.sh --script "script.lisp"
While evaluating the form starting at line 1, column 0
  of #P"/Users/cperivol/quicklisp/local-projects/sbcl-librarian/example/libcalc.asd":
Unhandled MISSING-DEPENDENCY in thread #<SB-THREAD:THREAD "main thread" RUNNING {70086705A3}>: Component "sbcl-librarian" not found, required by NIL

...
$ ls ~/quicklisp/local-projects 
sbcl-librarian   system-index.txt

I am probably using quicklisp wrong, any help would be much appreciated.

EDIT: Note that I am still using the change in example/script.lisp I mentioned.

[fakedrake]

I have a new sticking point: I can't find libsbcl.so.

My previous problem was fixed by using script.lisp

(require '#:asdf)

(asdf:load-asd (truename "./libcalc.asd"))
(asdf:load-system '#:libcalc)

(in-package #:sbcl-librarian/example/libcalc)

(build-bindings libcalc ".")
(build-python-bindings libcalc ".")
(build-core-and-die libcalc ".")

and use roswell to run it.

EDIT: You need to run sh ./make-shared-library.sh in the sbcl source root you use for building the example.

[stylewarning]

@fakedrake Did you get it to work with those additional steps? We can probably update the readme to make the steps more clear.

[fakedrake]

I was able to build libcalc.dylib and libcalc.h but I couldn't compile example.c because

$ gcc example.c -o example -lsbcl -lcalc -L.
example.c:36:5: error: implicit declaration of function 'lisp_release_handle' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    lisp_release_handle(expr);
    ^
1 error generated.

The generated libcalc.h is

#ifndef _libcalc_h
#define _libcalc_h

#if defined(CALC_API_BUILD)
#  if defined(_WIN64)
#    define CALC_API __declspec(dllexport)
#  elif defined(__ELF__)
#    define CALC_API __attribute__ ((visibility ("default")))
#  else
#    define CALC_API
# endif
#else
#  if defined(_WIN64)
#    define CALC_API __declspec(dllimport)
#  else
#  define CALC_API
#  endif
#endif

/* types */
struct expr_type__ { int unused; }; typedef struct expr_type__ *expr_type;
typedef enum { ERR_SUCCESS = 0, ERR_FAIL = 1, } err_t;
/* functions */
CALC_API err_t (*calc_int_literal)(int value, expr_type *result);
CALC_API err_t (*calc_int_literal_value)(expr_type expr, int *result);
CALC_API err_t (*calc_int_literal_p)(expr_type obj, int *result);
CALC_API err_t (*calc_sum_expression)(expr_type left, expr_type right, expr_type *result);
CALC_API err_t (*calc_sum_expression_left_arg)(expr_type expr, expr_type *result);
CALC_API err_t (*calc_sum_expression_right_arg)(expr_type expr, expr_type *result);
CALC_API err_t (*calc_sum_expression_p)(expr_type expr, int *result);
CALC_API err_t (*calc_simplify)(expr_type expr, expr_type *result);
CALC_API err_t (*calc_parse)(char* source, expr_type *result);
CALC_API err_t (*calc_expression_to_string)(expr_type expr, char* *result);
CALC_API err_t (*calc_remove_zeros)(expr_type expr, expr_type *result);
CALC_API int init(char* core);

#endif

[svetlyak40wt]

I was able to build libcalc.dylib and libcalc.h but I couldn't compile example.c because

How did you build libcalc.dylib? I use this command:

gcc -v -shared libcalc.o -o libcalc.so -L ~/.roswell/src/sbcl-2.2.2/src/runtime/ -lsbcl

But get such error:

Apple clang version 13.0.0 (clang-1300.0.29.30)
Target: arm64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
 "/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -dynamic -dylib -arch arm64 -platform_version macos 11.0.0 12.1 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o libcalc.so -L/Users/art/.roswell/src/sbcl-2.2.2/src/runtime/ -L/usr/local/lib libcalc.o -lsbcl -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a
Undefined symbols for architecture arm64:
  "_initialize_lisp", referenced from:
      _init in libcalc.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[fakedrake]

I didn't mention all my steps to avoid derailing the conversation but here are all my steps:

cd ~/.roswell/local-projects/
git clone https://github.com/quil-lang/sbcl-librarian
cd sbcl-librarian/example/

now in script.lisp, right after (require '#:asdf) insert (asdf:load-asd (truename "./libcalc.asd")). There's probably a better way to do this but I havent figured it out.

Then to generate libcalc.core libcalc.c libcalc.h and libcalc.py

ros run -l "script.lisp" -q

Now you need the sbcl library which is not shipped with roswell so you need to build it yourself from your own copy of the source

cd ~/Projects
git clone https://github.com/sbcl/sbcl
cd sbcl
./make-shared-library.sh

Now you have sbcl shared library so you can build libcalc

cd ~/.roswell/local-projects/sbcl-librarian/example/
gcc -dynamiclib -o libcalc.dylib libcalc.c -L$HOME/Projects/sbcl/src/runtime -lsbcl

So now you have libcalc.dylib but you may have the problem I mentioned in my previous comment when you try to use it.

As an aside, example.py doesn't work either and the error maybe is a peek into the future of trying to build example.c

$ ln -s $HOME/Projects/sbcl/src/runtime/libsbcl.so ./ # because I couldn't convince python SDLL to look to the right place
$ python3 example.py
fatal error encountered in SBCL pid 42443 pthread 0x104fac580:
core was built for runtime "m1mini1-snmsts-2022-02-27-19-00-19" but this is "host.local-cperivol-2022-03-14-00-51-53"

EDIT: I tried using the sbcl implementation rather than sbcl-bin. This one allows me to build against the roswell sbcl rather than a locally cloned copy but the error is very similar:

$ python3 example.py
fatal error encountered in SBCL pid 44421 pthread 0x104818580:
core was built for runtime "host.local-cperivol-2022-03-15-14-43-55" but this is "host.local-cperivol-2022-03-14-00-51-53"

The fundamental problem here is that sbcl builds the compiler and the library indifferent steps and therefore they are assigned different build ids.

[fakedrake]

I made an extremely dirty fix for the python problem:

# Found the actual id from the error message. The XXX in "build for runtime XXX". 
$ echo '"host.local-cperivol-2022-03-15-14-43-55"' > ~/.roswell/src/sbcl-2.2.2/output/build-id.inc
$ cd ~/.roswell/src/sbcl-2.2.2/
$ make -C src/runtime/ clean && ./make-shared-library.sh
$ cd ~/.roswell/local-projects/sbcl-librarian/example
$ python3 example.py
> (+ 10 30)

40

Traceback (most recent call last):
  File "/Users/cperivol/.roswell/local-projects/sbcl-librarian/example/example.py", line 36, in <module>
    libcalc.calc_release_handle(expr)
AttributeError: module 'libcalc' has no attribute 'calc_release_handle'

[karlosz]

Hi,

The core file mismatch is exactly that: The sbcl build you made the core with (i.e. ran the build script on) is not the same as the libsbcl.so dynamic library you built.

In the future this will be addressed in SBCL itself by having the binary sbcl distribution installation (including from package managers) include libsbcl.so in the dynamic library path of your OS, like how ECL provides their shared library support, so users won't need to download a copy of the source and build a differing library version themselves.

[svetlyak40wt]

I was able to build libcalc.dylib and libcalc.h but I couldn't compile example.c because

$ gcc example.c -o example -lsbcl -lcalc -L.
example.c:36:5: error: implicit declaration of function 'lisp_release_handle' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    lisp_release_handle(expr);
    ^
1 error generated.

Seems this code from handles.lisp:

(define-api handles (:function-prefix "")
  (:function
   (("lisp_release_handle" release-handle) :void ((handle :pointer)))
   (("lisp_handle_eq" handle-eq) :bool ((a :pointer) (b :pointer)))))

should also find it's way to some *.h file. but these declarations are missing from libcalc.h

[svetlyak40wt]

I was able to overcome this lisp_release_handle when compiling an example binary, by applying this patch:

diff --git a/library.lisp b/library.lisp
index 817ddef..3a82e0a 100644
--- a/library.lisp
+++ b/library.lisp
@@ -35,7 +35,8 @@ Here ERROR-MAP, FUNCTION-PREFIX, and SPECS identify the arguments of DEFINE-API
          ,@(callable-definitions-from-spec function-prefix error-map specs)
          (define-aggregate-library ,name
              (:function-linkage ,function-linkage)
-           ,library-api)))))
+           ,library-api
+           sbcl-librarian:handles)))))

After applying it and regenerating libcalc bindings using (build-bindings libcalc "."), these two declarations were added to libcalc.h:

[art@art-osx:...ects/lisp/sbcl-librarian]% diff libcalc.h libcalc-before.h
35,36d34
< CALC_API void (*lisp_release_handle)(void* handle);
< CALC_API int (*lisp_handle_eq)(void* a, void* b);

But this didn't help to solve "core file mismatch" problem :(

[fakedrake]

I was just pushing the exact same fix and it looks like @karlosz fixed it in a different way a few hours ago on the main repo.
For the core mismatch did you try what I did on my other comment (replacing for your own build ID)?

The example works fine for me now. I have made a few more changes to automate a part of the process and I updated the README (although it is now centered around roswell)

[kartik-s]

Just wanted to summarize my thoughts on the three distinct issues I see here:

ASDF can't find the libcalc system when running the Makefile

This is unrelated to Roswell. The problem here is that ASDF does not know to look for system definitions under local-projects unless Quicklisp has been loaded into the image.

Our Makefile does not currently ensure that Quicklisp is loaded before running script.lisp. There are two ways to fix this:

• Assume the user has run the ql:add-to-init-file function to add Quicklisp loading to their .sbclrc file, and then replace --script with --load since --script tells SBCL not to read .sbclrc.
• Explicitly load Quicklisp in the Makefile before running script.lisp (e.g. as done in the Quilc Makefile)

Missing libsbcl.so/core mismatch

As @karlosz noted above, this can be fixed properly by including libsbcl.so in the list of things that the SBCL installer installs. I have authored a quick patch to do this, and it can probably hit the next release:

From cc84c33e21e434095e48620f4733cef50da0b10b Mon Sep 17 00:00:00 2001
From: Kartik Singh <[email protected]>
Date: Tue, 15 Mar 2022 13:38:54 -0700
Subject: [PATCH] Add libsbcl.so to installation

---
 .gitignore | 1 +
 NEWS       | 1 +
 install.sh | 1 +
 make.sh    | 1 +
 4 files changed, 4 insertions(+)

diff --git a/.gitignore b/.gitignore
index e4ec7a286..9332c1488 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,7 @@
 *.d
 *.dSYM
 *.o
+*.so
 *.a
 *.fasl
 *.FASL
diff --git a/NEWS b/NEWS
index 56c4281a8..5ce72cb49 100644
--- a/NEWS
+++ b/NEWS
@@ -8,6 +8,7 @@ changes relative to sbcl-2.2.2:
   * optimization: fasls containing standard object literals are now smaller
     and load more efficiently.
   * bug fix: EQness of constants is now always preserved when block compiling.
+  * enhancement: install.sh now also installs libsbcl.so
 
 changes in sbcl-2.2.2 relative to sbcl-2.2.1:
   * platform support:
diff --git a/install.sh b/install.sh
index 1246ecf59..5497be59f 100755
--- a/install.sh
+++ b/install.sh
@@ -113,6 +113,7 @@ test -f "$BUILD_ROOT$SBCL_HOME"/sbcl.core && \
 
 cp src/runtime/$RUNTIME "$BUILD_ROOT$INSTALL_ROOT"/bin/
 cp output/sbcl.core "$BUILD_ROOT$SBCL_HOME"/sbcl.core
+cp src/runtime/libsbcl.so "$BUILD_ROOT$INSTALL_ROOT"/lib
 cp src/runtime/sbcl.mk "$BUILD_ROOT$SBCL_HOME"/sbcl.mk
 for i in $(grep '^LIBSBCL=' src/runtime/sbcl.mk | cut -d= -f2-) ; do
     cp "src/runtime/$i" "$BUILD_ROOT$SBCL_HOME/$i"
diff --git a/make.sh b/make.sh
index 9a8522631..a4ad5eb95 100755
--- a/make.sh
+++ b/make.sh
@@ -81,6 +81,7 @@ maybetime sh make-target-1.sh
 maybetime sh make-host-2.sh
 maybetime sh make-target-2.sh
 maybetime sh make-target-contrib.sh
+maybetime sh make-shared-library.sh
 
 # contrib/Makefile shouldn't be counted in NCONTRIBS.
 # "find contrib/* -name Makefile" would still find contrib/./Makefile.
-- 
2.24.3 (Apple Git-128)

The other workaround for now (besides overwriting the build ID as demonstrated by @fakedrake) is to build SBCL from source and run sh install.sh to install it over the installation from your package manager in addition to building the shared library.

C example fails to compile because of missing definitions

As noted by others this has been fixed by this commit.