I think earlier there was a problem that the hustle code had some parts (especially some wrappers that were written around some arrow functions) which required a particular version of the arrow to work correctly.

Seems like we're not using any modified arrow? Then shall we just make arrow a part of the CMake dependency?

Yes, it seems things are fine! If with the latest arrow version there's no issue with the tests, then we should be good!

We did download arrow using a custom script to be able to control the version of arrow.
@srsuryadev is right, this choice was a result of an incident where arrow updated some internal method we depended on.

Can we control the arrow version when we make arrow part of the cmake toolchain? --even if the answer is no I like the cleaner flow that making arrow part of the cmake toolchain gives us

One question you might want to answer between the 3 of you (@srsuryadev, @GindaChen, @mprammer2) is if we still depend on arrow methods that are not part of their external api which might change from version to version. -- if yes we might want to keep track of them and anticipate possible changes

Can we control the arrow version when we make arrow part of the cmake toolchain?

Yes - and we can still do custom repo to host our arrow in CMake.

Shall we depend on arrow API?

I don't know... eventually maybe not, but we may want to discuss this a bit further. Adding another abstraction on compute, not the type system, could be beneficial; but the whether the type system should depend on that is a big question.