Hi all,
I am in the process of converting a fairly complicated proprietary simulation model that I wrote in Python to Julia 1.5.3. The code is broken into around 20 different sub packages on the file system by their purposes for organization and reusability. Each sub package also contains a tests folder that contains the unit tests for the different modules inside the sub package. In Python this can be done very easily because each .py file is a module, each directory is a package containing these modules, all imports should be done explicitly (ie: from foo_dir.foo import stuff at the top of bar.py) and you can configure PyTest to systematically find and execute all unit tests in all packages in the entire project folder tree. I have spent a couple of days trying to figure out how to do all of this properly in Julia. Couple of things I have tried:
A. Use include("relative path to foo.jl") directly at the top of bar.jl
Pro:
- Very easy to use
- Very easy to understand.
- Both Juno and VSCode intellisense plays well with this method.
Con:
-
If foo.jl moves on the file system modification process is initially simple. However this quickly becomes a pain if a lot of barN.jl includes foo.jl all over the place. Python actually suffers from the very same problem but PyCharm tends to refactor this automatically so it’s not a super big issue.
-
Can cause redefinition of global constants in foo.jl if
include(foo.jl)were called multiple times due to multiple inclusions. No control over what’s included from foo.jl and what’s not which means name collisions can happen easily. This will basically make it useful for just very small projects. This is a deal breaker.
B. Wrap code in foo.jl in module Foo ... end, export desired public stuff from foo.jl. Import with using .Foo.x, .Foo.y or import .Foo.x, .Foo.y
Pro: AFAIK, none! More on this in the Con section.
Con:
-
This only “looks” more like what Python does but in order to actually import/using the module it either requires
include("foo.jl")just like A (and therefore inherit all of A’s cons) or adding the code to path. -
Actual name from Foo can live under all kinds of weird prefixes if done without Reexport and the @reexport macro. This feels hacky and tbh, pretty unintutive.
C. Build sub packages into actual packages. Do generate foo_dir/foo under Pkg mode and use dev foo_dir/foo in bar’s environment. Any sub packages (ie: bar) that want to use/extend foo can do import Foo.x, Foo.y etc in its code.
Pro:
-
This is by far the most well behaving solution and the one I am leaning towards. In practice everything pretty much behaves like how I want it to because AFAIK it’s kinda like “installing” the Foo package into the parent environment (in dev mode but whatever).
-
I haven’t tried out setting up all the unit tests for different sub packages yet but I imagine it’s pretty much the same process as setting up the unit tests for just a single package.
Con:
-
Environments everywhere. Every sub package gets its own environment with its own dependency (ie: 20 different Manifest.toml and Project.toml in the project repository at various locations). This also makes me worried about the possible clusterfuck I might have to deal with when some foo.jl inevitably has to move out/get merged from/into another (new) sub package during development.
-
It’s a pain to generate the proper file system tree for the sub packages. First of all the source files for each sub package now live under the
subpackage_dir/src/directory instead of at the subpackage root dir. TBH I can live with this. The other problem is in order to generate the package at the correct file system location you have to be careful with which environment you have activated right now. This is quite a bit more thinking and running commands and swapping environments than I like for something that should be very simple. -
VSCode intellisense seems to play with this badly. I am guessing this might have something to do with the cache its language server generates is based on hashing the version number of the package. Since the sub package isn’t really a package that is being distributed over some registry, its version number won’t be updated for different releases either (and therefore cache doesn’t get regenerated when say, foo.jl changes). Again, just a guess. At least Juno seems to do intellisense properly under this method.
It’s entirely possible I have missed something simple and obvious. Is there a recommended way to deal with this?
5 posts - 3 participants