So. The idiotic observation is that I can only compile correctly if I assume that all files are typechecked and compiled given all declarations imported in the project. That leads to a somewhat expensive design where the top module drives everything.
Somehow, this trivial observation took me half a week, and I am still not sure how to implement it right; i.e., how to translate this to performant and clear handling of data structures internally.
It simply isn't nice that a simple module needs to be compiled with a context including all the declarations, it breaks compositionality one assumes one has, in the project but there doesn't seem to be a way out. At least, not an easy way out.
And, at minimum, I want a design which can easily be extended to allow pre-compiled modules to be loaded at some point too. There's that too.
 Two modules may mutually recursively import each other's declaration, so there is already a need to import "forward" declarations. The top module may import declarations of modules which "shadow" imported declarations and lead to a name clash during linking.
 But it looks like other compilers don't have these compositionality invariants either. If I modify a file half-way through a C project that'll likely give me a core dump. And I can easily program C code such that all files will compile nicely individually but linking will fail. It's just an unsolvable problem.