Revisit using src/package_name directory instead of package_name directory

[radifar]

So around a month ago I read Hynek Schlawack nice article about testing and packaging, which also direct me to another article by Ionel MC. The main point of those articles are why it is important to move source code directory below src directory instead of inside the main directory. They argue that is better for testing and packaging.

Back then I would like to share this thought in here but I was reluctant as I wasn't completely get their point (but now I'm pretty much do). Until, just recently I'm having a testing problem with my project. Took me many painful hours for me to figure out what's wrong, basically I was testing the package import using monkeypatching. And no matter how I manipulate the sys.path and the environment the package is already imported before the test run. It took me many hours to understand that pytest traversing through the __init__.py that is above its path and execute them before running the actual test. Then I tried reading the articles by Hynek and Ionel again, and now I can appreciate their approach more.

Then I started restructure my project, move source code to src/project_name, move tests to main directory, update CI.yaml, MANIFEST.in, and setup.cfg. And the problem above is gone.

Before I submit this issue I checked if there is similar issue, and turns out @jaimergp has already mention it on #78 , and I have to say that I disagree with this statement

Ionel seems to steer more into the general web framework domain which is quite different from our current practices. (@dgasmith)

I believe that this apply to our current practice as well, and I think one of my favorite argument from Hynek is

Your tests do not run against the package as it will be installed by its users. They run against whatever the situation in your project directory is.

I hope that we can revisit this idea again, to avoid many potential problems in package testing.

[jaimergp]

A workaround I've been using to avoid that issue is to run the tests from a level up your directory tree. Annoying but it works!

[radifar]

Just tried what you said, it works, but it doesn't prevent you from loading the __init__.py on the project directory. Pytest discovery will still load any __init__.py above the test directory. Unless you place your tests directory at the same level as project directory.

[jaimergp]

Yes, I usually have them like that:

- my_package/
- tests/
- setup.py
- ...

[mikemhenry]

I'm also in favor of using the src/package_name structure

[loriab]

Thanks for your interesting links -- I've enjoyed reading them. Also a disclaimer -- I don't actually use cookiecutter-cms directly. That said, there were some points in those articles that alarmed me and that I think make src/pkgname less suitable for scientific software:

You need tooling to use the src-layout effectively. If you don't use a tool to manage virtualenvs and install your project like Tox then it's going to be painful.

Without src you get messy editable installs ("setup.py develop" or "pip install -e"). Having no separation (no src dir) will force setuptools to put your project's root on sys.path - with all the junk in it (e.g.: setup.py and other test or configuration scripts will unwittingly become importable).

It's extremely unlikely that the user of the library will run the tests instead of the library's developer.

In my experience, scientific software is more likely to be developed in a stack with multiple projects being edited at once. In this setup, pure-py projects are development freebies that are conveniently pip install -e .'d and tested in situ. On the user end, installations are often tested to be sure a collection of packages works harmoniously together, and test suites are more likely to depend on other packages than on mockup data structures.

The src/ structure seems to me a nice way to signal I'm-a-complicated-py-package-treat-me-like-c++. But it sounds like for many projects of the complexity that the cms cookiecutter addresses that a CI routine to test the packaged project (which I think the cookiecutter had once with building the conda recipe but was removed b/c it took too long) could avert a lot of the bugs that non-src allows, while preserving the ease of in-src development and testing.

[dgasmith]

Both articles lean heavily on the tox assumption, which isn't the case here. It is also worth noting that the citation for the change in cryptography has a pretty long thread on the matter pyca/cryptography#1468. I read those comments as "well this solves the solution and this guy really cares about so whatever" rather than "this is a solution that is robust and we're happy with". We should find projects of similar nature which are pure python and do not use tox (which complicates matters) and do not have rust and C components.

It is also notable that we suggest putting tests in the library itself so that downstream library users can run the tests themselves. In an ideal world this wouldn't be needed, but we generally find 1) people are poor at correctly pending their dependencies 2) developers are equally poor at following semantic versioning. Doing so correctly requires a framework of integration tests, release candidates, etc in an area where resources for unit tests themselves is rarely allocated. Therefore not a degradation of the developers, but a general trend of scientific computing and something that needs to be holistically fixed to truly build robust ecosystems.

Balancing pip, tox, setuptools, venv, pipenv, poetry, etc with conda is challenging as they overlap strongly. It sounds like your particular problem is more import structure related/some sort of specific path manipulation rather than the issues illuminated in the articles. We should understand the use case more before making changes and make sure we pull ideas from closely related communities.