Python 2 and 3 in single specfile

[Jan Matějek]

Fellow packagers,

after some discussions in our team, we figured that it would be neat if we could
build python modules for Python 2 and 3 from a single spec file. Now that a
great deal of python modules work with both pythons, this makes a lot of sense.

A good deal of work can be done by RPM macros. Then we could even seamlessly
start building modules for PyPy, Jython and other pythons.

A sample specfile, and the corresponding set of RPM macros, is attached. As you
can see, the appropriate subpackage is generated automatically by
%python_subpackages, build and install steps are handled through %python_build
and %python_install respectively. There is some more magic involved, as well as
possibilities for customization.

There is a minor problem with BuildRequires. As long as you only need
python-devel, it's not too bad to just list $flavor-devel requirements by hand.
It's worse when you require more subpackages, because then you need all of them
in both python2 and python3 versions.
It would be nice to be able to specify %{python_require Mako} and expand that to
all the necessary BuildRequires, but OBS blocks us here, because the limited
environment will not expand such macros.

mls, ro, or anyone from the OBS team, would it be possible to solve this?

Still, it basically doesn't matter if you list all the BuildRequires twice in
one spec or in two specs, so there.

Another thing I'm still unclear on are the filelists. It's certainly possible to
make a generic filelist, something along the lines of:

%package -n %flavor-%modname # python3-Mako
%defattr(-, root, root)
%{%flavor_sitelib}/*
%{%flavor_sitearch}/*

but this doesn't solve docs and possible other files.

Again, obviously, we can write the filelists by hand. But if you have good
suggestions for some smart macros here, I'd love to see them.

What do you folks think? Comments, questions?

m.

[Todd Rme]

A few thoughts:

1. The %python_install and %python_build macros need to be able to
support additional arguments. For example, to make a package use the
system version of a library instead of the bundled version.

2. There should be a macro to only run some commands in python 2 or
only run some commands in python 3. For example, if you need to
manually run 2to3 (not that common anymore, but needs to be
supported):

%ifpy3
2to3 -wn ./
%endif

3. Macros for %files can get complicated for packages that install
files in the root of sitelib or sitearch, since you have to deal with
__pycache__ for python 3 but not python 2.

4. What about %check? This is hard to get right with macros. There
are a variety of different ways to run unit tests. Which approach is
used varies from package to package, even amongst packages that use
the same unit testing framework. And you sometimes need to disable
some tests for one version of python or the other manualy.

5. What about packages with subpackages? matplotlib, for example, has
a ton of subpackages. Would these be created once or would they need
to be manually specified?

6. What about packages where the python and python 3 Requires are
different? For example, if a package requires a python 2 package that
backports a python 3 feature?

7. It would be really nice to have some sort of %unicode_env macro
that switches the build environment to utf8 for python packages that
need it (which is practically any package with non-pure-asciii
README).

Post by Jan Matějek
Fellow packagers,
after some discussions in our team, we figured that it would be neat if we could
build python modules for Python 2 and 3 from a single spec file. Now that a
great deal of python modules work with both pythons, this makes a lot of sense.
A good deal of work can be done by RPM macros. Then we could even seamlessly
start building modules for PyPy, Jython and other pythons.
A sample specfile, and the corresponding set of RPM macros, is attached. As you
can see, the appropriate subpackage is generated automatically by
%python_subpackages, build and install steps are handled through %python_build
and %python_install respectively. There is some more magic involved, as well as
possibilities for customization.
There is a minor problem with BuildRequires. As long as you only need
python-devel, it's not too bad to just list $flavor-devel requirements by hand.
It's worse when you require more subpackages, because then you need all of them
in both python2 and python3 versions.
It would be nice to be able to specify %{python_require Mako} and expand that to
all the necessary BuildRequires, but OBS blocks us here, because the limited
environment will not expand such macros.
mls, ro, or anyone from the OBS team, would it be possible to solve this?
Still, it basically doesn't matter if you list all the BuildRequires twice in
one spec or in two specs, so there.
Another thing I'm still unclear on are the filelists. It's certainly possible to
%package -n %flavor-%modname # python3-Mako
%defattr(-, root, root)
%{%flavor_sitelib}/*
%{%flavor_sitearch}/*
but this doesn't solve docs and possible other files.
Again, obviously, we can write the filelists by hand. But if you have good
suggestions for some smart macros here, I'd love to see them.
What do you folks think? Comments, questions?
m.

[Jan Matějek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

1. The %python_install and %python_build macros need to be able to support
additional arguments. For example, to make a package use the system
version of a library instead of the bundled version.

I will extend the macros to pass arguments directly to setup.py. I also expect
that as we come across more packages, the complexity will grow.
Of course, in the more complicated packages, it will be easier to loop over
%pythons manually, instead of using the macros.

2. There should be a macro to only run some commands in python 2 or only
run some commands in python 3. For example, if you need to manually run
%ifpy3 2to3 -wn ./ %endif

Good point. A specific define for each python that is present should work.

3. Macros for %files can get complicated for packages that install files in
the root of sitelib or sitearch, since you have to deal with __pycache__
for python 3 but not python 2.

We can have "generic python" and "generic python3" pattern, and the macros
will apply whichever is appropriate.
Not sure about packages that don't fit the generic pattern though.

4. What about %check? This is hard to get right with macros. There are a
variety of different ways to run unit tests. Which approach is used varies
from package to package, even amongst packages that use the same unit
testing framework. And you sometimes need to disable some tests for one
version of python or the other manualy.

True. I think that %check will need to be written manually in all cases.
Fortunately, "for $flavor in %pythons" shell-loop should help here.

5. What about packages with subpackages? matplotlib, for example, has a
ton of subpackages. Would these be created once or would they need to be
manually specified?

I'll try to extend my proof of concept so that it can handle subpackages by
itself. This will mean parsing the specfile more extensively...
Again, filelists complicate this part.

6. What about packages where the python and python 3 Requires are
different? For example, if a package requires a python 2 package that
backports a python 3 feature?

We have a dependency tracker for python, although I'm not sure how well it
works. This should solve a big part of the problem.

Things that cannot be picked up automatically, i'm not sure. How about a macro
that you could redefine in order to specify the requires?
%define requires_for_python python-six python-backports

That might not help in case you have subpackages with different requires, but,
well, if that's the case, you'll probably have to specify the subpackages
manually.
The macros must make sure that they don't redefine subpackages that are
already present.

7. It would be really nice to have some sort of %unicode_env macro that
switches the build environment to utf8 for python packages that need it
(which is practically any package with non-pure-asciii README).

not sure what you mean.
can you point me to a package where this is a problem?

Post by Jan Matějek
Fellow packagers,
after some discussions in our team, we figured that it would be neat if
we could build python modules for Python 2 and 3 from a single spec file.
Now that a great deal of python modules work with both pythons, this
makes a lot of sense.
A good deal of work can be done by RPM macros. Then we could even
seamlessly start building modules for PyPy, Jython and other pythons.
A sample specfile, and the corresponding set of RPM macros, is attached.
As you can see, the appropriate subpackage is generated automatically by
%python_subpackages, build and install steps are handled through
%python_build and %python_install respectively. There is some more magic
involved, as well as possibilities for customization.
There is a minor problem with BuildRequires. As long as you only need
python-devel, it's not too bad to just list $flavor-devel requirements by
hand. It's worse when you require more subpackages, because then you need
all of them in both python2 and python3 versions. It would be nice to be
able to specify %{python_require Mako} and expand that to all the
necessary BuildRequires, but OBS blocks us here, because the limited
environment will not expand such macros.
mls, ro, or anyone from the OBS team, would it be possible to solve this?
Still, it basically doesn't matter if you list all the BuildRequires
twice in one spec or in two specs, so there.
Another thing I'm still unclear on are the filelists. It's certainly
%package -n %flavor-%modname # python3-Mako %defattr(-, root, root)
%{%flavor_sitelib}/* %{%flavor_sitearch}/*
but this doesn't solve docs and possible other files.
Again, obviously, we can write the filelists by hand. But if you have
good suggestions for some smart macros here, I'd love to see them.
What do you folks think? Comments, questions?
m.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBAgAGBQJUCghXAAoJEIskb84DCy7LqmMP/RmcZuB6TUX8yaznzIdt520C
3eDIYN8Krk8NbQIr943IdITGfeNxeRXBp6EBV16kqcClWTn9WhsZtVnPL1jN+834
4/Lb2wpvR4DH0EiEnuxxJAiycuBbo41ruWTG7zHTCCFlHoPtmU3pH2UGhr3qwK43
NwYUS3H05XVdKb7J3aLxsTxqjkXPKJFccYoAB7jIU7PYe8M68uBajnerLTuw72xb
tkS0cGs28+9X0OU/5nW2CYBIkpNRi5mFZAhq174S8xg3sYIFjJ8rQqdkkX+3LQLa
4a1Un2vpSQKNYB152pZQxGsJgFZneDfF4zvaS8SLNhOXdUYyQuiP41fCnGBGFkoo
vKcKPDOaqa1qGTKCQ/K/ZC6QMwntuoSnGS+xQR14OKIVUW4s6uptPrcrMSLxuqYa
Mfx8lrySKJ1K01zCwFph8syyzb1lbrbDDT3Rh9nZHiTzflbeB3w2ANBYCKkacNvn
lpSTgPy4EFfek9JI7gsJw6GQ818KTPsUIwYN6huYk3bIli2lLIEuatAZwBMJj2UZ
MT5jjBdF6YpgYgycijTtsydEFSodw+j+ZCUK2M36p26Ll8Fz+jS5NrhNpph6QLGt
ZeHHIKMF2C7y930cB7G21AnsNc9AXf69b2mLIulZSxbdEJWQz96IDbYKA9LdXtpH
7uexMNGHcDhb+Vsa8CJ+
=1z6I
-----END PGP SIGNATURE-----

[Todd Rme]

Post by Jan Matějek

7. It would be really nice to have some sort of %unicode_env macro that
switches the build environment to utf8 for python packages that need it
(which is practically any package with non-pure-asciii README).

not sure what you mean.
can you point me to a package where this is a problem?

Try python3-tabulate, for example.

[Thomas Bechtold]

Hi,

I'm more or less new to rpm packaging but I agree that py2 and py3
packages should be build from a single source. So I created a hackweek
project [1]. I guess I'm not able to work on this alone but if some more
experienced packager would join, I would like to work on this.

Cheers,

Tom

[1] https://hackweek.suse.com/11/projects/438