OBS cleanup -> removal of outdated repositories

[Adrian Schröter]

Hi,

Please excuse the cross post.
You may have noticed that we have higher and higher build load on build.o.o.
While the new dispatcher algorithms are still good to prefer the "right" jobs,
it is still a sign that we may should cleanup.

My current plan is to remove all repositories in home projects, which have not been
touched this year. Not touched means, no source submission and no meta data has
been altered at all.
Additionally also all non-home projects, which have not been touch in last two years.

This will affect 5668 out of 15668 projects.

Why remove repos and not just disable the build ?
This will free disk space on our servers and also on all mirrors. As result
we can be mirrored more easily.

Will any source get lost ?
No.

How to enable it again ?
Just add the wanted repos again.

Which projects will get affected exactly ?
Find a full list here:
http://www.suse.de/~adrian/OBS-remove-list-candidates

There is a project which has not been touched, but the repos are still anyway important !
Just drop me a mail ....

Why not drop the entire project now that we have an undelete function ?
I thought about that, but currently the webui just says that the project does not exist.
It does not offer to undelete it, so that might be too agressive for now.

Why not drop people a mail and ask them to remove it ?
Way to many accounts have no valid email adress and past experience showed that people
are often not react when they lost interesst in their project.

May plan is to do the removal end of this week, except more discussion about this
is needed.

Just tell me your opinion, also when you support this ;)

thanks
adrian

[Greg Freemyer]

On Tue, Nov 23, 2010 at 11:45 AM, Adrian Schröter <***@suse.de> wrote:
<snip>

Post by Adrian Schröter
May plan is to do the removal end of this week, except more discussion about this
is needed.
Just tell me your opinion, also when you support this ;)
thanks
adrian

Adrian,

I'm surprised you have so many untouched projects. I assume anyone
that has added a recent distro repo will not see their packages
deleted? Or does that happen at the project level, and thus even
maintainers that have been adding new distros to their repo list will
see packages deleted?

I consider that a bad thing.

Also, just a comment that this is a major holiday week in the US and
lots of people are offline already. And starting Wed. COB, many will
not touch a computer again before Monday AM.

I don't know if that impacts the timing of your process or not.

Greg

[Adrian Schröter]

Post by Greg Freemyer
<snip>

Post by Adrian Schröter
May plan is to do the removal end of this week, except more discussion about this
is needed.
Just tell me your opinion, also when you support this ;)
thanks
adrian

Adrian,
I'm surprised you have so many untouched projects. I assume anyone
that has added a recent distro repo will not see their packages
deleted? Or does that happen at the project level, and thus even
maintainers that have been adding new distros to their repo list will
see packages deleted?

No, who ever touch any package source or touch project meta data (by adding a repo)
should not be affected.

The list may contain quite some projects, where never a real package have been built.
Just people who clicked wildly in the webui ;)

bye
adrian

[Cristian Rodríguez]

Post by Adrian Schröter
Hi,
Please excuse the cross post.
You may have noticed that we have higher and higher build load on build.o.o.
While the new dispatcher algorithms are still good to prefer the "right" jobs,
it is still a sign that we may should cleanup.
My current plan is to remove all repositories in home projects, which have not been
touched this year. Not touched means, no source submission and no meta data has
been altered at all.
Additionally also all non-home projects, which have not been touch in last two years.
This will affect 5668 out of 15668 projects.

I just sent a osc rdelete loop to remove mines. ;)

[Dave Plater]

Post by Adrian Schröter
Hi,
Please excuse the cross post.
You may have noticed that we have higher and higher build load on build.o.o.
While the new dispatcher algorithms are still good to prefer the "right" jobs,
it is still a sign that we may should cleanup.
My current plan is to remove all repositories in home projects, which have not been
touched this year. Not touched means, no source submission and no meta data has
been altered at all.
Additionally also all non-home projects, which have not been touch in last two years.
This will affect 5668 out of 15668 projects.
Why remove repos and not just disable the build ?
This will free disk space on our servers and also on all mirrors. As result
we can be mirrored more easily.
Will any source get lost ?
No.
How to enable it again ?
Just add the wanted repos again.
Which projects will get affected exactly ?
http://www.suse.de/~adrian/OBS-remove-list-candidates
There is a project which has not been touched, but the repos are still anyway important !
Just drop me a mail ....
Why not drop the entire project now that we have an undelete function ?
I thought about that, but currently the webui just says that the project does not exist.
It does not offer to undelete it, so that might be too agressive for now.
Why not drop people a mail and ask them to remove it ?
Way to many accounts have no valid email adress and past experience showed that people
are often not react when they lost interesst in their project.
May plan is to do the removal end of this week, except more discussion about this
is needed.
Just tell me your opinion, also when you support this ;)
thanks
adrian

If it helps to stop a few of my wip home project packages from staying
on scheduled, sometimes for more than a day, I'm +1
Dave P

[Peter Linnell]

Post by Dave Plater

Post by Adrian Schröter
Hi,
Please excuse the cross post.
You may have noticed that we have higher and higher build load on build.o.o.
While the new dispatcher algorithms are still good to prefer the "right" jobs,
it is still a sign that we may should cleanup.
My current plan is to remove all repositories in home projects, which have not been
touched this year. Not touched means, no source submission and no meta data has
been altered at all.
Additionally also all non-home projects, which have not been touch in last two years.
This will affect 5668 out of 15668 projects.
Why remove repos and not just disable the build ?
This will free disk space on our servers and also on all mirrors. As result
we can be mirrored more easily.
Will any source get lost ?
No.
How to enable it again ?
Just add the wanted repos again.
Which projects will get affected exactly ?
http://www.suse.de/~adrian/OBS-remove-list-candidates
There is a project which has not been touched, but the repos are still anyway important !
Just drop me a mail ....
Why not drop the entire project now that we have an undelete function ?
I thought about that, but currently the webui just says that the project does not exist.
It does not offer to undelete it, so that might be too agressive for now.
Why not drop people a mail and ask them to remove it ?
Way to many accounts have no valid email adress and past experience showed that people
are often not react when they lost interesst in their project.
May plan is to do the removal end of this week, except more discussion about this
is needed.
Just tell me your opinion, also when you support this ;)
thanks
adrian

If it helps to stop a few of my wip home project packages from staying
on scheduled, sometimes for more than a day, I'm +1
Dave P

Hi Adrian,

My only question is why it took so long :)

I had a look at that list and it is to my eyes a perfectly legitimate
move to delete these repos. On the other hand, it shows you are victim
of your OBS's popularity...

Cheers,

Peter

[Stefan Seyfried]

On Tue, 23 Nov 2010 17:45:30 +0100

Post by Adrian Schröter
Which projects will get affected exactly ?
http://www.suse.de/~adrian/OBS-remove-list-candidates

I just committed an unimportant fix (smp_mflags) to home:seife:cross, to
trigger a rebuild. With current speed of the buildservice, this will take
about two weeks to build (because it's a big project, and I consciously
avoid changing it if not strictly necessary, but you basically asked for
it so you get ist ;)

Post by Adrian Schröter
May plan is to do the removal end of this week, except more discussion about this
is needed.

I expect you to check the candidates again for activity before doing
this... :)

[Adrian Schröter]

Post by Stefan Seyfried
On Tue, 23 Nov 2010 17:45:30 +0100

Post by Adrian Schröter
Which projects will get affected exactly ?
http://www.suse.de/~adrian/OBS-remove-list-candidates

I just committed an unimportant fix (smp_mflags) to home:seife:cross, to
trigger a rebuild. With current speed of the buildservice, this will take
about two weeks to build (because it's a big project, and I consciously
avoid changing it if not strictly necessary, but you basically asked for
it so you get ist ;)

The way I collected the data was just by checking the sources, independend from build
results. So it does not matter if it builds or not until then.

I removed home:seife:cross from my delete list now.

Post by Stefan Seyfried

Post by Adrian Schröter
May plan is to do the removal end of this week, except more discussion about this
is needed.

I expect you to check the candidates again for activity before doing
this... :)

k, but it is in any cases better to drop me a mail in such cases.

[Patrick Shanahan]

* Adrian Schröter <***@suse.de> [11-23-10 11:47]:
...

Post by Adrian Schröter
May plan is to do the removal end of this week, except more discussion
about this is needed.

Does the ability exist to determine usage (download) of a repo?
Changes/reguilds may have not been necessary.

Post by Adrian Schröter
Just tell me your opinion, also when you support this ;)

absolutely.

[Cristian Rodríguez]

Post by Adrian Schröter
Hi,
Please excuse the cross post.
You may have noticed that we have higher and higher build load on build.o.o.
While the new dispatcher algorithms are still good to prefer the "right" jobs,
it is still a sign that we may should cleanup.

To cleanup your repos by your own try:


curl http://www.suse.de/\~adrian/OBS-remove-list-candidates |
awk -F " " '{print $1}' | sed -e ***@.pkg@@g | grep yourusername |
while read repo; do osc rdelete --force "$repo"; done

[Christian Boltz]

Hello,

Post by Adrian Schröter
You may have noticed that we have higher and higher build load on build.o.o.

Well, at least on the x86_64 build hosts. The ppc64 hosts are bored ;-)

Which brings me to a question and suggestion:


How to handle noarch packages?

As a rough number, on my 11.3 laptop I have 1940 x86_64 and 318 noarch
packages - that's about 15% noarch packages. (I'll take this number
instead of checking all packages in the distribution in the rest of this
mail.)

1) build targets

The noarch packges I checked in Factory are build for i586 _and_ x86_64,
and later only one of them is really used. The other one just wastes
build service power.

Wouldn't it be enough to build for one target, say i586?

That would save half of 15% = 7.5% of all builds, and is probably easy
to implement (scan specfile for noarch, then disable x86_64 builds for
them).


2) where to build noarch packages

Noarch packages could be build on any idle host, including ppc64, with
the same result.

This would move some load to the currently idle ppc64 build hosts and
remove some burden from the overloaded x86_64 hosts.

15% would mean that about 3 x86_64 build hosts would be freed for other
build jobs as long as there are enough idle ppc64 build hosts.

I don't know how difficult this would be on the technical side
(scheduler etc.), but hey, that's the advantage if you come up with
ideas without knowing all the technical details *g*


Regards,

Christian Boltz
--
Du weißt aber schon, was T-ONLINE heißt - oder?

T raurig,
-
O bwohl
N ichts
L äuft,
I st
N iemand
E rreichbar

[Peter Geerds in suse-linux]

[Jean Delvare]

Post by Christian Boltz
Hello,

Post by Adrian Schröter
You may have noticed that we have higher and higher build load on build.o.o.

Well, at least on the x86_64 build hosts. The ppc64 hosts are bored ;-)
How to handle noarch packages?
As a rough number, on my 11.3 laptop I have 1940 x86_64 and 318
noarch packages - that's about 15% noarch packages. (I'll take this
number instead of checking all packages in the distribution in the
rest of this mail.)
1) build targets
The noarch packges I checked in Factory are build for i586 _and_
x86_64, and later only one of them is really used. The other one
just wastes build service power.
Wouldn't it be enough to build for one target, say i586?
That would save half of 15% = 7.5% of all builds, and is probably
easy to implement (scan specfile for noarch, then disable x86_64
builds for them).
2) where to build noarch packages
Noarch packages could be build on any idle host, including ppc64,
with the same result.
This would move some load to the currently idle ppc64 build hosts and
remove some burden from the overloaded x86_64 hosts.
15% would mean that about 3 x86_64 build hosts would be freed for
other build jobs as long as there are enough idle ppc64 build hosts.
I don't know how difficult this would be on the technical side
(scheduler etc.), but hey, that's the advantage if you come up with
ideas without knowing all the technical details *g*

Sounds like a really great idea! Which I wholeheartedly second, also
knowing nothing about the technical details ;)

[Adrian Schröter]

Post by Christian Boltz
Hello,

Post by Adrian Schröter
You may have noticed that we have higher and higher build load on build.o.o.

Well, at least on the x86_64 build hosts. The ppc64 hosts are bored ;-)
How to handle noarch packages?
As a rough number, on my 11.3 laptop I have 1940 x86_64 and 318 noarch
packages - that's about 15% noarch packages. (I'll take this number
instead of checking all packages in the distribution in the rest of this
mail.)
1) build targets
The noarch packges I checked in Factory are build for i586 _and_ x86_64,
and later only one of them is really used. The other one just wastes
build service power.
Wouldn't it be enough to build for one target, say i586?

yes ... but if you need it for building x86_64 packages you need to maintain
export filters manually. And you can build dependency loops via cross architecture,
which are harder to detect and would increase the build time a lot.

So far we always believed it is too much trouble. But you can try it of course.

bye
adrian

[Christian Boltz]

Hello,

Post by Adrian Schröter

Post by Christian Boltz
Wouldn't it be enough to build for one target, say i586?

yes ... but if you need it for building x86_64 packages you need to
maintain export filters manually. And you can build dependency
loops via cross architecture, which are harder to detect and would
increase the build time a lot.

Hmm, what about a "first build wins" concept?

If a noarch package needs a rebuild, schedule it for every arch where a
rebuild is needed (i586, x86_64 and maybe additionally ppc64 if you want
to move some load there). So far, nothing new here except the additional
build job for ppc64.

Let's assume the build on ppc64 is finished first.
As soon as the package from ppc64 is finished,
- copy the RPM to i586 and x86_64
- remove the build of that package from the i586 and x86_64 scheduler
queue and mark it as successfully built. (You could call it a very
fast build ;-))
- (in case i586 build has already started, do not copy over the package
built on ppc64 to i586.)

Yes, I know that sounds hacky, but if I get your description right, it
should cause less problems than using a separate noarch scheduler.

If a BuildRequire'd package changes on one arch after copying in the
package from the ppc64 build, re-add it to the scheduler for rebuild -
but this will probably happen already.

Post by Adrian Schröter
So far we always believed it is too much trouble. But you can try it of course.

Adrian, there's a reason why I wrote "that's the advantage if you come
up with ideas without knowing all the technical details" in my first
mail ;-)

As long as we talk about the concept or some not-too-creative perl
sniplets, I can most probably follow you. However, I'm afraid my perl
knownledge is not good enough for something like the scheduler, and
(bigger problem) I don't have enough time to read and understand all the
code.


Regards,

Christian Boltz