wiki:WikiStart

Version 20 (modified by Bruno Cornec, 16 years ago) ( diff )

perfmon up to cms2pkg without good result

Welcome to Project-Builder Wiki

Design Ideas around the project-builder.org project

Since 2005 that I took over the maintenance of the ​MondoRescue project, I have passed time developing code on the periphery of the project to allow me to easily build packages from Subversion, deliver them, manage them, ...

I met a lot of Open Source project which are suffering from the same barrier for their adoption as mondo suffered in the past: without native packaging for your distribution you have much less people testing your software, trying it, less chance to be integrated in distributions, ...

Then I wanted to also work on another project called ​dploy.org which uses the great ​LinuxCOE. However that LinuxCOE project had exactly the same issue as previously mentioned, no packaging, on install from CVS, so smaller community. And in order for me to put something on a prouction server, I want it packaged :-)

So I decided it could be a good idea to reuse the scripts made for mondo to package LinuxCOE. However, they were too specific, even if the ideas could be reused (that was my 3rd generation of build scripts, so ideas were more advanced than the code).

Technical ideas around the project-builder.org (aka pb) project

Example of use on netperf (simple example)

As an example of how easy (or not :-) Project-Builder is to use, I decide to write that tutorial as a gift for another project which deserves it, the [www.netperf.org NetPerf] project .

You first have to create a .pbrc configuration file in your home directory, which will setup things globally: 

$ cat > ~/.pbrc << EOF
#
# Root of the temp file used to store info on the project
# Is also used to determine the root directory of the project
# No default option allowed here as they need to be all different
#
pbrc netperf = $ENV{'HOME'}/netperf/src
EOF

Should be obvious for the content. Note that the format is in general 

pb-keyword project = value

which means that here I named my project for netperf for Project-Builder (I can choose what I want but need to be consistent everywhere).

Then get the bits of the project you want to package. It could be using a SVN or CVS access if there is such a thing, or by downloading (at worse) the tar file of the project. Then extract that under a newly created directory that you referenced in your project configuration file: 

$ cd /tmp
$ wget ftp://ftp.netperf.org/netperf/netperf-2.4.4.tar.bz2
$ mkdir -p $HOME/netperf/src
$ cd $HOME/netperf/src
$ tar xvfj /tmp/netperf-2.4.4.tar.bz2
$ rm -f /tmp/netperf-2.4.4.tar.bz2

The netperf project is a nice project using a simple environment to build itself. It uses the GNU autoconf/autmake/litool mecanism, which makes it easy to package with pb.

You will now need to get the latest and greatest project-builder software from our ​ftp repository. Of course we provide lots of packages for your distribution ;-) Or a tar.gz file if you prefer. Once available, you can use the newproj option of pb to create the infrastructure that is mandatory for pb. It is mandatory that there is a correspondance between the directory mentioned in $HOME/.pbrc and the -r option: 

$ pb -p netperf -r `pwd` newproj
Creating /users/bruno/netperf/src/pbconf directory

Do not to forget to commit the pbconf directory in your CMS if needed
After having renamed the pkg1 directory to your package's name

Project: netperf
Action: newproj

Explore then the pbconf directory to familiarize yourself with some of the config file. the next one we want to edit, is the main configuration file which will be used as the central piece of information for pb.

Edit it and adapt it. After edition, here is the resulting file: 

$ ls -R pbconf
pbconf:
netperf.pb  pbfilter  pkg1

pbconf/pbfilter:
all.pbf  deb.pbf  md.pbf  novell.pbf  rpm.pbf

pbconf/pkg1:
deb  pbfilter  rpm

pbconf/pkg1/deb:
changelog  compat  control  copyright  pkg1.dirs  pkg1.docs  rules

pbconf/pkg1/pbfilter:

pbconf/pkg1/rpm:
pkg1.spec
$
$ vi pbconf/netperf.pb
[Modifications here]
$ cat pbconf/netperf.pb
#
# Project Builder configuration file
# For project netperf
#
# $Id$
#

#
# Which CMS system is used (Subversion, CVS or tar file content extracted)
#
#cms netperf = svn
#cms netperf = cvs
cms netperf = flat

#
# Packager label
#
packager netperf = "Bruno Cornec <bruno@project-builder.org>"
#

# For delivery to a machine by SSH (potentially the FTP server)
# Needs hostname, account and directory
#
sshhost netperf = www.project-builder.org
sshlogin netperf = bruno
sshdir netperf = /mondo/ftp
sshport netperf = 22

#
# For Virtual machines management
# Naming convention to follow: distribution name (as per ProjectBuilder::Distribution)
# followed by '_' and by release number
# a .vmtype extension will be added to the resulting string
# a QEMU rhel_3 here means that the VM will be named rhel_3.qemu
#
vmlist netperf = mandrake_10.1,mandrake_10.2,mandriva_2006.0,mandriva_2007.0,mandriva_2007.1,mandriva_2008.0,redhat_7.3,redhat_9,fedora_4,fedora_5,fedora_6,fedora_7,rhel_3,rhel_4,rhel_5,suse_10.0,suse_10.1,suse_10.2,suse_10.3,sles_9,sles_10,gentoo_nover,debian_3.1,debian_4.0,ubuntu_6.06,ubuntu_7.04,ubuntu_7.10

#
# Valid values for vmtype are
# qemu, (vmware, xen, ... TBD)
vmtype netperf = qemu

# Hash for VM stuff on vmtype
vmntp default = pool.ntp.org

# We suppose we can commmunicate with the VM through SSH
vmhost netperf = localhost
vmlogin netperf = pb
vmport netperf = 2225

# Timeout to wait when VM is launched/stopped
vmtmout default = 120

# per VMs needed paramaters
vmopt netperf = -m 384 -daemonize
vmpath netperf = /home/qemu
vmsize netperf = 5G

#
# Global version/tag for the project
#
projver netperf = 2.4.4
projtag netperf = 1

# Adapt to your needs:
# Optional if you need to overwrite the global values above
#
#pkgver pkg1 = stable
#pkgtag pkg1 = 3
#pkgver nil
#pkgtag nil

# Hash of default package/package directory
defpkgdir netperf = netperf-2.4.4

# Hash of additional package/package directory
#extpkgdir pkg1-doc = pkg1-docdir

# Hash of valid version names
version devel
#version stable

# List of files per pkg on which to apply filters
# Files are mentioned relatively to pbroot/defpkgdir
filteredfiles netperf = configure.ac
#filteredfiles pkg1-doc = configure.in

I skip the parameters begining with vm and ssh for the moment. I inform pb that I will produce packages with a version-tag of 2.4.4-1, that no local package should have a different version than the project version (pkgver/pkgtag), that my default package will be netperf (pb will work on it by default all the time) and that no additional packages will be produced. In addition to standard version with numbers, pb will accept the version devel. And finally, I give to pb the name of all files (per package) that it will have to filter to produce the packages - more on that later on. Once this is done, you may already try to see if pb can make something useful for you. Check the result of: 

$ pb -p netperf -r `pwd` cms2build
Project: netperf
Action: cms2build
Packages: netperf

Management of netperf 2.4.4-1 (rev flat)
Exporting /users/bruno/netperf/src/netperf-2.4.4 from DIR to /users/bruno/netperf/delivery/netperf-2.4.4... OK
Build files generated for
No Build files found for mandriva-2006.0,sles-9,mandrake-10.2,mandriva-2007.0,rhel-3,mandriva-2007.1,redhat-9,ubuntu-6.06,suse-10.3,rhel-5,ubuntu-7.10,gentoo-nover,suse-10.2,mandrake-10.1,mandriva-2008.0,sles-10,fedora-6,suse-10.1,fedora-5,fedora-4,ubuntu-7.04,fedora-7,suse-10.0,debian-4.0,rhel-4,debian-3.1,redhat-7.3
filter: no such variable
Can't use an undefined value as a HASH reference at /usr/lib/perl5/vendor_perl/5.8.8/ProjectBuilder/Base.pm line 1077.

Ok we tried to make a build file from ou Configuration Management System (CMS) which didn't work completely as our structure is incomplete. However, pb already recognized the project name, action, package to build, type of CMS (flat), used our -r option to create the intermediate delivery directory where all the final files|packages will be created.

So now, in order to be more useful, we need to create the structure for our package. Luckily, pb did most of the work for us, by creating a template pkg1 directory and files under pbconf. Rename them to netperf (name of our package) and explore and adapt its content: 

$ find . -name pkg1'*'
./pbconf/pkg1
./pbconf/pkg1/rpm/pkg1.spec
./pbconf/pkg1/deb/pkg1.dirs
./pbconf/pkg1/deb/pkg1.docs
$ mv ./pbconf/pkg1 ./pbconf/netperf
$ mv ./pbconf/netperf/rpm/pkg1.spec ./pbconf/netperf/rpm/netperf.spec
$ mv ./pbconf/netperf/deb/pkg1.dirs ./pbconf/netperf/deb/netperf.dirs
$ mv ./pbconf/netperf/deb/pkg1.docs ./pbconf/netperf/deb/netperf.docs

Now this is done, it's time to look at those and adapt them. At the begining, we try to make RPM packages. 

$ vi pbconf/netperf/rpm/netperf.spec
[Modifications here]
$ cat pbconf/netperf/rpm/netperf.spec
#
# $Id$
#

Summary:        The netperf tool is a benchmarking network tool
Summary(fr):    netperf est un outil de benchmark réseau

Name:           PBPKG
Version:        PBVER
Release:        PBTAGPBSUF
License:        GPL
Group:          PBGRP
Url:            PBURL
Source:         PBSRC
BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(id -u -n)
Requires:       PBDEP

%description
PBDESC

%description -l fr
netperf est un outil de benchmark réseau

%prep
%setup -q

%build
%configure
make %{?_smp_mflags} VERSION=%{version}

%install
%{__rm} -rf $RPM_BUILD_ROOT
make DESTDIR=$RPM_BUILD_ROOT install

%clean
%{__rm} -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root)
%doc ChangeLog
%doc INSTALL COPYING README AUTHORS NEWS

%changelog
PBLOG

That spec file looks normal except that it uses a lot of PBSOMETHING keywords. Those are Project-Builder macros that will be instantiate depending on the distribution during the cms2build step.

So now that we have the base for our spec file, we need to look at those filters: 

$ vi pbconf/pbfilter/all.pbf
[Modifications here]
$ cat pbconf/pbfilter/all.pbf
#
# $Id$
#
# Filter for all files
#
# PBSRC is replaced by the source package format
filter PBSRC = ftp://ftp.netperf.org/netperf/%{name}-%{version}.tar.gz

# PBVER is replaced by the version ($pbver in code)
filter PBVER = $pbver

# PBDATE is replaced by the date ($pbdate in code)
filter PBDATE = $pbdate

# PBLOG is replaced by the changelog if value is yes
filter PBLOG = yes

# PBTAG is replaced by the tag ($pbtag in code)
filter PBTAG = $pbtag

# PBREV is replaced by the revision ($pbrev in code)
filter PBREV = $pbrev

# PBPKG is replaced by the package name ($pbpkg in code)
filter PBPKG = $pbpkg

# PBPACKAGER is replaced by the packager name ($pbpackager in code)
filter PBPACKAGER = $pbpackager

# PBDESC contains the description of the package
filter PBDESC = "netperf is a network benchmarking tool"

# PBURL contains the URL of the Web site of the project
filter PBURL = http://www.netperf.org

We can now try again to use pb to create our build file: 

$ pb -p netperf -r `pwd` cms2build
Project: netperf
Action: cms2build
Packages: netperf

Management of netperf 2.4.4-1 (rev flat)
Exporting /users/bruno/netperf/src/netperf-2.4.4 from DIR to /users/bruno/netperf/delivery/netperf-2.4.4... OK
Build files generated for mandriva-2006.0,sles-9,mandrake-10.2,mandriva-2007.0,rhel-3,mandriva-2007.1,redhat-9,ubuntu-6.06,suse-10.3,rhel-5,ubuntu-7.10,suse-10.2,mandrake-10.1,mandriva-2008.0,sles-10,fedora-6,suse-10.1,fedora-5,fedora-4,ubuntu-7.04,fedora-7,suse-10.0,debian-4.0,rhel-4,debian-3.1,redhat-7.3
No Build files found for gentoo-nover
Files configure.ac have been filtered
Creating netperf tar files compressed... OK
Under /users/bruno/netperf/delivery/netperf-2.4.4.tar.gz

Much better. You have now your first useful result from pb :-) Congrats !

Example of use on LinuxCOE (more complex example)

When the project was mature enough to package correctly mondorescue and itself, I then decided it was time to try it on another unrelated project to see how difficult it could be and document that experience so that others could also begin to use pb for their own project.

You first have to edit your .pbrc configuration file in your home directory, which will setup things globally.

Here are my .pbrc LinuxCOE relative entries: 

$ grep -E '^#|linuxcoe|default' ~/.pbrc
#
# Root of the temp file used to store info on the project
# Is also used to determine the root directory of the project
# No default option allowed here as they need to be all different
#
pbrc linuxcoe = /users/bruno/LinuxCOE/cvs/pbrc

Then you have to go under the directory you planned to store the project you want to package (/users/bruno/LinuxCOE/cvs) and you have to create there a directory pbconf which will contain everything related to project-builder around your own project. The pbrc file you indicated in the configuation file will be managed automatically by pb and you don't have to care of it. Just to reference where it will be, which will in turn give to project-builder the information of the project's directory. 

$ cd /users/bruno/LinuxCOE/cvs
$ mkdir pbconf

The main configuration file under pbconf should be named with the name of the project (the one used above in the .pbrc file) with the pb extension. the one created for LinuxCOE looks like that: 

$ cat linuxcoe.pb
#
# Project Builder configuration file
# For project LinuxCOE
#
# $Id$
#

#
# Global version/tag for the project
#
projver linuxcoe = devel
projtag linuxcoe = 1

# Adapt to your needs:
# Optional if you need to overwrite the global values above
#
pkgver nil
pkgtag nil


# Hash of default package/package directory
#
defpkgdir systemdesigner = SystemDesigner

# Hash of additional package/package directory
#
extpkgdir systemdesigner-docs = docs
extpkgdir systemdesigner-fedora = SystemDesigner-Fedora

# Hash of valid version names
#
version devel
version stable

# List of additional files on which to apply filters
# Files are mentioned relatively to pbroot
#
filteredfiles systemdesigner =
filteredfiles systemdesigner-docs =

I inform pb that I will produce packages with a version-tag of devel-1, that no local package should have a different version than the project version (pkgver/pkgtag), that my default package will be systemdesigner (pb will work on it by default all the time) and that additional packages it may deal with are systemdesigner-docs and systemdesigner-fedora. In addition to standard version with numbers, pb will accpet the version devel and stable. And finally, I give to the name of all files (per package) that it will have to filter to produce the packages - more on that later on.

With that minimal configuration, you may begin to test project-builder. Try to produce a compressed tar file of your project. 

$ export PBPROJ=linuxcoe
$ pb -r `pwd` cms2build

The export order will allow you to avoid precising all the time to pb that you work on the linuxcoe project. If you prefer, you can invoke pb as follows: 

pb -p linuxcoe -r `pwd` cms2build

or make a shell alias, or whatever is convenient for you.

The -r option is needed at start to create the pbrc file you mentioned in your .pbrc configuration file initially. After that first invocation, you will only need to call it again, if you change the version you want to generate.

cms2build is generally the first action pb will be asked to execute as it will produce a compressed tar ball from your Confoguration Management System repository. pb is CMS agnostic, and can currently handle friendly SVN and CVS, and can be easily adapted to support others.

As a result of your commands you should see something like that: 

$ pb -r `pwd` cms2build
Project linuxcoe
Action: cms2build
Packages: systemdesigner

Management of systemdesigner devel-1 (rev CVS)
Exporting /users/bruno/LinuxCOE/cvs/SystemDesigner from CVS... OK
Extracting log info from CVS... OK
Build files generated for
No Build files found for mandriva-2007.1
Creating systemdesigner tar files compressed... OK
Under /users/bruno/LinuxCOE/cvs/delivery/systemdesigner-devel.tar.gz

So it creates for you the right tar file at the right place. But it warns you that no build file was found for your project. Normal we haven't created one yet ;-) It's our next step.

You'll first need to create the infrastructure needed to host the information for building packages: 

$ mkdir -p pbconf/systemdesigner/rpm
$ mkdir -p pbconf/systemdesigner/pbfilter

then thanks to the work done by Louis Bouchard to package LinuxCOE, you can take the spec file he created and adapt it to suit pb. 

$ cat pbconf/systemdesigner/rpm/systemdesigner.spec

Concerning trac

For a complete list of local wiki pages, see TitleIndex.

Trac is brought to you by ​Edgewall Software, providing professional Linux and software development services to clients worldwide. Visit ​http://www.edgewall.com/ for more information.

Note: See TracWiki for help on using the wiki.