Changeset 1549 in ProjectBuilder

Timestamp:

May 21, 2012, 2:17:30 AM (12 years ago)

Author:

Bruno Cornec

Message:

• pb: Improve build2pkg documentation so It can be referenced in build2ve. Document need to specify verpmtype for a rpm-style VE. Add substep of further fixes to filters and build files in build2ve. Provide example of needing an additional repo, which is more likely on centos. Fix bug with adapting owner since centos5.8 sets up so that we can't sudo until the sudoers file is fixed, but we don't do so until later. However, if we call su, then we need to keep the additional sudo. Apply the ftp/http proxy bits from the config file while setting up a VE. Don't try to mount /proc if it is already mounted. Fiddle with the substitution on requiretty; it wasn't working so moved to \s+ instead of [ \t]+ also left a comment to show users that the line was removed. Remove comment about applying ftp/http proxy to bootstrapping since we now do. Add note that we probably want a variable for where to get the source snapshot for source "installs" of project-builder.
• Filter.pm: Add support for multi-line filter variables since a description in an rpm or deb file is usually multiple lines long (this is now in addition to the support of the $/ variable). Add support for transforming those things since debian files use leading spaces, and rpm files don't. Transforms let you start with the same variable and fix it for the different OSs. Leave note about unknown "correct" documentation location.

Location:

devel/pb

Files:

• bin/pb (modified) (12 diffs)
• lib/ProjectBuilder/Filter.pm (modified) (2 diffs)

devel/pb/bin/pb

@@ -492 +492 @@
 =head2 build2pkg
 
-The build2pkg command takes the tar files created in the cms2build step and attempts to build the binary packages for your current operating system. In order for this step to work, you may need to edit the files in one of the $pbdefdir/$project/pbconf/$version/$project/{deb,rpm,pkg} directories. Those files will be used to try to build your package. Note that if you change those files, you need to re-run the cms2build step. Then you can run a command like:
+The build2pkg command takes the tar files created in the cms2build step and attempts to build binary packages for your current operating system. There are two steps:
+
+=over 4
+
+=item Update your filters and build files.
+
+You probably need to edit the files describing the build steps in one of the $pbdefdir/$project/pbconf/$version/$project/{deb,rpm,pkg} directories and the filters in $pbdefdir/$project/pbconf/$version/pbfilter. Note that you can define additional filters and transforms in the *.pbf files. The build files will be filtered by the filters defined in the *.pbf files to generate the inputs to the build step. Therefore, if you change those files, you need to re-run the cms2build step.
+
+=item Build the package.
+
+Then you can run a command like:
 
  % pb -p $project -r $version build2pkg
 
 To create the files in $project/build that comprise your binary package(s).
+
+=back
 
 =head2 newve
@@ -513 +525 @@
  vetype default = chroot
 
+If you are building for rpm style OS's, update the verpmtype option, and install the appropriate tool.
+
+ verpmtype default = rpmbootstrap
+
 You may also choose to specify a mirror for the OS packages, and optionally http/ftp proxies.  You can specify the proxies either through environment variables ($http_proxy/$ftp_proxy) or in the configuration file. The configuration file will be used if no corresponding environment variable has been set. For example, for debian and with a local squid proxy:
 
@@ -555 +571 @@
 The build2ve command is similar to the build2pkg command in that it will take the sources created by cms2build and turn them into binary packages. The command has two differences. First, it creates the packages in a virtual environment, i.e. the one made by an earlier setupve setup. Second it copies the resulting packages to a repository and builds the repository meta-data needed.
 
-Two sub-steps are needed:
+Three sub-steps are needed:
 
 =over 4
@@ -565 +581 @@
  sshhost $project = localhost
  sshdir $project = $home/cache-project-builder/repos
+
+You may also need to specify additional repository files to use or rpms to install. Note the URL for repositories is not the URL of the repository, but the URL of a file that can be put in the yum.repos.d or apt.sources.d directory.
+
+ addrepo centos-5-i386 = http://localhost/pb/centos-extras.repo,http://mirror.centos.org/centos/5/extras/i386/RPMS/chrpath-0.13-3.el5.centos.i386.rpm
+
+=item Update your filters and build files
+
+You may need to update your filter files (*.pbf) as in the build2pkg step if you are building for a new OS or architecture.
 
 =item Build the packages and copy them to the repository
@@ -2165 +2189 @@
     # ssh communication if not VE or CPAN
     # should use a hash instead...
-    my ($shcmd,$cpcmd,$cptarget,$cp2target);
+    my ($shcmd,$shcmdroot,$cpcmd,$cptarget,$cp2target);
     if ($cmt =~ /^VE/) {
         my $tp = pb_path_expand($vepath->{$ENV{'PBPROJ'}});
@@ -2173 +2197 @@
         my $arch = pb_get_arch();
         if ($vetype eq "chroot") {
-            $shcmd = "sudo /usr/sbin/chroot $tpdir /bin/su - $mac -c ";
+            $shcmdroot = "sudo /usr/sbin/chroot $tpdir ";
+            $shcmd = "$shcmdroot /bin/su - $mac -c ";
         } elsif ($vetype eq "schroot") {
             $shcmd = "schroot $tp -u $mac -- ";
         }
-        $sshcmd = "setarch i386 $sshcmd" if (($pbos->{'arch'} =~ /i?86/) && ($arch eq 'x86_64'));
+        $shcmd = "setarch i386 $shcmd" if (($pbos->{'arch'} =~ /i?86/) && ($arch eq 'x86_64'));
         $cpcmd = "sudo /bin/cp -r ";
         # We need to get the home dir of the target account to deliver in the right place
@@ -2233 +2258 @@
     # For VE we need to change the owner manually
     if ($cmt =~ /^VE/) {
-        pb_system("$shcmd \"sudo chown -R $mac $tdir\"","Adapt owner in $tdir to $mac");
+        if (defined $shcmdroot) {
+            # This should help overcome a CentOS 5.8 bug as well as having a simper sequence
+            pb_system("$shcmdroot \"chown -R $mac $tdir\"","Adapt owner in $tdir to $mac");
+        } else {
+            pb_system("$shcmd \"sudo chown -R $mac $tdir\"","Adapt owner in $tdir to $mac");
+        }
     }
 
@@ -2421 +2451 @@
     my $pbos = pb_distro_get_context($v);
     
+    my $ftp_proxy = pb_distro_get_param($pbos,pb_conf_get_if("ftp_proxy"));
+    my $http_proxy = pb_distro_get_param($pbos,pb_conf_get_if("http_proxy"));
+
+    $ENV{ftp_proxy} ||= $ftp_proxy if ((defined $ftp_proxy) && ($ftp_proxy ne ""));
+    $ENV{http_proxy} ||= $http_proxy if ((defined $http_proxy) && ($http_proxy ne ""));
+
     # Launch the VMs/VEs
     if ($vtype eq "vm") {
@@ -2995 +3031 @@
 
 # For VE we first need to mount some FS
-pb_system("mount -t proc /proc /proc");
+pb_system("mount -t proc /proc /proc") unless (-d "/proc/$$");
 
 EOF
@@ -3153 +3189 @@
 EOF
     print SCRIPT << 'EOF';
-    s/Defaults[ \t]+requiretty//;
+    s/Defaults\s+requiretty/# disable Defaults: requiretty/;
     print PBOUT $_;
 }
@@ -3205 +3241 @@
     }
     # We may need a proxy configuration. Get it from the local env
-
-    # TODO: make this apply to the bootstrapping also.
-    my ($ftp_proxy_map, $http_proxy_map) = pb_conf_get_if('ftp_proxy', 'http_proxy');
-    my $ftp_proxy = pb_distro_get_param($pbos, $ftp_proxy_map);
-    my $http_proxy = pb_distro_get_param($pbos, $http_proxy_map);
-
-    $ENV{'ftp_proxy'} = $ftp_proxy if ((defined $ftp_proxy) && ($ftp_proxy ne '') && (not defined $ENV{'ftp_proxy'}));
-    $ENV{'http_proxy'} = $http_proxy if ((defined $http_proxy) && ($http_proxy ne '') && (not defined $ENV{'http_proxy'}));
+    my $ftp_proxy = pb_distro_get_param($pbos,pb_conf_get_if("ftp_proxy"));
+    my $http_proxy = pb_distro_get_param($pbos,pb_conf_get_if("http_proxy"));
+
+    $ENV{'ftp_proxy'} ||= $ftp_proxy if ((defined $ftp_proxy) && ($ftp_proxy ne ""));
+    $ENV{'http_proxy'} ||= $http_proxy if ((defined $http_proxy) && ($http_proxy ne ""));
 
     if (defined $ENV{'http_proxy'}) {
@@ -3267 +3300 @@
             $cmdget = "mv /tmp/ProjectBuilder-$pbver1$pbextdir.tar.gz ProjectBuilder-latest.tar.gz ; mv /tmp/project-builder-$pbver2$pbextdir.tar.gz project-builder-latest.tar.gz";
         } else {
+            # TODO: shouldn't we have a variable to specify the source file?
+            # And shouldn't we be getting the one that matches the current pb version?
+            # For the moment, just take the latest stable one
             $cmdget = "wget --passive-ftp ftp://ftp.project-builder.org/src/ProjectBuilder-latest.tar.gz; wget --passive-ftp ftp://ftp.project-builder.org/src/project-builder-latest.tar.gz";
         }

devel/pb/lib/ProjectBuilder/Filter.pm

@@ -119 +119 @@
         while(<CONF>)  {
             if (/^\s*([A-z0-9-_]+)\s+([[A-z0-9-_]+)\s*=\s*(.+)$/) {
-                pb_log(3,"DEBUG creating entry $1, key $2, value $3\n");
-                $h{$1}{$2}=$3;
+                my ($what, $var, $value) = ($1, $2, $3);
+                pb_log(3,"DEBUG creating entry $what, key $var, value $value\n");
+                # Add support for multi-lines
+                while ($value =~ s/\\\s*$//o) {
+                    $_ = <CONF>;
+                    die "Still processing continuations for $what $var at EOF" if (not defined $_);
+                    s/[\r\n]//go;
+                    $value .= "\n$_";
+                }
+                $h{$what}{$var}=$value;
+            } elsif ((/^\s*#/o) || (/^\s*$/o)) {
+                # ignore
+            } else {
+                chomp();
+                warn "unexpected line '$_' in $f";
             }
         }
@@ -126 +139 @@
     }
     $ptr = $h{"filter"};
+
+    # TODO: find a place to document it. Why not in this file as pod and also documenting filters ?
+    # Handle transform
+    if (defined $h{transform}) {
+        while (my ($out_key,$spec) = each %{$h{transform}}) {
+            die "Unknown transform for $out_key '$spec' expected <out-key> <transform>" unless $spec =~ /^([\w\-]+)\s+(.+)$/;
+            my ($in_key, $expr) = ($1, $2);
+            local $_ = $ptr->{$in_key} || '';
+            eval $expr;
+            die "Error evaluating tranform for $out_key ($expr): $@" if $@;
+            $ptr->{$out_key} = $_;
+            pb_log(2, "Transform $in_key to $out_key\n$ptr->{$in_key}\n$ptr->{$out_key}\n");
+        }
+    }
 }
 pb_log(2,"DEBUG f:".Dumper($ptr)."\n") if (defined $ptr);