Changeset 1549

May 21, 2012, 2:17:30 AM (7 years ago)
  • pb: Improve build2pkg documentation so It can be referenced in build2ve. Document need to specify verpmtype for a rpm-style VE. Add substep of further fixes to filters and build files in build2ve. Provide example of needing an additional repo, which is more likely on centos. Fix bug with adapting owner since centos5.8 sets up so that we can't sudo until the sudoers file is fixed, but we don't do so until later. However, if we call su, then we need to keep the additional sudo. Apply the ftp/http proxy bits from the config file while setting up a VE. Don't try to mount /proc if it is already mounted. Fiddle with the substitution on requiretty; it wasn't working so moved to \s+ instead of [ \t]+ also left a comment to show users that the line was removed. Remove comment about applying ftp/http proxy to bootstrapping since we now do. Add note that we probably want a variable for where to get the source snapshot for source "installs" of project-builder.
  • Add support for multi-line filter variables since a description in an rpm or deb file is usually multiple lines long (this is now in addition to the support of the $/ variable). Add support for transforming those things since debian files use leading spaces, and rpm files don't. Transforms let you start with the same variable and fix it for the different OSs. Leave note about unknown "correct" documentation location.
2 edited


  • devel/pb/bin/pb

    r1548 r1549  
    492492=head2 build2pkg
    494 The build2pkg command takes the tar files created in the cms2build step and attempts to build the binary packages for your current operating system. In order for this step to work, you may need to edit the files in one of the $pbdefdir/$project/pbconf/$version/$project/{deb,rpm,pkg} directories. Those files will be used to try to build your package. Note that if you change those files, you need to re-run the cms2build step. Then you can run a command like:
     494The build2pkg command takes the tar files created in the cms2build step and attempts to build binary packages for your current operating system. There are two steps:
     496=over 4
     498=item Update your filters and build files.
     500You probably need to edit the files describing the build steps in one of the $pbdefdir/$project/pbconf/$version/$project/{deb,rpm,pkg} directories and the filters in $pbdefdir/$project/pbconf/$version/pbfilter. Note that you can define additional filters and transforms in the *.pbf files. The build files will be filtered by the filters defined in the *.pbf files to generate the inputs to the build step. Therefore, if you change those files, you need to re-run the cms2build step.
     502=item Build the package.
     504Then you can run a command like:
    496506 % pb -p $project -r $version build2pkg
    498508To create the files in $project/build that comprise your binary package(s).
    500512=head2 newve
    513525 vetype default = chroot
     527If you are building for rpm style OS's, update the verpmtype option, and install the appropriate tool.
     529 verpmtype default = rpmbootstrap
    515531You may also choose to specify a mirror for the OS packages, and optionally http/ftp proxies.  You can specify the proxies either through environment variables ($http_proxy/$ftp_proxy) or in the configuration file. The configuration file will be used if no corresponding environment variable has been set. For example, for debian and with a local squid proxy:
    555571The build2ve command is similar to the build2pkg command in that it will take the sources created by cms2build and turn them into binary packages. The command has two differences. First, it creates the packages in a virtual environment, i.e. the one made by an earlier setupve setup. Second it copies the resulting packages to a repository and builds the repository meta-data needed.
    557 Two sub-steps are needed:
     573Three sub-steps are needed:
    559575=over 4
    565581 sshhost $project = localhost
    566582 sshdir $project = $home/cache-project-builder/repos
     584You may also need to specify additional repository files to use or rpms to install. Note the URL for repositories is not the URL of the repository, but the URL of a file that can be put in the yum.repos.d or apt.sources.d directory.
     586 addrepo centos-5-i386 = http://localhost/pb/centos-extras.repo,
     588=item Update your filters and build files
     590You may need to update your filter files (*.pbf) as in the build2pkg step if you are building for a new OS or architecture.
    568592=item Build the packages and copy them to the repository
    21652189    # ssh communication if not VE or CPAN
    21662190    # should use a hash instead...
    2167     my ($shcmd,$cpcmd,$cptarget,$cp2target);
     2191    my ($shcmd,$shcmdroot,$cpcmd,$cptarget,$cp2target);
    21682192    if ($cmt =~ /^VE/) {
    21692193        my $tp = pb_path_expand($vepath->{$ENV{'PBPROJ'}});
    21732197        my $arch = pb_get_arch();
    21742198        if ($vetype eq "chroot") {
    2175             $shcmd = "sudo /usr/sbin/chroot $tpdir /bin/su - $mac -c ";
     2199            $shcmdroot = "sudo /usr/sbin/chroot $tpdir ";
     2200            $shcmd = "$shcmdroot /bin/su - $mac -c ";
    21762201        } elsif ($vetype eq "schroot") {
    21772202            $shcmd = "schroot $tp -u $mac -- ";
    21782203        }
    2179         $sshcmd = "setarch i386 $sshcmd" if (($pbos->{'arch'} =~ /i?86/) && ($arch eq 'x86_64'));
     2204        $shcmd = "setarch i386 $shcmd" if (($pbos->{'arch'} =~ /i?86/) && ($arch eq 'x86_64'));
    21802205        $cpcmd = "sudo /bin/cp -r ";
    21812206        # We need to get the home dir of the target account to deliver in the right place
    22332258    # For VE we need to change the owner manually
    22342259    if ($cmt =~ /^VE/) {
    2235         pb_system("$shcmd \"sudo chown -R $mac $tdir\"","Adapt owner in $tdir to $mac");
     2260        if (defined $shcmdroot) {
     2261            # This should help overcome a CentOS 5.8 bug as well as having a simper sequence
     2262            pb_system("$shcmdroot \"chown -R $mac $tdir\"","Adapt owner in $tdir to $mac");
     2263        } else {
     2264            pb_system("$shcmd \"sudo chown -R $mac $tdir\"","Adapt owner in $tdir to $mac");
     2265        }
    22362266    }
    24212451    my $pbos = pb_distro_get_context($v);
     2453    my $ftp_proxy = pb_distro_get_param($pbos,pb_conf_get_if("ftp_proxy"));
     2454    my $http_proxy = pb_distro_get_param($pbos,pb_conf_get_if("http_proxy"));
     2456    $ENV{ftp_proxy} ||= $ftp_proxy if ((defined $ftp_proxy) && ($ftp_proxy ne ""));
     2457    $ENV{http_proxy} ||= $http_proxy if ((defined $http_proxy) && ($http_proxy ne ""));
    24232459    # Launch the VMs/VEs
    24242460    if ($vtype eq "vm") {
    29963032# For VE we first need to mount some FS
    2997 pb_system("mount -t proc /proc /proc");
     3033pb_system("mount -t proc /proc /proc") unless (-d "/proc/$$");
    31543190    print SCRIPT << 'EOF';
    3155     s/Defaults[ \t]+requiretty//;
     3191    s/Defaults\s+requiretty/# disable Defaults: requiretty/;
    31563192    print PBOUT $_;
    32053241    }
    32063242    # We may need a proxy configuration. Get it from the local env
    3208     # TODO: make this apply to the bootstrapping also.
    3209     my ($ftp_proxy_map, $http_proxy_map) = pb_conf_get_if('ftp_proxy', 'http_proxy');
    3210     my $ftp_proxy = pb_distro_get_param($pbos, $ftp_proxy_map);
    3211     my $http_proxy = pb_distro_get_param($pbos, $http_proxy_map);
    3213     $ENV{'ftp_proxy'} = $ftp_proxy if ((defined $ftp_proxy) && ($ftp_proxy ne '') && (not defined $ENV{'ftp_proxy'}));
    3214     $ENV{'http_proxy'} = $http_proxy if ((defined $http_proxy) && ($http_proxy ne '') && (not defined $ENV{'http_proxy'}));
     3243    my $ftp_proxy = pb_distro_get_param($pbos,pb_conf_get_if("ftp_proxy"));
     3244    my $http_proxy = pb_distro_get_param($pbos,pb_conf_get_if("http_proxy"));
     3246    $ENV{'ftp_proxy'} ||= $ftp_proxy if ((defined $ftp_proxy) && ($ftp_proxy ne ""));
     3247    $ENV{'http_proxy'} ||= $http_proxy if ((defined $http_proxy) && ($http_proxy ne ""));
    32163249    if (defined $ENV{'http_proxy'}) {
    32673300            $cmdget = "mv /tmp/ProjectBuilder-$pbver1$pbextdir.tar.gz ProjectBuilder-latest.tar.gz ; mv /tmp/project-builder-$pbver2$pbextdir.tar.gz project-builder-latest.tar.gz";
    32683301        } else {
     3302            # TODO: shouldn't we have a variable to specify the source file?
     3303            # And shouldn't we be getting the one that matches the current pb version?
     3304            # For the moment, just take the latest stable one
    32693305            $cmdget = "wget --passive-ftp; wget --passive-ftp";
    32703306        }
  • devel/pb/lib/ProjectBuilder/

    r1434 r1549  
    119119        while(<CONF>)  {
    120120            if (/^\s*([A-z0-9-_]+)\s+([[A-z0-9-_]+)\s*=\s*(.+)$/) {
    121                 pb_log(3,"DEBUG creating entry $1, key $2, value $3\n");
    122                 $h{$1}{$2}=$3;
     121                my ($what, $var, $value) = ($1, $2, $3);
     122                pb_log(3,"DEBUG creating entry $what, key $var, value $value\n");
     123                # Add support for multi-lines
     124                while ($value =~ s/\\\s*$//o) {
     125                    $_ = <CONF>;
     126                    die "Still processing continuations for $what $var at EOF" if (not defined $_);
     127                    s/[\r\n]//go;
     128                    $value .= "\n$_";
     129                }
     130                $h{$what}{$var}=$value;
     131            } elsif ((/^\s*#/o) || (/^\s*$/o)) {
     132                # ignore
     133            } else {
     134                chomp();
     135                warn "unexpected line '$_' in $f";
    123136            }
    124137        }
    126139    }
    127140    $ptr = $h{"filter"};
     142    # TODO: find a place to document it. Why not in this file as pod and also documenting filters ?
     143    # Handle transform
     144    if (defined $h{transform}) {
     145        while (my ($out_key,$spec) = each %{$h{transform}}) {
     146            die "Unknown transform for $out_key '$spec' expected <out-key> <transform>" unless $spec =~ /^([\w\-]+)\s+(.+)$/;
     147            my ($in_key, $expr) = ($1, $2);
     148            local $_ = $ptr->{$in_key} || '';
     149            eval $expr;
     150            die "Error evaluating tranform for $out_key ($expr): $@" if $@;
     151            $ptr->{$out_key} = $_;
     152            pb_log(2, "Transform $in_key to $out_key\n$ptr->{$in_key}\n$ptr->{$out_key}\n");
     153        }
     154    }
    129156pb_log(2,"DEBUG f:".Dumper($ptr)."\n") if (defined $ptr);
Note: See TracChangeset for help on using the changeset viewer.