Changeset 1545 in ProjectBuilder


Ignore:
Timestamp:
May 14, 2012, 4:13:43 AM (12 years ago)
Author:
Bruno Cornec
Message:
  • pb: Add in better documentation for setupve, and add documentation for build2ve. Make sure that if we are doing a parallel build we will stop on errors if that option is set. Fix the code for determining which output files were made. The .dsc, .tar.gz, and .changes files are named based on the Source: option in the debian control file. The .deb files can be extracted from there or (as implemented) from the output of dpkg-deb. Verify that all expected files are present. Refactor code to avoid duplicating long filename construction. Fix bug in running in 32bit VE with 64bit host; the setarch command needs to be used to make the child think it is running in a 32 bit environment. Make the use of a key option to ssh optional so that if ssh-agent is in use no key-file is generated. Do *not* set the time in a virtual environment. It is getting it's time from the parent OS and I believe this piece of code was mis-setting the time since the VE thought it was in UTC rather than PST for the parent and so was shifting the clock by many hours. In the script generated in the sub-child, enable set -e if pb_stop_on_error is set so that if the package build fails we don't try to keep going. Having done that, /proc can remain around on failure, so tolerate that state. Fix the documentation on how the date is being set. Invert the loop that makes the bootstrapping project-builder so that we only include each of the .pm files once; if you have installed the pb-modules and have a locally built copy in @INC you could get both copies in the bootstrap version which lead to updating the source not changing the behavior. When getting the ssh-keyfile, if ssh-agent is running, and asked to use it then use it. Tolerate failures on the debsign step -- the key-file can validly not exist. (Eric Anderson)
  • Add a new option pbusesshagent in pb.conf to allow the user to specify whether he wants to use his existing SSH environement and agent, or if he prefers to create a new key pair for building. (Bruno Cornec)
Location:
devel
Files:
3 edited

Legend:

Unmodified
Added
Removed

  • devel/pb-modules/etc/pb.conf

    r1530 r1545  
    620620ospathcmd-halt solaris = /usr/sbin/halt
    621621
     622# Do you want pb to create a dedicated SSH key pair to dialog with VM|RM
     623# or do you prefer to use an existing SSH Agent instead and existing keys
     624pbusesshagent default = false
     625
    622626# How do you want to install pb in your VM/VE (pkg or file)
    623627pbinstalltype default = pkg

  • devel/pb-modules/etc/pb.conf.pod

    r1544 r1545  
    524524 Nature: Optional
    525525 Key: project (as defined in the -p option or PBPROJ environment variable)
    526  Value: The directory under which will be found your secret key file.
     526 Value: The directory under which will be found your secret GPG key file.
    527527 Conffile: home
    528528 Example: pbpasspath pb = /home/me/.gnupg
    … …  
    576576 Example: pburl linuxcoe = cvs+ssh://:ext:user@linuxcoe.cvs.sourceforge.net:/cvsroot/linuxcoe, pburl pb = svk:////local/pb
    577577
     578=item B<pbusesshagent>
     579
     580 Nature: Mandatory
     581 Key: project (as defined in the -p option or PBPROJ environment variable)
     582 Value: false means that you want pb to create a dedicated SSH key pair to dialog with VM|RM and false that you prefer to use an existing SSH Agent instead and existing keys
     583 Conffile: pb
     584 Example: pbusesshagent default = false
     585
    578586=item B<pbwf>
    579587

  • devel/pb/bin/pb

    r1542 r1545  
    533533=head2 setupve
    534534
    535 The setupve command prepares a virtual environment for use by project builder. In particular it installs project-builder from the packages into the virtual environment. For example:
     535The setupve command prepares a virtual environment for use by project builder. In particular it installs project-builder from the packages into the virtual environment. Two sub-steps are necessary:
     536
     537=over 4
     538
     539=item Update $project.pb
     540
     541You need to have a sshhost entry for setupve to work, so add one, even an invalid one, e.g.:
     542
     543 sshhost $project = foo.example.org
     544
     545=item Setup the virtual environment
    536546
    537547 % pb -v -p $project -m debian-6.0-i386 setupve
    538548
    539549If you prefer to install the current SVN version of project builder, you can substitute the setupve option by the sbx2setupv one.
     550
     551=back
     552
     553=head2 build2ve
     554
     555The build2ve command is similar to the build2pkg command in that it will take the sources created by cms2build and turn them into binary packages. The command has two differences. First, it creates the packages in a virtual environment, i.e. the one made by an earlier setupve setup. Second it copies the resulting packages to a repository and builds the repository meta-data needed.
     556
     557Two sub-steps are needed:
     558
     559=over 4
     560
     561=item Update $project.pb
     562
     563You need to have a valid sshdir and sshhost entry for build2ve to work, so add them. Note that you need to be able to ssh from the host you run the command on to the repository host, preferably without needing to type in a password, so using ssh-agent or having a special passwordless project-builder ssh key will make this step easier.
     564
     565 sshhost $project = localhost
     566 sshdir $project = $home/cache-project-builder/repos
     567
     568=item Build the packages and copy them to the repository
     569
     570 % pb -v -p $project -m debian-6.0-i386 build2ve
     571
     572=back
     573
     574*Debugging:* If the build fails (and you did not specify the --no-stop-on-error) option, then the virtual environment and scripts should still be present and configured to build the package. You can run a command like 'sudo setarch i386 chroot $path bash' in order to get into the environment. In your log you should see a command like that. From there you can go into the /home/pb directory as the pb user and run the same style of pb commands as you did when doing build2pkg. This will help you figure out what has gone wrong in the build in the virtual environment.
    540575
    541576=head1 AUTHORS
    … …  
    13291364    my $made = ""; # pkgs made during build
    13301365    my $pm;
    1331     $pm = new Parallel::ForkManager($pbparallel) if (defined $pbparallel);
     1366    my $all_ok = 1;
     1367
     1368    if (defined $pbparallel) {
     1369        $pm = new Parallel::ForkManager($pbparallel);
     1370        $pm->run_on_finish(sub { my ($pid, $code, $id, $signal, $dump) = @_;
     1371                                $all_ok = 0 unless (($code == 0) && ($signal == 0) && ($dump == 0)); });
     1372    }
    13321373
    13331374    # We need to communicate info back from the children if parallel so prepare a dir for that
    … …  
    14531494            # Get the name of the generated packages
    14541495            open(LOG,"$ENV{'PBTMP'}/system.$$.log") || die "Unable to open $ENV{'PBTMP'}/system.$$.log";
     1496            my $tmp = "";
    14551497            while (<LOG>) {
    14561498                chomp();
    1457                 my $tmp = $_;
    1458                 next if ($tmp !~ /^dpkg-deb.*:/);
    1459                 $tmp =~ s|.*\.\./(.*)_(.*).deb.*|$1|;
    1460                 $made="$made $tmp.dsc $tmp.tar.gz $tmp"."_*.deb $tmp"."_*.changes";
     1499                next unless (/^dpkg-deb.*:\s+building\s+package\s+.*\s+in\s+\`\.\.\/(\S+)\'\./o);
     1500                $tmp = $1;
     1501                die "Missing file $tmp" if (not -f $tmp);
     1502                $made = "$made $tmp";
    14611503            }
    14621504            close(LOG);
     1505            open(CTRL,"debian/control") or die "Unable to open debian/control: $!";
     1506            #$made="$made $tmp.dsc $tmp.tar.gz $tmp"."_*.deb $tmp"."_*.changes";
     1507            while (<CTRL>) {
     1508                next unless (/^Source: (\S+)/o);
     1509                foreach my $glob (("$1\_*.changes", "$1\_*.dsc", "$1\_*.tar.gz")) {
     1510                    my @file = glob($glob);
     1511                    die "Missing file for $glob" unless @file > 0;
     1512                    die "Too many files for $glob" if @file > 1;
     1513                    die "Missing file $file[0]" if (not -f $file[0]);
     1514                    $made .= " $file[0]";
     1515                }
     1516            }
     1517            close(CTRL);
     1518            pb_display_file("$ENV{'PBTMP'}/system.$$.log");
     1519
    14631520            chdir ".." || die "Unable to chdir to parent dir";
    14641521            pb_rm_rf("$pbpkg-$pbver");
    … …  
    16191676            $made .= " ".pb_get_content($f);
    16201677        }
     1678        die "Aborting, one or more of the children failed.\n" if ((not $all_ok) && ($Global::pb_stop_on_error));
    16211679        pb_rm_rf($tmpd);
    16221680    }
    … …  
    16611719
    16621720    # Keep track of what is generated so that we can get them back from VMs/RMs
    1663     open(KEEP,"> $ENV{'PBBUILDDIR'}/pbgen-$ENV{'PBPROJVER'}-$ENV{'PBPROJTAG'}-$pbos->{'name'}-$pbos->{'version'}-$pbos->{'arch'}") || die "Unable to create $ENV{'PBBUILDDIR'}/pbgen-$ENV{'PBPROJVER'}-$ENV{'PBPROJTAG'}-$pbos->{'name'}-$pbos->{'version'}-$pbos->{'arch'}";
     1721    my $pbkeep = "$ENV{'PBBUILDDIR'}/pbgen-$ENV{'PBPROJVER'}-$ENV{'PBPROJTAG'}-$pbos->{'name'}-$pbos->{'version'}-$pbos->{'arch'}";
     1722    open(KEEP,"> $pbkeep") || die "Unable to create $pbkeep: $!";
    16641723    print KEEP "$made\n";
    16651724    close(KEEP);
    … …  
    21102169        my ($ptr) = pb_conf_get("vetype");
    21112170        my $vetype = $ptr->{$ENV{'PBPROJ'}};
     2171        my $arch = pb_get_arch();
    21122172        if ($vetype eq "chroot") {
    21132173            $shcmd = "sudo /usr/sbin/chroot $tpdir /bin/su - $mac -c ";
    … …  
    21152175            $shcmd = "schroot $tp -u $mac -- ";
    21162176        }
     2177        $sshcmd = "setarch i386 $sshcmd" if (($pbos->{'arch'} =~ /i?86/) && ($arch eq 'x86_64'));
    21172178        $cpcmd = "sudo /bin/cp -r ";
    21182179        # We need to get the home dir of the target account to deliver in the right place
    … …  
    21482209    } else {
    21492210        my $keyfile = pb_ssh_get(0);
     2211        my $keyopt = defined $keyfile ? "-i $keyfile" : "";
    21502212        my $sshcmd = pb_check_req("ssh",1);
    21512213        my $scpcmd = pb_check_req("scp",1);
    2152         $shcmd = "$sshcmd -i $keyfile -q -o UserKnownHostsFile=/dev/null -p $nport $mac";
    2153         $cpcmd = "$scpcmd -i $keyfile -p -o UserKnownHostsFile=/dev/null -P $nport";
     2214        $shcmd = "$sshcmd $keyopt -q -o NoHostAuthenticationForLocalhost=yes -p $nport $mac";
     2215        $cpcmd = "$scpcmd $keyopt -p -o NoHostAuthenticationForLocalhost=yes -P $nport";
    21542216        $cptarget = "$mac:$tdir";
    21552217        if ($cmt =~ /^(V|R)Mbuild/) {
    … …  
    25052567my $pbos = shift;
    25062568
     2569# VE gets time from parent OS.
     2570return "/bin/true" if ($vtype) =~ /^ve/o;
     2571
    25072572my ($ntp) = pb_conf_get_if($vtype."ntp");
    25082573my $vntp = $ntp->{$ENV{'PBPROJ'}} if (defined $ntp);
    … …  
    25672632        $i++;
    25682633    }
     2634    print SCRIPT "set -e\n" if $Global::pb_stop_on_error;
    25692635    # Activate script verbosity if at least 2 for pbdebug
    25702636    print SCRIPT "set -x\n" if ($i gt 1);
    … …  
    25832649    }
    25842650
    2585     # VE needs a good /proc
     2651    # VE needs a good /proc, tolerate one being potentially left around after a failure
    25862652    if ($vtype eq "ve") {
    2587         print SCRIPT "sudo /bin/mount -t proc /proc /proc\n";
     2653        print SCRIPT "[ -d /proc/1 ] || sudo /bin/mount -t proc /proc /proc\n";
    25882654    }
    25892655
    … …  
    25932659    my $ntpline = pb_date2v($vtype,$pbos);
    25942660    print SCRIPT "# Time sync\n";
    2595     print SCRIPT "echo 'setting up date with '";
    2596     print SCRIPT "echo $ntpline\n";
     2661    print SCRIPT "echo setting up date with $ntpline\n";
    25972662    print SCRIPT "$ntpline\n";
    25982663    # Use potential local proxy declaration in case we need it to download repo, pkgs, ...
    … …  
    32553320
    32563321    # Adds pb_distro_get_context and all functions needed from ProjectBuilder::Distribution, Conf and Base
    3257     foreach my $d (@INC) {
    3258         my @f = ("$d/ProjectBuilder/Base.pm","$d/ProjectBuilder/Distribution.pm","$d/ProjectBuilder/Conf.pm");
    3259         foreach my $f (@f) {
     3322    foreach my $m ("ProjectBuilder/Base.pm","ProjectBuilder/Distribution.pm","ProjectBuilder/Conf.pm") {
     3323        foreach my $d (@INC) {
     3324            my $f = "$d/$m";
    32603325            if (-f "$f") {
    3261                 open(PBD,"$f") || die "Unable to open $f";
     3326                open(PBD,"$f") || die "Unable to open $f: $!";
    32623327                while (<PBD>) {
    32633328                    next if (/^package/);
    … …  
    32683333                }
    32693334                close(PBD);
     3335                # We just need the first one of each file wrt @INC - hopefully that's the right one.
     3336                last;
    32703337            }
    32713338        }
    … …  
    38413908my $create = shift || 0;    # Do not create keys by default
    38423909
     3910my ($pbagent) = pb_conf_get_if("pbusesshagent");
     3911# use ssh-agent if asked so.
     3912return(undef) if (($create == 0) && (defined $pbagent->{$ENV{'PBPROJ'}}) && ($pbagent->{$ENV{'PBPROJ'}} =~ /true/io));
     3913
    38433914# Check the SSH environment
    38443915my $keyfile = undef;
    … …  
    38533924$keyfile = "$ENV{'HOME'}/.ssh/id_dsa" if (-s "$ENV{'HOME'}/.ssh/id_dsa");
    38543925$keyfile = "$ENV{'HOME'}/.ssh/pb_dsa" if (-s "$ENV{'HOME'}/.ssh/pb_dsa");
    3855 die "Unable to find your public ssh key under $keyfile" if (not defined $keyfile);
     3926die "Unable to find your public ssh key under $ENV{'HOME'}/.ssh" if (not defined $keyfile);
    38563927return($keyfile);
    38573928}
    … …  
    41494220    }
    41504221    my $debsigncmd = pb_check_req("debsign",1);
    4151     pb_system("$debsigncmd -m\'$ENV{'PBPACKAGER'}\' $changes","Signing DEB packages") if ($changes ne "");
     4222    pb_system("$debsigncmd -m\'$ENV{'PBPACKAGER'}\' $changes","Signing DEB packages",undef,1) if ($changes ne "");
    41524223} else {
    41534224    pb_log(0,"I don't know yet how to sign packages for type $pbos->{'type'}.\nPlease give feedback to dev team\n");
Note: See TracChangeset for help on using the changeset viewer.