Changeset 1545 in ProjectBuilder for devel/pb/bin

Timestamp:

May 14, 2012, 4:13:43 AM (13 years ago)

Author:

Bruno Cornec

Message:

• pb: Add in better documentation for setupve, and add documentation for build2ve. Make sure that if we are doing a parallel build we will stop on errors if that option is set. Fix the code for determining which output files were made. The .dsc, .tar.gz, and .changes files are named based on the Source: option in the debian control file. The .deb files can be extracted from there or (as implemented) from the output of dpkg-deb. Verify that all expected files are present. Refactor code to avoid duplicating long filename construction. Fix bug in running in 32bit VE with 64bit host; the setarch command needs to be used to make the child think it is running in a 32 bit environment. Make the use of a key option to ssh optional so that if ssh-agent is in use no key-file is generated. Do *not* set the time in a virtual environment. It is getting it's time from the parent OS and I believe this piece of code was mis-setting the time since the VE thought it was in UTC rather than PST for the parent and so was shifting the clock by many hours. In the script generated in the sub-child, enable set -e if pb_stop_on_error is set so that if the package build fails we don't try to keep going. Having done that, /proc can remain around on failure, so tolerate that state. Fix the documentation on how the date is being set. Invert the loop that makes the bootstrapping project-builder so that we only include each of the .pm files once; if you have installed the pb-modules and have a locally built copy in @INC you could get both copies in the bootstrap version which lead to updating the source not changing the behavior. When getting the ssh-keyfile, if ssh-agent is running, and asked to use it then use it. Tolerate failures on the debsign step -- the key-file can validly not exist. (Eric Anderson)
• Add a new option pbusesshagent in pb.conf to allow the user to specify whether he wants to use his existing SSH environement and agent, or if he prefers to create a new key pair for building. (Bruno Cornec)

File:

• devel/pb/bin/pb (modified) (17 diffs)

devel/pb/bin/pb

@@ -533 +533 @@
 =head2 setupve
 
-The setupve command prepares a virtual environment for use by project builder. In particular it installs project-builder from the packages into the virtual environment. For example:
+The setupve command prepares a virtual environment for use by project builder. In particular it installs project-builder from the packages into the virtual environment. Two sub-steps are necessary:
+
+=over 4
+
+=item Update $project.pb
+
+You need to have a sshhost entry for setupve to work, so add one, even an invalid one, e.g.:
+
+ sshhost $project = foo.example.org
+
+=item Setup the virtual environment
 
  % pb -v -p $project -m debian-6.0-i386 setupve
 
 If you prefer to install the current SVN version of project builder, you can substitute the setupve option by the sbx2setupv one.
+
+=back
+
+=head2 build2ve
+
+The build2ve command is similar to the build2pkg command in that it will take the sources created by cms2build and turn them into binary packages. The command has two differences. First, it creates the packages in a virtual environment, i.e. the one made by an earlier setupve setup. Second it copies the resulting packages to a repository and builds the repository meta-data needed.
+
+Two sub-steps are needed:
+
+=over 4
+
+=item Update $project.pb
+
+You need to have a valid sshdir and sshhost entry for build2ve to work, so add them. Note that you need to be able to ssh from the host you run the command on to the repository host, preferably without needing to type in a password, so using ssh-agent or having a special passwordless project-builder ssh key will make this step easier.
+
+ sshhost $project = localhost
+ sshdir $project = $home/cache-project-builder/repos
+
+=item Build the packages and copy them to the repository
+
+ % pb -v -p $project -m debian-6.0-i386 build2ve
+
+=back
+
+*Debugging:* If the build fails (and you did not specify the --no-stop-on-error) option, then the virtual environment and scripts should still be present and configured to build the package. You can run a command like 'sudo setarch i386 chroot $path bash' in order to get into the environment. In your log you should see a command like that. From there you can go into the /home/pb directory as the pb user and run the same style of pb commands as you did when doing build2pkg. This will help you figure out what has gone wrong in the build in the virtual environment.
 
 =head1 AUTHORS
@@ -1329 +1364 @@
     my $made = ""; # pkgs made during build
     my $pm;
-    $pm = new Parallel::ForkManager($pbparallel) if (defined $pbparallel);
+    my $all_ok = 1;
+
+    if (defined $pbparallel) {
+        $pm = new Parallel::ForkManager($pbparallel);
+        $pm->run_on_finish(sub { my ($pid, $code, $id, $signal, $dump) = @_;
+                                $all_ok = 0 unless (($code == 0) && ($signal == 0) && ($dump == 0)); });
+    }
 
     # We need to communicate info back from the children if parallel so prepare a dir for that
@@ -1453 +1494 @@
             # Get the name of the generated packages
             open(LOG,"$ENV{'PBTMP'}/system.$$.log") || die "Unable to open $ENV{'PBTMP'}/system.$$.log";
+            my $tmp = "";
             while (<LOG>) {
                 chomp();
-                my $tmp = $_;
-                next if ($tmp !~ /^dpkg-deb.*:/);
-                $tmp =~ s|.*\.\./(.*)_(.*).deb.*|$1|;
-                $made="$made $tmp.dsc $tmp.tar.gz $tmp"."_*.deb $tmp"."_*.changes";
+                next unless (/^dpkg-deb.*:\s+building\s+package\s+.*\s+in\s+\`\.\.\/(\S+)\'\./o);
+                $tmp = $1;
+                die "Missing file $tmp" if (not -f $tmp);
+                $made = "$made $tmp";
             }
             close(LOG);
+            open(CTRL,"debian/control") or die "Unable to open debian/control: $!";
+            #$made="$made $tmp.dsc $tmp.tar.gz $tmp"."_*.deb $tmp"."_*.changes";
+            while (<CTRL>) {
+                next unless (/^Source: (\S+)/o);
+                foreach my $glob (("$1\_*.changes", "$1\_*.dsc", "$1\_*.tar.gz")) {
+                    my @file = glob($glob);
+                    die "Missing file for $glob" unless @file > 0;
+                    die "Too many files for $glob" if @file > 1;
+                    die "Missing file $file[0]" if (not -f $file[0]);
+                    $made .= " $file[0]";
+                }
+            }
+            close(CTRL);
+            pb_display_file("$ENV{'PBTMP'}/system.$$.log");
+
             chdir ".." || die "Unable to chdir to parent dir";
             pb_rm_rf("$pbpkg-$pbver");
@@ -1619 +1676 @@
             $made .= " ".pb_get_content($f);
         }
+        die "Aborting, one or more of the children failed.\n" if ((not $all_ok) && ($Global::pb_stop_on_error));
         pb_rm_rf($tmpd);
     }
@@ -1661 +1719 @@
 
     # Keep track of what is generated so that we can get them back from VMs/RMs
-    open(KEEP,"> $ENV{'PBBUILDDIR'}/pbgen-$ENV{'PBPROJVER'}-$ENV{'PBPROJTAG'}-$pbos->{'name'}-$pbos->{'version'}-$pbos->{'arch'}") || die "Unable to create $ENV{'PBBUILDDIR'}/pbgen-$ENV{'PBPROJVER'}-$ENV{'PBPROJTAG'}-$pbos->{'name'}-$pbos->{'version'}-$pbos->{'arch'}";
+    my $pbkeep = "$ENV{'PBBUILDDIR'}/pbgen-$ENV{'PBPROJVER'}-$ENV{'PBPROJTAG'}-$pbos->{'name'}-$pbos->{'version'}-$pbos->{'arch'}";
+    open(KEEP,"> $pbkeep") || die "Unable to create $pbkeep: $!";
     print KEEP "$made\n";
     close(KEEP);
@@ -2110 +2169 @@
         my ($ptr) = pb_conf_get("vetype");
         my $vetype = $ptr->{$ENV{'PBPROJ'}};
+        my $arch = pb_get_arch();
         if ($vetype eq "chroot") {
             $shcmd = "sudo /usr/sbin/chroot $tpdir /bin/su - $mac -c ";
@@ -2115 +2175 @@
             $shcmd = "schroot $tp -u $mac -- ";
         }
+        $sshcmd = "setarch i386 $sshcmd" if (($pbos->{'arch'} =~ /i?86/) && ($arch eq 'x86_64'));
         $cpcmd = "sudo /bin/cp -r ";
         # We need to get the home dir of the target account to deliver in the right place
@@ -2148 +2209 @@
     } else {
         my $keyfile = pb_ssh_get(0);
+        my $keyopt = defined $keyfile ? "-i $keyfile" : "";
         my $sshcmd = pb_check_req("ssh",1);
         my $scpcmd = pb_check_req("scp",1);
-        $shcmd = "$sshcmd -i $keyfile -q -o UserKnownHostsFile=/dev/null -p $nport $mac";
-        $cpcmd = "$scpcmd -i $keyfile -p -o UserKnownHostsFile=/dev/null -P $nport";
+        $shcmd = "$sshcmd $keyopt -q -o NoHostAuthenticationForLocalhost=yes -p $nport $mac";
+        $cpcmd = "$scpcmd $keyopt -p -o NoHostAuthenticationForLocalhost=yes -P $nport";
         $cptarget = "$mac:$tdir";
         if ($cmt =~ /^(V|R)Mbuild/) {
@@ -2505 +2567 @@
 my $pbos = shift;
 
+# VE gets time from parent OS.
+return "/bin/true" if ($vtype) =~ /^ve/o; 
+
 my ($ntp) = pb_conf_get_if($vtype."ntp");
 my $vntp = $ntp->{$ENV{'PBPROJ'}} if (defined $ntp);
@@ -2567 +2632 @@
         $i++;
     }
+    print SCRIPT "set -e\n" if $Global::pb_stop_on_error;
     # Activate script verbosity if at least 2 for pbdebug
     print SCRIPT "set -x\n" if ($i gt 1);
@@ -2583 +2649 @@
     }
 
-    # VE needs a good /proc
+    # VE needs a good /proc, tolerate one being potentially left around after a failure
     if ($vtype eq "ve") {
-        print SCRIPT "sudo /bin/mount -t proc /proc /proc\n";
+        print SCRIPT "[ -d /proc/1 ] || sudo /bin/mount -t proc /proc /proc\n";
     }
 
@@ -2593 +2659 @@
     my $ntpline = pb_date2v($vtype,$pbos);
     print SCRIPT "# Time sync\n";
-    print SCRIPT "echo 'setting up date with '";
-    print SCRIPT "echo $ntpline\n";
+    print SCRIPT "echo setting up date with $ntpline\n";
     print SCRIPT "$ntpline\n";
     # Use potential local proxy declaration in case we need it to download repo, pkgs, ...
@@ -3255 +3320 @@
 
     # Adds pb_distro_get_context and all functions needed from ProjectBuilder::Distribution, Conf and Base
-    foreach my $d (@INC) {
-        my @f = ("$d/ProjectBuilder/Base.pm","$d/ProjectBuilder/Distribution.pm","$d/ProjectBuilder/Conf.pm");
-        foreach my $f (@f) {
+    foreach my $m ("ProjectBuilder/Base.pm","ProjectBuilder/Distribution.pm","ProjectBuilder/Conf.pm") {
+        foreach my $d (@INC) {
+            my $f = "$d/$m";
             if (-f "$f") {
-                open(PBD,"$f") || die "Unable to open $f";
+                open(PBD,"$f") || die "Unable to open $f: $!";
                 while (<PBD>) {
                     next if (/^package/);
@@ -3268 +3333 @@
                 }
                 close(PBD);
+                # We just need the first one of each file wrt @INC - hopefully that's the right one.
+                last;
             }
         }
@@ -3841 +3908 @@
 my $create = shift || 0;    # Do not create keys by default
 
+my ($pbagent) = pb_conf_get_if("pbusesshagent");
+# use ssh-agent if asked so.
+return(undef) if (($create == 0) && (defined $pbagent->{$ENV{'PBPROJ'}}) && ($pbagent->{$ENV{'PBPROJ'}} =~ /true/io));
+
 # Check the SSH environment
 my $keyfile = undef;
@@ -3853 +3924 @@
 $keyfile = "$ENV{'HOME'}/.ssh/id_dsa" if (-s "$ENV{'HOME'}/.ssh/id_dsa");
 $keyfile = "$ENV{'HOME'}/.ssh/pb_dsa" if (-s "$ENV{'HOME'}/.ssh/pb_dsa");
-die "Unable to find your public ssh key under $keyfile" if (not defined $keyfile);
+die "Unable to find your public ssh key under $ENV{'HOME'}/.ssh" if (not defined $keyfile);
 return($keyfile);
 }
@@ -4149 +4220 @@
     }
     my $debsigncmd = pb_check_req("debsign",1);
-    pb_system("$debsigncmd -m\'$ENV{'PBPACKAGER'}\' $changes","Signing DEB packages") if ($changes ne "");
+    pb_system("$debsigncmd -m\'$ENV{'PBPACKAGER'}\' $changes","Signing DEB packages",undef,1) if ($changes ne "");
 } else {
     pb_log(0,"I don't know yet how to sign packages for type $pbos->{'type'}.\nPlease give feedback to dev team\n");