Wiki source code of Build Management

Version 19.1 by cds on 2013/04/04 12:40
Hide last authors
cds 19.1 1 = An Overview of KIELER Build Automation =
2
cds 11.1 3 In order to keep the KIELER code as stable as possible, we use automatic continuous integration builds. Once you push something into one of the KIELER repositories, you trigger the automatic build process. If the build fails for whatever reason – be it failed unit tests or simply compilation errors – you are notified of the problem.
tig 3.1 4
cds 11.1 5 This page describes the software and setup we use to implement all of this.
tig 3.1 6
cds 11.1 7 **Content**
tig 3.1 8
9
cds 11.1 10
11 {{toc/}}
12
cds 19.1 13 == Software We Use ==
cds 11.1 14
cds 16.1 15 To implement our automatic builds, we use the popular [[Maven>>url:http://maven.apache.org/||shape="rect"]] tool in conjunction with [[Tycho>>url:http://eclipse.org/tycho/||shape="rect"]], a set of Maven plug-ins that allow Maven to build Eclipse projects. Our KLay layouters library is built using [[Apache Ant>>url:http://ant.apache.org/||shape="rect"]]. To implement our continuous integration builds, we use [[Atlassian Bamboo>>url:http://www.atlassian.com/software/bamboo/overview||shape="rect"]].
cds 11.1 16
cds 19.1 17 === Maven / Tycho ===
cds 11.1 18
19 Maven is a build tool for Java projects. It takes care of dependency management, including in-build dependencies (the order in which packages are compiled) as well as dependencies to third-party libraries. The latter are automatically fetched from special Maven repositories. Without getting too technical, a Maven build consists of several phases, such as //compile// and //package//. Within each phase, several Maven plug-ins handle different //tasks// (or //goals//, as Maven calls them). The //maven-compile-plugin// for example compiles {{code language="none"}}.java{{/code}} files into {{code language="none"}}.class{{/code}} files.
20
21 To correctly compile a project, Maven needs to be told about the project. While the popular Ant build tool uses {{code language="none"}}build.xml{{/code}} files to describe the steps to be executed for building a project, Maven uses {{code language="none"}}pom.xml{{/code}} files to describe the project and figures out the steps for itself. The POM files may inherit settings from a parent POM file.
22
23 Tycho is a set of Maven plugins that handles compiling and dependency management as well as bundling of Eclipse plug-ins. Tycho understands Eclipse metadata files such as {{code language="none"}}plugin.xml{{/code}} or {{code language="none"}}feature.xml{{/code}}, provides dependency information extracted from those files, and provides an Eclipse instance for compiling and packaging Eclipse bundles.
24
cds 19.1 25 === Apache Ant ===
cds 16.1 26
27 Ant is a very popular Java build tool. While Maven wants to know metadata about a project and then knows what to do to build it, Ant works by specifying exactly what to do to build a project. These steps are configured in a {{code language="none"}}build.xml{{/code}} file. We try to avoid using Ant, but still have Ant build files around for jobs too specialized to be properly handled by Maven.
28
cds 19.1 29 === Bamboo ===
cds 11.1 30
31 While Maven and Tycho know how to compile KIELER, Bamboo knows when to compile KIELER and what to do with the compiled project. Bamboo has access to our source code repositories and triggers continuous integration builds every time someone pushes new code into a repository. It also does a full build every night and copies the results onto our nightly build update site to be accessed by people all around the world. And beyond. Tell your friends!
32
cds 19.1 33 == The Automatic Build Process ==
cds 11.1 34
cds 15.1 35 This section describes how our POM files are distributed throughout the repository structure, and how you can trigger an automatic build of KIELER.
cds 11.1 36
cds 19.1 37 === POM Files ===
tig 3.1 38
cds 15.1 39 The basic structure of the POM files can be seen below:
40
41 [[image:attach:pomfiles.png]]
42
43 Each plug-in and feature has a corresponding (usually rather small) POM file. The POM files in the {{code language="none"}}features{{/code}} and {{code language="none"}}plugins{{/code}} directories know about the different features and plug-ins. The parent POM file, which all other POM files copy basic configuration from, knows about the feature and plug-in POM files. In addition, the {{code language="none"}}build{{/code}} directory also contains a bunch of subdirectories. These also contain POM files specialized for building the P2 repositories necessary to publish our Eclipse features.
44
cds 19.1 45 === Using the KIELER Maven Build ===
cds 15.1 46
47 Using the KIELER Maven build requires knowledge about two aspects: necessary configuration / required libraries and invoking Maven.
48
cds 19.1 49 ==== Necessary Configuration ====
cds 15.1 50
51 KIELER needs to be built against a P2 repository generated from our Eclipse reference installation. The path to this repository is found in the parent POM file and must be changed if the repository location changes. Also, when changing the reference installation, the repository has to be updated using the following magic command (adapted accordingly):
52
53 {{code language="none"}}
54 java -jar eclipse_3.8/plugins/org.eclipse.equinox.launcher_*.jar \
tig 7.1 55 -application org.eclipse.equinox.p2.publisher.FeaturesAndBundlesPublisher \
56 -metadataRepository file:/home/java/repository/juno382 \
57 -artifactRepository file:/home/java/repository/juno382 \
58 -source /home/java/eclipse_3.8/ \
59 -publishArtifacts
cds 15.1 60 {{/code}}
tig 6.1 61
cds 19.1 62 ==== Invoking Maven ====
tig 6.1 63
cds 15.1 64 To actually build KIELER once all preliminaries are done, navigate to the {{code language="none"}}/build/de.cau.cs.kieler.parent{{/code}} directory and run the following command line:
65
66 {{code language="none"}}
cds 19.1 67 . /home/java/java-env # Only necessary when working on our servers
cds 15.1 68 mvn clean package -P <profile> # For a list of available profiles, see parent pom.xml
tig 6.1 69 {{/code}}
70
cds 15.1 71 Once Maven has finished, the different build artifacts may be found in the {{code language="none"}}/build/de.cau.cs.kieler.*.repository/target{{/code}} directories.
tig 9.1 72
cds 19.1 73 === What to Be Aware of ===
tig 3.1 74
cds 15.1 75 There are some things that people need to be aware of to keep the build files in a valid state.
tig 3.1 76
cds 15.1 77 1. Eclipse metadata and POM files are not synchronized. Thus, if the version of an Eclipse plug-in changes, its {{code language="none"}}pom.xml{{/code}} needs to be updated accordingly.
78 1. The repository POM directories contain product files and product icons. These are copies of the files found in the corresponding branding plug-ing (such as {{code language="none"}}de.cau.cs.kieler.core.product{{/code}}) and have to be manually synchronized.
tig 3.1 79
cds 19.1 80 === The Curious Case of Building the KLay Library ===
cds 16.1 81
82 We distribute our KLay layout algorithms in two library files: one that contains just our algorithms, and another one that also contains dependencies such as required EMF classes. To build the KLay libraries, navigate to the {{code language="none"}}/build/de.cau.cs.kieler.klay.libraries{{/code}} directory and do the following:
83
84 1. Set the {{code language="none"}}PRODUCT{{/code}} environment variable to point to a complete installation of KIELER. You can get one by using the automatic Maven build.
85 1. Run Ant on the {{code language="none"}}build.xml{{/code}} file. This will produce both versions of the KLay layout library.
86
cds 19.1 87 == Continuous and Nightly Builds ==
tig 3.1 88
cds 12.1 89 There are basically three different build plans for each of the KIELER projects:
tig 3.1 90
cds 13.1 91 1. **Continuous Plugins** – Compiles the plug-ins and runs the unit tests on them. None of the compiled artifacts are published anywhere. This plan is triggered by pushing stuff into the repositories, giving early feedback regarding whether committed changes break anything.
92 1. **Nightly Product** – Compiles the plug-ins and assembles distributable product files and update sites. Distributable files are published in the nightly build directory {{code language="none"}}/home/kieler/public_html/files/nightly{{/code}}. Update sites are published in {{code language="none"}}/home/kieler/public_html/updatesite/nightly{{/code}}. This plan is run once every night.
93 1. **Nightly Rating** – Compiles the plug-ins and runs our code quality rating doclet on them. The result is a website published at {{code language="none"}}/home/kieler/public_html/rating{{/code}}. This plan is run once every night.
cds 19.1 94 1. **Release Builds** – Continuous builds of release branches once a release is imminent. These usually run whenever changes are pushed into the repository.
tig 3.1 95
cds 19.1 96 The //Semantics// project has an additional build plan:
cds 14.1 97
cds 19.1 98 1. **Ptolemy Update Site** – Builds the Ptolemy library and produces an update site for it. The build is triggered whenever changes are pushed into the Ptolemy repository. This build plan is necessary since both, the //Pragmatics// and the //Semantics// projects use the Ptolemy libraries.
cds 14.1 99
cds 19.1 100 == Update Sites and Redistributables ==
101
102 Our automatic builds produce a bunch of so-called //artifacts//: redistributable applications as well as a number of update sites. This table lists all artifacts, the project or repository they belong to, the build file responsible for producing them, the Bamboo build plan that builds them, and the directory they are finally placed in.
103
104 |=(((
105 Artifact
106 )))|=(((
107 Repository
108 )))|=(((
109 Build File
110 )))|=(((
111 Bamboo Build Plan
112 )))|=(% colspan="1" %)(% colspan="1" %)
113 (((
114 Final Directory
115 )))
116 |(((
117 KLay Layout Libraries
118 )))|(((
119 Pragmatics
120 )))|(((
121 {{code language="none"}}
122 ...klay.libraries/build.xml
123 {{/code}}
124 )))|(((
125 KIELER Pragmatics -> Nightly Product
126 )))|(% colspan="1" %)(% colspan="1" %)
127 (((
128 {{code language="none"}}
129 /home/kieler/public_html/files/nightly/
130 {{/code}}
131 )))
132 |(((
133 KWebS RCA
134 )))|(((
135 Pragmatics
136 )))|(((
137 {{code language="none"}}
138 ...kwebs.repository/pom.xml
139 {{/code}}
140 )))|(((
141 KIELER Pragmatics -> Nightly Product
142 )))|(% colspan="1" %)(% colspan="1" %)
143 (((
144 {{code language="none"}}
145 /home/kieler/public_html/files/nightly/
146 {{/code}}
147 )))
148 |(((
149 KIELER Pragmatics Update Site
150 )))|(((
151 Pragmatics
152 )))|(((
153 {{code language="none"}}
154 ...pragmatics.repository/pom.xml
155 {{/code}}
156 )))|(((
157 KIELER Pragmatics -> Nightly Product
158 )))|(% colspan="1" %)(% colspan="1" %)
159 (((
160 {{code language="none"}}
161 /home/kieler/public_html/updatesite/nightly/pragmatics/
162 {{/code}}
163 )))
164 |(((
165 Papyrus Layout Update Site
166 )))|(((
167 Pragmatics
168 )))|(((
169 {{code language="none"}}
170 ...papyrus.repository/pom.xml
171 {{/code}}
172 )))|(((
173 KIELER Pragmatics -> Nightly Product
174 )))|(% colspan="1" %)(% colspan="1" %)
175 (((
176 {{code language="none"}}
177 /home/kieler/public_html/updatesite/nightly-papyrus/
178 {{/code}}
179 )))
180 |(((
181 KIELER RCA
182 )))|(((
183 Semantics
184 )))|(((
185 {{code language="none"}}
186 ...semantics.repository/pom.xml
187 {{/code}}
188 )))|(((
189 KIELER Semantics -> Nightly Product
190 )))|(% colspan="1" %)(% colspan="1" %)
191 (((
192 {{code language="none"}}
193 /home/kieler/public_html/files/nightly/
194 {{/code}}
195 )))
196 |(((
197 KIELER Semantics Update Site
198 )))|(((
199 Semantics
200 )))|(((
201 {{code language="none"}}
202 ...semantics.repository/pom.xml
203 {{/code}}
204 )))|(((
205 KIELER Semantics -> Nightly Product
206 )))|(% colspan="1" %)(% colspan="1" %)
207 (((
208 {{code language="none"}}
209 /home/kieler/public_html/updatesite/nightly/semantics/
210 {{/code}}
211 )))
212 |(((
213 Ptolemy Libraries Update Site
214 )))|(((
215 Ptolemy
216 )))|(((
217 {{code language="none"}}
218 ...ptolemy.repository/pom.xml
219 {{/code}}
220 )))|(((
221 KIELER Semantics -> Ptolemy Update Site
222 )))|(% colspan="1" %)(% colspan="1" %)
223 (((
224 {{code language="none"}}
225 /home/kieler/public_html/updatesite/ptolemy/
226 {{/code}}
227 )))

Navigation

Quick Links

KIELER on GitHub
RTSYS Homepage