From hours to minutes: GlassFish pool for Jakarta EE TCKs

Jakarta EE TCKs are notoriously slow. The bulk of the wall clock is not test execution but GlassFish cold-start: every test module unpacks a dist, boots a domain, deploys, undeploys, and stops the domain. With one hundred-ish test modules and several seconds of start/stop per module, a full TCK run easily takes hours. The arquillian-glassfish-server-pool module and the glassfish-pool-maven-plugin, both released as part of OmniFish arquillian-container-glassfish 2.2.0, eliminate that overhead by sharing a pool of pre-started GlassFish instances across the entire reactor.

The proof of concept: Faces TCK

The Jakarta Faces TCK historically consisted of two parts. The "old TCK" was the original Oracle suite: an Ant-driven JavaTest harness inherited from the JSF 1.x days, with around 5000 tests. The "new TCK" was the body of contributed and later-added tests built on JUnit + Arquillian, optionally with HtmlUnit or Selenium for browser interaction. Running both ends to end easily took over 3 hours on Jenkins CI, dominated by the old TCK.

Folding the old TCK into the new-TCK style was always the goal, but per-test manual conversion was prohibitively cumbersome; AI-assisted development is what finally made it feasible. Pull requests #2146, #2147, #2149, and #2150 mechanically migrated the entire old TCK, with Claude Code doing the bulk of the rewriting and consolidating the remaining HtmlUnit assertions onto Selenium along the way. WAR consolidation (one WAR per feature group instead of one per test) brought wall clock down to ~1h.

The second step was an in-house gf-pool prototype (#2156) that pre-started a pool of GlassFish instances and leased one slot per failsafe-forked JVM. With mvn clean verify -T8 (8 threads), the full Faces TCK reactor now finishes in under 4 minutes. The prototype proved the model works; the natural next step was extracting it into a reusable module so other TCKs do not have to copy-paste the wiring.

Phase	Linux	MacBook	Jenkins
Pre-migration	02:57 h	02:53 h	03:19 h
Post-migration	01:05 h	40:52 m	01:18 h
With gf-pool	3:46 m (-T8)	4:47 m (-T5)	13:06 m (-T2)

Linux: Intel Core i9-10900X with 32GB
MacBook: M1 Pro 10 Core with 16GB
Jenkins: Eclipse Jiro with "2 CPU" and "8 GB"

What gf-pool is

The pool ships in two artifacts:

• arquillian-glassfish-server-pool: a runtime Arquillian DeployableContainer that leases a slot for the duration of a test JVM and deploys against the leased slot's DAS through the standard CommonGlassFishManager.
• glassfish-pool-maven-plugin: lifecycle goals (up, down, provision, status, nuke) that provision and start slots before integration-test and stop them after.

Provisioning clones a single source GlassFish install into slot-1/, slot-2/, …, rewrites each slot's domain.xml so its ports land in a non-overlapping window (adminBase + (slot - 1) * portStride), and starts every slot in parallel. Each test JVM acquires an exclusive FileChannel.tryLock() on slot-N/lock, reads slot-N/ports.properties, and holds the lock for the JVM's lifetime. The lease protocol is pure Java; there's no -javaagent, no surefire argLine plumbing, and no shell scripts.

The pool grows on demand. A sequential build uses one slot; mvn clean verify -T4 grows to four; -T8 grows to eight. A JVM shutdown hook installed on Maven's own JVM stops every slot at session end (or on Ctrl+C), so no orphaned processes survive a hard build failure.

The optimal -TN for your machine is bounded by available RAM, not by core count. Each slot is a full GlassFish JVM plus a failsafe-forked test JVM, so the dominant cost is heap and resident memory, not CPU. Moreover, if your TCK drives a browser (as Faces does), each slot also spawns its own Chrome plus chromedriver, which pushes the total to ~1.5GB per slot. A 16-core box with 16GB will usually thrash at -T8 while a 8-core box with 32GB happily handles -T8; pick N by watching resident memory and swap, not number of processors.

Maven setup

Two plugin blocks: run the pool plugin (which resolves and unpacks GlassFish itself), and point failsafe at the same <poolDir>.

<build>
    <plugins>
        <plugin>
            <groupId>ee.omnifish.arquillian</groupId>
            <artifactId>glassfish-pool-maven-plugin</artifactId>
            <version>2.2.0</version>
            <configuration>
                <poolDir>${project.build.directory}/pool</poolDir>
                <poolSource>${project.build.directory}/dist/glassfish9</poolSource>
                <distribution>
                    <groupId>org.glassfish.main.distributions</groupId>
                    <artifactId>glassfish</artifactId>
                    <version>9.0.0-M2</version>
                    <type>zip</type>
                </distribution>
            </configuration>
            <executions>
                <execution><id>pool-up</id><goals><goal>up</goal></goals></execution>
                <execution><id>pool-down</id><goals><goal>down</goal></goals></execution>
            </executions>
        </plugin>

        <plugin>
            <artifactId>maven-failsafe-plugin</artifactId>
            <configuration>
                <systemPropertyVariables>
                    <gf.pool.dir>${project.build.directory}/pool</gf.pool.dir>
                    <gf.pool.source>${project.build.directory}/dist/glassfish9</gf.pool.source>
                </systemPropertyVariables>
            </configuration>
        </plugin>
    </plugins>
</build>

The <distribution> block tells the plugin to resolve the named artifact through your usual Maven repositories and unpack it under ${project.build.directory}/dist before provisioning runs. Staging is idempotent: re-runs fast-exit when the marker file written after a successful unpack is still present.

Add the runtime as a test-scope dependency, and an arquillian.xml that names the glassfish-pool qualifier:

<dependency>
    <groupId>ee.omnifish.arquillian</groupId>
    <artifactId>arquillian-glassfish-server-pool</artifactId>
    <version>2.2.0</version>
    <scope>test</scope>
</dependency>

<arquillian xmlns="http://jboss.org/schema/arquillian">
    <container qualifier="glassfish-pool" default="true">
        <configuration>
            <property name="poolDir">${project.build.directory}/pool</property>
        </configuration>
    </container>
</arquillian>

That's it. mvn clean verify works sequentially; mvn clean verify -T8 fans out across eight slots.

Need to peek at the pool? mvn glassfish-pool:status in a separate terminal redraws a top-style table once per second:

Overlays

For TCK-style builds that test a SNAPSHOT impl jar against a released distribution (or vice versa), the plugin can copy overlay jars into glassfish/modules/ after unpack and before slot cloning. Declare zero or more <overlay> blocks:

<configuration>
    <overlays>
        <overlay>
            <groupId>org.glassfish</groupId>
            <artifactId>jakarta.faces</artifactId>
            <version>5.0.0-SNAPSHOT</version>
            <destFileName>mojarra.jar</destFileName>
        </overlay>
    </overlays>
</configuration>

Each overlay accepts a <skip> child that wires to your existing per-profile property switches, so you can hold back individual jars per release line without forking the pom.

Bring your own unpack

If you'd rather use maven-dependency-plugin for the unpack (e.g. because your build already has one for unrelated reasons), drop the <distribution> block from the pool plugin and add a regular unpack execution that lands in the same directory <poolSource> points at:

<plugin>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>unpack-glassfish</id>
            <phase>process-test-classes</phase>
            <goals><goal>unpack</goal></goals>
            <configuration>
                <artifactItems>
                    <artifactItem>
                        <groupId>org.glassfish.main.distributions</groupId>
                        <artifactId>glassfish</artifactId>
                        <version>9.0.0-M2</version>
                        <type>zip</type>
                        <outputDirectory>${project.build.directory}/dist</outputDirectory>
                    </artifactItem>
                </artifactItems>
            </configuration>
        </execution>
    </executions>
</plugin>

The pool plugin will then skip staging and clone slots directly from ${project.build.directory}/dist/glassfish9.

JVM system properties at slot boot

Some properties have to be on the GF JVM at startup. The canonical example is javax.net.ssl.trustStorePassword: a PKCS12 truststore needs the password before SSL is used for the first time, because Java caches the default SSLContext after first use and never reloads from disk. The plugin's <systemProperties> hook bakes each key=value into every <java-config> of each slot's domain.xml at provisioning time:

<configuration>
    <systemProperties>
        javax.net.ssl.trustStorePassword=changeit
        java.awt.headless=true
    </systemProperties>
</configuration>

Adoption: Security TCK

The Jakarta Security TCK followed the same old-TCK / new-TCK split as Faces: a JavaTest "old TCK" from Oracle plus a JUnit + Arquillian "new TCK" of later contributions. The combined suite ran in just under 13 minutes. Unlike Faces, the old-TCK side wasn't the bottleneck: it deployed its apps onto a single long-running GlassFish domain and replayed all 83 of its JavaTest clients against them, so it was already efficient on its own. Migrating it was therefore not about runtime; it was about consolidating on a single test harness. Pull request #365 did exactly that, mechanically rewriting the old TCK into the new-TCK style with Claude Code doing the bulk of the assertion work. The unavoidable trade-off is that each migrated test now spins up its own Arquillian-managed GlassFish instead of sharing one domain, so single-threaded runtime nearly doubled to ~24 minutes. That is the cost of trading a shared harness for per-test isolation. Pull request #368 recovers that cost (and then some) by swapping the JVM-scoped arquillian-glassfish-server-managed container for arquillian-glassfish-server-pool. With mvn clean verify -T8, the Security TCK now finishes in under 2 minutes.

Per-test isolation is a much bigger deal for Faces, whose old TCK has ~5000 tests and takes 2+ hours on a single shared GlassFish; there the post-migration single-threaded runtime would be prohibitive without the pool. Security is the small-scale case where you can see the trade clearly; Faces is where parallelism stops being optional.

Phase	Linux	MacBook	Jenkins
Pre-migration	12:50 m	9:38 m	19:05 m
Post-migration	23:42 m	16:34 m	—
With gf-pool	1:49 m (-T8)	1:30 m (-T5)	—

Linux: Intel Core i9-10900X with 32GB
MacBook: M1 Pro 10 Core with 16GB
Jenkins: Eclipse Jiro with "2 CPU" and "8 GB"

Parallelism: what you'll discover

Moving to a shared pool surfaces parallelism issues that a sequential build hides. The Security TCK migration is a good cross-section. None of these are pool bugs; they're latent contracts that finally see daylight when two slots run side by side.

Hardcoded ports. Tests that embed an LDAP server, a Tomcat instance, or any other side-process on a fixed port collide as soon as two slots co-run. Pick distinct ports per module, or derive them from the slot index (gf.pool.slot is published as a system property by the leaser). The Security TCK's embedded LDAP modules split 33389 onto 33390 and 33391 for its two extra app-ldap variants.

Hardcoded URLs. Tests that publish http://localhost:8080/... URLs to an external party (OAuth callback URIs, OIDC issuer metadata) break the moment the slot's HTTP port is anything but 8080. Replace literals with UriInfo-derived or request-derived URLs at runtime, so the URL tracks the slot's actual HTTP port.

Pre-registered redirect URIs for every slot. External identity providers that need redirect URIs pre-registered (Mitre OIDC in the Security TCK case) have to be told about every slot the pool may grow to. Maven exposes ${session.request.degreeOfConcurrency} as the -TN value; bsh-property can promote it to a regular property if your plugin only consumes typed properties.

Cross-app singletons. A java:global/ DataSource shared across apps, or any other JNDI/CDI/resource a previous app deploy leaves behind, can leak into the next app's lookup on the same slot. The Security TCK adoption uncovered a related upstream GlassFish bug where ComponentEnvManagerImpl.getResourceId returned an empty string for ScopeType.GLOBAL, fixed in eclipse-ee4j/glassfish#26029. Worth re-running your TCK against this fix if you exercise cross-app global resources.

Persisted state across re-runs. Anything written to work/, sessions/, or other on-disk caches survives a slot's lease release and can resurrect into the next consumer. If your test relies on a known starting state, wipe the relevant directories on container start.

Aggregator goals on the reactor root. Goals like failsafe-report-only, cyclonedx:makeAggregateBom, or install-file bound to a per-module phase stall the -T reactor because Maven serialises them across modules. Move these to inherited=false on the reactor root only.

For Jakarta EE TCK maintainers

If your Jakarta EE TCK still drags a JavaTest "old TCK" alongside its JUnit + Arquillian "new TCK", or runs sequentially against a freshly-unpacked GlassFish per module, the migration path is the one Faces and Security walked. First, fold any remaining old TCK into the new-TCK style (JUnit + Arquillian, plus Selenium if your tests drive a browser, as Faces does); AI-assisted development handles the mechanical rewriting well enough that the bulk of the work is reviewing diffs, not writing them. Then wire arquillian-glassfish-server-pool and glassfish-pool-maven-plugin in. The result is a TCK that finishes in minutes instead of hours, and a build that still runs sequentially under mvn clean verify for vendors who prefer that.

The README at glassfish-pool-maven-plugin/README.md documents the full configuration surface; the working example at integration-tests/src/it/pool is ~150 lines of pom and runs as a smoke test in the project's own CI. Both Faces TCK (faces#2165) and Security TCK (security#368) are open and worth studying as real-world consumers.