Code Coverage in Scala 3

ScalaCon 2021

Chris Kipp

@ckipp01

A little about me

Switched careers into programming
I enjoy simple tech
Big Neovim fan
You'll find me working in Scala Tooling
Host the Tooling Talks Podcast
Software engineer at Lunatech
chris-kipp.io/talks

scoverage

Code coverage tool for Scala
Started in 2010 by Mikko Koponen
Then maintained by Stephen Samuel and Grzegorz Slowikowski for years
Over 1.5 million dowloads a month
sbt, Mill, Gradle, and Maven

How do you fit into the picture, Chris?

I write Scala, and needed code coverage...

So I decided to adopt it.

How does it even work?

Compiler plugin (in Scala 2)

sbt:scoverage-test> show Compile / compile / scalacOptions
[info] Compile / compile / scalacOptions
[info]  List(
[info]    -Xplugin:/my/path/to-plugin/2.0.0-M4/scalac-scoverage-plugin_2.13.6-2.0.0-M4.jar,
[info]    -P:scoverage:dataDir:/path-to-my-project/target/scala-2.13/scoverage-data,
[info]    -P:scoverage:sourceRoot:/path-to-my-project/scoverage-test,
[info]    -P:scoverage:reportTestName
[info]  )

Ok, so what does it do

 def isTwo(num: Int): Boolean =
  if (num == 2) {
    true
  } else {
    false
  }

We'll illustrate with a very simple piece of code

Compiler plugin (Scala 2)

case i: If =>
  treeCopy.If(
    i,
    process(i.cond),
    instrument(process(i.thenp), i.thenp, branch = true),
    instrument(process(i.elsep), i.elsep, branch = true)
  )

The tree would be instrumented here

  def isTwo(num: Int): Boolean = if ({
    scoverage.Invoker.invoked(
      1,
      "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
      scoverage.Invoker.invoked$default$3()
    );
    num.==(2)
  }) {
    scoverage.Invoker.invoked(
      3,
      "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
      scoverage.Invoker.invoked$default$3()
    );
    {
      scoverage.Invoker.invoked(
        2,
        "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
        scoverage.Invoker.invoked$default$3()
      );
      true
    }
  } else {
    scoverage.Invoker.invoked(
      5,
      "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
      scoverage.Invoker.invoked$default$3()
    );
    {
      scoverage.Invoker.invoked(
        4,
        "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
        scoverage.Invoker.invoked$default$3()
      );
      false
    }
  };

scoverage.coverage

1                                                       <-- id
a/src/main/scala/mypackage/BasicControlStructures.scala <-- source path
a                           <-- package name
BasicControlStructures      <-- class name
Object                      <-- class type
a.BasicControlStructures    <-- full class name
isTwo                       <-- method name
74                          <-- start offset
79                          <-- end offset
5                           <-- line number
scala.Int.==                <-- symbol name
Apply                       <-- tree name
false                       <-- is branch
0                           <-- invocations count
false                       <-- is ignored
num.==(2)                   <-- sign

/a/target/scala-2.13/scoverage-data

What's the Invoker?

Writes the ids that call out to it
Keeps track of how many times they are called
Adds the test name where it was triggered

scoverage.measurements

1 a.basiccontrolstructuresspec
9 a.basiccontrolstructuresspec
4 a.basiccontrolstructuresspec
8 a.basiccontrolstructuresspec
7 a.basiccontrolstructuresspec

a/target/scala-2.13/scoverage-data

Reports

Scala 3 coverage

You just explained it all, why not cross publish it and call it a day?

New compiler new rules
6 week release cycles are hard

lazy val bin212 =
  Seq(
    defaultScala212,
    "2.12.14",
    "2.12.13",
    "2.12.12",
    "2.12.11",
    "2.12.10",
    "2.12.9",
    "2.12.8"
  )
lazy val bin213 =
  Seq(
    defaultScala213,
    "2.13.6",
    "2.13.5",
    "2.13.4",
    "2.13.3",
    "2.13.2",
    "2.13.1",
    "2.13.0"
  )

Put it IN the compiler

No need to release every 6 weeks
Easily catch regressions
Have a friendly group of compiler devs looking it over
Plus Quentin Jaquier already has a working POC from like 4 years ago.
- https://github.com/Qjaquier/dotty-coverage-tools

So let's do this

Again, Quentin's work on this and the report he wrote is incredibly useful. Most of these examples come straight from Code coverage for Dotty.

http://guillaume.martres.me/code_coverage.pdf

Where do we put this?

❯ scala3-compiler -Xshow-phases
parser
typer
inlinedPositions
sbt-deps
extractSemanticDB
posttyper
prepjsinterop
sbt-api
SetRootTree
pickler
inlining
postInlining
staging
pickleQuotes
{firstTransform, checkReentrant, elimPackagePrefixes, cookComments, checkStatic, betaReduce, inlineVals, expandSAMs}
initChecker
{elimRepeated, protectedAccessors, extmethods, uncacheGivenAliases, byNameClosures, hoistSuperArgs, specializeApplyMethods, refchecks, tryCatchPatterns, patternMatcher}
{elimOpaque, explicitJSClasses, explicitOuter, explicitSelf, elimByName, stringInterpolatorOpt}
{pruneErasedDefs, uninitializedDefs, inlinePatterns, vcInlineMethods, seqLiterals, intercepted, getters, specializeFunctions, liftTry, collectNullableFields, elimOuterSelect, resolveSuper, functionXXLForwarders, paramForwarding, genericTuples, letOverApply, arrayConstructors}
erasure
{elimErasedValueType, pureStats, vcElideAllocations, arrayApply, addLocalJSFakeNews, elimPolyFunction, tailrec, completeJavaEnums, mixin, lazyVals, memoize, nonLocalReturns, capturedVars}
{constructors, instrumentation}
{lambdaLift, elimStaticThis, countOuterAccesses}
{dropOuterAccessors, checkNoSuperThis, flatten, renameLifted, transformWildcards, moveStatic, expandPrivate, restoreScopes, selectStatic, junitBootstrappers, Collect entry points, collectSuperCalls, repeatableAnnotations}
genSJSIR
genBCode

Where do we put this?

❯ scala3-compiler -Xshow-phases
...
SetRootTree
+ coverage
pickler
inlining
...

At the end of the front end phases
It's important to do this before things like dead code elimination or other optimizations that would remove or change code we want to instument

Different types of tree instrumentation

Fully Instrumented

These are things that can be fully instrumented as they are.

case tree: Literal => instrument(tree)
case tree: New => instrument(tree)
case tree: This => instrument(tree)
case tree: Super => instrument(tree)
case Select(qual, _) if (qual.symbol.exists && qual.symbol.is(JavaDefined)) =>
case tree: Select =>
  if (tree.qualifier.isInstanceOf[New]) {
    instrument(tree)
  } else {
    ...
  }

Not Instrumented

These are things that either their syntax don't allow it or they code may be removed during compilation

case tree: Import => tree
case tree: Ident if (isWildcardArg(tree)) => tree

Partially Instrumented

Cases where we don't want to instrument the entire tree

def instrumentCaseDef(tree: CaseDef)(using Context): CaseDef = {
  cpy.CaseDef(tree)(tree.pat, transform(tree.guard), transform(tree.body))
}

Some special considerations

object Main {
  def ourFunction(a: Int, b: Int, c: Int) = ???
}

Main.ourFunction(1, thisThrows(2), someOtherComputation())

Main.ourFunction(1, thisThrows(2), someOtherComputation())

Argument lifting

val x0 = Main
val x1 = 1
val x2 = thisThrows(2)
val x3 = someOtherComputation()

x0.ourFunction(x1, x2, x3)

Main.ourFunction(1, thisThrows(2), someOtherComputation())

Reusing some of the same logic from ETA expansion

But... be careful where you apply this

val x0 = doThing()

true || x0

This would show x0 as covered, when it shouldn't be

What can we re-use from scoverage?

Scala 2

Compiler Plugin

Invoker

Scala 2

Compiler Plugin

Domain

Serializer

Reporter

Scala 2

Compiler Plugin

Domain

(de)Serializer

Reporter

Scala 3

In the Compiler

Invoker

2.12, 2,13, 3

Scala Code Coverage

Unshared

Instrumenting your code
The invoker

Shared

Deserializing
Aggregation
Reporting

Demo

How to help

Test this out
https://github.com/lampepfl/dotty/pull/13880
https://github.com/ckipp01/scala3-example-project
Help out with maintenance
- Mill contrib module
- sbt-scoverage plugin
- Abandoned Maven plugin
- gradle-scoverage
- Abandoned examples repo

Code Coverage in Scala 3

A little about me

scoverage

How do you fit into the picture, Chris?

I write Scala, and needed code coverage...

So I decided to adopt it.

How does it even work?

Compiler plugin (in Scala 2)

Compiler plugin (in Scala 2)

Ok, so what does it do

Compiler plugin (Scala 2)

scoverage.coverage

What's the Invoker?

scoverage.measurements

Reports

Reports

Scala 3 coverage

You just explained it all, why not cross publish it and call it a day?

New compiler new rules

6 week release cycles are hard

Put it IN the compiler

So let's do this

Where do we put this?

Where do we put this?

Different types of tree instrumentation

Fully Instrumented

Not Instrumented

Partially Instrumented

Some special considerations

Argument lifting

But... be careful where you apply this

What can we re-use from scoverage?

Scala 2

Scala 2

Scala 2

Scala 3

Scala Code Coverage

Unshared

Shared

Demo

How to help