A dive into how Metals works

Chris Kipp

A little about me

  • Software engineer at Lunatech
  • Big Neovim fan
  • Enjoy working on tooling
  • One of the current Metals maintainers and author of the Metals Nvim/Vim extensions

@ckipp01

https://www.chris-kipp.io/slides/slides-a-dive-into-how-metals-works

Scalameta

Vadim Chelyshov

Gabriele Petronella

Olafur Geirsson

Tomasz Godzik

Plus so many more

What's so confusing?

Metals

LSP

BSP

DAP

SemanticDB

dotc

Bloop

Neovim

What is Metals?

  • Scala language server with rich IDE features

  • Provides language features like auto complete, go to definition, find all references etc.

  • Work with any editor that knows LSP... more or less
  • Scalameta + Language Server --> Metals

What is LSP?

The matrix problem

 

LSP

The idea behind the Language Server Protocol (LSP) is to standardize the protocol for how tools and servers communicate, so a single Language Server can be re-used in multiple development tools, and tools can support languages with minimal effort.

https://microsoft.github.io/language-server-protocol/

LSP

 

https://microsoft.github.io/language-server-protocol/overview

LSP

 

[Trace - 11:53:10 AM] Sending notification 'window/showMessage'
Params: {
  "type": 4,
  "message": "Compiling root 2s"
}


[Trace - 11:53:10 AM] Sending notification 'textDocument/publishDiagnostics'
Params: {
  "uri": "file:///.../test-project/src/main/scala/example/Other.scala",
  "diagnostics": []
}


[Trace - 11:53:10 AM] Sending notification 'window/logMessage'
Params: {
  "type": 4,
  "message": "INFO  time: compiled root in 2.67s"
}

LSP

 

LSP Client

Neovim

LSP Server

Metals

LSP

RPC-JSON

Communication

Cool, so Metals then compiles my code?

No

The matrix problem... 

again

 

Build Server Protocol

 

Protocol for IDEs and build tools to communicate about compile, run, test, debug and more.

https://build-server-protocol.github.io/

BSP

 

BSP Client

Metals

BSP Server

Bloop

BSP

RPC-JSON

Communication

BSP

[Trace - 03:17:38 PM] Sending request 'buildTarget/scalaTestClasses - (10)'
Params: {
  "targets": [
    {
      "uri": "file:/Users/ckipp/Documents/scala-workspace/sanity/project/?id\u003dsanity-build"
    }
  ]
}


[Trace - 03:17:38 PM] Received response 'buildTarget/scalaMainClasses - (9)' in 5ms
Result: {
  "items": [
    {
      "target": {
        "uri": "file:/Users/ckipp/Documents/scala-workspace/sanity/project/?id\u003dsanity-build"
      },
      "classes": []
    }
  ]
}

What's so confusing?

LSP

BSP

Wait, but how does

<enter build server here> 

actually know about my Build?

Situation 1

❯ sbt bspConfig
[info] welcome to sbt 1.5.5 (GraalVM Community Java 11.0.11)
[info] loading global plugins from /Users/ckipp/.sbt/1.0/plugins
[info] loading settings for project sanity-build from plugins.sbt ...
[info] loading project definition from /Users/ckipp/Documents/scala-workspace/sanity/project
[info] loading settings for project sanity from build.sbt ...
[info] set current project to sanity (in build file:/Users/ckipp/Documents/scala-workspace/sanity/)
[success] Total time: 0 s, completed 18 Jul 2021, 14:41:15

~/Documents/scala-workspace/sanity 7s
❯ bat .bsp/sbt.json
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: .bsp/sbt.json
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ {"name":"sbt","version":"1.5.5","bspVersion":"2.0.0-M5","languages":["scala"],"argv":["/Users/ckipp/.sdkman/candidates/java/21.1.0.r11-grl/bin
       │ /java","-Xms100m","-Xmx100m","-classpath","/Users/ckipp/Library/Application Support/Coursier/bin/sbt:/Users/ckipp/Library/Caches/Coursier/v1/h
       │ ttps/repo1.maven.org/maven2/io/get-coursier/sbt/sbt-runner/0.2.0/sbt-runner-0.2.0.jar:/Users/ckipp/Library/Caches/Coursier/v1/https/repo1.mave
       │ n.org/maven2/org/scala-sbt/sbt-launch/1.5.5/sbt-launch-1.5.5.jar","xsbt.boot.Boot","-bsp","--sbt-launch-jar=/Users/ckipp/Library/Caches/Coursi
       │ er/v1/https/repo1.maven.org/maven2/org/scala-sbt/sbt-launch/1.5.5/sbt-launch-1.5.5.jar"]}

Your build tool can be your build server

Situation 2

Bloop is your build server using Build Export

❯ ./mill mill.bsp.BSP/install

Build definition

Bloop definition

What's so confusing?

LSP

BSP

Build Definition

Situation 1

What's so confusing?

LSP

BSP

Build Definition

Situation 2

Build Export

How does Metals then do

everything else?

The Scala Presentation Compiler

Version of the compiler that is specifically designed for IDE-like tooling that is:

  • Asynchronous
  • Interruptible at every point
  • Can do targeted type-checking
  • Can stop after a specific point (type-checking) and provide a partial result.

https://www.chris-kipp.io/blog/an-intro-to-the-scala-presentation-compiler

The Scala Presentation Compiler

  • Completions
  • Signature help
  • Hover information
  • Backup definition provider (in times when it's not found in SemanticDB)
  • Insert inferred type
  • Selection ranges

https://github.com/scalameta/metals/blob/main/mtags-interfaces/src/main/java/scala/meta/pc/PresentationCompiler.java

SemanticDB

SemanticDB is a data model for semantic information such as symbols and types about programs in Scala and other languages. SemanticDB decouples production and consumption of semantic information, establishing documented means for communication between tools.

https://scalameta.org/docs/semanticdb/guide.html

SemanticDB

~/Documents/scala-workspace/semanticdb-example
❯ tree -L 4
.
├── build.sbt
├── project
│  ├── build.properties
│  └── metals.sbt
│  
└── src
  └── main
     └── scala
        └── Main.scala

An example app

SemanticDB

❯ pwd
semanticdb-example/.bloop/semanticdb-example/bloop-bsp-clients-classes/classes-Metals-ZrQOg_QhQGq6uyC-zEqr8A==/META-INF/semanticdb/src/main/scala

~/D/s/s/.b/s/b/c/M/s/s/m/scala
❯ metap Main.scala.semanticdb
src/main/scala/Main.scala
-------------------------

Summary:
Schema => SemanticDB v4
Uri => src/main/scala/Main.scala
Text => empty
Language => Scala
Symbols => 1 entries
Occurrences => 4 entries

Symbols:
_empty_/Main. => final object Main extends AnyRef with App

Occurrences:
[0:7..0:11) <= _empty_/Main.
[0:20..0:23) => scala/App#
[0:24..0:24) => java/lang/Object#`<init>`().
[1:2..1:9) => scala/Predef.println(+1).

An example app

SemanticDB

Wait, but then how does that work to navigate around external sources?

SemanticDB

Interactive SemanticDB

SemanticDB that is produced on-demand using the Presentation Compiler

What's so confusing?

LSP

BSP

Build Definition

Build Export

SemanticDB

(on disk)

Presentation

Compiler

SemanticDB

(in memory)

Ok. But what about Debugging?

Debug Adapter Protocol

The Debug Adapter Protocol (DAP) defines the abstract protocol used between a development tool (e.g. IDE or editor) and a debugger.

https://microsoft.github.io/debug-adapter-protocol/

DAP

workspace/executeCommand
  metals.debug-adapter-start

workspace/executeCommand
  "name": "example.Main",
  "uri": "tcp://127.0.0.1:63343"

Debuggee

Debugger

scalacenter/scala-debug-adapter

What's so confusing?

LSP

BSP

Build Definition

Build Export

SemanticDB

(on disk)

Presentation

Compiler

SemanticDB

(in memory)

DAP

Some bonus things.

Because we need more protocols and more arrows

What about worksheets?

LSP

mdoc

  • There is no BSP communication for this
  • mdoc is just used as a library
  • The Scala version is dependant on where your worksheet is located
  • What you have access to is dependant on where your worksheet is located

https://scalameta.org/mdoc/

What about Ammonite?

LSP

  • Ammonite is the BSP server

  • Metals uses alexarchambault/ammonite-runner

  • The Scala version is dependant on where your worksheet is located or a comment on the top of your file

Ammonite

BSP

What about Formatting?

LSP

  • Still reads from your .scalafmt.conf file

  • Metals uses scalafmt as a library

  • There are some extra goodies sprinkled in if you're using Scala 3

Scalafmt

What about Organize Imports?

LSP

  • Still reads from your .scalafix.conf file

  • Uses liancheng/scalafix-organize-imports

  • Probably more Scalafix integrations coming in the future

Scalafix

What's TVP?

Tree View Protocol

  • Protocol for Metals to show various tree views
  • Navigate external libraries and APIs
  • Jump to places in your code base from the tree and vice versa
  • Links to creating an issue or doing a clean compile
  • Compilation information

Questions???