For general server architecture, see README.md. This module implements the watchdog process.

Watchdog state

Most LSP clients only send us file diffs, so to facilitate sending entire file contents to freshly restarted workers, the watchdog needs to maintain the current state of each file. It can also use this state to detect changes to the header and thus restart the corresponding worker, freeing its public imports.

TODO(WN): We may eventually want to keep track of approximately (since this isn't knowable exactly) where in the file a worker crashed. Then on restart, we tell said worker to only parse up to that point and query the user about how to proceed (continue OR allow the user to fix the bug and then continue OR ..). Without this, if the crash is deterministic, users may be confused about why the server seemingly stopped working for a single file.

Watchdog <-> worker communication

The watchdog process and its file worker processes communicate via LSP. If the necessity arises, we might add non-standard commands similarly based on JSON-RPC. Most requests and notifications are forwarded to the corresponding file worker process, with the exception of these notifications:

• textDocument/didOpen: Launch the file worker, create the associated watchdog state and launch a task to asynchronously receive LSP packets from the worker (e.g. request responses).
• textDocument/didChange: Update the local file state so that it can be resent to restarted workers. Then forward the didChange notification.
• textDocument/didClose: Signal a shutdown to the file worker and remove the associated watchdog state.

Moreover, we don't implement the full protocol at this level:

• Upon starting, the initialize request is forwarded to the worker, but it must not respond with its server capabilities. Consequently, the watchdog will not send an initialized notification to the worker.
• After initialize, the watchdog sends the corresponding didOpen notification with the full current state of the file. No additional didOpen notifications will be forwarded to the worker process.
• File workers are always terminated with an exit notification, without previously receiving a shutdown request. Similarly, they never receive a didClose notification.

Watchdog <-> client communication

The watchdog itself should implement the LSP standard as closely as possible. However we reserve the right to add non-standard extensions in case they're needed, for example to communicate tactic state.

def Lean.Server.Watchdog.workerCfg : IO.Process.StdioConfig

inductive Lean.Server.Watchdog.WorkerEvent : Type

Events that worker-specific tasks signal to the main thread.

• terminated : WorkerEvent
• importsChanged : WorkerEvent
• crashed (exitCode : UInt32) : WorkerEvent
• ioError (e : IO.Error) : WorkerEvent

inductive Lean.Server.Watchdog.WorkerState : Type

• crashed : WorkerState
• cannotWrite : WorkerState
• terminating : WorkerState
• running : WorkerState

def Lean.Server.Watchdog.parseParams (paramType : Type) [FromJson paramType] (params : Json) : IO paramType

structure Lean.Server.Watchdog.RequestQueueMap : Type

• i : Nat
• reqs : Std.TreeMap JsonRpc.RequestID (Nat × JsonRpc.Request Json) compare
• queue : Std.TreeMap Nat (JsonRpc.RequestID × JsonRpc.Request Json) compare

@[implicit_reducible]

instance Lean.Server.Watchdog.instInhabitedRequestQueueMap : Inhabited RequestQueueMap

def Lean.Server.Watchdog.instInhabitedRequestQueueMap.default : RequestQueueMap

def Lean.Server.Watchdog.RequestQueueMap.empty : RequestQueueMap

@[implicit_reducible]

instance Lean.Server.Watchdog.RequestQueueMap.instEmptyCollection : EmptyCollection RequestQueueMap

def Lean.Server.Watchdog.RequestQueueMap.enqueue (m : RequestQueueMap) (req : JsonRpc.Request Json) : RequestQueueMap

def Lean.Server.Watchdog.RequestQueueMap.erase (m : RequestQueueMap) (id : JsonRpc.RequestID) : RequestQueueMap

def Lean.Server.Watchdog.RequestQueueMap.get? (m : RequestQueueMap) (id : JsonRpc.RequestID) : Option (JsonRpc.Request Json)

def Lean.Server.Watchdog.RequestQueueMap.contains (m : RequestQueueMap) (id : JsonRpc.RequestID) : Bool

@[implicit_reducible]

instance Lean.Server.Watchdog.RequestQueueMap.instForInProdRequestIDRequestJsonOfMonad {m : Type u_1 → Type u_2} [Monad m] : ForIn m RequestQueueMap (JsonRpc.RequestID × JsonRpc.Request Json)

structure Lean.Server.Watchdog.RequestData : Type

• requestQueues : Std.TreeMap Lsp.DocumentUri RequestQueueMap compare
• uriByRequest : Std.TreeMap JsonRpc.RequestID Lsp.DocumentUri compare

def Lean.Server.Watchdog.instInhabitedRequestData.default : RequestData

@[implicit_reducible]

instance Lean.Server.Watchdog.instInhabitedRequestData : Inhabited RequestData

def Lean.Server.Watchdog.RequestData.empty : RequestData

def Lean.Server.Watchdog.RequestData.clearWorkerRequestData (d : RequestData) (uri : Lsp.DocumentUri) : RequestData

def Lean.Server.Watchdog.RequestData.enqueue (d : RequestData) (uri : Lsp.DocumentUri) (req : JsonRpc.Request Json) : RequestData

def Lean.Server.Watchdog.RequestData.erase (d : RequestData) (uri : Lsp.DocumentUri) (id : JsonRpc.RequestID) : RequestData

def Lean.Server.Watchdog.RequestData.contains (d : RequestData) (uri : Lsp.DocumentUri) (id : JsonRpc.RequestID) : Bool

def Lean.Server.Watchdog.RequestData.getUri? (d : RequestData) (id : JsonRpc.RequestID) : Option Lsp.DocumentUri

def Lean.Server.Watchdog.RequestData.getRequest? (d : RequestData) (uri : Lsp.DocumentUri) (id : JsonRpc.RequestID) : Option (JsonRpc.Request Json)

def Lean.Server.Watchdog.RequestData.getRequestQueue (d : RequestData) (uri : Lsp.DocumentUri) : RequestQueueMap

@[reducible, inline]

abbrev Lean.Server.Watchdog.RequestDataMutex : Type

def Lean.Server.Watchdog.RequestDataMutex.new : BaseIO RequestDataMutex

def Lean.Server.Watchdog.RequestDataMutex.clearWorkerRequestData (m : RequestDataMutex) (uri : Lsp.DocumentUri) : BaseIO Unit

def Lean.Server.Watchdog.RequestDataMutex.enqueue (m : RequestDataMutex) (uri : Lsp.DocumentUri) (req : JsonRpc.Request Json) : BaseIO Unit

def Lean.Server.Watchdog.RequestDataMutex.erase (m : RequestDataMutex) (uri : Lsp.DocumentUri) (id : JsonRpc.RequestID) : BaseIO Unit

def Lean.Server.Watchdog.RequestDataMutex.contains (m : RequestDataMutex) (uri : Lsp.DocumentUri) (id : JsonRpc.RequestID) : BaseIO Bool

def Lean.Server.Watchdog.RequestDataMutex.getUri? (m : RequestDataMutex) (id : JsonRpc.RequestID) : BaseIO (Option Lsp.DocumentUri)

def Lean.Server.Watchdog.RequestDataMutex.getRequestQueue (m : RequestDataMutex) (uri : Lsp.DocumentUri) : BaseIO RequestQueueMap

structure Lean.Server.Watchdog.FileWorker : Type

• doc : DocumentMeta
• proc : IO.Process.Child workerCfg
• exitCode : Std.Mutex (Option UInt32)
• commTask? : Option (ServerTask WorkerEvent)
• state : Std.Mutex WorkerState

def Lean.Server.Watchdog.FileWorker.stdin (fw : FileWorker) : IO.FS.Stream

def Lean.Server.Watchdog.FileWorker.stdout (fw : FileWorker) : IO.FS.Stream

def Lean.Server.Watchdog.FileWorker.waitForProc (fw : FileWorker) : IO UInt32

def Lean.Server.Watchdog.FileWorker.killProcAndWait (fw : FileWorker) : IO UInt32

@[reducible, inline]

abbrev Lean.Server.Watchdog.ImportMap : Type

structure Lean.Server.Watchdog.ImportData : Type

Global import data for all open files managed by this watchdog.

• imports : ImportMap

  For every open file, the files that it imports (transitively).

• importedBy : ImportMap

  For every open file, the open files that it is imported by (transitively).

def Lean.Server.Watchdog.ImportData.update (d : ImportData) (uri : Lsp.DocumentUri) (imports : Std.TreeSet Lsp.DocumentUri compare) : ImportData

Updates d with the new set of imports for the file uri.

def Lean.Server.Watchdog.ImportData.eraseImportsOf (d : ImportData) (uri : Lsp.DocumentUri) : ImportData

Sets the imports of uri in d to the empty set.

structure Lean.Server.Watchdog.RequestIDTranslation : Type

• sourceUri : Lsp.DocumentUri
• localID : JsonRpc.RequestID

@[reducible, inline]

abbrev Lean.Server.Watchdog.PendingServerRequestMap : Type

structure Lean.Server.Watchdog.ServerRequestData : Type

• pendingServerRequests : PendingServerRequestMap
• freshServerRequestID : Nat

def Lean.Server.Watchdog.ServerRequestData.trackOutboundRequest (data : ServerRequestData) (sourceUri : Lsp.DocumentUri) (localID : JsonRpc.RequestID) : JsonRpc.RequestID × ServerRequestData

def Lean.Server.Watchdog.ServerRequestData.translateInboundResponse (data : ServerRequestData) (globalID : JsonRpc.RequestID) : Option RequestIDTranslation × ServerRequestData

structure Lean.Server.Watchdog.WaitForILeanRequest : Type

• uri : Lsp.DocumentUri
• id : JsonRpc.RequestID
• version : Nat

structure Lean.Server.Watchdog.ReferenceData : Type

• loadingTask : ServerTask Unit
• references : References
• finalizedWorkerILeanVersions : Std.TreeMap Lsp.DocumentUri Nat compare
• pendingWaitForILeanRequests : Array WaitForILeanRequest

def Lean.Server.Watchdog.ReferenceData.modifyReferences (rd : ReferenceData) (f : References → References) : ReferenceData

def Lean.Server.Watchdog.ReferenceData.modifyReferencesM {m : Type → Type u_1} [Monad m] (rd : ReferenceData) (f : References → m References) : m ReferenceData

def Lean.Server.Watchdog.ReferenceData.modifyPendingWaitForILeanRequests (rd : ReferenceData) (f : Array WaitForILeanRequest → Array WaitForILeanRequest) : ReferenceData

def Lean.Server.Watchdog.ReferenceData.modifyFinalizedWorkerILeanVersions (rd : ReferenceData) (f : Std.TreeMap Lsp.DocumentUri Nat compare → Std.TreeMap Lsp.DocumentUri Nat compare) : ReferenceData

inductive Lean.Server.Watchdog.LogMsg : Type

• deserialized (direction : JsonRpc.MessageDirection) (msg : JsonRpc.Message) : LogMsg
• serialized (direction : JsonRpc.MessageDirection) (msg : String) : LogMsg

@[implicit_reducible]

instance Lean.Server.Watchdog.instInhabitedLogMsg : Inhabited LogMsg

def Lean.Server.Watchdog.instInhabitedLogMsg.default : LogMsg

structure Lean.Server.Watchdog.LogData : Type

• cfg : Logging.LogConfig
• chan? : Option (Std.Channel LogMsg)

structure Lean.Server.Watchdog.ServerContext : Type

• hIn : IO.FS.Stream
• hOut : IO.FS.Stream
• logData : LogData
• args : List String

  Command line arguments.

• fileWorkersRef : IO.Ref (Std.TreeMap Lsp.DocumentUri FileWorker compare)
• initParams : Lsp.InitializeParams

  We store these to pass them to workers.

• workerPath : System.FilePath
• referenceData : Std.Mutex ReferenceData
• serverRequestData : IO.Ref ServerRequestData
• importData : IO.Ref ImportData
• requestData : RequestDataMutex

structure Lean.Server.Watchdog.ReferenceRequestContext : Type

• fileWorkerMods : Std.TreeMap Lsp.DocumentUri Name compare
• references : References

@[reducible, inline]

abbrev Lean.Server.Watchdog.ServerM (α : Type) : Type

def Lean.Server.Watchdog.readMessage : ServerM JsonRpc.Message

def Lean.Server.Watchdog.writeMessage (msg : JsonRpc.Message) : ServerM Unit

def Lean.Server.Watchdog.writeSerializedMessage (msg : String) : ServerM Unit

def Lean.Server.Watchdog.updateFileWorkers (val : FileWorker) : ServerM Unit

def Lean.Server.Watchdog.getReferences : ServerM References

def Lean.Server.Watchdog.modifyReferences (f : References → References) : ServerM Unit

def Lean.Server.Watchdog.modifyReferencesIO (f : References → IO References) : ServerM Unit

def Lean.Server.Watchdog.getFileWorker? (uri : Lsp.DocumentUri) : ServerM (Option FileWorker)

def Lean.Server.Watchdog.fileWorkerExists (uri : Lsp.DocumentUri) : ServerM Bool

def Lean.Server.Watchdog.eraseFileWorker (uri : Lsp.DocumentUri) : ServerM Unit

def Lean.Server.Watchdog.setWorkerState (fw : FileWorker) (s : WorkerState) : ServerM Unit

def Lean.Server.Watchdog.getWorkerState (fw : FileWorker) : ServerM WorkerState

def Lean.Server.Watchdog.errorPendingRequests (uri : Lsp.DocumentUri) (code : JsonRpc.ErrorCode) (msg : String) : ServerM Unit

def Lean.Server.Watchdog.eraseGetPendingRequest (uri : Lsp.DocumentUri) (id : JsonRpc.RequestID) : ServerM (Option (JsonRpc.Request Json))

def Lean.Server.Watchdog.erasePendingRequest (uri : Lsp.DocumentUri) (id : JsonRpc.RequestID) : ServerM Bool

def Lean.Server.Watchdog.handleILeanHeaderSetupInfo (fw : FileWorker) (params : Lsp.LeanILeanHeaderSetupInfoParams) : ServerM Unit

def Lean.Server.Watchdog.handleIleanInfoUpdate (fw : FileWorker) (params : Lsp.LeanIleanInfoParams) : ServerM Unit

def Lean.Server.Watchdog.handleIleanInfoFinal (fw : FileWorker) (params : Lsp.LeanIleanInfoParams) : ServerM Unit

def Lean.Server.Watchdog.emitServerRequestResponse {α : Type u_1} [ToJson α] (fw : FileWorker) (r : JsonRpc.Response α) : IO Unit

def Lean.Server.Watchdog.emitServerRequestResponseError (fw : FileWorker) (r : JsonRpc.ResponseError Unit) : IO Unit

structure Lean.Server.Watchdog.ModuleQueryMatchScore : Type

• isExactMatch : Bool
• score : Float

def Lean.Server.Watchdog.ModuleQueryMatchScore.compare (ms1 ms2 : ModuleQueryMatchScore) : Ordering

structure Lean.Server.Watchdog.ModuleQueryMatchextends Lean.Server.Watchdog.ModuleQueryMatchScore : Type

• isExactMatch : Bool
• score : Float
• decl : Name
• declAsString : String

def Lean.Server.Watchdog.ModuleQueryMatch.fastCompare (m1 m2 : ModuleQueryMatch) : Ordering

def Lean.Server.Watchdog.ModuleQueryMatch.compare (m1 m2 : ModuleQueryMatch) : Ordering

def Lean.Server.Watchdog.matchAgainstQuery? (query : Lsp.LeanModuleQuery) (decl : Name) : Option ModuleQueryMatch

def Lean.Server.Watchdog.handleQueryModule (fw : FileWorker) (id : JsonRpc.RequestID) (params : Lsp.LeanQueryModuleParams) : ServerM (ServerTask Unit × IO.CancelToken)

def Lean.Server.Watchdog.handleImportClosure (fw : FileWorker) (params : Lsp.LeanImportClosureParams) : ServerM Unit

Updates the global import data with the import closure provided by the file worker after it successfully processed its header.

def Lean.Server.Watchdog.findDefinitions (fw : FileWorker) (p : Lsp.TextDocumentPositionParams) : ServerM (Array Lsp.Location)

def Lean.Server.Watchdog.parseRequestParams? (α : Type) [FromJson α] (request : String) : Except String α

def Lean.Server.Watchdog.parseNotificationParams? (α : Type) [FromJson α] (notification : String) : Except String α

def Lean.Server.Watchdog.parseResponseResult? (α : Type) [FromJson α] (response : String) : Except String α

def Lean.Server.Watchdog.bendLocationLinks (fw : FileWorker) (req : JsonRpc.Request Json) (r : Array Lsp.LeanLocationLink) : ServerM Json

def Lean.Server.Watchdog.handleResponseSpecialCases (fw : FileWorker) (req : JsonRpc.Request Json) (resp : String) : ServerM String

def Lean.Server.Watchdog.startFileWorker (m : DocumentMeta) : ServerM Unit

def Lean.Server.Watchdog.terminateFileWorker (uri : Lsp.DocumentUri) : ServerM Unit

def Lean.Server.Watchdog.tryWriteMessage (uri : Lsp.DocumentUri) (msg : JsonRpc.Message) : ServerM Unit

def Lean.Server.Watchdog.notifyAboutStaleDependency (uri staleDependency : Lsp.DocumentUri) : ServerM Unit

Sends a notification to the file worker identified by uri that its dependency staleDependency is out-of-date.

def Lean.Server.Watchdog.handleReference (p : Lsp.ReferenceParams) : ReaderT ReferenceRequestContext IO (Array Lsp.Location)

structure Lean.Server.Watchdog.CallHierarchyItemData : Type

Used in CallHierarchyItem.data? to retain all the data needed to quickly re-identify the call hierarchy item.

• module : Name
• name : Name

@[implicit_reducible]

instance Lean.Server.Watchdog.instFromJsonCallHierarchyItemData : FromJson CallHierarchyItemData

def Lean.Server.Watchdog.instFromJsonCallHierarchyItemData.fromJson : Json → Except String CallHierarchyItemData

@[implicit_reducible]

instance Lean.Server.Watchdog.instToJsonCallHierarchyItemData : ToJson CallHierarchyItemData

def Lean.Server.Watchdog.instToJsonCallHierarchyItemData.toJson : CallHierarchyItemData → Json

def Lean.Server.Watchdog.CallHierarchyItemData.fromItem? (item : Lsp.CallHierarchyItem) : Option CallHierarchyItemData

Extracts the CallHierarchyItemData from item.data? and returns none if this is not possible.

def Lean.Server.Watchdog.handlePrepareCallHierarchy (p : Lsp.CallHierarchyPrepareParams) : ReaderT ReferenceRequestContext IO (Array Lsp.CallHierarchyItem)

def Lean.Server.Watchdog.handleCallHierarchyIncomingCalls (p : Lsp.CallHierarchyIncomingCallsParams) : ReaderT ReferenceRequestContext IO (Array Lsp.CallHierarchyIncomingCall)

def Lean.Server.Watchdog.handleCallHierarchyOutgoingCalls (p : Lsp.CallHierarchyOutgoingCallsParams) : ReaderT ReferenceRequestContext IO (Array Lsp.CallHierarchyOutgoingCall)

def Lean.Server.Watchdog.handlePrepareModuleHierarchy (p : Lsp.LeanPrepareModuleHierarchyParams) : ReaderT ReferenceRequestContext IO (Option Lsp.LeanModule)

def Lean.Server.Watchdog.sortModuleImports (imports : Array ModuleImport) : Array ModuleImport

def Lean.Server.Watchdog.handleModuleHierarchyImports (p : Lsp.LeanModuleHierarchyImportsParams) : ReaderT ReferenceRequestContext IO (Array Lsp.LeanImport)

def Lean.Server.Watchdog.handleModuleHierarchyImportedBy (p : Lsp.LeanModuleHierarchyImportedByParams) : ReaderT ReferenceRequestContext IO (Array Lsp.LeanImport)

def Lean.Server.Watchdog.handleWorkspaceSymbol (p : Lsp.WorkspaceSymbolParams) : ReaderT ReferenceRequestContext IO (Array Lsp.SymbolInformation)

def Lean.Server.Watchdog.handlePrepareRename (p : Lsp.PrepareRenameParams) : ReaderT ReferenceRequestContext IO (Option Lsp.Range)

def Lean.Server.Watchdog.handleRename (p : Lsp.RenameParams) : ReaderT ReferenceRequestContext IO Lsp.WorkspaceEdit

def Lean.Server.Watchdog.handleDidOpen (p : Lsp.LeanDidOpenTextDocumentParams) : ServerM Unit

def Lean.Server.Watchdog.handleDidChange (p : Lsp.DidChangeTextDocumentParams) : ServerM Unit

def Lean.Server.Watchdog.handleDidSave (p : Lsp.DidSaveTextDocumentParams) : ServerM Unit

When a file is saved, notifies all file workers for files that depend on this file that this specific import is now stale so that the file worker can issue a diagnostic asking users to restart the file.

def Lean.Server.Watchdog.handleDidClose (p : Lsp.DidCloseTextDocumentParams) : ServerM Unit

def Lean.Server.Watchdog.handleDidChangeWatchedFiles (p : Lsp.DidChangeWatchedFilesParams) : ServerM Unit

def Lean.Server.Watchdog.handleCancelRequest (p : Lsp.CancelParams) : ServerM Unit

def Lean.Server.Watchdog.forwardNotification {α : Type} [ToJson α] [Lsp.FileSource α] (method : String) (params : α) : ServerM Unit

def Lean.Server.Watchdog.forwardRequestToWorker (id : JsonRpc.RequestID) (method : String) (params : Json) : ServerM Unit

def Lean.Server.Watchdog.handleReferenceRequest (α β : Type) [FromJson α] [ToJson β] (id : JsonRpc.RequestID) (params : Json) (handler : α → ReaderT ReferenceRequestContext IO β) : ServerM Unit

def Lean.Server.Watchdog.handleRequest (id : JsonRpc.RequestID) (method : String) (params : Json) : ServerM Unit

def Lean.Server.Watchdog.handleNotification (method : String) (params : Json) : ServerM Unit

def Lean.Server.Watchdog.handleResponse (id : JsonRpc.RequestID) (result : Json) : ServerM Unit

def Lean.Server.Watchdog.handleResponseError (id : JsonRpc.RequestID) (code : JsonRpc.ErrorCode) (message : String) (data? : Option Json) : ServerM Unit

def Lean.Server.Watchdog.shutdown : ServerM Unit

inductive Lean.Server.Watchdog.ServerEvent : Type

• workerEvent (fw : FileWorker) (ev : WorkerEvent) : ServerEvent
• clientMsg (msg : JsonRpc.Message) : ServerEvent
• clientError (e : IO.Error) : ServerEvent

def Lean.Server.Watchdog.runClientTask : ServerM (ServerTask ServerEvent)

partial def Lean.Server.Watchdog.mainLoop (clientTask : ServerTask ServerEvent) : ServerM Unit

def Lean.Server.Watchdog.mkLeanServerCapabilities : Lsp.ServerCapabilities

def Lean.Server.Watchdog.initAndRunWatchdogAux : ServerM Unit

def Lean.Server.Watchdog.findWorkerPath : IO System.FilePath

def Lean.Server.Watchdog.startLoadingReferences (referenceData : Std.Mutex ReferenceData) : IO Unit

Starts loading .ileans present in the search path asynchronously in an IO task. This ensures that server startup is not blocked by loading the .ileans. In return, while the .ileans are being loaded, users will only get incomplete results in requests that need references.

def Lean.Server.Watchdog.runMessageLoggingTask (logData : LogData) : IO Unit

def Lean.Server.Watchdog.initAndRunWatchdog (args : List String) (i o : IO.FS.Stream) : IO Unit

def Lean.Server.Watchdog.watchdogMain (args : List String) : IO UInt32