Osmosys Osmosys Developer Documentation
From instrument file to finished result
Osmosys is the software layer between the instruments in your lab and the rest of your data infrastructure. It notices when an instrument produces a file, runs the right analysis automatically, keeps a record of what happened, and where it helps, can send a command back to the instrument. These pages explain how to use it, how it's built, and how to extend it.
In most labs there is a gap between an instrument finishing a run and a scientist getting an answer. Files pile up in folders, someone copies them to a cluster, someone else remembers to start the analysis, and by the time a result comes back the next sample is already loaded. Osmosys watches your instruments, processes their output as soon as it appears, and gives your team one place to see what every instrument is doing and what every run produced.
You don't have to replace anything to adopt it. Osmosys works alongside the instruments, storage, and analysis tools you already have, observing and orchestrating them rather than displacing them.
How this documentation is organized
Guide
A tour of the console
The console is the web application your team uses to run the lab day to day. This guide walks through each part of it, so that after ten minutes you know where to look for anything.
When you open Osmosys you land on Lab, the home screen. It shows how many analyses are running, how much of your compute is in use, and which of your instruments are active. Across the top is a command bar labelled Ask Osmosys, where you can type a plain-English question instead of hunting through menus. That is covered in Using the assistant.
This clip is a placeholder recorded against sample data, so you can see the interface. It will be replaced with a capture from a live system before release.
The five main areas
Everything in the console lives under five headings in the left-hand navigation. You rarely need more than these.
| Area | What you do here |
|---|---|
| Lab | Your starting point: live status for the whole lab, the running-jobs list, and the command bar. |
| Instruments | Every instrument Osmosys knows about, grouped by type (cryo-EM, sequencing, and so on). Open one to see its current state, recent activity, and the data it's sending. |
| Control Plane | The runs themselves: what's queued, what's in progress, and what finished. Open a run to watch it move through its steps. |
| Data | The results. Every dataset a run produced, searchable, each one traceable back to the instrument and run that made it. |
| Record | The lab's automatic logbook: a time-ordered account of everything that happened, written by the system rather than by hand. See The research record. |
Finding things quickly
Two shortcuts save the most time. Press ⌘K (or Ctrl-K) anywhere to open search and jump straight to an instrument, a run, or a dataset by name. On any list, the coloured status labels (running, queued, failed) let you scan for what needs attention without reading every row.
New to the lab? Spend your first session on the Instruments page. Once you understand what each instrument is sending and how healthy it is, the rest of the console reads naturally.
Guide
Connecting a new instrument
Bringing an instrument onto Osmosys takes one small program and about five minutes. This guide explains what that program is, how to set it up, and how to confirm it's working. The instrument itself does not need to change.
The idea in one paragraph
Every instrument writes its output to a folder. A microscope drops movie frames into a session directory; a sequencer writes reads as it goes. Osmosys learns about an instrument by watching that folder. You run a lightweight program called a watcher, point it at the folder, and from then on it notices each file as it appears and tells Osmosys what arrived. The watcher runs beside the instrument, on the acquisition PC or a nearby machine that can see the same folder. It never touches the instrument's own software.
Step by step
- Pick the right watcher. There is one watcher per instrument type: a cryo-EM microscope, an Oxford Nanopore sequencer, and so on. Each already knows what its instrument's files look like and which details to pull out of them. The full list is in the watcher reference.
- Tell it where to look, and who it is. A watcher is configured through a handful of environment variables: the folder to watch, and the credentials it uses to talk to Osmosys. Nothing is hard-coded, so the same watcher runs unchanged in every lab.
- Start it. Watchers ship as small containers, so starting one is a single
docker run. It signs in, begins watching the folder, and waits. - Land a file and confirm. Run a short acquisition, or drop a sample file into the folder. Within about two seconds the watcher checks that the file is complete and valid, then sends Osmosys a short message describing it. Open the Instruments page and you'll see the instrument report in.
What a watcher command looks like
Here is the Oxford Nanopore experiment watcher. The variables in blue are the only things you change per instrument.
docker run -d --name ont-experiment \ -v /data/ont/projects:/data:ro \ # the folder the sequencer writes to -e ONT_PROJECT_DIRECTORY=/data \ -e KAFKA_API_URL=https://gateway.yourlab.io/services/servicekafkaproducer/api/messages/dev-osmosys-experiment \ -e AUTHORIZATION_URL=https://auth.osmosys.io/realms/lab/token \ -e CLIENT_ID=$OSMOSYS_CLIENT_ID -e CLIENT_SECRET=$OSMOSYS_CLIENT_SECRET \ -e GRANT_TYPE=client_credentials \ registry.yourlab.io/ngs_ont_experiment:latest
The watcher validates before it reports. A half-written or malformed file is skipped rather than forwarded, so a sample sheet still being saved won't trigger a false start. It only tells Osmosys about files that are ready.
That's the whole setup. Once the instrument is reporting in, Osmosys can begin running analyses on what it produces, which is the subject of the next guide.
Guide
Running your first analysis
Once an instrument is connected, Osmosys can turn its raw output into a finished result without anyone babysitting the process. This guide follows a single file from the moment it lands to the moment a result appears.
What happens on its own
Most analyses start themselves. When a file lands, the watcher reports it, and Osmosys matches it to the pipeline set up for that instrument and assay. For a cryo-EM microscope that might be motion correction followed by CTF estimation; for a nanopore run it might be basecalling followed by quality checks. Each pipeline is a sequence of steps, and Osmosys runs them in order on your compute, moving to the next step only when the previous one passes its quality check.
You watch this happen on the Control Plane page. A run appears as soon as it's created, shows you which step it's on and how far along it is, and updates live. When it finishes, the results land in Data, already labelled with the instrument, the sample, and the run that produced them.
Starting one yourself
Sometimes you want to start an analysis by hand, to re-process an older dataset or to try different parameters. You can do that two ways:
- From the run list. On the Control Plane page, choose the pipeline you want, point it at the input data, and start it. The run joins the same live list as the automatic ones.
- By asking. Type what you want into the command bar, for example "process the latest apoferritin session with RELION", and the assistant sets up and submits the run for you. This is covered in Using the assistant.
When something fails
If a step fails, whether from a corrupt file or a quality gate that wasn't met, the run stops at that step and is marked failed rather than silently producing a bad result. The record for that run tells you which step failed and why, so you can fix the input and re-run just the part that broke instead of starting over.
Because a run only advances when each step passes, compute isn't spent finishing an analysis that was already doomed by its first frame, and you find out early rather than after the microscope has moved on.
Guide
Using the AI assistant
The assistant lets you ask the lab questions and give it instructions in plain English, instead of clicking through screens. This guide explains what it can do, how it does it, and how to get good answers.
You reach the assistant two ways: the Ask Osmosys command bar at the top of the Lab page, or the full chat view. Either way, you type a question and it answers using live information from the lab rather than general knowledge. When you ask "what instruments are running right now?", it doesn't guess. It looks up the current state of your instruments and tells you.
How it actually answers
Behind the scenes, the assistant has a set of tools: small, well-defined actions it can take, like "list the instruments," "fetch the logs for the last hour," or "start a motion-correction job." When you ask something, it decides which tool fits, calls it, reads the real result, and answers from that. This is why its answers are grounded. Every operational reply traces back to a real query against the running system. The complete set of tools is in the tools reference.
Things worth asking
| You type | What it does |
|---|---|
| "What instruments are online?" | Lists your instruments with their current health and when each was last seen. |
| "Why did the last run pause?" | Reads the run's history and explains the cause in a sentence, with the evidence. |
| "Show me everything that happened to sample CR-207." | Pulls the full record for that sample, end to end. |
| "Start basecalling on tonight's nanopore run." | Sets up and submits the job, then reports the run it created. |
The assistant only acts through its tools, and anything that changes the lab, such as starting a job or sending a command to an instrument, is recorded in the research record just like a manual action. You always have an account of what it did and why.
Guide
The research record
Osmosys keeps a time-ordered account of everything that happens in the lab, written automatically as it happens. This is the Record: a laboratory notebook that fills itself in.
Every meaningful event lands here: a file arriving, a pipeline step starting and finishing, a decision the system made and the evidence behind it, a command sent to an instrument. Each entry is timestamped and attributed, so you can reconstruct what happened to a sample from beginning to end without anyone having typed a note. This is where it differs from a traditional electronic lab notebook. Nobody wrote it by hand, so nothing depends on someone remembering to.
Reading it
Each entry carries a small badge showing how the action was decided: AUTO for something the system did on its own, ADVISED for a suggestion a person approved, and MANUAL for a direct human action. When an entry involved sending a command to an instrument, it's marked so you can see the moment the lab acted on the physical world.
You can filter the record to a single run or sample. The fastest way is to ask the assistant, "show me the full log for run CR-2026-002". Any slice of it exports to a spreadsheet for a colleague or a review.
Because the record is captured at the source as events happen, results can be traced back through the steps, parameters, and files that produced them, which is what makes a run reproducible months later. Note that this is an operational history, not a regulatory audit trail; see Security & compliance for what that distinction means today.
Architecture
How Osmosys works
This section is for engineers and IT teams who want to understand what's happening under the console. We'll follow a single file from the instrument to a finished, recorded result, naming each component as we go.
Osmosys is an event-driven pipeline. Nothing polls and nothing runs on a schedule waiting for work. A file appearing on an instrument is an event, and that event flows through a chain of components, each of which reacts and passes the next event along. Here is the whole chain:
validates, reports
an event stream
(dorado, CryoSPARC…)
results, lineage
Walking the chain
A watcher sits beside the instrument and turns "a file appeared" into a structured message. It runs the validation appropriate to that instrument, checking that a POD5 file or a sample sheet is complete, and only then reports. This is where the messy edge of the lab becomes well-described data. (See The event spine.)
The watcher sends its message to the producer, a service that writes events onto a durable event stream built on Kafka. The stream is the backbone of the system. Everything downstream reads from it, and because it's a log rather than a queue, the history of the lab stays replayable.
A set of consumers subscribe to the stream. Each one watches for a particular kind of event and does real work in response: running dorado to basecall a nanopore read, driving CryoSPARC through motion correction, splitting a POD5 file. When a consumer finishes, it records the outcome in the relevant backend service and often produces the next event. That is how one file landing turns into a whole analysis without a central scheduler orchestrating it.
Why it's built this way
Because every step is triggered by an event and every event is on a durable log, the system reacts in near real time rather than on a polling interval, and a run can be reconstructed by replaying its events. It also means you can add a new analysis stage by adding a consumer that listens for the events it cares about. Nothing else has to change.
The rest of this section goes one level deeper on each piece: the event spine, the control plane that governs it, how decisions feed back to the bench, and how it all deploys.
Architecture
The event spine
The event spine is the backbone that carries every fact about the lab from the instruments to the software that acts on it. It has three parts: the watchers that produce events, the stream they flow through, and the consumers that process them.
Producers: the watchers
Each instrument type has a dedicated watcher, written in Rust, that runs at the edge next to the instrument. It watches a directory and reacts when a relevant file appears, typically within a two-second debounce so that a burst of writes collapses into a single settled event. It validates before it reports: a nanopore watcher opens the POD5 file to confirm it's well-formed, a sequencing watcher checks the sample sheet's columns. Only valid files become events, which keeps malformed data out of the pipeline.
A watcher authenticates to your Osmosys gateway and sends each event as a small JSON message over HTTPS. It carries no analysis logic of its own. Its whole job is to notice, validate, and describe.
The stream: topics and consumers
Events are written to a Kafka stream, organised into a few named topics. A topic is a channel for one kind of event. A consumer subscribes to the topic it cares about and reads events in order, keeping its place so that nothing is processed twice and nothing is missed. Because the stream is a durable log rather than a queue that empties, the full history stays available for replay.
| Analysis stage | Reads topic |
|---|---|
kafka | dev-osmosys-instrument |
cryosparc | dev-osmosys-experiment |
ont-pod5-splitter | dev-osmosys-instrument |
ont-basecall | dev-osmosys-instrument |
ngs-management | dev-osmosys-experiment |
ngs-ont-concatenator | dev-osmosys-preprocessing |
Consumers: where the science happens
Each consumer turns an event into real work. It filters for the events it handles, say nanopore basecalling requests, and then shells out to the actual scientific tool, such as dorado or the CryoSPARC engine, to do the computation. When the tool finishes, the consumer reports the outcome back to the appropriate backend service and, where the workflow continues, emits the next event for the following stage to pick up.
Deployment note. The Kafka broker is provisioned as part of your deployment rather than bundled here, and today's delivery is best-effort (at-most-once). A durable local buffer with guaranteed replay is on the roadmap. Plan broker capacity and retention for your event volume; see Deployment.
Architecture
The control plane
The control plane is the layer that decides what to do with the events on the spine, turning them into coordinated analysis, quality control, and where warranted, action.
Its job is orchestration with context. As events flow past, it dispatches the right pipeline for each instrument and assay, enforces quality gates so that data failing an early check never consumes downstream compute, and tracks the state of every run so you can see which sample has completed which step. What makes it more than a job queue is that every unit of work carries its lineage with it: which instrument, which run, which quality score, which prior results. Decisions are made with that context rather than in isolation.
You interact with the control plane three ways, all over the same foundation: through the console, as an operator; through the assistant and its tools, in natural language; and through the gateway API, programmatically. All three read and write the same services, so they never disagree about the state of the lab.
The control plane is not a laboratory-automation scheduler and not a reporting dashboard. It is operational: agents act on it, and decisions feed back into the lab rather than only being displayed.
Architecture
Closing the loop
Osmosys can feed a decision back to the instrument while a run is still in progress, so that a failing experiment can be stopped rather than analysed after the fact.
We call this actuation: a command sent to an instrument in response to what its data is showing. It is treated as a recorded action rather than a side channel. When the system or a person decides to pause a degraded cryo-EM session, that decision becomes a command with a full status history (issued, sent, acknowledged) and appears in the research record like any other event. There is no separate "device control" panel to reason about. Actuation is another logged action on the spine.
Where it is worth the complexity
Actuation pays off only where the cost of continuing a bad run is high and there is a real window in which to decide. Two cases meet that bar:
| Case | Why it pays off | Status |
|---|---|---|
| Cryo-EM quick-QC | Checking the first exposures lets the system abort a degraded grid before hours of microscope time are spent on it. | GA |
| Nanopore read-until | Deciding per read whether to keep sequencing or eject conserves pore capacity for the reads that matter. | Beta |
| LC-MS quality gates | Triggering a rerun mid-batch instead of after it saves a full re-acquisition. | Preview |
Any command that physically moves hardware passes a safety gate first. It is either provably side-effect-free, or it requires human or policy approval before it fires.
Architecture
Which instruments Osmosys observes
The first question we get is usually "which instruments do you support?" The architectural answer is that the runtime does not know what it is watching, and that is intentional rather than a gap.
What it means for an instrument to be "on" the runtime
Every instrument already writes files somewhere. An instrument is on the runtime when something sits beside it, notices those files as they land, checks they are complete, and describes what arrived as a structured event. That something is a watcher, a small component at the edge. It does not talk to the instrument's control software, it does not need a vendor API or a plugin, and it does not require the acquisition PC to be modified. It reads a folder.
Everything downstream is instrument-agnostic. The event stream, the pipelines, the control plane, and the record cannot tell a transmission electron microscope from a sequencer, and they do not need to.
Why this isn't a connector catalogue. A connector company sells breadth, where the integration count is the product and each new instrument is another vendor integration to build and maintain. Osmosys is built the other way round. Only two things are ever instrument-specific: the file pattern and validation at the edge, and the scientific semantics in the app layer. The substrate in between does not change. Coverage here is therefore a contract rather than a count, and meeting it for a new instrument class is a small piece of work rather than a platform project.
Classes on the runtime today
These are the instrument classes with watchers in production. The models named are the ones we have exercised, but a class is defined by the file contract rather than the model. Another microscope writing the same movie files uses the same watcher and is not a new integration.
| Class | Exercised on | Observes | Status |
|---|---|---|---|
| Cryo-EM (TEM) | Thermo Fisher Krios G4, Glacios | Movie frames (.tiff/.mrc) as they're acquired, plus project and workspace state | Stable |
| Nanopore sequencing | Oxford Nanopore PromethION 24 | .pod5 reads, sample sheets, basecall and workspace activity | Stable |
| Illumina sequencing | NovaSeq X Plus | Run folders and basecall output | Preview |
The app layer's vocabulary covers more ground than the watchers do. The tool catalogue carries components for proteomics and spatial work (see the tool layer). Those assays run today on data brought in through the other routes described in Integrating your systems; a dedicated watcher is simply the tidiest route.
Adding a class
Bringing a new instrument class onto the runtime means implementing one contract:
- Observe. Watch the directory the instrument writes to, and decide which files matter. This is usually an extension and a naming pattern.
- Validate. Confirm the file is complete and well-formed before saying anything. A half-written file should produce silence rather than a false start.
- Emit. Describe what arrived as an event, naming the file, the assay and the run, and post it to the event gateway.
That is the whole obligation. Nothing in the control plane, the stream, or the record changes to accommodate a new class. The work is a small edge component, plus the scientific meaning you want to attach to it.
Architecture
The tool layer
Osmosys does not reimplement science. It runs the tools your field already uses, in containers with a defined contract, and exposes a second set of tools that an assistant can invoke. Those are the two halves of the tool layer.
Containerised compute components
A component is one scientific tool, pinned to a version, packaged as a container, with a declared set of inputs and outputs. Because the contract is explicit, components compose. The runtime chains them into a pipeline, feeding each step's outputs into the next, and schedules them on your compute. Pinning is what makes a run reproducible, because the exact image is recorded alongside the result.
The catalogue below is generated from the component registry, so these are the versions that ship.
Cryo-EM
| Component | Tool | Image |
|---|---|---|
| MotionCor2 | MotionCor2 | cryoem/motioncor2:1.6.4 |
| CTFFIND4 | CTFFIND | quay.io/biocontainers/ctffind4:4.1.14--h2ba8f37_2 |
| crYOLO | crYOLO | sphire/cryolo:1.9.9 |
| RELION 2D classification | RELION | relion/relion:5.0.0 |
| RELION 3D classification | RELION | relion/relion:5.0.0 |
| RELION 3D auto-refine | RELION | relion/relion:5.0.0 |
| RELION PostProcess | RELION | relion/relion:5.0.0 |
Sequencing (NGS)
| Component | Tool | Image |
|---|---|---|
| bcl-convert | Illumina bcl-convert | nfcore/bclconvert:4.2.7-2 |
| FastQC | FastQC | quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0 |
| fastp | fastp | quay.io/biocontainers/fastp:0.23.4--hadf994f_2 |
| BWA-MEM | BWA | quay.io/biocontainers/bwa:0.7.17--he4a0461_11 |
| STAR | STAR | quay.io/biocontainers/star:2.7.11a--h0033a41_0 |
| samtools sort | samtools | staphb/samtools:1.19 |
| GATK MarkDuplicates | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK BaseRecalibrator | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK ApplyBQSR | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK HaplotypeCaller | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK Mutect2 | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK FilterMutectCalls | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK CollectReadCounts | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK DenoiseReadCounts | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK ModelSegments | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK CallCopyRatioSegments | GATK | broadinstitute/gatk:4.5.0.0 |
| GATK Funcotator | GATK | broadinstitute/gatk:4.5.0.0 |
Proteomics & mass spec
| Component | Tool | Image |
|---|---|---|
| msconvert | ProteoWizard | proteowizard/pwiz-skyline-i-agree-to-the-vendor-licenses:latest |
| MSFragger | FragPipe | fragpipe/fragpipe:22.0 |
| Percolator | Percolator | quay.io/biocontainers/percolator:3.6.5--h6351f2a_0 |
| Philosopher | FragPipe | prvst/philosopher:5.1.0 |
| MaxQuant | MaxQuant | quay.io/biocontainers/maxquant:2.4.2.0--py310hdfd78af_0 |
| DIA-NN | DIA-NN | ghcr.io/vdemichev/diann:1.9.2 |
Licensing. Several of these tools carry their own vendor or academic licences, and obtaining them is your responsibility. Packaging a tool is not the same as licensing it. One image name states this in the image name itself (pwiz-skyline-i-agree-to-the-vendor-licenses), and tools such as MotionCor2, CryoSPARC and gated model weights each carry their own terms. Check the licence for any component before running it in production. See Dependencies & licensing.
Tools an assistant can invoke
The second half of the tool layer is the set of operations exposed to an assistant over MCP. A component is what runs; an MCP tool is what can be asked for. List the instruments, read the last hour of logs, submit a motion-correction job, poll its status. The assistant chooses the right one and answers from the real result rather than from memory, which is what keeps its answers grounded.
There are 19 of them today, split between looking things up and running analyses. The full list, generated from source, is in the assistant tools reference.
Both halves sit above the runtime rather than inside it. The substrate does not know what dorado or RELION are; it knows how to schedule a container and record what it produced. That separation is what lets a new tool arrive without touching the platform.
Architecture
Integrating with the systems you already run
Osmosys sits between your instruments and the rest of your stack. It is not a replacement for your notebook, your warehouse, or your identity provider. It produces well-described context and hands it on to whatever you already run.
People usually arrive with a version of the question "what does this replace?" For most teams the answer is nothing. Osmosys occupies the gap between the instrument and your existing platform, which is normally filled by people copying files around. Everything it produces leaves through one of four routes.
The four routes in and out
| Route | What it's for | Status |
|---|---|---|
| The event gateway inbound | Anything that can make an authenticated HTTPS call can put an event on the stream: a custom instrument, a bespoke internal platform, an existing pipeline. This is the same route the watchers use, and there is no privileged path. | Stable |
| The gateway API both | Read and write the domain directly (runs, samples, experiments, datasets, instruments) over REST with your OAuth2 token. This is how another platform pulls state out or pushes records in. | Stable |
| Record-system connections both | A dedicated model for external notebooks and LIMS: a connection with credentials and direction, the records it is scoped to, and a log of everything synced. | Beta |
| Assistant tools (MCP) outbound | External agents query the lab through the tool layer rather than against a database, so they get grounded answers and you get a record of what they asked. | Stable |
Connecting a record system (ELN / LIMS)
Notebooks and LIMS get a dedicated model because they are the systems labs most often need kept in step. A connection names the external system, how to authenticate to it (OAuth2, an API key, or basic auth), and which way records flow: inbound, where it is the source of truth for samples; outbound, where Osmosys pushes results to it; or bidirectional. Connections are scoped to the record types they cover, typically samples and experiments, and every exchange is written to a sync log you can inspect when the two systems disagree.
The model ships with shapes for the common commercial systems (Benchling, LabWare, STARLIMS, Clarity) and a custom provider for in-house systems, which is the one most large organisations use.
On top of connections sit delivery rules. A rule binds one pipeline step's output, such as a VCF, a BAM or a report, to a connection, so the artefact is delivered where it belongs without anyone exporting it by hand.
Connections and the sync log are Beta. Delivery rules are modelled and configurable today, but automatic delivery on every run is Preview. Confirm the behaviour for your deployment before depending on it.
Common integration patterns
Most requests land in one of these shapes. None of them require a bespoke connector from us.
| You want to… | The pattern |
|---|---|
| Keep an electronic lab notebook or LIMS in step | A record-system connection, with direction set by which system owns the truth, plus delivery rules for results. |
| Feed a data warehouse or lakehouse | Outbound. Either a delivery rule writes the artefact where your loader expects it, or your existing ETL subscribes to the event stream and reads data products through the gateway API. We do not ship a warehouse-specific connector, because the event stream and the gateway API are already the documented interfaces and your platform team owns the loader. |
| Pull sample or inventory records in | Inbound: a connection from the system that accessions samples, or direct writes through the gateway API. |
| Use your identity provider | Standard OpenID Connect. The gateway relays your token to every service, so your existing accounts and groups apply throughout. |
| Integrate a bespoke internal platform | The event gateway inbound, the gateway API outbound, and a custom connection if it is a record system. This is the usual answer for anything home-grown. |
| Let your own agents query the lab | Point them at the MCP tool layer. |
| Run analyses on your compute | The components are containers, so they run where you tell them to. See the tool layer. |
The principle behind all four routes is the same. Osmosys makes instrument data legible, meaning described, validated and attributable, and then hands it on. If you already have a platform you trust, it stays where it is and receives better data sooner.
Architecture
Deployment & topology
Osmosys runs as a set of containerised services, and it is the same software whether it sits on an appliance at the bench, in your cloud, or across both. This page explains how the pieces are deployed and discovered. For the practical steps, see Install & run.
The backend is a fleet of independent services, each responsible for one domain such as instruments, experiments, samples or workflows. They register themselves with a service registry (Consul) as they start, and a single gateway routes every request to the right service by name, relaying the caller's identity token. Because services find each other through the registry, you can scale, restart, or relocate any of them without reconfiguring the rest.
Rollouts are health-gated, so a new version of a service must report healthy before it takes traffic, and deployments pull from a private image registry so that every environment runs an identical, versioned build. Identity is handled by a standard OpenID Connect provider (Keycloak), so the same accounts and permissions apply across the console, the API, and the watchers.
Form factors
The Kafka broker is provisioned per deployment (its address is supplied by configuration), and is not bundled in the service repositories. Size it for your event throughput and retention needs.
Deploy & operate
System requirements
What has to exist before Osmosys will run. The platform is a set of containers, but it expects a few pieces of infrastructure to be in place first.
Infrastructure prerequisites
| Component | Requirement | Notes |
|---|---|---|
| Container runtime | Docker | Everything ships as containers. Service images are built for amd64, so on arm64 hosts they run under emulation and will be slower. |
| Service registry | Consul, reachable on :8500 | Services register on start and refuse to boot without it. |
| Identity | An OpenID Connect provider (Keycloak) | Every service validates JWTs against the issuer, and must be able to reach it at boot or the service will crash-loop. |
| Database | PostgreSQL on :5432 | One database per service, created ahead of time. Schema migrations run automatically; the databases themselves do not. |
| Event broker | Apache Kafka on :9092 | Provisioned by you rather than bundled. Size it for your event throughput and retention. |
| Image registry | A registry the deploy host can pull from | The deploy scripts pull service images by tag. |
Capacity
Each backend service is a JVM and is deployed with a memory cap of roughly 640 MB. Running the full fleet therefore wants on the order of 11 to 12 GB of RAM for the services alone, before the broker, the database, and any compute. Plan the deploy host accordingly. Without caps, a full fleet will exhaust a 16 GB machine.
Where a GPU is required
Most of the platform does not need one. The services, the gateway, the watchers, and the edge stack are all CPU-only. A GPU matters for specific analysis tools:
| Workload | GPU |
|---|---|
| RELION 2D/3D refinement | Strongly accelerated, and typically given a dedicated GPU host |
| MotionCor2 | GPU-only. It will not run without one. |
| CryoSPARC | GPU count is passed through per job |
| DeepVariant | Optional, and requires the GPU image variant |
| AlphaFold 3 | Expects a GPU host |
| dorado basecalling | Preview The current container image ships without CUDA, so basecalling falls back to CPU and will be slow. Treat GPU basecalling as not yet supported. |
Watcher hosts
A watcher runs beside the instrument and needs only Docker, read access to the instrument's output directory, and network reachability to your event gateway and identity provider. One watcher polls rather than using filesystem events, which makes it safe over network mounts.
Deploy & operate
Installing and running
Three things get deployed: the backend services, the watchers beside your instruments, and the edge stack that fronts the console and the assistant.
Backend services
Each service is pulled from your registry and run with its configuration supplied as environment variables. The pattern below is what the deploy scripts execute for each one. The environment overrides are load-bearing rather than decorative; the note underneath explains why each is there.
docker run -d --name <service> \ --network <network> --restart unless-stopped --memory 640m \ -e SPRING_PROFILES_ACTIVE=dev,api-docs \ -e SPRING_CLOUD_CONSUL_HOST=<consul-host> \ -e SPRING_CLOUD_CONSUL_PORT=8500 \ -e SPRING_SECURITY_OAUTH2_CLIENT_PROVIDER_OIDC_ISSUER_URI=https://<keycloak>/realms/<realm> \ -e SPRING_CLOUD_CONSUL_DISCOVERY_PREFER_IP_ADDRESS=true \ -e SPRING_CLOUD_CONSUL_DISCOVERY_IP_ADDRESS=<advertise-ip> \ -e SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE=5 \ -e JAVA_TOOL_OPTIONS="-Xmx448m" \ -p <port>:<port> <registry>/<service>:<tag>
Then wait for the service to report healthy before moving on:
curl -fsS http://localhost:<port>/management/healthWhy these particular overrides. Consul must arrive as an environment variable, because the default configuration binds too late, falls back to localhost, and the container crash-loops. The OIDC issuer must be an environment variable because the OS environment outranks both the registry's key/value store and the image's bundled configuration. The memory cap and pool size stop a full fleet from exhausting the host's RAM and the shared database's connection limit. Use JAVA_TOOL_OPTIONS rather than JAVA_OPTS, which the image entrypoint ignores. Finally, stagger the service starts, because starting the fleet at once will rate-limit the registry.
The gateway is deployed the same way but runs the production profile and needs its database connection supplied, because the development profile expects an embedded database that the production image does not ship.
An instrument watcher
Watchers take everything from the environment: the directory to watch, and the credentials to reach your gateway. See Configuration for the per-watcher variables.
docker run -d --name <watcher> \ -v /path/on/instrument:/data:ro \ -e DIRECTORY_TO_WATCH=/data \ -e KAFKA_API_URL=https://<gateway>/services/servicekafkaproducer/api/messages/<topic> \ -e AUTHORIZATION_URL=https://<keycloak>/realms/<realm>/protocol/openid-connect/token \ -e CLIENT_ID=<id> -e CLIENT_SECRET=<secret> \ -e GRANT_TYPE=client_credentials \ -e RUST_LOG=info \ <registry>/<watcher>:<tag>
Watchers read every setting with no fallback, so a missing variable stops the container at start-up rather than letting it run with a default. This is intentional, on the grounds that a watcher pointed at a guessed directory is worse than one that refuses to start, but it does mean the environment has to be complete.
The edge stack
The reverse proxy, the assistant's chat surface, the model proxy, and the console bundle are brought up together with docker compose up -d from the infrastructure repository, after copying the example environment file and filling in your keys.
Deploy & operate
Configuration reference
Every component is configured through environment variables, with nothing baked into an image. This is the complete set.
Watchers: common to all
| Variable | Required | Purpose |
|---|---|---|
KAFKA_API_URL | yes | Full URL of the event gateway endpoint, including the topic. |
AUTHORIZATION_URL | yes | OIDC token endpoint. |
CLIENT_ID · CLIENT_SECRET | yes | Credentials for the watcher's service account. |
GRANT_TYPE | yes | Normally client_credentials. |
RUST_LOG | no | Log level. Defaults to info, and is the only variable with a default. |
Watchers: per instrument class
| Watcher | Additional variables |
|---|---|
cryoem_instrument | DIRECTORY_TO_WATCH, WORKSPACE_UID, PROJECT_UID, GAIN_FILE_PATH, P_SIZE_A, CS_MM, TOTAL_DOSE_E_PER_A2, ACCELERATION_KV |
cryoem_cryosparc | CRYOSPARC_DIRECTORY |
ngs_ont_instrument | DIRECTORY_TO_WATCH, OPERATION, BASECALL_MODEL, NUMBER_OF_READS_TO_SPLIT, BASECALL_TO_FASTQ, WORKSPACE_UID, PROJECT_UID |
ngs_ont_basecall | none beyond the common set |
ngs_ont_experiment | ONT_PROJECT_DIRECTORY |
ngs_ont_workspace | none beyond the common set |
ngs_illumina_instrument | DIRECTORY_TO_WATCH |
osmosys-filewatcher | none beyond the common set |
The cryo-EM instrument watcher additionally carries the acquisition optics as configuration (pixel size, spherical aberration, total dose, and accelerating voltage) because those values travel with every movie event.
Backend services
| Variable | Purpose |
|---|---|
SPRING_PROFILES_ACTIVE | Which profile to run. The gateway requires the production profile. |
SPRING_CLOUD_CONSUL_HOST · _PORT | Service registry. Must be supplied as environment, not config. |
SPRING_CLOUD_CONSUL_DISCOVERY_IP_ADDRESS | The address this service advertises to the registry. |
SPRING_SECURITY_OAUTH2_CLIENT_PROVIDER_OIDC_ISSUER_URI | Your OIDC issuer. Overrides everything else. |
SPRING_DATASOURCE_* | Database URL and credentials. |
SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE | Connection cap. Required so the fleet doesn't exhaust the shared database. |
JAVA_TOOL_OPTIONS | JVM options. JAVA_OPTS is ignored by the image entrypoint. |
The event producer additionally takes the Kafka broker address as configuration.
The control plane (assistant tools)
The tool server is configured through a set of variables covering where data lives and where each scientific tool can be found:
| Group | Variables |
|---|---|
| Data | OSMOSYS_DATA_ROOT (root of the on-disk data lake), OSMOSYS_WATCHER_LOG_DIR, OSMOSYS_WATCHER_LOG_MAX_MSG |
| Databases | OSMOSYS_NGS_DSN, OSMOSYS_CRYOEM_DSN: read-only connection strings for the log and query tools |
| DeepVariant | OSMOSYS_DEEPVARIANT_IMAGE, OSMOSYS_DEEPVARIANT_RUNTIME |
| AlphaFold | OSMOSYS_ALPHAFOLD_IMAGE, _RUNTIME, _MODEL_DIR (gated weights), _DB_DIR |
| RELION | Per-binary overrides (OSMOSYS_RELION_*_BIN), OSMOSYS_RELION_CTFFIND_EXE (required), OSMOSYS_RELION_MOTIONCOR2_EXE (optional), OSMOSYS_RELION_MPIRUN |
| Remote execution | OSMOSYS_RELION_SSH_HOST, _USER, _KEY, _PORT, _OPTS: run RELION on a GPU host over SSH |
Remote execution requires passwordless key authentication, and both hosts must mount the same shared storage at the same absolute path, because job directories and inputs are referenced by identical paths on both sides.
Deploy & operate
Canonical ports
The default listening ports for every component. Service ports are generated from each service's own configuration.
Backend services
| Service | Port |
|---|---|
gateway | 8080 |
servicekafkaproducer | 8085 |
servicecryoemmanagement | 8086 |
servicengsmanagement | 8087 |
serviceassaymanagement | 8088 |
servicecomputemanagement | 8089 |
servicefleetmanagement | 8090 |
serviceexperimentmanagement | 8091 |
serviceinstrumentmanagement | 8092 |
serviceworkflowmanagement | 8093 |
servicedatasetmanagement | 8094 |
serviceconfigmanagement | 8095 |
servicesamplemanagement | 8096 |
servicealertmanagement | 8097 |
serviceactivitymanagement | 8098 |
serviceprojectmanagement | 8099 |
servicestoragemanagement | 8100 |
serviceinventorymanagement | 8101 |
servicecomponentmanagement | 8102 |
servicepipelinemanagement | 8103 |
servicelimsmanagement | 8104 |
servicefilewatchermanagement | 8105 |
servicekafkaconsumermanagement | 8106 |
Infrastructure
| Component | Port | Notes |
|---|---|---|
| Consul | 8500 | Service registry |
| PostgreSQL | 5432 | One database per service |
| Kafka broker | 9092 | You provision this |
| Keycloak | 443 / 9080 | HTTPS in production; the LAN default is plain HTTP and should be overridden |
| Image registry | 5000 | Deploy hosts pull from here |
| Assistant tool server | 8765 | Binds all interfaces, so firewall it |
| Tool REST bridge | 8000 | Fronted by the reverse proxy |
| Model proxy | 4000 | OpenAI-compatible |
| Chat surface | 3000 | Behind the reverse proxy |
| Reverse proxy | 80 / 443 | The only ports that need to be publicly exposed |
Only the reverse proxy should be reachable from outside the deployment network. Everything else, including the services, registry, database, broker and tool server, is internal.
Deploy & operate
Dependencies & licensing
What Osmosys is built on, and which third-party scientific tools carry their own licences that you need to obtain. The second half of this page is the part that usually matters.
The Osmosys platform itself
Osmosys is proprietary software. It is not open source and no open-source licence is granted. Your rights to use, deploy, and modify it are defined solely by your commercial agreement. Nothing on this page grants a licence.
Platform stack
| Layer | Built on |
|---|---|
| Instrument watchers | Rust (2024 edition), using filesystem notification, HTTP, and format-validation libraries |
| Backend services | Java 17 · Spring Boot 3.4.5 · PostgreSQL · Consul · Keycloak |
| Stream consumers | Java 17 · Apache Kafka clients |
| Control plane tools | Python |
| Console | Angular |
| Edge stack | Caddy, an OpenAI-compatible model proxy, and a chat surface |
Scientific tools you must license yourself
Osmosys orchestrates these tools but does not license them to you. Packaging a tool into a pipeline is not the same as granting the right to use it. Before running any of them in production, and particularly for commercial work, confirm your own entitlement.
| Tool | What you need to do | Commercial use |
|---|---|---|
| CryoSPARC | Obtain a licence from Structura and run your own instance. Osmosys connects to it and never installs it. | Licence required |
| AlphaFold 3 | Model weights are gated, so you request access and stage them yourself. The container is built locally and there is no public image. | Non-commercial terms. Review before commercial use. |
| MotionCor2 | Download from UCSF under its own terms and point the configuration at the binary. Not bundled. | Academic terms |
| Illumina bcl-convert | Obtain from Illumina and accept their licence. | Licence acceptance required |
| Vendor mass-spec formats | Reading vendor raw formats requires accepting the vendor licences carried by the conversion tooling. | Vendor terms |
| RELION, CTFFIND, dorado, samtools, GATK | Open source, but each under its own licence. Review them for your use case. | Open source |
Two of these are structural rather than a formality. CryoSPARC has to already be running, because Osmosys drives it, so a deployment without a CryoSPARC licence has no cryo-EM processing path at all. AlphaFold 3's weights are gated and carry non-commercial terms, so if your use is commercial, resolve that with the model's owners before relying on it.
Deploy & operate
Security & compliance posture
A straight account of what the platform does today for security and regulated environments, and what it does not. If you are evaluating Osmosys for a regulated workflow, read this page before the others.
Osmosys holds no certifications. There is no SOC 2, no ISO 27001, no HIPAA attestation, and no completed GxP validation. The platform is intended for non-GxP research use. We would rather state that here than have you find it out during a security review.
What is implemented
| Area | Today |
|---|---|
| Authentication | OAuth2 and OpenID Connect across the gateway and every service. JWTs are validated on issuer and audience, and sessions are stateless. |
| Authorisation | Role-based, with a small standard role set. There is no separate signer, reviewer, or quality role, and therefore no segregation of duties. |
| Service identity | Watchers and integrations authenticate as service accounts using client credentials. |
| Transport | TLS terminates at the reverse proxy. Internal service-to-service traffic is unencrypted by default and assumes a trusted deployment network. |
| Activity record | An activity-event service records what happened, and the console surfaces it. |
What is not implemented
These are commonly assumed to exist in a platform of this shape. They do not exist here today, and you should plan as though they do not:
- A tamper-evident audit trail. The activity record is a normal, mutable table. There is no append-only guarantee, no hash chain, and no write-once storage. Do not treat it as a regulatory audit trail.
- Per-record audit metadata. Domain records do not currently carry created-by or modified-by auditing.
- Electronic signatures. There is no signing, no signer roster, and no 21 CFR Part 11 controls. Any sign-off language in the interface describes the roadmap rather than working function.
- Encryption at rest. Not configured by the platform. Use your database and storage layer's own encryption.
- Retention, legal hold, and purge. Not implemented.
- Validation artefacts. No IQ/OQ/PQ, no validation plan, no traceability matrix.
Deployment hardening, which is your responsibility
- Rotate every credential before use. Treat all default and example credentials shipped with the platform as public and compromised. Generate your own database passwords and OAuth client secrets.
- Terminate TLS and force HTTPS to your identity provider. Some defaults point at plain HTTP, so override them.
- Expose only the reverse proxy. Keep the services, registry, database, broker and tool server on an internal network. See Ports.
- Health and metrics endpoints are unauthenticated by default. Do not expose them publicly.
- Stream consumers hold Docker access so they can launch analysis containers, which is root-equivalent on their host. Isolate that host.
The defensible summary, if you need one for an internal review: Osmosys is built on an OAuth2/OIDC-authenticated microservice architecture with an activity-event service, and a compliance surface on the roadmap. Claims beyond that, such as immutable audit, electronic signature, Part 11 or a validated state, are not supported by the platform as it stands.
Reference
Instrument watchers
There is one watcher per instrument type. The table and details below are generated directly from each watcher's source, so the configuration keys and behaviour here always match what actually ships. For how to deploy one, see Connecting an instrument.
| Watcher | Instrument / assay | Detection | Status |
|---|---|---|---|
cryoem_instrument | CRYOGENIC_ELECTRON_MICROSCOPY | inotify | Stable |
cryoem_cryosparc | any | inotify | Stable |
ngs_ont_instrument | NEXT_GENERATION_SEQUENCING_OXFORD_NANOPORE | inotify | Stable |
ngs_ont_basecall | NEXT_GENERATION_SEQUENCING_OXFORD_NANOPORE | inotify | Stable |
ngs_ont_experiment | NEXT_GENERATION_SEQUENCING_OXFORD_NANOPORE | inotify | Stable |
ngs_ont_workspace | NEXT_GENERATION_SEQUENCING_OXFORD_NANOPORE | poll (5s) | Beta |
ngs_illumina_instrument | any | inotify | Stable |
osmosys-filewatcher | any | inotify | Preview |
Configuration
Every watcher is configured through environment variables: the folder to watch, and the credentials it uses to reach Osmosys. These keys are shared across the fleet, and the full per-watcher set is in Configuration.
| Variable | Purpose |
|---|---|
KAFKA_API_URL | The gateway endpoint the watcher sends its events to. |
AUTHORIZATION_URL | The sign-in endpoint for the watcher's service account. |
CLIENT_ID, CLIENT_SECRET | The credentials for that account. |
GRANT_TYPE | The sign-in method, usually client_credentials. |
Per-watcher detail
Watches for CRYOGENIC_ELECTRON_MICROSCOPY output via inotify. Configured with: KAFKA_API_URL, AUTHORIZATION_URL, CLIENT_ID, CLIENT_SECRET, GRANT_TYPE, RUST_LOG, DIRECTORY_TO_WATCH, WORKSPACE_UID, PROJECT_UID, GAIN_FILE_PATH, P_SIZE_A, CS_MM, TOTAL_DOSE_E_PER_A2, ACCELERATION_KV.
Watches for files output via inotify. Configured with: KAFKA_API_URL, AUTHORIZATION_URL, CLIENT_ID, CLIENT_SECRET, GRANT_TYPE, RUST_LOG, CRYOSPARC_DIRECTORY.
Watches for NEXT_GENERATION_SEQUENCING_OXFORD_NANOPORE output via inotify. Configured with: KAFKA_API_URL, AUTHORIZATION_URL, CLIENT_ID, CLIENT_SECRET, GRANT_TYPE, RUST_LOG, DIRECTORY_TO_WATCH, OPERATION, BASECALL_MODEL, NUMBER_OF_READS_TO_SPLIT, BASECALL_TO_FASTQ, WORKSPACE_UID, PROJECT_UID.
Watches for NEXT_GENERATION_SEQUENCING_OXFORD_NANOPORE output via inotify. Configured with: none.
Watches for NEXT_GENERATION_SEQUENCING_OXFORD_NANOPORE output via inotify. Configured with: KAFKA_API_URL, AUTHORIZATION_URL, CLIENT_ID, CLIENT_SECRET, GRANT_TYPE, ONT_PROJECT_DIRECTORY.
Watches for NEXT_GENERATION_SEQUENCING_OXFORD_NANOPORE output via poll (5s). Configured with: RUST_LOG.
Watches for files output via inotify. Configured with: KAFKA_API_URL, AUTHORIZATION_URL, CLIENT_ID, CLIENT_SECRET, GRANT_TYPE, DIRECTORY_TO_WATCH.
Watches for files output via inotify. Configured with: none.
Reference
Backend services
The backend is a fleet of 22 independent services, one per domain. Each is reached through the gateway at /services/{service}/api. The entities and fields below are generated from each service's data model, so they stay in step with the running system.
| Service | Port | Holds | Status |
|---|---|---|---|
serviceactivitymanagement | 8098 | ActivityEvent | Stable |
servicealertmanagement | 8097 | Alert | Stable |
serviceassaymanagement | 8088 | AnalysisJob, Assay, Platform | Stable |
servicecomponentmanagement | 8102 | ComponentParam, ComponentTool | Stable |
servicecomputemanagement | 8089 | ComputeNode, ResourcePool | Stable |
serviceconfigmanagement | 8095 | ConfigFile | Stable |
servicecryoemmanagement | 8086 | Project, Workspace | Stable |
servicedatasetmanagement | 8094 | Dataset, DatasetLineage | Stable |
serviceexperimentmanagement | 8091 | Experiment | Stable |
servicefilewatchermanagement | 8105 | none | Preview |
servicefleetmanagement | 8090 | FleetNode | Stable |
serviceinstrumentmanagement | 8092 | ActuationCommand, IngestionSession, Instrument | Stable |
serviceinventorymanagement | 8101 | InventoryItem | Stable |
servicekafkaconsumermanagement | 8106 | none | Preview |
servicekafkaproducer | 8085 | none | Preview |
servicelimsmanagement | 8104 | LimsConnection, LimsDeliveryRule, LimsSyncEvent | Stable |
servicengsmanagement | 8087 | Job, Project, Workspace | Stable |
servicepipelinemanagement | 8103 | Pipeline, PipelineStep | Stable |
serviceprojectmanagement | 8099 | Milestone, Project | Stable |
servicesamplemanagement | 8096 | Sample | Stable |
servicestoragemanagement | 8100 | StorageVolume | Stable |
serviceworkflowmanagement | 8093 | WorkflowRun, WorkflowStep, WorkflowTemplate | Stable |
Every service exposes standard create / read / update / delete operations on its entities at /api/{entities}. Click a service for its full model, or see two representative examples below.
Instruments & actuation
Reached at /services/serviceinstrumentmanagement/api with your access token. com.osmosys.serviceinstrumentmanagement.
ActuationCommand /api/actuation-commands
| Field | Type | |
|---|---|---|
commandUid | UUID | required |
action | ActuationAction {START,STOP,PAUSE,RESUME,CALIBRATE,ACQUIRE,RESET} | required |
params | String | |
status | CommandStatus {PENDING,SENT,ACKED,FAILED} | required |
issuedBy | String | |
issuedAt | Instant | required |
IngestionSession /api/ingestion-sessions
| Field | Type | |
|---|---|---|
sessionUid | UUID | required |
experimentRef | UUID | |
datasetName | String | required |
bytesIngested | Long | |
fileCount | Integer | |
status | IngestionStatus {RECEIVING,COMPLETE,FAILED} | required |
startedAt | Instant | required |
completedAt | Instant |
Instrument /api/instruments
| Field | Type | |
|---|---|---|
instrumentUid | UUID | required |
name | String | required |
vendor | String | |
model | String | |
kind | String | required |
assayRef | UUID | |
assayName | String | |
status | InstrumentStatus {ONLINE,IDLE,BUSY,OFFLINE,ERROR} | required |
site | String | |
lastSeen | Instant |
The event producer
Reached at /services/servicekafkaproducer/api with your access token. com.osmosys.servicekafkaproducer.
This service carries no stored entities of its own.
Service reference
serviceactivitymanagement
Handles the activity domain. Reached at /services/serviceactivitymanagement/api; runs on port 8098.
Reached at /services/serviceactivitymanagement/api with your access token. com.osmosys.serviceactivitymanagement.
ActivityEvent /api/activity-events
| Field | Type | |
|---|---|---|
eventUid | UUID | required |
category | ActivityCategory {ACTUATION,INGESTION,WORKFLOW,CONFIG_CHANGE,DATASET,JOB,SAMPLE,AUTH,ALERT} | required |
action | String | required |
actor | String | |
targetRef | String | |
targetName | String | |
summary | String | |
severity | ActivitySeverity {INFO,NOTICE,WARNING} | required |
occurredAt | Instant | required |
Service reference
servicealertmanagement
Handles the alert domain. Reached at /services/servicealertmanagement/api; runs on port 8097.
Reached at /services/servicealertmanagement/api with your access token. com.osmosys.servicealertmanagement.
Alert /api/alerts
| Field | Type | |
|---|---|---|
alertUid | UUID | required |
severity | AlertSeverity {INFO,WARNING,CRITICAL} | required |
source | AlertSource {INSTRUMENT,JOB,NODE,STORAGE,WORKFLOW,PIPELINE,INGESTION} | required |
title | String | required |
detail | String | |
status | AlertStatus {OPEN,ACKNOWLEDGED,RESOLVED} | required |
entityRef | String | |
entityName | String | |
raisedAt | Instant | required |
acknowledgedBy | String | |
acknowledgedAt | Instant | |
resolvedAt | Instant |
Service reference
serviceassaymanagement
Handles the assay domain. Reached at /services/serviceassaymanagement/api; runs on port 8088.
Reached at /services/serviceassaymanagement/api with your access token. com.osmosys.serviceassaymanagement.
AnalysisJob /api/analysis-jobs
| Field | Type | |
|---|---|---|
jobRef | UUID | required |
sampleRef | UUID | |
experimentRef | UUID | |
sampleName | String | |
analyte | Analyte {DNA,RNA,PROTEIN} | |
status | JobStatus {QUEUED,RUNNING,COMPLETED,FAILED,PAUSED} | required |
progress | Double | |
stage | String | |
stageIndex | Integer | |
submittedBy | String | |
startedAt | Instant | |
finishedAt | Instant | |
etaMinutes | Integer | |
gpus | Integer | |
cpus | Integer | |
nodeName | String | |
priority | JobPriority {LOW,NORMAL,HIGH} |
Assay /api/assaies
| Field | Type | |
|---|---|---|
assayId | UUID | required |
name | String | required |
fullName | String | required |
category | String | required |
description | String | |
status | AssayStatus {ACTIVE,PLANNED,RETIRED} | required |
accentColor | String | |
iconKey | String |
Platform /api/platforms
| Field | Type | |
|---|---|---|
platformId | UUID | required |
name | String | required |
vendor | String | |
kind | PlatformKind {PIPELINE,INSTRUMENT} | required |
Service reference
servicecomponentmanagement
Handles the component domain. Reached at /services/servicecomponentmanagement/api; runs on port 8102.
Reached at /services/servicecomponentmanagement/api with your access token. com.osmosys.servicecomponentmanagement.
ComponentParam /api/component-params
| Field | Type | |
|---|---|---|
name | String | required |
kind | ParamKind {INPUT,OUTPUT,PARAM,REFERENCE} | required |
dataType | DataType {BCL,FASTQ,BAM,CRAM,VCF,BED,INTERVAL_LIST,REFERENCE_GENOME,ENUM,STRING,INTEGER,FLOAT,BOOLEAN,FILE} | required |
mandatory | Boolean | required |
cliFlag | String | |
defaultValue | String | |
allowedValues | String | |
description | String | |
ordinal | Integer |
ComponentTool /api/component-tools
| Field | Type | |
|---|---|---|
componentUid | String | required |
name | String | required |
toolFamily | String | |
version | String | required |
assayType | AssayType {NGS,CRYOEM,SPATIAL,PROTEOMICS,GENERAL} | required |
stage | ComponentStage {BASECALL_CONVERSION,QC,TRIMMING,ALIGNMENT,POST_ALIGNMENT,VARIANT_CALLING,CNV,ANNOTATION,UTILITY} | required |
dockerImage | String | required |
commandTemplate | String | |
description | String | |
documentationUrl | String | |
cpuCores | Integer | |
memGb | Integer | |
gpu | Boolean | |
status | ComponentStatus {DRAFT,PUBLISHED,DEPRECATED} | required |
source | ComponentSource {STANDARD,CUSTOM} | required |
pipelineTags | String | |
registeredBy | String | |
registeredAt | Instant | required |
Service reference
servicecomputemanagement
Handles the compute domain. Reached at /services/servicecomputemanagement/api; runs on port 8089.
Reached at /services/servicecomputemanagement/api with your access token. com.osmosys.servicecomputemanagement.
ComputeNode /api/compute-nodes
| Field | Type | |
|---|---|---|
nodeId | String | required |
name | String | required |
role | NodeRole {GPU,CPU,MEMORY,STORAGE} | required |
online | Boolean | required |
gpuUtil | Double | |
cpuUtil | Double | |
memUtil | Double | |
jobCount | Integer |
ResourcePool /api/resource-pools
| Field | Type | |
|---|---|---|
poolId | String | required |
name | String | required |
kind | ResourceKind {GPU,CPU,MEMORY,STORAGE,NETWORK} | required |
unit | String | |
total | Double | required |
used | Double | required |
Service reference
serviceconfigmanagement
Handles the config domain. Reached at /services/serviceconfigmanagement/api; runs on port 8095.
Reached at /services/serviceconfigmanagement/api with your access token. com.osmosys.serviceconfigmanagement.
ConfigFile /api/config-files
| Field | Type | |
|---|---|---|
fileUid | UUID | required |
name | String | required |
path | String | |
format | ConfigFormat {YAML,JSON,TOML,INI,ENV} | required |
content | TextBlob | |
version | Integer | required |
category | String | |
assayRef | UUID | |
updatedBy | String | |
updatedAt | Instant | required |
active | Boolean |
Service reference
servicecryoemmanagement
Handles the cryoem domain. Reached at /services/servicecryoemmanagement/api; runs on port 8086.
Reached at /services/servicecryoemmanagement/api with your access token. com.osmosys.servicecryoemmanagement.
Project /api/projects
| Field | Type | |
|---|---|---|
directoryToWatch | String | required |
projectUid | String | required |
gainReferenceFilePath | String | required |
pSizeA | Double | required |
csMm | Double | required |
totalDoseEPerA2 | Long | required |
accelerationKv | Integer | required |
projectJson | String | required |
Workspace /api/workspaces
| Field | Type | |
|---|---|---|
workspaceUid | String | required |
workspaceJson | String | required |
projectWorkspaceUid | String | required |
Service reference
servicedatasetmanagement
Handles the dataset domain. Reached at /services/servicedatasetmanagement/api; runs on port 8094.
Reached at /services/servicedatasetmanagement/api with your access token. com.osmosys.servicedatasetmanagement.
Dataset /api/datasets
| Field | Type | |
|---|---|---|
datasetUid | UUID | required |
name | String | required |
description | String | |
assayRef | UUID | |
assayName | String | |
experimentRef | UUID | |
workflowRunRef | UUID | |
modality | Modality {STRUCTURE,SEQUENCE,IMAGE,SPECTRUM,TABULAR,MIXED} | required |
format | String | |
sizeBytes | Long | |
fileCount | Integer | |
storageUri | String | |
status | DatasetStatus {DRAFT,CURATED,PUBLISHED,ARCHIVED} | required |
fairScore | Integer | |
schemaVersion | String | |
owner | String | |
createdAt | Instant | required |
metadataJson | String |
DatasetLineage /api/dataset-lineages
| Field | Type | |
|---|---|---|
lineageUid | UUID | required |
nodeType | LineageNodeType {INSTRUMENT,INGESTION,EXPERIMENT,WORKFLOW,DATASET} | required |
refUid | UUID | |
label | String | required |
ordering | Integer | required |
status | String |
Service reference
serviceexperimentmanagement
Handles the experiment domain. Reached at /services/serviceexperimentmanagement/api; runs on port 8091.
Reached at /services/serviceexperimentmanagement/api with your access token. com.osmosys.serviceexperimentmanagement.
Experiment /api/experiments
| Field | Type | |
|---|---|---|
experimentUid | UUID | required |
name | String | required |
assayRef | UUID | required |
assayName | String | required |
status | ExperimentStatus {PLANNED,ACTIVE,COMPLETED,ABORTED} | required |
sampleRef | String | |
operator | String | |
startedAt | Instant | |
completedAt | Instant | |
notes | String |
Service reference
servicefilewatchermanagement
Handles the filewatcher domain. Reached at /services/servicefilewatchermanagement/api; runs on port 8105.
Reached at /services/servicefilewatchermanagement/api with your access token. com.osmosys.servicefilewatchermanagement.
This service carries no stored entities of its own.
Service reference
servicefleetmanagement
Handles the fleet domain. Reached at /services/servicefleetmanagement/api; runs on port 8090.
Reached at /services/servicefleetmanagement/api with your access token. com.osmosys.servicefleetmanagement.
FleetNode /api/fleet-nodes
| Field | Type | |
|---|---|---|
nodeUid | UUID | required |
name | String | required |
site | String | required |
formFactor | FormFactor {LAB_IN_A_RACK,LAB_IN_A_CART,LAB_IN_A_BOX,LAB_IN_A_CUBE,MASS_STORAGE_EXTENDER} | required |
status | NodeStatus {ONLINE,DEGRADED,OFFLINE} | required |
gpuModel | String | |
gpus | Integer | required |
cpuCores | Integer | required |
memTb | Integer | required |
storageTb | Integer | required |
gpuUtil | Double | |
cpuUtil | Double | |
memUtil | Double | |
storageUtil | Double | |
jobs | Integer |
Service reference
serviceinstrumentmanagement
Handles the instrument domain. Reached at /services/serviceinstrumentmanagement/api; runs on port 8092.
Reached at /services/serviceinstrumentmanagement/api with your access token. com.osmosys.serviceinstrumentmanagement.
ActuationCommand /api/actuation-commands
| Field | Type | |
|---|---|---|
commandUid | UUID | required |
action | ActuationAction {START,STOP,PAUSE,RESUME,CALIBRATE,ACQUIRE,RESET} | required |
params | String | |
status | CommandStatus {PENDING,SENT,ACKED,FAILED} | required |
issuedBy | String | |
issuedAt | Instant | required |
IngestionSession /api/ingestion-sessions
| Field | Type | |
|---|---|---|
sessionUid | UUID | required |
experimentRef | UUID | |
datasetName | String | required |
bytesIngested | Long | |
fileCount | Integer | |
status | IngestionStatus {RECEIVING,COMPLETE,FAILED} | required |
startedAt | Instant | required |
completedAt | Instant |
Instrument /api/instruments
| Field | Type | |
|---|---|---|
instrumentUid | UUID | required |
name | String | required |
vendor | String | |
model | String | |
kind | String | required |
assayRef | UUID | |
assayName | String | |
status | InstrumentStatus {ONLINE,IDLE,BUSY,OFFLINE,ERROR} | required |
site | String | |
lastSeen | Instant |
Service reference
serviceinventorymanagement
Handles the inventory domain. Reached at /services/serviceinventorymanagement/api; runs on port 8101.
Reached at /services/serviceinventorymanagement/api with your access token. com.osmosys.serviceinventorymanagement.
InventoryItem /api/inventory-items
| Field | Type | |
|---|---|---|
itemUid | UUID | required |
name | String | required |
category | ItemCategory {REAGENT,KIT,CONSUMABLE,ENZYME,ANTIBODY,BUFFER,GRID,FLOWCELL} | required |
vendor | String | |
catalogNo | String | |
lotNo | String | |
quantity | Double | required |
unit | String | |
reorderLevel | Double | |
location | String | |
status | ItemStatus {IN_STOCK,LOW,OUT_OF_STOCK,EXPIRED,ON_ORDER} | required |
custodian | String | |
receivedAt | Instant | |
expiresAt | Instant |
Service reference
servicekafkaconsumermanagement
Handles the service-kafkaconsumer domain. Reached at /services/servicekafkaconsumermanagement/api; runs on port 8106.
Reached at /services/servicekafkaconsumermanagement/api with your access token. com.osmosys.servicekafkaconsumermanagement.
This service carries no stored entities of its own.
Service reference
servicekafkaproducer
Handles the kafka-producer domain. Reached at /services/servicekafkaproducer/api; runs on port 8085.
Reached at /services/servicekafkaproducer/api with your access token. com.osmosys.servicekafkaproducer.
This service carries no stored entities of its own.
Service reference
servicelimsmanagement
Handles the lims domain. Reached at /services/servicelimsmanagement/api; runs on port 8104.
Reached at /services/servicelimsmanagement/api with your access token. com.osmosys.servicelimsmanagement.
LimsConnection /api/lims-connections
| Field | Type | |
|---|---|---|
connectionUid | String | required |
name | String | required |
provider | LimsProvider {BENCHLING,LABWARE,STARLIMS,GENOLOGICS,CUSTOM} | required |
baseUrl | String | required |
direction | LimsConnectionDirection {INBOUND,OUTBOUND,BIDIRECTIONAL} | required |
authMode | LimsAuthMode {API_KEY,OAUTH2,BASIC,NONE} | required |
status | LimsConnectionStatus {CONNECTED,DEGRADED,DISCONNECTED,DISABLED} | required |
entityScopes | String | |
lastCheckedAt | Instant | |
notes | String | |
mappingsJson | String |
LimsDeliveryRule /api/lims-delivery-rules
| Field | Type | |
|---|---|---|
ruleUid | String | required |
connectionRef | String | required |
connectionName | String | required |
pipelineRef | String | required |
pipelineName | String | required |
stepOrder | Integer | required |
outputName | String | required |
dataType | String | required |
targetEntity | LimsTargetEntity {SAMPLE,EXPERIMENT,ATTACHMENT} | required |
enabled | Boolean | required |
lastFiredAt | Instant | |
notes | String |
LimsSyncEvent /api/lims-sync-events
| Field | Type | |
|---|---|---|
eventUid | String | required |
connectionRef | String | required |
connectionName | String | required |
provider | LimsProvider {BENCHLING,LABWARE,STARLIMS,GENOLOGICS,CUSTOM} | required |
direction | LimsSyncDirection {INBOUND,OUTBOUND} | required |
entityType | LimsEntityType {SAMPLE,EXPERIMENT} | required |
entityRef | String | required |
entityName | String | required |
externalId | String | required |
externalUrl | String | |
operation | LimsSyncOperation {CREATE,UPDATE,DELETE} | required |
status | LimsSyncStatus {SUCCESS,FAILED,PENDING,CONFLICT} | required |
message | String | |
occurredAt | Instant | required |
payloadJson | String |
Service reference
servicengsmanagement
Handles the ngs domain. Reached at /services/servicengsmanagement/api; runs on port 8087.
Reached at /services/servicengsmanagement/api with your access token. com.osmosys.servicengsmanagement.
Job /api/jobs
| Field | Type | |
|---|---|---|
jobUid | String | required |
jobType | String | required |
jobJson | String | |
status | JobStatus {STARTED,STOPPED,FAILED} | required |
startTime | Instant | required |
stopTime | Instant | |
metadata | String |
Project /api/projects
| Field | Type | |
|---|---|---|
projectDirectory | String | required |
projectUid | String | required |
basecallModel | String | |
projectJson | String |
Workspace /api/workspaces
| Field | Type | |
|---|---|---|
workspaceUid | String | required |
workspaceJson | String | |
operation | String |
Service reference
servicepipelinemanagement
Handles the pipeline domain. Reached at /services/servicepipelinemanagement/api; runs on port 8103.
Reached at /services/servicepipelinemanagement/api with your access token. com.osmosys.servicepipelinemanagement.
Pipeline /api/pipelines
| Field | Type | |
|---|---|---|
pipelineUid | String | required |
name | String | required |
description | String | |
assayType | AssayType {NGS,CRYOEM,SPATIAL,PROTEOMICS,GENERAL} | required |
status | PipelineStatus {DRAFT,PUBLISHED,ARCHIVED} | required |
createdBy | String | |
createdAt | Instant | required |
PipelineStep /api/pipeline-steps
| Field | Type | |
|---|---|---|
stepOrder | Integer | required |
alias | String | |
componentUid | String | required |
componentName | String | |
bindingsJson | String |
Service reference
serviceprojectmanagement
Handles the project domain. Reached at /services/serviceprojectmanagement/api; runs on port 8099.
Reached at /services/serviceprojectmanagement/api with your access token. com.osmosys.serviceprojectmanagement.
Milestone /api/milestones
| Field | Type | |
|---|---|---|
milestoneUid | UUID | required |
name | String | required |
status | MilestoneStatus {PLANNED,IN_PROGRESS,DONE,MISSED} | required |
ordering | Integer | required |
dueDate | Instant | |
completedAt | Instant |
Project /api/projects
| Field | Type | |
|---|---|---|
projectUid | UUID | required |
code | String | required |
name | String | required |
description | String | |
program | String | |
status | ProjectStatus {PROPOSED,ACTIVE,ON_HOLD,COMPLETED,ARCHIVED} | required |
phase | String | |
lead | String | |
team | String | |
assayRef | UUID | |
priority | ProjectPriority {LOW,NORMAL,HIGH,CRITICAL} | required |
budgetUsd | Double | |
startedAt | Instant | required |
targetDate | Instant | |
completedAt | Instant |
Service reference
servicesamplemanagement
Handles the sample domain. Reached at /services/servicesamplemanagement/api; runs on port 8096.
Reached at /services/servicesamplemanagement/api with your access token. com.osmosys.servicesamplemanagement.
Sample /api/samples
| Field | Type | |
|---|---|---|
sampleUid | UUID | required |
name | String | required |
sampleType | SampleType {CELL_LINE,TISSUE,PLASMA,SERUM,DNA,RNA,PROTEIN,CRYO_GRID,LIBRARY} | required |
organism | String | |
source | String | |
prepProtocol | String | |
status | SampleStatus {REGISTERED,IN_PREP,READY,IN_ANALYSIS,ARCHIVED,DEPLETED} | required |
custodian | String | |
assayRef | UUID | |
experimentRef | UUID | |
storageLocation | String | |
volumeUl | Double | |
concentrationNgUl | Double | |
collectedAt | Instant | required |
notes | String |
Service reference
servicestoragemanagement
Handles the storage domain. Reached at /services/servicestoragemanagement/api; runs on port 8100.
Reached at /services/servicestoragemanagement/api with your access token. com.osmosys.servicestoragemanagement.
StorageVolume /api/storage-volumes
| Field | Type | |
|---|---|---|
volumeUid | UUID | required |
name | String | required |
tier | StorageTier {HOT,WARM,COLD,ARCHIVE} | required |
backend | StorageBackend {S3,NFS,CEPH,LUSTRE,TAPE} | required |
status | VolumeStatus {ONLINE,DEGRADED,READONLY,OFFLINE} | required |
capacityTb | Double | required |
usedTb | Double | required |
mountPath | String | |
site | String | |
datasetCount | Integer | |
quotaTb | Double | |
custodian | String | |
lastScrubAt | Instant |
Service reference
serviceworkflowmanagement
Handles the workflow domain. Reached at /services/serviceworkflowmanagement/api; runs on port 8093.
Reached at /services/serviceworkflowmanagement/api with your access token. com.osmosys.serviceworkflowmanagement.
WorkflowRun /api/workflow-runs
| Field | Type | |
|---|---|---|
runUid | UUID | required |
experimentRef | UUID | |
datasetRef | String | |
status | RunStatus {QUEUED,RUNNING,COMPLETED,FAILED,CANCELED} | required |
submittedBy | String | |
submittedAt | Instant | required |
finishedAt | Instant | |
targetCluster | String | |
progress | Double |
WorkflowStep /api/workflow-steps
| Field | Type | |
|---|---|---|
stepUid | UUID | required |
name | String | required |
stepType | StepType {ANALYSIS,TRANSFORM} | required |
ordering | Integer | required |
status | StepStatus {PENDING,RUNNING,COMPLETED,FAILED,SKIPPED} | required |
tool | String | |
startedAt | Instant | |
durationSec | Integer |
WorkflowTemplate /api/workflow-templates
| Field | Type | |
|---|---|---|
templateUid | UUID | required |
name | String | required |
assayRef | UUID | |
category | WorkflowCategory {ANALYSIS,TRANSFORM} | required |
description | String | |
stepCount | Integer |
Reference
Assistant tools
These are the actions the assistant can take on your behalf. Each is a precise, named operation; the assistant picks the right one for your question and answers from its result. Descriptions are generated from the source. For how to use them conversationally, see Using the assistant.
Looking things up
| Tool | What it returns |
|---|---|
list_instruments | Enumerate scientific instruments currently registered in the control plane, with last-seen timestamps and current health state. |
query_rest_logs | Query RESTful API logs over a time window. Optional filters on HTTP status code and endpoint path glob (e.g. "/api/v1/*"). Returns up to `limit` most-relevant records. |
query_kafka_consumer_log | Read entries from the Kafka consumer application log, optionally filtered to a single `topic`. Returns up to `limit` records. |
query_file_watcher_events | Query the downstream PARQUET store of persisted file-watcher events (created/modified/moved/deleted) under an optional path prefix. Returns up to `limit` records. NOTE: this store is written by a separate consumer and is frequently NOT populated. For the raw file-watcher activity itself -- what files each watcher detected/created, event kinds, validation outcomes, token fetches, and HTTP posts -- use query_file_watcher_logs (it reads the watcher process logs directly). |
query_file_watcher_logs | Read the raw stdout/stderr logs emitted by the Rust file-watcher binaries (the file-watcher side): cryoem_instrument, cryoem_cryosparc, ngs_ont_instrument, ngs_ont_basecall, ngs_ont_experiment, ngs_ont_workspace, and ngs_illumina_instrument. Logs are stored gzip-compressed -- one file per watcher (<watcher>.log.gz) -- and are read on the fly without decompressing to disk. This tool returns the watcher *process* log lines themselves (startup config, token fetches, per-file events, validation, HTTP posts, retries, and errors). PREFER THIS TOOL for questions about recent file-watcher activity -- what files a watcher detected or created, event kinds, the latest experiment/run, validation results, or post failures. It reads the watcher output directly and is the live source when the parquet event store (query_file_watcher_events) is not populated. Args: watcher: restrict to one watcher by name (omit to search all). An unknown name fails with the list of available watchers. since / until: bound results by each line's leading [RFC3339-UTC] timestamp. Continuation lines of a multi-line record (e.g. bcl-convert stderr) inherit the timestamp of the record they belong to. level: keep only DEBUG/INFO/WARN/ERROR records. contains: case-sensitive substring the raw line must contain. regex: Python regular expression the raw line must match (re.search). newest_first: return the most recent matches (default). Set False for the oldest matches. Across multiple watchers, results are merged by timestamp. limit: maximum number of lines to return. Keep this modest -- every record counts against the model's context window. Prefer narrowing with level/contains/regex and a small limit over pulling large windows; DEBUG payload lines are large, so use level="INFO" for summaries. Returns up to `limit` records, each {watcher, ts (ISO-8601 or null), level, message}. Messages longer than WATCHER_LOG_MAX_MSG (default 500) chars are truncated with a marker. |
query_db_transactions | Run a read-only SQL query (a single SELECT) against one of the management service databases. These are separate Postgres databases -- you cannot join across them in one query; pick the one that holds the data you need. database="ngs" -> ngs_management (public schema): project(id, project_directory, project_uid, basecall_model, project_json) workspace(id, workspace_uid, workspace_json, operation, project_id -> project.id) job(id, job_uid, job_type, job_json, status, start_time, stop_time, metadata, workspace_id -> workspace.id) status enum: 'STARTED' | 'STOPPED' | 'FAILED' database="cryoem" -> cryoem_management (public schema): project(id, directory_to_watch, project_uid, gain_reference_file_path, p_size_a, cs_mm, total_dose_e_per_a_2, acceleration_kv, project_json) workspace(id, workspace_uid, workspace_json, project_workspace_uid, project_id -> project.id) Connection: set OSMOSYS_NGS_DSN / OSMOSYS_CRYOEM_DSN to DSNs backed by a role with SELECT-only grants. |
plot_timeseries | Render a time-series chart from data the model has already gathered. Returns a Vega-Lite spec the host UI can display inline. Multiple series are distinguished by the `label` field. |
get_deepvariant_run_status | Poll a DeepVariant job submitted via submit_deepvariant_run. Returns the job metadata with `status` in {running, succeeded, failed} and, on failure, the tail of stderr to help diagnose. |
get_alphafold_run_status | Poll an AlphaFold 3 job submitted via submit_alphafold_run. Returns the job metadata with `status` in {running, succeeded, failed}, the container exit code on completion, and a tail of stderr if it failed. |
get_colabfold_run_status | DEMO STUB. Does not run ColabFold. Values are synthetic and must not be used. Poll a ColabFold job submitted via submit_colabfold_run. While running, returns the current pipeline `stage` (MSA generation, template search, model inference, relaxation) and `approx_seconds_remaining`. When complete, returns `status: "succeeded"` with per-model pLDDT and pTM scores in `models[]` (rank 1 is the top model), plus `msa_depth` and `template_count`. |
get_relion_job_status | Poll any RELION job submitted via submit_relion_motioncorr_ctf, submit_relion_autopick, submit_relion_class2d, or submit_relion_refine3d. Returns the job metadata with `status` in {running, succeeded, failed}, the process exit code on completion, and -- on failure -- the tail of stderr to help diagnose (e.g. a missing CTFFIND binary or too few MPI ranks). Args: job_id: the id returned by a submit_relion_* tool. pipeline: optional stage ("motioncorr_ctf" | "autopick" | "class2d" | "refine3d") to skip the cross-stage lookup. job_ids are globally unique, so this is only an optimization. |
Running analyses
| Tool | What it does |
|---|---|
submit_deepvariant_run | Submit a DeepVariant variant-calling run (https://github.com/google/deepvariant) as a background container job and return the job_id immediately. WGS runs can take many hours -- the agent should poll get_deepvariant_run_status with the returned job_id rather than waiting on this call. Args: reads: absolute host path to a BAM/CRAM (index .bai/.crai must sit alongside). ref: absolute host path to a reference FASTA (.fai must sit alongside). model_type: WGS | WES | PACBIO | ONT_R104 | HYBRID_PACBIO_ILLUMINA. sample_name: optional sample name to embed in the VCF. regions: optional region string, e.g. "chr20:10,000,000-10,010,000". num_shards: parallelism (defaults to host CPU count). use_gpu: pass --gpus all (requires the GPU-tagged image). |
submit_alphafold_run | Submit an AlphaFold 3 prediction (https://github.com/google-deepmind/alphafold3) as a background container job. AF3 runs take minutes (inference only) to several hours (full pipeline with MSA search). Poll get_alphafold_run_status with the returned job_id; do not block on this call. Supply exactly one input: input_json: dict matching the AF3 schema, e.g. {"name": "...", "sequences": [{"protein": {"id": ["A"], "sequence": "..."}}], "modelSeeds": [1], "dialect": "alphafold3", "version": 1} json_path: absolute host path to an existing fold_input.json. Phases (both default on): run_data_pipeline: CPU MSA/template search; requires OSMOSYS_ALPHAFOLD_DB_DIR. run_inference: GPU structure prediction; requires OSMOSYS_ALPHAFOLD_MODEL_DIR (weights obtained from DeepMind via their access form). use_gpu: pass --gpus all to the runtime (turn off for CPU-only data pipeline). |
submit_colabfold_run | DEMO STUB. Does not run ColabFold. Values are synthetic and must not be used. Submit a ColabFold (https://github.com/sokrypton/ColabFold) structure prediction. By default this blocks until completion and returns the full result -- per-model pLDDT and pTM scores in `models[]` (rank 1 is the top model), MSA depth, template count, and output paths. Set wait_for_completion=False to run asynchronously and poll with get_colabfold_run_status instead. Args: sequence: single-letter amino-acid sequence. name: short identifier used in output filenames. num_models: how many ranked models to produce (1-5 typical). num_recycles: number of recycling iterations. msa_mode: MSA generation strategy. use_templates: include PDB template search. use_gpu: run inference on GPU. wait_for_completion: block and return final results (default), or return a job_id immediately for asynchronous polling. |
build_relion_movies_star | Build the RELION movies STAR file that seeds the pipeline (the "Import" step that runs BEFORE submit_relion_motioncorr_ctf). It scans a directory for raw movie files matching a regex and writes a RELION 3.1+/4.x/5.x movies.star (a data_optics block carrying the acquisition parameters + a data_movies block listing the movies), then returns the path to feed into the `movies_star` argument of submit_relion_motioncorr_ctf. This is synchronous (just a directory scan + file write) -- it returns the result directly, with num_movies and a sample so you can sanity-check the regex matched the expected files BEFORE launching motion correction. Because the acquisition parameters live in the STAR's optics group, supply the SAME angpix/voltage/cs/amp_contrast you will pass to submit_relion_motioncorr_ctf. Args: movies_dir: directory holding the raw movies (searched recursively by default -- EPU/SerialEM sessions nest movies under subdirectories). angpix: the raw movie/detector pixel size in Angstroms (rlnMicrographOriginalPixelSize). Required. pattern: Python regex matched (re.search, case-sensitive) against each file's path RELATIVE to movies_dir. The default matches common movie extensions; narrow it to disambiguate movies from other files in the dir (e.g. r"Fractions.*\\.mrc$" or r"_fractions\\.tiff$"). voltage: acceleration voltage in kV (rlnVoltage). cs: spherical aberration in mm (rlnSphericalAberration). amp_contrast: amplitude contrast fraction (rlnAmplitudeContrast), e.g. 0.1. optics_group_name: name for the single optics group (letters/digits/_). recursive: walk subdirectories (default True) or only the top level. output_star: where to write the STAR. Defaults to a fresh RELION_JOBS_DIR/import/<id>/movies.star (on the shared data root, so a remote RELION host sees it). Movie paths inside are ABSOLUTE so they resolve regardless of the job's working directory. Returns {movies_star, num_movies, optics_group, angpix, voltage, cs, amp_contrast, pattern, movies_dir, sample (first few matched paths)}. |
submit_relion_motioncorr_ctf | Submit the FIRST step of a RELION (https://relion.readthedocs.io) cryo-EM single-particle pipeline: beam-induced motion correction followed by CTF estimation, chained in one background job. Returns a job_id immediately -- poll get_relion_job_status; do not block on this call. Motion correction (relion_run_motioncorr) aligns/averages and dose-weights the movie frames, writing corrected_micrographs.star; CTF estimation (relion_run_ctffind, which drives an external CTFFIND4 binary) then estimates the defocus on those micrographs, writing micrographs_ctf.star -- the input to submit_relion_autopick. The two run as `motioncorr && ctffind`, so the job fails if either stage fails. Args: movies_star: STAR file listing the raw movies, from build_relion_movies_star. angpix: pixel size in Angstroms. Always supplied (passed to both stages); required whenever the input lacks optics-group pixel-size metadata. voltage: acceleration voltage in kV (also the CTFFIND --HT). dose_per_frame: electron dose per movie frame (e/A^2); drives dose weighting. pre_exposure: pre-exposure dose before the first frame (e/A^2). cs: spherical aberration in mm (CTFFIND --CS). amp_contrast: amplitude contrast fraction (CTFFIND --AmpCnst), e.g. 0.1. backend: "own" for RELION's built-in (CPU) implementation, or "motioncor2" to drive an external MotionCor2 binary (GPU; needs motioncor2_exe). motioncor2_exe: path to the MotionCor2 executable (backend="motioncor2" only; falls back to OSMOSYS_RELION_MOTIONCOR2_EXE / the RELION_MOTIONCOR2_EXECUTABLE env var). gain_ref: optional gain-reference MRC for detector normalization. first_frame_sum / last_frame_sum: 1-based frame range to sum (-1 = last). patch_x / patch_y: number of patches for local (anisotropic) alignment. group_frames: average this many frames before estimating shifts (low-dose). bin_factor: Fourier-crop binning of the corrected micrographs. bfactor: B-factor (px^2) applied during alignment. dose_weighting: apply dose-dependent frequency weighting (recommended). save_no_dw: also write the non-dose-weighted average (used for CTF if present). ctffind_exe: path to the CTFFIND4 executable (falls back to OSMOSYS_RELION_CTFFIND_EXE / the RELION_CTFFIND_EXECUTABLE env var). The CTF stage cannot run without it. box: CTFFIND box size in pixels. res_min / res_max: low/high resolution limits (A) for the CTF fit. dfmin / dfmax: defocus search range in Angstroms. fstep: defocus search step in Angstroms. dast: expected astigmatism in Angstroms (restraint). do_phaseshift: also estimate a phase shift (Volta phase plate data). threads: per-process threads (--j) for both stages. nr_mpi: MPI ranks; >1 runs the _mpi binaries under mpirun. gpu_ids: colon-separated GPU ids for MotionCor2 (e.g. "0:1:2:3"); ignored by RELION's own (CPU) implementation. |
submit_relion_autopick | Pick particles from CTF-estimated micrographs using RELION's reference-FREE Laplacian-of-Gaussian (LoG) auto-picker (relion_autopick --LoG), then -- by default -- extract the picked particles into a particles STAR file ready for submit_relion_class2d (relion_preprocess --extract). Returns a job_id immediately; poll get_relion_job_status. LoG picking needs NO template/reference (it is a blob detector tuned by a particle-diameter band), and is CPU-only -- it does not use the GPU. Args: micrographs_star: micrographs_ctf.star from submit_relion_motioncorr_ctf. diam_min / diam_max: min/max particle diameter in Angstroms (the LoG band). angpix: pixel size (A). Usually read from the STAR optics; pass it only for bare-MRC inputs that lack optics metadata. lowpass: low-pass filter in Angstroms applied before picking. adjust_threshold: shift the picking threshold (+ = fewer, - = more picks). upper_threshold: reject candidates above this score (removes contamination). invert_picks: pick white-on-black particles (--Log_invert). shrink: micrograph down-scaling for speed (0 = automatic). pickname: suffix tag for the coordinate files (_<pickname>.star); letters, digits, and underscores only. extract: also extract the picked particles (default True). Set False to pick only (coordinates), e.g. to inspect before committing to a box size. extract_size: extraction box size in pixels (before any rescaling). rescale: rescale extracted particles to this box size in pixels (binning for speed); None keeps extract_size. bg_radius: background-mask radius in pixels (in extract_size pixels) for normalization; None uses 0.375*extract_size. invert_contrast: invert densities so particles are white (recommended). normalize: normalize each extracted particle (background to mean 0, sd 1). nr_mpi: MPI ranks for picking and extraction (>1 runs the _mpi binaries). |
submit_relion_class2d | Run reference-free 2D classification on extracted particles (relion_refine), to sort them into 2D class averages for cleaning the data set. Returns a job_id immediately; poll get_relion_job_status. RELION 4+/5 defaults 2D classification to the gradient-driven VDAM algorithm (--grad), which is faster and needs no class-number tuning. VDAM CANNOT run under multi-rank MPI (RELION aborts), so this tool requires nr_mpi=1 when algorithm="vdam"; use algorithm="em" for the classic expectation-maximization method if you want to parallelize across MPI ranks. Args: particles_star: particles.star from submit_relion_autopick (the extracted set). num_classes: number of 2D classes (K). particle_diameter: circular-mask diameter in Angstroms (~ longest dimension). algorithm: "vdam" (gradient, default, single-rank) or "em" (classic EM). iters: iterations; default 200 for VDAM, 25 for EM. tau2_fudge: regularization parameter T (2 is typical for 2D). psi_step: in-plane angular sampling in degrees. offset_range / offset_step: translational search range/step in pixels. ctf: do CTF correction (recommended). ctf_intact_first_peak: leave the CTF untouched up to the first peak (for very low-resolution / phase-plate data); requires ctf=True. zero_mask: mask the solvent area to zero (recommended). flatten_solvent: flatten the solvent during reconstruction. pool: particles processed together per thread (memory/speed trade-off). pad: Fourier padding factor (2 = standard). threads: per-process threads (--j). nr_mpi: MPI ranks (must be 1 for VDAM; >1 allowed only for EM). gpu_ids: "" = use all available GPUs (auto), "0:1" = specific devices, or None to run on CPU only. |
submit_relion_refine3d | Reconstruct a 3D map by gold-standard 3D auto-refinement (relion_refine_mpi --auto_refine --split_random_halves): the two independent half-sets are refined to convergence and their FSC gives an unbiased resolution estimate. Returns a job_id immediately; poll get_relion_job_status. This is the "3D reconstruction" step after 2D classification. It REQUIRES an initial 3D reference (ref_map) -- typically a de-novo initial model or a low-pass-filtered prior map -- and runs ONLY under MPI with at least 3 ranks (1 master + 2 half-set leaders), so nr_mpi must be >= 3. Args: particles_star: particles.star of the cleaned particle set (post-2D-selection). ref_map: initial 3D reference map (.mrc). Mandatory for auto-refine. particle_diameter: circular-mask diameter in Angstroms. ini_high: initial low-pass filter (A) applied to the reference (limits starting-model bias). symmetry: point-group symmetry, e.g. "C1", "D2", "C6", "I". ctf: do CTF correction (recommended). ref_uncorrected: set ONLY if ref_map has NOT been CTF-amplitude corrected (adds --ctf_uncorrected_ref). RELION assumes the reference IS corrected by default, so a map from a prior RELION reconstruction needs this left False. healpix_order: initial angular-sampling order (2 = 7.5 degrees). auto_local_healpix_order: order at which to switch to local angular searches. offset_range / offset_step: translational search range/step in pixels. low_resol_join_halves: low-pass (A) at which to join the half-maps for the first few iterations (helps small/elongated particles); None = off. zero_mask: mask the solvent area to zero (recommended). flatten_solvent: flatten the solvent during reconstruction. pool / pad: particles-per-thread pool and Fourier padding factor. threads: per-process threads (--j). nr_mpi: MPI ranks; must be >= 3 for split-half auto-refine. gpu_ids: "" = use all available GPUs (auto), "0:1" = specific devices, or None to run on CPU only. |
Reference
Changelog
Notable changes across the Osmosys codebase, drawn from the commit history and rebuilt whenever these docs are regenerated. Routine maintenance and internal refactoring are left out.