marspylib¶
Mars - Molecule ARchive Suite
Pure-Python library for reading and writing Mars Molecule Archives (.yama)
and utility functions for working with them — no JVM, no Fiji install
required. Full API reference documentation is available at
marspylib.readthedocs.io. Complete
Molecule ARchive Suite (Mars) documentation including a guide to working
with mars data structures in python can be found at
mars-docs.
Installation¶
conda install -c conda-forge marspylib
or
pip install marspylib
Only dependency requirements are numpy, pandas, and matplotlib — this
package can be installed in any plain Python/conda environment and does not
require Fiji, ImageJ, or a JVM of any kind.
Reading/writing archives on S3 (see Using S3 below) needs
boto3, kept as an optional extra so it’s not a hard dependency for users
who only ever work with local files:
pip install marspylib[s3]
Usage¶
import marspylib.yama as yama
archive = yama.open("experiment.yama")
print(archive.properties.number_of_molecules)
for molecule in archive:
df = molecule.table # pandas.DataFrame
if "accepted" in molecule.tags:
...
molecule = archive["some-uid"] # random access by UID
archive.save("experiment_out.yama")
Creating a new archive¶
You don’t need Fiji to build a .yama from scratch — construct
Properties, a Molecule (or one of its subclasses, matching the archive
type — see Supported archive types), and
MarsMetadata/regions/positions as needed, then put() each record into a
fresh Archive and save:
import pandas as pd
import marspylib.yama as yama
properties = yama.Properties(archive_type=yama.ARCHIVE_TYPES["SingleMoleculeArchive"])
archive = yama.Archive(properties, metadata={}, molecules={})
metadata = yama.MarsMetadata(microscope="Nikon Ti2", source_directory="/data/2024-01-15")
archive.put_metadata(metadata)
for i in range(3):
molecule = yama.SingleMolecule(metadata_uid=metadata.uid)
molecule.add_tag("accepted")
molecule.parameters["dwell"] = 5.5
molecule.table = pd.DataFrame({
"T": [0.0, 1.0, 2.0],
"Intensity": [10.1 + i, 10.5 + i, 10.9 + i],
})
archive.put(molecule) # molecule.uid was auto-generated -- see below
archive.save("new_experiment.yama")
Leaving uid unset when constructing a Molecule/MarsMetadata (as above)
auto-generates one in mars-core’s own format — the same Base58 encoding of a
random UUID that Fiji itself uses (MarsMath.getUUID58()), so records you
create in Python get UIDs that are unique right alongside ones created in
Fiji, which matters if the two ever get merged into the same archive. You
can also call the generator directly, or supply your own uid=:
yama.new_molecule_uid() # e.g. "8eoHfZ1GdvNBeWGNNDY3hp" -- full-length
yama.new_metadata_uid() # e.g. "mnAgQYYn63" -- fixed 10 characters, matching mars-core
molecule = yama.SingleMolecule(uid="my-own-id") # or supply your own
yama.ARCHIVE_TYPES maps every supported short name (SingleMoleculeArchive,
DnaMoleculeArchive, DefaultMoleculeArchive, ObjectArchive,
TransverseFlowArchive) to the archive-type string mars-core expects — use
whichever matches the Molecule subclass you’re building records with.
Removing records¶
archive.remove("some-uid") # drop a molecule
archive.remove_metadata("meta-uid") # drop a metadata record
archive.save()
For a .yama.store archive, remove()/remove_metadata() delete the
underlying file immediately (matching mars-core), not deferred to the next
save(); for a single-file archive it just drops the record from memory,
taking effect the next time you save().
Virtual archives (.yama.store)¶
Large archives saved from Fiji as a .yama.store directory (rather than a
single .yama file) open the same way — yama.open() detects it from the
path — but records are loaded lazily, one at a time, instead of all at once:
archive = yama.open("experiment.yama.store") # nothing loaded yet
print(len(archive), "molecules")
for molecule in archive: # each one read from disk as you reach it
if "accepted" in molecule.tags:
...
molecule = archive["some-uid"] # read once, then cached for reuse
Writing works the same archive.save(...) call as single-file archives,
dispatched purely on whether the target path ends in .yama.store:
archive["some-uid"].add_tag("reviewed")
archive.save() # update the .yama.store in place
archive.save("subset.yama") # flatten into a single file instead
archive.save("copy.yama.store") # write out as a separate virtual store
A .yama.store archive must still exist on disk for as long as you’re
reading from it — a molecule you haven’t touched yet is only read from its
file the moment you access it, so don’t delete, move, or let a temporary
directory holding one go out of scope while you’re still using the archive.
Molecules in a .yama.store are loaded lazily and LRU-cached (default 128
at a time) — mutating one you already hold a reference to (like
molecule.add_tag(...) above) is safe as long as it’s still cached when you
call save(). If you’re touching more distinct molecules than that between
editing one and saving, or want to be certain, use archive.put(molecule)
to pin it so it’s guaranteed to be written regardless of cache pressure:
molecule = archive["some-uid"]
molecule.add_tag("reviewed")
archive.put(molecule) # pins it -- guaranteed to persist even if evicted
# ... touch hundreds of other molecules ...
archive.save()
put()/put_metadata() are also how you add a brand-new record (a UID
that wasn’t already in the archive) — see the mapping table below.
Using S3¶
Archives (single .yama files or .yama.store virtual archives) can be
opened and saved directly on S3 or any S3-compatible endpoint, with no local
copy needed — open_s3()/archive.save_s3() mirror open()/archive.save()
exactly, including the lazy loading and immediate-delete-on-remove()
behavior for .yama.store.
A location is three things: the endpoint host (server_address), the
bucket, and the key (path to the file or .yama.store directory within
the bucket). Passing these three separately is the recommended way — it’s
unambiguous, unlike a combined URL, which can be misread if a bucket or host
name happens to contain a stray .:
import marspylib.yama as yama
archive = yama.open_s3(server_address="storage.example.org",
bucket="my-bucket", key="path/to/experiment.yama")
archive["some-uid"].add_tag("reviewed")
archive.save() # no args -- saves back to the same S3 location it was opened from
archive.save_s3(server_address="storage.example.org",
bucket="my-bucket", key="path/to/experiment.yama.store")
server_address defaults to https; for a plain-http (no TLS) endpoint,
either pass secure=False or just include the scheme directly in
server_address ("https://..."/"http://..." are both accepted and
override secure accordingly) — whichever you find easier to read.
If you already work with combined virtual-hosted-style URLs
(https://<bucket>.s3.<server_address>/<key>), those are also supported, as
a single location argument in place of the three separate fields:
archive = yama.open_s3("https://my-bucket.s3.storage.example.org/path/to/experiment.yama")
archive.save_s3("https://my-bucket.s3.storage.example.org/path/to/experiment.yama.store")
The .s3. in that combined form is a fixed separator token, not necessarily
part of the real endpoint host — it’s stripped back off when parsed, so both
forms above point at the same server_address="storage.example.org". (Real
AWS S3 is a case where the actual endpoint host, s3.amazonaws.com,
separately happens to also start with s3. — that’s a coincidence of AWS’s
own naming, not something this parsing depends on.)
Both forms work with .yama.store too, dispatched the same way save()
dispatches locally — by whether key ends in .yama.store.
Credentials are never passed through this API directly. By default,
open_s3()/save_s3() resolve credentials the standard boto3 way —
environment variables, ~/.aws/credentials, SSO cache, IAM role — the same
chain the AWS SDK for Java (and so Fiji/mars-core) uses, so if your
credentials are already configured locally, nothing further needs to be set
up here. To use a specific profile or override resolution, pass your own
boto3.Session:
import boto3
session = boto3.Session(profile_name="my-profile")
archive = yama.open_s3(server_address="storage.example.org", bucket="my-bucket",
key="path/to/experiment.yama", session=session)
Supported archive types¶
Every SingleMoleculeArchive, DnaMoleculeArchive, and
DefaultMoleculeArchive opened from Fiji gets its own Molecule subclass
(SingleMolecule, DnaMolecule, DefaultMolecule) — currently identical
to plain Molecule, since in mars-core these three only differ in name, not
in the fields they store today. Every archive type gets a dedicated Python
class uniformly, even when (as here) there’s nothing archive-type-specific
about it yet, so a future mars-core field addition to any one of them only
means filling in that one class here, not restructuring anything.
Two other archive types carry one extra field per record, so they get their
own Molecule subclasses:
ObjectArchive (mars-core’s object package) uses MartianObject,
which adds .shapes: a dict[int, PeakShape] mapping timepoint → tracked
polygon outline.
archive = yama.open("tracked_objects.yama") # archive_type is ObjectArchive
obj = archive["some-uid"]
shape = obj.shapes[12] # PeakShape at timepoint 12
shape.x, shape.y # coordinate arrays (same length)
obj.shapes[13] = yama.PeakShape(x=[...], y=[...])
archive.save()
TransverseFlowArchive (the separate mars-transverseflow module) uses
TransverseFlowMolecule, which adds .replication_fork_shapes: a
dict[int, ReplicationForkShape] mapping timepoint → replication fork
geometry (parental/leading/lagging strand outlines, each with an optional
per-channel intensity profile).
archive = yama.open("forks.yama") # archive_type is TransverseFlowArchive
mol = archive["some-uid"]
shape = mol.replication_fork_shapes[7]
shape.parental_x, shape.parental_y # not-yet-replicated duplex outline
shape.leading_x, shape.leading_y # leading-strand daughter outline
shape.lagging_x, shape.lagging_y # lagging-strand daughter outline
shape.leading_intensity["GFP"][7] # per-channel intensity, keyed by coordinate
Coming from Java/Groovy/Fiji: method name mapping¶
This library’s classes cover the same data as mars-core’s Java classes, but
follow Python naming conventions (snake_case, attributes instead of
getters/setters) rather than mirroring the Java API name-for-name. This
applies equally to Groovy scripts written against Mars in Fiji’s script
editor — Groovy calls Java methods with identical syntax (molecule.addTag(tag)
is valid in both), so everywhere this table says “Java” it means Groovy too.
Where a Java/Groovy call was a getter/setter pair, the Python side is
usually just a plain attribute you can read or assign directly.
Archive (de.mpg.biochem.mars.molecule.MoleculeArchive)
Java |
Python |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(metadata collection) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The molecule_*/metadata_* lookups above are the same “don’t load the full
record just to check one field” trick Java gets from its index: for a
.yama.store opened with an indexes.sml present, they answer from that
index alone (tags/channel/image/metadata linkage only — not parameters,
regions, positions, or the table, which aren’t indexed in mars-core either).
For a single-file archive, or a record you’ve already loaded/put() in this
session, they just read the in-memory object directly, which is equally
cheap. archive[uid].tags etc. still works too, of course — it’s just not
free for a not-yet-touched .yama.store record the way these are.
Molecule / MarsRecord (de.mpg.biochem.mars.molecule.Molecule, MarsRecord)
Java |
Python |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MarsMetadata (de.mpg.biochem.mars.metadata.MarsMetadata)
Java |
Python |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
MarsRegion / MarsPosition (de.mpg.biochem.mars.util)
Java |
Python |
|---|---|
|
|
|
|
MoleculeArchiveProperties
Java |
Python |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
document access |
|
Note archive.save() recomputes properties.tag_set/channel_set/
parameter_set/region_set/position_set/table_column_set/
segment_table_names/number_of_molecules/number_of_metadata from the
archive’s actual current contents before writing (mirroring mars-core’s own
rebuildIndexes()), so these always reflect reality after a put() even
though nothing updates them incrementally as you go.