Leo He

Introduction

At first glance, the question "Are these two molecules the same?" seems trivial. A trained chemist can often tell at a glance. But for a computer processing millions of molecules in a chemical database, this question — known as graph isomorphism — is one of the most fundamental and computationally demanding problems in cheminformatics.

Two molecules may be drawn differently, have their atoms numbered differently, or be represented in entirely different file formats, yet they are chemically identical. To determine this algorithmically, we need robust and efficient methods for testing whether two molecular graphs are isomorphic.

This is where Nauty and Bliss come in — two of the most powerful and widely used algorithms for graph canonical labeling and graph isomorphism testing. In this post, I will explain what these algorithms do, how they work, and why they are indispensable tools in computational chemistry.

1. The Graph Isomorphism Problem

1.1 Definition

Two graphs \(G_1 = (V_1, E_1)\) and \(G_2 = (V_2, E_2)\) are isomorphic if there exists a bijection \(f: V_1 \rightarrow V_2\) such that \((u, v) \in E_1\) if and only if \((f(u), f(v)) \in E_2\).

In plain language: two graphs are isomorphic if one can be transformed into the other by relabeling vertices. The structure (connectivity) is identical; only the labels differ.

1.2 Computational Complexity

The graph isomorphism problem occupies a peculiar position in complexity theory:

• It is in NP (a solution can be verified in polynomial time).
• It is not known to be NP-complete.
• It is not known to be in P (solvable in polynomial time) for general graphs.
• László Babai showed in 2015 that it can be solved in quasipolynomial time — a landmark theoretical result.

In practice, however, algorithms like Nauty and Bliss solve graph isomorphism extremely efficiently for the vast majority of real-world graphs, including molecular graphs.

1.3 Canonical Labeling as a Solution

Instead of directly testing whether two graphs are isomorphic (a pairwise comparison), a more powerful approach is to compute a canonical form (also called a canonical labeling) for each graph.

A canonical labeling is a unique, deterministic relabeling of the graph's vertices such that:

Two graphs are isomorphic if and only if their canonical forms are identical.

This reduces the isomorphism test to a simple string (or hash) comparison, enabling efficient indexing, deduplication, and lookup of molecules in large databases.

2. Nauty: The Pioneer

2.1 Overview

Nauty (No AUTomorphisms, Yes?) was developed by Brendan McKay at the Australian National University, first released in 1981 and continuously improved over four decades. It is arguably the most influential software for graph automorphism and canonical labeling ever created.

Nauty computes:

1. The automorphism group of a graph (all symmetries).
2. A canonical labeling of the graph.

2.2 Core Algorithm: Individualization-Refinement

Nauty's algorithm is based on the individualization-refinement paradigm. Here is a high-level description:

Step 1: Vertex Partition Refinement

Start with an initial partition of vertices (e.g., based on vertex degree or atom type in a molecular graph). Then iteratively refine this partition:

• For each cell (group) in the partition, split it based on the adjacency pattern to other cells.
• Continue until the partition is equitable — no further splitting is possible.

This refinement process is similar to the Weisfeiler-Leman (WL) algorithm (specifically, the 1-dimensional WL or "color refinement"). It can distinguish many non-isomorphic graphs but is not sufficient for all cases.

Step 2: Individualization (Branching)

If the equitable partition is not discrete (i.e., some cells still contain multiple vertices), Nauty selects a vertex from a non-singleton cell and individualizes it — placing it in its own cell. This creates a branch in a search tree.

After individualization, refinement is applied again. This process continues recursively until a discrete partition is obtained (every cell has exactly one vertex), which corresponds to a specific labeling of the graph.

Step 3: Pruning and Canonical Selection

The search tree can be exponentially large in theory. Nauty employs several powerful pruning strategies:

• Automorphism detection: When two branches lead to the same partition, the mapping between them reveals an automorphism. This automorphism is used to prune equivalent branches.
• Canonical selection rule: Among all the discrete partitions (leaves of the search tree), Nauty selects one as the canonical form using a deterministic comparison rule (e.g., lexicographic ordering of the adjacency matrix).

Step 4: Output

Nauty outputs:

• A set of generators for the automorphism group.
• The canonical labeling (a permutation that transforms the original graph into its canonical form).

2.3 Performance

Nauty is remarkably fast in practice:

• For most molecular graphs (typically < 100 vertices, sparse, with few symmetries), Nauty runs in microseconds.
• It has been tested on graphs with millions of vertices.
• Performance degrades on highly symmetric graphs (e.g., complete graphs, hypercubes), where the automorphism group is very large.

2.4 Traces

In recent versions, Nauty includes an alternative algorithm called Traces, also by McKay and Piperno. Traces uses a different search strategy (based on breadth-first rather than depth-first exploration of the search tree) and often outperforms classic Nauty on large, sparse graphs.

3. Bliss: A Modern Alternative

3.1 Overview

Bliss (A Tool for Computing Automorphism Groups and Canonical Labelings of Graphs) was developed by Tommi Junttila and Petteri Kaski at Aalto University (Helsinki University of Technology), first published in 2007.

Like Nauty, Bliss computes automorphism groups and canonical labelings. It was designed with a focus on:

• Large, sparse graphs (common in many applications including chemistry).
• Memory efficiency.
• Clean, modern C++ implementation.

3.2 Algorithm

Bliss follows the same general individualization-refinement framework as Nauty but introduces several key innovations:

3.2.1 Refined Partition Refinement

Bliss uses an optimized version of partition refinement that is particularly efficient for sparse graphs. The refinement procedure is implemented with careful attention to data structures and cache performance.

3.2.2 Search Tree Pruning

Bliss introduces novel pruning techniques:

• Component-based pruning: Exploits the connected component structure of the graph to prune the search tree more aggressively.
• Orbit-based pruning: Uses the orbits of the automorphism group (computed so far) to avoid redundant branches.

3.2.3 Cell Selection Heuristics

The choice of which vertex to individualize (the cell selector) has a dramatic impact on the size of the search tree. Bliss experiments with several heuristics:

• First non-singleton cell (F): Simple but often effective.
• Largest first (FL): Choose the largest non-singleton cell.
• First largest (LF): Among the first occurring cells, choose the largest.
• First maximum connected (FMC): Consider connectivity in the selection.

Different heuristics perform better on different graph families. For molecular graphs, the default heuristics typically work well.

3.3 Performance Comparison with Nauty

Extensive benchmarks show:

For typical molecular graphs (< 200 atoms, sparse, few symmetries), both Nauty and Bliss are extremely fast and the performance difference is negligible.

4. Application in Chemical Molecule Isomorphism Detection

4.1 Molecules as Graphs

A molecule can be naturally represented as a labeled graph:

• Vertices = atoms, labeled with element type (C, N, O, S, ...), charge, isotope, and other properties.
• Edges = bonds, labeled with bond type (single, double, triple, aromatic) and stereochemistry.

Two molecules are identical (ignoring conformation) if and only if their labeled molecular graphs are isomorphic.

4.2 Why Canonical Labeling Matters for Chemistry

4.2.1 Database Deduplication

Chemical databases like PubChem (> 100 million compounds), ChEMBL, and ZINC need to detect duplicate entries. Computing the canonical form of each molecule allows:

• O(1) lookup using hash tables.
• Efficient deduplication across databases.

Without canonical labeling, checking if a new molecule already exists would require pairwise comparison with every existing entry — computationally infeasible at scale.

4.2.2 Canonical SMILES Generation

SMILES (Simplified Molecular Input Line Entry System) is the most widely used linear notation for molecules. However, the same molecule can be written as many different SMILES strings. For example, ethanol can be written as:

CCO
OCC
C(O)C
[CH3][CH2][OH]

A canonical SMILES is a unique SMILES string for each molecule. Generating canonical SMILES requires solving the graph canonical labeling problem. Under the hood, tools like RDKit and OpenBabel use canonical labeling algorithms (often inspired by or directly using Nauty/Bliss) to produce canonical SMILES.

4.2.3 InChI Generation

The IUPAC International Chemical Identifier (InChI) is another canonical molecular representation. The InChI algorithm uses its own canonical labeling procedure (based on Morgan-like algorithms with tie-breaking rules) to assign a unique identifier to each molecule.

4.2.4 Substructure Search

While substructure search (subgraph isomorphism) is a different problem (NP-complete in general), canonical labeling plays a supporting role:

• Precomputing canonical forms of molecular fragments.
• Pruning search spaces using automorphism information.
• Accelerating pattern matching in chemical databases.

4.2.5 Symmetry Perception

The automorphism group computed by Nauty/Bliss reveals the symmetries of a molecule. This has direct chemical significance:

• Equivalent atoms: Atoms in the same orbit of the automorphism group are chemically equivalent (e.g., the three hydrogens in a methyl group).
• NMR prediction: Equivalent atoms produce the same NMR signal. Symmetry perception is essential for predicting NMR spectra.
• Reaction site enumeration: When enumerating possible reaction sites, equivalent positions need to be identified to avoid redundant calculations.
• Combinatorial library design: Symmetry information reduces redundancy in virtual library enumeration.

4.3 How Canonical Labeling Works for Molecules

When applying Nauty/Bliss to molecular graphs, the process is adapted as follows:

1. Construct the vertex-colored, edge-colored graph:

   • Each atom becomes a vertex colored by its element type (and possibly charge, isotope, etc.).
   • Each bond becomes an edge colored by its bond type.

2. Initialize the partition:

   • The initial partition groups atoms by their color (element type). For example, all carbons in one cell, all oxygens in another, etc.
   • This dramatically reduces the search space compared to starting with a uniform partition.

3. Run canonical labeling:

   • Nauty/Bliss refines the partition, individualizes vertices, and searches for the canonical form.
   • Edge colors are incorporated into the refinement step (adjacency to different bond types provides additional distinguishing information).

4. Output the canonical permutation:

   • The canonical permutation defines a unique ordering of atoms.
   • This ordering is used to generate a canonical SMILES, canonical adjacency matrix, or canonical hash.

4.4 Integration in Cheminformatics Toolkits

Several major cheminformatics libraries use or are inspired by Nauty/Bliss:

Some research tools and specialized applications directly call Nauty or Bliss via their C/C++ APIs for high-performance canonical labeling of molecular graphs.

5. Worked Example

Consider two representations of the same molecule, isobutane (2-methylpropane):

Representation 1:

Atom 1: C, connected to 2, 3, 4
Atom 2: C, connected to 1
Atom 3: C, connected to 1
Atom 4: C, connected to 1

Representation 2:

Atom A: C, connected to B
Atom B: C, connected to A, C, D
Atom C: C, connected to B
Atom D: C, connected to B

These look different, but the underlying graphs are isomorphic (both are the star graph \(K_{1,3}\) with all carbon labels).

Step 1: Partition Refinement

• Initial partition: {1, 2, 3, 4} (all carbons) for Representation 1.
• After refinement by degree: {1} (degree 3), {2, 3, 4} (degree 1).
• The partition is now equitable.

For Representation 2:

• After refinement: (degree 3), {A, C, D} (degree 1).

Step 2: Canonical Form

Both representations produce the same canonical adjacency structure:

• One central vertex connected to three leaf vertices.

Nauty/Bliss would output the same canonical hash for both → isomorphism confirmed.

Bonus — Automorphism Group:

The automorphism group is \(S_3\) (symmetric group on 3 elements), corresponding to the 6 permutations of the three equivalent methyl groups. This tells us the three terminal carbons are chemically equivalent.

6. Performance on Molecular Graphs

Molecular graphs have properties that make them particularly amenable to canonical labeling:

As a result, Nauty and Bliss can canonicalize typical drug-like molecules in microseconds. Even for large molecules like natural products or peptides with hundreds of atoms, the computation rarely exceeds milliseconds.

Benchmark Results

In a typical benchmark on a dataset of ~1 million drug-like molecules:

(These numbers are approximate and depend on hardware and implementation details.)

7. Beyond Simple Isomorphism

7.1 Stereoisomer Handling

Canonical labeling of the molecular graph alone does not distinguish stereoisomers (e.g., R vs. S enantiomers, E vs. Z isomers). To handle stereochemistry:

• Stereochemical information must be encoded as additional vertex or edge labels.
• CIP (Cahn-Ingold-Prelog) priority rules are applied after canonical labeling.
• The canonical SMILES or InChI includes stereochemical descriptors.

Nauty/Bliss handles this naturally: stereochemistry is encoded as additional colors, making stereoisomers non-isomorphic in the colored graph.

7.2 Tautomer Handling

Tautomers (molecules that differ by proton position) have different molecular graphs but are often considered "the same" compound. Canonical labeling must be combined with tautomer standardization for proper deduplication.

7.3 Molecular Similarity and Subgraph Isomorphism

While Nauty/Bliss solves exact isomorphism, chemistry often requires:

• Subgraph isomorphism (substructure search): Is molecule A a substructure of molecule B? This is solved by algorithms like VF2 or Ullmann's algorithm, not by canonical labeling.
• Maximum common subgraph (MCS): What is the largest common substructure between two molecules? This is NP-hard in general.
• Graph similarity: How similar are two molecular graphs? Approached via fingerprints, graph kernels, or graph neural networks.

Canonical labeling complements these methods but does not replace them.

8. Practical Usage

8.1 Using Nauty

Nauty is distributed as C source code. A basic usage example:

#include "nauty.h"

// Define graph with MAXN vertices
// Set up adjacency using ADDONEEDGE macro
// Call densenauty() or sparsenauty()
// Compare canonical forms

Nauty also provides command-line tools:

• dreadnaut: Interactive interface for graph manipulation and canonical labeling.
• shortg: Removes isomorphic duplicates from a list of graphs.
• geng: Generates all non-isomorphic graphs of a given size.

8.2 Using Bliss

Bliss provides a clean C++ API:

#include "bliss/graph.hh"

bliss::Graph g(n_vertices);
g.change_color(vertex_id, color);  // Set atom type
g.add_edge(v1, v2);               // Add bond

// Compute canonical form
const unsigned int* canonical_labeling = g.canonical_form(stats, NULL, NULL);

8.3 Python Wrappers

Both Nauty and Bliss have Python bindings:

• pynauty: Python interface to Nauty.
• PyBliss: Python interface to Bliss.
• igraph: The igraph library includes Bliss integration via Graph.canonical_permutation().
• NetworkX: Can interface with external tools for canonical labeling.

Example using igraph:

import igraph as ig

# Create molecular graph
g1 = ig.Graph(edges=[(0,1), (1,2), (1,3)])
g1.vs["label"] = ["C", "C", "C", "C"]

g2 = ig.Graph(edges=[(0,1), (0,2), (0,3)])
g2.vs["label"] = ["C", "C", "C", "C"]

# Get canonical permutation (uses Bliss internally)
perm1 = g1.canonical_permutation(color=g1.vs["label"])
perm2 = g2.canonical_permutation(color=g2.vs["label"])

# Compare canonical forms
g1_canon = g1.permute_vertices(perm1)
g2_canon = g2.permute_vertices(perm2)

print(g1_canon.isomorphic(g2_canon))  # True

9. Comparison with Other Approaches

10. Conclusion

Nauty and Bliss are foundational tools in computational graph theory, and their impact on cheminformatics cannot be overstated. By efficiently solving the graph canonical labeling problem, they enable:

• Unique molecular identification through canonical SMILES, InChI, and hash-based lookups.
• Database deduplication across chemical databases with millions or billions of entries.
• Symmetry perception for NMR prediction, reaction enumeration, and conformational analysis.
• High-throughput virtual screening where rapid isomorphism checks are essential.

For typical molecular graphs — small, sparse, and with limited symmetry — both algorithms perform in microseconds, making them practical for real-time applications. While the two algorithms share the same theoretical framework (individualization-refinement), they differ in implementation details and performance characteristics. Bliss tends to excel on large, sparse graphs with its memory-efficient design, while Nauty (especially with Traces) is often faster on dense or highly symmetric graphs.

As chemical databases continue to grow and computational chemistry demands ever-higher throughput, the importance of efficient canonical labeling algorithms will only increase. Whether you are building a chemical registration system, mining the patent literature, or developing a new virtual screening platform, understanding Nauty and Bliss is essential knowledge for any computational chemist or cheminformatician.

References

1. McKay, B. D., & Piperno, A. (2014). Practical graph isomorphism, II. Journal of Symbolic Computation, 60, 94–112.
2. Junttila, T., & Kaski, P. (2007). Engineering an efficient canonical labeling tool for large and sparse graphs. Proceedings of the Meeting on Algorithm Engineering & Experiments (ALENEX), 135–149.
3. Weininger, D. (1988). SMILES, a chemical language and information system. Journal of Chemical Information and Computer Sciences, 28(1), 31–36.
4. Heller, S. R., McNaught, A., Pletnev, I., Stein, S., & Tchekhovskoi, D. (2015). InChI, the IUPAC International Chemical Identifier. Journal of Cheminformatics, 7(1), 23.
5. McKay, B. D. (1981). Practical graph isomorphism. Congressus Numerantium, 30, 45–87.
6. Babai, L. (2016). Graph isomorphism in quasipolynomial time. Proceedings of the 48th Annual ACM Symposium on Theory of Computing (STOC), 684–697.
7. Cordella, L. P., Foggia, P., Sansone, C., & Vento, M. (2004). A (sub)graph isomorphism algorithm for matching large graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(10), 1367–1372.

Introduction

Chemical structure recognition — the task of converting graphical depictions of molecular structures into machine-readable formats (such as SMILES, InChI, or MOL files) — is a long-standing challenge at the intersection of computer vision, cheminformatics, and document analysis. Millions of chemical structures are locked inside scanned journal articles, patents, and hand-drawn lab notebooks. Unlocking these structures in a machine-readable form is essential for drug discovery, reaction database construction, and automated literature mining.

In this blog post, I provide a comprehensive overview of the major approaches to chemical structure recognition, tracing the evolution from early rule-based systems to modern deep learning architectures.

1. Problem Definition

A chemical structure diagram typically consists of:

• Atoms represented by element symbols (C, N, O, S, etc.), with carbon atoms often implied at vertices.
• Bonds depicted as lines (single), double lines (double), triple lines (triple), wedge/dash (stereochemistry).
• Rings and functional groups arranged in 2D space.
• Charges, radicals, isotope labels, and other annotations.

The goal is to take a raster image (or sometimes a vector graphic) of such a diagram and produce a canonical, machine-readable molecular representation.

2. Rule-Based and Classical Approaches

2.1 Early Pioneers

The earliest systems for Optical Chemical Structure Recognition (OCSR) emerged in the 1990s and relied heavily on hand-crafted image processing pipelines.

Key systems include:

• KEKULÉ (1990s): One of the first published systems. It used thinning algorithms to extract a skeleton from the binary image, then applied geometric heuristics to identify bonds and nodes.
• CLiDE (Chemical Literature Data Extraction): Developed by Peter Johnson's group, CLiDE used vectorization of the raster image, followed by rule-based classification of line segments as bonds and text regions as atom labels.
• OSRA (Optical Structure Recognition Application): An open-source tool developed at the NIH/NCI. OSRA combines image segmentation, thinning, line detection, and OCR (using GOCR/Tesseract) with a set of heuristic rules to assemble the molecular graph. It has been one of the most widely used tools in the field.

2.2 General Pipeline of Rule-Based Methods

Most classical approaches follow a common pipeline:

1. Preprocessing: Noise removal, binarization, and deskewing of the input image.
2. Segmentation: Separating text (atom labels) from graphical elements (bonds, rings).
3. OCR for Atom Labels: Recognizing element symbols and associated charges/hydrogens using optical character recognition.
4. Bond Detection: Extracting line segments from the thinned skeleton and classifying them as single, double, triple, wedge, or dashed bonds.
5. Graph Assembly: Connecting detected atoms and bonds into a molecular graph, resolving implicit carbon atoms, and applying valence rules for validation.
6. Output Generation: Converting the graph into SMILES, InChI, or MOL format.

2.3 Limitations

• Highly sensitive to image quality (resolution, noise, color).
• Fragile in the presence of non-standard drawing styles.
• Difficulty handling stereochemistry, abbreviations (e.g., "Ac", "Boc"), and Markush structures.
• Extensive manual tuning required for each sub-module.

3. Machine Learning–Enhanced Classical Methods

Before the deep learning revolution, several systems began incorporating machine learning components into the traditional pipeline.

3.1 Imago

Imago, developed by EPAM/GGA Software, improved on earlier tools by using trainable classifiers for character recognition and bond-type classification. It combined adaptive image preprocessing with learned models for symbol segmentation, achieving better robustness than purely rule-based approaches.

3.2 MolVec

MolVec, developed by Daniel Lowe and others, adopted a hybrid approach that uses heuristic vectorization combined with learned components for OCR and bond classification. It remains one of the most competitive open-source tools for OCSR.

3.3 ChemInfty and Other Tools

Several other systems (ChemInfty, chemOCR) used SVMs or random forests for local classification tasks (e.g., distinguishing atom labels from bond lines) while retaining rule-based graph assembly.

4. Deep Learning–Based Approaches

The advent of deep learning has fundamentally transformed chemical structure recognition. Modern approaches can be broadly categorized into image-to-sequence models, image-to-graph models, and segmentation-based models.

4.1 Image-to-Sequence (Encoder–Decoder) Models

These methods treat chemical structure recognition as an image captioning problem: encode the image with a CNN (or Vision Transformer), then decode a sequence — typically a SMILES string or InChI — using an autoregressive language model.

4.1.1 DECIMER

DECIMER (Deep Learning for Chemical Image Recognition), developed by Rajan et al., uses an Inception-based or EfficientNet encoder coupled with a Transformer decoder to directly predict SMILES from images. DECIMER has been iteratively improved:

• DECIMER 1.0: CNN encoder + LSTM decoder trained on a large corpus of synthetic chemical images.
• DECIMER 2.0: Adopted a Transformer-based decoder and significantly expanded the training data, improving accuracy on diverse image styles.
• DECIMER Segmentation: A dedicated module for detecting and segmenting individual chemical structures from full document pages.

4.1.2 SMILES-based Seq2Seq Models

Several research groups have independently developed encoder-decoder models that predict SMILES:

• Staker et al. (2019): Used a ResNet encoder and LSTM decoder with attention, training on millions of RDKit-rendered images.
• Khokhlov et al.: Applied a similar architecture with data augmentation strategies (varying fonts, line widths, rotations) to improve generalization.

4.1.3 SwinOCSR

SwinOCSR uses a Swin Transformer as the image encoder paired with a Transformer decoder. By leveraging the hierarchical feature extraction of Swin Transformers, it achieves strong performance on both clean and noisy images.

4.1.4 MolScribe

MolScribe (by Qian et al., 2023) is an image-to-graph approach that first uses an encoder-decoder framework to predict atom and bond information, then assembles a molecular graph. It combines the benefits of sequence prediction with explicit graph reasoning:

• Predicts atom positions and types.
• Predicts bonds between detected atoms.
• Applies chemical validity constraints during post-processing.

MolScribe achieves state-of-the-art accuracy on several benchmarks.

4.1.5 Img2Mol

Img2Mol (Clevert et al., 2021) takes a slightly different approach: instead of decoding SMILES token by token, it encodes the image into a continuous molecular fingerprint representation (CDDD — Continuous and Data-Driven Descriptors), then retrieves or decodes the closest valid molecule. This approach is more robust to minor image variations but depends on the quality of the learned embedding space.

4.2 Image-to-Graph Models

Rather than generating a linear sequence, these approaches directly predict a molecular graph (atoms as nodes, bonds as edges).

4.2.1 Graph-Based Reconstruction

Some methods detect atoms (keypoints) and bonds (edges) using object detection or keypoint detection networks, then assemble them into a graph:

• AtomLenz / ChemGrapher: Uses a two-stage approach — first detect atom positions and labels, then classify bonds between nearby atoms.
• MolGrapher (2024): A more recent approach that combines a keypoint detector with a graph neural network for bond prediction, achieving competitive results.

4.2.2 Advantages of Graph-Based Methods

• Explicit spatial reasoning about atom positions.
• Easier to enforce chemical validity constraints (valence rules, aromaticity).
• Better handling of stereochemistry (wedge bonds, E/Z isomerism).

4.3 Segmentation-Based Approaches

Some methods frame the problem as semantic segmentation:

• Segment pixels into categories: atom, single bond, double bond, triple bond, wedge bond, etc.
• Post-process segmentation maps to extract graph structure.

These methods can leverage powerful segmentation architectures (U-Net, Mask R-CNN) but often struggle with the combinatorial complexity of graph assembly.

5. Training Data and Data Augmentation

A critical factor in the success of deep learning approaches is training data. Most modern systems rely on synthetic data generation:

1. SMILES/MOL → Image rendering: Tools like RDKit, Indigo, CDK, or ChemDraw are used to render millions of molecular structures as images.
2. Augmentation strategies:
   • Varying bond lengths, angles, and line widths.
   • Random fonts for atom labels.
   • Adding noise, JPEG artifacts, and background textures.
   • Simulating scanning artifacts and low resolution.
   • Adding Markush structures, R-group labels, and reaction arrows as distractors.
3. Real-world datasets for evaluation:
   • USPTO: Patent chemical structure images.
   • UOB, CLEF, JPO: Benchmark datasets with paired images and ground-truth structures.
   • Staker benchmark: Curated set of challenging real-world images.
   • RealWorldMol: A more recent benchmark with diverse image sources.

The domain gap between synthetic and real images remains a key challenge.

6. Evaluation Metrics

Evaluating OCSR systems is non-trivial because there are multiple valid SMILES representations for the same molecule. Common metrics include:

• Exact match accuracy: The predicted SMILES/InChI is canonicalized and compared with the ground truth. This is the strictest metric.
• Tanimoto similarity: Molecular fingerprint similarity between predicted and ground-truth molecules, providing a "soft" evaluation.
• InChI match: Comparing InChI strings layer by layer (connectivity, stereochemistry, charge, etc.).
• Graph edit distance: Measuring the minimum number of atom/bond insertions, deletions, and substitutions to transform the predicted graph into the ground truth.

7. Handling Special Cases

7.1 Stereochemistry

Recognizing wedge and dash bonds for R/S chirality, as well as E/Z double bond geometry, remains challenging. MolScribe and some graph-based methods handle this explicitly; many sequence-based methods rely on SMILES stereochemistry tokens (@ and /) which can be difficult to predict correctly.

7.2 Abbreviated Groups

Chemical structures often contain abbreviations like "OMe", "Boc", "Ac", "Ph", etc. Systems need either:

• A dictionary of common abbreviations, or
• The ability to learn abbreviation patterns from data.

DECIMER and MolScribe have made progress on this front by including abbreviated structures in training data.

7.3 Markush Structures and R-Groups

Patent documents frequently contain generic (Markush) structures with variable R-groups. This remains an open problem, as the output representation must capture combinatorial variability, going beyond standard SMILES/InChI.

7.4 Reaction Schemes

Full reaction scheme recognition (reactants → reagents → products) requires:

• Detecting individual molecular structures.
• Recognizing arrows and reaction conditions.
• Parsing the overall reaction topology.

Tools like RxnScribe extend chemical structure recognition to full reaction diagrams.

8. Comparison of Major Tools

9. Current Challenges and Future Directions

9.1 Domain Adaptation

The gap between synthetic training images and real-world images (from old scanned papers, hand-drawn structures, or low-resolution patents) remains the biggest obstacle. Self-supervised pretraining, domain adaptation, and few-shot learning are promising research directions.

9.2 Multimodal and LLM-based Approaches

With the rise of large vision-language models (GPT-4V, Gemini, LLaVA), there is growing interest in using general-purpose multimodal models for chemical structure recognition. Early experiments show that these models can recognize simple structures, but they still lag behind specialized tools on complex molecules. Fine-tuning multimodal LLMs on chemical data is an exciting frontier.

9.3 End-to-End Document Processing

Moving beyond isolated structure recognition to full-page or full-document processing — extracting structures, reactions, tables, and text in context — is increasingly important. This requires integrating OCSR with layout analysis, table parsing, and NLP.

9.4 3D Structure and Conformations

Current OCSR deals with 2D depictions. Extending recognition to 3D molecular representations or even predicting 3D conformations from 2D diagrams is a nascent area.

9.5 Interactive and Human-in-the-Loop Systems

For practical applications, systems that allow chemists to verify and correct recognition results interactively can dramatically improve usability and accuracy.

10. Conclusion

Chemical structure recognition has come a long way from the hand-crafted rule-based systems of the 1990s to today's deep learning models that can achieve >90% exact match accuracy on many benchmarks. The field has benefited enormously from:

• Large-scale synthetic data generation using cheminformatics toolkits.
• Encoder-decoder architectures borrowed from image captioning and machine translation.
• Graph-based reasoning that respects the discrete, combinatorial nature of molecules.

Yet significant challenges remain — handling diverse image quality, stereochemistry, abbreviated groups, and integration into full document understanding pipelines. As multimodal foundation models continue to improve and more annotated real-world data becomes available, we can expect chemical structure recognition to become an increasingly reliable and indispensable tool in the chemist's digital toolkit.

If you found this post useful, feel free to share it. For questions or suggestions, leave a comment below!

References

1. Rajan, K., Zielesny, A., & Steinbeck, C. (2020). DECIMER: towards deep learning for chemical image recognition. Journal of Cheminformatics, 12, 65.
2. Qian, Y. et al. (2023). MolScribe: Robust Molecular Structure Recognition with Image-to-Graph Generation. Journal of Chemical Information and Modeling.
3. Clevert, D.A. et al. (2021). Img2Mol – accurate SMILES recognition from molecular graphical depictions. Chemical Science.
4. Staker, J. et al. (2019). Molecular Structure Extraction from Documents Using Deep Learning. Journal of Chemical Information and Modeling.
5. Filippov, I. & Nicklaus, M. (2009). Optical Structure Recognition Software to Recover Molecular Information: OSRA. Journal of Chemical Information and Modeling.
6. Oldenhof, M. et al. (2024). MolGrapher: Graph-based Visual Recognition of Chemical Structures. ICCV.