MIT's GenCAD-3D: Scan to CAD, Explained
Summary
- GenCAD‑3D turns point clouds or meshes into editable parametric CAD programs (feature trees).
- It aligns CAD and geometry in one latent space and uses a diffusion model to generate the command sequence.
- Code, models, and datasets are public, including 51 real scanned parts. Onshape export is available.
- Balanced training cuts invalid outputs and boosts accuracy on longer programs. Mesh encoders help on clean data. Noisy normals can hurt.
- Best for scan‑to‑CAD and reverse engineering. You start reviews on a feature tree, not a mesh.
GenCAD-3D is a research project from MIT. It takes 3D scans (point clouds or meshes) and turns them into editable CAD models with a feature tree.
It works by connecting scan data and CAD data in the same “space” so it can understand both. Then it uses a generative model to create the CAD commands step by step.
The result isn’t just an STL—it’s a full parametric CAD model you can review and edit.
Why this matters
Engineering teams get a lot of non‑parametric data: supplier meshes, legacy STLs, quick scans from the shop floor. Those files are hard to edit or dimension. What you really need for design work is a feature tree you can modify, constrain, and carry through reviews and PLM. GenCAD‑3D targets this gap by going geometry‑to‑CAD. It shortens “scan‑to‑CAD,” helps with reverse engineering, and gives you a starting point you can actually iterate on.
What GenCAD‑3D is
At a high level, the system learns how CAD programs and 3D geometry relate. It encodes each modality, pulls their embeddings into a shared space, and conditions a latent diffusion model to produce the CAD command sequence. The generated program then compiles to a B‑Rep using a geometry kernel. The key idea: don’t predict triangles; predict the parametric recipe.
Approach
- Convert CAD models and 3D scans into numeric form using different encoders.
- Train the system so CAD and scan data for the same part line up in the same space.
- Use a diffusion model, guided by the scan, to generate the CAD program.
- Compile that program into a B-Rep solid mod
Key Design Choices
- Point cloud encoder: DGCNN, scaled up for higher capacity.
- Mesh encoder: FeaStNet, adapted to handle full meshes and their connectivity.
- Compilation: Uses OpenCascade (via pythonocc).
What they released
The project includes research code, pre‑trained models, and datasets. There’s a main dataset with synthetic parts and paired geometry, plus a smaller set of real, 3D‑scanned parts for evaluation. If you want to try it quickly, you can run inference on a single STL and inspect the resulting feature tree. If you work in Onshape, there’s a route to push the generated program into a Part Studio.
- Code & inference scripts (MIT‑licensed) with quickstart commands for visualization and generation; optional Onshape export via API keys.
- Datasets on Hugging Face:
- GenCAD3D (~127 GB): CAD programs + point clouds + meshes + STLs + STEP.
- GenCAD3D_Scans (~700 MB): 51 3D‑printed & laser‑scanned parts with aligned CAD.
- GenCAD3D_SynthBal (~109 GB): balanced/augmented set.
- GenCAD3D_SynthBal_1M (~9 GB): 1 M synthetic CAD programs (programs only).
- License is listed as MIT on the model/data pages.
How it works (a bit deeper)
CAD models can be broken down into numbers so a computer can learn patterns in them. 3D scans, like point clouds or meshes, are also processed by separate tools that learn the shape’s structure.
During training, the system makes sure that CAD models and 3D scans of the same shape end up close to each other in its “memory space.”
When you give it a new scan, the system uses that learned memory to generate a CAD version. A decoder then translates that into a list of CAD commands (like extrude, cut, fillet). Those commands can be compiled into a real, editable CAD model.
So in practice: you can scan a part, and the system will turn it into an editable CAD file in one step.
Technical details
- Point cloud encoder: DGCNN with KNN neighborhoods, trained for 300 epochs, subsampling 2,048 points per pass for stability.
- Point cloud + normals: Same encoder, but adds per-point normals as extra input.
- Mesh encoder: FeaStNet, which uses mesh adjacency and vertex-density-weighted pooling.
The mesh and point-cloud-with-normals encoders capture more surface detail. On clean data, they improve both retrieval and reconstruction.
- Latent diffusion: Runs in the learned CAD latent space (not directly on tokens). Conditioned on the geometry embedding, it generates the program latent, which the decoder turns into a command sequence.
Where this fits in your workflow
Use GenCAD‑3D as an assist, not a replacement. If you start with a scan or supplier mesh, generate a parametric baseline first. Now your review is about editing features and dimensions instead of wrestling with triangles. You can also retrieve similar CAD programs from a library to jump‑start work on variants. Once a baseline exists, your normal processes—feedback, markups, change tracking—apply cleanly because there’s a feature tree under the hood.
- Scan → CAD program. Start from a point cloud or mesh and generate an editable model rather than a mesh you can’t modify. This shortens the time from scan to a usable feature tree.
- Cross‑modal retrieval. Use geometry to retrieve “nearest” CAD programs from a library—handy when you prefer to adapt a known starting point.
- Design alternatives. The generative model can produce multiple candidates you can evaluate, tweak, and document.
If you already centralize design reviews and supplier feedback, this kind of model can pre‑populate the parametric baseline you’ll review—so you’re discussing edits to the feature tree, not fighting a mesh.
How to try it (quick, realistic path)
Plan for a decent GPU and some downloads. Install the repo and dependencies, then run inference on a test file.
1. Set up code and data. Clone the repo, install PyTorch and dependencies, and (optionally) pythonocc‑core if you want local compilation. Download model weights and, if needed, a subset of the dataset.
2. Run inference on your own STL.
python -m diffusion.evaluation.visualize_diffusion_inference \
-encoder_type mesh_feast \
-contrastive_model_name mesh_SynthBal_1M_SBD \
-filenames examples_files/00152170.stl
Swap encoder_type and model names if you’re using point clouds instead of meshes.
3. Export to a CAD system. If you prefer Onshape, create an API key and run the provided script to translate the generated program into a new Part Studio
Practical cautions
The datasets are large, so budget storage and time. The command set focuses on sketch‑and‑extrude workflows common in industry, but not every advanced CAD feature. Expect drift on parts that rely on specialized operations or heavy references. Scan quality matters too: noisy normals can hurt performance, so keep inputs simple if needed. As always, validate results on your own geometry before you scale up.
How it compares to tools you already use
Generative design tools—Autodesk Fusion 360 Generative Design, nTopology, Siemens NX Generative Engineering, MSC Apex Generative Design, and PTC Creo GDX—optimize shapes against goals like stiffness, mass, or manufacturability. They help you explore design space and return optimized geometry, often with smooth, printable forms. Spectral Labs (SGS‑1) and GenCAD‑3D aim at a different step: converting simple inputs—2D sketches, scans, or meshes—into editable, parametric CAD. If you think of the first group as “create new shapes under constraints,” think of this group as “recover the feature tree so you can keep engineering.” Both can live in the same workflow.
Resources and Link
GenCAD‑3D — Paper (arXiv abstract)
GenCAD‑3D — ASME JMD article page
GenCAD‑3D — Datasets (Hugging Face)
Direct dataset downloads:
https://huggingface.co/datasets/yu-nomi/GenCAD_3D/resolve/main/GenCAD3D.zip
https://huggingface.co/datasets/yu-nomi/GenCAD_3D/resolve/main/GenCAD3D_Scans.zip
https://huggingface.co/datasets/yu-nomi/GenCAD_3D/resolve/main/GenCAD3D_SynthBal.zip
https://huggingface.co/datasets/yu-nomi/GenCAD_3D/resolve/main/GenCAD3D_SynthBal_1M.zip
