A Transformer Model for Symbolic regression towards Scientific Discovery

模型训练命令模型评估命令模型导出命令模型推理命令

# linux
wget -c https://paddle-org.bj.bcebos.com/paddlescience/datasets/transformer4sr/data_generated.tar.gz
# windows
# curl https://paddle-org.bj.bcebos.com/paddlescience/datasets/transformer4sr/data_generated.tar.gz -o data_generated.tar.gz
# unzip it
tar -xzvf data_generated.tar.gz
python transformer4sr.py

# download srsd dataset from huggingface
git clone https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_easy/
git clone https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_medium/
git clone https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_hard/
# running
python transformer4sr.py mode=eval EVAL.pretrained_model_path=https://paddle-org.bj.bcebos.com/paddlescience/models/transformer4sr/transformer4sr_pretrained.pdparams

python transformer4sr.py mode=export

python transformer4sr.py mode=infer

预训练模型	指标
transformer4sr_pretrained.pdparams	Mean ZSS distance(srsd-feynman_easy): 0.658 +- 0.390 Hit rate(srsd-feynman_easy): 8/30 Mean ZSS distance(srsd-feynman_medium): 0.674 +- 0.331 Hit rate(srsd-feynman_medium): 8/37 Mean ZSS distance(srsd-feynman_hard): 0.737 +- 0.188 Hit rate(srsd-feynman_hard): 1/39

1. 背景简介

符号回归（SR）搜索最能描述数值数据集的数学表达式，它大致分为三类，即 GP-based SR（基于遗传编程的符号回归）、ML-based SR（基于机器学习的符号回归）、DL-based SR（基于深度学习的符号回归）。基于遗传编程的 SR 算法的计算成本通常很高，因此该案例使用了一种针对符号回归的新 Transformer 模型，并将最佳模型应用于 SRSD 数据集（科学发现数据集的符号回归）进行推理和测试。

2. 问题定义

作者提出了一种基于 Transformer 网络的 SR 模型，称为 Transformer4SR (A Transformer Model for Symbolic regression towards Scientific Discovery)，该模型用于处理封闭库问题，将符号回归的预定义词汇用特定的方法转化为 tokens。在该案例中，输入数据通过程序生成，数据通过模型后得到输出结果，再将其转化回符号表示，最终得到数据的符号回归结果。

下图为该方法的网络结构图，该结构基于 Transformer 有编码器 Encoder 和解码器 Decoder 两个主要部分。 作者提出三种编码器架构：MLP、Att 或 Mix，在本案例中，主要实现了 Mix 下的编码器结构。解码器是标准的 Transformer 解码器。在训练期间，编码器接收表格数据集，解码器接收 tokens 的真实值序列，而在推理过程中，解码器是独立的，并以自动回归的方式预测 tokens。

transformer4sr 模型结构图

3. 问题求解

接下来开始讲解如何将问题一步一步地转化为 PaddleScience 代码，用深度学习的方法求解该问题。 为了快速理解 PaddleScience，接下来仅对模型构建、方程构建、计算域构建等关键步骤进行阐述，而其余细节请参考 API文档。

3.1 数据集生成与下载

3.1.1 数据生成

该案例使用的训练数据为案例自主生成的数据：先用符号库中的符号随机生成大量公式；再通过一定的筛选机制筛除无效或不合规的公式；最后再根据这些公式进行采样；最终得到输入数据集和标签数据。

生成数据的参数信息如下：

DATA_GENERATE:
  # output path
  data_path: "./data_generated/"
  # filters
  num_nodes: [2, 15] # number of nodes
  num_nested_max: 6 # multiple levels of nesting
  num_consts: [1, 1] # number of constants(C)
  num_vars: [1, 6] # number of variables(x1,x2,...)
  seq_length_max: 30
  order_of_mag_limit: 1.0e+9 # magnitude of value
  # others
  num_init_trials: 100000 # number of initial trials
  num_sampling_per_eq: 25 # number of times to evaluate constants for each unique equation
  sampling_times: 50 # the number of observations
  var_type: "normal" # variable representation, 'normal' is (y, x1, x2, ...), 'log' is log(abs(y, x1, x2, ...)), or 'both'
  num_zfill: 8
DATA:

其中num_init_trials为随机产生的初始方程数，这个值越大生成的数据越多，在原论文中这个值为 1000000。

设置相关参数后，可使用如下命令生成数据集：

python generate_datasets.py

3.1.2 数据下载

我们也提前生成了一个比原始训练数据规模小 10 倍的数据集（即num_init_trials为 100000），以便简单的进行模型训练，并提供了下载链接：

wget -c https://paddle-org.bj.bcebos.com/paddlescience/datasets/transformer4sr/data_generated.tar
tar -xvf data_generated.tar

该案例在开源符号回归数据集 SRSD(Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery) 上进行模型验证，因此需要预下载此数据集，该数据集存放在 huggingface 上，地址分别为srsd-feynman_easy、srsd-feynman_medium、srsd-feynman_hard。可从对应网页下载或使用 git 下载：

git clone https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_easy/
git clone https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_medium/
git clone https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_hard/

3.2 数据读取

由于数据读取和转换比较复杂，数据集相关的函数被定义在 functions_data.py 文件中，并在模型训练、验证等过程中被调用。

训练时需要读取自主生成的数据集：

# data
data_funcs = DataFuncs(
    cfg.DATA.data_path,
    cfg.DATA.vocab_library,
    cfg.DATA.seq_length_max,
    cfg.DATA.ratio,
    shuffle=True,
)

验证时需要读取 SRSD 数据集：

# data
data_funcs = SRSDDataFuncs(
    cfg.DATA.data_path_srsd,
    cfg.DATA.sampling_times,
    cfg.DATA.response_variable,
    cfg.DATA.vocab_library,
    cfg.DATA.seq_length_max,
    shuffle=True,
)

数据相关参数定义在 yaml 文件中：

data_path_srsd: ["./srsd-feynman_easy/"]
ratio: [0.8, 0.1, 0.1]
sampling_times: ${DATA_GENERATE.sampling_times}
seq_length_max: 30 # ${DATA_GENERATE.seq_length_max}
response_variable: ["y", "x1", "x2", "x3", "x4", "x5", "x6"] # maximum number of variables is len(response_variable)=7
vocab_library: [
    "add",
    "mul",
    "sin",

3.3 模型构建

在本问题中，我们使用神经网络 Transformer 作为模型。

# set model
num_var_max = len(cfg.DATA.response_variable)
vocab_size = len(cfg.DATA.vocab_library) + 2
model = ppsci.arch.Transformer(
    **cfg.MODEL,
    num_var_max=num_var_max,
    vocab_size=vocab_size,
    seq_length=data_funcs.seq_length_max,
)

为了在计算时，准确快速地访问具体变量的值，我们在这里指定网络模型的输入变量名是 ("input", "target_seq")，输出变量名是 ("output", )，这些命名与后续代码保持一致。

3.4 优化器构建

本案例使用一种自定义的学习率策略 LambdaDecay，该学习率策略支持自定义学习率衰减函数。训练过程会调用优化器来更新模型参数，此处选择较为常用的 Adam 优化器。

# set optimizer
def lr_lambda(step, d_model=cfg.MODEL.d_model, warmup=cfg.TRAIN.lr_warmup):
    if step == 0:
        step = 1
    lr = d_model ** (-0.5) * min(step ** (-0.5), step * warmup ** (-1.5))
    return lr

lr_scheduler = ppsci.optimizer.lr_scheduler.LambdaDecay(
    **cfg.TRAIN.lr_scheduler,
    lr_lambda=lr_lambda,
)()
optimizer = ppsci.optimizer.Adam(lr_scheduler, **cfg.TRAIN.adam)(model)

3.5 约束构建

在本案例中，我们使用监督数据集对模型进行训练，因此需要构建监督约束 SupervisedConstraint：

# set constraint
sup_constraint = ppsci.constraint.SupervisedConstraint(
    {
        "dataset": {
            "name": "NamedArrayDataset",
            "input": {
                "input": data_funcs.values_train.astype(paddle.get_default_dtype()),
                "target_seq": data_funcs.targets_train[:, :-1],
            },
            "label": {"output": data_funcs.targets_train[:, 1:]},
        },
        "batch_size": cfg.TRAIN.batch_size,
        "sampler": {
            "name": "BatchSampler",
            "drop_last": False,
            "shuffle": True,
        },
        "num_workers": 1,
    },
    ppsci.loss.FunctionalLoss(cross_entropy_loss_func),
    name="sup_constraint",
)

SupervisedConstraint 的第一个参数是监督约束的读取配置，其中 dataset 字段表示使用的训练数据集信息，各个字段分别表示：

1. name： 数据集类型，此处 NamedArrayDataset 表示数据集的类型为 Array；
2. input： 输入数据；
3. label： 标签数据；

batch_size 字段表示 batch的大小；

sampler 字段表示采样方法，其中各个字段表示：

1. name： 采样器类型，此处 BatchSampler 表示批采样器；
2. drop_last： 是否需要丢弃最后无法凑整一个 mini-batch 的样本；
3. shuffle： 是否需要在生成样本下标时打乱顺序；

第二个参数是损失函数，此处的 FunctionalLoss 为 PaddleScience 自定义 loss 函数类，该类支持编写代码时自定义 loss 的计算方法，loss 的具体实现为其参数 cross_entropy_loss_func， 这是一个被定义在 functions_loss_metric.py 文件中的函数，如下所示：

def cross_entropy_loss_func(output_dict, label_dict, *args):
    custom_loss = paddle.nn.CrossEntropyLoss(ignore_index=0, label_smoothing=0.0)
    loss = custom_loss(output_dict["output"], label_dict["output"])
    return {"ce_loss": loss}

第三个参数是约束条件的名字，我们需要给每一个约束条件命名，方便后续对其索引。

在约束构建完毕之后，以我们刚才的命名为关键字，封装到一个字典中，方便后续访问：

# wrap constraints together
constraint = {sup_constraint.name: sup_constraint}

3.6 评估器构建

在训练过程中通常会按一定轮数间隔，用验证集(测试集)评估当前模型的训练情况，因此使用 ppsci.validate.SupervisedValidator 构建评估器，构建过程与 约束构建 类似。

# set validator
sup_validator = ppsci.validate.SupervisedValidator(
    {
        "dataset": {
            "name": "NamedArrayDataset",
            "input": {
                "input": data_funcs.values_val.astype(paddle.get_default_dtype()),
                "target_seq": data_funcs.targets_val[:, :-1],
            },
            "label": {"output": data_funcs.targets_val[:, 1:]},
        },
        "batch_size": cfg.TRAIN.batch_size,
        "num_workers": 1,
    },
    ppsci.loss.FunctionalLoss(cross_entropy_loss_func),
    metric={"metric": ppsci.metric.FunctionalMetric(compute_inaccuracy)},
    name="sup_validator",
)

# wrap validator together
validator = {sup_validator.name: sup_validator}

其中评估指标为自定义的指标计算函数 compute_inaccuracy，在 functions_loss_metric.py 文件中：

def compute_inaccuracy(
    output_dict: Dict[str, paddle.Tensor],
    label_dict: Dict[str, paddle.Tensor],
    *args,
) -> Dict[str, paddle.Tensor]:
    """Calculate the ratio of incorrectly matched tokens to the total number."""
    preds = output_dict["output"]
    labels = label_dict["output"]
    padding_not_mask = labels != 0
    correct_bool = paddle.equal(paddle.argmax(preds, axis=-1), labels)
    correct_bool = paddle.logical_and(
        correct_bool,
        padding_not_mask,
    )
    inacc = 1 - paddle.sum(correct_bool) / paddle.sum(padding_not_mask)
    return {"inaccuracy_mean": inacc}

3.7 超参数设定

设置训练轮数等参数，如下所示。

      "x4",
      "x5",
      "x6",
    ] # vocab_size=len(vocab_library)+2(because add and mul are binary operators)
# model settings
MODEL:
  input_keys: ["input", "target_seq"]

3.8 模型训练、评估

完成上述设置之后，只需要将上述实例化的对象按顺序传递给 ppsci.solver.Solver，然后启动训练、评估。

# initialize solver
solver = ppsci.solver.Solver(
    model,
    constraint,
    optimizer=optimizer,
    validator=validator,
    cfg=cfg,
)

# train model
solver.train()

# evaluate after finished training
solver.eval()

3.9 模型验证与结果可视化

该案例在开源符号回归数据集 SRSD(Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery) 上进行模型验证，验证时采用自回归的方式运行解码器，不需要运行编码器。验证时的指标为一种基于树编辑距离的归一化指标 ZSS。

num_repeat = cfg.EVAL.num_repeat if isinstance(data_funcs, SRSDDataFuncs) else 1
num_samples = data_funcs.values_test.shape[0]
zss_dist = np.zeros((num_repeat, num_samples))
for i in tqdm(range(num_repeat), desc="Evaluating"):
    encoder_input = paddle.to_tensor(
        data_funcs.values_test, dtype=paddle.get_default_dtype()
    )
    preds = model.decode_process(encoder_input, is_tree_complete)
    labels = paddle.to_tensor(data_funcs.targets_test)

    for j in range(num_samples):
        try:
            pred_simplify = simplify_output(preds[j], "tensor")
            zss_dist[i][j] = compute_norm_zss_dist(pred_simplify[0], labels[j])
        except Exception:
            zss_dist[i][j] = np.nan

    if i != num_repeat - 1:
        # reload data to increase randomness
        data_funcs.init_data("test")

zss_dist_mean = np.nanmean(zss_dist, axis=0)
zss_dist_std = np.nanstd(zss_dist, axis=0)
zss_dist_min = np.nanmin(zss_dist, axis=0)
zss_dist_max = np.nanmax(zss_dist, axis=0)

try:

可视化的代码定义在文件 functions_vis.py 中，除了对验证集中的结果进行可视化外，还提供了一个公式为 \(25*x1+x2*log(x1)\) 的 demo 的可视化结果：

visualizer = VisualizeFuncs(model)
visualizer.visualize_valid_data(data_funcs.targets_test, data_funcs.values_test, 10)
visualizer.visualize_demo()

4. 完整代码

# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
Reference: https://github.com/omron-sinicx/transformer4sr
"""

import hydra
import numpy as np
import paddle
from functions_data import DataFuncs
from functions_data import SRSDDataFuncs
from functions_loss_metric import compute_inaccuracy
from functions_loss_metric import cross_entropy_loss_func
from functions_vis import VisualizeFuncs
from omegaconf import DictConfig
from tqdm import tqdm
from utils import compute_norm_zss_dist
from utils import is_tree_complete
from utils import simplify_output

import ppsci


def train(cfg: DictConfig):
    # data
    data_funcs = DataFuncs(
        cfg.DATA.data_path,
        cfg.DATA.vocab_library,
        cfg.DATA.seq_length_max,
        cfg.DATA.ratio,
        shuffle=True,
    )

    # set model
    num_var_max = len(cfg.DATA.response_variable)
    vocab_size = len(cfg.DATA.vocab_library) + 2
    model = ppsci.arch.Transformer(
        **cfg.MODEL,
        num_var_max=num_var_max,
        vocab_size=vocab_size,
        seq_length=data_funcs.seq_length_max,
    )

    # set optimizer
    def lr_lambda(step, d_model=cfg.MODEL.d_model, warmup=cfg.TRAIN.lr_warmup):
        if step == 0:
            step = 1
        lr = d_model ** (-0.5) * min(step ** (-0.5), step * warmup ** (-1.5))
        return lr

    lr_scheduler = ppsci.optimizer.lr_scheduler.LambdaDecay(
        **cfg.TRAIN.lr_scheduler,
        lr_lambda=lr_lambda,
    )()
    optimizer = ppsci.optimizer.Adam(lr_scheduler, **cfg.TRAIN.adam)(model)

    # set constraint
    sup_constraint = ppsci.constraint.SupervisedConstraint(
        {
            "dataset": {
                "name": "NamedArrayDataset",
                "input": {
                    "input": data_funcs.values_train.astype(paddle.get_default_dtype()),
                    "target_seq": data_funcs.targets_train[:, :-1],
                },
                "label": {"output": data_funcs.targets_train[:, 1:]},
            },
            "batch_size": cfg.TRAIN.batch_size,
            "sampler": {
                "name": "BatchSampler",
                "drop_last": False,
                "shuffle": True,
            },
            "num_workers": 1,
        },
        ppsci.loss.FunctionalLoss(cross_entropy_loss_func),
        name="sup_constraint",
    )

    # wrap constraints together
    constraint = {sup_constraint.name: sup_constraint}

    # set validator
    sup_validator = ppsci.validate.SupervisedValidator(
        {
            "dataset": {
                "name": "NamedArrayDataset",
                "input": {
                    "input": data_funcs.values_val.astype(paddle.get_default_dtype()),
                    "target_seq": data_funcs.targets_val[:, :-1],
                },
                "label": {"output": data_funcs.targets_val[:, 1:]},
            },
            "batch_size": cfg.TRAIN.batch_size,
            "num_workers": 1,
        },
        ppsci.loss.FunctionalLoss(cross_entropy_loss_func),
        metric={"metric": ppsci.metric.FunctionalMetric(compute_inaccuracy)},
        name="sup_validator",
    )

    # wrap validator together
    validator = {sup_validator.name: sup_validator}

    # initialize solver
    solver = ppsci.solver.Solver(
        model,
        constraint,
        optimizer=optimizer,
        validator=validator,
        cfg=cfg,
    )

    # train model
    solver.train()

    # evaluate after finished training
    solver.eval()


def evaluate(cfg: DictConfig):
    # data
    data_funcs = SRSDDataFuncs(
        cfg.DATA.data_path_srsd,
        cfg.DATA.sampling_times,
        cfg.DATA.response_variable,
        cfg.DATA.vocab_library,
        cfg.DATA.seq_length_max,
        shuffle=True,
    )

    # set model
    num_var_max = len(cfg.DATA.response_variable)
    vocab_size = len(cfg.DATA.vocab_library) + 2
    model = ppsci.arch.Transformer(
        **cfg.MODEL,
        num_var_max=num_var_max,
        vocab_size=vocab_size,
        seq_length=data_funcs.seq_length_max,
    )
    ppsci.utils.save_load.load_pretrain(model, path=cfg.EVAL.pretrained_model_path)
    model.eval()

    # evaluate
    num_repeat = cfg.EVAL.num_repeat if isinstance(data_funcs, SRSDDataFuncs) else 1
    num_samples = data_funcs.values_test.shape[0]
    zss_dist = np.zeros((num_repeat, num_samples))
    for i in tqdm(range(num_repeat), desc="Evaluating"):
        encoder_input = paddle.to_tensor(
            data_funcs.values_test, dtype=paddle.get_default_dtype()
        )
        preds = model.decode_process(encoder_input, is_tree_complete)
        labels = paddle.to_tensor(data_funcs.targets_test)

        for j in range(num_samples):
            try:
                pred_simplify = simplify_output(preds[j], "tensor")
                zss_dist[i][j] = compute_norm_zss_dist(pred_simplify[0], labels[j])
            except Exception:
                zss_dist[i][j] = np.nan

        if i != num_repeat - 1:
            # reload data to increase randomness
            data_funcs.init_data("test")

    zss_dist_mean = np.nanmean(zss_dist, axis=0)
    zss_dist_std = np.nanstd(zss_dist, axis=0)
    zss_dist_min = np.nanmin(zss_dist, axis=0)
    zss_dist_max = np.nanmax(zss_dist, axis=0)

    try:
        keys = data_funcs.keys_test
        assert len(keys) == num_samples
    except Exception:
        keys = [f"sample_{i}" for i in range(num_samples)]

    print(
        f"zss_distance and accuracy in {num_repeat} attempts of {num_samples} samples with format: name => mean +- std | min ~ max"
    )
    for i in range(num_samples):
        key = keys[i]
        print(
            f"{key} => {zss_dist_mean[i]:.3f} +- {zss_dist_std[i]:.3f} | {zss_dist_min[i]:.3f} ~ {zss_dist_max[i]:.3f}"
        )

    print("-----------")
    print(
        f"=> Mean ZSS distance: {np.nanmean(zss_dist):.3f} +- {np.nanstd(zss_dist):.3f}"
    )
    print(f"=> Hit rate: {np.sum(np.any(zss_dist==0, axis=0))}/{zss_dist.shape[1]}")

    # visualize prediction
    visualizer = VisualizeFuncs(model)
    visualizer.visualize_valid_data(data_funcs.targets_test, data_funcs.values_test, 10)
    visualizer.visualize_demo()


def export(cfg: DictConfig):
    def temporary_complete_func(seq_indices):
        ".utils.is_tree_complete is not work in static gragh now."
        arity = 1
        for n in seq_indices:
            n = n.item()
            if n == 0 or n == 1:
                continue
                print("Predict padding or <SOS>, which is bad...")
            if n == 2 or n == 3:
                arity = arity + 2 - 1
            elif n in range(4, 13):
                arity = arity + 1 - 1
            elif n in range(13, 20):
                arity = arity + 0 - 1
        if arity == 0:
            return True
        else:
            return False

    class WarppedModel(ppsci.arch.Transformer):
        def __init__(self, *args, complete_func, **kwargs):
            super().__init__(*args, **kwargs)
            self.complete_func = complete_func

        def forward(self, x):
            return {"output": self.decode_process(x["input"], self.complete_func)}

    # set model
    num_var_max = len(cfg.DATA.response_variable)
    vocab_size = len(cfg.DATA.vocab_library) + 2
    warpped_model = WarppedModel(
        **cfg.MODEL,
        num_var_max=num_var_max,
        vocab_size=vocab_size,
        seq_length_max=cfg.DATA.seq_length_max,
        complete_func=temporary_complete_func,
    )
    warpped_model.eval()

    # initialize solver
    solver = ppsci.solver.Solver(
        warpped_model,
        pretrained_model_path=cfg.INFER.pretrained_model_path,
    )

    # export model
    from paddle.static import InputSpec

    input_spec = [
        {
            "input": InputSpec(
                [None, cfg.DATA.sampling_times, len(cfg.DATA.response_variable), 1],
                "float32",
                name="input",
            )
        }
    ]
    solver.export(input_spec, cfg.INFER.export_path)


def inference(cfg: DictConfig):
    import sympy

    from deploy.python_infer import pinn_predictor

    predictor = pinn_predictor.PINNPredictor(cfg)

    C, y, x1, x2, x3, x4, x5, x6 = sympy.symbols(
        "C, y, x1, x2, x3, x4, x5, x6", real=True, positive=True
    )
    y = 25 * x1 + x2 * sympy.log(x1)
    print("The ground truth is:", y)

    x1_values = np.power(10.0, np.random.uniform(-1.0, 1.0, size=50))
    x2_values = np.power(10.0, np.random.uniform(-1.0, 1.0, size=50))
    f = sympy.lambdify([x1, x2], y)
    y_values = f(x1_values, x2_values)
    dataset = np.zeros((50, 7))
    dataset[:, 0] = y_values
    dataset[:, 1] = x1_values
    dataset[:, 2] = x2_values
    encoder_input = dataset[np.newaxis, :, :, np.newaxis].astype(np.float32)
    output_dict = predictor.predict({"input": encoder_input}, cfg.INFER.batch_size)
    output_dict = {
        store_key: output_dict[infer_key]
        for store_key, infer_key in zip(("output",), output_dict.keys())
    }
    sympy_pred = simplify_output(output_dict["output"][0], "sympy")
    print("The prediction is:", sympy_pred)


@hydra.main(version_base=None, config_path="./conf", config_name="transformer4sr.yaml")
def main(cfg: DictConfig):
    if cfg.mode == "train":
        train(cfg)
    elif cfg.mode == "eval":
        evaluate(cfg)
    elif cfg.mode == "export":
        export(cfg)
    elif cfg.mode == "infer":
        inference(cfg)
    else:
        raise ValueError(f"cfg.mode should in ['train', 'eval'], but got '{cfg.mode}'")


if __name__ == "__main__":
    main()

5. 结果展示

下方展示了模型在公式 \(25*x1+x2*log(x1)\) 上的预测结果。

demo 公式上的预测结果

其中 \(C\) 表示常量，可以看到模型预测结果与真实公式基本一致。

6. 参考资料

• A Transformer Model for Symbolic regression towards Scientific Discovery
• 参考代码

@inproceedings{lalande2023,
    title = {A Transformer Model for Symbolic Regression towards Scientific Discovery},
    author = {Florian Lalande and Yoshitomo Matsubara and Naoya Chiba and Tatsunori Taniai and Ryo Igarashi and Yoshitaka Ushiku},
    booktitle = {NeurIPS 2023 AI for Science Workshop},
    year = {2023},
    url = {https://openreview.net/forum?id=AIfqWNHKjo},
}