Benchmarks
Avaliamos as ferramentas econométricas e o agente do MAGI contra benchmarks públicos e artigos publicados. Cada número leva à execução de origem, disponível para download.
O que é avaliado
Duas camadas: as ferramentas econométricas (chamadas na especificação de referência) e o agente analista (que escolhe ferramentas e argumentos e cita os números calculados).
- Verificabilidade
- Cada pacote inclui o verify_bundle.py que vai dentro dele (só biblioteca padrão do Python) para conferir hashes, anexos e recalcular o resumo a partir dos resultados.
- Limites
- O agente responde apenas por chamadas de ferramentas registradas; lacunas de cobertura são informadas.
- Os benchmarks externos são públicos; eventual presença nos dados de treino é informada por suíte.
Como ler os números
Cada métrica mede uma etapa distinta e aponta para a execução de origem.
- Cobertura das ferramentas
- itens do benchmark atendidos pelas ferramentas registradas.
- Cobertura alcançável pelo agente
- itens resolvíveis pelos dados recebidos pelo agente com as ferramentas atuais.
- Fidelidade de implementação
- ferramenta comparada a uma implementação independente da mesma especificação.
- Fidelidade de replicação
- ferramenta na especificação dos autores contra o valor publicado no artigo ou gabarito.
- Concordância com o caminho de referência
- a mesma conta quando o caminho foi montado pelo MAGI a partir dos metadados.
- Acurácia do analista
- taxa de acerto do modelo ao vivo, bruta e condicionada ao acerto em T1.
- Métrica nativa
- métrica oficial de cada benchmark.
- Linhas de base triviais
- chute constante zero, regressão simples sem controles (ingênua) e roteador por regras.
Por suíte
| Suíte | Cobertura (ferramenta · agente) | Fidelidade (implementação · replicação) · concordância do caminho | Acurácia do analista | Dado T1 | Métrica nativa | Linhas de base (zero · ingênua · regras) | Comparador | Outro conjunto de itens | Licença | Exclusões |
|---|---|---|---|---|---|---|---|---|---|---|
| causcibench_real | Fidelidade de implementação: — Fidelidade de replicação: — Concordância com o caminho de referência: 45%50/112v0.1.2 da versão 0.1.2, não a atual | — | — | — | nenhum no mesmo conjunto de itens | 70,83 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, OpenAI-o3, Method Sel. Accuracy, Direct) +31 referências publicadas77,17 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, OpenAI-o3, Method Sel. Accuracy, CoT) 68,33 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, OpenAI-o3, Method Sel. Accuracy, PoT) 72,6 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, OpenAI-o3, Method Sel. Accuracy, ReAct) 52,56 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, OpenAI-o3, Mean Rel. Error, Direct) 48,96 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, OpenAI-o3, Mean Rel. Error, CoT) 63,1 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, OpenAI-o3, Mean Rel. Error, PoT) 66,48 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, OpenAI-o3, Mean Rel. Error, ReAct) 69,4 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-5-mini, Method Sel. Accuracy, Direct) 70,95 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-5-mini, Method Sel. Accuracy, CoT) 68,31 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-5-mini, Method Sel. Accuracy, PoT) 70,32 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-5-mini, Method Sel. Accuracy, ReAct) 59,05 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-5-mini, Mean Rel. Error, Direct) 55,78 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-5-mini, Mean Rel. Error, CoT) 59,5 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-5-mini, Mean Rel. Error, PoT) 53,56 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-5-mini, Mean Rel. Error, ReAct) 42,86 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, Gemini-2.5-Flash-Lite, Method Sel. Accuracy, Direct) 51,63 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, Gemini-2.5-Flash-Lite, Method Sel. Accuracy, CoT) 51,91 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, Gemini-2.5-Flash-Lite, Method Sel. Accuracy, PoT) 54,34 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, Gemini-2.5-Flash-Lite, Method Sel. Accuracy, ReAct) 67,64 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, Gemini-2.5-Flash-Lite, Mean Rel. Error, Direct) 66,08 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, Gemini-2.5-Flash-Lite, Mean Rel. Error, CoT) 71,82 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, Gemini-2.5-Flash-Lite, Mean Rel. Error, PoT) 72,09 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, Gemini-2.5-Flash-Lite, Mean Rel. Error, ReAct) 56,52 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-4o, Method Sel. Accuracy, Direct) 59,34 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-4o, Method Sel. Accuracy, CoT) 56,14 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-4o, Method Sel. Accuracy, PoT) 59,41 method_selection_accuracy · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-4o, Method Sel. Accuracy, ReAct) 67,09 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-4o, Mean Rel. Error, Direct) 67,44 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-4o, Mean Rel. Error, CoT) 74,2 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-4o, Mean Rel. Error, PoT) 66,58 mean_relative_error · N = 175 Acharya, S., Zhang, T. J., Kim, A., et al. (2026). CauSciBench: Can LLMs Automate Causal Inference in Real-World Scientific Research? Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 332-384. (Table 6 (appendix F.1), Dataset Real, GPT-4o, Mean Rel. Error, ReAct) Calculado sobre outros itens: mostrado à parte, nunca como diferença. | Benchmark code MIT (CauSciBench README); each dataset keeps its own licence as data/README.md…Benchmark code MIT (CauSciBench README); each dataset keeps its own licence as data/README.md lists it (CC0 1.0, MIT, CC BY 4.0, CC BY-NC 2.0, GPL 3.0 or none stated), recorded per item; the CSVs are fetched at the pinned revision, never hosted | 22 itens excluídos
| ||
| closed_form | Cobertura das ferramentas: — Cobertura alcançável pelo agente: — | Fidelidade de implementação: — Fidelidade de replicação: — Acurácia das ferramentas contra os valores esperados: 100%48/48 (93%–100%)v0.1.0 da versão 0.1.0, não a atual | — | — | — | Linha de base zero: — Caminho ingênuo: — Roteador por regras: — | nenhum no mesmo conjunto de itens | nenhum | Generated at evaluation time; the published figures are cited, not copied data | nenhuma |
| crb_papers | Fidelidade de implementação: — Fidelidade de replicação: 88%139/158v0.1.2 da versão 0.1.2, não a atual | — | — | — | nenhum no mesmo conjunto de itens | 0,810345 strategy_correct · N = 174 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 5, Identification Metrics, row 'Strategy correct') +9 referências publicadas0,747126 causal_quantity_correct · N = 174 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 5, Identification Metrics, row 'Causal quantity correct') 0,706897 controls_correct · N = 174 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 5, Identification Metrics, row 'Controls correct') 0,37931 identification_spec_correct · N = 174 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 5, Identification Metrics, row 'Identification spec correct (all checks)') 0,833 estimate_within_solution_ci · N = 174 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 5, Estimation Metrics, row 'Estimate Within gold CI') 0,793 null_hypothesis_decision_correct · N = 174 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 5, Estimation Metrics, row 'Null Hypothesis Agreement') 13,5 percentage_error (median) · N = 174 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 5, Estimation Metrics, row 'Median percentage error') 0,597 identification_spec_correct (Difference-in-Differences) · N = 67 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 6, row 'Difference-in-Differences', column 'Full ID (%)') 0,114 identification_spec_correct (Regression Discontinuity) · N = 44 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 6, row 'Regression Discontinuity', column 'Full ID (%)') 0,364 identification_spec_correct (Instrumental Variable) · N = 22 Sawarni, Tan & Syrgkanis, CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, arXiv 2602.20571v3 (2026-09-27), GPT-5.3 baseline (Table 6, row 'Instrumental Variable', column 'Full ID (%)') Calculado sobre outros itens: mostrado à parte, nunca como diferença. | MIT (benchmark) plus each dataset's own terms; data files fetched, never hosted | 5 itens excluídos
| ||
| dgp | Cobertura das ferramentas: — Cobertura alcançável pelo agente: — | Fidelidade de implementação: — Fidelidade de replicação: — Acurácia das ferramentas contra os valores esperados: 100%12/12 (76%–100%) | — | — | — | Linha de base zero: — Caminho ingênuo: — Roteador por regras: — | nenhum no mesmo conjunto de itens | nenhum | Generated at evaluation time (magi_bench.generators) | nenhuma |
| e2e_memo | Cobertura das ferramentas: — Cobertura alcançável pelo agente: — | Fidelidade de implementação: — Fidelidade de replicação: — Taxa de aprovação do protocolo: 100%7/7 (65%–100%)v0.1.0 da versão 0.1.0, não a atual | — | — | — | Linha de base zero: — Caminho ingênuo: — Roteador por regras: — | nenhum no mesmo conjunto de itens | nenhum | internal; the bills are validation texts written for MAGI | nenhuma |
| nist_anova |
T2 e T3 aparecem separados, nunca somados. T3 pontua o desenho; a estimativa é registrada. gemini-3.8-flash | — | — | Linha de base zero: — Caminho ingênuo: — Roteador por regras:
| nenhum no mesmo conjunto de itens | nenhum | Public domain (US Government work, NIST/ITL StRD); files fetched from commit- or…Public domain (US Government work, NIST/ITL StRD); files fetched from commit- or timestamp-pinned mirrors with sha256, never committed | nenhuma | ||
| nist_univariate |
T2 e T3 aparecem separados, nunca somados. T3 pontua o desenho; a estimativa é registrada. gemini-3.8-flash | — | — | Linha de base zero: — Caminho ingênuo: — Roteador por regras:
| nenhum no mesmo conjunto de itens | nenhum | Public domain (US Government work, NIST/ITL StRD); files fetched from commit- or…Public domain (US Government work, NIST/ITL StRD); files fetched from commit- or timestamp-pinned mirrors with sha256, never committed | nenhuma | ||
| official | Cobertura das ferramentas: — Cobertura alcançável pelo agente: — | Fidelidade de implementação: — Fidelidade de replicação: — | — | — | — | Linha de base zero: — Caminho ingênuo: — Roteador por regras: — | nenhum no mesmo conjunto de itens | nenhum | Public statistics; PNADc and POF microdata come from magi_data's local cache and are never downloaded here | nenhuma |
| paper_repro |
T2 e T3 aparecem separados, nunca somados. T3 pontua o desenho; a estimativa é registrada. gemini-3.8-flash | 87%48/55v0.1.1 da versão 0.1.1, não a atual | any_cited_would_pass 0,245283n=153v0.1.1 da versão 0.1.1, não a atual | nenhum no mesmo conjunto de itens | nenhum | Datasets are fetched, never hosted: each keeps its publisher's terms (the authors' pages, the…Datasets are fetched, never hosted: each keeps its publisher's terms (the authors' pages, the Econometric Society's supplements). Bundles hold digests, tool results and short attributed excerpts only. | nenhuma | |||
| qrdata_socsci | Fidelidade de implementação: — Fidelidade de replicação: — Concordância com o caminho de referência: 89%17/19v0.1.2 da versão 0.1.2, não a atual | — | — | — | nenhum no mesmo conjunto de itens | 57,9 accuracy (eval.py), all 411 items · N = 411 Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, Yansong Feng. Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data. Findings of the Association for Computational Linguistics: ACL 2024, pages 9215-9235. (Table 2, Code Interpreter Assistants, GPT-4, Overall Acc.) 38,9 accuracy (eval.py), all 411 items · N = 411 Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, Yansong Feng. Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data. Findings of the Association for Computational Linguistics: ACL 2024, pages 9215-9235. (Table 2, ReAct-style Prompting, Gemini-Pro, Overall Acc.) Calculado sobre outros itens: mostrado à parte, nunca como diferença. | CC BY-NC 4.0 (QRData); question text quoted with attribution, data files fetched, never hosted | nenhuma | ||
| strd | Cobertura das ferramentas: — Cobertura alcançável pelo agente: — | Fidelidade de implementação: — Fidelidade de replicação: — Acurácia das ferramentas contra os valores esperados: 100%20/20 (84%–100%) | — | — | — | Linha de base zero: — Caminho ingênuo: — Roteador por regras: — | nenhum no mesmo conjunto de itens | nenhum | Public domain (US Government work, NIST); files committed under suites/data/strd | nenhuma |
| tool_selection | Cobertura das ferramentas: — Cobertura alcançável pelo agente: — | Fidelidade de implementação: — Fidelidade de replicação: — | — | — | — | nenhum no mesmo conjunto de itens | nenhum | internal | nenhuma |
Registro de execuções
Execuções mais recentes de cada suíte de benchmark.
Cadeia do registro íntegra: 62 eventos, cada um com o sha256 do anterior.
| Data | Execução | Situação | Cadeia | Commit | Agente | Modelo | Taxa de acerto ± IC 95% | Tokens | Custo | Resumo (sha256 de SHA256SUMS) |
|---|---|---|---|---|---|---|---|---|---|---|
| 15/10/2026, 15:35 | 20261015T183517Z-suite-paper_repro-gemini-0cda63 | concluída | íntegra | a167d5ead931 | gemini | gemini-3.8-flash | paper_repro
| 9.462.607 | USD 18,256 estimativa | de545076236a |
| 15/10/2026, 15:33 | 20261015T183314Z-suite-nist_anova-gemini-6acf8c | concluída | íntegra | a167d5ead931 | gemini | gemini-3.8-flash | nist_anova
| 489.000 | USD 0,974898 estimativa | 2428cc68cd79 |
| 15/10/2026, 15:28 | 20261015T182814Z-suite-nist_univariate-gemini-dd5d14 | concluída | íntegra | a167d5ead931 | gemini | gemini-3.8-flash | nist_univariate
| 780.586 | USD 1,54581 estimativa | 0a04f1dc0202 |
| 15/10/2026, 14:14 | 20261015T171414Z-suite-official-scripted-391f79 | concluída | íntegra | 2925aeab2262 | scripted | — | official 86% (49%–97%) | 0 | — | 41a2118bddb5 |
| 15/10/2026, 14:14 | 20261015T171401Z-suite-tool_selection-baseline-a1426d linha de base | concluída | íntegra | 2925aeab2262 | baseline | rules-1 | tool_selection 94% (86%–98%) | 0 | — | 82a6279c7d1b |
| 15/10/2026, 14:13 | 20261015T171333Z-suite-e2e_memo-scripted-8076a5 | concluída | íntegra | 2925aeab2262 | scripted | fake | e2e_memo 100% (65%–100%) | 0 | — | 1913e25f398f |
| 15/10/2026, 14:13 | 20261015T171311Z-suite-dgp-scripted-505dfe | concluída | íntegra | 2925aeab2262 | scripted | — | dgp 100% (76%–100%) | 0 | — | aab927559ab5 |
| 15/10/2026, 14:13 | 20261015T171300Z-suite-closed_form-scripted-9fb914 | concluída | íntegra | 2925aeab2262 | scripted | — | closed_form 100% (93%–100%) | 0 | — | 7ac29f56955f |
| 15/10/2026, 14:12 | 20261015T171252Z-suite-strd-scripted-d8a322 | concluída | íntegra | 2925aeab2262 | scripted | — | strd 100% (84%–100%) | 0 | — | ab3339433da2 |
| 15/10/2026, 14:11 | 20261015T171142Z-suite-causcibench_real-naive-d6494d linha de base | concluída | íntegra | 2925aeab2262 | naive | naive-ols | causcibench_real 7% (3%–11%) sem os 2 gabaritos defeituosos: 7% | 0 | — | e51187e6c780 |
| 15/10/2026, 14:10 | 20261015T171027Z-suite-crb_papers-naive-54c3f2 linha de base | concluída | íntegra | 2925aeab2262 | naive | naive-ols | crb_papers 41% (31%–51%) sem o gabarito defeituoso: 41% | 0 | — | 17150e88100b |
| 15/10/2026, 14:00 | 20261015T170037Z-suite-qrdata_socsci-naive-761cb5 linha de base | concluída | íntegra | 62cdbe0860ad | naive | naive-ols | qrdata_socsci 3% (0%–9%) sem os 7 gabaritos defeituosos: 4% | 0 | — | c25a2cb8ad3b |