This report documents the results of a simulation based calibration (SBC) run for RBesT. The calibration data will be generated whenever relevant changes to the gMAP function were made. The calibration runs are performed for typical use cases of gMAP. These include the three likelihoods (binomial, gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a run with a very/less conservative prior choice for between-trial heterogeniety parameter.

The calibration data presented here has been generated at and with the RBesT git version as:

The MD5 hash of the calibration data file presented here must match the above listed MD5:

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Hierarchical intercept only (random-effects intercept) model for binomial, gaussian and Poisson likelihood

Likelihood:

Binary \[y_i|\theta_{j} \sim \mbox{Bernoulli}(\theta_j), \] \[g(\theta) = \mbox{logit}(\theta)\]
Normal \[y_i|\theta_{j} \sim \mbox{Normal}(\theta_j, \sigma^2), \] \[g(\theta) = \theta\]
Poisson \[y_i|\theta_{j} \sim \mbox{Poisson}(\theta_j), \] \[g(\theta) = \log(\theta)\]

Hierarchical prior:

\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]

\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]

The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Sampler Diagnostics Overview

family	data_scenario	sd_tau	N	total_divergent	total_divergent_sim_fraction	min_ess	max_Rhat	total_large_Rhat	min_lp_ess_bulk	min_lp_ess_tail
binomial	dense	0.5	10000	322	0.028	109	1.010	0	234	142
binomial	dense	1.0	10000	353	0.029	58	1.032	0	136	129
binomial	sparse	0.5	10000	1229	0.058	9	1.217	1	19	9
binomial	sparse	1.0	10000	2373	0.113	12	1.279	1	109	58
gaussian	dense	0.5	10000	521	0.028	30	1.071	0	227	164
gaussian	dense	1.0	10000	339	0.023	42	1.039	0	145	72
gaussian	sparse	0.5	10000	21734	0.467	3	1.840	1	12	21
gaussian	sparse	1.0	10000	41989	0.402	2	3.895	1	9	17
poisson	dense	0.5	10000	742	0.057	64	1.035	0	147	50
poisson	dense	1.0	10000	1888	0.110	12	1.233	1	123	185
poisson	sparse	0.5	10000	26234	0.396	2	2.722	1	6	48
poisson	sparse	1.0	10000	39505	0.411	2	11.114	1	4	16

Note: Large Rhat is defined as exceeding 1.2.

Summary Statistics

\(\chi^2\) Statistic, \(\mu\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
1	dense	binomial	0.5	mu	53.901	63	0.786
13	dense	binomial	1	mu	49.830	63	0.886
25	dense	gaussian	0.5	mu	64.499	63	0.424
37	dense	gaussian	1	mu	49.498	63	0.893
49	dense	poisson	0.5	mu	49.395	63	0.895
61	dense	poisson	1	mu	68.493	63	0.296
73	sparse	binomial	0.5	mu	61.363	63	0.535
78	sparse	binomial	1	mu	61.222	63	0.540
83	sparse	gaussian	0.5	mu	50.317	63	0.876
88	sparse	gaussian	1	mu	49.126	63	0.900
93	sparse	poisson	0.5	mu	58.509	63	0.637
98	sparse	poisson	1	mu	57.498	63	0.672

\(\chi^2\) Statistic, \(\tau\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
2	dense	binomial	0.5	tau	63.091	63	0.473
14	dense	binomial	1	tau	58.048	63	0.653
26	dense	gaussian	0.5	tau	34.266	63	0.999
38	dense	gaussian	1	tau	88.768	63	0.018
50	dense	poisson	0.5	tau	62.938	63	0.479
62	dense	poisson	1	tau	61.850	63	0.517
74	sparse	binomial	0.5	tau	78.451	63	0.091
79	sparse	binomial	1	tau	82.598	63	0.049
84	sparse	gaussian	0.5	tau	71.245	63	0.223
89	sparse	gaussian	1	tau	60.966	63	0.549
94	sparse	poisson	0.5	tau	52.416	63	0.827
99	sparse	poisson	1	tau	50.355	63	0.875

\(\chi^2\) Statistic, group estimates \(\theta\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
3	dense	binomial	0.5	theta[10]	48.013	63	0.919
4	dense	binomial	0.5	theta[1]	74.291	63	0.156
5	dense	binomial	0.5	theta[2]	60.979	63	0.549
6	dense	binomial	0.5	theta[3]	62.451	63	0.496
7	dense	binomial	0.5	theta[4]	55.142	63	0.749
8	dense	binomial	0.5	theta[5]	54.362	63	0.773
9	dense	binomial	0.5	theta[6]	68.800	63	0.288
10	dense	binomial	0.5	theta[7]	67.597	63	0.323
11	dense	binomial	0.5	theta[8]	104.563	63	0.001
12	dense	binomial	0.5	theta[9]	80.819	63	0.065
15	dense	binomial	1	theta[10]	65.562	63	0.388
16	dense	binomial	1	theta[1]	63.437	63	0.461
17	dense	binomial	1	theta[2]	53.325	63	0.802
18	dense	binomial	1	theta[3]	61.670	63	0.524
19	dense	binomial	1	theta[4]	59.840	63	0.590
20	dense	binomial	1	theta[5]	58.944	63	0.622
21	dense	binomial	1	theta[6]	79.194	63	0.082
22	dense	binomial	1	theta[7]	45.120	63	0.957
23	dense	binomial	1	theta[8]	55.667	63	0.733
24	dense	binomial	1	theta[9]	79.411	63	0.079
27	dense	gaussian	0.5	theta[10]	66.355	63	0.362
28	dense	gaussian	0.5	theta[1]	74.342	63	0.155
29	dense	gaussian	0.5	theta[2]	59.840	63	0.590
30	dense	gaussian	0.5	theta[3]	86.797	63	0.025
31	dense	gaussian	0.5	theta[4]	96.845	63	0.004
32	dense	gaussian	0.5	theta[5]	56.000	63	0.722
33	dense	gaussian	0.5	theta[6]	81.382	63	0.059
34	dense	gaussian	0.5	theta[7]	58.342	63	0.643
35	dense	gaussian	0.5	theta[8]	77.542	63	0.103
36	dense	gaussian	0.5	theta[9]	58.419	63	0.640
39	dense	gaussian	1	theta[10]	78.784	63	0.087
40	dense	gaussian	1	theta[1]	70.234	63	0.248
41	dense	gaussian	1	theta[2]	64.486	63	0.424
42	dense	gaussian	1	theta[3]	55.398	63	0.741
43	dense	gaussian	1	theta[4]	56.474	63	0.706
44	dense	gaussian	1	theta[5]	61.939	63	0.514
45	dense	gaussian	1	theta[6]	45.504	63	0.953
46	dense	gaussian	1	theta[7]	44.352	63	0.964
47	dense	gaussian	1	theta[8]	59.878	63	0.588
48	dense	gaussian	1	theta[9]	66.189	63	0.367
51	dense	poisson	0.5	theta[10]	48.166	63	0.916
52	dense	poisson	0.5	theta[1]	64.038	63	0.440
53	dense	poisson	0.5	theta[2]	64.397	63	0.427
54	dense	poisson	0.5	theta[3]	62.528	63	0.493
55	dense	poisson	0.5	theta[4]	66.432	63	0.360
56	dense	poisson	0.5	theta[5]	68.557	63	0.295
57	dense	poisson	0.5	theta[6]	74.291	63	0.156
58	dense	poisson	0.5	theta[7]	59.366	63	0.607
59	dense	poisson	0.5	theta[8]	63.987	63	0.442
60	dense	poisson	0.5	theta[9]	62.464	63	0.495
63	dense	poisson	1	theta[10]	78.861	63	0.086
64	dense	poisson	1	theta[1]	53.837	63	0.788
65	dense	poisson	1	theta[2]	73.510	63	0.172
66	dense	poisson	1	theta[3]	63.181	63	0.470
67	dense	poisson	1	theta[4]	62.528	63	0.493
68	dense	poisson	1	theta[5]	87.270	63	0.023
69	dense	poisson	1	theta[6]	60.890	63	0.552
70	dense	poisson	1	theta[7]	78.643	63	0.088
71	dense	poisson	1	theta[8]	60.019	63	0.583
72	dense	poisson	1	theta[9]	92.864	63	0.009
75	sparse	binomial	0.5	theta[1]	61.837	63	0.518
76	sparse	binomial	0.5	theta[2]	55.142	63	0.749
77	sparse	binomial	0.5	theta[3]	50.918	63	0.863
80	sparse	binomial	1	theta[1]	59.226	63	0.612
81	sparse	binomial	1	theta[2]	66.189	63	0.367
82	sparse	binomial	1	theta[3]	76.493	63	0.118
85	sparse	gaussian	0.5	theta[1]	51.674	63	0.845
86	sparse	gaussian	0.5	theta[2]	64.678	63	0.418
87	sparse	gaussian	0.5	theta[3]	68.442	63	0.298
90	sparse	gaussian	1	theta[1]	56.768	63	0.697
91	sparse	gaussian	1	theta[2]	82.701	63	0.049
92	sparse	gaussian	1	theta[3]	63.155	63	0.471
95	sparse	poisson	0.5	theta[1]	71.552	63	0.215
96	sparse	poisson	0.5	theta[2]	65.882	63	0.377
97	sparse	poisson	0.5	theta[3]	72.179	63	0.200
100	sparse	poisson	1	theta[1]	67.162	63	0.337
101	sparse	poisson	1	theta[2]	54.413	63	0.771
102	sparse	poisson	1	theta[3]	56.294	63	0.712

Session Info

## R version 4.4.2 (2024-10-31)
## Platform: aarch64-apple-darwin20
## Running under: macOS Sequoia 15.3.1
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## time zone: Europe/Zurich
## tzcode source: internal
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] purrr_1.0.2        rstan_2.32.6       StanHeaders_2.32.9 here_1.0.1        
##  [5] ggplot2_3.5.1      broom_1.0.6        tidyr_1.3.1        dplyr_1.1.4       
##  [9] assertthat_0.2.1   knitr_1.47        
## 
## loaded via a namespace (and not attached):
##  [1] sass_0.4.9         utf8_1.2.4         generics_0.1.3     digest_0.6.35     
##  [5] magrittr_2.0.3     evaluate_0.24.0    grid_4.4.2         fastmap_1.2.0     
##  [9] rprojroot_2.0.4    jsonlite_1.8.8     pkgbuild_1.4.4     backports_1.5.0   
## [13] gridExtra_2.3      fansi_1.0.6        QuickJSR_1.2.2     scales_1.3.0      
## [17] codetools_0.2-20   jquerylib_0.1.4    cli_3.6.2          rlang_1.1.4       
## [21] munsell_0.5.1      withr_3.0.0        cachem_1.1.0       yaml_2.3.8        
## [25] parallel_4.4.2     inline_0.3.19      colorspace_2.1-0   curl_5.2.1        
## [29] vctrs_0.6.5        R6_2.5.1           matrixStats_1.3.0  stats4_4.4.2      
## [33] lifecycle_1.0.4    V8_4.4.2           pkgconfig_2.0.3    RcppParallel_5.1.7
## [37] pillar_1.9.0       bslib_0.7.0        gtable_0.3.5       loo_2.7.0         
## [41] glue_1.7.0         Rcpp_1.0.12        xfun_0.45          tibble_3.2.1      
## [45] tidyselect_1.2.1   htmltools_0.5.8.1  rmarkdown_2.27     compiler_4.4.2

Simulation based calibration for RBesT

Sebastian Weber

Mon Mar 17 11:11:15 2025