cfa63825a153b84fbe6cc292995235c7e3d9dd65

1 /* This file is part of libDAI - http://www.libdai.org/

2 *

3 * libDAI is licensed under the terms of the GNU General Public License version

4 * 2, or (at your option) any later version. libDAI is distributed without any

5 * warranty. See the file COPYING for more details.

6 *

7 * Copyright (C) 2008-2010 Joris Mooij [joris dot mooij at libdai dot org]

8 */

11 /** \file

12 * \brief Contains additional doxygen documentation

13 *

14 * \todo Replace all Name members by virtual functions (or add virtual functions returning the Name)

15 *

16 * \idea Adapt (part of the) guidelines in http://www.boost.org/development/requirements.html#Design_and_Programming

17 *

18 * \idea Use "gcc -MM" to generate dependencies for targets: http://make.paulandlesley.org/autodep.html

19 *

20 * \idea Disentangle structures. In particular, ensure that graphical properties are not

21 * entangled with probabilistic properties. For example, a FactorGraph contains several components:

22 * - a BipartiteGraph

23 * - an array of variable labels

24 * - an array of variable state space sizes

25 * - an array of pointers to factor value vectors

26 * In this way, each factor could be implemented differently, e.g., we could have

27 * some sparse factors, some noisy-OR factors, some dense factors, some arbitrary

28 * precision factors, etcetera.

29 *

30 * \idea Use boost::uBLAS framework to deal with matrices, especially, with 2D sparse matrices.

31 * See http://www.boost.org/libs/numeric/ublas/doc/matrix_sparse.htm

32 * However: I read somewhere that boost::uBLAS concentrates more on correct implementation than on performance.

33 *

34 * \idea Introduce naming scheme:

35 * - all Vars should be named v_..., e.g. v_i instead of i

36 * - all VarSets should be named vs_..., e.g. v_i instead of i

37 * - all Factors should be named f_..., e.g. f_I instead of I

38 * - all indices should be named _..., e.g. _k instead of k

39 * - all iterators should be named i_, e.g. i_i is an iterator to i

40 * - all const_iterators should be named ci_, e.g. ci_i is an iterator to i

41 **/

44 /** \mainpage Reference manual for libDAI - A free/open source C++ library for Discrete Approximate Inference methods

45 * \author Joris Mooij, Frederik Eaton

46 * \version git HEAD

47 * \date February 11, 2010 - or later

48 *

49 * <hr size="1">

50 * \section about About libDAI

51 * libDAI is a free/open source C++ library (licensed under GPL 2+) that provides

52 * implementations of various (approximate) inference methods for discrete

53 * graphical models. libDAI supports arbitrary factor graphs with discrete

54 * variables; this includes discrete Markov Random Fields and Bayesian

55 * Networks.

56 *

57 * The library is targeted at researchers. To be able to use the library, a

58 * good understanding of graphical models is needed.

59 *

60 * The best way to use libDAI is by writing C++ code that invokes the library;

61 * in addition, part of the functionality is accessibly by using the

62 * - command line interface

63 * - (limited) MatLab interface

64 * - (experimental) python interface

65 * - (experimental) octave interface.

66 *

67 * libDAI can be used to implement novel (approximate) inference algorithms

68 * and to easily compare the accuracy and performance with existing algorithms

69 * that have been implemented already.

70 *

71 * \section features Features

72 * Currently, libDAI supports the following (approximate) inference methods:

73 * - Exact inference by brute force enumeration;

74 * - Exact inference by junction-tree methods;

75 * - Mean Field;

76 * - Loopy Belief Propagation [\ref KFL01];

77 * - Fractional Belief Propagation [\ref WiH03];

78 * - Tree-Reweighted Belief Propagation [\ref WJW03];

79 * - Tree Expectation Propagation [\ref MiQ04];

80 * - Generalized Belief Propagation [\ref YFW05];

81 * - Double-loop GBP [\ref HAK03];

82 * - Various variants of Loop Corrected Belief Propagation

83 * [\ref MoK07, \ref MoR05];

84 * - Gibbs sampler;

85 * - Conditioned Belief Propagation [\ref EaG09].

86 *

87 * These inference methods can be used to calculate partition sums, marginals

88 * over subsets of variables, and MAP states (the joint state of variables that

89 * has maximum probability).

90 *

91 * In addition, libDAI supports parameter learning of conditional probability

92 * tables by Expectation Maximization.

93 *

94 * \section limitations Limitations

95 * libDAI is not intended to be a complete package for approximate inference.

96 * Instead, it should be considered as an "inference engine", providing

97 * various inference methods. In particular, it contains no GUI, currently

98 * only supports its own file format for input and output (although support

99 * for standard file formats may be added later), and provides very limited

100 * visualization functionalities. The only learning method supported currently

101 * is Expectation Maximization (or Maximum Likelihood if no data is missing)

102 * for learning factor parameters.

103 *

104 * \section rationale Rationale

105 *

106 * In my opinion, the lack of open source "reference" implementations hampers

107 * progress in research on approximate inference. Methods differ widely in terms

108 * of quality and performance characteristics, which also depend in different

109 * ways on various properties of the graphical models. Finding the best

110 * approximate inference method for a particular application therefore often

111 * requires empirical comparisons. However, implementing and debugging these

112 * methods takes a lot of time which could otherwise be spent on research. I hope

113 * that this code will aid researchers to be able to easily compare various

114 * (existing as well as new) approximate inference methods, in this way

115 * accelerating research and stimulating real-world applications of approximate

116 * inference.

117 *

118 * \section language Language

119 * Because libDAI is implemented in C++, it is very fast compared with

120 * implementations in MatLab (a factor 1000 faster is not uncommon).

121 * libDAI does provide a (limited) MatLab interface for easy integration with MatLab.

122 * It also provides a command line interface and experimental python and octave

123 * interfaces (thanks to Patrick Pletscher).

124 *

125 * \section compatibility Compatibility

126 *

127 * The code has been developed under Debian GNU/Linux with the GCC compiler suite.

128 * libDAI compiles successfully with g++ versions 3.4, 4.1, 4.2 and 4.3.

129 *

130 * libDAI has also been successfully compiled with MS Visual Studio 2008 under Windows

131 * (but not all build targets are supported yet) and with Cygwin under Windows.

132 *

133 * Finally, libDAI has been compiled successfully on MacOS X.

134 *

135 * \section download Downloading libDAI

136 * The libDAI sources and documentation can be downloaded from the libDAI website:

137 * http://www.libdai.org.

138 *

139 * \section support Mailing list

140 * The Google group "libDAI" (http://groups.google.com/group/libdai)

141 * can be used for getting support and discussing development issues.

142 */

145 /** \page license License

146 * <hr size="1">

147 * \section license-license License

148 *

149 * libDAI is free software; you can redistribute it and/or modify

150 * it under the terms of the GNU General Public License as published by

151 * the Free Software Foundation; either version 2 of the License, or

152 * (at your option) any later version.

153 *

154 * libDAI is distributed in the hope that it will be useful,

155 * but WITHOUT ANY WARRANTY; without even the implied warranty of

156 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

157 * GNU General Public License for more details.

158 *

159 * <hr size="1">

160 * \section license-gpl GNU General Public License version 2

161 *

162 * \verbinclude COPYING

163 */

166 /** \page citations Citing libDAI

167 * <hr size="1">

168 * \section citations-citations Citing libDAI

169 *

170 * If you write a scientific paper describing research that made substantive use

171 * of this program, please:

172 * - mention the fashion in which this software was

173 * used, including the version number, with a citation to the literature,

174 * to allow replication;

175 * - mention this software in the Acknowledgements section.

176 *

177 * An appropriate citation would be:\n

178 *

179 * Joris M. Mooij et al. (2010) "libDAI 0.2.4: A free/open source C++ library for Discrete

180 * Approximate Inference", http://www.libdai.org

181 *

182 * or in BiBTeX format:

183 *

184 * <pre>

185 * \@misc{mooij2010libdai,

186 * author = "Joris M. Mooij et al.",

187 * title = "lib{DAI} 0.2.4: A free/open source {C}++ library for {D}iscrete {A}pproximate {I}nference",

188 * howpublished = "http://www.libdai.org/",

189 * year = 2010

190 * }

191 * </pre>

192 *

193 * Moreover, as a personal note, I would appreciate it if you would email

194 * (citations of) papers referencing this work to joris dot mooij at libdai dot org.

195 */

198 /** \page authors Authors

199 * \section authors-authors People who contributed to libDAI

200 *

201 * \verbinclude AUTHORS

202 */

205 /** \page build Building libDAI

206 * <hr size="1">

207 * \section build-unix Building libDAI under UNIX variants (Linux / Cygwin / Mac OS X)

208 *

209 * \subsection build-unix-preparations Preparations

210 *

211 * You need:

212 * - a recent version of gcc (at least version 3.4)

213 * - GNU make

214 * - doxygen

215 * - graphviz

216 * - recent boost C++ libraries (at least version 1.34 if you have

217 * a recent version of GCC, otherwise at least version 1.37; however,

218 * version 1.37 shipped with Ubuntu 9.04 is known not to work)

219 *

220 * On Debian/Ubuntu, you can easily install all these packages with a single command:

221 * <pre> apt-get install g++ make doxygen graphviz libboost-dev libboost-graph-dev libboost-program-options-dev</pre>

222 * (root permissions needed).

223 *

224 * On Mac OS X (10.4 is known to work), these packages can be installed easily via MacPorts.

225 * If MacPorts is not already installed, install it according to the instructions at http://www.macports.org/.

226 * Then, a simple

227 * <pre> sudo port install gmake boost doxygen graphviz</pre>

228 * should be enough to install everything that is needed.

229 *

230 * On Cygwin, the prebuilt Cygwin package boost-1.33.1-x is known not to work.

231 * You can however obtain the latest boost version (you need at least 1.37.0)

232 * from http://www.boost.org/ and compile/install it with:

233 *

234 * <pre> ./configure

235 * make

236 * make install

237 * </pre>

238 *

239 * \subsection build-unix-libdai Building libDAI

240 *

241 * To build the libDAI source, first copy a template Makefile.* to Makefile.conf

242 * (for example, copy Makefile.LINUX to Makefile.conf if you use GNU/Linux).

243 * Then, edit the Makefile.conf template to adapt it to your local setup.

244 * Especially directories may differ from system to system. Platform independent

245 * build options can be set in Makefile.ALL. Finally, run

246 * <pre> make</pre>

247 * The build includes a regression test, which may take a while to complete.

248 *

249 * If the build is successful, you can test the example program:

250 * <pre> examples/example tests/alarm.fg</pre>

251 * or the more extensive test program:

252 * <pre> tests/testdai --aliases tests/aliases.conf --filename tests/alarm.fg --methods JTREE_HUGIN BP_SEQMAX</pre>

253 *

254 *

255 * <hr size="1">

256 * \section build-windows Building libDAI under Windows

257 *

258 * \subsection build-windows-preparations Preparations

259 *

260 * You need:

261 * - A recent version of MicroSoft Visual Studio (2008 is known to work)

262 * - recent boost C++ libraries (version 1.37 or higher)

263 * - GNU make (can be obtained from http://gnuwin32.sourceforge.net)

264 *

265 * For the regression test, you need:

266 * - GNU diff, GNU sed (can be obtained from http://gnuwin32.sourceforge.net)

267 *

268 * \subsection build-windows-libdai Building libDAI

269 *

270 * To build the source, copy Makefile.WINDOWS to Makefile.conf. Then, edit

271 * Makefile.conf to adapt it to your local setup. Platform independent

272 * build options can be set in Makefile.ALL. Finally, run (from the command line)

273 * <pre> make</pre>

274 * The build includes a regression test, which may take a while to complete.

275 *

276 * If the build is successful, you can test the example program:

277 * <pre> examples\\example tests\\alarm.fg</pre>

278 * or the more extensive test program:

279 * <pre> tests\\testdai --aliases tests\\aliases.conf --filename tests\\alarm.fg --methods JTREE_HUGIN BP_SEQMAX</pre>

280 *

281 *

282 * <hr size="1">

283 * \section build-matlab Building the libDAI MatLab interface

284 *

285 * You need:

286 * - MatLab

287 * - The platform-dependent requirements described above

288 *

289 * First, you need to build the libDAI source as described above for your

290 * platform. By default, the MatLab interface is disabled, so before compiling the

291 * source, you have to enable it in Makefile.ALL by setting

292 * <pre> WITH_MATLAB=true</pre>

293 * Also, you have to configure the MatLab-specific parts of

294 * Makefile.conf to match your system (in particular, the Makefile variables ME,

295 * MATLABDIR and MEX). The MEX file extension depends on your platform; for a

296 * 64-bit linux x86_64 system this would be "ME=.mexa64", for a 32-bit linux x86

297 * system "ME=.mexglx". If you are unsure about your MEX file

298 * extension: it needs to be the same as what the MatLab command "mexext" returns.

299 * The required MEX files are built by issuing

300 * <pre> make</pre>

301 * from the command line. The MatLab interface is much less powerful than using

302 * libDAI from C++. There are two reasons for this: (i) it is boring to write MEX

303 * files; (ii) the large performance penalty paid when large data structures (like

304 * factor graphs) have to be converted between their native C++ data structure to

305 * something that MatLab understands.

306 *

307 * A simple example of how to use the MatLab interface is the following (entered

308 * at the MatLab prompt), which performs exact inference by the junction tree

309 * algorithm and approximate inference by belief propagation on the ALARM network:

310 * <pre> cd path_to_libdai/matlab

311 * [psi] = dai_readfg ('../examples/alarm.fg');

312 * [logZ,q,md,qv,qf] = dai (psi, 'JTREE', '[updates=HUGIN,verbose=0]')

313 * [logZ,q,md,qv,qf] = dai (psi, 'BP', '[updates=SEQMAX,tol=1e-9,maxiter=10000,logdomain=0]')</pre>

314 * where "path_to_libdai" has to be replaced with the directory in which libDAI

315 * was installed. For other algorithms and some default parameters, see the file

316 * tests/aliases.conf.

317 *

318 * <hr size="1">

319 * \section build-doxygen Building the documentation

320 *

321 * Install doxygen, graphviz and a TeX distribution and use

322 * <pre> make doc</pre>

323 * to build the documentation. If the documentation is not clear enough, feel free

324 * to send me an email (or even better, to improve the documentation and send a patch!).

325 * The documentation can also be browsed online at http://www.libdai.org.

326 */

329 /** \page changelog Change Log

330 * \verbinclude ChangeLog

331 */

334 /** \page terminology Terminology and conventions

335 *

336 * \section terminology-graphicalmodels Graphical models

337 *

338 * Commonly used graphical models are Bayesian networks and Markov random fields.

339 * In libDAI, both types of graphical models are represented by a slightly more

340 * general type of graphical model: a factor graph [\ref KFL01].

341 *

342 * An example of a Bayesian network is:

343 * \dot

344 * digraph bayesnet {

345 * size="1,1";

346 * x0 [label="0"];

347 * x1 [label="1"];

348 * x2 [label="2"];

349 * x3 [label="3"];

350 * x4 [label="4"];

351 * x0 -> x1;

352 * x0 -> x2;

353 * x1 -> x3;

354 * x1 -> x4;

355 * x2 -> x4;

356 * }

357 * \enddot

358 * The probability distribution of a Bayesian network factorizes as:

359 * \f[ P(\mathbf{x}) = \prod_{i\in\mathcal{V}} P(x_i \,|\, x_{\mathrm{pa}(i)}) \f]

360 * where \f$\mathrm{pa}(i)\f$ are the parents of node \a i in a DAG.

361 *

362 * The same probability distribution can be represented as a Markov random field:

363 * \dot

364 * graph mrf {

365 * size="1.5,1.5";

366 * x0 [label="0"];

367 * x1 [label="1"];

368 * x2 [label="2"];

369 * x3 [label="3"];

370 * x4 [label="4"];

371 * x0 -- x1;

372 * x0 -- x2;

373 * x1 -- x2;

374 * x1 -- x3;

375 * x1 -- x4;

376 * x2 -- x4;

377 * }

378 * \enddot

379 *

380 * The probability distribution of a Markov random field factorizes as:

381 * \f[ P(\mathbf{x}) = \frac{1}{Z} \prod_{C\in\mathcal{C}} \psi_C(x_C) \f]

382 * where \f$ \mathcal{C} \f$ are the cliques of an undirected graph,

383 * \f$ \psi_C(x_C) \f$ are "potentials" or "compatibility functions", and

384 * \f$ Z \f$ is the partition sum which properly normalizes the probability

385 * distribution.

386 *

387 * Finally, the same probability distribution can be represented as a factor graph:

388 * \dot

389 * graph factorgraph {

390 * size="1.8,1";

391 * x0 [label="0"];

392 * x1 [label="1"];

393 * x2 [label="2"];

394 * x3 [label="3"];

395 * x4 [label="4"];

396 * f01 [shape="box",label=""];

397 * f02 [shape="box",label=""];

398 * f13 [shape="box",label=""];

399 * f124 [shape="box",label=""];

400 * x0 -- f01;

401 * x1 -- f01;

402 * x0 -- f02;

403 * x2 -- f02;

404 * x1 -- f13;

405 * x3 -- f13;

406 * x1 -- f124;

407 * x2 -- f124;

408 * x4 -- f124;

409 * }

410 * \enddot

411 *

412 * The probability distribution of a factor graph factorizes as:

413 * \f[ P(\mathbf{x}) = \frac{1}{Z} \prod_{I\in \mathcal{F}} f_I(x_I) \f]

414 * where \f$ \mathcal{F} \f$ are the factor nodes of a factor graph (a

415 * bipartite graph consisting of variable nodes and factor nodes),

416 * \f$ f_I(x_I) \f$ are the factors, and \f$ Z \f$ is the partition sum

417 * which properly normalizes the probability distribution.

418 *

419 * Looking at the expressions for the joint probability distributions,

420 * it is obvious that Bayesian networks and Markov random fields can

421 * both be easily represented as factor graphs. Factor graphs most

422 * naturally express the factorization structure of a probability

423 * distribution, and hence are a convenient representation for approximate

424 * inference algorithms, which all try to exploit this factorization.

425 * This is why libDAI uses a factor graph as representation of a

426 * graphical model, implemented in the dai::FactorGraph class.

427 *

428 * \section terminology-inference Inference tasks

429 *

430 * Given a factor graph, specified by the variable nodes \f$\{x_i\}_{i\in\mathcal{V}}\f$

431 * the factor nodes \f$ \mathcal{F} \f$, the graph structure, and the factors

432 * \f$\{f_I(x_I)\}_{I\in\mathcal{F}}\f$, the following tasks are important:

433 *

434 * - Calculating the partition sum:

435 * \f[ Z = \sum_{\mathbf{x}_{\mathcal{V}}} \prod_{I \in \mathcal{F}} f_I(x_I) \f]

436 * - Calculating the marginal distribution of a subset of variables

437 * \f$\{x_i\}_{i\in A}\f$:

438 * \f[ P(\mathbf{x}_{A}) = \frac{1}{Z} \sum_{\mathbf{x}_{\mathcal{V}\setminus A}} \prod_{I \in \mathcal{F}} f_I(x_I) \f]

439 * - Calculating the MAP state which has the maximum probability mass:

440 * \f[ \mathrm{argmax}_{\mathbf{x}}\,\prod_{I\in\mathcal{F}} f_I(x_I) \f]

441 *

442 * libDAI offers several inference algorithms, which solve (a subset of) these tasks either

443 * approximately or exactly, for factor graphs with discrete variables. The following

444 * algorithms are implemented:

445 *

446 * Exact inference:

447 * - Brute force enumeration: dai::ExactInf

448 * - Junction-tree method: dai::JTree

449 *

450 * Approximate inference:

451 * - Mean Field: dai::MF

452 * - (Loopy) Belief Propagation: dai::BP [\ref KFL01]

453 * - Fractional Belief Propagation: dai::FBP [\ref WiH03]

454 * - Tree-Reweighted Belief Propagation: dai::TRWBP [\ref WJW03]

455 * - Tree Expectation Propagation: dai::TreeEP [\ref MiQ04]

456 * - Generalized Belief Propagation: dai::HAK [\ref YFW05]

457 * - Double-loop GBP: dai::HAK [\ref HAK03]

458 * - Loop Corrected Belief Propagation: dai::MR [\ref MoR05] and dai::LC [\ref MoK07]

459 * - Gibbs sampling: dai::Gibbs

460 * - Conditioned Belief Propagation: dai::CBP [\ref EaG09]

461 *

462 * Not all inference tasks are implemented by each method: calculating MAP states

463 * is only possible with dai::JTree and dai::BP, calculating partition sums is

464 * not possible with dai::MR, dai::LC and dai::Gibbs.

465 *

466 * \section terminology-learning Parameter learning

467 *

468 * In addition, libDAI supports parameter learning of conditional probability

469 * tables by Expectation Maximization (or Maximum Likelihood, if there is no

470 * missing data). This is implemented in dai::EMAlg.

471 *

472 * \section terminology-variables-states Variables and states

473 *

474 * Linear states are a concept that is used often in libDAI, for example for storing

475 * and accessing factors, which are functions mapping from states of a set of variables

476 * to the real numbers. Internally, a factor is stored as an array, and the array index

477 * of an entry corresponds with the linear state of the set of variables. Below we will

478 * define variables, states and linear states of (sets of) variables.

479 *

480 * \subsection terminology-variables Variables

481 *

482 * Each (random) \a variable has a unique identifier, its \a label (which has

483 * a non-negative integer value). If two variables have the same

484 * label, they are considered as identical. A variable can take on a finite

485 * number of different values or \a states.

486 *

487 * We use the following notational conventions. The discrete

488 * random variable with label \f$l\f$ is denoted as \f$x_l\f$, and the number

489 * of possible values of this variable as \f$S_{x_l}\f$ or simply \f$S_l\f$.

490 * The set of possible values of variable \f$x_l\f$ is denoted

491 * \f$X_l := \{0,1,\dots,S_l-1\}\f$ and called its \a state \a space.

492 *

493 * \subsection terminology-variable-sets Sets of variables and the canonical ordering

494 *

495 * Let \f$A := \{x_{l_1},x_{l_2},\dots,x_{l_n}\}\f$ be a set of variables.

496 *

497 * The \a canonical \a ordering of the variables in \a A is induced by their labels.

498 * That is: if \f$l_1 < l_2\f$, then \f$x_{l_1}\f$ occurs before \f$x_{l_2}\f$ in the

499 * canonical ordering. Below, we will assume that \f$(l_i)_{i=1}^n\f$ is

500 * ordered according to the canonical ordering, i.e., \f$l_1 < l_2 < \dots < l_n\f$.

501 *

502 * \subsection terminology-variable-states States and linear states of sets of variables

503 *

504 * A \a state of the variables in \a A refers to a joint assignment of the

505 * variables, or in other words, to an element of the Cartesian product

506 * \f$ \prod_{i=1}^n X_{l_i}\f$ of the state spaces of the variables in \a A.

507 * Note that a state can also be interpreted as a mapping from variables (or

508 * variable labels) to the natural numbers, which assigns to a variable (or its

509 * label) the corresponding state of the variable.

510 *

511 * A state of \a n variables can be represented as an n-tuple of

512 * non-negative integers: \f$(s_1,s_2,\dots,s_n)\f$ corresponds to the

513 * joint assignment \f$x_{l_1} = s_1, \dots, x_{l_n} = s_n\f$.

514 * Alternatively, a state can be represented compactly as one non-negative integer;

515 * this representation is called a \a linear \a state. The linear state

516 * \a s corresponding to the state \f$(s_1,s_2,\dots,s_n)\f$ would be:

517 * \f[

518 * s := \sum_{i=1}^n s_i \prod_{j=1}^{i-1} S_{l_j}

519 * = s_1 + s_2 S_{l_1} + s_3 S_{l_1} S_{l_2} + \dots + s_n S_{l_1} \cdots S_{l_{n-1}}.

520 * \f]

521 *

522 * Vice versa, given a linear state \a s for the variables \a A, the

523 * corresponding state \f$s_i\f$ of the \a i 'th variable \f$x_{l_i}\f$ (according to

524 * the canonical ordering of the variables in \a A) is given by

525 * \f[

526 * s_i = \left\lfloor\frac{s \mbox { mod } \prod_{j=1}^i S_{l_j}}{\prod_{j=1}^{i-1} S_{l_j}}\right\rfloor.

527 * \f]

528 *

529 * Finally, the \a number \a of \a states of the set of variables \a A is simply the

530 * number of different joint assignments of the variables, that is, \f$\prod_{i=1}^n S_{l_i}\f$.

531 */

534 /** \page fileformats libDAI file formats

535 *

536 * \section fileformats-factorgraph Factor graph (.fg) file format

537 *

538 * This section describes the .fg file format used in libDAI to store factor graphs.

539 * Markov Random Fields are special cases of factor graphs, as are Bayesian

540 * networks. A factor graph can be specified as follows: for each factor, one has

541 * to specify which variables occur in the factor, what their respective

542 * cardinalities (i.e., number of possible values) are, and a table listing all

543 * the values of that factor for all possible configurations of these variables.

544 *

545 * A .fg file is not much more than that. It starts with a line containing the

546 * number of factors in that graph, followed by an empty line. Then all factors

547 * are specified, using one block for each factor, where the blocks are seperated

548 * by empty lines. Each variable occurring in the factor graph has a unique

549 * identifier, its label (which should be a nonnegative integer). Comment lines

550 * which start with # are ignored.

551 *

552 * \subsection fileformats-factorgraph-factor Factor block format

553 *

554 * Each block describing a factor starts with a line containing the number of

555 * variables in that factor. The second line contains the labels of these

556 * variables, seperated by spaces (labels are nonnegative integers and to avoid

557 * confusion, it is suggested to start counting at 0). The third line contains

558 * the number of possible values of each of these variables, also seperated by

559 * spaces. Note that there is some redundancy here, since if a variable appears

560 * in more than one factor, the cardinality of that variable appears several

561 * times in the .fg file; obviously, these cardinalities should be consistent.

562 * The fourth line contains the number of nonzero entries

563 * in the factor table. The rest of the lines contain these nonzero entries;

564 * each line consists of a table index, followed by white-space, followed by the

565 * value corresponding to that table index. The most difficult part is getting

566 * the indexing right. The convention that is used is that the left-most

567 * variables cycle through their values the fastest (similar to MatLab indexing

568 * of multidimensional arrays).

569 *

570 * \subsubsection fileformats-factorgraph-factor-example Example

571 *

572 * An example block describing one factor is:

573 *

574 * <pre>

575 * 3

576 * 4 8 7

577 * 3 2 2

578 * 11

579 * 0 0.1

580 * 1 3.5

581 * 2 2.8

582 * 3 6.3

583 * 4 8.4

584 * 6 7.4

585 * 7 2.4

586 * 8 8.9

587 * 9 1.3

588 * 10 1.6

589 * 12 6.4

590 * 11 2.6

591 * </pre>

592 *

593 * which corresponds to the following factor:

594 *

595 * \f[

596 * \begin{array}{ccc|c}

597 * x_4 & x_8 & x_7 & \mbox{value}\\

598 * \hline

599 * 0 & 0 & 0 & 0.1\\

600 * 1 & 0 & 0 & 3.5\\

601 * 2 & 0 & 0 & 2.8\\

602 * 0 & 1 & 0 & 6.3\\

603 * 1 & 1 & 0 & 8.4\\

604 * 2 & 1 & 0 & 0.0\\

605 * 0 & 0 & 1 & 7.4\\

606 * 1 & 0 & 1 & 2.4\\

607 * 2 & 0 & 1 & 8.9\\

608 * 0 & 1 & 1 & 1.3\\

609 * 1 & 1 & 1 & 1.6\\

610 * 2 & 1 & 1 & 2.6

611 * \end{array}

612 * \f]

613 *

614 * Note that the value of \f$x_4\f$ changes fastest, followed by that of \f$x_8\f$, and \f$x_7\f$

615 * varies the slowest, corresponding to the second line of the block ("4 8 7").

616 * Further, \f$x_4\f$ can take on three values, and \f$x_8\f$ and \f$x_7\f$ each have two possible

617 * values, as described in the third line of the block ("3 2 2"). The table

618 * contains 11 non-zero entries (all except for the fifth entry). Note that the

619 * eleventh and twelveth entries are interchanged.

620 *

621 * A final note: the internal representation in libDAI of the factor above is

622 * different, because the variables are ordered according to their indices

623 * (i.e., the ordering would be \f$x_4 x_7 x_8\f$) and the values of the table are

624 * stored accordingly, with the variable having the smallest index changing

625 * fastest:

626 *

627 * \f[

628 * \begin{array}{ccc|c}

629 * x_4 & x_7 & x_8 & \mbox{value}\\

630 * \hline

631 * 0 & 0 & 0 & 0.1\\

632 * 1 & 0 & 0 & 3.5\\

633 * 2 & 0 & 0 & 2.8\\

634 * 0 & 1 & 0 & 7.4\\

635 * 1 & 1 & 0 & 2.4\\

636 * 2 & 1 & 0 & 8.9\\

637 * 0 & 0 & 1 & 6.3\\

638 * 1 & 0 & 1 & 8.4\\

639 * 2 & 0 & 1 & 0.0\\

640 * 0 & 1 & 1 & 1.3\\

641 * 1 & 1 & 1 & 1.6\\

642 * 2 & 1 & 1 & 2.6

643 * \end{array}

644 * \f]

645 *

646 *

647 * \section fileformats-evidence Evidence (.tab) file format

648 *

649 * This section describes the .tab fileformat used in libDAI to store "evidence",

650 * i.e., a data set consisting of multiple samples, where each sample is the

651 * observed joint state of some variables.

652 *

653 * A .tab file is a tabular data file, consisting of a header line, followed by

654 * an empty line, followed by the data points, with one line for each data point.

655 * Each line (apart from the empty one) should have the same number of columns,

656 * where columns are separated by one tab character. Each column corresponds to

657 * a variable. The header line consists of the variable labels (corresponding to

658 * dai::Var::label()). The other lines are observed joint states of the variables, i.e.,

659 * each line corresponds to a joint observation of the variables, and each column

660 * of a line contains the state of the variable associated with that column.

661 * Missing data is handled simply by having two consecutive tab characters,

662 * without any characters in between.

663 *

664 * \subsection fileformats-evidence-example Example

665 *

666 * <pre>

667 * 1 3 2

668 *

669 * 0 0 1

670 * 1 0 1

671 * 1 1

672 * </pre>

673 *

674 * This would correspond to a data set consisting of three observations concerning

675 * the variables with labels 1, 3 and 2; the first observation being

676 * \f$x_1 = 0, x_3 = 0, x_2 = 1\f$, the second observation being

677 * \f$x_1 = 1, x_3 = 0, x_2 = 1\f$, and the third observation being

678 * \f$x_1 = 1, x_2 = 1\f$ (where the state of \f$x_3\f$ is missing).

679 *

680 * \section fileformats-emalg Expectation Maximization (.em) file format

681 *

682 * This section describes the file format of .em files, which are used

683 * to specify a particular EM algorithm. The .em files are complementary

684 * to .fg files; in other words, an .em file without a corresponding .fg

685 * file is useless. Furthermore, one also needs a corresponding .tab file

686 * containing the data used for parameter learning.

687 *

688 * An .em file starts with a line specifying the number of maximization steps,

689 * followed by an empty line. Then, each maximization step is described in a

690 * block, which should satisfy the format described in the next subsection.

691 *

692 * \subsection fileformats-emalg-maximizationstep Maximization Step block format

693 *

694 * A maximization step block of an .em file starts with a single line

695 * describing the number of shared parameters blocks that will follow.

696 * Then, each shared parameters block follows, in the format described in

697 * the next subsection.

698 *

699 * \subsection fileformats-emalg-sharedparameters Shared parameters block format

700 *

701 * A shared parameters block of an .em file starts with a single line

702 * consisting of the name of a ParameterEstimation subclass

703 * and its parameters in the format of a PropertySet. For example:

704 * <pre> CondProbEstimation [target_dim=2,total_dim=4,pseudo_count=1]</pre>

705 * The next line contains the number of factors that share their parameters.

706 * Then, each of these factors is specified on separate lines (possibly

707 * seperated by empty lines), where each line consists of several fields

708 * seperated by a space or a tab character. The first field contains

709 * the index of the factor in the factor graph. The following fields should

710 * contain the variable labels of the variables on which that factor depends,

711 * in a specific ordering. This ordering can be different from the canonical

712 * ordering of the variables used internally in libDAI (which would be sorted

713 * ascendingly according to the variable labels). The ordering of the variables

714 * specifies the implicit ordering of the shared parameters: when iterating

715 * over all shared parameters, the corresponding index of the first variable

716 * changes fastest (in the inner loop), and the corresponding index of the

717 * last variable changes slowest (in the outer loop). By choosing the right

718 * ordering, it is possible to let different factors (depending on different

719 * variables) share parameters in parameter learning using EM. This convention

720 * is similar to the convention used in factor blocks in a factor graph .fg

721 * file (see \ref fileformats-factorgraph-factor).

722 *

723 * \section fileformats-aliases Aliases file format

724 *

725 * An aliases file is basically a list of "macros" and the strings that they

726 * should be substituted with.

727 *

728 * Each line of the aliases file can be either empty, contain a comment

729 * (if the first character is a '#') or contain an alias. In the latter case,

730 * the line should contain a colon; the part before the colon contains the

731 * name of the alias, the part after the colon the string that it should be

732 * substituted with. Any whitespace before and after the colon is ignored.

733 *

734 * For example, the following line would define the alias \c BP_SEQFIX

735 * as a shorthand for "BP[updates=SEQFIX,tol=1e-9,maxiter=10000,logdomain=0]":

736 * <pre>

737 * BP_SEQFIX: BP[updates=SEQFIX,tol=1e-9,maxiter=10000,logdomain=0]

738 * </pre>

739 *

740 * Aliases files can be used to store default options for algorithms.

741 */

743 /** \page bibliography Bibliography

744 * \anchor EaG09 \ref EaG09

745 * F. Eaton and Z. Ghahramani (2009):

746 * "Choosing a Variable to Clamp",

747 * <em>Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS 2009)</em> 5:145-152,

748 * http://jmlr.csail.mit.edu/proceedings/papers/v5/eaton09a/eaton09a.pdf

749 *

750 * \anchor EMK06 \ref EMK06

751 * G. Elidan and I. McGraw and D. Koller (2006):

752 * "Residual Belief Propagation: Informed Scheduling for Asynchronous Message Passing",

753 * <em>Proceedings of the 22nd Annual Conference on Uncertainty in Artificial Intelligence (UAI-06)</em>,

754 * http://uai.sis.pitt.edu/papers/06/UAI2006_0091.pdf

755 *

756 * \anchor HAK03 \ref HAK03

757 * T. Heskes and C. A. Albers and H. J. Kappen (2003):

758 * "Approximate Inference and Constrained Optimization",

759 * <em>Proceedings of the 19th Annual Conference on Uncertainty in Artificial Intelligence (UAI-03)</em> pp. 313-320,

760 * http://www.snn.ru.nl/reports/Heskes.uai2003.ps.gz

761 *

762 * \anchor KFL01 \ref KFL01

763 * F. R. Kschischang and B. J. Frey and H.-A. Loeliger (2001):

764 * "Factor Graphs and the Sum-Product Algorithm",

765 * <em>IEEE Transactions on Information Theory</em> 47(2):498-519,

766 * http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=910572

767 *

768 * \anchor KoF09 \ref KoF09

769 * D. Koller and N. Friedman (2009):

770 * <em>Probabilistic Graphical Models - Principles and Techniques</em>,

771 * The MIT Press, Cambridge, Massachusetts, London, England.

773 * \anchor Min05 \ref Min05

774 * T. Minka (2005):

775 * "Divergence measures and message passing",

776 * <em>MicroSoft Research Technical Report</em> MSR-TR-2005-173,

777 * http://research.microsoft.com/en-us/um/people/minka/papers/message-passing/minka-divergence.pdf

778 *

779 * \anchor MiQ04 \ref MiQ04

780 * T. Minka and Y. Qi (2004):

781 * "Tree-structured Approximations by Expectation Propagation",

782 * <em>Advances in Neural Information Processing Systems</em> (NIPS) 16,

783 * http://books.nips.cc/papers/files/nips16/NIPS2003_AA25.pdf

784 *

785 * \anchor MoK07 \ref MoK07

786 * J. M. Mooij and H. J. Kappen (2007):

787 * "Loop Corrections for Approximate Inference on Factor Graphs",

788 * <em>Journal of Machine Learning Research</em> 8:1113-1143,

789 * http://www.jmlr.org/papers/volume8/mooij07a/mooij07a.pdf

790 *

791 * \anchor MoK07b \ref MoK07b

792 * J. M. Mooij and H. J. Kappen (2007):

793 * "Sufficient Conditions for Convergence of the Sum-Product Algorithm",

794 * <em>IEEE Transactions on Information Theory</em> 53(12):4422-4437,

795 * http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4385778

796 *

797 * \anchor MoR05 \ref MoR05

798 * A. Montanari and T. Rizzo (2005):

799 * "How to Compute Loop Corrections to the Bethe Approximation",

800 * <em>Journal of Statistical Mechanics: Theory and Experiment</em> 2005(10)-P10011,

801 * http://stacks.iop.org/1742-5468/2005/P10011

802 *

803 * \anchor StW99 \ref StW99

804 * A. Steger and N. C. Wormald (1999):

805 * "Generating Random Regular Graphs Quickly",

806 * <em>Combinatorics, Probability and Computing</em> Vol 8, Issue 4, pp. 377-396,

807 * http://www.math.uwaterloo.ca/~nwormald/papers/randgen.pdf

808 *

809 * \anchor WiH03 \ref WiH03

810 * W. Wiegerinck and T. Heskes (2003):

811 * "Fractional Belief Propagation",

812 * <em>Advances in Neural Information Processing Systems</em> (NIPS) 15, pp. 438-445,

813 * http://books.nips.cc/papers/files/nips15/LT16.pdf

814 *

815 * \anchor WJW03 \ref WJW03

816 * M. J. Wainwright, T. S. Jaakkola and A. S. Willsky (2003):

817 * "Tree-reweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching",

818 * <em>9th Workshop on Artificial Intelligence and Statistics</em>,

819 * http://www.eecs.berkeley.edu/~wainwrig/Papers/WJW_AIStat03.pdf

820 *

821 * \anchor YFW05 \ref YFW05

822 * J. S. Yedidia and W. T. Freeman and Y. Weiss (2005):

823 * "Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms",

824 * <em>IEEE Transactions on Information Theory</em> 51(7):2282-2312,

825 * http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1459044

826 */

829 /** \page discussion Ideas not worth exploring

830 * \section discuss_extendedgraphs Extended factorgraphs/regiongraphs

831 *

832 * A FactorGraph and a RegionGraph are often equipped with

833 * additional properties for nodes and edges. The code to initialize those

834 * is often quite similar. Maybe one could abstract this, e.g.:

835 * \code

836 * template <typename Node1Properties, typename Node2Properties, typename EdgeProperties>

837 * class ExtFactorGraph : public FactorGraph {

838 * public:

839 * std::vector<Node1Properties> node1Props;

840 * std::vector<Node2Properties> node2Props;

841 * std::vector<std::vector<EdgeProperties> > edgeProps;

842 * // ...

843 * }

844 * \endcode

845 *

846 * Advantages:

847 * - Less code duplication.

848 * - Easier maintainability.

849 * - Easier to write new inference algorithms.

850 *

851 * Disadvantages:

852 * - Cachability may be worse.

853 * - A problem is the case where there are no properties for either type of nodes or for edges.

854 * Maybe this can be solved using specializations, or using variadac template arguments?

855 * Another possible solution would be to define a "class Empty {}", and add some code

856 * that checks for the typeid, comparing it with Empty, and doing something special in that case

857 * (e.g., not allocating memory).

858 * - The main disadvantage of this approach seems to be that it leads to even more entanglement.

859 * Therefore this is probably a bad idea.

860 *

861 * \section discuss_templates Polymorphism by template parameterization

862 *

863 * Instead of polymorphism by inheritance, use polymorphism by template parameterization.

864 * For example, the real reason for introducing the complicated inheritance scheme of dai::InfAlg

865 * was for functions like dai::calcMarginal. Instead, one could use a template function:

866 * \code

867 * template<typename InfAlg>

868 * Factor calcMarginal( const InfAlg &obj, const VarSet &ns, bool reInit );

869 * \endcode

870 * This would assume that the type InfAlg supports certain methods. Ideally, one would use

871 * concepts to define different classes of inference algorithms with different capabilities,

872 * for example the ability to calculate logZ, the ability to calculate marginals, the ability to

873 * calculate bounds, the ability to calculate MAP states, etc. Then, one would use traits

874 * classes in order to be able to query the capabilities of the model. For example, one would be

875 * able to query whether the inference algorithm supports calculation of logZ. Unfortunately,

876 * this is compile-time polymorphism, whereas tests/testdai needs runtime polymorphism.

877 * Therefore this is probably a bad idea.

878 */