Talk:Principle of maximum entropy

2003 dicussion

I find this discussion very doctrinaire and probably incomprehensible to most mathematicians for lack of context; maybe I'll do some more substantive editing later. Michael Hardy 17:46 Mar 30, 2003

Doctrinaire - a person inflexibly attached to a practice or theory without regard to its practicality. Online dictionary definition. Hey, I just tried to describe what it is - whether or not it's valid is an issue that requires its own subsection. Since most of what I've read on the subject of the validity of PME was written by its proponents, I have the information to give only one side of the story (from as N a POV as I can manage).Cyan 07:37 Apr 1, 2003 (UTC)

I don't claim to be a mathematician, and yet with a few terms of calculus and discrete math under my belt I find this presentation to be very accessible. I don't see how it can be made any more accessible without sacrificing content. I learned a few new things from this page (ie proving that ME solution is also ML solution) that I haven't come across when browsing papers on maxent.

I have some knowledge on how the algorithms that approximate maximum entropy solution work (the GIS and the IIS), if there's demand for it, perhaps I should post some info? yaroslavvb Jun 3, 2003 (PST)

Absolutely. But the PME page is already rather long. I suggest you create a new page (or pages) for these algorithms, and provide links to and from the PME page. Cyan 04:35 5 Jun 2003 (UTC)

<Mild chagrin> See also the second rule 25 on Wikipedia Anti-Rules. Cyan 21:53 Apr 3, 2003 (UTC)

What I meant by "doctrinaire" is that it imitates closely the language of Edwin Jaynes and may be incomprehensible to those unfamiliar with Jaynes' writings. One of these days I'll edit this article, but for the Time Being I have other obligations. Michael Hardy 01:36 Apr 4, 2003 (UTC)

equation minus sign

I think the minus sign on the equations to find lambda values is wrong. I'll remove it. --163.117.155.37 18:01, 12 January 2007 (UTC)

Epistemic probability?

I've never seen that term used. It seems out of place in a mathematical context, and more appropriate to philosophy. I recommend changing it to the more standard term "Bayesian".

Who are you, that the fact that YOU have never seen it should be considered significant? I think it conveys the idea better than "Bayesian". Michael Hardy 21:12, 11 October 2005 (UTC)

I think it's a good term here, underlining that we're talking about probabilities being used to handle a lack of complete knowledge. Bayesian writers are often keen to stress that Bayesian inference and Bayesian methods are part of epistemology -- ie how to handle knowledge and incomplete knowledge; rather than ontology -- statements about the actual nature of the world. They are also clear on the value of always keeping the two clearly distinguished. Jheald 15:36, 20 October 2005 (UTC)

MLE and bayesianism

In the current article, we can read

maximum entropy principle is like other Bayesian methods in that

implying that MLE is a bayesian method. But, up to my knowledge, this claim is controversial (for instance R.Neal said :Maximum entropy is not consistent with Bayesian methods). Should we modify this sentence? For more information on this debate, here is starting a discussion, with good pointers.--Dangauthier 16:48, 16 February 2007 (UTC)

Looked at your blog quickly, didn't work through the example, but the result looks very fishy to me.

Given that the Principle of Maximum Entropy with respect to an invariant measure is essentially the same thing as Kullback's Minimum Discrimination Information (MDI) principle, you might like to look at Relative_entropy#Principle_of_minimum_discrimination_information.

That seems to me to show why MDI should replicate Bayes where Bayes is applicable.

Can you diagnose why your example is different ? Jheald 17:49, 16 February 2007 (UTC)

Even Bayesians disagree what "Bayesian" means. IE, is MAP Bayesian? Or must inference about models use model averaging? There's no consensus. There have been papers showing how MaxEnt can be massaged to look like a special case of Bayesian approach, and vica versa, the whole disagreement is mostly about semantics. --yaroslavvb

There's another discussion of the example here with a useful contribution from John Baez.

Basically, there are two different scenarios that need to be distinguished: Does the constraint only apply to our prior probability assessment, so that once we have formed that prior probability, we can forget the constraint and just apply Bayes' theorem using that prior distribution? Or must the constraint also apply to the posterior distribution - in which case it has to be included as a nuisance variable in its own right in the model, and explicitly conditionalised on in a Bayesian inference. Jheald 23:47, 3 March 2007 (UTC)

There are basically three positions in the literature on the relationship of Maximum Entropy's "constraint rule" and Bayesian conditionalization: 1) the two conflict; 2) the two are complementary; and 3) MaxEnt generalizes Bayesian conditionalization. The somewhat scattered literature on this is surveyed and summarized nicely in:

Uffink, Jos (1996). "The Constraint Rule of the Maximum Entropy Principle" (PDF). Studies in History and Philosophy of Modern Physics. 27 (1): pp. 47-79. {{cite journal}}: |pages= has extra text (help)

Recall that we're just here to summarize the literature, not to ourselves resolve the argument. --Delirium 01:44, 4 August 2007 (UTC)

neccessary and sufficient condition for a sufficient statistic

(what a confusing title...) I was intrigued by the comment about the n&s condition for the existence of a sufficient statistic, so I read Pitman-Koopman_theorem and it says: "...only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases." I rephrased the comment slightly but I know very little about the subject, so if any of you knows a little about the subject please consider fixing it...

Amitushtush 11:30, 22 September 2007 (UTC)

Paradoxes

Jaynes had a lifelong controversy with authors who claimed that his maxent principle would allow paradoxes, most famously the Dawid-Stone-Zideck paradox. After his death, the paradox was acknowledged by Kavin van Horn. I do not see any mention of paradoxes at all. -- Zz (talk) 13:30, 18 January 2008 (UTC)

least biased?

The term "least biased" in the 2nd sentence seems likely to be incorrect, at least if "bias" has its usual statistical meaning Least incorrect? Least additional information? Melcombe (talk) 10:09, 30 April 2008 (UTC) Reading again, perhaps it would be better to just have "It finds the distribution that..."Melcombe (talk) 10:14, 30 April 2008 (UTC)

Relativity

That relativity bit in the lead seems pretty creative to me, and I'm not sure I buy the metaphor. Is this attributable to anyone outside of WP editor(s)? Cretog8 (talk) 03:11, 12 July 2008 (UTC)