[Maspace] Confirmation and refutation

Fri, 03 Jan 2003 12:29:22 -0500

--=====================_-1796913295==.ALT
Content-Type: text/plain; charset="us-ascii"; format=flowed

Hi...

Is there a semantic argument going on here?!  Brian says that "The 
(inescapable) problem with confirmationism is that many theories may be 
supported (i.e. not refuted) by the same evidence..."  This sounds like an 
implicit endorsement of the refutation approach... although I may be 
reading too much into the parenthetical remark.

My problem with Popper (insofar as I am entitled to have a problem with the 
great and the good) is that virtually all theories can be refuted provided 
you open up the system sufficiently.  And this is (probably) precisely what 
Brian is talking about when he talks about "confounding 
complexity".  Popper's approach applies to a world of closed experiments 
wherein individual causal mechanisms and their effects are isolated and 
observed.

In social science closed experiments are impossible.  And... much as we 
might like to style models as computational experiments they are not.  A 
model's referent remains the external, open, and complex world, no matter 
how abstract the model becomes, unless we see ourselves as playing with 
models, just for the sake of it.

On a related line of thought, how do we all feel about the Monte Carlo 
approach in this context -- I've always had a nagging feeling that there is 
a logical problem with it.  Let's say we run a model 1000 times to get a 
feel for the variability in its behavior (its predictions, if you 
like).  We now turn to actual observed reality and note that it falls 
'inside' the 95% confidence interval of those 1000 runs -- ignore for a 
moment the difficulty of making that determination, since it hinges on 
having methods for comparing complicated multivariate observations.  We 
happily conclude that the model is plausible... or can we?  This approach 
actually seems to be turning things on their heads by asking if observable 
reality is plausible assuming the model to be true.  Effectively treating 
the model as a null hypothesis that we would like to prove, which of course 
is completely the opposite of how we are supposed to go about 
things.  Also, the approach seems to ignore the commonplace feeling that 
lots of odd and unexpected things happen as actual history unfolds -- I 
mean, how likely were the events of the last few years?  (Questionably 
elected US President, subsequently rendered electoral gold by unlikely 
terrorist mega-event?).

Unfortunately 'conjecture and refutation' won't get us very far 
either.  Progress by this approach would consist of building dozens of 
models that fail which would eventually (supposedly) narrow down the range 
of models that might be true.  But there doesn't seem to be a great deal of 
point in building models that don't work, does there?  (This has always 
seemed to me a weakness of Popper's approach -- it simply doesn't sound 
like a description of how science progresses.  Admittedly my reading of 
Popper has been superficial at best).

All this leaves still up against Brian's question:
"how do you decide:
(1) that you have correct theory (or model).
(2) that you have enough confirmatory evidence to support your choice."

The answer to both of these seems inescapably to be "by some sort of social 
process of negotiation" and not via any particular technical 
approach.  Proponents of particular theories will always prefer their own 
theory to others, and we will find it next to impossible (in a complex open 
world) to use models to prove or disprove any particular theory, provided 
it passes some minimal standards of logical coherence.  For proof of that 
look no farther than neo-classical economics...  That leaves us in the same 
place as a lot of other social scientists -- arguing the case for our 
theories because we honestly believe in them.  And what's so bad about that?!

Happy New Year!

David

--------------------------------------
Penn State Geography - GeoVISTA Center
University Park, PA 16802, USA
--=====================_-1796913295==.ALT
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi...

Is there a semantic argument going on here?!  Brian says that
"The (inescapable) problem with confirmationism is that many
theories may be supported (i.e. not refuted) by the same
evidence..."  This sounds like an implicit endorsement of the
refutation approach... although I may be reading too much into the
parenthetical remark.

My problem with Popper (insofar as I am entitled to have a problem with
the great and the good) is that virtually all theories can be refuted
provided you open up the system sufficiently.  And this is
(probably) precisely what Brian is talking about when he talks about
"confounding complexity".  Popper's approach applies to a
world of closed experiments wherein individual causal mechanisms and
their effects are isolated and observed.  

In social science closed experiments are impossible.  And... much as
we might like to style models as computational experiments they are
not.  A model's referent remains the external, open, and complex
world, no matter how abstract the model becomes, unless we see ourselves
as playing with models, just for the sake of it.

On a related line of thought, how do we all feel about the Monte Carlo
approach in this context -- I've always had a nagging feeling that there
is a logical problem with it.  Let's say we run a model 1000 times
to get a feel for the variability in its behavior (its predictions, if
you like).  We now turn to actual observed reality and note that it
falls 'inside' the 95% confidence interval of those 1000 runs -- ignore
for a moment the difficulty of making that determination, since it hinges
on having methods for comparing complicated multivariate
observations.  We happily conclude that the model is plausible... or
can we?  This approach actually seems to be turning things on their
heads by asking if observable reality is plausible assuming the model to
be true.  Effectively treating the model as a null hypothesis that
we would like to prove, which of course is completely the opposite of how
we are supposed to go about things.  Also, the approach seems to
ignore the commonplace feeling that lots of odd and unexpected things
happen as actual history unfolds -- I mean, how likely were the events of
the last few years?  (Questionably elected US President,
subsequently rendered electoral gold by unlikely terrorist
mega-event?).

Unfortunately 'conjecture and refutation' won't get us very far
either.  Progress by this approach would consist of building dozens
of models that fail which would eventually (supposedly) narrow down the
range of models that might be true.  But there doesn't seem to be a
great deal of point in building models that don't work, does there? 
(This has always seemed to me a weakness of Popper's approach -- it
simply doesn't sound like a description of how science progresses. 
Admittedly my reading of Popper has been superficial at best).

All this leaves still up against Brian's question:

"how do you
decide: 

(1) that you have correc=
t
theory (or model). 

(2) that you have enough
confirmatory evidence to support your choice."

The answer to both of these seems inescapably to be "by some
sort of social process of negotiation" and not via any particular
technical approach.  Proponents of particular theories will always
prefer their own theory to others, and we will find it next to impossible
(in a complex open world) to use models to prove or disprove any
particular theory, provided it passes some minimal standards of logical
coherence.  For proof of that look no farther than neo-classical
economics...  That leaves us in the same place as a lot of other
social scientists -- arguing the case for our theories because we
honestly believe in them.  And what's so bad about that?!

Happy New Year!

David

-------------------------------------- 

Penn State Geography - GeoVISTA Center 

University Park, PA 16802, USA

--=====================_-1796913295==.ALT--