George, my main use of noise reduction is in recording music, when the noise is usually (strictly speaking) an unwanted part of the signal I’ve put in myself, such as mains hum or amplifier hiss. So I will be distinguishing noise from signal by selecting for sampling a section where my repetitive but creative bass is not included to be considered noise. Too much noise reduction destroys signal too.
I assume that all noise reduction algorithms will have some similarly intelligent criteria for deciding in advance what will qualify as signal. And that underlines to me that “signal” cannot be considered apart from some assumptions about both its sender and the receiver.
Jon, the discussion seems to focus on entropy, and this seems to be equated with noise. Entropy in my field is a property of a system and the energetics of reactions are understood in terms of making/breaking chemical bonds, and the entropy of the resulting products. e,g, let us say we break up H2O into H2 and O2 (2H2O => 2H2 + O2). We need energy to break the H-O bonds, but the entropy increases because we have gases instead of a liquid. In this case, we understand what entropy means.
I have frequently dealt with instruments that have a given noise to signal feature, and we often reduce noise by accumulating the signal of interest over a number of scans. In this way, the signal strength grows, and the noise is reduced as a consequence. In this case, I cannot see how entropy would be relevant.
If we consider a receiver that simply responds to every audio (and perhaps other parts of the spectrum) signal, we would have a jumble, but that jumble would contain information on every audio+ source that emits the various frequencies - the jumble(if I understand Joshua correctly) would be very high disorder (the meaning of disorder may also be entropy for this example), and have high information content. It may be extremely difficult for us to extract all of the information, although experts can identify some portions of such a jumble (eg a car horn)…
I agree that we must accept a source(s) for these matters, and a receiver. In terms of digital signals that are sent across fibres or wires, the technical details would be more complicated, but I think we still must think of sender, medium, receiver as the overall context.
When information is discussed for DNA, I tend to retreat, as the context is so vastly different.
Perhaps @Swamidass Joshua may wish to elucidate further.
I hope that I can jump in and ask a question somewhat tangential to the conversation. Since ID has been discussed can you tell me what is intelligence from an information science point of view?
Rather I am saying that the most potent source of information (but not the exclusive source) is noise or randomness. So a better (and not mathematically precise) equation my qualitatively be…
entropy = information = noise + mechanism + intelligent_input
We can further break down noise into…
noise = indepedent_noise + shared_noise
And mechanism into…
mechanism = known_mechanism + unknown_mechanism
So what you are saying here is correct…
What you are doing here is subtracting out the noise that can be traced to the instrument.
If you are averaging across multiple runes, the final processed signal is a type of mutual information, where you look for the information incommon (which you call signal here).
If you are improving the instrumentation, you are finding ways to remove noise from the process, so that it pollutes the signal less_, and this will reduce the information in the data._
If you are using Fourier like this, the information content does not increase it stays constant. But if you know the frequencies that are relevant to your question and those that are likely to be noisy (because you know the mechanism by which some of the noise is produced), you can delete or zero out the frequencies that you know are noise or irrelevant.
If you are using an MP3 encoder to compress a song, this is how it works. Deleting the frequencies that cannot be heard by the human ear.
If you are using a lossy JPEG/PNG compression, this also is how it works, deleting the frequencies (or wavelets) most difficult for the human eye to perceive.
In all these cases, removing noise deletes information from the data, and usually quite a bit of information. Because the noise signal has a very high amount of entropy.
And what are we left with? We still have information in the remaining data…
Now if we have a good model of the mechanism, we could see how well it fits, the data to estimate known_mechanism. That we can now subtract, which will give us the unexplained information (or the error of the final model)…
How much of the poor fit is explained by each of these components? There is no way from information theorey alone to discriminate these. There are some ways (based on replicates in physics and neutral theory in biology) to estimate the information content of the independent noise. In biology we think is very high (noise to signal is about 1000 to 1), but these estimates are doubted by skeptics of evolution.
That is my point.
Not all information is noise, but noise is information too. In physics where we do tightly controlled experiments we can reduce noise, but can never completely eliminate it. In biology, however, noise is much much higher. Even when we do experiments, our replicates are not high enough to reduce it to the levels that physicists are accustomed to. In the case of evolution (just like is the case for much in astrophysics), we cannot do well controlled experiments for many important questions, so we are left without way to reduce noise to zero.
Then we fit models to the data, and see error.
How much of that error is noise? Unknown mechanisms? Or the input of intelligence? We do not know.
So even when you do all this work to remove noise from your data, you might actually be throwing away semantic information. There is absolutely no way to be sure what is noise
Also (in my field), we are finding out there is important information in images that is not perceptible to humans. We thought it was noise, but it is not.
Small diversion here … you’ll all be happy to know I downloaded the entire internet during a slow patch yesterday afternoon. There’s a lot there! And I ran the whole thing through repeated data compressions to make it more manageable for storage. Here is my completely compressed result: 1
I have to warn you though; the decompression algorithm is a monster!
(probably old programmer humor that I heard a long time ago somewhere.)
…and while I’m digging holes, let’s throw in a bit of Steven Wright humor: “I read the dictionary the other day. Figured that was easier than having to read every book.” or this: I really wish my first word as a child had been ‘quote’. Then just before I die I could say ‘unquote’.
Joshua, I may (I hope) be slowly catching on that you are using ‘information’ in the broadest sense of that word, and that what you have called ‘semantic’ information is usually only a tiny subset of that whole. To some one who is admiring the Mona Lisa, the actual textural peculiarities of some particular brush stroke in a corner may be noise so far as their enjoyment of the whole painting is concerned, but it would absolutely NOT be noise to the investigator inspecting it to see if it is a forgery. It sounds like what is considered the ‘semantic’ portion of our larger body of information is determined by the present interests of the beholder. Is that right?
Thanks for the detailed and thoughtful response Joshua.
I must confess I am even more puzzled as to the meaning of entropy in your field. I understand that background “noise” can just as easily be understood as a huge amount of data indiscriminately accumulated by a detector (manages to detect the entire spectrum), and that we may only process (or comprehend) a small portion, and for the purposes of an experiment, we may find ways to eliminate the unknown “noise” and so isolate the information of interest.
But if theoretically, we can examine every portion of the spectrum, we would obtain a great deal of information (frequencies, amplitude, equated with various sources etc).
So where does this notion of entropy = information come into all of this?
In biology, I understand attempts at using four entities in DNA in virtually endless combinations may be seen analogous to 0 1 combination in computing. However computing has an intelligent source, and a given medium (electronics) with a comprehensible output. So how would you characterise the context of your [quote=“Swamidass, post:67, topic:35327”]
to estimate the information content of the independent noise. In biology
[/quote]
I am sorry, but although I find this discussion fascinating, I cannot see how you can make your latter point re biology.
Perhaps you may be saying there is a hope that if a mechanism is found, then some sense can be made of the huge noise to signal ratio - at this point in time, you comments may suggest you have “noise” but no signal. At least that is how it come over.
As one simple example, let us consider the human genome.
We can ask how much of the genome is explained by common descent. In evolution, we look to see the overlap between the information in the human genome and that in other organisms. That brings us to the human and chimpanzee genomes. By some measures, they are about 98% the same. A large proportion of proteins coding sequences, are identical.
Now, we can ask how much of the difference is explained. Turns out that neutral theory gives us a formula.
Today’s Distance = 2 * Rate of Change * Time Since Separation
For the relevant quantities for this specific measure of distance, (which can be directly measured), we get an estimate of 2%. So now, at this level of detail, we have quant explanation for nearly 100% of the information in the human genome. There is a small amount of error here. How much is noise? How much is a mechanism we missed? How much of it mutations directly inspired by God? We have no idea.
We can do the same exercise with mice and rat genomes, and their differences. Remember, they are about 20% different from us (using the same metric that puts humans and chimps just 2% apart). Using the same formula, we we get close (about 2 to 3 fold off, well within the formulas error) to explaining 100% of the information again. How much left is noise, mechanism or God? We just do not know.
We can ask this question in more precise and careful ways too. We find different rates in Y chromosomes. Sure enough, this corresponds to a difference in divergences too. This precise patterns hold up at multiple scales of the genome.
Even if evolution is wrong, it quantitatively explains massive amount of precise information in our genomes. Does it explain all? No. How much of the unexplained information is noise, new mechanisms, or God’s action? We do not know. And it hard to imagine a theory, other than common descent, that could approach evolutions’ ability to explain >99% of the data this way.
So, no, biology it is not that there is no signal. Rather the signal is very strong. The problem is that the math has large error bars, and we cannot do more replicates. For this reason, we cannot rule out God’s action, but we cannot demonstrate it either.
I guess we have found our area of puzzlement and skepticism - in my world, if a theory were to explain nearly 100% of the data, the error bars in our maths would be extremely small and I for one, would proclaim a mechanism that is quantitatively and qualitatively robust.
Yet with ToE we are still searching for some mechanistic explanation, while you chaps continue (from Darwin on) with arguments from similarity and analogical explanations.
I am not sure what I can make of your formula, since from your own admission, we cannot reproduce (or perhaps test) it - so what does the difference or similarity in the genome amount to? Certainly I cannot for the life of me see entropy in this, nor some formula that says, X = human, Y = chimp and the noise is Z…
So I think I understand what you may mean by [quote=“Swamidass, post:72, topic:35327”]
Even if evolution is wrong, it quantitatively explains massive amount of precise information in our genomes.
[/quote]
but if my comment about X=human and Y = chimp cannot be made from current data and a testable formula, then I cannot see a sound argument, even if you are satisfied with an explanation.
I am sorry Joshua, but explanations have been given in the Sciences for decades and centuries, but they are all opinions - those that have been tested and withstood withering sceptical enquiry have remained.
Can you see why this statement from you is problematic? I read that to mean your belief in the signal from evolutionary biology is strong, but you at least acknowledge the error bars are large.
That is neither noise nor information but “potential information” as we will never give up hope to decipher it one day. This is not hypothetical - we have many such data streams from the past.
I am not sure what this means. A search shows this Wikipedia quote: "Entropy is a measure of unpredictability of the state, or equivalently, of its average information content. " Hence Entropy is just an attribute of Information.
Claude Elwood Shannon (April 30, 1916 – February 24, 2001) was an American mathematician, electrical engineer, and cryptographer known as “the father of information theory”. His focus was information, not entropy or anything else.
That’s what I have been saying for a while - a key is absolutely mandatory. The link you provide contains this: "Assume two pads of paper containing identical random sequences of letters were somehow previously produced and securely issued to both. "
Exactly. As long as they don’t issue us a key.
I am not sure what you want and why you seem perpetually hostile. Perhaps you should refrain from replying to my comments.
Yes. The signal (data) can be anything until deciphered. That’s when it becomes information. For that we need a key. Your random tweets can be mistaken for information but not for long. Mistakes - info as noise and noise as info - happen all the time, but noise looking like info never lasts too long (info as noise is just a bad communication channel).
Noise just cannot generate information. That’s why all “infinite monkey” experiments have failed, that’s why “randomness” is just being abused in evolution talk, and that’s why nothing arises in this universe without a will. http://nonlin.org/random-abuse/ http://nonlin.org/arising-of-everything/
Going back to the original claim, Information ≠ Entropy because the key needs to be transmitted too. More than that, some redundant data need to be sent from time to time to re-synchronize the sender and receiver and this reduces the entropy.
One: I don’t see you making any headway in understanding what @Swamidass and a few others are trying to teach you about how they make a living.
Two: Before anyone even knew anything about DNA, genetic information was creating millions of life forms every day. Doesn’t this pretty much defeat your logic completely? You insist information can only exist with a “key” … so based on your logic, for thousands (millions?) of years, DNA had zero information and promulgated zero information. . . . because there were no humans to Interpret what the DNA was doing?
This is just a statement that biology (for obvious reasons) does not work like physics. Mutation rates change over time. And there is uncertainty in assigning divergence times. Using experimental data and the fossil record, we can put our best estimates in and make a computation. The data we observe is within the error bars, but the error bars are not very tight.
Now, we can do this over and over again (in different organisms and different parts of the genome). This is a way of “repeating” the experiment. Taken together, we end up with very high confidence that our model is not a fluke. It is predicting something in the data that nothing else can.
I assure you, evolution has withstood “withering sceptical inquiry and remained.” I am using “explain” in a very precise sense. So please do not be condescending.
Evolution remains the only theory we have that quantitatively explains (models? predicts? computes?) why humans and chimps are 10x less different than mice and rats. This is no different than saying the BAO model is the only one that explains the frequency pattern in CMB radiation, or that relativity is the theory that explains why time slows with acceleration.
I will do my best not to appear condescending, but you should make an effort to follow the line of reasoning in this exchange.
We are discussing noise. error, information, and ToE. You brought up a 98% figure regarding the same DNA between humans and chimps. I have pointed out that any correlation in any discipline of the physical sciences that has a correlation of 98%, or an error of 2%, is in virtually all cases the same.
Chimps are not humans, and humans are not chimps. This has nothing to do with any theory, be it common descent, or anything else - it is a line of reasoning you propose.
I am at a loss on how to respond to this - if biology is obviously different, than why bring in physics theories as being no different to those of biology, to your discussion? Physics looks at errors bars very closely and I cannot find instances where large errors are dismissed as , “oh well that is physics”, or high correlations in data as a convenient way to support a notion, even if other observations are inconsistent.
In any event, if this discussion has reached an end for useful and meaningful exchanges, I will be happy to end it here.
If entropy is disorder, and information is found in DNA, than expound on this.
My apologies for not noticing sooner your comment directed towards me.
I think you misunderstand my response to you. I’m not perpetually hostile towards you. However, it might seem that way because when I see you dismiss facts as mere opinion, or even as untrue, it compels me to call you out on such behavior.
What do I want? I want you to show some ability to cope with the reality that some things people are explaining to you are real and valid.
In the realm of information, @Swamidass is not offering up to you something controversial or something still waiting for more evidence to verify. It’s rock solid; it’s what he works with professionally.
Indeed, In your very last response to a posting from me, I specifically responded to your comments; your reply? You answered that I didn’t respond to your comments and refused to further respond.
As you can see, this did not go over well with the adminstrators. If you are going to participate on these boards, your obligation is to be able to acknowledge the difference between fact vs.opinion, rather than to treat everything written by someone else here as mere opinion.
Perhaps you are too entrenched in the discussion methods you employ in your own blog, where you make all the rules and you daily dismiss whatever you choose?
There is a common saying among school Science Teachers: “If it wriggles, it’s Biology, if it fizzes, it’s Chemistry, and if it doesn’t work, it’s Physics.” And there are numerous jokes along the lines of the Spherical Cow (see Wikipedia).
Not that that’s anything to do with this thread, I just thought it would lighten it up!
There was a context here you are forgetting. Biology is (1) not physics, but (2) it is like physics in some specific ways.
Point one, biology is not physics. For example we discussed a formula based on mutation rates that is accurate. However, mutation rates change over time, so the final predictions have a larger error range than if mutation rates did not change. To draw an analogy, the gravitational constant does not change over time, so we can predict its effect on objects with greater precision. One way to explain this is the formulas in near neutral theory and population genetics are very accurate, but they are also low precision and we often low replicates. In physics, laws are higher precision and there are many more replicates which even further reduces the error bars; it is just a cleaner system.
Point two, my use of the words “explains” is in the same way as it is used in physics, which means there is a quantitative and mechanistic theory that can compute accurate predictions based on some data to predict other data. This is exactly what we do with evolutionary theory. The fact that we have lower precision estimates does not change the fact that neutral theory “explains” almost all the information we see in genomes. This is one way biology is like physics.
Point one and two are not in conflict. Biology is (1) not physics, but (2) it is like physics in some specific ways.
And point two is the reason why this dismissal of evolutionary theory is a grave misrepresentation of the situation. The “data” and “information” I am sharing here is not mere opinion with no basis. The fact that evolution explains so much of biology is a fact, even if we find evolution itself is ultimately wrong. I cannot be glibly dismissed.
If evolution is false, we need some explanation for why its theory explains so much of biology.
