Final Report 2010 - Revision history

A1161917: /* Aims and Objectives */

2010-10-26T02:24:07Z

Aims and Objectives

← Older revision		Revision as of 11:54, 26 October 2010
Line 6:		Line 6:

	==Aims and Objectives==		==Aims and Objectives==
	For decades the written code that is linked to the Somerton Man Mystery has been undecipherable and delivered no solid information about his history or identity. An initialism is an abbreviation of a sentence or phrase and is formed using the first letters of each word.		For decades the written code that is linked to the Somerton Man Mystery has been undecipherable and delivered no solid information about his history or identity. An initialism is an abbreviation of a sentence or phrase and is formed using the first letters of each word. Results from past year projects indicate that the mystery code is likely to be an initialism with the possibility of a substitution cipher being attached to it. The project aims to bring light to the unsolved mystery by tapping into the wealth of information available on the internet. This is done by first verifying the results from past years, and then creating a web crawler and text parser to specifically search for words and patterns similar to the Somerton Man’s code and using statistical analysis to decipher the results we obtain. Of course the ideal goal of this project would be to crack the code, but this is very unlikely and the key sub-goals of creating the web crawler and text parser are the main focus.
	Results from past year projects indicate that the mystery code is likely to be an initialism with the possibility of a substitution cipher being attached to it. The project aims to bring light to the unsolved mystery by tapping into the wealth of information available on the internet. This is done by first verifying the results from past years, and then creating a web crawler and text parser to specifically search for words and patterns similar to the Somerton Man’s code and using statistical analysis to decipher the results we obtain.

	==Background==		==Background==

A1161917: /* Summary and Conclusions */

2010-10-24T10:44:57Z

Summary and Conclusions

← Older revision		Revision as of 20:14, 24 October 2010
Line 562:		Line 562:
	The main aim of the project was to create a web crawler and text parsing algorithm for use in this project as well as general non-specific use. We believe that we were quite successful in this element because our code is quite capable of parsing through a single file or several directories and storing the results into another text file. The code can be improved by attempting to remove some of the limitations stated earlier.		The main aim of the project was to create a web crawler and text parsing algorithm for use in this project as well as general non-specific use. We believe that we were quite successful in this element because our code is quite capable of parsing through a single file or several directories and storing the results into another text file. The code can be improved by attempting to remove some of the limitations stated earlier.

	This project attempted to prove that the Somerton Man's code belonged to a specific English text type by testing the frequency of initialism occurring; however, based on our testing, we could not find any significant differences in frequency over any of the text types. Perhaps the most promising lead is the lack of results found in The Rubaiyat; it seems incredibly unlikely that such small frequency would occur unless it was intentional. Although it seems highly suspect, more testing is needed to determine if this is significant or not. As stated earlier, testing poetry texts truncated to the same size as The Rubaiyat would prove very beneficial.		This project attempted to prove that the Somerton Man's code belonged to a specific English text type by testing the frequency of initialism occurring; however, based on our testing, we could not find any significant differences in frequency over any of the text types. The results found show that the text type with the greatest amount of initialisms found was the Revised Standard Version of the Bible, however this is only due to the length of the text; the text type with the greatest amount of initialisms found relative to it's length was The Rubaiyat for 3 letter initialisms and science texts for 4 letter initialisms. It is currently undetermined whether this has any significance or not.

			Perhaps the most promising lead is the lack of results found in The Rubaiyat; it seems incredibly unlikely that such small frequency would occur unless it was intentional. Although it seems highly suspect, more testing is needed to determine if this is significant or not. As stated earlier, testing poetry texts truncated to the same size as The Rubaiyat would prove very beneficial.

	Due to the sheer number of different initialisms, the pattern initialism results were not able to be completely analysed and we were unable to determine what kind of substitution cipher, if any, was being used. To further this area of the project, we need to analyse the greatest occurring initialisms for each pattern and cross reference them with every other pattern to narrow down the possible substitutions used. Due to time constraints, this was not able to be completed.		Due to the sheer number of different initialisms, the pattern initialism results were not able to be completely analysed and we were unable to determine what kind of substitution cipher, if any, was being used. To further this area of the project, we need to analyse the greatest occurring initialisms for each pattern and cross reference them with every other pattern to narrow down the possible substitutions used. Due to time constraints, this was not able to be completed.

A1161917: /* Further Research */

2010-10-24T10:42:08Z

Further Research

← Older revision		Revision as of 20:12, 24 October 2010
Line 485:		Line 485:
	Our results were supposed to attempt to find the type of text that the Somerton man’s code is from; however there wasn’t any single text type with much more greater than expected occurrences. In fact, other than The Rubaiyat, there were no probabilistic irregularities found. In order to determine if this is significant to unraveling the Somerton Man’s code, further testing is required. This can be done by running the text parser for more code segments, as well as choosing other texts to obtain results on.		Our results were supposed to attempt to find the type of text that the Somerton man’s code is from; however there wasn’t any single text type with much more greater than expected occurrences. In fact, other than The Rubaiyat, there were no probabilistic irregularities found. In order to determine if this is significant to unraveling the Somerton Man’s code, further testing is required. This can be done by running the text parser for more code segments, as well as choosing other texts to obtain results on.

	Additionally, further testing with poems is needed. Only a small amount of the poem texts tested match the same format as the Rubaiyat (four lines per poem), so tests of other four line poems are needed to determine if the exact initialism results are specific to the Rubaiyat or all/most poems of a similar size. Another method would be to truncate the poem texts to the same size. The Rubaiyat is about 64 poems of 4 lines each, so we can truncate the other poem texts to either 64 poems, or possibly ~~256~~ lines and then determine if there are any differences in the results.		Additionally, further testing with poems is needed. Only a small amount of the poem texts tested match the same format as the Rubaiyat (four lines per poem), so tests of other four line poems are needed to determine if the exact initialism results are specific to the Rubaiyat or all/most poems of a similar size. Another method would be to truncate the poem texts to the same size. The Rubaiyat is about 84 poems of 4 lines each, so we can truncate the other poem texts to either 84 poems, or possibly 300-400 lines and then determine if there are any differences in the results. While this truncation method has been done for the tested set of poetry texts, the results can be more accurate with more poetry texts.

	Also, at the moment, our results assume that all of the words in the text and in the code are independent of each other. We know this isn’t actually the case and that there is some sort of dependence between initial letters of text. Our results could be expanded by taking this into account and using Markov chains similar to last year’s project group<ref>https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/Markov_models</ref>.		Also, at the moment, our results assume that all of the words in the text and in the code are independent of each other. We know this isn’t actually the case and that there is some sort of dependence between initial letters of text. Our results could be expanded by taking this into account and using Markov chains similar to last year’s project group<ref>https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/Markov_models</ref>.

A1161917: /* Discussion */

2010-10-24T10:37:58Z

Discussion

@@ Line 281: / Line 281: @@
 [[Image:compare poems1.jpg|center|530px]]
-[[Image:compare poems2.jpg|center|530px]]
-[[Image:compare poems3.jpg|center|530px]]
 [[Image:compare poems4.jpg|center|530px]]
 These exact initialism results indicate that the Somerton man’s code could be a substitution cipher of initialisms found in the Rubaiyat, based on the fact that it is incredibly unlikely that none of the tested code segments were found in the Rubaiyat. To further test this theory, we began testing for patterns of initialisms to see if there was any correlation between texts and attempt to narrow down what the code has been substituted from.

A1161917: /* Discussion */

2010-10-24T10:24:29Z

Discussion

← Older revision		Revision as of 19:54, 24 October 2010
Line 277:		Line 277:

	Of particular note was that neither of the 4 or 3 character long code segments appeared much at all in the Rubaiyat. This seems quite highly suspect, and there is a possibility that someone intentionally created the mystery code in such a way that this would happen because is it very unlikely that almost none of the code segments would be found in the Rubaiyat. More testing is probably needed to prove this by testing more poems, and possibly truncating other poems to the same size as the Rubaiyat.		Of particular note was that neither of the 4 or 3 character long code segments appeared much at all in the Rubaiyat. This seems quite highly suspect, and there is a possibility that someone intentionally created the mystery code in such a way that this would happen because is it very unlikely that almost none of the code segments would be found in the Rubaiyat. More testing is probably needed to prove this by testing more poems, and possibly truncating other poems to the same size as the Rubaiyat.

			To test if The Rubaiyat results were meaningful or just an artifact of length, the same assorted poetry texts were truncated to a size of 400 to 500 lines down from several thousand (a similar size to The Rubaiyat text file) and then the same tests were run again. The results for this are shown in the graphs below.

			[[Image:compare poems1.jpg\|center\|530px]]
			[[Image:compare poems2.jpg\|center\|530px]]
			[[Image:compare poems3.jpg\|center\|530px]]
			[[Image:compare poems4.jpg\|center\|530px]]

	These exact initialism results indicate that the Somerton man’s code could be a substitution cipher of initialisms found in the Rubaiyat, based on the fact that it is incredibly unlikely that none of the tested code segments were found in the Rubaiyat. To further test this theory, we began testing for patterns of initialisms to see if there was any correlation between texts and attempt to narrow down what the code has been substituted from.		These exact initialism results indicate that the Somerton man’s code could be a substitution cipher of initialisms found in the Rubaiyat, based on the fact that it is incredibly unlikely that none of the tested code segments were found in the Rubaiyat. To further test this theory, we began testing for patterns of initialisms to see if there was any correlation between texts and attempt to narrow down what the code has been substituted from.

A1161917: /* Method */

2010-10-24T10:19:33Z

Method

← Older revision		Revision as of 19:49, 24 October 2010
Line 263:		Line 263:

	[[Image:3 letter greatest hits.jpg\|center\|1050px]]		[[Image:3 letter greatest hits.jpg\|center\|1050px]]


	[[Image:4 letter greatest hits.jpg\|center\|1050px]]		[[Image:4 letter greatest hits.jpg\|center\|1050px]]

A1161917: /* Method */

2010-10-24T10:18:44Z

Method

← Older revision		Revision as of 19:48, 24 October 2010
Line 260:		Line 260:

	A raw comparison of actual number of hits can be seen in the tables below.		A raw comparison of actual number of hits can be seen in the tables below.


	[[Image:3 letter greatest hits.jpg\|center\|1050px]]		[[Image:3 letter greatest hits.jpg\|center\|1050px]]
	[[Image:4 letter greatest hits.jpg\|center\|1050px]]		[[Image:4 letter greatest hits.jpg\|center\|1050px]]


	The table shows the total about of forward and backward initialisms found in each different text, as well as the total amount of words found in the text. This is important because longer texts will obviously have a greater chance of having more initialisms, so in order to properly compare between texts, we have to determine how many initialisms were found relative to the length of the texts (which is the row titled "Ratio Relative to Total"). This is done for forward and backward initialisms separately as well as together.		The table shows the total about of forward and backward initialisms found in each different text, as well as the total amount of words found in the text. This is important because longer texts will obviously have a greater chance of having more initialisms, so in order to properly compare between texts, we have to determine how many initialisms were found relative to the length of the texts (which is the row titled "Ratio Relative to Total"). This is done for forward and backward initialisms separately as well as together.

A1161917: /* Method */

2010-10-24T10:17:03Z

Method

← Older revision		Revision as of 19:47, 24 October 2010
Line 261:		Line 261:
	A raw comparison of actual number of hits can be seen in the tables below.		A raw comparison of actual number of hits can be seen in the tables below.

	[[Image:3 letter greatest hits.jpg\|center\|~~530px~~]]		[[Image:3 letter greatest hits.jpg\|center\|1050px]]
	[[Image:4 letter greatest hits.jpg\|center\|~~530px~~]]		[[Image:4 letter greatest hits.jpg\|center\|1050px]]

			The table shows the total about of forward and backward initialisms found in each different text, as well as the total amount of words found in the text. This is important because longer texts will obviously have a greater chance of having more initialisms, so in order to properly compare between texts, we have to determine how many initialisms were found relative to the length of the texts (which is the row titled "Ratio Relative to Total"). This is done for forward and backward initialisms separately as well as together.

			From the tables, we can see that the text with the greatest amount of initialisms found was the Revised Standard Edition of the Bible. However, this is like to only be the case because the text files in the RSV Bible have far greater words in them compared to the Rubaiyat or poems. We can actually see that relative to the length of the texts tested, The Rubaiyat has the most hits for 3 letter initialisms whilst the science texts has the most hits for the 4 letter initialisms.

	====Discussion====		====Discussion====

A1161917: /* Method */

2010-10-24T10:00:56Z

Method

← Older revision		Revision as of 19:30, 24 October 2010
Line 257:		Line 257:

	The remaining direct initialism graphs can be found on [[Direct Initialism Graphs\|this page]].		The remaining direct initialism graphs can be found on [[Direct Initialism Graphs\|this page]].


			A raw comparison of actual number of hits can be seen in the tables below.

			[[Image:3 letter greatest hits.jpg\|center\|530px]]
			[[Image:4 letter greatest hits.jpg\|center\|530px]]

	====Discussion====		====Discussion====

A1161917: /* Method */

2010-10-22T04:57:15Z

Method

← Older revision		Revision as of 14:27, 22 October 2010
Line 248:		Line 248:
	Where n is the size of the segment we are looking for.		Where n is the size of the segment we are looking for.

	Using the results, several tables and graphs were generated to simplify viewing. The raw tables and excel sheets can be found in the appendix. Some of the graphs are below.		Using the results, several tables and graphs were generated to simplify viewing. The raw tables and excel sheets can be found in the appendix. Some of the graphs are below. The graphs indicate the proportions of each sequence found in the selected texts; the red bar graphs showing the code sequence found as is in the Somerton Man's code and the blue bar graphs showing the reverse of the sequence (in cases where the sequence is a palindrome, a blue bar graph is displayed for both). The far left bar of each graph shows the expected probability of the sequence occurring in regular English text(found using the table above), and then shows the actual proportions found in each of the text types. This was done in order to easily compare whether the sequence was found more or less than expected for each text type.