Editing
Cracking the Voynich code 2015
Jump to navigation
Jump to search
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
== Supervisors == *[[Derek Abbott|Prof Derek Abbott]] *[[Brian W.-H. Ng|Dr Brian Ng]] *[[Maryam Ebrahimpour]] ==Honours students== *[[Andrew McInnes]] *[[Lifei Wang]] ==Project guidelines== *[http://www.eleceng.adelaide.edu.au/personal/dabbott/project_handbook_2009.pdf Project Handbook] ==General project description== The Voynich Manuscript is a mysterious 15th century book that no one today know what it says or who wrote it. The book is in a strange alphabet. See details [https://en.wikipedia.org/wiki/Voynich_manuscript here]. Fortunately the whole book has been converted into an electronic format with each character changed to a convenient ascii character. We want you to write software that will search the text and perform statistical tests to get clues as to the nature of the writing. Does the document bear the statistics of a natural language or is it a fake? We already have Support Vector Machine (SVM) and Multiple Discriminant Analysis (MDA) software that you can adapt for your purposes. This software is set up to test if two texts are written by the same author or not. The great thing about our software is that it is independent of language. So you could compare it against the existing writings of Roger Bacon, who is a suspected author ==Useful notes== * Download the digital Voynich from [http://www.ic.unicamp.br/~stolfi/voynich/99-01-16-concordance/ here]. * The UN Declaration of Human Rights is translated into every language in the world and in principle you can compare the Voynich to all the existing languages for statistical proximity. Electronic access is [http://www.ohchr.org/EN/UDHR/Pages/Introduction.aspx here]. ==Specific tasks== * '''Phase 1:''' Characterize the text. Write scripts that count its features. How many words? How many tokens? Word frequencies? Compare these in a table with known languages obtained by running your same code on the Declaration of Human Rights. Don't forget to get a short paragraph of English and manually count everything and then run it on your code to cross check it is counting correctly. You must always validate your code or you will lose marks. Tabulate results for all 16 versions of the Voynich. Which Voynich tokens ''only'' appear at the start of words and which are ''only'' at the end? * '''Phase 2:''' Investigate just using English text, how you can separate the alphabet from other ascii tokens such as &, $, (, ), +, =, 3, etc. If you were an alien without ''a priori'' knowledge, how would you do it? Characterize English text to see how token frequency, token recurrence interval, and statistics of token pairs varies between the alphabet and other characters. Think up some of your own ideas to try out. Think up only very simple ideas. Simplicity is the key. * '''Phase 3:''' Investigate Linguistic Morphology. Initially started looking at determining the varying possible word stems and affixes but eventually only used basic character sequences at the start and end of words to determine 'affix' frequencies for comparisons. * '''Phase 4:''' Illustration Investigation. * '''Phase 5:''' Collocation Investigation. Collocations give a quantitative measure of word association. As collocations are considered to be relatively frequent in natural languages, this may be able to help determine if the Voynich is a hoax. ==Deliverables== Re-check required deliverables and dates ===Semester A=== * Proposal seminar (''' Week 5 - 31/03/2015 - 3:05pm ''') ::[[File:Proposal Seminar Slides.pdf]] ::[http://www.eleceng.adelaide.edu.au/personal/dabbott/honours/Voynich_Sem1_talk_2015.mp4 Proposal Seminar Video] * Research Project Proposal and Progress Report draft ('''Week 6a - 17/04/2015 - 3:00pm''') ::[[File:Draft Research Project Proposal - Project 31.pdf]] ::[[Draft Research Project Proposal - Wiki Page Convert]] * Research Project Proposal and Progress Report - only one report needed in wiki format (''' Week 12 - 05/06/2016 - 3:00pm ''') ::[[Research Project Proposal and Progress Report]] ::''NOTE: A .pdf version must be generated and submitted by each member via MyUni, these must clearly show the contributions of the specific member'' ===Semester B=== * Final seminar (''' Tuesday, 13/10/15, Week 10 ''') ::[[File:Final Seminar Slides.pdf]] * Final Report/Thesis (''' Week 11 ''') ::[[Cracking the Voynich Code 2015 - Final Report]] ::The following pdf is a copy of the separate submitted thesis version as per the new course requirements ::[[File:Cracking the Voynich Manuscript- Using basic statistics and analyses to determine linguistic relationships.pdf]] * Poster (''' Friday, 23/10/15 ''') ::[[File:Poster Draft.pdf]] ::[[File:Project 31 Expo Poster.pdf]] * Project Exhibition - Ingenuity 2015 (''' Monday - Tuesday, 26/10/15 - 27/10/15 ''') * Labelled CD or USB stick containing your whole project directories. Only one is needed but it should contain two project directories, ie. one for each group member (''' By Friday 06/11/15 ''') * YouTube video summarizing project in exciting way - add the URL to this wiki - only one needed (''' By Friday 06/11/15 ''') ::[https://www.youtube.com/watch?v=Zt2C4UgWuKI Project 31: Cracking the Voynich Code 2015] * Optional: any number of instructional how-to YouTube videos on running your software etc. == Code == As particular code bases become complete, or relatively close to complete, they will be listed below. [[Characterization Code]] Note: Brief details of the codes are now compiled within the final report. == Voynich word search == *[[Search engine]] == Weekly progress and questions == This is where you record your progress and ask questions. Make sure you update this every week. *[[Cracking the Voynich Code 2015 weekly progress]] Code files, work logs, meeting minutes, etc. can also be found on the project team's Google Drive. This can be viewed below: * [https://drive.google.com/a/student.adelaide.edu.au/folderview?id=0B3xk_r8iaE_IYURhTEhLd1dyeVk&usp=sharing Voynich Manuscript Project 2015 Google Drive] ==Approach and methodology== We expect you to take a structured approach to both the validation and the writing of the software. You should carefully design the big-picture high-level view of the software modules, and the relationships and interfaces between them. Think also about the data transformations needed. == Expectations == * We don't really expect you to crack the Voynich, though that would be cool if you do and you'll become very famous overnight. * To get good marks we expect you to show a logical approach to decisively eliminating some languages and authors, and finding some hints about the statistical nature of the words. * In your conclusion, you need to come up with a short list of possible hypotheses and a list of things you can definitely eliminate. * We expect you to critically look at the conclusions of the previous work and highlight to what extent your conclusions agree and where you disagree. * It is important to regularly see your main supervisors. Don't let more than 2 week go by without them seeing your face briefly. * You should be making at least one formal progress meeting with supervisors per month. It does not strictly have to be exactly a month, but roughly each month you should be in a position to show some progress and have some problems and difficulties to discuss. On the other hand the meetings can be very frequent in periods when you have a lot of activity and progress to show. * The onus is on you to drive the meetings, make the appointments, and set them up. == Relationship to possible career path== Whilst the project is fascinating as you'll learn about a specific high-profile mystery—and we do want you to have a lot of fun with it—the project does have a hard-core serious engineering side. It will familiarize you with techniques in information theory, probability, statistics, encryption, decryption, signal classification, and datamining. It will also improve your software skills. The new software tools you develop may lead to new IP in the areas of datamining, automatic text language identification, and also make you rich/famous. The types of jobs out there where these skills are useful are in computer security, comms, digital forensics, internet search companies, and language processing software companies. The types of industries that will need you are: the software industry, e-finance industry, e-security, IT industry, Google, telecoms industry, [http://www.asio.gov.au/ ASIO], [http://www.asis.gov.au/ ASIS], defence industry (e.g. [http://www.dsd.gov.au/ DSD]), etc. So go ahead and have fun with this, but keep your eye on the bigger engineering picture and try to build up an appreciation of why these techniques are useful to our industry. Now go crack the Voynich...this message will self-destruct in five seconds :-) ==See also== * [[Cracking the Voynich code|Voynich Project, MAIN Page]] * [[Cracking the Voynich Code 2015 weekly progress|Voynich 2015 (Andrew and Lifei) Weekly Progress]] * [https://www.eleceng.adelaide.edu.au/students/wiki/projects/index.php/Projects:2014S1-44_Cracking_the_Voynich_Manuscript_Code Elec Eng 2014 Project Wiki] * [https://drive.google.com/a/student.adelaide.edu.au/folderview?id=0B3xk_r8iaE_IYURhTEhLd1dyeVk&usp=sharing Voynich Manuscript Project 2015 Google Drive] ==Useful papers we wrote== [http://www.eleceng.adelaide.edu.au/personal/dabbott/publications/PLO_ebrahimpour2013.pdf] [[Maryam Ebrahimpour|M. Ebrahimpour]], [[Tālis J. Putniņš|T. J. Putniņš]], [[Matthew J. Berryman|M. J. Berryman]], [[Andrew G. Allison|A. Allison]], [[Brian W.-H. Ng|B. W.-H.-Ng]], and '''[[Derek Abbott|D. Abbott]]''', "Automated authorship attribution using advanced signal classification techniques," ''PLoS ONE'', '''Vol. 8,''' No. 2, Art. No. e54998, 2013, http://dx.doi.org/10.1371/journal.pone.0054998 [http://www.eleceng.adelaide.edu.au/Personal/dabbott/publications/FNL_berryman2003.pdf] [[Matthew J. Berryman|M. J. Berryman]], [[Andrew G. Allison|A. Allison]], and '''[[Derek Abbott|D. Abbott]],''' "Statistical techniques for text classification based on word recurrence intervals," ''Fluctuation and Noise Letters'', '''Vol. 3''', No. 1, pp. L1–L12, 2003. == References and useful resources== If you find any useful external links, list them here: * [https://archive.org/details/TheVoynichManuscript Scanned copy of the Voynich] * [http://www.ic.unicamp.br/~stolfi/voynich/99-01-16-concordance/ Digital download for the Voynich] * [http://ebooks.adelaide.edu.au/ Adelaide Uni Library e-book collection] * [http://www.gutenberg.org/wiki/Main_Page Project Gutenburg e-books] * [http://onlinebooks.library.upenn.edu/archives.html#foreign Foreign language e-books] * [http://www.ohchr.org/EN/UDHR/Pages/Introduction.aspx UN Declaration of Human Rights - different languages] * [http://portal.acm.org/citation.cfm?id=1389095.1389425 Evolutionary algorithm for decryption of monoalphabetic homophonic substitution ciphers encoded as constraint satisfaction problems] * [http://www.bckelk.ukfsn.org/words/etaoin.html List of letter rankings for different languages] * [http://www.compellingpress.com/voynich/ The Curse of the Voynich] * [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0067310&representation=PDF Statistical properties of the Voynich] * [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0066344&representation=PDF Information-theoretic analysis of the Voynich] * [http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=voynich Books on the Voynich (Amazon)] * [http://www.ic.unicamp.br/~stolfi/voynich/ Voynich resources] * [http://ixoloxi.com/voynich/tools.html Voynich tools] * [http://en.wikibooks.org/wiki/The_Voynich_Manuscript/Jargon Guide to Voynich jargon] ==Back== *[https://myuni.adelaide.edu.au/webapps/portal/frameset.jsp Back to MyUni] *[http://www.eleceng.adelaide.edu.au/personal/dabbott Back to Derek Abbott's homepage] *[http://www.eleceng.adelaide.edu.au Back to EEE Department page] *[http://www.adelaide.edu.au Back to the University of Adelaide homepage]
Summary:
Please note that all contributions to Derek may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see
Derek:Copyrights
for details).
Do not submit copyrighted work without permission!
Cancel
Editing help
(opens in new window)
Navigation menu
Personal tools
Not logged in
Talk
Contributions
Create account
Log in
Namespaces
Page
Discussion
English
Views
Read
Edit
View history
More
Search
Navigation
Main page
Recent changes
Random page
Help about MediaWiki
Tools
What links here
Related changes
Special pages
Page information