Talk for me application - Revision history

A1803076 at 02:47, 5 November 2024

2024-11-05T02:47:55Z

← Older revision		Revision as of 12:17, 5 November 2024
Line 54:		Line 54:
	Due to our NDA, strict methods are not able to be provided, but can be summarised as accessing the existing code, and modifying/adding to it to gain the desired functions/changes.		Due to our NDA, strict methods are not able to be provided, but can be summarised as accessing the existing code, and modifying/adding to it to gain the desired functions/changes.

	~~==Possible extension==~~
	~~TBA~~

	== Expectations ==		== Expectations ==
Line 77:		Line 75:
	The the coding and project management skills you pick up in the project will serve you for a wide range of careers.		The the coding and project management skills you pick up in the project will serve you for a wide range of careers.

	~~== References and useful resources==~~
	~~If you find any useful external links, list them here:~~
	*
	*

	==Back==		==Back==

A1803076 at 02:47, 5 November 2024

2024-11-05T02:47:38Z

← Older revision		Revision as of 12:17, 5 November 2024
Line 3:		Line 3:
	*[[Matthew J. Berryman\|Dr Mathew Berryman]]		*[[Matthew J. Berryman\|Dr Mathew Berryman]]

	==Honours students==		== Honours students ==
	*~~'''2024:'''~~ [[Keefe Dela Cruz]], [[Addy Dhingra]] ~~and~~ [[Matthew Fowler~~]], see [[Talk for me application 2024~~]]		*[[Keefe Dela Cruz]]
			*[[Addy Dhingra]]
			*[[Matthew Fowler]]

	==~~Project guidelines==~~		==General project description==
	*[http://www.eleceng.adelaide.edu.au/personal/dabbott/project_handbook_2009.pdf Project Handbook]

	~~==Project~~ description==
	The Talk For Me app provides a fully customisable yet easy to use grid to point to communicate for people who can't talk, and supports text-to-speech output and image-recognition suggested text. This project is about extending the capability further, from words to sentences, possibly involving the use of ChatGPT. You will require a Mac computer since this is an iOS app, and familiarity with Swift and SwiftUI is essential.		The Talk For Me app provides a fully customisable yet easy to use grid to point to communicate for people who can't talk, and supports text-to-speech output and image-recognition suggested text. This project is about extending the capability further, from words to sentences, possibly involving the use of ChatGPT. You will require a Mac computer since this is an iOS app, and familiarity with Swift and SwiftUI is essential.

	== ~~Aims~~ ==		==Specific tasks==


	Talk For Me is a text-to-speech (TTS) iOS application that processes uploaded images to generate contextual text suggestions, which can t hen be spoken aloud by the device. The application is currently in alpha, offering only basic text suggestions. The aim is to enhance this feature and incorporate additional elements to improve the application. Improving these features and subsequently integrating the application into the neurodivergent community ensures future contributions to the advancement of the health caregiving market.

	~~== Background ==~~		The tasks identified by the group to complete for this project focus around improving three areas, the UI, the functionality of the App, and the performance of the app through LLM optimisation.


	Nonverbal individuals often face significant challenges with communication. While tools like sign language have provided solutions, they are not universally effective as it is not widely known. Consequently, TTS applications have been developed to enable nonverbal individuals to ~~communicate in real-time~~ with ~~others using a common language.~~		UI
			* Update the UI to fit with similar apps on the market

	Artificial Intelligence (AI), particularly through the use of Large Language Models (LLMs), is the key tool driving the improvements for Talk For Me. LLMs can perform language generation and natural language processing, enabling users to input keywords in the form of images and generate complex sentences. With numerous LLM options available, this project will also evaluate their performance to enhance communication speed. Additionally, the project will explore LLMs capable of operating locally on the user’s mobile phone.

	~~== Motivation ==~~		Functionality of the App
			* Receive Location/Time information to make informed suggestions
			* Suggest Menu Items based off the users location
			* Suggest previously used sentences given a selection of words
			* Generate 1+ messages from the LLM
			* Sort the keywords presented to the users based off the acquired Location/Time information gained above.


	~~The Dr Matthew Berryman, of Across the Cloud, inspired the idea for Talk For Me after experiencing a stroke~~ and ~~facing communication struggles. As a results, the application development project is driven by~~ the ~~need~~ to ~~help neurodivergent individuals nonverbally specify their needs to the public. In the past, TTS applications have been slow and cumbersome because users had to manually type and edit every~~ word ~~they wanted to be spoken. However, with~~ the ~~integration~~ of ~~AI, these interactions are expected to become more natural and seamless, eliminating the pauses caused by manual input. AI will achieve this by predicting~~ the ~~most likely sentences based on various factors such~~ as ~~location, keywords, and previous behaviour or chosen sentences. It will~~ the ~~give contextual suggestions~~ to the ~~users allowing them to communicate easier~~.		Large Language Models
			* Run and compare the time take to respond over a number of datasets reflecting 1-3 word combinations
			* Compare the number of outliers in each performance
			* Compare the models "reliability" as in how well it follows the provided instructions to respond with a response that contains the expected words (and a sentence that can be used for TTS).

	== ~~Objectives~~ ==		==Deliverables==

			===Semester 1===
			* Project Proposal [[File:TalkForMe-ProjectPlan.pdf]]
			* Seminar [https://adelaide.zoom.us/rec/share/ZmSdxd6Drx1VFdIszz-0N71q885vowWd4ge_iUgfHhZevbzT__buvo9mldA4GLDO.AkJ5dqBwcs8UOFh0?startTime=1718062694000] password: 3LC!+USl
			* Progress Report: [[File:TalkForMe-ProgressReport.pdf]]

	To ensure success within the allotted time frame and considering the team's expertise, the project focuses solely on enhancing the application's user interface (UI), functionality, performance, and analytics. By narrowing the scope, the purpose is to develop a high-quality, scalable application. Specifically, the objectives are to make UI improvements, integrate location-based systems, and LLM exploration. UI improvements will enhance both the readability and marketability of the product, particularly for neurodivergent audiences. Integrating location-based systems will create a more personalized user experience, significantly improving the user's quality of life. Exploring and experimenting with Large Language Models (LLMs) will ensure the generation of consistently appropriate sentences, reducing the frequency of errors.		===Semester 2===

	~~== Project management ==~~		* Ingenuity Poster [[File:Talk For Me Ingenuity Poster.pdf]]
			* Ingenuity Video
			* Final Report [[File:TalkForMe-FinalReport.pdf]]

	Regular meetings with the team will be held to ensure tasks are complete. Furthermore, regular bi-weekly meetings with Dr Berryman will also ensure that the project remains aligned with the company's needs and values. To adhere to the time constraints, the schedules outlined in Appendix A: Gantt Chart will be strictly followed.		== Weekly progress and questions ==
			This is where you record your progress and ask questions. Make sure you update this every week.
	== ~~Report overview~~ ==		*[[Talk for me application 2024 weekly progress]]


	This ~~report provides a review of relevant literature pertaining to the project objectives. It offers a comprehensive overview of the objectives, outcomes,~~ and ~~strategies implemented to achieve them~~. ~~Additionally, it provides details on the current progress of each approach and the plan to complete them in the future~~.

	~~==Useful notes==~~
	~~As useful bits of information come to light, just list them here:~~

	==Approach and methodology==		==Approach and methodology==
	~~List dot points:~~		The approach involved a number of methods listed in our Progress and Final Reports.
	SS		Due to our NDA, strict methods are not able to be provided, but can be summarised as accessing the existing code, and modifying/adding to it to gain the desired functions/changes.

	==Possible extension==		==Possible extension==
			TBA


	== Expectations ==		== Expectations ==
			* To get good marks we expect you to show a logical approach to coding.

	* To get good marks we expect you to show a logical approach.		* In your conclusion, you need to come up with a short list of likely possibilities and a list of things you can definitely eliminate.

	* In your conclusion, you need to come up with a list of ~~to do items for future improvement~~.

	* We expect all the written work to be place on this wiki. No paper reports are to be handed up.		* We expect all the written work to be place on this wiki. No paper reports are to be handed up.
Line 74:		Line 75:

	== Relationship to possible career path==		== Relationship to possible career path==
	~~This will enhance your software skills~~ and project management skills ~~and prepare~~ you for a wide range of ~~job opportunities~~.		The the coding and project management skills you pick up in the project will serve you for a wide range of careers.

	~~==See also==~~
	~~List any pages you create here:~~


	== References and useful resources==		== References and useful resources==
	If you find any useful external links, list them here:		If you find any useful external links, list them here:
			*
			*

	==Back==		==Back==
Line 90:		Line 87:
	*[http://www.eleceng.adelaide.edu.au Back to EEE Department page]		*[http://www.eleceng.adelaide.edu.au Back to EEE Department page]
	*[http://www.adelaide.edu.au Back to the University of Adelaide homepage]		*[http://www.adelaide.edu.au Back to the University of Adelaide homepage]
			*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/Cipher_Cracking_2012 Back to Cipher Cracking 2012 homepage]

A1793854: /* Useful notes */

2024-08-25T07:51:24Z

Useful notes

← Older revision		Revision as of 17:21, 25 August 2024
Line 46:		Line 46:
	==Useful notes==		==Useful notes==
	As useful bits of information come to light, just list them here:		As useful bits of information come to light, just list them here:



	==Approach and methodology==		==Approach and methodology==

A1793854: /* Approach and methodology */

2024-08-25T07:51:12Z

Approach and methodology

← Older revision		Revision as of 17:21, 25 August 2024
Line 51:		Line 51:
	==Approach and methodology==		==Approach and methodology==
	List dot points:		List dot points:
			SS

	==Possible extension==		==Possible extension==

A1793854: /* Motivation */

2024-08-25T07:46:00Z

Motivation

← Older revision		Revision as of 17:16, 25 August 2024
Line 27:		Line 27:


	The Dr Matthew Berryman, of Across the Cloud, inspired the idea for Talk For Me after experiencing a stroke and facing communication struggles. As a results, the application development project is driven by the need to help neurodivergent individuals nonverbally specify their needs to the public. In the past, TTS applications have been slow and cumbersome because users had to manually type and edit every word they wanted to be spoken. However, with the integration of AI, these interactions are expected to become more natural and seamless, eliminating the pauses caused by manual input. AI will achieve this by predicting the most likely sentences based on various factors such as location, keywords, and previous behaviour or chosen sentences. It will the give contextual suggestions to the users allowing them to communicate easier.		The Dr Matthew Berryman, of Across the Cloud, inspired the idea for Talk For Me after experiencing a stroke and facing communication struggles. As a results, the application development project is driven by the need to help neurodivergent individuals nonverbally specify their needs to the public. In the past, TTS applications have been slow and cumbersome because users had to manually type and edit every word they wanted to be spoken. However, with the integration of AI, these interactions are expected to become more natural and seamless, eliminating the pauses caused by manual input. AI will achieve this by predicting the most likely sentences based on various factors such as location, keywords, and previous behaviour or chosen sentences. It will the give contextual suggestions to the users allowing them to communicate easier.


	== Objectives ==		== Objectives ==

A1793854: /* Project management */

2024-08-25T07:45:52Z

Project management

← Older revision		Revision as of 17:15, 25 August 2024
Line 38:		Line 38:


	Regular meetings with the team will be held to ensure tasks are complete. Furthermore, regular bi-weekly meetings with Dr Berryman will also ensure that the project remains aligned with the company's needs and values. To adhere to the time constraints, the schedules outlined in Appendix A: Gantt Chart will be strictly followed.		Regular meetings with the team will be held to ensure tasks are complete. Furthermore, regular bi-weekly meetings with Dr Berryman will also ensure that the project remains aligned with the company's needs and values. To adhere to the time constraints, the schedules outlined in Appendix A: Gantt Chart will be strictly followed.


	== Report overview ==		== Report overview ==

A1793854: /* Objectives */

2024-08-25T07:45:39Z

Objectives

← Older revision		Revision as of 17:15, 25 August 2024
Line 33:		Line 33:


	To ensure success within the allotted time frame and considering the team's expertise, the project focuses solely on enhancing the application's user interface (UI), functionality, performance, and analytics. By narrowing the scope, the purpose is to develop a high-quality, scalable application. Specifically, the objectives are to make UI improvements, integrate location-based systems, and LLM exploration. UI improvements will enhance both the readability and marketability of the product, particularly for neurodivergent audiences. Integrating location-based systems will create a more personalized user experience, significantly improving the user's quality of life. Exploring and experimenting with Large Language Models (LLMs) will ensure the generation of consistently appropriate sentences, reducing the frequency of errors.		To ensure success within the allotted time frame and considering the team's expertise, the project focuses solely on enhancing the application's user interface (UI), functionality, performance, and analytics. By narrowing the scope, the purpose is to develop a high-quality, scalable application. Specifically, the objectives are to make UI improvements, integrate location-based systems, and LLM exploration. UI improvements will enhance both the readability and marketability of the product, particularly for neurodivergent audiences. Integrating location-based systems will create a more personalized user experience, significantly improving the user's quality of life. Exploring and experimenting with Large Language Models (LLMs) will ensure the generation of consistently appropriate sentences, reducing the frequency of errors.


	== Project management ==		== Project management ==

A1793854: /* Background */

2024-08-25T07:45:32Z

Background

← Older revision		Revision as of 17:15, 25 August 2024
Line 22:		Line 22:
	Nonverbal individuals often face significant challenges with communication. While tools like sign language have provided solutions, they are not universally effective as it is not widely known. Consequently, TTS applications have been developed to enable nonverbal individuals to communicate in real-time with others using a common language.		Nonverbal individuals often face significant challenges with communication. While tools like sign language have provided solutions, they are not universally effective as it is not widely known. Consequently, TTS applications have been developed to enable nonverbal individuals to communicate in real-time with others using a common language.

	Artificial Intelligence (AI), particularly through the use of Large Language Models (LLMs), is the key tool driving the improvements for Talk For Me. LLMs can perform language generation and natural language processing, enabling users to input keywords in the form of images and generate complex sentences. With numerous LLM options available, this project will also evaluate their performance to enhance communication speed. Additionally, the project will explore LLMs capable of operating locally on the user’s mobile phone.		Artificial Intelligence (AI), particularly through the use of Large Language Models (LLMs), is the key tool driving the improvements for Talk For Me. LLMs can perform language generation and natural language processing, enabling users to input keywords in the form of images and generate complex sentences. With numerous LLM options available, this project will also evaluate their performance to enhance communication speed. Additionally, the project will explore LLMs capable of operating locally on the user’s mobile phone.


	== Motivation ==		== Motivation ==

A1793854: /* Aims */

2024-08-25T07:45:24Z

Aims

← Older revision		Revision as of 17:15, 25 August 2024
Line 15:		Line 15:


	Talk For Me is a text-to-speech (TTS) iOS application that processes uploaded images to generate contextual text suggestions, which can t hen be spoken aloud by the device. The application is currently in alpha, offering only basic text suggestions. The aim is to enhance this feature and incorporate additional elements to improve the application. Improving these features and subsequently integrating the application into the neurodivergent community ensures future contributions to the advancement of the health caregiving market.		Talk For Me is a text-to-speech (TTS) iOS application that processes uploaded images to generate contextual text suggestions, which can t hen be spoken aloud by the device. The application is currently in alpha, offering only basic text suggestions. The aim is to enhance this feature and incorporate additional elements to improve the application. Improving these features and subsequently integrating the application into the neurodivergent community ensures future contributions to the advancement of the health caregiving market.


	== Background ==		== Background ==

A1793854: /* Project description */

2024-08-25T07:45:17Z

Project description

← Older revision		Revision as of 17:15, 25 August 2024
Line 11:		Line 11:
	==Project description==		==Project description==
	The Talk For Me app provides a fully customisable yet easy to use grid to point to communicate for people who can't talk, and supports text-to-speech output and image-recognition suggested text. This project is about extending the capability further, from words to sentences, possibly involving the use of ChatGPT. You will require a Mac computer since this is an iOS app, and familiarity with Swift and SwiftUI is essential.		The Talk For Me app provides a fully customisable yet easy to use grid to point to communicate for people who can't talk, and supports text-to-speech output and image-recognition suggested text. This project is about extending the capability further, from words to sentences, possibly involving the use of ChatGPT. You will require a Mac computer since this is an iOS app, and familiarity with Swift and SwiftUI is essential.


	== Aims ==		== Aims ==