Joe Wong, Harsavardhan M G R, Kenneth Y T Lim, Swee Ling Leong (2024). Exploring The Use of ChatGPT4 API in Approaching Math Word Problems. In: Human Interaction and Emerging Technologies (IHIET 2024). AHFE Open Access, vol. 157, pp. 221–229. https://doi.org/10.54941/ahfe1005481
With the evolving educational landscape precipitated by the COVID-19 pandemic, online education becomes increasingly prevalent. Much help is needed to provide innovative solutions to address the challenges faced by both students and teachers during this time of crisis. This paper describes an independent research project conducted by a pair of high school students between April 2023 and February 2024, under the mentorship of a senior research scientist at the National Institute of Education in Singapore. The project investigates various methods of Tesseract OCR text recognition, OpenCV image processing, Flask web development and OpenAI’s Large Language Models to improve mathematics-solving applications.Our program extracts text using Tesseract OCR, utilising it as input for the GPT-4 API, enabling a conversational presentation of mathematics problems. Users interact by inputting the image address of the math problem that they would like the AI to solve, and GPT-4 provides solutions with detailed step-by-step explanations. OpenCV improves the provided image’s quality such as making the text or diagrams more distinct to reduce the possibility of them being misinterpreted. Through evaluation by testing with different types of maths problems of varying difficulty, our findings underscore the potential for advanced language models in educational tools, offering interactive and intuitive maths problem-solving experiences. There were a few limitations encountered during experimentation, such as challenges with extraction of non-Latin alphabets and accuracy of the OpenAI’s Large Language Modules when solving more complex diagram problems, highlighting the need for further refinement to enhance the system's robustness and adaptability. Future work involves addressing these limitations to broaden the system's applicability for educational purposes and beyond.
Background and Purpose of the Research Area In the dynamic tech landscape, Python shines as a versatile language for various projects, from simple web apps to complex machine learning with deep-learning algorithms. Our project leverages Python’s potential, merging Tesseract OCR, OpenCV, and the OpenAI API for innovative math problem- solving. Designed to assist students, our approach provides quick, step- by-step answers to both word and diagram problems, showcasing the
power of Python in creating comprehensive solutions for diverse educational challenges.
Hypothesis of the Research Our hypothesis centres on the idea that the integration of optical character recognition (Tesseract OCR) and image processing (OpenCV) with advanced natural language processing (OpenAI API, specifically GPT-3.5 and GPT-4 Vision) can significantly enhance the accuracy and efficiency of maths problem-solving. This amalgamation of technologies will provide solutions and detailed step-by-step explanations, revolutionising how users interact with mathematical content.
METHODOLOGY/MATERIALS Tesseract OCR
Employing Tesseract OCR, we extracted text, focusing on mathematical expressions and laying the foundation for subsequent processing. Tesseract OCR employs algorithms for line and word detection, character identification, and accommodates variations in spacing due to different fonts (Smith, 2007). The versatility of Tesseract OCR, with algorithms like fixed pitch detection and proportional word finding, is crucial for reading user handwriting with varying fonts (NYU, 2024). These features enhance the accuracy of our program, making it resilient to diverse font styles and attributes.
OpenCV Image Processing Utilising OpenCV, we captured images from the webcam and sent them to the Tesseract OCR to extract text. However, many challenges existed, such as shadows and varying lighting conditions (Computer vision engineer, 2023), as shown in Figure 1. This affects the Tesseract OCR significantly and prevents it from extracting data accurately (Klippa, 2022).
Figure 1: Original image with bad lighting.
Exploring the Use of ChatGPT4 API in Approaching Math Word Problems 223
To solve this problem, an adaptive threshold is used instead. The algorithm determines the threshold required based on the surrounding pixels (Kumar, 2019). There will be different thresholds calculated for different regions of the image, allowing the text which is darker to become more visible regardless if there is a shadow on the image (Rosebrock, 2021).
This code snippet changes the image to grayscale, then uses adaptive thresholding to make the darker areas more visible, as shown in Figure 2.
Figure 2: Image after adaptive thresholding.
Through OpenCV’s webcam and adaptive thresholding feature, the image can become clearer and users would not need to worry about poor lighting or blocking text when scanning their images.
OpenAI GPT-3.5 API We integrated OpenAI’s GPT-3.5 API to process prompts derived from extracted data. Users could pose mathematical queries, and GPT-3.5 provided detailed, conversational solutions. Our utilisation involved a straightforward API call to GPT-3.5, enabling prompt processing and solution generation within the program. GPT-3.5 is a learning language model that uses transformer neural networks. Neural networks have multiple layers of algorithms where they would recognise the underlying relationships in the input, and the transformer model takes the context into more detail, improving performance on NLP tasks. It analyses natural language queries to predict the best possible response based on their understanding of language. The transformer neural network architecture also uses self- attention mechanisms to focus on different parts of the input text during each processing step. These GPT models rely on the knowledge they gain after they are trained with hundreds of billions of parameters on massive language datasets (AWS, n.d.).
OpenAI GPT-4 Vision API In addition to GPT-3.5, we explored the capabilities of GPT-4 Vision for image analysis.
This code snippet shows a GPT-4 Vision API call. This multimodal model integrated image inputs into text, enabling a more comprehensive understanding of the mathematical content presented in images.
Integration of Flask for Interactive Problem Solving Flask is a simple Python Web framework that has useful tools and features that make website development easier. It provides developers with flexibility and is easily accessible as a website can be easily set up with only one Python file. Flask also offers various customisations with different elements like text boxes and buttons and a wide selection of colours. We leveraged Flask to design an intuitive web interface where users can effortlessly submit math problems for our custom solver. The website features a user-friendly input box where individuals can paste the URL of an image containing a maths problem. This seamless interaction is made possible through Flask’s capabilities in handling HTTP requests and rendering dynamic HTML content. Upon submission of the image URL, Flask orchestrates the communication between the user interface and our Python maths problem- solving code. Using Flask routes, we directed form submissions to the appropriate function, initiating the process of solving the maths problem extracted from the provided image.
Dynamic Rendering With Jinja2 Flask’s integration with the Jinja2 template engine allowed us to dynamically render HTML templates. The result of the math problem-solving process
Exploring the Use of ChatGPT4 API in Approaching Math Word Problems 225
is seamlessly displayed on the website, providing users with immediate feedback. This dynamic rendering capability enhances the user experience and ensures a fluid interaction with the system.
Scalability and Future Development The modular structure of Flask ensures scalability and facilitates future enhancements to our system. As our project evolves, Flask provides a solid foundation for incorporating additional features, refining the user interface, and optimizing the overall user experience (Saini, 2023).
In essence, Flask serves as the backbone of our web-based math problem- solving platform, seamlessly connecting users with the power of our custom Python code and OpenAI API integration.
This code snippet shows how Flask is used to set up the website and to call functions when a button is clicked. Figure 3 shows the user interface of the website, in which the user can paste an image URL and submit it in order to get a response.
Figure 3: The user interface of the website.
RESULTS/DISCUSSION We have developed a model using Tesseract OCR to extract text from images, OpenCV to modify these images for the computer to read, and GPT- 3.5 to generate answers to the questions from the text extracted. Hence, we investigated the difference between our model and the GPT-4 Vision model.
Test 1 (GPT-4) | Extract Equations From Image | Fail
Figure 4: The image (Engineering discoveries, 2020) we tested our model with.Figure 5: The result from our model.
Comments: Heavily inaccurate. Misinterpretation of the equation due to font type. A potential solution involves training a custom machine learning algorithm with various fonts, which was not feasible within the project’s time constraints.
Test 2 (Vision) | Text Problem | Success
Figure 6: The image (Wells, 2019) we tested the vision model with.
Exploring the Use of ChatGPT4 API in Approaching Math Word Problems 227
Figure 7: The result from the model.
Comments: GPT-4 Vision demonstrated superior capabilities in solving various problems compared to Tesseract OCR. Notable successes were observed in both text and diagram-based problems.
Test 3 (Vision) | Diagram Problem | Success
Figure 8: The image (Talwalkar, 2016) we tested the vision model with.Figure 9: The result from the model.
Comments: GPT-4 Vision was able to identify angles but unable to name angles and solve for x despite knowing the steps to the solution.
CONCLUSION
Future Plans Unfortunately, due to the limited time constraints, we were unable to implement everything we had hoped to previously. Firstly, we would like to utilise Dall-E, an image generation module developed by OpenAI, to help draw diagrams or models for our users to better understand the working presented by GPT-4. Secondly, we would like to format the large chunk of text provided by GPT-4 into steps that users can easily read and follow. Thirdly, we hope that our app could have been more interactive by asking the user for the next step to follow or to fill in the working after telling the user the next step to solve the question. Lastly, our web application requires some refining in its User interface to be more readable and user-friendly.
Conclusion
In conclusion, our project successfully integrated Tesseract OCR, OpenCV, and the OpenAI API, showcasing the synergy of diverse tools for comprehensive math problem-solving. Tesseract OCR excelled in text extraction, while OpenCV addressed image quality issues, and GPT-3.5 and GPT-4 Vision provided insightful solutions for both text- and diagram-based problems. Our findings underscore the potential of these technologies in creating interactive learning experiences.
However, our experimentation revealed certain limitations. GPT-4 Vision faces challenges with non-Latin alphabets, impacting global optimization. Enlarging small text improves readability but risks cropping essential details. Rotation of text or images may lead to misinterpretation, and struggles with diverse visual elements and spatial reasoning tasks were observed. Future work involves refining our approach to enhance robustness and adaptability, aiming to broaden the system’s applicability for educational purposes and beyond.
REFERENCES
AWS (n.d.). What is GPT AI? - Generative Pre-Trained Transformers Explained.
Available at: https://aws.amazon.com/what-is/gpt/#:~:text=More%20specificall y%2C%20the%20GPT%20models,on%20their%20understanding%20of%20l anguage (Accessed: 4 January 2024). Computer vision engineer (2023). OpenCV tutorial for beginners | FULL COURSE in 3 hours with Python. Available at: https://www.youtube.com/watch?v=eDIj5L uIL4A (Accessed: 4 January 2024). Engineering Discoveries (2020). ‘A trip back to high school: Can you solve this math problem?’, Engineering Discoveries, 13 October. Available at: https://engineeringdiscoveries.com/a-trip-back-to-high-school-can-you-solve- this-math-problem/ (Accessed: 4 January 2024). Klippa (2022). ‘Tesseract OCR: What is it, and Why Would You Choose it in 2023?’,
Klippa, 20 October. Available at: https://www.klippa.com/en/blog/information/tes seract-ocr/ (Accessed: 4 January 2024). Kumar, S. (2019). ‘A straightforward introduction to Image Thresholding using python’, Spinor, 2 October. Available at: https://medium.com/spinor/a-stra ightforward-introduction-to-image-thresholding-using-python-f1c085f02d5e (Accessed: 4 January 2024).
Exploring the Use of ChatGPT4 API in Approaching Math Word Problems 229
NYU (2024). Tesseract OCR Software Tutorial. Available at: https://guides.nyu.edu /tesseract (Accessed: 4 January 2024). Rosebrock, A. (2021). ‘Adaptive Thresholding with OpenCV (cv2.adaptiveThreshold)’, PyImageSearch, 12 May. Available at: https: //pyimagesearch.com/2021/05/12/adaptive-thresholding-with-opencv-cv2-ad aptivethreshold/ (Accessed: 4 January 2024). Saini, A. (2023). ‘An Easy introduction to Flask Framework for Beginners’, Analytics
Vidhya, 23 August. Available at: https://www.analyticsvidhya.com/blog/2021/10/ flask-python/ (Accessed: 4 January 2024). Smith, R. (2007). ‘An Overview of the Tesseract OCR Engine’, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007). Curitiba, Brazil, 23–26 September. Los Alamitos: IEEE Computer Society, pp. 629–633. doi: 10.1109/ICDAR.2007.4376991. Talwalkar, P. (2016). ‘The Hardest Easy Geometry Problem – Sunday Puzzle –
Mind Your Decisions’, Mind Your Decisions, 4 September. Available at: https://mindyourdecisions.com/blog/2016/09/04/the-hardest-easy-geometry- problem-sunday-puzzle/ (Accessed: 4 January 2024). Wells, J. (2019). ‘The power of plugging numbers in’, Wells Academic Solutions, 20 September. Available at: https://www.wellsacademics.com/the-power-of-plugging -numbers-in/ (Accessed: 4 January 2024).
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET 2024)
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-33-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional