A migration agent’s review of ChatGPT: Is it a reliable and accurate resource for Australian immigration advice? [Part 1]
TLDR: It may not be 100% accurate, but it’s impressive how much it appears to get right and it can be a useful tool if used in the right way. However, it’s still best to check official sources or seek professional help.
We’ll look at some examples of what it gets right, and wrong, so you can decide how useful it is.
To get us started, here is a response from ChatGPT about its capability to provide Australian immigration advice.
How does ChatGPT work?
ChatGPT is a powerful language model tool. It works by using deep learning algorithms to process large amounts of data and learn from it. This is how it generates responses to questions by guessing what the answer might be.
ChatGPT was trained on many sources including books, articles, and websites. It claims to have been trained on immigration legislation, regulations, and case law, and to be able to “provide accurate and up-to-date legal advice” for people who want to apply for a visa. Our experience suggests that this claim may not be entirely accurate.
Can ChatGPT be relied on for Australian immigration advice?
No, in our opinion ChatGPT offers information rather than migration advice. We tested ChatGPT by asking it immigration questions and found that its answers were general in nature and easy to understand. For these reasons, ChatGPT can be a useful starting point for someone without prior immigration knowledge to research visa options.
But it is important to remember that ChatGPT’s answers are a summary of the immigration information available on the internet and other publicly available sources. ChatGPT cannot make an assessment of the user’s individual circumstances against specific immigration regulations. It may also rely on outdated information. This means that ChatGPT’s information may not be accurate enough and if solely relied upon, could lead to problems.
Further, ChatGPT uses a wide range of sources to generate its responses, including both official government websites and other sources available on the internet, which could be unverified. While ChatGPT can appear to provide accurate and up-to-date information, it may not always be reliable or comprehensive. Therefore it is recommended that users verify any information from ChatGPT with official sources or seek professional advice from a registered migration agent.
What does it get right and how it can help with a visa application
Using a machine for legal guidance may be a challenging idea, but ChatGPT’s ability to provide surprisingly relevant and usually correct answers to almost any questions makes it a very valuable tool. In the current version, its strength is in creative writing. The ability to weave facts from user prompts into a narrative can make it a useful tool for writing a short visa submission.
When ChatGPT was good at providing Australian immigration information
One of the most significant advantages of using ChatGPT is its ability to draw on immigration sources to provide easy-to-understand answers. Here is an example of ChatGPT providing a useful and detailed answer when asked about visa options in Australia. Note that ChatGPT’s replies are related to the previous questions asked. Therefore all answers shown in this article are related to the topic we discussed, which is Australian immigration.
However, ChatGPT made a mistake by mentioning an old immigration list, the CSOL. This makes us wonder if its information is up-to-date. This limitation is something that will be explored further in this article. But, the rest of the content is generally correct and informative enough for the user to continue researching their visa options.
Here is another example of ChatGPT providing a general but informative answer on a specific aspect of the migration process. It was able to point out all of the correct skills assessing authorities for the occupation of Accountant as set out by the Legislative Instrument.
However, it is still advisable to double-check against the current Legislative Instrument or seek professional immigration assistance as the consequence of getting the skills assessing authority wrong could be disastrous.
Can it provide specific answers based on Australian immigration legislation and regulation?
ChatGPT does a good job at providing general information but does not appear to be able to provide specific information based on legislation or regulation that will help with the application. At least not right away. Take for example the below question.
It is technically correct that the occupation of Accountant is on a Skilled Occupation List (SOL) but to be truly accurate and useful, the user needs to know which Occupation List the occupation falls into (MLTSSL, STSSL, and ROL) so they can determine which skilled visa they are eligible for. This information can be found in the Legislative Instrument which it does not appear to have referenced.
Because we know that the above answer was not accurate enough, we gave it this next question.
The answer is inaccurate. Currently, the occupation of Accountant is in the MLTSSL. ChatGPT already mentioned in the previous answer that Accountant is in the SOL, which is not accurate enough. It also mentions the outdated CSOL again, so this answer is not correct.
This was also the first indication that after a certain point, ChatGPT’s effectiveness is reliant on the user possessing some level of immigration knowledge to ask the right questions. An immigration expert would be able to see where ChatGPT made a mistake. The harder and more specific the question, the more an expert is needed.
Does ChatGPT provide up-to-date information?
In short, no. As an AI language model, its knowledge cut-off is set to September 2021. This means any new legislation or regulations that have been updated since then will not be reflected in the responses.
It is important to use the latest laws and regulations when dealing with immigration law. ChatGPT may tell users to seek help from an expert. Even so, sometimes ChatGPT appears confident about its answers and can even hallucinate answers. We will explore this more in the next article.
How good is it at providing practical guidance on visa application and process?
Many people struggle to find helpful visa application advice online because there is too much information to sort through. Concerns about full accuracy aside, we found ChatGPT to be helpful in pulling relevant insight from large volumes of immigration information and summarising that information into practical guidance. This is shown in the example below.
But although ChatGPT’s answer is correct, it is incomplete. It is not able to determine eligibility and give important details that a migration expert would provide, such as how many points the applicant can claim. A migration agent would assess the applicant’s unique circumstances against specific regulations and requirements, lay out how the applicant would be eligible for the visa, and explore potential cost effective strategies, all of which ChatGPT cannot do.
From what we’ve seen so far, ChatGPT provides general information but does not consider individual circumstances. Users still need to do their own research to determine their eligibility and requirements for the visa and skills assessment process. A skilled migration agent can provide personalised advice and assess eligibility more efficiently, saving time, cost and effort for the applicant.
Check out the Part 2 of this article to find out what other details ChatGPT can get right or wrong, as we put it under further tests and ask it to perform a visa eligibility assessment.
Disclaimer: The contents of this article is based on the writer’s personal opinion and experience of using ChatGPT-3, not ChatGPT-4. The information in this article is intended to serve as a general guide only and should not be relied upon as legal or professional advice. It is advisable to seek the guidance of a registered migration agent for tailored advice that is specific to your individual circumstances. We do not accept any liability for any reliance on the information contained on this website.