Coinbase has recently compared the outcomes of conventional token security evaluations by utilizing ChatGPT to execute automated token assessments using its ERC20 token evaluation framework. The firm says that it will continue to look at ChatGPT even though it did not meet the accuracy level which might have made it obvious that it deserved to be included in the asset assessment process. They intended to evaluate how effectively ChatGPT, developed by OpenAI, would function as a forefront tool used on a large scale as opposed to merely as a one-off code reviewer, given the excitement surrounding its capacity to find security issues. Coinbase revealed this information in its latest blog.
ChatGPT Misclassifies a High-Risk Asset as Low-Risk
The company contrasts ChatGPT’s 20 smart contract risk assessments with those from a human security analysis. 12 times, ChatGPT delivered the same outcome as the human evaluation. However, out of the 8 misses, 5 included ChatGPT misclassifying high-risk assets as low-risk, making it the worst-case scenario because underestimating a risk rating is much more serious than overestimating.
A ChatGPT review is remarkably effective, but there are still several restrictions that reduce the tool’s accuracy. To do a thorough security analysis, ChatGPT is mostly unable of realising when it lacks context. As a result, there are coverage gaps when new dependencies are not examined. Each time, a preliminary analysis would be necessary to check the review for the tool to avoid any coverage gaps.
Second, the application can sometimes be inconsistent; when ChatGPT was asked the same question repeatedly, they didn’t always receive the same response. Moreover, it looked to be impacted by code comments and occasionally seemed to fall back on comments instead of function logic. Lastly, OpenAI keeps updating ChatGPT versions, which results in increased output instability.
Certain instructions that may have formerly produced consistent outcomes may suddenly create diverse outcomes as a consequence of a version change. Timely repair and output quality assurance may be needed to ensure consistent results and prevent any operational flaws.
Coinbase Claims that ChatGPT won’t be used as the Primary Tool
They’ll keep experimenting with novel tools that appear promising as they continue to explore worthwhile methods to integrate AI advancements into the security procedure. ChatGPT will only be utilized as a supporting tool, not as the main tool until an automated or intelligent system can demonstrate the same level of accuracy as their current methods. They are eager to investigate novel techniques that may enhance our capacity to safeguard the development of the crypto economy.